[Q] Is Your platform HIPAA compliant? Do you have current healthcare clients in the USA?
[A] Not at the moment. However, we are ISO27001 certified and we leverage the security measures offered by Amazon AWS.
[Q] Can you help me understand how you distinguish between “manual tester” and “tester”? Is a “manual tester” someone who doesn’t use tools? …or a tester who doesn’t write code?
[A] I’m mostly referring to testers that don’t write code. They do use tools for test management, issue management and test specification but not for test automation.
[Q] How do you access applications your clients want to test? Do your systems have direct access to internal company servers?
[A] Our solution runs in the Cloud. The way it works is that for every test execution we spin up a dedicated server on our Amazon platform – on this server we launch specific browsers and use this remote browser to navigate and access the testing application. In order to allow a secure access to internal servers our customers can simply whitelist our IP on their own firewall. Alternatively, we offer a virtual private cloud on Amazon that allows customers to have their own separate test environment for execution as part of our platform and create a VPN tunnel between the virtual private cloud and the customer network. The latter option is considered more secure and satisfies customers in highly regulated industries, such as banking, insurance, financial services etc. We also have an on-prem version where everything runs in the customer’s network, but in this case the customer loses some of the benefits of using a SaaS platform.
[Q] It seems like your tool is customized – how can we do coding? Which languages does it support?
[A] We are targeting mostly manual testers and don’t require any coding skills. However, we do allow our customers to write javascript code and execute it as part of a test on our platform. We also offer “Selenium Bricks” that enables the import of Selenium libraries into our platform and make them accessible to manual tester through the same interface.
[Q] How long is the free trial?
[A] The trial lasts 3 days with an option to extend it if necessary. We kickoff the trial with a discussion about customer infrastructure and testing requirements. We then set the success criteria for the trial with the customer and continue with a short training session. The customers then continue on their own with our support as needed. We assign each customer with a dedicated success manager to make sure they deliver results.
[Q] How do you get around challenges with the application when some parts of it don’t interact with automation tools? Some tools are not sufficient…
[A] Our focus is on supporting web applications on different platforms – this means desktop browsers, of course, but also mobile browsers. We allow customers to write code although we are targeting manual testers. If a customer has coding skills or an engineering team they will be able to write javascript code and can have their testers use that code as part of the test. If there’s an area of the tested application that isn’t supported natively by our platform, adding code assets might help. In addition, Selenium Bricks allows us to import existing Selenium libraries that might offer certain features that aren’t natively available on our platform. We then make the Selenium Bricks accessible through our platform to testers using the same visual UI. So in short, either import Selenium libraries or you can use your own code and apply it on our visual test platform.
[Q] Can we configure TestCraft with different tools such as HP QC, Microsoft TFS, etc.? JIRA?
[A] We do support JIRA for ticketing, Jenkins, TeamCity, Travis, Circle Codechips and, Microsoft Visual Studio for execution triggering. We also support test management solutions such as PractiTest.
[Q] Who will define elements in TestCraft and how will TestCraft detect these elements?
[A] Our platform natively supports the vast majority of elements and actions that might be available on a web application. The platforms identifies the elements automatically as the user is creating tests.
[Q] Are BDD test cases supported?
[A] No, our platform is visual based, rather than BDD based. We believe it’s easier for testers without coding skills to build their test scenarios visually.
[Q] Could you explain the codeless strategy once again?
[A] Sure. Our platform is built as a layer on top of Selenium. Automated test scenarios can be created in one of two ways, both does not require coding knowledge. We then translate those test scenarios into Selenium code. The first way is by a simple drag and drop of test steps on a canvas to create test scenarios. This way enables testers to create test scenarios even before the code is ready (“shift left”). The second way is creating the test scenarios in a “live” mode while the app is running, by simply going through the scenarios. It may appear as a “record replay” solution, but in fact we’re creating a model and not a recording which can be adapted to changes. The codeless strategy is also relevant to maintenance – a great pain point in code-based test automation. We use AI technology that automatically overcomes most changes in the app (more than 80%). If a test does break eventually (“the other 20%), fixing it is very simple – you don’t need to look for the error in the code, fix it and deploy all over again – you just point and click on the correct element to fix it.
[Q] Is there support for codeless Mobile testing and Rest API testing?
[A] We support mobile web testing and have different mobile simulators to run the tests on. We do not currently support native mobile apps. We can incorporate Rest API calls within test
[Q] How can you convince a company who is using Selenium to use TestCraft?
[A] We believe that there are many advantages of using TestCraft, even for those that does not begin automation from scratch and are already using Selenium. First advantage is maintenance. TestCraft dramatically reduces maintenance overhead – using AI technology, we automatically overcome 81% of the changes in the app, most of which would break tests in plain Selenium. In the event of a broken test, fixing it in TestCraft takes a fraction of the time it would take using plain Selenium. It doesn’t require finding the error in the code, fixing it and deploy all over again. Instead, we will point out where the error occurred, and the tester would then need to simply point and click to fix it. Second advantage is ease and speed of building new test scenarios in TestCraft vs. plain Selenium. Third advantage is the reusability of test elements which makes the whole automation process much more productive and maintainable. Forth advantage is the ability to run tests within the TestCraft platform, on different browsers and work environment, without the need to maintain such grids or pay for external test labs. Fifth advantage is “Selenium Bricks” which enables our customers to easily import Selenium libraries created by the Selenium community into the platform.
[Q] Can we use TestCraft to test Hybrid Mobile Apps?
[A] Yes!
[Q] Does TestCraft use Selenium in the background?
[A] Yes, Selenium is at the core of our platform. You can think of our platform as a layer on top of Selenium that allows Selenium to become more accessible to business and manual testers. We are working hard to deliver more of what the Selenium community has to offer using our Selenium Bricks technology, mentioned previously.
[Q] How well does the AI handle dynamic data sets, i.e. e-commerce ticketing platforms where specific performance and seat availability will change frequently? Is it able to accommodate for these situations automatically?
[A] Yes, it does. This is part of what the AI algorithm is accomplishing. It’s there to analyze such dynamic changes and fix the test accordingly, without user intervention.
[Q] Can your solution work on MAM wrapped applications?
[A] It depends – if it’s mobile web apps then yes.
[Q] How would your automation be executed, using what tools (Jenkins)? Any integration with Jenkins, JIRA, Slack?
[A] We have integrations with all the tools you’ve mentioned – you can run the tests and receive their results from any CI/CD tool, including Jenkins. You can open issues in Jira and receive system notifications on Slack. We also integrate with Test Management tools and with Applitools.
[Q] How do you manage “motivation” to move a manual tester to be an automation tester?
[A] Manual testers bring a lot of value to an organization. They are very knowledgeable about the business processes and applications, but in some cases they are concerned about the future. We see some companies trying to take manual testers and turn them into software engineers. I believe it might defeat the purpose as there is no guarantee they will become good engineers after taking coding courses. With a codeless platform they can move to the next level of becoming great automation engineers – they would leverage their knowledge and experience to create excellent, resilient automated tests.
[Q] What verification/review process is available to confirm differences that the machine learning process fixes are correct and valid?
[A] There is a fine threshold over which we break the test and ask for the user to point to the correct element. Our sophisticated set of AI algorithms is able to ascertain what that threshold should be in order to assure there are (almost) no false positives, while keeping the number of broken tests to a minimum. So, the verification process you have asked about is done automatically by the platform itself with great success.
[Q] How do you decide what to automate first? Defects or functional tests?
[A] It depends. We don’t do the automation ourselves. We allow our users to automate on their own. I would say you should start with functional testing – but it depends on the specific tester and application.
[Q] Can this tool talk to Defect Management Tools?
[A] Yes, we are integrated with Jira. If you are using a different tool – we can integrate with it as well. See answer number 18 for more info.
[Q] Is there any functionality in Selenium that is not in TestCraft?
[A] I’m sure there is. Selenium has a vibrant community around it that keeps adding new functionality to it. We can make new functionality available with our recently announced Selenium Bricks technology – it allows you to take any Selenium library and make it accessible to manual testers through our platform. If a customer has a specific need that is currently not supported we will work with them to identify and implement the right Selenium Brick. We foresee a marketplace for Selenium Bricks being developed with time.
[Q] Can the test scripts be kept in the internal company repository for proper versioning? (We use very strict instance of Gigiscloud)
[A] At the moment no. Just to explain how it works – we do have a versioning option on our platform so you can manage branches and versions but you can’t integrate with external Git platforms or other version management tools. We do however allow a customer to download the Selenium code that is generated by our platform which is very important for continuity – if any of our customers decides to step away from our platform for some reason, they can rest assure that their investment won’t go to waste. It’s not our intention to lose them but, this is something we offer that is unique.
[Q] The new automation tester only records the scripts? Do you only use Selenium RC?
[A] We aren’t recording the script, recorders in general are limited. We offer the user experience of a recording, so our users can go through an application and point-and-click to elements on the screen and have them captured as single actions. We then build those single elements into a model on a virtual canvas that can be very easily edited and maintained.
[Q] Does your platform have API testing capabilities?
[A] Not at the moment. This is something we are working on right now.
[Q] Does the platform integrate with continuous software development platforms like Jenkins/TFS? If so, what platforms do you support?
[A] Yes, we support many different CI platforms and if we are missing one we can easily add it. We have integrations with Jenkins, TeamCity, Travis, Circle Codechips and Microsoft Visual Studio.
[Q] Does the tool support desktop applications and Sabre (blue screen)?
[A] No
[Q] What skills should a manual tester possess in order to use TestCraft?
[A] We believe that testers should know the app they are testing inside and out and learn as much as possible about testing techniques. We don’t believe that knowing how to work with a specific tool/ technology is important. There is no need to know code.
[Q] Some testers have compared TestCraft to record and playback. Please respond to this view.
[A] TestCraft can be sometimes mistaken for a record-replay because of its ease of use and because the “live” way of creating test scenarios (while the app is running) may appear similar to record-replay. However, it is completely different because: First, we create a model, not a recording. In case the app changed there is no need to record all over again, from scratch. All you need to do is simply add/ delete or bind the changed element by a simple click. In most cases you won’t even need to do that, as our AI technology will automatically overcome the changes in the app. Second, we have our own internal lab which you can use to run the tests on different browsers and work environments and upon shceudle. Third, our visual way of creating test scenarios enables testers to create test scenarios even before the app is ready – shift left. You can start testing earlier in the dev process which will lead to major savings and increase productivity – you can’t do that with a recording.
[Q] The biggest problem with Selenium is when programmers change the name or ID of an element. How does TestCraft handle this problem?
[A] Our sophisticated AI algorithms identify such changes and by “considering” many different attributes, both technical and contextual, are able to fix such tests automatically without breaking (for most changes to elements).
[Q] What are you using for multi platform validation? What are you running the mobile OS on (simulation or actual devices?)
[A] We have our own browser lab that supports the common browsers such as Chrome, FireFox and Internet Explorer as well as mobile simulators. We integrate with external lab solutions and allow customers to execute test from our platform on such labs, which include real mobile devices.
[Q] What does your reporting look like?
[A] We have a dashboard that displays information about the state of testing in an organization this includes, number of tests, execution details, information about test creation and execution, etc. This will allow you to drill down to a specific project or user level. All of the executions are being stored and will allow customers to browse through them. The results of every execution will include every type of platform execution it’s working on, details on each of the flows in a suite, if there is a success or a failure, and screenshots of every step of the way. This information can also be exported to external tools such as CI platforms, test management platforms, etc. This is how we become an integral part of the CI and DevOps model.
[Q] Could you automate 100% of an app?
[A] You can, but we don’t believe it’s good testing. You are welcome to watch our webinar about “How to become better at test automation” to better understand why and how automation should be done.
Dror Todress – Co-Founder & CEO, TestCraft