To effectively implement an automation program, the same approach and similar effort is needed when building complex software. You will be integrating a system with many parts as well as software (i.e. scripts and glue software). Additionally, to complicate things the automation software will be interfacing with another application (application under test). Therefore the same lessons learned in building systems and software should be applied to your automation project.
Goals and Objectives
It is vital to define the high level goals of the automation system. The most important
questions to answer are:
- What type of application features are we trying to test – is it performance testing,
unit testing, or functionality testing? - What is the extent of automation that you are attempting and what is the business need?
- How do you propose to support this automation once implemented?
Once these questions are answered the next set of items to be identified are the features
you require in the automation system. These include:
- Logging capabilities
- Graphical automation scenario creator
- API for remote procedure calls
- Reporting
- Resource management of test/lab resources
- Concurrent scripts
- Capability to prepare or “clean up” the test environment
- Versioning of test scripts
- Documentation for administration/test script creation
Designing and Documenting Automation Architecture
Once these items are identified then you can design the automation framework. The automation framework lays the foundation and rules for the automation toolset. For example, a shared software library for automation scripts should be implemented and used by all automation script writers. Similar points are listed below:
- Extensibility: Automation requirements will grow in step with the requirements of the software under test, and the automation system must be able to cope with that growth.
- Instrumentation for debugging purposes: Running scripts is the easy part. Debugging scripts is what takes the most time. There should be a system in place that quickly summarizes all the problems from a test execution run.
- Scripts should be modularized.
- Maintaining statistics of executions: Identify test cases with a high false positive rate and either fix the root cause or remove from your executions. Conversely, identify tests that find real software defects.
- Automation scripts re-usability: Could scripts developed for feature testing later be re-used for integration testing or performance testing. Could other groups re-use the scripts?
Tool Selection
Once you have the automation framework defined then pick a tool to develop the automation framework. Open source, commercial or in-house tools are your typical options based on whether the tool fits current and future requirements. Other factors in tool selection include:
- Support requirements to maintain the application
- Average time required for a feature request or bug fix to be implemented
- If open source, is the community active in developing features and fixing defects?
- If the software is home grown do you have developers in-house to support the product?
Resource Management
Full-time resources must be qualified and fully committed to building and maintaining an effective automation program. When staffing up for an automation project create a list of job requirements for the new team members. If the focus of the team is divided between multiple complicated projects, it will reduce the chances of success of the automation program.
Automation programs contribute substantially to reducing the cost of testing and improving, but it is critical to plan them carefully with the same rigor as you would in software development.
About the Author
Sreekanth Singaraju Sreekanth Singaraju has more than 12 years of senior technology leadership experience and leads Alliance’s QA & Testing organization in developing cutting edge solutions. He is a respected technology thought leader with focus in testing Agile projects, test process automation and metrics-based development approaches and utilizing lean techniques in software development.
Sreekanth has helped design and architect a Robotium-based test automation framework that integrates with enterprise test automation frameworks to provide comprehensive test coverage along with the test deployment process to create a lights-out lab for mobile app testing.