Software testing is more complicated, labor intensive, and time consuming than ever before. Applications support multiple platforms, browsers, devices and/or languages and end users are expecting better usability, performance and integration with the applications they use every day. All these factors have multiplied the number of variables that may affect the functionality and performance of the application, making the task of software testing more complicated and time consuming than ever before.
The same forces that drive the demand for these applications also require that these products be delivered to market as quickly as possible—if not sooner. Software development projects that used to release a few times a year are releasing weekly or even daily. QA teams are under increasing pressure to complete their testing in ever-shorter time frames. And, as if this were not enough, these challenges are compounded by hectic development schedules that introduce problems of their own. New features may suddenly appear or disappear, and the design and functionality of product features often change to meet the demands of the marketplace or to accommodate tight release schedules.
As if these challenges were not enough, many testing organizations continue to rely on spreadsheets and documents to manage their testing which add a whole new set of challenges! While spreadsheets are cost effective and require very little training to use, let’s take a look at some of the pitfalls to relying on them.
Challenges of Paper-based Systems
In a typical paper-based testing system all activities are recorded and tracked in static lists, tables, outlines, and matrices created in word processing documents or spreadsheets. These systems are difficult to maintain, update, and track, and as a result testing strategies are necessarily simple and limited in scope because testing teams are straitjacketed from the very start of the process. Poor or inaccessible documentation and inadequate channels for communication make it difficult for QA organizations to adjust to sudden changes in product design. Test plans are frequently based on out-dated requirements documents, and this can lead to test cases that do not adequately test product features and functionality.
Paper-based testing solutions also limit standardization, lack effective security control, increase response time for urgent issues, make it difficult to see the whole picture and finally, they do not scale well as projects grow. We will discuss each of these challenges in detail and then look at how more robust test management solutions can help eliminate these problems.
One of the most often mentioned challenges of utilizing a paper based approach to testing is enforcing standardization. More often than not the rule is “different team, different spreadsheets”.
While teams do strive to standardize on the data that will be tracked and the format that will be used, the lack of meaningful security makes it very difficult. When format changes are needed or additional data points are added, the changes need to be rolled out to each spreadsheet or document individually and there is typically no easy way to ensure that all documents are actually updated correctly, if at all. Furthermore because different teams are working on different project schedules some changes may need to be delayed until a new project begins. Finally, the lack of security also opens up the possibility that changes may be made accidentally or independent of established standards.
The lack of security, or user permission control, makes standardization particularly challenging for organizations utilizing paper-based systems, but poor system security also raises additional challenges. First and foremost, users are normally given all-or-nothing visibility and all-or-nothing access. A user that has permission to edit any part of the spreadsheet can usually edit the entire spreadsheet, and these users can usually even delete items. To make matters worse most paper-based solutions do not have the ability to track who has made a change, when, and what the change was. The lack of user permissions and security makes it very difficult to have a defined testing workflow or a comprehensive audit trail of what activities have taken place.
“Test plans are frequently based on out-dated requirements documents, and this can lead to test cases that do not adequately test product features and functionality.” — Jason Hammon
When used for test execution, documents and spreadsheets do not provide real-time results since results need to be compiled once the individual documents or spreadsheets are completed. Delays between the execution and the compilation of the results can lead to a delay for comprehensive reports and test status updates. When blocking issues or other urgent issues that require immediate attention are discovered within a test cycle, teams must rely on individuals to manually report them to avoid delays.
Seeing the Whole Picture
Another challenge created by paper based solutions is that it is difficult to see the whole picture. As teams develop test coverage to validate the various functional areas and requirements for a product they usually break out those test cases into multiple documents or spreadsheets. Managing these items in separate documents makes sense because the test cases will ultimately be run in a set and quite possibly by a single owner, but this approach makes it nearly impossible to see the overall status of a test project. Results can be viewed fairly easily by functional area but additional spreadsheets are needed to compile all of the results leading to additional delays. And finally, manual and automated tests cannot be viewed together when using paper based systems, and it is often impossible to see cross project results.
Paper-based solutions simply do not scale well. Each of the challenges discussed above becomes more severe as projects become larger and more complex. More testing in a paper-based solution leads to less visibility. There are also practical challenges that come up with larger projects because it becomes difficult to update many items at once. If a product name changes or a major function is revised multiple documents may need to be updated. The manual process of updating the documents may result in missed items or inconsistent updates, and the lack of comprehensive searching ability across all test documents makes it difficult to find all of the items that need to be changed.
Moving Beyond Spreadsheets: Quality Management Benefits
To meet the challenges of delivering more complex software in less time many quality assurance organizations are pursuing several different strategies:
- Beginning their test planning earlier in the development life cycle
- Leveraging the results of previous test efforts
- Improving the management of testing data
- Reusing existing test coverage
Additionally many teams are abandoning paper-based systems and turning to software-based Test Management solutions to organize, plan, and analyze their testing efforts. These software platforms enable businesses to work more intelligently and efficiently, and provide the following key benefits:
Manage Knowledge and Facilitate Communication
A test management solution should provide testers with a centralized and secure tool through which they may review controlled documents and research previously reported bugs. Knowledge collected by the testing group is accessible to all project members at all times.
Enforce the Standardization of Test Cases
A test management solution enables the testing group to define the data they wish to track and to design a standard interface for collecting and tracking that data. Standards increase efficiency and ensure that comparable data is present across products or teams.
Promote the Creation of Reusable Test Cases
A test management solution makes it easy for testing groups to save, manage, and reuse test cases. Test case reuse allows teams to leverage existing coverage across multiple projects and releases.
Respond to Issues More Quickly
A test management solution should provide real-time access to all testing data so that testing groups can immediately respond to critical test blockers and other urgent issues. The faster the team is able to address the critical issues the sooner the test team may resume their testing.
Stay on Track
A test management solution provides QA Managers and Leads with a complete picture of their testing project so they can better manage their project and make the necessary adjustments to meet testing objectives and deadlines. A test management solution can take the guesswork out of scheduling and assigning test cycles, increasing efficiency and raising product quality.
The Proof is in the Results
Whether your organization is using manual testing alone or automating your testing, a quality management solution should present you with the results in one consolidated view. All of your test data should be visible in dashboards as well as in reports that are available to share with your team and other stakeholders.
This article was featured in the December 2012 Issue of Software Test & QA Magazine.
About the Author
Jason Hammons Jason Hammon is the Director or Product Management at TechExcel, Inc, a privately held software company dedicated to developing intelligent solutions that effectively manage the software development lifecycle. Jason has over ten years of experience in software development as well as a diverse mix of creative and analytical experiences in entertainment, marketing and statistical analysis. Prior to joining TechExcel, Jason worked for Commerce One and was a recording artist for Universal Music. Jason received a J.D. from the University of California, Hastings School of Law and holds a bachelor’s degree from California State University East Bay.