Some time ago I was confronted with an assertion made by a project manager that testing was costing the project far too much. I asked what she meant by that. The response was along the lines of, “We have all these test cases and test sets, time has been spent building test scripts that can be run and the results reviewed and you guys are doing a bunch of testing. But the project is over budget and testing is costing too much.”

I asked if the test group for the project was over its estimate. “No, not yet, but everyone else was and you aren’t done so I think you will be over budget, too. We can’t afford that.”

Recently I encountered a similar conversation. The test group was reviewing requirements and kept asking questions and was working with the designers and developers to make sure both groups had the same understandings. In some cases we did, in others we did not. So we each asked questions around the documented requirements. Something similar happened.

The test group was using more of its estimated (which resulted in budgeted) time for reviewing requirements than it was supposed to.

In both cases, the end of the conversation was the same: testing costs too much.

While volunteering at an event recently, I struck up a conversation with a fellow volunteer. In the course of our conversation, we found we were both involved in software. He was a developer/ designer who had transitioned into project management and was working as an independent consultant for firms whose projects tended to get into trouble. He asks hard questions and is unafraid to deliver unpopular messages to management about the projects and methodologies used. I asked him about these conversations and how similar they were.

His response was interesting. He said, “Testing is at the end of the project (or iteration or release). That is where the mistakes show up so they are the ones who usually get blamed for project delays and cost overruns. Other people (groups and departments) have cost overruns, take longer than expected and generally spend more of the budget than was originally allocated for them. They shift these costs down the road. Testing does not have anyone to shift the costs, and therefore the blame, too. They are the end of the line.”

The fact that this lined up with my general views on reducing the cost of testing made me smile. He also said that most project managers who work for an organization as a full-time, regular employee will not deliver that kind of a message because the emotions and politics around the ideas expressed are too volatile. The repercussions will be more than they are willing to accept.

Testing and Software Development
Many learned and wise people will explain the true nature around the cost of testing or, what I think of as the management’s view of the same question, the ROI for testing. There are models and formulas that can explain many things involved in the cost and benefits of testing. Some colleagues, whose views I respect, have long presentations and essays on the opportunity cost and other ideas around the cost of testing.

Let us consider how software is developed. By that, I mean generally what needs to be done, not how we specifically do it. We consider what is wanted, or what is believed to be wanted, by the clients and customers.

We consider the requirements we need to fulfill in order to deliver what is wanted. We evaluate these requirements and the intent of the effort and create a design. We then write the code and in turn test the code. The exact sequence of these activities and how they are done can vary by organization and methodology and I realize that other activities have been left out, and in some environments, many of these activities are happening very close to one another, if not exactly at the same time.

There has been a small voice in the back of my mind for some time, starting with the first time I heard the assertion that we needed to reduce the cost of testing. In fact, when I heard that assertion the very first time, I naively presumed that there were similar efforts around other aspects of software development. At that company, it was not the case.

Reducing the Cost or Working Better?
Come to think of it, I rarely hear references to reducing the cost of project management or requirements definition. There is a great deal of material on doing those things better. Still, doing them better is not an explicit directive to reduce the cost of doing them. Similarly, doing testing better does not explicitly mean reducing the cost. Doing testing, or anything else, better means to do it better.

If we, as software development professionals, are not looking to improve all aspects of our profession and craft, then we are doing ourselves and fellow craftsmen a huge disservice. All of us need to find ways to do things better.

There may be an implication that doing specific aspects within software development will, by definition, reduce the cost of that activity. However, most credible writing I have found does not support that reasoning. Instead, I’d suggest that the improvement in a specific area may actually increase the cost of that activity. Likewise, I suggest that an activity done better will have an impact on the cost of activities done later in the process.

If we do a better job describing and defining business needs and the purpose behind the effort, will we not do a better job with the requirements discovery and definition? If we do a better, more thorough, perhaps more thoughtful job on requirements, would we not expect the design process to be smoother? If we follow this to its logical conclusion, doing a better on each of these tasks, will reduce the cost of testing.

The greatest single cost in testing, in my experience, is reconciling ambiguities in the software product.

Reducing the Cost of Testing
People make presumptions every day. The problem for software projects in general, and testing in particular, is that if there is the opportunity for multiple interpretations around an idea, that opportunity will be maximized.

These ambiguities, these opportunities, begin to manifest themselves at the very beginning of the effort. Unclear business needs or intent lead to imprecise or highly variable requirements which leads to designs that do not address the business need and result in software with many defects that are time-consuming, hence costly, to completely investigate, advocate, correct then re-exercise.

Still, we do not see calls for reducing the cost of requirements definition or software design. Likewise suggesting that we need an ROI on software coding seems painfully obtuse. It is precisely in those areas where the true reduction in the cost of testing can be achieved.

The benefits of testing, the point of it and the true return for a software shop making commercial software, is you stay in business. People buy your product and continue to do so. Customers buy the latest version because it makes their life better, not because the previous version is no longer supported or will not run on their new platform. How one measures that, I do not know. Staying in business and not being sued for criminal negligence from your faulty software has a value beyond a simple first year finance class exercise can calculate.

Expensive and slow testing is a symptom of other problems and other costs that were not paid earlier in the project.

I challenge managers of all levels, and testers of all variety to carefully review materials they see on reducing the cost of testing and calculating the testing return on investment. Some material is very good and needs to be considered carefully. If you find your team is doing good work effectively, there may be little fat to cut in the area of software testing when called to reduce the cost of testing. Look to cost savings around testing in unclear or ambiguous requirements, poor design, and other shoddy work done earlier in the process. If you look to these things, the cost of testing may just take care of itself.


About the Author

Peter Walen Pete Walen has been in software development for over 25 years. After working many years as a programmer, he moved to software testing and QA. Following a brief foray in Project Management and Business Analysis, he returned to software testing. He has worked in the fields of Insurance and Finance, Manufacturing, Higher Education/Universities, Retail, Distribution and Point Of Sale Systems. Pete is an active member of several testing associations and an active blogger on software testing.