May 24, 2010 When I am asked to review projects, one of the things that I ask to see is the quality plan. Almost always I am handed a test plan, and usually the project manager is very proud of the fact that they have a well-documented test plan. There seems to be a common belief that you can test quality into a project–or perhaps worse, that testing is the same as quality.
So I’d like to make this a bit of a “back to basics” article and just explore quality and testing–and perhaps help you identify some ways that you can improve the way that you manage quality on your own projects.
Quality and Testing
Let’s start off by defining what we are talking about here, at least for the purposes of this article. Google will of course give you more definitions than you could ever want to deal with, but from my perspective we are dealing with the following:
Quality – the way that we ensure that standards are achieved
Testing – the measurement of performance against standards
Both of those use the word “standards”; that’s simply the level that we want to reach. Consider an exam: If the pass mark is 50 out of 100, then that’s the standard. If, on the other hand, the pass mark is 75 out of 100, then that is a higher standard. You have delivered quality if you achieve that standard–in one case, 60 out of 100 is a quality outcome; in the other it isn’t.
You could argue that you should strive for 100. In fact, the parents of school-age children reading this tell their kids that regularly. I won’t argue with that, but in our projects it’s different. Assuming that the standard is appropriate, if we are significantly exceeding the standard then we are unnecessarily spending resources (effort, time, money) that could be diverted elsewhere.
But back to our definitions: Quality is all about the way that we do our work. It’s is about ensuring that the way we do our projects is likely to allow us to achieve the standard. Consider another example–we are driving 100 miles and wish to use no more than 3 gallons of gas for the journey. Think about how you would achieve that–timing your journey to avoid rush hour, avoiding heavy acceleration and braking, etc. These are actions that we plan and then execute in order to help us achieve our goal (the standard). Testing only comes in at the end–how much gas did we actually use and did we achieve the standard?
In projects, the measures are more complex but the principles remain the same. There will be more criteria that we have standards for–performance, accuracy, speed, number of users, uptime, etc.–and as a result there will need to be many more measures taken or tests carried out.
Planning for Quality
Quality standards can come from a number of places–they may be defined organization wide, they may be part of the contract with the customer or they may simply be a series of goals that the project team sets for themselves. Regardless of the source of the standards, you need to plan how to achieve that quality. Your organization should have done a lot of the work for you by providing consistent processes through your PMO that are designed to ensure that the organizational standards are met. You may need to enhance these for your own project if there are additional standards that need to be achieved, and there will likely be project or product-specific standards that need to be achieved, especially when we deal with IT systems. These project specific requirements should be readily available to you from your requirements documentation.
To be successful, these standards need to be understood and planned for right at the start of the project. You can’t achieve a quality target of 100,000 concurrent users if your system design assumed a user base of 1,000. Similarly, you can’t expect to redesign a system to move from 1,000 users to 100,000 if you only provide one day for systems redesign.
This is something that we readily accept in our personal lives: We don’t expect to be able to buy a custom kitchen for $200 and have it ready in half a day, and yet we tend to forget it in our work environment. We want high quality but then don’t budget time or money to achieve it, instead relying on testing to save us (do you really need to “test” the $200 half-day custom kitchen to know that it isn’t what you want?).
The Role of Testing
Testing really is nothing more than determining whether the quality standard has been achieved. In some cases this can be a very black-and-white quantitative measure–is it fast enough, does it support sufficient concurrent users, etc. At other times we are dealing more with shades of gray, like qualitative measures–is the UI intuitive enough, is the documentation clear enough, etc.
When we test we are not looking to measure the quality processes that were put in place by the organization and the project team (that’s a quality audit, and quite different). Instead we are looking to measure the project output to see whether the planning that was done around quality–and the execution of that plan–resulted in a quality product.
Testing needs to be planned for–we need to establish what it is that’s going to be tested, how it is going to be tested and what it is going to be tested against (it’s measured against requirements, but qualitative measures need some kind of measurement scale).
Similarly, testing should only focus where there are quality standards. If there is no requirement for the quality of documentation or training materials, then they shouldn’t be tested. Can that result in problems reaching customers? Yes, but if the project hasn’t identified that as a quality target then it doesn’t matter–we have to assume that the requirements are both accurate and complete.
I often hear of a product not being released because the testing didn’t “pass it”, but to me that’s not a testing decision. Testing merely establishes whether the standard has been achieved; the decision as to whether a product that doesn’t meet the standard should be released is a management one. The answer may sound obvious, but consider the recent Winter Olympics–the weather meant that some of the courses didn’t meet the standard, but the event still went ahead because time was the critical factor.
Testing can Improve Quality
Although the two disciplines of quality and testing are very different, they are also very closely tied together. Testing can identify a number of trends that may help identify weaknesses in the way that we try to achieve quality. In a simple example, if we find that one developer consistently has more bugs in their code than the other developers, then it may identify a training need for that developer (although it may simply be a symptom of them producing more code than anyone else, and the bugs per lines of code may be lower).
Usually things are much more complex than that, but test results are a vital input to root-cause analysis that helps identify the cause of failures to achieve quality (and by extension helps to identify the corrective action that needs to be taken). These failures are usually related to processes or training so we can see that testing becomes a vital input to the continuous improvement process for quality processes.
Conclusion
There’s nothing profound in this article, but I have tried to provide a reminder to one of the cornerstones of project management. In my experience it’s something that is commonly forgotten, causing huge problems on projects. I have seen projects where the bug tracking database recorded thousands of defects and delayed the project by months. The simple fact was that quality had been ignored in the planning and execution of the project and testing had been relied upon to catch the problems. Testing did catch the issues, but at what cost?
Plan for quality in the first place and you won’t need to spend so much time fixing the things that have gone wrong.
Andy Jordan is President of Roffensian Consulting Inc., an Ontario, Canada-based management consulting firm with a comprehensive project management practice. Andy always appreciates feedback and discussion on the issues raised in his articles and can be reached at andy.jordan@roffensian.com.
Copyright © 1999-2010 gantthead.com, Inc. Reproduced by permission of gantthead.com, Inc.
About the Author
Andy Jordan Andy Jordan is President of Roffensian Consulting Inc, a consulting firm in Ontario, Canada specializing in program and project management and PMO development. Andy has led business critical projects in many industries including finance, software and telecommunications, as well as building and leading a number of high profile PMOs. Andy teaches project management and leadership and is a regular author for Gantthead.com where he is frequently the featured author.