Reflections on Software Testing

Testing software for a living is challenging. The habits and traits that make a good tester are well known: the mind of a detective; good attention to detail; tenacity; skepticism; a passion for analysis; curiousity; enjoying seeing things break; understanding how software is designed and coded; etc. Unfortunately, the better you are at testing, the more likely it is you’ll have to deal with the politics and inter-personal issues that arise from being the bearer of bad news.

Software developers often consider testers to be ‘the enemy’ – someone whose job it is to find problems in their code. Most of the software developers I’ve worked with treat software development as some type of artistic endeavour – they pour their heart and soul into their software and don’t take kindly to someone pointing out their mistakes. In my entire career, I’ve come across only a handful of software developers who were professional enough to recognize the value of independent testing.

Working with managers who truly appreciate the value of testing is also rare. I’ve found that most managers think of testing as a ‘necessary evil’. On one hand, they know it is needed, but they can’t shake the feeling that testing is the real reason why the project schedule keeps slipping. Even when presented with clear evidence of an error-prone development environment, they still cling to the belief that testing is the real problem.

Every professional tester I know has at least one horror story dealing with the bearing of the bad news. I was the lead Test Engineer on a mission-critical project to automate very large industrial vehicles. The software development team was manned by an inexperienced group of people who thought they were much better than they actually were, writing very poor quality code. My system test team was performing very well – we were finding lots of defects. This naturally led to a high number of test-fix cycles, which led to schedule slippages, which eventually led to the missing of some key milestones. Can you guess what the Project Manager’s solution to this problem was? Rather than institute some corrective actions to improve quality in the software development team, he chose what he thought was a much simpler solution: he got rid of the entire testing team, opting to let the software developers do the system testing. As a direct result of that myopic decision, the project wallowed for several years longer than originally scheduled.

Over the course of my long career, I’ve occasionally had those ‘aha!’ moments when I’ve discovered some cosmic truth about testing. I got into the habit of writing them down, and now I’d like to share them with you. I’ve summarized them as a series of tips - advice for testers and for management. You may find that some of them have a slightly cynical tone, but keep in mind that every tip is based on a real-world situation I’ve encountered.

Advice to Testers

1. Whenever possible, perform each test more than once. If a test passes the first time, that’s just luck. If it passes the second time, that’s a coincidence. If it passes the third time, then there’s a good chance the thing you’re testing might actually be working properly.
2. Question EVERYTHING. Be skeptical, but not cynical.
3. Don’t refer to software problems as ‘bugs’. Bugs are little creatures that crawl around in the shadows - we humans generally have no control over them. Call them what they are: defects. Software defects do not sneak into software when no one is looking – they were put there by software developers. Everyone involved in a software project (especially software developers) needs to acknowledge this.
   
   [ Historial Note: the first (and possibly last) known real computer bug was found in 1947 – a moth that got stuck in an electrical relay of one of the first primitive mechanical/electrical computers ]
4. Keep your defect reports succinct, factual and impersonal. Describe the behaviour observed and why it isn’t what was expected.
5. Software development is a process consisting of two conflicting tasks: feature development and defect injection.
6. Version 1.0 of anything probably sucks.
7. Assume all software is defective until proven otherwise.
8. A really minor software change can still result in a really major defect.
9. Document all test plans and test results. If you don’t document your testing, it never happened.
10. Software developers are like motorists - most of them think they are above average. [think about that for a minute...]
11. When in doubt as to how to test something, throw lots of data at it.
12. A last-minute change to software will probably introduce a defect, especially if that change was done in a hurry.
13. The probability of a test failing is inversely proportional to the amount of time left before the project deadline. [The last test performed will probably fail, especially if it is performed right before a deadline.]
14. An untested feature will probably have defects.
15. In the presence of a changing software environment, a feature that used to work, but hasn’t been tested in a long time, probably doesn’t work quite so well anymore.
16. Be the quality advocate for the project. In your role as a Tester, you are in a unique position to identify problems and propose solutions that will improve product quality.

Advice to Management

1. A Tester’s job is to evaluate a product and report on its quality. If the product has defects (before or after shipping), remember that they were not put there by the Tester.
2. When evaluating the performance of a Tester, three key areas should be considered: the ability to identify defects; the ability to communicate information about those defects to the relevant parties; and how many defects are reported after testing is complete.
3. Don’t shoot the messenger: a Tester’s job is to find defects. If a Tester is finding lots of defects, that means two things. First, it means that the Tester is doing his job properly. Second, it means that other people are not doing their job properly.
4. In order to test something, a Tester needs to know what that thing is supposed to do. That means there needs to be a complete set of unambiguous, testable requirements.
5. Testing takes time. Often, it takes a long time. The more defects that are found, the longer testing will usually take. If a product is delivered to the Tester at the last minute, the schedule will have to move to the right, or not all the testing will be done (or both!).
6. Planning for testing takes time – sometimes it can take longer than the actual testing effort. But that doesn’t mean the time spent planning for testing is time wasted.
7. To ensure that testing runs efficiently, a Tester should be assigned to a project as soon as the requirements are written. Ideally, a Tester should be part of the requirements review process. This will ensure that all requirements are testable, test planning can occur in parallel with product development, and the Tester gains an early understanding of what the system is supposed to do.
8. Testing software is often an iterative process, something I like to describe as peeling an onion. The onion in Figure 1 represents the software delivered to the testing team. The green core represents the software that actually meets the customer’s needs and has an acceptably low number of defects. The outer, orange layers represent the defects that must be removed before the software can be shipped.

With a real onion, you generally have to remove the layers one at a time. The same is often true with software - some defects hide or block other ones. The first pass (or passes) of testing might not identify all the defects – some of the defects that are found must first be fixed before others will appear. And, like an onion, some layers are thicker than others and require more work before the inner layers are exposed.

About the Author
Bruce Butler (a.k.a. ‘a software developer’s worst nightmare’) has 27 years of experience designing and testing embedded software systems in such diverse areas as mining automation, autonomous underwater vehicles, electronic charting, vessel surveillance, marine navigation and telecommunications. Bruce is registered as a Professional Engineer in British Columbia, Canada, is a Senior Member in the American Society for Quality and is certified by the ASQ as a Software Quality Engineer. When he's not making life miserable for software developers, Bruce likes to participate in ultramarathons and triathlons.