Test consultant Rex Black and Scrum Master Bob Galen have been at odds lately about agile methods and their effect on software testing. Black claims to spend much of his time helping clients overcome the new challenges they encounter “when using agile methods, typically Scrum.” Galen calls this view “typical of traditional testing consultants.” Both have more than two decades of industry experience.
A Certified Scrum Master Practicing since 2004, Galen says that Black’s perspective could not have been “gathered from the perspective of an actual agile tester, one with serious experience working in mature and high performing agile teams.” Black admits that some of what he sees may have arisen from “improper understanding or implementation of the chosen methodology,” and that not all the challenges are inherent in agile methodologies.
Still, the two disagree on many agile-related issues. Take unit testing, a cornerstone of the agile’s code a little, test a little philosophy. According to Black, the practice often falls short of the mark. “Unit testing has limited bug-finding utility. In research published in CrossTalk, the Journal of Defense Software Engineering, author Capers Jones found unit testing to be only 25 to 30 percent effective at finding and removing defects.” RBCS assessments, have shown system testing, on the other hand, to be about 85 percent effective, he added. “Organizations that care about quality should [perform] unit testing and system testing.”
“We need to stop quoting Capers Jones as I suspect his data and studies do not reflect the dynamics of mature agile teams,” counters Galen. “Agile methods stress automated testing of all kinds—not simply unit tests. No agile team to my knowledge views unit tests as the end-all of testing.”
“While unit testing is often thought of as the developer’s job, I have found that many programmers just don’t do it or deliver cursory attempts,” continues Black. Such sloth often leads to complications for the test team in its role as “bug finder of last resort,” he says, with hardships further compounded by Scrum’s short test execution periods. “This can lead to code churn during testing, since so much code must change to fix the bugs.” This much change, Black concludes, ultimately outstrips even the best automated regression testing system to keep up “[This] leads to lower defect detection effectiveness for the test team.”
Galen addresses the failure of developers’ to deliver unit tests by clearly defining their role. “I expect the testers to pair with the developers to improve these practices, and I expect developers to deliver solid unit tests before they can be considered ‘done.’ In fact, he adds that as part of delivering “done” software increments, the team must decide on the set of tests, which might include unit, functional, acceptance, non-functional, and regression tests. “[These] should be run within the Sprint in order to meet a customer-ready deliverable. Every team member should be defining these testing tasks, with the testers taking the lead.”
About the Author
Edward J Correia