"The only exhaustive testing there is is so much testing that the tester is exhausted!"
- Attributed to Bill Hetzel
"
Extensive testing has no fixed meaning. To management, and to anyone not versed in testing, ALL testing LOOKS extensive. This is because testing bores the hell out of most people, and even a little of it seems like a lot." - James Bach,
yesterday.
So last week I got exhausted by testing -- but to describe it I need to start with a bit about our test environment at Socialtext.
I test web-based software. The software works on a linux box, and we have two different flavors - a development environment that is unique by userId, and, also, we have the capability to set up and test true appliances in the cloud.
Conceptually, the dev-env is identical to the appliance, but there's always the possibility of something being different, such as when a new package is added to the dev-env but it is not added to the code to install on an appliance.
We also have a desktop application that runs on top of flash and adobe air. This application provides secure micro-blogging, like twitter, and adobe air provides an abstraction layer making it cross-platform, linux/win32/mac.
That's covers the environment; now I want to tell you a testing story. To do that, I'd like to talk about a bug. Now it's easy to read this story and think "Matt is saying his devs stink"; that's not the point. Our developers are
extremely good, but I want to talk about a real problem on software projects. It might only happen once in a blue moon at Socialtext, but we even have this problem here; I want to make an environment where it's safe to talk about it. To do that, I need to tell a story that involves mistakes.
If you can't talk about mistakes, we've got a problem, right? So here goes.
To download the desktop app you need to find it's core location, typically from us, and people with an on-site appliance are not used to going to our website. So we had a story to provide a link in the traditional web-based app allowing the user to download Desktop. Of course, desktop needs flash and air, so the link needs to actually open up a light-box that includes either:
(A) A link to the desktop or,
(B) A link to the runtime to install flash, air, or both.
I needed to test this under IE7, IE6, Safari, and FF3 (4 cases), likely IE8. Because I was dealing with an OS-level runtime, I
wanted to test it under Mac and PC for Safari and FF3. (That's uh, seven combinations, right?)
For each combination, I needed to test:
(A) With no software installed,
(B) With Flash only,
(C) With Air only,
(D) With Flash and air
That's twenty-eight test cases, which painful but possible. I could do a quick installfest and be done, right?
If it worked.
Which it did not.
So, on my first build, I try an appliance, click the link to download and see ... an absolutely blank white screen.
The devs had never tested it on an appliance, and it turned out to be a packaging problem. I waited a busines day or so, got a new build and tried again, on a vz, starting with flash and air removed.
... and immediately got a javascript stack dump.
[caption id="attachment_964" align="alignleft" width="359" caption="Javascript Stack Dump"]
[/caption]
Not good, right?
This time, it looked like the developers hadn't tried to install with both flash and air removed. Maybe they just tested the 'flat' install, I'm not sure. But, again, I mark up the story, tag in "in development", and wait for a new build.
... time passes ...
Now, in between this time, I'm reviewing a presentation by Lanette Creamer in PDF format, and it seems to have Mojibake on my Macintosh, so I install the newest Adobe reader on my VMWare PC within my Mac; I install version 9.3. A new build comes from dev, and I decide to test it on the PC, and -- the lightbox pops up to install flash and air, I install everything, then click again and ... the lightbox comes up again.
Ok, I'll close the browser and try again. Nope.
Ok, I'll reboot the box and try again. Nope, the lightbox still says I need flash and air.
I'm not certain, but I believe the newest Adobe Reader installs one of those two components, causing an installation problem.
Now, again, I'm not trying to beat up on developers. People make mistakes and code has unintended consequences; if it didn't, I wouldn't have a day job. Better developers make mistakes less often -- but even world-class folks are going to have days like this -- likely, less often. The lesson here is not "hire stronger devs" or "testers shouldn't have to put up with stuff like this."
Let's not define process improvement as "other people have to change to make my life easier." Sure, the devs might have room to improve dev practices, but this is a
testing blog, and I'd like to point out a testing problem.
Think about the condition of the team at the end of that story. Yes, the devs were frustrated and wanted the story to move to awaiting signoff. The PM's have deadlines and schedules and wanted it to move.
But me, I was exhausted. I wanted to be done with the dang thing. Done, done, done. C'mon. It's
ONE LINK. It should not be this hard.
To mark it as "passed" not only did I have to the feature a half-dozen times, each test required me to add/remove programs, restart the browsers, and perhaps reboot the box.
Nobody wanted that story to move to awaiting signoff more than I did.
And each time It came back in QA, I 'felt' I knew more about what could go wrong, and re-tested only a logical subset of the app. By the last time, if it had just @$#$% passed, I would have tested it in maybe three browsers and three combinations and signed off.
This is a real problem. Let me give an example from manufacturing:
In his novel
The Majors, W.E.B. Griffin tells the story of the fictionalized "big bad baird", an prototype army helicopter developed in the 1950's and equipped with rockets. The rockets from the baird were purchased from a manufacturing plant that had the opposite problem - the rockets were nearly always correct. In thousands of cases in a row, the inspector found no problems, so, eventually, he just marked them correct and went on to more important cases.
Except in the one case they were not correct. The baird failed in it's first demonstration run, killing it's pilot.
Both of these point to the same problem - tester exhaustion. And, while the first story is a good attention-grabber, the second is more common: The dang software works fine, so we do regression testing for five minutes and call it good. Things are a little better for those of us with a tad bit of Aspergers Syndrome - it doesn't occur to us as often to pencil-whip tests.
I was once thanked for sloging through detailed by-hand testing. It was an odd compliment to me; I hadn't realized I had a choice of just "calling it good" like other folks on the team had. Strictly speaking, I still don't think of that as a valid choice.
Have you ever experienced this? What solutions have you found?
One option that I'm going to bring up is, when a tester reaches the point of exhaustion, give them the ability to switch stories with someone else, to get a fresh set of eyes on the problem. Otherwise, we run the risk of getting a sort of myopia where we miss bugs in order to just
get the dang thing done.
And if you don't think that can happen, you should get out my chapter of
Beautiful Testing sometime. :-)