Yesterday night a bunch of local testers (and a couple programmers) met in Grand Rapids to talk about testing. Pete Walen
led a discussion on test process improvement. There was much beer and chicken wings to be had, and it was good.
First off, Pete's talk put the business environment and situation
into account - that a team do an analysis of it's strength and
weaknesses, along with the environment's opportunities and threats.
Then instead of some grand plan, you ask "what should we change next",
and iterate. At least, that's my take on the core elements of the
What I really enjoyed, though, was the free-flowing narrative of the discussion.
There was no powerpoint; we did not look at handouts.
Pete just talked. He was interrupted a lot. Some of the interrupts
were me; some of them were bringing out the main course or getting a
refill. We explored some tangents, but slowly brought things back to
the main points.
Overall, it was more a sharing session than a powerpoint-driven-lecture.
It was /great
/. (If you'd like to hear it from his own words, check it out, Pete will be presenting the talk
at STPCon 2011 in March in Nashville!)
... but I digress. I was going to tell the story of Bungalow Bill.
So picture it, West Michigan, years and years ago - back when there was
no grey in my hair and I was a software developer for a satellite office
for a fortune-500 company. At the time, our little office was going
through one of those classic transitions where we had decided to spool
up a testing team. (Or at least create a full-time tester role who would
manage the ad-hoc team created during crunch time on projects. Yes,
this was back in the days when 'agile' was always lower-case and meant
So we had our tester, who i'll call Bunaglow Bill. Now it's important
to note that I'm not picking on Bill -- he is an extremely talented man,
who came out of tech support. In tech support he quickly rose to be
the guy that everyone turned to when they ran out of ideas.
Not only did they turn to him, but he solved the problem. It might take
him two hours with a customer, but he'd figure out what was wrong and
get it fixed. As a result, his metrics for calls resolved were lower
than everyone else, but the whole team knew he was the best. (Come to
think of it, that whole metrics thing is a different post.)
The new management team want to promote Bill but, sadly, they didn't
really have a place to put him. Tech Support had a management team, and
Bill wanted a new challenge. So they created a tester job for him.
Now Bill didn't know much about "testing" per se, but he was curious and
good tinker-er. Over time, he learned about testing and became
excellent, but the start was ... rocky.
Especially when you got beyond Bill's expertise, which was in our Bill-of-Materials software and AutoCad.
So, of course, I'm the new guy, and one of my first assignments as a programmer is to create some software to take an export from several different systems (outlook, outlook express, lotus notes, and a few more) and import them into our software.
It was pretty standard text file manipulation, no big deal, and I wrote the code, fired up the systems, filled in the contact manager, exported, and tested the files.
Then I passed the code to Bill to test, along with a note "Matt doesn't have a great deal of confidence in this -- could you really put the app through it's paces?"
... and got an email back "Sure. Please email me some text files to test wait."
Wait ... what?
Here's the problem: By replicating the exact same input conditions that Matt tested with, Bill would be doing the exact same test -- adding no value to the process. I (Matt) wanted him to explore different combinations, like leaving last name blank, or having long names, or special characters, or multiple fields in the street address field ... the combinations were limitless.
... And Bill didn't think to try anything but exactly the same things Matt was doing. (To be fair, he might have tried it on different versions of the windows operating systems, but the risk that it would succeed on Windows98 and fail on Windows95 was pretty slim.)
The reason I thought of Bill was because of a term that came up at the User's Group last night -- "Happy Path Testing", the first level of the three-step model.
Three Levels of Testing
Probably the best one-parapgraph explanation I know of comes from a gentleman named Lee Copeland, who also has a book on coming up with test ideas
. Lee said that when we get our first job and the boss says "test this", we take the simple, happy path. We run it once. We give the software middle-of-the-road input, see success, and tell the boss "okay, I tested it."
Bill was doing this sort of testing -- testing "the happy path."
After the happy path, when we begin to see ourselves as professional software "breakers", we may get really excited about making the application fall over. We may be able to "break" anything, do some crazy stuff, and cackle with glee as the webserver crashes. That's the second level, and, at least in terms of finding defects quickly, it's an improvement over the first.
In Lee's words, the third level is "where we really start to think about and talk about risk." To add a bit to that, in my words, that means thinking in terms of business value, the risk of failure for a test idea, and the consequences if it did fail. It might mean triaging our tests to only run the most important ones, or taking a hard look at the schedule impact of various kinds of testing -- to seek the best bang for the buck.
In the years that passed since then, both Bill and I have continued to learn and grow professionally, and I'm pleased to report that one interesting anecdote does not make a pattern or color Bill's spectacular career.
It's a neat little story, but the only reason I thought of it was because of the fun and fellowship that drew it out.
I hope you have a place to gather, at least every couple of months, to talk about testing with other peers in a safe environment. To pause, reflect, maybe to remember. If you don't, you might want to think about starting one -- it can make a huge difference.
Just an idea.