Metric Maladies

Testing Software Metrics Test and QA Tools Trends

Earlier I mentioned the AST board meeting in Madison, Wisconsin, my conversation with Cem Kaner and Lynn McKee, and how I found it extremely helpful. If I had to express Lynn and Cem's ideas in a very few words, it would be that when we talk about test techniques, we run the risk of portraying testing as a commodity skill, where you 'just' slide a specification and some software in over here, go get some coffee, and check back to see bug reports and test reports 'popping' over on the right.

If we really think of testing that way, then it does make sense to outsource to a low-wage country, or to teach a three-day class and give out a certificate.

I turns out that there is more to testing than that. (I know! Crazy!)

Cem referred to those skills - quick attacks, boundaries and equivilance classes - as 'junior tester skills.' These are skills that are obvious and visible. They may give the appearance of being complete, but in fact there are piles of other test skills (like interviewing skills, or scenario-building skills) that more experienced testers may use to figure out the state of the software for an average user.

It turns out those senior tester skills are harder to teach, too. (For those at STPCon, Dawn Haynes talked about this a little bit on Friday during her Rapid Fire Session.)

I am very keen on senior tester skills and how to teach them. If you have ideas or want to talk, drop me a note.

But wait, there's more!

There's at least one other thing that we talked about during the board meeting, and I thought I should blog about here -- software metrics.

Folks who follow my work know that I am not keen on reducing qualitative measures ("how well are we doing?") to a single number - at least, not as a baseline used for comparison, for purposes to evaluating performance or conformance to plan.    I call those control metrics.

Used to figure out what is going on? I use those metrics all the time. Here's an example of two metrics: "We've had three code rollbacks in the past three weeks. What's that about?"

Notice that an inquiry metric could turn into a control metric. For example, if I check the bug system for how many metrics were filed last week vs. the week before, that might be an inquiry metric. If I create a policy to track these over time and post them to the wall, then use the numbers as an input to an annual review process, I've created a control metric.

At the same time, I've likely introduced dysfunction, as people will begin to figure out how I'm using the numbers, then manipulate the numbers to get the reward they want. (You may say "but my team has high integrity!" and that's wonderful, but I say it's some kind of wrong, some kind of unfair, to put people in a system that tempts and rewards them for doing the wrong thing, like gaming a metric.)

Now the hope here is that if we don't have dysfunction - if he haven't perverted the data - that a simple inquiry will normalize -- that if we have 2x the bugs this week than last, then we can use some systems thinking it figure out what that means. I do want to be careful; perhaps the best three testers came back from vacation, etc., but in general, there is some hope that inquiry metrics can do more good than harm here.

Dr. Kaner's Metrics Paper

During the discussion in Madison, I brought up Dr. Kaner's Paper, Software Engineering Metrics: What do They Measure and How do We Know, which is a wonderful discussion of these issues, in some depth, if written for an academic audience. I meant to express my appreciation for the paper.

Dr. Kaner agreed, a bit reluctantly, that the paper was good for it's time, but he brought up a concern. You see, since that paper was introduced in 2004, a good number of consultants have used it as a bit of an axe, or battering ram, to attack the idea of software metrics at all.

"You See!" we've shouted from the rooftops "You See! Software Test Metrics are terrible! Just give up! There is no metric, no metric at all, that is any good for software! Abandon Hope, all ye who enter here!"

Except, well, that isn't what the paper said, really.

What the paper did was lay out some requirements for a good metric, and it pointed out that we don't really have any in the industry yet.

That's not to say that we couldn't have some tomorrow.

If you think about it, many fields have metrics that ain't great. Accounts, for example, use Cost Accounting, a measurement which Eli Goldratt found critical flaws in -- and made his career suggesting alternatives -- In his Classic Book The Goal: A Process of Ongoing Improvement.

Meanwhile, the CFO's suite knows those numbers are problematic. So goes the external auditor, and so does the CEO's office -- yet they are able to use some of those numbers to good effect to manage the business.

In other words, those numbers don't have to be perfect, they need to be good enough -- and, if you have wisdom, observation, good judgement, and common sense, they simply neeed to do more good than harm.

Wouldn't it be nice, if, over the next few decades, we could provide some numbers to the CIO's suite, along with some guidance about them, instead of screaming "No" over and over?

So let's reframe.

On one hand, we could reject all control metrics entirely. Or, we could recognize the weaknesses of the numbers and come up with some advice, rules of thumbs, and numbers that are substantially less terrible than the trivial ones often suggested in the literature.

Talking with Dr. Kaner, I couldn't come up with one paper I knew that existed today that covered that space well, and he couldn't think of one offhand - at least not the realm of quantitative assessment. We know a bit about qualitative assessment, but even that space, in my mind, leaves room for improvement.

I'm not looking for an easy answer today (though links to papers, or examples, are welcome); I suspect this problem may take years for us to make significant progress on.

Instead, I'd just like to say: I'm on the prowl.

If this is an area you have interest in, look me up.

I'll be around.

Next: Collaboration Wins Previous: STPCon Wrap-Up

Comments

You must be logged in to comment.

Retrieving Comments...

Related Items

Tweets You Care About

Seapine (Seapine Software) Check out Search Software Quality's wrap-up of #STPcon http://t.co/jRTJ4vetYF
monkeypunky (nik billings) RT @jerrywelch: Hey @SoftwareTestPro ... Remember that one time ... at #STPCon last week in San Diego? That was pretty cool.
lvcodesmith (Scott Scoble) @BrianKCopeland Thanks for your recommendation to attend #stpcon, three of my team went and it was a great conference!
cowboytesting (Curtis Stuehrenberg) @jitterted I'll be conducting two sessions at #STPCon. Even if they don't overlap I don't think I can swing two weeks off in the same month
rvansteenbergen (Rob van Steenbergen) RT @mheusser: Missed #STPCon? My lightning talk 'the death star was an inside job' is available online - http://t.co/AJxDyjJvy7
MentoraGroup (Mentora) RT @softwaretestpro: Thanks! @MentoraGroup Mentora #STPCon Spring 2013: Recap from the Show! http://t.co/Z6TEY1Vugw @SoftwareTestPro
JA_Harrison ( JeanAnn Harrison) @lanettecream @joolery @lisacrispin @zuzuzka @IrisClasson Haha I went from NH to SD for my hair appt - of course I was there for #STPCon too
cowboytesting (Curtis Stuehrenberg) #STPCon this fall has a water park, a steak house, AND cowboys. Take note #BSCWest. You'd better start bringing it.
Seapine (Seapine Software) Our own @pvarhol delivers his wrap-up of #STPcon: http://t.co/XFt8M7Jv9U

Get Involved

STP Community Forum

Events

Software Test Professionals Conference & Expo

Training

Why Should You Consider Training?

Software Test & Quality Assurance Magazine

Resources

Community

Join an Association…

Services

Publications

Author

Rating

Published