About STP / 877.257.9531
Log In Join Now





Tuesday March 6th 2012 7pm

Your Time to Shine

Software Test and QA Development Editorial Metrics
Climbing the software testing ladder takes a long time.

Even the 'tricks' which can accelerate you - getting your PhD, working at Google, Microsoft or Facebook, writing a respectable, if not best-selling book - these things take a lot of work.  Without the tricks, you'll have to slog through years, if not decades of testing and community involvement to make your reputation.

Every now and again, though, comes an opportunity to hit the big time right now. 

I'm talking about metrics.

Let me explain.

Every (Good) Software Testing Metrics Post Ever

So I am on a plane, going out to a training engagement with Pete Walen.  Pete is surfing the  twittersphere, and announces that a few people have forwarded on a post about metrics from a global-trotting consultant we both know by reputation.  I point out that I expect it'll be good, but it will probably suffer from the first rule of good metrics posts.

So Pete goes and reads and, and replies that it's a good post, but what is the first rule of good metrics posts, anyway?

I don't have a quick tag line, but the rule comes down to this:  Good metrics articles seem to provide a checklist of what a good metric would be, then don't provide any examples.

This goes along with the hand-wavy answer of the test consultant: They'll provide the checklist, with no answers, then, when pressed, say something like "Well, every context is different.  You know, I can't possible suggest something for your organization without knowing about it." 

So you say that's fine, just give me a case study, a detailed example in context of what someone else did.  And you get back ... nothing.    (Example: Here's video of Lee Copeland, a man I admire, at Google, presenting on metrics.  Didya see it?)

And that's the good ones!

The bad ones will give you an example that ignores nuance, is gameable, rewards the wrong behaviors, and introduces dysfunction.  (If you reward the wrong thing, you are going to get it.  This is the classic problem with counting and rewarding bugs, test cases, etc.  You've seen the Dilbert; I don't need to dwell on it, right?)

That, my friends, is the opportunity:  Provide the community some case studies of metrics programs that don't suck.

No, really, that's it.  I've invested huge amounts of time and life-energy in this, and I have a few ideas, and none of them satisfy all the requirements of a "good [control] metric."

It's harder than you might think. 

Some Caveats

Now wait a minute.  If my boss asks me how many tasks are left and I say "six" and that they should take me "about an hour each" and I expect to be done "by the end of the day", I just used three different numbers to measure and predict, right?  How is that not a metric?

I'm not talking about inquiry metrics; I'm fine with those.  Nor am I talking about greater, holistic measures of the software development process.  One of my main clients uses a Kanban board and measures 'points' delivered per week ('throughput'), the average time it takes to complete a story ('cycle time'), and, to some extent the variation on those numbers.  Those are development numbers, but they are inquiry numbers -- every time they've been used as a goal, things go south rather quickly. 

No, I am talking about control metrics for testing.

In 2008, at STPCon, my colleague Michael Bolton said he knew a lot of 'requirements' for 'good' metrics, yet he knew of no metric - not one - that actually fulfilled that checklist.

I don't know if he still takes that position, but if it's true, it certainly explains the first law of good metrics blog posts.
The Bottom Line

Like I've implied, the real "ask", or request, here are for metrics than can be used to control, evaluate, and predict the testing process.  These will enable "management by spreadsheet", give confidence in estimates, and help evaluate difference ways of doing things in order to decide which approach is more effective.

In that, they sort of conflate issues.  Are we using the metrics to evaluate people?  Teams?  To set goals, or to predict?  As I said before, measurements used as part of a balanced breakfast, to understand what is going on instead of to control it, and especially at the level of the entire development process, I am much more comfortable with.  

I'm talking about numbers used to control the test process.

The more I think about it, the more I am with Tom DeMarco and the less interested I am in control.  At this point, it's just an interesting intellectual exercise - could those control metrics keep their promises at all?

I have some confidence they can not, at least not with any sort of broad applicability.

You want to make your reputation, to give the big keynote speeches and charge the high fees?

Prove me wrong.  Show some examples.  I would find it fascinating.

But I can make it a little easier.

A More Reasonable Challenge

To hit the big time with this one, you don't need to hit it out of the park and fulfill some dream of control.  No, I'd be happy with a few detailed, serious, rigorous case studies of what you did on a project - even inquiry metrics.  No, make that especially inquiry metrics, because I find those the most helpful, valuable, and real.

As a field, we could benefit from more big visible charts and ways to report status.  I have contributed a few, I can contribute more, but the hole in the literature in that space is gaping wide.

If you want to make your mark, here's a place.  If your ideas are good, I'd be happy to point you in the right direction to get it published, or to speak about it, or help refine your writing, or whatever.

Let's be clear: If your ideas are good, you won't need me.  The world will beat down your door when you open your mouth.  Yes, the opportunities are that big if you can deliver.

We might not be able to come up with perfect metrics, but I'm certain we can find more that can have some value to decision makers, used for the right reasons at the right time.

Swerve To The Left

As I was reviewing this post with my friend, Wade Wachs, Wade pointed out two things I happen to totally agree with.  

First, he asked me if other fields with similar levels of complexity have similar problems with metrics, and how they overcome them?  They do; cost accounting in manufacturing produces numbers that are all kinds of problematic.  As another friend recently challenged me, one thing we could do is to create metrics that are good enough to have some value that can be taken with a grain of salt -- just like the CFO knows to use his cost metrics.

Wade's second point was that we need a better way of explaining what we do, and the language executive speak is the language of numbers.  I'd use different words for this, but those of you that know the Story of Biff know that there was a project manager who slipped the project for four months at a time, every four months.  Anyone with a brain in their head had every metric reason to not believe him -- but they did anyway, because they trusted him.

In many settings a request for metrics is a polite way of saying "I don't trust you."  (Sadly, in software, we spent most of the 1990's breaking promises.  In many cases, the customer shouldn't believe us.)

Yet there are many ways to build trust, and metrics are only one of them.  I wish I knew what Biff did, and how he did it, and if we could do that while keeping our integrity.

Thank you, Wade.

So control metrics, inquiry metrics, ways to communicate what we do, ways to build trust.  Pick your path; all of them could be ways to both make your reputation and make your mark in this industry.

The field is green, The opportunity, clear.

Go make your mark.


You must be logged in to comment.
Retrieving Comments...

STPCon Spring 2017

Friend SoftwareTestPro on Facebook
Follow @SoftwareTestPro on Twitter
Create or Join a Crew

Upcoming Virtual Training

New Classes Coming Soon!

Explore STP