'Proving' The Embedded Software Test Team

Software Testing

Recently on the Agile-Testing list, Michael Heutterman asked about arguments for embedded testers in with developers. By that I mean having the testers, and developers, all the technical folk, really, on one single project team, instead of having a "dev team" and a "test team", when assembling some from each group to create a "project team", which you pull apart when the project ends.

Mr. Heutterman had just finished a book "DevOps for Developers", where he made the claim that his "whole team" approach was more effective -- and he was looking for statistics to back up the claim.

The reply to his request was underwhelming. On the internet, you can't really hear crickets chirp or a pin drop, but if you could, you would have -- there were no solid responses.

've got plenty of /qualitative/, /systems-thinking/ answers, and experiences to

share.

But the author asked for hard numbers.

I am afraid that I do not have a lot of quantitative numbers, as most software

engineering metrics lack construct validity. (Or, in different words, "All bugs

are not created equal.")

Two arguments I have for embedded teams:

1) Tracking the amount of time testers spent blocked on different clients over a

period of years, I see a correlation between tester-is-expensive-paperweight

waiting for a build and external test teams. Of course, there are too many

independent variables to track here; the improved teams didn't just embed the

testers, they also generally moved toward one-piece flow, pair programming, etc.

2) Tracking the number of handoffs that occur, from dev-to-test-to-dev. When

people sit next to each other, it is easier to show the dev your test

methodology, so they will poke test the app before handing it back - not just

fix the first blocking bug you found as a tester. Also, allowing

mentions-in-passing and desk pairing increses the chance the bug will get

fixed-fixed on the first go round.

3) Tracking the costs of hand-offs. If testers have to file a bug report that

needs to get triaged before it gets fixed, then costs increase over a

conversation and a fix. The most project management layers on top of this, the

more the fixes cost.

That said, I'm sorry, I don't have ROI numbers "FIVE HUNDRED PERCENT

IMPROVEMENT!"

My ROI numbers, working with clients, have been so obvious that we didn't need a

spreadsheet to tell us that there was an improvement.

I /do/ think these examples /are/ generally plan-do-check-act, in the Deming

sense, and I'm willing to share them in more detail. Dr. Deming, the metrics

man himself, is known of the claim that often the most important "metrics" are

unknown or unknow-able. (See his wikipedia page, or

http://mikeneiss.wordpress.com/2009/04/30/unknown-and-unknowable/ )

Lisa Crispin asked why the no responses, and I stepped in with this:

I have plenty of /qualitative/, /systems-thinking/ answers, and experiences to share, but the author asked for hard numbers.

I am afraid that I do not have a lot of quantitative data, as most software engineering metrics lack construct validity. (Or, in different words, "All bugs are not created equal.")

Three arguments I have for embedded teams:

1) Track the amount of time testers spent blocked on different clients over a period of years, I see a correlation between tester-is-expensive-paperweight waiting for a build and external test teams. Of course, there are too many independent variables to track here; the improved teams didn't just embed the testers, they also generally moved toward one-piece flow, pair programming, etc.

2) Track the number of handoffs that occur, from dev-to-test-to-dev. When people sit next to each other, it is easier to show the dev your test methods, so the programm can 'poke test' the app before handing it back - not just fix the first blocking bug you found as a tester. Also, allowing mentions-in-passing and desk pairing increses the chance the bug will get fixed-fixed on the first go round.

3) Track the costs of hand-offs. If testers have to file a bug report that needs to get triaged before it gets fixed, then costs increase over a conversation and a fix. The most project management layers on top of this, the more the fixes cost.

That said, I'm sorry, I don't have ROI numbers. No "Five hundred percent improvement" claims in big bold letters with many asterisks here.

My ROI numbers, working with clients, have been so obvious that we didn't need a spreadsheet to tell us that there was an improvement.

I do think these examples are generally plan-do-check-act, in the Deming sense, and I'm willing to share them in more detail. Dr. Deming, the metrics man himself, is known of the claim that often the most important "metrics" are unknown or unknow-able.

I think that was a pretty good post. Yet a few hours after I clicked submit, I had a few things to add, and I thought this audience might appreciate it.

Here are some more thoughts:

1) When people ask for data, they rarely actually want the data. On one recent panel, an expert asked if anyone had actually been successful using Kanban to increase predictability in software delivery. I said that yes, and I could point him to some case studies.

Things went downhill from there.

I would have been better off to say "yes, at my current client, and Kathy, from the technical staff, is right over there." (She really was too. Man. Not my finest hour.)

Beyond that, the request for data is often an attempt to find a reason to reject a claim.

The important thing here is that the person has already made up their mind. Like asking an would-be-entreprenuer for a business plan, it is a polite way to look interested without having to take any action.

Sometimes the request is sincere - but be careful for where it is not, otherwise you wind up having unproductive conversations at best, and, possibly, damaged reputations or relationships.

2) Data does not exist in a vacuum.

Anytime anyone says "Team X changed from doing process X to process Y, and (measurement) went up (percentage!)" it is an appeal to dig into details.

Example: Team moves from multiple kinds of stories to only one and two point stories and throughput goes up 40%!

Okay, what else changed?

In this case, I would ask "what is a point? How do you know that a single point story six months ago is the same as a single-point story today?"

Sometimes, you get lucky. The team invites you over to lunch to look at the cards and see how it's done. I recently got invited to visit ETsy, in Brooklyn, and just two weeks ago, the technical staff at Zappos, just because I asked.

Sometimes, they don't. The response to you inquiry is "don't you trust us?", or a personal attack. Sometimes it is nothing at all.

Once again, there are several ways to make a case, and numbers are one of them. I'm just advising caution.

Next Time

As for next week, we could dig further into ways to argue, drawing examples from a pre-recorded software testing debate (no, really, these exist), or I could get back into examples of finding real bugs in software -- let me know in the comments, okay folks?

More to come.