Hang around software development or testing long enough, read enough blogs and books, and sooner or later, you are likely to hear the great crisis talk
. It typically goes something like this:
(A) Software Development is in a state of crisis!
(B) (Quote the Chaos Report)
(C) Something must be done!
(D) Therefore - whatever the presenter is selling
The presenter could be selling anything - from process improvement, to 'getting the requirements right', to 'prevention', or 'inspection' - it doesn't really matter.
The point is, software development is in a state of crisis and we must be fix it!
Wait a minute. Let's hold our horses. What is this Chaos report, exactly?
It turns out that a business think-tank called the Standish group conducts a series of surveys to determine the overall state of the health of IT; the 1995 report is available on-line free
. At the very least, it is an interesting read.
The Standish's report (the "Chaos Report") takes IT projects and divides them into three buckets:
Three Project Outcomes
Resolution Type 1, or project success: The project is completed on-time and on-budget, with all features and functions as initially specified.
Resolution Type 2, or project challenged: The project is completed and operational but over-budget, over the time estimate, and offers fewer features and functions than originally specified.
Resolution Type 3, or project impaired: The project is canceled at some point during the development cycle.
Three buckets. Really?
Imagine, for a moment, we used the same criteria to evaluate movies. Did you know that the movie Titanic
had it's original release date of June 1995 pushed back to December? Under these definitions it would be "challenged", even though it was the single largest grossing film in the history of cinema
Forget Titanic; Star Wars
was six months late! That's right kids; the single largest iconic geek movie of the 20th century - the one that launched five sequels, countless action figures, lunch boxes, breakfast cereals, toy light sabres that make whoosing sounds and it's own version of "Monolopy" - should be caulked up as a failure. (It did run $3 million over budget
, after all, costing $11 million instead of $8)
Oh, bull pucky.
Even worse than late or over-budget, the Chaos report considers projects that were "canceled early" a failure, when in my career I have worked on projects that delivered 90% of the value in 50% of the allotted time. Likewise, some projects should
be canceled if the competitive environment changes and the project becomes irrelevant.
What all this means is the report doesn't allow for the possibility of change - positive or negative. Instead of measuring the execution of the software team, it actually measures the ability of the estimators at the beginning of the project
; not only on their estimation, but also on their ability to predict the future
(changes to the environment) that will happen before the project finishes.
Pictured this way, the 'shocking' and 'abysmal' 'failure' rates actually mean that, as an industry, we aren't so good at predicting the future.
I don't think this is all that shocking, nor would I call it a crisis.
Where did we go wrong?
The biggest problem I see is that the chaos report is only looking at a single dimension
- "did the project hit the deadline" instead of asking a more reasonable question, like, for example "did the delivered software have a good return on investment?"
Or, perhaps, one single question just isn't enough. It's like asking "what's your weight" to someone; it's hard to draw any conclusions without knowing their height. Even then, a weightlifter can come off "overweight", you need body fat index to really tell. And that might not be healthy; you'd need to know cholesterol
.. you get the point. Drawing conclusions with a single dimension can be dangerous.
How can we do it you one better? Well, the usual. Develop in small chunks. Adapt to change. Plan to re-plan. Think while we do. Limit our work in progress, so that if a new opportunity arises, we have to throw away less work. Plan phase gates that release working software - not documents that may or may not be useful.
And, as testers, try not to be fooled by happy-talk - or even crisis-talk.
Meanwhile if you want to build a new framework to analyze projects in our industry - one with depth, that, to paraphrase Gandhi, is the change we want to see in the world - please, drop me a line; I'd be interested.