The Rise and Fall of the Chaos Report Figures

This weekend I got an email from X (Chris Verhoef). He noticed I used the research of the Chaos Report and his research in a blog post. Chris and J. Laurenz Eveleens wrote an article in IEEE Software (January and February issue) on the Chaos Report. They stated; “Although the Standish Group’s Chaos reports are often used to indicate problems in application software development project management, the reports contain major flaws.”

They concluded:

  1. Misleading definitions: They’re misleading because they’re solely based on estimation accuracy of cost, time, and functionality. But Standish labels projects as successful or challenged, suggesting much more than deviations from their original estimates.
  2. Unrealistic Rates: The next issue is whether the Standish estimation accuracy definitions are sound. They are not. The Standish Group’s measures are one-sided because they neglect underruns for cost and time and overruns for the amount of functionality.
  3. Perverting Accuracy: The third problem is that steering on the Standish definitions causes large cost and time overestimations (and large functionality underestimations), which perverts rather than improves estimation accuracy.

Chris applied the definitions to data from projects from four organizations. Read the article for the stunning results.

From an Agile perspective it is difficult to define the success of a project on the original estimation of the functionality, time and cost. The scope of the functionality will change. That’s the point of Agile development. So are all Agile projects challenged?  Isn’t  the real measure: the business value?

Not long ago a witnessed the celebration of finishing of  release. A video was shown of the landing on the moon. This project was a success. A few weeks later I got a demo of the application…… Success is in the eyes off the beholder.

3 thoughts on “The Rise and Fall of the Chaos Report Figures”

  1. It's interesting to look at non-software projects through the lense of the report. The Sydney Opera House, one of the two landmarks of Australia, for example, wouldn't be considered a success by the criteria used by the Standish Group. As probably wouldn't most of todays really complex and creative engineering projects.

  2. Ronald Doelen

    Hi Ilja,

    Interesting point, a friend of mine is responsible for projects in utility construction (hospitals, factories). We have found interesting parallels: changing requirements …

  3. There is an interesting Discovery Channel tv series called "Kings of Construction" – the subtitle is "The world's most ambitious engineering projects". The parallels to software development projects are amazing…

Leave a Reply

Your email address will not be published. Required fields are marked *

CommentLuv badge