The Chaos Report
© The Standish Group 1995
Terminology
• Large Company: more than $500 million/year
in revenue.
• Medium Company: between $200 and $500
million/year in revenue.
• Small Company: between $100 and $200
million/year in revenue.
Why do we care?
• In the United States - more than $250 billion
spent each year on approximately 175,000
projects.
• Avg Cost large company is $2,322,000
• Medium company, it is $1,331,000
• small company, it is $434,000.
That wouldn’t be bad but...
• Estimates that in 1995 American companies
and government agencies will spend $81
billion for cancelled software projects
• As well as an additional $59 billion for
software projects that exceed their original
time estimates.
What was found:
• A staggering 31.1% of projects will be
cancelled before they ever get completed.
• On success side, avg. is 16.2% for on time +
budget. (Worse for large company’s, it’s only
9%)
• About 52.7% of projects will cost 189% of
their original estimates.
• Risk may be higher for newer project but
many are mundane.
Example
• City of Denver – failure to produce reliable
software to handle baggage at the new
Denver airport is costing the city $1.1 million
per month.
• Originally scheduled for October 31, 1993
ended on February 28, 1995.
• Originally single system for all three
concourses, was actually a separate one for
each concourse.
How was the report created?
• Across industry (e.g. banking, securities,... etc)
• Respondents were IT executive managers.
• Across corporation size. (Small, medium and
large companies).
• Total sample size was 365, and represented
8,380 applications (or projects).
How was the report created? Cont.
• Resolution Type 1, - Success : On-time, on-budget
with all features and functions initially specified.
(16.2%) (lowest in Large companies)
• Resolution Type 2, - Challenged: Completed and
operational, but over-budget, over time, and/or
fewer features/functions. (52.7%) (largest in
Large companies)
• Resolution Type 3 – Impaired: The project was
cancelled at some point during development
cycle. (31.1%) (largest in medium companies)
Reasons why projects are not Type 1:
• Restarts: For every 100 project there were 94
restarts (not 94/100 some may have restarted more than once.)
–
See California Department of Motor Vehicles.
• Overruns: This may be time or cost or both.
– For Type 2 and Type 3 projects almost a third had a
cost overrun of 150 to 200%.
– Over a third was 200 to 300% over their estimated
time.
• Content Deficiencies: For challenged projects
only, there was an average of 61% of originally
specified features.
Enough about stats, why did they fail?
• Top three things that help make or break a
project:
– User involvement. (15.9%) (12.8%)
– Executive management support. (13.9%) (7.5%)
– Clear statement of requirements. (13.0%) (12.3% /
11.8%)
• Some others included lack of resources, planning,
time frames, expectations, objectives.
% - when participants were asked about factors that cause projects to succeed.
% - when participants were asked about factors that cause projects to be challenged.
Some quotes from people in the field
(from the focus groups):
• John, Director of MIS at a government agency added:
"Probably 90% of application project failure is due to
politics!"
• Bill, the Director of MIS at a securities firm, "Changes,
changes, changes; they're the real killers."
• "Brain-dead users, just plain brain-dead users," said
Peter, an application analyst at a bank.
• Sid, a project manager at an insurance company. "The
project was two years late and three years in
development, We had thirty people on the project. We
delivered an application the user didn't need. They had
stopped selling the product over a year before."
Case Study1: California DMV
(Department of Motor Vehicles)
• 1987, the California DMV tried to overhaul their
drivers license and registrations application
process.
• 1993, after $45 million dollars were spent the
project was cancelled.
• Their objective: To have a system that was able to
rapidly respond to change.
• Reasons why it failed: No monetary payback,
lack of executive management, no user
involvement, poor planning, poor design
specifications, and unclear objectives.
Case Study2: Hyatt Hotels
• Their objective: To update their reservation
system.
• They succeeded, total cost was $15 million
(under budget), and ahead of schedule.
• Reasons why it succeeded: High user
involvement, executive management
support, a clear statement of requirements,
proper planning, and small milestones.
Conclusion
• Still need a more comprehensive report.
• Developed a success potential chart (which
weights each component like user involvement)
to help predict the success of a project.
• Failures and successes have to be continually
examined, and reported.
• Suggested a similar process to what was talked
about in the SCUM video. (That is short
iterations, deploy small elements, etc.)
Problem with the report....
• Specific data cannot be made publically available,
so no way to verify data.
• Problem with definitions of the Types. (i.e. a
project which arrives on budget and time, but
didn’t have all the features doesn’t have a specific
type.)
• Estimations are one-sided (ignores projects that
are under-cost, time, or have more features)
• There must have been biases/generalizations
made in order to get the cross intuitional figures.
• More details in Rise and Fall of Chaos.
References
• Chaos Report:
http://www.projectsmart.co.uk/docs/chaosreport.pdf
• The Rise and Fall of Chaos:
http://www.cs.vu.nl/~x/chaos/chaos.pdf