Data - The Next Quality Disaster
American Society for Quality,
Baltimore Section
Thomas C. Redman, Ph.D.
Navesink Consulting Group
732-933-4669
www.dataqualitysolutions.com
December 10, 2002
ASQ-DQ Disaster
© Navesink Consulting Group
My messages
Bad data are at the root of the events of
national and international importance. They
capture our interest and won’t let go.
Fortunately, most data quality issues don’t
make the newspaper. Unfortunately they
are insidiously hidden in everyday work,
increasing cost, causing customer dissatisfaction, and making it harder to make
decisions.
© Navesink Consulting Group
Outline
•
•
•
•
•
•
Data Quality Defined
Data Quality Disasters in the News
Everyday DQ Disasters
COPDQ
We Know What To Do About It
Final Remarks: A Clarion Call
© Navesink Consulting Group
What Makes Quality Data?
Data are of high quality if they are fit for their
intended uses (by customers) in operations, decisionmaking, and planning (after Juran).
Data that’s fit for use
free of defects:
- accessible
- accurate
- timely
- complete
- consistent with other sources
- etc.
possess desired features:
- relevant
- comprehensive
- proper level of detail
- easy-to-read
- easy-to-interpret
- etc.
Customers are the ultimate arbiters of quality!!
© Navesink Consulting Group
Poor Data Quality in the News
“Wall Bursts and Water Pours In, Trapping 9 Miners
240 Feet Down”
Though Pennsylvania state regulations require a 200 foot
safety wall between adjoining mines, outdated maps
of area mines failed to show the miners’ proximity to
an existing mine. After 77 difficult hours, all nine
men were freed.
The New York Times, July 26, 2002, Francis X. Clines
© Navesink Consulting Group
Poor Data Quality in the News
“Right Answer, Wrong Score: Test Flaws Take Toll”
The testing industry is coming off its three most
problem-plagued years. Its missteps have affected
millions of students who took standardized
proficiency tests in at least 20 states. An examination
of recent mistakes and interviews with more than 120
people involved in the testing process suggest that
the industry cannot guarantee the kind of error-free,
high-speed testing that parents, educators and
politicians seem to take for granted.
New York Times, May 20, 2001, Diana B. Henriques
© Navesink Consulting Group
Poor Data Quality in the News
“Indecision 2000: The Ballad of the Ballots”
The 2000 Presidential Election in the United States held
our attention for weeks as we tried to determine:
• the accuracy of various counts and recounts
• how the end-to-end "election chain" is supposed to
work
• the final result and consequences of the above.
The Associated Press, November 19, 2000, Ted Anthony and
Jerry Schwartz
© Navesink Consulting Group
Poor Data Quality in the News
“Voting: What is, What Could Be”
Four to six million votes, a number which is double
the population of Chicago, were lost in the 2000
presidential election due to problems with ballots,
equipment, registration or at the polling place,
according to a joint analysis by experts at the
California Institute of Technology and the
Massachusetts Institute of Technology.
MIT News, July 16, 2001
© Navesink Consulting Group
Poor Data Quality in the News
“Primary Day in Florida Goes On, and On, and On”
The primary election results for the Democratic
nomination for governor of Florida were held up
for 3 days while Miami-Dade election officials
tried to sort out problems at polling locations –
despite the new $32 million voting system which
replaced the one blamed in the 2000 Presidential
Election.
The New York Times, September 14, 2002, Adam Nagourney
© Navesink Consulting Group
Poor Data Quality in the News
"Holes in System Hid Links in Sniper Attacks"
.
According to the authors, "In the month before the sniper
attacks that left 10 dead and terrorized in the
Washington area, John Muhammad and Lee Malvo are
believed to have killed or wounded seven people in four
states. But because they kept on the move and the
police gave only routine attention to what seemed like
run-of-the-mill crimes, investigators say, Mr.
Muhammad and Mr. Malvo were able to slip through
the cracks, gaps and blind spots where jurisdictions
meet and crime-fighting databases end."
New York Times, November 29, 2002
© Navesink Consulting Group
Poor Data Quality in the News
“CIA Fires Officer Blamed in Bombing of Chinese
Embassy”
Using out-dated information, the CIA selected the
address of an armory for a bombing target. At the
time of the bombing, however, the building housed
the Chinese Embassy.
The New York Times, April 9, 2000
© Navesink Consulting Group
Poor Data Quality in the News
“Mail from INS Stuns Flight School”
As the result of mismanaged documents, a Florida
flight school received notice form the INS of
approval of student visas for two of the September
11th terrorists – six months to the day after the
attack on America.
USA Today, March 13, 2002, Kevin Johnson
© Navesink Consulting Group
Poor Data Quality in the News
“FBI Director Details Blunder on McVeigh Records”
The FBI disclosed that it committed a "serious error"
when it failed to turn over thousands of pages of
interview reports in the 1995 Oklahoma City
bombing case to lawyers for Timothy J. McVeigh,
who was sentenced to death for the crime.
The New York Times, May 17, 2001, David Johnston
© Navesink Consulting Group
Data Quality In the News
• United Nations Resolution 1441 mandated that Iraq
declare any current holdings of weapons of mass
destruction
• Does anyone seriously think that a 12,000 page
document, supported by 4 CD-ROMS will be errorfree?
© Navesink Consulting Group
Poor Data Quality in the News
“The Wrong Foot, and Other Tales of Surgical Error”
At least 500 times since 1996, surgeons in hospitals
in this country have operated on the wrong arm,
leg, eye, kidney or other body part, or even on the
wrong patient. The figure does not include nearmisses (when surgeons started to operate on the
wrong site or patient because no one collects this
information.
The New York Times, December 11, 2001, Lawrence K. Altman, MD
© Navesink Consulting Group
Poor Data Quality in the News
“Right Drug, Wrong Patient”
In 1997, a nationwide survey conducted by the trade
publication Drug Topics, 53 percent of pharmacists
admitted having made errors in the preceding two
months. A June 1996 survey of 3361 pharmacists
in California and Oregon revealed that errors
occurred at an annual rate of 324 per pharmacy –
nearly one a day!
Reader’s Digest, September 1998
© Navesink Consulting Group
Poor Data Quality in the News
“To Err Is Human: Building a Safer Health System”
This report, initiated by the Institute of Medicine, found that:
• Adverse events occurred in 2.9 percent of hospitalizations in Colorado
and Utah and 3.7 percent in New York.
• 8.8 percent of adverse events led to death in Colorado and Utah,
compared with 13.6 percent in New York hospitals.
• Medication errors alone, occurring either in or out of the hospital, are
estimated to account for over 7,000 deaths annually.
• More people die in a given year (between 44,000 and 98,000) as a
result of medical errors than from motor vehicle accidents, breast
cancer or AIDS.
• The national costs of preventable adverse events are estimated to be
between $17 billion and $29 billion.
“To Err is Human,” Committee on Quality of Health Care in America,
Institute of Medicine, November 1999.
© Navesink Consulting Group
Poor Data Quality in the News
“Accounting Wars”
Between 1997 and 1999, long before the current
financial reform outcry, at least 364 companies
restated financial statements.
Business Week, September 25, 2000, p.158
© Navesink Consulting Group
Poor Data Quality in the News
“The Great Divide”
The Enron scandal, on the other hand, clearly was
about us. It told us things about ourselves that we
probably should have known, but had managed not
to see. I predict that in the years ahead Enron, not
Sept. 11, will come to be seen as the greater turning
point in U.S. society.”
The New York Times, January 1, 2002, Paul Krugman
© Navesink Consulting Group
Everyday Data Quality Disasters
• Pricing errors on the internet
– United Airlines sold about 1,000 tickets for all flights
from and through Chicago and Denver for $5
– W Hotels – New Years Eve in NYC for $25/night
instead of $259.
– Hilton Hotels listed rooms for $0 at least 3 times in 6
months.
– Amazon.com sold a memory board valued at $329.99
for $9.99 and another worth $999.99 for $10.77
• Hyundai over-reported horsepower ratings by as much as
9.6% on some vehicles sold since 1992.
• Bear, Sterns ordered $4 billion stocks sold – order should
have been for $4 million.
© Navesink Consulting Group
© Navesink Consulting Group
Cost-of-Poor-Data-Quality
COPDQ =
+
+
+
+
+
+
Cost to Prevent Errors
Cost to Find and Fix Errors
Cost to Make Good for Customers
Cost of Bad/Delayed Decisions
Cost of Lost Opportunities
Cost of Damage to Reputation
+ +
Can (usually)
be estimated*
(Usually) more
important
*Doing so can be tricky. If the organization will respond to COPDQ, we
recommend:
– Doing so at the “project level” first,
– Then expanding to a supplier or information chain
– Working up to the entire company.
© Navesink Consulting Group
COPDQ – The Evidence
• Proprietary studies: “easy-to-measure” (finding and fixing
errors) COPDQ are 10-20% of revenue.
• The Data Warehousing Institute: estimates that, in the US,
$611 billion a year is lost as a result of poor customer data
(name and address data).
• As a result of the embassy bombing, the US paid the
Chinese government $27 million. But the real cost is much
higher.
• As a result of the 2000 Presidential Election, Florida spent
$32 million on a new system. The federal government will
spend $3.9 billion in the next 3 years to help states replace
outdated voting machines and train personnel.
• The error on W Hotel’s website cost $175,266 in bookings
to one hotel. The web error was posted for only 7 days.
© Navesink Consulting Group
Data Underlie Everything We Do
Perhaps they are “hidden in plain sight”
© Navesink Consulting Group
A Database is Like a Lake
To Clean Up The Lake, One Must First
Eliminate The Sources Of Pollutant
© Navesink Consulting Group
It is so easy for accountability to shift
downstream!!!
Here’s how you
do number 3,
son
cos2(x) + sin2(x) = 1
© Navesink Consulting Group
Lead, follow, or risk being acquired
Leading organizations have improved by two orders of
magnitude and reduced COPDQ by two-thirds.
Best-practice:
• Adopt the philosophy of prevention
• Leadership
• Management Accountability at the points of data creation
• Focus on customer
• Measurement, control, improvement
• Data Culture
© Navesink Consulting Group
A Clarion Call
Poor quality data are the norm and no one is immune to their
effects. Most issues are mundane, but their impact is
stunning. They:
• Increase cost – approximately 20% of revenue
• Anger customers
• Make decision-making more difficult
• Make it harder to implement new technologies
• Put company image at risk
© Navesink Consulting Group