PowerPoint® Slides to Accompany
A Gift of Fire: Social, Legal, and
Ethical Issues for Computers and the
Internet
(2nd Edition)
by
Sara Baase
San Diego State University
PowerPoint slides created by Sherry Clark
Copyright 2003 Prentice Hall
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
1
A Gift of Fire
Can We Trust the Computer?
What Can Go Wrong?
Case Study: The Therac-25
Increasing Reliability and Safety
Perspectives on Failures, Dependence, Risk, and Progress
Computer Models
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
2
What Can Go Wrong?
Facts About Computer Errors
• Error-free software is not possible.
• Errors are often caused by more than one factor.
• Errors can be reduced by following good procedures and professional
practices.
Q: How do we distinguish between tolerable or unavoidable errors in software
versus careless software development?
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
3
What Can Go Wrong?
The Roles of People in Computer-related
Problems:
Computer User
• At home or work, users should understand the limitations of computers
and the need for proper training and responsible use.
Computer Professional
• Understanding the source and consequences of computer failures is
valuable when buying, developing, or managing a complex system.
Educated Member of Society
• Personal decisions and political, social, and ethical decisions depend on
understanding computer risks.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
4
What Can Go Wrong?
Categories of Computer Errors and Failures
Problems for Individuals:
• usually in their role as consumers.
• who are incorrectly identified by inaccurate law enforcement databases.
System Failures:
• affecting large numbers of people and/or costing large amounts of
money.
Safety-Critical Applications:
• where people may be injured or killed.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
5
What Can Go Wrong?
Problems for Individuals
Billing Errors
• Lack of tests for inconsistencies and inappropriate amounts.
Database Accuracy Problems
• Incorrect information resulting in wrongful treatment or acts.
Q: Describe a computer error or failure that has affected you.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
6
What Can Go Wrong?
Problems for Individuals (cont’d)
Causes:
•
•
•
•
•
•
Large population.
Human common sense not part of automated processing.
Overconfidence in the accuracy of data from a computer.
Errors in data entry.
Information not updated or corrected.
Lack of accountability for errors.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
7
What Can Go Wrong?
System Failures
Communications:
• Telephone, online, and broadcast services.
Business:
• Inventory and management software.
Financial:
• Stock exchange, brokerages, banks, etc..
Transportation:
• Reservations, ticketing, and baggage handling.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
8
What Can Go Wrong?
System Failures (cont’d)
Causes:
•
•
•
•
Insufficient testing and debugging time.
Significant changes in specifications (during and after project begun).
Overconfidence in system.
Mismanagement of the project.
Q: Describe a recent system failure that affected many people or resulted in a
great monetary loss.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
9
What Can Go Wrong?
Safety-Critical Applications
Military
Power Plants
Aircraft
Trains
Automated Factories
Medicine
…others.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
10
What Can Go Wrong?
Safety-Critical Applications
Causes:
•
•
•
•
•
Overconfidence.
Lack of override features.
Insufficient testing.
Sheer complexity of system.
Mismanagement.
Q: What activities do you participate in that are controlled by safety-critical
applications?
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
11
Case Study: The Therac-25
The Therac-25 was a software-controlled
radiation-therapy machine used to treat
people with cancer.
Overdoses of radiation
• Normal dosage is 100–200 rads.
• It is estimated that 13,000 and 25,000 rads were given to six people.
• Three of the six people died.
Q: What determines whether the risks associated with a safety-critical
application are acceptable?
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
12
Case Study: The Therac-25
Therac-25 Radiation Overdose
Multiple Causes:
•
•
•
•
•
•
Poor safety design.
Insufficient testing and debugging.
Software errors.
Lack of safety interlocks.
Overconfidence.
Inadequate reporting and investigation of accidents.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
13
Increasing Reliability and Safety
What Goes Wrong?
Computer Systems Fail Because:
• The job they are doing is inherently difficult, and
• The job is done poorly.
Compounding the Reliability Issue:
• Developers and users exhibit overconfidence in the system.
• Reused system software may not work in different environments.
Q: Identify the elements needed as an incentive to increase reliability and
safety.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
14
Increasing Reliability and Safety
Professional Techniques
•
•
•
•
•
•
•
Follow good software practices.
Exhibit professional responsibility at all levels of development and use.
Construct well-designed user interfaces.
Take human factors into account.
Include built-in redundancy.
Incorporate self-checking where appropriate.
Follow good testing principals and techniques.
Q: What human interface features should be considered for ordinary business
applications?
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
15
Increasing Reliability and Safety
Law and Regulation
Criminal and Civil Penalties:
• to recover loss from faulty or unsafe systems.
Liability and Civil Laws:
• to provide incentives to produce reliable and safe systems.
Warranties:
• to guarantee a certain level of quality.
Federal or State Regulations:
• to protect the public.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
16
Increasing Reliability and Safety
Law and Regulation (cont’d)
Database Accuracy Enforcement:
• to protect the public from inaccurate information maintained by private
companies and government.
Mandatory Licensing of Software Developers:
• to ensure proper training, competency, and continuing education.
Q: How can consumers protect themselves from faulty software?
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
17
Perspectives on Failures,
Dependence, Risk, and Progress
Failures
• What are acceptable rates of failures?
• How accurate should software be?
Dependence
• How dependent on computer systems are our ordinary activities?
• How useful are computer systems to our ordinary activities?
Risk and Progress
• How do new technologies become safer?
• Can progress in software safety keep up with the pace of change in
computer technology?
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
18
Computer Models
Points to Consider:
• Models are simplifications of either physical or intangible systems.
• Those who design and develop models must be honest and accurate
with results.
• Computer professionals and the general public must be able to evaluate
the claims of the developers.
Q: What problems in your community have been or could be studied with
computer models?
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
19
Computer Models
Evaluating Models
Why Models Might Not Be Accurate:
•
•
•
•
•
Developers have incomplete knowledge of the system being modeled.
Data might be incomplete or inaccurate.
Power of the computer might be inadequate.
Variables are difficult to numerically quantify.
Political and economic motivation to distort results.
Q: For each item above, give an example of a model or simulation that was
inaccurate.
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
20
Computer Models
Evaluating Models (cont’d)
Regarding the Car-Crash Models Described in the Text:
• How well do the modelers understand the system and/or materials
being studied? How accurate and complete are the data?
• What are the assumptions and simplifications in the model?
• Do the results or predictions correspond with the real world?
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
21
Computer Models
Evaluating Models (cont’d)
Regarding the Climate Models Discussed in the Text:
• How well do the modelers understand the system and/or materials
being studied? How accurate and complete are the data?
• What are the assumptions and simplifications in the model?
• Do the results or predictions correspond with the real world?
A Gift of Fire, 2ed
Chapter 4: Can We Trust the Computer?
22