USING CONCLUSION, ANALYSIS AND EVIDENCE DIAGRAMS TO SUPPORT
STAKEHOLDER ANALYSIS IN ACCIDENT REPORTS
Nick Chozos
Glasgow Accident Analysis Group
Department of Computing Science
University of Glasgow
1. Short Summary
Accident reports describe the reasons for a system failure. In many cases, problematic software has been
identified as a cause of accidents in accident reports. Although the technical nature of a software failure is
identifiable through available techniques, the subsequent organizational and managerial issues that lead to
the software failures are an abstract, complex and unquantifiable problem to consider. This paper uses a
case study (SIT, 2000) in order to examine the organizational and political dimensions of the causes of an
accident resulting from a software failure, and suggests ways to reason about them from a systems
perspective.
2. Rationale
From the 1st of January, 2000 untill the 20th of May, 2000, 158 pregnant women were screened
incorrectly for the risk of giving birth to a child with Downs Syndrome, due to a software malfunction in
the Immunology Department of the Sheffield Northern General Hospital. In order to investigate this
event, a formal inquiry was conducted. The inquiry team characterise this incident as:
"...an intricate tale with its roots going back to the late eighties. From early on events
occurred, things were done and decisions were made which had a material bearing on the
failure of the Downs Screening software at the Northern General to perform correctly for a
period from 1st January 2000." (SIT, 2000, p. 76).
The inquiry team argued that the software failure was mainly a result of organizational and managerial
decisions and actions taken during the development and use of the software system. As we shall see,
organizational problems not only contributed to the failure of the software, but also allowed it to produce
false results for a period of over four months.
In general, although the technical faults of a software component can be traceable, for instance through
techniques such as software fault trees (Leveson et al., 1991), it is challenging to discuss the further
organizational and systemic problems that lead to these technical faults taking place, and, most
importantly, going undetected and unreported throughout the lifecycle of the development and use of the
application (Johnson, 2002).
3. Theoretical background
In this paper, we have used two techniques. Stakeholder Analysis, and Conclusion, Analysis and
Evidence Diagrams. They are both briefly discussed in this section.
3.1 Stakeholder Analysis
Stakeholder analysis originates from the business and management science. However, since its origins, it
has been applied in a number of different areas, such as marketing, production (Carroll, 1989), finance,
accounting, and information systems’ development (Pouloudi and Whitley, 1997). There are various
definitions of what a ‘Stakeholder’ is (Barrett, 2001; Papazafeiropoulou et al., 2001). Freeman (1984, p.
46) defined a stakeholder within an organization as “any group or individual who can affect or is affected
by the achievement of an organization’s objectives”. Another approach, provided by Clarkson (1995),
describes stakeholders as:
“persons or groups that have, or claim, ownership, rights, or interests in a corporation and its
activities, past, present, or future. Such claimed rights or interests are the result of transactions
with, or actions taken by, the corporation, and may be legal or moral, individual or collective”
(p. 106).
1
The approach overall examines social relationships in organizations. Issues such as personal interests,
politics and power among stakeholders can also create or promote instability within systems and thus
affect the development and use of an information system, especially in complex contexts like healthcare
(Atkinson et al., 2001). In this sense, their identification and analysis is an imperative task.
Stakeholder theory has three main aspects (Donaldson and Preston, 1995):
1)
The descriptive aspect of stakeholder theory concerns the identification, description and possibly
explanation of characteristics and behaviours of an organization.
2)
The instrumental aspect considers the identification of internal and external organizational
connections.
3)
The normative aspect, where the theory attempts to elaborate on the function of the organization,
and identify various moral and ethical issues that stem from the interconnections within the
organizational network. Ethical issues, as stated by Key (1999), are not considered in the traditional
philosophical sense; rather, their identification aims at measuring the “ ‘fit’ for utilitarianism” (p. 320),
or, bringing to light the necessary trade-offs that need to be made between stakeholders’ goals and the
organization’s goals.
3.1.1
Stakeholder Analysis In Information Systems Development
Stakeholder analysis has been identified in literature as a tool to facilitate information systems
development, by being integrated with development approaches such as Soft Systems Development
Methodology (Vidgen, 1997), Soft Systems and Information Technology Methodology, Participative
Simulation Modelling (Atkinson et al, 2001).
It has been suggested as a tool to study various parameters of development projects’ success or failure,
such as level of commitment, attitudes and expectations of various information systems stakeholders
(Pouloudi, 1999).
3.2 Conclusion, Analysis and Evidence Diagrams
Conclusion, Analysis and Evidence (CAE) diagrams are graphs. The roots represent individual
conclusions from an accident report. Lines of analysis that are connected to items of evidence support
these. Each item of evidence either weakens or strengthens a line of analysis. CAE Diagrams can help to
reason about the conclusions against the evidence provided (Johnson, 2001).
Analysis1.1
Evidence1.1.1
Evidence1.1.2
Conclusion1
Analysis1.2
Evidence1.2.1
Evidence1.2.2
Analysis1.3
Evidence1.3.1
Evidence1.3.2
Figure 1: Example CAE Diagram
4. Methodology
As mentioned, our aim is to further investigate the findings of the inquiry, in order to understand the
organizational and political dimensions of the software failure, and consider an approach to deal with
them from a systems’ design perspective.
In order to do so, we have used stakeholder analysis and CAE Diagrams. More specifically, we have
decided on one important conclusion drawn by the inquiry team, and produced the CAE Diagram for it.
In the text corresponding to the findings of the CAE diagram, there is a number of stakeholders
mentioned. Therefore, the CAE Diagram has helped in identifying and focusing our analysis on the
2
specific persons and organizations that were important to the specific conclusion about the incident, thus
filtering from the abundance of people that are mentioned throughout the report.
According to Turner (1978), organizations are vulnerable to socio-technical accidents. Disasters arise
from an interaction of the human and organizational arrangements of the socio-technical systems set up to
manage complex and ill-structured risk problems (Pidgeon, 1997). Hence, the purpose of stakeholder
analysis in this case, is to discuss people’s roles, perceptions and relationships, in order to measure the
socio-political context of the specific system’s failure that is reflected in the conclusion the CAE diagram
has elaborated upon.
5. Case Study: The Downs Syndrome Screening Software Failure in the Sheffield Northern
General Hospital
The Downs Syndrome Screening Software Failure incident took place in the Immunology Department of
the Sheffield Northern General Hospital. There, clinicians were using a software program to calculate the
risk factor of pregnant women giving birth to children with the Downs Syndrome.
The system had faced a number of problems through the 12 years of its life. In particular, it is wasn’t
Y2K compliant, although some actions had been taken to assure its safety against the bug, the result was
158 women screened wrongly as being at low risk, where in fact they were in the high-risk category. Out
of the 158 wrongly screened women, four Down'
s syndrome pregnancies went undetected. Two of the
women later gave birth to babies with the Down'
s syndrome, while two other women terminated their
pregnancies (BBC, 2000).
Following this incident, a formal inquiry was conducted, in order to identify the reason behind the failure,
and to make recommendations for the improvement of the system’s safety.
5.1 The Downs Syndrome Screening Process
The Department of Immunology at the Northern General Hospital provides the Downs Screening service
for GPs in Sheffield and surrounding areas and ten NHS Trusts and hospitals.
The process of Downs Screening is a complex multidisciplinary process involving obstetricians,
radiologists and the diagnostic laboratory. Several blood tests are taken at 15 – 17 weeks which are sent
to the laboratory for analysis. Calculations are then used to assess the risk of the foetus being affected by
Downs syndrome. These start with the age-related risk which is derived from known incidence of
pregnancies affected by Downs syndrome based on maternal age. This risk is modified using a likelihood
ratio of the presence of an affected foetus derived from the concentrations of the markers to provide the
final risk value. The calculation relies critically on accurate estimation of the projected maternal age at
delivery which, in turn, is based on the ultrasound measurements of foetal age and the date of birth of the
mother.
The PathLAN software makes the necessary calculations to produce the final likelihood ratio, and the
results are then passed on the obstetrician who will suggest a more definite diagnostic procedure to
women that are assessed as being at high-risk.
5.2 Regarding the Software Development
Downs Screening had begun in Sheffield within the Immunology Department at the Royal Hallamshire
Hospital in 1988 under the direction of a Dr A, as mentioned in the report. The software used by the
Sheffield Northern General NHS Trust to calculate the risk of foetus being affected by Downs Syndrome,
was written by an external consultant and paid for from Dr A’s research funds. The software was based
on an established algorithm. It was written in GWBasic operated on a dedicated PC.
Between 1988 and January 2000, the algorithm has been used in different pieces of software, the software
had undergone changes, had been moved between two hospitals, had been set up on different operating
systems, and had been maintained either in-house by people with self-acquired IT skills, or external
consultants, with self- acquired IT skills (SIT, 2000, p. 76).
5.3 The Inquiry Team Report
3
The inquiry team, made up by medical experts and a solicitor, who also sought the advice of computing
experts, performed a formal inquiry that lasted over five months. The overall conclusion is that
organizational factors led to the software failure in the Sheffield Great Northern Hospital (SIT, 2000, p.
77).
The report covers facts spanning twelve years, from when the software was originally developed, till the
day where patients were notified of the problem. The document of 120 pages mainly discusses
managerial decisions that were made, the people involved, their actions, and the affect those decisions
and actions had on the software.
5.4 Conclusion, Analysis and Evidence Diagram
The conclusion we have chosen to examine is the following:
C1: Nobody therefore had a means of knowing, or even of acquiring a feel, whether an individual result
was unusual (SIT, 2000, p. 77). Our aim is to discuss why the error went undetected for a period of 4
months, and this conclusion is deemed as one of the main reasons why, according to the inquiry team. In
figure 2, the CAE diagram for conclusion C1 is depicted, and then its components are presented1.
C1: Nobody therefore had a means
of knowing, or even of acquiring a
feel, whether an individual result
was unusual (p. 77).
A2: To the Systems Managers,
to Mr W and to the technical
staff operating the software daily,
the algorithm was like a black
box: information was fed in and a
result was generated. (p. 77)
A1: Only Dr.
A understood
the algorithm.
(p. 77)
E1
E2.2
E2.1
A3: There was no
error-trapping
(rewording p. 77)
E2.3
E3
Figure 2: CAE Diagram
Throughout the analysis and evidence components of the CAE Diagram, a number of stakeholders were
identified. In the following section, they are presented and discussed.
5.5 Stakeholder Analysis
The three aspects of stakeholder theory, as presented in section 3.2 were applied with regard to the
positions, views and relationships of the stakeholders identified in the previous section.
5.5.1 The Descriptive Aspect: Stakeholder Identification
In brief, the stakeholders identified in the CAE diagram in 5.3 are: (Immunology Dept., Sheffield
Hospital) Dr A, Dr B, Mr R, Mr K, Mr L, a Liaison sister, lab staff and clinical staff and (Hartlepool) Mr
W, as well as a Mid wife Coordinator in another hospital. In table 12, some of the stakeholders are
presented, in terms of their roles, interests, perceptions and relation to the PathLAN.
Stake
holder
Dr
A
Dr B
1
2
Responsibility
HoD until Dec 1999. In charge
of PRU, NEQAS and PQA
Limited. Also holds a post in
Sheffield University.
HoD after Dec 1999
Influence on
the System
Relationship to the
Software
Perceptions, views,
concerns
Very High
Original designer, only one
that understands the algorithm
Highly Supportive, and
confident that the
software is safe
High
No direct relation
Beginning of May
2000, stressed concerns
about logs not being
Due to space restrictions, figure 2 is not discussed
Due to space restrictions, table 2 is not fully presented
4
Mr R
Retired in Dec. 1999
Very High
Technical Manager
Laboratory Manager
maintained (p. 70)
Had stressed concerns
about Y2K in ’96 (p.
46).
Mr K
Computer Systems manager
Laboratory Manager after Mr
R’s retirement
High
Following Mr R’s retirement,
shared Mr R’s responsibilities
with Mr L, although the role
‘Computer Systems manager’
had cessed.
Was seeking assurances
by Hartlepool and
received them (p. 54).
Hartle
pool
Maintenance and support at
SGN’s request
Very High
Maintained the software,
suggested PathLan, only other
hospital that used it. Stopped
using it a year prior to the
accident.
Hartlepool was assured
PathLan is Y2K
compliant, and no other
bugs resided in the code
(p. 65).
Laboratory IT Manager at
Hartlepool, subcontracted to
Great Northern
High
Technical Support
Has stressed concerns
about Y2K compliance
(p. 65)
Nurses and Doctors
Medium
Mr W
Clinical
and
Lab Staff
Users, Keep Log files after Jan
Confident in the
2000
systems’ success (p. 77)
Table 1: Stakeholder Identification
5.5.2 The Instrumental Aspect: Stakeholder Relationships
The instrumental aspect of stakeholder analysis considers the identification of internal and external
connections of stakeholders and stakeholder groups. The relationships of stakeholders discussed in the
previous section, are depicted in figure 33.
Dr. B
Dr A
Mr W
algorithm
MR R
Mr M
Mid
Wife Coordinator
Clinical
Staff
PathLan
Hartlepool
Trust
Immunology Department, Sheffield Northern
General Hospital
Other
Hospital
Figure 3: Stakeholder Relationships
5.5.3 The Normative Aspect: Discussing Ethical Issues
The normative aspect of stakeholder analysis revolves around the discussion on ethical issues. It has been
suggested that moral issues can be identified by studying stakeholder relationships (Kujala, 2001).
According to ethical reasoning, issues of morality appear whenever a person (or an ‘actant’- or a
stakeholder) is confronted with a dilemma, and one’s decision has a consequent affect on others (Artz,
1994). The identification of points where a stakeholder was faced with one or more dilemmas, and the
decisions made (and not made) can potentially tell us a lot about what happened, or what can happen, as
making a wrong decision when faced with a dilemma, could well be part of the causes of the system’s
failure.
3
This is still work in progress, and findings are not complete
5
A problem with ethical discussion in the field of computing is that it is often very difficult to predict
consequences due to the complexity that characterises computerized information systems, and even more
difficult to determine a link between action and consequence, before something happens (Artz, 1994).
However, by examining the incident with hindsight, we are already aware of points were a ‘wrong’
decision was made. Therefore, rather than studying all stakeholder relationships, the discussion on
stakeholder relationships in this case focuses on examining how a specific decision which we can
determine as eventually critical according to its consequences relates to various stakeholders, in terms of
how it was caused and what its affects were to other people.
As an example, we can consider the following: There are five incidents mentioned in the report were
stakeholders, mainly users (Mid-wife coordinator, Liaison Sister) were raising concerns to people in the
Immunology Dept. (Mr M, Dr B, Mr K), however, no actions were taken. It is therefore important to
study the reasons why the decision not to follow up on concerns was taken by people responsible.
5.5.4
Using the Findings of Stakeholder Analysis
Stakeholder analysis has been used in the information systems area extensively for assisting in the
requirements capture (Atkinson et al, 2001). It is known for being applied along with Soft Systems
Development Methodology (SSDM), and other ‘soft’ approaches, such as Soft Information Systems and
Technology Methodology (SISTeM) and Participative Simulation Modelling. For instance, SSDM is a
problem-based development approach- it is used to bring about improvement in areas of social concern
(Lehaney and Paul, 1996). Stakeholder Analysis is used to ‘shed light’ on issues within an organization
that can be an obstacle to an information system development project (Pouloudi, 1999).
Currently, we are considering the use of Soft Systems Models in order to model the problems that are
identified in the findings of stakeholder analysis in this paper.
6. Discussion
The incident was the ending of a story lasting over a decade. Although the obvious cause was the
incompliance of the software to Y2K, identifying why this happened has taken the inquiry team five
months and 120 pages, discussing facts, people and their decisions and actions taken over 12 years.
The technical part of the failure is deemed as obvious, and once concerns finally reached the people
responsible, it took 37 minutes to spot and fix (SIT, 2000, p. 78). However, the organizational, political
and ethical issues behind the development and use of the software program are not as straightforward.
Stakeholder analysis is a tool that has primarily been used during the analysis stage of information
systems’ development. It has been suggested as a tool to study various parameters of development
projects’ success or failure, such as level of commitment, attitudes and expectations of various
information systems stakeholders (Pouloudi, 1999). Although stakeholder analysis has not been used in
accident analysis, here we have applied it in order to produce a ‘rich picture’ regarding one aspect of the
software failure. One conclusion drawn by the Inquiry team, and all the corresponding information was
initially selected from the report using CAE diagrams, and it was then analysed with stakeholder analysis.
6.1 Ideas for Collaboration
The purpose of this paper is to present stakeholder analysis as a tool that can provide with valuable
insights about social and political characteristics of an adopting organization during a software
development project, although it could be used in accident analysis as well. The organizational decisions
and actions that contributed significantly to this software failure, and the political context within which
they were formulated are the issues that stakeholder analysis opts to measure during a development
project, before they determine its outcome. However, its integration with design tools and techniques,
such as Soft Systems Development Methodology is necessary in order to utilise stakeholder analysis’
findings, and facilitate an effective development approach.
Therefore, once the analysis of this case study is concluded, it will be interesting to consider design
methods that can address the social problems that had a great impact on the outcome of this information
system. Finally, we are going to apply stakeholder analysis in a similar system made up by various
medical centres and a hospital here in Glasgow, in order to contrast some of the issues discussed in this
6
paper in a ‘real’ case. Again, the three aspects of stakeholder analysis should be integrated with design
techniques to utilise its findings.
7
7. References
Adelakum O., and Jennex E. M. (2002). Stakeholder Process Approach to Information Systems
Evaluation. Americas Conference on Information Systems 2002. [available at pal.lternet.edu/lter/biblio/
finalms/jennex_AMCIS-2002-final-submital1.rtf]
Artz, J. (1994) Virtue vs. Utility: Alternative Foundations of Computer Ethics. Proceedings of the Ethics
in the Computer Age Conference. Gatlinburg, Tenn. November 11-13, 1994.
Atkinson C. J. , Eldabi T., Paul R. J. and Pouloudi A (2001). Integrated Approaches to Health
Informatics Research and Development. Logistics Information Management, 15 (2), pp. 138-152.
Barrett, M. (2001). A Stakeholder Approach to Responsiveness and Accountability in Non-Profit
Organizations. Social Policy Journal of New Zealand, 17, pp. 37-51.
BBC News (2001). Millennium Bug Blamed for Test Errors. [Internet:
http://news.bbc.co.uk/1/hi/health/1541557.stm (Last accessed: 2 November, 2004).
Available
at]
Clarkson, M.B.E. (1995). A Stakeholder Framework for Analysing and Evaluating Corporate Social
Performance. Academy of Management Review, 20(1), 92-117.
Freeman, R.E. 1984. Strategic Management: A Stakeholder Approach. Boston: Pitman.
Johnson, C. W. (2000). Viewpoints and Bias in Accident and Incident Reporting. Foresight and
Precaution: Proceedings of European Safety and Reliability Conference ESREL 2000, Eds. M.P. Cottam,
D.W. Harvey, R.P. Pape and J. Tait, Balkema, Rotterdam, The Netherlands, pp. 1395-1402
Johnson, C. W. (2001). The London Ambulance Service, Computer Aided Despatch System: A Case
Study in the Integration of Accident Reports and the Constructive Design of Safety-Critical Computer
Systems, Reliability Engineering and Systems Safety, 71 (3), pp. 311- 326.
Johnson, C. W. (2002). Forensic Software Engineering: Are Software Failures Symptomatic of Systemic
Problems? , Elsevier's Safety Science, One of the `Best Papers' from SAFECOMP 2000, 40 (9), pp. 835847.
Key, S. (1999). Toward a New Theory of the Firm: A Critique of Stakeholder “Theory”. Management
Decision, 37 (4), pp. 317-28.
Kujala, J. (2001). Analysing Moral Issues in Stakeholder Relations. Business Ethics: A European Review,
10 (3), pp. 233-247.
Lehaney, B. and Paul R. J. (1996) Soft Systems Methodology and Simulation Modelling.
Proceedings on the 1996 Winter Simulation Conference, 1996, pp. 693-700
Leveson N.G., Cha S.S. and Shimeall T.J. (1991), Safety Verification of Ada Programs using Software
Fault Trees, IEEE Software, 8(7), pp. 48-59.
Papazafeiropoulou, A., Pouloudi, A. and Currie, W.L. (2001). Applying the Stakeholder Concept to
Electronic Commerce: Extending Previous Research to Guide Government Policy Makers, Proceedings
of the 34th Hawaii International Conference on System Sciences.
Pidgeon, N.F. (1997). The limits to safety? Culture, politics, learning and man-made disasters, Journal of
Contingencies and Crisis Management, 5(1), 1-14.
8
Pouloudi A. and Whitley E.A. (1997). Stakeholder identification in inter-organizational systems: Gaining
insights for drug use management systems. European journal of information systems 6(1), pp. 1-14.
Pouloudi, A (1999). Aspects of the Stakeholder Concept and their Implications for Information Systems
Development. Proceedings of the 32nd Hawaii International Conference on System Sciences.pp. 1-17.
Pouloudi A. and Whitley E. A. (2000) Representing human and non-human stakeholders: On speaking
with authority. In Organizational and social perspectives on information technology (Baskerville
Richard, Jan Stage and Janice I De Gross eds.), pp. 339-354, Kluwer, Aalborg, Denmark.
Sheffield Inquiry Team (2000). Report of the Inquiry into the Computer Software Error in Downs
Syndrome Screening. [Available at: http://www.nhsetrent.gov.uk/press/nghreport.htm] (Last accessed: 2
November, 2004).
Smith (2002). Relationship between Quality, Safety and Organisational Behaviour, Quality and Safety in
Health Care. 11, pp. 98-100
Turner, B. A. (1978). Man Made Disasters. Wykeham Science Press, London.
Vidgen, R. (1997). Stakeholders, Soft Systems and Technology: Separation and Mediation in the
Analysis of Information System Requirements. Information Systems Journal (7), pp. 21-46.
9