Modeling Safety Outcomes on Patient
Care Units
Anita Patil
Department of Electrical and Computer Engineering
University of Arizona
apatil@email.arizona.edu
Judith Effken
College of Nursing
University of Arizona
jeffken@nursing.arizona.edu
Kathleen Carley
Department of Social and Decision Sciences
Carnegie-Mellon University
kathleen.carley@cmu.edu
Ju-Sung Lee
Department of Social and Decision Sciences
Carnegie-Mellon University
jusung@andrew.cmu.edu
1.1 Introduction
In its groundbreaking report, “To Err is Human,” the Institute of Medicine
reported that as many as 98,000 hospitalized patients die each year due to
medical errors (IOM, 2001). Although not all errors are attributable to nurses,
nursing staff (registered nurses, licensed practical nurses, and technicians)
comprise 54% of the caregivers. Therefore, it is not surprising, that AHRQ
commissioned the Institute of Medicine to do a follow-up study on nursing,
particularly focusing on the context in which care is provided. The intent was to
identify characteristics of the workplace, such as staff per patient ratios, hours
on duty, education, and other environmental characteristics. That report,
Modeling safety outcomes on patient care units
2
“Keeping Patients Safe: Transforming the Work Environment of Nurses” was
published this spring (IOM, 2004).
In the last five years, AHRQ also funded a number of research studies
directed at patient safety issues. The Impact Study described in this paper is one
of those efforts. In the Impact Study, we chose to focus on studying workplace
characteristics (organization and patient care unit) that can affect patient safety
and quality outcomes. To do so, we collected data from 37 patient care units in
12 Arizona hospitals. Hospitals that participated in the research included
teaching and non-teaching hospitals, as well as public and privately funded
hospitals ranging in size from 60 to over 400 beds. To ensure a more
comparable analysis across units, only adult medical or surgical units were
included in the sample. Data were collected in two “waves;” patient care units
from half the hospitals were assigned to each wave. Each wave of data
collection required six months to complete. Data related to each of the model
components were collected through surveys of patients, staff, managers, quality
improvement (QI) departments, and information services (IS). In all, 1179
patients and 867 staff were surveyed.
Organizational Characteristics
(Hospital Level)
Hospital Group Culture (Manager)
Hospital Developmental Culture
(Manager)
Lifecycle: Admission Change 2000-01
Patient Characteristics
(Unit Level)
Patient Outcomes
(Unit Level)
% High Complexity
% Self-Pay
% Patient > age 75
Average Number of ICD-9 Codes
Safety: Medication Errors, Falls with &
without Injury (1000 P.D)
Quality: Symptom Management , Self
Care and Satisfaction.
Unit Characteristics
(Unit Level)
Nursing Culture Composite
Team Culture Composite
Staffing Composite
Turbulence Composite
Figure 1. The Conceptual Model Used to Guide the Impact Study showing only
the variables used for computational modeling
The framework for the research was based on the 1996 American Academy
of Nursing Health Outcomes Model (Fig. 1) (Mitchell, Ferketich & Jennings,
1998). Patient safety outcomes studied included medication errors and patient
falls, with and without injury. Quality outcomes investigated included symptom
management, simple and complex self care, and perception of being well cared
Modeling safety outcomes on patient care units
3
for. In this paper, we will focus only on the safety outcomes. The data were
subjected to descriptive analysis, as well as causal modeling prior to their being
used to develop the computational model. Our emphasis was on identifying
changes that nurse managers could make at the patient unit level, because
organizational changes are not under their control. Similarly, nurse managers
cannot change the characteristics of the patients they see on their units; instead
patient characteristics were treated as risk adjusters in the analysis.
1.2 Using OrgAhead to Create Virtual Patient Care Units
1.2.1 Orgahead
We used OrgAhead, a computational modeling program, to create virtual units
that functionally matched our actual units and then evaluated the patient safety
outcomes for the virtual units under various conditions. OrgAhead is a
theoretically based computational modeling program developed by Dr. Kathleen
Carley and her team at Carnegie Mellon University. The theoretical perspectives
grounding OrgAhead derive from social network, complexity, and
organizational theories (Carley & Hill, 2001) and are consistent with the
theoretical basis of the conceptual model used for the Impact Study. OrgAhead
had been used in a number of military and business settings, but this provided its
first application in healthcare.
1.2.2 The modeling process
The computational modeling process (Fig. 2) included four distinct steps: First,
variables in our research model were matched to the variables in OrgAhead.
Variables that were found to be significant in our model that were not initially in
OrgAhead were created for OrgAhead in collaboration with Dr. Carley and her
team at Carnegie Mellon University. Next, we determined the range for each
independent variable in OrgAhead. In the third step, values were set for all other
variables used in OrgAhead. Finally, experiments were run to validate the
virtual model with actual data and then to generate hypotheses about the kinds
of changes that might be made to improve patient outcomes (Effken et al., 2003,
Effken et al., in preparation) using a static version of the model.
One of the strengths of computational modeling is its ability move beyond
static, snap-shot analyses to examine organizational performance over time. To
explore how patient safety outcomes might change under various changes in
structure and unit characteristics, we utilized the annealing characteristics of the
model. In Orgahead, individual learning occurs through a standard stochastic
learning model for boundedly rational agents (Carley, 1996). Organizational
adaptation or learning occurs as a simulated annealing process. The annealing
model was developed originally to solve complex combinatorial optimization
Modeling safety outcomes on patient care units
4
problems (Kirkpatrick, 1983; Rutenbar,1989).
Simulated annealing is a
heuristic for optimization and a computational analog of the physical process of
annealing (heating and cooling) a solid, in which the goal of the process is to
find the atomic configuration that minimizes energy costs. In organizations, this
is analogous to a design problem in which the organization is trying to optimize
organizational performance under various constraints (Carley, 1997). For our
purposes, we assume that the patient care unit endeavors to optimize
performance (e.g., achieving desired quality and patient safety outcomes) while
reducing or maintaining costs.
Organizational adaptation, for our modeling purposes, has two components:
executive decisions about particular restructuring goals and strategies and
individual employees’ experiential learning (Carley, 1996).
Executive
decisions are commonly assumed to be “satisficing,” rather than optimizing.
That is, the manager doesn’t consider all possible strategies are compared;
instead the first one that seems likely to move the organization toward the goal
is selected (March, 1958; Simon, 1954). Similarly, nurse managers do not
consider every possible intervention, but select the first one that seems likely to
work, given their current constraints.
2. Select independent
variables and range of
values they will take.
1. Match variables in
Research Model to
OrgAhead Variables
Examples:
Research Model  OrgAhead
Medication Errors  Accuracy
Control over practice 
SOP
(standard operating procedures)
4. Conduct
experiments
3. Set non-core
OrgAhead variables
based on research
data.
Examples:
Independent Variables: Task
complexity, control over
practice, training period
Non-core variables: Number
of staff levels (RNs, LPNs,
etc.), number of staff at each
level, probability of hiring staff
at each level.
Figure 2. The Computational Modeling Process
Modeling safety outcomes on patient care units
5
1.2.3 Modeling the patient care unit
The patient care unit is modeled as two interlocking networks: an authority
structure (who reports to whom?) and a resource management structure (who
has access to which resources?). We assume a 3-layered structure with RNs at
the top level, LPNs and patient care technicians (PCTs) at the second level, and
unit clerks at the bottom level. Individuals may have one or more subordinates
and report to one or more managers. This allows us to model teams, hierarchies,
matrix structures, etc.
Both the organization and individual employees operate in a “task”
environment where a “task” equals a patient.
In Orgahead, patients are
modeled as 9-bit binary choice tasks. For each “patient,” which is modeled as a
9-bit series of a’s and b’s, the organization (patient care unit, in our research)
has to determine whether there are more a’s or b’s in the string. Different levels
of staff see different numbers of bits. RNs see 4 bits, LPNs and PCTs see 2 and
Unit Clerks see 3. No individual can make a patient unit decision alone
(assuming bounded rationality and distributed decision making); instead the unit
decision is created as a majority vote of the individual decisions. For more
details, see Carley, 1998.
1.3 Results and Discussion
For the static experiments with the 32 virtual units, the correlation coefficient (r) of the rank order for accuracy (virtual units) and total errors (actual
units) was found to be 0.83. This exceeded our target of acceptable level of
correspondence of 0.80 (Effken et al, 2003). The correlation co-efficient for the
actual values for accuracy and total errors was found to be -0.55, which is
acceptable at the value level.
The results of our static experiments to improve performance by varying the
initial values of various independent variables (task complexity, workload,
turbulence, standard operating procedures (SOP), training and memory) are
summarized in Table 1 for 6 pilot units. These 6 pilot units are different in their
key characteristics, as well as in their safety outcomes. Improvement in
accuracy was obtained by varying the independent variables in the static
experiments is given. A 5% improvement in accuracy in the virtual world
corresponds to a decrease in 5 errors in the real world. Managers could
potentially select a number of strategies to decrease task complexity or increase
training, for example, to achieve this level of improvement in their real units.
Dynamic simulations involved setting up annealing parameters and
allowing the virtual units to adapt over time. Figure 3 shows a snapshot of how
the 6 pilot units adapt over time by changing the authority and resource
management structures resulting in the performance varying over time.
Modeling safety outcomes on patient care units
Unit
Task
Complexity
Training
Memory
6
A
15
(17)
834
(234)
500
(100)
B
12
(14)
1000
(365)
1000
(100)
C
8
(11)
903
(303)
900
(100)
D
9
(12)
943
(343)
800
(100)
E
8
(9)
404
(404)
100
(100)
F
11
(13)
903
(403)
800
(100)
0.4
(0.0)
0.8
(0.6)
0.9
(0.62)
80.16
(77.72)
0.3
(0.28)
0.6
(0.33)
0.99
(0.46)
83.78
(82.22)
0.1
(0.30)
0.5
(0.29)
0.9
(0.39)
86.07
(78.16)
0.3
(0.33)
0.8
(0.23)
0.9
(0.42)
85.16
(76.55)
0.1
(0.24)
0.5
(0.35)
0.9
(0.18)
86.39
(79.98)
0.3
(0.51)
0.6
(0.71)
0.8
(0.67)
78.19
(74.95)
2.44
1.56
7.91
8.61
6.41
3.24
SOP
(a) RN
(b) LPN/PCT
(c) Unit Clerk
% Accuracy
%Improvement
Table 1. Percentage Improvement in Accuracy (based on static modeling)
achieved for 6 Pilot Units modeled. The numbers shown in parentheses are the
values for the corresponding actual unit; all other numbers are those values used
in OrgAhead.
85
Unit A
% Accuracy
80
Unit B
Unit C
75
Unit D
70
Unit E
Unit F
65
60
0
1
2
3
4
5
6
7
8
9
Sim ulation Tim e (per 1000 change cycles)
10
Figure 3. Accuracy (safety outcomes) by Time for 6 pilot units
The average accuracy over time for the dynamic run for each unit was
consistent with the total errors reported on the actual units. For example, Figure
3 shows that Units C and D initially had similar levels of accuracy, but over time
performed differently, reflecting their actual performance values. The actual
Modeling safety outcomes on patient care units
7
Unit C produced 4.73 errors, but its corresponding virtual unit showed better
performance (avg. accuracy = 79.99%) as compared to Unit D, for which the
actual unit reported total errors of 26.18, and the virtual unit showed an average
accuracy of 70.04%.
After validating that the dynamic average accuracy values for the 6 modeled
units reflected their actual outcomes, we selected some of the best and the worst
accuracy values during the adaptation process and studied the respective
authority and resource management matrices to see how they influenced the
outcomes. This gave us some insight into the structures that lead to both
successful and unsuccessful performance. Of these values, we picked only
those structures that are most likely to occur in the real scenario. The result of
one such successful structure for Unit E along with its initial structure is as
shown in Figure 4.
Interestingly, the better performing structures were flatter. This generally
resulted in eliminating the unit clerk position, which is probably not realistic
because of their pivotal roles on the unit. This may represent a limitation of the
modeling, which assumes that all members of the team are working on the same
problem. In fact, the unit clerk has a rather different role. These results suggest
that, in the future, we may need to omit the clerks from the models and perhaps
include the nurse manager instead. Further modeling will be needed to inform
that decision.
Task-Resource Structure
Task Resource Structure
Authority Structure
Authority Structure
111100000
001111000
000111100
000011110
000001111
RN
LPN
/PCT
Unit
Clerk
110000000
001100000
000011000
000000110
101010000
(A) Initial Structure
111100000
001111000
000111100
000011110
RN
RN
LPN
/PCT
LPN
/PCT
Unit
Clerk
Unit
Clerk
001100000
000011000
000100110
100000000
111010100
RN
LPN
/PCT
Unit
Clerk
(B) High Performance Structure
Figure 4. The initial and high performance structures for Unit E.
1.4 Conclusion
Using OrgAhead, we created 32 virtual patient care units that were functionally
similar to the actual units in their key characteristics and safety outcomes with
the outcomes (accuracy with total errors) having an acceptable level of
Modeling safety outcomes on patient care units
8
correspondence. Using these virtual units, for the static model we varied the
various independent variables for 6 pilot units, and studied how each would
improve the safety outcomes. We then used OrgAhead dynamically, and
verified that the average dynamic accuracy is consistent with the safety
outcomes. Using the dynamic runs, we were able to pick the authority and
resource management structures that produced the best and worst safety
outcomes. Our computational model could be a very effective tool for nurse
managers to visualize various structures and strategies that they could use to
improve patient safety outcomes on their units, before extending them to the real
world.
References
Carley, K.M. (1996). Adaptive organizations: A comparison of strategies for
achieving optimal performance, in Proceedings of the 1996 International
Symposium on Command and Control Research and Technology (pp. 322330), June, Monterey, CA.
Carley, K.M. (1998). Organizational adaptation. Annals of Operations
Research, 75, 25-47.
Carley, K.M. (Unpublished manuscript). Validating computational models.
Carley, K.M., & Hill, V., 2001, Structural Change and Learning Within
Organizations, in Dynamics of organizations: Computational modeling and
organization theories, edited by A. Lomi and E. R. Larsen , MIT
Press/AAAI (Menlo Park, CA).
Effken, J.A., Brewer, B.B., Patil, A., Lamb, G.S., Verran, J.A., & Carley, K.M.,
2003, Using computational modeling to transform nursing data into
actionable information, Journal of Biomedical Informatics, 36, 351-361.
Effken, J., Verran, J., Lamb, G., Patil, A., Garcia-Smith, D., Shea, K., Brewer,
B., & Carley, K., in preparation, What does computational modeling add
to traditional data analysis?
Institute of Medicine, 2001, To err is human: Building a safer healthcare
system, National Academy Press (Washington, DC).
Institute of Medicine, 2004, Keeping patient safe: Transforming the work
environment of nurses, National Academy Press (Washington DC).
Kirkpatrick, S., Gelatt, C.D., & Vecchi, M.P., 1993, Optimization by simulated
annealing, Science, 220(4598), 671-680.
March, J.G., 1958. Organizations, Wiley (New York).
Mitchell, P., Ferketich, S. & Jennings, B.M., 1998. Quality Health Outcomes
Model, Image: Journal of Nursing Scholarship, 30(1), 43-46.
Rutenbar, R.A., 1989, Simulated Annealing Algorithms - an Overview. IEEE
Circuit Devices Magazine, 5(1):19-26.
Simon, H.A., 1954, Decision-making and administrative organizations, Public
Administration Review, 4, 316-331.