ARTICLE IN PRESS
Applied Ergonomics xxx (2008) 1–9
Contents lists available at ScienceDirect
Applied Ergonomics
journal homepage: www.elsevier.com/locate/apergo
The effects of clinical information presentation on physicians’ and nurses’
decision-making in ICUs
Anne Miller a, *, Carlos Scheinkestel b, Cathie Steele c
a
School of Nursing & Center for Peri-operative Research in Quality, Medical Arts Building, Vanderbilt University Medical Center, Suite 732, 1211 21st Avenue South,
Nashville, TN 37212, USA
b
Intensive Care Unit, Alfred Hospital, Commercial Road, Prahran 3181, Victoria, Australia
c
Australian Centre for Health Innovation, Alfred Hospital, Commercial Road, Prahran 3181, Victoria, Australia
a r t i c l e i n f o
a b s t r a c t
Article history:
Received 7 October 2007
Accepted 28 July 2008
Available online xxx
This research evaluated physicians’ agreement about patients’ diagnoses and nurses’ ability to detect
patient change using traditional charts (TC) and a work domain analysis-based paper prototype (PP) and
also sought to determine whether differences persisted when the PP was represented as an electronic
prototype (EP). Nurses’ change detection improved using the PP and EP compared to TC (PP vs TC,
t(df¼6) ¼ 1.94, p < 0.03; EP vs TC, t(df¼6) ¼ 3.14, p < 0.01) and detection was better using the EP compared
with the PP (t(df¼6) ¼ 5.96, p < 0.001). Physicians were more likely to agree about failed physiological
systems using the EP compared with the PP (t(df¼10) ¼ 3.14, p < 0.01), but agreement about patient
diagnoses was higher using the PP compared with the EP (t(df¼10) ¼ 2.23; p < 0.02). These results are
attributed to information grouping around physiological functions and the direct association of causeand-effect relations in clinical information design.
Ó 2008 Elsevier Ltd. All rights reserved.
Keywords:
Patient change detection
Diagnostic agreement
Clinical information systems
1. Introduction
Advocates of electronic clinical information systems (CIS) that
store and present patient and treatment related data assert that
these systems can reduce medical error (Bates et al., 1998, 1999;
Fraenkel et al., 2003; Kohn et al., 2000; Instit. of Med., 2001),
although this assertion has not been tested (Wears and Berg, 2005).
Traditional paper-based CIS have evolved informally in response to
changes in work practices and technology, but they have not been
evaluated either. Given paper’s widespread use, it has been
reasonable to assume that traditional paper chart designs that
include mixed graphs, tables and free-form text should be
successful when implemented in an electronic format. However,
reports and problems are emerging with electronic CIS information
designs that have been copied from paper antecedents; some of
these problems have the potential to create new types of errors
(Ash et al., 2004; Koppel et al., 2005; Morris et al., 2005; Wears and
Berg, 2005).
Berg et al. (1998) argue that many of the observed problems
are grounded in assumptions about hospital work. Commercially
available electronic CIS, for example, are typically implemented on
computer terminals that were intended for use by office workers.
Unlike office work, clinical decisions are collaborative activities that
* Corresponding author. Tel.: þ1 615 936 7349; fax: þ1 615 936 7373.
E-mail address: anne.miller@vanderbilt.edu (A. Miller).
depend on team members being able to develop common understandings from information they can all see (Klein, 2001; Endsley
et al., 2003). Small screen sizes make it difficult for team members
to analyse large amounts of information. Paper charts are typically
laid out on tables where separate pieces of information can be
bought together given the issues at hand. Transferred to computer
screens this capacity is lost as information is fragmented across
multiple screens, making relationships between pieces of information difficult to see. Finally, only one person can use single input
devices (eg a mouse) thus limiting collaborative interaction among
people and with information.
To address these deficiencies new frameworks and approaches
to information design have been developed based on the principle
that electronic CIS should reflect the work of its main users, that is,
nurses and physicians (Berg et al., 1998; Burns and Hajdukiewicz,
2004; Endsley et al., 2003; Gibson, 1979; Powsner and Tufte, 1994;
Rasmussen et al., 1994; Vicente, 1999; Xiao, 2005; Zhang et al.,
2002). Work domain analysis (WDA) is a framework for describing
the relationships among elements (eg information) in a work
environment in ways that usefully guide design (Rasmussen et al.,
1994: Vicente, 1999). A WDA (reported in Miller, 2004) was
completed for medical Intensive Care Units (ICUs) and used as
a basis for information design within the ecological interface design
approach (Burns and Hajdukiewicz, 2004). The research reported in
this paper evaluates the effects of two CIS designs developed using
the same WDA but displayed using different media. The overall
purpose of the work was to determine whether there were
0003-6870/$ – see front matter Ó 2008 Elsevier Ltd. All rights reserved.
doi:10.1016/j.apergo.2008.07.004
Please cite this article in press as: Miller, A., et al., The effects of clinical information presentation on physicians’ and nurses’ decision-making in
ICUs, Applied Ergonomics (2008), doi:10.1016/j.apergo.2008.07.004
ARTICLE IN PRESS
2
A. Miller et al. / Applied Ergonomics xxx (2008) 1–9
differences in physicians’ diagnostic agreement and nurses’ ability
to detect patient change when using traditional paper charts (TC)
versus a WDA-based paper prototype (PP), and if so, whether these
differences persisted when the PP was converted to an electronic
prototype (EP). Conversion to an EP involved dividing information
displayed in the PP. The EP’s design rationale was the same as the
PPs and information was divided using WDA principles. Two
experiments were completed as described in the following
sections.
2. Method
The two experiments were conducted in two major metropolitan tertiary teaching hospitals in different Australian cities in
different years; both are recognised research centres. The ICUs
were comparable in terms of patient demographics and severity of
illness, bed numbers, types of technology and procedures undertaken. In summary, the first experiment (Brisbane, 2002) tested
role-relevant aspects of ICU nurses’ and physicians’ performance
when using traditional paper charts (TC) compared to a paper
prototype (PP). Subsequent funding in 2006 allowed this experiment to be extended. In experiment 2 (Melbourne, 2006), ICU
nurses and physicians used an electronic prototype (EP) to complete
tasks instead of the PP used in experiment 1. In both experiments
the clinicians’ experimental tasks reflected their clinical roles:
nurses were asked to detect parameters associated with patient
change whereas physicians completed diagnostic tasks. Nurses’
data were analysed using signal detection, whereas the physicians’
data were analysed according to inter-physician agreement. Human
Research Ethics Committees in both hospitals approved the
protocol.
2.1. Participants
The experimental design was a within-participants, 2 (control
and prototype) 4 (four patient data sets) counterbalanced design
that required eight participants. Eight experienced bedside nurses
from each of the two hospitals (N ¼ 16) volunteered to participate.
Only four of the six possible physicians volunteered to participate
in experiment 1 (Brisbane); one physician was on sabbatical and
one was on recreational leave. As a consequence, counterbalancing
for the experiment 1 physicians was incomplete for the patient data
sets but was complete for the designs. Eight physicians volunteered
to participate in experiment 2 (Melbourne) (N ¼ 12).
2.2. Materials
Two sets of materials were used: 1) demographic surveys and
task response sheets, and 2) three information designs that served as
experimental stimuli. The demographic surveys elicited information
about participants’ age, academic qualifications, years of ICU clinical
experience, and hours of weekly patient contact. The nurses’ task
response sheet included tables with columns for writing the time
and day of patient change events, the parameters and variables
associated with the change event and the direction of the change
(deteriorated or improved). The physicians’ task response sheets
included space for listing the name of physiological systems that had
failed and for writing the patient’s current diagnoses.
As part of the PP design process ten ICU patient data sets had
been collected from archieved records with approval from a third
hospital. Data sets were selected in consultation with the ICU
medical director of the third hospital ICU who considered them to
be typical of complex ICU patients. The criteria used to select the
data sets included ICU length of stay of at least five days, failure in
more than three physiological systems, and data sets that represented a comprehensive range of ICU treatments. Four data sets
were randomly selected from the original ten and put aside for use
in evaluation studies; these data sets were not used in the design
process. Coming from a third hospital the patients represented in
the data sets were unknown to the participants. The profiles of the
randomly selected patients are provided in Table 1.
Each of the patient data sets were presented in the TC design
used in experiment 1 and 2, in the PP design used in experiment 1
and in the EP used in experiment 2. The three designs are summarised below (Table 2).
The TC design used in this study was based on charts used in
the hospital from which the patient data sets were collected. This
hospital’s chart designs were based on examples taken from other
hospitals and were considered to be representative of such
designs. The grouping of information was also similar to information groupings in the proprietary electronic CIS used in the
experiment 1 hospital and was similar to the mixed paper and
electronic system used at the time of the study in the experiment
2 hospital (Fig. 1).
The PP design, represented schematically in Fig. 2a and as an
annotated photograph in Fig. 2b was based on the previous WDA
(Miller, 2004). The overall information architecture mirrored the
functional whole-part dimension of the WDA. Four functions were
represented including 1) Neurological functions: Executive (eg
consciousness, etc.); sensory and perceptual (eg vision, response to
pain), and autonomic (eg temperature regulation) functions; 2)
circulation and communication functions with three levels: i) system
level variables, eg blood pressure; ii) organ level variables, eg heart
rate and iii) tissue level variables eg blood cell analyses and other
blood tissue profiles and 3) Two metabolic functions (fluid and
electrolyte–renal; and gas exchange–respiratory functions). The
functional arrangement in Fig. 2a schematic is understood to have
two advantages for human information processing: 1) horizontal
scanning across information at the same level of functional
description (eg across circulation, fluid and electrolyte and gas
exchange functions at the system, organ or tissue levels) allows
clinicians to quickly assess functional relationships, whereas 2)
vertical scanning within each function facilitates more detailed
analyses without loss of orientation to the data set as a whole
(Burns and Hajdukiewicz, 2004; Vicente, 1999).
Next, 62 individual elements like those in Fig. 2a were designed.
The principle guiding element design was that treatment variables
(eg drugs) should be grouped with associated goals and patient
parameters (eg blood pressure) to reveal cause-and-effect relationships. Moreover, relationships were shown over time to highlight therapeutic time lags and physiological dynamics. The
temporal dimension was represented using data compression and
vanishing-point perspective with the past receding into the
‘distance’. As Burns and Hajdukiewicz (2004) point out diagnoses
do not need to be represented in these types of displays because
higher order abstractions emerge from patterns in the data.
With 62 elements, the PP was too large to present on a single
standard-sized computer screen; we encountered the same limited
screen real-estate problems transferring the WDA paper design to
electronic form as encountered when transferring existing designs
to electronic form. Thus for the EP design, the displayed information was divided according to the four physiological functions of the
WDA used in the PP (ie neurological, circulatory, respiratory and
renal), but in the EP each function was presented on a separate
screen. In all other respects information grouping was the same as
the PP and thus the rationale for information grouping and
presentation remained the same. The only difference between the
PP and the EP was the separation of physiological functions across
different screens; physiological functions in the EP could no longer
be viewed at the same time. The EP was developed in MS
PowerpointÔ. Hyperlinks across the top of each screen allowed
participants to move between screens (Fig. 3).
Please cite this article in press as: Miller, A., et al., The effects of clinical information presentation on physicians’ and nurses’ decision-making in
ICUs, Applied Ergonomics (2008), doi:10.1016/j.apergo.2008.07.004
ARTICLE IN PRESS
A. Miller et al. / Applied Ergonomics xxx (2008) 1–9
3
Table 1
Profiles of the patient data sets that were represented in all designs
Pt Admission diagnosis (AD) and summary of past history (pH)
No.
Summary of admission interventions and outcomes
1
Required non-invasive respiratory support, pulmonary artery catheter monitoring,
inotropic drug support, DC cardio-version. Discharged to ward.
2
3
4
AD: collapse for investigation with hypotension, acute pulmonary oedema and
incontinence.
pH: myocardial ischaemia, myocardial infarction, congestive cardiac failure,
Amiodarone-induced thyrotoxicosis, interstitial lung disease. Aged: 80 yrs.
AD: post-coronary artery by-pass grafts 3, left ventricular outflow obstruction,
Klebsiella septicaemia and pneumonia, acute confusional state.
pH: myocardial ischaemia, hypertension. Aged: 69 yrs.
Complex post-operative course with septic shock and left ventricular outflow
obstruction. Required reintubation 2 due to bronchospasm and fatigue leading to poor
oxygenation. Mechanically ventilated. Pulmonary artery catheter monitoring while on
inotropes. Discharged to ward.
AD: purpura fulminans with septic shock (no organism identified).
Multi-system organ failure (neurological, cardiovascular, respiratory, coagulation,
haematological, hepatobiliary, renal, metabolic) requiring mechanical ventilation,
pH: haemochromatosis; splenectomy in childhood, hereditary spherocytosis,
haemolytic anaemia. No history of immunization post-splenectomy. Aged 53 yrs. sedation, pulmonary artery catheter monitoring, inotropic support, haemodialysis.
Discharged to ward.
AD: acute respiratory distress, preceded by chest pain.
Developed acute pulmonary oedema following acute myocardial infarction. Further
pH: recently diagnosed but not treated (at time of admission) multiple myeloma. complicated by Streptococcus pneumoniae septicaemia & multiple resistance
Staphylococcus aureus infection. Required inotropic drug support and pulmonary artery
Aged: 68 yrs.
catheter monitoring, mechanical ventilation, renal support. Transferred to another
hospital.
To limit variation between the PP and the EP information presented in the EP could not be manipulated. Information was displayed in a graph format, although the graphic elements used in the
PP could not be transferred to the onscreen environment without
reducing participants’ ability to read them. Instead, while
preserving cause-and-effect relations presented in the PP, twodimensional graphs were used that included multiple related
parameters displayed on single x–y axes. Limited screen size also
meant that only four instead of the six days of patient data presented in the PP were displayed.
2.3. Procedure
The purpose of experiment 1 was to test for differences in
nurses’ ability to detect patient change and in physicians’ agreement about patients’ states, depending on whether the TC or PP
were used. The purpose of experiment 2 was to determine whether
the results obtained in experiment 1 persisted when the PP was
implemented in the EP format. Thus participants in both experiments received the TC but participants in experiment 1 received
the PP and participants in experiment 2 received the EP. The
counterbalanced schedule in Table 3 was used to present the
designs. Each participant was tested four times – twice with the
traditional and twice with either the paper or the electronic
prototype designs (all four patient data sets).
Nurses and physicians had different experimental tasks.
According to Miller and Sanderson (2005), nurses’ dominant use
of clinical information is tactical monitoring and patient change
detection; highly detailed analyses over relatively short timeframes. Thus, nurses were asked to identify parameters associated with patient change. In contrast, physicians’ dominant use of
clinical information is strategic; including a broader range of
information over larger timeframes, and so they were asked: (1)
to name the organ systems that had failed for each patient, and
(2) to diagnose the patient’s condition on the last day of the
presented data; which was not the last day of the patients’
admissions. All information about patients’ diagnoses, other than
the patient’s admission history, had been omitted from all
designs. (It should be noted that strategic and tactical distinctions
are relative not absolute tendencies; highly experienced nurses
may take a more strategic orientation whereas inexperienced
physicians may tend to be more tactically focussed. Irrespective
of who assumes these roles, both high-level diagnoses and
detailed monitoring need to be supported.)
Following an initial briefing, each participant was given a 10 min
orientation to the TC or PP/EP design; no other training was provided.
Using a demonstration display, the briefing included a verbal
description of the display space including the location of key groups
of physiological and treatment information and the structure of the
display elements. During this time participants could ask questions
about the design’s structure and could manipulate the EP hyperlinks
if needed. The researcher then read the admission history for the first
patient, and participants had 20 min to complete their respective
tasks. The process was repeated for the second patient using the same
Table 2
Comparative summaries of each of the designs
Traditional paper charts (PC)
Paper prototype (PP)
Electronic prototype (EC)
Physical dimensions
Design composition
1 1.2 m
Composed of mixed graphs and tables
with free-form text commentary.
Grouping of information reflects the
original data source eg biochemistry
results presented on biochemistry
department forms, drugs presented on
drug chart.
30 38 cm Computer screen
Two-dimensional graphs with
annotations were presented on each of
four functional screens (neurological,
circulatory, respiratory, and renal). The
sensed information, treatment and goal
groupings from the PP were preserved.
Only the functional division of
information and its 2-dimensional
graph representation was different.
Temporal representation
One day of data is displayed on each
page. Five pages were presented per
patient.
1.2 1.4 m
62 Three-dimensional graphs with
annotations. Information grouping
reflects whole-part dimension of the
WDA including neurological,
circulatory, respiratory and renal
functions. Within each function patient
information (sensed information) is
grouped with relevant treatments
(effectors) and goals were represented
with the sensed information and
treatments.
Five days of data were presented on one
page per patient.
Five days of data were presented on
each of five screens representing five
physiological functions.
Please cite this article in press as: Miller, A., et al., The effects of clinical information presentation on physicians’ and nurses’ decision-making in
ICUs, Applied Ergonomics (2008), doi:10.1016/j.apergo.2008.07.004
ARTICLE IN PRESS
4
A. Miller et al. / Applied Ergonomics xxx (2008) 1–9
7. Regular drugs
Text and numerical
table
8. As needed (prn)
drugs
Text and numerical table
1. Haemodynamic
variables
Graphs and
numerical tables
9. Once only (stat)
drugs
Text and numerical
table
10. Intravenous
fluids
Numerical table
2. Respiratory
variables
(patient and
ventilator)
11. Haemofiltration
Numerical table
3. Continuous
infusions
Numerical table
12. Fluid balance
chart
Numerical table
4. Consciousness
Free-form text
5.Biochemistry
results
Numerical table
13. Specialist
reports
Free form text
6. Haematology
results
Numerical table
Scale approx 1:10mm
Fig. 1. Annotated photograph of traditional charts (TC). Accessible from: http://www.healthinnovation.com.au/Documents/Ergo/Figure1.pdf.
design, and was followed by a 10 min briefing in preparation for the
second design. The same process ensued for the remaining two
patient data sets. The same overall procedure was used in experiment
2, except that the participants were given 14 instead of 20 min to
complete tasks because four instead of six days of data were presented in the electronic prototype.
discriminate meaningful information from random variation. SDT
measures this form of discrimination using two values. d0 (d-Prime,
or sensitivity) is a participant’s normalized correct responses minus
his or her false positives, and b (beta, or response bias) is the ratio of
normalized correct responses to false positives. d0 and b were
calculated for each nurse when using the TC and the PP or EP.
2.4. Data collation
2.4.2. Diagnostic agreement
Continuity of patient care requires that strategic patient goals
continue across shifts. However, physicians must first reach
agreement about what the patient’s situation is; hence, physicians’
responses were analyzed using inter-participant agreement. This
approach was also taken because the patient data sets were
incomplete and because the physicians who would have been
asked to provide standard diagnoses met criteria for participation
in this study. Given that all participant physicians were experienced, all of their responses for each patient data set were pooled
and used as a standard inventory of responses. Physicians exposed
to the same patient data sets in the same design were paired. The
proportions of agreed inventory responses were then collated for
each pair using the traditional and prototype designs.
2.4.1. Detecting patient change
In consultation with physicians who did not participate in the
study, a patient change event was operationally defined as an
episode where: 1) at least one parameter deviated from a previous
state (not necessarily a state within the ‘normal’ range) within
a specifiable time range, ie if the timeframe over which the variable
changed from start to resolution was 5 h then any time nominated
within that frame was considered valid if the remaining criteria
were also met; 2) at least one parameter reached a value differing
by more than 20% of its preceding value; and 3) the new state was
sustained or required intervention. Using these three criteria
a coding scheme was developed for each day of each patient’s
admission represented in the designs. A ‘day’ was defined as the
period between 0001 and 2400 h.
Signal Detection Theory (SDT) was used to calculate scores for
each nurse (Green and Swets, 1966; Proctor and van Zandt, 1994).
SDT is a useful method to assess real-world detection situations
because it adjusts a person’s correct responses for their false
positive responses. Some people readily identify changes, that is,
they have a liberal response bias. Such people have a large number
of correct responses but also have more false positive responses. A
person with a conservative response bias may identify fewer
correct responses but will also identify less false positives. In
patient care, conservative response biases may lead to nurses
under-reporting patient change, whereas nurses with liberal
response biases may lead to over-reporting; both could lead to
patient harm. The ideal response is one where people correctly
3. Results
3.1. Participant characteristics
Table 4 summarises the demographic characteristics of experiment 1 and 2 participants. The profiles for physicians represent
a relatively homogenous sample.
The demographic and performance data were screened and
tested for effects related to different gender proportions in experiment 1 and 2 nurses, for differences in nurses’ years of experience
and their hours of patient contact, for patient data set effects and
for learning effects (the difference in performance between the first
and second presentations of a design). No statistically significant
Please cite this article in press as: Miller, A., et al., The effects of clinical information presentation on physicians’ and nurses’ decision-making in
ICUs, Applied Ergonomics (2008), doi:10.1016/j.apergo.2008.07.004
ARTICLE IN PRESS
A. Miller et al. / Applied Ergonomics xxx (2008) 1–9
5
a
Executive functions
Sensation &
perceptual functions
Day
1 Day 2
2. Information allocated to the
display space based on WDA
whole-part dimension
(physiological functions)
Autonomic
Functions
Day 3
Day 4
Gas exchange
Water & electrolytes
Circulation &
communication
System information
System information
System information
Organ information
Organ information
Organ information
Tissue information
Tissue information
Tissue information
Day 5
Day 6
3. Two partially annotated design
elements:
- light bars are sensed patient data
- dark bars (except pH) are
treatment related variables
Minute ventilation
PIP
Tidal volume
-patient
-ventilator
Flow
Day
1
Day 2
pH
Respiratory Rate
-patient’s rate
-set ventilator rate
Ventilator Mode
-Spontaneous
-CPAP
-SIMV
1. Parameter grouping from prior WDA
Miller, 2004)
Day 3
Day 4
Day 5
Element scale
is approximately
actual size
Day 6
b
4. Consciousness
7. All drugs and
devices
Distributed with
appropriate functions
2. Specialist reports
Distributed within
appropriate functions
1. Haemodynamic
variables
Graphs and numerical
tables
2. Respiratory
variables
(patient and
ventilator)
3. Fluid input
and outputs
Approx scale 1:10mm
Variations in background colour are a consequence of
the non-uniform reflection of light offplastic laminate that
was used to protect the quality of the prototypes during
the study.
5.Biochemistry results are
distributed within
appropiate functions
6. Haematology results
linked with related
haemodynamic variables
Fig. 2. a. Schematic representation of paper prototype (PP). b. Annotated photograph of WDA-based paper prototype (PP). Accessible from: http://www.healthinnovation.com.au/
Documents/Ergo/Figure2a_2b.pdf.
Please cite this article in press as: Miller, A., et al., The effects of clinical information presentation on physicians’ and nurses’ decision-making in
ICUs, Applied Ergonomics (2008), doi:10.1016/j.apergo.2008.07.004
ARTICLE IN PRESS
6
A. Miller et al. / Applied Ergonomics xxx (2008) 1–9
differences in performance were identified depending on nurses’
gender, experience, hours of patient contact, nor were patient or
learning effects observed.
parameters compared with nurses who used the PP (mean
d0 PP ¼ 1.94 0.24 vs EP ¼ 2.9 0.38: t(df¼6) ¼ 5.96, p < 0.001).
There were no significant differences between the response biases
for experiment 1 and 2 nurses using either prototype.
3.2. Patient changes detected by nurses
3.3. Diagnostic agreement among physicians
3.2.1. Experiment 1: paper prototype (PP) compared with
traditional charts (TC)
Two-tailed matched sample t-tests (Howell, 2002) were used to
test the hypothesis of no difference between the mean d0 and b for
changed parameter detection using the PP and TC designs. Nurses
identified more correct changed parameters (mean d0 PP ¼ 1.94 0.24 vs TC ¼ 1.71 0.24: t(df¼6) ¼ 2.4, p < 0.05), and reported fewer
false positives due to a more conservative response bias (mean
b PP ¼ 138.19 59.6 vs TC ¼ 76.88, 50.75: t(df¼6) ¼ 2.86, p < 0.05)
when using the paper prototype.
3.2.2. Experiment 2: electronic prototype (EP) compared with
traditional charts (TC)
One-tailed matched sample t-tests (Howell, 2002) were used to
test the hypothesis that changed parameter detection by nurses
remained significantly improved using the EP compared with TC. It
was also hypothesised that the more conservative response bias
observed when using the PP compared to the TC would also persist
when using the EP. Nurses using the EP identified more correctly
changed parameters (mean d0 EP ¼ 2.9 0.38 vs TC ¼ 2.35 0.45:
t(df¼6) ¼ 3.14, p < 0.01), and also reported fewer false positives due
to a more conservative response bias (mean b EP ¼ 204.51 68.08
vs TC ¼ 135.53 89.15: t(df¼6) ¼ 1.943, p < 0.05).
3.2.3. Experiment 1 and 2: paper prototype (PP) compared with
electronic prototype (EP)
Hypotheses of no difference between experiment 1 and 2 nurses
when using the prototypes were tested using a two-tailed Student’s
t-test for independent samples of equal variance (Howell, 2002).
Nurses using the EP were able to detect more correctly changed
3.3.1. Experiment 1: paper prototype (PP) compared with
traditional charts (TC)
Student’s t-tests of equal sample variance were used to test
hypotheses of no difference in agreed organ systems failure and
current state diagnoses, depending on whether the PP or TC were
used. There were no significant differences in agreement about
failed organ systems. However, physicians using the PP achieved
greater mean proportions of agreement about the patients’ current
state diagnoses than they achieved using the TC (mean
PP ¼ 0.59 0.07 vs TC ¼ 0.36 0.09: t(df¼3) ¼ 4.1; p < 0.01).
3.3.2. Experiment 2: electronic prototype (EP) compared with
traditional charts (TC)
Based on data from experiment 1, a two-tailed Student’s t-test
for matched samples (Howell, 2002) were used to test the
hypothesis of no difference in physicians’ mean proportions of
agreement about patient’s failed organ systems depending on use
of the EP or TC designs. Contrary to experiment 1 Physicians using
the EP achieved greater proportions of agreement about patients’
failed organ systems than they did using the TC (mean
EP ¼ 0.51 0.11 vs TC ¼ 0.36 0.14; t(df¼6) ¼ 3.14, p < 0.01). Also
contrary to experiment 1 there were no significant differences in
physicians’ mean proportions of agreement about patients’ current
diagnoses regardless of whether the EP or TC were used.
3.3.3. Experiment 1 and 2: paper (PP) compared with electronic
prototype (EP)
The data from experiment 1 and 2 were analysed using twotailed Student’s t-tests of no difference for independent samples of
Opening Page
Access to all functional systems is provided by
hyperlinks across the top of the page.
Hyperlinks are presented on all pages in the
prototype so that all functional systems could be
accessed from all other functional systems.
Where parameters were relevant to more than one
functional system hyperlinks from the parameter to
the related system(s) were also provided within the
screen.
Neurology
Circulation
Respiration
Top table presents data related to
executive and sensory/perceptual
functions in the patient.
The top graph presents heart rate.
Beneath the graph other indicators
(CO CI) as well as relevant electrolytes
(K, Ca, Mg) are presented with
replacement doses and cardioactive drugs.
The top graph presents oxygenation
indicators and associated treatment
options (FiO2).
Renal
The top two graphs present fluid
inputs. The third graph shows the
cumulative and cumulative daily
input/output balances.
Above this table any goals related
Acid-base indicators are represented
neurological function are listed.
below oxygenation and beneath this are Urine output is presented in the fourth
graph.
patient and mechanical ventilation
Drugs and drug doses intended to
parameters.
Beneath this are kidney related
effect executive or sensory functions Similarly, beneath heart rate are
blood pressure, related indicators and Below these are examination and chest electrolytes (Urea, creatinine, Cl, etc)
are presented in the Table below
drugs and beneath this are pulmonary x-ray reports with relevant drugs.
and drugs.
pressures where relevant.
Fig. 3. Schematic representation of the electronic prototype. Accessible from: http://www.healthinnovation.com.au/Research/ergo.htm.
Please cite this article in press as: Miller, A., et al., The effects of clinical information presentation on physicians’ and nurses’ decision-making in
ICUs, Applied Ergonomics (2008), doi:10.1016/j.apergo.2008.07.004
ARTICLE IN PRESS
A. Miller et al. / Applied Ergonomics xxx (2008) 1–9
Table 3
Counterbalanced experimental design
Part 1st Design
#
presentation
2nd Design
presentation
Experiment 1 and 2 nurses and
experiment 2 physicians
1
2
3
4
5
6
7
8
T (Pt 1, Pt 2)
T (Pt 3, Pt 4)
T (Pt 2, Pt 4)
T (Pt 1, Pt 3)
P (Pt 1, Pt 2)
P (Pt 2, Pt 4)
P (Pt 2, Pt 4)
P (Pt 1, Pt 3)
P
P
P
P
T
T
T
T
(Pt
(Pt
(Pt
(Pt
(Pt
(Pt
(Pt
(Pt
3, Pt 4)
1, Pt 2)
1, Pt 3)
2, Pt 4)
3, Pt 4)
1, Pt 2)
1, Pt 3)
2, Pt 4)
Experiment 1 physicians
1
2
3
4
T (Pt 1, Pt 2)
P (Pt 1, Pt 2)
P (Pt 1, Pt 2)
T (Pt 1, Pt 2)
P
T
T
P
(Pt
(Pt
(Pt
(Pt
3, Pt
3, Pt
3, Pt
3, Pt
4)
4)
4)
4)
T ¼ traditional design, P ¼ prototype design (paper for experiment 1 OR electronic
for experiment 2 participants); Pt ¼ patients 1, 2, 3 or 4.
equal variance. When using the EP physicians achieved greater
mean proportions of agreement about the patients’ failed organ
systems compared to when using the PP (mean EP ¼ 0.51 0.11 vs
PP ¼ 0.27 0.08: t(df¼10) ¼ 3.14, p < 0.01), and physicians using the
PP achieved higher mean proportions of agreement about the
patients’ current state diagnoses than they achieved using the EP
(mean PP ¼ 0.59 0.07 vs EP ¼ 0.31 0.2: t(df¼10) ¼ 2.23; p < 0.02).
4. Discussion
This study compared clinically relevant aspects of physicians’
and nurses’ performance when using traditional charts (TC)
compared with two versions of a WDA-based prototype design
(paper, PP, and electronic, EP). Findings suggest that as clinical
information on paper charts is transferred to electronic media,
traditional chart designs are unlikely to yield optimal physician/
nurse performance. Specifically, results indicate that the presentation of information affects diagnostic agreement among physicians and nurses’ ability to detect patient change events. New
approaches to design that integrate rather than fragment information would appear to improve these aspects of clinical performance. However, performance improvements were not uniform
across the professional groups; design changes that enhance ICU
nurses’ performance may adversely affect physicians or vice versa.
4.1. Professional differences
Physicians and nurses have different training regimes, different
accreditation, remuneration and organisational structures. In
Table 4
Participant profiles (percentages)
Number
Gender (%)
Age (%)
Years in current
position (%)
Years post-ICU
graduated (%)
Hours of work with
patients per week (%)
Male
Female
>20–29
30–39
>40
>3–5
6–10
>10
>3–5
6–10
>10
>20–29
30–39
>40
Experiment 1, 2002
Experiment 2, 2006
Nurses
Physicians
Nurses
Physicians
8
37
63
12
63
25
12
13
75
25
62
13
12
88
–
4
100
–
–
25
75
25
50
25
25
50
25
25
50
25
8
63
37
13
87
–
25
37
38
25
50
25
50
25
25
8
80
20
–
60
40
40
40
20
20
20
60
40
40
20
7
practice their roles are highly interdependent. They often share
information displays albeit for different reasons. Physicians are
primarily responsible for diagnoses, medium to longer-term care
planning and treatment prescription. Bedside nurses are primarily
responsible for identifying and reporting patient change and for
monitoring the effects of prescribed treatments that they are often
responsible for administering (Miller and Sanderson, 2005). This
study assessed the effects of information presentation on limited
but important aspects of healthcare team performance.
Supporting their change identification role (Miller and Sanderson, 2005), nurses were better able to identify changed parameters
correctly using either prototype compared to the traditional charts
(TC), and the electronic prototype (EP) better supported change
detection than did the paper prototype (PP). Effective patient
monitoring and change detection depend on nurses being able to
closely track patient variables in the presence or absence of treatment interventions. Both PP and EP prototypes directly linked these
variables, whereas the traditional design did not. The EP may have
further enhanced change detection performance because the
separation of data into the four functional areas helped nurses
narrow their scanning focus to details within specific functions.
This explanation could be tested using eye-tracking technology to
analyze the focus of nurses’ attention. Confirmation would provide
further evidence for the need to better link patient and treatment
variables in design.
The electronic (EP) and paper (PP) prototypes had opposite
effects on physicians’ diagnostic agreement. Physicians achieved
the highest levels of diagnostic agreement when using the PP;
performance using the EP degraded to no better than that when
using the TC. The diagnosis and care planning of ICU patients
typically involve assessment across multiple physiological functions. The integration of information in one display space as in the
PP may have enhanced patient assessments and inter-participant
agreement through one or more of the following mechanisms
(Endsley et al., 2003): 1) reduced short-term memory requirements
(short-term memory quickly becomes overloaded and degrades
when searching multiple screens), 2) improved search time and
efficiency may free cognitive resources that can be used for other
interpretive or planning tasks and 3) laid out in one space, complex
patterns in the information may be more readily apparent. The
advantage afforded by the integrated PP design was lost when it
was divided functionally in the EP.
The EP appeared to better support agreement among physicians
about organ systems that had failed. Like nurses’ change detection
tasks, identifying failed organ systems required a more detailed
assessment of a narrow range of functionally related information.
Dividing information according to physiological functions may
have better supported this task because each physiological system
could be assessed in isolation. However, while physicians were
better able to identify failed physiological systems using the EP,
they did not integrate these diagnoses overall and hence agreement
decreased.
4.2. Clinical implications
Continuity of patient care involves the transfer of information,
including goals and plans, from one person or group to another
across work shifts. Agreement about the meaning and significance
of patterns in treatment and patient variables is the basis for
agreement about the directions and goals of care. Information
designs that integrate rather than divide clinical information
appear to enhance diagnostic agreement whereas interfaces that
divide information appear to reduce it. The results of this study
suggest that information displays that enhance agreement among
individuals about current states may better support continuity of
care across personnel and across work shifts.
Please cite this article in press as: Miller, A., et al., The effects of clinical information presentation on physicians’ and nurses’ decision-making in
ICUs, Applied Ergonomics (2008), doi:10.1016/j.apergo.2008.07.004
ARTICLE IN PRESS
8
A. Miller et al. / Applied Ergonomics xxx (2008) 1–9
The results of this study also suggest approaches that may be used
to further assess the real-world effects of CIS. Nurses using both
prototypes had fewer false positives. False positives are problematic
for several reasons. First, for nurses and the physicians with whom
they work, false positives unnecessarily interrupt and divert attention
from important tasks while adding to overall work load. Second, false
positives may increase the risk of unnecessary interventions that can
adversely affect the quality and cost of care. Hence, false positives
attributable to poor information design may have direct and
measurable outcomes in the ICU. Pronovost et al. (2003) note that it is
often difficult to establish causal links between team-based interventions and common patient outcome measures, such as length of
patient stay. With further verification false positives may provide
a direct link between CIS implementation, interruptions and broader
indicators of care quality such as patient outcomes.
4.3. Implications for design
CIS are used by and need to support teams whose members have
different information uses. Physicians tend to use clinical information
to diagnose patient states and prescribe interventions. Nurses tend to
use information to monitor and respond to change in patient status.
These uses suggest that physicians need more global, integrated views
of information, whereas, nurses’ need detailed information views that
highlight cause-and-effect relationships; members of both professions use the same information but they use it for different reasons.
Thus functionality that allows team members to ‘zoom-in and out’ of
different levels of detail is needed in future CIS designs.
Goals represent physicians’ plans for patient management, and
they are the criteria by which nurses monitor cause-and-effect
relations. Clinical goals, therefore, connect tactical monitoring to
strategic care planning and are likely to be a key component in
maintaining continuity of patient care delivery. The integrated
presentation of information should ideally integrate goals directly
with treatment and patient-related information and should also
represent these relationships over time. Further investigations are
needed to determine how clinical goals are developed, tracked and
handed over from one work shift to the next and how they should
be represented in design.
The types of screens used to display clinical information also
need to be considered. Standard office-sized computer screens and
current graphic models do not allow the concurrent presentation of
large volumes of information across physiological functions, as was
possible in the paper prototype and has been the norm in many
traditional chart designs that were spread over medium sized
tables. This may mean moving towards much larger screens with
higher resolutions, and innovative presentations of clinical
information.
4.4. Limitations
The limited sample size in this study is common to studies
involving professional participants. Alternatively, the generalizability of results from a larger pool of typically students to
experienced practitioners is questionable. A ‘within subjects’
counterbalanced experimental design reduces this problem by
allowing multiple measures on single individuals. Learning
effects and individual differences are distributed evenly across all
experimental conditions and thus act as constants.
4.5. Conclusion
In conclusion, new approaches to clinical information design
can enhance nurses’ ability to detect patient change and physicians’
diagnostic agreement. The important aspects of design appear to be
the integration of information according to physiological functions
and the association of cause-and-effect relations. However, the
effects were not uniform. In this study differences in performance
resulted from constraints on design that were imposed by
computer screen sizes that are better suited to office work. Clearly,
software design is only one aspect of system design; platform
issues are also important and further research is urgently needed to
better optimise computer platforms for the collaborative work that
occurs in the ICU.
Acknowledgements
This research was jointly funded by the Australian Centre for
Health Innovation and the Alfred Hospital, Intensive Care Unit. The
authors gratefully acknowledge the participation of all medical and
nursing staff at the Princess Alexandra Hospital, Brisbane, the
Alfred Hospital, the Austin Hospital and the Epworth Hospitals in
Melbourne, Australia. Prof. Penelope Sanderson provided advice to
experiment 1 as the first author’s PhD supervisor, and Dr. Matthew
Weinger and Dr. Peter Buerhaus assisted in preparing this
manuscript.
References
Ash, J.S., Berg, M., Coiera, E., 2004. Some unintended consequences of information
technology in health care: the nature of patient care information systemrelated errors. J. Am. Med. Inform. Assoc. 11 (2), 104–112.
Bates, D.W., Leape, L.L., Cullen, D.J., Laird, N., Petersen, L.A., Teich, J.M., Burdick, E.,
Hickey, M., Kleefield, S., Shea, B., Vander Vliet, M., Seger, D.L., 1998. Effect of
computerized physician order entry and a team intervention on prevention of
serious medication errors. JAMA 280 (15), 1311–1316.
Bates, D.W., Teich, J.M., Lee, J., Seger, D., Superman, G.J., Ma’Luff, N., Boyle, D.,
Leape, L., 1999. The impact of computerized physician order entry on medication error prevention. J. Am. Med. Inform. Assoc. 6 (4), 313–321.
Berg, M., Langenberg, C., Berg, I., Kwakkernaat, J., 1998. Considerations for sociotechnical design: experiences with an electronic patient record in a clinical
context. Int. J. Med. Inform. 52 (1–3), 243–251.
Burns, C., Hajdukiewicz, J.R., 2004. Ecological Interface Design. CRC Press, Boca
Raton, FL.
Endsley, M.R., Bolte, B., Jones, D.G., 2003. Designing for Situation Awareness: An
Approach to User-Centered Design. Taylor & Francis, New York, NY.
Fraenkel, D.J., Cowie, M., Daley, P., 2003. Quality benefits of an intensive care clinical
information system. Crit. Care. Med. 31 (1), 120–125.
Gibson, J.J., 1979. The Ecological Approach to Visual Perception. Houghton Mifflin
Company, Boston.
Green, D.M., Swets, J.A., 1966. Signal Detection Theory and Psychophysics. Wiley,
New York.
Howell, D.C., 2002. Statistical Methods for Psychology, fifth ed. Duxbury, Pacific
Grove, CA.
Institute of Medicine, 2001. Crossing the quality chasm: a new health system for the
21st Century. National Academy Press, Washington, DC.
Klein, G., 2001. Features of team coordination. In: McNeese, M., Salas, E., Endsley, M.
(Eds.), New Trends in Cooperative Activities: Understanding System Dynamics
in Complex Environments. Human Factors and Ergonomics Society, Santa
Monica, CA.
Kohn, L.T, Corrigan, J.M., Donaldson, M.S., 2000. To err is human: building a safer
health system. National Academy Press, Washington, DC.
Koppel, R., Metlay, J.P., Cohen, A., Abaluck, B., Localio, A.R., Kimmel, S.E., Strom, B.L.,
2005. The role of computerised physician order entry systems in facilitating
medication errors. JAMA 293 (10), 1197–1203.
Miller, A., 2004. A work domain analysis framework for modelling intensive care
unit patients. Cogn. Tech. Work 6, 207–222.
Miller, A., Sanderson, P, 2005. Clinical information use by medical and nursing staff
in ICU: outcomes of a coded table analysis. Proceedings of the Human Factors
and Ergonomics Society 49th Annual Meeting, pp. 954–988.
Morris, C.J., Savelyich, B.S.P., Avery, A.J., Cantrill, J.A., Sheikh, A., 2005. Patient safety
features of clinical computer systems: questionnaire survey of GP views. Qual.
Safe Health Care 14 (4), 164–168.
Powsner, S.M., Tufte, E.R., 1994. Graphical summary of patient status. Lancet 344,
386–389.
Proctor, R., van Zandt, T., 1994. Human Factors in Simple and Complex Systems.
Allyn & Bacon, Boston, MA.
Pronovost, P., Berenholtz, S., Dorman, T., Lipsett, P.A., Simmonds, T., Haraden, C., 2003.
Improving communication in the ICU using daily goals. J. Crit. Care 18 (2), 71–75.
Rasmussen, J., Pejtersen, A.M., Goodstein, L.P., 1994. Cognitive Systems Engineering.
John Wiley & Sons Inc., New York.
Vicente, K., 1999. Cognitive Work Analysis: Toward Safe, Productive, and Healthy
Computer-Based Work. LEA, Mahwah, NJ.
Wears, R.L., Berg, M., 2005. Computer technology and clinical work: still waiting for
Godot. JAMA 293 (10), 1261–1263.
Please cite this article in press as: Miller, A., et al., The effects of clinical information presentation on physicians’ and nurses’ decision-making in
ICUs, Applied Ergonomics (2008), doi:10.1016/j.apergo.2008.07.004
ARTICLE IN PRESS
A. Miller et al. / Applied Ergonomics xxx (2008) 1–9
Xiao, Y., 2005. Artifacts and collaborative work in healthcare: methodological, theoretical and technological implications of the tangible. J. Biomed. Inform. 38, 26–33.
Zhang, Y., Drews, F.A., Westenskow, D.R., Forest, S., Agutter, J., Bermudez, J.C.,
Blike, G., Loeb, R., 2002. Effects of integrated graphic displays on situation
awareness in anaesthesiology. Cogn. Tech. Work 4, 82–90.
Dr Anne Miller was awarded her PhD in 2004 from the University of Queensland,
Australia where she also held the position of Coordinator of the Human Factors Online
Post-graduate Program. She is currently a post-doctoral research fellow with the
School of Nursing and the Center for Peri-operative Research in Quality at Vanderbilt
University Medical Center, USA. Dr Miller has extensive consulting experience in IT
enabled change management (PwC) and human–computer interaction (telecommunications). She was formerly a Registered Nurse specialising in intensive care nursing.
Assoc. Prof, Carlos Scheinkestel is the Director of the Department of Intensive
Care at the Alfred Hospital. He has been practising full time in intensive care
9
since 1987. His roles have included: Chairman of the Victorian Branch and
a member of the Board of Directors of the Australian and New Zealand Intensive
Care Society and appointment by the Victorian Department of Human Services to
the Ministerial Emergency and Critical Care Committee. He is a reviewer for
national and international journals and has been a clinical examiner for the
Colleges of Physicians, Anaesthetists and Emergency Medicine.
Ms Cathie Steele is the General Manager of the Australian Centre for Health
Innovation at The Alfred Hospital in Melbourne. The Centre supports the
development, testing and demonstration of systems and technologies that
improve patient safety and quality of care. Cathie has had many years experience
in health executive and management roles including physiotherapy, quality,
patient safety, risk management and strategic planning. She has held the position
of Clinical Associate Professor of Physiotherapy at Melbourne University, has
worked in health consultancy, and is a surveyor with the Australian Council on
Healthcare Standards.
Please cite this article in press as: Miller, A., et al., The effects of clinical information presentation on physicians’ and nurses’ decision-making in
ICUs, Applied Ergonomics (2008), doi:10.1016/j.apergo.2008.07.004