Detecting abnormal
vital signs on six
observation charts: An
experimental
comparison
Report prepared for the Australian Commission on
Safety and Quality in Health Care’s program for
Recognising and Responding to Clinical Deterioration
Mark S. Horswill, Megan H. W. Preece, Andrew Hill,
and Marcus O. Watson
May 2010
Detecting abnormal vital signs on six observation charts: An experimental comparison
Preface
Chief Investigator:
Dr Mark Horswill
School of Psychology
The University of Queensland
Project Manager:
Ms Megan Preece
School of Psychology
The University of Queensland
Principal Investigator:
Associate Professor Marcus Watson
Queensland Health
Skills Development Centre
Senior Consultant:
Mr Andrew Hill
School of Psychology
The University of Queensland
Corresponding author:
Dr Mark Horswill
m.horswill@psy.uq.edu.au
(07) 3346 9520
School of Psychology, The University of Queensland, St Lucia, QLD 4072, Australia
Acknowledgments
We would like to thank all of the interested parties who provided us with observation charts from
their hospital or area health service for inclusion the current study. We would also like to thank the
following people for providing the de-identified patient data used in the experiment: Rozemary
Karamatic from Royal Brisbane and Women’s Hospital; Lynette Adams from the Patient Safety
Centre, Queensland Health; Kevin McCaffery from the Royal Children’s Hospital, Brisbane; Imogen
Mitchell and Heather McKay from The Canberra Hospital, ACT Health; and Helen Ward and Aaron
Armstrong from The Prince Charles Hospital, Brisbane.
We thank David Collard (Information Division, Queensland Health) for adapting our draft versions of
the Adult Deterioration Detection System (ADDS) charts into a desktop publishing format
for use in the current experiment.
We would also like to thank Nicola Dunbar and Alexandra Sonsie from the Commission and Jillann
Farmer, Lynette Adams, and Kirsten Price from the Patient Safety Centre, Queensland Health, for
their support throughout the project. We thank Mark Wetton for his assistance with the Media
software. Thanks also to members of the Royal Brisbane and Women’s Hospital for their help in
recruiting participants, and to staff at the Skills Development Centre, Queensland Health for their
help in running the experiment.
Finally we thank the research assistants who helped to prepare materials and collect the data for the
current study: Melany Christofidis, Georgina Parker, Natasha Innes, Olivia Ng, John Manly, and Nonie
Finlayson.
2
Detecting abnormal vital signs on six observation charts: An experimental comparison
Contents
Preface ....................................................................................................................................... 2
Acknowledgments...................................................................................................................... 2
Executive Summary.................................................................................................................... 4
1.
Introduction ....................................................................................................................... 5
1.1 General background ................................................................................................... 5
1.2 Background to the project ......................................................................................... 6
1.3 The Adult Deterioration Detection System (ADDS) chart .......................................... 6
1.4 Rationale for the current study .................................................................................. 7
2.
Method .............................................................................................................................. 9
2.1 Participants................................................................................................................. 9
2.2 Materials .................................................................................................................... 9
2.3 Design and procedure .............................................................................................. 13
2.4 Data analysis............................................................................................................. 14
3.
Results .............................................................................................................................. 14
3.1 Participant characteristics ........................................................................................ 14
3.2 Error rates................................................................................................................. 14
3.3 Response times ........................................................................................................ 18
4. Discussion............................................................................................................................. 22
References ............................................................................................................................... 26
Appendix A: The ADDS chart with systolic blood pressure table ............................................ 27
Appendix B: The ADDS chart without systolic blood pressure table ....................................... 29
Appendix C: Single parameter track and trigger chart ............................................................ 31
Appendix D: Multiple parameter track and trigger chart ........................................................ 33
Appendix E: No track and trigger graphical chart .................................................................... 35
Appendix F: No track and trigger numerical chart .................................................................. 37
Appendix G: Questionnaire and Test For Novice Group ......................................................... 38
Appendix H: Questionnaire For Health Professional Group .................................................... 41
Appendix I: Summary sheet of background information ........................................................ 46
Appendix J: Participant Knowledge Test .................................................................................. 48
Appendix K: Participant Knowledge Test Answers .................................................................. 50
Appendix L: Testing manual for novice group ......................................................................... 51
Appendix M: Testing manual for health professional group ................................................... 57
3
Detecting abnormal vital signs on six observation charts: An experimental comparison
Executive Summary
Paper-based observation charts are the principal means of monitoring changes to patients’ vital
signs. There is considerable variation in the design of observation charts and a lack of empirical
research on the performance of different designs. This report describes the results of a study carried
out as part of a project funded by the Australian Commission for Safety and Quality in Health Care
and Queensland Health to investigate the design and use of observation charts in recognising and
managing patient deterioration, including the design and evaluation of a new adult observation
chart that incorporated human factors principles. The first phase of this project involved using a
procedure known as heuristic analysis to review 25 observation charts from Australia and New
Zealand. 1,189 usability problems, which could lead to errors in recording data and identifying
patient deterioration, were identified in the charts. The results from the heuristic analysis were used
to design a new chart (the Adult Deterioration Detection System [ADDS] chart) based on human
factors principles and current best practice.
The study described in this report involved an empirical comparison of six charts (two versions of the
ADDS chart, two existing charts rated as “well designed” in the heuristic analysis, one existing chart
rated as being of “average design”, and one existing chart rated as “poorly designed”). Novices
(individuals who were unfamiliar with using patient charts) and health professionals (doctors and
nurses) were recruited as participants. Each chart design was shown to each participant four times
displaying different physiological data with one abnormal vital sign (e.g. a high systolic blood
pressure), and four times displaying different normal physiological data. After memorising the
normal ranges for each vital sign, participants had to classify the physiological data on the charts as
“normal” or “abnormal”. Error rates (the proportion of trials where participants made an incorrect
normal/abnormal judgement) and response time (the time to read the chart and make the
judgement) were measured.
Results indicated that chart design had a statistically significant effect on both error rates and
response time, with the charts identified as having better design tending to yield fewer errors and
shorter decision times. Specifically, the two versions of the ADDS chart outperformed all the existing
charts on both metrics, where the other charts yielded between 2.5 and 3.3 times as many errors as
the ADDS chart. There was no significant difference between novices and health professionals in
error rates for any chart, but the health professionals were significantly faster than novices at
making their decisions for the charts rated as “average” and “poor”. There was no significant
difference between doctors and nurses on either of the two performance measures for any of the
charts.
These data indicate that differences in the design of observation charts have a profound impact on
chart users’ decisions regarding patients’ vital signs as well as the time it takes to make such
decisions. Based on the current data, it appears that the ADDS chart is significantly better at
signalling patient deterioration than other currently available charts.
4
Detecting abnormal vital signs on six observation charts: An experimental comparison
1.
Introduction
1.1 General background
Improving the recognition and management of patients who deteriorate whilst in hospital is a
priority both at the national and state level. The Australian Commission on Safety and Quality in
Health Care (ACSQHC) has launched a national program for ‘Recognising and Responding to Clinical
Deterioration’ (1). In parallel, Queensland Health’s Patient Safety Centre have released a strategy
options paper discussing gaps in the recognition and management of the deteriorating patient (2).
Changes in physiological observations or ‘vital signs’ commonly precede serious adverse events such
as cardiac or respiratory arrest, unplanned Intensive Care Unit (ICU) admission, or unexpected death
(3-8). Several studies report that derangements in vital signs are observable up to 48 hours before
the adverse event (3, 5, 6, 9). This suggests that if deterioration is recognised early and
appropropriately managed, then complications arising from delays could be reduced (e.g. morbidity,
unexpected ICU admissions, extended length of stays in hospital), and some serious adverse events
could potentially be avoided altogether (10-13).
Paper-based observation charts are the principal means of recording and monitoring changes to
patients’ vital signs. However, vital signs are not always correctly recorded or appropriately acted
upon (3, 6, 9, 10, 14). The design of the observation charts themselves may contribute to failures in
the ability of medical and nursing staff to record vital signs and recognise deterioration.
There is considerable variation in the design of observation charts in current use in Australia. They
vary in both the number and selection of vital signs monitored. Observation charts also exhibit
diversity in the way in which they display information. For instance, respiration rate may be
displayed on one chart as a row containing boxes in which to write the number of breaths taken by a
patient per minute at each time-point, while on another chart it may be plotted as a graph over
time. Finally, observation charts also vary in the degree to which they incorporate track and trigger
systems based on clinical criteria to help users recognise a deteriorating patient and respond
appropriately.
There is presently a lack of empirical research on the design and use of observation charts. In
Australia, observation charts tend to be designed at the local hospital or individual health service
area level, resulting in a nationwide duplication of effort (10). Some observation charts appear to
have been trialled in specific wards before full implementation or evaluated by means of a staff
survey. Rigorous empirical evaluation is lacking in most cases.
There are indicative findings that efforts to improve the design of observation charts can produce
benefits for patients, staff, and the hospital. In the United Kingdom, Chatterjee et al. carried out an
empirical evaluation of five observation charts in use at a district general hospital (15). They
reported that the design of the charts had a significant effect on the ability of staff to recognise
patient deterioration (with a detection rate as low as 0% for one vital sign), and that no single
existing chart was best for all vital signs. As a result, they designed and implemented a new chart
incorporating a track and trigger system. They found that there was a significant improvement in
5
Detecting abnormal vital signs on six observation charts: An experimental comparison
staff’s ability to recognise deterioration (all detection rates over 90%), after the re-design and
implementation of the new chart. Their new chart produced improvements in the detection of four
forms of deterioration, hypoxia (45% increase in detection), tachypnoea (41% increase in detection),
tachycardia (29% increase in detection), and fever (16% increase in detection). A recent Australian
project to improve the early detection of patient deterioration, which included improvements to
observation chart design (together with other interventions such as training), was found to produce
statistically significant gains in the frequency of recording vital signs, as well as decreasing unplanned
ICU admissions, decreasing the rate of cardiac arrests, and decreasing the rate of hospital deaths
(16).
1.2 Background to the project
The study described in this report was part of a research project carried out at The University of
Queensland and funded by the Australian Commission for Quality and Safety in Health Care and
Queensland Health. The aim of the project was to investigate the design and use of observation
charts in recognising and managing patient deterioration, including the design and evaluation of a
new adult observation chart that incorporated human factors principles. The initial phase of the
project was a systematic usability evaluation of the quality and extent of design problems in 25
existing observation charts (17). A total of 1,189 usability problems were identified in the
observation charts. Usability problems were identified as affecting the observation charts’ page
layout, information layout, recording of vital signs, integration of track and trigger systems, language
and labelling, cognitive and memory load, use of fonts, use of colour, photocopying legibility, and
night-time legibility. In compiling lists of the various usability problems present in the observation
charts, principles for producing a better designed observation chart were developed.
1.3 The Adult Deterioration Detection System (ADDS) chart
Using the information obtained from the heuristic analysis, a new chart was designed by combining
what were considered to be the best design features of existing charts (see reference 18 for details).
The chart was largely based on: (a) The Prince Charles Hospital chart (Brisbane, Queensland), which
in turn was based on the Compass chart developed at The Canberra Hospital, ACT Health, and (b) the
Children’s Early Warning Tool (CEWT) paedriatric chart developed at Royal Children’s Hospital
(Brisbane, Queensland). The new chart was named the Adult Deterioration Detection System (ADDS)
chart and incorporated the following features designed to minimize the design problems that might
lead to human error in both recording and interpreting patient data (see Appendices A and B to view
the two versions of the ADDS chart). Note that the key function of the ADDS chart was to detect
patient deterioration, rather than to act as a general observation chart.
The ADDS chart featured both a single parameter and a multiple parameter colour-coded
track and trigger system to facilitate the detection of deterioration. The single parameter
system (in which a medical emergency response was required when any single patient vital
sign was outside a given range) had the advantage of simplicity of use. The multiple
parameter system (in which vital signs were scored using a colour-coded key and scores
6
Detecting abnormal vital signs on six observation charts: An experimental comparison
were summed to give an overall indication of the patient’s condition) was potentially more
sensitive to deterioration and could lead to earlier detection of deterioration or fewer false
alarms (see reference 18 for further details and discussion of this issue).
Chart colours were chosen such that colour density correlated with the extent to which the
patient’s vital signs were outside the normal range (apart from being an intuitive
progression, this strategy would aid colour-blind users).
All information required for use (for example, the colour key, the medical emergency
criteria, and the actions to be taken when different levels of deterioration were detected)
was provided on the same page as the vital signs data. This was in order to reduce cognitive
load (for example, to avoid the user having to retain vital sign data in memory while turning
the page to access more information).
Terms and abbreviations used on the chart were selected in part based on the preferences
expressed among a large sample of health professionals.
Only vital signs considered to be the most important for detecting deterioration were
included on the chart. If additional information had been included, this less important
information would potentially compete with the more important information for the user’s
attention.
Each vital sign was presented as a separate graph. Many existing charts either displayed data
numerically (making it difficult to see data trends and hence making deterioration harder to
detect) or included graphs with multiple vital signs plotted on the same graph area
(increasing visual clutter, and potentially making deterioration harder to detect).
The most critical vital signs were placed towards the top of the page, as this is where users
would look first. Most existing charts did not follow this practice.
Scales were labelled on both the left and right of each graph and bold vertical lines were
placed every 3 columns. These features were designed to minimize the chance of users
reading from the wrong column or row.
There was space to record modifications to vital sign thresholds. This information was placed
so that it would be in view when a user first picked up the chart.
1.4 Rationale for the current study
The evidence from the heuristic analysis used to develop the ADDS chart was opinion-based and
therefore required empirical verification. Given that a key aim of the new chart was to minimise
errors in recognising patient deterioration, the present study was designed to investigate whether
the ADDS chart was successful in this regard, relative to a range of existing charts. Six charts were
7
Detecting abnormal vital signs on six observation charts: An experimental comparison
selected for empirical comparison: two versions of the ADDS chart and four charts in current use in
hospitals in Australia. The four existing charts were chosen on the basis of ratings from the heuristic
analysis: two of these charts were rated as “well designed”, one chart was rated as “average”, and
one chart was rated as “poorly designed”.
The objective of the experiment was to present actual patient data on each of the charts and ask
both novices (individuals unfamiliar with using patient charts) and health professionals whether any
of the vital signs were outside of pre-defined normal ranges for any timepoint. Their judgements
were then coded as correct or incorrect. It was hypothesized that the charts that were rated as being
better designed would yield fewer incorrect responses. It was also hypothesized that participants
would be faster to reach their decision for the better-designed charts. Finally, we hypothesized that
the new ADDS chart would perform at least on the same level as the charts rated as “well designed”,
if not better. We also planned to compare the performance of the novices with the health
professionals (for example, it could be that different chart designs favoured different groups) and,
within the health professional group, compare doctors and nurses.
The six charts involved in the comparison are described below.
Two versions of the ADDS chart
Two versions of the ADDS chart were included (see Appendices A and B). The first version included a
systolic blood pressure table to allow the patient’s usual systolic blood pressure to be taken into
account when deciding the normal range for this vital sign. The second version did not have this
table; instead, the normal range was based on the assumption that the patient’s usual systolic blood
pressure was 120 mmHg. The second version was potentially simpler to read than the first version,
but the first version was likely to yield a more accurate decision as to whether a patient’s blood
pressure was abnormal or not.
Two existing charts rated as “well designed” in the heuristic analysis
Two charts were chosen because they were rated as “well designed” and were currently being
widely used in Australia. Both of these charts used colour as part of a track and trigger system and
displayed nearly all of the vital signs as graphs. The first chart used a single parameter track and
trigger system, with two bandings of colour (yellow and red) to denote different levels of patient
deterioration outside the normal range (the chart can be viewed in Appendix C). With a single
parameter track and trigger system, if any vital sign was outside its normal range, this would be
signalled by the data point being recorded against a coloured background, indicating that action
should be taken. The second chart used a multiple parameter track and trigger system (the chart can
be viewed in Appendix D). With this system, different bandings of colour (based on the severity of
deterioration) were used to score each vital sign, and scores for key vital signs were summed to
provide a single number summarising the patient’s overall condition. This score was used to
determine the action that should be taken. The second chart (unlike the first) also used a systolic
blood pressure table to allow a patient’s usual systolic blood pressure to be taken into account.
8
Detecting abnormal vital signs on six observation charts: An experimental comparison
One chart rated as “average” in the heuristic analysis
One chart was chosen to represent a chart of “average” quality according to the heuristic analysis
(the chart can be viewed in Appendix E). The chart was monochrome and did not have a track and
trigger system. Three of the key vital signs were graphed (the rest were represented numerically).
One chart rated as “poor” in the heuristic analysis
One chart was chosen to represent a chart of “poor” quality according to the heuristic analysis (the
chart can be viewed in Appendix F). The chart was navy and light blue and did not have a track and
trigger system. All of the key vital signs were presented as numbers rather than as graphs.
2.
Method
2.1 Participants
A novice group (n = 44) and a professional group (n = 45) were recruited for the experiment. Novices
were recruited from members of The University of Queensland and were paid $20 for participating
in the study. Health professionals were recruited from staff of the Royal Brisbane and Women’s
Hospital and were paid $100 for participating in the study.
2.2 Materials
The following materials were developed for use in this study.
Questionnaire for novice participants
Before the experiment, novice participants completed a questionnaire asking for their age and sex,
as well as whether they had ever worked as a health professional and whether they had ever used a
hospital observation chart before. Note that there were other questions not analysed in this report.
After the experiment, participants were asked whether they had previously used a chart similar to
any used in the experiment. See Appendix G for the full questionnaire.
Questionnaire for health professionals
Before the experiment, the health professionals group were asked to complete a questionnaire that
recorded their occupation (nurse/doctor/other) and further details about their nursing or medical
role. They were asked the number of years that they had been registered, the postcode of their
institution, and where they spent the greatest proportion of time working
(ward/emergency/ICU/theatre/outpatient clinic/management etc). They were also asked to record
their age and sex. They were asked whether they used observation charts as part of their current
role and how frequently. They were also asked whether they recorded information into observation
charts and how frequently. Finally they were asked what training they had received in the use of
observation charts (none/read the instructions/informal/formal/other). Note that there were other
questions not analysed in this report. After the experiment, the health professionals completed the
9
Detecting abnormal vital signs on six observation charts: An experimental comparison
same post-experiment questionnaire as the novices (including asking whether they had previously
used a chart similar to the ones used in the experiment). See Appendix H for the full questionnaire.
Training videos
All participants viewed a number of training videos before commencing the experiment. Note that
all videos are available for viewing online (please contact m.horswill@psy.uq.edu.au for further
information).
Background video: The first video covered background information essential to understanding the
different vital signs and why they are monitored, as well as explaining the normal ranges for each
vital sign that were applicable to the experiment. Also explained were cut-off scores and early
warning scores. The novice and professional groups received different versions of the video. The
novice version (13 minutes 6 seconds) assumed no previous knowledge of vital signs and so included
basic introductory information (e.g. an explanation of patient deterioration and definitions and
explanations of different vital signs). The health professionals’ version (4 minutes 55 seconds) was a
truncated version of the novice video, where basic knowledge about vital signs was assumed (the
focus was on the normal ranges to be used in the experiment as well as explanations of cut-off
scores and early warning scores).
The normal ranges used in this study were taken from the track and trigger systems used by three of
the charts used in the study. The ranges are shown in Table 1.
Table 1: Normal ranges of vital signs used in this study.
Vital Sign
Normal Range
Respiratory rate
Between 9 – 20 breaths per minute
Oxygen Delivery
Patient is receiving oxygen at equal to or less than 1 L/min
Oxygen Saturation
Between 93 – 100%
Systolic Blood Pressure
Between 100 – 160 mmHg
Pulse
Between 50 – 100 beats per minute
Temperature
Between 36.1 – 37.9 Celsius
Consciousness
Patient is classified as “Alert”
Pain
Patient is in no pain
Chart videos: Training videos were created for each of the six charts used in the experiment. The
videos described how to read each chart, including interpreting cut-off scores or early warning
scores as appropriate.
Experiment instruction video: This short video described the procedure to be followed in the
experiment (see section 2.3 below).
Participant Knowledge Test
All participants completed a 10-item four-option multiple-choice examination testing key elements
of the information that they were required to have learned while watching the training videos.
Questions 1 to 8 asked about the normal ranges for the eight vital signs described in the videos. The
10
Detecting abnormal vital signs on six observation charts: An experimental comparison
final two questions asked for definitions of cut-off scores and early warning scores. The full
examination can be viewed in Appendix J, with answers given in Appendix K.
Patient data
De-identified patient data was collected from a number of locations (Royal Brisbane and Women’s
Hospital, Royal Children’s Hospital, The Canberra Hospital, The Prince Charles Hospital). For each
patient whose observations were used in the experiment, the data spanned at least 13 consecutive
time points (out of the six charts included in the experiment, the minimum number of given time
points that could be recorded was 13).
The cut-offs for normal ranges were taken from those provided as part of the track and trigger
systems on three of the charts in the study (the two ADDS charts and the multiple parameter track
and trigger chart, which were consistent with one another). Note that two of the other three charts
used did not provide cut-offs and the only other chart to have a track and trigger system (the single
parameter track and trigger system chart) was modified to bring the cut-offs in line with the other
charts. Any vital sign outside of these cut-offs was designated “abnormal”.
Forty-eight cases were obtained (cut-offs are consistent with the normal ranges presented in the
background training video and tested in the knowledge test):
3 cases of hypothermia (temperature below 36.1 C)
3 cases of fever (temperature above 37.9 C)
3 cases of hypotension (systolic blood pressure below 110 mmHg)
3 cases of hypertension (systolic blood pressure above 160 mmHg)
3 cases of bradycardia (heart rate fewer than 50 beats per minute)
3 cases of tachycardia (heart rate greater than 100 beats per minute)
6 cases of hypoxia (oxygen saturation less than 93%)
24 cases of normal data
For the abnormal cases, only one observation (across the 13 time points recorded for each case) was
abnormal for each patient. These abnormalities were restricted to one vital sign per patient to
standardise the amount of deterioration present in each case. That is, there was no patient for
whom more than one data point was outside the normal ranges (across all vital signs and across all
time points).
The patient data was modified in the following circumstances:
(1) If a data point was missing for a particular vital sign, the average of the data points from
the closest available time points before and after the missing value was inserted.
(2) For cases showing deterioration, if a particular vital sign became abnormal at more than
one time point, the data were adjusted so as to be outside the normal range at only one
time point (to ensure homogeneity across the abnormal cases).
(3) For six cases (five stable, one hypotension), we only had data for 12 out of the 13 time
points. In these cases, an extra time point was inserted between two existing values.
Data for this extra time point was generated by calculating the mean of the data from
11
Detecting abnormal vital signs on six observation charts: An experimental comparison
the time points before and after. For a seventh case (hypoxia), two extra sets of
observations were inserted in this way.
Preparation of charts
Six charts were evaluated in this study and are detailed in Table 2 (also described earlier).
Table 2: Descriptions of the six charts used in the experiment.
Chart
Description
1.ADDS chart with
A new chart (the ADDS chart) developed in an earlier phase of the
systolic blood pressure
current project, including both single parameter and multiple parameter
table
track and trigger systems. The format was A3 double-sided and the cutoffs for systolic blood pressure could take account of a patient’s usual
blood pressure (in this study, usual blood pressure was taken as 120
mmHg). The chart can be viewed in Appendix A.
2. ADDS chart without
A second version of the ADDS chart, which did not have systolic blood
systolic blood pressure
pressure cut-offs tailored to a participant’s usual blood pressure (it
table
assumed that usual blood pressure was 120 mmHg for all patients). The
chart can be viewed in Appendix B.
3. Multiple parameter
A double-sided A4 chart with a multiple parameter track and trigger
track and trigger chart
system. The cut-offs for blood pressure could take account of a patient’s
usual blood pressure (in this study, usual blood pressure was taken as
120 mmHg). The chart can be viewed in Appendix D.
4. Single parameter track A double-sided A3 chart with a single parameter track and trigger
and trigger chart
system, without systolic blood pressure cut-offs tailored to usual blood
pressure. The chart can be viewed in Appendix C.
5. No track and trigger
A double-sided monochrome A3 chart with no track and trigger system,
graphical chart
where some of the vital signs were plotted as graphs but others were
not. The chart can be viewed in Appendix E.
6. No track and trigger
A single-sided A4 chart with no track and trigger system and all vital sign
numerical chart
data written as numbers. The chart can be viewed in Appendix F.
The single parameter track and trigger chart was modified using Jasc Paint Shop Pro 8 where
necessary to bring its cut-offs in line with those shown on other charts (the modification resulted in
lower cut-offs than were originally used on this chart, so that any given vital sign would be more
likely to be classified as outside the normal ranges). This was crucial to the experiment to ensure
that the charts were being compared on the basis of their design and not their chosen cut-offs (that
is, without the modification, we would be unable to tell if resulting differences between charts were
due to differences in design or differences in cut-offs). The other five charts were not modified.
The 48 patient cases were plotted onto the six charts (6 x 48 = 288 charts in total). Data was plotted
by hand by a team of trained research assistants, with the Project Manager double-checking all
entries for errors.
12
Detecting abnormal vital signs on six observation charts: An experimental comparison
2.3 Design and procedure
The study was granted ethics approval by the Royal Brisbane and Women’s Hospital’s Human
Research Ethics Committee and by The University of Queensland’s School of Psychology Ethics
Review Officer. The novice group were tested at the University of Queensland and the health
professional group were tested at the Skills Development Centre, Royal Brisbane and Women’s
Hospital, Queensland, Australia.
Participants were tested one at a time in quiet room under good lighting by two researchers. First,
participants were asked to complete a consent form and the initial questionnaire.
All participants were shown the background video explaining the different vital signs and their
normal ranges (the explanation was truncated for health professionals). They were then given a
laminated handout summarising the cut-offs which they were required to memorise (see Appendix
I).
Participants’ knowledge of the normal ranges for each vital sign was tested using the Participant
Knowledge Test. If they obtained less than 10 out of 10 answers correct for the test, then they were
asked to review the information again and retake the test until they were able to answer all of the
questions correctly.
Following the test, participants were shown the training videos explaining each of the six charts. To
minimise the chances of one chart receiving an advantage over another (e.g., due to primacy or
recency effects), the chart training videos were shown in a different random order for each
participant using custom-made software. Finally, all participants were shown the video explaining
the experimental protocol.
Each participant completed 48 trials in which they viewed each of the 48 sets of patient data plotted
on one of the six charts. Each chart design was therefore viewed eight times, four times with normal
data and four times with abnormal data (for each chart design, these four cases contained
derangements in temperature, systolic blood pressure, pulse, and oxygen saturation). Each session
was recorded using a digital voice recorder to allow checking of responses at a later date.
The protocol for each trial was as follows. First, a chart with 13 sets of observations completed was
placed in front of the participant. A timer was started when the participant starting viewing the
chart. The participant had to indicate whether any of the vital signs were abnormal, or whether all of
the observations were normal (based on the normal ranges given in the instructions). If they
responded “abnormal”, then they had to say which vital sign was abnormal. Subsequently, the
remaining 47 charts were presented to the participant, one-at-a-time, and the same procedures
were followed. Note that, to prevent order effects, the 48 charts were presented in an order that
was individually randomised for each participant.
The testing session took approximately one and a half hours per participant. The full experiment
protocol and instructions can be viewed in Appendices L (novice group) and M (health professional
group).
13
Detecting abnormal vital signs on six observation charts: An experimental comparison
2.4 Data analysis
Participant responses (normal vs. abnormal) and response times for each chart were entered into
Microsoft Excel 2007. Data was then exported into SPSS for Windows version 17 for statistical
analysis.
3.
Results
3.1 Participant characteristics
The novice group (n = 44) consisted of both medical students (n = 6) and other members of The
University of Queensland recruited from a research participant pool (n = 38). The health
professionals (n = 45) consisted of both doctors and nurses recruited from the Royal Brisbane and
Women’s Hospital. Characteristics of the two groups are listed in Table 3.
Two participants were excluded (both novices) for having overall error rates greater than chance
(50%), given that they were unlikely to be following instructions. Note that these exclusions had no
effect on the pattern of results reported.
Table 3: Participant characteristics of the two groups.
Novices (n = 44)
Age in years
23.7 (4.4)a
Gender
Female: 70.5%
Male: 29.5%
Health profession
Years registered
-
Health professionals (n = 45)
32.6 (8.2)a
Female: 64.4%
Male: 35.6%
Doctor: 53.3%
Nurse: 46.6%
Doctor: 2.8 (2.3)a
Nurse: 9.8 (10.8)a
Note. aValues are mean (standard deviation).
3.2 Error rates
Overall error rates (the proportion of incorrect responses) are given in Figure 1.
14
Detecting abnormal vital signs on six observation charts: An experimental comparison
45
40
Percentage Errors
35
30
25
20
Novices
15
Professionals
10
5
0
ADDS with
ADDS
Single
Multiple No track andNo track and
SBP table without SBP parameter parameter
trigger
trigger
table
track and track and graphical numerical
trigger chart trigger chart
chart
chart
Figure 1: Percentage of errors in detecting abnormal vital signs for the six charts. Error bars are
standard errors of the mean.
A mixed-design analysis of variance was conducted with chart type as a repeated-measures
independent variable and participant group as a between-subjects independent variable. Mauchly’s
test of sphericity indicated no significant sphericity in the data (Mauchly’s W = 0.771, p = 0.076).
There was a significant main effect of chart type on error rates, F(5,435) = 42.08, p < 0.001.
However, there was no significant effect of participant group on error rates; that is, health
professionals made the same number of errors as novices overall, F(1,87) = 0.06, p = 0.807.
However, there was a significant interaction between chart type and participant group, F(5,435) =
2.33, p = 0.042, potentially indicating that the health professionals made proportionally fewer errors
on the monochrome charts (though see later for simple effects analyses of this interaction).
Pairwise comparisons between charts were carried out separately for novices and health
professionals using the Bonferroni-Holm method of correcting for multiple comparisons. Table 4
gives comparisons for the novices and Table 5 for the health professionals.
15
Detecting abnormal vital signs on six observation charts: An experimental comparison
Table 4: Pairwise comparisons between charts for error rates for the novice group, arranged in order
of effect size.
Comparison
Observed p
Numerical chart vs. ADDS without SBP table
No track and trigger numerical chart vs. ADDS with
SBP table
No track and trigger graphical chart vs. ADDS
without SBP table
No track and trigger graphical chart vs. ADDS with
SBP table
Single parameter track and trigger chart vs. ADDS
without SBP table
Multiple parameter track and trigger chart vs. ADDS
without SBP table
Single parameter track and trigger chart vs. ADDS
with SBP table
Multiple parameter track and trigger chart vs. ADDS
with SBP table
No track and trigger graphical chart vs. Single
parameter track and trigger chart
No track and trigger numerical chart vs. Single
parameter track and trigger chart
No track and trigger numerical chart vs. Multiple
parameter track and trigger chart
No track and trigger graphical chart vs. Multiple
parameter track and trigger chart
No track and trigger numerical chart vs. No track and
trigger graphical chart
Single parameter track and trigger chart vs. Multiple
parameter track and trigger chart
ADDS without SBP table vs. ADDS with SBP table
Note. SBP = systolic blood pressure.
16
Comparison
significant at
5% level?
< 0.001
< 0.001
Critical p
for
significance
at 5% level
0.003
0.004
< 0.001
0.004
Significant
< 0.001
0.004
Significant
< 0.001
0.005
Significant
< 0.001
0.005
Significant
< 0.001
0.006
Significant
< 0.001
0.006
Significant
< 0.001
0.007
Significant
< 0.001
0.008
Significant
< 0.001
0.010
Significant
< 0.001
0.013
Significant
0.57
0.017
Not significant
0.79
0.025
Not significant
0.92
0.050
Not significant
Significant
Significant
Detecting abnormal vital signs on six observation charts: An experimental comparison
Table 5: Pairwise comparisons between charts for error rates for the health professionals group,
arranged in order of effect size.
Comparison
Observed p
Comparison
significant at
5% level?
< 0.001
Critical p
for
significance
at 5% level
0.003
No track and trigger graphical chart vs. ADDS with
SBP table
No track and trigger numerical chart vs. ADDS with
SBP table
Multiple parameter track and trigger chart vs. ADDS
with SBP table
Single parameter track and trigger chart vs. ADDS
with SBP table
No track and trigger graphical chart vs. ADDS
without SBP table
No track and trigger numerical chart vs. ADDS
without SBP table
Multiple parameter track and trigger chart vs. ADDS
without SBP table
Single parameter track and trigger chart vs. ADDS
without SBP table
ADDS without SBP table vs. ADDS with SBP table
< 0.001
0.004
Significant
< 0.001
0.004
Significant
< 0.001
0.004
Significant
< 0.001
0.005
Significant
< 0.001
0.005
Significant
< 0.001
0.006
Significant
< 0.001
0.006
Significant
0.135
0.007
0.148
0.008
0.157
0.010
0.190
0.013
0.212
0.017
0.694
0.025
1.000
0.050
Not
significant
Not
significant
Not
significant
Not
significant
Not
significant
Not
significant
Not
significant
No track and trigger numerical chart vs. Single
parameter track and trigger chart
No track and trigger numerical chart vs. Multiple
parameter track and trigger chart
No track and trigger graphical chart vs. Multiple
parameter track and trigger chart
No track and trigger graphical chart vs. Single
parameter track and trigger chart
No track and trigger numerical chart vs. No track and
trigger graphical chart
Single parameter track and trigger chart vs. Multiple
parameter track and trigger chart
Note. SBP = systolic blood pressure.
Significant
Despite the significant interaction between chart type and novice/professional group differences
(examining Figure 1 there appears to be a trend indicating that professionals produced fewer errors
for the no track and trigger charts), there was no statistically reliable difference between novices
and professional participants for any one chart. This means that we cannot be confident in this
interpretation of the interaction. See Table 6 for pairwise comparisons with a Bonferroni-Holm
correction for multiple comparisons.
17
Detecting abnormal vital signs on six observation charts: An experimental comparison
Table 6: Comparisons between novice and health professionals’ error rates for the six charts,
arranged in order of effect size.
Chart (novice vs. professional comparison)
Observed p
Critical p for
Comparison
significance
significant at
at 5% level
5% level?
No track and trigger numerical chart
0.078
0.008
Not significant
No track and trigger graphical chart
0.099
0.010
Not significant
ADDS chart with SBP table
0.153
0.013
Not significant
ADDS chart without SBP table
0.465
0.017
Not significant
Multiple parameter track and trigger chart
0.795
0.025
Not significant
Single parameter track and trigger chart
0.942
0.050
Not significant
Note. SBP = systolic blood pressure.
There was no overall significant difference between the accuracy rates of doctors and nurses across
all charts, F(1,43) = 1.00, p = 0.324, nor for any chart individually (see Table 7).
Table 7: Comparisons between doctors and nurses’ error rates for the six charts, arranged in order of
effect size.
Chart (doctor vs. nurse comparison)
Observed p
Critical p for
Comparison
significance
significant at
at 5% level
5% level?
No track and trigger graphical chart
0.017
0.008
Not significant
ADDS chart with SBP table
0.187
0.010
Not significant
No track and trigger numerical chart
0.586
0.013
Not significant
Single parameter track and trigger chart
0.626
0.017
Not significant
Multiple parameter track and trigger chart
0.635
0.025
Not significant
ADDS chart without SBP table
0.663
0.050
Not significant
Note. SBP = systolic blood pressure.
3.3 Response times
Overall response times (regardless of whether responses were correct or incorrect) are given in
Figure 2.
18
Detecting abnormal vital signs on six observation charts: An experimental comparison
25
Response time in seconds
20
15
Novices
10
Professionals
5
0
ADDS with
ADDS
Single
Multiple No track andNo track and
SBP table without SBP parameter parameter
trigger
trigger
table
track and track and graphical numerical
trigger chart trigger chart
chart
chart
Figure 2: Response times in for the six charts. Error bars are standard errors of the mean.
A mixed-design analysis of variance was run with chart type as a repeated-measures independent
variable and participant group as a between-subjects independent variable. Mauchly’s test of
sphericity indicated a significant departure from sphericity in the data (Mauchly’s W = 0.068, p <
0.001) and so Greenhouse-Geisser estimates are reported for the degrees of freedom for all
repeated-measures effects.
There was a significant main effect of chart type on response times, F(2.056, 178.875) = 48.96, p <
0.001. Also, there was a significant effect of participant group on response times: Health
professionals were faster to reach a decision than novices, F(1,87) = 8.69, p = 0.004. There was also a
significant interaction between chart type and participant group: health professionals appeared to
be proportionally faster than novices with the no track and trigger charts compared with the
coloured charts, F(2.056, 178.875) = 3.86, p = 0.002.
Pairwise comparisons between charts were carried out separately for novices and health
professionals using the Bonferroni-Holm method of correcting for multiple comparisons. Table 8
gives comparisons for the novices and Table 9 is for the health professionals.
19
Detecting abnormal vital signs on six observation charts: An experimental comparison
Table 8: Pairwise comparisons between charts for response times for the novice group, arranged in
order of effect size.
Comparison
Observed p
No track and trigger numerical chart vs. ADDS
without SBP table
No track and trigger graphical chart vs. ADDS
without SBP table
Multiple parameter track and trigger chart vs. ADDS
without SBP table
No track and trigger graphical chart vs. ADDS with
SBP table
No track and trigger numerical chart vs. ADDS with
SBP table
Single parameter track and trigger chart vs. ADDS
without SBP table
No track and trigger numerical chart vs. Single
parameter track and trigger chart
Multiple parameter track and trigger chart vs. ADDS
with SBP table
No track and trigger graphical chart vs. Single
parameter track and trigger chart
No track and trigger numerical chart vs. No track and
trigger graphical chart
No track and trigger numerical chart vs. Multiple
parameter track and trigger chart
ADDS without SBP table vs. ADDS with SBP table
No track and trigger graphical chart vs. Multiple
parameter track and trigger chart
Single parameter track and trigger chart vs. ADDS
with SBP table
Single parameter track and trigger chart vs. Multiple
parameter track and trigger chart
Note. SBP = systolic blood pressure.
20
< 0.001
Critical p for
significance
at 5% level
0.003
Comparison
significant at
5% level?
Significant
< 0.001
0.004
Significant
< 0.001
0.004
Significant
< 0.001
0.004
Significant
< 0.001
0.005
Significant
< 0.001
0.005
Significant
< 0.001
0.006
Significant
< 0.001
0.006
Significant
< 0.001
0.007
Significant
0.001
0.008
Significant
0.001
0.010
Significant
0.007
0.010
0.013
0.017
Significant
Significant
0.010
0.025
Significant
0.199
0.050
Not
significant
Detecting abnormal vital signs on six observation charts: An experimental comparison
Table 9: Pairwise comparisons between charts for response times for the health professionals group,
arranged in order of effect size.
Comparison
Observed p
< 0.001
Critical p for
significance
at 5% level
0.003
Comparison
significant at
5% level?
Significant
No track and trigger numerical chart vs. ADDS
without SBP table
No track and trigger numerical chart vs. ADDS with
SBP table
Multiple parameter track and trigger chart vs. ADDS
with SBP table
No track and trigger graphical chart vs. ADDS
without SBP table
Multiple parameter track and trigger chart vs. ADDS
without SBP table
No track and trigger graphical chart vs. ADDS with
SBP table
No track and trigger numerical chart vs. Single
parameter track and trigger chart
Single parameter track and trigger chart vs. ADDS
without SBP table
No track and trigger numerical chart vs. Multiple
parameter track and trigger chart
No track and trigger numerical chart vs. No track and
trigger graphical chart
Single parameter track and trigger chart vs. ADDS
with SBP table
No track and trigger graphical chart vs. Single
parameter track and trigger chart
Single parameter track and trigger chart vs. Multiple
parameter track and trigger chart
No track and trigger graphical chart vs. Multiple
parameter track and trigger chart
ADDS without SBP table vs. ADDS with SBP table
< 0.001
0.004
Significant
< 0.001
0.004
Significant
< 0.001
0.004
Significant
< 0.001
0.005
Significant
< 0.001
0.005
Significant
< 0.001
0.006
Significant
< 0.001
0.006
Significant
< 0.001
0.007
Significant
< 0.001
0.008
Significant
0.001
0.010
Significant
0.003
0.013
Significant
0.034
0.017
0.244
0.025
0.477
0.050
Not
significant
Not
significant
Not
significant
Note. SBP = systolic blood pressure.
Overall, health professionals were significantly faster than novices at reaching a decision, F(1,87) =
8.69, p = 0.004. This difference was significant for the two no track and trigger charts but not the
coloured charts when a Bonferroni-Holm correction for multiple comparisons was applied (see Table
10).
21
Detecting abnormal vital signs on six observation charts: An experimental comparison
Table 10: Comparisons between novices and health professionals in response times for the six charts,
arranged in order of effect size.
Chart (novice vs. professional comparison)
Observed p
Critical p for
Comparison
significance
significant at
at 5% level
5% level?
No track and trigger graphical chart
0.001
0.008
Significant
No track and trigger numerical chart
0.007
0.01
Significant
Single parameter track and trigger chart
0.030
0.0125
Not significant
Multiple parameter track and trigger chart
0.032
0.017
Not significant
ADDS chart with SBP table
0.037
0.025
Not significant
ADDS chart without SBP table
0.229
0.050
Not significant
Note. SBP = systolic blood pressure.
There was no overall significant difference between the response times of doctors and nurses across
all charts, F(1,43) = 0.05, p = 0.824, nor for any chart individually (see Table 11).
Table 11: Comparisons between doctors and nurses for response times across the six charts,
arranged in order of effect size.
Chart (doctor vs. nurse comparison)
Observed p
Critical p for
Comparison
significance
significant at
at 5% level
5% level?
ADDS chart with SBP table
0.951
0.008
Not significant
ADDS chart without SBP table
0.516
0.010
Not significant
Single parameter track and trigger chart
0.556
0.013
Not significant
Multiple parameter track and trigger chart
0.688
0.017
Not significant
No track and trigger graphical chart
0.272
0.025
Not significant
No track and trigger numerical chart
0.488
0.050
Not significant
Note. SBP = systolic blood pressure.
4. Discussion
Improving the recognition and management of patients who deteriorate whilst in hospital is a
frequently cited priority for improving patient safety (1, 2). One way to improve the recognition and
management of deteriorating patients is to improve the design of paper-based adult observation
charts (where patient management can be potentially influenced if action plans are included on a
chart). The aim of the current study was to compare the ability of six observation charts to signal
patient deterioration.
De-identified patient data was presented on two versions of the new ADDS chart (designed to
conform to human factors principles), two existing charts rated as “well designed”, one existing
chart rated as being of “average design”, and one existing chart rated as “poorly designed”. Chart
novices and health professionals were recruited as participants. Participants had to classify the
22
Detecting abnormal vital signs on six observation charts: An experimental comparison
physiological data on the charts as “normal” or “abnormal”. Error rates and response times were the
main outcomes of interest.
It was hypothesized that the charts that were rated as being better designed would yield fewer
incorrect responses. It was also hypothesized that participants would be faster to reach their
decision for the better designed charts. Finally, we hypothesized that the new ADDS chart would
perform at least on the same level as the charts rated as “well designed”, if not better.
The results can be summarized as follows:
1. There were significant differences in both error rate and response time between the charts,
indicating that chart design influences both of these performance measures. This is the first study to
investigate both of these effects in novices and health professionals (a previous study by Chatterjee
and colleagues (14) only investigated the effect of chart design on error rates among non-novice
health professionals).
2. Error rates for both the novice and health professional groups were substantial, reaching 32.6%
for one chart (the no track and trigger numerical chart, both groups combined), where 50% errors
indicates chance performance. This is despite participants making their judgements under test
conditions without distractions, when they were aware that they were under scrutiny. Even the best
performing chart (the ADDS chart with systolic blood pressure table) yielded an error rate of 9.8%. It
is possible that under real world conditions (where individuals are faced with external pressures,
distractions, and a lack of scrutiny) error rates for some or all of the charts could be higher.
3. For the novice group, the error rates on the charts were grouped as follows: both versions of the
ADDS chart had similar error rates to one another but yielded fewer errors than both existing
coloured charts (the single parameter track and trigger chart and the multiple parameter track and
trigger chart), which did not differ from one another. All of the coloured charts with track and trigger
systems were superior to the no track and trigger charts, which did not differ from one another. This
ordering is largely consistent with the conclusions of the heuristic analysis from Phase 1 of this
project, which predicted that charts considered to be better designed would yield fewer errors (16).
For the health professional group, the error rates on the charts indicated that the ADDS charts were
superior to all four existing charts, which did not differ from one another.
4. The response time data yielded more pronounced differences between the charts, with
approximately the same rank ordering of best to worse. For novices, there were significant
differences in response time between all the charts except the two existing coloured charts (the
single parameter track and trigger chart and the multiple parameter track and trigger chart). The
ADDS chart without the systolic blood pressure table yielded the fastest decisions, followed by the
ADDS chart with the systolic blood pressure table, and the existing coloured charts. The no track and
trigger charts took the longest to interpret, with the no track and trigger numerical chart being the
slowest, consistent with predictions from the heuristic analysis. For the health professionals, the
rank ordering of charts by response time was the same as for novices, although there was no
significant difference between the two versions of the ADDS chart, between the two existing
23
Detecting abnormal vital signs on six observation charts: An experimental comparison
coloured charts, or between the no track and trigger graphical chart and the multiple parameter
track and trigger chart.
5. There was no evidence for any difference between the novices and the health professionals in
their ability to detect unstable vital signs, with both groups yielding the same, high error rates.
However, the health professionals were significantly faster at reaching their decisions when
confronted with the no track and trigger charts than were the novices. This suggests that the health
professionals had a speed advantage when the charts did not incorporate any track and trigger
systems (and participants were instead relying on the memorised cut-offs). It also potentially
indicates the presence of a speed-accuracy trade-off, where it is possible that the health
professionals might have been able to obtain lower error rates than the novices for the no track and
trigger charts if they had taken the same amount of time over their decisions. However a further
experiment (where viewing times are standardised) would be required to test this hypothesis.
6. There was no difference between doctors and nurses in either their ability to detect unstable vital
signs or their speed of reaching a decision.
Overall, the outcomes of this study indicate that the design of patient observation charts has
dramatic effects on both the ability of individuals to detect abnormal vital signs as well as the time
taken to make those judgements. Given that the detection of abnormal vital signs is critical to
patient safety, we are in the position of being able to make the strong assertion that the way that
observation charts are designed is likely to have a substantial impact on patient safety. For
example, the error rate for the worst performing chart was 3.31 times the error rate of the best
performing chart.
The results are broadly consistent with the results of the heuristic analysis in that the four existing
charts in the study emerged in more or less the rank order predicted. While it is not possible to
provide definitive statements about each of the design principles highlighted in the heuristic analysis
because the charts vary on more than one dimension, there are a number of indicative findings that
could be followed up in future studies. First, the coloured charts tended to outperform the no track
and trigger charts (where the coloured charts all had track and trigger systems incorporated into the
vital signs’ graphs while the no track and trigger charts did not). This supports the idea that coloured
track and trigger systems are beneficial. Second, the finding that the no track and trigger graphical
chart that outperformed the no track and trigger numerical chart in terms of decision time (if not
error rate) is consistent with the idea that graphed data is superior to numerical. However, a series
of additional experiments, involving charts that differ on one design feature only, would be required
to provide more convincing evidence about the utility of each of these design principles.
The results also indicated that the ADDS chart outperformed all the existing charts, even those that
were considered to be well designed in the heuristic analysis (namely the single parameter track and
trigger chart and the multiple parameter track and trigger chart). The single parameter track and
trigger chart yielded 2.50 times the error rate of the ADDS chart with the systolic blood pressure
table and the multiple parameter track and trigger chart had 2.46 times the error rate of the ADDS
chart with the systolic blood pressure table (both groups together). This indicates that, based on the
metrics used in the present study, the ADDS chart should be recommended as current best practice
24
Detecting abnormal vital signs on six observation charts: An experimental comparison
(bearing in mind that this recommendation may change depending on the results from studies
investigating other issues).
25
Detecting abnormal vital signs on six observation charts: An experimental comparison
References
1.
Australian Commission on Safety and Quality in Health Care. Recognising and responding to
clinical deterioration: Background paper: Australian Commission on Safety and Quality in Health
Care, Department of Health and Ageing, Commonwealth of Australia2009 Jun.
2.
Patient Safety Centre. Recognition and management of the deteriorating patient: Core
strategy options paper. Brisbane, Queensland, Australia: Patient Safety Centre, Queensland
Health2010.
3.
Franklin C, Mathew J. Developing strategies to prevent inhospital cardiac arrest: Analyzing
responses of physicians and nurses in the hours before the event. Critical care medicine
1994;22(2):189-91.
4.
Buist M, Bernard S, Nguyen TV, Moore G, Anderson J. Association between clinically
abnormal observations and subsequent in-hospital mortality: A prospective study. Resuscitation.
2004;62(2):137-41.
5.
Hillman KM, Bristow PJ, Chey T, Daffurn K, Jacques T, Norman SL, et al. Antecedents to
hospital deaths. Internal Medicine Journal. 2001;31(6):343-8.
6.
Goldhill DR, White SA, Sumner A. Physiological values and procedures in the 24 h before ICU
admission from the ward. Anaesthesia. 1999;54:529-34.
7.
McQuillan P, Pilkington S, Allan A, Taylor B, Short A, Morgan G, et al. Confidential inquiry
into quality of care before admission to intensive care. British Medical Journal. 1998;316:1853-8.
8.
Kause J, Smith G, Prytherch D, Parr M, Flabouris A, Hillman K. A comparison of antecedents
to cardiac arrests, deaths and emergency intensive care admissions in Australia and New Zealand,
and the United Kingdom-the ACADEMIA study. Resuscitation. 2004;62:275-82.
9.
Endacott R, Kidd T, Chaboyer W, Edington J. Recognition and communication of patient
deterioration in a regional hospital: A multi-methods study. Australian Critical Care. 2007;20(3):1005.
10.
Lim P. Recognition and management of the deteriorating patient. Technical discussion
paper: Patient Safety Centre, Queensland Health, 2009.
11.
Bristow PJ, Hillman KM, Chey T, Daffurn K, Jacques TC, Norman SL, et al. Rates of in-hospital
deaths and intensive care admissions: the effect of a medical emergency team. Medical Journal of
Australia. 2000;173:236-40.
12.
Bellomo R, Goldsmith D, Uchino S, Buckmaster J, Hart GK, Opdam H, et al. A prospective
before-and-after trial of a medical emergency team. Medical Journal of Australia. 2003;179(6):283-7.
13.
Bellomo R, Goldsmith D, Uchino S, Buckmaster J, Hart G, Opdam H, et al. Prospective
controlled trial of medical emergency team on postoperative morbidity and mortality rates. Critical
Care Medicine. 2004;32(4):916-21.
14.
Van Leuvan CH, Mitchell I. Missed opportunities? An observational study of vital sign
measurements. Critical Care & Resuscitation. 2008;10(2):111-5.
15.
Chatterjee MT, Moon JC, Murphy R, McCrea D. The "OBS" chart: An evidence based
approach to re-design of the patient observation chart in a district general hospital setting.
Postgraduate Medical Journal. 2005;81:663-6.
16.
Mitchell IA, McKay H, Van Leuvan C, Berry R, McCutcheon C, Avard B, et al. A prospective
controlled trial of the effect of a multi-faceted intervention on early recognition and intervention in
deteriorating hospital patients. Resuscitation. 2010.
17.
Preece M, Horswill M, Hill A, Karamatic R, Hewett D, Watson M. Heuristic Analysis of 25
Australian and New Zealand Adult General Observation Charts. Sydney, New South Wales, Australia:
Australian Commission on Safety and Quality in Health Care2009.
18.
Preece MHW, Horswill MS, Hill A, Watson MO. The development of the Adult Deterioration
Detection System (ADDS) chart. Sydney, New South Wales, Australia: Australian Commission on
Safety and Quality in Health Care, 2010.
26
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix A: The ADDS chart with systolic blood pressure table
Note: Chart is printed at A3 size double-sided.
27
Detecting abnormal vital signs on six observation charts: An experimental comparison
28
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix B: The ADDS chart without systolic blood pressure table
Note: Chart is printed at A3 size double-sided
29
Detecting abnormal vital signs on six observation charts: An experimental comparison
30
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix C: Single parameter track and trigger chart
Note: Chart is printed at A3 size double-sided and has been de-identified
31
Detecting abnormal vital signs on six observation charts: An experimental comparison
32
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix D: Multiple parameter track and trigger chart
Note: Chart is printed at A4 size double-sided and has been de-identified
33
Detecting abnormal vital signs on six observation charts: An experimental comparison
34
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix E: No track and trigger graphical chart
Note: Chart is printed at A3 size double-sided and has been de-identified
35
Detecting abnormal vital signs on six observation charts: An experimental comparison
36
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix F: No track and trigger numerical chart
Note: Chart is printed at A4 single sided -sided and has been de-identified.
37
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix G: Questionnaire and Test For Novice Group
Questions About Your Background
All responses are completely anonymous. Please answer the following questions as
honestly and accurately as possible. There are no right or wrong answers.
1. Your sex:
 Female
 Male
2. Your age:
_____ years
3. Are you colour-blind?
 Yes
 No
If you answered „Yes‟:
4. What type of colour-blindness do you have? (E.g. “red-green”)
_____________________________
5. Does your colour-blindness impact on your work or studies?
 Yes
 No
6. Do you wear glasses or contact lenses in order to read?
 Yes
 No
7. Was English your first language?
 Yes
 No
8. Did you complete the majority your professional training or schooling in
English?
 Yes
 No
38
Detecting abnormal vital signs on six observation charts: An experimental comparison
9. Please indicate the description-step that better reflects the psycho-physical
state you have experienced in the last ten minutes:
 = extremely alert
 = very alert
 = alert
 = rather alert
 = neither alert nor sleepy
 = some signs of sleepiness
 = sleepy, no effort to stay awake
 = sleepy, some effort to stay awake
 = very sleepy, great effort to keep awake, fighting sleep
10. Have you ever worked as a health professional (e.g. nurse)?
 Yes
 No
If you answered „Yes‟, please describe your work as a health
professional:
________________________________________________________
________________________________________________________
________________________________________________________
11. Have you ever used a Hospital Observation Chart before?
 Yes
 No
If you answered „Yes‟, please describe your experience with Charts:
________________________________________________________
________________________________________________________
________________________________________________________
39
Detecting abnormal vital signs on six observation charts: An experimental comparison
Post-Experiment Questions
1. Have you previously used a chart that was very similar to any of the
Observation Charts that were presented in the study?
 Yes
 No
If you answered „Yes‟, please tell the researcher so that they can show
you copies of the Charts, then you can indicate which ones you have
previously used.
2. Please indicate the description-step that better reflects the psycho-physical
state you have experienced in the last ten minutes:
 = extremely alert
 = very alert
 = alert
 = rather alert
 = neither alert nor sleepy
 = some signs of sleepiness
 = sleepy, no effort to stay awake
 = sleepy, some effort to stay awake
 = very sleepy, great effort to keep awake, fighting sleep
40
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix H: Questionnaire For Health Professional Group
Questions About Your Background
All responses are completely anonymous. Please answer the following questions as
honestly and accurately as possible. There are no right or wrong answers.
1. What is your occupation?
 Nurse
 Doctor
 Other health professional (please specify:
_____________________________________ )
2. Which of the following best describes your nursing role?










Enrolled Nurse
Nursing Assistant
Registered Nurse
Clinical Nurse
Nurse Unit Manager
Clinical Nurse Consultant
Nurse Educator
Nursing Director
Nurse Practitioner
Other (please specify:
____________________________________________________ )
3. Which of the following best describes your medical role?












Post-graduate year 1 (Intern)
Post-graduate year 2 and not in an accredited training program
Post-graduate year 3 and not in an accredited training program
Post-graduate year 4+ and not in an accredited training program
Post-graduate and in an accredited training program
Hospitalist
Career Medical Officer
Senior Medical Officer
Visiting Medical Officer
Staff Specialist
Senior Staff Specialist
Other (please specify:
____________________________________________________ )
41
Detecting abnormal vital signs on six observation charts: An experimental comparison
4. How many years have you been registered?
______ years
5. What is the postcode of your institution or place of work?
_______
6. At your institution, where do you spend the greatest proportion of your time
working?







Ward
Emergency
ICU
Theatre
Outpatient Clinic
Management/administration
Other (please specify:
____________________________________________________ )
7. Your sex:
 Female
 Male
8. Your age:
______ years
9. Are you colour-blind?
 Yes
 No
If you answered „Yes‟:
a. What type of colour-blindness do you have? (E.g. “red-green”)
_____________________________
b. Does your colour-blindness impact on your work?
 Yes
 No
42
Detecting abnormal vital signs on six observation charts: An experimental comparison
10. Do you wear glasses or contact lenses in order to read?
 Yes
 No
11. Was English your first language?
 Yes
 No
12. Did you complete the majority your professional training (e.g. BNurs, MB BS)
in English?
 Yes
 No
13. Please indicate the description-step that better reflects the psycho-physical
state you have experienced in the last ten minutes:
 = extremely alert
 = very alert
 = alert
 = rather alert
 = neither alert nor sleepy
 = some signs of sleepiness
 = sleepy, no effort to stay awake
 = sleepy, some effort to stay awake
 = very sleepy, great effort to keep awake, fighting sleep
43
Detecting abnormal vital signs on six observation charts: An experimental comparison
The next few questions are about your current use of Observation Charts.
14. Do you use Observation Charts as part of your current role?
 Yes
 No
15. How frequently do you use Observation Charts?







More than once a day
Once a day
More than once a week, but less than once a day
Once a week
More than once a month, but less than once a week
Once a month
Less than once a month
16. Do you record information in Observation Charts as part of your current
role?
 Yes
 No
17. How frequently do you record information in Observation Charts?







More than once a day
Once a day
More than once a week, but less than once a day
Once a week
More than once a month, but less than once a week
Once a month
Less than once a month
18. What training have you received in the use of Observation Charts? (tick all
that apply)





None
Read the instructions
Informal (e.g. by co-worker)
Formal (e.g. in-service or workshop)
Other (please specify:
____________________________________________________ )
44
Detecting abnormal vital signs on six observation charts: An experimental comparison
Post-Experiment Questions
3. Have you previously used a chart that was very similar to any of the
Observation Charts that were presented in the study?
 Yes
 No
If you answered „Yes‟, please tell the researcher so that they can show
you copies of the Charts, then you can indicate which ones you have
previously used.
For researcher:
__________________________________________________________
__________________________________________________________
__________________________
4. Please indicate the description-step that better reflects the psycho-physical
state you have experienced in the last ten minutes:
 = extremely alert
 = very alert
 = alert
 = rather alert
 = neither alert nor sleepy
 = some signs of sleepiness
 = sleepy, no effort to stay awake
 = sleepy, some effort to stay awake
 = very sleepy, great effort to keep awake, fighting sleep
45
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix I: Summary sheet of background information
Vital Signs Summary
Vital sign
Normal physiological range
Respiratory rate
9 to 20 breaths per minute
Oxygen saturation
93% to 100%
Oxygen delivery
rate
0 to 1 litre of oxygen per minute
Systolic blood
pressure
110 to 159 mmHg
Heart rate
50 to 99 beats per minute
Temperature
36.1 to 37.9 °C
Consciousness
Alert
Pain
No pain
46
Detecting abnormal vital signs on six observation charts: An experimental comparison
Scoring Summary
System
Definition
Cut-off scores
Cut-off scores compare a patient’s current observations
with a simple set of criteria for the vital signs (the “cutoffs”)
Early warning
scores
Early warning scores allow a patient’s current
observations to be weighted and combined into a single
total score
47
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix J: Participant Knowledge Test
Multiple Choice Questions
Please circle the most correct answer for the following questions:
1. For a healthy adult at rest, what is the normal range for their respiratory rate?
a. 1 to 8 breaths per minute
b. 9 to 20 breaths per minute
c. 21 to 32 breaths per minute
d. 33 to 40 breaths per minute
2. What is the normal range for oxygen saturation?
a. 69 – 76%
b. 77 – 84%
c. 85 – 92%
d. 93 – 100%
3. What is the normal range for oxygen delivery?
a. 0 to 1 litre of oxygen per minute
b. 1 to 2 litres of oxygen per minute
c. 2 to 3 litres of oxygen per minute
d. 3 to 4 litres of oxygen per minute
4. For a healthy adult at rest, what is the normal range for their systolic blood
pressure?
a. 10 to 59 mmHg
b. 60 to 109 mmHg
c. 110 to 159 mmHg
d. 160 to 209 mmHg
5. For a healthy adult at rest, what the normal range for their pulse?
a. 50 to 99 beats per minute
b. 100 to 149 beats per minute
c. 150 to 199 beats per minute
d. 200 to 249 beats per minute
48
Detecting abnormal vital signs on six observation charts: An experimental comparison
6. For a healthy adult sitting in a room with a normal temperature, what is the
normal range for their body temperature?
a. 32.3 to 34.1 °C
b. 34.2 to 36.0 °C
c. 36.1 to 37.9 °C
d. 38.0 to 39.8 °C
7. A healthy adult who is fully conscious is described as being what?
a. Alert
b. Responsive to voice only
c. Responsive to pain only
d. Unresponsive
8. A healthy adult at rest should be what amount of pain?
a. No pain
b. None to some amount of pain
c. None to a moderate amount of pain
d. None to a considerable amount of pain
9. Which of the following is a definition of Cut-Off Scores?
a. Cut-off scores are indicators of a physiological processes that are
essential for life
b. Cut-off scores compare a patient‟s current observations with a simple
set of criteria for the vital signs
c. Cut-off scores are a set of criteria for when to stop monitoring a
patient‟s vital signs
d. Cut-off scores allow a patient‟s current vital signs to be weighted and
combined into a single total score
10. Which of the following are features of Early Warning Scores?
a. They are calculated to signal early signs that a patient is deteriorating
b. They allow a patient‟s current vital signs to be weighted and combined
into a single total score
c. Early warning scores look at all the vital signs together, not just a single
vital sign in isolation
d. All of the above
49
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix K: Participant Knowledge Test Answers
MCQ Scoring Key
1b
2d
3a
4c
5a
6c
7a
8a
9b
10d
50
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix L: Testing manual for novice group
Recognition Study Testing Manual – Students
Very Important
We only have 1 copy of each of the charts. Make sure the
charts are kept in a high, safe place and that water bottles
are always placed on the floor. Also, apart from when
having to fill in various sheets, make sure the participant
does not have a pen when doing the experiment.
Preparation
Make sure that you are:
Ready and able to explain any part of testing to the participant
Clear and articulate when giving instructions
Aware of building rapport with the participant and setting them at ease, whilst
maintaining consistency in test administration and minimising your influence on
their responses
o It’s fine to chat to the participant before & after testing, but no talking during
testing please
Before testing, check that you have:
The participant number
The correct chart scoring sheet
Observation charts in the correct order for that participant
A Participant Information and Consent Form
A questionnaire (with the participant number filled in)
Laminated participant cut-off summary
MCQ scoring key
A Participant Debrief Sheet
Payments to Research Participants Form for paid participants
$20 for paid participants
Computer set up with the Instructions presentation in Media (with correct
parameter file for that participant)
o Megan will log in each morning
o Click Media on the desktop
o Click Deterioration Study 3.4 shortcut (looks like filmstrip) on the Desktop
o Click Run Experiment
51
Detecting abnormal vital signs on six observation charts: An experimental comparison
o The ‘Loading…’ window appears, click ‘Browse’. Under ‘Data’, select the
correct participant number and click ‘Open’
o Click ‘Load’ in the ‘Loading…’ window
o The ‘Subject identifier’ window appears, enter the participant’s number e.g.
“01”
o The first screen of the Background Information presentation will appear
Pens
Stopwatch
Voice recorder/mike
Turned off your mobile phone (even receiving messages on Silent can distract the
participant)
Be ready and waiting for the participant at least 10 minutes before the session is
scheduled to begin.
There will be 2 Research Assistants (RAs) running the sessions, one should be in charge
of interacting with the participant (i.e., explaining all instructions, handing them materials)
and the other should be in charge of preparing materials and collecting data (i.e., getting
forms ready for the other RA, setting up the voice recorder, timing & transcribing the
participant’s responses in the experiment). To avoid fatigue, you can alternate between
these roles across sessions.
Start of the Session
When the participant arrives, welcome them (thank them for taking the time to
participate) and invite them to come in and sit down at the computer. Introduce yourself
and explain the procedure.
“My name is ___________ and I am a research assistant for the School of
Psychology, thanks for coming today. We are conducting a study to identify which
patient observation chart design best assists health professionals in recognising
patient deterioration. First, you will fill in a background questionnaire. Second, you
will be shown patients’ vital signs displayed on 6 different observation chart designs.
This session will take up to 2 hours of your time – give or take. There are no trick
questions; you can take every task at face value. All we ask is that you try your best
on the tasks.”
Give the participant the Participant Information and Consent Form.
“The Participant Information and Consent Form explains what the study is about.
Your participation is completely voluntary and confidential, and you are free to
withdraw at any time. If you are happy with the information on the sheet, sign the
bottom of the second page. Please feel free to ask me any questions before we
begin.”
52
Detecting abnormal vital signs on six observation charts: An experimental comparison
Give the participant the Questions About Your Background sheet.
“First, can you fill in this questionnaire?”
When they finish the questionnaire, introduce the Background Information
presentation on the computer and show them the Participant Cut-off Summary briefly.
“Next, I would like you watch a presentation on the computer. It explains
important background information for the experiment. It’s about 35 minutes long. You
will have to answer 10 multiple choice questions at the end to check that you
understand the information. But you will be given this summary sheet at the end to
help you remember the important points. Feel free to adjust the volume of the
headphones”
Get the participant to wear the headphones and let them adjust the volume if they want
to. They will tell you when the presentation ends. While they are watching the presentation,
one RA should stay in the room (the other RA can take a break if they want, but must be
back before the instructions end).
After the participant has watched the presentation, prepare them for the Multiple
Choice Questions and give them the Participant Cut-off Summary to study, plus a pen and
paper.
“Soon, I’ll be giving you 10 multiple choice questions to check that you
understand the background information. But first, here’s a summary of the
information from the videos that you will need to know to pass the test. You can take
as long as you like now to study this information, and here’s a pen and paper in case
writing will help you memorise it all. When you’re confident that you know everything
on this summary sheet, let me know and I’ll give you the test to complete. After you
finish the test, I’ll compare your answers with the correct answers. If you get any of
the questions wrong, you will have to study the summary sheet again and re-attempt
all of the questions, because we can’t start the experiment until you can get 10 out of
10.”
When the participant says that they are ready, give them the Multiple Choice
Questions. Leave the computer as it is.
Once the participant is finished, check how many they answered correctly with the MCQ
scoring key, without showing the participant the scoring key. If the participant gets any
questions wrong, get them to look at the Cut-off Summary again (and then re-attempt all
questions on a blank form). Let the participant know how many they got wrong, but not
which questions.
Next, introduce the Experiment Instructions presentation on the computer.
“Next, I would like you watch a very short presentation on the computer. It
explains the instructions for the experiment”.
53
Detecting abnormal vital signs on six observation charts: An experimental comparison
Play the video. You re-activate the video by clicking in the top-left of the screen (there’s
a “secret button” here). Make sure the participant puts the headphones on again. If
something goes wrong in Media, you can play the back-up WMV file on the desktop.
Afterwards, ask the participant whether they understood the test instructions. If not,
play the test instructions again.
The Experiment
Make sure the desk is free of water bottles and pens.
Explain that we will be recording the experiment in case the scribe RA writes down the
wrong answers, the participant won’t be able to be identified and the data is only used for
preventing our errors.
“Now, I’m going to turn on this mike, this is just to record your responses during
the experiment in case the scribe misses something. We’re not using the recording
for any other reason”.
Make sure the mike is plugged in to the mains, and press the record button twice to
activate recording. The red light under the Play/Rec label should be red and not flashing.
Explain that for the experiment, 1 RA will give the participant the instructions and the
charts, whereas the other RA will be in the room to read out the correct chart number and
write down the participant’s answers. Tell the participant that it’s important that they
speak in a loud, clear voice throughout the experiment. Tell the participant not to damage
the charts as we only have 1 copy of each set of data.
Make sure that the participant has a clear tabletop in front of them. When the
participant is ready, begin the experiment. The “scribe” RA should read out the trial number
then the chart number for the trial and the “instructor” RA should confirm that they have
the correct chart before placing it in front of the participant (so that we can hear it on the
recording).
Place the 1st chart face-down in front of the participant. Instructor RA:
“This is the first trial. Ready, set, go!” Begin timing as you say, “Go!”
Scribe RA: When the participant says “Normal” or “Abnormal”, stop timing. Write down
their answer in correct column of the Chart Scoring Sheet, then write down the time in the
next cell MM:SS.
If a participant answers “Abnormal” Instructor RA should ask them which vital sign they
think is abnormal and the Scribe RA should write down their answer. If they answer this
question by pointing, the Instructor RA should say which vital sign the participant is pointing
to (like paraphrasing) so that the participant confirms it out loud.
Take the finished chart away from the participant as soon as possible.
54
Detecting abnormal vital signs on six observation charts: An experimental comparison
Throughout the experiment, always ask the participant to repeat anything that you did
not hear properly, whether you are the instructor or the scribe.
When you are ready, repeat the cycle.
After trials 16 and 32, offer the participant a break.
If a participant wants to stop during a trial (even though we’ve asked them not to do
this), stop timing them and take back or flip over the observation chart (so that they can’t
keep looking at the observations). When the problem is resolved, resume the trial by
flipping the chart over again on “Go!” and resume timing. Make a note on the sheet of
which trial this happened on and what the problem was. Note what the time (on the
stopwatch) was when the interruption occurred.
Post-experiment
After completing all 48 trials, give the participant the Post-Experiment Questions sheet.
If a participant has had previous experience with a chart, note which one(s) and how they
are familiar with them.
End of the Session
Thank the participant for participating and give them a copy of the Participant Debrief
Sheet.
“Thank you for participating in our study. The research question was whether
observation chart design can affect rates of recognising patient deterioration. The
research is explained in more detail on the Debrief Sheet. Do you have any
questions?”
Answer any questions that the participant might have. If the participant is interested in
the results, make a note of their email address and Megan will get back to them.
Paid participants: Give them $20, and get them to complete and sign the Payments to
Research Participants Form. It is very important that paid participants sign this form, or
else we cannot be reimbursed for the participant payment.
Afterwards
After testing, you must:
Check that there are enough consent forms, questionnaires, scoring sheets, debrief
sheets, and money for the next testing session
Enter the data from the questionnaire & experiment into the Excel spreadsheet
Back-up the voice recording (Megan’s computer & markhgrp)
Back-up the Excel spreadsheet (Megan’s computer & markhgrp)
55
Detecting abnormal vital signs on six observation charts: An experimental comparison
Store the consent form & questionnaire
Order of testing summary
1. Participant Information and Consent Form
2. Questions About Your Background sheet – pt to fill in
3. Background Information presentation on the computer (need correct pt ID) – pt to
watch wearing headphones
4. Participant Cut-off Summary (plus pen & paper) – pt to study
5. Multiple Choice Questions – pt to fill in & get all correct (MCQ scoring key)
6. Experiment Instructions presentation – pt to watch wearing headphones
7. The Experiment
a. Make sure the desk is free of water bottles and pens.
b. Turn on voice recorder
c. Scribe RA needs Chart Scoring sheet with correct pt ID, stopwatch, & pen
d. Instructor needs charts in the correct order for that participant
8. Post-Experiment Questions (pt to fill in)
9. Participant Debrief Sheet
10. Pay participant & get pt to sign Payments to Research Participants Form
56
Detecting abnormal vital signs on six observation charts: An experimental comparison
Appendix M: Testing manual for health professional group
Recognition Study Testing Manual – RBWH Staff
Very Important
We only have 1 copy of each of the charts. Make sure the
charts are kept in a high, safe place and that water bottles
are always placed on the floor. Also, apart from when
having to fill in various sheets, make sure the participant
does not have a pen when doing the experiment.
Preparation
Make sure that you are:
Ready and able to explain any part of testing to the participant
Clear and articulate when giving instructions
Aware of building rapport with the participant and setting them at ease, whilst
maintaining consistency in test administration and minimising your influence on
their responses
o It’s fine to chat to the participant before & after testing, but no talking during
testing please
Before testing, check that you have:
The participant number
The correct chart scoring sheet
Observation charts in the correct order for that participant
A Participant Information and Consent Form
A questionnaire (with the participant number filled in)
Laminated participant Cut-off Summary
MCQ scoring key
A Participant Debrief Sheet
Computer set up with the Instructions presentation in Media (with correct
parameter file for that participant)
o [reference to computer username and login removed]
o Click Deterioration Study on the desktop
o Click Deterioration Study 3.4 shortcut (looks like filmstrip) in that folder
o Click Run Experiment
o The ‘Loading…’ window appears, click ‘Browse’. Under ‘Data’, select the
correct participant number and click ‘Open’
57
Detecting abnormal vital signs on six observation charts: An experimental comparison
o Click ‘Load’ in the ‘Loading…’ window
o The ‘Subject identifier’ window appears, enter the participant’s number e.g.
“01”
o The first screen of the Background Information presentation will appear
Pens
Stopwatch
Voice recorder/mike
Turned off your mobile phone (even receiving messages on Silent can distract the
participant)
Be ready and waiting for the participant at least 10 minutes before the session is
scheduled to begin.
There will be 2 Research Assistants (RAs) running the sessions, one should be in charge
of interacting with the participant (i.e., explaining all instructions, handing them materials)
and the other should be in charge of preparing materials and collecting data (i.e., getting
forms ready for the other RA, setting up the voice recorder, timing & transcribing the
participant’s responses in the experiment). To avoid fatigue, you can alternate between
these roles across sessions.
Start of the Session
When the participant arrives, welcome them (thank them for taking the time to
participate) and invite them to come in and sit down at the computer. Introduce yourself
and explain the procedure.
“My name is ___________ and I am a research assistant for the School of
Psychology, thanks for coming today. We are conducting a study to identify which
patient observation chart design best assists health professionals in recognising
patient deterioration. First, you will fill in a background questionnaire. Second, you
will be shown patients’ vital signs displayed on 6 different observation chart designs.
This session will take up to 1 and a half hours of your time – give or take. There are
no trick questions; you can take every task at face value. All we ask is that you try
your best on the tasks.”
Give the participant the Participant Information and Consent Form.
“The Participant Information and Consent Form explains what the study is about.
Your participation is completely voluntary and confidential, and you are free to
withdraw at any time. If you are happy with the information on the sheet, sign the
bottom of the second page. Please feel free to ask me any questions before we
begin.”
Give the participant the Questions About Your Background sheet.
“First, can you fill in this questionnaire?”
58
Detecting abnormal vital signs on six observation charts: An experimental comparison
When they finish the questionnaire, introduce the Background Information
presentation on the computer and show them the Participant Cut-off Summary briefly.
“Next, I would like you watch a presentation on the computer. It explains
important background information for the experiment. It’s about 25 minutes long. You
will have to answer 10 multiple choice questions at the end to check that you
understand the information. But you will be given this summary sheet at the end to
help you remember the important points. Feel free to adjust the volume of the
headphones”
Get the participant to wear the headphones and let them adjust the volume if they want
to. They will tell you when the presentation ends. While they are watching the presentation,
one RA should stay in the room (the other RA can take a break if they want, but must be
back before the instructions end).
After the participant has watched the presentation, prepare them for the Multiple
Choice Questions and give them the Participant Cut-off Summary to study, plus a pen and
paper.
“Soon, I’ll be giving you 10 multiple choice questions to check that you
understand the background information. But first, here’s a summary of the
information from the videos that you will need to know to pass the test. You can take
as long as you like now to study this information, and here’s a pen and paper in case
writing will help you memorise it all. When you’re confident that you know everything
on this summary sheet, let me know and I’ll give you the test to complete. After you
finish the test, I’ll compare your answers with the correct answers. If you get any of
the questions wrong, you will have to study the summary sheet again and re-attempt
all of the questions, because we can’t start the experiment until you can get 10 out of
10.”
When the participant says that they are ready, give them the Multiple Choice
Questions. Leave the computer as it is.
Once the participant is finished, check how many they answered correctly with the MCQ
scoring key, without showing the participant the scoring key. If the participant gets any
questions wrong, get them to look at the Cut-off Summary again (and then re-attempt all
questions on a blank form). You can let the participant know how many they got wrong, but
not which questions.
Next, introduce the Experiment Instructions presentation on the computer.
“Next, I would like you watch a very short presentation on the computer. It
explains the instructions for the experiment”.
Play the video. You re-activate the video by clicking in the top-left of the screen (there’s
a “secret button” here). Make sure the participant puts the headphones on again. If
something goes wrong in Media, you can play the back-up WMV file on the desktop.
59
Detecting abnormal vital signs on six observation charts: An experimental comparison
Afterwards, ask the participant whether they understood the test instructions. If not,
play the test instructions again.
The Experiment
Make sure the desk is free of water bottles and pens.
Explain that we will be recording the experiment in case the scribe RA writes down the
wrong answers, the participant won’t be able to be identified and the data is only used for
preventing our errors.
“Now, I’m going to turn on this mike, this is just to record your responses during
the experiment in case the scribe misses something. We’re not using the recording
for any other reason”.
Make sure the mike is plugged in to the mains and there’s a red light for the 2CH
recording option, and press the record button twice to activate recording. The red light
under the Play/Rec label should be red and not flashing.
Explain that for the experiment, 1 RA will give the participant the instructions and the
charts, whereas the other RA will be in the room to read out the correct chart number and
write down the participant’s answers. Tell the participant that it’s important that they
speak in a loud, clear voice throughout the experiment. Tell the participant not to damage
the charts as we only have 1 copy of each set of data.
Make sure that the participant has a clear tabletop in front of them. When the
participant is ready, begin the experiment. The “scribe” RA should read out the trial number
then the chart number for the trial and the “instructor” RA should confirm that they have
the correct chart before placing it in front of the participant (so that we can hear it on the
recording).
Place the 1st chart face-down in front of the participant. Instructor RA:
“This is the first trial. Ready, set, go!” Begin timing as you say, “Go!”
Scribe RA: When the participant says “Normal” or “Abnormal”, stop timing. Write down
their answer in correct column of the Chart Scoring Sheet, then write down the time in the
next cell SS:MsMs.
If a participant answers “Abnormal” Instructor RA should ask them which vital sign they
think is abnormal and the Scribe RA should write down their answer. If they answer this
question by pointing, the Instructor RA should say which vital sign the participant is pointing
to (like paraphrasing) so that the participant confirms it out loud.
Take the finished chart away from the participant as soon as possible.
60
Detecting abnormal vital signs on six observation charts: An experimental comparison
Throughout the experiment, always ask the participant to repeat anything that you did
not hear properly, whether you are the instructor or the scribe.
When you are ready, repeat the cycle.
After trials 16 and 32, offer the participant a break.
If a participant wants to stop during a trial (even though we’ve asked them not to do
this), stop timing them and take back or flip over the observation chart (so that they can’t
keep looking at the observations). When the problem is resolved, resume the trial by
flipping the chart over again on “Go!” and resume timing. Make a note on the sheet of
which trial this happened on and what the problem was. Note what the time (on the
stopwatch) was when the interruption occurred.
Post-experiment
After completing all 48 trials, give the participant the Post-Experiment Questions sheet.
If a participant has had previous experience with a chart, note which one(s) and how they
are familiar with them.
Then, give the participant another copy of the Multiple Choice Questions to do 1 more
time. They don’t need to repeat it if they get any wrong (checks that they remembered
all/most of the ranges for the duration of the experiment).
End of the Session
Thank the participant for participating and give them a copy of the Participant Debrief
Sheet.
“Thank you for participating in our study. The research question was whether
observation chart design can affect rates of recognising patient deterioration. The
research is explained in more detail on the Debrief Sheet. Do you have any
questions?”
Answer any questions that the participant might have. If the participant is interested in
the results, make a note of their email address and Megan will get back to them.
Afterwards
After testing, you must:
Check that there are enough consent forms, questionnaires, scoring sheets, and
sheets for the next testing session
Enter the data from the questionnaire & experiment into the Excel spreadsheets
Back-up the voice recording (testing computer/the external hard drive)
61
Detecting abnormal vital signs on six observation charts: An experimental comparison
o Plug mike into USB on computer, on mike: Menu, scroll down to USB, “enter”
USB, then Storage, and it should go to USB –PC mode & will be recognised on
the computer.
o Find FOLDER01 and copy the files across (then rename according to
participant no.), note that these files are big and can take awhile if you have
several to back up
Back-up the Excel spreadsheet (testing computer/Melany’s laptop/external hard
drive)
Store the consent form & questionnaire
Order of testing summary
11. Participant Information and Consent Form
12. Questions About Your Background sheet – pt to fill in
13. Background Information presentation on the computer (need correct pt ID) – pt to
watch wearing headphones
14. Participant Cut-off Summary (plus pen & paper) – pt to study
15. Multiple Choice Questions – pt to fill in & get all correct (MCQ scoring key)
16. Experiment Instructions presentation – pt to watch wearing headphones
17. The Experiment
a. Make sure the desk is free of water bottles and pens.
b. Turn on voice recorder
c. Scribe RA needs Chart Scoring sheet with correct pt ID, stopwatch, & pen
d. Instructor needs charts in the correct order for that participant
18. Post-Experiment Questions & MCQ again (pt to fill in)
19. Participant Debrief Sheet
20. Pay participant & get pt to sign Payments to Research Participants Form
62