Brief Report
I
RELIABILITY OF
THE BEDSIDE SHIVERING
ASSESSMENT SCALE
NTERRATER
By DaiWai M. Olson, RN, PhD, CCRN, Jana L. Grissom, RN, CCRN, Rachel A.
Williamson, BS, Stacey N. Bennett, RN, MSN, ACNP, Steven T. Bellows, BE, and
Michael L. James, MD
CNE
1.0 Hour
Notice to CNE enrollees:
A closed-book, multiple-choice examination
following this article tests your understanding of
the following objectives:
1. Identify the bedside assessment that can be
used in scoring a patient’s shivering.
2. Describe the components of the assessment
tool in evaluating a patient’s shivering.
3. Discuss the benefits and limitations of the tool
and research study used for assessing a
patient’s shivering.
To read this article and take the CNE test online,
visit www.ajcconline.org and click “CNE Articles
in This Issue.” No CNE test fee for AACN members.
©2013 American Association of Critical-Care Nurses
doi: http://dx.doi.org/10.4037/ajcc2013907
70
Background Since its early development, the Bedside Shivering
Assessment Scale (BSAS) has had only initial psychometric
testing. Before this instrument is incorporated into routine
practice, its interrater reliability should be explored in a diverse
group of practitioners.
Methods This prospective nonrandomized study used a panel
of 5 observers who completed 100 paired assessments.
Observers independently scored patients for shivering by
using the BSAS. Kappa statistics were determined by using
SAS version 9.4 with BSAS scores treated as ordinal data.
Results A weighted kappa value of 0.48 from 100 paired observations of 22 patients indicates moderate agreement of the
BSAS scores. Most of the BSAS scores were 0 or 1; dichotomizing shivering as little or no shivering versus significant
shivering resulted in a kappa of 0.66 (substantial agreement).
No relationship was found between timing of assessment or
the role of the practitioner and the likelihood of both observers
assigning the same BSAS score.
Conclusion The BSAS has adequate interrater reliability to be
considered for use among a diverse group of practitioners.
(American Journal of Critical Care. 2013;22:70-75)
AJCC AMERICAN JOURNAL OF CRITICAL CARE, January 2013, Volume 22, No. 1
www.ajcconline.org
T
herapeutic temperature management (TTM) is being instituted with increasing
frequency in patients after cardiac arrest and acute brain injury.1-5 Preclinical data
and results of clinical studies support the use of hypothermia and fever reduction
to reduce secondary brain injury.4,6-8 However, authors cite shivering as either a
complication of TTM or a hurdle to overcome, often functioning as a barrier to
implementation of TTM.9-13
As an attempt to maintain core temperature at
some level set by the hypothalamus, shivering is
known to induce increased intracranial pressure,
increased metabolic demand, and asynchrony with
mechanical ventilation.14-17 No consensus has been
reached on the impact, incidence, or best practice
for treating shivering during TTM. Although multiple publications9,10,18-21 describe modes of treating
shivering during TTM, adequately exploring the
impact of shivering requires a valid metric to score
shivering episodes.
The Bedside Shivering Assessment Scale (BSAS)
was developed to standardize scoring of shivering.13
This 1-item, 4-level tool uses observation and palpation to score shivering. The BSAS requires raters to
observe the patient for 2 minutes; this includes
visual inspection as well as palpating the neck, thorax, arms, and legs. The scale is independently associated with the hypermetabolic index: increased
oxygen consumption, resting energy expenditure,
and carbon dioxide production.13 Additionally,
interobserver reliability is high in patients being
actively cooled after stroke.22 Thus, the BSAS has
potential utility as a clinical and research tool for
the evaluation of shivering in the setting of TTM.
It is important to evaluate the interrater reliability of the BSAS at multiple institutions and
among multiple members of the care team. Therefore, the purpose of this study is to evaluate the
interrater reliability of the BSAS in a diverse group
About the Authors
DaiWai M. Olson is an assistant professor of medicine
/neurology at Duke University Health System in Durham,
North Carolina. Jana L. Grissom is a staff nurse, level IV,
at Duke University Health System. Rachel A. Williamson
is a medical student, level I, at Duke University in Durham,
North Carolina. Stacey N. Bennett is a neurocritical care
nurse practitioner at Duke University Health System.
Steven T. Bellows is a medical student, level II, at Baylor
College of Medicine in Houston, Texas. Michael L. James
is an assistant professor of anesthesiology and of medicine/neurology at Duke University Health System.
Corresponding author: DaiWai M. Olson, Box 2900, Duke
University, Durham, NC 27710 (e-mail: olson006@mc.duke
.edu).
www.ajcconline.org
of care providers in an institution outside of the
development site.
Methods
This nonrandomized prospective study received
approval from the institutional review board at
Duke University Health System to enroll staff and
patients at Duke University Hospital. Patients were
candidates for the study if they were receiving TTM
via a temperature management device (Medivance
and Zoll Medical) approved by the Food and Drug
Administration. Patients receiving paralytic medications (neuromuscular blocking agents) were excluded
from the study. Because the outcome of interest was
the care provider’s assessment of shivering, neither
the reason for temperature management nor the
type of device used was recorded. Before enrollment,
a 5-member panel (a physician, a
nurse practitioner, a medical student, a charge nurse, and a staff
nurse) provided consent for the
study and was taught how to use
and interpret the BSAS. The primary
investigator (D.M.O.) taught panel
members how to use the BSAS
before recruitment of participants
into the study; no assessment was
done after the training. This group
was purposely selected to represent the diverse continuum of hospital providers tasked with assessing
shivering. During the first half of the study period,
one of the medical students left and was replaced
by an incoming student.
After the patients had provided consent, the
study coordinator contacted the assessment panel
and paired assessments (convenience) were completed. Paired assessments were completed simultaneously in a blinded fashion. Assessors were not
provided with information about the patient that
might give them a clue about the degree of shivering (eg, nursing interventions to reduce shivering).
First, the assessor would observe the patient from
the foot of the bed; next, the assessor would place
his or her hand on the patient’s jaw and upper
Studies support
the use of
hypothermia and
fever reduction to
reduce secondary brain injury.
AJCC AMERICAN JOURNAL OF CRITICAL CARE, January 2013, Volume 22, No. 1
71
Table
Raw scores on the Bedside Shivering Assessment
Scale (BSAS) for observers A and Ba
Observer B’s BSAS score
Observer A’s BSAS score
0
1
2
3
Total
0
65
16
0
0
81
1
5
12
0
0
17
2
0
0
1
0
1
3
0
1
0
0
1
70
29
1
0
100
Total
a
Values represent paired observations.
chest to palpate for shivering. Finally, the assessor
would enter the BSAS assessment onto a paper
record by placing an “X” to indicate the level of
shivering (1 sheet per assessment). The panel members remained blinded to the assessment results
until all 100 assessments had been completed. The
BSAS paper scores were entered into an Excel spreadsheet for data cleaning and storage. Analyses were
performed by using SAS v9.4 (SAS Inc) with BSAS
scores treated as ordinal data.
(76%) for BSAS scores = 0 (no shivering). The computed κ value of 0.46 (weighted κ = 0.48) indicates
moderate agreement.
There was complete agreement in 78% of the
paired assessments. There was complete agreement in
66% of the paired assessments (19 of 29 instances)
where 3 or more raters evaluated a patient simultaneously. Because only 2 patients received BSAS scores
of 2 or 3, we further explored the data by dichotomizing ratings. Exploring BSAS scores dichotomized to
“no shivering” (BSAS score = 0) versus “any shivering”
(BSAS score = 1, 2, or 3) resulted in moderate agreement (weighted κ = 0.47). Dichotomizing BSAS scores
as “little or no shivering” (BSAS score = 0 or 1) versus
“significant shivering” (BSAS score = 2 or 3) resulted
in substantial agreement (κ = 0.66).
The time of each observation was recorded. Most
assessments were performed in the afternoon hours
(see Figure). No relationship was found between
the time an assessment was performed and the likelihood of both observers assigning the same BSAS
score. Scores did not differ between assessors with a
nursing background and assessors with a medical
background (P = .98).
Results
Paired observations from 22 patients were completed between February 2010 and May 2011 (see
Table). Patients were primarily male (55%) with a
mean age of 54 years (range, 29-79 years); 73% of
patients were white and 27% were African American. Among 100 paired assessments of 22 patients
performed by 5 observers, agreement was highest
Discussion
The results of this study indicate that the BSAS
has adequate interrater reliability among a diverse
group of care providers. A key assumption is that
agreement between 2 scores most likely represents
both agreement and reliability.23 Reliability is an
expression of how likely a given score is to represent
Number of paired observations
45
40
35
30
25
20
15
10
5
0
Before 10
AM
10-11:59
Noon-1:59
AM
PM
2-3:59
PM
After 3:59
PM
Time when observations were taken
Figure Observations by time of day across a 12-hour shift.
72
AJCC AMERICAN JOURNAL OF CRITICAL CARE, January 2013, Volume 22, No. 1
www.ajcconline.org
the true score plus any systematic error. If 2 observers
incorrectly agree that the BSAS score is 3, error is
introduced to validity, but the agreement score
remains high.
Somewhat surprising is that 2 of 100 evaluations
were of BSAS scores greater than 1. In 1 paired
observation, both observers awarded a BSAS of 2;
in the other paired observation, observer A awarded
a BSAS score of 3 and observer B awarded a BSAS
score of 1. These findings are similar to results of a
recent study24 in which 93.9% (801/853) reported
observations were scored as a BSAS score of 0 or 1.
Construct validity evaluates 2 instruments
designed to measure a similar construct correlate.
Prior work by pioneers such as Holtzclaw25 should
be considered in studies of instrument validity.
Recently, May et al24 reported adequate correlation
between BSAS scores and derived electromyography
scores. That study24 also introduces the concept of
continuous monitoring for shivering, an impractical
application of the BSAS.13 We could find no other
reports regarding the BSAS from which to evaluate
construct validity.
Content validity is an assurance that the scale
has enough items to cover the domain being explored
adequately.26 DeVellis27 supports using more items
and indicates that the less specific the concept, the
more broad the categories and items will be. The
BSAS has not been evaluated for content validity,
but conceptually it is reasonable that it should be a
relatively specific and limited scale as it attempts to
address only shivering.
Limitations
Recognized limitations include the lack of randomization and the convenience of assessments’
timing between 2 or more observers. One medical
student involved in the study left and was replaced
by an incoming student; we were unable to obtain
paired assessments for these 2 practitioners and it is
possible that differences existed. The study’s use of
a team of 5 observers from diverse backgrounds is
both a strength and a limitation. Not comparing
the BSAS scores with a known reference standard
limits the external validity of these findings. The
external validity of the study is enhanced by the
diverse sample; however, future studies with higher
internal validity would be beneficial to fully understand the psychometrics of this instrument.
The observance of few BSAS scores greater than
1 is a significant limitation. The study design enrolled
subjects without regard to antishivering measures,
and it is likely that nurses and physicians had
employed measures to reduce shivering. However,
www.ajcconline.org
our observations parallel the findings of the only
other study24 exploring the BSAS outside of Columbia. Because the outcome of interest was interrater
reliability and not a patient-related outcome, observations were made of any patient with any form of
temperature management. It is unknown if the
instrument would perform differently for discrete
populations. It is unknown if the BSAS would be
more or less discriminatory at higher levels of shivering. Future studies should explore the applicability of the BSAS at higher levels of shivering.
Conclusion
The BSAS has adequate interrater reliability
across a diverse group of observers and should be
considered as an assessment tool for TTM. The relatively few instances of BSAS scores exceeding 1
deserves additional investigation.
FINANCIAL DISCLOSURES
Research funding for this study was provided by The
Medivance Corporation, the Duke Translational Nursing
Institute, and a Scientist Development Grant from the
American Heart Association.
eLetters
Now that you’ve read the article, create or contribute to an
online discussion on this topic. Visit www.ajcconline.org
and click “Submit a response” in either the full-text or
PDF view of the article.
REFERENCES
1. Axelrod YK, Diringer MN. Temperature management in acute
neurologic disorders. Neurol Clin. 2008;26:585-603, xi.
2. The Hypothermia After Cardiac Arrest Study Group. Mild
therapeutic hypothermia to improve the neurologic outcome after cardiac arrest. N Engl J Med. 2002;346:549-556.
3. Sydenham E, Roberts I, Alderson P. Hypothermia for traumatic
head injury. Cochrane Database Syst Rev. 2009:CD001048.
4. Marion D, Bullock MR. Current and future role of therapeutic
hypothermia. J Neurotrauma. 2009;26:455-467.
5. MacLellan CL, Clark DL, Silasi G, Colbourne F. Use of prolonged hypothermia to treat ischemic and hemorrhagic
stroke. J Neurotrauma. 2009;26:313-3236.
6. Dine CJ, Abella BS. Therapeutic hypothermia for neuroprotection. Emerg Med Clin North Am. 2009;27:137-149.
7. Sydenham E, Roberts I, Alderson P. Hypothermia for traumatic
head injury. Cochrane Database Syst Rev. 2009:CD001048.
8. Arrich J. Clinical application of mild therapeutic hypothermia after cardiac arrest. Crit Care Med. 2007;35:1041-1047.
9. Weant KA, Martin JE, Humphries RL, Cook AM. Pharmacologic options for reducing the shivering response to therapeutic hypothermia. Pharmacotherapy. 2010;30:830-841.
10. Mahmood MA, Zweifler RM. Progress in shivering control.
J Neurolog Sci. 2007;261:47-54.
11. Badjatia N, Kowalski RG, Schmidt JM, et al. Predictors and
clinical implications of shivering during therapeutic normothermia. Neurocrit Care. 2007;6:186-191.
12. Pitoni S, Sinclair HL, Andrews PJ. Aspects of thermoregulation physiology. Curr Opin Crit Care. 2011;17:115-121.
13. Badjatia N, Strongilis E, Gordon E, et al. Metabolic impact
of shivering during therapeutic temperature modulation:
the Bedside Shivering Assessment Scale. Stroke. 2008;39:
3242-3247.
14. Polderman KH, Herold I. Therapeutic hypothermia and controlled normothermia in the intensive care unit: practical
considerations, side effects, and cooling methods. Crit Care
Med. 2009;37:1101-1120.
AJCC AMERICAN JOURNAL OF CRITICAL CARE, January 2013, Volume 22, No. 1
73
15. Reynolds L, Beckmann J, Kurz A. Perioperative complications of
hypothermia. Best Pract Res Clin Anaesthesiol. 2008;22: 645-657.
16. Kurz A. Physiology of thermoregulation. Best Pract Res Clin
Anaesthesiol. 2008;22:627-644.
17. Polderman KH, Tjong Tjin Joe R, Peerdeman SM, Vandertop
WP, Girbes AR. Effects of therapeutic hypothermia on
intracranial pressure and outcome in patients with severe
head injury. Intensive Care Med. 2002;28:1563-1573.
18. Bicer C, Esmaoglu A, Akin A, Boyaci A. Dexmedetomidine
and meperidine prevent postanaesthetic shivering. Eur J
Anaesthesiol. 2006;23:149-153.
19. Doufas AG, Lin CM, Suleman MI, et al. Dexmedetomidine
and meperidine additively reduce the shivering threshold in
humans. Stroke. 2003;34:1218-1223.
20. Mokhtarani M, Mahgoub AN, Morioka N, et al. Buspirone
and meperidine synergistically reduce the shivering threshold. Anesth Analg. 2001;93:1233-1239.
21. Choi HA, Ko SB, Presciutti M, et al. Prevention of shivering
during therapeutic temperature modulation: The Columbia
anti-shivering protocol. Neurocrit Care. 2011;14:389-394.
22. Badjatia N, Strongilis E, Prescutti M, et al. Metabolic benefits
of surface counter warming during therapeutic temperature
74
modulation. Crit Care Med. 2009;37:1893-1897.
23. Bakeman R, Gottman JM. Observing Interaction: An Introduction to Sequential Analysis. New York, NY: Cambridge
University Press; 1997.
24. May T, Seder DB, Fraser GL, et al. Association of the bedside
shivering assessment scale and derived emg power during
therapeutic hypothermia in survivors of cardiac arrest.
Resuscitation. 2011;82:1100-1103.
25. Holtzclaw BJ. Shivering in acutely ill vulnerable populations.
AACN Clin Issues. 2004;15:267-279.
26. Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. New York, NY:
Oxford University Press; 1995.
27. DeVellis RF. Scale Development: Theory and Applications.
Thousand Oaks, CA: Sage Publications Inc.; 2003.
To purchase electronic or print reprints, contact The
InnoVision Group, 101 Columbia, Aliso Viejo, CA 92656.
Phone, (800) 899-1712 or (949) 362-2050 (ext 532); fax,
(949) 362-2049; e-mail, reprints@aacn.org.
AJCC AMERICAN JOURNAL OF CRITICAL CARE, January 2013, Volume 22, No. 1
www.ajcconline.org