BRADLEY ET AL., TECHNICAL APPENDIX
Note that the content of this technical appendix is directly from a paper by Rothman et al., in
J Biomedical Informatics.1
In this Appendix, we describe criteria for variable selection and how the selected variables are
combined to create the final Rothman Index, the measure of patient condition at a specific
point in time during hospitalization.
A survey of EMR data collected yielded approximately 7,000 variables (nearly 6,500 flow sheet
inputs and 500 laboratory tests). From these, variables were selected that were: a) related to
the patient condition, b) regularly collected on all patients, and c) susceptible to change over
the course of a patient’s hospital stay. Demographic or descriptive variables that do not change
during a patient’s time in the hospital, such as age, sex, diagnosis, comorbidities, and
hospitalization history were excluded. These criteria reduced the dataset to 43 candidate
variables: 13 nursing assessments, 6 vital signs, 23 laboratory tests, and cardiac-monitoring
rhythms.
For each of these 43 clinical variables, an excess risk function was computed. “Excess risk” is
defined as the percent increase in 1-year all-cause mortality associated with any value of a
clinical variable, relative to the minimum 1-year mortality identified for that variable.2 For
example, Figure 1 shows the excess risk function for white blood cell count (WBC). The points
represent average excess 1-year post-discharge all-cause mortality versus average WBC at
discharge; data from 22,265 patient discharges are bucketed by WBC range. The regression line
is a polynomial fit to the data, normalized to the lowest risk value. Above and below clinical
value extrema, where data are sparse, the function is set to a constant.
1
Figure 1. Excess 1-year mortality risk as a function of white blood cell count.
For values of 1 or below, the value is set at 80%. For values of 34 and above, the value is set
at 60%. Diamonds show 2004 data, circles show 2005-2006 data.
Excess risk values were also determined for nursing assessment results. Nursing assessment
data are collected in the course of a “head-to-toe” or “body system” patient examination3
performed at least once per nursing shift and recorded in the EMR in one of two ways. If
charting by exception,4 the nurse answers a master question for each physiological system,
such as “Is the patient’s respiratory function within normal limits?” (“normal” limits might be
defined as respiration at 12-24 breaths/minute, nail beds pink, bilateral breath sounds).
Alternatively, the nurse may answer a series of questions, such as, “What are the breath
sounds?”, “What color are the nail beds?” etc. The answer to a master question is “pass” or
“fail,” and when there are multiple questions per assessment, we map the entire category to a
“fail” if any answer reflects a deviation from normal. Assessment questions may vary between
hospitals but share the aim of noting non-normal physiological system fundamentals. Example
definitions of standards for each nursing assessment are shown in Table 1.
Table 1. Nursing Assessment Standards
Pulse regular, rate 60-100 BPM, skin warm and dry. Blood Pressure
Cardiac Standard:
less than 140/90 and no symptoms of hypotension.
No difficulty with chewing, swallowing or manual
Food/Nutrition Standard: dexterity. Patient consuming >50% of daily diet ordered as
observed or stated.
Abdomen soft and non-tender. Bowel sounds present. No nausea
Gastrointestinal
or vomiting. Continent. Bowel pattern normal as observed or
Standard:
stated
Voids without difficulty. Continent. Urine clear, yellow to amber as
Genitourinary Standard:
observed or stated. Urinary catheter patent if present.
Musculoskeletal
Independently able to move all extremities and perform functional
Standard:
activities as observed or stated (includes assistive devices).
2
Pain Standard:
Neurological Standard:
Without pain or VAS (visual analogue pain scale) <4 or
experiencing chronic pain that is managed effectively.
Alert, oriented to person, place, time, and situation. Speech is
coherent.
Peripheral/Vascular
Standard:
Extremities are normal or pink and warm. Peripheral pulses
palpable. Capillary refill <3 sec. No edema, numbness or tingling.
Psychosocial Standard:
Behavior appropriate to situation. Expressed concerns and fears
being addressed. Adequate support system.
Resp. 12-24/min at rest, quiet and regular. Bilateral breath sounds
clear. Nail beds and mucous membranes pink. Sputum clear, if
present.
Safety/Fall risk factors not present. Patient is not a risk to self or
others.
Skin clean, dry and intact with no reddened areas. Patient is alert,
cooperative and able to reposition self independently. Braden
scale >15.
Respiratory Standard:
Safety/Fall Risk Standard:
Skin/Tissue Standard:
Excess risk for each nursing assessment category was calculated from the difference in 1-year
mortality between patients who passed and patients who failed their last assessment prior to
discharge. In Fig. 2, we show excess risk computed for 12 nursing assessments (excluding the
Braden score) for 2 separate 1-year periods at SMH to show the stability of the relative impact
of failing a particular nursing assessment. The 2004 results were used in the model
development.
Fig. 2. Excess 1-year mortality risk for each of 12 simplified nursing assessments.
With all 43 variables on a common 1-year excess mortality risk scale, multi-collinearity was
determined using Pearson correlation coefficients. If any pair of variables had a Pearson
3
correlation coefficient greater than 0.7, the less frequently collected variable was excluded. The
relative importance of the remaining variables was determined using forward stepwise logistic
regression (SAS Version 9.2) of the excess risk values against 1-year mortality. Variables were
added subject to the added regression coefficient having a p-value of less than 0.05. The final
set of variables is shown in Table 2. The logistic regression is used simply to select variables; its
coefficients are not used.
The final RI includes the 26 variables, which in development of the RI were shown, based on
final values prior to discharge, to best predict 1-year all-cause mortality with the condition that
if any pair of variables had a Pearson correlation coefficient greater than rho = 0.70, the less
commonly collected variable was excluded. Only hemoglobin and hematocrit were deemed
collinear, and hematocrit was removed. The final set of 26 variables selected by the logistic
regression analysis for inclusion in the RI is shown below.
Table 2. Variables (n=26) chosen as inputs to the RI
Vital Signs
Temperature
Diastolic Blood Pressure
Systolic Blood Pressure
Pulse Oximetry
Respiration Rate
Heart Rate
Nursing
Assessments
(Head-to-Toe)
Cardiac
Respiratory
Nursing
Assessments
(Other)
Braden Score
Laboratory
Tests (blood)
Cardiac Rhythm
Creatinine
Sodium
- asystole
- sinus rhythm
Gastrointestinal
Chloride
- sinus bradycardia
Genitourinary
Neurological
Skin
Safety
Peripheral
Vascular
Food/Nutrition
Psychosocial
Musculoskeletal
Potassium
BUN
WBC
Hemoglobin
-
sinus tachycardia
atrial fibrillation
atrial flutter
heart block
junctional rhythm
- paced
- ventricular fibrillation
- ventricular tachycardia
Using these 26 variables, a transform, termed an “excess risk” function, and defined as the
relationship between the final measured value of a variable at discharge and the “excess” 1year post-discharge all-cause mortality above a base rate, was computed for each variable. The
RI score, which ranges from -91 to +100 (from poorest condition to healthiest condition), is
calculated as follows shown in Equation (1):
Equation (1)
The scale factor was computed empirically so that in general RI scores for patients on a medical
surgical ward would be above zero. A score of 100 is achieved only when all input variables are
4
at a minimum (zero excess risk) value. Critically ill patients may have negative RI values (the
minimum possible RI score is -91).
As it is unusual for all 26 variables to be measured at the same time, the model must
accommodate missing data elements. If a variable is completely missing for a particular patient,
or if an existing variable is more than 15 hours old, zero excess risk is assigned. Laboratory tests
are generally collected less frequently than vital signs and nursing assessments (less than every
15 hours). To utilize information from laboratory tests without introducing inaccuracies
stemming from the inclusion of old lab data, the RI model is comprised of 2 sub-models (RInoLab
and RIwithLab). Both sub-models are computed as in Equation (1). RInoLab uses only nursing
assessments and vital signs, whereas RIwithLab uses nursing assessments, vital signs, and
laboratory tests. As the laboratory data ages, its relevance to the patient’s current condition
diminishes; therefore, RIwithLab is blended by a linear decay with RInoLab. After 48 hours, RInoLab is
used solely, until new laboratory data becomes available. At a minimum, computing a patient’s
RI requires complete information on a set of vital signs and on all by up to two nursing
assessments. Stepwise forward logistic regression was used to select variables for each submodel. Variables selected for RInoLab include all inputs listed in Table 1, except the laboratory
results, which are missing by definition in RInoLab. The procedure for RIwithLab yielded 24 variables.
The model is thus a simple linear combination of the two sub-models as a function of time,
based on the most recent available laboratory data, as shown in Equation (2):
Equation (2):
Where “TimeSinceLabs” has a maximum value = 48 hours.
5
References
1.
Rothman MJ, Rothman SI, Beals IV J. Development and validation of a continuous
measure of patient condition using the Electronic Medical Record data. J Biomed
Inform. 2013; DOI: 10.1016/j.jbi.2013.06.011.
2.
Rothman SI, Rothman MJ, Solinger AB. Placing clinical variables on a common linear
scale of empirically-based risk as a step toward construction of a general patient acuity
score from the Electronic Health Record: A modelling study. BMJ Open. 2013;
3:e002367. DOI: 10.1136/bmjopen-2012-002367.
3.
Baid H. The process of conducting a physical assessment: a nursing perspective. Br J
Nurs. 2006; 15: 710-714.
4.
Kerr SD. A comparison of four nursing documentation systems. J Nurs Staff Dev. 1992; 8:
27-31.
6
Appendix Table A. Comparison of derivation and validation samples
Age
18-44 years
45-64 years
65 and older
Mean (SD)
Sex
Male
Female
Insurance type
Medicare
Medicaid
Blue Cross/Commercial
Other/Uninsured
Service type
Medical
Surgical
Readmission
Yes
No
Rothman Index [Mean (SD)]
At admission
At 48 hours before discharge
At discharge
Derivation
Sample
(N=2,781)
Validation
Sample
(N=2,730)
578 (20.8%)
1066 (38.3%)
1137 (40.9%)
59.6 (18.30)
563 (20.6%)
1023 (37.5%)
1144 (41.9%)
60.0 (18.56)
1356 (48.8%)
1425 (51.2%)
1373 (50.3%)
1357 (49.7%)
1356 (48.8%)
602 (21.6%)
772 (27.8%)
51 (1.8%)
1348 (49.4%)
579 (21.2%)
748 (27.4%)
55 (2.0%)
1868 (67.2%)
913 (32.8%)
1846 (67.6%)
884 (32.4%)
449 (16.1%)
2332 (83.9%)
450 (16.5%)
2280 (83.5%)
72.6 (17.77)
74.2 (17.39)
77.3 (13.24)
71.6 (17.84)
73.8 (13.98)
77.3 (13.34)
Unadjusted
P-value1
0.729
0.450
0.255
0.912
0.722
0.734
0.038
0.267
0.998
1
P-values derived from chi-square tests and independent t-tests for categorical and continuous
variables, respectively.
7
Appendix Table B. Fully adjusted logistic regression model examining associations with
unplanned readmission, untransformed parameters (N=2,730)
Adjusted1
Estimate (95% CI)
Age
18-44 years
0.23 (-0.16 - 0.62)
45-64 years
0.18 (-0.14 - 0.49)
65 and older
REF
Sex
Male
REF
Female
0.08 (-0.14 - 0.31)
Insurance type
Medicare
0.41 (-0.57 - 1.40)
Medicaid
0.31 (-0.67 - 1.29)
Blue Cross/Commercial
0.42 (-0.55 - 1.39)
Other/Uninsured
REF
Service type
Medical
0.23 (-0.13 - 0.60)
Surgical
REF
Rothman index at discharge
Highest risk (<70)
0.97 (0.54 - 1.41)**a
Medium risk (70-79)
0.87 (0.45 - 1.305)**a
Low risk (80-89)
0.34 (-0.05 - 0.76)
Lowest risk (>90)
REF
*P < 0.05; **P < 0.01
a
Coefficient significantly different from the “Low risk (80-89)” category.
1
Adjusted for covariates shown as well as discharge diagnosis (modeled with 185 dummy variables).
8