Beyond the Volume-Outcome
Volume Outcome Relationship:
Deliberately Learning to Improve
Quality of Care
IIngrid
id M.
M Nembhard,
N bh d PhD
Yale Schools of Public Health & Management
Anita Tucker, DBA
Harvard Business School
AcademyHealth Annual Meeting, Boston, MA (June 2010)
The Volume
Volume-Outcome
Outcome Relationship
Ou
utcome
(e.g., procedu
ure time, mortalityy)
Learning-by-doing
a.k.a. “The Learning Curve”
-- Average for X
organizations
i i
0
5
10
15
20
Volume
l
(i.e., number of procedures, patients, cases)
2
A motivating puzzle:
Significant variation among hospitals
Proce
edure tim
me in hourrs
(Adjuste
ed for illn
ness seve
erity)
-- Average of 16 hospitals implementing MICS
Mountain Medical Center (The Best)
Chelsea Hospital
6
What explains
hospital
variation in
improvement
rates?
5
4
3
0
5
10
15
20
Number of procedures (Case number)
Sources: Pisano, GP., Bohmer RMJ, Edmondson AC. 2001. Organizational Differences in Rates of Learning:
Evidence from the Adoption of Minimally Invasive Cardiac Surgery." Management Science; 47(6); Edmondson AC,
3
Bohmer RMJ, Pisano, GP. Speeding Up Team Learning. Harvard Business Review (October 2001 ): 125-132
Explaining organizational differences in
i
improvement
t rates
t
What didn
didn’tt matter:



Institution type: All leading academic medical centers and community
hospitals, Highly successful in traditional approach to cardiac surgery
High-level management support: All had senior management support
Project leaders status: All reputable surgeons, some department heads,
some senior surgeons, some junior surgeons

Training: All went through identical training to use MICS

Attempts: All attempted to implement MICS in their organizations

Patient characteristics: No discernable differences

D
Data
collection/reporting:
ll
i /
i
All produced
d
d retrospective
i summary reports
4
Research question & hypotheses

Research q
question: Does the use of deliberate
learning activities (DLA) offers performance benefits beyond
those provided by cumulative volume (i.e., experience)?


DLA: deliberate activities to create, acquire, or transfer
knowledge, e.g., soliciting staff ideas, dry runs, etc.
Hypotheses:
Interdisciplinary collaboration
Use of deliberate learning activities
learning activities
H2: (+)
H1: (+)
Patient outcomes
5
Research methods


Setting: Vermont Oxford Network (VON) NIC/Q2002 national
collaborative
ll b ti to
t improve
i
neonatal
t l care (44 NICUs)
NICU )
Data
a a Sources:
Sou es
1. Organizational Assessment Survey (2 parts)
1.
2.
2.


Part I – completed by all staff; measured interdisciplinary collaboration
Part II – completed by QI teams; measured use of learning activities
VON NICU-patient database
Sample:
 23 NICUs in the United States and Canada (58% response)
 1,440 health care professionals (46% response), 265 for Part II
 2159 ELBW infants treated in 23 NICUs during study period (‘03-04)
Analysis: Logistic regression models with clustering by NICU, RSE
6
Measures
Variable
Measured by
Patient Outcome:
Patient Mortality
An indicator variable to denote whether each ELBW
infant in the NICU lived (1 = lived, 0 = died)
C
Cumulative
l ti Experience
E
i
S
Sum
off the
th number
b off infants
i f t treated
t t d
Improvement team members’ report of the use of 7
learning activities (=.88):
(= 88):
Use of Deliberate
Learning Activities
Interdisciplinary
C ll b ti
Collaboration
-
Solicitation of ideas from staff
- Pilot runs
Staff feedback before implementation - Dry runs
Education sessions with staff
- Team meetings
Problem-solving cycles (Plan-Do-Study-Act)
Nurses’’ and
N
d respiratory
i t
th
therapists’
i t ’ reported
t d llevell off
agreement with 6 survey items (=.88) e.g.,
- “Communication between nurses and physicians is open and
positive.”
- “The input of respiratory therapists and/or other ancillary staff
7
is regularly sought when developing treatment plans.”
Control variables
Variables
Individual patient-level
variables
i bl
Measured by
From risk-adjustment
F
i k dj t
t model
d l developed
d
l
d by
b the
th Vermont
V
t
Oxford Network (Rogowski et al. 2004):
gestational age, gestational age squared, gender,
small for gestational age, member of multiple birth,
vaginal delivery, outborn status, mother’s race, and 1minute Apgar score
NICU-level variables
(p < .10)
10)
teaching status
urban location
improvement team size
average staff tenure (~ staff turnover)
prior performance (Standardized Mortality Ratio in 2001)
Excluded variables
(p > .10)
10)
hospital
p
ownership
p status (e.g.
( g for-profit
p
hospital),
p ),
number of NICU beds, staff-to-beds ratio, average staff
hours worked weekly, percentage of physicians on the
improvement team, number of times participated in a
collaborative, number of improvement projects during
the collaborative, and level of evidence supporting the
8
NICU’s improvement project portfolio
Result 1:
Deliberate learning activities matters
Model 1: All years
Model 2: By year
DV = Mortality
OR
RSE
p
for 2003 infants
Cumulative experience
0.97
0.10
Cumulative experience for 2004 infants
0.80*
0.07
D lib t llearning
Deliberate
i activities
ti iti for
f 2003 iinfants
f t
1 18*
1.18*
0 08
0.08
Deliberate learning activities for 2004 infants
0.82*
0.07
0.84
0.11
Variables
Cumulative experience
Deliberate learning activities
OR
0.93
RSE
0.06
1.01
0.07
Year of infant birth (1 = infant born in 2003)
Control variables
Urban location
1.19
0.19
1.33*
0.19
Teaching hospital
0.66*
0.12
0.66*
0.12
Improvement team size
0.84**
0.06
0.85*
0.06
Staff tenure
1.20*
0.10
1.18^
0.10
Prior performance (2001 SMR)
1.29*
0.13
1.28*
0.13
Pseudo R-squared
0.149
0.154
Notes: N = 2159; * p < 0.05; ** p < .01; Infant-level risk adjusters included; clustered by NICUs
9
Result 2:
Collaboration does not moderate
DV = Mortality
OR
RSE
Cumulative experience
0.94
0.07
Deliberate learning activities
1.14
0.13
Interdisciplinary collaboration
0.76**
.07
Deliberate learning activities x Collaboration
1 00
1.00
.08
08
Urban location
1.92**
0.37
Teaching hospital
0.56**
0.12
Improvement team size
0 78**
0.78
0 04
0.04
Staff tenure
1.27**
0.09
Prior performance (2001 SMR)
1.20*
0.10
Variables
H2: Not
Supported
Control variables
Pseudo R-squared
^ < 0.10, * p < 0.05; ** p < .01; Infant-level risk adjusters included;
0.155
10
A re-conceptualization:
Mediation not Moderation
Two possibilities:
1.
Interdisciplinary collaboration as a mediator
Use of deliberate l
learning activities
2.
Interdisciplinary Collaboration
ll b
Patient Outcomes
Deliberate learning activities as a mediator
Interdisciplinary Collaboration
Use of deliberate g
learning activities
Patient Outcomes
11
Test of M1:
I t di i li
Interdisciplinary
collaboration
ll b ti as a mediator
di t
Mediation
Supported
Interdisciplinary Collaboration
Step 2: 
B = .21**
Use of deliberate Use
of deliberate
learning activities
OR = .82* Step 3: 
OR = .82* -> 1.03
Step 1: 
Patient Patient
mortality
Step 3: 
^ < 0.10, * p < 0.05; ** p < .01 ; 2004 data
Control variables included: Urban location, teaching hospital, improvement team size, prior12
performance, staff tenure, psychological safety, infant-level risk adjusters
Test of M2:
Use of DLA as a mediator
Use of deliberate
learning activities (DLA)
Step 2: 
Interdisciplinary Interdisciplinary
collaboration
Mediation
Not Supported
B = -.01
Step 1: 
OR =.79**
Patient Patient
mortality
^ < 0.10, * p < 0.05; ** p < .01; 2004 data
Control variables included: Urban location, teaching hospital, improvement team size, prior13
performance, staff tenure, psychological safety, infant-level risk adjusters
Summary of results:
Two paths to high quality care
In the long-term:
Autonomous
Learning:
Cumulative
experience
(e.g., volume)
Induced
Learning:
Use of deliberate
learning activities
(e.g., dry runs)
Patient
mortality
Interdisciplinary
Collaboration
14
Contributions and implications
p



Validates the systematic study of the deliberate learninglearning
outcome relationship
 Short
Short-term
term negative relationship suggests the need for
extra vigilance during initial phase of use
 Long-term
g
positive relationship
p
p suggests
gg
alternate strategy
gy
for outcomes improvement
Shows that deliberate learning activities influences interdisciplinary collaboration, a key determinant of quality of care
Informs the policy debate: “The Volume-Outcome Conundrum”
 Use of deliberate learning activities as an equally effective
15
alternative to solve the conundrum long-term?
Acknowledgements

Collaborators from the Vermont Oxford Network




F d
Funders




Hospitals that participated
Jeffrey Horbar, MD (CEO)
Joseph Carpenter, MS
Harvard Business School Division of Research
Wharton’ss Fishman-Davidson Center for Service and
Wharton
Operations Management
Sloan Industry Studies Fellowship
Colleagues from Harvard Business School


Amy Edmondson,
A
Ed
d
PhD
Richard Bohmer, MD, MPH
16
THANK YOU!
Supplemental Slides
The Volume
Volume-Outcome
Outcome Relationship
Volume 301 (25): 1364-1369
1364 1369
December 20, 1979
Should operations be regionalized?
The empirical relation between surgical volume and mortality
HS Luft
Luft, JP Bunker
Bunker, and AC Enthoven
Abstract
This study examines mortality rates for 12 surgical procedures of varying complexity in 1498
hospitals to determine whether there is a relation between a hospital's surgical volume and its surgical
mortality. The mortality of open-heart surgery, vascular surgery, transurethral resection of the prostate, and
coronary bypass decreased with increasing number of operations. Hospitals in which 200 or more of
these operations were done annually had death rates, adjusted for case mix, 25 to 41 per cent
lower than hospitals with lower volumes. For other procedures, the mortality curve flattened at lower
volumes. For example, hospitals doing 50 to 100 total hip replacements attained a mortality rate for this
procedure almost as low as that of hospitals doing 200 or more. Some procedures, such as cholecystectomy,
showed no relation between volume and mortality. The results may reflect the effect of volume or experience
on mortality,
l
or referrals
f
l to institutions with
h better
b
outcomes, as wellll as a number
b off other
h factors,
f
such
h as
patient selection. Regardless of the explanation, these data support the value of regionalization for
certain operations.
19
The Volume-Outcome Conundrum
The Volume–Outcome Conundrum
Kenneth W. Kizer, MD, MPH
V l
Volume
349 (22) :2159-2161
2159 2161
November 27, 2003
Volume and Outcome – It Is Time to Move Ahead
Arnold
o M. Epstein,
p
, MD
Volume 346 (15) :1161-1163
April 11, 2002
20
The “Volume-Outcome
Volume Outcome Conundrum
Conundrum”

Implications of the effectiveness of Cumulative Experience





Regionalization of care to high-volume institutions
Selective purchasing (e.g., Leapfrog Group)
Selective referral to high-volume
high volume institutions
Selective avoidance of low-volume institutions
Concerns and challenges







Will increased caseload deteriorate performance?
Will high-volume providers use market power to inflate costs?
Will innovators still be able to enter the market?
Will the disruption to the delivery system be too great?
What will the impact of lost patients be on financially strained centers?
C the
Can
th opposition
iti off hospitals
h
it l and
d physicians
h i i
be
b overcome??
What to do with the low-volume providers?
21
Kizer K. “The Volume-Outcome Conundrum;” and Epstein A. “Volume and Outcome – It Is Time to Move Ahead”
NICU Participants v. Non-participants
Characteristics
Hospital ownership
Hospital type
Participant (N=23)
Non-Participant (N=17)
22 Not-for-profit
1 Government
13 Not-for-profit
1 Government
2 For-profit,
F
fit 1 Other
Oth
17 Teaching
6 Non-teaching
Level of care in NICU a, b, c
13 Teaching
4 Non-teaching
2 No major surgery
9 No cardiac surgery
12 All surgeries
2.4 (.7)
0 No major surgery
13 No cardiac surgery
4 All surgeries
2.2 (.4)
8 None
11 one prior
4 two prior
.8 (.7)
8 None
7 one prior
2 two prior
.7(.7)
NICU-OB
NICU
OB relations,
relations Pain and sedation,
sedation
Infection, Respiratory care, Discharge,
Staffing, Family-centered care
NICU-OB
NICU
OB relation,
relation Pain and sedation,
sedation
Infection, Respiratory care, Discharge,
Staffing, Family-centered care
# of NICU Beds
41.4 (12.2)
Not available
No. of ELBW babies in 2001 b
45.2 (22.4)
47.6 (29.5)
% of babies with APGAR score < 3
33.9 (10.8)
30.8 (10.4)
% of babies born outside the NICU
29.5 (33.9)
18.5 (24.6)
Babies’ birthweight (grams)
772.6 (29.7)
775.9 (37.2)
26 (.5)
26 (0.7)
74 (12.5)
74.8 (9.5)
No of past collaboratives a,, b
No.
Target Area
Gestational age at birth (weeks) a, b, e
Length of stay (days)
a, b, e
22
Sample: Patient characteristics
Variable
Mean or %
SD
Min
Max
0 15
0.15
0 36
0.36
0
1
1a. Mortality for 2003 infants (n=1098)
0.15
0.35
0
1
1b. Mortality for 2004 infants (n=1061)
0.16
0.36
0
1
2. Gestational age (weeks)
25.96
2.00
21
35
3. Birthweight (grams)
769.65
146.86
415
1000
g
4. Small for ggestational age
0.16
0.37
0
1
5. 1-minute APGAR score
4.74
2.31
0
9
6. Gender (1 = male)
0.50
0.50
0
1
7. Mother’s
h
race - White
hi
0.60
0.49
0
1
8. Mother’s race - Black
0.23
0.42
0
1
9. Mother’s race - Hispanic
0.10
0.30
0
1
10. Mother’s race - Asian
0.05
0.21
0
1
11. Mother’s race - Other
0.03
0.16
0
1
12 Vaginal
12.
V i l delivery
d li
0 30
0.30
0 46
0.46
0
1
13. Member of multiple birth
0.27
0.45
0
1
14. Born at another hospital
0.26
0.44
0
1
1 Mortality (2003 and 2004)
1.
23
Sample: NICU characteristics
Variable
Mean
SD
Min
Max
1. Mortality (2003 & 2004)
0.17
0.06
0.08
0.27
141.33
76.5
21.7
322.3
-0.40^
3. Deliberate learning
activities
3.46
0.47
2.29
4.37
-0.11
-0.11
4. Interdisciplinary
collaboration
5 22
5.22
0 60
0.60
4 04
4.04
6 33
6.33
-0.28
0 28
-0.03
0 03
0 30
0.30
5. Urban hospital
0.04
0.21
0.00
1.00
-0.21
0.48*
-0.09
0.21
6 Teaching hospital
0.70
0.47
0.00
1.00
-0.09
0.09
-0.23
-0.14
0.14
7. Improvement team size
12.91
6.33
5.00
27.00
-0.42^
0.28
0.27
-0.10
0.18
-0.25
8. Staff tenure
3.86
0.52
2.89
4.80
0.36^
-0.30
-0.30
0.20
0.01
0.09
-0.30
9. Prior performance
(2001 SMR)
0 99
0.99
0 40
0.40
0 39
0.39
1 86
1.86
0 27
0.27
-0 13
-0.13
0 37^
0.37
-0 15
-0.15
-0 04
-0.04
-0 02
-0.02
0 12
0.12
2. Cumulative experience
(2002 2004)
(2002-2004)
1
2
3
4
5
6
7
8
-0 23
-0.23
24
Summary:
y Staff characteristics
Variable
Mean
S.D.
1
2
3
4
5
6
7
8
1 P
1.
Professional
f i l status
t t - 3 groups:
Physicians v. nurses v.
therapists
2 03
2.03
.43
43
2. Professional status - 2 groups:
High v.
v low status
.08
.27
.71**
3. Psychological safety
5.31
1.08
.15**
.14**
4. Leader inclusiveness
4.62
.55
<.01
-.01
.29**
5. Engagement in quality
improvement work
5.45
1.06
.12**
.17**
.48**
.21**
6. Gender
1.10
.30
.23**
.56**
.05
-.04
.02
7 Y
7.
Years iin any NICU§
4 17
4.17
1 19
1.19
.08**
08**
.10**
10**
-.05
05
.09**
09**
-.03
03
.08**
08**
8. Years as hospital employee§
3.97
1.26
.03
.01
-.04
.02
-.02
.07*
.73**
9. Years in current NICU§
3.84
1.31
.05
.03
-.04
.02
-.02
.08**
.81**
.94**
10. Hours per week in NICU
33.84
13.46
.35**
.38**
.05
-.03
.12**
-.06*
-.08**
§
.10**
9
-.08**
This is a categorical variable: 1 = less than one year, 2 = one to less than two years, 3 = two to less than five years, 4= five to less than
ten years, 5 = ten or more years
* p < .05,
05 ** p < .01
01
25
Control
Co
o variable:
a ab e Prior
o performance
pe o a ce
Prior Performance as measured by SMR in 2001
(SMR = Standardized Mortality Ratio)
1.
2.
3.
4
4.
Logistic regression model (clustered by NICU)
 Dependent variable: Morality (1 = lived, 0 = died)

Independent variables: Established infant-level risk factors
P di t probability
Predict
b bilit off death
d th for
f each
h baby
b b
By NICU, sum up probability of death, actual deaths
Compute ratio:
Actual deaths

SMR Mortality 
 Expected deaths



SMR < 1 Outcomes BETTER than expected
SMR = 1 Outcomes equal to expected
SMR > 1 Outcomes
O t
WORSE than
th expected
t d
26
Test of M1:
U off DLA CollaborationMortality
Use
C ll b ti M t lit
Criterion for mediation being tested
Criterion 1
Criterion 2
Criterion 3
Dependent variable
Mortality
Collaboration
Mortality
Coefficients reported
OR
RSE

RSE
0.09^
0.05
OR
RSE
Variables
Cumulative experience
Cumulative experience for 2003 infants
0.97
0.10
0.91
0.09
Cumulative experience for 2004 infants
0.80*
0.07
0.79**
0.07
Deliberate learning activities
0.21**
0.07
Deliberate learning activities for 2003 infants
1 18*
1.18
0 08
0.08
1 45**
1.45
0 15
0.15
Deliberate learning activities for 2004 infants
0.82*
0.07
1.03
0.12
0.84
0.11
.81^
0.09
.82*
0.08
Year of infant birth (1 = 2003)
Interdisciplinary collaboration
Control variables (all previous)
Observations
2159
1081
2159
Pseudo R-squared (R-squared)
0.15
(0.65)
0.16
Mediation Supported
27
Test of M2:
Collaboration  Use of DLA Mortality
Criterion for mediation being tested
Criterion 1
Criterion 2
Dependent variable
Mortality
Deliberate Learn
Coefficients reported
RSE
OR

RSE
Variables
Cumulative experience
Cumulative experience for 2003 infants
0.94
0.09
Cumulative experience for 2004 infants
0.84*
0.07
Year of infant birth (1 = 2003)
0.83
0.12
Interdisciplinary collaboration
0.79**
0.07
-0.08
0.07
-0.01
0.12
Deliberate learning activities
Deliberate learning activities for 2003 infants
Deliberate learning activities for 2004 infants
Control variables (all previous)
Observations
2159
244
Pseudo R-squared (R-squared)
.16
.10
Mediation NOT Supported
28
A qualitative look at deliberate learning

Deliberate learning
g activities create opportunities
pp
for all staff to
engage in operational and conceptual learning -- while
adapting work practices to fit the organizational context
Adaptation
Learning
(via
p
)
experimentation)
Staff
involvement
“We made our best guess using the [existing bits and
pieces of] evidence to figure out how we could implement it
in a way that was reasonable for our people
people. . . We used the
isolation room and a rubber ball to simulate a neonate, and
we tried everything. We conducted dry-runs to see what it
would be like to try this and that...We
that We tried lots and lots of
ways, practicing.
“We
W use this
thi procedure
d
iin the
th d
delivery
li
room 100% off the
th
time. Even now, every once in awhile, someone will say,
‘What if we did this?’ And they suggest a little bit of a
modification ”
modification.
Neonatologist, NICU 3
29
Qualitative support for collaboration
as a mediator


“We had to change how we worked together as a team,
too, because now we had to do things a little differently,
so now you needed your buddy to do something a little
differently, so you could do something a little differently.”
- NICU 3 Respiratory
p
y Therapist
p
“We
We practiced because this requires a certain amount of
dexterity. Now we predefine before we go to delivery who
does what job. All of us can do all of the jobs. . . Sometimes
Rick and I are back there, and I’ll intubate and he’ll put the
surfactant in. Sometimes, he’ll intubate.”
- NICU 3 Neonatalogist
30
I
Induced
Autono
omous
An unanswered question:
Th path
The
th to
t the
th use off deliberate
d lib t learning
l
i
Cumulative
experience
(e.g., volume)
Leader
inclusiveness
Psychological
safety
Deliberate
learning activities
Patient
mortality
Interdiscp.
Collaboration
(e g dry runs)
(e.g.,
 LLeader
d inclusiveness:
i l i
l d actions
leader
ti
that
th t indicate
i di t an invitation
i it ti and
d appreciation
i ti for
f
others’ contributions
 Psychological safety: feeling safe to speak up with questions, concerns, suggestions31
Cumulative experience * Collaboration
Logistic
g
regression:
g
Std. Err. adjusted
j
for 23 clusters
------------------------------------------------------------------Robust
mortality | Odds Ratio
Std. Err.
95% Conf. Interval
-------------+----------------------------------------------------Cum. experience
| .9981499* .0009491
.9962914 1.000012
Deliberate learning | .8684743
.1393244
.6341669 1.189352
Collaboration
| .4803655* .1460516
.2647099
.871712
Experience*Collab
| .9316546
.1919105
.6221813 1.39506
Urban
| 2.417283
1.577743
.6725958 8.687617
Teaching
| .4691311** .0838228
.330525
.6658618
T
Team
size
i
| .9533597**
9533597** .010311
010311
.9333632
9333632
.973784
973784
Prior performance
|
1.841424* .4697671
1.116871 3.036019
Staff tenure
|
1.444116* .2368301
1.047147 1.991573
32
Hypothesis 1


Hypothesis The use of DLA is positively associated
with better patient outcomes, after accounting for
the benefit of cumulative experience.
experience
R ti
Rationale:
l DLA helps
h l overcome 3 challenges:
h ll



Rapid knowledge growth -> limited ability to draw on past
Variability and uncertainty ->unable to gain experience
Errors may cause harm -> low motivation to change
Use of deliberate Use
of deliberate
learning activities
H1
Patient Patient
outcomes
33
Hypothesis 2:
Th moderating
The
d ti effect
ff t off collaboration
ll b ti


Hypothesis:
yp
The p
positive relationship
p between use
of DLA and patient outcomes will be stronger in
g p with greater
g
collaboration.
workgroups
Why collaboration might matter: It leads to:



Better decision-making (Milliken and Martins 1996)
Better coordination (Gittell 2002)
Better error detection and recovery (Weick et al. 1999)
Interdisciplinary collaboration
Use of deliberate Use
of deliberate
learning activities
H2
Patient Patient
outcomes
34
Research question
Does the use of deliberate learning
activities (DLA) account for differences
in outcomes improvement?
DLA defined: deliberate activities to create
create,
acquire, or transfer knowledge, e.g., soliciting
staff ideas
ideas, dry runs
runs, etc
etc.
35
Research objective


examine whether the use of deliberate learning activities
(DLA) offers performance benefits beyond those provided
byy cumulative volume ((i.e.,, experience)
p
)
 DLA: deliberate activities to create, acquire, or transfer
knowledge, e.g., soliciting staff ideas, dry runs, etc.
assess whether the effectiveness of deliberate learning
activities
i i i depends
d
d on a critical
i i l interaction
i
i in
i workgroups,
k
interdisciplinary collaboration.
Interdisciplinary collaboration
Use of deliberate Patient H2
learning activities
learning activities
outcomes
H1
36
Hypotheses


Hypothesis 1: The use of DLA is positively associated with
better patient outcomes, after accounting for the benefit of
cumulative experience.
p
Hypothesis 2: The positive relationship between use of DLA
and patient outcomes will be stronger in workgroups with
greater collaboration.
Interdisciplinary collaboration
Use of deliberate learning activities
learning activities
H2
H1
Patient outcomes
37