APPENDIX
We examined the sensitivity of estimates of the percent of variance at different levels of
the hierarchy to an alternative modeling approach, specifically to using a Bayesian hierarchical
model. There are several advantages to the Bayesian hierarchical model. One, whereas PROC
MIXED in SAS requires the assumption that the dependent variable (the ratio of the number of
eligible goals/processes achieved) can be approximated by a normal distribution, the Bayesian
model allows us to model the number of eligible goals/processes achieved as a binomial
distribution. The binomial distribution is a better approximation to the data than the normal
distribution. Second, PROC MIXED is not able to distinguish variation across patients from
residual variation in the number of goals/processes achieved (though it is able to estimate the
sum of those two types of variation). The Bayesian model can estimate variation across patients,
which can then be compared to variation at higher levels of the hierarchy. However, in the
Bayesian model, when examining the technical quality measures, random variation cannot be
easily estimated on the same scale as the other measures of variation (which are on a logit scale)
and hence it is not possible to easily obtain the sum of residual plus patient level variation. When
examining the satisfaction measures, all estimates from the Bayesian model are on the same
scale and hence both patient-level and random variation can be distinguished.
In what follows, we describe the Bayesian model.
Let
qfj = quality at facility j
qtk = quality delivered by team k
qpm = quality delivered by provider m
qpan = quality associated with patient n’s behavior
MDC-D-09-00625
Page 1
We assume
qfj | m0, pr0 ~ N(m0, pr0) (i.e., normally distributed with mean m0 and precision pr0)
qtk | qff(k) , pr1 ~ N(qff(k) , pr1) (where f(k) indicates the facility of team k)
qpm | qtt(m), pr2 ~ N(qtt(m), pr2) (where t(m) indicates the team of provider m)
qpan | qpp(n), pr3 ~ N(qtp(n), pr3) (where p(n) indicates the provider of patient n)
For the technical quality measures, let logit(probn) = qpan
Yn = number of processes performed/goals met by the nth patient
Nn = number of processes/goals for which the nth patient was eligible
We assume
Yn ~ Nn , probn ~ binomial(Nn , probn)
For the satisfaction measures, let Yn = measure value for the nth patient. We assume
Yn ~ qpan , pr4 ~ N(qpan, pr4)
We placed vague priors on m0 (uniform(0,5)) and the standard deviations (i.e., s0, s1, s2, s3, and
s4) (uniform(0, 10) from which precision is calculated (e.g., pr0 = 1/s02). Our interest is in the
posterior variances.
s02 : variance in quality across facilities
s12 : variance in quality across teams
s22 : variance in quality across providers
s32: variance in quality across patients
s42: random variance in outcomes for a patient (relevant only for the satisfaction measures)
MDC-D-09-00625
Page 2
We used Gibbs sampling as implemented in WinBUGS 4.1 to estimate the posterior distribution
of parameters. We report the mean of the estimates from the Gibbs samples and the percent of
total variance that occurs at different levels of the hierarchy for technical quality (Appendix
Table 1) and patient satisfaction (Appendix Table 2). It is important to note that for the technical
quality measures the Bayesian model variances are on the logit scale. Hence, the magnitude of
the variances is not directly comparable to the variances estimated using PROC MIXED.
However, the percentages attributed to each of level of the hierarchy are comparable. For the
satisfaction measures, the magnitudes of variance at each level of the hierarchy are on the same
scale and hence more comparable. We calculated the fraction of explainable variance (excluding
the patient-level residual variance) to compare the estimates at the provider, team, and medical
center levels obtained from SAS and WinBUGS (Appendix Tables 1-2).
To “back into” estimates of the percent of the residual variation from PROC MIXED (which
includes both patient and random variation) that is attributable to the patient, we did the
following:
1) Assume the ratio of patient-level variation to system-level variation is the same in the
Bayesian model and the PROC MIXED model (something that seems reasonable given
the similarity of estimates of components of system-level variation from both
approaches);
2) Multiply the ratio of patient-level variation (s32) to system-level variation from the
Bayesian model by system-level variation from PROC MIXED. This provides an
estimate of patient-level variation from the PROC MIXED results. Subtract patient-level
MDC-D-09-00625
Page 3
variation from the total residual variation from PROC MIXED to estimate random
variation from the PROC MIXED results.
We can partially validate the assumption of the comparability by examining results for the
patient satisfaction analyses, as follows:
1) Calculate the ratio of residual variation to system-level variation from both the PROC
MIXED and Bayesian results.
2) Compare the ratios from the PROC MIXED and Bayesian models. .
As shown in Appendix Table 3, the results are comparable.
We can use results from the Gibbs samples to examine a number of interesting questions that
shed light on the degree of certainty associated with our conclusions. We examined the
following questions in regard to the technical quality measures:
1) How confident are we that provider level variation is greater than team level variation?
Than medical center variation?
2) How confident are we that team level variation is greater than medical center level
variation?
3) How confident are we that patient level variation is greater than system level variation?
Than 1.5 times system level variation?
We examine these questions by recording the proportion of the Gibbs samples where, for
example, provider level variance is greater than team level variance. We show results in
Appendix Table 4. With the exception of AMI, we are pretty sure (over 0.74 probability)
MDC-D-09-00625
Page 4
that provider level variation is greater than team level variation and almost certain (over 0.98
probability) that provider-level variation is greater than medical center level variation. With
the exception of cancer, there is a pretty good chance (over 0.63 probability) that team level
variation is greater than medical center variation. We are certain that patient variation is
greater than system-level variation and there is a reasonably good chance (over 0.59
probability) it is at least 50% greater than system-level variation.
For the patient satisfaction measures, we can distinguish patient variation from random
variation. In Appendix Table 5, we show results for the following questions:
a. How confident are we that patient-level variation is greater than system-level
variation?
b. How confident are we that random variation is greater than system-level
variation?
c. How confident are we that random variation is greater than patient-level
variation?
MDC-D-09-00625
Page 5
[Appendix] Table 1. Percent Variance Accounted for Among Technical Quality Measures at the
Provider, Team and Medical Center Level: Comparing Estimates from SAS and WinBUGS
Level of Care
Disease Ratio
Provider
Team
Medical Center
SAS
WinBUGS
SAS
WinBUGS
SAS
WinBUGS
Diabetes
61.1%
61.5%
21.4%
26.3%
17.5%
12.1%
Hypertension
71.9%
71.5%
16.6%
15.0%
11.4%
13.5%
Cancer
46.5%
52.0%
24.3%
20.7%
29.2%
27.3%
Immunizations
55.3%
50.0%
33.0%
37.3%
11.7%
12.8%
Acute Myocardial
Infarction
50.0%
43.6%
41.9%
47.0%
8.1%
9.4%
MDC-D-09-00625
Page 6
[Appendix] Table 2. Percent Variance Accounted for Among Patient Satisfaction Measures at
the Provider, Team and Medical Center Level: Comparing Estimates from SAS and WinBUGS
Level of Care
Measure/Scale
Provider
SAS
Team
Medical Center
WinBUGS
SAS
WinBUGS
SAS
WinBUGS
Single-Item Overall
Quality
34.3%
34.0%
34.4%
34.3%
31.3%
31.7%
Delivery System
Index
40.6%
40.3%
27.7%
27.5%
31.7%
32.2%
Doctor/Patient
Interaction Index
59.9%
59.9%
19.4%
19.0%
20.7%
21.0%
MDC-D-09-00625
Page 7
[Appendix] Table 3. Validation Results for SAS Patient-Level Variance Components (Estimated
from WinBUGS)
Measure/Scale
SAS
WinBUGS
Estimated
Patient +
Random
System
Level
Ratio*
Estimated
Patient +
Random
System
Level
Ratio*
0.719
0.035
20.802
0.710
0.035
20.333
Delivery System
Index
374.880
28.882
12.980
373.613
29.150
12.817
Doctor/Patient
Interaction Index
353.770
33.156
10.670
353.489
33.303
10.614
Single-Item Overall
Quality
*The ratio was calculated by dividing the estimated patient plus random variance by the systemlevel variance (provider, team, and medical center levels).
MDC-D-09-00625
Page 8
[Appendix] Table 4. Probability In Support of the Indicated Relationship Between Variances for
Different Levels of the Hierarchy: Technical Quality Measures
Relationship of
Variances for Indicated
Levels
Diabetes
Hypertension
Cancer
Immunizations
Acute
Myocardial
Infarction
Team>Medical Center
0.78
0.63
0.26
0.96
0.96
Provider>Team
0.84
1.00
0.98
0.74
0.49
Provider>Medical
Center
0.99
1.00
0.98
0.99
0.83
Patient>System
1.00
1.00
1.00
1.00
1.00
Patient>1.5*System
0.61
0.80
0.59
0.93
0.98
MDC-D-09-00625
Page 9
[Appendix] Table 5. Probability In Support of the Indicated Relationship Between Variances for
Different Levels of the Hierarchy: Patient Satisfaction Scores
Relationship of
Variances for
Indicated Levels
Overall
Quality
Delivery
System
Doctor/Patient
Interaction
Patient>System
0.74
1.00
0.0
Residual>System
1.00
0.32
1.00
Residual>Patient
1.00
0.0
1.00
MDC-D-09-00625
Page 10