DISMEVAL
A STUDY ON DISEASE MANAGMENT EVALUATION IN
EUROPE
BERT VRIJHOEF PH.D.,
PROFESSOR OF CARE FOR THE CHRONICALLY ILL
INTEGRATED CHRONIC DISEASE MANAGEMENT FORUM,
DEPARTMENT OF HEALTH,
29TH AUGUST 2011, MELBOURNE
A few facts
The Netherlands
•
•
•
•
•
•
41,526 sq km
Amsterdam (capital)
16,6 million inhabitants
Infant mortality rate:
4.4/1000
Life expectancy: 78.3 yrs for
men and 82.3 yrs for women
GDP: 843 billion US dollars
www.state.gov
Australia
•
•
•
•
•
•
7,7 million sq km
Canberra (capital)
22,6 million inhabitants
Infant mortality rate:
4.7/1000
Life expectancy: 79.3 yrs for
men and 83.9 yrs for women
GDP: 1.3 trillion US dollars
www.state.gov
A few facts
The Netherlands
•
Hockey Champions Trophy 2011
Australia
•
Tour de France 2011
A few facts
The Netherlands
•
Planking
Australia
•
Planking
Contents
 Disease Management
 Disease Management Evaluation
 DISMEVAL, the project
 DISMEVAL, the Netherlands
Take home messages
 ‘Disease management’ presents a spectrum of chronic care
improvement programmes ranging from no till substantial
attempt to redesign practice
 Measuring disease management programme performance at
the population level in a scientifically sound fashion while
being practicable for routine operations remains a challenge
 DISMEVAL illustrates that other designs (than RCT)
constitute valuable designs for the evaluation of complex
interventions and provide in-depth insights
 Future evaluation of ‘disease management’ should focus on
the identification of subgroups and target these subgroups
DISEASE MANAGEMENT
• ‘Disease management’ is a global buzzword in health care
and it is often used by different people to mean different
things
• Structured ‘disease management ‘has been proposed as a
means to improve quality and reduce the costs of health care
and to improve health outcomes for the chronically ill
• ‘Disease management’ presents a wide spectrum of chronic
care improvement programmes
DISEASE MANAGEMENT
• Coleman et al. [2008] distinguish programmes by the degree
to which they work to redesign the patients’ medical care
DISEASE MANAGEMENT
 Deficiencies in current health care systems:
• rushed practitioners not following evidence based
guidelines
• lack of care coordination
• lack of active follow-up to ensure the best outcomes
• patients inadequately trained to manage their illness
DISEASE MANAGEMENT
 Why system change?
• to put emphasize on system instead of only on physician
• to categorize characteristics of successful interventions in
an useful way
• to appreciate commonalities across chronic conditions
Contents
 Disease Management
 Disease Management Evaluation
 DISMEVAL, the project
 DISMEVAL, the Netherlands
DISEASE MANAGEMENT EVALUATION
• What we know about the impact of interventions to manage chronic
disease(s) tends to be based, mainly, on small studies that frequently
focus on high risk patients, and are often undertaken in academic
settings (Bodenheimer, Wagner and Grumbach, 2002).
• The effects of large, population-based programmes and initiatives
are less well understood (Mattke, Seid and Ma, 2007).
• Four examples: Steuten et al. (2006); Mattke et al. (2007);
Coleman et al. (2008, 2009); Peikes et al. (2009).
DISEASE MANAGEMENT EVALUATION
•
Steuten et al. (2006) found that a link between aims of disease
management and evaluated indicators does not exist in a substantial
part of published studies on disease management of diabetes and
asthma/COPD, especially when efficiency of care is concerned.
DISEASE MANAGEMENT EVALUATION
•
Mattke et al. (2007) report that the evidence on the role of
disease management in reducing utilization of health service is
inconclusive. Support for disease management “is more an article
of faith than a reasoned conclusion grounded on well researched
facts”.
DISEASE MANAGEMENT EVALUATION
•
Coleman et al. (2008) found that the degree to which disease
management redesigns practice predicts more positive outcomes.
The impact of the Chronic Care Model on health care costs and
revenues remains uncertain and probably varies by condition (2009).
DISEASE MANAGEMENT EVALUATION
• Peikes et al. (2009) conclude that care coordination, as practiced
by the programs participating in the demonstration from 2002 to 2006,
holds little promise of reducing total Medicare expenditures for
beneficiaries with chronic illnesses.
DISEASE MANAGEMENT EVALUATION
• The overall results of disease management evaluation studies are
hard to interpret.
• This is in part because of the variation in interventions, their
constituting components, and the lack of standardization of initiatives
(e.g. design, scope, scale, operational detail and providers involved
vary widely).
• Another challenge is the lack of widely accepted evaluation methods
to measure and report programme performance at the population
level in a scientifically sound fashion and that are also practicable for
routine operations.
•
Contents
 Disease Management
 Disease Management Evaluation
 DISMEVAL, the project
 DISMEVAL, the Netherlands
DISMEVAL
• Aim: to support the evaluation of disease management by
identifying and validating evaluation methods and performance
measures for disease management programmes or equivalent
approaches and to make recommendations to policymakers,
programme officials and researchers on best practices that are
both scientifically sound and operationally feasible.
• DISMEVAL brings together a multi-disciplinary team of 10
partners in 7 EU countries. It is funded under the European
Commission’s 7th Framework Programme, theme Health and
runs from 1 January 2009 to 31 December 2011.
DISMEVAL
Evaluating chronic disease management in Europe -- A review
of evaluation methods and performance measures in use by
Conklin A, Nolte E, Vrijhoef HJM (submitted)
1/3
• Review on approaches to chronic disease management in 12
European countries and data on methods and metrics to evaluate
these approaches
• Austria, Denmark, England, Estonia, France, Germany, Hungary,
Italy, Latvia, Lithuania, Netherlands, Switzerland
• key informants per country were identified and asked to collect data
on: (i) chronic disease management approach and (ii) evaluation
categories as defined in template
• data synthesis: narrative approach
DISMEVAL
Evaluating chronic disease management in Europe -- A review
of evaluation methods and performance measures in use by
Conklin A, Nolte E, Vrijhoef HJM (submitted)
2/3
• most chronic disease management approaches underwent some
form of evaluation, except in Latvia and Lithuania
• nature and scope of evaluations varied considerably
• mix of controlled and non-controlled studies measuring
predominantly clinical process measures, patient behaviour and
satisfaction, cost and utilisation
• effects were usually observed over 1-3 years on a population of
patients with a single, commonly prevalent, chronic disease
DISMEVAL
Evaluating chronic disease management in Europe -- A review
of evaluation methods and performance measures in use by
Conklin A, Nolte E, Vrijhoef HJM (submitted)
3/3
• majority of approaches were targeting diabetes and/or
cardiovascular disease(s)
• most commonly aimed to assess the performance of a given
approach with rarely specified information about targets against
effects might be compared
• about half of evaluations used non-experimental designs without
means for comparison
• large differences in indicators of effect, length of observation
• evaluations are carried out by different actors
DISMEVAL
Disease management evaluation – A comprehensive review of
current state of the art by Conklin A, Nolte E (RAND 2010)
1/3
• aim: to help advance the methodological basis for chronic disease
management evaluation by providing a comprehensive inventory of
current evaluation methods and performance measures, and by
highlighting the potential challenges to evaluating complex
interventions such as disease management
• two-stage literature review: (i) methodology papers, (ii) evaluation
of disease management
• 111 papers (89 journal articles, 6 book chapters, 10 working
papers, 6 other)
DISMEVAL
Disease management evaluation – A comprehensive review of
current state of the art by Conklin A, Nolte E (RAND 2010)
2/3
• challenges identified are conceptual, methodological, and analytical
in nature
• conceptual: clarification of characteristics of disease management
intervention and the selection of evaluation measures
• methodological and analytical: the establishment of the
counterfactual is a key challenge -- in the context of multi-component,
multi-actor disease management initiatives, RCTs are frequently not applicable
because randomisation is not possible (or desirable) for reasons such as cost,
ethical considerations, generalisability, and practical difficulties of ensuring
accurate experimental design.
DISMEVAL
Disease management evaluation – A comprehensive review of
current state of the art by Conklin A, Nolte E (RAND 2010)
3/3
• as alternative strategies become less of a controlled experiment, there are
more threats to the validity of findings from possible sources of bias and
confounding which can undermine the counterfactual and reduce the utility of
the evaluation
• other analytical challenges: selection of suitable control group, determining
statistical power, case mix, etc.
• a clear framework of the mechanisms of action and expected effects that
draws on an understanding of the characteristics of disease management,
those of the intervention and target populations, an adequate length of
observation to measure effects and the logical link between performance
measures and the intervention’s aims, elements and underlying theory driving
the anticipated behaviour change
Contents
 Disease Management
 Disease Management Evaluation
 DISMEVAL, the project
 DISMEVAL, the Netherlands
DISMEVAL: The Netherlands
Going beyond the ‘grand mean’ – Advancing approaches to disease
management evaluation in the Netherlands by Elissen A, Duimel-Peeters
IGP, Spreeuwenberg C, Vrijhoef HJM
• most experience in disease management approach in NL is related
to care for people with diabetes
• disease management is organized by ‘care groups’: provider
networks based in primary care
• insurers pay a single fee to care groups, to cover an integrated
bundle of care for a specific chronic disease over a fixed period of
time (usually one year)
• minimal required services are stipulated in national care standards
DISMEVAL: The Netherlands
Going beyond the ‘grand mean’ – Advancing approaches to disease
management evaluation in the Netherlands by Elissen A, Duimel-Peeters
IGP, Spreeuwenberg C, Vrijhoef HJM
• aim: to advance current methods of disease management
evaluation by testing and validating potentially valuable research
designs on data from existing programmes set in the Netherlands
• assumption: advanced methods are necessary to go beyond a
‘grand mean effect of disease management’ and gain insight into
what (combination of) components of the approach work(s) and for
whom.
• use was made of data from 9 care groups, already collected by
Dutch National Institute for Public Health and the Environment, and
an other 9 care groups retrospectively retrieved for this study
DISMEVAL: The Netherlands
Going beyond the ‘grand mean’ – Advancing approaches to disease
management evaluation in the Netherlands by Elissen A, Duimel-Peeters
IGP, Spreeuwenberg C, Vrijhoef HJM
• experimental comparisons are not feasible, as the DM approach to
T2DM care has been dispersed across all regions of the Netherlands
• two research designs were applied, i.e. meta-analysis and metaregression, which are assumed to be particularly suitable for
evaluating a heterogeneous care strategy that differs across settings
in terms of the specific interventions being offered, targeted patients,
and level of implementation
• included outcomes are restricted to clinical endpoints, as data on
more patient-centered measures, such as quality of life or satisfaction
with care, as well as on the costs of care could not be retrieved
retrospectively.
DISMEVAL: The Netherlands
>8000
2000-8000
<2000
●●● Frontrunner groups
●●● Non-frontrunner groups
• 18 care groups
• data of over 105,000
diabetes patients
• baseline 8-12 months
and follow-up 12 months
DISMEVAL: The Netherlands
DISMEVAL: The Netherlands
Going beyond the ‘grand mean’ – Advancing approaches to disease
management evaluation in the Netherlands by Elissen A, Duimel-Peeters
IGP, Spreeuwenberg C, Vrijhoef HJM
• meta-analysis of individual patient data (IPD), i.e. the raw
individual level data for each study, which is considered by some as
the ‘gold standard’ of systematic review (Thompson & Higgins, 2005; Stewart,
Tierney & Clarke, 2011). We applied meta-analysis methods to determine
the impact of the programmatic approach on IPD concerning changes
in eight clinical outcomes between baseline and follow-up.
• meta-regression can examine multiple individual and group level
factors together, the results of which may facilitate stratified
medicine, and adjust for baseline factors
Care group
1
Frontrunner status1
Size of patient population2
Price of care
bundle (€) 3
Collaboration with specialist
care4
351,00
Limited
367,48
Limited
1
No
2
No
3
No
18,531
345,72
Limited
4
No
7913
455,23
Extensive
5
No
1308
358,00
Limited
6
No
9392
380,04
Extensive
7
No
5778
390,00
Extensive
8
No
7192
458,20
Limited
9
No
13,754
298,92
Limited
10
Yes
348
-
-
11
Yes
4948
-
-
12
Yes
1539
-
-
13
Yes
9756
-
-
14
Yes
1072
-
-
15
Yes
7636
-
-
16
Yes
1893
-
-
17
Yes
1614
-
-
18
Yes
2174
-
-
1480
8728
Frontrunners are subsidized by the government as part of the experimental pilot into bundled payments for integrated diabetes care; 2 Size of patient population during groupspecific research period of either 18 or 24 months (see figure 2); 3 Price per care bundle per year on 01/01/2010; 4 Limited collaboration is defined as ‘primary care providers having
an agreement with specialists that enables them to consult the latter for advice’, whereas extensive collaboration implies ‘more intensive cooperation, for instance in the form of
patients being seen by specialists in the primary care setting’.
Care group
Frequency (median)
HbA1c
Range (%)1
Cholesterol
LDL
HDL
Triglycerides
SBP
DBP
BMI
1
1
1
1
1
1
4
4
3
69.7
2
3
1
1
1
1
4
4
2
55.7
3*
-
-
-
-
-
-
-
-
86.7
4
3
1
1
1
1
3
3
2
44.4
5
2
1
1
1
1
4
4
3
65.4
6*
-
-
-
-
-
-
-
-
84.6
7*
-
-
-
-
-
-
-
-
84.5
8
2
2
2
2
2
4
4
3
74.7
9
4
1
1
1
1
3
3
1
44.4
10
2
1
1
1
1
3
3
3
57.2
11
2
1
1
1
1
3
3
3
74.0
12
4
1
1
1
1
4
4
4
76.8
13
2
1
1
-
1
4
4
4
73.0
14
2
1
1
1
1
3
3
3
64.1
15*
-
-
-
-
-
-
-
-
65.6
16
1
1
1
1
1
2
2
1
73.9
17
1
1
1
1
1
3
3
2
55.1
18
4
1
1
1
1
4
4
2
81.6
Range was operationalized as the percentage of patients having registrations of all eight included clinical outcomes during the follow-up year. *Care groups 3, 6, 7, and 15 registered
solely the endpoints measured during the annual check-up of patients. Hence, information on frequency of measurements was unavailable for these groups.
1
Characteristics
Patients for whom characteristic is known
Estimate
(total =105,056)
Baseline age
Baseline diabetes duration
Sex
% (N)
Mean ± SD
99.9 (105,014)
65.7 ± 11.9
71.9 (75,498)
4.8 ± 5.6
% (N)
% (N)
99.3 (104,369)
Male
49.3 (51,421)
Female
Medication
50.7 (52,948)
65.0 (68,298)
No
36.0 (24,606)
Yes
Type of medication
64.0 (43,692)
41.6 (43,692)
Oral
80.5 (35,163)
Insulin
7.9 (3460)
Both
Comorbidity1
11.6 (5069)
94.5 (99,278)
None
84.2 (75,357)
One or more
Smoking status
15.8 (14,165)
74.6 (78,384)
No or Ex-smoker
81.6 (63,943)
Current smoker
18.4 (14,441)
% (N)
Mean ± SD
Baseline HbA1c (mmol/mol) [target <53]
71.5 (75,127)
50.2 ± 9.8
Baseline total cholesterol (mmol/l)
58.4 (61,376)
4.5 ± 1.0
Baseline LDL (mmol/l) [target < 2.5]
55.9 (58,697)
2.6 ± 0.9
Baseline HDL (mmol/l)
51.8 (54,456)
1.2 ± 0.4
Baseline triglycerides (mmol/l)
58.1 (61,078)
1.8 ± 0.9
Baseline SBP (mmHg) [target < 140]
69.9 (73,437)
140.4 ± 18.0
Baseline diastolic blood pressure (mmHg)
69.6 (73,115)
78.60 ± 9.6
Baseline BMI (kg/m2) [target < 25]
60.3 (63,341)
29.7 ± 5.2
1Included were four major comorbidity associated with diabetes mellitus: angina pectoris, myocardial infarction (MI), stroke, transient ischemic attack (TIA).
DISMEVAL: The Netherlands
Going beyond the ‘grand mean’ – Advancing approaches to disease
management evaluation in the Netherlands by Elissen A, Duimel-Peeters
IGP, Spreeuwenberg C, Vrijhoef HJM
• Main results from meta-analysis:
Endpoint
Care groups (N)
Patients (N)
Mean difference [95%CI]
Heterogeneity (I2, P)
HbA1c (%)
18
75,127
0.17 [-0.60, 0.93]
98%*
Total cholesterol (mmol/l)
18
61,376
-0.10 [-0.14, -0.06]*
90%*
LDL (mmol/l)
18
58,697
-0.09 [-0.13, -0.05]*
93%*
HDL (mmol/l)
17
54,456
0.02 [0.00, 0.03]*
92%*
Triglycerides (mmol/l)
18
61,078
-0.05 [-0.07, -0.03]*
75%*
SBP (mmHg)
18
73,437
-0.95 [-1.25, -0.64]*
57%*
DBP (mmHg)
18
73,115
-0.80 [-0.93, -0.67]*
34%
BMI (kg/m2)
18
63,341
-0.04 [-0.10, 0.02]
0%
* Statistically significant (p<0.05)
DISMEVAL: The Netherlands
Going beyond the ‘grand mean’ – Advancing approaches to disease
management evaluation in the Netherlands by Elissen A, Duimel-Peeters
IGP, Spreeuwenberg C, Vrijhoef HJM
• subgroup analysis were performed on patient, care group and
process characteristics. Most notably:
• DM is more effective for patients who have poor baseline clinical
values (HbA1c, LDL, SBP) than for those with better baseline health
• impact of DM for BP is greater in patient in higher age categories
• care group characteristics did not reduce heterogeneity
• patients whose cholesterol was measured more frequently show
better results
DISMEVAL: The Netherlands
• Main results from meta-regression:
Groups (N)
HbA1c (mmol/mol)
Care group characteristics
Change in variance
Intercept
Frontrunner subsidy
Care group size
Process characteristics
Change in variance
Intercept
Measurement frequency
Measurement range
Length of follow-up
Patient characteristics
Change in variance
Intercept
Age
Disease duration
Baseline HbA1c (mmol/mol)
Comorbidity
Smoking status
Significant covariates
Change in variance
Intercept
Measurement range
Length of follow-up
Age
Disease duration
Baseline HbA1c (mmol/mol)
Smoking status
* Statistically significant (p<0.05)
18
18
14
13
13
Patients (N)
75127
75127
48235
42659
42659
Coefficient
τ2
0.1810
-0.5230
0.9698
0.000038
-0.4845
-0.02744
-0.09095*
0.1141*
20.8566*
-0.03388*
0.1471*
-0.3869*
0.1954*
0.2458*
21.1761*
-0.1441*
0.06357*
-0.03288*
0.1468*
-0.3873*
0.2477*
σ2
ICC
2.8001*
2.6095*
-6.8%
62.6311*
62.6311*
0%
3.5274
26.0%
62.5406
-0.1%
2.4510*
-12.5%
47.9356*
-23.5%
2.5665
-8.3%
47.8876
-23.5%
4.28%
DISMEVAL: The Netherlands
Going beyond the ‘grand mean’ – Advancing approaches to disease
management evaluation in the Netherlands by Elissen A, Duimel-Peeters
IGP, Spreeuwenberg C, Vrijhoef HJM
• meta-regression revealed that most variance exists within groups
• no evidence was found for relationship between care group
characteristic on any clinical outcome
• process characteristics: as the duration of care increases, the
positive effects of DM diminish; higher measurement frequency is
associated with better outcomes
• patient characteristics: impact of DM is greater as patients’ baseline
values are poorer and this is more determinative than other
characteristics
DISMEVAL: The Netherlands
Going beyond the ‘grand mean’ – Advancing approaches to disease
management evaluation in the Netherlands by Elissen A, Duimel-Peeters
IGP, Spreeuwenberg C, Vrijhoef HJM
• from a simple pre-post comparison one would conclude that DM
does not achieve the intended goals
• from mete-regression and meta-analysis it was found that the
variance within care groups is much higher than between care groups
• the poorer patients’ baseline values of a particular endpoint are, the
more beneficial frequent measurement of that outcome is
• in other words: intensive DM programs targeting patients at high
risk of diabetes complications have great potential for costeffectiveness
Take home messages
 ‘Disease management’ presents a spectrum of chronic care
improvement programmes ranging from no till substantial
attempt to redesign practice
 Measuring disease management programme performance at
the population level in a scientifically sound fashion while
being practicable for routine operations remains a challenge
 DISMEVAL illustrates that other designs (than RCT)
constitute valuable designs for the evaluation of complex
interventions and provide in-depth insights
 Future evaluation of ‘disease management’ should focus on
the identification of subgroups and target these subgroups
THANK YOU
H.J.M.VRIJHOEF@UVT.NL
INTEGRATED CHRONIC DISEASE MANAGEMENT FORUM,
DEPARTMENT OF HEALTH,
29TH AUGUST, MELBOURNE