How the California Health
Benefits Review Program
Assesses Medical
Effectiveness
June 30, 2009
Janet M. Coffman, MA, MPP, PhD
Assistant Professor
Philip
h l R. Lee Institute for
f Health
l h Policy
l
Studies
d
&
Department of Family & Community Medicine
University of California, San Francisco
Outline
Medical effectiveness review process
Medical effectiveness findings
Challenges encountered
2
Medical Effectiveness
Review Process
3
Purpose of CHBRP’s
CHBRP s
Medical Effectiveness Reviews
Assess the medical effectiveness of health
insurance benefit mandates based on the
best available scientific evidence
Form a foundation for the cost and public
health analyses
4
Steps in CHBRP’s
CHBRP s Medical
Effectiveness Review Process
Prepare literature
literat re re
review
ie specifications
(consult with content expert)
Perform literature search (medical librarians)
Review abstracts
R t i
Retrieve
pertinent
ti
t articles
ti l
Extract information from articles
M k a qualitative
Make
lit ti ““call”
ll” on th
the lit
literature
t
Summarize quantifiable evidence for specific
h lth outcomes
health
t
if ffeasible
ibl
5
Sources of Evidence
 Relyy heavily
y on PubMed,, the Cochrane databases,,
and other indices of peer-reviewed publications
 Consult sources of grey literature such as the
National Guideline Clearinghouse (NGC) and the
National Institute for Health and Clinical Excellence
(NICE)
 Apply same hierarchy of evidence to peer-reviewed
literature and grey literature
6
Hierarchy of Evidence
 Meta-analyses
 Systematic reviews
 Evidence-based guidelines
 Well-designed
Well designed randomized controlled trials
 Randomized controlled trials with major
weaknesses
 Nonrandomized studies with comparison groups
 Case series and case reports
 Clinical/practice guidelines based on expert
opinion and narrative reviews
7
Review Criteria
Strength
St
th off research
h design
d i
Statistical significance
Direction of effect
Size of effect
Generalizability to California’s population
8
Findings from Medical
Effectiveness Reviews
Findings
CHBRP h
has completed
l t d medical
di l
effectiveness analyses for 51 of the 58
reports
t issued
i
d tto date
d t
Some bills do not lend themselves to a
y
traditional medical effectiveness analysis
10
Findings
Among
g bills for which medical
effectiveness analyses completed
# of Reports
p
(n = 51)
Evidence of effectiveness from studies with strong
research
hd
designs
i
25
Evidence of effectiveness from studies with weak
research designs
5
Insufficient or conflicting evidence
3
Strength of evidence varied across diseases or
services addressed
7
No studies of effectiveness identified
11
11
Challenges in
Conducting Medical
Effectiveness Reviews
12
Challenges
Political dynamics
Differences in expertise in health
services research
Limitations of the medical literature
13
Challenges
 Limitations of the medical literature







Types of interventions assessed may vary
Outcomes may not be measured consistently
Literature may not address all important outcomes
Evidence may be inconsistent
N rigorous
No
i
studies
t di
Direct evidence of effectiveness may not be available
Strength of evidence may vary across conditions
addressed
14
Challenges
 Variation in interventions assessed:
AB 264 – Pediatric Asthma Education
Type of Education Provided*
Provided
Group classes
Individual In-person
In person
Internet/computer-based
Number of Sessions
Duration
(n = 47 studies)
38%
55%
19%
2 – 90
2 wks. - 18 mths.
* Percentages do not sum to 100% because some interventions encompassed more
than one type of education
15
Challenges
 Variation in strength of evidence across conditions:
AB 1774 – Gynecological Cancer Screening
Cancer
Cervical
Endometrial
Ovarian
State of Evidence re Screening
Evidence that screening reduces
morbidity and mortality
No studies of screening
Screening
g can detect cancer at an
earlier stage but has no impact on
survival
16
Challenges
 Lack of studies:
SB 1634 – Orthodontics for Oral Cleft
 Orthodontic procedures are a component of standard
care for oral clefts along with surgery and speech
therapy
 No
N studies
t di off th
the added
dd d b
benefit
fit off orthodontic
th d ti
procedures
 Guidelines are based on expert opinion
17
Conclusions
 Challenging but possible to systematically review
medical
di l lit
literature
t
within
ithi th
the titight
ht titime constraints
t i t off th
the
legislative process
 There is evidence of effectiveness for many of the health
care services for which benefit mandates are proposed
 However, evidence is lacking for many other services
 Communicating findings to legislators is an ongoing
challenge
18