The Free Market Place of
US Guidelines
Salomeh Keyhani MD MPH
Department of Health Evidence and Policy
Mount Sinai School of Medicine
James J. Peters Bronx VAMC
Acknowledgement
• Azalea Kim MS1
• Micah Mann MD
• Deborah Korenstein MD
Dr. Keyhani is funded by a VA HSR&D
Career Award. This project was not
directly funded by any grants.
Background
• High costs of care has led many researchers
and policymakers to call for evidence based
practice to improve the quality of health care
delivery.
• Less attention paid to the quality of guidelines
developed in the US.
• No one national entity sets health care
standards and the quality of the guidelines
issued by different organizations is unclear.
Background
The Institute of Medicine has identified eight important
attributes of guidelines:
•
•
•
•
•
•
•
•
Validity
Reproducibility
Reliability
Clinical applicability
Clinical flexibility
Clarity
Multidisciplinary process
Scheduled review
Research Objective
• --To examine the number, quality and
specificity of current screening guidelines
available for cancer and cardiovascular
disease prevention in the US.
• --To compare government and non
government issued guidelines in terms of
quality and recommendations.
Methods-Sample
Cancer Prevention (n=11)
Breast Cancer
Cervical Cancer
Colorectal Cancer
Prostate Cancer
Ovarian Cancer
Bladder Cancer
Lung Cancer
Pancreatic Cancer
Oral Cancer
Testicular Cancer
Skin Cancer
Cardiovascular Disease
Prevention n=6
Diabetes Mellitus
Hypertension
Lipid Disorders
Abdominal Aortic Aneurysm
Carotid Artery Stenosis
Peripheral Vascular Disease
Method-Data Collection
• Systematically collected guidelines from the National
Guideline Clearinghouse, organization websites and
Medline.
Inclusion Criteria:
• Unique screening guidelines in the relevant medical
and/or surgical specialty or subspecialty, disease groups
and national entities
• 2002Exclusion Criteria:
• Information directed toward patients
• Best practices guides
Methods—Data Collection
• Guidelines classified as developed by:
1- Government
2-Non Government
- Medical society
- Disease organization
- Other entity
Methods—Data Collection
•
Modified clinical practice guideline appraisal instrument developed
by Shaneyfelt et al.
•
Items divided into four domains based on content area:
1) General Guideline Development Details
(purpose, panelists, area of expertise, multidisciplinary, intended audience)
2) Evidence Extraction and Synthesis
(time period evidence extraction, identifying evidence, data extraction, grading
evidence, systematic review, outside review, update )
3) Specificity of Guideline Content
(start age, stop age, interval, high risk population, harms, benefits)
4) Costs
Methods--Analysis
• Determined number of unique guidelines for each disease.
• Each guideline was evaluated using the modified version
Shaneyfelt tool.
• Determined the proportion of criteria each guideline met.
• Compared government-issued guidelines to pooled nongovernment-issued guidelines using chi square statistics.
• Examined the impact of each guideline on the number of
life time screens for that condition.
• Comparisons were based on a healthy, low risk patient
with a life expectancy of 75 years.
• Analysis was performed using STATA statistical software
version 9.0.
Results
• We identified 63 unique screening guidelines.
• 19 government, 24 medical society, 10 disease
group and 10 from “other” category.
• Disease groups: AHA, ADA, ACS
• Other groups: included the National Cancer
Care Network and the Institute for Clinical
Systems Improvement.
• 89% of government guidelines were from the
USPSTF
• On average each condition had 3 guidelines -range 1-9.
Results
Government
Sponsored
Non
Government
Sponsored
p-value*
General Guideline
Development Details
N=19
N=44
Purpose of guideline
specified
100%
100%
ns
Rational and importance
explained
100%
93%
ns
Participants and their areas
of expertise specified
94.7%
29.5%
<0.001
Is the panel multidisciplinary
100%
64%
<0.001
Patient population defined
100%
97%
ns
Intended audience described
100%
79.5%
p<0.05
Results
Government
Sponsored
Non
Government
Sponsored
p-value*
Evidence Extraction and Synthesis
N=19
N=44
Time period of evidence is specified
100%
31.8%
p<0.001
Method of identifying evidence is
specified
100%
18.1%
p<0.001
Method of data extraction is specified
100%
27.2%
p<0.001
Method for classifying scientific
evidence specified
100%
47.7%
p<0.001
Formal method of combining
evidence is described
100%
40.9%
p<0.001
Was a systematic review performed?
100%
20.4%
p<0.001
Recommendations graded according
to the strength of the evidence
89%
36.3%
p<0.001
Evidence used is identified by citation
100%
68.1%
P<0.01
Criteria for reassessment of evidence
and/or date of review is specified
5.2%
38.6%
p<0.001
Results
Specificity of Guideline content and Costs
• No difference between guidelines in terms of
specification of a start age, stop age or
screening interval.
• Few guidelines of any type recommended a stop
age for screening (28%-36%).
• Guidelines developed by the government more
commonly discussed benefits (100% vs. 70%,
p<0.05) and harms (81.8% vs. 39.4%, p<0.05) of
screening.
• The impact of screening on health care costs or
provide any cost-effectiveness analysis was
rarely addressed (10-11%).
Results-Recommendations
• The average number and range of screens
varied by issuing entity.
• The ACS guidelines would lead to 36 breast
cancer screens for a healthy 40 year old woman,
ICSI called for 13 screens in the same time
period.
• The ACS guidelines would lead to an average of
25.9 cervical screens during a healthy young
woman’s life. In contrast, USPSTF called for 15
screens in the same time period.
• Greater variability in recommendations for
prostate cancer screening--- “insufficient
evidence to recommend for or against
screening” to 15 recommended screens.
Results-Recommendations
• The population for which screening was
recommended also varied.
Example: Diabetes
• ACOG---screen all women over the age of 45 for
diabetes every 3 years.
• USPSTF screen adults with sustained blood
pressure > 135/80 ( interval unknown).
• AHA calls for screening of all patients every 5
years.
Conclusions
• Guidelines are issued by medical societies,
disease organizations, government bodies, and
institutes.
• The majority of guidelines issued by nongovernment entities appear to be of poor quality
and do not meet IOM standards.
• Same evidence drawn upon to develop
guidelines, but screening recommendations
differ.
Limitations
• Only publicly information used in
assessing guidelines.
• Our estimates of life time screens were
based on several simple assumptions that
may not accurately reflect the trajectory of
screening for a patient during their life
time.
• A more rigorous analysis would have
included decision analysis and the impact
of a positive screen on the trajectory of
screening.
Implications
• The lack of a national entity that sets
standards and monitors quality has led to
the development of a free market place of
guidelines in the US.
• This variability in recommendations can
impede efforts by policymakers to improve
the quality of care delivered in the US.
• It is doubtful that such a plethora of
guidance is serving the best interests of
patients, physicians or the tax paying
public.