EVIDENCE BASED MEDICINE
A new approach to clinical care and research
Developed and presented by
Judy Tarselli, RN
Dubai, UAE
Karachi, Pakistan
October 2003
Organized by NKF cyberNephrology
University of Alberta, Canada
www.cyberNephrology.org
Special thanks to our sponsors Janssen-Cilag
PROGRAM OUTLINE
1.
2.
3.
4.
5.
6.
7.
Definition of EBM
Basic Steps
Trials, Studies and Reports
Pros, Cons and Limitations
EBM in Developing Countries
EBM Library
Advanced EBM
But first, a test…
WHAT IS THE BASIS OF YOUR
MEDICAL PRACTICE?
(Check all that apply)
A.
Training, clinical experience and consultation
with other professionals
B.
Convincing evidence (non-experimental) from
articles, case reports, product literature, etc.
C.
Preferences of the patient
D.
Active search of Randomized Controlled Trials,
Systematic Reviews, Meta-Analysis Reports
WHAT IS THE BASIS OF YOUR
MEDICAL PRACTICE?
EXCELLLENT!
A.
Training, clinical experience and consultation
with other professionals
B.
Convincing evidence (non-experimental) from
articles, case reports, product literature, etc.
C.
Preferences of the patient
D.
Active search of Randomized Controlled Trials,
Systematic Reviews, Meta-Analysis Reports
BUT… Past knowledge and practice
might be outdated or inadequate
Graduate Medical School
Practiced Physician
WHAT IS THE BASIS OF YOUR
MEDICAL PRACTICE?
FANTASTIC!
A.
Training, clinical experience and consultation with
other professionals
B.
Convincing evidence (non-experimental) from articles,
case reports, product literature, etc.
C.
Preferences of the patient
D.
Active search of Randomized Controlled Trials,
Systematic Reviews, Meta-Analysis reports
BUT… This evidence may be biased, outdated,
incorrect, or not applicable to your patient
JOURNALS (1987 to present)
ARTICLES
ADVERTISEMENTS
WHAT IS THE BASIS OF YOUR
MEDICAL PRACTICE?
WONDERFUL!
A.
Training, clinical experience and consultation with other
professionals
B.
Convincing evidence (non-experimental) from articles,
case reports, product literature, etc.
Mutual Respect +
Shared Goals =
Better Cooperation
and Compliance
C.
Preferences of the patient
D.
Active search of Randomized Controlled Trials,
Systematic Reviews, Meta-Analysis reports
The patient should be involved in
all important decisions
But this is NOT always an easy task!
And conflicts WILL occur!
No salt?
Lose weight?
Forget it!
Just give me a pill!
I WON’T take that medicine…
The side effects are
INTOLERABLE!
But doctor, I DO want
to have children!
And conflicts WILL occur!
No salt?
Lose weight?
Forget it!
Just give me a pill!
I WON’T take that medicine…
The side effects are
INTOLERABLE!
But doctor, I DO want
to have children!
Education about current alternatives and risks is often
needed… for both the Patient and the Doctor!
Wow…
I never knew that high
blood pressure could
be so dangerous at my
age!
Yes, I’d like to try that
new medication!
I’ll discuss those risks
with my husband.
Education about current alternatives and risks is often
needed… for both the Patient and the Doctor!
An important rule in Evidence Based Medicine…
It STARTS with the patient and ENDS with the patient.
The patient’s preferences MUST be considered!
WHAT IS THE BASIS OF YOUR MEDICAL
PRACTICE?
WOW!!! SUPERB!!!
A.
Training, clinical experience and consultation with other
professionals
B.
Convincing evidence (non-experimental) from articles,
case reports, product literature, etc.
C.
Preferences of the patient
D.
Active search of Randomized Controlled Trials,
Systematic Reviews, Meta-Analysis reports
In the practice of Evidence Based Medicine,
it is the physician’s duty to find the best and
most current information and apply it
judiciously for the benefit of the patient.
But… A practice based exclusively on science and math
is effective only if your patients are robots or clones!
Don’t forget to allow for individual human differences
and personal preferences!
WHAT IS THE BASIS OF YOUR
MEDICAL PRACTICE?
If you checked all 4 items…
A.
Training, clinical experience and consultation with other
professionals
B.
Convincing evidence (non-experimental) from articles,
case reports, product literature, etc.
C.
Preferences of the patient
D.
Active search of Randomized Controlled Trials,
Systematic Reviews, Meta-Analysis reports
CONGRATULATIONS!
You are practicing
EVIDENCE BASED
MEDICINE!
A.
Training, clinical experience and consultation with other
professionals
B.
Convincing evidence (non-experimental) from articles, case reports,
product literature, etc.
C.
Preferences of the patient
D.
Active search of Randomized Controlled Trials, Systematic
Reviews, Meta-Analysis reports
EVIDENCE BASED MEDICINE
A new approach to clinical care and research
1.
2.
3.
4.
5.
6.
7.
Definition of EBM
Basic Steps
Trials, Studies and Reports
Pros, Cons and Limitations
EBM in Developing Countries
EBM Library
Advanced EBM
“What is Evidence Based Medicine?”
“And where did it come from?”
A BRIEF HISTORY
1980’s: McMasters University in Ontario, Canada
Dr. David Sackett and colleagues proposed Evidence
Based Medicine (EBM) as a new way of teaching, learning
and practicing medicine.
Dr. Sackett defines EBM as:
“…The conscientious, explicit, and judicious use
of current best evidence in making decisions
about the care of individual patients.”
Evidence Based Medicine
It is a change in the way physicians practice medicine, teach and
learn, and handle research.
Clinical practice: Based on the best current evidence
(not necessarily on how it’s always been done)
Patient Care: Compassionate, patient-oriented
(less authoritarian)
Learning & Teaching: Problem-based, problem-solving
more investigative, less know-it-all-by-yesterday
Research: More stringent approach, better proof criteria
(more demanding of proof, less room for error)
THREE MAJOR
COMPONENTS of
EBM
PATIENT
Question
or
Problem
PHYSICIAN
INFORMATION
THE ADDED DETAILS
PATIENT
Values, Concerns Preferences,
Expectations
Life predicament
EBM
PHYSICIAN
Training & Experience
Current Expertise
Continued learning
Demand for proof
INFORMATION
Clinically relevant
Proven by research
Best up-to-date
evidence
OPTIONAL
COMPONENTS
PATIENT
Values, Preferences
Concerns, Expectations
Life predicament
TO BE ADDED BY
THE PHYSICIAN
CHARITY
EBM is not a
required practice
(yet)
HUMILITY
Non-authoritarian
practice
EBM
PHYSICIAN
Training
Expertise
Continued Learning
Demand for proof
INFORMATION
Clinically relevant
ENTHUSIASM
Challenge, Variety, Proven by research
Current, up to date
Change
“Isn’t this the way
we have always
practiced medicine?”
“Aren’t these just the
same old ingredients
tossed into a new
recipe?”
When am I supposed to find
the time to do that?
The basic steps of EBM
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
The Clinical Question
The FIRST step
The HARDEST step
The MOST IMPORTANT step!
FACT: We all have informational needs!
That is not a problem!
Problems arise
• if we fail to recognize those needs
• if we fail to bridge the information gap
• if we fail to ask the right questions
Asking good questions
is a skill to be learned.
Lee, exactly how
much time did you
spend on that big
project?
It will make life
easier for you...
And also for others
around you!
Hmmm… Is he
about to give me a
BONUS?
Or is he about
to FIRE me?
Lee, can you give me an
accounting of the extra time
you spent on that project so
that I can charge it back to
the client?
A GOOD QUESTION…
•
Is focused and relevant
•
Provides clear
communication
•
Clarifies your goal or need
•
Will reduce the amount of
time needed to obtain the
answer
Oh sure! I’ll have it
on your desk by
tomorrow!
WHEN PRACTICING EBM,
a good question must also:
•
Be specific
Identify the problem, clarifiy
the clinical issue
•
Be answerable
through the literature
•
Contain multiple aspects
(patient, options,
comparisons, etc)
ACTUAL CASE SCENARIO
Large cauc male, age 40
2mo ago: Presented with classic
nephrotic syndrome, significant
symptoms. Bx showed IgAN. Cr
1.4, incr to 2 range, now 1.6
Tried prednisone 60mg qd tolerated poorly w/tremors and
depression.
Needs new regimen, but others
are aimed more at nephritic IgA
rather than nephrotic syndrome.
Suggestions?
It should NOT involve a
question of Personal Preference
or Local Concern.
THE EVIDENCE BASED RESPONSE
Posted on Nephrol 4/13/03
Respondant recommends cyclophosphamide and
prednisolone (assuming secondary causes excluded) - a
combination that allows for lower dose prednisolone…
“In the study below, proteinuria and renal
function improved on this combination:
Ballardie FW, Roberts IS. Controlled prospective
trial of prednisolone and cytotoxics in progressive
IgAN. J Am Soc Nephrol 2002 Jan….”
“I have patients on this regime who have
benefitted.”
Regards,
Dr. Paulose P. Thomas
Nephrologist - Belhoul Apollo Hospital, Dubai, UAE
BACKGROUND and FOREGROUND QUESTIONS
(all part of EBM)
FOREGROUND QUESTIONS
NEW POSSIBILITIES
INDEFINITE ANSWERS
“Where do we want to go,
and how else might
we get there?”
“Where are we now?
And which way are we headed?”
BASIC & CONCRETE
BACKGROUND QUESTIONS
STUDENT
GRAD
EXPERT
BACKGROUND QUESTIONS
BASIC & CONCRETE
1. QUESTION
• Who, What, Where, When, Why, How
2. VERB
• is, causes, does, treats, reduces, cures, prevents, affects
3. GENERAL KNOWLEDGE ABOUT DISORDER
clinical manifestations of disease, patient findings, differential
diagnosis, etiology, patient experience, comorbid condition,
screening and diagnostic tests, prognosis, therapy, risk factors,
etc.
STUDENT
GRAD
EXPERT
FOREGROUND QUESTIONS
NEW POSSIBILITIES
INDEFINITE ANSWERS
PT AND/OR PROBLEM Differential dx, Unusual presentation, uncertain
etiology, pt’s prior experience, comorbid conditions
INTERVENTION Exposure, test. Prognostic factor,
treatment, pt perception, etc.
COMPARISON INTERVENTION
OUTCOMES
STUDENT
GRAD
EXPERT
EBM QUESTION: Should include multiple factors
(Examples)
P
PATIENT
type of patient or population
Ex: 47 yr male w/DM2 and cellulitis toe, 25 yr female w/DVT and chest pain
E
EXPOSURE
environmental, personal, biological
Ex: TB, tobacco, drug, diet, pregnancy or menopause, MRSA, allergy
I
INTERVENTION
clinical intervention
Ex: medication, procedure, test, surgery, radiation, drug, vaccine
C
COMPARISON compare alternative treatment
Ex: other prior, new or existing therapy
O
OUTCOME
clinical outcome of interest
Ex: Reduced death rate in 5 yrs, decreased infections, fewer hospitalizations
FRAMING THE QUESTION (Example: PICO)
ELEMENT
PROMPTS THE QUESTION:
Patient
Intervention
Comparison
Outcome
How would I describe a group of patients similar to mine?
What main action am I considering?
What is/are the other options?
What do I (or the patient) want to happen (or not happen)?
Example:
P:
In kids under age 12 with poorly controlled asthma on metered
dose inhaled steroids…
I: would the addition of salmetrol to the current therapy
C: compared to increasing the dose of current steroid
O: lead to better control of symptoms without increasing side effects?
CATEGORY OF QUESTION
MAJOR CATEGORIES
1.
2.
3.
4.
Diagnosis
Prognosis
Therapy/ Treatment
Harm (iatrogenic, other)
MISCELLANEOUS
• Quality of care
• Health economics
• Office Management
• Etc.
PICO
PEO
THE PATIENT’S QUESTIONS
Must be considered!
Often QUALITATIVE (not based on measureable outcomes)
Feelings, ideas, experiences, preferences, concerns, fears, beliefs,
ethnicity
Usually based on LIMITED BACKGROUND
Perception of problem
Self-diagnosis
Treatment wanted or needed
Alternatives (read, heard, considered, tried)
What is the patient hoping to avoid?
What benefits does the patient want or need most?
Etc.
QUANTITATIVE vs QUALITATIVE QUESTIONS
QUANTITATIVE: “Solid Evidence”
•
Measurable answer or response
•
Necessary for scientific study
•
Necessary for the practice of EBM
QUALITATIVE: “Quality of Life”
•
“Fuzzy” data - Impact on daily life, work, family, etc.
•
May be very important and influential to decisions –
especially for the patient
•
Creates added challenge or twist to practice of EBM
QUALY: QUALITY ADJUSTED LIFE YEAR
CHRONIC MEDIOCRITY
PERFECT HEALTH
10
9
8
7
6
5
4
3
2
1
2
3
4
5
6
7
8
9
10
10
9
8
7
6
5
4
3
2
1
3
4
5
6
7
8
9
10
5-yrTREATMENT COMPARISON
DETERIORATING HEALTH
9
8
7
6
5
4
3
2
1
0
2
10
8
Chemo&Rad
6
4
Radiation only
2
Chemo only
0
2
3
4
5
6
7
8
9
10
1
2
3
4
5
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
Find the Best Evidence
“The Literary Search”
HINT: If your desk looks like this, it’s probably the
LAST place you should start looking!
Find the Best Evidence
“The Literary Search”
The BEST EVIDENCE is:
External - from outside resources (researchers, experts)
Current – not out of date, most recent
High Quality - accurate, precise, effective, safe
Patient focused - applicable and appropriate for your individual
patient
FIVE STEPS TO FINDING THE BEST EVIDENCE
1.
IDENTIFY NEEDS:
What type of information is needed?
2.
IDENTIFY RESOURCES:
Types, Availability, Timeliness,Costs?
3.
SEARCH & RETRIEVE:
Use efficient strategies
4.
REVIEW :
Check quality and usefulness of info
5.
INTERPRET:
Help patient understand info, application
WHAT TYPE OF INFORMATION IS NEEDED?
WHAT CATEGORY IS THE QUESTION?
•
•
•
•
Diagnosis
Prognosis
Therapy
Harm
WHAT STUDY DESIGN FITS IT BEST?
There are MANY study designs!
EXPERIMENTAL TRIALS
(Answers questions of diagnosis or treatment)
Randomized Controlled Trials (RCTs)
Controlled studies
Blinded vs Open
ETC.
OBSERVATIONAL STUDIES
Descriptive reports
Retrospective studies
Cohort studies
Case Control
ETC.
EXAMPLE
Randomized Controlled Trials (RCT)
“Gold Standard” of research
Ideal experimental design - Best design for TREATMENT questions
Must identify objective of treatment
(Ex: cure, prevent complication, palliation, reassurance)
Still not always the right intervention for individual patient at that particular time and
place
What type of evidence best addresses the question, problem or issue?
CLINICAL PRACTICE
APPROPRIATE DESIGN FOR CLINICAL RESEARCH
Diagnosis, Dx testing
Cross-sectional study – not randomized trial
Prognosis
Follow-up studies of patients evaluated at same early point of illness
Therapy, treatment
RCT or Systematic review of multiple RCTs must be used
Avoid non-experimental approaches to avoid false conclusions about efficacy
Exceptions:
When treatment may be successful in an otherwise fatal condition
When no studies are available (rare conditions, new treatments, etc.)
Harm
RCT, Cohort, Case-control
OTHER INFORMATIONAL
Explore hypothesis
History-taking
Individual trial & error
Following clinical course
Recordkeeping
Quality of Care research
Qualitative research
Case control study
n of 1 trial
Cohort study
Systematic registry-based (computer supported) research
Individual peer review, Process Evaluation
MISCELLANEOUS
Basic Science, Genetics, Immunology, etc.
WHAT FORM OF INFORMATION?
Case report
Controlled Trial
Systematic review
Meta-analysis
Clinical guidelines
etc.
LITERARY SEARCH: NEXT STEP
IDENTIFY YOUR RESOURCES
Colleagues
Consultation, Discussion
(Caution: Response may be an outdated “This is what we do”)
Paper resources
books, reports, journals
Electronic databases
Health Literature Services
specialized librarians, staff
Review services, Abstract Services, etc.
SEARCH AND RETREIVE THE BEST EVIDENCE
Learn and Practice various SEARCH STRATEGIES:
• To find useful information quickly
• To eliminate irrelevant, inappropriate or weak information
Try to develop the habit of learning as you go;
Not just in lengthy formal sessions!
LITERARY SEARCH STRATEGY
ASK FOR HELP!
SPECIALIZED PERSONNEL
• track down information, textbooks, articles,
guidelines
• may provide electronic search support or training
EXAMPLES
• Medical Librarians
• Medical Informatics Specialists
• Specially trained staff member
LITERARY RESOURCES
•
TEXTBOOKS (caution – most obsolete!)
• Traditional
• Evidence Based
•
JOURNALS (may be outdated)
•
REVIEW ARTICLES (summaries, abstracts)
•
SYSTEMATIC REVIEWS (prepared in systematic, rigorous
manner) Ex: Cochrane Collection
•
META-ANALYSIS
•
CLINICAL PRACTICE GUIDELINES
Summarized and easily digestible information
ELECTRONIC RESOURCES, DATABASES, INTERNET
Bibliographic Database
Example: Medline, PubMed
Medical Information Services: Medscape, HDCN
Review Services
Subjective
Systematic Reviews
Meta-analysis
Examples:
• Cochrane,
• Best Evidence,
• Up to Date
MORE GREAT INTERNET RESOURCES
Websites
cyberNephrology, National Kidney Foundation. NIDDK,
American Heart Association, American Cancer Society.
National Institutes of Health, etc
Listserve Discussion Groups
CyberNephrology, C-span, etc.
Specialty Electronic Databases
Psyclit
CancerLit
CINAHL
(allied health and nursing journals)
Etc
OTHER RESOURCES
Tapes
Videos
CD-ROMs
Specialty seminars
Product information and comparisons
A closer look at some Internet Resources…
MEDLINE
WHAT IS IT?
Searchable database of medical information compiled by National Library of
Medicine in US 1966-present
Catalogs articles from approx 4000 world journals (of estimated 12-15k total)
SEARCH METHODS
Any word or words (title, abstract, content, author name, institution, etc.)
Medical Subject Heading (MeSH) terms
A restricted thesaurus of medical titles
Articles categorized by most specific possible MeSH heading
COST: FREE!
Or may subscribe to companies with specialized search strategies:
•
Ovid Technologies (ovid)
•
Silver Platter Information (WinSPIRS)
BENEFITS
Free
Vast database
LIMITATIONS
Not all articles are indexed on Medline (only 1/3 of approx 10 million!)
Much material listed and described on Medline can only be accessed through
journal article
MEDLINE: ELECTRONIC SEARCH STRATEGIES
Search through “Clinical Queries” service of PubMed
http://www.ncbi.nlm.nih.gov/clinical.html
Medical Subject Headings (MeSH)
Search filters
Search by a text word can supplement a MeSH search
Boolean search: “and”, “not”, etc.
To increase sensitivity
• use “explode” command
• avoid using subheadings
Online Tutorial is available!
COCHRANE LIBRARY
Cochrane Database of Systematic Reviews
-systematically compiled reviews of intervention
Cochrane Controlled Trials Register
-citations of controlled trials identified anywhere in the world
Cochrane Review Methodology Database
-methodological papers relating to systematic reviews
Etc.
BEST EVIDENCE
Electronic version of two publications:
•
Evidence Based Medicine
•
American College of Physicians Journal Club
Covers broad topics of information
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
CRITICAL APPRAISAL
Interpreting the evidence
• How to read a paper
• How to do the math
CRITICAL APPRAISAL
IMPORTANT!
You do NOT have to become a researcher,
epidemiologist, or statistician to practice EBM.
Focus on how to USE research reports –
not on how to generate them!
CRITICAL APPRAISAL
HOWEVER…
You must have a solid understanding of
basic research principles and
study designs in order to understand
and interpret the evidence!
TYPES OF STUDIES AND REPORTS
Randomized Controlled Trial - “The Gold Standard”
Systematic review
Meta-analysis
Retroactive vs Prospective
Incidence
Prevalence
Case Control
Cohort (Follow-up)
Cross-sectional
Ecologic
Longitudinal
Experimental
Blinded vs Open
Qualitative Screening
DETOUR
BASIC RESEARCH PRINCIPLES
STUDY DESIGNS
THE TIME FACTOR
When was the study done?
What was its duration?
In what time direction is it headed?
RETROSPECTIVE
PROSPECTIVE
THE TIME FACTOR
When was the study done?
What year?
What technology? (ie: test, drug, equipment, procedure)
Any associated social factor or historical event?
THE TIME FACTOR
What was the Study Duration?
Was it an appropriate length of time for the
intended goal?
Limited time study or ongoing?
Was study completed? Stopped early?
In what direction is it headed?
RETROSPECTIVE
“LOOKING BACK”
Historical Review or
Investigation
PAST
PROSPECTIVE
“LOOKING FORWARD”
Future Results
The Great Unknown
PRESENT
FUTURE
In what direction is it headed?
RETROSPECTIVE
PROSPECTIVE
PRO
•Lower risk of bias
PRO
•May provide good
direction for future study
CON:
May get faulty results based
on incomplete data or
insignificant subgroups
“Hind Sight is 20/20”
CON:
•Prone to Bias
•A“Fishing Expedition” for
positive results
PRESENT
(Example of Error: Untreated
hypertension unlikely to cause
cardiac event in child, so treatment
is unnecessary below age 18yrs)
“Was there a similar comparison group?”
No comparison group
All subjects receive Experimental Intervention
Experimental
Intervention
UNCONTROLLED STUDIES
NO EVENT
Experimental
Intervention
OUTCOME
EVENT
“Trial and Error?” or “Before & After?”
UNCONTROLLED STUDIES
Generally NOT accepted:
Potentially Dangerous and Flawed
Prone to BIAS!
“Traditional Study Method”
May produce strong results
“Trial and Error”
“Before & After”
PROBLEMS
BENEFITS
POSITIVE OUTCOME MAY BE DUE
TO:
•Other factors
•Natural course of disease (some
get better, some don’t!)
Can answer some questions
about:
•likelihood of response
•adverse effect, etc.
•Spontaneous change of health
•Placebo Effect
VERY PATIENT-SPECIFIC!
•Hawthorne Effect
NEGATIVE OUTCOME
May be due to study treatment.
Could be disastrous!
MAY BE ONLY OPTION
Rare conditions
Previously unknown conditions
UNCONTROLLED TRIALS:
“TRIAL AND ERROR”
Example#1
SMALLPOX
VACCINATION
James Phipps,
age 8 years
GOOD!
Resistant to
Cowpox and
Smallpox
(NO DISEASE
OUTCOME)
SMALLPOX VACCINE
1. 1796: Edward Jenner inoculates 8yr-old James Phipps with cowpox virus
from a milkmaid’s hands.
Child develops illness, recovers.
2. Two weeks later, inoculates same child with smallpox virus.
Child survives, no illness.
(Centuries later, smallpox eradicated!)
n=1
UNCONTROLLED TRIALS:
“TRIAL AND ERROR”
Example #2
Drinks culture of
H.pylori
Dr. Marshall
Microbiologist
n=1
NO
OUTCOME
SEVERE
GASTRITIS
HELICOBACTER PYLORI - GASTRIC ULCERS
1982: Australian microbiologist Barry J. Marshall presents evidence showing a
possible infectious cause for gastric ulcers. Suggests they may be treatable with
antibiotics.
Findings are met with disinterest and disbelief by medical community. Lacks
support for further study.
5 years later: Prepares a broth of live organisms isolated from a gastric ulcer
patient and drinks it. Becomes violently ill, develops severe acute gastritis.
1990’s Antibiotics are used routinely to cure some gastric ulcers!
UNCONTROLLED TRIAL
RECOVERED
Experimental
Intervention
May represent the ONLY
treatment option for a new or rare
disease
Present
DIED
FUTURE
STRONGLY PREFERRED! Reduces BIAS. Provides stronger results.
Experimental
Intervention
Control
Group
CONTROLLED STUDY
Only the TEST group receives the Experimental Intervention
ExperimentalI
ntervention
Control group may receive…
Nothing
Placebo
Observation only
Other
IMPORTANT
All other differences
should be minimized or
eliminated to reduce
potential BIAS
Gold Standard
Treatment
RANDOMIZED CONTROLLED TRIAL (RCT)
“The Gold
Standard”
Experimental
Intervention
Control
Group
THE FIRST RANDOMIZED CONTROLLED TRIAL
By Sir Austin Bradford Hill
Streptomycin
(n=50)
(BLINDED)
Bedrest
(n=50)
1944 TUBERCULOSIS TREATMENT: Streptomycin vs Bedrest
OPEN vs BLINDED STUDIES
Experimental
Intervention
OPEN
Control
Group
OPEN vs BLINDED STUDIES
BLINDED
TRIAL
BLINDED
BLINDED
TRIAL
BLINDING
SINGLE BLINDED:
Pt unaware of what group s/he is in
DOUBLE BLINDED:
Pt and MD unaware
OPEN LABEL:
Everyone is aware
RANDOMIZED vs NON-RANDOMIZED TRIALS
Experimental
Intervention
How is this
group divided?
Control
Group
NON-RANDOMIZED
Experimental
Intervention
Assigned to
groups, usually
by the
researcher
Control
Group
Potential for RESEARCHER BIAS!
RANDOMIZED
Experimental
Intervention
Random method of
assignment used
Control
Group
Maximizes “sameness,” Eliminates BIAS!
RANDOMIZED CONTROLLED TRIAL (RCT)
(EXPERIMENTAL TRIAL)
Experimental
Intervention
“The Gold Standard”
Control
Group
Present
FUTURE
One patient, series of tests
n=1
TRIAL SERIES FOR INDIVIDUAL PATIENT
GOOD
GOOD
Experimental
Intervention
Experimental
Intervention
Trial of Medicine
1
Trial of Medicine
2
Or placebo
Or placebo
NO CHANGE
OR BAD
NO CHANGE
OR BAD
Why a TRIAL SERIES for one patient?
BENEFIT
Produces data most applicable to the individual patient
EXAMPLES:
Trial of different medications and/or placebo for child reported
to have ADHD symptoms that are not clinically apparent
Trial of different analgesics for patient with chronic pain from
a combination of diseases not previously studied
PATIENT
•Must be blinded
•Must keep diary or complete
questionnaire
PHYSICIAN
•May need to be blinded (enlist help
of pharmacist!)
•Must treat patient as usual in all
other respects
CROSSOVER TRIALS
ONE GROUP, MULTIPLE TESTS
(Best if participants are blinded)
Intervention
A
Intervention
Intervention
A
Intervention
B
B
ASSESS
OUTCOMES #1
ASSESS
OUTCOMES #2
COMPARE OUTCOMES
CROSSOVER TRIALS
PROS & CONS
Fewer participants needed than a RCT!
Intervention
A
Intervention
Intervention
A
Intervention
B
B
ASSESS
OUTCOMES #1
ASSESS
OUTCOMES #2
All are in experimental group
Lower costs
CROSSOVER TRIALS
PROS & CONS
MUST HAVE SHORT CARRYOVER EFFECT
MUST HAVE SHORT WASHOUT EFFECT
Intervention
A
Intervention
Intervention
A
Intervention
B
B
ASSESS
OUTCOMES #1
ASSESS
OUTCOMES #2
(OR WAIT A SUITABLY LONG WASHOUT TIME!)
CASE CONTROL
(“A LOOK BACK”)
RISK FACTOR?
(PAST)
Present
CASE CONTROL
(“A LOOK BACK”)
HEALTHY
NEVER
SMOKED
RISK FACTOR
LUNG CANCER
SMOKER
(PAST)
Present
CASE CONTROL
(“A LOOK BACK”)
NON-DIABETIC
NORMAL
WEIGHT
RISK FACTOR
DM TYPE II
OBESITY
Present
COHORT“FOLLOWUP DESIGN”
IS RISK
FACTOR
PRESENT?
(Exclude those
with outcome
already!)
Future Outcome
COHORT
TO INVESTIGATE ETIOLOGY
OR HYPOTHETICAL CAUSE
OF DISEASE/OUTCOME
IS RISK
FACTOR
PRESENT?
“FOLLOWUP DESIGN”
Present
Future Outcome
COHORT
EXAMPLE
RISK FACTOR
Hgb <9
DIALYSIS PATIENTS
Present
Measures future outcome for
dialysis pts w/o treatment of anemia
CROSS SECTIONAL DESIGN
? Cause ? Risk factors
A look back
CROSS SECTIONAL DESIGN
OTHER CAUSES
RISK = SLEEP PRONE
SIDS DEATHS
INFANT
DEATHS
Problems of looking back
NON-SIMILAR
CONDITIONS
Social
Personal
Comorbid conditions
Other treatments
Etc.
Not usually
accepted by
medical journals
(accepted in
popular press,
not reviewed)
VARIATION IN
TREATMENT
OR METHOD
NO
CONTROL
OVER
CONTROL
GROUP
CURRENT
GROUP OF
PATIENTS
RANDOMIZED & CONTROLLED TRIAL (RCT)
Experimental
Intervention
MAY BE
BLINDED
Control
Group
PROSPECTIVE
START WITH YOUR TARGET POPULATION
START WITH YOUR TARGET POPULATION
Set CRITERIA for
INCLUSION / EXCLUSION
This will determine:
ELIGIBILITY at the start
VALIDITY at the end
START WITH YOUR TARGET POPULATION
ELIMINATE THOSE WHO DO NOT MEET THE CRITERIA
NEXT: GATHER A SAMPLE GROUP
THE SAMPLE GROUP WILL:
•Represent the target population
•Meet the criteria for inclusion / exclusion
SIDE NOTES…
Study should be approved by an
Ethics Committee
Informed consent should be
obtained from study participants
SAMPLE GROUP MAY BE SUBDIVIDED FURTHER
STRATIFICATION
Divide into subgroups based on
important similar characteristics
RANDOMIZATION
Divide into sub-groups based on
unknown confounders
STRATIFICATION
“important similar characteristics”
Examples:
• Male or Female
• Age
• Stage of illness
• Prior illness or treatment
• Hospital vs Office groups
• Comorbid condition
• Etc.
EXAMPLE OF
STRATIFICATION
FEMALE
MALE
RANDOMIZATION
“unknown confounders”
Examples:
• Postal code
• Month of birth
• Random number
• Etc.
EXAMPLE OF
RANDOMIZATION
DX IN JANUARY-JUNE
DX IN JULY-DECEMBER
Next… Divide your sample group(s) into STUDY GROUPS
Experimental
Intervention
Control
Group
“Test Group”
“Baseline Group”
Next… Divide your sample group(s) into STUDY GROUPS
“Test Group”
Experimental
Intervention
Receives Experimental
Intervention
“Baseline Group”
Control
Group
•
•
•
•
•
Nothing
Observation
“Same” miscellaneous
intervention (nonexperimental)
Placebo
“Gold Standard” therapy especially if unethical to do
otherwise!
ASSIGN PATIENTS TO STUDY GROUPS
Experimental
Intervention
Use caution against bias!
Control
Group
Sample Group
Study Groups
STUDY INVESTIGATOR
 usually assigns
patients to study
groups.
Experimental
Intervention
 usually has a
personal preference
for the treatment or
patient
 might unconsciously
Control
Group
“work harder” to
make the study work
with non-preferred
candidates
= POTENTIAL FOR
BIAS
RANDOMIZED CONTROLLED TRIAL (RCT)
Experimental
Intervention
Use random
separation
and assignment!
Control
Group
RANDOMIZED CONTROLLED TRIAL (RCT)
Experimental
Intervention
Control
Group
RANDOMIZED CONTROLLED TRIAL (RCT)
Experimental
Intervention
Control
Group
Present
Proceed with study
FUTURE
RANDOMIZED CONTROLLED TRIAL (RCT)
Experimental
Intervention
EXPERIMENTAL EVENT
RATE (EER)
Control
Group
CONTROL
EVENT RATE
(CER)
RANDOMIZED CONTROLLED TRIAL (RCT)
“The Gold
Standard”
Experimental
Intervention
EXPERIMENTAL EVENT
RATE (EER)
Control
Group
Present
CONTROL
EVENT RATE
(CER)
FUTURE
Disadvantages of RCT
Expensive
large # pts needed
Prolonged recruitment and follow-up time needed
Funding difficult to obtain except w/support of
pharmaceutical companies (problematic!)
RETURN FROM DETOUR
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
CRITICAL APPRAISAL
Interpreting the evidence
• How to read a paper
• How to do the math
EVALUATE WRITTEN EVIDENCE FOR…
Quality
Usefulness
…BY ASSESSING
Validity
Reliability
Relevance
Clinical importance
Critical Appraisal: VALIDITY
What was the original purpose of the study?
When was it prepared?
By whom?
•
•
credentials?
affiliations?
Sample population
Did the subjects represent an appropriate test group?
How were they selected?
Were controls used?
Were groups similar for important prognostic characteristics?
VALIDITY
How was the information gathered and processed?
Were groups treated equally except for trial therapy?
Were appropriate criteria used to measure results?
Were criteria applied rigorously?
Was the study completed?
(Or ended early for a specified reason?)
Did the study account for all test subjects?
Including subjects lost to follow-up?
Were ALL pts analyzed in their allocated groups?
(ie: INTENTION TO TREAT - not “completed treatment” analysis)
VALIDITY
Information
Does the paper support its claims?
Is the information accurately presented?
Does it represent the truth?
Results
Are the results believable?
To what degree of confidence?
Ex: Disagreement is not uncommon on angiograms, EKGs, radiographs, pathology, PAP
tests, etc.
VALIDITY
Comprehensiveness
Size: Was it large enough to yield credible results?
Thoroughness: Was it complete enough?
Duration: Was it long enough?
CRITICAL APPRAISAL: RELIABILITY
Do we trust the information and results?
1. APPROPRIATE TYPE OF STUDY
2. REPRODUCEABILITY
3. INTERPRETATION OF RESULTS
4. BIAS
RELIABILITY
APPROPRIATE TYPE OF STUDY
Was the type of study design used proper for the question?
Example:
RCT would be choice for questions on TREATMENT
RELIABILITY
Are the Measurements and Results reproducibile?
Different determinations may be caused by:
• Variation in measurement methods
• Different interpretation of results
• Lack of agreement
Example:
BP checks on same patient may vary. Are differences result of pt factor, examiner
factor, treatment factor, normal variance
Would the same results be obtained if patient is re-measured?
(with identical procedure)
• at another time?
• by another person?
Were any similar studies done?
• Was the information comparable?
• Did the results agree?
RELIABILITY
INTERPRETATION OF RESULTS
Is there consistency among researchers?
Different determinations may be caused by:
• Variation in measurement methods
• Different interpretation of results
• Lack of agreement
EXAMPLE:
BANFF CONFERENCE - Setting standards in Transplant Pathology
established by Kim Solez, MD
Were any new questions or controversies raised by the study?
RELIABILITY
IS THERE ANY EVIDENCE OF BIAS?
A dangerous pitfall!
•
PATIENTS
•
RESEARCHERS
PATIENT BIAS
Social Desirability Bias
Patient responds in the way
they perceive as correct
Patient denies unhealthy
behavior, gets misclassified
•
•
Ex: Smoker vs Non-smoker
to support MD
to support a preconceived
notion (ie: foods vs ADD)
PATIENT BIAS
Hawthorne Effect
People act differently when
they know they are being
watched.
Authors must take steps to
reduce this bias by treating
all equally!
Ex: Follow more careful diet
when regular weigh-ins are
scheduled
Ex: Weigh all patients with same
frequency, even for group not on
special diet
RESEARCHER BIAS
Who sponsored or funded the study?
Personal gain or loss from results?
Affiliates
Special interests
Conflict of interest
Biased goal?
To satisfy editors and reviewers… rather than solve
real life clinical problems
RESEARCHER BIAS
Criteria bias?
Risk-avoidance by researchers
(will focus energy on topics that produce positive results)
Bias toward patients?
Sample selection criteria used (inclusive, exclusive)
Assignment to test group or control - Random? Blind?
RESEARCHER BIAS
Data collection methods used
•
applied similarly to all subjects, including controls?
•
starting point – prospective/retrospective, stage of patient?
•
Was assessment blind?
Data analysis
•
Were all potential subjects included in denominator or otherwise
accounted?
•
Were they evaluated in originally designated group?
(INTENTION TO TREAT)
REDUCING OR ELIMINATING BIAS AND ERROR
CONDUCT BLIND STUDIES
•
•
Single
Double-blinded
USE INDEPENDENT OBSERVERS
•
•
When doctor and/or patient can not be blinded, blinded IO measures outcome
IO may even be unaware of study hypothesis
USE MULTIPLE OBSERVERS
Ex: Send subject slides to multiple pathologists for interpretation
ESTABLISH CLEAR STANDARDS
•
•
Exact methods to use to reduce variation in technique among researchers
Clear wording on surveys, etc
VALIDATING INSTRUMENTS
•
•
Repeat screening to check for correct answers on surveys
More frequent evaluations or surveys prevent guesstimates common to less
frequent evaluation
NEXT STEP IN CRITICAL APPRAISAL:
RELEVANCE
QUESTION: Is the report applicable to our…
Problem?
“Does it address the questions raised?”
Patient(s)?
“Will my patient respond like those in the study?”
Practice?
“Can it be done within my practice or circle?”
ARE THE STUDY PATIENTS
•
Comparable within the study? (similar traits, age,
socioeconomic group, stage of illness, treatment, etc.)
•
Comparable to your patient?
ARE THE STUDY PROFESSIONALS
•
Comparable to you?
(general/specialist, primary care/teaching hospital, etc.)
NEXT STEP in CRITICAL APPRAISAL
CLINICAL IMPORTANCE
Information can be true and interesting in theory,
yet useless in clinical practice!
1. Is the information clinically important?
2. If yes, how important is it?
•
study design - See: Hierarchy of Evidence
•
weight of results
HEIERARCHY OF EVIDENCE
(value of study design to maximize wt, minimize bias)
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
Systematic Review of all relevant RCTs
At least one properly designed RCT
Trials and case studies
Well-designed Controlled Trial without Randomization
Well designed Cohort or Case Control Studies, preferably from >1
centre or group
Multiple Time series with or without intervention
(Exception: Dramatic results in uncontrolled trials, such as
introduction of PCN in the 1940s)
Opinions of respected authorities, based on
Clinical expertise
Descriptive studies
Reports of Expert Committees
RANDOMIZED CONTROLLED TRIAL (RCT)
Evaluation of RCT
Were all clinically appropriate outcomes measured?
Did an ethics committee approve the study?
Any statistically significant results also clinically significant?
Any significant adverse reactions?
Was follow-up procedural analysis identical?
Was continuous data analysis vs end of trial data used?
Interpreting the evidence
• How to read a paper
• How to do the math
HOW TO DO THE MATH
Incidence
Prevalence
Statistical Formulas
+/- Predictive Values - Probability - The p value
Relative Risk
Risk Reduction
Odds Ratios
NNT (Number Needed to Treat) – Risk Reduction
Confidence Intervals
Sensitivity and Specificity
Regression Analysis
Subgroup Analysis
Health Status Evaluation
Health Economics
OUTCOMES: NOT “STUDY FAILURES”
ACCOUNT FOR ALL even if
•Non-compliant
•Lost to follow-up
OUTCOMES RELATE TO EVERYDAY CLINICAL PRACTICE, including…
•
Deaths
•
Poor compliance
•
Wrong treatment received
•
Lost to follow-up
•
Etc.
Analyze as a member of the originally assigned group!
Analysis SHOULD BE BASED ON
• INTENTION TO TREAT
• NOT on “completed treatment” analysis
INCIDENCE & PREVALENCE
NEPHROL, a service of NKF cyberNephrology
7/10/03 10:17:12AM
Dear Nephrolers,
I would like to know how to calculate incidence and
prevalence of B and C virus in HD.
Thank you in advance.
Mario Cuba, MD
Servicio de Nefrologia
Hospital Lucia Iniguez Landin
Holguin, Cuba
INCIDENCE & PREVALENCE
Response from Michel Jadoul, MD
NEPHROL, a service of NKF cyberNephrology
Prevalence: total number of positive patients divided by total
number of patients: 20+/200=10%
Incidence: number of new positive cases/total number of
cases negative at start of period (e.g; year)/period (year?)
thus : for instance 2 new positive cases /100 negative cases
at start of year=2%/year.
M.Jadoul, M.D.
B
PREVALENCE
B
10 cases Hep B in 100 patients
10 / 100 = 0.10
B
PREVALENCE = 10%
B
B
B
B
B
B
B
B
INCIDENCE
B
B
1 year period
90 HBsA(-) at start of study
B
5 new cases
B
5 / 90 = 0.06
INCIDENCE = 6% per year
B
B
B
B
B
B
B
B
B
B
B
B
RECALCULATED PREVALENCE
B
16 cases Hep B in 100 patients
16 / 100 = 0.16
B
B
NEW PREVALENCE = 16%
B
B
B
B
B
B
B
B
B
B
COMPARISON STUDIES: NEW DIAGNOSTIC TESTS
COMPARING A NEW TEST AGAINST THE GOLD STANDARD TEST
RESULTS OF GOLD STANDARD TEST
EXPERIMENTAL TEST
DISEASE
PRESENT
POSITIVE
NO DISEASE
TRUE (+) FALSE (+)
a
b
NEGATIVE FALSE (-)
c
TRUE (-)
d
ACCURACY OF TEST - COMPARE TO GOLD STANDARD
What is the usefulness of the test in various groups and subgroups of pts?
TESTS ARE RARELY 100%
ACCURATE
EXPERIMENTAL TEST
THEY MUST BE COMPARED
AGAINST THE
GOLD STANDARD
TEST
POSITIVE
a+b
TEST
NEGATIVE
c+d
RESULTS OF GOLD STANDARD TEST
DISEASE
PRESENT
a+c
NO DISEASE
PRESENT
b+d
TRUE (+) FALSE (+)
a
b
FALSE (-)
c
TRUE (-)
d
ACCURACY OF TEST - COMPARE TO GOLD STANDARD
What is the usefulness of the test in various groups and subgroups of pts?
TESTS ARE RARELY
100% ACCURATE
EXPERIMENTAL
TEST
THEY MUST BE
COMPARED AGAINST
THE GOLD STANDARD
RESULTS OF GOLD
STANDARD TEST
TOTALS
DISEASE
PRESENT
DISEASE NOT
PRESENT
TEST
POSITIVE
TRUE (+)
a
FALSE (+)
b
a+b
TEST
NEGATIVE
FALSE (-)
c
TRUE (-)
d
c+d
a+c
b+d
TOTALS
a+b+c+d
ACCURACY OF TEST - COMPARE TO GOLD STANDARD
What is the usefulness of the test in various groups and subgroups of pts?
TESTS ARE RARELY
100% ACCURATE
EXPERIMENTAL
TEST
THEY MUST BE
COMPARED AGAINST
THE GOLD STANDARD
TEST
POSITIVE
RESULTS OF GOLD
STANDARD TEST
TOTALS
DISEASE
PRESENT
DISEASE NOT
PRESENT
TRUE (+)
a
FALSE (+)
b
a+b
TRUE (-)
d
c+d
SENSITIVITY
TEST
NEGATIVE
FALSE (-)
c
SPECIFICITY
TOTALS
a+c
b+d
a+b+c+d
SENSITIVITY AND SPECIFICITY
SENSITIVITY = PATIENT (+) TEST (+)
Probability that patient WITH disease
will have ABNORMAL result
(instead of False Negative)
SPECIFICITY = PATIENT (-) TEST (-)
Probability that patient WITHOUT disease
will have NORMAL result
(instead of False Positive)
OVERALL DISCRIMINATION OF TESTS
High SENSITIVITY = low number false negatives
High SPECIFICITY = low number of false positives
Best accuracy if both factors are close to 100%
SENSITIVITY = a / (a + c)
PATIENT (+) TEST (+)
SPECIFICITY = d / (b + d)
PATIENT (-) TEST (-)
POSITIVE PREDICTIVE VALUE = a / (a + b)
If pt tests (+), what is the likelihood s/he has the disease?
NEGATIVE PREDICTIVE VALUE = d / (c + d)
If pt tests (-), what is the likelihood s/he does NOT have the disease?
PREVALENCE = (a + c) / (a + b + c + d)
ACCURACY = (a + d) / (a + b + c + d)
Proportion of results that correctly identify pts with and without disease
(True + and True - as proportion of all results)
LIKELIHOOD RATIO = sensitivity / (1 - specificity)
How likely is it that + result accurately indicates disease, and - result no disease?
LIKELIHOOD RATIO (LR)
Because Sensitivity and Specificity are NOT always 100%
How likely is it that + result accurately indicates disease, and - result no disease?
LIKELIHOOD RATIO FOR A POSITIVE RESULT (LR+)
Probability of (+) result in diseased subject
divided by
Probability of (+) result in a healthy subject
- or worded differently -
Sensitivity
divided by
100% - Specificity
LIKELIHOOD RATIO FOR A NEGATIVE RESULT (LR-)
100% - Sensitivity
divided by
Specificity
DISCRIMINATION = ZERO IF LR = 1
EVALUATING STUDY RESULTS
Example:
Mortality rates in 4444 pts x 5.4 trial years:
11.5% Placebo
8.2% Medicine
RRR 29% (Relative Risk Reduction)
ARR 3.3% (Absolute Risk Reduction)
NNT 30 (Number needed to treat for 5.4
years to save 1 life
QALY = QUALITY ADJUSTED LIFE YEAR
QUALITY RATING
NOT specific for disease or treatment!
Value rating - subject to different values of patients, physician, community
• Patient-based
• Economy-based - cost-utility/cost-effectiveness analysis
Compares outcomes of conditions or intervention(s)
• State of health - vs -Time spent in it
QUALY: QUALITY ADJUSTED LIFE YEAR
CHRONIC MEDIOCRITY
PERFECT HEALTH
10
9
8
7
6
5
4
3
2
1
2
3
4
5
6
7
8
9
10
10
9
8
7
6
5
4
3
2
1
3
4
5
6
7
8
9
10
5-yrTREATMENT COMPARISON
DETERIORATING HEALTH
9
8
7
6
5
4
3
2
1
0
2
10
8
Chemo&Rad
6
4
Radiation only
2
Chemo only
0
2
3
4
5
6
7
8
9
10
1
2
3
4
5
TEST RESULTS: VARIABILITY
FACT: Study results may vary.
Group too small
Not representative
of larger group
May be discovered
or identified through
study
Etc.
Age, sex, race, condition, culture, etc.
Compliancy issues (patient and
physician!)
TEST RESULTS: VARIABILITY
FACT: Variability may or may not be significant
Group too small
Not representative
of larger group
May be discovered
or identified through
study
Etc.
Age, sex, race, condition, culture, etc.
Compliancy issues (patient and
physician!)
TEST RESULTS: VARIABILITY
Obviously faulty studies should be eliminated.
Group too small
Not representative
of larger group
May be discovered
or identified through
study
Etc.
Age, sex, race, condition, culture, etc.
Compliancy issues (patient and
physician!)
TEST RESULTS: VARIABILITY
Some variability should be expected in the rest.
Group too small
Not representative
of larger group
May be discovered
or identified through
study
Etc.
Age, sex, race, condition, culture, etc.
Compliancy issues (patient and
physician!)
TEST RESULTS: VARIABILITY
Some factors are completely unexpected.
Group too small
Not representative
of larger group
May be discovered
or identified through
study
Etc.
Age, sex, race, condition, culture, etc.
Compliancy issues (patient and
physician!)
The statistics allow us to distinguish between
and variability due to
PROBABILITY
Statistically significant
Results are measurable
and predictable
Affected by sample size
(1 in 20 is less convincing than
1 in 10,000)
WARNING
PROBABILITY should
NOT
be confused with
CHANCE!
CHANCE
No statistical
significance
Random, unpredictable
PROBABILITY
A Study in Probability…
QUESTION #1:
What percentage of patients will develop diarrhea while
taking Antibiotic A?
QUESTION #2:
Will the results be the same, better or worse on
Antibiotic B?
QUESTION #1:
What percentage of patients will develop diarrhea while
taking Antibiotic A?
50
50
50
50
50
50
50
50
50
50
50
50
50
20 study
groups
50
50
50
50
50 pts each
study
50
50
50
1000 patients
total
QUESTION #1:
What percentage of patients will develop diarrhea while
taking Antibiotic A?
NUMBER OF STUDIES
Conduct
studies
Organize
results
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16%
IDENTIFY STATISTICALLY SIGNIFICANT RESULTS
NUMBER OF STUDIES
The Bell Curve
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16%
MODE
NUMBER OF STUDIES
Most COMMON result =
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16%
MEDIAN
NUMBER OF STUDIES
The CENTER of distribution =
PERCENTAGE OF PATIENTS WITH DIARRHEA
4%
6
8
10
12
14
16%
DETERMINE RESULTS OF STUDY
NUMBER OF STUDIES
MODE (most COMMON result) = 10%
MEDIAN (the CENTER of distribution) = 10%
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16%
CONCLUSION
NUMBER OF STUDIES
“10% of patients will develop diarrhea while
taking Antibiotic A.”
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16%
Wait… What about the other 15 study groups?
NUMBER OF STUDIES
You can’t just
ignore them!
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16
Or can
you?
Wait… What about the other 15 study groups?
You can’t just
ignore them!
15 groups
x 50 per group
= 750 patients
(75%!)
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16
Or can
you?
Wait… What about the other 15 study groups?
You can’t just
ignore them!
Studies must
account for
ALL patients
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16
Or can
you?
Wait… What about the other 15 study groups?
NUMBER OF STUDIES
You can’t just
ignore them!
Results should
not be ignored,
but
STATISTICAL
SIGNIFICANCE
may be
questioned.
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16
Or can
you?
NUMBER OF STUDIES
So, how is
STATISTICAL
SIGNIFICANCE
determined?
My head is
starting to feel
heavy…
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16
STATISTICAL SIGNIFICANCE
PROBABILITY is determined by it.
CHANCE is not related to it at all!
How is it determined?
Let’s use our study on “Antibiotic A” as the example
20 groups were tested.
Only 1 of 20 groups landed at each end of the Bell Curve….
NUMBER OF STUDIES
WHY?
SAMPLE
VARIATION?
or CHANCE?
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16%
NUMBER OF STUDIES
It is tempting to say,
“There is a 1 in 20 chance that other patients
will land in these categories.”
But that
would NOT
be a correct
statement!
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16%
NUMBER OF STUDIES
It is tempting to say,
“There is a 1 in 20 chance that other patients
will land in these categories.”
Why?
Because
CHANCE
can not be
used to
predict
future
results!
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16%
CHANCE is based on RANDOM possibility.
Example: THE COIN TOSS
Coins tossed: 20
“Heads”
“Tails”
17
3
(85%)
(15%)
Statistical Significance: ZERO!
The next coin toss will still produce a random result!
Random results can not be used to calculate
Statistical Probability.
So instead of measuring “chance”…
“There is a 1 in 20 chance that patients will
land in one of these two categories.”
1 in 20
chance
1 in 20
chance
PERCENTAGE OF PATIENTS WITH DIARRHEA
4%
6
8
10
12
14
16%
We need to determine the PROBABILITY!
“There is a 5% probability that patients will
land in one of these two categories.”
1 in 20
chance
5%
Translates into
probability
PERCENTAGE OF PATIENTS WITH DIARRHEA
4%
6
8
10
12
14
16%
The results now look like this:
There is a
5% probability
that 4% of
patients will
develop
diarrhea on
Antibiotic A.
There is a
5% probability
that 16%of
patients will
develop
diarrhea on
Antibiotic A.
PERCENTAGE OF PATIENTS WITH DIARRHEA
4%
6
8
10
12
14
16%
And now… Let’s abbreviate it some more!
p = 0.05
DECIMEL
Probability
= 5%
1 in 20
chance
FRACTION
PERCENTAGE
PERCENTAGE OF PATIENTS WITH DIARRHEA
4%
6
8
10
12
14
16%
…by changing “% probability” to the “p value”
p = 0.05
DECIMEL
Probability
= 5%
1 in 20
chance
FRACTION
PERCENTAGE
PERCENTAGE OF PATIENTS WITH DIARRHEA
4%
6
8
10
12
14
16%
The “p value” is statistically important!
p = 0.05
DECIMEL
Probability
= 5%
1 in 20
chance
FRACTION
PERCENTAGE
PERCENTAGE OF PATIENTS WITH DIARRHEA
4%
6
8
10
12
14
16%
It determines statistical PROBABILITY.
p = 0.05
“p value”
Probability
= 5%
1 in 20
“chance”
4%
6
8
10
12
14
16%
PROBABILITY vs CHANCE
NUMBER OF STUDIES
…but PROBABILITY is
very important!
It tells us the likelihood that
something will happen.
So, CHANCE
has ZERO
significance
PERCENTAGE OF PATIENTS WITH DIARRHEA
4%
6
8
10
12
14
16%
OUR PROBABILITY STATEMENT
“There is a 5% probability that our study patients
will fall into either of these categories.”
p = 0.05
p = 0.05
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16%
THAT IS A LOW PROBABILITY!
Anything less than 5% (p= 0.05)
MAY be due to chance.
p = 0.05
p = 0.05
PERCENTAGE OF PATIENTS WITH DIARRHEA
4
6
8
10
12
14
16%
THAT IS A LOW PROBABILITY!
And anything less than 1% (p= 0.01)
is MOST LIKELY due to chance!
p = 0.01
4
6
8
10
12
14
16%
p = 0.01
THAT IS A LOW PROBABILITY!
Anything less than 5% (p= 0.05) MAY be due to chance.
Anything less than 1% (p= 0.01) is MOST LIKELY due to chance.
p = 0.05
p = 0.01
p = 0.05
4
6
8
10
12
14
16%
p = 0.01
But when compared to another study…
(p= 0.05) becomes VERY SIGNIFICANT
And (p= 0.01) becomes HIGHLY SIGNIFICANT!
4
6
8
10
12
14
16
18%
But when compared to another study…
(p= 0.05) becomes VERY SIGNIFICANT
And (p= 0.01) becomes HIGHLY SIGNIFICANT!
4
6
8
10
12
14
16
18%
NUMBER OF STUDIES
Antibiotic A
PERCENTAGE OF PATIENTS WITH DIARRHEA
4%
6
8
10
12
14
16
18%
NUMBER OF STUDIES
Antibiotic B
PERCENTAGE OF PATIENTS WITH DIARRHEA
4%
6
8
10
12
14
16
18%
DIFFERENCE = STATISTICALLY SIGNIFICANT
(p= 0.05) becomes VERY SIGNIFICANT
And (p= 0.01) becomes HIGHLY SIGNIFICANT!
4
6
8
10
12
14
16
18%
The P value
Measures Probability
How often is this finding expected to occur?
Determines Statistical Significance
What is the likelihood these findings are TRUE or FALSE?
Do the comparative findings show a significant difference?
Meaningful ranges
p >0.05
Not significant
p <0.05
Statistically SIGNIFICANT
p <0.01
HIGHLY SIGNIFICANT!
Does probability provide PROOF?
NO! We could be misled by it.
The sample size is very important when determining probability!
SAMPLE SIZE and PROBABILITY
EXAMPLE:
100 pieces of fruit are in a bin: APPLES and ORANGES
You close your eyes and pick 10 of them:
Question: Does your sample accurately
represent what is in the bin?
Answer: No!
Larger samples provide a closer approximation of the
populations they represent... But the only way to get
100% proof is to examine “all of the fruit in the bin!”
The P value
Meaningful ranges
p >0.05
Not significant
p <0.05
Statistically SIGNIFICANT*
p <0.01
HIGHLY SIGNIFICANT!**
*Significance only means that CHANCE is an unlikely explanation for the
results
LIMITATION
The p value determines LIKELIHOOD… Not proof!
CAUTION
Statistical significance does not necessarily imply any clinical
significance!
EXAMPLE: Looking through a pinhole will improve vision in most people… But would
this be an appropriate treatment for your myopic patients? (Key Topics in EBM )
MISCELLANEOUS STUFF
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
PATIENT
Question
or
Problem
PHYSICIAN
INFORMATION
(The Three Major Components of EBM)
PATIENT
PHYSICIAN
INFORMATION
PATIENT
PATIENT
“A METHODOLOGICAL
MINEFIELD”
PATIENT
“A
METHODOLOGICAL
Difficult time
Personal MINEFIELD”
understanding
priorities may
background
conflict with
information
yours
PHYSICIAN
INFORMATION
PATIENT
Recognize:
Needs
Choices
Preferences
Values
Socioeconomic
concerns
CONFLICT?
SEPARATE THE ISSUES!
Respect the
personal
priorities of the
patient
PHYSICIAN
Help the patient
to negotiate a
decision on
intervention,
treatment
Help the patient to
understand and
interpret available
information
INFORMATION
PATIENT
And then help the
patient pull it all
together again
PATIENT
PHYSICIAN
INFORMATION
KEY POINTS
PARADIGM SHIFT
OLD:
Doctor had authority
(despite the pile of unread journals!)
NEW: Current Best Evidence leads medical practice
but it MUST be individually applied
THE INDIVIDUAL PATIENT
Every patient is different. Treat YOURS and not others
The “ideal” course of action is not necessarily best for THIS patient.
EBM + Psychosocial factors =
THIS patient should be advised to take
THIS therapy at
THIS point in time.
THE FIVE BASIC STEPS OF EBM
1. Clinical Question
Patient-focused, problem-oriented
2. Find Best Evidence
Literary Search
3. Critical Appraisal
Evaluate evidence for quality and usefulness
4. Apply the Evidence
Implement useful findings in clinical practice
5. Evaluate
The information, intervention, and EBM process
Evaluate
INFORMATION
Adequate resources?
Ease or Difficulty of finding and getting desired information?
Costs?
INTERVENTION
Patient response or acceptance?
Ease or Difficulty of Application?
Clinical outcomes?
EBM PROCESS
EFFECT ON PRACTICE
Will this particular experience change our thinking or practice?
SELF EVALUATION
How did we do? (Question, Search, Appraise, Apply)
How could we improve our own EBM performance?
EBM: PROS, CONS and LIMITATIONS
PROS
Clinicians update knowledge base routinely
Improved understanding of research methods
Physician becomes more critical in use of data
Increased confidence in management decisions
Increased computer literacy, data search technology
Better reading habits
Provides framework for group problem solving, team generated
practice
Transforms weakness or paucity of knowledge into positive change
OK to be uncertain
OK to be skeptical
OK to be flexible
Integrates medical education, research and clinical expertise
Can be learned by non-clinicians – other HCWs, patient groups,
purchasers, etc.
Allows us to keep up with our better-educated patients!
Increased contribution of junior MDs
Increased patient benefit
Better communication with patients re: rationale of management
decisions
Promotes better and more appropriate use of limited resources
May reduce costs or medical care or practice by eliminating outdated or
unnecessary factors
Can be learned at any stage of physician’s career
CONS
Time consuming
Information overload
Time to learn and practice
Time may be needed for team conferencing, planning and review
Takes $$$ to establish resource infrastructure – library, office, etc.
computers, peripherals
Internet costs
Programs, software information, CD-ROMS
Subscription costs – online and paper resources
May increase cost of care (but hopefully offset by elimination of
unnecessary medical interventions, tests, journals, etc. – plus save time
in getting proper intervention)
Online references made to unavailable journals or references
Exposes gaps in the evidence (but provides ideas for researchers!)
Requires computer skills (but can be done with minimal
computer literacy and skill)
May expose your current practice as obsolete or dangerous
(loss of authority and respect)
LIMITATIONS
Lack of evidence (shortage of studies)
Difficulty applying evidence to care of a particular patient
Barriers to the practice of high quality medicine
Lack of skills (search, appraise, etc.)
(foster development of new
skills!)
Lack of time to learn and practice EBM (promotes lifelong learning thru
better focus)
Lack of physician resources for instant access to evidence (EBM has
worldwide applicability)
RESTRICTED AVAILABILITY OF LAB TESTS
NON-TEXTBOOK CASE
co morbidity, additional risk factors
AFFORDABILITY (MD & PT)“I can’t afford to practice EBM.”
Language barriers – available evidence may be unreadable, should be
included
Physician attitude: Can be the greatest limitation!
“It decreases the importance of my clinical expertise”
(that’s a necessary component!)
“It only applies to those involved in research.”
(promotes cooperation among multiple physicians)
“It ignores patient values and preferences.”
“It’s just another cookbook approach to medicine.”
“It’s a poorly disguised way to cut medical costs.”
(cost of care may actually increase)
“It’s a way to ration care and resources.”
(Provides better utilization of avail resources)
DISAGREEMENT
Pt’s comfort, choice, acceptance, values preferences
Vs MD’s recommendations
DOES RISK OR SIDE EFFECTS OF TREATMENT OUTWEIGHT THE
BENEFITS?
The unanswered question…
“DOES EBM REALLY MAKE A DIFFERENCE?”
Effect of practicing EBM on patient outcome is actually
unknown – no studies done
EBM good based on population studies:
(ie: Pts who rec’d ___ generally fare better than those who don’t)
EBM IN DEVELOPING COUNTRIES
LIMITED RESOURCES
May help to eliminate unnecessary or poor quality
screening tests (ie: resting EKG to screen for
CAD = high false negative and false positive
rates)
LIMITED DRUG REGULATION
Approval for drug marketing easy - promotes
insurgence of new drugs for questionable
indications, limited effectiveness, false claims,
inflated prices based on ad response (include
“more expensive is better”)
EBM IN DEVELOPING COUNTRIES
LIMITED CAPACITY FOR CME
Drug companies - may sponsor meetings that are little
more than captive marketing sessions or biased
education sessions (drug education vs promo)
Result may be push for more expensive, less effective
treatments (ie push for CCB’s over BB’s) - calc channel
blockers over Beta Blockers
EBM IN DEVELOPING COUNTRIES
LIMITED ACCESS TO LITERATURE DATABASES
Desktop computer with CD ROM reader and modem
($900)
Electricity
1 yr subscription to MedLine on CD ROM (?500)
Internet connection $25/mt
Convince administrators of expense:
Publicly cite how searches help with lectures, research
and patient care management decisions
Get equipment from drug companies
(usually strings attached)
EBM IN DEVELOPING COUNTRIES
LIMITED ACCESS TO ADEQUATE LIBRARY FACIILITIES
ALMOST INEVITABLE IN DEVELOPING COUNTRIES
Identify resources via search, but then unable to retrieve articles!
A top EBM practitioner (Philippines) recommends:
1. Top 3 medical libraries in your country
2. Multinational drug company libraries
3. Friends and colleagues - including in other countries
EBM IN DEVELOPING COUNTRIES
QUESTIONABLE APPLICABILITY OF ARTICLES
RETRIEVED
Article describes a treatment that worked in one country, but
seems impossible in yours
Check…
•
•
•
•
•
Are there pathophysiologic differences?
Will patient differences diminish the treatment response?
Patient compliance issues?
Provider compliance issues?
Co-morbid conditions which will alter the benefits or risks?
EBM IN DEVELOPING COUNTRIES
OBSTACLES TO TEACHING OR LEARNING EBM
Your Hospital or Institution does not reimburse for time spent on
Continuing Medical Education programs
The standard 5-day workshop would be far too costly to provide or
attend!
Need to learn the basics - computer skills, etc.
TRY THESE!
Combine efforts to learn more and practice EBM with handful of
colleagues (small group learning)
Ask about basis for information provided by drug reps, medical supply
companies, etc. It will prompt them to provide you with on the spot
teaching and better information, too!
EBM LIBRARY
BASIC REQUIREMENTS
Convenient – easy access at point of contact with patient if
possible
Current – Up to date information
Electronic Database – Should be included
• Online
• CD-ROM
ELECTRONIC DATABASES
Evidence-Based Medicine Reviews (EBMR) – from Ovid
(ovid.com)
- combines Cochrane, Best evidence, Evidence Based
Mental health, EB Nursing, Cancerlit, healthstar, AIDSline,
Medline, and journal links (Described by one EBM specialist
as “the best”)
Cochrane Library – “Gold Standard” for systematic reviews
Best Evidence
Medline – world’s largest, free resource – over 10 million
references
PERSONNEL
Medical Librarian
Informatics Specialist
“We can learn a great deal about current best information
sources from librarians and other experts in medical
informatics, and should seek hands-on training from them
as an essential part of our clinical training.”
(ch 2 p29-30 – Blue circled 2)
PRINTED RESOURCES
TEXTBOOKS
most obsolete!
Some updated yearly, plus heavy references and scientific
evidence for support
Clinical Evidence (BMJ Publishing Group & ACP – 1999present)
Evidence-Based On Call (http//cebm.jr.ox.uk/eboc/eboc.html)
Up To Date (General medicine, CD format, Medline abstracts
used for evidence)
Scientific American Medicine – limited references from
Medline, Harrisons
JOURNALS
Traditional Journals
subject to author submissions
specialists need to read and evaluate
may subscribe to services that send articles of interest to
your specialty
timely, instant information at time of publication
Ex: NEJM, Clinical Nephrology, etc.
Evidence Based journals
selects best studies from multiple journals of interest,
summarizes best evidence
Good for use by generalists
Lag time from original publication: 3-6 months
Ex: Evidence Based Medicine, Evidence Based
Nursing, Evidence Based CV Medicine, etc.
SPECIAL RESOURCES
WHO Blue trunk
Hinari
PATIENT RESOURCES
Medical treatments www.nlm.nih.gov/medlineplus
Medical guidelines www.guideline.gov
THE NEXT LEVEL: ADVANCED EBM
SUMMARIZE AND STORE INFORMATION
Future reference
SHARING INFO
Local Colleagues
Author paper
TEACHING
New skills or treatments
EBM practices
OTHER APPLICATIONS
Care of the individual patient
Team protocols
Hospital or practice guidelines
EBM in Medical Education
Message to medical educators from Trisha Greenhalgh, MD,
co-author of Evidence Based Health Care Workbook:
“An important challenge for medical educators… is to
recognize that the competent student (and clinician!) is one
who knows how to cope with an immense and rapidly
changing body of knowledge and not one who excels in
recalling the traditional or memorizing the ephemeral.
The deans of medical and nursing schools must develop an
infrastructure that allows problem-based, self-directed
learning methods to develop within the didactic, lecturebased curricula, which have seen no fundamental changes
for two centuries or more.”
ADVANCED EBM: ADVANCED APPLICATIONS
APPLY METHODS TO…
Care of the individual patient
Team protocols
Hospital or practice guidelines
Continued Learning: problem-based approach
Teaching
SELF-DIRECTED LEARNING
JAMA Series of User Guides
“Clinical Epidemiology: A Basic Science for Clinical
Medicine”
Week-long workshops
On-the-job learning (in your own practice)
EVIDENCE BASED MEDICINE
A new approach to clinical care and research
Developed and presented by
Judy Tarselli, RN
Dubai, UAE
Karachi, Pakistan
October 2003
Organized by NKF cyberNephrology
University of Alberta, Canada
www.cyberNephrology.org
Special thanks to our sponsors at Janssen-Cilag of Dubai
EVIDENCE BASED MEDICINE
A new approach to clinical care and research
Developed and presented by
Judy Tarselli, RN
Dubai, UAE
Karachi, Pakistan
October 2003
Organized by NKF cyberNephrology
University of Alberta, Canada
www.cyberNephrology.org
Special thanks to our sponsors at Janssen-Cilag of Dubai