Drug and Therapeutics
Committee
Session 3. Assessing Medicine Efficacy
Drug Efficacy
2
Objectives
 Understand the importance of determining efficacy
and evaluating the clinical literature
 Discuss the major types of medicine study design
 Describe the key components of a journal article
 Understand how to evaluate and interpret results
of a randomized controlled trial
 Discuss the use of systematic reviews and metaanalyses in evaluating medicines
Outline
 Introduction
 Assessing medicine studies in the clinical
literature
 Systematic review and meta-analysis
 Activities
 Summary
Introduction
 DTC medicine selection responsibilities—
 Evaluate and recommend appropriate
medicines for the formulary based on efficacy,
safety, quality, and cost
 Screen ineffective or costly medicines that
have no increased benefit
Getting Started in Evaluating an
Article: Formulating the Question
 The first step in assessing whether a new medicine is
effective is to formulate the clinical question that is
important to your DTC.
 This question should be constructed to specify—
 The patient population (P)
 The medicine intervention that you are interested in (I)
 The comparative treatment already available (C)
 The outcome that is important to clinicians and patients (O)
 Example: In a diabetic patient over age 60 (P), does
metformin (I) compared to glibenclamide (C) reduce the risk
of stroke (O)?
Evidence: What Kind and How to Find It
 What kind of evidence:
 Secondary research or reviews—overviews,
systematic reviews, meta-analysis
 Studies from primary literature—medicine trials,
observational studies, surveys, experiments
 For the DTC, the most reliable evidence is the
systematic review that contains several
randomized clinical trials and a meta-analysis
 Finding it:
 Depends on your resources
 Many online databases (e.g., PubMed, Medline,
Cochrane)
Assessing the Quality of the Evidence:
What Makes a Good Clinical Trial?
 Abstract—summary
information about the
article
 Introduction—why the
authors decided to do the
research
 Methods—how they did it
and how they analyzed
their results
 Results—what they
found
 Discussion—what the
results mean, in the author's
opinion
 Conclusion— are these
justified by the results?
 References—list of
references used in the study
 Acknowledgments—study
funding sources, potential
conflict of interests, authors’
affiliations
Evaluation of a Clinical Medicine Study
Why was the
study done?
How well was the
study conducted?
Was bias
minimized?
What type of study
was done?
Clinical
Medicine
Study
Was the study
design appropriate
to the research
question?
Who is the study
about?
Was the design of the
study sensible?
Checklist for Evaluating Medicine
Studies (1)
Why was the study done; what is the clinical
question?
 Introductory statement should clearly state the
study question and the hypothesis that the authors
have decided to test
 When presented in a negative statement, it is
referred to as the “null hypothesis”
 Example – “Antibiotics do not improve the symptoms
and recovery time of upper respiratory tract
infections”
Checklist for Evaluating Medicine Studies (2)
What type of study was done?
 Many different types of studies
 Reviews, experiments, trials, and surveys
 Observational studies
 Cohort (prospective or retrospective; cross-sectional or
longitudinal)
 Case control (used primarily to evaluate an adverse event
thought to be medicine-related)
 Case series (single case report)
 Randomized controlled trials (experimental)
 Randomized controlled trial (RCT)
 Most reliable study design
 Should use this type of study - If not used, must determine why
they have elected not to use
Example of an RCT (1)
Target population
R
Intervention
Outcome
(example)
Group A
Group B
% reduction
% reduction in
in morbidity or Comparison morbidity or
mortality
mortality
Example of an RCT (2)
Meropenem vs. Imipenem/Cilastatin
Prophylaxis for 176
patients
R
Intervention Meropenem
Imipenem/
Cilastatin
% reduction in
Outcome % reduction
in infection Comparison
Infection
(example)
(11.4%)
(13.6%)
Manes et al. (2003); Pancreas
Example of an RCT (3)
Meropenem vs. Imipenem/Cilastatin (Continued)
Treatment for 182 patients
with lower RTI, UTI and
other infections
R
Meropenem
Imipenem/
Cilastatin
Bacterial
Eradication (86%)
Efficacy rates (90%)
Bacterial
Eradication (86%)
Efficacy rates (87%)
Comparison
Fang et al. (2002); Chinese Medical Journal
Checklist for Evaluating Medicine
Studies (3)
Was the study design appropriate to the
research question?
 Consider whether the study design used is the
preferred design for the research question,
examples:
 Testing efficacy—RCT
 New diagnostic test—RCT plus cohort study
 Screening—cross-sectional cohort study
 Prognosis (outcome of a disease over time)—
longitudinal cohort study
 Causation—toxin, ADRs, adverse drug events—
case control or cohort study
Checklist for Evaluating Medicine
Studies (4)
Was the design of the study sensible?
 What specific intervention was being considered and
what was it compared with?
 Example—is it reasonable to compare a new product
for hypertension with a half dose of an ACE inhibitor
or medicines no longer used because of side-effects
(e.g., reserpine)?
 What outcome was measured in the study and how?
 Is it an outcome that is clinically important?
Checklist for Evaluating Medicine
Studies (5)
Who is studied?
 How were the subjects for the study recruited?
 Who was included in the study?
 Representative of the population in which the medicine will
be used?
 Who was excluded from the study?
 Is it likely to lead to false conclusions about the effect of an
intervention
 What was the setting of the study and does it relate to the local
environment?
Checklist for Evaluating Medicine
Studies (6)
How well was the study conducted? Was
systematic bias avoided or minimized?
 Very important part of a critical appraisal
 Important elements to determine how well a study
was conducted—some important definitions:
 Bias–anything that leads to deviation of the results from the truth
or processes leading to such deviation
 Randomization—process of assigning patients to treatment
groups (new medicine, comparator medicine, placebo) by chance
 Observer blind—the person measuring the outcomes in a study is
not told what treatment patients have received
Checklist for Evaluating Medicine
Studies (7)
 Important definitions (continued)
 Double-blind—neither the observer or the patient in a trial
knows what treatment the patients received
 Allocation—the process of assigning patients to treatment
groups
 Intention-to-treat population—the total number of patients
assigned to receive a particular treatment, irrespective of
whether they actually received it or not
 Confounding factor or variable—a variable that can cause or
prevent the outcome of interest, is not intermediate variable,
and is associated with the factor under investigation
Example: Summary of Key Sources of Bias in RCTs
(Source: Greenhalgh, T. 1997. British Medical Journal 315:305–08)
Quality of RCTs: What to Look for (1)
 Features that are most important in determining
reliability of a RCT
 Methods used to randomize patients
 Blinding (double blind)
 Are all patients followed up and included in the
analysis?
Quality of RCTs: What to Look for (2)
 Randomization and concealment of allocation
 Assigning patients by chance removes the likelihood
that the investigator will select patients, either
consciously or unconsciously, for the experimental
treatment who are more or less likely to respond to it.
 If the paper you are reviewing does not tell you how
patients were randomized and how the allocation
process was concealed, it may have unreliable
results.
Quality of RCTs: What to Look for (3)
 Double-blind versus open trials
 Three people who can influence a trial
 Patients
 Treating physician
 Person measuring the outcome
 Ideally, patient, physician, and observer all must be
unaware of the treatment group.
 In summary, double-blind, placebo, controlled trials
with objectives and outcomes judged by an
independent outcome committee are the gold
standard.
Quality of RCTs: What to Look for (4)
 Inclusion of all patients in the statistical analysis
 A trial is less likely to have bias if all patients recruited
and allocated to treatment are accounted for.
 Trials of new interventions that do not report what
happened to all patients and do not report an
intention-to-treat analysis (includes all recruited
patients even if they did not receive treatment) should
be treated with more uncertainly than those that do.
Non-Randomized Trials: What to Look For
 Non-randomized trials—main difference with RCT
is that risk of selection bias is higher.
 Judgment needs to be made concerning the
significance of selection bias in the study.
 If selection bias is significant, then the study must
be suspect.
 No amount of statistical adjustment can overcome
the fundamental bias that is introduced by the
absence of randomization.
Understanding the Numbers (1)
 What did the authors think they would find?
 A trial should be big enough and long enough to have a
high chance of detecting an effect of treatment.
 What did the authors decide was an important difference
before they did the trial and how many patients did they
calculate they would need ?
 How are the results described ?
 Simplest description of trial results is to use proportions,
that is, the number of patients in treatment group who
have the outcomes compared with the total number in
the group.
Understanding the Numbers (2)
Key Concepts for the DTC to Consider
 Different types of data need different statistical tests.
 Comparing the effect of treatment in one group relative
to the effect in the other is necessary.
 Comparing the absolute value of the results in one
group with those in the other is critical.
 The difference in the effects of treatment (if any) can
be described as the estimate of effect size:
 Confidence interval expresses the range of plausible
results.
 P-value expresses the probability that the difference is
real and not due to chance.
Understanding the Numbers (3)
P-Values and Confidence Intervals (CI)
 P-value <0.05 indicates that there is—
 5% chance (1 in 20 probability) of observing a result that
does not truly exist
 95% chance that any observed result is a true result
(e.g., difference in outcome with different medicines) that
exists in the population
 A 95% CI shows the range of results that 95 times
out of 100 will include the mean (i.e., the study
result)
 5% chance that the true result will fall outside the range
 The larger the sample size, the smaller the CI, and the
more confident we are that these results are reliable
Understanding the Numbers (4)
Comparing
treatment
groups with
placebo
or other
medicines
Event rate
Relative
risk
(RR)
Absolute
risk
difference
(ARD)
Absolute
risk
reduction
(ARR)
Number
needed
to treat
(NNT)
Relative
risk
reduction
(RRR)
Odds ratio
(OR)
Understanding the Numbers (5)
 Event rate
(over specified time
period)
No. of patients with event
Total no. of patients
 Relative risk (RR)
(same as rate ratio or
risk ratio)
Event rate in treatment group
Event rate in control group*
 Absolute risk
difference
Event rate in treatment group –
event rate in control group*
* or comparator medicine group
Understanding the Numbers (6)
 Number needed to treat
(No. patients that need to be
treated during the study
period before an effect may
be realized)
 Relative risk reduction
(RRR)
(this is not a helpful statistic)
 Odds ratio (OR)
________1___________
Absolute risk difference
1 – Relative risk
No. patients who have the event
No. who do not have the event
Systematic Reviews (1)
 An overview of individual studies that contains an explicit
statement of objectives, materials, and methods and has
been conducted according to explicit and reproducible
methodology
 Use of explicit methods limits bias in identifying and rejecting
studies
 Not same as narrative reviews done to prove a point
 Conclusions are generally more reliable and accurate because
of the methods that are used
 Large amounts of information can be assimilated quickly by
DTC
 Cochrane Collaboration undertakes many systematic reviews;
abstracts are free of charge at www.cohrane.org
Systematic Reviews (2)
 Results of different studies can be formally compared
to establish the generalizability of findings and
consistency of results.
 Reasons for heterogeneity (inconsistency in results
across studies) can be identified and new hypotheses
generated about particular subgroups.
 Where appropriate, results of individual studies can
be statistically combined using meta-analysis to
provide a single summary estimate of the effect of an
intervention.
Systematic Reviews (3)
 To fully understand and interpret systematic
review, one must consider—
 How the trials included in the systematic review
were found, and the potential for publication bias:
search strategies and inclusion criteria
 The use of meta-analysis
 The use of sensitivity analysis in interpreting the
results
 Analysis re-run with different values for key variables and
the impact of these changes on the results is observed
 Tests the robustness of assumptions made in analysis
 Interpreting inconsistent results (heterogeneity)
Meta-analysis
 Refers to the statistical techniques used to
combine the results of a clinical trial into a
single estimate of effect
 Can be thought of as a weighted average
effect
 Used to calculate pooled or summary
estimates for all metrics (RR, ARD, NNT, OR)
 Presented graphically as a forest plot
Meta-analysis—Forest Plot
Source: Dale, K.M. et al. 2006. Journal of the American Medical Association 295:74–80
Potential Clinical Study Problems:
Objectives
 Comparing medicines
 Medicine is tested against placebo, not the standard
medicine in its class.
 Medicine is tested against a medicine with poor
performance.
 Insufficient information about the disease
outcomes and effects of the medicine study are
given
 Clinically unimportant outcomes may be used.
Potential Clinical Study Problems:
Methods (1)
 Study sample problems
 Study patients are not representative of the
population that will actually take the medicine.
 Number of participants is too small.
 Randomization
 Patients are not randomized correctly to the
treatment, control or comparator groups.
 Dropouts
 Patients with more side-effects or less effect are
more likely to drop out and not complete the study.
Potential Clinical Study Problems:
Methods (2)
 Confounding factors have not been controlled
rigorously and results of the study may be from the
confounding factor
 Bias introduced by the researcher
 May have an extremely important effect but be
difficult to assess
 Blinding
 Blinding is not done effectively
 Statistical significance of a trial is valid, but clinical
significance is weak
Potential Clinical Study Problems:
Methods (3)
 Dosing regimens
 Efficacy and safety are based on one dosage regimen
and do not provide opportunity to learn more from
various doses.
 Medicine studies use fixed doses to compare different
medicines.
 Outcomes (end points)
 Outcomes in the analysis are not optimal or were
selected after study completion.
 Not all outcome measures are reported.
 Outcome measures may be discordant (some show
improvement and others do not).
Potential Clinical Study Problems:
Results and Conclusions
 Study is not subject to a peer-review process
 Throw away journal
 Symposia proceedings
 Study funding is provided by a pharmaceutical
company
 Misleading data presentation and analysis
 Published data can be represented in different ways
and taken out of context.
 Conclusions do not agree with the results
Systematic Review Problems
 Meta-analysis does not answer the clinical
question.
 The search process and inclusion criteria of
articles is not defined and may be incomplete.
 Negative studies not included so giving overly
positive results
 Appraisal of individual studies not described
 In meta-analysis, how were data combined
statistically and was a sensitivity analysis done?
 How was inconsistent results (heterogeneity)
interpreted?
Activities
 Activity 1–Comparing Antimicrobial Medicines for
Pneumonia
 Activity 2–Interpreting the Data: The Helsinki
Heart Study
 Activity 3–Critically Evaluating an Article
 Activity 4–Interpreting the Data: A Medicine Trial
to Compare Artesunate with Mefloquine to Treat
Malaria
Summary
 Critical evaluation of the clinical literature is
important but difficult and time consuming.
 Best source is systematic review but may have to
review primary studies.
 Methodology needs careful evaluation because
many studies are subject to bias.
 Use of clinical literature evaluation techniques will
improve the DTC’s ability to assess, evaluate, and
select the best medicines for the formulary.