Chapter 16
Treatment Outcomes: Research
Methods and Data Analysis
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Chapter Overview
• Understand why assessment of the effectiveness of
therapeutic interventions on patient outcomes is the
centerpiece of clinical research.
• Be able to describe the tenets of evidence-based practice.
• Appreciate the value of evidence-based practice in making
clinical decisions.
• Understand that the “best” research evidence ideally consists
of patient-oriented evidence from well-conducted clinical
trials.
• Appreciate the issues involving study design, statistical
analysis, and interpretation of the results of clinical trials
that assess treatment outcomes.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Building an Infrastructure to Measure
Treatment Outcomes
• Clinical trials assessing treatment outcomes must be
planned ahead of time.
• An infrastructure needs to be created for the regular
collection of treatment outcomes (disease-oriented and
patient-oriented measures).
• The means by which outcomes data will be collected and
managed must be determined:
– paper-based surveys
– entering data electronically
– long term follow-up surveys administered by phone or
by mail
• The performance of clinical trials often involves multiple
clinical sites.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Study Designs
• The design of a clinical trial is the most important factor that
influences the “level of evidence” stemming from that study’s results.
• The two most common schema for classifying levels of evidence:
–
Centre for Evidence-Based Medicine (CEBM)
–
Strength of Recommendation Taxonomy (SORT)
The CEBM has five general categories of levels of evidence (5 is lowest
and 1 is highest).
• Level 5 Evidence: Expert Opinion and Disease-Oriented Evidence
(basic and translational research)
• Level 4 Evidence: Case Series
• Level 3 Evidence: Case Control Studies (retrospective in design)
• Level 2 Evidence: Prospective Cohort Studies (lack randomization)
• Level 1 Evidence: Randomized Controlled Trials (gold standard for
clinical trials methodology)
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
CEBM: Five General Categories of
Levels of Evidence
Level 4 Evidence: Case Series
• A case series represents the reporting of the clinical outcomes of a
number of patients with the same pathology who are treated with
the same or similar intervention.
• Does not utilize an experimental design, which limits the internal
validity.
Level 3 Evidence: Case Control Studies
• The clinical outcomes of two groups of patients are examined after
interventions are administered .
• Patients are treated with the respective treatments based on
clinician judgment and not because of experimental allocation.
Level 2 Evidence: Prospective Cohort Studies
• Involve baseline measurement before patients receive the
prescribed treatment.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
CEBM: Five General Categories of
Levels of Evidence (continued)
Level 1 Evidence: Randomized Controlled Trials
• Assigning interventions randomly limits the potential of confounding biases
between groups.
• Guidelines to assure quality in the design and reporting of RCTs:
– CONSORT statement
– PEDro scale
• Dropouts in RCTs: the data should be analyzed using intention to treat
analysis.
• Intention to treat analysis: subjects are analyzed in their original group
regardless of whether they received the full course of intervention.
• Missing data points in RCTs: imputation of data should be performed.
• Imputation: involves carrying the subjects’ most recent scores forward by
using those scores for all future time points as well.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
CONSORT Flow Chart
• Recommended in reporting RCTs to illustrate issues of subject retention
throughout the entire course of the trial.
(CONSORT Group, 2008. Accessed from http://www.consort-statement.org/consort-statement/flow-diagram/February 11,
2010.)
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Data Analysis: Statistical Significance
vs. Clinical Meaningfulness
• It is possible to have statistically significant results that are
not clinically meaningful.
• It is possible for results not to be statistically significant but
to have clinical meaningfulness.
• Statistical analysis assesses the statistical significance of
the involved comparisons, not the clinical meaningfulness
of the results.
• Hypothesis testing should be viewed as an integral part of
data analysis but not the only means of analysis.
• Hypothesis testing does not provide information about the
magnitude of mean differences between comparisons.
• Measures not easily understood at face value should
employ other analysis concepts: confidence intervals, effect
sizes, and minimally important clinical differences may be
employed.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Confidence Intervals
• Confidence intervals are estimates of dispersion, or
variability, around a point estimate.
–
In treatment outcomes data, the point estimate is typically
a mean, or average, value for a given sample.
• The confidence intervals between two comparisons can be
evaluated visually by assessing whether the confidence
intervals overlap.
• Confidence intervals are determined by M + w.
– M = point estimate
– w = width of the confidence interval
• W = Tc(SE)
Tc = an established critical value to be used as a multiplier
SE = standard error or SD/√n
• The width is influenced by two factors: the variance in the data
(indicated by the SD) and the sample size.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Effect Sizes
• Effect sizes provide an estimate of the strength of a
treatment effect and an indication of the meaningfulness of
results.
• Effect sizes are on a standardized, unitless scale.
• There are several ways to compute effect sizes. The most
straightforward method is Cohen’s d.
•
•
•
•
Effect sizes greater than 0.8 are considered “strong.”
Those between 0.5 and 0.8 are considered “moderate.”
Those between 0.2 and 0.5 are considered “small.”
Those less than 0.2 are considered “weak.”
• In general, effect sizes are reported as positive values
unless there is a specific reason to report them as negative
values.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Minimally Clinically Important Difference
• ROC curves applied to analyses of data related to treatment
outcomes and prognosis.
• Minimally clinically important difference (MCID) or
minimally important difference (MID): a value
established when ROC analysis identifies the change in a
health status measure associated with improvements
meaningful to patients.
• ROC analysis is used in diagnostic studies when the
measure in question is continuous rather than
dichotomous.
• Continuous measure: the number of factors present
following patient evaluation that are related to the outcome
of treatment.
• Multiple approaches to estimating an MCID: ROC analysis,
use of standard error measurement values and effect size
estimates. Use of a ROC analysis is not always preferable.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Clinical Prediction Rules
• Flynn et al. (2002) derived a clinical prediction rule
to guide the treatment of patients with low back pain
with spinal manipulation.
• Ultimately, five factors were selected that best
discriminate between patients who have a favorable
response to spinal manipulation, defined as a greater
than 50% improvement on the Modified Oswestry
Disability Questionnaire, and those who do not.
• The papers by Stucki et al. (1995) and Flynn et al.
(2002) are examples of the utility of ROCs across
varying research purposes.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Comprehensive Data Analysis
• Traditional hypothesis testing is the most common
way to analyze data statistically, but it has
limitations in assessing clinical meaningfulness.
• Alternative techniques such as:
– confidence intervals
– effect sizes
– minimal clinically important differences
should be used with hypothesis testing as part of a
comprehensive data analysis plan.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Chapter Summary and Key Points
• The importance of building a strong research infrastructure for clinical
outcomes research cannot be overemphasized.
• The design of a clinical trial is the greatest factor influencing the “level
of evidence” stemming from that study’s results.
• A case series represents the reporting of the clinical outcomes of a
number of patients with the same pathology who are treated with the
same or similar intervention.
• Case control studies are by definition retrospective in nature.
• The randomized controlled trial (RCT) is the “gold standard” for
clinical investigation.
• The individual responsible for coordinating the randomization should
not be directly involved in deciding the inclusion criteria of potential
subjects or the measurement of outcomes directly from patients.
• Hypothesis testing does not provide information about the magnitude
of mean differences between comparisons; this is the limitation of this
technique.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins