N7000:
INTRODUCTION TO
EBP
Week Seven: Experimental Studies
Presented by: Corina Lelutiu-Weinberger, PhD
Objectives
1. Define experimental (in particular randomized control trials) research studies
2. Compare/contrast to other types of research studies
3. Identify strengths/limitations of experimental studies
4. Discuss rapid appraisal of this type of research
Meets 7000 Course Objectives:
4. Appraise the scientific merit and clinical relevance of research reports that use the following research designs:
• Systematic reviews and meta-analysis
• Experimental studies including randomized controlled trials
• Observational (cohort and case control) studies
• Cross sectional studies
• Qualitative studies
5. Interpret research findings presented in tables, figures, and graphical displays.
7. Evaluate and engage in meaningful conversation regarding the appropriateness of scientific evidence for use in their
clinical practice area.
Circling Back Around…
• You are working as an RN on in an oncology
ambulatory surgery center. You begin to notice there
are high preoperative anxiety levels in women
undergoing breast surgery. Your institution recently
hired a music therapist….and you wonder if music
therapy may impact your patients positively.
What could be considered a starting PICO question?
PICO Ideas
Which question is the most precise?
1. How does music impact anxiety?
2. What are effective methods for reducing anxiety in women
with breast cancer?
3. Does music decrease levels of anxiety?
4. What is the impact of music therapy on women with breast
cancer?
5. In women with breast cancer undergoing breast surgery,
what is the impact of music therapy compared to no
therapy on anxiety levels?
And the answer is….
• In women with breast cancer undergoing
breast surgery, what is the impact of music
therapy compared to no therapy on anxiety
levels?
PICO
And the search begins
• Precise search terms
• Limits (years;
geographical area)
• # and relevance of
results
Start with Most Relevant One
But which
elements in the
article will be
most
informative?
EXPERIMENTAL
STUDIES: THE BASICS
What is an experimental study?
• Experimental studies aim to
determine efficacy of an
intervention (whether
intended impact of
intervention on outcome is
statistically significant
compared to control).
• Can include randomization
or not (RCT stronger than
pre-post comparisons or
open trials)
What is a randomized control trial study?
• RCTs are ranked high on the level of
evidence pyramid
• Includes intervention/treatment
group vs. control group (or more)
• Considered to be strongest
methodology type used to test
efficacy of treatment
• Adaptive designs are gaining
ground
• Aims to establish cause and effect
• Follows patients FORWARD in time
Experimental Studies
Experimental
Study
– Introduce an intervention or
treatment
Randomized
– Assess its impact on one or
more outcomes
Yes
No
Randomized
Controlled
Trial (RCT)
QuasiExperimental
Screened (N = 14043)
Screening
Eligible, not participating (N = 847)
Withdrew (n = 37)
Eligible (N = 1893)
Dropped (n = 84)
Completed baseline assessment (N = 536)
Baseline
Excluded (n = 231)
•Ineligible after rescreen (n = 17)
•No sex risk (n = 136)
•No HD risk (n = 41)
•HIV positive (n = 13)
•Incomplete testing (n = 16)
•Inconsistent data (n = 8)
Excluded (n = 2)
Randomized to Comunică: n = 153
Ineligible (n = 12150)
•Under age 16 (n = 113)
•Not residing in Romania in the next 12
months (n = 830)
•Not male gender (n = 1066)
•Doesn’t have sex with men (n = 1869)
•No CAS (n = 9452)
•No HDD (n = 6911)
•PrEP adherent (n = 413)
•HIV positive (n = 898)
•No mobile device (n = 155)
•Hospitalized for mental illness
(n = 184)*
•Symptoms of psychosis (n = 946)*
•Symptoms of mania (n = 787)*
•Took antipsychotic or bipolar disorder
medication (n = 542)*
* As of 9/16/20 mental health screening
questions are no longer part of
eligibility criteria.
Randomized to EAC: n = 152
Withdrew (n = 6)
Withdrew (n = 2)
4-month follow-up
Due 4M FU: n = 134
Retained: n = 120 (90%)
Due 4M FU: n = 138
Retained: n = 124 (90%)
8-month follow-up
Due 8M FU: n = 122
Retained: n = 106 (87%)
Due 8M FU: n = 120
Retained: n = 101 (84%)
12-month follow-up
Due 12M FU: n = 112
Retained: n = 104 (93%)
Due 12M FU: n = 108
Retained: n = 95 (88%)
Study completion
(assessment + testing)
n = 98
n = 88
Figure 1. Consort diagram.
M = months; FU = follow-up
CAS = condomless anal sex
HDD = heavy drinking days
Where Does It Fall?
Where does it fall?
Examples
1. Vitamin B12 supplementation in treating major
depressive disorder: a randomized controlled trial
2. Efficacy of a Web-Based Intervention to Increase
Uptake of Maternal Vaccines: An RCT
3. Effectiveness of Cranberry Capsules to Prevent
Urinary Tract Infections in Vulnerable Older Persons: A
Double-Blind Randomized Placebo-Controlled Trial in
Long-Term Care Facilities
CONSIDERATIONS FOR IMPLEMENTING AN
INTERVENTION
Extreme calorie intake
Smoking
Things we know are
Cyanide
harmful…not ethical
Race
Where one was raised
Exposures in utero
Things we can’t change
about a person…not
feasible
Immunotherapy drug for cancer
Outpatient drug treatment program
Procedures for central line maintenance
Things we can expose people
to…but carefully
Guiding research principles e.g.
informed consent, favorable
risk-benefit ratio (clinical
equipoise)
Important Terms
A Note About Bias
To review:
• Bias is anything that distorts study results and
comes from study methodology and results
reporting.
• Bias can occur at any point of the study
Intervention Research Components
(Minimize Bias)
1. Selection
2. Randomization
3. Measurement
4. Contamination
5. Reporting
6. Attrition
Intervention Research Components
1. Selection
a) Representativeness of sample
b) Methods: probability, respondent driven
sampling (RDS), convenience (includes
snowballing), cluster
c) Factors that affect sampling:
• Location, time of day, method (flyer, social
media, direct recruitment, clinician
referral)
Intervention Research Components
2. Randomization
•Allocation Concealment
• Can investigators or participants predict in advance
who will be assigned to each group?
•Blinding
• Are participants and investigators adequately
blinded? If not, is this likely to introduce bias?
Randomization (continued)
•Randomization in study conditions:
• Group allocation by chance (at random)
• Groups will be comparable participant-wise
•When judging an intervention:
• Is group allocation random? Does every participant
have an equal and independent probability of being
assigned to each group?
• Stratification (balancing groups by characteristics):
• Groups should ideally have equivalent age,
gender, risk, etc, distributions
Randomization (continued)
Allocation vs. Blinding Concealment
Allocation concealment occurs BEFORE the
participant is enrolled in the study.
Blinding occurs AFTER the participant is
enrolled in the study
Intervention Research Components
3. Measurement
a)
Conceptual (dictionary definition)
Pain is defined as “physical suffering or discomfort caused by
illness or injury”
b) Operational (how is it actually measured)
Pain is measured using FACES scale
Operationalization
1. Which of these is clearly operationalized and why?
a) The study measured mental health
b) The study measured depression using the Beck’s
Depression Inventory
Validity
1. Internal- the degree in which
changes to the dependent variable
occur as a result of the independent
variable
2. External- the extent to which results
may be generalized to the population
at large
Take home points:
- There is no perfect balance between the two
- Interventions may be adapted
Intervention Research Components (to
minimize!)
4. Contamination
Participants assigned to the non-intervention (control) arm
are exposed/receive elements of the intervention.
What does contamination affect?
- validity
Intervention Research Components (to
minimize!)
4. Contamination (continued)
A researcher intends to minimize cross-groups
contamination in their intervention. Which of the strategies
below would NOT prevent cross-group contamination?
a) Not allow participants in one group to be exposed to
materials from the other group
b) Have different staff deliver the intervention and the
control group materials
c) Ensure that the two groups are exposed to their
conditions for an equal amount of time
Intervention Research Components
5. Reporting
Selective Outcome Reporting
• Do the investigators report all of their results?
Only “desirable” results are reported
- Null findings are typically under-reported or
journals are less likely to accept manuscripts
What are some implications of under-reporting?
- treatments with low impact may be used
Intervention Research Components (to
minimize!)
6. Attrition
The portion of the sample that dropped out of the study.
- 20% attrition is reasonable
- Power calculations
- Important for analyses (did people at higher risk drop
out?)
- Conduct analyses to see if there were any differences
between retained and lost participants
- Be transparent when reporting limitations and
interpreting results
Variables: Beyond Independent and Dependent
Confounding variables- a variable related to
both your independent variables and affects your
dependent variable(s)
• It must be correlated with the independent
variable.
• May or may not be a causal relationship
• It must be causally related to the dependent
variable.
• How can you manage them?
Examples of Odds Ratios
Associations of Sexual Minority Stigma and Alcohol/
Drug Use at Last Sexual Encounter
“For the three substance-related outcomes, a one standard deviation increase in stigma was associated with a
19% increase in the odds of sex under the influence of alcohol at the last sexual encounter, a 27% increase in
the odds of sex under the influence of cannabis at the last sexual encounter, and a 49% increase in the odds of
sex under the influence of illicit drugs at the last sexual encounter.
Table 2. Regression models for sexual behavior under the influence and HIV testing awareness predicted by sexual orientation stigma (n = 2,087).
Age
Missing covariates
(ref. = no)
Missing stigma (ref.
= no)
University education
(ref. = basic)
Employed (ref. =
unemployed)
Bisexual orientation
(ref. = gay)
City dweller (ref. =
small city/town)
Had a steady partner
(ref. = no)
Sexual orientation
stigma (z-score)
Last sex: Under the influence
of alcohol
B
AOR
95% CI
-0.02 0.98**
[0.97, 0.99]
Last sex: Under the influence
of cannabis
B
AOR
95% CI
-0.01 0.99
[0.97, 1.01]
Last sex: Under the influence
of club drugs
B
AOR
95% CI
-0.01 0.99
[0.97, 1.01]
Knowing where to get an HIV
test
B
AOR
95% CI
0.04 1.04**
[1.01, 1.06]
0.21
1.24
[-0.07, 0.49]
0.52 1.68*
[1.05, 2.69]
0.16 1.18
[0.71, 1.95]
-0.52
0.59*
[0.39, 0.90]
1.06
2.88***
[0.47, 1.65]
0.46 1.58
[0.60, 4.17]
0.66 1.93
[0.75, 4.94]
-1.02
0.36**
[0.19, 0.68]
-0.13
0.88
[0.72, 1.07]
-0.43 0.65*
[0.45, 0.96]
0.04 1.04
[0.71, 1.53]
0.29
1.33
[0.93, 1.92]
-0.29
0.75*
[0.59, 0.96]
-0.07 0.93
[0.59, 1.47]
-0.32 0.73
[0.45, 1.19]
0.03
1.03
[0.67, 1.58]
0.44
1.55***
[1.22, 1.98]
0.55 1.74**
[1.14, 2.64]
0.83 2.30***
[1.52, 3.47]
-0.25
0.78
[0.52, 1.17]
0.24
1.27*
[1.03, 1.55]
0.31 1.37
[0.91, 2.05]
0.23 1.25
[0.82, 1.92]
0.40
1.50*
[1.06, 2.12]
-0.26
0.77*
[0.63, 0.95]
-0.34 0.71
[0.49, 1.04]
-0.28 0.76
[0.52, 1.11]
0.23
1.26
[0.88, 1.79]
0.18
1.19***
[1.07, 1.33]
0.24 1.27*
[1.03, 1.56]
0.40 1.49***
[1.22, 1.82]
0.23
1.26*
[1.01, 1.59]
Notes. N = 2087. * p < 0.05; ** p < 0.01; *** p < 0.001.
All models were adjusted for demographic characteristics significantly associated with sexual orientation stigma.
B = unadjusted coefficient; AOR = Adjusted coefficient
32
Variables: Beyond Independent and Dependent
Extraneous variables- not a focus of your
investigation but can affect results.
Aspects you are not manipulating or measuring
but that can affect your findings:
• Neighborhood dilapidation (can affect
asthma data)
• Diet (can affect fertility treatment outcomes)
• Can you give us examples?
Variables: Beyond Independent and Dependent
•
How can you manage extraneous variables?
- Include in your models (“adjust” or
“control” for them)
- Publicly available data (census, etc)
- If cannot be measured
•
•
results interpretation and/or
limitations section mention (+ future
directions)
Extraneous vs. Confounding Variables
https://www.scribbr.com/methodology/extraneous-variables/
Strengths of Experimental Studies
• Randomization
• helps to minimize selection bias
• Blinding of study information from participants and
research team
• can reduce demand characteristics/social desirability
and experimenter effects biases
Strengths of Experimental Studies (continued)
• Dedicated groups: treatment and control group (or
more) - only type of study to determine causality
• Have specific inclusion and exclusion sample criteria
which leads to…
• Higher internal validity
Weaknesses of Experimental Studies
• Placebo effect
• Ethics
• Protocol rigidity
• implementation science
• Threats of extraneous factors
EXPERIMENTAL
STUDIES: CRITICAL
APPRAISAL
Critical Appraisal
To Review:
Critical appraisal is the “systematic unbiased,
careful examination of all aspects of studies to
judge their strengths, limitations, trustworthiness,
meaning and applicability to practice”
(Burns & Grove, 2021)
Critical Appraisal
Critical appraisal is “one of the most valuable
skills that clinicians can possess in today’s
healthcare environment. Distinguishing the best
evidence from unreliable evidence and unbiased
evidence from biased evidence lies at the root of
the impact the clinicals’ actions will have in
producing their intended outcomes”
(Melnyk & Fineout-Overholt, 2019, p. 109)
Critical Appraisal: Deeper Dive
1. Title and Abstract
- Does the title of the article indicate the type of study
that was conducted?
- Is the abstract comprehensive (i.e. includes study aims,
design, sample, intervention, results and implications)?
Critical Appraisal
2. Introduction: Problem Statement and Purpose
Statement
- Are the significance and background of the problem
described?
- Is the purpose statement clearly written?
- Do the authors describe how their findings will
contribute to science, practice and/or policy?
Critical Appraisal
3. Review of Literature
- Were previous studies described?
- How old were studies in the review of literature?
- Did researchers critically describe studies in their review of
literature (e.g., pointed to poor or strong sampling
procedures, limitations of methodologies)?
- Did the researchers point to the gap in the knowledge they
aim to address (e.g., whether assertiveness in healthcare
utilization may be increased with peer navigation)?
- Were there research questions and/or hypotheses?
Critical Appraisal
4. Research design
- What is the specific study design (RCT, cluster randomized
controlled trial, open pilot)?
- Was the treatment or intervention described in detail?
- If there was a comparison made, what was/were the
comparison group(s) and are they sufficiently described?
- If more than one study group was used, how were
participants assigned to their groups?
- Were extraneous variables identified?
Critical Appraisal
5. Population, sample, setting
- Are inclusion/exclusion criteria clearly defined?
- What type of sampling method was used?
- What is the sample size?
- What was the attrition rate?
- Was IRB approval obtained, was informed consent
conducted?
- What is the study setting?
Critical Appraisal
6. Variables
- Were variables identified and defined both conceptually
(assertiveness) and operationally (Rathus Assertiveness
Scale)?
- Independent variable clearly defined (impact of peer
navigation intervention vs. reading infographics on
assertiveness)?
- Dependent variable clearly defined (blood pressure)?
- Are demographic variables clear (was the sample on the
older side)?
Critical Appraisal
7. Measurement
- What were the measurement instruments? (Validated
scales or researcher-developed? Was permission for
use needed and gained?)
- What type of measurement instrument was used (Likert
scale? Physiologic measurement? EMR data?)
- Was the reliability of instrument used provided?
- Was validity of the scale or instrument used provided?
Critical Appraisal
8. Data management and statistical analyses
- How were the data collected, scales calculated?
- What statistical tests were used?
- Were analyses done to test all proposed hypotheses?
- Was the level of significance provided?
9. Results
- Were the results clearly reported and based on hypotheses?
- Was anything left out?
Critical Appraisal
10. Discussion
- How were results interpreted by researchers?
- Were findings expected?
- How were surprising findings interpreted?
- Did the researchers interpret their findings in
relation/comparison to existing literature (confirm, extend,
disprove)
9. Limitations
- Were study limitations identified (any, enough)?
Critical Appraisal
Critical Appraisal