Conducting flawless research – the right
question, the right design
Dr. Dick Menzies
July 10th, 2015
The starting point in Research – a Problem
• An unusual clinical observation
– An unusual cluster (Outbreak of XDR in S Africa)
– An unusual complication (Acquired Rifamycin resistance)
• An observation from reported data
– Temporal trend (Increase in TB in Foreign-born)
– Geographic clustering (TB in hospitals)
• Problems in diagnosis
– Delayed diagnosis (cultures, DST)
– Missed diagnosis (AFB smears in HIV infected)
• Problems in treatment:
– Treatment too long (Standard “short-course” therapy)
– Treatment too toxic (INH)
– Treatment too long AND too toxic (MDR, 9INH)
An unusual clinical observation – example
Extensively drug-resistant tuberculosis as a cause of death
in patients with TB and HIV in rural South Africa
Neel R Gandhi, Anthony Moll, A Willem Sturm, Robert Pawinski,
Thiloshini Govender, Umesh Lalloo, Kimberly Zeller, Jason Andrews,
Gerald Friedland
• Among 475 patients with culture-confirmed TB:
– 39% (185 patients) MDR-TB
– 6% (30 patients) XDR-TB.
• Of XDR-TB:
–
–
–
–
45% had been previously treated for TB ;
67% had a recent hospital admission.
100% of tested for HIV were co-infected.
98% died, with median survival of 16 days
From clinical problem (XDR outbreak)
to Research question
•
•
•
•
•
•
Questions arising:
How (and why) did the patients acquire XDR?
Why the association with HIV co-infection?
Why the association with hospitalization?
Why was mortality so high?
Why was mortality so rapid?
• Each of these questions needs a separate study, and
unique design - to be answered.
• WHICH question should be answered?
• IS this the right problem?
Choosing the right study question
Step 1: Definition of the problem
– Who gets the problem,
– How important is it - does it affect death?
Costs? Quality of life?
Step 2: Understanding the biology?
•
•
•
Manifestations - what organs are affected
Pathogenesis - probable or know
Transmission/hosts/reservoirs
Step 3: make the question as precise as
possible
– (PICO or pseudo-PICO format)
Choosing the right study question
“That year in the lab saved me a day in the library”
Step 4. Review the literature (again) (and again)
– This is often under-utilized - can avoid wasted time
– Avoid repeating errors others have made
– Do not study a problem that has been well described.
• RCT of FDCs (have been 15),
• Comparing positive TST vs IGRA (have been hundreds)
– Know what has been described
• Effect of standard treatment
• Accuracy of standard diagnostics
Step 5. Get the opinions of others
– Speak to experts (or pseudo-experts)
– Speak to providers
– Speak to patients, community members, others affected
From Research qst to study design
The biggest determinants of study design:
1) Time – investigators’ and patients’
2) Money (“The biggest determinant of sample size is the
budget”)
3) Resources – what do you have to work with? (Diagnostic
Laboratory, Molecular Epi, Molecular Biology, Genetics)
4) Collaborators – who do you have to work with? What
can they do?
These 4 factors determine the kinds of studies you can do
– which determines the question you can answer.
Hierarchy of study designs
• Descriptive studies
• Analytic studies
– Observational
•
•
•
•
Cross-sectional
Ecologic studies
Case control studies
Cohort studies
– Experimental studies
• Uncontrolled trials
• Randomized controlled trials
– Individual level
– Field trials = group level
Increasing costs and complexity
• Case reports, and case series
• Reported data,
• Prevalence surveys
Reported Data
• Useful to:
–
–
–
–
Define incidence/prevalence
Identify geographic or temporal differences
Describe clinical characteristics
Describe outcomes.
• Implicit comparison with general population
– Risk factors can be identified.
– Useful if data is COMPLETE and ACCURATE
19
53
19
56
19
5
19 9
62
19
65
19
6
19 8
71
19
74
19
7
19 7
80
19
83
19
8
19 6
89
19
92
19
95
19
98
20
01
TB incidence (per 100,000 ha.)
Temporal trends may indicate clues to causal exposure
Coal use and TB in USA: 1953 – 2003
60
100
50
80
40
60
30
40
20
20
10
0
0
Years
Coal use (
) and TB incidence (
).
Total residential coal consumption
(Million short tons)
120
100
95
90
85
80
75
70
65
60
55
50
45
40
35
30
25
20
15
10
5
0
35
30
25
20
15
10
5
0
1978
1980
1982
1984
1986
1988
1990
1992
1994
1996
Years
Total coal combustion (
) Notified cases of TB (
).
1998
2000
2002
2004
Total coal consumption
(exajoules)
TB notification rate
(per 100,000 ha.)
Temporal trends may indicate clues to causal exposure
Coal use and TB in China: 1978 - 2004
Ecologic Studies
Advantages
• Usually very easy and quick studies
• Take advantage of already gathered data
– Exposures
– Diseases
Disadvantages
• Relationship may be due to completely unmeasured
factors
• VERY substantial potential for confounding
Directionality in research
• Retrospective: Start with persons with disease and ‘look
backward’ in time to ascertain exposures.
– Advantages: Biggest is convenience – do not have to
wait a long time, because disease HAS happened.
Makes these studies much quicker to complete, and
much cheaper.
• Prospective: Start with a population and observe them
‘going forward’ in time. Exposures are measured first,
and health events are measured afterward, as they
occur.
– Advantages: Biggest is accuracy of exposure, and
certainty that exposure precedes disease.
Cross-sectional or Prevalence Studies
General approach:
Measure disease(s) and exposure(s) all at once in a population.
Estimate effect: Prevalence odds ratio
Advantages
• Good for common/chronic diseases
• Good for common exposures
• Allows one to measure multiple disease or conditions and
multiple exposures
Disadvantages
• Measurement of exposure may be difficult
– Recall problems if long latency
– May change over time (Alcohol, smoking, blood pressure)
• Can not distinguish cause and effect
– (Tobacco Industry defense)
Tuberculin (or IGRA) Surveys
A special type of cross-sectional survey
• Once TST or IGRA convert to positive with TB infection –
they remain positive lifelong (some exceptions)
• Cross-sectional survey – detects all with positive tests –
from recent or remote
• From prevalence of positive test at a given age – can
calculate average annual risk of TB infection.
• Can compare prevalence in different populations
• If different ages, or different exposure periods can
estimate trends in infection
University students in Brazil
(All BCG vaccinated in infancy)
% of Reaction
Silva et.al. IJTLD2000; 4:420-426
18
16
14
12
10
8
6
4
2
0
Engineering Students
Medical Students
Early 0-2
Intermediate
3-4
Senior 5-6
Years of training
Average ARI: Preclinical
Clinical
Incidence of TB in Brazil:
0.2%
2.9%
75/100 000
Example of Kaplan-Meier analysis: General
Hospital Ventilation and time to TST conversion
Case Control Studies
General Approach:
• Measure exposures retrospectively in persons with disease (cases)
and without disease (controls)
• Estimate odds of exposure for disease (Odds ratio)
Advantages
• Relatively cheap and quick
• Particularly useful for studying rare conditions
– Or conditions with long latency
Disadvantages
• Controls, Controls, Controls
– Very difficult to select proper controls
– This is the source of most problems in case control studies
– And is why they are generally considered weak evidence.
• Difficulties of retrospective exposure assessment
– particularly if long latency
Fitness costs of drug-resistance mutations in
MDRTB: a household-based case-control study.
Salvatore…. and Cohen T, JID 2015
• Cases: MDR index patients with at least 1 other HH
member with same strain
• Controls: MDR but no-one in HH with same strain
• Matching:
– Same number of contacts in House-Hold
– Same extent of Drug Resistance
– Same time to detection
• Not matched: Extent of disease, Cavitation, smear,
duration of symptoms, prior episodes, Size of house
• Findings: katG Ser315Thr mutation associated (OR:
2.39; 95% CI: 1.21- 4.70). BUT combination of katG
Ser315Thr & rpsL-Lys43Arg mutations was protective.
Nested Case Control study – example
TB outbreak in Inuit village
Behr, Fox, Khan, Lee, Menzies & many others
• Major ‘outbreak’ of TB in small village in N Quebec
• 695/940 residents investigated as contacts during outbreak
– Newly infected
– 50 Confirmed disease, 19 probable disease
• Case control study of housing, nutrition, lifestyle
determinants
– “Nested” within all 695 investigated as contacts
• Case-control1: New Infection
– Cases=Newly infected, Controls = Not infected
• Case-control2: Disease
– Cases=Confirmed or Probable, Controls = Infected but
no disease
Cohort Studies
General approach:
• Start with a population free of disease. Measure exposures, and
follow them all.
• Estimate of effect: Risk ratio (Risk disease in Exposed/unexposed)
Advantages:
• Can measure many exposures, and many diseases
• Temporal relationship clearer (cause before disease)
Disadvantages
• Long and expensive (often very $$$)
• Good for common diseases (some cancers, cardiovascular).
• But not for rare diseases or long latency
• Also what if you fail to measure key determinants
– (Solution = freezer)
Average Annual Incidence of Tuberculosis Among Navy Recruits
By History of Household Contact
Average anual rate per 100,000
160
140
120
100
80
60
40
20
Contact with TB
No Contact with TB
0
0
2.5
5
7.5
10 12.5 15 17.5 20 22.5 25 27.5 30
Mm. of induration on enlisted
Comstock, Edwards, and Livesay; Am Rev Respir Dis 1974; 110:576
Country
N
Test
Incidence of active TB
in IGRA+ groups
The Gambia [Hill et
al. 2008]
2348
ELISPOT (inhouse)
9/1000 person-yr
Turkey [Bakir et al.
2008]
908
ELISPOT
(T-SPOT.TB)
21/1000
S Africa [Mahomed
et al. 2009
(abstract)]
5248
QFT
6/1000 person-yr
Colombia [del Corral
et al. 2009]
2060
In-house wholeblood CFP-10
assay
11/1000 person-yr
Senegal [Lienhardt
et al. PLoS One
2010]
2679
ELISPOT (inhouse)
14/1000 person-yr
person-yr
Randomized Trials
Advantages
• Best method to evaluate an intervention
• Best control of bias and confounding
Disadvantages
• Not easy or feasible for all interventions
• Very Expensive and long
• Not for studies of risk factors or natural history
• Substantial refusal or drop-out rates can restrict
generalizability
• Population selected may not be representative
– Younger adults with only one condition
– Often exclude pregnant woman, kids, elderly!
Solution – Biomarker?
– 2 month culture conversion instead of fail/relapse
Five Phase 2 trials:
2 month culture conversion with FQN
(all patients received 4 months HR in continuation)
Author
Year
Patients
(N)
FQN
regimen
Control
Regimen
2 Month Culture
Conversion
FQN
Control
Burman
2006
277
HRZM
HRZE
71%
71%
Rustomjee
2008
205
HRZG/M
HRZE
79%/42%
64%/36%
Conde
2009
146
HRZM
HRZE
80%
63%
Dorman
2009
328
MRZE
HRZE
60%
55%
Wang
2010
123
HRZEM
HRZE
90%
78%
Three Phase 3 trials of
4 month regimens with FQN
Study
Pts.
(N)
FQN – 4 months
Standard - 6 months
Cure
Death
Fail/
Relapse
Cure
Death
Fail/
Relapse
OFLUTB
1585
79%
1%
16%
83%
1%
10%
Remox
1900
83%
1%
11%
92%
1%
3%
Rifaquin
592
82%
0
18%
95%
1%
4%
Experimental Community or Field Trials
General Design
• Pick an intervention to be applied at a community
level
– Fluoride in water, public education, vaccination
• Find several communities or population groups
• Apply intervention to some and not others
– Randomly again
• Measure outcomes at population or group level
Cluster randomized Trials – example
Effect of improved tuberculosis screening and isoniazid
preventive therapy on incidence of tuberculosis and death in
patients with HIV in clinics in Rio de Janeiro, Brazil: a stepped
wedge, cluster-randomised trial
Betina Durovni, Valeria Saraceni, Lawrence H Moulton, Antonio G Pacheco,
Solange C Cavalcante, Bonnie S King, Silvia Cohn, Anne Efron,
Richard E Chaisson, Jonathan E Golub
• 29 HIV clinics in Rio de Janeiro.
• Staff trained in TB screening, TST and INH.
• Clinics randomly allocated to date of starting the
intervention. 2 clinics started every 2 months
starting from Sept 2005, until Aug 2009
• Outcome: TB incidence +/- death
A randomized trial to compare 4
months Rifampin vs 9 months
INH for the treatment of LTBI
Phase 1: Compliance and completion
Completed in 2003
Phase 2 – Adverse events and costs
Completed in 2007
Phase 3: Efficacy and effectiveness
Enrolment completed in 2014: 840 children, 6020 adults
Follow-up will finish in 2016
Publication in ???
Thanks