Rationale and Statistical Needs for
Pragmatic Clinical Trials
Robert M Califf MD
Deputy Commissioner for Medical Products and
Tobacco
Penn Conference
Statistical Issues Arising in Pragmatic Clinical
Trials
April 15th, 2015
1
Statistical Needs in PCTs
• Why is pragmatism so important
now?
• What is a pragmatic clinical trial?
• PCTs are a team sport
• Role of the statistician
2
Premise
• Choices about health and health care are
best made when patients, clinicians and
policy makers have access to convincing
scientific evidence to inform and guide their
decisions
• Unfortunately, healthcare decisions too often
based on subjective impressions or indirect
comparisons from studies not designed to
inform decision-making with high quality
level of evidence
3
Premise
• Field of clinical research is maturing and
tools have grown increasingly sophisticated
• Frailty of existing body of medical knowledge
used to support practice has been revealed
• Solution: data-rich integration of research
and practice
4
Deaths per 100,000
Mortality
in
the
20th
Century
3000
Better treatment of
cardiovascular disease,
low birth-weight infants
2000
1000
Reduced infectious disease
mortality (clean water, sewers,
antibiotics, better nutrition)
0
1900 1910 1920 1930 1940 1950 1960 1970 1980 1990
82
Life Expectancy (years)
White females
78
Black females
White males
74
70
Black males
66
62
1975
Adapted from Harper et al,
JAMA 2007;297:1224–1232
1985
1995
Year
Life Expectancy at Birth
2005
% In-hospital Mortality
Link Between Overall ACC/AHA Guidelines Adherence
and Mortality
8
Adjusted
Unadjusted
6
4
2
0
Every 10%  in guidelines adherence
 11%  in mortality
≤25
25–50%
50–75%
≥75
Hospital Composite Quality Quartiles
Peterson et al, ACC 2004
Which Treatment is Best for Whom? High-Quality Evidence is Scarce
< 15% of guideline recommendations supported by high quality
evidence
8
Tricoci P et al. JAMA 2009;301:831-41
Tufts study accounts for direct
costs, failures, and investors’ opportunity cost
The Tufts $2.6B figure for the average cost of bringing an
NME to market has three components:
a) Direct cash costs of
developing the NME
b) Cash expenditures
on failed drugs
Currently Available
Cost Decomposition in Tufts (2014)
Cash
costs
1600
$M
1400
1200
1000
800
c) Cost of capital
600
(i.e. expected rate of return)
400
200
Note: Not all Tufts 2014 study
parameters have been released
Economics Staff
Office of Program and Strategic Analysis (OPSA)
0
Pre-clinical
Clinical
Cost of capital Direct costs + Cost of failures
Source: Tufts 2014
•9
Studies of in-house drug R&D
spending do not exactly agree on trends
Basic cost decomposition across
studies of in-house R&D
Studies are consistent on
rising clinical costs but
inconsistent on spending
on in-house preclinical
development
1,200
1,000
800
Tufts 2003
600
Paul 2010
M-F 2012
400
Tufts 2014
200
In-house R&D development duration
0
Cash pre-clinical
Tufts 2014
M-F 2012
Paul 2010
Tufts 2003
0
50
100
150
200
Months
Preclinical
CDER Economics Staff
Phase 1
Phase 2
Phase 3
NDA
Cash clinical
Studies seem to suggest
that length of Phases 2
and 3 is stable but
duration of preclinical
phase for in-house R&D
may be falling
Office of Program and Strategic Analysis (OPSA)
•10
Summary
• Enormous gap between evidence and
need for evidence
• New technology development costs are
skyrocketing
• Therefore a new path is needed
CDER Economics Staff
Office of Program and Strategic Analysis (OPSA)
•11
General Classification
• Mechanistic trials
– Intent to evaluate a biological or
mechanistic hypothesis
• Pragmatic trials
– Intent to inform decision makers about
health and healthcare
12
Elements of PCTs
• Compare clinically relevant alternatives
• Enroll diverse study population
• Recruit from a variety of practice
settings
• Measure a broad range of relevant
health outcomes
» Tunis, Stryer and Clancy JAMA
13
Practical Adaptation of PCT
Definition
(1) an intent to inform decision-makers (patients,
clinicians, administrators, and policymakers), as
opposed to elucidating a biological or social
mechanism;
(2) an intent to enroll a population relevant to the
decision in practice and representative of the
patients/populations and clinical settings for whom the
decision is relevant; and
(3) either an intent to
– (a) streamline procedures and data collection so that the trial can focus
on adequate power for informing the clinical and policy decisions
targeted by the trial
14
– (b) measure a broad range of outcomes
Pragmatic Clinical Trial
Fit for the purpose of informing decisionmakers regarding the comparative balance of
benefit and risk of a biomedical or behavioral
health intervention at the individual or
population level
We should be striving for pragmatism in every
clinical trial
15
Learning health care systems
Increasing Level of Difficulty
Re-engineering the Clinical Research Enterprise
Plan and start a few demonstration
networks
Simplify complex regulatory systems –
demonstration projects
Plan for networks in place for all institutes
Funding mechanism to sustain national
system through consensus of all
constituents (“1% solution”)
Simplified regulatory system in place for
networks
National Clinical Research System
creates effectiveness data that moves
rapidly into the community AND data
on outcomes and quality of care;
sustained efficient infrastructure to
rapidly initiate large clinical trials;
scientific information for patients,
families, advocacy groups
Establish repositories of biological
specimens and standards for collection
Standardize nomenclature, data
standards, core data, forms for most major
diseases
Start a library of these elements shared
between institutes and NLM
Develop efficient network administration
infrastructure at NIH
Develop standards for capturing images
for research
Data standards shared across NIH
institutes
ONE medical nomenclature with
national data standards (agreed to by
NIH, CMS, FDA, DOD, CDC)
Data standards updated ‘in real time”
through networks
National repository of images and
samples
Critical national “problem list”
Most efficient network funding
mechanisms in place across NIH
Create NIH standards to provide “safe
haven” for clinical research
Inventory and evaluate existing publicprivate partnerships, networks, CR
institutions, and regulatory systems
Establish FORUM(S) of all stakeholders
Establish standards for and pilot creation
of a National Clinical Research Corps
Demonstration/planning grants to
enhance/evaluate/develop model
networks
NIH standards for safe haven in place
Regulations and ethics harmonized with
FDA, CMS
Public private partnership mechanisms in
place
100,000 members of certified “Clinical
Research Corps”
Standards shared across NIH
1-3 years
Funding mechanisms evaluated to
determine which are most efficient
4-7 years
Time
Participation in research is a
professional standard (taught in all
health professions schools)
Study, evaluation and training
regarding clinical research a part of
every medical school, nursing school,
pharmacy school
Clinical research practices
documented and updated regularly to
maintain safe haven
Networks provide detailed training
about network specific issues
8-10 years
Conclusion Clinical trials registered in
ClinicalTrials.gov are dominated by small
trials and contain significant heterogeneity
in methodological approaches, including
reported use of randomization, blinding,
and DMCs.
•18
18
What Have We Learned?
• More trials than previously imagined (>380/week)
• Unexplained heterogeneity in
– Trial design
– Trial oversight
• Lack of fidelity to trial protocol is common
• Apparent lack of expertise among trial personnel
• Published literature does not reflect the “CRE”
– Only about 67% of trials ever published
– Only some outcome measures published
• Discrepancies among various sources of
summary data raise questions about validity
•19
Prioritization
• When deciding upon the question there are
several approaches
–
–
–
–
Traditional peer review (open market place of ideas)
Net present value calculations (value to a business)
Value of information analysis (value to society)
Programmatic decisions based on gaps and needs
(“top down”)
• Like other decisions priorities should be
informed by evidence
– Where are the gaps?
– What could be gained by a trial?
20
The Team
•
•
•
•
•
Clinical investigators
Statisticians
Informaticians
Operational experts/Regulatory scientists
PATIENTS AND PATIENT
REPRESENTATIVES
• PAYORS/HEALTH SYSTEMS
21
NY Times Dec 25th, 2014
Headline: M.B.A. Programs Start to Follow
Silicon Valley Into the Data Age
“Meanwhile, across the country, colleges are
adding new courses in statistics, data
science and A/B testing, which often
involves testing different web page designs
to see which attracts more traffic.”
Even Dilbert is using AB Testing
(Randomization)
Protocol Design
• Traditional statistician roles
– Keeping the team focused on hypothesis testing
– Calculations for samples size, power, subgroup analysis,etc.
• Enhanced roles
– Helping to assess data sources
– Choosing basic design (CRT, traditional RCT, adaptive, etc.)
– Dealing with magnified issues of missing data or uncertainty
about key outcomes
– Quality by design plan
• Statistical process control
24
A well designed pragmatic trial
25
Our usual Clinical Trial after
Regulatory/FDA/Academic Interactions
26
One person’s data can be in different
places
Data in claims
Data in ambulatory EHRs
Care public health
clinic
Prescription fills paid
out of pocket
Data in inpatient EHRs
Study Start Up
•
•
•
•
Efficiency of procedures
Practicality of procedures/missingness
Assessment of data quality early on
Statistical process control procedures
28
http://www.nytimes.com/2014/08/18/technology/for-bigdata-scientists-hurdle-to-insights-is-janitor-work.html?_r=2
Implementation—Trial Monitoring
• Interim analysis
• Quality by design
– Focusing on errors that matter
• For CRTs think about Intraclass Correlation
Coefficient
• Adaptation to observations
30
Analysis
• More people at the table
• Complex steering committees
• Patients and patient advocates
32
“What a couple of clucks we are. Here’s another error up here!”
Page 33
Dissemination
•
•
•
•
•
•
•
ClinicalTrials.gov
Placing the result in context
Traditional manuscripts
Clinical practice guidelines
Payers
Patient advocacy
Transparency
34
Special Article
Compliance with Results Reporting at
ClinicalTrials.gov
Monique L. Anderson, M.D., Karen Chiswell, Ph.D., Eric D. Peterson, M.D.,
M.P.H., Asba Tasneem, Ph.D., James Topping, M.S., and Robert M.
Califf, M.D.
N Engl J Med
Volume 372(11):1031-1039
March 12, 2015
Cumulative Percentage of Clinical Trials That Reported Results to ClinicalTrials.gov,
According to the Time after the Primary Completion Date.
Anderson ML et al. N Engl J Med 2015;372:1031-1039
An Aspiration of the Clinical Research
System– a reliable product
• consistently good in quality or performance; able to be
trusted.
• "a reliable source of information"
• Synonyms: dependable, good, well founded, authentic,
valid, genuine, sound , true
• The biostatistician is in the best position to be the product
manager and assessor
• As data get more complicated confidence in the reliability
of the product is increasingly dependent on:
– Replication
– A track record of reliability
The Pipeline—Sheer Numbers
• There is an absolute shortage of qualified
statisticians who can produce reliable results
• We need new methods
• PhD programs seem more focused on the
historical profession than filling the huge
societal need
• MS programs not linked enough with the clinical
research community
• We need a national program to increase the
supply of quantitatively competent people
Pipeline-Integration
• Quantitative Sciences
–
–
–
–
–
Biostatistics
Statistics
Computer science
Bioinformatics
Data analytics
• Clinical Research
– People educated in both clinical science and quantitative science
– What quantitative skills should be expected of
• Clinical investigators
• Practitioners