Pilot studies
Karla Hemming
The University of Birmingham
Pilot or feasibility study?


No consensus on the definition
Pilot study:



Mini version of the full trial
Tests protocol
Feasibility study:


Tests individual components
Refines / develops intervention
2
Why have a pilot study?

70% of publically funded trials fail to recruit
required number of participants

NIHR increasingly require demonstration of
proof of principal that the TRIAL will work (not
the intervention)
3
What pilot studies are not
about

Not around estimating effect size or CI of
effect size

Why?



Pilot studies are small
Any estimate of effect will be highly uncertain
Mean estimate tells us nothing!
4
Objectives of a pilot study


Show proof of principal of intervention (?)
Test out data collection methods


Inform outcome


Acceptability of questionnaires
Only insofar as what is important to the patient
Estimate important parameters to inform
sample size calculation

(not effect size!)
5
Proof of principal

Need a well defined surrogate outcome
That surrogate outcome would have to have
a very clear link to important clinical outcome

Example – weight management programme




Important clinical outcome BMI
Show adherence improved in a pilot?
Difficulty – to show adherence improved might
need as large a sample as for BMI
6
Important parameters to
estimate

Examples include but are not limited to:





Recruitment rate (informs how long the trial will
have to last for)
Consent rate (informs how amenable patients
are to be randomised)
Retention rate (inform drop out rate)
Continuous outcomes – SD
Binary outcomes – underlying event rate
7
Sample size justification

Sample size should be justified

Depends on objective
8
Objective: estimate SD

Browne 1995

Sample size of 30 is reasonable to give an
estimate of the SD

BUT! Don’t just use mean estimate of SD in
subsequent power calculations – use upper
95 percentile value
9
Objective: estimate a process
rate



Show this rate is above some value
Or within some range (CI)
Called a precision based approach
10
Example



Want to show retention rate (p) is above
some value (c)
Retention should not differ between arms
(blinded trial) – so include both arms in calculation
Precision based approach



Show retention rate above c%
Need an estimate of actual retention rate p%
Specify significance or width of CI (precision)
11
Example cont…

Show retention rate above 60%




Estimate actual retention rate 80%
Recruit SS of 35, number retained 28 (80%)
95% CI would be: 63% to 92%
Great! Showed retention rate above 60% and
can proceed to definitive trial
12
Example cont…

What if retention rate only 70%




That is, recruit 35
But, only 24 are retained
95% CI would be 51% to 83%
Oh no… haven't shown retention rate above
60%
13
Power of this sample size

Method was motivated around CI or precision
based approach (common in pilot studies)

But, the power of a SS of 35 to show that the
retention rate is above 60% is only 50%
(assuming event rate observed is 80%)
14
Precision based methods

Precision methods (CI) crucially depend on
actual observed rate

Associated power of this approach is much
lower than 80%

So, if you use the precision based approach
do not incorporate stringent criteria to follow
to definitive trial
15
Recommendations




Application viewed unfavourably if geared
around effect sizes and hypothesis testing
Rather, should be focused around testing
TRIAL protocol (or components)
Sample size needs to be justified
Don’t have stringent stop go criteria for full
trial
16
Other things…

More than one objective – take the larger SS

Internal pilots

Full trial fundable?

If full trial not likely to be fundable by NIHR, RfPB
unlikely to fund
17