Statistical considerations
for grants
Brian Healy
Comments from previous class
Change time of course
 Available on-line power calculators

– http://www.cs.uiowa.edu/~rlenth/Power/
Two-sided vs. one-sided
 Comparison of statistical packages

Review
Type I error
 Type II error
 Ways to increase power
 Power/sample size calculation with
continuous outcome

Type I error
We could plot the distribution of the sample
means under the null before collecting data
 Type I error is the probability that you reject
the null given that the null is true
a = P(reject H0 | H0 is true) Notice that the shaded

a
area is still part of the
null curve, but it is in
the tail of the
distribution
Type II error
Definition: when you fail to reject the
null hypothesis when the alternative is in
fact true (type II error)
 This type of error is based on a specific
alternative
b= P(fail to reject the H0 | HA is true)

Power

Definition: the probability that you reject
the null hypothesis given that the
alternative hypothesis is true. This is what
we want to happen.
Power = P(reject Ho | HA is true) = 1 - b

Since this is a good thing, we want this to
be high
Outline
Aspects of statistical considerations
section of a grant
 Example statistical analysis section
 Worked example from dataset from
students in class
 Management of data collection/
spreadsheet

Aspects of statistical considerations

Overarching statistical issues:
– Data management
– Methodological issues esp. related to data collection
(ex. Image processing)
– Handling missing data
– Clustering/correlation of observations

Specific aims:
– Identify outcomes/explanatory variables
– Type of analysis
– Power calculation
Research study
I.
Study design
•
•
Experimental question- What are you trying to
learn? How will you prove this?
Sample selection- Who are you going to study?
II. Data collection
•
What should be collected?
III. Analysis of data
•
•
Results- Was there any effect?
Conclusions- What does this all mean? To whom
do results apply?
Experimental question:
What? How?
Sample selection: Who? How many?
Collect Data
Analysis: Is there an effect?
Conclusion: To whom?
How is statistics related to each
stage?
I.
Study design
• Experimental question- Define
outcome, sources of variability, unit
and analysis plan
• Sample selection- Sample size, type of
sample
Experimental question

In a grant, the experimental question is written
as the specific aims
– Generally, specific aims can be easily translated to a
null hypothesis
– If specific aims are more general, the specific null
hypotheses are listed in the grant after the aims
– This is the critical step in the grant because
everything else is based on the aims
– Usually easiest if can set up hypothesis as Y/N
question
Example
Dr. Janet Hall kindly provided a grant to use as
an example
 One goal of the grant was to investigate
whether age had an effect on estrogen
treatment in post-menopausal woman

– Is there an interaction between estrogen and age?
– The treatment is given to increase resting metabolic
activity in the brain as measured by PET and other
neuroimaging modalities

In addition, the effect of age on resting
metabolic activity at baseline (untreated) was of
interest
Specific aim 1

SPECIFIC AIM #1: To determine the effect of
aging on changes in baseline (resting state)
cortical function and their responses to estrogen
using FDG-PET.
– Hypotheses:
– Resting state metabolic activity, as measured by FDGPET at baseline, is decreased in the dorsolateral
prefrontal cortex (DLPFC) and increased in the
hippocampus as a function of age.
– Estrogen exposure results in progressive increases in
resting metabolic activity in the DLPFC over time in
young postmenopausal women that is not seen in
their older counterparts.
Hypothesis 1
Resting state metabolic activity, as measured by
FDG-PET at baseline, is decreased in the
dorsolateral prefrontal cortex (DLPFC) as a
function of age.
 What is the experimental question?

– Is the FDG-PET level different in the hippocampus or
DLPFC for women of different ages?
– What is the outcome?
– What is the explanatory variable?
Types of variables
The outcome is FDG-PET level and this is a
continuous variable
 The explanatory variable, age, could be
considered continuous, but for this grant it was
decided to group patients into young postmenopausal women (age 45-55) vs. old postmenopausal women (age 70-80)
 What type of analysis would we use in this case?

– Are the data approximately normal?
Sample selection

Our sample selection is based on the
definition of the groups
– What is the effect of this definition? Does it
affect the generalizability of the findings?

For this study, we plan to sample small
groups from a single site
– Could another approach have been used?
– What is the advantage of a single site?
Disadvantage?
Sample size calculation
We have defined our null hypothesis, outcome
and sample selection
 What sample size do we need?
 In this case, previous data showed mean (SD)
FDG-PET at DLPFC in the young group of 83.0
(7.3) and in the old group of 76.2 (7.3)

– What else do we need for our sample size calculation?
– Power=0.8, a=0.05

Assuming equal groups, we need 20 patients per
group
Additional considerations

Multiple comparisons
– We have two outcomes so should we adjust
the significance level for the two
comparisons?
– Bonferroni correction for significance level
Do any specifics regarding the
measurement need to be discussed?
 Confounders/adjustment

Abbreviated grant section
Analysis plan: The two groups will be compared
using a two-sample t-test.
 Power calculation: Previous data has estimated
the mean (SD) FDG-PET in the young group of
83.0 (7.3) and in the old group of 76.2 (7.3).
Group sample sizes of 20 and 20 achieve 82%
power to detect a difference of 6.8 between the
two assuming a standard deviations of 7.3 in
each group and a significance level of 0.05 using
a two-sided two-sample t-test.

Hypothesis 2
Estrogen exposure results in progressive
increases in resting metabolic activity in
the DLPFC over time in young
postmenopausal women that is not seen
in their older counterparts.
 What is the experimental question?

– Is the effect of estrogen on the FDG-PET level
in the DLPFC different for women of different
ages?
Types of variables
One potential outcome is change in FDGPET level and this is a continuous variable
 Age group and treatment are the
explanatory variables
 How many FDG-PET levels are measured
and how many observations contribute to
the analysis?
 What type of analysis could we use in this
case?

Data set-up
We measure the change in four types of patients
(young/treated, young/placebo, old/treated,
old/placebo)
 We can estimate the mean change in all groups,
but what is truly of interest for our hypothesis?

– Interaction between the two measures
– Linear regression/two-way ANOVA
Young
Old
Treated
MeanYoung,treated
MeanOld,treated
Placebo
MeanYoung,placebo
MeanOld,placebo
Mean in treated
old patients
Mean in
untreated old
patients
Mean in treated
young patients
Mean in
untreated
young patients
Sample selection
Now that our outcome and explanatory
variable are clearly defined
 Our sample selection in this case is a little
more complex

– Age group is defined by enrollment
– Patients in each group were randomized to
treatment or placebo
– What does the randomization get for us?
Sample size calculation
We have defined our null hypothesis,
outcome and sample selection
 What sample size do we need?

– What preliminary data would we need or what
would we need to hypothesize to calculate the
sample size?

Some resources for this complex design
on-line, but likely you should consider
speaking to a statistician for this
Abbreviated grant section
Analysis plan: The effect of age on the
treatment effect of estrogen in post-menopausal
women will be investigated using a two-way
ANOVA. The outcome for the analysis will be the
change in the FDG-PET level before and after
the treatment and the two factors will be age
and treatment group. The focus of the analysis
will be the interaction between the two factors.
 Power calculation: Given our preliminary data
and available sample size, we will have 80%
power to detect a hypothesized difference of x
using a two-way ANOVA.

Alternative analysis strategy

Rather than focusing on the difference between
the before and after treatment measurements,
we could have included all of the measurements
in a single model
– Each patient contributes a before and after treatment
measurement rather than a difference

The analysis of this approach requires
accounting for the repeated measures within a
subject
– Repeated measures ANOVA or mixed effects model
Advantages of this approach
Handles missing data more easily
 Generalizes to more than two
measurements easily
 Power calculations with mixed effects
models can be completed as well

Conclusions
Each hypothesis needs an analysis plan
that describes the type of data and
statistical approach used to analyze the
data
 Each hypothesis also requires a sample
size or power calculation
 Additional issues (missing data,
confounding) must be included in the
statistical analysis section

Worked example
Kidney transplant research

Students in the class are investigating the effect
of genetics of the donor/recipient pair on various
outcomes
– Creatinine level measured at time of transplant, 3
month, 6 month, 12 month and 36 months after
transplant
– Time to rejection of the transplant
– Type of rejection (acute/chronic)

Genetic factor of interest is large deletion
polymorphisms at 20 sites
Study design

Patients have been followed at 4 different sites
since 1995
–
–
–
–

Korea
Finland
BWH
MGH
Only HLA genetic data is available at the
moment, but would like to genotype sufficient
numbers of patients to determine if there is an
effect
Experimental question

Specific aim: To explore the potential
contribution of a new class of large deletion
polymorphisms on the development of acute and
chronic renal allograft rejection following renal
transplantation.
– Hypotheses:
– Donor/recipient pairs with matching deletion
polymorphisms will have lower creatinine levels at all
time points compared to non-matched pairs
– Donor/recipient pairs with matching deletion
polymorphisms will have fewer acute/chronic
rejection events compared to non-matched pairs
Definition of groups
Both the donor and recipient for each transplant
will be genotyped and classified as either having
the deletion or not having the deletion
 We decided to treat each group separately
initially. What type of variable is the explanatory
variable?

Donor with
deletion
Recipient with
deletion
Group 1
Donor w/o deletion Group 2
Recipient w/o
deletion
Group 3
Group 4
Creatinine levels
Here are the initial
values for the
creatinine for one of
the populations
 Note the outliers at
the end of the
distribution. These
would be very
important to model
 Turned out they were
incorrect data

Analysis plan
Initially, we will compare each creatinine
measurement separately
 Since I have 4 groups (categorical
outcome) and a continuous outcome, I will
compare across the groups using ANOVA

– The corrected data look sufficiently normal to
make this analysis plan reasonable
– An alternative option would be to use a
Kruskal-Wallis test, which is a rank-based test
that is not sensitive to the outliers
Abbreviated analysis plan

Analysis plan: The four groups of donor/recipient
pairs will be compared using ANOVA. If a
significant difference between the groups is
observed, the pairwise comparisons will be
completed with the appropriate correction for
multiple comparisons. Although we could
investigate the main effect of the donor’s and
recipient’s deletion status in a two-way ANOVA
model, our interest is in the four group
comparison given the relationships seen in
previous work.
Additional considerations

Rather than modeling each creatinine separately,
should we model them together?
– Trend with time?
– Multiple comparisons if treat separately?

Confounders:
– Age
– Gender
– HLA status

Should we treat all 20 deletion separately?
Power calculation
Unlike the previous example, we have no
preliminary data regarding the effect of
these deletions
 How can we complete a power
calculation?

– Option 1: Propose a sample size from each
group and determine the difference between
groups you could detect
– Option 2: Estimate the effect using an
available measurement/literature value
Available measurement
In the dataset, we have HLA status and
can calculate the mean (SD) in each of the
four groups
 Using this preliminary data, we can
perform a power calculation and assume
the effect size for the deletions will be
similar to HLA

– How good of a surrogate is HLA for deletion?
Abbreviated power calculation

Power calculation: Our preliminary data have
shown that the mean (SD) month 12 creatinine
levels of the recipient was 1.21 (0.29) in HLA
identical donor/recipient pairs and 1.28 (0.33) in
HLA non-identical donor/recipient pairs. We
anticipate that recipients who are deletion
matches will behave like the HLA identical
recipients and recipients who are not deletion
matches will behave like the HLA non-identical
recipients. A sample size of 202 per group is
required to have 80% power to detect the
proposed difference between the groups at the
0.05 level using one-way ANOVA
Additional considerations
Pairwise tests
 Is there a better approximation for the
group means?
 Clustering by country

Proportion with acute rejection
Another outcome for the study is the proportion
of patients who experience acute rejection
 The table at the end of the study would look like
this:

Dyes/Ryes
Acute
rejection
No acute
rejection
Dno/Ryes
Dyes/Rno
Dno/Rno
Abbreviated analysis plan

Analysis plan: The proportion of patients
who have acute rejection will be compared
across the groups using a chi-square test
for each deletion separately. In order to
investigate the combined effect of
deletions, multiple logistic regression
models will also be fit.
Power analysis
As previously, there is no preliminary data,
but let’s try the set sample size approach
now
 Assume that we have two groups,
matched and non-matched, and we have
200 matched patients and 400 nonmatched patients
 What type of power analysis could we
complete?

Abbreviated power analysis

Power analysis: Given our sample size (200
matched patients and 400 non-matched
patients) and the assumption that matching
would decrease the proportion with acute
rejection, we will have at least 80% power to
detect the differences presented in Table xx.
Proportion with acute
rejection among matches
0.2
Proportion with acute
rejection among mismatches
0.33
0.3
0.4
0.5
0.45
0.55
0.65
Additional considerations

Clustering by region
– Stratified analysis
Management of data collection
Example Excel sheet
 All relevant information should be included
as a column (Loss to follow-up date)
 No symbols in column names (ex. #) and
column names should be as short as
possible
 No empty rows or columns (white space)

Conclusions
Experimental question must be well
defined to set up an appropriate analysis
plan
 Sample size calculation based on analysis
plan. If uncertain of power calculation,
consult a statistician
 Attempt to address other aspects of your
data

THANK YOU!!!