The analysis of longitudinal studies
in rheumatology:
Is it done correctly?
6th Annual Clinical Research Methodology
Course
Friday, December 16, 2011
Emmanuel Lesaffre
Department of Biostatistics, Erasmus MC, Rotterdam, the Netherlands
L-Biostat, K.U.Leuven, Leuven, Belgium
In collaboration with
Karolina Sikorska, Maurits de Rotte, Jolanda Luime, Mieke Hazes
1
Conclusions
1. Many longitudinal studies in rheumatology are not using an appropriate
statistical methodology.
2. Repeated significance tests and summary statistics methods are
inappropriate in practice and can not show the evolution of an outcome
over time.
3. Classical repeated measurement methods are inappropriate in practice.
4. The mixed effects model and the GEE approach can make use of all
available cases independently whether they have data on all time points
studied.
5. The mixed effects model and the GEE approach are sparsely used in
longitudinal studies rheumatology.
2
Description JIA study
3
Juvenile Idiopathic Arthritis
(JIA) study
• Description
– Longitudinal study: 5 time points (0, 3, 6, 9, 12 months)
– Recruitment at University Medical Centre Utrecht (UMCU) and
Wilhelmina Children’s Hospital, the Netherlands
– Children diagnosed with JIA according to the ILAR criteria and started
methotrexate (MTX) therapy between 1990 and 2006
– All patients and their parents gave their informed consent and approved
by the medical ethics committee of the UMCU
– Treatment was tight-controlled using a standardized report form on
disease activity every 3 months
– Information on MTX usage, disease activity, route of administration,
dosing of MTX, reasons for ending MTX treatment, concomitant therapy
and laboratory parameters were collected
4
JIA study
• Description
– 183 JIA patients of our test cohort were genotyped for ABCB1
3435C>T.
– 2 groups: genotype CC homozygous (53 patients, 28%)  T-carrier
(129 patients, 72%)
– Continuous response = erythrocyte sedimentation rate (ESR) was
obtained on all 5 visits for 106/183 patients (58%): CC homozygous (31
patients, 29%), T-carrier (75 patients, 71%) = complete cases
– ESR was log-transformed because of a skewed distribution.
– Binary response = ACR30 response, obtained for all 183 patients on all
visits. Of the 183 patients, 46 (25%) responded at 3 months, 85 (46%)
at 6 months, 110 (60%) at 9 months and 113 (62%) at 12 months.
5
Classical analyses for ESR
6
JIA study
4.0
• Classical analysis (ESR)
3.5
CC-homozygous
T-carrier
+
2.5
O
+
+O
O
+
2.0
+
O
1.5
45
118
48
1.0
ln(ESR)
3.0
O
0
2
42
107
48
105
4
6
50
96
8
103
P= 0.44
P= 0.33
P= 0.62
P= 0.12
P= 0.066
10
12
TIME (months)
7
Classical approaches
for comparing 2 groups over time
1. Repeated significance tests (t-tests, Wilcoxon tests, etc)
2. Summary Statistic Method: summarizing whole curve by e.g. AUC
and compare AUC between 2 groups with t-test
3. Repeated measurements ANOVA
4. MANOVA
8
260
NS
NS
240
NS
NS
220
P < 0.05
200
MEAN CHOL+/- SEM
280
300
1. Repeated significance tests
1
2
3
4
5
VISIT
9
1. Repeated significance tests
JIA study (ESR)
• Here unpaired t-tests giving P= 0.066, 0.12, 0.62, 0.33, 0.44
• Nowhere significant  no real issue here
• Suppose
P= 0.066, 0.03, 0.62, 0.33, 0.44 … what to conclude?
• Or
P= 0.066, 0.12, 0.62, 0.33, 0.03 … what to conclude?
• If no adjustment of P-value, then problem of multiple testing
10
1. Repeated significance tests
• Popular approach in medical research
• But
– Inefficient: only patients are included that are present at visit
– Likely to be biased: missing data process can bias analysis
– No good insight in individual evolution: link between subsequent
observations is lost
– Turns longitudinal study into several cross-sectional studies
• Not suitable for contemporary rheumatology studies
11
JIA study
• All individual profiles (ESR)
CC homozygous
T-carrier
12
JIA study
5
• 3 individual profiles (ESR)
*
3
DROPOUT
*
o
2
INTERMITTENT MISSING
+
COMPLETE CASE
o
o
+
o
1
ln(ESR)
4
*
o
0
2
4
6
8
+
10
12
TIME (months)
13
2. Summary Statistic Method
JIA study (ESR)
• Compute Area-under-the-Curve (AUC) for each profile
• On complete cases only:
CC-Homozygous – T-carrier= -1.33, P = 0.54, 95% CI=(-5.61; 2.96)
• Complete the incomplete cases by LOCF:
CC-Homozygous – T-carrier= -3.78, P = 0.04, 95% CI=(-7.33; -0.21)
• In addition one could compare
– Averages of the profiles
– ….
14
JIA study
5
• 3 individual profiles (ESR): incomplete profiles completed by
Last-Observation-Carried-Forward (LOCF)
*
3
2
*
o
LOCF
+
o
o
+
o
LOCF
1
ln(ESR)
4
*
o
0
2
4
6
8
+
10
12
TIME (months)
15
2. Summary Statistic Method
• Not frequently used in medical research
• Recommended for balanced data
– Balanced: all patients are examined at all (regular) visits
– Difficult for unbalanced data, i.e. when there are missing
observations and patients can come at all times
– LOCF is a popular approach to make data more balanced
• LOCF
– Imputes unrealistic values
– Underestimates variability of the data
• Not suitable for contemporary rheumatology studies
16
3. Repeated Measurements ANOVA
• One of only 2 approaches available for analysis of repeated
measurements 50 years ago
• Requires balanced data
– Patients with missing values are excluded
– Time points must be regular, patients with irregular time points are
excluded
– Restrictive assumptions on correlation between subsequent responses:
correlations are equal (corrections available but not sufficient)
– Splits treatment effect up into time, group and group*time effect
• Not suitable for contemporary rheumatology studies
17
3. Repeated Measurements ANOVA
JIA study (ESR)
• On complete cases only:
– CC-Homozygous – T-carrier=
• Time effect:
P < 0.001
• Genotype effect:
P=0.47
• Time*genotype effect:
P=0.02
• Complete the incomplete cases by LOCF:
– CC-Homozygous – T-carrier=
• Time effect:
P < 0.001
• Genotype effect:
P=0.04
• Time*genotype effect:
P=0.34
18
3. Repeated Measurements ANOVA
JIA study (ESR)
• Assumption equal correlation??
TIME (months)
0
3
6
9
12
0
1
0.71
0.51
0.46
0.42
3
0.71
1
0.61
0.47
0.48
6
0.51
0.61
1
0.70
0.64
9
0.46
0.47
0.70
1
0.69
12
0.42
0.48
0.64
0.69
1
19
4. MANOVA
• Other of only 2 approaches available for analysis of repeated
measurements 50 years ago
• Requires balanced data
– Patients with missing values are excluded
– Time points must be regular, patients with irregular time points are
excluded
– No structure allowed on correlations  large studies are required
– Splits treatment effect up into time, group and group*time effect
– Output more difficult to understand
• Not suitable for contemporary rheumatology studies
20
4. MANOVA
JIA study (ESR)
• On complete cases only:
– CC-Homozygous – T-carrier=
• Time effect:
P < 0.001
• Genotype effect:
P=0.47
• Time*genotype effect:
P=0.03
• Complete the incomplete cases by LOCF:
– CC-Homozygous – T-carrier=
• Time effect:
P < 0.001
• Genotype effect:
P=0.04
• Time*genotype effect:
P=0.36
21
Modern analyses for ESR
22
“Modern” approaches
for comparing 2 groups over time
1. Types of missing data processes
2. Approaches to deal with missing data
3. Mixed effects models
4. GEE models
23
1. Types of missing data processes
• What are missing data?
• Data that are not observed
• How are missing data generated?
• Patients/clinicians forget to fill item(s) in questionnaire
• Patients refuse to fill item(s) in questionnaire
• Lost or damaged biological sample
• Patients miss a visit because …
• Patients decide not to return to clinician anymore
• Patient died
• …
 Different reasons why data are missing
24
Impact of missing data
• What is believed
• Loss of efficiency
• In reality
• Loss of efficiency
• Biased results (often)
 Classical statistical methods need to be adapted
• Approach
• CLEVER explicit or implicit imputation of missing data
• But: can NEVER replace true data
25
Terminology
•
Monotone missing
• Also called dropouts
1
2
3
4
1
2
3
4
1
2


n
•
Non-monotone missing
• Also called intermittent missingness
1
2


n
26
More terminology
• Missing completely at random
• Missing at random
• Missing not at random – informative missing
• Terminology might be confusing – Rubin (1975)
27
Missing completely at random (MCAR)
• Probability of missingness is independent of all responses
• Examples
• A random selection of teeth in mouth are taken in the study
• A blood tube is dropped
• Patient died in a car accident, but careful: patient could have
experienced a sleep attack when taking a dopamine agonist
• Then
• Simple mean of response is unbiased estimate of true mean
• Classical statistical techniques (repeated t-tests, Summary statistics
method, repeated measurements ANOVA, MANOVA) can be used
• Impact: loss of efficiency
28
Missing at random (MAR)
• Probability of missingness depends on observed responses
• Examples
• Study design specifies that if blood pressure is not lowered patient will
be removed from anti-hypertensive trial
• Multi-stage screening: data are missing at subsequent stages due to
result at initial stage (negative test)
• Then
• Simple mean of response is biased estimate of true mean
• Classical statistical techniques canNOT be used
• Statistical tests to distinguish versus MCAR exist
• Impact: loss of efficiency + bias, but likelihood analysis can
correct for bias due to missingness
29
Missing not at random (MNAR)
• Probability of missingness depends on observed responses &
unobserved responses
• Examples
• Patient shows a flare up in the disease unobserved in the study +
patient decides to leave the study
• Then
• Simple mean of response is biased estimate of true mean
• Classical statistical techniques canNOT be used
• No test to distinguish versus MAR
• There is no satisfactory analysis, ONLY sensitivity analysis
• Impact: loss of efficiency + bias, only a sensitivity analysis
can shed light on the problem
30
Bias of simple mean
COMPLETE
1
2
3
4
5
20
15
10
RESPONSE
5
0
0
0
5
10
RESPONSE
15
15
10
5
0
RESPONSE
MAR
20
20
MCAR
0
1
VISIT
2
3
4
5
0
1
2
VISIT
3
4
VISIT
COMPLETE
2.07
4.41
6.87
9.03
11.57
13.30
MCAR
2.14
4.07
6.50
8.53
10.95
12.49
MAR
2.07
4.41
7.41
10.30
17.14
20.22
31
5
2. Approaches to deal with missing
data
• Prevention and planning
• Analytical remedies
• Complete case analysis
• Available case analysis
• Imputation techniques
• Likelihood-based analyses
• Weighted analyses
• Sensitivity analyses
 Most appropriate statistical solutions are COMPLICATED
32
2. Approaches to deal with missing
data
• Complete case analysis (MCAR)
– Default analysis in many packages, only ok for MCAR
– If not MCAR: substantial bias can be the result
• Available case analysis (MCAR)
– Use for each variable/time point all observations available
• Single value imputation (MCAR, MAR??)
– Mean value imputation
– Hot decking
– LOCF
methods are based on unrealistic models & underestimate variance
33
Example: LOCF
LOCF
time
34
2. Approaches to deal with missing
data
• Multiple imputation (MAR)
– Explicit imputation of missing data
– Incorporate random mechanism
– Generate M different completed imputed data sets
– Combine M means and M variances  1 overall mean & variance
method is based on statistical model for imputation
35
2. Approaches to deal with missing
data
• Likelihood-based models (MAR)
– Implicit imputation of missing values
– Model-based:
• Linear & generalized linear regression
• Linear & generalized linear mixed models
• …
method is based on statistical model for response
• GEE models (MCAR) and weighted GEE models (MAR)
• Bayesian models (MAR)
36
2. Approaches to deal with missing
data
• MNAR models
– Model also missing data mechanism
– Complex modelling
– Never completely satisfactory  sensitivity analysis
– BUT: if time points are close to each then MNAR close to MAR
37
3. Mixed effects models
• Two examples
RANDOM INTERCEPT MODEL
RANDOM INTERCEPT + SLOPE MODEL
250
140
eps _43
200
150
RESPONSE
120
100
100
80
RESPONSE
b04
6
7
8
TIME
9
10
6
7
8
9
10
TIME
38
3. Mixed effects models
• Assumption in mixed effects models
39
3. Mixed effects models
• Not enough used YET in medical research
• Recommended for contemporary rheumatology studies
– Allows unbalanced data
– None of the patients are deleted from the study
– Irregular time points are allowed
– No explicit imputation of responses
– Time evolution, effect of covariates and correlation structure: all can be
flexible and modelled
• BUT
– One needs statisticians for the job
– Luckily: they are OFTEN cheap
40
3. Mixed effects models
• Linear mixed effects models
– Response = continuous
– Normality assumptions
• Generalized linear mixed effects models
– Response = continuous or discrete
– Normality assumptions
41
3. Mixed effects models
JIA study (ESR)
• On ALL cases:
– CC-Homozygous – T-carrier=
• Time effect:
P < 0.001
• Genotype effect:
P=0.03
• Time*genotype effect:
P=0.09
• + ESTIMATES and 95% CIs
42
3. Mixed effects models
JIA study (ESR)
43
4. GEE models
• Can cope only with MCAR
• Makes only few assumptions about distribution of data
• Most popular for discrete responses
• Can also be used for continuous response
44
4. GEE models
JIA study (ACR30)
• Binary response
• No missing data in the study
• Possible analyses
– Chi-square tests (Fisher’s exact tests) at each time point
– Generalized mixed effects model
– GEE model
• Same comments as before
• Here all analyses gave non-significant difference between genotype
groups
45
Literature review
46
Literature review
•
Literature search (criteria) :
– PubMed
– January 1- December 31, 2008
– Annals of the Rheumatic Diseases & Arthritis and Rheumatism
– Patients with arthritis were followed up in time over > 3 measurements
– Response that could vary over time
•
Information collected:
–
–
–
–
•
Research aim
Primary outcome, secondary outcomes
Used technique for handling with missing data
Statistical analysis techniques used
Eligible studies were ranked according to degree they correctly
analyzed the longitudinal research question
47
Literature review
Study characteristics
•
203 longitudinal studies in ARD or in AR
•
156 out of these studies dealt with arthritis
– 11 studies with JIA patients
– 58 studies described a RCT
– 98 studies described a longitudinal cohort study
•
110 studies were included (excluded: 30 studies not longitudinal, 1
study was not first published in 2008, 16 studies did not include
patients with arthritis)
Statistical techniques used
•
110 studies were ranked
•
17 made optimal use of modern statistical methods
48
Literature review
Ranking
Description
Suitable for repeated measurement analysis
Number of studies
110
1
Repeated character of the data not taken into account
78
2
Use of Bonferroni correction for multiple testing
7
2
Use of Repeated Measurements ANOVA or MANOVA
8
3
Appropriate repeated measurement techniques used on
all suitable research questions
17
49
Conclusions
50
Conclusions
1. Many of the longitudinal studies in rheumatology are not using an
appropriate statistical methodology.
2. Repeated significance tests and summary statistics methods are
inappropriate in practice and can not show the evolution of an outcome
over time.
3. Classical repeated measurement methods are inappropriate in practice.
4. The mixed effects model and the GEE approach can make use of all cases
independently whether they have data on all time points studied.
5. The mixed effects model and the GEE approach are sparsely used in
longitudinal studies rheumatology.
51
Thank you for your attention
52