Technical Appendix to Tiffin, Dowell & McLachlan BMJ 2012; 344: e1805.
Full methods and results of missing data sensitivity analysis by multiple
imputation
Background and rationale
Missing data theory generally classifies missing values into three categories: 1
1. Missing completely at random (MCAR)- whether the data is missing and the
values of the missing observations are the work of complete chance (e.g. a
respondent accidently skipping a question) and are therefore unrelated to
outcome or other observed variables.
2. Missing at random (MAR)- whether the observations are missing and the
unobserved values are not the work of pure chance but are related to observed
variables.
3. Non-ignorable missing data/missing not at random (MNAR) data- this data
systematically missing data that cannot be related to the values of observed
variables.
In the present study whether an applicant had missing advance qualification and
socioeconomic status (SES) data was related to observed variables (including the
outcome of offer) and was therefore not MCAR. However, if the data were MAR rather
than MNAR then we could be relatively confident that the findings from the multi-level
multilevel logistic regression models were robust despite these unobserved values. This
is because in a multiple regression the predictor variables are conditioned on the other
independent variables. If missing values could also be predicted from observed variables
then the results would be unlikely to change significantly. Therefore in this case, rather
than relying on imputed values, we used multiple imputation as a form of sensitivity
analysis in order to estimate the robustness of our inferences in the face of the relatively
high proportion of missing values for advanced qualifications and SES.
Methods
The relationships between missing data status and other variables were initially
evaluated using logistic regression in order to establish that the data were not MCAR.
Therefore in order to estimate the extent to which these missing data could be
considered MAR a series of sensitivity analyses were performed using multiple
imputation procedures. 2 For the SES data ten imputed data sets were produced. A
univariate imputation procedure was utilised whereby the dichotomous imputed values
(low SES present or absent) were conditioned, via a logistic regression, on the observed
predictor variables. Advanced qualification data was excluded from this missing data
modelling as in around 10% of cases candidates were missing both information SES and
advanced qualifications. Thus, this would have led to an inability to impute SES values
for this minority of candidates. Whilst imputation procedures can be used to emulate
clustering in multilevel models this was unnecessary in this case as the variables to be
imputed (SES and advanced qualifications) were missing at the individual level.
Consequently, missing SES categories could be imputed for 4,346 candidates. Only
values relating to 259 candidates could not be imputed due to missing values on one or
more other sociodemographic variables (aside from advanced qualifications).
Similarly for the missing advanced qualification data ten imputed data sets were
created. Again, a univariate imputation procedure was used though this time the imputed
values (standardised UCAS tariff scores) were treated as continues and condition via a
linear regression on the observed predictor variables (with the exception of SES, again
due to the high rate of missingness). In order to allow for the censored nature of these
standardised UCAS tariffs (i.e. no candidate could obtain more than around 1.27
standard deviations above the mean as only the grades from best of three or equivalent
A levels were counted) imputed values above 1.27 were replaced with the theoretical
maximum standardised tariff observable. Values could be imputed for 4,729 candidates
where advanced qualification data had been missing. Imputed values for 188 candidates
could not be generated due to missing predictor variables required for the missing data
modelling.
Results
As expected, the imputed datasets contained higher rates of low academic achievement
and low SES than observed in the non-imputed data. As the data relating to applications
contained higher rates of missingness compared to that related to entrants the multilevel
logistic regression analyses relating to obtaining conditional and unconditional offers
were repeated using the multiply imputed data sets, with results being pooled. The
results from the analysis of the imputed datasets were generally very similar to those
relating to the original, non-imputed data. For the data with missing SES categories
imputed the ORs obtained for the probability of obtaining a conditional offer only differed
by an average of 8% (range 1% to 25%) from those originally obtained. For three
interaction terms the status of the statistical significance of the coefficient changed; the
interaction between an application to a Group 2 (compared to a Group 3) university and
ethnicity became statistically significant (p=.01) whilst the interaction between a Group 1
application (compared to Group 3) and SES become non-significant (p=.1), as did the
interaction term between UKCAT score and ethnicity (p=.06). For the model relating to
unconditional offers the OR obtained differed by an average of 5% (range 0% to 10%)
from those originally obtained. For two predictor variables the status of the statistical
significance of the coefficient changed; SES became a predictor variable of only
borderline significance (p=.08) whilst EASL became statistically significant (p=.02).
For the data with missing advanced qualification attainment imputed the ORs
obtained for the probability of obtaining a conditional offer differed by an average of 12%
(range 0% to 45%) from those originally obtained. Those coefficients for age of applicant
and EASL status became statistically significant (p=.002 and p<.001 respectively).
Those coefficients relating to the group of university group applied to also changed in
significance (Group 1 became non-significant (p=.1) whilst Group 2 became significant
(p=.03). For three interaction terms the status of the statistical significance of the
coefficient also changed; the interaction between an application to a Group 1 (compared
to a Group 3) and SES became of borderline significance (p=.06) whilst the interaction
terms between UKCAT score and age of applicant became non-significant (p=.4) as did
the interaction term between UKCAT score and ethnicity (p=.7). The ORs obtained for
the probability of obtaining an unconditional offer differed by an average of 9% (range
1% to 26%) from those originally obtained. The coefficients for age of applicant (p=.2)
and SES became non-significant (p=.1) whilst EASL status became statistically
significant (p<.001). In addition the interaction term between an application to a Group 1
(compared to Group 3) university and UKCAT score became non-significant (p=.9).
Conclusions
Overall the results from the analysis of the imputed data sets differed relatively little from
those relating to the non-imputed data, with an average of around 5-10% variation in the
ORs recovered for the predictor variables. This implies that the missing values were
likely to be generally MAR. However, in a small minority of cases the coefficients (and
hence ORs) for independent variables and interaction terms did change in statistical
significance status and therefore it may that at least some of the missing data are nonignorable. This could be possibly related to a subgroup of applicants (e.g. mature
applicants) where missing values are may not be so closely related to observed
covariates and outcomes. Overall the results suggest we should be relatively confident
about our findings relating to offer. However, some caution may be required when
specifically considering the impact of the missing advanced qualification values on the
ability to model the probability of conditional offers.
References
1. Rubin DB. Inference and missing data. Biometrika 1976;63:581-92.
2. Verbeke G, Molenberghs G, Thijs H, Lesaffre E, Kenward MG. Sensitivity analysis for
non-random dropout: a local influence approach. Biometrics 2001;57:7-14.