Donna L. Coffman
Joint Prevention Methodology Seminar

The purpose of this talk is to illustrate how to
obtain propensity scores in multilevel data
and use these to strengthen causal inferences
about mediation.



Many prevention programs randomly
assigned at school level and hypothesized to
affect mediators at individual level.
Although intervention is randomly assigned,
the mediator is not.
Need to take into account confounders of the
mediator and outcome.
◦ e.g., poverty, individual’s prior delinquency


One solution is to use propensity scores,
which is based on potential outcomes
framework.
One assumption of this framework is that
there is no interference, such that one
individual’s assignment to treatment level
does not influence another individual’s
outcome.



No-interference is different from a nested
data structure.
Nested data does not necessarily mean that
we need propensity scores at both the school
and individual level.
Our motivating problem is what is sometimes
referred to as 2-1-1 or lower level mediation.

Three-level treatment: control, usual
intervention, intervention adapted for rural
◦ Cluster-randomized at the school level

Outcome- Alcohol use:
◦ Binary because highly skewed.
◦ 0 = no use and 1 = any use




Wave 2 refusal confidence about drinking
Wave 2 peer alcohol use norms
Wave 2 positive expectancies about drinking
Scores of 1 are good - indicate greater
refusal confidence, less positive expectancies,
and lower substance use norms.


Demographics: age, gender, race/ethnicity
Lifetime and wave 1 measures of recent
substance use (including alcohol)
◦ Also age at first use


Wave 1 measures of mediators
Wave 1 measures of delinquency
◦ Stealing, destroying property, school expulsion

Coping, anxiety, rebelliousness, family
relationships & rules, parental knowledge &
monitoring



Imputation model included a lot of variables
due to all the confounders in the propensity
score model.
Many of the variables were not normally
distributed so joint model (e.g., NORM) would
not work.
We used chained equations. Even then, this
step was computationally intensive and we
decided to impute only one data set for
illustration purposes.

Propensity score model for treatment
◦ Multinomial logistic regression with fixed effects for
school
◦ Differences are not that large initially but we
wanted to reduce variance.

Computed stabilized weights
◦ Predicted probability from intercept-only model
divided by predicted probability from model of
treatment regressed on all confounders.
◦ Helps to reduce variability of weights and extreme
weights.

Propensity score model for each mediator
◦ Logistic regression of each mediator on all
confounders and fixed effects for school and
treatment level

Computed weights
◦ Numerator of weights was predicted probability
from a logistic regression of each mediator on the
treatment level

Assess effect of intervention on mediators
◦ Logistic regression for each mediator
◦ Use only weights for intervention

Assess effect of intervention and mediators
on alcohol use.
◦ Logistic regression models for each mediator
◦ Using product of intervention and mediator
weights.




Intervention did not have statistically
significant effect on refusal confidence or
positive expectancies.
Mediators did have statistically significant
effect on alcohol use.
Intervention did not have a statistically
significant direct effect on alcohol use.
Intervention did not have a statistically
significant total effect on alcohol use.

Refusal Confidence
◦ Not statistically significant when either weighted or
unweighted

Peer Alcohol Use Norms
◦ Significant effect: For the usual DRS intervention there is
a 19% reduction in lower norms (-.214, odds ratio
=.807) when unweighted and a 20% reduction in lower
norms (-.228, odds ratio = .796) when weighted.
◦ p = .025 and .035 for unweighted and weighted

Positive Expectancies
◦ Not statistically significant when either weighted or
unweighted

Refusal Confidence

Peer Alcohol Use Norms

Positive Expectancies
◦ Statistically significant effect: -.363, p = .038
◦ When refusal confidence is high, alcohol use decreases –
there is a 30% reduction in alcohol use. Odds Ratio =
.696
◦ Statistically significant effect: -1.10, p < .001
◦ When norms are low, alcohol use decreases – there is a
67% reduction in alcohol use. Odds Ratio = .333
◦ Statistically significant effect: -1.07, p < .001
◦ When positive expectancies are low, alcohol use
decreases – there is a 66% reduction in alcohol use. Odd
Ratio = .343

Effect of treatment on alcohol use using
logistic regression without the mediators in
the model.
◦ No significant effect either with or without the
weights.


We recognize that the mediators may not be
independent. We performed an analysis that took
this into consideration but results were not
different.
We did not multiply coefficients together for the
effect of the intervention on the mediators and
the effect of mediators on alcohol use because
with binary mediators and outcomes, this value is
not interpretable as an indirect effect although
the indirect effect can be tested for significance
(Coffman & Zhong, 2012).


It is important to proceed with mediation
analysis even in the absence of a total effect.
It is important to look at the effects
separately.
◦ We learned that the mediators do indeed have
strong effects on alcohol use, even when
controlling for many confounders.
◦ But the intervention did not have statistically
significant effects on the mediators, perhaps due to
implementation issues.


Mediation analyses should consider that
individuals and/or groups are not randomly
assigned to levels of the mediator(s) and there
may be confounders of the mediator and
outcome even if the intervention is randomly
assigned.
This is an active area of research and future work
when examine under what conditions propensity
score models should include random effects,
when propensity scores are needed at each level
of nested data, and how to handle multiple
mediators simultaneously.




Bethany Bray
Wanghuan Chu
Michael Hecht and John Graham
NIH NIDA P50 DA10075