Signature Pedagogy
and CPED Convention Theme
• DPELFS’ Signature Pedagogy is curriculumbased embedded fieldwork that provides
engagement with and service to the local
community.
• One of the 2015 Convening Themes is
Community Connection and Partnership
Building
Section I:
Experimental Design for Data Collection
• True Experimental Design: Randomized Trials
to avoid biasness
• Quasi-Experimental Design: Matched samples
to establish equivalency between groups
• Q: How to match numerous covariates on
multiple dimensions?
Proactive Approach: Pursuing Group
Matching Prior to Data Collection
• “Groupwise matching: select control comparison
to be groupwise similar to treatment group, e.g.,
schools with similar demographics, geography,
etc.
Generally a good idea.
• Individual matching: select individuals from the
potential control pool that match treatment
individuals on one or more observed
characteristics. May not be a good idea.”
• It is too difficult to match cases on all relevant
variables for a fair comparison in nonequivalent
designs.
Method Overview (1)
• After Data Collection: When to use each
method?
Method Overview (2)
• Given two groups of students, we gather the
group id and covariates we wish to balance;
• Compute propensity score, i.e., the predicted
possibility for the group affiliation using
SPSS, SAS, or R;
• Select cases from these two groups based on
the propensity score match;
• Add outcome measures to examine the group
difference for the study.
Illustration in SPSS (Steps 1)
Illustration in SPSS (Steps 2)
The Latest Development in SPSS 22
The Latest Development in SPSS 22
Intellectual Merit
The propensity score approach
1. Can support apples-to-apples comparison
under field-based, non-randomized
conditions;
2. Provides a way to summarize covariate
information for balanced subject selections;
3. Can be used to adjust group settings via
research design (i.e., matching).
Quote from Christopher Baum (2013)
The key concern is that of similarity. How can we
find individuals who are similar on all observable
characteristics in order to match treated and nontreated individuals? … Propensity score matching
(PSM) allows this matching problem to be reduced
to a single dimension: that of the propensity score
… Thus, rather than matching on all values of the
variables, individual units can be compared on the
basis of their propensity scores alone.
Benefit of the Field-based
Data Analysis for Practitioners
This approach
. Tends to be more representative of realworld conditions than randomized trials
in true experimental designs.
. Can be used to adjust for the estimate of
the treatment effect when randomization
is not possible.
. Can effectively control extraneous
variables on multiple dimensions.
Match Indicator Construction:
ies.ed.gov/pdf/CommonGuidelines.pdf
Q&A
(A short break between the two sub-sections:
design for data collection and consideration for
data analysis)
Section II:
Consideration for Data Analysis
Integrity of Data Analysis under
• Missing Treatments
-- No way for imputation
• Missing Observations
-- No complication from Multiple Imputations
(1) Missing is missing
(2) Model for imputation?
(3) How many imputations to make it conclusive?
(4) How to aggregate non-additive outcome?
Illustration of Missing Data Impact on
SS Configuration in ANOVA: S(𝑋 − 𝑋)2
Private School
Attending
Class
Yes
No
mean
Attending
Class
mean
7
mean
5
7
9
8
8
Yes
No
Public School
4
6
6
5
Private School
Public School
mean
8
5
6.5
8
5
6.5
8
5
Four Options in Standard Software
Method Choice
“The question of which set of sums of squares is the
Right Thing provokes low-level holy wars on R-help
from time to time. ”
-- http://cran.r-project.org/doc/FAQ/R-FAQ.html#Whydoes-the-output-from-anova_0028_0029-depend-on-theorder-of-factors-in-the-model_003f
In balanced designs, so-called "Type I," "II," and "III"
sums of squares are identical.
-- Because most textbooks use examples from balanced
data, the issue of method choice does not surface in
most courses in intermediate statistics.
SPSS Example
data list free/ses gender score.
begin data.
1 1 9 1 1 4 1 1 11 2 1 8 2 1 12 2 1 13 3 1 18 3 1 17 3 1 15 1 2 2 1 2 6
1 2 4 2 2 9 2 2 10 2 2 17 3 2 6 3 2 8 3 2 4
end data.
Examine score impacts from gender and socioeconomic status (SES)
R Links Online
In unbalanced designs, Type I assumes examination of
sequential effects (e.g., “Quantifying population genetic
differentiation from Next-Generation Sequencing data”)
Type-II and Type-III tests are reasonably construed as
tests of main effects and interactions.
-- https://stat.ethz.ch/pipermail/r-help/2006-August/111913.html
Type III SS [default]: Compare the full model to the models
with each factor removed one at a time.
Type IV SS is used to handle missing treatments.
Type II SS: How does it handle “each
factor removed one at a time”?
Given factors A, B, and C, the removal occurs to
all the components that involve a particular
factor!
• To test the AB interaction, Type-II SS will drop
both AB and ABC interactions in the null
model;
• The test for the A main effect assumes that
the ABC, AB, and AC interaction are absent
-- https://stat.ethz.ch/pipermail/r-help/2006-August/111913.html
Conclusion
• For balanced data analyses, the results are
identical for all models
• For unbalanced dada with missing observations,
(1) Type I results are designed for testing factors
with sequential effects
(2) Types II or III SS are used for testing the effect
of each intervention (non-sequential)
• For missing treatments, Type IV SS can estimate
certain effects of researchers’ choice
Type IV Sum of Squares (SS)
• Column Contrasts
(2 pairs of cells for C2 and C3)
(3 pairs of cells for C1 vs. C2; C3 vs. C4)
Type IV SS: Estimable Functions
• Row Contrasts
(not unique)
References
• Analysis of Messy Data (three volumes) by
Milliken and Johnson
• http://mcfromnz.wordpress.com/2011/03/02/an
ova-type-iiiiii-ss-explained/ (No Type IV)
• http://www1.umn.edu/statsoft/doc/statnotes/st
at05.txt (SAS based discussion)
• http://statisticscahn.blogspot.com/2007/10/type-i-iv-sum-ofsquares.html (Re-revise Type IV for SPSS default?)
• Propensity Score Analysis: Fundamentals and
Developments (Pan & Bai, 2015/Guilford Press)