Preliminary Findings
Social and Character Development
in Elementary School:
The Effectiveness of the Making
Choices Program
Mark W. Fraser, PI, School of Social Work, UNC-Chapel Hill
Steven H. Day, School of Social Work
Shenyang Guo, School of Social Work
Alan Ellis, Sheps Center and School of Social Work
Roderick A. Rose, School of Social Work
Maeda J. Galinsky, School of Social Work
Kim Dadisman, Co-PI, Center for Developmental Science, UNC-CH
Dylan Robertson, Center for Developmental Science
Tom Farmer, School of Education, Pennsylvania State University
This presentation was given at the School of Social Work, University of Maryland, Baltimore, MD, on April 9, 2009.
Portions of this report were presented at the annual meeting of SACD Project grantees on June 13, 2008 in
Washington, DC
Agenda
• Theoretical Bases and programs
• Design and challenges
• Analytic strategies
• Analytic methods (skim – see slides)
• Findings
Acknowledgments
• This project was support by a cooperative agreement
(R305L030162) with the Institute of Education Sciences
at the U.S. Department of Education (US DOE).
Funding for the project was appropriated by the US DOE
and the Centers for Disease Control and Prevention.
• We thank Paul Rosenbaum (U Penn), Ben Hansen (U of
Michigan), and Matthias Schonlau (Rand Corp) for their
consultation on methodological issues related to this
presentation.
Teachers Talk about
Making Choices
• Changes in Classroom
Atmosphere
• Observable Differences
in Student Behaviors
• Measurable Academic
Achievement
Classroom Atmosphere
“I noticed that the
classroom
started working
more as one big
group instead of
individuals.”
Gr.5 Sandy Grove Elementary,
Hoke County
Observable Behaviors
“The students tend to
be less critical of
each other and
more
understanding of
each other’s
differences.”
Gr. 5 Sandy Grove Elementary,
Hoke County
Academic Achievement
“ The program uses excellent books to support
the goals of being a good friend and not hurting
others.… I use them during Language Arts time.”
Gr. 4 Tommy’s Road Elementary, Wayne County
Observable Behaviors …
“It provided
a way for
students to
put their
feelings into
words.”
I am
feeling
really mad!
Gr. 2, Bunn Elementary,
Franklin County
Academic Achievement
“My students spend more time on task. They
seem less distracted by annoying behavior.”
Gr. 5 Scurlock Elementary, Hoke County
Make the right
choice!
Children are actually stopping and thinking
about making the right choices, and I have
heard a lot of children say to themselves,
“Make the right choice.”
It is great to hear.
Kdg. Bunn Elementary, Franklin County
“It made a difference with teaching children how to
deal with their feelings using better methods rather
than having tantrums or hitting.”
Kdg. Bunn Elementary, Franklin County
Oh, boy! I need
that Making
Choices
program.
Classroom Atmosphere
“This program
provided a
foundation on
which we could
build a
classroom
community.”
Gr. 1 North Drive,
Wayne County
Children in my school need…
…social skills to make friends and deal with
interpersonal problems.
…lessons that teach children respect toward others
and responsibility for their own actions.
…a program designed to reduce disruptive
behavior and promote academic achievement.
Making Choices
Research Question:
Does social and character
education work?
(i.e., is Making Choices effective?)
Intervention Research Perspective:
The Design and Development Approach
1. Specify the problem and develop a
program theory
2. Create and revise program materials
3. Refine and confirm program components
(sequential experimentation perspective)
4. Assess effectiveness in a variety of
practice circumstances and settings
5. Disseminate findings and program
materials
Source: Fraser, M. W., Richman, J. M., Galinsky, M. J., & Day, S. H. (2009). Intervention
research: Developing social programs. New York, NY: Oxford University Press.
Social and Character Development in Childhood:
A Risk and Resilience Orientation
PROBLEM THEORY:
Perspectives on Conduct Problems and Academic
Achievement in Elementary School
•Developmental risk perspective
•Ecological theory
•Social information processing theory
Eco-Developmental Risk Cascade
POINT: Risk factors for poor developmental outcomes vary over time. Lacking effective intervention,
the potential for poor outcomes increases – and cascades – as function of complex bio-social
processes. To promote positive outcomes, we must disrupt malleable risk mechanisms.
Biological Risks
Parenting
Family-School
Pre-School
Climate
Neighborhood
School Readiness
Processing Skills
Parenting
Family-School
School Climate
Neighborhood
Increasingly Broad Repertoire
of Potentially Damaging and
Aggressive Behaviors
Peer Rejection
Academic Failure
Parenting
Family-School
School Climate
Neighborhood
Cognitive Mediation Model
(in Developmental Sciences)
Biological
Predisposition
Biological
Predisposition
Parenting
•Monitoring
•Bonding
Peers
•Deviancy training
•Contagion effect
•False consensus
effect
Sociocultural
Context
•Stress/poverty
•Racism
•Street codes
•Acute/chronic stress
Mental
Processes
•Social knowledge
•Scripts
•Schema/skills
Sociocultural
Context
•Stress/poverty
•Racism
•Street codes
•Acute/chronic stress
Conduct
Problems
•Conduct disorder
•Fighting
•Drug use
Adapted from: Dodge, K. A., &
Pettit, G. S. (2003, p. 351). A
biopsychosocial model of the
development of chronic
conduct problems in
adolescence. Developmental
Psychology, 39(2), 349-371.
Social Information Processing Theory:
SIP Skills and Emotional Regulation as Malleable Mediators?
Interpret
social cues
Encode
social cues
Assess
outcomes
State the
problem
Arousal, Emotions,
Social Knowledge
Select &
enact the
best solution(s)
Set
goal(s)
Generate
potential
solutions
Evaluate
potential
solutions
Social Knowledge: Life experiences producing scripts, schemata, skills, and beliefs
PROGRAM THEORY
(specifies how a program is to work)
Intervention
Program Structure
Such as:
Targeting Unit:
Classroom
Entire school
Other (after school,
family)
Curriculum Structure:
Distinct activities
Embedded in curriculum
Activities to address
SACD Goals
Such as:
Character education
Violence prevention/peace
promotion
Social and emotional
development
Tolerance and diversity
Risk prevention and health
promotion
Behavior management
Social and Character Development:
Prevention Model
Behavior
Social Emotional
Competence
(mediator)
Attitudes about aggression
Self-efficacy
Empathy
School Climate
(mediator)
School connectedness
Victimization
Feelings of safety at school
Parent involvement
Positive Behavior
Responsible behavior
Prosocial behavior
Self-regulation
Cooperation
Negative behavior
Aggression
Minor delinquency
Disruptive classroom behavior
Academics
Academic competence
School engagement
Grades
Standardized test scores
Moderating Factors
Child
Gender
Socioeconomic status
Race/ethnicity
Risk status
Prior test scores/grades
Family
Parenting practices
Home atmosphere
Community
Community risk factors
Social capital
Program
Fidelity
Intensity and dosage
School
Activities to promote social and character development
Organizational structure
Social Development Model Perspective
Engaged school behavior
...
Focusing on work
Paying attention
Following instructions
Instruction in social & emotional
skills . . .
Empathy
Anger management
Problem solving & impulse
control
School success
Reductions in . . .
Classroom disruptive
behavior
Anxiety
Anger
Opportunity to . . .
Discuss and identify feelings
Acquire language and
communication skills
Practice solving problems
Observe models
Social & emotional skills . . .
Empathy
Anger management
Problem solving &
impulse control
Reductions in . . .
Problem behavior
Aggression
Peer rejection
Reinforcement and
generalization of learning
through . . .
Naturally occurring
opportunities in school
Home discussion of materials
Intervention
Immediate outcomes
Knock-on outcomes
Outcomes
PROGRAMS
The Competence Support Program*
Social
Dynamics
Training
Social Skills
Training
for students
for teachers
Group randomization:
Cohort 1: Hoke and
Wayne Counties (10
schools randomized to
5 intervention; 5
control)
Cohort 2: Franklin
County (4 schools
randomized to 2
intervention; 2 control)
Classroom
Behavior
Management Training
and Consultation
for teachers
*Developed by the program investigators, the intervention simultaneously focuses on the
characteristics of children and on the classrooms in which they learn. The intervention
combines three components.
Program Elements
• Making Choices: Skills Training curriculum for students in
elementary school. In-service training introduced teachers to
the risks of peer rejection and social isolation, including poor
academic outcomes and conduct problems. Throughout the
school year, teachers received consultation and support (2
times per month) in providing lessons designed to enhance
children’s social information processing and other skills. As a
part of the Standard Course of Study, the program was
integrated into routine class instruction.
• Classroom Behavior Management provided teacher
consultation on classroom management strategies designed
to strengthen engagement in instructional activities.
• Social Dynamics Training provided teacher consultation on
classroom contexts, social groupings, and interactional
patterns that can be used to reinforce academic achievement
and prosocial behavior.
Theory of Change: Making Choices
Random Assignment
Core
#1
Training the
Teacher or
Counselor
Core
#2
Application
of Making
Choices by
Teacher or
Counselor
Characteristics
of the Teacher
or Counselor
Treatment as
Usual Control
Condition
Core
#3
SIP skills of
the Children
in the
School
Characteristics
of the Children
and the
Classroom
Core
#4
Impact on
Social
Engagement and
Peer
Rejection
Core
#5
Impact on
Disruptive
Behavior
and
Academics
•Assess
implementation
of
•Test the
degree to which
training
the intervention is
•Assess
if teacher
acquires
delivered
as intended,
skills from training/supervision
e.g., specific activities
Note. In a randomized
trial, you must figure out
a way to measure each
of the core elements.
Make Program Manuals
• From risk mechanisms, mediators, and
logic models to the design of a program
• Specifying program activities that target
the malleable mediators and have cultural
congruence
• Example: Making Choices
For a discussion of issues in the development and use of treatment manuals, see: Galinsky,
M. J., Terzian, M. A., & Fraser, M. W. (2006). The art of group work practice with manualized
curricula. Social Work with Groups, 29(1), 11-26.
Warning: It is easy to under
estimate the difficulty of
developing a program
manual.
“That Sunk Feeling”
If you start in the wrong
place, it usually does
not help to dig deeper!
Source: Don Moyer, Harvard Business Review (October, 2004, p. 160)
How to begin in the right place…
Start with theory and
research, plus practice
experience…
Develop a template for each
lesson or session
Grade 2
Recognizing Your Feelings
Objectives:
 The learner will recognize that certain situations bring out feelings in all of us.
Lesson 2
 The learner will practice recognizing their own feelings.
 The learner will use personal experiences and knowledge to interpret written
and oral messages. (SCS- LA 3.01)
Overview
 The learner will write structured, informative presentations and narratives
when given help with organization. (SCS- LA 4.08)
Materials:
Prep
Materials
Penguin Facts page, Response Sheets, Write About It worksheets A and B
Introduction
Review the idea that we all experience a variety of emotions and responses
to emotions. Even when we experience the exact same situation, we may
have different responses to the situation. Our responses to our feelings can
cause us to do good things, but at times they can also cause us to do things
that are not helpful.
Activity 1
Answers
Activity I: Pete the Penguin
Using two columns, list on the board the emotions presented in Lesson 1 of
the book, The Way I Feel.
Column I- Emotions that Feel Good: happy, silly, excited, proud, or
thankful
Column II- Emotions that Don’t Feel Good: scared, sad, disappointed, bored,
angry, or jealous
Process Tip
Introduce the students to Pete the Penguin using the penguin puppet. Pass out the
Penguin Facts page and discuss the factual information about penguins. Explain to
the students that Pete has experienced events that have brought out many different
emotions. Sometimes his emotions feel good, but at other times they don’t feel very
good at all.
Review the emotions listed in the columns on the board. Then give each student four
small pieces of paper (about the size of a note card). Read aloud the following
events involving Pete the Penguin. After reading each event, ask the students, “How
would you feel?” Give the students enough time to record their responses on one of
Standard
Course of
Study
Review
Prop
their blank pieces of paper. They can use the emotions on the board to express how
they would feel or they may provide their own responses.
Avoid
labeling
Scenarios
Activity 2:
Write
About It!
After you read each situation, collect a few responses randomly and read them aloud
(so as not to bring attention to specific student responses). As you read through
each response, discuss whether the event brought out a good feeling or a not-sogood feeling. The texts are ambiguous so that students can develop their own
interpretations—not all students will feel the same way about each situation. Discuss
the idea that everyone heard the same event, yet the feelings were different in many
instances.

Today Pete walked in the classroom. As he walked to his desk, Pete noticed
Susan and Tony talking quietly and laughing. They both looked up at Pete and
giggled. If you were Pete, how would you feel?

When Pete was on the playground, he saw a group of students playing ball.
He went to join them, and they told him he could play as soon as they started
the next game. If you were Pete, how would you feel?

At lunch, Pete was sitting next to Jermaine. Jermaine opened his lunch and
Pete looked inside. All he saw was two cookies and a drink box. If you were
Pete, how would you feel?

Pete’s teacher told him he could play a game with Juan as soon as he finished
his writing assignment. If you were Pete, how would you feel?
After discussing the above events, ask the students how they recognize when they
are feeling certain emotions. “What happens when you start to feel angry?” “Happy?”
“Frustrated?” and so on. (Example response: When you are getting angry- you might
get hot, start to shake, get tense, grit your teeth, etc.)
Leave the list of emotions on the board to use in Activity II.
Activity II: Write About It
Give the students the Write About It page. On the top of the sheet, have
students write about an event in their life that caused them to experience an
emotion that made them feel good. On the bottom of the sheet they can write
about an experience that caused an emotion that didn’t feel good. Each
narrative should describe the emotion, what caused it, and how they responded to
the emotion. Students can refer to the columns on the board to choose the emotions
they want to write about. Share the following examples aloud or on a transparency:
Example 1:
Once I felt excited when I was going to my friends party.
I knew I felt this way because I was smiling and jumping around.
Develop all worksheets and
artwork
Activity II: Sheet A
NAME: _________________
A good feeling:
Once I felt ___________________ when
____________________________________
____________________________________
____________________________________
____________________________________
I knew I felt this way because__________
____________________________________
____________________________________
Activity II: Sheet B
NAME: ____________________
A not so good feeling:
Once I felt _____________________ when
___________________________________
___________________________________
___________________________________
___________________________________
I knew I felt this way because__________
___________________________________
___________________________________
“Pete the Penguin” Poster for Grade 2
Sample Lesson
Activities
from
Making Choices
Gr. 3 Lesson - Intentions
SYMBOLS
Grrrrr!
MEAN
FRIENDLY
CAN’T TELL
Intentions: Mean or Friendly?
LAUGHS
WITH
YOU
FRIENDLY
HITS
YOU
SHARES
WITH
YOU
MAKES
A FACE
AT YOU
HUGS
YOU
IGNORES
YOU
FRIENDLY
CAN’T TELL
Grrrrr!
TALKS
ABOUT
YOU
BITES
YOU
HELPS
YOU
FRIENDLY
MEAN
Grade 3 Lesson 10
GOAL SETTING
GOAL: Something a person wants or
something a person wants to see happen.
RELATIONSHIP GOAL: Goals that involve
wanting to get along with another person.
Grade 4 Lesson 6
Are these Relationship Goals?
(thumbs up or thumbs down)
•
•
•
•
•
•
•
•
•
I want to make an “A” on my math test.
I want to play more often with my friend.
I want a new video game for my birthday.
I want to eat out at a restaurant for dinner.
I want to become friends with the new student.
I want to join in the basketball game at recess.
I want to sit with Jose on the bus.
I want to be in the class play this fall.
I want to stop getting upset when friends ignore me.
GOAL SETTING
• Set a relationship goal for these situations:
I was playing basketball at recess with some
friends. Terrell, who is not very good at
basketball, asked if he could play with us.
Set a Relationship Goal
Denise just made
me really upset. She
tried to pick a fight
with me by saying
things that are not
true. I am feeling
angry with her right
now.
Set a Relationship Goal
Yesterday, my mom gave me a really cool pen that writes
in all different colors. When I brought it to school this
morning, Stacey asked me if she could borrow it. Last time
I let Stacey borrow something she lost it, but if I say no she
might get angry with me.
EVALUATION DESIGN:
Cluster Randomized Trial
with Ten Schools Randomly Assigned to
Treatment (j=5) and Control (j=5) Conditions
Cohort Design: Intervention provided in grades 3, 4, and 5
Prior Studies
1. Single-group qualitative trial of MC intervention (8th grade girls)
2. Two-group cluster randomized trial at classroom level in one middle school (6th
grade only)
3. Two-group cluster randomized trial at classroom level in one school (3rd grade)
4. Two-group, MC+SF intervention randomized trial (11 sites, 3rd – 4th grade)
5. Cohort sequential study by classroom in two schools (3rd grade)
6. (Current) Two-group cluster randomized trail at 14 elementary schools
SAMPLE
Two Overlapping Samples
Grade 3
n=571
Grade 4
n=557
Grade 3-4 Sample
• 3rd and 4th graders
• 10 schools
• Any consented students on a 3rd or
4th grade roster
• Change=entrants-leavers
Grade 5
n=433
n=414
n=370
Grade 3-4-5 Sample
• 3rd, 4th, and 5th graders
• 9 schools*
• Only consented students on a 5th grade roster
• Change=addition of entrants
* One treatment school was reorganized into a different building and dropped the program between
4th and 5th grade; students from that school were excluded from the 3-4-5 sample.
Equivalence of Intervention and Control Groups on Selected Child, Family, and
School Attributes: Grade 3 Cohort 1
N
Total
Control
Intervention
p-value
Male child, %
618
47.7
50.0
45.3
.243
Age of child, years
618
7.92
7.94
7.89
.235
Black child, %
White child, %
618
618
34.5
48.1
21.9
59.3
48.0
35.8
.001
.001
Hispanic child, %
618
10.0
13.7
6.1
.001
564
563
550
13.5
48.5
26.0
15.0
43.7
24.6
11.7
54.0
27.6
.259
.014
.416
School (aggregate, school level)
Black, %*
10
41.3
28.4
51.4
.001
Free Lunch, %*
Adequate Yearly Progress*#
10
10
44.7
81.9
37.4
84.7
50.7
79.0
.001
.068
Pupil/teacher ratio, mean*
10
16.3
16.2
16.4
.825
Child (analysis sample)
Family (analysis sample)
Primary caregiver, not a HS graduate , %+
Two biological parents not in household, %+
Income-to-needs < = 1, %+
Note. + MPR data from baseline child level data file. * NCES school level data (CCD 2003-2004)
across all schools. # AYP Performance Composite score, Year 1 of the SACD study.
Difference in School-Level Academic
Performance: Percentage at Grade Level
Cohort 1 AYP Performance Composite
Percent of students achieving at grade level
90
85
80
75
Treatment
Control
70
65
60
55
50
02-03
03-04
04-05
05-06
06-07
Test results for 2005-06 and 2006-07 are based on a revised accountability model and are not comparable to those from previous years.
Equivalence of Intervention and Control Groups on Selected Site Specific Outcomes
Grade 3 Sample (Pretest): Cohort 1
N
Total
Control
Intervention p-value
SIP Skill Level Assessment - Child
Encoding
420
44.2
43.0
45.6
.149
Goal formulation
Response decision making
412
410
67.6
66.0
65.5
64.1
70.1
68.3
.074
.225
Social contact
Cognitive concentration
Social competence
549
549
549
3.8
3.2
3.3
3.7
3.2
3.3
3.8
3.2
3.3
.121
.952
.945
Social Aggression
549
4.1
4.1
4.0
.183
Aggression
548
2.5
2.4
2.6
.095
Academic competence
Popularity
548
548
5.1
4.9
5.1
4.9
5.1
4.8
.948
.798
Aggression
502
38.7
42.6
34.4
.152
Prosocial skills
502
82.8
84.8
80.6
.552
Carolina Child Checklist - Teacher
Interpersonal Competence Scale Teacher
Peer Interpersonal Assessment
Sample sizes vary because pretest measures were collected from different respondents (teachers, students) at different times. SLA and Peer assessment
pretest were collected from students at the end of 2nd grade. CCC and ICST were collected from teachers at the beginning of 3rd grade. SLA=Skill Level
Assessment (SIP skill – HOME Scale adaptation by Dodge, 1980). CCC=Carolina Child Checklist (Macgowan et al. 2002 – Research on Social Work
Practice). ICST-Interpersonal Competency Scale – Teacher (Xie et al., 2002, Social Development)
Teacher and Classroom Characteristics by Intervention Status
Characteristic
Demographics
White %+
Greater than a bachelor degree %+
Years of teaching, mean+
Years teaching at current school,
mean+
Has regular/advanced teaching
certificate, %+
SACD classroom activities
N of SACD classroom strategies,
mean+
Violence prevention hours, mean+
Social and emotional development
hours, mean+
Classroom observations
Number of feedback and structure
exemplars in place, mean#
Intervention
Teachers
(n = 21)
Non
Intervention
Teachers
(n = 23)
Difference
p-value
73.7
28.6
11.6
95.5
4.6
14.1
-11.8
24.0
-2.5
.051
.033
.385
4.1
8.0
-3.9
.040
85.0
90.9
-5.9
.566
14.8
7.8
13.7
2.7
1.1
5.1
.267
.039
10.9
2.9
8.0
.002
6.2
4.2
2.0
.019
Note. +MPR baseline data (spring). #Observations were conducted with the Classroom Observation Form (COF) by an
intervention specialist blind to the treatment or control condition of each school. The COF assesses seven domains relevant to
the intervention: daily routines, time and task management, consequences and follow-through, teaching alternative behaviors,
communication and feedback, and group processes and peer support.
Causation in Research Design:
Randomization Is Supposed to Produce the Counterfactual
Note. We let the control group serve as evidence for what would have
happened counter to the fact of participation in intervention (the stat class).
Randomization is supposed to create equivalence or balance between the
intervention (taking the stat class) and control (not taking the stat class)
groups. But it didn’t. On several observed and an unknown number of
unobserved measures, the intervention and control group schools differ.
Four evaluation challenges
 Selection Bias:


Covariates are not balanced
between treated and control groups
Missing Data: No baseline data on enterers and
lost data on leavers = constant churning of sample
Rater Effects: Outcome ratings were made by the
same teachers within grades, but different
teachers over grades 3, 4, and 5
•
Piecewise analyses – change scores within grade level
 Treatment Contamination/History:
High
intervention content in control schools*
Note. Student Citizen Act (SL 2001-363) was passed into law by the Legislature in 2001. The Act required local boards of
education to develop and incorporate character education instruction into standard curricula. Local boards of education began
implementation in the 2002-2003 school year.
MEASURES
Site-Specific Outcomes
• Skill Level Assessment Activity (SLA): Based on the Dodge
Home Scale (1980), the SLA uses students’ responses to questions about
hypothetical social situations. After viewing picture scenarios, students
answer questions measuring different aspects of social information
processing skill: encoding (α=.78), goal formulation (α=.76), and response
decision making (α=.80).
• Carolina Child Checklist (CCC): The CCC is a 35 item teacher
questionnaire that yields factor scores on children’s behavior including
social contact (α=.90), cognitive concentration (α=.97), social competence
(α=.90), and social aggression (α=.91).
• Interpersonal Competence Scale-Teacher (ICST): The ICST
is an 18-item teacher questionnaire that yields factor scores on children’s
behavior including aggression (α=.84), academic competence (α=.74), and
popularity (α=.78).
• Peer interpersonal assessments: Peer interpersonal
assessments were used to examine classmates’ perceptions of
participants’ social and behavioral characteristics including aggression
(α=.92), prosocial skills (α=.84 ), and internalizing behavior (α=.67 ).
Summary of Data Collection Occasions
Time of year
Cohort 1 (C1)
Cohort 2 (C2)
Grade 2
Wave 1
March
2003-04
2004-05
Wave 2
Sept
2004-05
2005-06
Grade 3
Wave 3
March
2004-05
2005-06
Wave 4
April
2004-05
2005-06
Wave 5
Sept
2005-06
2006-07
Grade 4
Wave 6
March
2005-06
2006-07
Wave 7
April
2005-06
2006-07
Grade 5
Wave 8
Wave 9
Sept
April
2006-07 2006-07
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C1
C1
C2
C1
C2
C1
C2
C1
C1
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C1
C1
C1
C1
C1
Instrument (in-house data)
Making Choices
Student report (SLA)
Friends and Groups
Student report (Social Cognitive
Maps)
Peer groups
Teacher report (PG)
Carolina Child Checklist
Teacher report (CCC)
Interpersonal Competence Scale
Teacher report (ICST)
Peer Interpersonal Assessments
Student report (PNOMS)
Ratings of school adjustment
Teacher report (TASS)
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
MPR data (across-site data)
Student reported
Teacher reported
Parent reported
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C1
C1
C1
C2
C1
C2
C1
C2
C1
C1
C1
Minutes of Skills Training Instruction in 3rd and 4th
Grades by Student
Benchmark=1,140 minutes
Below benchmark: 19%
Above benchmark: 81%
(overall n=571)
ANALYTIC
PROCEDURES
Multiple Imputation of Missing Data
(The imputation models employed both
predictor and outcome variables, but the
analysis models employed imputed
missing values for predictor variables
only). 50 imputations for each outcome
variable
Estimation of propensity scores
using Generalized Boosted
Modeling (gbm) -- aims to
optimize balance on observed
covariates between treated and
control groups
Heckman sample selection
Model (Predictors of the
selection equation are similar
to the input of gbm)
Dose (efficacy subset)
analysis) using Abadie et al.
Matching estimator
Analytic Procedures:
Flow Chart for Use
of “Bias-Correcting”
Statistical Methods
Optimal pair matching
using propensity scores
estimated by gbm
Piecewise change score
HLM analysis using
propensity score weighting
(propensity scores
estimated by gbm)
Post-pair-matching
with regression
adjustment
Optimal full matching
using propensity scores
estimated by gbm
Post-full-matching
with HodgesLehmann aligned
rank test
Procedures for multiple imputation
of missing data





Test for MCAR (Little, 1988) confirms models are not
MCAR.
Assumption of MCAR not required if imputation model is
informed (i.e., data may be missing at random)
A diagnostic stage identified models that resulted in 99%
relative efficiency for all analysis variables.
50 simulations (copies of the raw data set) generated
using MI.
DVs and predictors both used in imputation; imputed DVs
discarded after imputation (MID procedure; von Hippel,
2007).
Missing Data Diagnostics: Proportion without Missing Data
and Proportion of Missing Data Points
General Procedure for Propensity Score Analysis
Step 1: Logistic regression
 Dependent variable: log
odds of receiving treatment
Search an appropriate set of
conditioning variables
(boosted regression, etc.)
 Estimated propensity scores:
predicted probability (p) or
log[(1-p)/p].
Step 2:Analysis using propensity
scores
Analysis of weighted mean
differences using kernel or local
linear regression (difference-indifferences model of Heckman
et al.)
Step 2: Analysis using propensity
scores:
Multivariate analysis using
propensity scores as weights
Step 2: Matching
Greedy match (nearest neighbor
with or without calipers)
 Mahalanobis with or without
propensity scores
 Optimal match (pair matching,
matching with a variable number
of controls, full matching)
Step 3: Post-matching
analysis
 Multivariate analysis
based on matched
sample
Step 3: Post-matching
analysis
 Stratification
(subclassification)
based on matched
sample
Estimating propensity scores

Need relevant conditioning variables

Obtain “best” logistic regression (i.e., best
functional forms); however, no way to know

Used Multiple Additive Regression Trees (MART)
to run logistic regression. Rand Generalized
Boosted Modeling (gbm): Aims for best balance on
observed covariates between treated and
controlled groups. Iteration stops when the sample
average standardized absolute mean difference
(ASAM) is minimized.
Example of gbm output: Does gbm reduce the
difference between treated and control schools?
Point: After using gbm propensity score weights, all pretreatment differences are ns.
STR=treatment group; LTR=control group; ASAM= average standardized absolute mean difference
between treatment and control cases; pretreatment covariates: red solid diamonds= p-values before use
of gbm weights (if below line then significant); black diamond outline = p-values after weights applied
Predictors of the propensity score model
________________________________________________________________________________________________________________________
Outcome
___________________________________________________________________________
ICSTAGG ICSTACA
ICSTINT
CCCSCOM CCCPROS CCCEREG CCCRAGG
Predictor
Academ ic
Social
Aggression Com petence Internalizing Com petence
Prosocial
Em otion
Regulation
Relational
Aggression















White

Hispanic

Prim ary caregiver's education (years of schooling)

Ratio of incom e to poverty threshold

Prim ary caregiver full-tim e em ploym ent (part-tim e is referrence)

Father's presence in fam ily: Yes (absence is reference)





















































Age at baseline (year)
Gender fem ale (m ale is reference)
Race (Other is reference)
African Am erican
ICST-aggrestion at baseline
ICST-academ ic com petence at baseline







CCC-social com petence at baseline


CCC-prosocial at baseline



CCC-relational aggression at baseline


________________________________________________________________________________________________________________________
ICST-internalizing at baseline
CCC-em ontion regulation at baseline
Note. Predictors vary by outcome variable. Following the convention of propensity score analysis,
we did not include predictors that are highly correlated with the outcome variable.
Propensity score weighting

When estimating the treatment effect, can use propensity
scores as sampling weights.
(Hirano & Imbens, 2001; McCaffrey et al., 2004; Rosenbaum, 1987)

Suppose p is the propensity score of receiving treatment.
Then:
 Average treatment effect for the treated (ATT):



control weight = p/(1-p)
treatment weight = 1
Average treatment effect for the population (ATE):


control weight = 1/(1-p)
treatment weight = 1/p
Post-optimal-matching analysis


For the matched sample created by optimal pair
matching, regress pairwise differences in Y
between treated and control cases on pairwise
differences in X vector between treated and
control cases (Rubin, 1979). In doing so, use the
intercept of the regression to estimate the
treatment effect and its p-value as a significance
test.
For matched sample created by optimal full
matching or optimal variable matching, use the
signed-rank test of Hodges and Lehmann (1962)
to estimate the average treatment effects.
Dose analysis using Matching
estimator
• The dose analysis evaluates the outcome difference
between a dose group (i.e., low, benchmark, or high) and
a comparison group using Matching estimator developed
by Abadie et al. (2004).
• Under the exogeneity assumption, this method imputes
the missing potential outcome by using average outcomes
for individuals with “similar” values on observed covariates.
• The estimator uses the vector norm (i.e., ||x||v=(x’Vx)1/2
with positive definite matrix V) to calculate distances
between one treated case and each of the matched
multiple nontreated cases, and chooses the outcome of
the nontreated case whose distance is the shortest among
all as the predicted outcome for the treated case.
Comparing model features
_________________________________________________________________________________________________
Model
The Model Controls for:
____________________________________
Level at
which
treatment
was tested
Multiple
imputation
of missing Rater's
data
effect
Selection
bias
Clustering
____________________________________________________
________ ________ ________ ________ ________
Piecewise change with propensity score weighting (ATE) School
Yes
Yes
Yes
Yes
Piecewise change with propensity score weighting (ATT) School
Yes
Yes
Yes
Yes
Optimal pair matching with regression adjustment
School
Yes
Yes
Yes
Yes
Optimal full matching with Hodges-Lehmann test
Student
Yes
Yes
Yes
No
Efficacy subset analysis using Matching estimator
Student
Yes
Yes
Yes
No
_________________________________________________________________________________________________
Note. Regression models include covariates age at baseline, female, black, white, latino, primary caregiver education,
income-to-poverty ratio, primary caregiver fulltime employment, father in household, and midyear change in teacher.
FINDINGS
Findings: Treatment effects measured
by changes in the 3rd Grade (g34)
Outcome Variable
Piecewise
Change
with
Propensity
Score
(gbm)
Weighting
Model 1:
Hypothetical
ATE
Sign
Grade 3
Piecewise
Change
with
Propensity
Score
(gbm)
Weighting
Model 2:
ATT
Grade 3
Change
Score Using
Change
Optimal FullScore Using
Matching
Optimal Pair- (gbm) and
Matching
Hodges(gbm) and
Lahmann
Regression
Aligned
Adjustment
Rank Test
Grade 3
Grade 3
Approximate Sample Size Used in Analysis
≈571
≈571
≈542
≈571
ICSTAGG - Aggression
-.10
-.04
-.10
-.01
ICSTACA - Academic competence
+
.12+
.08
.11
-.08***
ICSTINT - Internalizing
.14+
.13+
.14
.17
CCCSCOM - Social competence
+
-.22
-.25+
-.25
-.25
CCCPROS - Prosocial
+
-.25 +
-.26 +
-.25
-.19+
CCCEREG - Emotion regulation
+
-.16
-.18
-.20
-.24
CCCRAGG - Relational aggression
.17
.21*
.18
.28
_________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings: Treatment effects measured
by changes in the 4th Grade (g34)
Outcome Variable
Piecewise Piecewise
Change
Change
Change
with
with
Score Using
Propensity Propensity Optimal
Score
Score
Pair(gbm)
(gbm)
Matching
Weighting Weighting (gbm) and
Model 1: Model 2: Regression
Hypothetical
ATE
ATT
Adjustment
Sign
Grade 4
Grade 4
Grade 4
Change
Score Using
Optimal
FullMatching
(gbm) and
HodgesLahmann
Aligned
Rank Test
Grade 4
Approximate Sample Size Used in Analysis
≈557
≈557
≈550
≈557
ICSTAGG - Aggression
-.13
-.14
-.14
-.17
ICSTACA - Academic competence
+
-.13+
-.10
-.11
-.08
ICSTINT - Internalizing
-.00
.04
-.02
.18
CCCSCOM - Social competence
+
-.01
-.02
.06
.05
CCCPROS - Prosocial
+
-.00
-.03
.05
.07+
CCCEREG - Emotion regulation
+
-.00
-.01
.06
.03
CCCRAGG - Relational aggression
-.12
-.12
-.13
-.06
______________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings: Treatment effects measured
by changes in the 3rd Grade (g345)
Outcome Variable
Piecewise
Change
with
Propensity
Score
(gbm)
Weighting
Model 1:
Hypothetical
ATE
Sign
Grade 3
Piecewise Change
Change
Change
Score
Score Using
with
Using
Optimal FullPropensity Optimal
Matching
Score
Pair(gbm) and
(gbm)
Matching
HodgesWeighting (gbm) and
Lahmann
Model 2: Regression
Aligned
ATT
Adjustment Rank Test
Grade 3
Grade 3
Grade 3
Approximate Sample Size Used in Analysis
≈370
≈370
≈314
≈370
ICSTAGG - Aggression
-.15
-.12
-.12
-.08
ICSTACA - Academic competence
+
.15
.12
.03
.28
ICSTINT - Internalizing
.09
.09
.13
.19+
CCCSCOM - Social competence
+
-.02
-.03
-.04
-.03
CCCPROS - Prosocial
+
-.03
-.05
-.04
-.01
CCCEREG - Emotion regulation
+
-.01
-.01
-.04
.03
CCCRAGG - Relational aggression
.09
.09
.07
.07
_______________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings: Treatment effects measured
by changes in the 4th Grade (g345)
Outcome Variable
Piecewise
Change
with
Propensity
Score
(gbm)
Weighting
Model 1:
Hypothetical
ATE
Sign
Grade 4
Piecewise
Change
with
Propensity
Score
(gbm)
Weighting
Model 2:
ATT
Grade 4
Change
Score Using
Optimal
Change
FullScore Using Matching
Optimal Pair- (gbm) and
Matching
Hodges(gbm) and
Lahmann
Regression
Aligned
Adjustment Rank Test
Grade 4
Grade 4
Approximate Sample Size Used in Analysis
≈414
≈414
≈380
≈414
ICSTAGG - Aggression
-.17
-.21
-.18
-.21+
ICSTACA - Academic competence
+
-.08
-.08
-.06
-.13
ICSTINT - Internalizing
-.07
-.03
-.02
.13
CCCSCOM - Social competence
+
.15+
.18*
.20
.13
CCCPROS - Prosocial
+
.15 +
.14 +
.17
.11+
CCCEREG - Emotion regulation
+
.17+
.20+
.22
.21+
CCCRAGG - Relational aggression
-.15
-.18
-.18
-.15+
_________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings: Treatment effects measured
by changes in the 5th Grade (g345)
Outcome Variable
Piecewise
Change
with
Propensity
Score
(gbm)
Weighting
Model 1:
Hypothetical
ATE
Sign
Grade 5
Piecewise
Change
with
Propensity
Score
(gbm)
Weighting
Model 2:
ATT
Grade 5
Change
Score Using
Optimal
Change
FullScore Using Matching
Optimal Pair- (gbm) and
Matching
Hodges(gbm) and
Lahmann
Regression
Aligned
Adjustment Rank Test
Grade 5
Grade 5
Approximate Sample Size Used in Analysis
≈433
≈433
≈350
≈433
ICSTAGG - Aggression
-.08
-.08
-.12
-.01
ICSTACA - Academic competence
+
.20*
.20*
.16
.16
ICSTINT - Internalizing
-.17+
-.18+
-.17
-.24+
CCCSCOM - Social competence
+
.27**
.25**
.29
.28*
CCCPROS - Prosocial
+
.29***
.27***
.29
.28*
CCCEREG - Emotion regulation
+
.24*
.22*
.27
.27+
CCCRAGG - Relational aggression
-.20
-.22
-.24
-.24+
_______________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings: Growth curve and dose
models (g345)
Low Confidence:
Do Not Cite
Growth
Growth
curve
curve
Dose (cum. Dose (cum. Dose (cum.
(intervntn* (intervntn* minutes)
minutes)
minutes)
Hypothetical months)
months)
Grade 3
Grade 4
Grade 5
Outcome Variable
Sign
ATE
ATT
ATE
ATE
ATE
(9 months) (9 months) (8 hours)
(8 hours)
(8 hours)
Approximate Sample Size Used in Analysis
≈472
≈472
≈370
≈414
≈433
ICSTAGG - Aggression
-.01
-.02
-.00
-.09
.12
ICSTACA - Academic competence
+
.03
.03+
.03+
.08
-.03
ICSTINT - Internalizing
-.13***
-.14***
.02
-.06
-.08
CCCSCOM - Social competence
+
.07**
.06**
.03
.10
-.05
CCCPROS - Prosocial
+
.06***
.05*
.02
.07+
-.10
CCCEREG - Emotion regulation
+
.07***
.07***
.02
.07
-.10
CCCRAGG - Relational aggression
-.05*
-.05*
.00
-.19**
.22+
____________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings of Efficacy Subset Analysis:
Treatment effects measured by changes in
the 3rd and 4th Grades (g34)
__________________________________________________________________________________________________
Low
Benchmark
High
Exposure
Exposure
Exposure
Adequate
(<900)
(900-1044)
(1045+)
Exposure
versus
versus
versus (240+) versus
Hypothetical Comp. (0)
Comp. (0) Comp. (0)
Comp. (0)
Outcome Variable
Sign
Grade 3
Grade 3
Grade 3
Grade 4
Approximate Sample Size Used in Analysis
343
372
446
545
ICSTAGG - Aggression
.11
-.09
.01
-.25**
ICSTACA - Academic competence
+
-.10
.10
.06
-.12
ICSTINT - Internalizing
-.04
.10
.15
.09
CCCSCOM - Social competence
+
-.50***
-.17+
-.23*
.08
CCCPROS - Prosocial
+
-.51***
-.20*
-.24**
.06
CCCEREG - Emotion regulation
+
-.39**
-.21*
-.16+
.08
CCCRAGG - Relational aggression
.38***
.10
.21*
-.22**
__________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings of Efficacy Subset Analysis:
Treatment effects measured by changes in
the 3rd, 4th, and 5th Grades (g345)
Outcome Variable
Hypothetical
Sign
Low
Exposure
(<900)
versus
Comp. (0)
Grade 3
Benchmark
Exposure
(900-1044)
versus
Comp. (0)
Grade 3
High
Exposure
Adequate
Adequate
(1045+)
Exposure
Exposure
versus (240+) versus (240+) versus
Comp. (0)
Comp. (0)
Comp. (0)
Grade 3
Grade 4
Grade 5
Approximate Sample Size Used in Analysis
240
260
295
354
354
ICSTAGG - Aggression
.16
-.08
-.10
-.30***
-.10
ICSTACA - Academic competence
+
.10
.14
.03
-.06
.10
ICSTINT - Internalizing
-.17
.19
.14
.04
-.24*
CCCSCOM - Social competence
+
-.28
-.08
-.02
.22*
.29**
CCCPROS - Prosocial
+
-.43*
-.08
-.07
.20*
.30**
CCCEREG - Emotion regulation
+
-.27
-.09
.03
.23*
.27**
CCCRAGG - Relational aggression
.36+
.05
.11
-.31**
-.20+
__________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Findings of Efficacy Subset Analysis: Treatment
effects measured by changes in the 4th & 5th
Grades using subsets of Grade 3 exposure (g345)
Outcome Variable
Hypothetical
Sign
G3 Low
Exposure
(<900)
versus
Comp. (0)
Grade 4
G3
Benchmark G3 High
G3 Low
Exposure
Exposure Exposure
(900-1044)
(1045+)
(<900)
versus
versus
versus
Comp. (0) Comp. (0) Comp. (0)
Grade 4
Grade 4
Grade 5
G3
Benchmark G3 High
Exposure
Exposure
(900-1044)
(1045+)
versus
versus
Comp. (0) Comp. (0)
Grade 5
Grade 5
Approximate Sample Size Used in Analysis
221
246
285
234
252
288
ICSTAGG - Aggression
-.09
-.45***
-.32**
.34
-.05
-.26*
ICSTACA - Academic competence
+
.61**
-.09
-.22*
.19
.17
.02
ICSTINT - Internalizing
-.16
-.02
.12
-.50+
-.12
-.17
CCCSCOM - Social competence
+
.38*
.09
.19*
.27
.29*
.28**
CCCPROS - Prosocial
+
.36*
.12
.15
.33
.25+
.28**
CCCEREG - Emotion regulation
+
.39*
.10
.22*
.09
.29*
.28**
CCCRAGG - Relational aggression
-.19
-.24+
-.32**
.09
-.09
-.33**
______________________________________________________________________________________________________________________
*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.
Summary
From different methods of analysis, a pattern of small,
cumulative program effects emerges across grades 3, 4, and
5. These analyses exclude one poorly performing that was
dissolved in third year of the study.
Positive cumulative effects on:
 Social competence – including
• Prosocial behavior and
• Skill in regulating emotions
 Internalizing behavior
 Relational aggression
By HLM and efficacy subsets, promising effects observed on:
 Academic competence
 Aggression
Focuses on Program Development and Steps
in Intervention Research
Focuses on (Selection) Bias-Correction
Statistical Methods
For a description of Making Choices and copies of sample lessons, see http://ssw.unc.edu/jif/makingchoices/
References
Abadie, A., Drukker, D., Herr, J. L., & Imbens, G. W. (2004). Implementing matching estimators for average
treatment effects in Stata. The Stata Journal 4(3), 290-311.
Fraser, M. W., Day, S. H., Galinsky, M. J., Hodges, V. G., & Smokowski, P. R. (2004). Conduct problems and
peer rejection in childhood: A randomized trial of the Making Choices and Strong Families programs.
Research on Social Work Practice, 14(5), 313-324.
Fraser, M. W., Galinsky, M. J., Smokowski, P. R., Day, S. H., Terzian, M. A., Rose, R. A., & Guo, S.
(2005).Social information-processing skills training to promote social competence and prevent
aggressive behavior in third grade. Journal of Consulting and Clinical Psychology, 73(6), 1045-1055.
Fraser, M. W., Nash, J. K., Galinsky, M. J., & Darwin, K. E. (2000). Making choices: Social problem-solving
skills for children. Washington, DC: NASW Press.
Fraser, M. W., Richman, J. M., Galinsky, M. J., & Day, S. H. (2009). Intervention research: Developing social
programs. New York, NY: Oxford University Press.
Galinsky, M. J., Terzian, M. A., & Fraser, M. W. (2006). The art of group work practice with manualized
curricula. Social Work with Groups, 29(1), 11-26.
Guo, S., & Fraser, M. W. (in press). Propensity score analysis: Statistical methods and applications.
Thousand Oaks, CA: Sage Press.
Hansen, B. B. (2007). Optmatch: Flexible, optimal matching for observational studies. R News, 7, 18-24.
Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153-161.
Heckman, J. J. (2005). The scientific model of causality. Sociological Methodology, 35, 1-97.
Hirano, K., & Imbens, G. (2001). Estimation of causal effects using propensity score weighting: An
application to data on right heart catheterization. Health Services and Outcomes Research
Methodology, 2, 259-278.
Hodges, J. L., & Lehmann, E. L. (1962). Rank methods for combination of independent experiments in the
analysis of variances. Annals of Mathematical Statistics, 33, 482-497.
McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation with boosted regression
for evaluating causal effects in observational studies. Psychological Methods, 9, 403-425.
Rosenbaum, P. (1987). Model-based direct adjustment. Journal of the American Statistical Association, 82,
387-394.
Rosenbaum, P. (2002). Observational studies (2nd ed.). New York: Springer-Verlag.
Rubin, D. B. (2008). For objective causal inference, design trumps analysis. The Annals of Applied
Statistics, 2(3), 808-840.
Rubin, D. B. (1979). Using multivariate matched sampling and regression adjustment to control bias in
observational studies. Journal of the American Statistical Association,74(366), 318-328.
Rubin, D. B. (1979). Using multivariate matched sampling and regression adjustment to control bias in
observational studies. Journal of the American Statistical Association,74(366), 318-328.
Von Hippel, P. T. (2007). Regression with missing Ys: An improved strategy for analyzing multiply imputed
data. Sociological Methodology, 37(1), 83-117.
Note. CCC=Carolina Child Checklist; ICST = Interpersonal Competency Scale - Teacher
Four Stages in the Development of Program Manuals
Integrated across the Five Steps in Intervention Research
St eps in Int ervention Research
Step 1:
Specify
Problem and
Develop
Program
Theory
Step 2:
Create and
Revise
Program
Materials
Step 3:
Refine and
Confirm
Program
Components
Step 4:
Assess
Effectiveness in
Variety of
Settings and
Circumstances
Step 5:
Disseminate
Findings and
Program
Materials
Stage 1
Formulation
Stage 2
Revision
Stage 3
Differentiation
Stage 4
Translation/Adaptation
Source: Fraser, M. W., Richman, J. M., Galinsky, M. J., & Day, S. H. (2009). Intervention research: Developing social
programs. New York, NY: Oxford University Press.