Implementation of
Randomized Trials
David Myers
American Institutes for Research
Washington, DC
Prepared for IES/NCER Summer Research Training Institute, 2008
1
Context
• Conducting effectiveness trials and not efficacy trials
• Design and implementation of random assignment
should not distort the program/intervention or the
population served
• Programs should run at capacity
2
Illustrative Studies: Personal Involvement
• Upward Bound
– Nationally representative study
– 67 sites and more than 3,000 students
– Student RA within site
–
http://www.mathematica-mpr.com/publications/PDFs/upboundimpact.pdf
• Workplace Literacy
– 3 sites and about 500 adult learners
– Learner RA within site
–
"Addressing Literacy Needs at Work: Implementation and Impact of Workplace Literacy
Programs. Final Report." Washington, DC: Mathematica Policy Research, Inc., October
1998
3
Illustrative Studies: Personal Involvement
• 21st Century Community Learning Centers
– 12 grantees (elementary school study)
– Student RA within site
–
http://www.mathematica-mpr.com/publications/PDFs/elementaryschools.pdf
• NYC Voucher Experiment
– 2000 students
– Family RA
–
http://www.mathematica-mpr.com/publications/PDFs/nycfull.pdf
4
Illustrative Studies: Personal Involvement
• Reading Comprehension
– 10 districts and 40 schools
– School RA
–
http://www.mathematica-mpr.com/publications/PDFs/readcomp.pdf
• Closing the Reading Gap
– 32 schools (4 interventions)
– Student RA to reading groups within schools
–
http://ies.ed.gov/ncee/pdf/20084013.pdf
5
Illustrative Studies: Personal Involvement
• Quantum Opportunities
– 7 sites and about 1000 students
– Student RA within site
–
http://www.mathematica-mpr.com/publications/PDFs/QOPfinalimpacts.pdf
6
Practical Problems in Implementation
• Technical
• Political and Ethical Challenges
• Recruitment (not independent of P and E Challenges)
7
Further Exploration: Technical
• Cross-overs
– Members of C get into T or something like T (use UB
example and James Comer example)
• Into T -- straight forward adjustment (AIR, 1996)
• Something like T -- ?; affects interpretation
• Attrition
– Post baseline -- “straight forward” adjustment (e.g., MI)
– Before baseline -- problematic, no information
– How to fix?????
8
Further Exploration: Technical
• Unbalanced designs
– Don’t demand additional recruitment
• Large variance in selection probabilities (weights)
– “over subscription” didn’t meet expectations (UB example)
• Dishonest assignment -- post randomization
– Sites don’t tell all students they have been selected for the
program (UB example)
– Sites ignore RA and move controls into T
9
Further Exploration: Ethical and Political
Challenges
• Random assignment isn’t fair
– Programs afraid of denying services to students
• Is it fair to never give a student a chance? (UB example)
• Random assignment will force a program/teacher to
serve a different population
– Role of stratification to serve the desired mix
• Some students (units) must be served
– Role of the “wild card” before randomization
– Wild cards excluded from analysis
10
Further Exploration: Ethical and Political
Challenges
• All “seats” must be filled
– Role of the waiting list
• Random selection within strata, if needed and desired by
program operators
• T and C groups analyzed as implemented at initial
randomization
• When programs believe recruitment will distort the
population
– Identify “most likely” and “least likely” to serve
– Prior to randomization and stratify in analysis
11
Further Exploration: Ethical and Political
Challenges
• New treatments/interventions are hard to sell
– Core programs vs. supplemental programs
– Does the core program align with other curriculum and state
assessments?
• Concerns about making AYP
– Will a supplemental program reduce hours of instruction in a
core area such as ELA?
12
Strategies for Recruitment: Schools
• Start with assistant superintendent for instruction
(reading, math, science) or someone of similar
stature
• Quickly develop a relationship with an office and not
just an individual -- staff come and go
• Determine who needs to approve and to buy into
participation
–
–
–
–
Superintendent -- they may need to go to the school board
Principals
Teachers
Parents and community groups
13
Strategies for Recruitment: Schools
• Be prepared to meet with them
– Have a recruitment team
• Technical expertise in the design
– Be prepared to tell audience why RA is valuable and
not in a technical sense!
• Deep knowledge about the intervention -- most important
• Example of a meeting (next slide)
14
Irrefutable Evidence
15
Strategies for Recruitment: Schools
• Establish expectations for the researchers and the
schools
– Minimize legal talk
• Will bring in their general counsel and … .
– Responsibilities and timelines
•
•
•
•
Obtaining consent
Data collection
Administering instruments and tests
Answering questions from parents and others
16