S-STEM
Program Evaluation
S-STEM PI Meeting
Arlington, VA
October 2012
Program Evaluation Overview
 NSF need for a comprehensive examination of
implementation, effectiveness and impact of S-STEM
 Dual evaluation focus
– Scholarship recipients and role of S-STEM in their
educational opportunities and employment outcomes
– Institutions and the student support they provide
 Evaluation team: Abt Associates, SageFox, Abt SRBI
Abt Associates | pg 2
Key Evaluation Features
 Collaborative relations with NSF, projects, and other
experts
 Evaluation based in program theory
 Involves 494 projects
– Funded from 2006 to 2010
– Involve undergraduate students
Abt Associates | pg 3
Challenges and Opportunities
 Challenges
– Tracking scholarship recipients
– Obtaining cooperation and high response rates
– Identifying an appropriate comparison
 Opportunities
– Collecting data representative of projects and program
– Providing NSF with a strong understanding of investment
– Contributing to understanding of student supports and
success
Abt Associates | pg 4
Study Components
 Implementation study
– Ways in which S-STEM projects recruit and retain students in STEM fields,
allocate scholarship funds, and provide educational and support
programming for scholarship recipients
 Relational study
– Associations between project characteristics and practices and recipient
outcomes
 Benchmarking comparison
– Comparisons of recipients’ educational and academic support experiences
to national trends
 Quasi-experimental study
– Use of Propensity Score Matching (PSM) to compare the educational and
career outcomes of S-STEM scholarship recipients to a matched national
sample
Abt Associates | pg 5
Multiple Data Sources

S-STEM Monitoring System
–

Projects’ Annual Reports
–

Operation and implementation of S-STEM awards
Survey data from S-STEM scholarship recipients
–

In-depth information about project implementation in particular contexts
Survey data from S-STEM Principal Investigators
–

Descriptions of activities and implementation
Site visits to a purposive sample of S-STEM projects
–

Components and strategies employed by S-STEM projects
Information on educational experiences, and r educational and career outcomes
Beginning Postsecondary Students Longitudinal Study (BPS) survey
–
Extant data on outcomes from a matched comparison group
Abt Associates | pg 6
Use of Monitoring System
 Identify population
– 5,478 at two-year institutions
– 14,914 at four-year institutions
 Identify supports and activities (2-year n=74, 4-year
n=350)
– Academic support services (96%, 94%)
– Participation in meetings or conferences (86%, 63%)
– Career counseling (76%, 70%)
– Mentoring by faculty or professionals (73%, 94%)
Abt Associates | pg 7
Site Visits to Selected Awards
 Emerging themes
– Building on an existing structure
– Developing a community of scholars
– Providing talent for the local workforce
– Recruiting through an existing program
– Providing early access to research
– Recruiting to the university
– Strategic alignment with other university initiatives
– Mentoring/advising
– Creating pathways
Abt Associates | pg 8
PI Survey Topics
 S-STEM activities
 Recruitment and selection
 Scholarship recipients
 Recipient outcomes
 Benefits and challenges
Abt Associates | pg 9
Recipient Survey Topics
 Education background
 College experiences
– Supports
– Work
– Financial aid
 Future plans
 Employment
Abt Associates | pg 10
Critical Role of PIs and Recipients
 Ensure high response rate on surveys
– OMB threshold is 85%
 Cooperation with requests from Abt Associates
– Complete surveys
– Encourage participation
 Timeframe
– NSF and OMB review required before fielding
– Surveys in field Spring or Fall 2013
Abt Associates | pg 11
Measure Progress toward Goals
 Improved educational opportunities for students
 Increased retention of students to degree
achievement
 Improved student support programs at institutions of
higher education
– Increased numbers of well-educated and skilled employees
in technical area of national need
Abt Associates | pg 12
Uses of Evaluation
 Reporting progress and accomplishments to
stakeholders
– Congress, OMB, GAO, OSTP, NSF, the public
 Accountability
– Numbers of STEM graduates, time to degree, retention,
success of project activities
 Program improvement
 Inform similar types of programs
Abt Associates | pg 13
Questions
alina_martinez@abtassoc.com
Abt Associates | pg 14