2013 Summer Pilot Study
Utilizing the PEAR Common Instrument & Dimensions of Success Program Observation Tools
The Boston Private Industry Council (PIC) in partnership with the Massachusetts Afterschool Partnership (MAP) is
conducting a pilot study, using student interest and program evaluation tools developed by Harvard University
and McLean Hospital’s Program in Education and Afterschool Resiliency (PEAR). The primary objective of this pilot
is to boost regional STEM networks’ capacity to promote and implement data collection strategies, while also
helping to foster a community of practice around program quality in the out-of-school time community.
Ultimately, the data collection made possible by PEAR’s Common Instrument and Dimensions of Success tools will
provide more detailed information on Massachusetts students’ interest in STEM subjects and the range of
summer programs available to them.
The Common Instrument (CI) is a three-page survey that youth will complete, answering questions about their
interest in science, technology, engineering, and math. It will be administered twice, once at the start of the
program and once at the end, to gauge shifts in their feelings on STEM subjects. The CI includes five questions
from the National Assessment of Educational Progress (NAEP) so that the results can be compared with a national
sample of youth.
The Dimensions of Success (DoS) observation protocol focuses on measuring STEM programs in a variety of
settings. Observations are conducted in pairs to establish inter-rater reliability. Observers watch and take notes
for 15 minutes and then utilize a rubric (see below) to rate what was seen for an additional 10 minutes. Reports
are administered to students and activity leaders for the last 10 minutes of the observation. All scores are then
entered into an online database.
ELIGIBILITY
Any community- or school-based program that provides STEM summer programming to middle school youth (rising 5th
through 9th graders) is eligible to apply. Programs or partners must serve at least 20 youth for a minimum of 4 hours
per week, over the course of 4 weeks or more. Programs that are shorter in length (1-2 weeks) but offer more STEMintensive experiences are also encouraged to apply.
Students will be asked to provide 2012-2013 end of the year (EOY) report card data, which will allow school
achievement data to be linked to survey data collected from participants and facilitators. However, this is optional -students will be able to participate without providing this information.
STIPEND AWARDS
Successful applicants are expected to send staff to the PEAR two-day training and to administer the Common
Instrument within the first sessions of the summer program and again at the conclusion of the program; as well as
participating in two observations utilizing the Dimensions of Success tool. Upon completion of the pilot program,
participating programs will receive stipends of $1000. Awardees will receive:
1
SUMMER 2013 PILOT STUDY: UTILIZING THE PEAR COMMON INSTRUMENT & DIMENSIONS OF SUCCESS PROGRAM OBSERVATION TOOLS – APPLICATION




Free two-day training for one staff member to become adept in the use of the PEAR tools; the staff member
will become PEAR-credentialed after taking part
Access to individual program results that will help organizations better understand the impact of their
programs on youth interest in STEM
Stipend of $1000 upon successful implementation of pre- and post- CI surveys on summer cohort, participation
in two DoS observations, and attendance at two-day training and fall 2013 learning forum
Long-term access to the CI and DoS tools, allowing interested programs to self-sustainably continue their use
after the pilot program ends
TIMELINE










May 24 – All OST providers submit their qualifications to participate in the project
Week of May 27 – Selected OST providers are invited to participate
June 5 & 6, 9:00 A.M. - 4:00 P.M., Worcester United Way – Two-day training in the use of PEAR tools
June/July – All CI pre-tests are administered and data-entered by programs
July-August – DoS tool conducted at program sites, once by programs and once by STEM network staff
August 30 – All CI post-tests are administered and data entered
September 15 – PEAR conducts analysis on individual programs and produces a report for each to review and
comment on their data
September 30 – Feedback due to PEAR and MAP
October – PEAR drafts pilot report, soliciting feedback and questions from networks and programs
November/December – STEM networks and MAP host learning forum to share outcomes and results
APPLICATION INSTRUCTIONS
To apply, please complete the application on the following pages. Narrative responses should be no more than two
typed pages. Applications should be e-mailed to your regional STEM network contact, listed below; please contact that
representative for any additional information or questions. All applications must be received by 4:00PM on Friday, May
24.
Network
Boston
Central
MetroWest
Northeast
Contact
Lynne Powers, lynne.powers@bostonpic.org
Karin Lebeau, karin.lebeau@umassmed.edu
Kevin Thurston, kthurston1@framingham.edu
Marjorie Dennis, marjorie_dennis@uml.edu
2
SUMMER 2013 PILOT STUDY: UTILIZING THE PEAR COMMON INSTRUMENT & DIMENSIONS OF SUCCESS PROGRAM OBSERVATION TOOLS – APPLICATION
COMPLETED APPLICATION MUST BE RECEIVED BY 4:00 PM on May 24
Submissions should be e-mailed. Faxed applications will not be accepted.
E-mail: E-mail completed application and attachments to local STEM network representative,
with subject line “Summer Pilot Study Application”
Eligibility: Any community- or school-based program that provides STEM summer programming to middle school youth
(rising 5th through 9th graders) is eligible to apply. Programs or partners must serve at least 20 youth for a minimum of 4
hours per week, for at least 4 weeks, in order to effectively implement the Common Instrument and Dimensions of
Success tools. Programs that are shorter in length (1-2 weeks) but offer more STEM-intensive experiences are also
encouraged to apply. Application narrative responses are limited to two typed pages.
PART I: Organizational Information
Please provide organization information and community demographics.
Organization:
_____________________________________________________________
Organization Name
_____________________________________________________________
Address
City
Zip
_____________________________________________________________
Phone
Fax
Website
Check all that apply:
CBO (Community-Based Organization)
21st Century Community Learning Center
Contact Information:
ASOST Grantee
School/School District
_____________________________________________________________
Executive Director
Phone
E-mail
_____________________________________________________________
Project Coordinator (if different)
Availability:
Phone
E-mail
What are the start & end dates of your program? ______________________
What is the start date for your summer program staff? ______________________
Community Information:
Please indicate the percentage of students in your program in the following priority
areas.

Gender: Female:______% Male:______%

Low-income (receiving free or reduced lunch and/or contracts/vouchers under the
Department of EEC): ______%

Children or youth with disabilities: ______%

English Language Learners: ______%

Students from low-performing districts: ______%

Racial/ethnic demographics (please add lines as needed):
__________________________ - ______%
__________________________ - ______%
__________________________ - ______%
__________________________ - ______%
3
SUMMER 2013 PILOT STUDY: UTILIZING THE PEAR COMMON INSTRUMENT & DIMENSIONS OF SUCCESS PROGRAM OBSERVATION TOOLS – APPLICATION
PART II: Narrative
Please respond to the following questions briefly and concisely (no more than 2 typed pages).
1. Provide a brief description of your program.
2. Please describe your organization’s STEM offerings.
3. Who will be your Project Director/point person? Why was this person selected to lead the project? Indicate
here if any other staff or volunteers will be part of this project and how they will participate.
4. What agencies, organizations, or other service providers have agreed to cooperate with your organization on
this project to support summer STEM learning?
PART III: Required Commitment
Please confirm that your program agrees to project commitments by checking the boxes below and signing.
________________________________ agrees to the following:
Organization name
One staff member will serve as the project’s point person and will commit to attending the two-day training,
coordinate the program’s participation in the project, and serve as the main program site contact.
My program will administer the Common Instrument within the first two days of summer programming and
again within the last week of the program, and will enter all data into the database.
My program will conduct at least two Dimension of Success protocols at its site and will report results of selfevaluations and improvement strategies in program development.
My program will send a representative to the follow-up forum to share in the findings from the experience.
My program will communicate regularly with MAP, PEAR, and regional STEM Network organizers.
My program will provide summer STEM programming to at least 20 youth for a minimum of 4 hours per week
over at least 4 weeks or a shorter in length (1-2 weeks), more STEM-intensive experience.
My program will work with PEAR on parental consent for requesting report card data, and to help remove
identifying features from the data in a way that will allow it to be linked to survey data collected from
participants and facilitators.
Signed: __________________________________________________
Date:________________________
Authorized Signature
__________________________________________________
Name (Printed)
4
SUMMER 2013 PILOT STUDY: UTILIZING THE PEAR COMMON INSTRUMENT & DIMENSIONS OF SUCCESS PROGRAM OBSERVATION TOOLS – APPLICATION