THE FROYO EFFECT:
USING SMALL INCENTIVES TO ACHIEVE A
51% UNIVERSITY-WIDE RESPONSE RATE
Office of Institutional Research and Analysis
Director, Janel Sutkus, Ph.D.
Research Designer and Analyst, Hyun Kyoung Ro, Ph.D.
Our goal for this presentation
To describe – in detail – how we administered
a small incentive program for our spring census
survey that resulted in a 51% response rate
(a response rate we had not achieved since
Spring 2005)
Outline
Study background
Response rate history and goal
Survey schedule by cohorts
The big RQ
Survey populations (on-campus and off-campus)
Marketing plan
Administration
Calendar
Invitation and incentive messages
Outcomes
Response rates and odds of responding
Redemption of small incentives
Questions and discussion
Study background
The Carnegie Mellon University-Student Experience Study (CMU-SES)
is a university-wide study designed to understand:
 how well, if at all, we are meeting our educational objectives for students
 which experiences, resources, and activities affect those educational outcomes
 when they do affect the outcomes, in what direction and magnitude
The CMU-SES was designed to replace the End of Year Survey which was
administered to undergraduates in 2000, 2001, 2004, 2005, and 2009.
It will be administered to all CMU students on a three-year cycle,
with the next administrations in the spring semesters of 2015 and 2018.
The core planning team began meeting on June 1, 2011 with these working
principles:
 this will not be a revision of past EOY’s, but an entirely new study
 the development process will be research question-driven, rather than survey
item-driven, and the instrument will be created hand-in-hand with the analysis
plan (we will hypothesize that certain predictors affect particular outcomes,
and will test those hypotheses using regression methods)
 satisfaction is not one of our primary educational objective, so this
will not be a satisfaction survey
 we cannot study everything we want to know, so we will set our priorities
and study a few things in a thorough manner
We spent ten months creating and testing this survey.
Survey and assessment schedule 11-12 to 18-19
Year
2011-12
2012-13
F
S
CIRP
SES
F
2013-14
S
F
2014-15
S
F
2015-16
S
F
2016-17
S
F
2017-18
S
F
2018-19
S
F
S
Cohort
Fall 2011
Fall 2012
BCSSE
ADV
ADV
SES
ADV
ADV
SES
ADV
SES
ADV
ADV
SES
ADV
ADV
SES
ADV
ADV
SES
NSSE
Fall 2013
NSSE
CIRP
Fall 2014
Fall 2015
BCSSE
Comparisons:
FY
ADV
CIRP
NSSE
Comparisons:
SES
Fall CIRP to Spring SES
BCSSE
NSSE /ADV
Soph
ADV
FY SES to SR SES
ADV
Junior
ADV
Soph ADV to JR ADV
SES
Senior
SES
NSSE/ADV
ADV
NSSE
Comparisons:
Fall BCSSE to Spring NSSE
ADV
FY ADV to JR ADV
FY NSSE to SR NSSE
SES
JR ADV to SR ADV
FY ADV to Soph ADV
ADV
Soph ADV to SR ADV
ADV
CIRP: Cooperative Institutional Research Program Freshman Survey (UCLA-HERI)
Spring 2012
SES
BCSSE: Beginning College Student Survey of Engagement (Indiana University)
Spring 2013
ADV
NSSE: National Survey of Students Engagement (Indiana University)
Spring 2014
ADV
SES: Carnegie Mellon University Student Experience Study (CMU IR&A)
Spring 2015
SES
ADV: Carnegie Mellon University Undergraduate Advising Survey (CMU IR&A)
Spring 2016
ADV
Spring 2017
ADV
Spring 2018
SES
Spring 2019
ADV
NSSE (FY and senior only)
NSSE (FY and senior only)
NSSE (FY and senior only)
Response rate history for spring census surveys
58.6%
53.0%
39.7%
32.7%
32.3%
31.6%
33.0%
UG pop randomly split in two
April 2000
End of Year
April 2001
End of Year
April 2004
End of Year
April 2005
End of Year
April 2009
End of Year
April 2009
Advising
April 2011
Advising
The big goal and associated RQ
By using a series of small incentives, can we return our spring
census response rate to the levels we had in the mid-2000’s?
We hedged our bets by also providing a large “raffle-type”
incentive, but our plan was to do analysis at the end to
determine whether small incentives could replace large
incentives in the future.
RQ: Are there differences in the odds of responding during a
particular small incentive window for students who are eligible
for small incentives compared to students who are not eligible
for small incentives?
Survey population
ON CAMPUS: 5517 students Were eligible for a drawing
for one of two iPad3’s (or $500 at Bookstore) AND
received a small incentive for completing the survey
OFF CAMPUS: 117 students on study abroad were eligible
for a drawing for one of two iPad3’s (or $500 at Bookstore)
LIMITATION AND CAVEAT: Other than the big difference of
being off-campus and having study abroad experience, the
two groups were much the same in terms of demographics
and ultimate survey responses
Off campus students were not aware what they were missing
Marketing plan
“Who says? CMU-SES!” branding campaign
Banner in University Center
Table tents at dining locations
Flyers
“Tabling” on the Cut with balloons and branded candy
Core team had CMU-SES t-shirts with iron-on transfers
Who says? CMU-SES
Student
Experience
Study
Administration calendar
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
25
26
27
28
29
SURVEY LAUNCH
11:00 PM
30
LAUNCH EVENT AT UC
4-6 PM
31
1
2
GIVEAWAY #1
ENDS AT 10 PM
REMINDER #1
SENT AT 10 PM
3
GIVEAWAY #1
AT UC 3-6 PM
(FroYo)
4
5
6
7
8
9
10
11
GIVEAWAY #2
ENDS AT 10 PM
12
GIVEAWAY #2
AT UC 3-5 PM
(Fitness class card)
REMINDER #2
SENT AT 10 PM
13
14
15
16
17
18
GIVEAWAY #3
ENDS AT 10 PM
REMINDER #3
SENT AT 10 PM
19
GIVEAWAY #3
AT CARNIVAL MIDWAY
3:30-5 PM
(Funnel cake)
20
21
22
23
24
25
GIVEAWAY #4
ENDS AT 10 PM
REMINDER #4
SENT AT 10 PM
26
GIVEAWAY #4
AT UC 3-5 PM
(Maggie Murph coffee)
FINAL REMINDER
SENT AT 10 PM
27
SURVEY CLOSES 2 PM
WRAP EVENT AT UC
5-7 PM
(CMU gear and
iPad3 drawing)
28
Invitation
Dear Andrew,
We are writing to invite your participation in the Carnegie Mellon University Student Experience Study (CMU-SES).
Every three years the university conducts a study of the entire student body to gain insight into the educational experience at
CMU. Your input is extremely valuable and should take fewer than 10 minutes using the link below.
Students who complete the survey will be placed into a drawing for one of two iPad3’s (the recipients may choose to substitute
$500 of merchandise at the University Stores). In addition, all students who complete the survey between now and 10:00 PM
on Monday, April 2, will be eligible to pick up a certificate for a free FroYo at Skibo Café.
Please click here to participate in the survey: <<link>>
Or copy and paste the URL below into your internet browser: <<URL>>
Not included
for students
studying abroad
This study is sponsored by the Vice Provost for Education and the Dean of Student Affairs and is administered by the Office of
Institutional Research and Analysis. Your individual responses will be kept confidential and will only be reported in the aggregate.
No information that could identify your responses will be reported.
The survey will be open until Friday, April 27, at 2:00 PM.
Thank you for your participation. If you have any questions about the study, please contact Janel Sutkus, Director of Institutional
Research and Analysis, at jsutkus@cmu.edu.
Sincerely,
Amy Burkert, Vice Provost for Education
Gina Casalegno, Dean of Student Affairs
End of survey messages
On campus:
Thank you for completing the CMU-SES!
Watch your e-mail for details on how to claim your thank you gift and for
your invitation to the closing party on April 27 where we will announce the
winners of the two iPad3's (you do NOT need to attend to win).
Studying abroad:
Thank you for completing the CMU-SES!
You have been entered into the drawing for one of two iPad3's.
E-mail upon completion
Dear Andrew,
Thank you for participating in the CMU-Student Experience Study. This e-mail is your ticket for TWO things:
1) You have won a certificate for a free FroYo at the Skibo Café.
Please pick up your certificate at:
The University Center, Wean Commons on Tuesday, April 3, from 3:00 to 6:00 PM.
You will need your student ID and this e-mail to receive your certificate.
You can either print this e-mail or show it on your smartphone. The certificate will be valid until May 11.
2) You are invited to the CMU-SES closing party at:
The University Center, Connan Room on Friday, April 27, from 5:00 to 7:00 PM.
The party will include refreshments and music, and students who attend will receive tickets for drawings for CMU gear.
We will also announce the winners of the two iPad3's (you do not need to attend to win).
You will need your student ID and this e-mail to attend. You can either print this e-mail or show it on your smartphone.
We look forward to seeing you there.
Janel Sutkus and Hyunny Ro
Office of Institutional Research and Analysis
Amy Burkert
Gina Casalegno and Holly Hippensteel
Office of the Vice Provost for Education Office of the Dean of Student Affairs
Certificates
This certificate is good for
one Group X fitness class
(includes yoga classes)
at the University Center.
This certificate is good for
one FroYo at the Skibo Café.
Thanks for participating in CMU-SES!
#
Expires
5/11/12
Thanks for participating in CMU-SES!
#
This certificate is good for
one funnel cake at the
A Phi O Booth at Carnival.
This certificate is good for
one 16 ounce brewed coffee
or tea at Maggie Murph Café.
Thanks for participating in CMU-SES!
#
Expires
4/21/12
Expires
5/4/12
Thanks for participating in CMU-SES!
#
Expires
5/11/12
Each certificate was hand-numbered in green ink and samples were given to each vendor.
Responses by date
662
484
Reminder
Survey
launched
at 11 PM
354
257
199
86
Final
Reminder
Reminder
41
163
159
Reminder
44 44
Reminder
88
71
48
3/29 3/30 3/31
4/1
4/2
4/3
4/4
4/5
4/6
4/7
4/8
4/9
4/10 4/11 4/12 4/13 4/14 4/15 4/16 4/17 4/18 4/19 4/20 4/21 4/22 4/23 4/24 4/25 4/26 4/27
Responses by incentive
Invited R4
Invited R3
Invited R2
Invited R1
5635
4301
Responded
1334
3669
Responded
632
3209
Responded
460
Invited R5
3104
Responded
315
Responded
105
Incentive
FroYo
Fitness class
Funnel cake
Coffee
None
Response rate
23.7%
14.7%
12.5%
3.3%
10.1%
Odds of
responding
89.8%
higher odds
no difference
in odds
688.3%
higher odds
no difference
in odds
no difference
in odds
in the specific incentive window for a student in the small incentive population compared to a student
not in the small incentive population, and controlling for gender, race, citizenship, and class year
Incentive redemption
Week
Incentive
Days
active
Cost
Responses
N
Collected
%
Collected
Dist at
Closing
Party
Cost if all
redeemed
N
redeemed
%
redeemed
Actual
cost
20
$609
136
67.0%
$408
5
29.4%
$266
62
81.6%
$217
1
FroYo
4
$3.00
1334
183
13.7%
2
Fitness
Class
10
Gift
632
17
2.7%
3
Funnel
Cake
6
$3.50
460
76
16.5%
4
Coffee
7
$1.25
105
7
6.7%
10
$21
12
70.6%
$15
2531
283
11.2%
30
$947
215
76.0%
$640
ALL
27
Total costs
Incentives
Cost
Marketing
Cost
All small incentives
$640
Banner
$144
Two iPad3’s
$998
Balloons
$25
T-shirts, candy, supplies
$1638
$122
Closing Party
Cost
Balloons and decorations
$146
Food (pizza, popcorn,
cookies, hummus trays)
$468
Drinks
$291
$32
$646
$2575
Compared to the most recent census survey (April 2011 Undergraduate Advising),
we paid $143 per additional response percentage point or $2.61 per additional student
Future plans
CMU-SES in Spring 2015:
GOAL: Maintain response rate in the low 50’s
Marketing and branding continues
No large incentives
Keep small incentives (replace fitness class and coffee)
Anticipated budget - $1700
New “health behaviors” survey in Fall 2013:
GOAL: Response rate in low 50’s to match with
FY’s and sophs from CMU-SES in Spring 2013
Create marketing and branding campaign
Guaranteed small incentives (FroYo, other tasty but healthy treats)
Raffle every two days for Fitbugs
In the time since. . .
58.6%
53.0%
51.0%
39.7%
32.7%
32.3%
31.6%
33.4%
33.0%
One large
incentive:
iPad Mini
UG pop randomly split in two
April 2000
End of Year
April 2001 April 2004
End of Year End of Year
April 2005
End of Year
April 2009
End of Year
April 2009
Advising
April 2011
Advising
April 2012
Student
Experience
Study
April 2013
Advising
For further information
Janel
jsutkus@cmu.edu
Hyunny
hro@cmu.edu