Lessons Learned from Years of
Administering a Multi-Institution Online
Alumni Survey
American College Personnel Association
March 2013
Amber D. Lambert, Ph.D.
Angie L. Miller, Ph.D.
Center for Postsecondary Research, Indiana University
Presentation Outline
 Literature Review: Importance of alumni
assessment and survey issues
 Lessons from the Strategic National
Arts Alumni Project (SNAAP)
 Survey administration challenges
 How schools are implementing survey
results
Literature Review
 As funding to higher education institutions
continues to be cut, colleges and universities
are often required to show measures of their
effectiveness (Kuh & Ewell, 2010)
 Surveys are used in many areas of higher
education (Kuh & Ikenberry, 2009; Porter,
2004)
 Alumni surveys can provide valuable information on
student satisfaction, acquired skills, strengths and
weaknesses of the institution, and current career
attainment
Literature Review
 A major concern with all surveys, and alumni
surveys in particular, is low response rates
 Over the last decade survey response rates have
been falling (Atrostic, Bates, Burt, & Silberstein,
2001; Porter, 2004)
 Alumni surveys often have lower response rates
than other types of surveys (Smith & Bers, 1987)
due to:
 Bad contact information
 Suspicion of money solicitation
 Decreased loyalty after graduation
Lessons Learned from the
Strategic National Arts Alumni
Project (SNAAP)
SNAAP
 As an example, we will discuss some best practices for
survey administration and share results from the
Strategic National Arts Alumni Project (SNAAP)
 What is SNAAP?
 On-line annual survey designed to assess and improve
various aspects of arts-school education
 Investigates the educational experiences and career paths
of arts graduates nationally
 Findings are provided to educators, policymakers, and
philanthropic organizations to improve arts training,
inform cultural policy, and support artists
Who does SNAAP survey?
 Participants drawn from:
 Arts high schools
 Independent arts colleges
 Arts schools, departments, or programs in comprehensive
colleges/universities
 Over 5 years, SNAAP has been administered at nearly
300 institutions of various focuses, sizes, and other
institutional characteristics
 Cohort Year Sampling
 2008 and 2009 Field Tests: 5, 10, 15, & 20 years out
 2010 Field Test: 1-5, 10, 15, & 20 years out
 2011 and forward: all years to generate the most
comprehensive data possible
Increasing Numbers…
 2010 Field Test
 Over 13,000 respondents
 154 Institutions
 2011 Administration
 More than 36,000 respondents
 66 institutions
 2012 Administration
 More than 33,000 respondents
 70 institutions
 Now able to combine 2011 and 2012 respondents to create a
“SNAAP Database” with over 68,000 respondents
Questionnaire Topics

Formal education and degrees

Institutional experience and satisfaction

Postgraduate resources for artists

Career

Arts engagement

Income and debt

Demographics
Survey Administration
Challenges
Survey Administration Challenges:
Locating the Lost
 Important that contact
information is accurate and
up-to-date
 Encourage proactive efforts
 Newsletters
 Websites, social networking
 Alumni tracking
 Contracted with Harris
Connect, a direct marketing
firm
Survey Administration Challenges:
Response Rates
 Response rates are directly
related to the accuracy of
contact information
 Incentives: only minimally
effective
 “Open enrollment” features can
increase number of responses
 Social networking sites
 Need to verify respondents
Survey Administration Challenges:
Response Rates
 Email invitations to
participate in the
survey
 Is it better to have
HTML or plain text?
 For the 2011
administration, we
created visually
appealing email
invitations in HTML
format
Survey Administration Challenges:
Response Rates
 For the 2012 administration, we systematically
compared the effectiveness of HTML invites to
plain text invites across the 5 email contacts sent to
participants
 Results of this experiment suggested that a
combination of message types gets the highest
response rates
 Plain text was more effective for the initial contact
 HTML was more effective for follow-up contacts
 Potential reasons: plain text may reach larger
numbers, but HTML may give the project legitimacy
Survey Administration Challenges:
Response Rates
 Long-term strategies can also influence the tendency of
alumni to respond
 Consider building recognition of the alumni survey
presence while they are still students, so there is
familiarity with the project once contacted to
participate as alumni
 Sharing data on campus
 Involving students in campus or curricular changes made
based on survey results
 Connecting alumni surveys with senior exit surveys
(SNAAP plans to test this model with select institutions in
the 2014 administration)
Implementing Survey
Results
Using SNAAP for Curricular
Assessment
Using SNAAP for Curricular
Assessment (cont.)
Using SNAAP for Curricular
Assessment (cont.)
 Most important skills:
1.
2.
3.
4.
5.
Creative thinking and problem solving
Listening and revising
Interpersonal relations and working collaboratively
Broad knowledge and education
Critical thinking and analysis of arguments and
information
 Recommend that faculty:
 Incorporate open-ended projects (top skill #1) and group
projects (top skill #3)
 Require analysis of theories or reviews/critiques (top skill
#5) and provide opportunities for feedback and revision
(top skill #2)
 Ensure curricula include a firm knowledge foundation in a
wide variety of areas (top skill #4)
Using SNAAP for Curricular
Assessment (cont.)
Using SNAAP for Curricular
Assessment (cont.)
Using SNAAP for Curricular
Assessment (cont.)
 Can identify strengths:
 What skills and competencies have the highest
percentages of alumni reporting the institution
helped them develop “very much” or “quite a bit”?
 Can identify areas for improvement:
 What skills and competencies have the highest
percentages of alumni reporting the institution
helped them develop “very little” or “not at all”?
 Peer group information provides context:
 Do other institutions have similar strengths and
weaknesses?
Examples From 2011 Aggregate Findings
100%
90%
90%
80%
77%
81%
71%
70%
60%
50%
40%
30%
26%
22%
20%
10%
0%
Artistic technique
Financial and business
management skills
Entrepreneurial skills
"Very" or "Somewhat Important" to profession or work life
Institution contributed "Very much" or "Some" to development of skill
Using SNAAP for Curricular
Assessment (cont.)
 Alumni receive strong training in learning artistic
techniques
 Discrepancies between those who say a skill is
important for their work and those who say the
institution helped them develop that skill suggest
some improvements that could be made, such as:
 Requiring business and financial classes, or
incorporating these elements into existing courses
 Include classes looking at the “nontraditional” career
paths of arts graduates
Using SNAAP for Program Assessment
Using SNAAP for Program
Assessment (cont.)
Using SNAAP for Program
Assessment (cont.)
 Programs and services with low satisfaction may
need to be revised
 Career advising had 59% report either “very
dissatisfied” or “somewhat dissatisfied”
 Additional resources could be devoted to
developing new components of career advising
such as:
 Alumni career panel presentations
 Résumé or portfolio building sessions
 Networking opportunities for graduating students
Examples:
Sharing on Campus
Assessment Brief #62
October 12, 2011
Miami University
Assessment
Using Feedback from Miami Alumni
to Improve Educational Effectiveness
Surveying Alumni
Miami students are frequently surveyed throughout
their college experiences. However, assessing the
long-term impact of students’ Miami education can
also require reaching out to students after they
graduate. Feedback from alumni, who are now using
the skills they developed at Miami, can greatly
improve educational effectiveness.



The Strategic National Arts Alumni Project
(SNAAP) survey gathers information about fine arts
alumni to better understand the relationship between
arts education and arts-related occupations. The
SNAAP participants from Miami University
consisted of 220 undergraduate fine arts alumni who
graduated in the following years: 1990, 1995, 2000,
and 2005-2009.
Respondents were least satisfied with
opportunities to network with alumni and others,
advice about further education, career advising,
and work experience.
The vast majority of respondents reported
developing critical and creative thinking skills
while at Miami and found these skills important
in their future careers.
Fine arts alumni were less likely to report that
Miami helped them to develop business and
technological skills related to their field.
Student Satisfaction
The survey included questions about institutional
experiences and career choices. To capture
institutional experiences, the survey prompted alumni
to report their overall satisfaction with their education
as well as their satisfaction with specific areas (e.g.,
academic advising, freedom to take risks). In the
career section, alumni reported their current and
previous occupations, their satisfaction with these
jobs, and their current level of fine arts engagement.
To explore the intersection between institutional
experience and careers, the survey asked alumni
about the skills and competencies they developed at
Miami University as well as which skills were most
important in their current job.
By reviewing these results, faculty and staff can
better understand how students’ experiences at
Miami prepare them for their career.
Key Findings
 Fine arts alumni were satisfied with their
experiences at Miami University; 94% of
undergraduate arts alumni rated their overall
experience as good or excellent.
 Arts alumni were especially satisfied with their
sense of belonging at Miami and with their
instructors.
Recommendations
The SNAAP survey highlights the importance of
gathering alumni feedback. Such feedback is a
valuable resource for assessing educational impact
and improving educational effectiveness across the
university. The SNAAP survey helps faculty and
staff in fine arts by identifying the following:
 Common occupations and post-secondary
degrees among graduates
 Skills and competencies that students will
frequently use in their careers
 Levels of student satisfaction with various
aspects of their Miami experience
These results can help the division improve retention
and graduation rates and better prepare students for
their future careers.
If you have comments or questions, please contact the Center for the Enhancement of Learning, Teaching and University
Assessment at celt@muohio.edu or 513-529-9266. Previous Briefs are available online at:
http://www.units.muohio.edu/celt/assessment/briefs/.
Examples:
Alumni & Donor Outreach
Examples:
Recruitment
Conclusions
 Assessing alumni can provide important information
on institutional effectiveness, but alumni surveys can
pose several obstacles
 When administering alumni surveys, some steps can
be taken to update contact information and increase
response rates
 The results from alumni surveys can be useful in
multiple areas, including curricular and program
assessment, campus information sharing, alumni and
donor outreach, and recruitment
Questions or Comments?
 Contact Information:
 Amber D. Lambert adlamber@indiana.edu
 Angie L. Miller anglmill@indiana.edu
Strategic National Arts Alumni Project (SNAAP)
www.snaap.indiana.edu
(812) 856-5824
snaap@indiana.edu
References
Atrostic, B. K., Bates, N., Burt, G., & Silberstein, A. (2001). Nonresponse in
U.S. government household surveys: Consistent measure, recent
trends, and new insights. Journal of Official Statistics, 17(2), 209226.
Kuh, G. D. & Ewell, P. T. (2010). The state of learning outcomes assessment in
the United States. Higher Education Management and Policy, 22(1),
1-20.
Kuh, G. D. & Ikenberry, S. O. (2009). More than you think, less than we need:
Learning outcomes assessment in American higher education,
Urbana, IL: University of Illinois and Indiana University, National
Institute of Learning Outcomes Assessment.
Porter, S.R. (2004). Raising response rates: What works? New Directions for
Institutional Research, 121, 5-21.
Smith, K., & Bers, T. (1987). Improving alumni survey response rates: An
experiment and cost-benefit analysis. Research in Higher
Education, 27(3), 218-225.