Alumni Data For
Program Evaluation and
Policy Analysis:
Reality to Envisioned Reality,
University of Wisconsin-Madison
Jocelyn Milner, Director,
Academic Planning and Analysis,
Clare Huhn, Policy and Planning Analyst,
Academic Planning and Analysis
AIRUM
November 10-11, 2005
Acknowledgements

Our Wisconsin Alumni Association (WAA)
collaborators
 Heidi Zoerb, Sr. Marketing and Research Manager
(until 11/05)
 Angie Nash, VP for Information Services

Peter Spear, Provost (until 11/05)
Paula Bonner, President of WAA


Funded in part by grants from the UW-Madison
Assessment Fund (2004-05 and 2005-06)
2
UW-Madison/WAA Alumni Data Project
Outline
Context, ideas, and bigger ideas
 Phase I – Feasibility Study (did these
ideas have a future?)
 Phase II – Development (making the
ideas work)
 Comments and advice from you

3
UW-Madison/WAA Alumni Data Project
Research on Alumni
A long and
rich history
Under-developed
4
Development and giving
 Services to alumni, effective
communications strategies
 Learning outcomes,
engagement, academic and
curricular planning, recruiting

UW-Madison/WAA Alumni Data Project
Starting Assumptions,
Now Assertions

Information about alumni is important
to academic programs for assessment
and evaluation
(External stakeholders expect we know)

5
Gaps in the available information are
worth making an effort to fill
UW-Madison/WAA Alumni Data Project
Idea – Alumni Profile



Build a “profile” of alumni information for
each academic program
Base it on unit record data
Include degree information, addresses,
employment, continued engagement,
perceived value of a UW-Madison degree,
evident value of a UW-Madison degree
6
UW-Madison/WAA Alumni Data Project
Alumni Profile
Level: Bachelors
Major: Jurassic Studies
1-5 years
6-10
out
years out
% Female
50%
42%
% Minority
12%
8%
% Living in WI
45%
36%
% Providing employment info
100%
73%
% Employed, by occupation, by sector
100%
98%
% Employed in occupation related to major
68%
87%
% Who agree that UW-Madison education “made
a difference”
100%
100%
% Continued engagement (through WAA)
77%
95%
% Ever made a donation
3%
75%
7
UW-Madison/WAA Alumni Data Project
Bigger Idea –
If we have a unit record alumni
data set,
then we have a powerful resource
for analysis and for serving our
public accountability needs.
8
UW-Madison/WAA Alumni Data Project
Public “Accountability”




Use data, analysis to counter assumptions
and mis-information (“brain drain”)
Measuring Up (national report card)
Accreditation: HLC-NCA, ABET, and most
others
AAU Institutional Data Committee – what
do graduates do?
9
UW-Madison/WAA Alumni Data Project
Public “Accountability”
HEA (S.1932)
“Requires institutions to report to the DoE
information on employment placement of
graduates and graduates that enroll in
graduate education as gathered from such
sources as alumni surveys, student
surveys, NSSE, state data systems, other
relevant sources.”
(from 10/28/05 AAU summary)
10
UW-Madison/WAA Alumni Data Project
Questions?
What do graduates do after college?
 Did their education make a difference to
them? (private interest)
 Did their education make a difference to
the community? (public interest)

11
UW-Madison/WAA Alumni Data Project
External / Internal

External utility (IR office, provost’s office,
external affairs)


aggregated analysis at the institutional or
school/college level
Internal utility (school/college,
department or program level)

Alumni Profiles, lists of alumni
12
UW-Madison/WAA Alumni Data Project
The timing was right (Spring 2004)





UW-Madison gave up on PS-Advancement
The Registrar’s Office gave post-graduation
records to WAA and UW Foundation
WAA invested in on-line collection interface
and robust data base
Potential arose for unified alumni records
Alumni records now separated from student
records
13
UW-Madison/WAA Alumni Data Project
14
UW-Madison/WAA Alumni Data Project
Outline
Context, ideas, and bigger ideas
 Phase I – Feasibility Study (did these
ideas have a future?)
 Phase II – Development (making the
ideas work)
 Comments and advice from you

15
UW-Madison/WAA Alumni Data Project
Phase I – Feasibility Study







Can we actually produce a prototype Alumni
Profile, use unit record alum data for analysis?
What data elements are held centrally?
Where are the gaps?
Who are the data custodians?
Are they willing partners?
Can the data sources can be linked?
Can they be used analytically?
16
UW-Madison/WAA Alumni Data Project
Project Goals



Expand the collection of information on UW-Madison alumni to
include more elements of interest for program evaluation and
academic program development.
Store the information in a data base structure that makes is
accessible for analysis and aggregate reporting.
Make use of the enriched alumni data sources to develop regular
reports and analytic resources




to support program faculty and staff and others for academic
program evaluation and development.
of provide students, prospective students and their families with
information on the post-graduation experiences of UW-Madison
students.
to describe the value of a UW-Madison degree to the people of
Wisconsin, state and federal legislators, news media, accreditors,
and other external audiences.
Make use of the enriched alumni data sources to conduct research
and analysis related to academic issues and initiatives.
17
UW-Madison/WAA Alumni Data Project
Slide Credit: Joanne Berg, AVC Enrollment Mgmt 8/05
Enrollment Management Continuum
Prospective
Applicants
Applicants
For
Admissions
Admitted
Applicants
Enrolled
Students
Alumni
Graduates
PreCollege
Program
Participants
Each box represents points along the continuum when source data is
collected. Source data for individuals represented by boxes above
the line are stored in a central data source. All reports should be
generated from source data.
PreCollege
Program
Prospects
“...tracking, internal program planning and assessment, information for
development, grant writing, recruitment...” Change 2004
18
UW-Madison/WAA Alumni Data Project
What we have.
What we want.
Fragmented
alumni data
with limited
use for
programmatic
or analytical
purposes.
Potential to expand
the WAA alumni
data collection
system.
19
Unified collection
system structured for
operational and programmatic
uses.
UW-Madison/WAA Alumni Data Project
Slide Credit: Joanne Berg, AVC Enrollment Mgmt 8/05
Information Flow
Distributed Entry

Collection

Storage
Audit and Analysis

(who knows the data best *)

Reporting
 Official reporting
 Internal reporting
20
UW-Madison/WAA Alumni Data Project
Central,
shared
system
*Data
Custodians
*IR
Collecting alumni information
It’s given voluntarily.
Persuade by:
 service delivery (transcripts, career
advising, free email)
 alumni directory searches
 alumni involvements (reunions)
 ***department/program encouragement***
21
UW-Madison/WAA Alumni Data Project
Available data elements

UW-Madison Registrar’s Office:


Degree records, student records,
demographics
Wisconsin Alumni Association (WAA)
and UW Foundation

22
Alumni addresses, employment
information, spouses, engagement (WAA
membership, donations, etc)
UW-Madison/WAA Alumni Data Project
Alumni Profile
1-5 years
6-10
out
years out
% Female
% Minority
% Living in WI
% Providing employment info
% Employed, by occupation, by sector
% Employed in occupation related to major
% Who agree that UW-Madison education “made
a difference”
% Continued engagement (through WAA)
% Ever made a donation
23
UW-Madison/WAA Alumni Data Project
Alumni Profile Prototype
All bachelors major programs
 All masters major programs
 All doctoral major programs


24
See handout
UW-Madison/WAA Alumni Data Project
Data Availability and Gaps







Degree information 100%
Alumni addresses >90% (NCOA)
Employer <50%
Job title (free form) ~50%
Occupation title pick list – new
UW-Madison “made a difference”, “related to my
major” – unconnected, scattered survey data
WAA engagement, UW Foundation donor - 100%
25
UW-Madison/WAA Alumni Data Project
Policy Analysis
Geographic Distribution of Alumni
 Evidence of workforce contributions to the
WI economy
 Evidence of added value of a university
education
 How programs compare with each other

26
UW-Madison/WAA Alumni Data Project
Geographic Distribution of Alumni
Other Midwest (not incl.
WI)
•24% Overall
13% WI Resident students
47% Non-resident students
Wisconsin
•51% Overall
69% WI Resident students
11% Non-resident
students
Northeast
•8% Overall
3% WI Resident students
17% Non-resident
students
New
Information!
Pacific
•7% Overall
6% WI Resident students
11% Non-resident
students
(includes AK and HI)
West
•3% Overall
3% WI Resident students
3% Non-resident students
27
UW-Madison/WAA Alumni Data Project
See
handout.
South
•7% Overall
6% WI Resident students
10% Non-resident
students
Analytical Questions
Top 10 lists by program….
% living in WI
% engaged with the university (WAA)
% ever made a donation
% providing employment information
28
UW-Madison/WAA Alumni Data Project
Top 10 List – Job Titles Provided
% Providing Job Title Information
School/
College
Major
Code
Major Name
ALS
639
Meat and Animal Science
78
EGR
175
Civil and Environmental Engineering
77
ALS
247
Dairy Science
75
ALS
054
Agronomy
72
ALS
027
Agricultural Economics
72
NUR
712
Nursing
70
BUS
004
Accounting
69
BUS
882
Risk Management and Insurance
69
BUS
009
Actuarial Science
67
ALS
814
Poultry Science
67
29
UW-Madison/WAA Alumni Data Project
Top 10 List – Donors
% Who have ever Donated $
School/
College
Major
Code
Major Name
EGR
175
Civil and Environmental Engineering
53
ALS
247
Dairy Science
51
BUS
004
Accounting
49
ALS
639
Meat and Animal Science
45
ALS
036
Agricultural Engineering
45
EGR
711
Nuclear Engineering
44
BUS
882
Risk Management and Insurance
44
BUS
009
Actuarial Science
43
ALS
027
Agricultural Economics
43
EGR
148
Chemical Engineering
40
30
UW-Madison/WAA Alumni Data Project
Phase I – Findings





WAA, UW Foundation, Registrar are custodians
“IR” office and WAA developed a willing and energetic
partnership
Address, demographic, engagement information is
available; gaps in employment information, value
perceptions
Produced a prototype Alumni Profile for all majors
Student and alumni data can be linked and used
analytically, but within narrow limits
31
UW-Madison/WAA Alumni Data Project
Outline
Context, ideas, and bigger ideas
 Phase I – Feasibility Study (did these
ideas have a future?)
 Phase II – Development (making the
ideas work)
 Comments and advice from you

32
UW-Madison/WAA Alumni Data Project
To move ahead …
1.
2.
We need more alumni to put their
information in the WAA collection system
We need to collect more kinds of
information in a structure that can be
used analytically
33
UW-Madison/WAA Alumni Data Project
How to move ahead?
1.
2.
Convince schools/colleges/departments
to use the WAA system
Expand and improve the WAA data
collection and data base structure for
broad purposes
34
UW-Madison/WAA Alumni Data Project
Convince the
schools/colleges/departments



Academic units are pivotal in directing
graduating students to use the WAA
collection interface
Get them to share shadow systems
Need their input on expanded collections
35
UW-Madison/WAA Alumni Data Project
Barriers





Schools/colleges/departments want control
Concerns about just-in-time delivery of
information
Concerns about privacy for alumni
One-size doesn’t fit all
Time and money
36
UW-Madison/WAA Alumni Data Project
Incentives
Academic units want services
 The WAA already provides free, just-intime services
 Under-used by academic units

37
UW-Madison/WAA Alumni Data Project
WAA Services

Alumni directory look-up



Career services
Lists of alumni by program


By name, who works at Ford Motor Co, who
lives in Singapore
age, gender, race/ethnicity, geographic region,
with spousal information
http://www.uwalumni.com/lists/
38
UW-Madison/WAA Alumni Data Project
39
UW-Madison/WAA Alumni Data Project
40
UW-Madison/WAA Alumni Data Project
41
UW-Madison/WAA Alumni Data Project
Expanded WAA Collection

Employment information re-design to
support analysis


Use standard occupation/sector
formats to allow for benchmarking
Design and add survey-type questions
Unit record format
 Connects to alum and student records

42
UW-Madison/WAA Alumni Data Project
Employment Collection Structure
Possible taxonomies:
 Occupation: Standard Occupational
Classification System (SOC)
 Sector: North American Industry
Classification System (NAICS)
 US Census Current Employment Question
See Handout.
43
UW-Madison/WAA Alumni Data Project
Modified from V. Borden, NDIR, V126, 2005
Impact Indicators for Bachelors Degree Holders
Percent of Program Alumni
Indicator
<=2
years
out
Employed full time
Pursing a graduate or professional degree
Completed a graduate or professional degree
Pursing other continuing education
Employment is related to their undergraduate studies (somewhat or
directly related)
Education prepared them well for current job (somewhat or very well)
First job after graduation was related to undergraduate major(s)
(somewhat or strongly)
UW-Madison education “made a positive difference” in graduate’s life
(agree or strongly agree)
44
UW-Madison/WAA Alumni Data Project
3-5
years
out
6-10
years
out
>10
years
out
Progress Report
Sept 2005
July 2006
As planned:
 Active conversations with
colleagues in 4 of 13
schools/colleges
 Design for employer,
occupation, sector
collection progressing
 Survey questions in design
45
Unexpected:
 Improving UW-Madison’s
external image identified
as a high priority
 Provost transition – how
will leadership make a
difference?
 Partner transition –
personnel change at WAA
UW-Madison/WAA Alumni Data Project
Comments from You




Your questions for us?
What is your experience with alumni
data collections, reporting, and
analysis?
Your ideas for reducing barriers,
overcoming resistance?
Are there data sets or sources we’ve
missed?
46
UW-Madison/WAA Alumni Data Project
Selected References and
Further Reading
1.
2.
3.
4.
5.
Borden, V.H.M. 2005. Using Alumni Research to Align Program
Improvement with Institutional Accountability. New Directions for
Institutional Research. V126: 61-72.
Cabrera, A. F., D.J. Weerts, and B.J. Zulick. 2005. Making an
Impact with Alumni Surveys. New Directions for Institutional
Research. V126: 5-18.
Ewell, P.T. 2005. Alumni Studies as Instruments of Public Policy:
The US Experience. New Directions for Institutional Research.
V126: 19-30.
Pearson, J. 1999. Comprehensive Research on Alumni
Relationships: Four Years of Market Research at Stanford University.
New Directions for Institutional Research. V101: 5-21.
Pettit, J. 1999. Now What Should You Do? New Directions for
Institutional Research. V101: 101-105.
47
UW-Madison/WAA Alumni Data Project
UW-Madison / WAA
Alumni Data Project
For more information:
http://wiscinfo.doit.wisc.edu/obpa/Alumni/alumni.htm
Jocelyn Milner – jlmilner@wisc.edu
Clare Huhn – chuhn@vc.wisc.edu
48
UW-Madison/WAA Alumni Data Project