Christine Keller
Voluntary System of Accountability
Paul Gore
University of Utah
Session #19
Southern Association for Institutional Research
Orlando, Florida ▪ September 22-25, 2012
• Review of original goals of VSA and student
learning outcomes
• Selected findings and recommendations from
NILOA evaluation of VSA Pilot
• Application of evaluation results: technical
work group, communications advisory group
• Next steps for VSA: wrap up pilot, new
reporting options, refocus/rebranding
Initiative by public universities to supply
straightforward, comparable information on
the undergraduate
student experience
through a common
web report – the
College Portrait.
www.collegeportraits.org
• VSA developed, launched in 2007
• Sponsored by APLU and AASCU
• 320 participating public universities
• Three original objectives
– Demonstrate accountability, transparency
– College search tool
– Support measurement and reporting of
student learning outcomes
• Goal: Directly measure, publicly report
student learning gains (value-added) at
institution level using a common method
– Skills: critical thinking, analytic reasoning,
problem solving, written communication
– Tests: CAAP, CLA, ETS Proficiency Profile
• Reporting includes link to institution
specific learning outcomes data
• Pilot period ends December 2012
• Conducted by National Institute for Learning
Outcomes Assessment (NILOA)
– Focus groups
– Interviews
– Surveys
– Google Analytics
– College Portrait statistics
NILOA’s mission is to document student
learning outcomes assessment work, identify
and disseminate best practices, and support
institutions in their assessment efforts.
SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ● OCCASIONAL
PAPERS ● WEBSITE ● RESOURCES ● NEWSLETTER ● LISTSERV ● PRESENTATIONS
● TRANSPARENCY FRAMEWORK ● FEATURED WEBSITES ● ACCREDITATION
RESOURCES ●
ASSESSMENT EVENT CALENDAR ●
ASSESSMENT NEWS ●
MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ● ENVIRONMENTAL SCAN
www.learningoutcomesassessment.org
• Both participating and nonparticipating
institutions agree that VSA in 2007 was a wise,
timely, useful, necessary response to the
accountability , transparency demands of the time
• Many eligible institutions - about 1/3 - do not
participate in the VSA
• Nearly half of the participants have not yet met
expectations of student learning outcomes pilot
– 50% of participating institutions have yet to post
student learning outcomes information (Sept 2012)
College Cost Estimator
College Cost and Financial Aid
Students
Admissions
Student Success and Progress
Classes & Campus Life
Majors, Graduation, & Next
Student Learning Outcomes
Student Experiences
0
20
40
Percentage of Total Views
60
• College Portrait: “Information posted may
not reflect the needs of prospective students,
families or provide the information they seek to
make decisions about where to attend college.”
• Student Learning Outcomes: “The
standardized test measures of student learning
outcomes lack broad credibility and acceptance
in the higher education community,
undermining institutional participation and
engagement with the VSA and campus faculty
and staff support of the VSA initiative.”
“Mend it, don’t end it”
• Expand range of assessment tools and
approaches
• Focus on specific audiences and
communicating meaningful information
• Technical work group considered alternative
measures of learning outcomes to recommend
to VSA Board
• Confirmed importance of student learning
outcomes reporting within the VSA
• No perfect of measure of student learning
exists for all audiences
– External accountability
– Institutional improvement
– College selection
• Continue use of value-added measurement
using CAAP, CLA, ETS Proficiency Profile
• Introduce option to use VALUE Rubrics based
on AAC&U essential learning outcomes
– written communication
– critical thinking
• Introduce option to use aggregate scores from
professional and graduate admissions exams
such as the GRE, GMAT, LSAT, and MCAT
Work Group Ratings of Recommended VSA Instruments in Key Areas
Instrument
1= lowest rating
10=highest rating
CLA
CAAP
ETS PP
(short
form)
Allow comparisons, benchmarking across
institutions?
6
8
9.5
9.5
6
7
Offer a representative sample?
7
7
7
7
7
2
inst-2
class-7
5
2
2
9
3
Transparency of method, results
6
8
8
8
9
8
External accountability
8
7
7
7
5
5.5
Ease of administration on campus
5
6
7
6
4
9.9
Reasonable costs (time, resources)
5
7
7
6
startup -3
Motivation for students to do well?
3
3
3
3
9
9.9
Likely interest to consumers?
2
2
2
2
5
6
Valid learning outcomes instrument for
average student?
5.5
6
6
6
portfolios -9
assign -7
3
Usefulness to faculty/students for learning
improvement?
ETS PP
(long
form)
VALUE
Rubrics
GRE
marginal- 8
9
• College Portrait re-focus from “college
selection tool” to “consumer information
tool to demonstrate effectiveness of
educational programs”
• Instrument options:
– CAAP, CLA, ETS Proficiency Profile
– AAC&U VALUE rubrics – critical thinking,
written communication
– GRE General Test
• Reporting options: value-added, benchmarking
• VSA participants describe, publish pilot
project experiences using three part form
– Required for institutions with no SLO results
published
– Strongly encouraged for all institutions
– Data entry opens week of Oct 1, closes Jan 7
• New SLO reporting options released for 201213 data cycle (week of Jan 18, 2013)
– Deadline: Spring of 2015
– No data more than 3 years old
• Report overall experiences participating in
the pilot – whether 1 of the 3 value-added
tests was administered or not.
• Provide information about an additional
institutional assessment initiative on
campus, including results from a recent
administration and/or cycle.
• Disclose which of the newly approved SLO
reporting options your campus is likely to
consider for future VSA reporting.
http://www.collegeportraits.org/slo_preview.html
• Includes individuals from public affairs,
communications, government affairs
– Develop outreach strategy to recruit new
participants and connect with key audiences
– Identify and promote key elements on
College Portrait: success and progress rate,
cost of attendance, financial aid, future
plans
– Create streamlined view targeted at policy
makers, government affairs professional
• Unprecedented public challenges for higher
education to be more accountable for
results.
• Accompanying external demands for
information about student and institutional
performance are growing calls for
institutions and accreditors to become
more transparent about what they do and
the results they achieve. (Ewell, 2010)
Reports & Presentations
www.voluntarysystem.org
Christine Keller, VSA Executive Director
ckeller@aplu.org
Teri Hinds, VSA Associate Director
thinds@aplu.org
Paul Gore, U of Utah
paul.gore@utah.edu