Christine Keller
VSA Executive Director
Teri Lyn Hinds
VSA Associate Director
Higher Learning Commission Annual Conference
Chicago, IL ▪ April 7, 2013
• Review original goals of VSA, student learning
outcomes pilot project
• Discuss selected findings, recommendations
from NILOA evaluation of SLO Pilot, participant
reports
• Describe use of evaluation results: technical
work group, communications advisory group
• Outline next steps for VSA: new reporting
options, refocus/rebranding
Initiative by public universities to supply
straightforward, comparable information on
the undergraduate
student experience
through a common
web report – the
College Portrait.
www.collegeportraits.org
• Developed, launched in 2007
– 70 institutions, 80 leaders
– APLU and AASCU partnership
• Three original objectives
– Demonstrate accountability, transparency
– College search tool
– Support innovation in the measurement,
reporting of student learning outcomes
• 300 participating public universities
• Goal: Directly measure, publicly report
student learning gains (value-added) at
institution level using a common method
– Skills: critical thinking, analytic reasoning,
problem solving, written communication
– Tests: CAAP, CLA, ETS Proficiency Profile
• Reporting includes link to institution
specific learning outcomes data
• Pilot period ended December 2012
• Conducted by National Institute for Learning
Outcomes Assessment (NILOA)
– Focus groups
– Interviews
– Surveys
– Google Analytics
– College Portrait statistics
NILOA’s mission is to document student
learning outcomes assessment work, identify
and disseminate best practices, and support
institutions in their assessment efforts.
SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ● OCCASIONAL
PAPERS ● WEBSITE ● RESOURCES ● NEWSLETTER ● LISTSERV ● PRESENTATIONS
● TRANSPARENCY FRAMEWORK ● FEATURED WEBSITES ● ACCREDITATION
RESOURCES ●
ASSESSMENT EVENT CALENDAR ●
ASSESSMENT NEWS ●
MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ● ENVIRONMENTAL SCAN
www.learningoutcomesassessment.org
• VSA in 2007 was a wise, timely, useful,
necessary response to the accountability ,
transparency demands of the time
• Many eligible institutions - about 1/3 - do not
participate in the VSA
• 50% of participating institutions missing
student learning outcomes information
• The student learning outcomes section of
attracts little traffic
• College Portrait: “Information posted may
not reflect the needs of prospective students,
families or provide the information they seek to
make decisions about where to attend college.”
• Student Learning Outcomes: “The
standardized test measures of student learning
outcomes lack broad credibility and acceptance
in the higher education community,
undermining institutional participation and
engagement with the VSA and campus faculty
and staff support of the VSA initiative.”
“Mend it, don’t end it”
• Expand range of assessment tools and
approaches
• Focus on specific audiences and
communicating meaningful information
• Technical work group considered alternative
measures of learning outcomes to recommend
to VSA Board
• Confirmed importance of student learning
outcomes reporting within the VSA
• No perfect of measure of student learning
exists for all audiences
– External accountability
– Institutional improvement
– College selection
Work Group Ratings of Recommended VSA Instruments in Key Areas
Instrument
1= lowest rating
10=highest rating
CLA
CAAP
ETS PP
(short
form)
Allow comparisons, benchmarking across
institutions?
6
8
9.5
9.5
6
7
Offer a representative sample?
7
7
7
7
7
2
inst-2
class-7
5
2
2
9
3
Transparency of method, results
6
8
8
8
9
8
External accountability
8
7
7
7
5
5.5
Ease of administration on campus
5
6
7
6
4
9.9
Reasonable costs (time, resources)
5
7
7
6
startup -3
Motivation for students to do well?
3
3
3
3
9
9.9
Likely interest to consumers?
2
2
2
2
5
6
Valid learning outcomes instrument for
average student?
5.5
6
6
6
portfolios -9
assign -7
3
Usefulness to faculty/students for learning
improvement?
ETS PP
(long
form)
VALUE
Rubrics
GRE
marginal- 8
9
• Continue use of value-added measurement
using CAAP, CLA, ETS Proficiency Profile
• Introduce option to use VALUE Rubrics
based on AAC&U essential learning
outcomes
• Introduce option to use aggregate scores
from professional and graduate admissions
exams such as the GRE, GMAT, LSAT, and
MCAT
Broader Focus & Outreach
– Evolution from college selection website to
consumer information tools
– New audience focus: state officials,
accreditors
Expanded Learning Outcomes Options
– Original tests - CAAP, CLA, ETS PP
– Plus AAC&U VALUE rubrics – critical thinking,
written communication
– Both value-added, benchmarked reporting
• Pilot project wrap-up: VSA participants
describe, publish pilot project
experiences using three part form
• 92 institutions completed the Pilot
Experience Report
– 65 participated – administered or tried to
administer one of the instruments
– 27 did not participate
• 27 institutions posted results to their
College Portrait
– 16 found the results useful on campus
– CLA was the most frequently reported
• 38 institutions tried to administer an
instrument
– 14 reported difficulty meeting minimum sample
size requirements
– 8 reported campus concerns with sample sizes,
even when they met publisher recommendations
• 27 institutions did not participate
– Campus perceptions that the instruments
wouldn’t provide useful results at 9
institutions
– 7 institutions has more pressing institutional
priorities
– 6 institutions were concerned about the
monetary cost of the instruments
• Institutions asked to select all options under
consideration by campus
– 28 institutions considering more than one
option
– Many institutions exploring benchmarking and
value-added for the same instrument
• 44 institutions considering the ETS PP
– Either/both value-added, senior-only benchmark
administrations
• 34 institutions considering AAC&U VALUE
Rubrics
• Form Communication Advisory Group from
public affairs, communications, government
affairs
• Promote key elements on College Portrait:
success and progress rate, cost of
attendance, net price calculator, financial
aid, future plans
• Re-connect with key audiences: assessment
experts, state/system officials, accreditors
• Institutions can publish Success &
Progress Rate graphs and tables for
customized student groups
– Part-time students, veterans, Pell-recipients
• Augmented cost, financial aid info
– Net price and financial aid by income range,
average indebtedness, average loan
payment
• VSA Affordability Estimator improvements
provide more accurate net price
estimates, clearer presentation of results
(May)
• Learning Outcomes Pilot Project Report
will summarize campus experiences,
lessons from 4-year pilot (May)
• At-a-Glance tool for government affairs
staff will offer streamlined, summarized
view of key indicators (August)
• Unprecedented challenges for public
higher education to be more
accountable and more transparent
concerning value, outcomes
• Work to position the VSA/College
Portrait as a tool to respond to those
challenges
College Portrait website:
www.collegeportraits.org
Reports & Presentations:
www.voluntarysystem.org
Twitter: @VSAvoice
Blog: http://vsavoice.blogspot.com/
Christine Keller
Executive Director
ckeller@aplu.org
Teri Lyn Hinds
Associate Director
thinds@aplu.org