VOLUNTARY SYSTEM OF
ACCOUNTABILITY AND LEARNING
OUTCOMES: AN UPDATE
Teri Hinds
Voluntary System of Accountability
Natasha Jankowski
National Institute for Learning Outcomes Assessment
October 30, 2012
Assessment Institute






Overview
NILOA’s work with Transparency
Internal and External Communication
Voluntary System of Accountability
Evaluation of VSA College Portrait Pilot
Future of Student Learning Outcomes on College
Portrait
VSA as a communication tool
The Heat is On


Unprecedented public challenges for higher
education to be more accountable for results.
Accompanying external demands for information
about student and institutional performance are
growing calls for institutions and accreditors to
become more transparent about what they do
and the results they achieve.
(Ewell, 2010)
Overview of NILOA
NILOA’s mission is to document student learning
outcomes assessment work, identify and disseminate
best practices, and support institutions in their
assessment efforts.
SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ● OCCASIONAL PAPERS ● WEBSITE
● RESOURCES ● NEWSLETTER ● LISTSERV ● PRESENTATIONS ● TRANSPARENCY FRAMEWORK ●
FEATURED WEBSITES ● ACCREDITATION RESOURCES ● ASSESSMENT EVENT CALENDAR ●
ASSESSMENT NEWS ● MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ● ENVIRONMENTAL
SCAN ● Degree Qualifications Profile
www.learningoutcomesassessment.org
www.learningoutcomeassessment.o
Reports on Transparency
Question


What information does your institution share
publically on its website?
How are results of student learning shared
internally?
Transparency Framework
http://www.learningoutcomesassessment.org/TransparencyFramework.htm
Transparent Communication



Internal communication
External communication
Telling an institutional story about student learning
Questions


What types of stories does your institution tell about
its students?
What types of stories does your institution tell about
how it assures quality?
Background
VSA launched in 2007
 Partnership between APLU and AASCU
 320 participating institutions
 Three original objectives

 Demonstrate
accountability and transparency
 College search tool
 Support innovation in the measurement of
student learning outcomes
Student Learning Outcomes Pilot






Student learning outcomes page includes link to
institution specific learning outcomes data;
Selection of the CLA, CAAP or ETS Proficiency
Profile to measure value-added learning gains
Report results no more than four years after
joining
Link to information on test administration, sample
and response rate
Pilot period ends 12/31/12
What was learned from the pilot?
Evaluate the Effectiveness of SLO Pilot
 Focus groups
 Interviews
 Surveys
 Google Analytics
 College Portrait statistics
Findings related to Transparency


Both VSA-participating and nonparticipating institutions agree
that VSA in 2007 was a wise, timely, useful, and necessary
response to the accountability and transparency demands of the
time
Yet, nearly half of the participants have not yet met
expectations of College Portrait student learning outcomes pilot

45% of participating institutions have yet to post student

learning outcomes information
Information that was provided on student learning outcomes
lacked explanatory or contextual material

82% of participating institutions had traffic on their
student learning pages but almost 1 in 5 did not have
a single visitor to their student learning outcomes page
since 2009.
College Cost Estimator
College Cost and Financial Aid
Students
Admissions
Student Success and Progress
Classes & Campus Life
Majors, Graduation, & Next
Student Learning Outcomes
Student Experiences
0
20
40
Percentage of Total Views
60


Information posted on the College Portrait may
not reflect the needs of prospective students and
families or provide the information they actually
seek to make decisions about where to attend
college.
The authorized standardized test measures of
student learning outcomes lack broad credibility
and acceptance in the higher education
community – it is thought the tests do not
accurately reflect student performance within
the institution.


Concerns about displaying results include:
 Inability to understand what scores represent
 Questions regarding what tests measure
 Issues of reliability and validity
 Difficulty in using scores to improve student learning
 Motivation of students & difficulty to obtain a sample
 Questions regarding quality of results
81% of survey respondents stated that the pilot requirements did not
align with their assessment practices and that the results were not
usable for campus improvement efforts

Participating institutions agree - expanding the
number & nature of student learning measures will:



produce more accurate portrayals of student attainment,
provide more useful information for campuses,
make information more meaningful for all audiences.
Recommendations






Focus on specific audiences and communicating
meaningful information to them
Contextualize information
Expand range of assessment tools and approaches
Increase traffic to site
Include examples of use of evidence to improve learning
Consider creating template which could be used by all
postsecondary institutions
Working Group Deliberations
 Technical work group considered alternative
measures of learning outcomes to recommend to
VSA Board
 Confirmed importance of student learning
outcomes reporting within the VSA
 No perfect of measure of students learning exists
for all audiences
 External
accountability
 Institutional improvement
 College selection
Working Group Deliberations
 Make value-add measurement using existing
measures (CAP, CLA, ETS Proficiency Profiler)
optional
 Introduce option to use VALUE Rubrics based on
AAC&U essential learning outcomes (written
communication and critical thinking)
 Introduce option to use aggregate scores from
professional and graduate admissions exams
such as the GRE, GMAT, LSAT, and MCAT
Work Group Ratings of Recommended VSA Instruments in Key Areas
Instrument
1= lowest rating
10=highest rating
CLA
CAAP
ETS PP
(short
form)
Allow comparisons, benchmarking across
institutions?
6
8
9.5
9.5
6
7
Offer a representative sample?
7
7
7
7
7
2
inst-2
class-7
5
2
2
9
3
Transparency of method, results
6
8
8
8
9
8
External accountability
8
7
7
7
5
5.5
Ease of administration on campus
5
6
7
6
4
9.9
Reasonable costs (time, resources)
5
7
7
6
startup -3
Motivation for students to do well?
3
2
3
2
3
2
3
2
5.5
6
6
6
Usefulness to faculty/students for learning
improvement?
Likely interest to consumers?
Valid learning outcomes instrument for
average student?
ETS PP
(long
form)
VALUE
Rubrics
GRE
marginal- 8
9
9
9.9
5
6
portfolios -9
assign -7
3
VSA Board Decisions



College Portrait re-focus from “college selection tool for
prospective students” to “consumer information tool for
institutions to demonstrate effectiveness of educational
programs.”
 Primary audiences: state officials, policy makers, consumers
Instrument options:
 CAAP, CLA, ETS Proficiency Profile
 AAC&U VALUE rubrics – critical thinking, written
communication
 GRE General Test
Reporting options: value-added, benchmarking
Implementation/Next Steps

All VSA institutions report results of pilot by end of 2012
to fulfill reporting requirement
 3 part report with tabbed display
 Did
you participate (or try to)? If so, how’d it go? If not, why
not?
 Give an example of another institutional-level evaluation
 Which of the new options are you considering for the future?
 Reporting
Format Preview:
http://www.collegeportraits.org/slo_preview.html
Implementation/Next Steps

Communications Task Force working with College Portrait
Support Team on display of new options
Emphasis on more graphical displays
 Additional guidance on what the scores and results mean
 Click-through links for additional details, e.g., subscores



New options released for 2012-13 data cycle (Jan 2013)
Develop outreach strategy to recruit new participants and
connect with key audiences – state officials, accreditors
Websites
www.voluntarysystem.org
www.collegeportraits.org
www.learningoutcomesassessment.org
Teri Hinds: thinds@aplu.org
Natasha Jankowski: njankow2@illinois.edu
Let’s Discuss.