VOLUNTARY SYSTEM OF ACCOUNTABILITY AND LEARNING OUTCOMES: AN UPDATE Teri Hinds Voluntary System of Accountability Natasha Jankowski National Institute for Learning Outcomes Assessment October 30, 2012 Assessment Institute Overview NILOA’s work with Transparency Internal and External Communication Voluntary System of Accountability Evaluation of VSA College Portrait Pilot Future of Student Learning Outcomes on College Portrait VSA as a communication tool The Heat is On Unprecedented public challenges for higher education to be more accountable for results. Accompanying external demands for information about student and institutional performance are growing calls for institutions and accreditors to become more transparent about what they do and the results they achieve. (Ewell, 2010) Overview of NILOA NILOA’s mission is to document student learning outcomes assessment work, identify and disseminate best practices, and support institutions in their assessment efforts. SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ● OCCASIONAL PAPERS ● WEBSITE ● RESOURCES ● NEWSLETTER ● LISTSERV ● PRESENTATIONS ● TRANSPARENCY FRAMEWORK ● FEATURED WEBSITES ● ACCREDITATION RESOURCES ● ASSESSMENT EVENT CALENDAR ● ASSESSMENT NEWS ● MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ● ENVIRONMENTAL SCAN ● Degree Qualifications Profile www.learningoutcomesassessment.org www.learningoutcomeassessment.o Reports on Transparency Question What information does your institution share publically on its website? How are results of student learning shared internally? Transparency Framework http://www.learningoutcomesassessment.org/TransparencyFramework.htm Transparent Communication Internal communication External communication Telling an institutional story about student learning Questions What types of stories does your institution tell about its students? What types of stories does your institution tell about how it assures quality? Background VSA launched in 2007 Partnership between APLU and AASCU 320 participating institutions Three original objectives Demonstrate accountability and transparency College search tool Support innovation in the measurement of student learning outcomes Student Learning Outcomes Pilot Student learning outcomes page includes link to institution specific learning outcomes data; Selection of the CLA, CAAP or ETS Proficiency Profile to measure value-added learning gains Report results no more than four years after joining Link to information on test administration, sample and response rate Pilot period ends 12/31/12 What was learned from the pilot? Evaluate the Effectiveness of SLO Pilot Focus groups Interviews Surveys Google Analytics College Portrait statistics Findings related to Transparency Both VSA-participating and nonparticipating institutions agree that VSA in 2007 was a wise, timely, useful, and necessary response to the accountability and transparency demands of the time Yet, nearly half of the participants have not yet met expectations of College Portrait student learning outcomes pilot 45% of participating institutions have yet to post student learning outcomes information Information that was provided on student learning outcomes lacked explanatory or contextual material 82% of participating institutions had traffic on their student learning pages but almost 1 in 5 did not have a single visitor to their student learning outcomes page since 2009. College Cost Estimator College Cost and Financial Aid Students Admissions Student Success and Progress Classes & Campus Life Majors, Graduation, & Next Student Learning Outcomes Student Experiences 0 20 40 Percentage of Total Views 60 Information posted on the College Portrait may not reflect the needs of prospective students and families or provide the information they actually seek to make decisions about where to attend college. The authorized standardized test measures of student learning outcomes lack broad credibility and acceptance in the higher education community – it is thought the tests do not accurately reflect student performance within the institution. Concerns about displaying results include: Inability to understand what scores represent Questions regarding what tests measure Issues of reliability and validity Difficulty in using scores to improve student learning Motivation of students & difficulty to obtain a sample Questions regarding quality of results 81% of survey respondents stated that the pilot requirements did not align with their assessment practices and that the results were not usable for campus improvement efforts Participating institutions agree - expanding the number & nature of student learning measures will: produce more accurate portrayals of student attainment, provide more useful information for campuses, make information more meaningful for all audiences. Recommendations Focus on specific audiences and communicating meaningful information to them Contextualize information Expand range of assessment tools and approaches Increase traffic to site Include examples of use of evidence to improve learning Consider creating template which could be used by all postsecondary institutions Working Group Deliberations Technical work group considered alternative measures of learning outcomes to recommend to VSA Board Confirmed importance of student learning outcomes reporting within the VSA No perfect of measure of students learning exists for all audiences External accountability Institutional improvement College selection Working Group Deliberations Make value-add measurement using existing measures (CAP, CLA, ETS Proficiency Profiler) optional Introduce option to use VALUE Rubrics based on AAC&U essential learning outcomes (written communication and critical thinking) Introduce option to use aggregate scores from professional and graduate admissions exams such as the GRE, GMAT, LSAT, and MCAT Work Group Ratings of Recommended VSA Instruments in Key Areas Instrument 1= lowest rating 10=highest rating CLA CAAP ETS PP (short form) Allow comparisons, benchmarking across institutions? 6 8 9.5 9.5 6 7 Offer a representative sample? 7 7 7 7 7 2 inst-2 class-7 5 2 2 9 3 Transparency of method, results 6 8 8 8 9 8 External accountability 8 7 7 7 5 5.5 Ease of administration on campus 5 6 7 6 4 9.9 Reasonable costs (time, resources) 5 7 7 6 startup -3 Motivation for students to do well? 3 2 3 2 3 2 3 2 5.5 6 6 6 Usefulness to faculty/students for learning improvement? Likely interest to consumers? Valid learning outcomes instrument for average student? ETS PP (long form) VALUE Rubrics GRE marginal- 8 9 9 9.9 5 6 portfolios -9 assign -7 3 VSA Board Decisions College Portrait re-focus from “college selection tool for prospective students” to “consumer information tool for institutions to demonstrate effectiveness of educational programs.” Primary audiences: state officials, policy makers, consumers Instrument options: CAAP, CLA, ETS Proficiency Profile AAC&U VALUE rubrics – critical thinking, written communication GRE General Test Reporting options: value-added, benchmarking Implementation/Next Steps All VSA institutions report results of pilot by end of 2012 to fulfill reporting requirement 3 part report with tabbed display Did you participate (or try to)? If so, how’d it go? If not, why not? Give an example of another institutional-level evaluation Which of the new options are you considering for the future? Reporting Format Preview: http://www.collegeportraits.org/slo_preview.html Implementation/Next Steps Communications Task Force working with College Portrait Support Team on display of new options Emphasis on more graphical displays Additional guidance on what the scores and results mean Click-through links for additional details, e.g., subscores New options released for 2012-13 data cycle (Jan 2013) Develop outreach strategy to recruit new participants and connect with key audiences – state officials, accreditors Websites www.voluntarysystem.org www.collegeportraits.org www.learningoutcomesassessment.org Teri Hinds: [email protected] Natasha Jankowski: [email protected] Let’s Discuss.