LEARNING-CENTERED ASSESSMENT: MODELS & STRATEGIES FOR ASSESSING INSTITUTIONAL LEARNING OUTCOMES Ashley Finley, Ph.D Senior Director of Assessment & Research, AAC&U National Evaluator, Bringing Theory to Practice Kutztown University January 19, 2012 WHY ARE WE HERE? What does it mean to be What are you doing? Why are you doing it? How do you know you’re doing it well? learning-centered at KU? What practices promote and advance student learning? How pervasive are these? What is the change you want to see in your students when they leave the institution? What are the foundational skills and competencies students should acquire throughout their learning experiences? What are the effective approaches for implementing a cycle of inquiry and improvement? THE NATIONAL WORK ON LEARNING-CENTERED ASSESSMENT, LIBERAL EDUCATION OUTCOMES, & GENERAL EDUCATION Liberal Education: A philosophy of learning that empowers & prepares individuals to deal with complexity, diversity, & change. = Broad knowledge combined w/ in-depth study To help students develop a sense of social responsibility, strong & transferable intellectual and practical skills & a demonstrated ability to apply knowledge. Vast majority of institutions have articulated learning outcomes ANY institution can achieve the goals of a liberal education – 3 Major National Initiatives Compass Project (State University Systems – CA, OR, WI) Quality Collaboratives VALUE Rubric Project WHAT EMPLOYERS SAY ABOUT LIBERAL EDUCATION SKILLS: % of Employers who agree with each statement • Our company is asking employees to take on more responsibilities and to use a broader set of skills than in the past 91% • Employees are expected to work harder to coordinate with other departments than in the past 90% • The challenges employees face within our company are more complex today than they were in the past 88% • To succeed in our company, employees need higher levels of learning and knowledge today than they did in the past 88% Source: Raising the Bar (AAC&U, 2010) % of Employers who agree that two- and four-year colleges should place MORE emphasis on helping students develop the following: Effective oral/written communication Critical thinking/ analytical reasoning Knowledge/skills applied to real world settings Analyze/solve complex problems Connect choices and actions to ethical decisions Teamwork skills/ ability to collaborate Ability to innovate and be creative Concepts/developments in science/technology 89% 81% 79% 75% 75% 71% 70% 70% Source: “Raising the Bar: Employers’ Views on College Learning in the Wake of the Economic Downturn” (AAC&U and Hart Research Assoc. 2010) AND IN THESE AREAS… Locate/organize/evaluate information 68% Understand global context of situations/ decisions Global issues’ implications for future 67% Understand & work with numbers/statistics 65% 63% Understand role of U.S. in the world 57% Knowledge of cultural diversity in US/world 57% Civic knowledge, community engagement 52% “Raising the Bar: Employers’ Views on College Learning in the Wake of the Economic Downturn” (AAC&U and Hart Research Assoc. 2010) HOW MUCH DOES A LIBERAL EDUCATION PAY?: “IMAGINATION IS MORE IMPORTANT THAN KNOWLEDGE” -ALBERT EINSTEIN Mean Earnings: Use of Speaking Skills earnings 60,000 30,000 Earni… 0 q1(low) q2 q3 q4 q5(high) quintiles Source: Georgetown University, Center for Education and the Workforce (Anthony Carnavale) Mean Earnings: Use of Originality earnings 60,000 Earnings 30,000 0 q1(low) q2 q3 quintiles q4 q5(high) Source: Georgetown University, Center for Education and the Workforce Mean Earnings: Use of Judgement & Decision-Making Skills earnings 70,000 35,000 Earnings 0 q1(low) q2 q3 q4 q5(high) quintiles Source: Georgetown University, Center for Education and the Workforce earnings 60,000 Mean Earnings: Use of Social Skills 30,000 Earnings 0 q1(low) q2 q3 q4 q5(high) quintiles Source: Georgetown University, Center for Education and the Workforce Articulating Student Learning and Development Content Outcomes •Theoretical traditions •Methodology •Topical areas How do we assess these? • Course Exams •Course Papers •Individual/ Group course projects •In-Class work/ participation Learning Outcomes •???? •???? •???? Kutztown Mission: “provide a high quality education at the undergraduate and graduate levels in order to prepare students to meet lifelong intellectual, ethical, social, and career challenges.” How do we assess these? •Goal 1: Cultivate intell. & pract. skills •Knowledge of Human •Inquiry & Analysis Cultures & the Physical & •Decision making Natural World •Written & Oral Communication •Content areas •Quantitative Literacy Institutional •Intellectual & Practical •Information Literacy Skills •Teamwork & problem solving •Inquiry & Analysis •Wellness •Practiced throughout curr, w/ progressively •Critical & Creative Thinking more challenging probs, projects & standards •Written & Oral Comm. •Goal 2: Develop understanding of human •Reading cultures & physical and natural world •Quantitative Literacy •Sciences, mathematics, social sciences, •Information Literacy humanities, histories, languages, arts •Teamwork & Problem-solving •Engage w/ big questions, contemp. & enduring •Personal & Social Resp. •Goal 3: Sense of personal & social resp. •Civic Knowledge •Local/Global civic knowledge •Intercultural Knowledge & •Intercultural knowledge & competence Competence •Ethical reasoning & action •Ethical Reasoning •Personal qualities & attitudes (passion, •Lifelong Learning curiosity, self-confidence, imagination, •Integrative & Applied cooperation, commitment, support) •Active inv. w/diverse comm. & real world chall. Learning Outcomes Gen.Ed./ Major Course What is the story you want to tell about student learning? Assess Learning Outcomes Learning Outcomes Curriculum Assessment Outcomes What is the story you want to tell about student learning? Assess Learning Outcomes Learning Outcomes Basic Logic Model INPUTS ACTIVITIES Resources are needed Actions or for activities processes – What is necessary to needed to produce start or keep outputs – something What going? activities •Human need to occur • Financial to produce •Technological evidence? •Curricular/ Program OUTPUTS OUTCOMES Products used to assess outcomes What can be counted as evidence of change? Expected Changes and Benefits – What are the goals you want to reach? Assumptions • Short-term • Intermediate • Long-term Sample Logic Model INPUTS •GE •Depts. •Advising •Writing Center •Stud. Aff. •Health & Wellness Center •Alumni Engagement •Multicult. Services •Athletics •Distance Ed •Career Services •Inst. Res. ACTIVITIES •Experiential learning •Civic learning •Interaction •Engagement in life’s big questions •Practiced extensively & progressively •Active involve. In diverse comm. & real-world challenges OUTCOMES OUTPUTS • Goals & •Signature Domains of assignments Learning •Reflection • Understanding papers of human •Group cultures… projects • Intellectual •Communityand practice based skills projects • Critical thinking • Personal & Social Responsibility • Civic engagement VALUE PROJECT (WWW.AACU.ORG/VALUE) Project Goals 1) Create dialogue and develop shared understanding of common learning outcomes 2) Create template for direct assessment of student work (in text and non-text formats) 3) Create student-friendly format to engage students in selfevaluation Rubric Development Teams of faculty/scholars nationwide Reviewed existing rubrics to identify commonalities, clarify language, and develop broad agreement on outcomes criteria (openedpractices.org) To date accessed by over 3000 institutions/organizations, 11,000 individuals Domestic & international, K-12, state systems Reliability study LIST OF VALUE RUBRICS Knowledge of Human Cultures & the Physical & Natural Worlds Civic Knowledge & Engagement Intercultural Knowledge & Competence Ethical Reasoning Foundations & Skills for Lifelong Learning Content Areas No Rubrics Intellectual and Practical Skills Inquiry & Analysis Critical Thinking Creative Thinking Written Communication Oral Communication Reading Quantitative Literacy Information Literacy Teamwork Problem-solving Personal & Social Responsibility Integrative & Applied Learning Integrative & Applied Learning Located at: http://www.aacu.org/value/index.cfm The Anatomy of a VALUE Rubric Criteria Levels Performance Descriptors THE CALIBRATION TRAINING PROCESS Scoring Steps: Review rubric to familiarize yourself with structure, language, performance levels Ask questions about the rubric for clarification or to get input from others regarding interpretation Read student work sample Connect specific points of evidence in work sample with each criterion at the appropriate performance level (if applicable) Calibration Steps: Review scores Determine common score(s) Hear from outliers Discuss Determine final score THE GROUND RULES This is not grading. We are not changing the rubric (today). Our work is time sensitive. Go with your instinct. Think globally about student work and about the learning skill. Think beyond specific disciplinary lenses or content. Start with 4 and work backwards. Pick one performance benchmark per criterion. Avoid “.5”. Zero does exist. Assign “0” if work does not meet benchmark (cell one) performance level. N/A exists. Assign “not applicable” if the student work is not intended to meet a particular criterion. SIGNATURE ASSIGNMENTS Assignment should enable attainment of criteria Break down criteria to determine key components for assignment What should students do with content to meet criteria? E.g. What are the pieces to be analyzed, compared, integrated? Will the assignment be used for more than one outcome? What are the types of assignments that will be most helpful for allowing students to demonstrate competency?