The Nuts and Bolts of Outcome Assessment Terri Manning Denise Wells Planning and Research Central Piedmont Community College Much confusion! There are so many words being used in education right now that involve some sort of measurement that people are generally confused, throw up their hands and walk away. What are the differences if any between testing, measurement and assessment? Are objectives and outcomes the same thing? What is the difference between a learning outcome, an outcome target, an educational output, an outcome objective….. and the list goes on and on…… Our Task…... 1. To explain all these definitions (with examples) so you can understand them. 2. To prepare you to guide your department/unit in establishing and measuring outcomes. Stages of Grief for Outcome Measurement Stage 4 Stage 5 Acceptance & adaptation Challenge & competition Catalyst - Proactive Depression Compliance - Passive reactive Stage 3 Bargaining - no time/no money Seek outside sources Stage 2 Anger and antagonism Resistant & Reactive Stage 1 Disbelief & Denial Paralysis - Passive resistance In Education… We’ve learned that things come and they go Most of these trends are purely academic exercises What we do on a daily basis doesn’t change regardless of the mission, vision and goals There will be new “edu-babble” things to do next year Changes in Higher Education We have moved from: a culture of unexamined assumptions a culture of implicit, individually held hopes, preferences and beliefs a model of higher education as primarily a quantitative, additive process To: a culture of assessment and evidence a culture of explicit broadly shared goals, criteria and standards a model that is fundamentally qualitative and transformative Changes in Higher Education, continued We have moved from: a culture that tends to ignore costs a teaching culture which ignores what is known about human learning a culture that emphasizes and privileges individual struggle for private advantage To: a culture that attempts to realistically account for direct, deferred and opportunity costs one which applies relevant knowledge to improve learning one which encourages collaboration for the common good and individual advancement Long Story Short…. Accountability in higher education is here to stay! Measuring Outcomes is here to stay and makes good sense! The national educational climate is skeptical about accountability! They want us to prove that students are learning, that their lives are improving and that we are good stewards of funds! The Spellings Commission Some things the draft report called for: The creation of an overall measurement of an institution's "bottom line," including measures of institutional costs and performance that let parents and policy makers view institutional results; A mandate that institutions measure student learning outcomes, disseminate the results to students, and report them publicly in the aggregate; The development of a national student unit-record database to follow the progress of each student; The establishment a national accreditation framework that includes comparable performance measures, and making the findings of reviews easily accessible to the public; SACS Requirements and Proposed Changes Core Requirement 2.5 (from 2003) The institution engages in ongoing, integrated and institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (1) results in continuing improvement and (2) demonstrates that the institution is effectively accomplishing its mission (Institutional Effectiveness.) Proposed Change 12/06 The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that (1) incorporate a systematic review of institutional mission, goals and outcomes; (2) result in continuing improvement in institutional quality and (3) demonstrate that the institution is effectively accomplishing its mission (Institutional Effectiveness) SACS, continued Comprehensive Standard 3.3.1 states “The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.” Proposed and Tabled – (12/06) The institution identifies expected outcomes, assesses whether it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: educational programs, to include student learning outcomes at the program and individual level administrative support services educational support services research within its educational mission, if appropriate community/public service within its educational mission, if appropriate SACS, continued Comprehensive Standard 3.4.1 The institution demonstrates that each educational program for which academic credit is awarded (a) is approved by the faculty and (b) establishes and evaluates program and learning outcomes. Changed (12-06) (Educational programs: all educational programs (includes all on-campus, off-campus and distance learning programs and course work) The institution demonstrates that each educational program for which academic credit is awarded is approved by the faculty and the administration. SACS, continued Comprehensive Standard 3.5.1 The institution identifies college-level competencies within the general education core and provide evidence that graduates have attained those competencies. Changed (12-06) The institution identifies college-level general education competencies and uses best practices in assessment to provide evidence that graduates have attained them. So an institution needs to have: Ongoing, integrated, and institution-wide research-based planning and evaluation processes which includes: A list of expected outcomes, the assessment of those outcomes and evidence of improvement based on analysis of those outcome results in each of the following areas: a systematic review of institutional mission, goals and outcomes results in continuing improvement in institutional quality demonstrates that the institution is effectively accomplishing its mission educational programs (student learning outcomes at the program and individual level) administrative support services educational support services Identified college-level general education competencies (based on best practices in assessment) and provide evidence that graduates have attained them But Why? Are we doing this only because of SACS? Who is SACS? Shouldn’t we periodically take a serious look at our students? Are they learning? Who is learning best? Are they achieving the outcomes we expected? Should we make changes in programs and services? Do we need more in-depth services? Do we need a new curriculum or a change in methodology? The Great Fallacy Grades In this day of social promotion, grade inflation and different teaching/learning philosophies, grades tell you virtually nothing. They are not a measure of outcome achievement. Two teachers will grade a student differently for the exact same work. They cannot be used! How it Works in Most Schools… Mission Values Philosophy Goals Unit Objectives Goals Unit Objectives Outcome Objectives Unit Objectives Outcome Objectives Goals Unit Objectives Outcome Objectives Unit Objectives Outcome Objectives Unit Objectives Outcome Objectives Mission Statement It is the first step in any planning process It establishes an identity for the organization that is more expansive than what one individual or small group can bring to the organization It has to be an enduring vision of future direction and values It identifies the organizations worthwhile work Mission Statement Helps the organization accomplish goals, receive recognition, accomplish set tasks, observe program success, move the organization forward and earn respect Is wide in scope, directional in purpose, and allows for a wide range of goals It should be broad enough to allow for creativity, expansion and new possibilities The Mission Statement Misunderstanding…. It doesn’t necessarily drive all the daily activities of a unit. Counselors are going to advise students into classes at a community college regardless of the college’s mission. However, each unit needs to own the mission and set some goals to help the college meet its mission. Therefore some of what a unit chooses to do is driven by the mission. People who say “everything we do is driven by the mission” are either naïve or don’t really know what is going on across the college. What is a Goal A statement of a quantifiable desired future state or condition It is driven by the mission A goal: lacks deadlines is usually long-range is relatively broad in scope provides guidance for the establishment of objectives (the specifics) What is a Goal It is reported on anecdotally It is accomplished by activities It takes several goals to accomplish a mission statement It takes several objectives to accomplish a goal Due to the nature of goals, a unit may never accomplish a goal… …but the unit makes progress toward the fulfillment of a goal. We are used to setting goals But goals are not outcomes Outcomes are program-specific Outcomes represent a new way of thinking Outcomes have become widely accepted by our various publics They are here to stay We used to measure ourselves by our activities Program Outcome Model INPUTS ACTIVITIES OUTPUTS Resources Services Products or Results of Activities Staff Buildings Facilities State funds FTE Education (classes) Services Counseling Student activities Numbers served FTE (input next year) # Classes taught # Students recruited Constraints Laws State regulations United Way model Program Outcomes Model INPUTS > ACTIVITIES > OUTPUTS > OUTCOMES Benefits for People (Outcomes answer the “so what” question) *New knowledge *Increased skills *Changes in values *Modified behavior *Improved condition *Altered status *New opportunities Which Is It? An input An activity An output An outcome 1st Activity GED Preparation College provides well trained faculty. 200 students complete their GED. Students move from public housing. 30 courses are offered each semester. 150 FTE are generated. Students’ reading level improves. Students are gainfully employed. Student Services Four counselors are hired. Students successfully transfer courses and enter the university system. Students receive financial aid at the university. Students attend campus activities. Students are able to pay back their student loans. Nursing Program 100% of students pass their state board exam. 10 faculty members deliver 600 hours of clinical experience for students. Two students become actively involved in the state nursing association. Agreements are established between the college and area hospitals for student training opportunities. General Education Courses Students receive creative classroom experiences. Faculty members receive a grant to offer multimedia opportunities to students. Students become active participants in County civic activities. 700 FTE are generated in core courses. Students’ math skills improve. Outcomes are ……... Driven by the mission. Related to overall program goals. Specific to the teachings/activities of your program. Determined by faculty and front-line staff. Measured carefully and specifically. Inputs through Outcomes: The Conceptual Chain Long-range Intermediate Initial Outputs Activities Inputs OUTCOMES But are there different types of outcomes? Different Types of Outcomes Learning Outcomes Program Outcomes Administrative Outcomes Definitions and Examples Learning Outcomes: What changes in knowledge, skills, attitude, awareness, condition, position (etc.) occur as a result of the learning that takes place in the classroom. These are direct benefits to students. Examples: general learning skills (e.g. improved writing and speaking abilities), ability to apply learning to the work environment (e.g. demonstrate skills in co-op), program-specific skills developed or enhanced (e.g. take blood pressure.) Definitions and Examples Program Outcomes: The benefits that results from the completion of an entire program or series of courses. Are there benefits for students who get the AAS in welding versus those who take a few courses? If so what are they? Typical examples are: licensure pass rates, employment rates, acceptance into 4-year schools, lifelong learning issues, contributions to society, the profession, etc.) Definitions and Examples Administrative Outcomes These are outcomes established by programs and services that have nothing to do with student learning or completion of programs. They have to do with benefits to the faculty/staff, the department and to the college. The English Department wants all faculty to attend one professional meeting annually so they can stay up-to-date in their field. What are the benefits to the faculty, the college, the students? Counseling may want students to lose few credits when they transfer or to recruit a new counselor with expertise in working with first-generation students (huge population for us). What are the benefits to the counseling department, the college and students? Facilities services may want all college units to feel that they respond quickly to maintenance needs and security issues. It they do this, what are the benefits to the college and to students? What is an Outcome Objective? A short-term, measurable, specific activity having a time limit or timeline for completion around a specific outcome They measure outcomes and are used to show progress toward goals They specify who, will do what, under what condition, by what standard and within what time period Go Over Definitions Example – The College Foundation Mission Statement (portion): To enable county residents to accomplish their educational goals. Goal: Enhance accessibility to the college through building private sector support for scholarships. Objective: The College Foundation will increase private sector contributions for scholarships in the amount of 5% by June 30, 2006. Outcome Objective: 100 additional students will receive need-based scholarships by August 31, 2006. How to Set Objectives There’s no magic number e.g. 80% or 90% What is reasonable? What can you afford? What realistically can your staff accomplish? What percent shows you’re not committed and what percent shows you’re naïve? How to Set Objectives Examples: Fifty percent of students will be able to communicate effectively in writing (complete the writing exam with a grade of 60 [D] or better) By the end of the spring term, 95% of faculty and staff will have completed 20 contact hours of professional development (workshops, college courses, conferences, onsite trainings, etc.) More Realistic Seventy percent of students will be able to communicate effectively in writing (complete the writing exam with a grade of 75 [C+] or better) By the end of the spring term, the professional development office will increase their offerings for faculty and staff by 10% over what was offered last year (workshops, college courses, conferences, onsite trainings, etc.) How to Set Objectives The first time you set objectives, be conservative Allow yourself a pilot semester or year to determine the appropriate levels of change that can be expected Don’t pull a rabbit out of your hat (e.g. let’s grow enrollment by 10%) May need to benchmark (what does it mean) Fall Curriculum Enrollment at ABC College 17,188 17,500 16,982 17,000 16,660 16,000 15,500 (1.5%) 16,245 16,500 15,208 15,488 15,724 15,997 (1.2%) (2.6%) (1.6%) (1.7%) 14,975 (1.5%) 15,000 (1.8%) (1.6%) 14,500 Headcount 14,000 13,500 1997 1998 1999 2000 2001 2002 2003 2004 2005 Should they set a goal for a 5% enrollment growth for next fall? Administrative Objectives Many units do not directly serve students or they want results within their units that are not truly outcomes. They want to improve services or approach an old problem in a new way. They want to become more efficient and effective. They will set administrative objectives. My Administrative Objectives 1. 80% of faculty/staff responding to the faculty/staff survey will perceive that Planning and Research responds quickly to their requests for data. 2. 80% of faculty/staff responding to the survey will perceive that Planning and Research makes a significant contribution to the College. 3. 80% of faculty/staff responding to the survey will perceive that Planning and Research contributes to the effectiveness of CPCC. 4. 80% of faculty/staff responding to the survey will indicate that Planning and Research produces enough reports to meet the planning and information needs of faculty and staff. Why is This Hard? Because it is education Because the best results may not happen for years Because we are so busy doing what we are doing…. we forget why we are doing it Let’s Look at Healthcare When you have strep throat and go to the doctor for your antibiotics (your intervention) What are your intended outcomes? Would the doctor ever tell you: We believe curing disease is a developmental process We believe there is value in the activity of taking pills and receiving shots We’re not sure if you’ll get better, it is how all doctors have treated the disease since we learned about it What would you say to that? Health Outcomes Are Easy Your health faculty have been doing this for many years Everything in healthcare is about outcomes People go to doctors, receive treatment, endure surgery and physical therapy – only for the outcomes… no other reason How to Measure Program/Student Outcomes Identifying Outcomes Faculty/staff in an area are often the worst in defining outcomes because they are too close to the subject. New Teachers 1st Year 2nd- 3rd Middle Years of Year Teachers Teaching Older Teachers Unconsciously Unskilled Consciously Unskilled Consciously Skilled Unconsciously Skilled Don’t realize they don’t know what they are doing Realize they don’t know what they are doing – begin to ask for help Know what works Can’t remember and why why they do what they do – just know it works Sources of Ideas for Outcomes * * * * * * * * * * program documents program faculty and staff national associations/credentialing boards key volunteers former students parents of students records of complaints programs/agencies/employers that are the next step for your students other colleges with similar programs, services and students as yours outside observers of your program in action How Often Should we measure objectives or student learning outcomes every year? When does measurement become too time consuming? Units need time to put into effect the changes made as a result of outcome assessment before they are thrown back into another cycle. They need time to reflect on changes and results. Disappointing Outcome Findings: Why Didn’t We Meet Our Objectives? Internal Factors: * * * * * Sudden staff turnover New teaching philosophy/strategy Curricular change (campus move) Unrealistic outcome targets Measurement problems (lack of followthrough, no effective tracking) Disappointing Outcome Findings External Factors: * * * * * Community unemployment increases State funding changes Related programs (BS or MS programs) close Public transportation increases fares or shuts down some routes serving your campus or time slot Employment trends change Use Your Findings Internal Uses for Outcome Findings Provide direction for curricular changes Improve educational and support programs Identify training needs for staff and students Support annual and long-range planning Guide budgets and justify resource allocations Suggest outcome targets (expected change) Focus board members’ attention on programmatic issues Help the college expand its most effective services Facilitates an atmosphere of change within the institution External Uses of Outcome Findings Recruit talented faculty and staff Promote college programs to potential students Identify partners for collaboration (hospitals, businesses, etc....) Enhance the college’s public image Retain and increase funding Garner support for innovative efforts Win designation as a model or demonstration site So will someone help me do this… help me select and measure outcomes for my program? Ideas for Outcomes Typical general education goals: Students will demonstrate the ability to obtain meaning from printed, electronic, and graphical resources Students will effectively communicate both orally and in writing. Students will demonstrate the ability to locate, critically evaluate, and present information. Students will apply mathematical concepts and skills to analyze, manipulate, and interpret quantitative data. Students will demonstrate the basic computer skills necessary to function in a technological world. Students will demonstrate the ability to identify, analyze, question, and evaluate content as a guide to understanding and action. Students will demonstrate knowledge of cultural differences. Students will demonstrate an understanding of the influence of the individual on group behavior and conversely, the influence of the group on the individual. Students will demonstrate comprehension of the major steps of the scientific method. Students will demonstrate knowledge of the humanities and critical skills in assessing cultural/artistic merit and significance. Let’s work through a few. How does the welding program improve math skills in students? How does the culinary arts program teach students to write effectively? How does the Nursing program teach students to think critically? How do English faculty help students understand the impact of the individual on society? How does the GED program improve student computer skills? Top Ten Skills for the Future Work ethic, including self-motivation and time management. Physical skills, e.g., maintaining one's health and good appearance. Verbal (oral) communication, including one-on-one and in a group Written communication, including editing and proofing one's work. Working directly with people, relationship building, and team work. Influencing people, including effective salesmanship and leadership. Gathering information through various media and keeping it organized. Using quantitative tools, e.g., statistics, graphs, or spreadsheets. Asking and answering the right questions, evaluating information, and applying knowledge. Solving problems, including identifying problems, developing possible solutions, and launching solutions. The Futurist Update (Vol. 5, No. 2), an e-newsletter from the World Future Society, quotes Bill Coplin on the “ten things employers want [young people] to learn in college” Let’s work through a few? How does the early childhood program improve the work ethic of childcare workers or the children in childcare centers? How do IT programs improve teamwork skills? How does Engineering improve students’ ability to use spreadsheets and read tables/graphs? How do you improve a student’s ability to: Influence people Have time management skills Ask the right questions Solve problems in the work environment Learning Outcomes for the 21st Century Students in the 21st Century will need to be proficient in: Reading, writing, speaking and listening Applying concepts and reasoning Analyzing and using numerical data Citizenship, diversity/pluralism Local, community, global, environmental awareness Analysis, synthesis, evaluation, decision-making, creative thinking Collecting, analyzing and organizing information Teamwork, relationship management, conflict resolution and workplace skills Learning to learn, understand and manage self, management of change, personal responsibility, aesthetic responsiveness and wellness Computer literacy, internet skills, information retrieval and information management (The League for Innovation’s 21st Century Learning Outcomes Project.) Let’s work through a few. How does any given program or course improve: Listening skills Environmental awareness Creative thinking Relationship management Conflict resolution Self-management Wellness Information management How can we measure it and use the results. Outcomes and the Community College Outcome evaluation is at the heart of institutional effectiveness. Institutional effectiveness is at the heart of SACS’ philosophy and more so with the new criteria. Data from outcome measurement should be used in as many ways as possible planning, improvement, accreditation. Challenges Identifying and defining outcomes is the easy part. The devil is in the details. How do we track it, where does it all go, how do we score it, compile it, turn it into a comprehensive report. How do we “demonstrate improvement in institutional quality.” Things to Remember Outcome measurement must be initiated from the unit/department level (promotes ownership of process). Measure only what you are teaching or facilitating. Put as much time in to “thinking through” the tracking process as you do into the definition of outcomes. Spend the time up front in planning and the process will flow smoothly. It will prove to be energy well spent. Look at Early Childhood Examples The Insanity Principle Doing the same thing we have always done but expecting different results.