From Here, Go Anywhere Assessment: Method for Understanding Student Learning “Effective assessment must begin with real concerns of the stakeholders and result in useful information and recommendations related to the purpose of assessment.” – D.W. Farmer & E.A. Napieralski Michael E. Bryan Office of Student Outcomes Assessment April 13, 2004 Objectives From Here, Go Anywhere Know what is assessment “Many faculty have been ‘doing Examine the driving forces assessment’ for their own sake, and have not been overwhelmed behind assessment (Why) with the task, without particular Understand the benefits & experience in evaluation methodology.” – O.J. Nichols (1995) tools of assessment Define course goals and objectives Determine when & why to use assessments Identify strategies & methods to collect data Put it all together From Here, Go Anywhere What is Assessment? The systematic collection and analysis of information to improve student learning. “…a learner-centered, teacher-directed approach designed to improve student learning in the individual classroom.” – B.D. Wright (1991) From Here, Go Anywhere Assessment and Grades Grades are global evaluations that represent the overall proficiency of students. They don’t tell you about student performance on individual learning goals. “…assessment involves taking a second look at materials generated in the classroom so that in addition to providing a basis for grading students, these materials allow faculty to evaluate their teaching.” – B.D. Wright (1991) Why Assess? From Here, Go Anywhere “…assessment involves taking a second look at materials generated in the classroom so that in addition to providing a basis for grading students, these materials allow faculty to evaluate their teaching.” – B.D. Wright (1991) Rationale and Driving Forces “The institution demonstrates that each educational program for which academic credit is awarded is (a) approved by the faculty and the administration, and (b) establishes and evaluates program and learning outcomes.” “The institution places primary responsibility for the content, quality, and effectiveness of its curriculum with its faculty.” Principles of Accreditation, Standards for All Educational Programs, Section III, 1 and 12, August 2003 From Here, Go Anywhere Reporting Mandates “Reports of Institutional Effectiveness (ROIE) is an annual report required by the State Council of Higher Education for Virginia (SCHEV) and is intended to provide meaningful information on the academic quality and operational efficiency of Virginia’s public institutions of higher education.” “Each institutional report is organized into five sections: institutional mission, college profile measures, system-wide measures, and institutionspecific measures, and the core competencies.” ROIE Homepage, http://research.schev.edu/roie/ “One of the most untruthful things possible, you know, is a collection of facts, because they can be made to appear so many different ways.” - Karl A. Menninger From Here, Go Anywhere Reporting Mandates Assessment processes at VCCS colleges have been mandated for the past eighteen years. Senate Document No. 14, 1986 Virginia General Assembly, provided the legislative mandate for assessment reporting and SCHEV established the initial guidelines for colleges and universities to respond. The Final Report of the Governor’s Blue Ribbon Commission on Higher Education (February 2000) called for the creation of a Quality Assurance Plan to: (1) define core competencies for written communications, oral communications, quantitative reasoning, scientific reasoning, critical thinking, and information literacy; (2) identify measures to assess students’ knowledge and skills; (3) provide a vehicle to present results publicly. Report of the VCCS Task Force on Assessing Core Competencies, July 18, 2002 From Here, Go Anywhere Local Reporting Mandates “Annual reports will be produced to account for what has been accomplished in enhancing current programs and implementing new ones, and to identify emerging developments that call for updates to the curriculum planning process.” TCC Comprehensive Five-Year Curriculum Plan (2003-2008), Page 1, August 2003 “While you are experimenting, do not remain content with the surface of things. Don't become a mere recorder of facts, but try to penetrate the mystery of their origin.” - Ivan Pavlov From Here, Go Anywhere Local Reporting Mandates Curriculum planning at TCC will occur in accord with the following principles: The curriculum will be structured and articulated in ways that provide students with multiple opportunities for connection and advancement as they enter, progress through, exit, and potentially return to the college. The curriculum will be responsive to the distinctive needs of business, industry, government, agencies, and the larger community, continually reinventing itself in relation to the immediate requests and long-term developments in various occupational and technical fields. The curriculum will be developed in consultation with employers and transfer institutions that are the direct recipients of graduates. All curricula will be built and maintained on a foundation of excellence. “In the college transfer area, the college will benchmark itself to the most rigorous standards of achievement in 4-year baccalaureate degree-granting institutions. In the fields of occupational-technical study, the college will look to business and benchmark its performance goals to the standards of the relevant industries themselves” (Bearings on the Future: The TCC Strategic Plan). TCC Comprehensive Five-Year Curriculum Plan (2003-2008), Pages 1 & 2, August 2003 From Here, Go Anywhere Local Reporting Mandates TCC will pursue the following corresponding strategies: The college will regularly review programming at peer community colleges across the nation with the goal of identifying current and emerging curricular areas for possible development and implementation. As resources permit, faculty and administrative teams will… bring back model programs and practices for adaptation and implementation at TCC. Through focus groups, advisory committees, and other means of formal and informal relationships with employers and the higher education community, the college will actively solicit input for needed program enhancements and development as well as communicate the range and availability of existing programs. The college will actively partner with those who are the primary stakeholders in the successful outcome of students, looking for ways to engage their support and sponsorship of students throughout the learning process. The college will put into place a systematic process for assessing programmatic and student learning outcomes, the results from which will be used for continuous improvement of disciplines and programs. Other college activities, processes, and initiatives (e.g., faculty hiring, adjunct faculty initiative, accreditation, professional development and renewal, budget development, grant proposals, major gifts campaign) will be aligned with the pursuit of curricular excellence. From Here, Go Anywhere Recap Why We Assess External Needs Requirements for SACS Accreditation Report of Institutional Effectiveness for SCHEV Quality Assurance Plan for VCCS Internal Needs Curriculum Planning Process & Annual Reports for TCC Intellectual Curiosity Professionalism From Here, Go Anywhere How is TCC doing as a learning institution? What do you really want students to know and be able to do by the end of your course? What are you trying to measure? What do you need to measure? Do you know why your data results are what they currently are? What can you do to help students learn what you believe they need to know? Is your plan for improvement based on actual instructional strategies? Essential Questions Process From Here, Go Anywhere Steps to the Process 1. 2. 3. Identify & articulate what students should learn in your class (Goals and Objectives) Review & implement tools to compile, measure and analyze student learning data Use the information gathered to improve/adapt curricula, pedagogy, and goals “One important distinction in assessment methods is between techniques that directly determine whether students have mastered the content of their academic programs and those that ask students to reflect on their learning.” – C.A. Palomba & W. Banta (1999) Step 1 From Here, Go Anywhere Establish Goals and Objectives Course Goals Describe broad learning outcomes and concepts What do you want students to learn/do/apply/take with them? Expressed in general terms e.g., clear communication, problem-solving skills “Much education today is monumentally ineffective. All too often we are giving young people cut flowers when we should be teaching them to grow their own plants.” - John W. Gardner Goals From Here, Go Anywhere Difficulty Writing Goals? Why do you use current assignments, course structure, and activities? What is it you want to help students learn through these course elements? What do you want your students to learn and in what ways do you want them to grow? In the past, have your goals for students been realistic? What do your students usually learn and in what ways do they usually grow? Where do students have difficulty; what do they consistently not get? If you ran into a student who had taken your class the previous semester, what would you hope the student would say about what s/he took away from your course? Objectives From Here, Go Anywhere Course Objectives Objectives describe specific learning behaviors that students should exhibit in the context of the course. Objectives are the specific skills, values and attitudes students should exhibit that reflect the broader goals (e.g., for students in a freshman writing course, this might be “students are able to develop a cogent argument to support a position”). Often in the assessment literature, “objectives”and “outcomes” are used interchangeably. “A student drove himself so hard that he missed the learning curve.” - Pun of the Day Objectives From Here, Go Anywhere Effectives Course Objectives Use action words that specify definite, observable behaviors Indicate an appropriate level of attainment Are assessable through one or more indicators Comprehensively and meaningfully define a goal “Two mathematicians arguing about even Are realistic and achievable numbers were at odds.” Use simple language – Pun of the Day From Here, Go Anywhere Levels of Objectives Word Power From Here, Go Anywhere Objectives From Here, Go Anywhere Difficulty Writing Objectives? For each of your stated goals, what are the specific student behaviors, skills, or abilities that would tell you this objective is being achieved? Ideally and briefly, what would a skeptic need (what evidence needs to be present, what specific behavior needs to be visible) in order to see that your students are achieving the major objectives you have set out for them? In your experience, what evidence tells you when students have met these objectives – how do you know when they’re “getting” it? Syllabus From Here, Go Anywhere Review the Course Syllabus Step 2 From Here, Go Anywhere Measure Attainment of Goals and Objectives What assessments do you currently use? Are the assessments clearly tied to your goals and objectives? Do you need to revise/improve existing assessments to determine student proficiency or goal attainment? Do you need to identify/develop new assessments? “The great difficulty in education is to get experience out of ideas.” - George Santayana Current Use From Here, Go Anywhere Identify & Review Existing Assessments/Measures What information on student learning/performance do you currently collect? How do these data sources relate to your newly articulated goals and objectives? Are there gaps between the information you collect and your course objectives? What other information do you need to have to understand whether students are meeting these objectives? From Here, Go Anywhere Need for Improvement? Additions, Deletions, and Modifications Do you need to revise/improve existing assessments to determine student proficiency or goal attainment? Do you need to identify/develop new assessments? “You can observe a lot by just watching.” “We made too many wrong mistakes.” - Yogi Berra Timing From Here, Go Anywhere Assess at Specific Points in Time or Over Time? Think about the length and scope of the assessment techniques you are currently administer or you’d like to implement. Key Questions From Here, Go Anywhere From Here, Go Anywhere Specific Points in Time Key Points in the Semester: at the start of the semester after a particularly in-depth lecture at times when you think ideas are not flowing as freely as they might to clear up potential areas of confusion to encourage reflective thinking on particular issues or topics “Too often we give children answers to remember rather than problems to solve.” - Roger Lewin From Here, Go Anywhere Specific Points in Time Types of Assessments: Primary Trait Analysis Minute Papers Misconception/Preconception Check Muddiest Point Punctuated Lectures Chain Notes Classroom Opinion Polls Reading Reaction Paper Reaction Over the Course From Here, Go Anywhere Assessing Student Learning Over the Course of the Semester: Tracking student achievement over time is one of the best ways that you have to document that students are really accomplishing what you intend. Incorporating classroom assessment into your teaching and curriculum design facilitates specific documentation of results that clearly demonstrate student learning from the beginning of the semester until the end of the course. “A math professor in an unheated room is cold and calculating.” – Pun of the Day Over the Course From Here, Go Anywhere Types of Assessments Over the Course of the Semester: Primary Trait Analysis Systematic progression of assignments Pre/Post Test Portfolio Analysis “Education is what survives when what has been learned has been forgotten.” - B. F. Skinner Step 3 From Here, Go Anywhere Closing the Loop: Interpreting and Using the Results of Classroom Assessment Where Do We Begin? Knowing WHY you are assessing is KEY to deciding WHAT to do with the data after you collect it. Gather Data for a Variety of Reasons: Classroom-based improvement in teaching and learning Institution-wide program/discipline review requirements External accreditation initiatives Analysis From Here, Go Anywhere Organize the Information Action plan From Here, Go Anywhere Document Your Effectiveness Reporting From Here, Go Anywhere Assessment Matrix So Much More From Here, Go Anywhere Tip of the Assessment Iceberg Countless types of assessments such as performance, portfolio, instructionally-oriented, computer-aided, standardized Item analysis - modern test theory (IRT) Briefly touched on selecting when and how often to assess Assessing student background and experience Engaging students throughout the process Preparing students for taking assessments Considerations for large lecture classes Tracking individual/cohort students Closure Activity From Here, Go Anywhere Fun Activity to Illustrate Importance of Scoring Guides (Time Permitting) Additional Information Need Assistance? Clarification? From Here, Go Anywhere Visit the Student Outcomes Assessment (SOA) Website http://www.tcc.edu/welcome/collegeadmin/OIE/SOA/ index.htm Call or Email Mike Bryan (757) 822-1073 mbryan@tcc.edu Questions? From Here, Go Anywhere