Academy for Planning, Assessment, and Research Creating SLO Assessment Plans Presented by ORIE Team Summer 2013 Table of Contents I. II. Defining Student Learning Outcomes Creating PLO or CLO Statements A. Bloom’s Taxonomy B. Habits of Mind III. Selecting Measures A. Direct vs. Indirect Measures B. Program Level Measures vs. Course Level Measures IV. Selecting Evaluation Methods A. Score Weighting Guide B. Rubrics VI. Setting Performance Criteria for Individual Students VII. Specifying Target Outcome for Group Success in meeting the SLO Defining Student Learning Outcomes The student learning outcome is a statement that explains what the student is learning, including the accumulated and demonstrated knowledge, skills, abilities, behaviors, and habits of mind, as a result of actively participating in the course or program of study. Program Learning Outcomes answer the question: What will students know or be able to do when they exit the program? Course Learning Outcomes answer the question: What will students know or be able to do when they complete the course? Creating Student Learning Outcome Statements Student Learning Outcome Statements Should align with the purpose of the program or the course; Should be specific and measurable, using action verbs. Bloom’s Taxonomy-Three Learning Domains: Cognitive Affective Psychomotor Bloom’s Taxonomy Knowledge: Recall data or information SLO Examples: Students will be able to cite, count, define, describe, draw, identify, know, label, list, match, name, outline, point, quote, read, recall, recite, record, recognize, repeat, reproduce, select, state, tabulate, tell, trace, underline. Describe history, purpose, and scope of physical therapy Identify the social, political, economic and cultural influences and differences that affect the development process of the individual. Bloom’s Taxonomy Comprehension: Understand the meaning, translation, interpolation, and interpretation of instructions and problems. State a problem in one's own words. comprehend, compute, convert, defend, differentiate, discuss, distinguish, estimate, explain, express, extrapolate, extend, generalize, give examples, infer, interpret, paraphrase, predict, report, restate, review, rewrite, summarize, translate. Examples of SLO: Students will be able to Distinguish important aspects of the western moral theories, from the virtue theory of ancient philosophers to the modern theories of act ethics. Explain georeferencing of photos or images to maps. Summarize the principles of magnetism. Bloom’s Taxonomy Application: Use a concept in a new situation or applies what was learned in the classroom into novel situations in the work place. apply, calculate, change, compute, construct, demonstrate, determine, discover, dramatize, employ, examine, illustrate, interpret, locate, manipulate, modify, operate, predict, prepare, produce, relate, show, solve, use. SLO Examples Students will be able to Implement plan of care for patients and families within the legal, ethical, and regulatory parameters. Apply appropriate organizational skills for selected types of speeches Conduct basic laboratory experiments involving classical mechanics. Bloom’s Taxonomy Analysis: Separates material or concepts into component parts so that its organizational structure may be understood. Distinguishes between facts and inferences. analyze, appraise, break down, compare, contrast, diagram, deconstruct, differentiate, discriminate, distinguish, identify, illustrate, infer, outline, relate, select, separate. SLO Examples: Students will be able to Analyze and design electrical and electronic circuits and systems, using Knowledge of mathematics and basic sciences. Analyze self-practice in relation to the roles of the professional nurse. Bloom’s Taxonomy Synthesis: Builds a structure or pattern from diverse elements. Put parts together to form a whole, with emphasis on creating a new meaning or structure. categorize, combine, compile, compose, create, devise, design, explain, generate, integrate, modify, organize, plan, rearrange, reconstruct, relate, reorganize, revise, rewrite, summarize, tell, write SLO Examples: Students will be able to Integrate the pathophysiological assessment findings to formulate a field impression. Coordinate human information and material resources in providing care for patients and their families. Bloom’s Taxonomy Evaluation: Make judgments about the value of ideas or materials. SLO Examples: appraise, compare, conclude, contrast, criticize, critique, defend, describe, discriminate, evaluate, explain, interpret, justify, relate, summarize, support Use critical thinking and a systematic problem-solving process for providing comprehensive care. Students will be able to Analyze and evaluate economic events that apply to the preparation of financial statements. Habits of Mind Arthur Costa and Bena Kallick (2000)’s Habits of mind: Persisting, thinking flexibly, and striving for accuracy Examples of Habits of Mind: Develop and use effective time management skills Follow directions correctly Appreciate negatives and failures as opportunities to learn and discover choices SLO Examples: Students will demonstrate time management skills, adhering to all deadlines for assignments, tests, and projects. Students will demonstrate effective therapeutic patient communication/relationship. Selecting Measures A. Direct vs. Indirect Measures To measure student learning outcomes, use the direct measures primarily, but also use indirect measures for triangulation of data Direct Measures: Refer to the type of data that can be used to directly measure students’ knowledge and skills Examples: Pre-test, post-test, comprehensive subject matter exam, licensure exam, portfolio, thesis, writing assignment, internship evaluation, capstone assignment, and performance demonstration (recital, art exhibit, or science project) Selecting Measures Indirect Measures: Refer to the type of data that can be used to infer student learning or achievement Examples: Surveys, Interviews, Focus Group Studies, Document Analyses, or Audit Measures (course grades, retention/graduation/job placement rates) Selecting Measures B. Program Level Measures vs. Course Level Measures PLO Measures: CLO Measures: Exit exam (standardized, licensure/professional exams) Thesis, Dissertation Capstone Course Paper or Project Portfolios Exams, Tests Rubric-Based Assignments: Projects Essays Portfolios Presentations Performance Demo. Selecting Measures C. How to Select Measures? Measuring Knowledge, understanding Tests, Exams, Essays, Portfolios, Presentations Measuring Skills Performance Demonstrations, Projects, Presentations, Internship Evaluation Measuring Habits of Mind Surveys, Interviews, Behavior Observations, Focus Groups, Document Audit Examples of Measures Aligned with SLOs SLOs Measures (Data Related to Student Work /Performance /Opinion) Students will be able to describe history, purpose, and scope of physical therapy (Knowledge) Exam on the history and purpose of Physical Therapy Essay describing the scope of physical therapy Students will be able to distinguish important aspects of the western moral theories, from the virtue theory of ancient philosophers to the modern theories of act ethics. (Understanding) Exam or Essay Questions on the Exam summarizing the principles of ethical theories Essay comparing the similarities and differences between the ancient and modern theories Examples of Measures Aligned with SLOs SLOs Measures (Data Related to Student Work /Performance /Opinion) Students will be able to apply appropriate organizational skills for selected types of speeches. (Skills) Demonstration of public speaking Speech in writing Students will demonstrate time management skills, adhering to all deadlines for assignments, tests, and projects. (Habit of Mind) Grade Book Audit Survey on student time management skills Selecting Evaluation Methods Aligned with Measures Measures (Data Related to Student Work /Performance /Opinion) Evaluation Methods (How do instructors assess students’ work or performance?) Exam on the history and purpose of Physical Therapy Essay describing the scope of physical therapy Score Weighting Guide with scores assigned to each exam question Grading Rubric specifying levels of quality in identified areas Essay Questions on the Exam summarizing the principles of ethical theories Essay comparing the similarities and differences between the ancient and modern theories Grading Rubric specifying levels of quality in identified areas Grading Rubric specifying levels of quality in identified areas Selecting Evaluation Methods Aligned with Measures Measures (Data Related to Student Work /Performance /Opinion) Evaluation Methods (How do instructors assess students’ work or performance?) Demonstration of public speaking Grading Rubric Speech in writing Grading Rubric Grade Book Audit Grade Book Record Survey on student time management skills Survey Instrument Creating a Rubric A rubric is a systematic scoring guide to evaluate student performance. Three Essential Parts of a Rubric: 1) A Set of Indicators of Learning 2) A Scale 3) A Set of Descriptors Creating a Rubric Indicators for a rubric-based assignment (essay, project, speech, performance) should be aligned with the instructional objectives. If an instructor’s objectives of teaching an essay are related to helping students compose an essay that is focused, coherent, organized, etc., then these features should serve as indictors of learning. Example of Indicators for an Essay for an English Class: Focus Coherence Organization Sentence Structure Word Choice Creating a Rubric A scale indicates points to be assigned in scoring a piece of work on a continuum of quality. High numbers are typically assigned to the best work. Scale examples: Needs Improvement (1)... Satisfactory (2)... Good (3)... Exemplary (4) Beginning (1)... Developing (2)... Accomplished (3)... Exemplary (4) Needs work (1)... Good (2)... Excellent (3) Novice (1)... Apprentice (2)... Proficient (3)... Distinguished (4) Creating a Rubric Descriptors are used to describe signs of performance at each level; the description needs to include both strengths and weaknesses (Weaknesses should be described particularly in lower levels of performance). Examples of Descriptors: Word Choice 4-- Vocabulary reflects a thorough grasp of the language appropriate to the audience. Word choice is precise, creating a vivid image. Metaphors and other such devices may be used to create nuanced meaning. 3-- Vocabulary reflects a strong grasp of the language appropriate to the audience. Word choice is accurate, but may be inappropriate in a couple of places. 2-- Vocabulary reflects an inconsistent grasp of the language and may be inaccurate or inappropriate to the audience. 1-- Vocabulary is typically inaccurate and inappropriate to the audience. Word choice may include vague, non-descriptive, and/or trite expressions. Example of a Rubric (A Snapshot) Indicators Explanation of Issues Evidence Critical Thinking VALUE Rubric (AACC) Capstone Milestones 4 3 2 Issue/problem to be Issue/problem to be Issue/problem to be considered critically is considered critically considered critically is stated clearly and is stated but description described stated, described, leaves some terms comprehensively, and clarified so that undefined, ambiguities delivering all relevant understanding is not unexplored, information necessary seriously impeded by boundaries for full understanding. omissions. undetermined, and/or backgrounds unknown. Information is taken from source(s) with enough Interpretation/ evaluation to develop a comprehensive analysis or synthesis. Information is taken from source(s) with enough interpretation /evaluation to develop a coherent analysis or synthesis. Information is taken from source(s) with some interpretation/ evaluation, but not enough to develop a coherent analysis or synthesis. Benchmark 1 Issue/problem to be considered critically is stated without clarification or description. Information is taken from source(s) without any interpretation/evaluation. Setting Criteria for Meeting SLO To determine whether students have successfully met the learning outcomes, we need to specify what score or rating an individual student needs to get in order to be considered as meeting the learning outcome. Examples: Measure: Exam on the history and purpose of Physical Therapy Evaluation Method: Score Weighting Guide with scores assigned to each exam question Criterion for Meeting the SLO: A score of 70 or above on the exam Measure: Essay describing the scope of physical therapy Evaluation Method: Grading Rubric specifying levels of quality in identified areas Criterion for Meeting the SLO: A score of 3 or above on a scale of 1-4 Specifying Target Outcomes To determine whether students as a group have successfully met the learning outcomes, we need to specify what % from the group should meet the learning outcome. Example I: Measure: Exam on the history and purpose of Physical Therapy Evaluation Method: Score Weighting Guide with scores assigned to each exam question Criterion for Meeting the SLO: A score of 70 or above on the exam Target Outcome: 85% of the students will score 70 or above on the exam. Specifying Target Outcomes To determine whether students as a group have successfully met the learning outcomes, we need to specify what % from the group should meet the learning outcome. Example II: Measure: Essay describing the scope of physical therapy Evaluation Method: Grading Rubric specifying levels of quality in identified areas Criterion for Meeting the SLO: A score of 3 or above on a scale of 1-4 Target Outcome: 70% of the students will score a 3 or above on their essays Recap: SLO Assessment Plan Student Learning Outcome Assessment Plan Mission: PLO or CLO Statement Measure (Student Teacher's Work/Performanc Evaluation e) Method Criterion (What cut-off score is considered as meeting PLO or CLO for each student?) Target Outcome (Group Success: % of the group meeting the PLO or CLO) PLO Assessment Plan: (Example) Associate Degree in Political Science Program Learning Outcomes Mission: The Political Science Department is committed to producing students who are knowledgeable about the operation of Texas and national government. Students should also have the ability to think critically about important political topics that will prepare them for transfer to higher institutions of learning and provide a framework for lifetime learning skills. PLO Statement Measure (Student Work/Performance) Teacher's Evaluation Method Criterion (What cut-off score is considered as meeting PLO for each student?) Target Outcome (Group Success: % of the group meeting the PLO) Students will understand the American politics subfield of political science. GOVT2304 (Intro to Political Science) Essay Question on American Politics Essay Assessment Rubric A score of 3 or above 80% of the students on essay (on a scale of answering the essay 1-5) question will get a score 3 or above. Students will understand GOVT2304 Essay the political theory Question on Political subfield of political Theory science. Essay Assessment Rubric A score of 3 or above 80% of the students on essay (on a scale of answering the essay 1-5) question will get a score 3 or above. Students will demonstrate Internship Project a level of civic engagement Comments on strengths 70% of the comments and weaknesses of the in the evaluation letter students’ internship are positive. performance in an evaluation letter 95% of the students will get a positive evaluation letter from their internship supervisor. CLO Assessment Plan: (Example) CLO Statement Engl 1302 Rhetoric and Composition Course Learning Outcome Assessment Plan Measure (Student Teacher's Evaluation Criterion Target Outcome (Group Work/Performance) Method (What cut-off score is Success: % of the group considered as meeting meeting the CLO) CLO for each student?) Students will be able to Persuasive Essay present a valid argument using convincing evidence. Essay Grading Rubric A score of 3 or above 80% of the students (1-5 scale) taking the course will receive a score of 3 or above. Students will be able to demonstrate appropriate use of standard English. Grammar Quiz Writing Portfolio a. Quiz grading standard a. A score of 70 on the b. Portfolio Grading quiz Rubric b. A score of 3 or above (1-5 scale) 75% of the students taking the Engl1302 course will receive a score of 70 for the quiz and a score of 3 or above for the portfolios. Student will apply researched information in writing a research paper. Summary of Readings Research Paper a. Summary Grading Rubric b. Research Paper Grading Rubric 75% of the students taking the Engl1302 course will receive a score of 3 or above for the summary assignment and for the research papers. a. A score of 3 or above (1-5 scale) b. A score of 3 or above (1-5 scale) Do’s and Don’ts When Creating an SLO Assessment Plan Do’s 1. When creating PLO or CLO statements, DO include both lower level and higher level of cognitive skills: not only knowledge, comprehension, but also application, analysis, evaluation, and synthesis. 2. When selecting assessment measures for PLOs, DO include both course-embedded assignments and program level measures (thesis, exit exams, or national exams, or graduate portfolios, or internship project, or capstone course project). 3. Encourage faculty participation in the design of the Assessment Plan. 4. Try to use common assessment assignments or tools, but allow for flexibility. Do’s and Don’ts When Creating an SLO Assessment Plan Don’ts Don’t be overly ambitious—Create only 5 to 7 PLOs or CLOs, and use only 7-10 pieces of data! Academy for Planning, Assessment, and Research