MARKING AND GIVING FEEDBACK Introduction to marking Using rubrics to mark accurately and reliably Estimated duration: 60 minutes Carry out some further research on rubrics and gather some inspiration by looking at the following: The examples below showing rubrics of different types. Brown, G, Bull, J, & Pendlebury, M. (1997) Assessing student learning in higher education. New York: Routledge. Walvoord, B. & Anderson, V. (1999) Effective Grading Practices. San Francisco, CA: Jossey-Bass. Examples of rubrics Contents Introduction ................................................................................................. 2 Example 1: Rubric for a short answer text question ................................................ 3 Example 2: Rubric for scoring writing (1) ............................................................. 4 Example 3: Rubric for scoring writing (2) ............................................................. 6 Example 4: Rubric for scoring student presentations (1) ........................................... 7 Example 5: Rubric for scoring student presentations (2) ........................................... 9 Example 6: Rubric for scoring an exam question .................................................. 10 Example 7: Rubric for first component of proposal paper ....................................... 13 Example 8: Rubric for quizzes and homework ...................................................... 14 Example 9: Rubric for evaluating senior design project .......................................... 15 Example 10: Rubric for problem-based learning discussion lab ................................. 17 Example 11: Using rubrics to help students develop their own ability to assess their work ......................................................................... 18 1 Introduction The examples of rubrics given in this document demonstrate how varied they can be. For example, levels of performance can be expressed in a range of ways… They might be laid out on a scale with the points along the scale given descriptive words, like this: 0 1 2 3 Unacceptable Minimally acceptable Acceptable Exceeds expectations They might be more elaborate and informative, like this: Unacceptable The answer has several errors in both conceptual level and communication clarity. Multiple mechanical errors or complete failure to answer the question Minimally acceptable The answer contains one or two basic facts that are correct, but may also have incorrect statements as well. No connections or comparisons provided. Acceptable The answer contains most (75%) of the points that needed to be included. The writing is clear, if uninspired. Correct attempts to integrate the points. Exceeds expectations The answer not only contains the main points but goes beyond them to provide a critique of their veracity. The writing is clear and measured. Rubrics frequently indicate the point value or level in addition to or in place of the scale shown above, like this: Component All main ideas are included Ideas have been analyzed Ideas have been connected Writing is clear and coherent Possible point value 5 points total 5 points total 3 points 3 points As you read through the examples, make a note of elements that you consider particularly effective or relevant to your own situation. Reflective questions You may also wish to reflect upon the following questions as you read each rubric: How much room does the rubric leave for individual interpretation or subjectivity in marking? Can you see any ways of making it more specific? Imagine using each rubric. Are there any areas you feel would require further clarification before you were able to make a start? Is it clear how you are to decide between each level of attainment? What is an appropriate level of detail for a rubric? Some academics have argued that, while it is helpful to be as minutely specific as possible, there is also value in taking a step back and marking more holistically, and that the final grade should reflect a balance between these two approaches. Do you agree? Why (not)? And where on the spectrum of minutiae/holistic does each rubric stand? Our thanks to all those who volunteered examples from their own courses. 2 Example 1: Rubric for a short answer test question Source: Marilla Svinicki, University of Texas at Austin College of Education Introduction This rubric might be used for scoring a written essay exam question. Note that there is space for the grader to keep track of unanticipated responses and how they were scored so that all papers will be consistent any time the unanticipated answer is encountered. Question Explain in 100 words or less how you would apply the theory of vicarious reinforcement to exercise more control over students in an elementary level class. (10 points) Rubric Criteria Accurate description of vicarious reinforcement as a concept Accurate use for classroom management suggested Complete, well-structured answer less than 100 words If incorrect terminology used, subtract Points 3 5 2 -3 if frequent, -2 if occasional Notes to grader Frequent misconceptions used (and disposition of each instance) Frequently used incorrect terminology (and disposition of each instance) 3 Example 2: Rubric for scoring writing (1) Source: Sally Kuhlenschmidt, Western Kentucky University Introduction This more detailed rubric is designed for assessing writing assignments. CATEGORY Most points Zero points Clarity Grammar, spelling, and style make it easy for the reader to follow. Uses words correctly and avoids jargon unless it is the most precise word. Occasional (2 or 3 per page) grammar, spelling or style problems. Tendency to use vague words or excessive jargon. Accuracy All statements are accurate relative to the resource being examined and/or to the material in the textbook. Discrepancies between sources are indicated. All statements are No more than one minor accurate relative to the inaccuracy per page. resource being examined and/or to the material in the textbook. Discrepancies are missed. More than one inaccuracy per page. Precision/ Logic Statements are at the best level of information that answers the question-not too vague but not "lost in the details" and missing the big picture. Statements are mutually supporting and follow from one another. Any contradictions are explained. Most statements are at the best level of information that answers the question--not too vague but not "lost in the details" and missing the big picture. Statements are usually mutually supporting and follow from one another. Any contradictions are explained. Fails to provide a level of information that answers the question--either too vague or filled with trivial details. Statements are internally contradictory without explanation. Problems in grammar, spelling or style that interfere with the author's statements. (Multiple problems in each paragraph). Statements are sometimes on target and sometimes off center. Segments of the paper hang together but other parts are unclear or contradictory with no good resolution. Significant problems in grammar, spelling or style that make it challenging to follow the author's statements. 4 Relevance/ Significance Achieves the learning objectives of the task. Topic is significant to both the course and in larger senses (e.g., to individual, to the region). Makes a case for that significance. Depth/Breadth Response displays a full understanding of the complexity of the issue addressed and multiple points of view. Recognizes varied interpretations and implications. No Plagiarism Rephrases in most cases. Uses only occasional quotation that is in quotation marks and is correctly cited following APA style. Achieves the learning Achieves learning objectives of the task. objectives minimally. Topic Topic is obviously central choice is only vaguely to the course. related to the assignment. Fails to achieve the learning objectives of the task. The topic is trivial or irrelevant to assignment. Of the following 2 tasks, does one well and the other partially or does both partially a) Recognizing varied points of view b) Exploring the topic in depth from one point of view. Response is both narrower than appropriate and superficial. Fails to recognize varied interpretations and implications of topic. Of the following 2 tasks, does one well and the other not at all or does both minimally a) Recognizing varied points of view b) Exploring the topic in depth from one point of view. Quotes properly but Plagiarized. Used excessive use of quotations. more than 3 consecutive words from a source without quotation marks. 5 Example 3: Rubric for scoring writing (2) Source: Michael Theall, Youngstown State University, USA Introduction This is a generic rubric that can be used for holistic grading of any general writing assignment. Rubric Holistic: Overall, does the essay hold together as a piece of college-level writing? DESCRIPTORS/SCORES: ACCEPTABLE 4. The essay shows adequate structure, support for major points, and command of the language 5. and may show quite good structure, support for all points, and good command of the language 6. and shows some insight, creativity, and very good command of the language. DESCRIPTORS/SCORES: UNACCEPTABLE 3. The essay may wander a bit, fail to support points adequately, or may show unacceptable command of some sentence-level features 2. and may show weakness in several of the areas, or serious weakness in one or more areas 1. and may show seriously flawed writing in most areas. Focus: Does the writer follow directions and address the topic adequately? DESCRIPTORS/SCORES: ACCEPTABLE 4. Follows directions, addresses the topic, and lets the reader know what's coming 5. and shows some insight into the topic 6. and even shows some creativity. DESCRIPTORS/SCORES: UNACCEPTABLE 3. Drops a minor part of the directions or misses a point or stance central to the topic 2. and loses touch with the topic at times 1. and may not follow directions or may lose touch with the topic entirely. Validity: Does the writer use accurate, non-trivial evidence and logic as support? DESCRIPTORS/SCORES: ACCEPTABLE 4. The evidence is reasonably accurate and non-trivial 5. and may show an above-average range of knowledge of the topic 6 Example 4: Rubric for scoring student presentations (1) Source: Norback & Utschig Presentation Scoring System, University of Georgia Introduction This is part of a much larger rubric for evaluating student presentations. The complete rubric has four components: Customizing to the audience (shown below) Telling the story Displaying key information Delivering the presentation. Each component is described in much greater detail in the final full rubric. Some examples of behavioral descriptions of a ‘wow’ performance are shown below. Rubric: Customizing to the audience Audience member characteristics are identified ahead of the presentation as observed through presentation details tailored to audience needs and interests." No Connects with the audience, referring directly to their needs Uses appropriate language for audience, describing concepts at just the right level Provides relevant details familiar to audience, using concrete examples where relevant Takes questions adeptly and answers them satisfactorily Yes, but 3 3 Yes Wow! 1 1 Not much 2 2 4 4 5 5 1 2 3 4 5 1 2 3 4 5 Examples of behavioral descriptors of a "wow!" level of performance. "Greets audience and uses names of audience members if possible." "Demonstrates that audience questions have been anticipated." "Describes how skills/knowledge/experience was important in the work behind each key point." 7 Example 5: Rubric for scoring student presentations (2) Source: Marilla Svinicki, University of Texas at Austin College of Education, USA Introduction This rubric was developed for scoring a presentation made by graduate students in a course about applying educational theories to special populations. Rubric Circle the level for each component and add weighted scores I. Clear description of population characteristics and challenges (40% weight) 0 Not satisfactory Incorrect, unclear or incomplete description 1 2 Satisfactory Complete, correct and clearly presented for both 3 4 Very well done Complete, correct and clearly presented, but shows an appreciation of connections between population and challenges; OR goes beyond clear description to very sophistication presentation or connection II. Reasonable choice and description of theories that would be useful and why others are not useful (40% weight) 0 Not satisfactory Incorrect, unclear or incomplete choice, analysis, and/or description of theories in relation to topic 1 2 Satisfactory Correct theories chosen and tied to population being analyzed; standard analysis of usefulness 3 4 Very well done Meets all standards of satisfactory, but raises the analysis and comparison of theories to professional level of sophistication III. Professional and Engaging (20% weight) 0 Not satisfactory Not satisfactory Presentation too casual or too disconnected with poor support materials and visuals. Doesn't follow directions 1 2 Satisfactory Presentation smooth and relaxed with adequate visuals and support materials 3 4 Very well done Presentation has an extra degree of variety and creativity while still making the main points clearly and professionally 8 Example 6: Rubric for scoring an exam question Source: Michael Theall, Youngstown State University Question 1. In your work as a teacher who is a “reflective practitioner”, you must constantly make decisions. Some of these decisions allow you the time to investigate topics and make plans (as in designing a new course for a future offering), and some allow no time at all (as in the need for you to act immediately when a classroom situation arises). Please describe the foundations or principles upon which you make these decisions. OR 2. In your classroom, you face students whose preparation, abilities, skills, motivations, individual differences, and personal situations vary a great deal. Some are eager and able. Some are resistant and unskilled. Some have potential that is masked or limited by factors beyond their (and perhaps your) control. You have to teach them all. What do you do to try to succeed in this most challenging task? For either question … You can respond in several ways, but your response should touch on one or more of the following questions (and/or other points/issues you feel to be important): What factors do you consider and what issues do you take into account? What kinds of information are important? What are your beliefs and motivations? When you have time, what process do you use to design effective instruction? When you must respond immediately, what guides your thinking and actions? What resources do you call upon? What human/interpersonal issues come into play? What theories and “best practices” do you rely on? Complete answers to these question and sub-questions would fill a book. Do not try to put down everything you know. Rather, indicate the important things that guide and influence your decisions. If you had ‘the’ answers, you would become famous … but teaching and learning present unique situations and the most successful teachers know there is no, one answer. They understand that they must reflect on a variety of issues and adapt to the variety of students and situations they face. Reflect for us here. Tell us what do you do and why do you do it? Rubric follows on the next page. 9 RATINGS SCORE SKILLS SUPERIOR = 4 STRONG = 3 ACCEPTABLE = 2 MARGINAL = 1 ABSENT =0 Consider important factors in successful instruction Thorough outline of important factors =4 Most factors covered =3 Some factors covered =2 Many factors omitted =1 Absent = 0 3 Incorporate philosophy of teaching & beliefs about teaching & learning Strong beliefs: clear & relevant to tchng & lrng =4 Strong beliefs discussed =3 Some beliefs presented =2 Little philosophy or few beliefs included =1 Absent = 0 3 Demonstrate knowledge of basic instructional design ID process or model(s) discussed in some depth =4 Brief process or model description =3 ID model(s) or process mentioned =2 ID model(s) or process implied =1 Absent = 0 2 Show concern for students & learning Clear, learnercentered approach =4 Some discussion of l-c approach =3 Reference(s) to l-c approach =2 Learnercenteredness implied =1 Absent = 0 4 Demonstrate reflective practice in action Shows good understanding of reflective practice =4 Discusses reflective practice =3 Mentions reflective practice =2 Reflective practive implied =1 Absent = 0 3 Connect to "best practices" & theories Strong connections to theory & practice =4 Some connections to theory & practice =3 Few connections to theory & practice =2 Theory & practice links implied =1 Absent = 0 2 Points >>>>>>>>>>>> >>>> Writing 10 SCORING****** PASS = > 12 pts. RATER: M. Theall STUDENT FAIL = < 13 pts. deductions >>>>>> Total points >>>>>>>>>>>> GRADE >>>>>>>>>>>> >>> Pass 11 Example 7: Rubric for first component of proposal paper Source: Marilla Svinicki, University of Texas at Austin College of Education Note This rubric was used for scoring an initial paper in a graduate class. Purpose To create an understanding of the problem and the need for a solution that is instruction-based or behavior management based. Questions 1. What is the problem for which you are proposing a solution? 2. Why is it important? 3. Why is an instructional or behavioral management solution called for? Rubric Basics The paper must state the problem so that someone not in the area can understand what the target problem looks like from a behavioral, learning, instructional or management standpoint (2 pages). It also makes a case for why this problem needs to be solved, including what impact it is having on the individual or class of individuals who represent the target audience (2 pages). Finally it relates the problem to the purpose of this class: the use of psychological theories to design instruction to address the issue (1 page). I recommend providing headings to indicate the subcomponents of the paper. An ‘A’ paper For this to be an A paper, the above must be very clearly and concisely written, with convincing main arguments, and give the reader a clear sense of how solving this problem will benefit the target audience. The paper must build a strong tie between the problem and basing the solution on instructional or management theory or theories studied in this class. (For the last of these criteria, you may look ahead to theories we have not discussed if they are appropriate.) A ‘B’ paper For this to be a B paper, at least two of the three basics listed above must be included, but the writing or argumentation will be less clear, concise, or compelling than an A paper. There will be fewer examples of key evidence for the seriousness of this problem and fewer reasons why the problem needs to be resolved using an instructional or management solution. References to the theory or theories that make up this course may be fewer. A ‘C’ paper A paper will receive a C if it is not clear what the problem being addressed is or if only one of the three basics listed above is included. A C might also reflect a poorly written paper or one with a lot of mistakes, either conceptual or mechanical. 12 Example 8: Rubric for quizzes and homework in Environmental Science Source: Diane Ebert-May and Tsao, Michigan State University Note This rubric can be used with either homework problems or on short quizzes. Rubric Level of Achievement Exemplary General Approach • • • • Adequate • • • • Needs Improvement • • • • Comprehension Addresses the question. States a relevant, justifiable answer. Presents arguments in a logical order. Uses acceptable style and grammar (no errors). • Does not address the question explicitly, although does so tangentially. States a relevant and justifiable answer. Presents arguments in a logical order. Uses acceptable style and grammar (one error). • Does not address the question. States no relevant answers Indicates misconceptions. Is not clearly or logically organized. Fails to use acceptable style and grammar (two or more errors). • • • Demonstrates an accurate and complete understanding of the question. Backs conclusions with data and warrants. Uses 2 or more ideas, examples and/or arguments that support the answer. Demonstrates accurate but only adequate understanding of question because does not back conclusions with warrants and data. Uses only one idea to support the answer. Less thorough than above. Does not demonstrate accurate understanding of the question. Does not provide evidence to support their answer to the question. No Answer 13 Example 9: Rubric for evaluating senior design projects in Chemical Engineering Source: Ron Miller, Colorado School of Mines Introduction This rubric is used to evaluate very extensive complex projects that require students to produce original designs for senior level courses intended as capstone courses. Note that the rubric also relates each objective being evaluated in the left most column. Outcome ChE graduates will be able to design chemical engineering processes or systems which meet specified requirements by: a. identifying specific project objectives based on general project and client requirements b. gathering and using relevant background information c. generating and analyzing alternative solutions by synthesizing and applying appropriate chemical engineering knowledge d. choosing the optimal solution based on evaluation of technical and economic criteria 4 - Exemplary 3 - Proficient 2 - Apprentice 1 - Novice All important project objectives are identified Important objectives are identified but 1 or 2 minor ones are missing. Most objectives are identified but at least 1 or 2 important ones are missing. Most or all important objectives are not identified. All relevant information is obtained and used to support design. Three or more alternative solutions are considered; each is correctly analyzed for technical feasibility. Sufficient information is obtained and used to support design. At least 3 alternative solutions are considered; analysis is complete but contains minor procedural errors. Some information is obtained, but more is needed to support design. No significant background information is gathered. At least 2 alternative solutions are considered; analysis contains minor conceptual and/or procedural errors. Only one solution is recommended; analysis does not apply all relevant chemical engineering knowledge. Best solution is recommended based on stated criteria. Reasonable solution is recommended; other alternatives should have been developed Satisfactory solution is recommended; better solutions were available to be considered. Only solution considered; no optimization included; better solutions were avaialble. Score 14 ChE graduates will be able to analyze the economic profitability of chemical engineering projects or systems ChE graduates will demonstrate an ability to communicate effectively in writing. Economic analysis is complete and correct; all relevant economic factors are considered. Economic analysis omits 1 or 2 minor economic factors but computations are correct. Economic analysis omits 1 or more major economic factors, but computations are correct. Economic analysis contains major computational errors and/or omits more than 1 major economic factor. Written report is virtually errorfree, logically presents design recommendations and analysis, is well-organized and easy to read and contains high quality graphics. Written report presents design recommendations and analysis logically, s well organized and easy to read, contains high quality graphics and contains few minor grammatical or rhetorical errors. Written report is generally well written but contains some grammatical, rhetorical and/or organizational errors; design recommendations and analysis are mentioned but not fully discussed. Written report does not present design recommendations or analyses clearly, is poorly organized and/or contains major grammatical and/or rhetorical errors. 15 Example 10: Rubric for problem-based learning discussion lab Source: Source: Dr. Patrick Davis, University of Texas at Austin College of Pharmacy, USA Student 9 Student 8 Student 7 Student 6 These are possible concepts that may fall under these categories. Please use them globally, not as a checklist. Preparation & Clinical Skills (6 points) prepared and reviewed multiple resources appeared able to answer questions without hesitation defined terms and clarified concepts ahead of time volunteered meaningful answers throughout session asked relevant questions throughout session used appropriate vocabulary/pronunciation Group Skills & Communication Skills (6 points) encouraged others to contribute shared ideas with others exhibited tolerance and respect for others was assertive, confident (without dominating) was clear and concise analyzed contributions of others Administration (3 points) dressed appropriately for lab as per syllabus was professional and enthusiastic was punctual Student 5 Date: Student 4 Lab Day (circle one): M T W Th F Student 3 Case #: Student 2 Facilitator: Student 1 Introduction This rubric is used to evaluate student participation in a problem-based learning discussion laboratory session on a Pharmacotherapeutics course. Total Score (15 points possible): Please use the space below to write any comments (positive or negative) about individual students: 16 Example 11: Using rubrics to help students develop their own ability to evaluate their work Source: John Cowan, Napier University Introduction The following example shows rubrics can be used to help students learn how to make evaluative judgements of their own work. Shown below is one example of a class activity he uses in this manner. Matching performance to evaluative criteria Purpose: To engage each student in objectively and formatively evaluating their own performance against familiar criteria, which have previously been applied to similar work, but in that case by a grader or tutor. Outline a. This activity is based on a current task. It has been used, for example, in conjunction with a discursive essay in social sciences, a first year engineering design; and a reflective journal featuring a recent critical incident on a placement. b. On completion of the task, students are immediately offered a podcast guiding them through the process of evaluating their work, leading to an accumulation of suggestions for both encouraging and critical comments, as feedforward. c. Students should now assemble and summarise the elements in their self-judgement, the data on which it was based, and their evaluative judgements. d. Marking tutors, acting virtually as auditors at this stage, quickly check the evaluations, and return (without judgements) with comments only when the judgement has overlooked something significant, or appears unjustifiably generous – or harsh. e. Assessment, in the form of a mark or grade, is not declared or (preferably) negotiated (as per Taras, 2001), until all of the above steps have been completed. f. The tutor informs the student of the outcome of their auditing of the student’s self-judgement. The common outcome is that each student will have had a relevant and useful experience of identifying personally-related data, and matching it to multiple criteria, in an objective process. The impact on the students’ further development is strengthened by the immediacy of the details of feedback and feedforward, and the lack of emphasis on the mark or grade to be awarded. 17