Learning Objectives and Classroom Assessment Jeff Froyd, Texas A&M University Western Michigan University, 18 September 2003, Kalamazoo, Michigan Workshop Presenter • Jeff Froyd, Director of Academic Development • Educational Achievement Division, College of Engineering, Texas A&M University • Project Director, Foundation Coalition Western Michigan University, 18 September 2003, Kalamazoo, Michigan Acknowledgement • Russ Pimmel, NSF Program Officer • Former Professor of Electrical Engineering at the University of Alabama • Assembled much of the material for similar workshops Western Michigan University, 18 September 2003, Kalamazoo, Michigan Objectives: Participants will gain experience in • Describing the rationale for preparing and using learning objectives in an individual course • Preparing specifications for high quality learning objectives • Writing learning objectives for a single course • Preparing specifications for assessment processes/tools • Generating alternative assessment processes/tools for a single course • Selecting assessment processes/tools for a single course Western Michigan University, 18 September 2003, Kalamazoo, Michigan Agenda • Background for learning objectives • Specifications for learning objectives • Writing learning objectives • Background for classroom assessment • Specifications for assessment processes/tools • Background on alternatives for assessment processes/tools • Generating alternatives for assessment processes/tools • Selecting assessment processes/tools • Review workshop activities Western Michigan University, 18 September 2003, Kalamazoo, Michigan Agenda • Background for learning objectives • Specifications for learning objectives • Requirements for specifications • Team exercise: develop specifications for learning objectives • Workshop exercise: improve specifications for learning objectives • Writing learning objectives • Individual exercise: write objectives for a course or a portion of a course • Team exercise: review individual objectives • Workshop exercise: develop a list of suggestions for writing learning objectives • Individual exercises: revise objectives Team exercise: review individual objectives • Workshop exercise: reflection Western Michigan University, 18 September 2003, Kalamazoo, Michigan Agenda • Background for classroom assessment • Specifications for assessment processes/tools • Team exercise: develop a set of specifications for assessment processes/tools • Workshop exercise: improve sets of specifications for assessment processes/tools • Generating alternatives for assessment processes/tools • Team exercise: select some learning objectives and generate alternative assessment processes/tools • Background on alternatives for assessment processes/tools • Selecting assessment processes/tools • Individual exercise: select a set of learning objectives and generate alternative assessment processes/tools • Individual exercise: select one or more assessment processes/tools that you would use in your course • Team exercise: share and review choices of assessment processes/tools • Review workshop activities Western Michigan University, 18 September 2003, Kalamazoo, Michigan Learning Objectives Background for Learning Objectives Session Objective: At the end of the session, participants will describe themselves as more confident in their ability to hold productive conversations with their colleagues regarding the place and importance of learning objectives in the teaching-learning process Western Michigan University, 18 September 2003, Kalamazoo, Michigan Stakeholders for Learning Objectives • Who are the stakeholders in conversations about preparing and applying learning objectives? • • • • Faculty Students Employers Accreditation organizations Western Michigan University, 18 September 2003, Kalamazoo, Michigan EC Program Outcomes (a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs (d) an ability to function on multi-disciplinary teams (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively (h) the broad education necessary to understand the impact of engineering solutions in a global and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. Western Michigan University, 18 September 2003, Kalamazoo, Michigan Bloom’s Taxonomy of Cognitive Learning • Knowledge: defines, recalls, matches, reproduces • Comprehension: explains, gives examples • Application: discovering, assessing, computing • Analysis: breaking down, organizing, inferring • Synthesis: creating, putting together • Evaluation: appraising, judging, selecting Western WesternMichigan MichiganUniversity, University,1825September October 2002, 2003,Kalamazoo, Kalamazoo,Michigan Michigan Purpose of Learning Objectives • Communicate expectations for a course • Provide a context for what will be learned http://ublib.buffalo.edu/libraries/projects/tlr/importance.html Western Michigan University, 18 September 2003, Kalamazoo, Michigan Objectives and Students • Objectives help students • • • • Clarify their personal goals Provide framework for measuring their success. Reduce their anxiety Improve their studying effectiveness • Objectives help instructors • Guide preparation of classroom material • Make homework assignments • Aid in test design Source - http://ublib.buffalo.edu/libraries/projects/tlr/importance.html Western Michigan University, 18 September 2003, Kalamazoo, Michigan Strategies For Workshop Teams • Be positive, supportive, and cooperative • Limit critical or negative comments • At least 5 positive comments for every negative comment • Be brief and concise in discussions • Avoid lengthy comments, stories, or arguments • Stay focused Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Roles • Assign team roles & follow through on responsibilities • Coordinator -- Coordinates discussion & develops consensus • Recorder -- Writes down the ideas & reports them • Gatekeeper -- Keeps the team on the subject • Timer -- Makes sure the team stays on schedule • With smaller teams – combine gatekeeper & timer Western Michigan University, 18 September 2003, Kalamazoo, Michigan Workshop Team Roles • For first exercise • Coordinator – Individual with largest class last semester • Recorder/Reporter – Individual on left of coordinator • Gatekeeper/timer -- Individual on left of recorder • Timer -- Individual on left of gatekeeper • Roles rotate clockwise on subsequent exercise Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Form teams of four people • Time: 5 minutes • Develop at least four (4) advantages and four (4) disadvantages of preparing learning objectives for a course. Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • • • • • • • Advantages Disadvantages Communication with • Could be too narrow or students expectations and focused what they need to learn • Explicitly stating learning Offers focus in preparing objectives takes (too?) material to accomplish much time goals • Busy work for faculty Evaluation tool • Could turn into curriculum Tie a course to program or nightmare curriculum • Poorly written objectives Clarifying goals to don’t help students seeks buy-in and • Could be cumbersome, self-assessment ambiguous ABET Assessment tool • Could be more difficult to Michigan University, 18 Septembermeasure 2003, Kalamazoo, Michigan StatingWestern goals facilitates Learning Objectives Preparing Specifications for Learning Objectives Session Objectives At the end of the session, participants will Write specifications for learning objectives Describe themselves as more confident in their ability to describe quality learning objectives. Western Michigan University, 18 September 2003, Kalamazoo, Michigan Form of Learning Objectives • Write objectives as student outcome statements • Objectives should answer the questions • "What must students do to prove that they have succeeded?" • "What should students be able to do as a consequence of instruction?" Source - http://ublib.buffalo.edu/libraries/projects/tlr/importance.html Western Michigan University, 18 September 2003, Kalamazoo, Michigan Elements of an Objective • Objective must contain three basic elements: • Verb describing an observable action • Conditions of this action • “When given x you will be able to..." • Level of acceptable performance Source - http://ublib.buffalo.edu/libraries/projects/tlr/importance.html Western Michigan University, 18 September 2003, Kalamazoo, Michigan Verbs for Objectives Verbs for constructing concrete objectives: analyze compare demonstrate discuss identify justify grasp state compute appreciate direct display infer list report synthesize classify contrast derive evaluate integrate understand respond name collaborate define designate know interpret organize solicit explain Modified from http://www.brown.edu/Administration/Sheridan_Center/pubs/syllabus.html#coura Western Michigan University, 18 September 2003, Kalamazoo, Michigan Learning Objectives Verbs for Categories in Bloom’s Taxonomy • Knowledge • Define, describe, list, reproduce, enumerate • Comprehension • Classify, explain, discuss, give example, summarize • Application • Determine, develop, compute, chart, utilize • Analysis • Correlate, diagram, distinguish, outline, infer • Evaluation • Compare & contrast, critique, justify, conclude • Synthesis • Adapt, combine, compare, contrast, design, generate Western Michigan University, 18 September 2003, Kalamazoo, Michigan Complex Versus Simple Objectives • One complex objective versus several simple ones? • High-level versus low-level objectives Example • One complex objective (4 or 5 weeks of classes) “Given a verbal description of a digital module, develop an implementation using any of 7 different logic devices” • 15 to 20 simpler objectives (1 or 2 per class) “Given a verbal description, draw the truth table” “Given a truth table, obtain a minimum-cost equation” … “Draw the the NAND-gate implementation for an equation” Western Michigan University, 18 September 2003, Kalamazoo, Michigan Comparison Of Complex & Simple Objectives • Multiple simple objectives • More manageable “chunks” for students • Explicit objective(s) for each class • Simple (more manageable) homework problems and test questions • Single complex objective • Student's attention directed to the overall process • May lead to higher level learning • Students must deal with complexity • Students must subdivide problem on their own Western Michigan University, 18 September 2003, Kalamazoo, Michigan Advantages Of Simple Objectives • Advantages of simple objectives are more important in • Large classes rather than small classes • Introductory courses rather than advanced courses Western Michigan University, 18 September 2003, Kalamazoo, Michigan How To Deal With “Understand” In Objectives • How do you write objectives when you want students to “understand” a complex concept, system, or process • Identify specific tasks that indicate “understanding” • Specify objectives for each task • Similar comments apply to “know”, “appreciate”, “value” Western Michigan University, 18 September 2003, Kalamazoo, Michigan How To Deal With “Understand” In Objectives -- Example • In our computer architecture course we want students to “understand” a sample architecture made up of several modules • What would students be able to do if they “understood” • Objectives – • Students should be able to identify: All the modules and interconnecting signals Modules involved in a given system-level operation Output values for a given input values for each module Sub-module changes given a system level change … Western Michigan University, 18 September 2003, Kalamazoo, Michigan Writing Objectives – Piecemeal Approach • Writing low-level objectives for a whole course may be overwhelming • Use a “piecemeal” approach • Write your lectures and define the homework as usual • After each class -- write down what you expect the students to be able to do • These become a list of objectives • Give them to the students before each exam • Use them to write the exam • As semester progresses -- may become comfortable writing the objectives before you prepare your lecture Western Michigan University, 18 September 2003, Kalamazoo, Michigan Evolving Objectives • In a 3-credit semester course • Russ Pimmel (UA, now NSF) started with over 100 objectives • Four offerings later -- down to about 50 • Eliminated peripheral “stuff” that was not central • Broadening, informational, perspective material Western Michigan University, 18 September 2003, Kalamazoo, Michigan Objectives and Homework Assignments • Homework assignments should match objectives • Students need to practice and explore the skills, knowledge, and attitudes defined in objectives • Frequently require supplementary homework problems • In some of my courses • 1/3 of homework is from textbook • Rest are supplementary problems • With well defined objectives • Writing homework problems is straightforward • Same is true for exam questions Western Michigan University, 18 September 2003, Kalamazoo, Michigan Student’s Use Of Objectives • Survey in 400-level required course • Did you find the objectives helpful? Yes --- 52 % No ---- 48% • Did you read the objectives? Frequently ------ 22 % Occasionally -- 37 % Never ------------- 41 % Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise -- Guidelines For Learning Objectives • Task • Write 3 or 5 guidelines for good learning objectives • What are the common features? • What should objectives look like? • Think of guidelines as specifications • Methodology • • • • Brain storm individually -- 2 minutes Establish consensus as a team -- 5 minutes Report team results -- 3 minutes Revise guidelines as a team-- 2 minutes Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • • • • • • • • • • Must be testable and measurable Achievable Clearly and precisely articulated Appropriate to course and audience Relate to program objectives Linked to course outcomes to allow for assessment Simple, one sentence, common format Simple better than complex Should be specific and unambiguous Relate to topic coverage Western Michigan University, 18 September 2003, Kalamazoo, Michigan Learning Objectives Preparing Learning Objectives Session Objectives At the end of the session, participants will Write learning objectives for one or more courses that they teach Describe themselves as more confident in their ability to describe quality learning objectives. Western Michigan University, 18 September 2003, Kalamazoo, Michigan EC Program Outcomes (a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs (d) an ability to function on multi-disciplinary teams (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively (h) the broad education necessary to understand the impact of engineering solutions in a global and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. Western Michigan University, 18 September 2003, Kalamazoo, Michigan Bloom’s Taxonomy of Cognitive Learning • Knowledge: defines, recalls, matches, reproduces • Comprehension: explains, gives examples • Application: discovering, assessing, computing • Analysis: breaking down, organizing, inferring • Synthesis: creating, putting together • Evaluation: appraising, judging, selecting Western WesternMichigan MichiganUniversity, University,1825September October 2002, 2003,Kalamazoo, Kalamazoo,Michigan Michigan Challenges in Engineering Education • Challenges • Challenge of lifelong learning • Challenge of problem solving • Challenge of engineering design • Challenge of transfer Western Michigan University, 18 September 2003, Kalamazoo, Michigan ENGR111 (Mean) CVEN349 (Mean) Significance* Information Processing 60.68 60.29 0.930 Test Strategies 64.33 55.18 63.27 59.29 0.794 0.342 Anxiety 60.52 67.12 0.147 Attitude 42.47 63.30 34.56 59.29 0.080 0.397 Concentration 61.31 54.56 0.144 Self-testing 52.47 60.24 55.23 37.92 45.04 47.65 0.006 0.005 0.134 LASSI SCALE Skill Component Selecting Main Ideas Will Component Motivation Self-regulation Component Study Aids Time Management Lifelong Learning at Penn State Self-Directed Learning Readiness Survey (SDLRS), Guglielmino & Associates, http://www.guglielmino734. com/prod01.htm, March 2003. 27 “Although the data suggest a slight upward upward trend, the trend proved not to be statistically significant based upon an analysis of variance (ANOVA). Thus the cross-sectional study did not find evidence of an increase in readiness for selfdirected learning, even for students in the later semesters who are taking elective courses and their capstone courses.” Litzinger, T., Wise, J., Lee, S., and Bjorklund, S. (2003) Assessing Readiness for Self-directed Learning, Proceedings, ASEE Annual Conference Challenge of Problem Solving “Despite individual professors’ dedication and efforts to develop problem solving skill, “general problem solving skill” was not developed in the four years in our undergraduate program. Students graduated showing the same inability that they had when they started the program. Some could not create hypotheses; some misread problem statements. During the fouryear undergraduate engineering program studied, 1974-1978, the students had worked over 3000 homework problems, they had observed about 1000 sample solutions being worked on the board by either the teacher or by peers, and they had worked many open-ended problems. In other words, they showed no improvement in problem solving skills despite the best intentions of their instructors.” Woods, D. et al (1997) “Developing Problem Solving Skills: The McMaster Problem Solving Program,” Journal of Engineering Education, Western Michigan University, 18 September 2003, Kalamazoo, Michigan Challenge of Problem Solving • Ineffective approach #1. give the students open-ended problems to solve; This, we now see, is ineffective because the students get little feedback about the process steps, they tend to reinforce bad habits, they do not know what processes they should be using and they resort to trying to collect sample solutions and match past memorized sample solutions to new problem situations. Western Michigan University, 18 September 2003, Kalamazoo, Michigan Challenge of Problem Solving • Ineffective approach # 2: Show them how you solve problems by working many problems on the board and handing out many sample solutions • This, we now see, is ineffective because teachers know too much. Teachers demonstrate "exercise solving". Teachers do not make mistakes; they do not struggle to figure out what the problem really is. They work forwards; not backwards from the goal. They do not demonstrate the "problem solving" process; they demonstrate the "exercise solving" process. If they did demonstrate "problem solving" with all its mistakes and trials, the students would brand the teacher as incompetent. We know; we tried! Western Michigan University, 18 September 2003, Kalamazoo, Michigan Challenge of Problem Solving • Ineffective approach #3: Have students solve problems on the board • Different students use different approaches to solving problems; what works for one won't work for others. When we used this method as a research tool, the students reported "we learned nothing to help us solve problems by watching Jim, Sue and Brad solve those problems!" Western Michigan University, 18 September 2003, Kalamazoo, Michigan Challenge of Problem Solving • Through four research projects we identified why and how these and other teaching methods failed to develop process skills and which methods were successful in developing the skills • Woods, D.R., J.D. Wright, T.W. Hoffman, R.K. Swartman and I.D. Doig (1975) "Teaching Problem Solving Skills," Annals of Engineering Education, 1, 1, 238-243. • Woods, D.R. et al. (1979) "Major Challenges to Teaching Problem Solving" Annals of Engineering Education, 70, No. 3 p. 277 to 284, 1979 and "56 Challenges to Teaching Problem Solving" CHEM 13 News no. 155 (1985). • Woods, D.R. (1993a) "Problem solving - where are we now?" J. College Science Teaching, 22, 312-314. • Woods, D.R. (1993b) "Problem solving - what doesn't seem to work," J. College Science Teaching, 23, 57-58. • Woods, D.R. (1993c) "New Approaches for developing problem solving skills," J. College Science Teaching, 23, 157-158. Western Michigan University, 18 September 2003, Kalamazoo, Michigan Challenge of Engineering Design The literature is filled with positive comments from students, instructors, and industrial sponsors who have participated in capstone design courses. The vast majority of participants feel that the course benefited all involved. The nature of capstone design courses, however, often leads to a purely subjective evaluation with little or no “hard evidence” of actual benefits. Born, for example, does not attempt to prove the value of senior level design courses. He simply states that he is convinced from his experiences that such courses are valuable. Other educators have similar “feelings” as to the relative costs and benefits of capstone design courses. Dutson, A.J., Todd, R.H., Magleby, S.P., Sorensen, C.D., (1997) “A Review of Literature on Teaching Engineering Design Through Project-Oriented Capstone Courses.” Journal of Engineering Education Western Michigan University, 18 September 2003, Kalamazoo, Michigan Challenge of Transfer Researches posed this problem to people. "Suppose you are a doctor faced with a patient who has a malignant tumor in his stomach. It is impossible to operate on the patient, but unless the tumor is destroyed the patient will die. There is a kind of ray that can be used to destroy the tumor. If the rays reach the tumor all at once at a sufficiently high intensity, the tumor will be destroyed. Unfortunately, at this intensity the healthy tissue that the rays pass through on the way to the tumor will also be destroyed. At lower intensities the rays are harmless to healthy tissue, but they will not affect the tumor either. What type of procedure might be used to destroy the tumor with the rays, and at the same time avoid destroying the health tissue?" Western Michigan University, 18 September 2003, Kalamazoo, Michigan Challenge of Transfer Consider the following story "A small country was ruled from a strong fortress by a dictator. The fortress was situated in the middle of the country, surrounded by farms and villages. Many roads led to the fortress through the countryside. A rebel general vowed to capture the fortress. The general knew that an attack by his entire army would capture the fortress. He gathered his army at the head of one of the roads, ready to launch a full-scale direct attack. However, the general then learned that the dictator had planted mines on each of the roads. The mines were set so that small bodies of men could pass over them safely, since the dictator need to move his troops and workers to and from the fortress. However, any large force would detonate the mines. Not only would this blow up the road, but it would also destroy many neighboring villages. It therefore seemed impossible to capture the fortress. However, the general devised a simple plan. He divided his army into small groups and dispatched each group to the head of a different road. When all was ready he gave the signal and each group marched down a different road. Each group continued down it road to the fortress at the same time. In this way, the general captured the fortress and overthrew the dictator." Western Michigan University, 18 September 2003, Kalamazoo, Michigan Challenge of Transfer • After the subjects read and summarized this story, they were asked to solve the tumor problem under the guise of a separate experiment. • Given the clear analogy, you might think that performance would be near ceiling. Surprisingly, only 30% of the subjects offered a convergence solution. • Moreover, when these same subjects were given the suggestion that they should use the General story, 80% provided a convergence solution. • This finding demonstrates that half the subjects could apply the General story to the tumor problem when they were instructed to but did not do so on their own. Western Michigan University, 18 September 2003, Kalamazoo, Michigan Focusing Activity (8 minutes) • INDIVIDUALLY – use 3 minutes to write your description of learning, what it is, what it looks like, how you might recognize when it has occurred, etc. • AS A PAIR – use 5 minutes to discuss descriptions with someone sitting next to you. If you have additional time, develop a consensus description of learning. Western Michigan University, 18 September 2003, Kalamazoo, Michigan Focusing Activity • ?? Western Michigan University, 18 September 2003, Kalamazoo, Michigan Individual Exercise -- Writing Learning Objectives • Individually write a set of objectives for a topic representing a few classes • Something that you recently did in class • Follow your team’s guidelines • Questions to consider about your objectives • • • • Do they define student behavior? Are they observable, measurable? Can you write homework & exam problems? Are they consistent with the instructor’s intent? • Time: 15 minutes Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise -- Reviewing Learning Objectives • Review each other’s objectives • Questions to consider in reviewing objectives • Do they follow your team’s guidelines • Do they define student behavior? • Are they observable, measurable? • Can you write homework problems & exam questions? • Are they consistent with the instructor’s intent? • 15 minutes Western Michigan University, 18 September 2003, Kalamazoo, Michigan Individual Exercise -- Revising Your Learning Objectives • Rewrite your learning objectives based on your team’s review • Report on biggest improvement • 10 minutes Western Michigan University, 18 September 2003, Kalamazoo, Michigan Biggest Improvement • Put quantifiable goals that students would achieve • Clarify my objectives • Classify learning objectives to go from basic to specific • Make a different level of objectives from whole course to chapter to each lecture • Make few higher level objective to organize multiple lower level objectives to support • Identify more complex goals and separate into simpler components for teaching communications Western Michigan University, 18 September 2003, Kalamazoo, Michigan • Improve clarity Minute Paper • Write a one-sentence answer to the following question: • What is the “muddiest point” about learning objectives? • (What is the most confusing point?) Western Michigan University, 18 September 2003, Kalamazoo, Michigan Muddiest Point • How to incorporate those that relate to attitude and enthusiasm for lifelong learning • Why do we spend time writing things that we do and we do best? • Answer to my question about the level and number of learning objectives that are appropriate for publication in my syllabus. Are there “intermediate” not published objectives? • How to relate a given objective to Bloom’s Taxonomy level, and how important is it to do so? • Ways of measuring each learning objective • How many objectives should be used – level of detail – then how do we practically measure them? • “Being brief” vs. “Being precise and accurate” • Being able to link objectives to larger program outcomes or goals • Organizing subject matter to achieve learning objectives Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise -- Reflection on Learning Objectives • Assume that you are a debate team • Write the single best pro and con arguments for the statement • “Using learning objectives improves student leaning.” Western Michigan University, 18 September 2003, Kalamazoo, Michigan Classroom Assessment Western Michigan University, 18 September 2003, Kalamazoo, Michigan Feedback on Learning Objectives • ?? Western Michigan University, 18 September 2003, Kalamazoo, Michigan Agenda • Background for classroom assessment • Specifications for assessment processes/tools • Team exercise: develop a set of specifications for assessment processes/tools • Workshop exercise: improve sets of specifications for assessment processes/tools • Generating alternatives for assessment processes/tools • Team exercises: select some learning objectives and generate alternative assessment processes/tools • Background on alternatives for assessment processes/tools • Selecting assessment processes/tools • Individual exercise: select a set of learning objectives and generate alternative assessment processes/tools • Individual exercise: select one or more assessment processes/tools that you would use in your course • Team exercise: share and review choices of assessment processes/tools • Review workshop activities Western Michigan University, 18 September 2003, Kalamazoo, Michigan Classroom Assessment Background for Classroom Assessment Session Objective: At the end of the session, participants will describe themselves as more confident in their ability to hold productive conversations with their colleagues regarding the place and importance of assessment instruments and processes in the teaching-learning process Western Michigan University, 18 September 2003, Kalamazoo, Michigan Pre-Assessment • Exercise • Classroom Assessment • Individual • Write a one-sentence answer to the following question: • “What can you do in the last few minutes of class to determine how well your class learned what you taught that day?” Western Michigan University, 18 September 2003, Kalamazoo, Michigan Classroom Assessment • At end of session, participants will be able to: • Define several classroom assessment tools • Discuss the importance of using classroom tools in the teaching-learning process • Write assessment tools for their classes Western Michigan University, 18 September 2003, Kalamazoo, Michigan Resources on the Web • www.hcc.hawaii.edu/intranet/committees/FacDevCo m/guidebk/teachtip/assess-1.htm • www.hcc.hawaii.edu/intranet/committees/FacDevCo m/guidebk/teachtip/assess-2.htm • www.siue.edu/~deder/assess/catmain.html Western Michigan University, 18 September 2003, Kalamazoo, Michigan Types Of Assessment • Assessment • Used in many contexts • Used for several different foci • Classroom • • Teacher focus: concerns your performance Student focus: concerns their individual performances • Program • Alumni (as a group) focus: ABET Criterion 2 • Graduates (as a group) focus: ABET Criterion 3 • Individual graduate focus: Comprehensive exams Western Michigan University, 18 September 2003, Kalamazoo, Michigan Types Of Assessment • Classroom, teacher focus • • Question -- How effective was a lecture, assignment, lab? Tools -- One-minute paper, student survey • Classroom, student focus • • Question – Did a specific student achieve the learning objectives? Tools -- Exams, reports, presentation • Program, graduates (as a group) focus • • Question -- How well did a group of students achieve a set of objectives (outcomes) in a program or course? Tools -- Standardized tests, design project report analysis Western Michigan University, 18 September 2003, Kalamazoo, Michigan Classroom Assessment • Two fundamental questions: • How well are learners learning? • How effectively are teachers teaching? • Deals with better learning and more effective teaching • Provides feedback about effectiveness as teachers • How students learn • How they respond to particular teaching approaches. • Gives students a measure of their progress as learners Modified from www.hcc.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/assess-1.htm Western Michigan University, 18 September 2003, Kalamazoo, Michigan Incorrect Assumptions About Teaching & Learning • Instructors assume students learn what they teach • Tests, concept inventories, and term papers provide disappointing evidence to the contrary • Students have not learned as much or as well as expected • Gaps between what was taught and what was learned • Sometimes considerable gaps • Instructors notice gaps too late to remedy the problems • Classroom assessment can uncover gaps earlier www.hcc.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/assess-1.htm Western Michigan University, 18 September 2003, Kalamazoo, Michigan Classroom Assessment Getting Started • Planning • Select one, and only one, of your classes • Choose a simple and quick technique • Implementing • Make sure the students understand the procedure • Analyze student’s responses as soon as possible • Responding -- “Close the feedback loop” • Tell students what you learned and what you will do about it - motivates students to become actively involved www.hcc.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/assess-1.htm Western Michigan University, 18 September 2003, Kalamazoo, Michigan Classroom Assessment – Five Suggestions • Don’t use any technique that does not appeal to you • Don't make it into a self-inflicted chore or burden. • Try it yourself before you use it with students • Allow more time than you think you will need • To carry out the assessment • To respond to it • Make sure to "close the loop" www.hcc.hawaii.edu/intranet/ committees/FacDevCom/guid ebk/teachtip/assess-1.htm • Let students know • • What you learned from their feedback How you and they can use that information to improve learning Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Process • Brain storm individually -- 1 minutes • Establish consensus as a team -- 4 minutes • Task: Write 2 to 4 guidelines for good classroom assessment tools (Think of guidelines as specifications) • What are the common features? • What should they look like? • Do all guidelines have to apply to all tools? Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Tool should be efficient use of time for instructor and class • Easy for students to use • Can be used across different classes and situations • Built into teaching methodology and doesn’t look like an add-on • Tied to learning objectives or limited number of topics • Flexible, not dependent on particular technology • Done repeatedly throughout semester • Easy for faculty to administer and evaluate • Self-documenting, easy to show results to others • Clear and precise instructions Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Process • Brain storm individually -- 2 minutes • Generate different ideas as a team -- 3 minutes • Task: Generate as many possible good classroom assessment tools that might satisfy your team guidelines • Generate different ideas Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • • • • • • • • Muddiest point Quizzes on specific objectives What is clear?/What is unclear? What did you learn today? Strengths/Improvements/Insights (SII) Self or peer assessment Who did you share X with today? Weekly feedback on homework instruction – what was clear? What was unclear? • Continuous Improvement – What was beneficial? What would you change? Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Process • Brain storm individually -- 2 minutes • Generate different ideas as a team -- 3 minutes • Task: Generate as many possible good classroom assessment tools where the student work product is primarily graphical and the tools might satisfy your team guidelines • Generate different ideas Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • • • • • • • • • • List topics with smiling/frowning faces Flow charts/flow sheets Process flow of your problem solving process Road map to get from A to B Bar graph of confidence related to specific objective Free body diagram Web of association to a particular term Story board of set up or explanation of particular concept Fit concept into network or tree of concepts Draw a concept Western Michigan University, 18 September 2003, Kalamazoo, Michigan Examples of Assessment Techniques • Background Knowledge Probe • Students respond to short-answers or multiplechoice questions • General information on their level of preparation • Minute Paper (most widely used) • Students write brief response to • "What was the most important thing you learned during this class?" • “What important question remains unanswered?“ www.hcc.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/assess-2.htm Western Michigan University, 18 September 2003, Kalamazoo, Michigan Examples of Assessment Techniques (Cont.) • Muddiest Point (simplest technique, remarkably efficient) • Students jot down a quick response to one • "What was the muddiest point in ....... ?” • A lecture, a discussion, a homework assignment • One-Sentence Summary • Students answer the questions "Who does what to whom, when, where, how, and why?" (WDWWWWHW) • Synthesize answers into a simple, informative, grammatical sentence. www.hcc.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/assess-2.htm Western Michigan University, 18 September 2003, Kalamazoo, Michigan Examples of Assessment Techniques (Cont.) • What's the Principle? • Students state the principle that best applies to a few problems www.hcc.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/assess-2.htm Western Michigan University, 18 September 2003, Kalamazoo, Michigan Concept Map • A concept map is a set of nodes that represent concepts connected by a labeled links that describe a link between concepts. Concept A Describe how concept A and concept B are related? Concept B Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise Building a Concept Map • Start with a subset of the concepts on the following page and construct a concept map that shows the concepts you have selected and how they are related. • Exchange concept maps and share insights Western Michigan University, 18 September 2003, Kalamazoo, Michigan Feedback Derivative Finite Element Analysis Integral Linear Momentum Angular Momentum Energy Interest Mass Ideal Gas Law Fick’s First Law Fick’s Second Law Vectors: Dot Product Vectors: Cross Product Ordinary Differential Equations Kirchoff’s Voltage Law Second Law of Thermodynamics Kirchoff’s Current Law Modeling Problem-Solving Force Ohm’s Law Resistance Complex Numbers Logarithmic Function Electric Flux Decision Theory Divergence Indirect Cost Capacitance Bending Moment Feedback First Law of Thermodynamics Entropy Heat Electric Field Magnetic Field Partial Differential Equations Determinants Return on Investment Phasors Brainstorming Exponential Function Conductivity Chemical Kinetics Specific Heat Elasticity Malleability Plasticity Resiliency Permittivity Current Electric Potential Curl Presentation Skills Democracy Profit Density Molecule Phase Shear Rheology Frequency Response Eigenvalue, Eigenvector Sinusoidal Functions Work Displacement Velocity Acceleration Resistivity Leadership Hess’ Law Zeroth Law of Thermodynamics Electric Potential Magnetic Flux Design Maxwell’s Equations Power Ductility Spring Constant Stress Strain Partial Derivative Permeability Charge Magnetic Potential Gradient Paragraph Rate of Return Frequency Atom Root Locus Torque Inductance Torsion Polymer Kinetic Theory of Gases Cowan’s Teaching Examples • Bridge design • Design and build two different bridges and grade on the lower performance design • Problem-solving script • Illustrate script for one type of problem, ask students to develop a script for another type of problem Cowan, J. (1998) On Becoming an Innovative University Teacher: Reflection in Action. Buckingham: SRHE and Open University Press. Western Michigan University, 18 September 2003, Kalamazoo, Michigan Some Personal Observations Western Michigan University, 18 September 2003, Kalamazoo, Michigan Relating Student Performance On Exams To Objectives • Write exam using objectives • Select objectives for exam questions from list • Many objectives -- test questions represent a sample • Exam question may involve more than one objective • Use some “hard” and some “easy” questions • Identify questions (& objectives) a high percentage missed • Review idea in class -- give additional work • Modify lecture, reading, or homework for future • Change the objective Western Michigan University, 18 September 2003, Kalamazoo, Michigan Using Cooperative Learning In-class Exercises • Examine students work during the in-class exercise • If all have a good approach -- may be wasting time • If all are lost -- may need more explanation • If one-half to two-thirds have a good approach -- level and pace are right • Collect and show a few solutions to in-class exercises • If all have correct approach -- may be wasting time • If all are wrong -- may need more explanation • If one or two are correct -- level and pace are right Western Michigan University, 18 September 2003, Kalamazoo, Michigan “One-Minute” Papers • Common questions • • • • What one thing should be changed about ____? What one thing should not be changed about ____? What do you think about ____? What is the “muddiest” point about ____ ? • Ask about • • • • Course or lecture Text or chapter Assignment or test Teaching style or class activity Western Michigan University, 18 September 2003, Kalamazoo, Michigan Some Colleagues’ Observations Western Michigan University, 18 September 2003, Kalamazoo, Michigan Informal Assessment Techniques • General Guidelines • Keep them anonymous • Use then frequently – better feedback • Close the loop • Let students know results of the process David Cordes Western Michigan University, 18 September 2003, Kalamazoo, Michigan Informal Assessment Techniques - Daily Activities • One-minute paper • At the end of the lecture, ask students for: • The most important topic that we covered today • The one topic you are still confused about • Single sheet of paper, no names • Can read on the way back to the office • Look for “common problems” • Look for “did they understand my focus?” David Cordes Western Michigan University, 18 September 2003, Kalamazoo, Michigan ME 360 - Plus / Delta Assessment #1 • On one side of “sticky” pad • Put a “+” in upper left hand corner • What is something that worked well or made more sense in lab this week? • On other side of “sticky” pad • Put a “D” in upper left hand corner • What is something that could have been done better in lab this week? • “Stick” on the door on your way out Joey Parker Western Michigan University, 18 September 2003, Kalamazoo, Michigan ME 372 - 1st Day of Class • What are a valid set of units for a mass moment of inertia? (Dynamics concept) • What is the difference between a capacitor and a resistor? (Circuits concept) • What is the equation of the straight line that passes through the points X=2, Y=7 and X=7, Y=2? (Math concept) Joey Parker Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Process • • • • Brain storm individually -- 2 minutes Establish consensus as a team -- 5 minutes Report team results -- 3 minutes Revise products as a team -- 2 minutes • Task: Write 3 to 5 guidelines for good classroom assessment tools (Think of guidelines as specifications) • What are the common features? • What should they look like? • Do all guidelines have to apply to all tools? Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Simple, quick, easy to administer and evaluate • Provide immediate feedback to students – they need to know you’re using the results • Tied to a specific topic or objective • Gives usable, valuable information • Identification of strengths and areas of improvement • Give clear and precise instruction to the student • Using graphical and textual feedback • Result in action Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Process • • • • Brain storm individually -- 2 minutes Establish consensus as a team -- 5 minutes Report team results -- 3 minutes Revise products as a team -- 2 minutes • Task: Generate as many possible good classroom assessment tools that might satisfy your team guidelines • Generate different ideas that you have generated or have seen Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Ask students to identify an application based on concept presented in class • Person A teaches Person B, and then they write what was hard to convey and what was hard to learn • Identify an objective for class and rate their comfort level with number or bar graph • Ask student to give example to transfer example in class to another context • Students examine 2-3 methods/opinions/processes for similar purposes, ask groups to analyze similarities and differences, solicit results, and then invent their own that combines methods/opinions/processes • Ask students to identify concept based on a diagram. • Ask students to write problem that could be solved by conceptWestern presented in class Michigan University, 18 September 2003, Kalamazoo, Michigan Individual Exercise • Individually write a set of assessment tools for a class or a topic representing a few classes • Follow your guidelines • Consider the following questions about your tool • • • • • Can your students understand the task? Can your students do the task quickly? Can you analyze the results quickly? Can you summarize and report the results easily? Does it assess student learning? • 5 minutes Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Review each other’s objectives and assessment tools • Consider the following questions • • • • • • Does the tool follow your guidelines? Can your students understand the task? Can your students do the task quickly? Can you analyze the results quickly? Can you summarize and report the results easily? Does it assess student learning? • 15 minutes Western Michigan University, 18 September 2003, Kalamazoo, Michigan Individual Exercise • Rewrite your assessment tools based on your team’s review • Identify the major improvement • 5 minutes Western Michigan University, 18 September 2003, Kalamazoo, Michigan Minute Paper • Write a one-sentence answer to the following question: • What is the muddiest point about classroom assessment? Western Michigan University, 18 September 2003, Kalamazoo, Michigan Individual Exercise • How can students be more involved in assessing each other, as well as the self-assessment we have largely concentrated on, and do this in reasonable time. • How often is a reasonable amount of assessment? How much is too much? • Going from creating an assessment tool to actually implementing the assessment. • Incorporation of feedback will not be every time easy and straight forward • Will students see the value of classroom assessment? Yes, with good approaches. Yes, with good feedback. • Being able to design a clear, concise assessment tool that is both beneficial to students and professor Western Michigan University, 18 September 2003, Kalamazoo, Michigan Team Exercise • Assume that you are a debate team • Write the single best pro and con arguments for the statement • “Using classroom assessment tools improves student learning.” Western Michigan University, 18 September 2003, Kalamazoo, Michigan Workshop Objectives & Action Items • Recall objectives • At end of session, participants will be able to define, discuss, & write • Learning objectives for their courses • Assessment tools for their classes • Workshop provided a structure for & experience in writing • Learning objectives • Assessment tools • Your charge – In one of your courses next semester use • Learning objectives • Classroom assessment tools Western Michigan University, 18 September 2003, Kalamazoo, Michigan Questions? Western Michigan University, 18 September 2003, Kalamazoo, Michigan Some Personal Observations Western Michigan University, 18 September 2003, Kalamazoo, Michigan