LO EXAM SETTING GUIDELINES BLOOMS REVISED TAXONOMY Levels of cognitive demand Taxonomy of thinking skills - Levels of cognitive demand Question keywords are also called 'command' words. These are the words in your exam questions that tell you what the examiner wants you to do. By understanding these command words, you are on your way to understanding your exam questions. The table below lists some of these command words. Category Evaluation Definition Judge value of something regarding criteria. Support judgement; challenges for deeper meaning Analysis Understand how parts relate to a whole. Understand structure and motive. Application Transfer knowledge Demonstrate, use, guide, map, learned in one situation to chart, build, cook, arrange, illustrate, another; familiar or new apply, sketch, locate, construct, solve.... Demonstrate a basic Give examples, explain, summarise, understanding of translate, show symbols, edit, concepts and curriculum. discuss, identify, interpret.... Translate to other words; recall particular information; make own meaning Understanding (Comprehension) Action Words Evaluate, give opinion, viewpoint, recommend, criticize, choose/decide, estimate, judge, predict, rate/score, select, support, justify, argue, conclude, why do you think.... Investigate, classify, categorise, compare, distinguish, report on… Remembering (Knowledge) Ability to remember something previously learned, recognise, recall relevant information Tell, recite, list, memorise, remember, define, locate, name, match, recall, repeat, state, outline, repeat… Questions for the revised bloom’s taxonomy LEVEL 1- REMEMBERING LEVEL 2 - UNDERSTANDING LEVEL 3- APPLYING Exhibit memory of previously learned material by recalling facts, terms, basic concepts, and answers Demonstrate understanding of facts and ideas by organizing, comparing, translating, interpreting, giving descriptions, and stating main ideas Solve problems to new situations by applying acquired knowledge, facts, techniques and rules in different ways Key words Questions Key words Questions Key words define What is …? classify Apply find Where is …? compare How would you classify the type of …? how How did ___ happen? contrast How would you compare …? Contrast …? choose label list match name relate Why did …? When did …? How would you show …? select show tell what when where which who demonstrate explain extend illustrate interpret outline Who were the main …? Which one …? How is …? When did ___ happen? How would you explain …? How would you describe..? relate summarize translate construct develop Will you state or interpret in your own words …? How would you rephrase the meaning …? What facts or ideas show …? rephrase show build experiment with identify interview make use of model What is the main organize idea of …? plan Which select statements support …? solve Can you explain what is happening …? What is meant …? What can you say about …? utilize Questions How would you use …? What examples can you find to …? How would you solve ___ using what you’ve learned …? How would you organize ___ to show …? How would you show your understanding of …? What approach would you use to …? How would you apply what you Can you recall …? Which is the best answer …? learned to develop …? Can you select …? How would you summarize …? What other way would you plan to …? Can you list the three …? What would result if …? Who was …? Can you make use of the facts to …? What elements would you choose to change …? What facts would you select to show …? What questions would you ask in an interview with …? LEVEL 4 - ANALYSING LEVEL 5 - EVALUATING LEVEL 6 - CREATING Examine and break information into parts by identifying motives or causes. Make inferences and find evidence to support generalizations. Present and defend opinions by making judgments about information, validity of ideas, or quality of work based on a set of criteria. Compile information together in a different way by combining elements in a new pattern or proposing alternative solutions. Key Words Questions Key Words Questions Key Words Questions analyse What are the parts or features of …? agree Do you agree with the actions…? With the outcome…? propose What changes would you make to solve …? What is your opinion of …? change assume categorize classify appraise assess How is ___ award related to …? adapt build How would you improve …? compare conclusion contrast discover Why do you think …? What is the theme …? choose compare conclude What motive is there …? criteria distinguish Can you list the parts …? decide divide deduct What inference can defend you make determine …? dissect examine function inference inspect list criticize How would you prove …? Disprove…? choose Can you assess the value or importance of …? compile Would it be better if …? create Why did they (the character) choose …? disprove What conclusions dispute can you draw estimate …? What would you recommend…? evaluate What would you cite to defend the actions …? motive How would you relationships classify...? simplify How would you survey categorize...? take part in Can you test for identify the different theme parts …? What evidence can you find …? What is the relationship between …? explain importance influence interpret judge justify mark measure opinion perceive prioritize Can you distinguish between …? prove What is the function of …? recommend What ideas justify …? rate rule on select support combine compose construct delete design develop discuss What would happen if …? Can you elaborate on the reason …? Can you propose an alternative…? Can you invent …? How would you adapt ___ to create a different …? elaborate How would you rate the …? How could you determine…? estimate formulate happen imagine improve What choice would you have made …? invent How would you prioritize …? maximize What judgment would you make about …? modify Based on what you know, how would you explain …? originate make up minimize original plan predict What information propose would you use to solution support the view…? solve How would you justify …? suppose test How could you change (modify) the plot (plan) …? What could be done to minimize (maximize) …? What way would you design …? What could be combined to improve (change) …? Suppose you could ___ what would you do …? How would you test …? Can you formulate a theory for …? Can you predict the outcome if …? value What data was used to make the conclusion…? theory What was it better that …? How would you compare the ideas …? People …? How would you estimate the results for …? What facts can you compile …? Can you construct a model that would change …? Can you think of an original way for the …? Bloom's Taxonomy Question Stems Remembering Understanding Applying Analysing Evaluating Creating What happened after...? Do you know of another instance where...? Which events could not have happened? Is there a better solution to...? Can you group by characteristic s such as...? If ____ happened, what might the ending have been? If you were ____, what would you do differently? How many...? What is...? Who was it that...? Can you name...? Find the definition of ... Describe what happened after... Who spoke to...? Which is true or false...? Can you explain why...? Can you write in your own words...? How would you explain...? Can you write a brief outline...? What do you think could have happened next...? Who do you think...? What was the main idea? Can you clarify...? Which factors would you change if...? What questions would you ask of...? From the information given, can you develop a set of How is ___ similar to ___? What do you see as other possible outcomes? Why did ... changes occur? Can you explain what Judge the value of ____. What do you think about...? Can you defend your position about...? Do you think ___ is a good or bad thing? How would you have handled...? What changes to ____ would How many ways can you think of to...? Predict what would be true if____. How can you explain...? Hypothesiz e what would happen if ... Where is...? Which one...? When did...? Who were the main characters? Describe the setting. Can you tell three...? When did ____ happen? How did the story end? Can you illustrate...? instructions about...? Does everyone act in the way that ____ does? What examples can What are you find to some of the support...? problems of...? Think of a situation that Can you occurred to a distinguish person in the between...? selection and What were tell what you some of the would have motives done. behind...? What would What was the result if...? turning point? Using what What is the you know, theme of...? how would you solve...? What motive How would you compare/contras t...? What is meant by...? Retell the story. How did the character feel about...? Explain what is happening when the author says... Explain why the story has the title that it does. Look at the picture. Explain what happened before and after the picture. What would you do if you could go to the place where the main character lived? If you had to cook a meal for the main character, what would you make? Write/tell what you have learned and how you can use this information in your life. must have happened when...? is there...? What conclusions can you draw about...? How would you classify/categ orize...? What inferences can you make...? What was the problem with...? What was the funniest part? What was the most exciting part? you recommend? Do you believe...? How would you feel if...? How effective are...? What are the consequences ...? What influence will ___ have on our lives? What are the pros and cons of...? Why is ___ of value? What are the alternatives? Who will gain and who will lose? Compare two characters in the selection. Which was a better person and why? Which character would you most like to spend the day with and why? Do you agree with the actions of...? How would you improve...? What changes would you make to...? Suppose you could ____. What would you do? How would you rewrite the selection from ______'s point of view? Can you design a ... to...? Can you see a possible solution to...? If you had access to all resources, how would you deal with...? Why don't you devise your own way to...? What would have What was the saddest part? Distinguish between two facts and opinions. What would be a good title for this story? What choice would you have made about...? What data was used to make the conclusion? happen if...? How many ways can you...? Can you create new and unusual uses for...? Can you develop a proposal which would...? Levels of Questions with Bloom's Taxonomy (Image Credits) Level 1. Knowledge When was this picture taken? Where was this picture taken? Question cues: List, name, mention, outline Level 2. Comprehension What is happening in this picture? Why are these boys dressed like this? Question cues: Describe, define, identify Level 3. Application How would you describe the photograph to others? What caption would you write for this photograph (say, in a newspaper)? Question cues: Modify, solve, change, explain ,discuss Level 4. Analysis Why are these boys here and not in school? What do you know about their lives based on this photo? Question cues: Analyze, separate, compare, contrast Level 5. Synthesis What might these boys say about their work in an interview setting? What might they say about their future? Question cues: Create, construct, plan, role-play,analyse Level 6. Evaluation What is the significance of this photo for the time period depicted? Compare this photo with one of three boys from today of the same age. How are their lives similar? How are they different? Question cues: Suggest, criticize, propose SOME BASIC GUIDELINES ON EXAM SETTING The underlying principles of good question design Test items should be really difficult for people who don't understand the subject material, but they should be straightforward for those who do. If an item is difficult because of complicated wording (e.g., double negatives) or vocabulary, you will end up testing language skills rather than content. The principles that underlie good question design are – i. Clarity ii. Reliability, iii. Validity iv. Authenticity v. Fairness i) Clarity “Nothing in the content or structure of [a test] item should prevent an informed learner from responding correctly.” Gronlund (1998) The clarity of an exam question may be compromised by unclear test instructions, confusing and ambiguous terminology, overly verbose and complicated vocabulary and/or sentence structure plus unnecessary and distracting detail (Gay, L.R., &Airasian, P. 2000). The layout of a question is also very important in conveying clarity – particularly in longer, multisectioned or data handling styled questions. EXAMPLE Testing for clarity – contrast the following versions of the same exam question. (essay format answer required): Version A: Public health policy in the United Kingdom underwent a number of significant changes during the Twentieth Century that can be directly attributed to the needs and exigencies brought about by international conflict. Some of the changes and developments that resulted to health systems and service delivery are still with us today and it is important that we understand the background of circumstances that influenced the decisions that were made. Provide a short analysis charting what you consider to be the main transitions in public health policy brought about by the unique needs and challenges, both direct and indirect, of an environment of international conflict, within the UK health systems specifically, using the Second World War as an example. Version B: Compare the advances in UK public health policy pre- and post-Second World War. Think about points such as: unclear test instructions, confusing and ambiguous terminology, being overly verbose (too packed) using complicated vocabulary, difficult or poor sentence structure, unnecessary and distracting detail. ii) Reliability Does the question allow markers to grade it consistently and does it allow markers to discriminate between different levels of performance? This frequently depends on the quality of the marking guidance and clarity of the assessment criteria. The likelihood of eliciting an accurate measure of the learner’s ability will be increased when learners are provided with a variety of ways to demonstrate their knowledge and skills iii) Validity A valid examination question measures achievement of the intended learning outcomes of the topic (not just what is easy to measure!). The form of the examination question may also be of importance in ensuring validity. For example, examination questions that are short answer questions are a good way of assessing greater breadth of material covered in a course but tend to focus on testing attainment of knowledge and application of knowledge. Whilst longer essay style questions allow a more in depth exploration of subject material and require a candidate to build and structure an argument or explain a complex concept with wide reference to examples and readings. iv) Authenticity Authenticity is the need to match the style and approach of question setting to the reality of practice. When the exam seeks to test a candidate‟ s knowledge of how something works, the order or sequencing of events, the interplay between contributing factors etc – it can be very important to ensure this is built into the question formatting and context setting (original)– to allow authenticity. v) Fairness You need to give learners a fair chance to demonstrate what they know and can do and to be able to succeed in examinations. Fairness can be facilitated by being very clear about expectations in the learners performance. Multiple Choice Questions The grammar you use in your answer choices may influence how your learners choose an answer. Make sure all your choices are grammatically parallel. In other words, if you ask, “Where does a rabbit live?”, phrase all your answers as prepositional phrases. A. In a house B. In a car C. Under a bridge D. In a cage. If you offered choice E. dangerously close to the highway, it will obviously be incorrect because it is phrased differently than the other choices . Watch your vowels. Always give the possibility of either a or an when asking a question. You can include the choice either in the question itself or in the answers. What does a rabbit eat for breakfast? A. a carrot B. a protein shake C. an egg D. a piece of toast, (or a rabbit eats a(n)___________ for breakfast). Keep all your answers around the same length. If one answer is significantly longer or shorter than all the rest, it will likely be the best choice or an easy elimination. Try to write about the same amount for each answer option to avoid give away answers. Keep the stem short and to the point. Make sure that the stem does not include words from the distractors. Use similar statements in the distractors. Do not ask a question in the negative. Minimise repetitions of words in sentences. Difficulty Levels Start your test with the easiest questions and move toward those that are more difficult Though you might want to mix up the order to challenge the learners, going from easier to harder questions alleviates stress for the learners and makes for a better testing experience. Test multiple learning levels. The majority of your questions should target levels of recall, comprehension and application. Do not be afraid, however, to add questions testing higher levels of learning like analysis, synthesis and evaluation in Section B and C (see Bloom’s taxonomy of learning for more information on learning levels). Include the marks each section is worth. This way learners can budget their time to be most impactful for them. Neither you nor they want them to spend ten minutes struggling to answer a question of minimal importance. Let them have full knowledge of what is weightiest as they take the exam so they can prioritize as they take it. . Checklist for reviewing draft exam questions 1. What is the question intended to measure? (eg factual recall, data processing/analysis skills, problem-solving skills, policy analysis skills, critical analysis skills) 2. What else does it actually measure? (eg does it rely too much on factual recall?) 3. Does it measure what we said we would measure? (Is it aligned to the content covered in the ATP/CAPS?) 4. How well does the question relate to the content? 5. Is the language simple, clear, unambiguous and straightforward? 6. What are the key words describing the task? Are they clear?(eg: list, define, „suggest reasons behind the effect‟ are better than interpret, discuss, evaluate) 7. Is the language used easy to understand, including by candidates for whom English is not their first language (eg does it use colloquial phrases)? 8. Check punctuation and grammar as this can markedly change the meaning of sentences (eg “a sheep eats, leaves and grass”). 9. Does the question advantage or disadvantage any group of learners? 10. How reliably can the answers be marked? 11. If the question is in sections, is the division of percent of marks between sections appropriate? 12. Can the question be completed in the time available (including reading, thinking and reviewing time), including those for whom English is not their first language? 13. Does the question lead to answers which will distinguish between weak and strong candidates, eg are there elements for candidates to demonstrate distinction-level skills/knowledge? 14. Are the choice questions in Section C of an equal level of difficulty? 15. Are sources relevant, current and up to date? 16. Are the sources mainly a stimulus and do not include the answers for learners? 17. Are sources correctly acknowledged? 18. Does the marking guideline provide for many possible responses? 19. Does the marking guideline clearly show how marks are to be awarded? 20. Are there any repetition of questions? Check and countercheck To exclude unsatisfactory questions, it is useful to do the following: Prepare your suggested answers as you complete questions as it provides a check on the complexity of the questions set and whether the time allotted is realistic. Work as a team and test the questions for ambiguity. Check on technicalities. Is the paper well laid out, legible and free of typographical or technical errors? Are the instructions clear and concise? Is the time to be spent answering each question and the maximum marks allocated to each question indicated to guide learners NB: Set original questions. Do not include questions from previous papers exactly as they were asked. Do not include words directly from the textbooks. Marking guideline must be clear. Constantly refer to the ATP/CAPS for clarity and content focus. BE BRAVE, BE BOLD, BE CONFIDENT AND BE CERTAIN OF YOUR CONTENT! GOOD LUCK AND ENJOY THE EXPERIENCE!
