Guides for Learning and Teaching: Writing Learning Outcomes 1 NTU Centre for Professional Learning and Development Resources for Learning and Teaching Writing Aims and Learning Outcomes Introduction This guide contains information to help you create well-designed aims and intended learning outcomes whether for individual sessions, a module or a course. For convenience, the guide talks about aims and learning outcomes in the context of individual teaching sessions, although the lessons apply to writing aims and learning outcomes at module and programme levels as well. Since The Dearing Review (NCIHE, 1997) recommended that learning intentions be expressed as intended learning outcomes in programme specifications, the national Quality Assurance Agency (QAA) has championed outcomes-based design, an approach which is well-embedded into programme design at NTU through an outcomes-based methodology known as ‘constructive alignment’. In this, the ‘alignment’ refers to the teacher’s practice of ensuring that learning activities and assessments are designed to match or align with the learning outcomes. Thus, any aims or outcomes you set for your individual sessions will align with those of the module or wider programme overall to help ensure cohesion. Information supplied in this guide addresses the following questions: What’s the difference between an ‘aim’ and a ‘learning outcome’ and does it matter? (see Section 1) How do aims and learning outcomes relate to learning ‘objectives’? (see Section 2) What contribution do aims and learning outcomes make to session design and delivery? (see Section 3) Do I share my aims and learning outcomes with my students? When? And how? (see Section 3) What are the implications of that contribution for tutors writing aims and learning outcomes? (see Section 4) How do I create learning outcomes that are sufficiently specific? (see Section 5) What information do you need to consider when designing the aims and outcomes for any session? (see Section 5) What are the wider implications of aims and learning outcomes for session design, planning and delivery? (see Section 6) Tips on writing learning outcomes (see Section 7) How this guide relates to the UKPSF and NTU Professional Standards for Teaching and Supporting Learning in HE Areas of Activity Areas of Core Knowledge (Ks) Professional Values (Vs) 2 A1, A2 K2, K3 V3 Section 1: Defining aims and learning outcomes What’s the difference between a session ‘aim’ and session ‘learning outcome’? An ‘aim’ sums up the overall purpose of the session, module or course, and is written from the tutor’s perspective. It’s a summing-up of what I want my session to achieve. More general verbs are used such as ‘learn’ or ‘do’ or ‘understand’ or ‘get to know’ or ‘introduce’. Aims are: The overall intention, purpose or goal of the session The general ‘what’ usually summed up in one simple sentence (possibly two!) Written from the facilitator/tutor’s perspective, ‘In this session my aim is to…’ Typically 1 per session A learning outcome, on the other hand, is much more specific. Learning outcomes are more statements of what you intend the students to have the opportunity to learn as a result of completing the session, so it is helpful to phrase them as ‘By the end of this session you will have the opportunity to….’ They are more about the destination you intend your students to get to as a result of completing the session. As such, they have to measurable – if they state what you expect your students to learn, it implies that it should be possible to demonstrate and evaluate the extent to which your students have achieved them. The verbs in learning outcomes need, therefore, to be more specific and measurable. Tip- the abbreviation, ‘LO’ is often used for simplicity - and outcomes can be numbered for easy reference eg LO1 and LO2. Sometimes you’ll see the abbreviation ‘ILO’ used – stands for Intended Learning Outcome. Learning outcomes are: A statement of intention specifying what the student can be expected to ‘learn’ as a result of completing the session, ‘By the end of this session you will have the opportunity to be able to….’ Measurable / assessable (you need to be able to judge whether students have achieved what was intended) Typically 2-3 per session Example of an AIM: “The aim of my session is to help you understand the concept of…………..” Example of a Learning Outcome: “By completing this session, you will have the opportunity to define and distinguish the concepts of….” In Section 3 below we look at the key differentiating factor between aims and learning outcomes – the action verbs used to define the intended ‘learning’. Before that – just a quick word in Section 2 about another term you might see ‘learning objective’. Section 2: What are learning objectives? Sometimes the term, ‘learning objective’ is used. In some disciplines the term, ‘learning objective’ has the same meaning as ‘learning outcome’, whereas in others, it is synonymous with aims. In others, it might even be somewhere between aims and learning outcomes! 3 Should you come across the term, ‘objective’, check with the module, course or session leader the intended meaning of the term - whether the learning objectives relate to the learning outcomes or to aims. At NTU, the term, ‘outcome’ is recommended to ensure consistency. For further information please refer to the NTU Quality Handbook on the CADQ website, which includes templates. Section 3: Distinguishing learning outcomes from aims – the role of the action verbs Exercise 3.1: Select an aim and a couple of learning outcomes from any of your sessions, modules or courses and write them below: Aim: LO1: LO2: What would you say is the key distinguishing feature between your aims and outcomes? Exercise 3.1 Feedback: The problem with the terms ‘know’ and ‘understand’ Take a look at the verbs you use to define the type of learning you intend your students to be able to achieve. Do you find yourself using words like ‘know’ and ‘understand’ in your learning outcomes? Our first step is often to identify in general terms what we want our students to ‘understand’ or ‘know’ or ‘learn’ something. This is fine when we are defining the aims for the session. However, as learning outcomes are meant to be ‘specific’ and ‘measurable’, we have to ask how specific and measurable are terms such as ‘know’ and ‘understand’? What will ‘knowing’ or ‘understanding’ look like? How complex do we expect it to be? For instance do we want our students to be able to recognise, identify, state, explain, analyse, evaluate, or create? For those are very different demonstrations of ‘understand’ or ‘know’. Verbs such as ‘know’ or ‘understand’ are therefore, open to further interpretation and definition – they can be measured differently - to different levels of academic complexity. When creating outcomes, it is preferable, therefore, to use more specific verbs. If you were designing a session aimed at helping new first year students to ‘Understand how to cook eggs healthily’, you might interpret ‘understanding’ in different ways. For example – it could be defined as being able to list or explain different methods of cooking eggs, whereas others might view it as being able to apply that knowledge to create a cooked egg. 4 The words ‘know’ and ‘understand’ are too vague therefore for learning outcomes and need to be more specifically and precisely defined, to indicate the nature and scope of learning we intend our students to have the opportunity to achieve. This is illustrated with the following examples: Examples of learning outcomes which are actually aims, and how they could be made more specific and measurable to become learning outcomes: LO1: “To be familiar with research methodology” could become: LO1: By the end of this course you will be able to apply research methodology effectively, or LO1: By the end of this session you will be able to identify a number of research methodologies, and explain their significance when constructing a research proposal LO2: “Understand concepts and terminology of pattern making” could become: LO2 Create a shirt pattern, or LO2 Critically evaluate the contribution of pattern making to design LO3: “Understand why E=MC2 could become LO3 Explain why E=MC2 LO4: “Understand the responsibilities of staff development managers in organisations”, could become either of the following: LO4 Identify the key responsibilities of staff development managers within organisations, or LO4 Discuss critically the contribution staff development managers make to an organisation’s effectiveness LO5: “By the end of this session you will know how to relieve financial recession in the UK”, could become the following: LO5 By the end of this module, you will be able to critically evaluate the potential of quantitative easing for relieving UK recession, or LO5 By the end of this module, you will be able to identify and explain a range of methods for addressing UK financial recession And just to keep with the analogy of cooking with eggs used earlier: LO6: “By the end of this session you will have the opportunity to be able to understand how to cook eggs healthily” could become: LO6: Demonstrate your ability to successfully cook eggs in two different ways to produce healthy meals Note the more specific verbs (underlined and bold) used in the revised examples above. Now you can re-evaluate the design of the aims and learning outcomes you listed in Exercise 1 earlier. Are the outcomes you create sufficiently specific and measurable, or are they almost indistinguishable from aims? 5 Section 4: Things we can use to select appropriate action verbs in our learning outcomes In this section we consider the frameworks and standards which we can use to select appropriate action verbs for our learning outcomes. 4.1 Taxonomies of learning objectives (such as Bloom et all 1956 in Atherton 2009) are useful frameworks for helping us define learning more specifically. Bloom’s taxonomy defines the domain of ‘thinking’ skills. There is a wide variety of terms available to avoid falling into using the same terms repeatedly, and to enable us to pick terms which match more closely the learning we intend our students to achieve. There is so much information on the web about these taxonomies – I’ve put a few links at end of this guide. Two other domains have been identified – psychomotor and affective, and all 3 are defined and illustrated further in the table below: cognitive (knowledge and understanding – the thinking ) psychomotor (skills of doing something involving physical and motor coordination competently) affective (values, professional ethics and behaviours) Learning Domain Characterised as: Cognitive : The knowledge and understanding we want our students to be able to demonstrate What we want our learners to ‘know’ Psychomotor: The skills we want our students to be able to perform competently What we want our learners to ‘do’ Affective : The professional/disciplinary values, ethics, and behaviours we want our students to be able to own and display ‘What we want our learners to ‘be’ Example of action verbs: describe, define, list, identify, explain, classify, analyse, evaluate, critically analyse, critically evaluate, decipher, distinguish, differentiate, compare, contrast, critique, judge Copy, imitate, capture, produce, create, drive, cut, design, dress, perform, draw, write, manipulate, disassemble, assemble Demonstrate practice that aligns with particular values or standards Point to note: At NTU, in our course and module specification documents, we have Course and Module Learning Outcomes defined in two groupings: ‘knowledge and understanding’ and ‘skills and attributes’. Although these can be aligned with Bloom’s domains – there is not meant to be a hard and fast rule. Individual learning outcomes might address any combination of Bloom’s domains. Use Bloom’s taxonomy verbs to help you identify the range of learning for your learning outcomes 6 We can also use these taxonomies to help us define the level of learning complexity we intend tour students to achieve. If you take a look at information about the taxonomies, you see they distinguish different levels of complexity in learning, from surface to deep, or lower level to higher level as illustrated in the levels 1 (lower level, more simple knowledge) to level 5 or 6 (more complex, integrated knowledge and understanding) 7 Cognitive Affective Psychomotor knowledge attitude skills 1. Recall data 1. Receive (awareness) 1. Imitation (copy) 2. Understand 2. Respond (react) 2. Manipulation (follow instructions) 3. Apply (use) 3. Value (understand and act) 3. Develop Precision 4. Analyse (structure/elements) 4. Organise personal value system 4. Articulation (combine, integrate related skills) 5. Synthesize (create/build) 5. Internalize value system (adopt behaviour) 5. Naturalization (automate, become expert) 6. Evaluate (assess, judge in relational terms) Extract from http://www.businessballs.com/bloomstaxonomyoflearningdomains.htm 22.11.2013 We can use these taxonomies when creating learning outcomes to help us select action verbs which: reflect an appropriate domain(s) reflect appropriate complexity or level of learning challenge. Point to note: As mentioned earlier, the intended learning may span more than one domain simultaneously – that’s fine; there’s no need to stick to only one. You reflect the domains, or combination of the domains which reflect most closely the type of learning you intend your students to achieve. The following examples illustrate this: LO1 LO2 Create a shirt pattern Critically evaluate the contribution of pattern making to design In LO1, the learning focus is on ‘making’ a shirt pattern. As this involves drawing on their knowledge and understanding of pattern making, and applying it to do something competently, the learning outcome addresses a mix of cognitive and psychomotor domains at a higher order of learning. In LO2, although the intention is for students to be able to do something, the thing they are expected to do is the reasoning activity of ‘critically evaluate’ so they are working more predominantly in the cognitive domain. They are working at a higher level of learning. Lower levels would be characterised by simpler learning activities such as listing, remembering, or describing. The difference in level of learning is further illustrated by the following examples: LO3a Identify the key responsibilities of staff development managers within organisations LO3b Discuss critically the contribution staff development managers make to an organisation’s effectiveness Compare and contrast the action verbs in LO3a with those in LO3b. The word ‘identify’ not only implies the cognitive domain, but also different levels of challenge or complexity of learning. The ‘identify’ verb of LO3a suggests lower level learning as students would learn to list, name etc., rather than the more complex skills of creating evaluative frameworks to analyse, compare and contrast, and reach conclusions implied by the ‘critically discuss’ action verbs in LO3b. 8 4.2 Using NTU descriptors of academic level AT NTU, as part of our academic standards, we define typical level descriptors for each level in undergraduate study, and also for postgraduate masters level. This can also help us see what the differences in the different levels of study. The links are: See the NTU Academic Quality Handbook on the Centre for Academic Development and Quality website Such frameworks of programme standards help us achieve consistency of standards across all our modules and programme that the levels 4, 5, 6 and 7 (masters) on the NTU framework are not meant to align one on one with Levels 1-6 in Bloom’s taxonomies! In other words, all the learning outcomes for first year undergraduate session outcomes are not expected to be at Bloom’s level 1 alone. That was never Bloom’s intention. Instead, there will be a different balance of Bloom’s levels across different academic levels. A 5 year-old can synthesise and evaluate – but the scope of that evaluation is quite different to the scope of synthesis and evaluation expected of graduates or postgraduates. Use the taxonomies to define an appropriate scope of learning expectation and challenge across the different academic levels, giving more weight to higher order learning with each increase in academic level. 9 Section 5: Contribution of aims and learning outcomes to session design and planning Section 1 revealed how aims and learning outcomes are distinguished from each other. Those distinct purposes mean that aims and learning outcomes have distinctive contributions to our session design and delivery. Contribution of aims Help provide your students with an overall impression of your intentions for the session – the over-arching goal and purpose of the session without necessarily knowing the specific aspects of what they will be expected to learn. Contribution of learning outcomes Help provide students with a more specific indication of what they are being given the opportunity to learn Informs the design of learning opportunities (if we intend our students to achieve something, we need to design content that aligns with those intentions) Inform the teaching and learning approaches we will use in the session (the learning activities need to provide students with an opportunity to learn what is defined in the learning outcomes) Inform the design of any subsequent student assessment (we need to work out ways of assessing the extent to which our students have been able to achieve what we intended them to learn from the session). Section 6: Implications of aims and learning outcomes for session design, planning and delivery Let’s look at the use of learning outcomes for the design of our sessions, including our teaching, learning and assessment processes. We also have to ensure that the session content aligns with the intention expressed in the learning outcomes, otherwise, we aren’t creating the opportunity for students to learn what we intend them to. Likewise the design of any assessment should assess our students’ ability to demonstrate the learning defined in the learning outcomes. No point asking them to make a shirt, if we actually want them to demonstrate their ability to critically evaluate the contribution of pattern making to the design. We have to design a means of finding out /judging the extent to which students have been able to achieve the learning outcomes – whether we are doing this in sessions, or via more formal summative and formative assessments. 10 Section 7: Tips on writing learning outcomes successfully To help re-cap the main points from this guide, here are some tips (adapted from Race 2007, p22) for designing learning outcomes whether for sessions, individual learning activities, modules or programmes. Top tips for writing learning outcomes: o Work out exactly what you intend your students to be able to do or know or be, so you can reflect this as precisely as possible in the verbs you use in your outcomes o Avoid words like ‘know’ or ‘understand’ – find more specific action verbs to reflect the learning domain and complexity o Don’t be trivial – learning outcomes should create sufficient challenge for your students. Use them to address the main learning, rather than have endless lists of every item of information you want them to know. LOs are neither an index nor a syllabus. o Personalise the wording of the outcomes – avoid saying ‘students will…’ and say ‘you will …’instead. It’s more personable and sounds like you’re speaking with the student. o Use user-friendly language and don’t over-complicate them with convoluted language o Keep your sentences short – put other detail in separate narrative or explanations elsewhere in your students’ documents or discussions to avoid cluttering up your learning outcomes with too much detail Top tips for using learning outcomes: 11 o Avoid promising what you have no intention of delivering o Use your learning outcomes to help you focus and narrow down the specific content of any session to ensure it addresses something relevant to the module or course overall o Re-visit ones you’ve already written at regular intervals and revise/adapt them as appropriate o Use them at the start to help set the scene and to help students understand what is expected of them and what to expect o Re-visit them at the end of a session to help your students review and recap on what they have learned and review progress o Ensure learning outcomes align appropriately across sessions, modules and courses. o Ensure your summative assessments address all the outcomes (but avoid falling into the trap of thinking that means you need a separate assignment per outcome!) o Test-drive your learning outcomes on others to ensure they mean the same to others as you intended One useful point to remember is the relationship between learning outcomes, assessment criteria and grading descriptors. Learning outcomes define what students can be expected to be able to do or know as a result of completing an activity, session, module or course. Assessment criteria should be related to the learning outcomes. Grading descriptors define the standards against which the student’s learning will be judged (ie to what extent, or to what level of performance they have achieved each of the outcomes!). Top tips for helping your students understand and use learning outcomes: It’s not necessarily easy to get the hang of creating well-designed learning outcomes straight away. It’s often the case you won’t get them right first time – the important thing is to keep them under review and improve them as you go along using feedback from others to help you. Avoid relying solely on presenting the learning outcomes (and aims) in documentation such as module handbooks or a session plan, without also providing some further explanation that helps your students understand them further. It’s helpful to explain how the learning outcomes for a session relate to the bigger picture of a module and course. Likewise within any sessions, explain and demonstrate how the learning outcomes for any activities map to the session outcomes. It’s also helpful to convince them of the benefits of the learning indicated in the learning outcomes – how is it relevant to their course learning outcomes. o o o 12 Be ready with an answer to the question ‘why these learning outcomes?’ Be ready with a response to the reaction, ‘so what’s the point of the learning outcomes?’ Be ready with an answer to the question, ‘how will knowing the learning outcomes help me do well?’ Ask your students for feedback. For instance, at the end of a session (or in an on-line discussion) ask them to comment on whether they think you have achieved the learning outcomes for the session – if so, which? If not, why? Which ones missing? That can reveal much about how they interpret what is expected of them. Or just ask them to jot down on a post-it note, the 3 key things they have learned from the session – how do their responses compare with what you hoped they would learn? Section 8: References Atherton, J., (2009), Bloom’s Taxonomy of Learning at https://now.ntu.ac.uk/d2l/lms/content/preview.d2l?tId=340854&ou=101495 (last accessed 12.8.2011) More on Bloom’s taxonomy at http://www.businessballs.com/bloomstaxonomyoflearningdomains.htm (last accessed 12.8.2011) Race, P., (2007), The Lecturer’s Toolkit: A Practical Guide to Assessment, Learning and Teaching, London and New York: Routledge (3rd edition) CPLD January 2015 13