SPM Speaking Examiner Training Training for examiners of the Speaking component of SPM English SPM Speaking Master Trainer training: Course aims • To prepare Master Trainers to cascade training to Speaking Examiners • To familiarise Master Trainers with the content of the test materials and the level of the exam • To standardise the delivery of the Speaking tests • To standardise the application of assessment scales and scoring for Speaking • To offer tips and advice to Master Trainers for their own training of SEs and discuss issues which may occur during both training sessions and live examining Introduction from Cambridge Assessment English consultant Jill Buggey Outline of the training – Day 1 Course modules Focus 1. General overview of SPM Speaking rater training (Trainer’s perspective) • • • • • The role of the trainer Practicalities on the day of training Materials required for training Preparing for the training Challenges of running training sessions 2. SPM Speaking test detailed overview – key points to focus on • • • • Format of the test Testing aims Overview of test parts CEFR level and level of SPM test 3. SPM Speaking test procedure – key points to focus on • Speaking examiner roles • Interlocutor frame • The handling of test materials • Interlocutor behaviour • Watching a sample test • Peer practice and practice with volunteer candidates At the end of Day 1, there will be a live Q&A session to discuss any points arising from the day’s training content Outline of the training – Day 2 Course modules Focus 4. SPM Speaking test assessment - overview • CEFR and assessment scales overview • Analytical scales • Overall Spoken Production scale 5. SPM Speaking test assessment – rating of performances • Setting the standard • Applying the standard 6. SPM Speaking test – round up • Marks Collection procedure (online) • Summary of main aspects of training for cascading to Speaking examiners At the end of the Speaking section of Day 2, there will be a second live Q&A session to discuss any final points arising from the training content Module 1 General overview of SPM Speaking rater training (Trainer’s perspective) General overview of SPM Speaking rater training from the trainer’s perspective This module will take you through the following aspects: • The role of the trainer • What the training will cover • Practicalities on the day of training • Materials required for training • Preparing for the training • Challenges likely to be encountered The role of the trainer • To standardise Speaking examiners in test delivery and procedures • To standardise Speaking examiners in test assessment • To run a 2-day training programme (including Writing) • To monitor audio recordings of Speaking tests See the Guidelines for Examiner Training for more details. What Speaking Examiner training covers • • • • • • • Introductions and outline of programme Speaking test format and procedure Peer practice Practice with volunteer candidates Speaking assessment – setting the standard Speaking assessment – applying the standard Speaking assessment – marks collection (online) Practicalities Before the training day, check: • Facilities at your venue are suitable • Attendance list of Speaking examiners • All materials are ready – see following slide • Agenda is prepared/adapted as necessary Materials required for training • • • • • • • Guidelines for Examiner Training PowerPoint slides Videos and commentaries Materials for pre-training familiarisation Handouts for activities during session Sample Speaking test booklets Instructions to Examiners containing – assessment scales, – copies of mark sheets, etc. Pre-Training Checklist Before delivery of your training, it may be useful to consider the following questions: • Are you fully familiar with all the samples covered in training (videos and scripts)? • Have you read all the supporting commentaries and highlighted / annotated any areas which are likely to require explanation and/or further justification? • Have you adapted the presentation materials provided to suit your agenda/timetable? • Have you selected any additional handouts or other resources you would like to use during the session? • Have you sent an agenda for the training session to trainees? Challenges arising during training sessions Spend a few minutes now to make a note of any worries you have at this stage or issues you think may occur during your training session. E.g. technical issues with playing the videos, SEs dispute the marks on the videos, etc. At the end of the day, we can discuss any concerns you have in the Q and A session. Module 2 1. SPM Speaking test detailed overview 2. Key points to focus on during Speaking Examiner training Key points to focus on during Speaking Examiner training? Based on what we’ve looked at so far, consider for a moment what you think are the key aspects you will need to focus on in the training session before introducing the examiners to the Speaking test procedure and marking. Key points to focus on during Speaking Examiner training Here’s what we suggest: • A general overview of SPM Speaking test • A focus on the format of the test, timing and testing aims • A focus on the level of the Speaking test with reference to the Council of Europe Framework of Reference for Languages (CEFR) SPM Speaking Test Overview • The Speaking test is taken by all Form 5 students and constitutes 25% of their overall candidate grade (the other papers, i.e. Reading and Use of English, Listening and Writing also being worth 25% each). • The SPM Speaking test is paired format, i.e. 2 candidates take the tests together. • There are two examiners present during the test: the Interlocutor and the Assessor who mark the test in situ. • The Speaking test has three sections, all aimed at the A2-C1 CEFR range, with a variety of tasks to elicit different types of spoken language. • The duration of the test is approximately 13 minutes. SPM Speaking test format The underpinning format of the test is designed to put candidates at ease by starting off with more support from the examiner and then moving on to be less controlled, i.e. Part 1 Controlled, short, individual responses Part 2 Individual long turn with a listening candidate response question Part 3 Discussion task followed by decision question and then further discussion task • The test is ‘staged’ so as to allow for different interaction patterns as it progresses. • The candidates answer individual everyday questions, and then speak individually before a paired interactive discussion activity • The level of interlocutor involvement lessens as the test progresses • The candidates have more scope to improvise as the test progresses • The test has been designed to cover both the productive and interactive aspects of the CEFR. Part 1 – Example questions Phase 1 Main questions Back-up prompts What’s your name? (to both candidates) Thank you Should I call you …? Where do you live/come from? (to Candidate A) How do you come to school? (to Candidate B) Do you live in ….? Do you come to school by…? Phase 2 Now I’m going to ask you about your daily routine. Main questions What do you normally do after you wake up? Back-up prompts What is the first thing you do every day? What do you usually do after coming back from school? What do you do when you get home from school every day? How do you spend your leisure time? What do you do when you have free time? What do you do on weekends? What do you do on Friday/Saturday/Sunday? SPM Speaking Test Format (Part 1) Task Format Part Interaction Pattern Part 1 Interlocutor interviews Interview candidates 3-4 minutes Candidate Output Input Interlocutor frame provides all questions for interlocutor. Discourse Features Responding to questions Functions Giving factual information about self (bio data) Talking about present circumstances Expressing opinions Explaining and giving reasons Talking about future plans Talking about past experience Part 2 – Example task Part 2 – Example frame SPM Speaking Test Format (Part 2) Task Format Part Part 2 Individual long-turn task 3-4 minutes Interaction Pattern An individual long-turn task (1 minute) with a listening candidate response question (20 seconds). Candidate Output Input Spoken rubrics. Written prompts given in candidate booklet. Discourse Features Sustaining a long turn. Managing discourse: Coherence/clarity of message. Organisation of ideas. Accuracy and appropriacy of linguistic resources. Functions Talking about past / present / future experience. Explaining and giving reasons. Expressing and justifying opinions. Describing people, places and situations. Part 3 – Example task Part 3 – Example frame 1 Part 3 – Example frame 2 SPM Speaking Test Format (Part 3) Task Format Interaction Pattern Part 3 Candidates discuss a Discussion task based task on a mindmap. Then 4-5 minutes they have a decision question to discuss. Part Candidate Output Input Discourse Features Functions Candidates are given a topic with a central focus question and surrounding ideas to help them. Turn-taking and responding appropriately. Negotiating. Initiating. Developing topics. Explaining and giving reasons. Exchanging information and opinions. Expressing and justifying opinions. Negotiating agreement. Making and responding to suggestions. Discussing alternatives. The level of the SPM Speaking test Explain to the examiners the following key points before we move on to focus on Speaking in more detail. • The Speaking test primarily targets the B1-B2 range of language proficiency with the highest results reported at C1 • The assessment scales used to rate candidates’ performance are derived from the guidance provided by the CEFR • The exam is criterion referenced (i.e. to the CEFR) • In line with the positive ‘can do’ ethos of the CEFR, the descriptors aim to describe communicative ability rather than inability Grammatical Accuracy - CEFR and SPM Compare the CEFR descriptor on the left with the SPM assessment scales on the right and identify any key words. A2 Uses some simple structures correctly, but still systematically makes basic mistakes – for example tends to mix up tenses and forget to mark agreement; nevertheless, it is usually clear what he/she is trying to say. SPM Grammar Band 1 Can show sufficient control of simple grammatical structures. SPM Communicative Competence Band 1 Can make him/herself understood but may have pauses and false starts. Grammatical Accuracy - CEFR and SPM Compare the CEFR descriptor on the left with the SPM assessment scales on the right and identify any key words. A2 Uses some simple structures correctly, but still systematically makes basic mistakes – for example tends to mix up tenses and forget to mark agreement; nevertheless, it is usually clear what he/she is trying to say. SPM Grammar Band 5 Can show sufficient control of simple grammatical structures. SPM Communicative Competence Band 5 Can make him/herself understood but may have pauses and false starts. Grammatical Accuracy - CEFR and SPM Compare the CEFR descriptor on the left with the SPM assessment scales on the right and identify any key words. B1 Uses reasonably accurately a repertoire of frequently used ‘routines’ and patterns associated with more predictable situations. Communicates with reasonable accuracy in familiar contexts; generally good control though with noticeable mother tongue influence. Errors occur, but it is clear what he/she is trying to express. SPM Grammar Band 3 • Can communicate with reasonable accuracy. • Can show a good degree of control of simple grammatical structures and attempt some complex grammatical structures. SPM Communicative Competence Band 3 • Can express him/herself clearly though there is some hesitation. • Can ask for clarification and further details in order to move a discussion forward. Grammatical Accuracy - CEFR and SPM Compare the CEFR descriptor on the left with the SPM assessment scales on the right and identify any key words. B1 Uses reasonably accurately a repertoire of frequently used ‘routines’ and patterns associated with more predictable situations. Communicates with reasonable accuracy in familiar contexts; generally good control though with noticeable mother tongue influence. Errors occur, but it is clear what he/she is trying to express. SPM Grammar Band 3 • Can communicate with reasonable accuracy. • Can show a good degree of control of simple grammatical structures and attempt some complex grammatical structures. SPM Communicative Competence Band 3 • Can express him/herself clearly though there is some hesitation. • Can ask for clarification and further details in order to move a discussion forward. Grammatical Accuracy - CEFR and SPM Compare the CEFR descriptor on the left with the SPM assessment scales on the right and identify any key words. B2 SPM Grammar Band 5 Shows a relatively high degree of grammatical control. Does not make mistakes which lead to misunderstanding. Has a good command of simple language structures and some complex grammatical forms, although he/she tends to use complex structures rigidly with some inaccuracy. • • Can communicate with a good degree of accuracy. Can show a good degree of control of a range of simple and some complex grammatical structures. SPM Communicative Competence Band 5 • Can express him/herself with little hesitation. • Can initiate, maintain, develop and close a conversation with ease. • Can relate contributions to his/her partner and negotiate towards an outcome. Grammatical Accuracy - CEFR and SPM Compare the CEFR descriptor on the left with the SPM assessment scales on the right and identify any key words. B2 SPM Grammar Band 5 Shows a relatively high degree of grammatical control. Does not make mistakes which lead to misunderstanding. Has a good command of simple language structures and some complex grammatical forms, although he/she tends to use complex structures rigidly with some inaccuracy. • • Can communicate with a good degree of accuracy. Can show a good degree of control of a range of simple and some complex grammatical structures. SPM Communicative Competence Band 5 • Can express him/herself with little hesitation. • Can initiate, maintain, develop and close a conversation with ease. • Can relate contributions to his/her partner and negotiate towards an outcome. Grammatical Accuracy - CEFR and SPM Compare the CEFR descriptor on the left with the SPM assessment scales on the right and identify any key words. C1 Consistently maintains a high degree of grammatical accuracy; errors are rare and difficult to spot. SPM Grammar Band 6 • Consistently maintains a high degree of grammatical accuracy; errors are rare, difficult to spot and generally corrected when they do occur. SPM Communicative Competence Band 6 • Can express him/herself spontaneously with little obvious searching for expressions. Grammatical Accuracy - CEFR and SPM Compare the CEFR descriptor on the left with the SPM assessment scales on the right and identify any key words. C1 Consistently maintains a high degree of grammatical accuracy; errors are rare and difficult to spot. SPM Grammar Band 6 • Consistently maintains a high degree of grammatical accuracy; errors are rare, difficult to spot and generally corrected when they do occur. SPM Communicative Competence Band 6 • Can express him/herself spontaneously with little obvious searching for expressions. Speaking and the CEFR Activity Look at Handout S1 (SPM Speaking and the CEFR) on the following slides. For each pair of CEFR descriptors decide which level of ability they are describing. (For Trainers): Make a note of any questions you have on this activity or anything you want more information about for the Q and A session. Handout S1: Range – A2 or B1? For each pair of CEFR descriptors decide which level of ability they are describing. Range – A2 or B1? Descriptor A Descriptor B Uses basic sentence patterns with memorised phrases, groups of a few words and formulae in order to communicate limited information in simple everyday situations. Has enough language to get by, with sufficient vocabulary to express him/herself with some hesitation and circumlocutions on topics such as family, hobbies and interests, work, travel, and current events. Handout S1: Range – A2 or B1? Range – A2 or B1? Descriptor A Descriptor B Uses basic sentence patterns with memorised phrases, groups of a few words and formulae in order to communicate limited information in simple everyday situations. Has enough language to get by, with sufficient vocabulary to express him/herself with some hesitation and circumlocutions on topics such as family, hobbies and interests, work, travel, and current events. A2 level B1 level Handout S1: Accuracy – B2 or C1? Accuracy – B2 or C1? Descriptor A Descriptor B Consistently maintains a high degree of grammatical accuracy; errors are rare, difficult to spot and generally corrected when they do occur. Shows a relatively high degree of grammatical control. Does not make errors which cause misunderstanding and can correct most of his/her mistakes. Handout S1: Accuracy – B2 or C1? Accuracy – B2 or C1? Descriptor A Descriptor B Consistently maintains a high degree of grammatical accuracy; errors are rare, difficult to spot and generally corrected when they do occur. (C1) Shows a relatively high degree of grammatical control. Does not make errors which cause misunderstanding and can correct most of his/her mistakes. (B2) Handout S1: Fluency – B1 or B2? Fluency – B1 or B2? Descriptor A Descriptor B Can keep going comprehensibly, even though pausing for grammatical and lexical planning and repair is very evident, especially in longer stretches of free production. Can produce stretches of language with a fairly even tempo; although he/she can be hesitant as he or she searches for patterns and expressions, there are few noticeably long pauses. Handout S1: Fluency – B1 or B2? Fluency – B1 or B2? Descriptor A Descriptor B Can keep going comprehensibly, even though pausing for grammatical and lexical planning and repair is very evident, especially in longer stretches of free production. (B1) Can produce stretches of language with a fairly even tempo; although he/she can be hesitant as he or she searches for patterns and expressions, there are few noticeably long pauses. (B2) Handout S1: Interaction – A2 or B1? Interaction – A2 or B1? Descriptor A Descriptor B Can initiate, maintain and close simple face-to-face conversation on topics that are familiar or of personal interest. Can repeat back part of what someone has said to confirm mutual understanding. Can ask and answer questions and respond to simple statements. Can indicate when he/she is following but is rarely able to understand enough to keep conversation going of his/her own accord. Handout S1: Interaction – A2 or B1? Interaction – A2 or B1? Descriptor A Descriptor B Can initiate, maintain and close simple face-to-face conversation on topics that are familiar or of personal interest. Can repeat back part of what someone has said to confirm mutual understanding. (B1) Can ask and answer questions and respond to simple statements. Can indicate when he/she is following but is rarely able to understand enough to keep conversation going of his/her own accord. (A2) Handout S1: Coherence – B2 or C1? Coherence – B2 or C1? Descriptor A Descriptor B Can use a limited number of cohesive devices to link his/her utterances into clear, coherent discourse, though there may be some "jumpiness" in a long contribution. Can produce clear, smoothly flowing, well-structured speech, showing controlled use of organisational patterns, connectors and cohesive devices. Handout S1: Coherence – B2 or C1? Coherence – B2 or C1? Descriptor A Descriptor B Can use a limited number of cohesive devices to link his/her utterances into clear, coherent discourse, though there may be some "jumpiness" in a long contribution. (B2) Can produce clear, smoothly flowing, well-structured speech, showing controlled use of organisational patterns, connectors and cohesive devices. (C1) Module 3 SPM Speaking test procedure Key points to focus on during Speaking Examiner training Note for this Module you will need a copy of the Instructions to Speaking Examiners Module 3 SPM Speaking test procedure This module will take you through the following aspects to focus on with Speaking examiners: • Watching a sample test • Speaking examiner roles • The use of the Interlocutor frame • The handling of test materials • Interlocutor behaviour • Managing exam resources • Peer practice and practice with volunteer candidates Speaking Test Procedure Activity Look at Handout S2 (Speaking Test Procedure) on the next slide. Watch the sample Speaking test and put the steps for carrying out the test in the correct order (1 – 9). (For Trainers): Make a note of any questions you have on this activity or anything you want more information about for the Q and A session. Handout S2: Speaking test procedure Here are the steps for carrying out the SPM Speaking test. Watch the sample test and number them 1 - 9 in the correct order. The first one has been done for you. 1 a b Candidate B listens and answers a question related to the topic of Candidate A’s talk. c Candidate A is given a topic to talk about for about a minute. The topic and 4 bullet points with some ideas to use are in the candidate booklet. d e Interlocutor asks one question to both candidates related to the topic of the discussion. f Both candidates have a discussion about a topic and then come to a decision. They should use language functions appropriately to give suggestions, opinions and elaboration as well as to agree or disagree politely and negotiate agreement. g Candidate B talks about a different topic for about a minute, using a different topic in the booklet, also with 4 bullets with ideas to use. h i Interlocutor introduces him/herself and the Assessor. Candidate A listens and answers a question related to the topic of Candidate B’s talk. Candidates respond to questions about themselves (e.g. their hobbies, their homes or other personal matters) Assessor provides marks for three criteria – Grammar, Vocabulary and Communicative Competence and Interlocutor gives one mark for Overall Spoken Performance. A sample SPM Speaking test Focus on Procedure Video 1: Helmi and Ain Handout S2: Speaking test procedure Here are the steps for carrying out the SPM Speaking test, numbered in the correct order. 4 6 3 a b c Candidate B listens and answers a question related to the topic of Candidate A’s talk. 8 2 d e Interlocutor asks one question to both candidates related to the topic of the discussion. 7 f Both candidates have a discussion about a topic and then come to a decision. They should use language functions appropriately to give suggestions, opinions and elaboration as well as to agree or disagree politely and negotiate agreement. 5 g Candidate B talks about a different topic for about a minute, using a different topic in the booklet, also with 4 bullets with ideas to use. 1 9 h i Interlocutor introduces him/herself and the Assessor. Candidate A listens and answers a question related to the topic of Candidate B’s talk. Candidate A is given a topic to talk about for about a minute. The topic and 4 bullet points with some ideas to use are in the candidate booklet. Candidates respond to questions about themselves (e.g. their hobbies, their homes or other personal matters) Assessor provides marks for three criteria – Grammar, Vocabulary and Communicative Competence and Interlocutor gives one mark for Overall Spoken Performance. Speaking Training Quiz Activity Look at Handout S3 (SPM Speaking Training Quiz) on the following slides. Consider each question about the Speaking test before reading the answer on the next slide. (For Trainers): Make a note of any questions you have on this activity or anything you want more information about for the Q and A session. Speaking Training Quiz 1. How long should each part of the test take? • Part 1 = • Part 2 = • Part 3 = • Total = 2. How should examiners introduce themselves at the beginning of the test? 3. How should the seating be arranged? Speaking Training Quiz 1. How long should each part of the test take? • Part 1 = 3-4 minutes • Part 2 = 3-4 minutes • Part 3 = 3-4 minutes • Total = 10-13 minutes (This may depend on the level of the candidates) 2. How should examiners introduce themselves at the beginning of the test? • As scripted in the interlocutor frame: Good morning / afternoon / evening. I’m … and this is my colleague … . He/She’ll just listen to us. 3. How should the seating be arranged? • As in the ISEs, excerpt shown here: Speaking Training Quiz 4.To what extent should the interlocutor follow the frames provided in the materials? 5. What are the analytical criteria used in assessing candidates? • • • 6. Who leads/controls the interaction in Part 3 of the test? Speaking Training Quiz 4.To what extent should the interlocutor follow the frames provided in the materials? In order to deliver the test equally to all candidates, the interlocutor must adhere strictly to the interlocutor frame and instructions, making sure the candidates are addressed as indicated (Candidate A/B) but using the candidates’ names correctly at the required points in the test. Research has shown that the use of unscripted comments and asides creates an unfair test and can, therefore, have a negative impact on a candidate’s performance. 5. What are the analytical criteria used in assessing candidates? • Grammar • Vocabulary • Communicative Competence 6. Who leads/controls the interaction in Part 3 of the test? Although the task is set up by the interlocutor, the main interaction is lead/controlled by the candidates themselves. This is the semi-controlled part of the test, allowing for some genuine peer-peer interaction. Speaking Training Quiz 7. What kind of timepiece should you use? 8. When do you start timing the test? 9. Why should you keep to timing for each section? Speaking Training Quiz 7. What kind of timepiece should you use? SEs should use a reliable clock or watch which shows seconds, and which doesn’t have a loud tick. Clocks/watches must not be set to emit noises to signal the timings of the different parts of the test 8. When do you start timing the test? The timing of the test begins when the interlocutor greets the candidates, this should also be noted discretely. 9. Why should you keep to timing for each section? It is important, for reasons of test reliability and fairness to candidates, that all tests are of the same length and that the time spent on the individual parts of the test does not vary from that indicated. Speaking Training Quiz 10. Can you repeat the same materials? 11. When should you change roles (interlocutor to assessor)? 12. What information does the interlocutor complete on the mark sheet? Speaking Training Quiz 10. Can you repeat the same materials? There will be a selection of tasks to choose and these will be in sets that should not be separated. You should use the sets randomly and vary the sets in order to provide test security. 11. When should you change roles (interlocutor to assessor)? Roles are fixed. The assessor is the teacher from a nearby school/centre appointed by the Assessment and Examination Sector in the State Education Department (SPP, JPN) while interlocutors are teachers who teach in the candidate’s school/centre. The Principal or School Administrator is responsible in appointing the interlocutor(s) and arranging the interlocutors’ schedules throughout the Speaking exam period. The role of interlocutor will be assigned by the head teacher within each school/centre. 12. What information does the interlocutor complete on the mark sheet? The interlocutor uses the Overall Spoken Performance scale and refers to it after the end of the test (as well as while the candidates are involved in the collaborative tasks and possibly during the long turns) and they record these marks on their mark sheet. Speaking Training Quiz 13. How can you react to candidates who need support in understanding or speaking? 14. How can you interrupt a candidate or candidates discreetly when necessary? 15. Why mustn’t you discuss marks with your co-examiner? Speaking Training Quiz 13. How can you react to candidates who need support in understanding or speaking? Redirection/support may take the form of: •repetition of all or part of the rubric •pointing to a task in the Candidate booklet •use of a scripted back-up question/prompt. You can use gesture and/or repetition and any back-up questions, but you must never deviate from the frame. 14. How can you interrupt a candidate or candidates discreetly when necessary? Use the “hovering hand” and a firm “Thank you” to interrupt discreetly and retrieve the task quickly. 15. Why mustn’t you discuss marks with your co-examiner? Because the Analytical Scale and the Overall Spoken Performance Scale refer to different areas on spoken language and the two examiners have very different viewpoints of the candidates’ performance. Speaking Training Quiz 16. Should you make notes to aid assessment? 17. Does it matter that the Part 1 questions don’t change? 18. How many questions should you ask in Phase 2 of Part 1? Speaking Training Quiz 16. Should you make notes to aid assessment? No: making notes means there are parts of candidate performance that you will miss. 17. Does it matter that the Part 1 questions don’t change? No, because the main function is to put candidates at ease. 18. How many questions should you ask in Phase 2 of Part 1? You should ask one or more questions from the list, depending on the time available (Part 1 is between 3 and 4 minutes). Try to ask different questions to each candidate if possible, using the back-up prompts as needed, and encouraging them to extend their answers if necessary with a hand gesture or nod. Speaking Training Quiz 19. Do you have to record every test you do? 20. If you have a question about a mark and you need advice, what should you do? Speaking Training Quiz 19. Do you have to record every test you do? No: MES or your senior hierarchy member will inform you if and when a sample of your marking is required. 20. If you have a question about a mark and you need advice, what should you do? Always refer queries to the person directly above you in the hierarchy. If need be, they will escalate it higher but it is essential you do refer queries as a way to ensure standardised practices in assessment. Speaking Examiner roles The Interlocutor: • conducts the test by giving instructions and asking the candidates questions, using the prompts provided to scaffold the test takers’ responses appropriately. • handles the test materials and ensures each candidate has an equal opportunity to speak by keeping strictly to the wording of the interlocutor frame and also the timing for each part of the test. • awards the candidates a Global Achievement score, using the Overall Spoken Production scale. The Assessor: • takes no part in the exchanges, but observes the test, ensuring he/she can see and hear the candidates clearly. • awards the candidates scores according to the Analytical Assessment Scales. • completes the answer sheets accurately and fully. Interlocutor frame When using the interlocutor frame: • speak naturally while keeping to the script • use candidates’names as appropriate • stress key words and use intonation to make meaning clear • don’t rephrase in any way, if asked to repeat instructions or explain lexis • don’t make any unscripted comments • don’t deviate from the script in any way • don’t assume a dominant role in the interaction. Handling materials Security Efficiency Familiarity Interlocutor behaviour Interlocutor behaviour • • • • Interlocutor involvement Interlocutor intervention Interlocutor body language Interlocutor manner Activity: Managing exam resources 1. When should you organise the test materials needed for your exam session? 2. What else should you have with on the exam day besides the current materials and ISE booklet? 3. How should you arrange the test room? 4. What should you have on the table in the exam room when acting as Assessor? 5. What should you have on the table in the exam room when acting as Interlocutor? How should you arrange the materials on the table? Where should you place the assessment scales? Why? Peer Practice • Trainees, in groups of three or four, take it in turns to act as the interlocutor and practise reading the rubrics aloud and handling the materials. The interlocutor should pay attention to: • • • • • • Stressing key words Using intonation to convey meaning Keeping their speed of delivery to the expected level Not deviating from the frame Maintaining a degree of eye contact where relevant Handling the materials efficiently. • Two trainees should act as candidates and can act out their roles, as appropriate. • One trainee should watch the test and make notes using handout 4: SPM Observing Practice Tests in order to give feedback to his/her colleagues afterwards. Observing Practice Tests Activity Look at Handout S4 (SPM Observing Practice Tests) on the following slide. Watch a practice test (with peers or with volunteer candidates) and make notes for feedback. (For Trainers): Make a note of any questions you have on this activity or anything you want more information about for the Q and A session. Observing Practice Tests While watching the practice tests with peers or volunteer candidates, make notes on the following points and then give feedback to your colleagues on the points you noted: 1. Timing. 2. Eye contact. 3. Delivery (stress and intonation) of the interlocutor frame. 4. Accuracy of the interlocutor frame. 5. Seating arrangements. 6. Position of the assessor. 7. Handling of the materials. 8. Interaction patterns. Procedure: round up SPM Speaking Examiners should: • maintain the security and confidentiality of the Speaking test. • be familiar with all relevant sections of the Instructions to Speaking Examiners booklet. • be familiar with the test procedure. • be familiar with the test materials and use them as instructed. • switch off mobile phones. • minimise security risks by varying the test materials, using the full range of tasks, and not always starting a session with materials which appear at the front of the pack. • ensure that no test materials are taken out of their personal charge. • maintain the confidentiality of the examination throughout the examining period and thereafter. • create a non-intimidating atmosphere so as to encourage candidates to perform to the best of their ability. • ensure that all candidates are treated fairly. Q and A session 1 This is the end of Day 1 Training. Please be ready to participate in the live Q and A session where you’ll be able to ask any questions or discuss any concerns you have. Module 4 SPM Speaking test assessment Key points to focus on during Speaking Examiner training Note for this Module you will need a copy of the Instructions to Speaking Examiners Module 4 SPM Speaking test assessment This module will take you through the following aspects to focus on with Speaking examiners: • The Analytical assessment criteria • The Overall Spoken Performance assessment criteria • Assessing samples for standardisation: • Setting the standard (2 videos) • Applying the standard (2 videos) • Marks Collection (2 videos to be done online) SPM Assessment Scales The three analytical criteria are: • Grammar • Vocabulary • Communicative Competence The interlocutor uses the Overall Spoken Performance scale The level of the test is primarily targeting CEFR B1 to B2 with the highest results reported at C1 Focus on the Analytical and Global Scales Activity Look at Handout S5 (SPM Speaking Scales Sorting Activity) on the following slides. For each descriptor, decide which is the Band 6 (C1 level), Band 5 (B2 level), Band 3 (B1 level) and Band 1 (A2 level) descriptor. (For Trainers): Make a note of any questions you have or anything you want more information about for the Q and A session. Speaking Scales Sorting Activity Which is the Band 6 (C1 level), Band 5 (B2 level), Band 3 (B1 level) and Band 1 (A2 level) descriptor? OVERALL SPOKEN PERFORMANCE 1 Can understand questions and instructions directed at him/her though will sometimes have to ask for repetition of particular words and phrases. Requires some prompting and support. Can understand questions and instructions directed at him/her with ease. Can produce extended relevant stretches of language with very little prompting or support. Can understand questions and instructions directed at him/her with ease. Requires very little prompting and support. Can understand questions and instructions directed at him/her slowly, directly and repeatedly. Requires prompting and support. Speaking Scales Sorting Activity Which is the Band 6 (C1 level), Band 5 (B2 level), Band 3 (B1 level) and Band 1 (A2 level) descriptor? OVERALL SPOKEN PERFORMANCE 1 Can understand questions and instructions directed at him/her though will sometimes have to ask for repetition of particular words and phrases. Requires some prompting and support. Band 3 Can understand questions and instructions directed at him/her with ease. Can produce extended relevant stretches of language with very little prompting or support. Band 6 Can understand questions and instructions directed at him/her with ease. Requires very little prompting and support. Band 5 Can understand questions and instructions directed at him/her slowly, directly and repeatedly. Requires prompting and support. Band 1 Speaking Scales Sorting Activity Which is the Band 6 (C1 level), Band 5 (B2 level), Band 3 (B1 level) and Band 1 (A2 level) descriptor? OVERALL SPOKEN PERFORMANCE 2 Can give basic information on familiar topics using short phrases. Can maintain and develop the interaction by asking and responding to questions and suggestions on familiar matters using longer responses despite hesitation while searching for patterns and expressions. Can maintain and develop the interaction, compensating for gaps in their own vocabulary or grammar knowledge, including on abstract and complex topics. Can maintain the interaction by asking and responding to questions and suggestions on familiar matters in a simple and direct way. Speaking Scales Sorting Activity Which is the Band 6 (C1 level), Band 5 (B2 level), Band 3 (B1 level) and Band 1 (A2 level) descriptor? OVERALL SPOKEN PERFORMANCE 2 Can give basic information on familiar topics using short phrases. Band 1 Can maintain and develop the interaction by asking and responding to questions and suggestions on familiar matters using longer responses despite hesitation while searching for patterns and expressions. Band 5 Can maintain and develop the interaction, compensating for gaps in their own vocabulary or grammar knowledge, including on abstract and complex topics. Band 6 Can maintain the interaction by asking and responding to questions and suggestions on familiar matters in a simple and direct way. Band 3 Analytical criteria for Speaking Band 1 descriptors – A2 level Score Grammar Can communicate what he/she is trying to say. 1 Communicative Competence Can make him/herself understood but may Can use sufficient vocabulary to talk about have pauses and false everyday situations and starts. familiar topics. Can initiate, maintain Vocabulary Can show sufficient control of simple grammatical Can mostly convey meaning but may use structures. inappropriate Can use vocabulary or some connectors to link repetition. simple sentences. and close a conversation with some difficulty. Can ask for clarification when necessary. Analytical criteria for Speaking Band 3 descriptors – B1 level Score 3 Grammar Vocabulary Can communicate with reasonable accuracy. Can use a range of appropriate Can show a good vocabulary to talk degree of control of about everyday simple, and attempt situations and some complex grammatical structures. familiar topics. Can form longer sentences and link them together using basic cohesive devices. Communicative Competence Can express him/herself clearly though there is some hesitation. Can initiate, maintain, develop and close a conversation with little Can convey relevant difficulty. meaning but may Can ask for clarification have errors in and further details in vocabulary choice. order to move a discussion forward. Analytical criteria for Speaking Band 5 descriptors – B2 level Score 5 Grammar Vocabulary Can communicate with a good degree Can use a wide range of accuracy. of appropriate Can show a good vocabulary to give and degree of control of exchange views on a a range of simple wide range of familiar and some topics. complex Can convey relevant grammatical meaning with good structures. vocabulary choice. Can use a range of cohesive devices. Communicative Competence Can express him/herself with little hesitation. Can initiate, maintain, develop and close a conversation with ease. Can relate contributions to his/her partner and negotiate towards an outcome. Analytical criteria for Speaking Band 6 descriptors – C1 level Score 6 Grammar Vocabulary Consistently Can use a wide range maintains a high of appropriate degree of grammatical accuracy. vocabulary to give and exchange views on a Has a good command wide range of of a wide range of abstract, complex grammatical and unfamiliar topics. structures. Can convey specific Can use a wide range differences in of organisational meaning through patterns, cohesive appropriate choice of devices and vocabulary. connectors. Communicative Competence Can express him/herself spontaneously. Can initiate, maintain, develop and close a conversation with ease. Can relate contributions skilfully and effectively to those of his/her partner. Which of these terms appear in the Overall Spoken Performance scale? support clarification control word order understand connectors repetition interaction instructions familiar topics prompting expressions initiate a conversation pausing questions Which of these terms appear in the Overall Spoken Performance scale? support clarification control word order understand connectors repetition interaction instructions familiar topics prompting expressions initiate a conversation pausing questions What is behind the SPM Overall Spoken Performance scale? • Understanding the tasks – simple or more complex questions / instructions / suggestions – amount of repetition needed • Handling topics – basic information / familiar topics / complex and abstract • Length of utterances short phrases / maintain / develop the interaction • Support – requires (some / little / no) prompting and support Some Global Questions? Discuss these questions in pairs/groups: 1. Why is the Overall Spoken Performance mark important? 2. When should the mark be decided on? 3. What is the procedure for completing the mark on the mark sheet? 4. How does the Interlocutor decide on his/her mark? 5. What do we mean by ‘from the Interlocutor’s perspective?’ Some Global Answers Discuss these questions in pairs/groups: 1. Why is the Overall Spoken Performance mark important? In order to ensure fair assessment, both examiners assess candidate performance, with the interlocutor giving a mark for overall spoken performance which has the same value as each of the analytical marks awarded by the assessor. 2. When should the mark be decided on? See section ‘Marking procedure’ of ISEs (both during and immediately after the test). 3. What is the procedure for completing the mark on the mark sheet? See section ‘mark sheet completion’ in ISEs and Missing Marks Checklist on Overall Spoken Performance scale pages. 4. How does the Interlocutor decide on his/her mark? By using the specific elements of the Overall Spoken Performance scale and matching the overall performance to the descriptors in bands 1, 3, 5 and 6. 5. What do we mean by ‘from the Interlocutor’s perspective?’ The interlocutor is much closer to the interaction than the assessor and so has a different perspective of the overall impression of the way the candidates interact and deal with the tasks in each part of the test. Setting the standard 1 Watch and assess Speaking test sample: Video 1 (Helmi & Ain) Marks awarded: Grammar Vocabulary Communicative competence Helmi 3 4 3 Ain 5 5 5 Setting the standard 2 Watch and assess Speaking test sample: Video 2 (Farid & Farhan) Marks awarded: Grammar Vocabulary Communicative competence Farid 1 2 1 Farhan 2 3 2 Applying the standard 1 Watch and assess Speaking test sample: Video 3 (Anas and Tanisha) Examiners can then compare marks and discuss. Grammar Anas Tanisha Vocabulary Communicative competence Overall Spoken Performance Applying the standard 1 Watch and assess Speaking test sample: Video 3 (Anas and Tanisha) Marks awarded: Grammar Vocabulary Communicative Overall competence Spoken Performance Anas 4 4 5 5 Tanisha 5 5 5 5 Applying the standard 2 Watch and assess Speaking test sample: Video 4 (Aminah and Alia) Examiners can then compare marks and discuss. Grammar Vocabulary Aminah Alia Communicative Overall competence Spoken Performance Applying the standard 2 Watch and assess Speaking test sample: Video 4 (Aminah and Alia) Marks awarded: Grammar Vocabulary Communicative Overall competence Spoken Performance Aminah 5 6 5 6 Alia 6 6 6 5 Round-up • After this training, there will be ‘marks collection’ to further monitor the impact of the training and ensure consistency of marking. This activity needs to be done in isolation, with no conferring with fellow examiners. • This activity will be carried out using Google Forms. Further instructions will be sent to you separately. • Trainee feedback survey Q and A session 2 This is the end of the Speaking Training. Please be ready to participate in the live Q and A session where you’ll be able to ask any further questions or discuss any concerns you have. Thank you!
