Notes to accompany the PowerPoint for the AMEP Assessment Task Bank Professional
Development Kit. Developed by Marian Hargreaves for NEAS, 2013.
Slide 1: Front page
Slide 2: Starting Basics
These guidelines are for developing assessment tasks for learners in the Adult Migrant
English Program (AMEP) which is based on the curriculum of the Certificates in Spoken and
Written English (CSWE). The AMEP is funded by the Department of Immigration and
Citizenship (DIAC) and aims to assist migrants into settlement in Australia including into paid employment. Learning and assessment in the program therefore takes a skills-based approach to everyday topics and activities especially those that assist with settlement and employment.
These guidelines are very similar to those for developing a reading assessment task. Both listening and reading are receptive skills and tasks to assess these share many features.
However, there are some significant differences when developing tasks to assess listening and these will be highlighted.
Slide 3: Design and Development of a listening assessment task
Step 1: Decide

which Certificate level you want to assess

which Module, and

which Learning Outcome (LO).
Consider whether it might be integrated with another learning outcome which may also involve listening but may involve other macro-skills (ie reading, writing, speaking).
Slide 4:
Step 2: Check the relevant Learning Outcomes in the CSWE curriculum and the criteria for the one you have chosen. At this point you need to be very clear about the specifications for the learning outcome, what you want to assess, and whether the task that you are considering will actually assess the skill that you want to evaluate.
Step 3: Look at existing models in the online Assessment Task Bank to see how other people have interpreted and applied the specifications.
Step 4: Choose a suitable text, situation or theme that can be used for a listening assessment.
© NEAS Ltd 2014 1

Slide 5: Texts for a listening task
Think about the:

Topic
As with a reading text, a listening text should be on a suitable topic, one that is both interesting and relevant for the learners but is not so familiar that they could answer the questions without listening to the text. Many texts are good teaching texts but are not good for assessment tasks: students may be able to answer the questions from personal knowledge without actually listening to the text, eg texts about sport.
Slide 6: Example: Aussie Rules
Although this would seem to be a relevant task as migrants may not be familiar with a very popular Australian sport, anyone with any knowledge of the game could answer these questions without listening to the text. The task is therefore not assessing listening skills and is not valid.
Slide 7: Fruit Salad
The assessment may also be impractical, for example making a fruit salad could be great fun as a class activity but a nightmare (due to different sequencing possibilities) as an assessment, and therefore not reliable.
Slide 8: Sensitive topics should be avoided, especially if the learners are humanitarian refugees, and be careful using culturally specific texts (unless the text is intended to give information about cultural events etc).
Specialist texts, eg botanical descriptions or dialogue using engineering jargon, should not be used.
Slide 9:
Length
The text should be long enough to provide material for suitable questions, but not so long as to add difficulty to the task by its sheer length.

Level of difficulty
Each task should be tailored to the level of difficulty being assessed, however, as a general note, listening texts usually need to be easier than those for reading tasks as the learner cannot control the rate of delivery.

Vocabulary
Vocabulary should be appropriate to the topic and to the level of difficulty of the task. While learners may be able to use a dictionary for a reading task, it is more difficult to use dictionaries for a listening assessment – especially if the spelling of the word is unconventional. Uncommon words should therefore be avoided, although at intermediate levels, ie CSWE III and above, learners should be able to understand common idioms and colloquialisms, and the criteria of the learning outcome may require that they have the opportunity to demonstrate this eg CSWE III B1: Demonstrate understanding of a casual
© NEAS Ltd 2014 2

conversation with topic changes. Teachers should also make sure that any potentially difficult vocabulary has been explained before the learners are given the task to complete.

Criteria
The curriculum criteria of the task should always be checked, not only to ensure that the text meets the overall description but also the detailed criteria. If the task is supposed to assess listening skills at an interview, the text should not be a presentation. If the criterion asks for the learner to identify the topic, a suitable question must give the learner that opportunity.
Tasks should also be consistent with other tasks for the same learning outcome to ensure reliability.

Authenticity
Authentic texts give credibility to an assessment, especially if they relate to a situation which the learner might actually have to cope with eg CSWE I H1: Demonstrate understanding of a simple answering machine message. However, authentic texts, as they stand, may not lend themselves to sufficient questions to meet the criteria. The text may be too long, too short, contain difficult or inappropriate language. It is often necessary to adapt the text to meet requirements.
Listening tasks usually involve dialogue. This can be difficult to write. Authentic dialogue can also be very broken and disjointed (try listening to other people in a checkout queue). This can be very difficult to listen to in an assessment, especially with low level learners where the discourse structure needs to be clear and the syntactic complexity appropriate to the level of difficulty. Overlap, interruption and repetition can be removed, but care needs to be taken that the dialogue does not devolve into a written text that is being spoken.

Semi-scripted authentic texts
For learners who are not complete beginners, semi-scripted texts can work well, especially in the earlier stages of developing a listening task. Here, the speakers are given minimal instructions, such as topic/location/purpose of text, so that the resulting dialogue can be more natural and unforced. However, the constraints of the criteria still have to be considered, and care taken that the final recording has not omitted any crucial vocabulary or reference required to answer one of the questions in the task.

Nature of input
For listening texts, consider clarity of voice, speech rate, accent, quality of recording.

Copyright
Using authentic texts raises the issue of copyright.
If a text is only to be used in-house, in the classroom, copyright should not be a problem. If, however, the text will be used more extensively or published in any form, it is important to get permission to use it. Ideally, writing a text yourself avoids all questions of copyright but it is also often the most time consuming option.
For more information on copyright, go to: http://www.copyright.org.au/publications/infosheets.htm
Select Education Institutions GO48 for information on the use of copyright materials in educational settings.
© NEAS Ltd 2014 3

Slide 10: Step 5: Text Modification
Modifying the text, developing the questions and recording a script usually all happen together but for this presentation we’ll start with text modification.
Changes to a text for a listening assessment are almost always necessary given all the issues already raised plus the specifications of the learning outcome in the CSWE curriculum.
The text may be too long or too short. The information may be given too quickly, so you need to build in redundancy to space it out (and give students time to respond appropriately). For example, in a dialogue Speaker B could repeat what Speaker A has said.
This not only adds time for the learner to process the information but also reinforces that information. The structure may need tweaking to meet the genre requirements – you may need more discourse markers, more information or propositional content on which to base questions. You may need to remove ambiguities and/or unsuitable vocabulary. Avoid unfamiliar/difficult names for streets, places or people. Avoid words or numbers that can easily be mistaken. Check the criteria!
Slide 11: Step 6: Question items
These will obviously depend on what is required by the criteria. If a learner has to demonstrate an understanding of conditional clauses, as in CSWE
II F1: ‘Demonstrate understanding of spoken instructions’, then an appropriate question needs to be provided.
One of the primary considerations, however, is the validity of the task. As the task is intended to assess listening, the questions should not impose heavy demands on reading or writing.
Questions should not be overly long to read, or in language that is more difficult that the language being assessed. Learners should be able to understand the questions without any difficulty and be given the opportunity to go through the questions and clarify any queries, such as vocabulary or meaning in the questions, before they start listening to the recorded text and completing the task.
Answers to a listening task should not require a large amount of writing. Short answers are therefore preferable or answers that only involve a tick, cross or circle in the correct place. In many cases the criteria for CSWE learning outcomes stipulate that answers may be either spoken or written. If written, answers need not be grammatically correct or in sentence form.
These requirements inevitably restrict the type of questions used in listening tasks. Of the various forms available, the most suitable for the assessment of listening skills are multiple choice questions (MCQs), summary cloze questions, sentence completion and short answer questions.
Refer to the PD Kit on Question Items here, or just use the
Question Types overview handout (on website).
Slide 12: Question items cont.
© NEAS Ltd 2014 4


Watch out for the test method effect:
This refers to the difficulty of certain item types and the familiarity that learners have with different types. For example, if learners have not done a multiple choice activity before, they may not know how to do it and either waste time working out what they have to do or get it wrong because they didn’t know what was required. Test preparation courses usually include a lot of practice with the types of items that will be met in the test.

Use a range of item formats
In order to minimise test method effect, it is a good idea to use a range of item types in the task. In this way, learners who find one type difficult won’t be overly disadvantaged.
However, you need to be careful that the task isn’t too cognitively demanding because there are too many different item types – balance is important.
Slide 13: Recording
• Before developing the task – using an existing text. You can always start with an existing text, one that you think has potential for an assessment task, or something that you could use as a model (eg a weather report/supermarket announcement/telephone information message).
• During – develop as you go. Once you have decided on the Learning Outcome, criteria and topic (ie you have got all your ingredients) you can make and modify the recording and develop the questions.
• After the task has been designed. Get some feedback from other teachers and try out with some learners who are at the benchmark level of ability. By now you are too familiar with the task to be objective.
• After the task has been piloted. Check it again, and use it in class for assessment
Refer to Recording of Scripts for Listening tasks on the Assessment Task Bank handout on next page.
Slide 14: Rubrics
Step 8: Write the rubrics . Consider any additional instructions for the teacher that may be necessary. Remember to make the language of instructions to the learner at least as easy as the language of the text of the assessment itself. These instructions need to be simple.
You don’t want the learners to get stuck because they don’t understand the instructions!
Rubrics should be clear and consistent for all tasks in that LO and this includes instructions for the teacher/task administrator.
© NEAS Ltd 2014 5

Some general points

There should be clearly differentiated voices – ideally one man and one woman.

Authenticity.
Texts should be as authentic as possible. However, it is almost impossible to find a script/text that is just right for your CSWE level and learning outcome. And there are copyright problems too if you want to publish your finished task, so try making your own.

Each recording should have an introduction . The full, final version should be for example, “Certificate One in Spoken and Written English, Module D, Learning Outcome
1, Demonstrate understanding of a spoken information text, [ + title, eg Shopping
Announcement].
” But a draft version could just be Cert I, D1 Shopping Announcement.

If you find that directions to actors are necessary, eg for emphasis, impatience, sound effects such as coughing or laughing etc, these should be inserted in the hard copy of the script.

Speed of recorded speech: Not too fast, and not too slow (even for Cert I)! It is important not to speak too slowly as the rhythm and melody (prosody) of the English language will be lost. Learners need to be able to use prosody to understand English, even if word stress and intonation may not be taught explicitly, and they will not be able to use these skills if the recorded speech is too slow and stilted. In fact, it will become even harder for them to understand.

Pace of recorded speech: A common mistake made when trying to speak more slowly for a recording is to speak as normal (which may be quite quickly) and then leave large gaps. Pausing can give the learner more time to process the information, but care should be taken to make sure that words are not spoken too quickly. The average speed works out fine, but the actual pace of the spoken words may be much too fast. Listen to some existing recordings on the Assessment Task Bank.

Extra effects are generally added later. They are fun (cf the elephant in C2 D1
Melbourne Zoo) and can give more context to the recording, however, they are entirely optional (unless specifically included eg coughing at the doctor’s, in which case they should be part of the directions in the script).

Equipment: Use whatever you’ve got! Digital recorders, tape recorders, mobile phones, cameras with video.

If you are recording digitally and you make a mistake, post production editing will allow you to delete this error, so just pause and restart the sentence. You don’t need to go to the beginning of the text every time.

Be aware of ambient sound before you start.
For example, traffic outside, road works, a noisy class next door, an imminent lunch break bell.

Practice.
Do some run-throughs before you actually record. Or just do lots of recordings and choose the best one. If you are using the semi-scripted technique, the first recording is often the best!
 Don’t forget any acknowledgement that may be due.

Have fun! – you will sound more natural!

Post-production editing can be done using a sound editing program such as Audacity or NCH Sound Editor. This will allow you to delete unwanted sections and add introductions and effects.
© NEAS Ltd 2014 6

Slide 15: Answer Key
Step 9: Write and check the answer key/marking guide .
Ensure that there are only very few possible correct answers to each question.
Check whether the responses correspond to what you were expecting. If a question produces more than three alternative, correct answers, moderate the items and/or the text.
Make sure too much writing is not involved. If a question demands quite a long, written, response, consider using a different type of question, eg a gap fill or cloze.
Always make a final check especially if the text or questions have changed in any way.
If possible, identify the criterion to which each question relates. This will ensure that the task complies with the curriculum, and help teachers decide how many questions need to be correct for the learner to achieve the learning outcome.
Slide 16

C heck you haven’t forgotten anything
(use the checklists) and do the task yourself, from the beginning.

Get someone else to look at and evaluate the task. Nobody, however skilled and experienced, can write an appropriate and successful assessment task by themselves. Getting a colleague to look at the task will raise new ideas and questions.

Pilot the task on learners.
This is essential. Learners have very different perspectives from their teachers, and, as they struggle to understand and come to grips with living in Australia and using English, their interpretations of a task may produce some completely unexpected responses. Remember that when piloting a task it is the task that is being assessed, not the students! The students should be made aware of this and that their comments will be valued. Tasks should be piloted on learners who have reached competency and are about to be assessed, or who have recently moved up a level and are therefore only just above the skill level being tested.

Modify as necessary and appropriate, according to feedback from piloting

Get the task proofread by someone who has not been involved in its development.
This is particularly useful in checking the answer key as often the text and questions get modified, but the answer key is forgotten.
Slide 17: Activities
© NEAS Ltd 2014 7

Activity 1: Battery Disposal (CSWE I, II or III)
Develop a script
Materials required:

CSWE curriculum

Tape or digital recorder.
Using the scenario below (from CSWE II C2 A Load of Rubbish), develop a script and a draft recording for:
1. CSWE I H1 Demonstrate understanding of a simple answering machine message, or
2. CSWE II E1 Demonstrate understanding of a telephone message, or
3. CSWE III E1 Demonstrate understanding of a spoken information text.
Scenario for ‘Battery Disposal’
The following are frequently asked questions regarding the disposal of a car battery. Design a script for your local Council’s answering machine message that will give the necessary information. You can choose what to include, what to leave out, and add extra information as appropriate.
FAQs:

Can you put the car battery in the rubbish bin?

Will the council pick up old batteries from your home?

Where can you dump the battery?

What is the address of the Council waste management centre?

What are the opening hours during the week/weekend?

How much does it cost?
Produce a draft recording and a hard/soft copy of the script.
Activity 2: Battery Disposal
Develop the questions, answer key and rubrics
Either , use the script that you have just developed and develop questions, answer key and rubrics.
Or, swap scripts with another group, in which case Step 1 will be to assess the text against the criteria.
If you plan to use the task for real assessment of learners, the task should then be piloted, amended and validated.
© NEAS Ltd 2014 8

Activity 3:
Assess a script: New Square (CSWE I)
Materials required:

CSWE curriculum

Photocopies of the script for New Square (below)

Photocopies of the Checklist for evaluating a potential text for a listening assessment task
(below)
Assess the following script against the CSWE curriculum for CSWE I D1Demonstrate understanding of a spoken information text.
For the presenter/facilitator ONLY:
Some of the problems include:
Text is lexically very dense – more redundancy is needed for a spoken text.
Text does not flow well – more written than spoken.
Some information is ambiguous.
Not a highly familiar and relevant topic (unless you live in Melbourne).
Street names are too difficult for level 1.
Make a note of any other problems that your participants identify.
© NEAS Ltd 2014 9

1. Is the topic a. relevant to the students, b. interesting for the students, c. appropriate for current learning focii (eg, employment/settlement) d. practical to assess?
NB: Avoid very familiar topics, traumatic topics, and texts with undue cultural, gender or religious bias.
2. Is the text one that students have already seen or heard, or could answer the questions without listening to the recording?
3. Do you have a model that you could use as a guide?
4. Does the text have a lot of discrete information that can be used to formulate questions?
5. Consider the difficulty of the text including a. length b. density of information c. difficulty of vocabulary d. difficulty of grammar e. discourse structure f. contextualisation.
6. How much of the above can be modified to suit the level and LO without losing all authenticity?
You may need to a. Write in redundancy (to space out the items or add extra detail) b. Change the structure (to allow of a sequencing question) c. Remove ambiguities d. Include more discourse markers.
7. Is there a question of copyright regarding the text (ie has it come from some other publication)?
8. Will the text satisfy the requirements of the curriculum?
9. Does the text provide the necessary opportunity to ask appropriate question items?
© NEAS Ltd 2014 10

Activity 4:
Develop a Listening assessment task
(From Hargreaves, M. Guidelines for Developing Tasks to Assess Listening Skills in the
CSWE . AMEP RC, 2010)
Certificate III E1 Demonstrate understanding of an oral presentation
– ‘Harmony Day’
This is a task that was developed for the 2003 CSWE curriculum, but for a number of reasons was not included in the Task Bank for the 2008 CSWE curriculum.
NOTE this task is very similar to but is not the one currently available on the ATB.
Included here is the text for the spoken information.

Assess the text against the criteria of the 2008 CSWE curriculum.
(Refer to the Checklist for Selecting or Writing Authentic Texts )

Modify the text if necessary.

Develop o o o questions for the learners, directions for administering the task, and an Answer Key for the teachers.
(Refer to the Listening Task Evaluation Checklist )

Pilot the task on an appropriate group of learners.
(Refer to the Piloting Checklist )

Record the new text and finalise the formatting of the completed task.
Ensure that all participants have signed the Copyright form if necessary.
Further activities:

Modify the text so that it is suitable for a different level of learner:
- CSWE I D1 Demonstrate understanding of a spoken information text
- CSWE II D1 Demonstrate understanding of a spoken information text

Consider using the same topic for developing a task for a different Learning
Outcome:
- CSWE I F1 Demonstrate understanding of a short spoken description
- CSWE II B1 Demonstrate understanding of a casual conversation
- CSWE III E1 Demonstrate understanding of spoken information text/media interview

Develop questions, directions and answer keys as appropriate.

Pilot the new task.
© NEAS Ltd 2014 11

Good morning everyone. My name is Marion Visentin and I work as the NT Publicity Officer for the Department of Immigration and Citizenship, or D-I-A-C, DIAC.
I’m pleased to be here today to talk to you, to explain to you and give you information about a SPECIAL national day in Australia – Harmony Day. I’ll be talking about its origin and purpose, and ways in which the day is celebrated all over Australia.
In actua l fact, this is a relatively ‘new’ day as it has only been in existence for 8 years!
Australia’s first Harmony Day was celebrated on 21 st March in 1999 to coincide with the
International Day for the Elimination of Racial Discrimination. Since then it has been celebrated around Australia on that same date – March 21 st – every year.
Did you know that since 1945 more than six million people have settled in Australia from different countries overseas and that between us all in Australia we speak more than 200 languages, including 45 indigenous languages? These facts demonstrate clearly that
Australia is truly a culturally diverse nation.
The word ‘harmony’ carries with it the idea of agreement, or the reconciliation of differences.
So the idea behind Harmony Day is that we should celebrate our ability to live together in a united and peaceful way, despite our different cultural backgrounds. This idea is reflected in the Harmony Day motto: YOU + ME= US.
On Harmony Day people who want to show their support for these ideas try to wear something orange, often including an orange ribbon. Orange is a bright and happy colour and is worn on Harmony Day as a symbol of anti-racism.
All over Australia, and especially in primary schools, special activities and events take place to celebrate Harmony Day. These are lots of fun and include things such as dressing-up in national costumes, food fairs, dancing and singing songs from other countries.
It’s a day to enjoy being part of our diverse and multicultural Australia.
(331 words, inc title)
© NEAS Ltd 2014 12

Activity 5:
Make a recording from a semi-scripted text.
Option 1a
CSWE I: New Square
Following Activity 3,

Identify the scenario for an authentic dialogue.

Identify key elements/words that need to be included so that appropriate questions can be written for the task to enable you to have a ‘skeleton’, or semi-scripted script.

Choose two people to act out the dialogue.
 Check the handout on making ‘Recordings of Scripts for Listening Tasks on the
Assessment Task Bank’

Check your recording equipment.

Have fun!
Note that while this activity may only produce a draft recording from which question items can be developed, it is often the case that an early, or even the first recording needs little serious alteration. In this case, use a Sound Editing program (such as Audacity) to edit out mistakes, add an introduction, and add sound effects for a finished product.
Option 1b
CSWE III : A Job Interview
Following the same procedure as above, develop a semi-scripted outline and make a recording for a listening task involving a job interview.
© NEAS Ltd 2014 13

Option 2
CSWE II: At the Doctor’s
Use the following framework for a dialogue in the Doctor’s surgery (developed from a semiscript by Philip Nichols, WA Polytechnic).

Read through the dialogue to become familiar with the scenario.

Adapt as necessary.

Choose two people to act out the dialogue.
 Check the handout on making ‘Recordings of Scripts for Listening Tasks on the
Assessment Ta sk Bank’.

Check your recording equipment.

Have fun!
The Doctor
Greet Mrs Long
Sit down.
Problem?
What’s wrong?
Ask a few questions.
Smoke?
How many cigarettes a day?
Not too good.
What about alcohol?
Drink?
How many standard drinks a day?
What about coffee/energy drinks?
How many hours sleep/night?
What time go to bed?
What time wake up?
Good diet?
When eat main meal?
What kind of work?
Exercise?
Give advice re
Smoking
Drinking
Eating
Exercise
The Patient (Mrs Long)
Respond
Don’t feel too good.
Always tired.
Respond positive.
20
Not much
2/3
5/6 cups of coffee/day
7
Time
Time
Respond OK
Time quite late
Bank
Respond negative
Respond
Bye
Bye.
Next please
© NEAS Ltd 2014 14