AMEP Assessment Task Bank
Professional Development Kit
How to Develop a Reading Assessment Task
Notes to accompany the PowerPoint for the AMEP Assessment Task Bank Professional
Development Kit. Developed by Marian Hargreaves for NEAS, 2013.
Slide 1: Front page
Slide 2: Aims of this workshop
This workshop will



review the main steps in developing a task to assess reading skills
complete activities that practice task development skills
develop new assessment tasks
Slide 3: Design and Development of a reading assessment task
Step 1: Decide



what CSWE level you want to assess
which module, and
which Learning Outcome (LO)
For example CSWE I K1 Demonstrate understanding of a written recount.
Consider whether it might be integrated with another learning outcome which may also
involve listening but may involve other macroskills (ie reading, writing, speaking), for
example, CSWE I K2 Write a recount.
Slide 4: Designing a task cont.
Step 2: Check the relevant Learning Outcomes in the CSWE framework and the criteria for
the one you have chosen. At this point you need to be very clear about the specifications for
the learning outcome (LO), what you want to assess, and whether the task that you are
considering will actually assess the skill that you want to evaluate.
Step 3: Look at existing models in the Assessment Task Bank to see how other people have
interpreted and applied the specifications.
Step 4: Choose a suitable text that can be used as a reading text. Texts should be as
authentic and relevant as possible. [See the ‘Checklist for selecting authentic texts’ which is
at page 7 of this document and go through the main points.]
Slide 5: Points to consider
© NEAS Ltd 2014
1









Know your specifications (ie the curriculum with the criteria and conditions of
assessment).
Choose texts of appropriate length (The curriculum may specify a range, but in any
case you do not want to overburden your learners – while still giving them a task that
will show the extent of their ability).
Choose texts not dissimilar to those in existing assessment tasks so that the tasks
are comparable.
For specific skills eg scanning, find texts which have the specified elements that have
to be scanned for. These may have to be built into the text or adapted.
Choose texts that have a recognisable structure, eg recount or report.
Choose texts that are interesting but not over exciting or disturbing.
Consider candidates’ background knowledge – it may be very different between
students and should never be assumed.
Do not use texts that require cultural knowledge (especially Australian cultural
knowledge, unless it has been specifically taught, eg snags at a bbq).
Do not use texts which learners have already read in class or for homework.
Slide 6: Designing a task cont.
Step 5: Evaluate the text in detail according to the specifications for the LO in the CSWE
and consider the following criteria:
Topic
 interest level for learners
 bias (cultural, gender, religious and so on)
 avoid very familiar topics.
Difficulty
 length (density of information)
 difficulty of vocabulary (technical, colloquial)
 difficulty of grammar (such as amount of embedding (eg relative clauses), verb
tenses etc)
 structure of discourse or genre
 amount of contextualisation (for example, supporting graphics, introductions).
Nature of input
 consider the effects of formatting, fonts, legibility and so on. If the task uses colour,
will this photocopy well into greyscale?
Suggestion: Activity 1.1 Selecting and adapting a text, could be done at this point. A new
text could be sourced or written, or you could start by looking at the text supplied for the
activity.
Slide 7: Designing a task cont.
Step 6: Modify the text as necessary. Unless you specifically write the text yourself, some
change will almost always be necessary to take into account any of the issues listed above
or to make it fit the specifications for the LO. For a reading text this might mean cutting the
text, expanding, reformatting and so on. You might need to build in redundancy if items are
too close together, change the structure and include more discourse markers, take out
ambiguities, include more information/propositional content and so on.
© NEAS Ltd 2014
2
It is effective at this stage to have some idea of the question items that you plan to develop
to go with the text. For example, if the criteria asks for the learner to show knowledge of
vocabulary, appropriate lexical items need to be included.
Suggestion: Use PD Kit Question Items (on ATB website page) at this point
or hand out Question types overview.
Step 7: Develop the questions

Provide a variety of questions, but not so many as to be confusing.

Ensure that the students are familiar with the type of question and understand what
they are supposed to do.

Ensure that students know what is expected of them.

Start with some easy questions as a lead-in.

Ensure that questions follow the text ie the answer to Q1 comes before the answer to
Q2, and summary questions come at the end.
Be aware of test method effect. This refers to the difficulty of certain item types and the
familiarity that learners have with different types. For example, if learners have not done a
matching activity before, they may not know how to do it and either waste time working out
what they have to do or get it wrong because they didn’t know what was required. Test
preparation courses usually include a lot of practice with the types of items that will be met in
the test.
Use a range of item formats. In order to minimise test method effect, it is a good idea to
use a range of item types in the task. In this way, learners who find one type difficult won’t be
overly disadvantaged. However, you need to be careful that the task isn’t too cognitively
demanding because there are too many different item types – balance is important.
Slide 8: Designing a task cont.
Step 8: Write the rubrics. Consider any additional instructions for the teacher that may be
necessary. Remember to make the language of instructions to the learner at least as easy
as the language of the text of the assessment itself. These instructions need to be simple.
You don’t want the learners to get stuck because they don’t understand the instructions!
They should be consistent for all tasks in that LO.
Step 9: Write and check the answer key/marking guide.
Ensure that there are only a few possible correct answers. Check whether the
responses correspond to what you were expecting. If a question produces more than three
alternative, correct answers, moderate the items and/or the text. Similarly, if a question
demands quite a long, written, response, consider using a different type of question, eg a
gap fill or cloze.
Slide 9: Review.
Step 10: Do the task yourself, check you haven’t forgotten anything!
© NEAS Ltd 2014
3
Step 11: Get someone else to look at and evaluate the task. Nobody, however skilled and
experienced, can write an appropriate and successful assessment task by themselves.
Getting a colleague to look at the task will raise ideas and questions not previously thought
of.
Step 12: Pilot the task on students. This is essential. Learners have very different
perspectives from their teachers, and, as they struggle to understand and come to grips with
living in Australia and using English, their interpretations of a task may produce some
completely unexpected responses. Remember that it is the task that is being assessed, not
the students! The students should be made aware of this and that their comments will be
valued. Tasks should be piloted on learners who have reached competency and are about to
be assessed, or who have recently moved up a level and are therefore only just above the
skill level being tested.
Step 13: Modify as necessary and appropriate according to feedback from piloting.
Step 14: Get the task proofread by someone who has not been involved in its
development. This is particularly useful in checking the answer key as often the text and
questions get modified, but the answer key is forgotten.
Suggestion: Activity 3, on Piloting, could be done at this point.
Slide 10: Handout: How to develop a reading assessment task – overview (see below
for handout).
Slide 11: Activities
This is an outline of suggested activities, with material available in separate files. You may
choose to do the activities at the suggested points during the powerpoint presentation, or to
do one or more following the presentation. This slide is for the latter situation and may be
omitted if you have already done the activities.
Activity 1.1: selecting and adapting the text
CSWE I K1 Demonstrate understanding of a written recount (Moving House)
Activity 1.2: developing questions for the learning outcome
Activity 1.3: develop a Multiple Choice Question for criterion 1.1: Demonstrate
understanding of topic/participant/s
Activity 1.4: develop a sequencing question for criterion 1.2: Demonstrate
understanding of sequence of events
Activity 2: Sister Emily
CSWE 3 K1 Demonstrate understanding of a biography ‘Sister Emily’
This text is suitable for assessing and adjusting the difficulty level of a text. ‘Sister Emily’ is a
task currently on the Assessment Task Bank, but the text is not the same as the one for this
activity.
© NEAS Ltd 2014
4
This is an alternative text to assess, modify and develop questions items for a CSWE 3
learning outcome, cf Activity 1.1
Activity 3: Spot the problems! An activity to show the importance of piloting.
CSWE I GI Read a Description “Jane”
Slide 12: References
© NEAS Ltd 2014
5
How to develop a reading assessment task – overview
•
Decide what aspects of reading you want to assess.
•
Check the curriculum.
•
Consider corresponding/integrated Learning Outcomes (LO).
•
Select an authentic text for the LO. Refer to ‘Checklist for selecting authentic
texts’.
•
Modify the text and think about the question items that you want to use. Refer
to the Questions types overview handout.
•
Write the rubrics.
•
Write and check the answer key/marking guide.
•
Do the task yourself, check you haven’t forgotten anything!
•
Get someone else to look at and evaluate the task.
•
Pilot the task on students. See the Piloting Checklist and analysis grid. Use
the Teacher evaluation form if preferred. (These are located on the ATB
Website page).
•
Modify as necessary.
© NEAS Ltd 2014
6
Checklist for selecting authentic texts for reading and listening assessment
1. Choose the learning outcome/s that you want to assess.
2. Check the criteria from the curriculum.
3. Choose a topic. Consider topics that are:
 relevant to the learners
 interesting for the learners
 appropriate for the current learning focus, for example work issues
 practical to assess.
Avoid very familiar topics, traumatic topics, and texts with undue cultural, gender or
religious bias.
4. Look at any models you could use as a guide, but avoid texts that the students have
already read.
5. Collect as many potential texts as possible. Look for texts that have a lot of discrete
information that can be used to formulate questions.
6. Consider the difficulty of the text including:
• length
• density of information
• difficulty of vocabulary
• difficulty of grammar
• discourse structure
• contextualisation.
7. Start modifying the texts according to the criteria and the level of the students. Consider
how much can be modified to suit the level of difficulty and the Learning Outcome
without losing all authenticity.
You may need to:
•
•
•
•
write in redundancy (to space out the items or add extra detail)
change the structure (to allow for a sequencing question)
remove ambiguities
include more discourse markers.
Remember the importance of font and formatting for text comprehension.
8. At the same time as modifying the text, think about the sort of questions you would like
to write. Does the text provide the necessary opportunity to ask appropriate question
items? Use the Question types overview handout to remind yourself of the options
available. Don’t be tempted to include dichotomous items (yes/no; true/false).
9. Is there a question of copyright regarding the text? Has it come from some other
publication, do you need to get permission from any authors or participants?
Reading task evaluation checklist
© NEAS Ltd 2014
7
CSWE Level ______
Learning Outcome
Title of task ______________________________________________________________
Task conditions
Complies with curriculum and criteria
Use of dictionary is stated
Compulsory/optional questions (if any) are identified
Task characteristics
Complies with curriculum and criteria
Assesses a range of the skills/learning outcomes required for this level
Equivalent difficulty to other tasks at this level
Includes visual clues, introduction and so on to contextualise the task
Rubrics are clear
Format and layout are clear
Reading text
Complies with curriculum
Discourse difficulty is appropriate
Items
Complies with curriculum and criteria
Language of items is simple
Item types are familiar
Rubrics are clear
Items produce unambiguous responses
Text/item relationship
Items in text order
Limited number of possible responses for each item
Answer key and Marking Guide
Answer key indicates all acceptable responses
Directions to teachers are included (for example the marking for each question)
© NEAS Ltd 2014
8