Testing Reading Testing reading Specifying what the candidates should be able to do 1. 2. 3. 4. operations texts speed criterial level of performance Setting the tasks 1. 2. 3. 4. 5. selecting texts writing items possible techniques which language for items and responses? procedures for writing items Scoring 01 Specifying what the candidates should be able to do Is it easy to test reading skill? Reason #1 The exercise of receptive skills does not usually manifest itself directly in overt behavior (compared to writing and speaking). Reason #2 The tested skills vary depending on the reading purpose, the kind of text, and the purpose of the test 1. Operation Operation is referred to the skills that readers perform when reading a text 2. Operations are based on differences of purpose Arthur Hughes (2003) Operations Expeditious reading operations Require candidates to respond to items without having time to read the full contents of a passage • • • Skimming Search reading Scanning Careful reading operations • • • • • • Identify Interpret Outline Distinguish Recognize Make inferences readability/ difficulty range of vocabulary Texts grammatical structure types forms graphic features topics style Intended readership length Text types Text books, handouts, articles (in newspapers, journals or magazines), maps or plans, advertisements, postcards, timetables, novels (extracts) and short stories, computer Help systems, notices and signs. Text forms description, exposition, argumentation, instruction, narration. Graphic features tables, charts, diagrams, cartoons, illustrations. Topics may be listed or defined in a general way (such as non-specialist) or in relation to a set of candidates whose background is known (such as familiar to the students). Style ● may be specified in terms of formality. Intended readership ● can be quite specific (e.g. native speaking science undergraduate students) or more general (e.g. young native speakers). Length ● ● is usually expressed in number of words. The specified length will normally vary according to the level of the candidates and whether one is testing expeditious or careful reading (although a single long text could be used for both). Readability ● ● ● is an objective. not necessarily very valid, measure of the difficulty of a text. Where this is not used, intuition may be relied on. Range of vocabulary ● ● may be indicated by a complete list of words (e.g: for the Cambridge tests for young learners), by reference either to a word list or to indications of frequency in a learners' dictionary. Range: expressed more generally (e.g. except where explained in the text) Range of grammar ● may be a list of structures, or a reference to those to be found in a course book or (possibly parts of) a grammar of the language. Content validity & positive backwash Texts The texts included in the tests should be representative so that candidates are able to read successfully Speed words/min Reading speed is expressed in words per minute Careful & Expeditious Different speeds are expected for careful and expeditious reading The expected reading speed Combine with the number & the difficulty of the items, based on which, the test time can be determined. Criteria level of performance • Traditional passmarks expressed in percentages are hardly helpful, due to the lack of direct interpretation of such scores. • The best way is to use the test tasks themselves to define the level. • Compare performance on the test with the rating of candidates’ reading ability using scales like ACTFL, or ILR. 02 Setting the tasks Setting the tasks 1. 2. 3. 4. 5. selecting texts writing items possible techniques which language for items and responses? procedures for writing items Selecting texts: Experience, Judgment & Common Sense Avoid general knowledge Avoid texts made up of information test-takers’ general knowledge. Suitable length Choose texts of suitable length for the required task(s). Representative Select a representative sample as possible. Avoid too culturally laden Avoid too culturally laden texts New texts Use new texts for candidates. Writing Items Measure Elicit Permit Measure the ability with which we wish to test candidates. Elicit reliable behavior from test-takers. Permit highly reliable scoring Reading test techniques 01 Multiple Choices 02 Short Answer 03 Gap Filling 04 Information Transfer Multiple Choices Learners choose one of the alternatives Multiple Choices 50% True/False It has already been pointed out that True/False items, which are to be found in many tests, are simply a variety of multiple choice, with only one distractor and a 50 per cent probability of choosing the correct response by chance. Short Answer Example It is better with a unique correct response For testing the ability to identify referents In which city do the people describes in the “Urban Villagers’ live? • To which there is only one possible correct response, e.g: Bombay. • The response may be a single word or something slightly longer, e.g: China and Japan, American women Based on the newspaper article about smoking, p.150: What does the word “it” (line 26) refer to? Based on the newspaper article about smoking, p.150: For testing the ability to predict the meaning of a word in context Find a single word in the passage (between lines 1 and 26) which has the same meaning as “making of laws” (The word in the passage may have an ending like -s, -tion, -ing, ed, etc.) Example For testing the ability to make various distinctions Short Answer Example For testing scanning Short Answer Example For testing the ability to structure a text Short Answer Gap Filling To test if the candidate had grasped the main idea Gap Filling To test the ability to recognize detail presented support a main idea. To scan items Gap Filling Summary cloze A reading passage is summarized by the tester Gaps are left in the summery for completion by the candidate Gap Filling Summary cloze Gap Filling 1. 2. 3. 4. To test if the candidate had grasped the main idea To test the ability to recognize detail presented support a main idea. To scan items Summary cloze Conclusion This technique is particularly useful in testing reading. It can be used any time that the required response is so complex that it may cause writing (and scoring) problems. The Information transfer Minimising demands on writing ability Example: a table, following a route, labelling a picture, etc. The Information transfer Language for items & responses Items Less demanding than the text itself. Responses Minimal demands on writing ability Language Candidates share the same language Procedures for writing items Step 1 Careful reading of text with specified operations in mind. 4 5 Step 2 Take notes Pinpoint the appropriate tasks Step 5 2 Wonder what a competent reader should derive from the text. Step 3 Step 4 1 Add paragraph and line numbers if necessary 6 3 Step 6 Have the text and items assessed by colleagues. Practical advice on item writing 1. 2. 3. 4. Scanning: Present items in the order they are to be found in the text. Do not include items that candidates may be able to answer from general knowledge. Make items independent of each other. Be ready to make minor changes of a text to improve the items. 03 Scoring Requirements & What to focus on Effiective scoring requires RELIABILITY How to make scoring more reliable? Tip #1 Tip #2 Sufficient number of samples of behavior Avoid too difficult or too easy test items Tip #3 Tip #4 Clear instructions Detailed scoring key Tip Sufficient number of samples of behavior: #1 - the more testing items => the more reliable - the additional items should be independent example: IELTS reading test: 40 questions/3 sections Tip Avoid too difficult or too easy test items: - unable to classify students’ competence #2 Tip #3 Clear instructions: - Do not assume that candidates are aware of what to do! example: Yes,No,Not given in IELTS: Tip #4 Detailed scoring key: - Especially for short answers => how to avoid too varied possible responses: => rule no.3: Clear and detailed instructions: What to focus on when testing reading Focus on READING ABILITY (receptive skills) 1 Less focus on: Grammar, vocabularies, spelling, punctuation. 2 To test receptive and productive skills at the same time makes the test less valid! 3 Example: What is the writer's attitude towards TikTok trendy dances today? A. He hate Tiktok trendy dances. B. He hates Tiktok trendy dances. Thanks for listening!
