The International English Language Testing System (IELTS): A Review Stephen Spanos Monterey Institute of International Studies EDUC8540A: Language Assessment September 26 2014 2 IELTS REVIEW Introduction With over two million tests administered in 2013, IELTS is the most popular high-stakes English proficiency test for higher education and global migration in the world (IELTS, 2014b). IELTS offers both an academic test and a general test intended for candidates seeking non-academic training, to gain work experience, and for immigration purposes. The test is gaining popularity in the US as an alternative to the TOEFL for higher education, though I mainly chose to research because of its international outlook and emphasis on multiple varieties of English. I plan to teach English and train teachers in Indonesia for the Peace Corps in March, where close proximity to Australia will likely influence English language education and student goals. With the following research and analysis I hope to better understand IELTS and the challenges test takers face. History of the IELTS The IELTS origin dates to 1980 when the English Language Testing Service (ELTS), as it was then known, replaced the English Proficiency Test Battery (EPTB), a largely traditional, multiple choice test used from the 1960s to screen international applicants to British universities and colleges (IELTS, 2014a). The ELTS shifted focus to language use in academic contexts while also reflecting real world use, as the new format “was influenced by the growth in ‘communicative’ language learning and ‘English for specific purposes (IELTS, 2014a).’” ELTS offered a test for higher education in the UK and a Non-Academic test for vocational candidates. The higher academic test offered a choice of six modules covering general academic areas such as Life Sciences, Medicine, and Social Studies. The Non-Academic test consisted of study skills, writing, an individual interview, and a general reading and listening section. Though the tests IELTS REVIEW 3 themselves have changed, IELTS continues to offer separate academic and a vocational tests. The ELTS test numbers remained low (10,000 by 1985) until the exam was revised in 1989 to reduce redundancy, broaden international participation, and increase validity (Alderson and Clapham, 1993). The British Council and UCLES EFL (now known as Cambridge English Language Assessment) joined with the International Development Program of Australian Universities and Colleges (IDP), now known as IDP Education Australia, as well as several academics from Britain and Australia, to develop and conduct research on the test. The format of the test was simplified and shortened as the subject-specific sections were reduced from six to three and the General Module replaced the Non-Academic test (Alderson and Clapham, 1993). Further revisions were made in 1995 to align assessment methods with theoretical and practical updates in language testing. The subject-specific sections were replaced with more general academic modules to increase fairness between test takers of different backgrounds (Clapham, 1996) and data was gathered on test performance and candidate background so IELTS designers could monitor fairness relating to test use and users (Charge and Taylor, 1997). Most recently, IELTS introduced a revised Speaking Test in 2001, new assessment criteria in 2005, and a computerized pilot version of the entire test in 2005. The original focus on direct testing, communicative language competence, and distinction between Academic and General Training models continues today. IELTS is increasingly used by students across the globe to gain admission to universities in countries such as the USA, UK, and Australia, or to obtain a visa in English speaking countries such as 4 IELTS REVIEW Canada or Australia. The test is accepted by over 9,000 organizations worldwide and is administered by more than 900 test centers in over 130 countries (IELTS, 2014c). General Composition and Description of IELTS IELTS consists of one section for each of the four language skills: listening (thirty minutes), reading (sixty minutes), writing (sixty minutes), and speaking (eleven to fourteen minutes). The Academic and General Training versions are identical in time limits for all sections; the topics differ in the reading and writing sections, with the Academic Test focusing on subjects authentic to undergraduate, postgraduate, and professional candidates while the General Training test uses materials relevant to everyday life (IELTS 2014d). A more detailed comparison of the two tests is provided in Appendix A. Information regarding pricing and application forms for the test can be found in Appendix B. To ensure security of the test, there are stringent measures and codes at each test center, including procedures to check candidate identity, unique test versions, and certified examiners (IELTS 2014h). Wesche’s Framework I will use Wesche’s (1983) framework to provide a structured overview and analysis of the IELTS test based on four constructs: stimulus material, tasks posed to the learner, learners’ response, and scoring criteria (Wesche, 1983). Stimulus Material The first component of Wesche’s (1983) framework is stimulus material, or “whatever linguistic or non-linguistic information is presented to the learners to get them to demonstrate the skills or knowledge [testers] want to assess (Bailey, 1998, p. 13). Descriptions of stimulus materials for each test section are provided in Table 1. More 5 IELTS REVIEW detailed descriptions of each section are provided in Appendix C. Sample stimulus materials and their corresponding tasks posed to the learner are provided in Appendix D. Table 1 Stimulus Materials for IELTS Test Stimulus Material (Academic) Section Four recorded monologues and Listening conversations Three long reading passages Reading ranging from descriptive to analytical; non-verbal materials (i.e. diagrams, graphs). All are authentic materials (e.g. from books, newspapers). Two prompts: one that includes a Writing graph, chart, diagram, or table and another that describes a point of view or problem that must be discussed In face-to-face interview, Speaking stimulus materials include short questions, guided discussion prompts by interviewer, and a booklet which asks test takers to talk about a particular topic Stimulus Material (General Training) Four recorded monologues and conversations Three reading passages including short factual texts, short work-related texts, and one longer general interest text. All are authentic materials (e.g. from company handbooks, newspapers). Two short written prompts: one that describes a situation that requires action (e.g. writing a letter) and another that describes a point of view or problem which must be discussed In face-to-face interview, stimulus materials include short questions, guided discussion prompts by interviewer, and a booklet which asks test takers to talk about a particular topic The stimulus material for the listening, reading, and writing sections are authentic materials that appear in books, newspapers, and journals. The speaking section stimulus materials include a real person asking questions. Such authentic materials and face-toface interview prompts are a good starting point for IELTS to promote face validity, the idea that, “on the face of it, a test will appear to be valid to test-takers, teachers, parents, employers, and so on (Bailey, 2015, p. 255).” The prompts are selected to match what the test-takers are likely to encounter in the contexts they wish to enter. Of course, the stimulus materials need to be matched with appropriate tasks for the learners to ensure validity, which I will discuss later. 6 IELTS REVIEW Task Posed to the Learner The second component of Wesche’s framework, the task posed to the learner, refers to the cognitive processes test-takers use to understand the task and produce output (Bailey, 1998). The tasks are provided in Table 2. Table 2 Tasks Posed to Test Takers Test Task Posed (Academic) Section Listen to stimulus material and Listening understand the main idea and key details; remember details after listening in order to answer questions Read and understand three long Reading academic texts; comprehend main idea, recognize key points and terms, scan for information; make inferences about the author’s point of view and feelings; demonstrate understanding of rhetorical structure of essay or article Understand the visual and essay Writing prompts; plan and organize a connected, concise, well-written response that relates directly to the prompt Understand stimulus material, plan Speaking response, form a coherent, organized, and direct response. Repeat the process for any followup questions by interviewer Task Posed (General Training) Listen to stimulus material and understand the main idea and key details; remember details after listening in order to answer questions Read and understand everyday texts; comprehend main idea; recognize key points and terms; scan for information; make inferences about the author’s point of view and feelings; demonstrate understanding of rhetorical structure of passage or article Understand the situational and essay prompts; plan and organize a connected, concise, well-written response that relates directly to the prompt Understand stimulus material, plan response, form a coherent, organized, and direct response. Repeat the process for any follow-up questions by interviewer Although the tasks listed above attempt to isolate each language skill, “there is inevitably an element of integration in each component in the same way that language skills are integrated in the real world (IELTS, 2014d).” Since test takers need to read a prompt for the writing section and listen to prompts and questions during the speaking section, the tasks do not completely isolate each language skill. Still, the prompts are 7 IELTS REVIEW intended to be short and simple so test takers do not suffer if they struggle in one skill. Beyond the language skills, the hierarchical components of language, listed as discourse, syntax, lexicon, morphemes, and phonemes (Bailey, 2015), are highly integrated for many of the tasks, especially in the speaking and writing sections. For example, the speaking section task (test takers answer general questions about themselves) invokes all hierarchical components of both listening and speaking. As is usually the case for integrative and productive skills, these tasks are direct testing methods, or “those which the learners’ response to test items involves actually doing the skill being assessed (Bailey, 2015, p. 64).” Even many of the tasks that are traditionally linked with discretepoint testing are integrative, such as multiple choice and sentence completion in the listening and reading sections. Although some multiple choice questions measure grammar, many of the IELTS tasks require test takers to comprehend the overall concept of the stimulus material (discourse level) based on the sentence structure (syntactic) as well as relevant vocabulary (lexical). Learners’ Response The test takers’ response is the third component to Wesche’s framework. Essentially, the learners’ response is the physical answer the test taker produces – whether by filling in a multiple choice bubble, writing an essay on paper, or speaking to an interviewer. Descriptions of learners’ responses are provided in Table 3. Table 3 Learners’ Response Test Learners’ Response (Academic) Section Test takers answer items that Listening include Multiple Choice, Form Completion, Short Answer, Learner’s Response (General Training) Test takers answer items that include Multiple Choice, Form Completion, Short Answer, Sentence Completion, 8 IELTS REVIEW Reading Writing Speaking Sentence Completion, Matching, Diagram Labeling Test takers answer items that include Multiple Choice, Matching, Summary Completion, Short Answers, Matching Sentence Endings 2 responses: candidates must describe visual information in own words (at least 150 words) and write a short essay in response to prompted point of view or problem (at least 250 words). One hour to complete both tasks. 3 responses: test takers answer general questions about themselves and interests (4-5 minutes); prepare response to booklet prompt (1 minute prep, 2 minutes speaking); respond to further questions connected to booklet prompt (4-5 minutes) Matching, Diagram Labeling Test takers answer items that include Multiple Choice, Matching, Summary Completion, Short Answers, Matching Sentence Endings 2 responses: candidates must respond to a situation (e.g. write a letter) in at least 150 words and write an essay in response to a prompt with at least 250 words. One hour to complete both tasks. 3 responses: test takers answer general questions about themselves and interests (4-5 minutes); prepare response to booklet prompt (1 minute prep, 2 minutes speaking); respond to further questions connected to booklet prompt (4-5 minutes) It is important to note that these responses must be completed under strict time constraints. While such limits are necessary for test practicality, some students and teachers feel the time limits, especially for the writing section, do not reflect a research university setting (Lewthwaite, 2007) and many note that time limits cause added stress when forming responses. Scoring Criteria The scoring criteria are provided in Table 4. A more detailed explanation is provided in appendix E. There is no distinction between the Academic and General Training tests because both are graded on the same scale. Test Section Scoring Criteria Listening 40 items: each correct item is awarded one point. No points for wrong or unanswered items. Maximum raw score is 40. 9 IELTS REVIEW Reading Writing Speaking 40 items: each correct item is awarded one point. No points for wrong or unanswered items. Maximum raw score is 40. 4 criterion areas, each weighted 25%: task achievement (task 1)/task response (task 2) which can be understood as providing a relevant answer to the prompt and questions; coherence and cohesion; lexical resource; grammatical range and accuracy 4 criterion areas, each weighted 25%: fluency and coherence; lexical resource; grammatical range and accuracy; pronunciation The criteria, especially for the writing and speaking sections, reflect the integrative aspect of the hierarchical components, as mentioned previously. Additionally, IELTS claims that a variety of styles of English lexical, grammatical, and pronunciation use is allowed, including both US and British (IELTS, 2014d). Scoring System The scoring system reflects criterion-referenced testing since “a student’s score is interpreted relative to a preset goal or objective – the criterion rather than to the performances of other test-takers (Bailey, 2015, p. 56).” This is a move away from previously popular norm-referenced language tests. The goal of the IELTS is to measure test taker language competence, not to compare learners against each other. For the reading and listening sections, raw scores are used to give corresponding scores on a Band Scale from one to nine. The speaking and writing sections use the criteria listed above, mostly holistic and subjective, to produce a band score. Detailed descriptions for the speaking and writing sections’ band scores are provided in Appendixes F and G. The four section scores are then averaged to produce an Overall Band Score. All band scores are rounded and reported in whole bands or half bands, such as a 6.5 in writing and 7.0 in speaking (IELTS, 2014e). The website explains that because of difficulty between tests occasionally varies, “the band score boundaries are set so that 10 IELTS REVIEW all candidates’ results relate to the same scale of achievement. This means, for example, that the Band 6 boundary may be set at a slightly different raw score across versions (IELTS, 2014e).” The website does not specify exactly how such boundaries are created and adjusted to ensure fairness across tests. IELTS tests are graded at the test center by trained graders and examiners. To become an examiner, one must possess qualifications that include an undergraduate degree or equivalent, a TEFL/TESOL qualification from a recognized institution, and substantial relevant teaching experience (IELTS, 2014d). Qualified applicants undergo interviews, training, standardization, and certification, along with refresher workshops to increase reliability, the concept that the results of a test should be consistent (Brown, 2005). Public reports are available that illustrate reliability using Cronbach’s alpha for the reading and listening sections. However, such calculations to measure reliability for the writing and speaking sections, which rely on subjective ratings by examiners, are not possible (IELTS, 2014g). Room for improvement in rater reliability has been suggested, especially regarding how scores are rounded to a final score and rater behavior while using the band scales (Uysal, 2010). Swain’s Framework Swain’s (1984) framework consists of four components: start from somewhere, concentrate on content, bias for best, and work for washback. I will briefly analyze IELTS based on each component. Start from somewhere suggests that “test development should build from existing knowledge and examples (Swain, 1984, p. 188).” As explained above, IELTS’ primary initial goal was to measure communicative competence in authentic situations. Such IELTS REVIEW 11 objectives are reflected in the stimulus materials and tasks proposed to the test takers, as well as the integration of language skills and hierarchical components. Even though IELTS gives scores for each language skill, test administrators acknowledge that skills do overlap in the test as in the real world, which aligns with the framework that language skills are interrelated and mutually reinforcing (Kumaravadivelu, 2003). Additionally, the IELTS’ movement toward increasingly integrative and international assessment connects with Savignon’s comparison of test items based on discrete versus integrative tasks and their level of global response mode (Savignon, 1983). Concentrate on content “refers to both the content of the material used as the basis of communicative language behaviours (Swain, 1984, p. 190).” This principle emphasizes the importance that test items should contain appropriate content for the test takers. IELTS’ use of authentic material and practical items such as letter writing for the General Training test and academic journal reading for the Academic test illustrate its adherence to Swain’s second principle. The third component to Swain’s framework is bias for best, which means that the test should do everything possible to elicit the learner’s best performance. While there are some challenges for learners, such as time constraints, IELTS has made strides to aid test takers. One innovative aspect is the test’s recognition of various English dialects and an international focus. Recently appointed Executive Director Ariel Foster stated that the IETLS is “a truly international test, incorporating a range of global English, a wide variety of accents and ways of speaking, not just American English (IELTS, 2014b).” Such acceptance for accents, vocabulary, and grammar use could help test takers since they have different schemata and experience with English styles. 12 IELTS REVIEW The recognition of various Englishes could also affect the fourth component of Swain’s framework, work for washback. Washback is the effect a test has on teaching practices; another way to think about washback is by analyzing the “curriculum that is related to” the test (Brown, 2005, p. 242). Because the test is focused on authentic material and a communicative approach there seems to be a high potential for positive washback. However, this is not true in all regions, as a study conducted in Cambodia found that test takers and teachers believed the IELTS to be “too Eurocentric,” with Cambodians needing to “acquire more ‘world knowledge’ in addition to developing their English language proficiency skills to achieve a satisfactory band score (Moore, Stroupe, and Mahony, 2012, p. 1).” Additionally, IELTS in Cambodia was found to have little impact on Cambodian education in general. So, while IELTS has made progress with regards to Swain’s framework, there is much opportunity for further improvement. Validity There has been a great deal of effort by IELTS to conduct, publish, and publicize research on IELTS testing methods and test items to ensure validity. Brown (2005) calls validity “the degree to which a test measures what it claims, or purports, to be measuring (p. 220).” Several studies have compared students’ IELTS scores and their ability to cope with undergraduate and graduate courses. Most of the results suggest that “students were generally able to produce, in the context of their academic studies, the language behaviour implied by an IELTS test score (Ingram & Bayliss, 2007, p. 2).” Such conclusions are not surprising considering that the Academic Test’s stimulus materials and prompts mirror university assignments. Additionally, attempts to correspond IELTS 13 IELTS REVIEW scores with Common European Framework of Reference for Languages (CEFR) levels, though only rough estimates, illustrate further attempts to create validity. Conclusion There is a substantial amount of research on IELTS and I have only covered a small portion in this essay. The British Council, IDP, and Cambridge English Language Assessment, along with various academic institutions, have conducted extensive research and continue to study and publish their findings. Research on the test – hopefully from unrelated institutions – will only increase as IELTS continues to gain popularity. The test’s emphasis on international context and communicative competence will likely remain, and with further study and revisions IELTS has the potential to increase its validity, reliability, and positive washback. 14 IELTS REVIEW References Alderson, J C and Clapham, C (eds) (1993). Examining the ELTS Test: An account of the first stage of the ELTS revision project – research report 2. The British Council/University of Cambridge Local Examinations Syndicate. Bailey, K. M. (1998). Learning about language assessment: Dilemmas, decisions and directions. Boston: Heinle & Heinle. Bailey, K. M. & Curtis, A. (2015). Learning About Language Assessment (2nd ed.). Boston, MA: National Geographic Learning. Brown, J. D. (2005). Testing in language programs: A comprehensive guide to English language assessment. New York: McGraw-Hill. Charge, N and Taylor, L (1997) 'Recent developments in IELTS', English Language Teaching Journal, 51/4. Oxford: Oxford University Press. Clapham, C (1996) 'The development of IELTS: a study of the effect of background knowledge on reading comprehension', Studies in Language Testing, volume 4. Cambridge: LES/Cambridge University Press. IELTS (2014a). History of IELTS Retrieved from http://www.ielts.org/researchers/history_of_ielts.aspx IELTS (2014b). Media centre. Retrieved from http://www.ielts.org/media_centre.aspx IELTS (2014c). About us. Retrieved from http://www.ielts.org/about_us.aspx IELTS (2014d). Guide for Teachers. Retrieved from http://www.ielts.org/PDF/Guide_Teachers_2013.pdf IELTS (2014e). Researchers – Band descriptors, reporting and interpretation IELTS REVIEW 15 Retrieved from http://www.ielts.org/researchers/score_processing_and_reporting.aspx IELTS (2014f). Getting my results. Retrieved from http://www.ielts.org/test_takers_information/getting_my_results.aspx IELTS (2014g). Researchers – Test performance 2012. Retrieved from http://www.ielts.org/researchers/analysis_of_test_data/test_performance_2 012.aspx IELTS (2014h). Institutions – Security procedures. Retrieved from http://www.ielts.org/institutions/security_and_integrity/security_procedur es.aspx Ingram, D., & Bayliss, A. (2007). IELTS as a predictor of academic language performance, Part 1. In P. McGovern, & S. Walsh (Eds.), IELTS Research Reports (Vol. 7; pp. 137-204). Canberra, Australia: IELTS/ UK: British Council. Kumaravadivelu, B. (2003). Beyond methods: Macrostrategies of language teaching. New Haven and London: Yale University Press. Lewthwaite, M. (2007). Teacher and student attitudes to IELTS writing tasks: positive or negative washback? UGRU Journal, 5 – research report 2006. Savignon, S.J. (1983). Communicative competence: Theory and classroom practice. Reading, MA: Addison-Wesley, p. 253. Swain, M. (1984). Large-scale communicative language testing: A case study. In S. J. Savignon & M. Berns (Eds.), Initiatives in communicative language teaching (pp. 185-201). Reading, MA: Addison-Wesley. Uysal, H. H. (2010). A critical review of the IELTS writing test. ELT Journal, 64(3), IELTS REVIEW 314-320. Wesche, M. B. (1983). Communicative testing in a second language. The Modern Language Journal, 67, 41-55. 16