Applying CEFR principles to testing and assessment in practice ACERT Conference 15th March 2014, Prague Mila Angelova Aims of the session • To give you a short update on EAQUALS operation and activities • To introduce you to the EAQUALS Certification Scheme • To (re)-introduce you to the most salient features of the six global levels according to CEFR • To assess two oral performances for standardisation purposes The Association • Founded in 1991 • A company registered in UK as charity • International NGO with participatory status at the Council of Europe (2006) EAQUALS Mission The mission of EAQUALS is: • to contribute to the development of excellence in language education • to be the internationally recognised authority for quality standards and quality assurance in language education and training EAQUALS Values • • • • Plurilingualism Intercultural understanding International cooperation Lifelong learning EAQUALS Members A network connecting 35 countries Accredited members x 112 25 languages, private and state sector Associate members x 27 British Council; Instituto Cervantes; CIEP France; Goethe-Institut; Cambridge Assessment; Trinity London; g.a.s.t. Germany; 12 national quality associations Project partners x 5 Centro di Ricerca e di Servizio + Universita per Stranieri di Perugia School of Languages, Sabanci, Turkey Individual membership (2011) EAQUALS Expertise • Quality management: - design of quality accreditation schemes - training and standardisation - capacity building at local level • The Common European Framework of Reference for Languages (CEFR): - descriptors, curriculum, assessment, - portfolio, certification • Teacher management and professional development: - teacher competences (profiling grid) - teacher training and development - academic management EAQUALS Activities International accreditation of language education - standards, inspections; recommendations Support and pathways for institutional development - consultancy; self-assessment; training resources An international professional network - regular conferences; involvement in funded projects Certificates of achievement – CEFR levels - training support, numbered certificates, website archive EAQUALS Projects • Quality management - ISO: standard for providers of language learning - Council of Europe: code of practice for integration of adult migrants - Leonardo: Quality evaluation of on-line teacher training, Quality Guidelines • CEFR - Core Inventories (English; French) - Listening and Reading scenarios (Eng, French) - EAQUALS Certificate of achievement • Teacher management & development - profiling grid for teachers (EPG Project) - teacher development framework EAQUALS Accreditation • • • • international and plurilingual across educational sectors consultative and developmental based on EAQUALS Charters EAQUALS Standards • • • • designed for language education services 12 main areas of assessment specific indicators of quality for each area 50 key assessment criteria: - clear and transparent - derived from the Charters Becoming a Member Attend an EAQUALS Conference and get to know EAQUALS: details on www.eaquals.org Complete a Self-assessment and identify areas that may need improvement Have a One-day Advisory Visit and receive consultancy on achieving EAQUALS standard Arrange your First Inspection after working through the points raised The EAQUALS Certificate of Achievement A Guarantee for Quality Assessment Brief introduction to the scheme Main aims of the scheme • To increase EAQUALS’ profile: the first organisation to award CEFR certificates of achievement across different languages Europewide • To help members differentiate themselves from their competitors and demonstrate their competence • To open the possibility, at a second stage, of accrediting CEFR certificates awarded by schools that are not yet accredited members of EAQUALS Core message • A guarantee for quality assessment in the accredited institutions by thoroughly screening all assessment processes • CEFR-implemented approach to syllabus and curriculum design • Feasibility of standardisation and moderation techniques used by the institution EAQUALS Certificate of Achievement • Phase 1: voluntary – Accredited Members only: accreditation carried out as a “top up” to the EAQUALS inspection • Phase 2: compulsory for Accredited Members; voluntary for schools which are members of Associate Members: the accreditation process is fully integrated into the EAQUALS inspection and carried out as a top up to inspections of Associate Members Why EAQUALS Certificate? Benefits for: • the institution • the course participants • the stake holders • EAQUALS profile Main prerequisites to apply for the Certification Scheme A) CEFR-based curriculum B) Standardisation training C) Moderation techniques CEFR-based curriculum 1. CEFR-related level descriptions and syllabus documents o the levels referenced to CEFR (CEFR criteria grids adopted or adapted to local circumstances) o learning aims specified by „can-do” statements based on ELP or CEFR scales and adapted to the institution’s own needs 2. Written guidelines for teachers on levels, teaching and assessment 3. Application of CEFR in teaching o coherence between curriculum and syllabus aims and classroom practice Implementing CEFR-based curriculum and syllabuses • Core of the syllabuses is the practiceoriented background of the CEFR • Can-do-statements developed for each type of course and level / sublevel • Can-do-statements serve as a basis for tests development • Can-do-statements used as a selfassessment instrument Standardisation training - requirements • • • Familiarisation with CEFR levels and descriptors Systematic standarisation training with CEFR illustrative samples (DVDs or scripts: the EAQUALS Standardisation packs – for English, French and German) Procedures for CEFR-novice and CEFR-experienced teachers Examples of standardisation training • Means of writing standardisation (making use of local samples) • Means of speaking standardisation (with official samples) • Standardisation of the placement testing (with standardisation videos and other CEFR-related tasks) Moderation Systematic reduction of subjectivity in the assessment. Moderation techniques - requirements • Collective assessment techniques o grading by two assessors (all performances, a structured sample or with less-experienced staff), or o assessing initially as a panel • Quality control o liaison between teachers o systematic monitoring by academic managers Examples of moderation techniques • Doing buzz observations during testing periods to monitor application of assessment standards • Monitoring inexperienced teachers’ assessment of spoken production and interaction / written work Assessment procedures should include at least two of: a) teacher judgements through continuous assessment in relation to CEFR/ELP descriptors b) performances in specific spoken/writen tasks assessed with CEFR-related criteria c) use of tests (progress test giving an indication of good performance - e.g.15/20), or an independent, validated “CEFR level test” reporting a CEFR level Procedure to accredit assessment processes Step 1: Introduction of a CEFR-based assessment approach »Standardisation training »Assessment procedures »Moderation techniques Step 2: A formal application Step 3: Evaluation by the EAQUALS Assessment Panel Introduction Please, discuss the following questions with the person sitting next to you: o How often do you do standardisation training at your institutions? Is it enough? o Do all teachers have to take part in it? If not, why? o What problems do you face when you do standardisation training? Warmer • Look at the descriptors for spoken production for B1 level. Fill in the gaps with the missing salient features for the level. • Compare your ideas with the person sitting next to you. CEFR Speaking Skills overview • Arrange the descriptors for the six basic • levels in the correct order. Compare with your answers from the previous stage with the actual descriptor for B1 Oral sample • Watch a short performance for 3-4 min. Decide on the speakers’ level. Bell Krakow_Anna & Magda.mpeg • Write a short description of the speaking abilities of the students in the video. • Compare your answers with the CEFR descriptor. • Provide a rationale for your decision by referring to the most salient features of the descriptors. Criteria for speaking assessment • List the basic criteria that you use for speaking assessment. • Compare your ideas with a suggested set of criteria, used at one EAQUALS institution. Sample mark sheet Candidate A Candidate B 1 2 3 4 5 1 2 3 4 5 Pronunciation: Can produce a limited number of words / phrases in a relatively recognizable manner and make her/himself understood in simple everyday situations but there is very noticeable interference from mother tongue in terms of individual sounds, word and sentence stress 1 2 3 4 5 1 2 3 4 5 Interaction: Can ask and answer questions about personal details. Can interact in a simple way but communication is totally dependent on repetition, rephrasing and repair 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 Criteria Discourse management Can manage very short, isolated, mainly pre-packaged utterances, with much pausing to search for expressions, to articulate less familiar words, and to repair communication. Can link words or groups of words with very basic linear connectors like “and” or “then” Grammar and vocabulary: Has a very basic repertoire of words and simple phrases related to personal details and particular concrete situations. Shows only limited control of a few simple grammatical structures and sentence patterns in a memorized repertoire. Global achievement: Overall impression mark Initial standardisation Task 1: • Watch the standardisation video again and assess the speakers’ performance using the mark sheet • Discuss your grades with a partner, then compare ideas with another pair Thank you! www.eaquals.org