Using assessment For understanding and improving Language education John M. Norris University of Hawai´i at Mānoa “Ich lerne sehen” ~R. M. Rilke Dealing with change In language education Rethinking the value of language education “English language learners (ELLs) are the fastest growing student population in America. Today, one out of every nine students is learning English as a second language…By 2025, English language learners will make up one out of every four students in our classrooms.” Margaret Spellings (2005) – U.S. Secretary of Education Rethinking the value of language education “Americans need to be open to the world; we need to be able to see the world through the eyes of others if we are going to understand how to resolve the complex problems we face.” Daniel Akaka, U.S. Senator from Hawaii Rethinking the value of language education “The stakes are very high. The challenges college graduates face over the next 50, 60, and 70 years will intensify. They involve the understanding of different cultures, the balance of global power, the depletion of environmental resources, and the ability to continue to grow in a rapidly changing world. It is because the stakes are so high that I believe a focus on student learning is so critical and why FL programs have such an important role to play.” (Chase, 2006, p. 585). Geoffrey Chase – Dean of Undergraduate Studies San Diego State University Rethinking the value of language education Standards Standards for foreign language learning in the 21st century High-quality teachers NCATE – TESOL/ACTFL Teacher Development Program Standards MLA White Paper Foreign Languages and Higher Education: New Structures for a Changed World AAAL, ACTFL, MLA, Northeast Conference Modeling Representation of Foreign Language Education at the Federal Level in the United States Rethinking the value of language education How do we bridge the content vs. language divide? Are we seeking communicative competence or quality of mind? Language Education Should we adopt an instrumental, aesthetic, ethical, or moral justification for FL learning? What’s the relationship between How do we ensure L1, L2, and academic advanced language achievement? learning? Who trains teachers how? Determining the value of language education “What we assess is what we value” --Lauren Resnick “How we choose to assess will determine what gets valued” --Norris Why worry about assessment? Top 3 sources of pressure for assessment in college FL programs: 1.University administration 2.The dean 3.Accreditation process “As part of its reaccreditation, the university has required all undergraduate programs to create and implement outcomes-oriented assessment plans.” (survey respondent) “Time-consuming. Takes away from the business of teaching. Many aspects of learning can’t be measured.” (survey respondent) Why worry about assessment? NO CHILD COLLEGE STUDENT LEFT BEHIND College Leaving Exam “Does the Spellings Commission think about language education at all?” Michael Holquist (ADFL 2007 Summer Seminar West) Why worry about assessment? A S S E S S M E N T Frequent teacher reactions to assessment: “utilitarian technocracy” “thoughtless mastery” “Orwellian, punitive process” “bureaucratic nonsense” “fails to improve teaching” “duplicates existing efforts” “reductionist” “discourages teaching skills that are difficult to measure” Why worry about assessment? FLs need to develop capacity for change faculties of problem-solving BUT… Perception of assessment managerial model of efficiency not appropriate to academe Domna Stanton (2007, ADFL Summer Seminar West) Why worry about assessment? Despite such problems, college FL survey respondents desired increased use of assessments and related processes for: 1. Understanding & improving program outcomes 2. Understanding & improving program functions 3. Improving FL education on the whole 4. Understanding & Improving the worth of the program 5. Raising awareness about FL programs Why worry about assessment? And some expressed a professional ethic to engage in outcomes assessment: “We have a social and moral responsibility towards our students and towards society at large to state as clearly as we can what it is that we do for them and why what we do is valuable.” (Anonymous survey respondent) Why worry about assessment? NCATE Teacher Preparation Program Standards ACTFL TESOL •Language, linguistics •Language •Cultures, literatures •Culture •Acquisition, instruction •Planning, managing •Integration of standards •Assessment •Assessment •Professionalism •Professionalism Assessment as change agent Within this milieu, what is the role to be played by assessment? Key Challenge: Define Delimit Detract Dismiss Value of Language Education Key Opportunity: Enable Enhance Engender Empower Traditions, trends, and the status quo In language education assessment 100% 90% 80% Assessment articles 70% 60% %Assess %FL 50% %USCFL 40% %Proficiency 30% 20% 10% 0% ADFL FLA LT MLJ UP Percentage and type of FL assessment/evaluation articles in five journals, 1984-2002 35% US college FL assessment 30% 25% 20% 15% 10% 5% 0% ADFL 84-93 ADFL 94-02 FLA 84-93 FLA 94-02 LT 84-93 LT 94-02 MLJ 84-93 MLJ 94-02 UP 84-93 UP 94-02 High, low, and average yearly percentage of articles on U.S. college FL assessment/evaluation, 1984-1993 and 1994-2002 Received traditions of language assessment Swender (2002), on FL teacher professional development in assessment: “After all, if teachers do not know how to measure what students can do with language, how will they be able to determine whether their students are measuring up to the expectations of the 21st century”. Technocratic measurement problem Received traditions of language assessment WebCAPE Foreign Language Placement Exam Commercial assessment problem: One size fits most purposes & settings Received traditions of language assessment TSE – SPEAK – SLEP - TOEIC Commercial assessment problem: One size fits most purposes & settings Received traditions of language assessment When we do address assessment, in… We focus on… • FL teaching texts • How to measure the four skills • State of the art collections • How to rate oral proficiency • Professional organizations • How to test cultural knowledge • NCATE/ACTFL teaching standards • How to assign grades • Accountability requirements • How to give feedback Without much concern for… • WHY are we assessing? • What assessment methods fulfill what purposes? • How do we contextualize it? • What good is it doing? Advanced vocabulary learning: individual vocabulary notebooks Learners identify meaningful words to study Based on individual needs or interests Record 30/week in vocabulary notebook Revisit, review, and acquire 30 over the week From: Moir & Nation (2002) Advanced vocabulary learning: individual vocabulary notebooks Learners identify meaningful words to study Based on individual needs or interests Learners identify Record 30/week unsuitable words in vocabulary to study notebook Revisit, review, Cram the and acquire 30 night before over the week the weekly test Pass the recall test and promptly forget the words Received traditions of language assessment The ‘assessment mindset’ in teacher practice: “Although this student still has problems with grammar, the ideas are there. He is working through the choices the community has about their need for a better water supply system. Hmm, this is difficult. I just wish his grammar errors weren’t so bad then I could give him an ‘A’.” Mohan example, TBLT 2007 Conference Received traditions of language assessment What is the appropriate proficiency level to adopt as a student learning outcome for the 2-year language requirement? INTERMEDIATE - LOW ? ? ? Current trends in assessment Student Learning Outcomes Assessment Outcomes embody the essential purpose of an educational program: developments in knowledge, skills, dispositions of learners Requires rethinking of educational programs as something more than the delivery of experiences or the exposure of learners to information Calls for articulation of curriculum and instruction in support of targeted outcomes, demands integrated thinking Provides a clear statement of educational program value; answers the question “How do you know?” with evidence of educational effectiveness Current trends in assessment Student Learning Outcomes Assessment Process •State outcomes •Identify indicators •Measure them •Analyze the results •Then what? •(Let the chair/dean do it) Current trends in assessment Large public institution Accreditation pressures to assess learning We have to assess our “liberal studies” core, ASAP! Huge expenditure, $$$, time, effort How about an electronic portfolio? Thousands of student portfolios created BUT… Current trends in assessment Electronic Portfolio …faculty didn’t understand it …students thought: 1. *!%#$!@* 2. waste of time …administrators wanted to do something with it but weren’t sure exactly what NEVER GOT USED Current trends in assessment Barrington (2003), on assessment in the liberal arts: “To design and administer (intellectually honest) assessment plans that will measure such capabilities with a dozen or more standardized ‘learning objectives’ is next to impossible” leading to “pestilent repercussions” for the truly valued learning objectives that constitute the liberal arts, in that it “discourages teaching such skills because they are difficult to measure”. Perception problem Traditions, trends, and the status quo Traditions and trends Status quo 1. Focus on doing… 1. Not useful—not used! 2. Based on commercial testing, proficiency mvt. 2. Potential negative washback, reductionism, waste 3. Reactive v. proactive praxis 3. Not relevant to curriculum & instruction, program values 4. Driven by external impetuses 4. Not ours—mandated or purchased off the shelf 5. Technocratic measurement emphasis 5. Not perceived as worth the effort by faculty 6. Little scholarly investment 6. Minimal professional development Re-envisioning assessment in language education Resolving terminological confusion Measurement is the consistent elicitation of quantifiable indicators of well-defined constructs via tests or related observation procedures; it emphasizes efficiency, objectivity, and technical aspects of construct validity. Norris (2006) MLJ Perspectives Resolving terminological confusion Assessment is the systematic gathering of information about student learning in support of teaching and learning…It may be direct or indirect, objective or subjective, formal or informal, standardized or idiosyncratic…It provides locally useful information on learners and learning to those individuals responsible for doing something about it. Norris (2006) MLJ Perspectives Resolving terminological confusion Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology. Norris (2006) MLJ Perspectives The nature of useful evaluations Evaluation gets used when… (a) intended users of evaluation participate; (b) evaluation is pursued as a process, not an end-game; (c) sufficient time and resources are allocated; (d) evaluation produces interesting, credible, relevant findings; (e) findings are reported in a timely fashion; (f) Interpretations and recommendations are contextualized. Evaluative assessment Light (2001) on outcomes assessment: “…a process of evaluating and improving current programs, encouraging innovations, and then evaluating each innovation’s effectiveness. The key step is systematic gathering of information for sustained improvement. And always with an eye toward helping faculty or students work more effectively.” Richard Light (2001, p. 224) An evaluative approach to assessment Accountability Raising Awareness Motivating Learners PURPOSES Program Development Improving teaching Justifying $ requests Illumination Revising curriculum Improving learning Diagnosis Certification Articulation An evaluative approach to assessment Surveys Tests Interviews Focus groups Peer Assessment Performance assessment Journals METHODS Portfolios Language Profiles Meetings Quizzes Observations Standardized assessment Self assessment An evaluative approach to assessment Approaching assessment: •The most important consideration is supposed to be the use for which it is intended (cf. Bachman & Palmer, 1996; Brown, 2005) •BUT, in test development, what is the first question that gets asked? What are you going to measure? “Decide on purpose of assessment: •What abilities are you assessing? --What is your construct or model of these abilities?” Coombe, Folse, Hubley (2007, p. 4)) An evaluative approach to assessment WHO? Test Users WHAT? Test Information INTENDED ASSESSMENT USE IMPACT? Test Consequences WHY? Test Purposes An evaluative approach to assessment Primary Intended Users Negotiate & specify: Assessment Advisors • priority uses • methods Stakeholders & Audiences • analyses •reporting • timelines ENABLING USE EMPOWERING USERS An evaluative approach to assessment Products: (a)public documents on the exact roles to be played by assessments in the FL program and the different forms that those assessments take (b) program policies on assessment practice at the individual, classroom, and program levels (c) assessment methods that lead to local actions (d) evaluative/evidentiary justification for assessments and their uses (e) Identification of other factors in need of evaluation Why bother? Align assessment with curriculum, instruction, program values: foster ownership Raise awareness & buy-in among students, faculty, others about roles for assessment and Increase the likelihood that evaluation data will be used Decrease frequency and number of (useless) assessments and evaluations Support improvements in student learning Why bother? Received view of assessment: What do we test? How do we test? What tests are available? Evaluative Vision: What questions do we have about our learners, courses, and programs? How do we gather data appropriate to answering those questions? In what ways can we utilize those data to resolve the challenges we face? What does it look like? Using assessment to understand and improve language programs Example: Needs assessment for curricular revision Context Problem Role of assessment • Japanese at University of Hawaii • Perception of curricular • Identify learners’ ‘antiquation’ perceived L2 needs • 2-yr. requirement (mostly non-majors) • Relevance to learners’ wants and needs? • Identify teachers’ perceptions of same • @1000 students per year • Shifting standards for college FL learning • Identify gaps between the two • @50 instructors • New expectation to pursue performance outcomes • Prioritize outcomes expectations for 2-yr. program • How to evolve? What values to prioritize? • Provide basis for revision of materials • Diversity of students, diverse uses of Japanese Example: Needs assessment for curricular revision Instruments and Procedures Questionnaires Teachers (N = 46) L2 Use Domains Academic Students (N = 688) Social Occupational Likert scale ratings of priorities Tourism + Open-ended questions Example: Needs assessment for curricular revision Identified Learner Needs Teachers + Learners Prioritize Learners Prioritize, Teachers do not In Hawaii… In Japan… In Japan… •Academic routines •Academic routines •Socializing •Socializing •Medical interactions •Hosting guests •Surviving as a tourist •Working retail •Interacting w/tourists •Using computers •Participating in complex discussions •Residing longterm (housing, etc.) •Working abroad •Formal ceremonies Example: Needs assessment for curricular revision Other Curricular Implications • Instruction moves too rapidly • Overemphasis on Kanji memorization • Not enough attention to culture • Not enough focus on situationspecific speaking tasks, performance practice Example: Needs assessment for curricular revision Using Assessment for Curricular Change Related Changes • Basis for review of texts, materials & pedagogy development • Dean commits to Departmentlevel Assessment Specialist • Revised 2-yr expectations emphasize speaking, culture, performance • Evaluation utilized as campuswide example of ‘closing the loop’ back to curriculum • Attention to Japanese use in Hawaii & long-term residence abroad (both authentic to learners) • Development of performance-based outcomes assessments • Revised balance/pace of instruction • Increased enrollments, improved course evals • Principal Investigator tenured… See Iwai et al (1999) Example: Realizing the curriculum via assessment Context • German at Georgetown Univ. • New curriculum: Developing Multiple Literacies • Fully integrated Language & Content instruction • Task- & genre-based Problem • New curriculum, new instruction, old assessment • Accuracy of curricular learning trajectories? • Students meeting expected outcomes? • Feasible expectations for intensive and non? • Advanced L2 literacy • Relevance of external target assessments? Role of assessment • Operationalize curricular expectations • Provide evidence regarding learner development & outcomes • Enable between-level, and intensive/non comparisons • Fit seamlessly into teaching and learning processes!!! Example: Realizing the curriculum via assessment Prototypical performance writing tasks T E A C H E R S Level performance profiles Deliberation Development Task assignment sheets Analysis Revision Sample student performances L E V E L G R O U P S Example: Realizing the curriculum via assessment Consistent assignment framework Task Curricular level expectations Content Language Assignment 1 Assignment 2 Semester… Assignment 3 Assignment 4 Explicit performance criteria Prototypical Performance Writing Task Example: Realizing the curriculum via assessment “Assessment in this kind of a context is, I would almost say probably an indispensable aspect in order to clarify any number of things. Because it is in the discourse about assessment and how we would do that that our knowledge became articulated or the holes in that knowledge became clearer to ourselves. Or the cover-ups that we had engaged in were no longer possible if we wanted to be honest with ourselves about it.” Example: Realizing the curriculum via assessment Using Assessment for Curricular Change • Forced the curriculum to become real • Close specification of L2 progress within/across curricular levels • Disambiguation of learning outcomes in terms of task, content, language • Curricular ‘map’ for use by teachers and learners (what happens when?) • Forged agreement between levels on what can and cannot be expected See Byrnes (2002) Example: Assessing and improving effectiveness Context Problem • LAUSD • Does it help? • Kindergarten/First, ELLs with varying proficiencies (41%) • Does it get used? • Supplemental software introduced to enhance reading instruction • Are conditions appropriate for benefiting from it? • Do students and teachers understand • Wide-spread reading how it is supposed to be ‘failure’ used? • Should it be maintained? Role of assessment • Evidence of reading development • Enable comparisons between ages, proficiencies, treatment/control • Capture the context of implementation • Provide thorough interpretation of processes + outcomes Example: Assessing and improving effectiveness Instruments and Procedures 200 classrooms (100 treatment, 100 control) Enable estimation of added effect of new software Standardized, mandated reading assessments Indicate English reading abilities among children Extensive classroom observations Teacher interviews Provide for meaningful interpretation of findings in terms of classroom realities Example: Assessing and improving effectiveness Findings •No differences in reading ability between treatment and control classes or proficiency groupings •BUT, Low implementation in classes and by individuals •Lots of reasons: Lack of time, competition with standard curriculum, computer malfunction, lack of understanding by learners and teachers, lack of ‘on-task’ engagement Uses •Provide for professional development in software •Articulate/align with standard curriculum •Individualize for students with greatest need •Shift school schedule to allow for implementation to occur •Retain the software but continue assessing effectiveness with students See Llosa & Slayton (forthcoming) Facing change: Learning to see assessment as a useful process •There is more to useful assessment than just good measurement; measurement informs construct interpretations…assessment enables informed change •Contextualization in classes, curricula, institutions is essential for assessment to be interpretable and meaningful •Intentionality for specific purposes, by specific users, with specific consequences is essential for assessment to be used •These processes should be built in from the beginning of assessment design; specification of intended use is one heuristic for doing so •Educational assessment is only useful insofar as it does good; hold assessments accountable to that principle A few resources… University of Hawai‘i at Mānoa, Second Language Studies http://www.nflrc.hawaii.edu/evaluation Thank you!