Alternative Assessment for English Language Learners Christy Kim Boscardin Barbara Jones Shannon Madsen Claire Nishimura Jae-Eun Park CRESST Conference Los Angeles, CA - January, 2006 UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing CRESST/UCLA ‹#›/16 Issues of Content Assessment and ELLs • NCLB: Inclusion of ELLs in high-stakes tests • Are content-based assessments actually measuring students’ content knowledge or are these tests unintentionally assessing students’ language proficiency? • Content assessment confounded with language proficiency (Abedi & Leon, 1999, Bailey, 2000) • Accommodations for Standardized tests • Added challenge for Performance Assessment CRESST/UCLA ‹#›/16 Issues with Current Performance Assessment Evaluation Criteria • Language expectations/proficiencies evaluated implicitly (e.g. AP, SAT) • Lack of distinction between content and language skills • One single score - Insufficient • Language skills can compensate for lack of content knowledge CRESST/UCLA ‹#›/16 Research Questions • Are we able to differentiate the various cognitive demands (content knowledge and language skills) associated with successful completion of content assessment? • Specifically, how much of the content versus language skills contribute to the overall evaluation of student performance? • What are the demographic and instructional factors associated with higher achievement in performance assessment? CRESST/UCLA ‹#›/16 Assessment and New Scoring Framework • Biology Context • Focus on scientific text & Science explanation (National Science Education Standards) • Specialized content knowledge & language proficiency • CRESST Model-based assessment – Science Explanation Task • Explanation Dominant Genre of School-based Science Writing • Adding Language Evaluation Component: Functional Linguistic Approach • evaluate the effective use of the unique linguistic forms and structures which are specific to communicating scientific knowledge • Used in previous studies to evaluate student writing CRESST/UCLA ‹#›/16 Method: Integrated Learning Assessment • Text Passage, Reading Comprehension, Explanation Task : Genetics & Physiology • Holistic: understanding of key biology concepts, effective communication, and overall organization and structure (4-point scale) • Content: specifically understanding of the target biology content, use of supportive evidence, and inclusion of prior knowledge – organization, structure and language features not considered (4-point scale) • Language: abstraction, informational density, and technicality beyond overall structure of the essay (4-point scale) CRESST/UCLA ‹#›/16 Instructional Practice Indicator: Classroom Assignment Ratings • Based on Previous CRESST Research (Achbacher, 1999, Matsumura, 2000)- effective teachers: 1) maintain high standards for student achievement, 2) hold clear goals for student learning, and 3) align their classroom tasks with instructional goals and assessment criteria • Incorporation of classroom artifacts as data for instructional practice • Adapted to Create Measures for Opportunity to Learn Language Skills – 12 dimensions of OTL related to classroom assignments • Clarity of Content Goals: how clearly a teacher articulates the specific scientific skills and biology concepts students are to utilize and gain from completing the biology assignment • Clarity of Literacy Goals: looked for literacy goals that were clear, detailed, and specific as to what literacy skills and processes students were to be engaged in while completing the assignment. • Level of Literacy Challenge: the degree to which this assignment task provided students with the opportunity to engage meaningfully with biology text • Classroom observations – subset 16 teachers CRESST/UCLA ‹#›/16 Method: • Participants • 1,167 students (grades 8 – 11): 26% ELL • 14 high schools from 6 school districts • 26 teachers • Measures: • Outcome - Integrated Learning Assessment: 8.7% of the total variability (Holistic), 4.7% of the total variability (Content), and 5.8% of the total variability (Language) • Instructional Practice Indicator: Classroom Assignment Ratings • CST-Science (Biology), CST-English Language Arts • Teacher and Student Demographic Information (e.g. Years of Experience, ELL, Grade level, gender) CRESST/UCLA ‹#›/16 Research Question 1: Differentiation of Various Cognitive Demands Correlation: 1) Content knowledge and Language skills both Highly Correlated w/ Holistic Score 2) Language and Content slightly lower correlation 3) Slightly higher correlation with CST-Science than CST-ELA for all 3 ILA scores CST_Sc CST_ELA Holistic Content CST_Sc 1 CST_ELA 0.82 1 Holistic 0.56 0.53 1 Content 0.55 0.50 0.79 1 Language 0.60 0.57 0.77 0.70 CRESST/UCLA Language 1 ‹#›/16 Research Question 1: Differentiation Various Cognitive Demands Regression Analysis: • Predicting Holistic Score using Content and Language Scores – 84% of the variance explained by the model • Content and Language Equally attributed to the Holistic Score • Language skills are important component Content Score Language Score CRESST/UCLA B .52 .43 t 17.98 15.53 Sig .00 .00 ‹#›/16 Research Question 2: Factors Associated w/ Student Outcome Logistic Hierarchical Model (HM) Level-1 Model Prob[R = 1|B] = P'(1) = P(1) Prob[R <= 2|B] = P'(2) = P(1) + P(2) Prob[R <= 3|B] = P'(3) = P(1) + P(2) + P(3) Prob[R <= 4|B] = 1.0 log[P'(1)/(1 - P'(1)] = B0 + B1*(FEMALE) + B2*(MINORITY) + B3*(EL2) + B4*(GRADE) + B5*(CSTSC) + B6*(CSTELA) + B7*(RCG) log[P'(2)/(1 - P'(2)] = B0 + B1*(FEMALE) + B2*(MINORITY) + B3*(EL2) + B4*(GRADE) + B5*(CSTSC) + B6*(CSTELA) + B7*(RCG) + d(2) log[P'(3)/(1 - P'(3)] = B0 + B1*(FEMALE) + B2*(MINORITY) + B3*(EL2) + B4*(GRADE) + B5*(CSTSC) + B6*(CSTELA) + B7*(RCG) + d(3) Level-2 Model B0 = G00 + G01*(EXP) + G02*(CG_GOALS) CRESST/UCLA ‹#›/16 Research Question 2: Factors Associated w/ Student Outcome Teacher Experience Content – Quality of Goals Content – Cognitive Challenge Teacherlevel variables Content – Support for Cognitive Challenge Content – Quality of Evaluation Criteria Literacy – Quality of Goals Literacy – Cognitive Challenge Literacy – Support for Cognitive Challenge Literacy – Quality of Evaluation Criteria Gender Minority status Studentlevel variables EL status Grade level CST Science score CST ELA score ILA Reading Comprehension score + = significant at the .05 level - = significant at the .05 level N S – N ot statistically significant CRESST/UCLA ILA Holistic ILA Content ILA Language + NS NS + + + NS NS NS NS + + NS + NS + NS + + NS NS NS NS + NS NS + NS + + + NS NS NS NS + + NS ‹#›/16 Summary and Conclusions 1) Explicit Evaluation of Language Skills 2) Language skills significant factor in Performance assessment – Equally contributing to the overall score 3) Develop explicit rubric for language rather than implicit – instructional and assessment implications 4) Explicit Literacy instruction (integrated approach) positively associated with student outcome CRESST/UCLA ‹#›/16 Part A: Reading - Genetic Modification In this part of the test, you will read a passage on traditional cross-breeding and genetic engineering. As you read the passage, you should think about what you have learned about genetics. You may underline, take notes, and write in the margin to help you make sense of what you are reading. When you have finished, continue to part B of the booklet. Traditional Breeding vs. Genetic Engineering1 History of Cross-breeding The link between biotechnology and food dates back over 3,000 years. When biotechnology began in 1800 B.C., yeast was used to leaven bread and ferment wine. Deliberate crossbreeding, another technique of biotechnology, began only a few hundred years ago in the 1860's (by Gregor Mendel). Foods such as potatoes, corn, tomatoes, wheat, oat, and rice are all products of traditional cross-breeding. History of Genetic Engineering The most recent technique in biotechnology was developed in 1973 and is called genetic engineering. This refers to the ability to transfer genetic information between plants using molecular technology. In genetic engineering, one or more genes are removed from one organism and added to the genome of another organism. A gene holds information that will give the organism a trait. Genetic engineering is one type of genetic modification. Traditional plant cross-breeding also modifies the genetic composition of plants. Every time people cross two plants in order to improve their traits, they are genetically modifying the plants. CRESST/UCLA In recent times, crossbreeding has led to a vast increase in the variety and quality of foods available to consumers. ‹#›/16 Part D: Writing In this part of the test, you will write an essay about genetic modification. Write your essay on the following lined pages. Write as neatly as you can. If you want to change a word, cross it out and write a new word above it. When you have finished, you may close the booklet. Essay Imagine that your school is developing a new high school textbook to use next year and that it will be written by students. You have been asked to write an essay to be included in the textbook. The editor of the textbook has asked you to include information in your essay that you have learned during the year in biology class as well as information from the “Traditional Breeding vs. Genetic Engineering” passage in Part A. Specifically, your assignment is to explain how a scientist, using genetic engineering, would alter long grain white rice to make it less appetizing to insects that normally like to eat it. In your essay, include information on: A. Why inserting the new or different DNA into a cell alters the genetic composition of that cell. B. How this process of genetic modification is completed. C. At what point in time a new gene should be inserted into the genome of an organism in order for the new trait to be expressed. CRESST/UCLA ‹#›/16 Christy Kim Boscardin: christyk@ucla.edu CRESST/UCLA ‹#›/16