Alternative Assessment for
English Language Learners
Christy Kim Boscardin
Barbara Jones
Shannon Madsen
Claire Nishimura
Jae-Eun Park
CRESST Conference
Los Angeles, CA - January, 2006
UCLA Graduate School of Education & Information Studies
National Center for Research on Evaluation,
Standards, and Student Testing
CRESST/UCLA
‹#›/16
Issues of Content Assessment and
ELLs
• NCLB: Inclusion of ELLs in high-stakes tests
• Are content-based assessments actually
measuring students’ content knowledge or are
these tests unintentionally assessing students’
language proficiency?
• Content assessment confounded with language
proficiency (Abedi & Leon, 1999, Bailey, 2000)
• Accommodations for Standardized tests
• Added challenge for Performance Assessment
CRESST/UCLA
‹#›/16
Issues with Current Performance
Assessment Evaluation Criteria
• Language expectations/proficiencies evaluated
implicitly (e.g. AP, SAT)
• Lack of distinction between content and
language skills
• One single score - Insufficient
• Language skills can compensate for lack of
content knowledge
CRESST/UCLA
‹#›/16
Research Questions
• Are we able to differentiate the various cognitive
demands (content knowledge and language skills)
associated with successful completion of content
assessment?
• Specifically, how much of the content versus language
skills contribute to the overall evaluation of student
performance?
• What are the demographic and instructional
factors associated with higher achievement in
performance assessment?
CRESST/UCLA
‹#›/16
Assessment and New Scoring
Framework
• Biology Context
• Focus on scientific text & Science explanation (National Science
Education Standards)
• Specialized content knowledge & language proficiency
• CRESST Model-based assessment – Science
Explanation Task
• Explanation Dominant Genre of School-based Science Writing
• Adding Language Evaluation Component:
Functional Linguistic Approach
• evaluate the effective use of the unique linguistic forms and structures
which are specific to communicating scientific knowledge
• Used in previous studies to evaluate student writing
CRESST/UCLA
‹#›/16
Method: Integrated Learning
Assessment
• Text Passage, Reading Comprehension,
Explanation Task : Genetics & Physiology
• Holistic: understanding of key biology concepts,
effective communication, and overall
organization and structure (4-point scale)
• Content: specifically understanding of the target
biology content, use of supportive evidence, and
inclusion of prior knowledge – organization,
structure and language features not considered
(4-point scale)
• Language: abstraction, informational density,
and technicality beyond overall structure of the
essay (4-point scale)
CRESST/UCLA
‹#›/16
Instructional Practice Indicator:
Classroom Assignment Ratings
• Based on Previous CRESST Research (Achbacher, 1999, Matsumura,
2000)- effective teachers: 1) maintain high standards for student
achievement, 2) hold clear goals for student learning, and 3) align
their classroom tasks with instructional goals and assessment criteria
• Incorporation of classroom artifacts as data for instructional practice
• Adapted to Create Measures for Opportunity to Learn Language Skills
– 12 dimensions of OTL related to classroom assignments
• Clarity of Content Goals: how clearly a teacher articulates the specific
scientific skills and biology concepts students are to utilize and gain from
completing the biology assignment
• Clarity of Literacy Goals: looked for literacy goals that were clear,
detailed, and specific as to what literacy skills and processes students were
to be engaged in while completing the assignment.
• Level of Literacy Challenge: the degree to which this assignment task
provided students with the opportunity to engage meaningfully with
biology text
• Classroom observations – subset 16 teachers
CRESST/UCLA
‹#›/16
Method:
• Participants
• 1,167 students (grades 8 – 11): 26% ELL
• 14 high schools from 6 school districts
• 26 teachers
• Measures:
• Outcome - Integrated Learning Assessment: 8.7% of
the total variability (Holistic), 4.7% of the total variability
(Content), and 5.8% of the total variability (Language)
• Instructional Practice Indicator: Classroom
Assignment Ratings
• CST-Science (Biology), CST-English Language Arts
• Teacher and Student Demographic Information (e.g.
Years of Experience, ELL, Grade level, gender)
CRESST/UCLA
‹#›/16
Research Question 1: Differentiation of
Various Cognitive Demands
Correlation:
1) Content knowledge and Language skills both Highly
Correlated w/ Holistic Score
2) Language and Content slightly lower correlation
3) Slightly higher correlation with CST-Science than CST-ELA
for all 3 ILA scores
CST_Sc CST_ELA
Holistic Content
CST_Sc
1
CST_ELA
0.82
1
Holistic
0.56
0.53
1
Content
0.55
0.50
0.79
1
Language
0.60
0.57
0.77
0.70
CRESST/UCLA
Language
1
‹#›/16
Research Question 1: Differentiation
Various Cognitive Demands
Regression Analysis:
• Predicting Holistic Score using Content and
Language Scores – 84% of the variance
explained by the model
• Content and Language Equally attributed to the
Holistic Score
• Language skills are important component
Content Score
Language Score
CRESST/UCLA
B
.52
.43
t
17.98
15.53
Sig
.00
.00
‹#›/16
Research Question 2: Factors Associated
w/ Student Outcome
Logistic Hierarchical Model (HM)
Level-1 Model
Prob[R = 1|B] = P'(1) = P(1)
Prob[R <= 2|B] = P'(2) = P(1) + P(2)
Prob[R <= 3|B] = P'(3) = P(1) + P(2) + P(3)
Prob[R <= 4|B] = 1.0
log[P'(1)/(1 - P'(1)] = B0 + B1*(FEMALE) + B2*(MINORITY) + B3*(EL2) +
B4*(GRADE) + B5*(CSTSC) + B6*(CSTELA) + B7*(RCG)
log[P'(2)/(1 - P'(2)] = B0 + B1*(FEMALE) + B2*(MINORITY) + B3*(EL2) +
B4*(GRADE) + B5*(CSTSC) + B6*(CSTELA) + B7*(RCG) + d(2)
log[P'(3)/(1 - P'(3)] = B0 + B1*(FEMALE) + B2*(MINORITY) + B3*(EL2) +
B4*(GRADE) + B5*(CSTSC) + B6*(CSTELA) + B7*(RCG) + d(3)
Level-2 Model
B0 = G00 + G01*(EXP) + G02*(CG_GOALS)
CRESST/UCLA
‹#›/16
Research Question 2: Factors Associated
w/ Student Outcome
Teacher Experience
Content – Quality of Goals
Content – Cognitive Challenge
Teacherlevel
variables
Content – Support for Cognitive Challenge
Content – Quality of Evaluation Criteria
Literacy – Quality of Goals
Literacy – Cognitive Challenge
Literacy – Support for Cognitive Challenge
Literacy – Quality of Evaluation Criteria
Gender
Minority status
Studentlevel
variables
EL status
Grade level
CST Science score
CST ELA score
ILA Reading Comprehension score
+ = significant at the .05 level
- = significant at the .05 level
N S – N ot statistically significant
CRESST/UCLA
ILA
Holistic
ILA
Content
ILA
Language
+
NS
NS
+
+
+
NS
NS
NS
NS
+
+
NS
+
NS
+
NS
+
+
NS
NS
NS
NS
+
NS
NS
+
NS
+
+
+
NS
NS
NS
NS
+
+
NS
‹#›/16
Summary and Conclusions
1) Explicit Evaluation of Language Skills
2) Language skills significant factor in
Performance assessment – Equally
contributing to the overall score
3) Develop explicit rubric for language rather
than implicit – instructional and assessment
implications
4) Explicit Literacy instruction (integrated
approach) positively associated with student
outcome
CRESST/UCLA
‹#›/16
Part A: Reading - Genetic Modification
In this part of the test, you will read a passage on traditional cross-breeding and genetic
engineering. As you read the passage, you should think about what you have learned about
genetics. You may underline, take notes, and write in the margin to help you make sense of what
you are reading. When you have finished, continue to part B of the booklet.
Traditional Breeding vs. Genetic Engineering1
History of Cross-breeding
The link between biotechnology and food dates back over 3,000 years. When biotechnology
began in 1800 B.C., yeast was used to leaven bread and ferment wine. Deliberate crossbreeding, another technique of biotechnology, began only a few hundred years ago in the
1860's (by Gregor Mendel). Foods such as potatoes, corn, tomatoes, wheat, oat, and rice are all
products of traditional cross-breeding.
History of Genetic Engineering
The most recent technique in biotechnology was developed
in 1973 and is called genetic engineering. This refers to the
ability to transfer genetic information between plants using
molecular technology.
In genetic engineering, one or more genes are removed
from one organism and added to the genome of another
organism. A gene holds information that will give the
organism a trait. Genetic engineering is one type of genetic
modification. Traditional plant cross-breeding also modifies
the genetic composition of plants. Every time people cross
two plants in order to improve their traits, they are
genetically modifying the plants.
CRESST/UCLA
In recent times, crossbreeding has led to
a vast increase in the variety and quality
of foods available to consumers.
‹#›/16
Part D: Writing
In this part of the test, you will write an essay about genetic modification. Write your essay on the
following lined pages. Write as neatly as you can. If you want to change a word, cross it out and
write a new word above it. When you have finished, you may close the booklet.
Essay
Imagine that your school is developing a new high school textbook to use next year and that it will
be written by students. You have been asked to write an essay to be included in the textbook. The
editor of the textbook has asked you to include information in your essay that you have learned
during the year in biology class as well as information from the “Traditional Breeding vs. Genetic
Engineering” passage in Part A. Specifically, your assignment is to explain how a scientist, using
genetic engineering, would alter long grain white rice to make it less appetizing to insects that
normally like to eat it. In your essay, include information on:
A.
Why inserting the new or different DNA into a cell alters the genetic composition of that cell.
B.
How this process of genetic modification is completed.
C.
At what point in time a new gene should be inserted into the genome of an organism in order
for the new trait to be expressed.
CRESST/UCLA
‹#›/16
Christy Kim Boscardin:
christyk@ucla.edu
CRESST/UCLA
‹#›/16