Updated 4.13.2015
The Winter 2015 scoring cycle for Collection of Evidence (COE) has finished. Concerns are being raised over
limited student success with the Biology COE compared to the other content areas. Initially, we need to
acknowledge that though state-wide the results are disconcerting several districts had success with the Biology
COE. Across the state there were individual schools and districts that had pass rates in excess of 60% (some
above 85%).
So, as a system – state, district, school, and classroom – how do we evaluate and understand the current
situation?
System Design & Implementation:
First, the Biology COE allows students, who were unsuccessful with the End-of-Course (EOC) assessment, an
alternative way to demonstrate their proficiency of the learning standards. Although the EOC and the COE
each measure the 2009 Washington State K-12 Science Learning Standards, these standards are being
measured in very different ways between the two assessments. The Biology EOC, being an on-demand
assessment (point-in-time test taking), is designed mostly of multiple choice items. The COE is, for the most
part, a performance assessment consisting entirely of constructed-response items and designed to mirror
classroom work; even the on-demand aspect of the COE allows for students to complete all parts of the task
across multiple days.
Second, the COE program, by law (RCW.28A.655.065 Section 5(d)(ii)), must be of “equal or greater rigor” to the
main assessment. Development of the Biology COE employed the same methods used to develop the Biology
EOC; in both instances this methodology, reliant upon the expert judgment of state biology teachers, has
provided for consistency of expectation and alignment to the learning standards. Biology COE scenarios and
items are written then reviewed by Washington biology educators using the same item specifications,
vocabulary, match to content standard expectations, and attention to fairness issues as with the EOC. Where
short-answer item templates exist for the EOC, those same templates are used in the COE. Washington biology
educators also make up the range finding committees that determine the scoring criteria for student
responses. Professional scorers for the COE are trained using the same protocols and quality control measures
as for the EOC.
Cut-scores for the Biology COE were established in August 2014 following the same procedural model as used
in setting the cut scores for the other COE assessments; further, the process used was parallel to the model
employed for setting cut-scores for the Biology EOC aiding the intent of “equal or greater rigor”. Following
similar procedures ensures that the minimum performance required for demonstrating proficiency is
comparable across the main assessment and the Biology COE. The steps followed were:





Convening a committee of 15 educators composed of content and grade-level expert educators.
Having committee members that participated in the EOC cut-score setting (in 2012) included as
participants in the COE cut-score setting (approximately 25% of committee members).
The Performance Level Descriptors used for setting the cut scores on the EOC were the same for the
COE activity.
The committee was shown portions of the Ordered Item Booklet from the EOC standard setting as a
means to familiarize members with the difficulty of items around the proficiency cut-score for the EOC.
The standard setting process, like that for EOC, included 3 rounds of voting with opportunities for
conversation preceding each round.
1
Updated 4.13.2015
Third, when looking at the threshold of performance to achieve proficiency with the Biology COE as compared
to the other content areas (Reading, Writing, and Math), the cut-scores are consistent across all four –
approximately 75% of available points are required to pass the COE in all areas.
Work Sample Analysis
After taking into consideration these system considerations, we need to understand the nature of the biology
submissions in total and what aspects of student work are inadequate in demonstrating proficiency against the
learning standards. From the two submission cycles to date – Summer 2014 and Winter 2015 – there have
been consistent patterns in student work that have a negative impact on the resulting scores.
Issues A through F that follow are the types of issues found during scoring with accompanying examples of
actual student work revealing the differences in response that distinguish between a typical low scoring
submission and that of a typical high scoring submission:
Issue A
Students provide incomplete responses to the respective questions, if responses are provided at all (numerous
instances of blank or “I don’t know” responses).
Example: The following question is worth 1 point. Beneath the question are two sample student
responses.
Describe one change to a desert ecosystem that could allow a population of beetles to increase rapidly.
In your description, be sure to:
 Identify one change to the desert ecosystem.
 Describe how that change could cause a population of beetles to increase.
Student Response
Scoring Comments
Lots of rain
lots of rain, more plants would grow so the
beetles would have more food
Response addresses first bullet only
Response addresses both bullets in prompt
Score
Earned
0
1
Issue B
Students demonstrate misconceptions and/or lack of knowledge in a variety of science content areas.
Example: The following question is worth 1 point. Beneath the question are two sample student
responses.
Describe scientific evidence that could show a close evolutionary relationship between two organisms.
Student Response
Scoring Comments
size
Get DNA from the two different organisms
and see how similar it is.
Incomplete/vague
Correct scientific content with a complete
description.
2
Score
Earned
0
1
Updated 4.13.2015
Example: The following question is worth 1 point. Beneath the question are three sample student
responses.
The common housefly has 12 chromosomes in its body cells. A housefly produces many new cells as it
grows. Describe the process that produces new cells in the housefly.
In your description, be sure to:
 Describe the process that increases the number of cells in a housefly as the housefly grows.
 Identify the number of chromosomes in each cell at the beginning and at the end of the process.
Student Response
Meiosis makes new cells. You start out with one
cell with 12 and end up with 4 cells with three
chromosomes each.
Mitosis takes a fly cell with 12 chromosomes and
splits it into two cells with 12 chromosomes each.
Scoring Comments
Score
Earned
Incorrect process and number of chromosomes
0
Correct scientific content as well as a complete
description
1
Issue C
Students appear to be turning to outside sources to generate responses causing the work sample to be alerted
as an irregularity and potentially leading to score invalidation for plagiarism or student-to-student copying.
Example: The following question is worth 1 point. Beneath the question are two sample student
responses.
Nitrogen cycles through ecosystems. Both insects and trees have nitrogen-containing molecules in their
cells. Describe how nitrogen can become part of an insect.
Student Response
Air is about 79% nitrogen. Nitrogen is needed for
life. It is an important part of proteins, DNA, and
RNA. In plants, nitrogen is needed for
photosynthesis and growth. Nitrogen fixation is
needed to change the nitrogen in air (N2) into
forms that can be used by life. Most nitrogen
fixation is done by micro-organisms called
bacteria. These bacteria have an enzyme that
combines N2 with hydrogen gas (H2) to make
ammonia (NH3) Some of these bacteria live in the
roots of plants (mostly legumes). In these roots,
they make ammonia for the plant and the plant
gives them carbohydrates. Other plants take
nitrogen compounds out of the soil through their
roots. All nitrogen in animals comes from eating
plants.
Insects get nitrogen by eating organisms that ate
plants.
Scoring Comments
This response would be alerted by a
professional scorer as coming from an outside
source. This response is copied from the
following website :
http://simple.wikipedia.org/wiki/Nitrogen_cycle
Complete description and correct scientific
content
3
Score
Earned
0
1
Updated 4.13.2015
Issue D
Diagrams are submitted without labels included - per the protocols of the program, unlabeled diagrams are
not scored.
Example: The following question is worth 1 point. Beneath the question is a sample student response.
The common housefly has 12 chromosomes in its body cells. A housefly produces many new cells as it
grows. Describe the process that produces new cells in the housefly.
In your description, be sure to:
 Describe the process that increases the number of cells in a housefly as the housefly grows.
 Identify the number of chromosomes in each cell at the beginning and at the end of the process.
Student Response
Scoring Comments
Score
Earned
Unlabeled diagram cannot be scored.
0
Issue E
The Biology COE is used by large numbers of students that are struggling with understanding across several
content areas. Example – numerous English learners access the Biology COE and in many instances the content
is more than the students are prepared to manage.
Issue F
Many student responses indicate that students may not have had opportunities to use available resources
(sample tasks, released items, templates, scoring training videos) to increase their understanding of the types
of questions and level of performance expected on the COE as well as the EOC. Resources are organized by
item specification and hyperlinks provided in the table below.
Item Spec
Example Items, Rubrics and Scored Responses for Biology COE
Cross Spec Description
Location
SYSA (2)
SYSC(1) (2)
Feedback loops
Models/predicting
INQB(1)
INQB(2)
Contr Expt
Field study
INQC(1)
Conc from data
INQF(1)
INQF(2)
APPB(2)
Validity
Reliability
Research/explore
Moths & Trees sample task
Scoring trng, Updates 2012, Updates 2014,
Moths & Trees sample task
Templates, Updates 2012,
Scoring trng, Templates, Updates 2013,
Updates 2014
Templates, Updates 2012, Updates 2013,
Salmonberry sample task
Salmonberry sample task
Salmonberry sample task
Templates, Updates 2012, Salmonberry
sample task
4
Updated 4.13.2015
Item Spec
APPC(1)/
APPB(1)*
APPC(2)
APPC(3)
APPE(1)
LS1B(2)
LS1D(2)
LS1E(3)
LS1E(6)
LS1H(3)
LS2A(1)
LS2A(3)
LS2B(1)
LS2B(3)
LS2C(1)
LS2E(2)
Cross Spec
Description
Constraints on solutions
Location
Templates, Updates 2013, Salmonberry
sample task
Templates, Moths & Trees sample task
Templates
Scoring trng, templates, Updates 2014,
Moths & Trees sample task
Salmonberry sample task
Salmonberry sample task
Salmonberry sample task
Updates 2014
Updates 2013, Salmonberry sample task
Salmonberry sample task
Scoring trng, Moths & Trees sample task
Moths & Trees sample task
Updates 2012
Salmonberry sample task
Moths & Trees sample task
Test solutions
Redesign
Unintended Consequence
SYSB(4)
SYSB(4)
SYSB(4)
SYSB(4)
SYSB(4)
LS3A(4)
LS3B(2)
LS3E(1)
Cellular respiration
Cell transport processes
DNA--> proteins
Comp strand
Mitosis #Chrom
Carbon cycle
N cycle
Population increase
Population density
Limits to pop growth
organism
interrelationships
Inherited variability
Mutations
Evolutionary relationships
Moths & Trees sample task
Moths & Trees sample task
Moths & Trees sample task
*Template and examples for APPC(1) also apply to APPB(1) items regarding constraints on a solution.
Best Score Model
The Biology COE is scored using a “best works” scoring model. What this means is that not every piece of work
submitted by the student is used to determine their score. Instead, the highest scores from each strand are
used to determine the total score. This allows the student to submit some work that does not demonstrate
evidence of understanding the content standards without being penalized for that work. The scoring algorithm
calculates the required number of scores from each strand, choosing those that yield the highest overall score
for the collection.
For more information on how the best score model is applied refer to the link under the title of “Sample Scoring Matrix” that
can be found at: http://www.coe.k12.wa.us/Page/303.
Additional Support
OSPI is continuing to develop additional resources to assist the field with understanding the expectations of
the Collection of Evidence, not only in Biology, but across all content. In the coming months OSPI will release
various tools and program updates to aid the understanding and expectations of the COEs with the intent to
facilitate teacher and student collaboration in fulfilling graduation requirements. In the meantime, any
questions should be directed to the appropriate contact person within the COE program.
5