Accountability, Testing,
and Common Exams
Tammy Howard
Jim Kroening
Jennifer Preston
October 25, 2012
Overview: 2012-13
Assessments
• EOGs and EOCs
• English Language
Arts (3-8, English II)
• Mathematics (3-8,
Algebra I/Integrated I)
• Science (5,8,
Biology)
• Delayed Scoring
(October 2013)
Accountability
• Ready Accountability
Model
• General Assembly’s
School Performance
Grades
• Delayed Reporting
(October 2013)
Common
Exams
• Administration
• Addressing
Concerns
2
READY Accountability
Model
3
Accountability
Purpose – Indicators – Key Ideas
▲
Provide incentives and supports to
• Improve Student Learning Outcomes
• Raise Graduation Rates
• Close Achievement Gaps
4
Elementary and Middle
School Accountability
Purpose – Indicators – Key Ideas
▲
Status
Indicators
Growth
Indicators
Progress
Indicators
“this year”
“combined
individual
student’s growth”
“trend over time
for groups of
students”
End of Grade
3-8 Math
End of Grade
3-8 ELA
EOG
3-8 Math
EVAAS
Growth
School-wide
Δ End of Grade
3-8 Math
Δ End of Grade
3-8 ELA
End of Grade
Δ End of Grade
5 & 8 Science
5 & 8 Science
5
High School
School Accountability
Purpose – Indicators – Key Ideas
▲
Status
Indicators
Growth
Indicators
Progress
Indicators
“this year”
“combined
individual
student’s growth”
“trend over time
for groups of
students”
End of Course
ACT
Graduation Rates
Math Course Rigor
WorkKeys
Graduation Project
Δ End of Grade
EOG
3-8 Math
3-8 Math
EVAAS
Growth
School-wide
Δ End of Course
Δ ACT
Δ Graduation Rates
Δ Math Course Rigor
Δ WorkKeys
6
How each indicator is defined
Performance Composite
(Elementary and High)
• Percent of proficient tests in a school
• All tests, subjects, and grade levels
• Uses the EOG/EOC test data
Algebra II/Integrated III
• Percent of graduates who take and pass Alg. II
or Int. Math III
• Excludes the 1% population
Graduation Rate
• Percent of students that graduate within 4
years and within 5 years
(4-year cohort graduation rate)
WorkKeys
• Percent of seniors who are CTE concentrators
who achieve a Silver certificate, or better, on
the WorkKeys assessment
ACT
• Percent of students who meet college
readiness targets
Graduation Project
• Did a school implement a graduation project ?
7
School Performance
Grades
8
Responding to School Performance
Grades (SPG)
• The SBE must respond to the General
Assembly “…annually by January 15 on
recommended adjustments to the school
performance grade elements and scales
for award of scores and grades.”
• Additionally, SECTION 7A.3.(f) indicates:
“It is the intent of the General Assembly to
add a student growth component to school
performance grades.”
• Operational in 2012-13
9
What are the basics of the SPGs?
Elementary/Middle
Schools
• Performance Composite
Total Points 0-100
100 points
Total Points 0-500
High Schools
•
•
•
•
•
Performance Composite
Algebra II/Integrated III
Graduation Rate
WorkKeys
ACT
100 points
100 points
100 points
100 points
100 points
+ Growth
10
How each indicator is defined
Performance
Composite (Elementary
and High)
• Percent of proficient tests in a school
• All tests, subjects, and grade levels
• Use the EOG/EOC test data
Algebra II/Integrated III
• Percent of 4-year cohort graduates who take and
pass Alg. II or Int. Math III
• Excludes the 1% population
Graduation Rate
• Percent of students that graduate within 4 years
(4-year cohort graduation rate)
WorkKeys
• Percent of seniors who are CTE concentrators who
achieve a Silver certificate, or better, on the
WorkKeys assessment
ACT
• Percent of Grade 11 students who meet a college
readiness standard
11
Alignment Between Indicators in High
School
High School Performance
Grades
End of Course
Math Course Rigor
Graduation Rates
WorkKeys
ACT
Graduation Project
•
•
•
•
•
Performance Composite
Algebra II/Integrated III
Graduation Rate
WorkKeys
ACT
Key Point: The set of indicators are
shared and set a college and careerready expectation. The Graduation
Project is not part of the school grade.
12
Assessment
A Closer Look at the Next
Generation of State
Assessments
13
2012-2013 School Calendar - Draft
Oct
Sept
Aug
July
June
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
2012
New State Tests (3-8 ELA and Math; 5 & 8 Science; 3 High School Exams)
New Assessments
Last
ABCs
Report
Compilation of New Data
Standard Setting
READY
Report
to SBE
ACT, PLAN, EXPLORE, WORKKEYS
ACT
PLAN
EXPLORE
WorkKeys March 5, 2013
March 19, 2013 (make-up)
March 5-19 (accommodations testing)
WorkKeys for Early Graduates in Dec
SMARTER Balanced (Development Work)
Teacher teams write
field test items
Pilot test of summative and
interim assessment items
and performance tasks
2nd Tech Readiness Tool
Collection Windows
14
A Change in the Claim of Testing
Claims in the Past:
Proficiency
Claims in the Future:
Career- and College- Ready
15
Changing what we think of as “State Assessments”
This is what
we’ve known
Summative
Interim Tools
Formative Processes
• Constructed
Response
+ • Performance Tasks
• Computer Adaptive
Testing
“Along the Way”
“Every Day”
16
Assessment
How do I get READY for
the new summative
assessments?
17
Getting Ready
Step 1: Dig into the Standards.
Where do I go to do that?
For Common Core
www.ncpublicschools.org/acre/standards/common-core-tools/
For Essential Standards
http://www.ncpublicschools.org/acre/standards/support-tools/
18
Getting Ready
Step 2: Internalize Content Specifications
Where do I go to do that?
Content Specifications
http://www.ncpublicschools.org/acre/assessment/online/
19
Getting Ready
Step 3:
Review New Item Types, Items and
Released Forms
Where do I go to do that?
Item Types
http://thismeeting.wikispaces.com/ under Summer Institute 2012
Released Forms
Coming in Fall of 2012
Tutorial of New Assessments
http://go.ncsu.edu/nctdemo
20
Getting Ready
Step 3:
New Item Types and Items
Constructed-Response and Technology-Enhanced Items
•
Mathematics: gridded response items
o
•
Calculator Inactive: Grades 3-8 and Math I (Algebra
I/Integrated I)
o
o
•
Grades 5-8 and Math I (Algebra I/Integrated I)
One-third to one-half of grades 3-8
One-third of Math I (Algebra I/Integrated I)
English II: short constructed response, technologyenhanced and multiple choice
21
Grade 5 Science TE Item
Distance Time Graph
5.P.1.2 Infer the motion of
objects in terms of how
far they travel in a certain
amount of time and the
directions in which they
travel (RBT =
Conceptual/Understand;
DOK = 2)
This distance/time graph shows the distance covered by an insect crawling across a
table. Using the information on the graph, place (click and drag) the remaining distances
to complete the table below.
Distance traveled in 10 seconds
Distance traveled in 20 seconds
Distance traveled in 25 seconds
20 cm
30 cm
40 cm
50 cm
22
Grade 8 Science TE Item
List of Species and Their Characteristics
• Estuarine species inhabit coastal areas; they are adapted to a mix of salt and fresh water
• Desert species are adapted to hot, arid areas which have large temperature fluctuations
• Very rare tree species inhabit the Amazon basin
• Reef-building coral inhabit coastal areas; their health and growth are negatively affected by
pollutants
• Salamander species in the southern U.S. depend on flowing water to keep oxygen levels high
enough for survival
If the changing conditions in the chart below occur, which species listed above will likely go extinct? Drag
and drop the species that will most likely go extinct for each changing condition.
8.L.4.1 Summarize the
use of evidence drawn
from geology, fossils,
and comparative
anatomy to form the
basis for biological
classification systems
and the theory of
evolution (RBT =
Conceptual/
Understand; DOK = 2)
Changing Conditions
Species Likely to Go Extinct
Sea levels rise rapidly
Estuarine
Prolonged drought
Deforestation in South America
Catastrophic oil spill in a coastal
region
Amazon rare tree
Desert
Salamander
Reef-building coral
23
Grade 6 Constructed Response Item
A company is having a picnic. The expenses will be for music and refreshments.
• The music will cost $150.
• The refreshments will cost $125.
• Tickets will be sold for $2.50 per employee.
What is the minimum number of tickets that must be sold to pay for the picnic
expenses?
Note that
students
receive
instructions
and
practice for
gridding
answers.
Calculator: Inactive
DOK: Skill/Concept
Domain: Expressions and
Equations
Standard: 6.EE.7 Solve realworld and mathematical
problems by writing and
solving equations of the form
x + p = q and px = q for
cases in which p, q and x are
all nonnegative rational
numbers.
24
Math I Constructed Response Item
The daily profit, P, of a business that sells x units of a product each day is given by the
function P = –2x2 + 200x + 3,000. The number of units sold on Tuesday was 10 less
than the number needed for maximum possible profit. What is the difference between
the actual profit on Tuesday and the maximum possible profit?
Enter your response here:
Only 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ., -, and / are allowed.
Calculator: Inactive
DOK: Skill/Concept
Domain: Interpreting Functions
Standard: F.IF.4 For a function that models a relationship
between two quantities, interpret key features of graphs and
tables in terms of the quantities, and sketch graphs showing
key features given a verbal description of the relationship. Key
features include: intercepts; intervals where the function is
increasing, decreasing, positive, or negative; relative
maximums and minimums; symmetries; end behavior; and
periodicity.
25
English II Multiple-Choice Item
Moonrise
by Jenette Purcell
City night sky
gives itself to me again
when I have so little left to receive it.
I am dark, crumbling
and you are rivers and trees away
searching your own night sky for a sign.
The strong gates of your heart
are wide open to me always, but,
if only.
So I wait, as seasons before, decades before,
fathers and mothers before me still inside
watch and listen.
Suddenly,
bamboo, bones, fiber, fences,
water, glistening koi,*
all the tiny rooms,
paths and places I hold your memories
relax
in audible, reverent wonder
at the fullness forming
on this horizon’s edge.
*koi:
colorful fish that symbolize love and friendship
Which line from the poem describes the
speaker’s feelings about loving someone?
“when I have so little left to
receive it”
“are wide open to me always, but”
“paths and places I hold your
memories
“at the fullness forming”
DOK: Skill/Concept
Standard: Reading for Literature (RL.1)
Key Ideas and Details: Cite strong and
thorough textual evidence to support
analysis of what the text says explicitly
as well as inferences drawn from the
text.
26
English II Constructed-Response Item
Moonrise
by Jenette Purcell
City night sky
gives itself to me again
when I have so little left to receive it.
I am dark, crumbling
and you are rivers and trees away
searching your own night sky for a sign.
The strong gates of your heart
are wide open to me always, but,
if only.
So I wait, as seasons before, decades before,
fathers and mothers before me still inside
watch and listen.
Suddenly,
bamboo, bones, fiber, fences,
water, glistening koi,*
all the tiny rooms,
paths and places I hold your memories
relax
in audible, reverent wonder
at the fullness forming
on this horizon’s edge.
*koi:
colorful fish that symbolize love and friendship
In Moonrise, explain how the theme is
developed throughout the poem. Use specific
details to support your answer.
DOK: Strategic Thinking
Standard: Reading for Literature (RL.2)
Key Ideas and Details: Determine a
theme or central idea of a text and
analyze in detail its development over the
course of the text, including how it
emerges and is shaped and refined by
specific details; provide an objective
summary of the text.
27
Scoring Rubric for ConstructedResponse Item
In Moonrise, explain how the theme is developed throughout the
poem. Use specific details to support your answer.
Score
(points)
Rubric
2
• Identifies the theme of the poem
• Uses at least one example of how the theme is revealed
in the poem
• Writes a response that explains how the theme is
developed throughout the poem
1
• Identifies the theme of the poem
• May or may not use at least one example of how the
theme is revealed in the poem
• Writes a response that may or may not explain how the
theme is developed throughout the entire poem
0
• Fails to identify the theme of the poem
• Fails to use at least one example of how the theme is
developed in the poem
• Fails to write a response that explain show the
theme is developed throughout the poem
28
English II Technology-Enhanced Item
Select (by clicking) the synonym that can replace reverent in the poem.
Excerpt from Moonrise
by Jenette Purcell
Suddenly,
bamboo, bones, fiber, fences,
water, glistening koi,*
all the tiny rooms,
paths and places I hold your memories
relax
in audible, reverent wonder
at the fullness forming
on this horizon’s edge.
respectful
redundant
DOK: Skill/Concept
Standard: Reading for Literature
(RL.4)
Craft and Structure: Determine
the meaning of words and phrases
as they are used in the text,
including figurative and
connotative meanings; analyze the
cumulative impact of specific word
choices on meaning and tone (e.g.,
how the language evokes a sense
of time and place; how it sets a
formal or informal tone).
amazed
significant
29
Common Exams
A Library of Common Exams
is being designed for non-tested
subjects for district use to populate
Standard 6
30
Focusing on the “Why”
So why have statewide Measures of Student
Learning/Common Exams?
1. North Carolina has a statewide evaluation system to ensure
that every teacher receives a fair and consistent evaluation,
regardless of his or her employing LEA
2. Teachers in all content areas should receive a Standard Six
rating based on the growth of their own students on their
content-specific standards
3. Most LEAs do not have the capacity to design their own
assessments for all non state-tested grades and subjects
31
Principles for Administration
1. Every English Language Arts, Science,
Mathematics, and Social Studies teacher in
grades 4 – 12 has a value-added score
2. Teacher growth values will be calculated
based on all students a teacher teaches and,
when multiple assessments are required, on
all data generated through the assessments
32
Decision Tree for Administration
33
Resources for Districts
• Implementation Guide
• Administration Timelines
• Assessment Specifications
• Guide to Measuring Student Growth
• Local Planning Template
34
Implementation Options - Timing
•
Administration of the high school MSLs in the fall is
optional
•
If a district chooses to administer, all MSLs must be
administered (1st semester only - Geometry & Algebra II
optional)
•
If a district chooses to administer, results will be used to
determine the sixth standard rating
35
District Flexibility
•
Administration online, paper/pencil or hybrid
•
Date of administration
•
Administration during class period or testing week
•
Use in student grade
•
Which assessments are administered
•
How to ensure secure administration
36
Addressing Concerns
Who has designed the Common Exams, and how
have they been designed?
•Same basic process as state assessments with the creation
of assessment blueprints, generation of items, review of
items, review of forms, and final production
•Over 800 teachers from across the State have been
involved in the blueprint creation and form review processes
•NCDPI psychometricians and test measurement specialists
have been involved and will analyze (and remove from
results) any poor-performing items before growth is
calculated
37
Addressing Concerns
Why doesn’t anyone know what will be on the
MSLs?
•Assessment specifications are available at:
http://www.ncpublicschools.org/educatoreffect/measures/spe
cifications/
•General information on rubrics released to C&I leaders on
October 19 (and posted to website)
•Online module will provide training on how to use rubrics to
score performance tasks
•Each item has its own specific rubric
38
Addressing Concerns
How will the performance items be scored?
•There must be at least one scorer who is not the student’s
teacher of record AND who has the content knowledge
necessary to score the item
•With the exception of ELA, performance items can be
administered early to allow time for scoring
•Scoring of work is necessary for what the 800 teachers
deemed to be authentic assessment for new, concept-based
standards
39
Addressing Concerns
MSLs hurt students and teachers
•The Common Exam administration process should not
affect students any differently than the administration of a
teacher-created final exam
•MSL scores do not need to be used as final exam
grades
•Percent correct provided by Winscan is a suggestion for
a grade
•It is only fair to base SOME part of a teacher’s evaluation
on the growth of his or her students
40
Contact Information
educatoreffectiveness@dpi.nc.gov
http://www.ncpublicschools.org/educatoreffect/
41