Assessment and
Accountability:
Embracing Transition
Collaborative Conference for
Student Achievement
March 25, 2013
Agenda
• Introduction and Increasing Expectations: Tammy Howard
• ESEA Waiver/Federal Accountability: Tammy Howard
• School Performance Grades:/State Accountability: Scott Beaudry
• Online Assessments: Jim Kroening
• SMARTER Balanced: Nadine McBride
2
Increasing Expectations:
Content Standards, Assessments,
and Performance Standards
Tammy Howard
Accountability Services Director, NCDPI
3
Changing what we think of as “State Assessments”
This is what
we’ve known
Summative
Interim Tools
Formative Processes
• Constructed
Response
+ • Performance Tasks
• Computer Adaptive
Testing
“Along the Way”
“Every Day”
Draft – March 2012. Check http://www.ncpublicschools.org/ready/resources/ for Updates to this Presentation
4
Purpose of the Test 101
5
School Accountability Tests
End-of-Grade (EOG)
When:
• English Language Arts/Reading
Grades 3–8
• EOCs/EOGs – End of
Semester/End of Year
• Mathematics Grades 3–8
• ACT – March
• Science Grades 5 and 8
• WorkKeys - February
End-of-Course (EOC)
Used For:
•
English II
•
Algebra I/Integrated I
• School accountability
(state and federal)
•
Biology
• Public reporting
WorkKeys 12th
ACT 11th
Draft – March 2012. Check http://www.ncpublicschools.org/ready/resources/ for Updates to this Presentation
6
Common Core State Standards and
NC Essential Standards
• More rigorous content standards
• New item types on new assessments
• New achievement standards aligned to
college and career readiness standards
(October 2013)
7
Draft – March 2012. Check http://www.ncpublicschools.org/ready/resources/ for Updates to this Presentation
Grade 5 Science TE Item
Distance Time Graph
5.P.1.2 Infer the motion of
objects in terms of how
far they travel in a certain
amount of time and the
directions in which they
travel (RBT =
Conceptual/Understand;
DOK = 2)
This distance/time graph shows the distance covered by an insect crawling across a
table. Using the information on the graph, place (click and drag) the remaining distances
to complete the table below.
Distance traveled in 10 seconds
Distance traveled in 20 seconds
Distance traveled in 25 seconds
40 cm
50 cm
20 cm
30 cm
Draft – March 2012. Check http://www.ncpublicschools.org/ready/resources/ for Updates to this Presentation
8
Grade 6 Constructed Response Item
A company is having a picnic. The expenses will be for music and
refreshments.
• The music will cost $150.
• The refreshments will cost $125.
• Tickets will be sold for $2.50 per employee.
What is the minimum number of tickets that must be sold to pay for the
picnic expenses?
Calculator: Inactive
Note that
DOK: Skill/Concept
students
Domain: Expressions and
receive
Equations
Standard: 6.EE.7 Solve realinstructions
world and mathematical
and
problems by writing and
practice for
solving equations of the form
gridding
x + p = q and px = q for
answers.
cases in which p, q and x are
all nonnegative rational
numbers.
9
Draft – March 2012. Check http://www.ncpublicschools.org/ready/resources/ for Updates to this Presentation
English II Constructed-Response Item
Moonrise
by Jenette Purcell
In Moonrise, explain how the theme is
developed throughout the poem. Use specific
details to support your answer.
City night sky
gives itself to me again
when I have so little left to receive it.
I am dark, crumbling
and you are rivers and trees away
searching your own night sky for a sign.
The strong gates of your heart
are wide open to me always, but,
if only.
So I wait, as seasons before, decades before,
fathers and mothers before me still inside
watch and listen.
Suddenly,
bamboo, bones, fiber, fences,
water, glistening koi,*
all the tiny rooms,
paths and places I hold your memories
relax
in audible, reverent wonder
at the fullness forming
on this horizon’s edge.
DOK: Strategic Thinking
Standard: Reading for Literature (RL.2)
Key Ideas and Details: Determine a
theme or central idea of a text and
analyze in detail its development over the
course of the text, including how it
emerges and is shaped and refined by
specific details; provide an objective
*koi: colorful fish that symbolize love and friendship
summary of the text.
Draft – March 2012. Check http://www.ncpublicschools.org/ready/resources/ for Updates to this Presentation
10
English II Technology-Enhanced Item
Select (by clicking) the synonym that can replace reverent in the poem.
Excerpt from Moonrise
by Jenette Purcell
Suddenly,
bamboo, bones, fiber, fences,
water, glistening koi,*
all the tiny rooms,
paths and places I hold your memories
relax
in audible, reverent wonder
at the fullness forming
on this horizon’s edge.
respectful
DOK: Skill/Concept
Standard: Reading for Literature
(RL.4)
Craft and Structure: Determine
the meaning of words and phrases
as they are used in the text,
including figurative and
connotative meanings; analyze the
cumulative impact of specific word
choices on meaning and tone (e.g.,
how the language evokes a sense
of time and place; how it sets a
formal or informal tone).
redundant
amazed
significant
11
Draft – March 2012. Check http://www.ncpublicschools.org/ready/resources/ for Updates to this Presentation
Why is this the right work right now?
100%
95%
The data show very encouraging progress in North Carolina…. North Carolina Graduation Rates
90%
…at the same time, there is profound room for improvement.
85%
80%
75%
70%
77.9%
70.3% 71.7%
74.2%
85,986 graduated
77.9% =
110,377 students that were eligible 65%
110,377 – 85,986 = 60%
24,391 students who could have graduated and didn’t
12
College and Career Readiness
It is important to remember that we have raised
expectations significantly in the 2012-13 school
year.
Claims in the Past:
Grade-level Proficiency
Claims in the Future:
Career- and College-Ready
13
Reading and Math Performance:
1992-93 to 2011-12
The North Carolina State Testing Results, 2011-12
Figure 1. 1992-93 to 2011-12 End-of-Grade General Test Multiple-Choice Test Results
Statewide Percent of Students At or Above Level III in Both Reading and Mathematics
Grades 3-8
100
80
60
50
40
1
30
20
10
52.9
55.5
58.1
60.0
66.3
69.1
69.9
74.7
80.8
81.3
62.5
80.9
71.7
61.7
Beginning of ABCs K-8 accountability model
Percent of Students
70
65.8
Implementation of more rigorous reading standards
90
Implementation of more rigorous mathematics standards
52.6
58.1
59.4
60.6
55.1
0
1992-93 1993-94 1994-95 1995-96 1996-97 1997-98 1998-99 1999-00 2000-01 2001-02 2002-03 2003-04 2004-05 2005-06 2006-07 2007-08 2008-09 2009-10 2010-11 2011-12
Notes:*N counts equal the number of students tested in both reading and mathematics; previous years are comparable.
Prior to 2002-03, the end-of-grade reading scale score range was 114-187. From 2002-03 to 2006-07, the end-of-grade reading scale score range was 216-290. From 2007-08 and beyond, the end-of-grade
reading scale score range is <330 to >370.
Prior to 2000-01, the end-of-grade mathematics scale score range was 98-208. From 2000-01 to 2004-05, the end-of-grade mathematics scale score range was 218-310. From 2005-06 and beyond,
the end-of-grade mathematics scale score range is <328 to >386.
The "Percent At or Above Level III in Both Reading and Mathematics" is calculated by dividing the number of students passing both reading and mathematics tests at or above Achievement Level III by
the number of students with valid scores in both reading and mathematics, therefore, the data do not include students tested only in reading or mathematics or taking alternate assessments.
End-of-grade retest data are not included in this figure. See Appendix A for student proficiency with retest scores included starting in 2008-09.
Data received from LEAs and charter schools after September 6, 2012 are not included in this figure.
Prepared by the NCDPI Division of Accountability Services/Test Development Section.
14
Standard Setting Timeline
1. Collect data from assessments: important for districts to
upload data daily (May/June)
2. Contractor constructs the developmental scale for grades
3-8 (June/July)
3. Contractor conducts standard setting with panels of
teachers (July/August)
4. Review standards, recommend to State Board of
Education (September)
5. Publish READY Accountability report (October)
ESEA Waiver:
Federal Accountability via
Annual Measureable
Objectives
Tammy Howard
Accountability Services Director, NCDPI
16
ESEA Flexibility
By demonstrating a state’s plans to address the 4 Principles….
States can receive
Flexibility Around Certain Provisions of No Child Left Behind
17
ESEA Flexibility
•
Schools not designated as having met or not met Adequate Yearly Progress
• For each school, report Annual Measureable Objectives
• Number of targets
• Number met
• Percent met
• Flexibility waiver included state accountability model: goal was one accountable model
18
ESEA Flexibility
•
Targeted subgroups’ minimum N = 30
• Previously was 40
• Pending USED Approval
• No Safe harbor
• No Targeted Assistance School (TAS) Options
• No AYP Growth Trajectory
• NO Averaging Participation Rates
19
ESEA Flexibility Participation
• The minimum participation rate for all subgroups will remain at 95%; however, consequences have changed:
• Year 1: Letter to parents with plan for improving participation
• Year 2: School labeled as “persistently low‐participating school” and develops an intensive action plan
• Year 3: non‐participating students (to meet 95 %) counted as not‐proficient
20
Principle 2
Recognition, Accountability and Support System
Annual Measurable Objectives
Reduce Not‐
Proficient by ½ in 6 years (by 2017)
Sets targets for subgroups to impact closing of gaps
21
Understanding New AMOs
Two important results of this method are
1) Acknowledges that subgroups have different starting points
2) Differentiated targets are ambitious and achievable 22
Understanding New AMOs
2010‐2011 2011‐2012 2012‐2013 2013‐2014 2014‐2015 2015‐2016 2016‐2017 Baseline
Targets
Targets
Targets
Targets
Targets
Targets
100 ‐ 57.6 means 42.4% are not proficient. Reading Reading Reading Reading Reading Reading Reading
Decrease by half in equal increments over 6‐years means
Subgroup GS HS GS HS GS HS GS HS GS HS GS HS GS
point improvement over 6 years
42.4/2 = 21.2. Total (All students)
72.8 84.2 75.1 85.5
77.3 86.8 79.6 88.2 81.9 89.5 84.1 90.8 86.4
21.2
HS
92.1
Native American 57.7 72.8 61.2 75.1 64.8 77.3 68.3 79.6 71.8 81.9 75.3 84.1 78.9 86.4
21.2/6 ≈ 3.5 point increase every year
Asian 79.1 81.0 80.8 82.6 82.6 84.2 84.3 85.8 86.1 87.3 87.8 88.9 89.6 90.5
Black 54.0 74.7 57.8 76.8 61.7 78.9 65.5 81.0 69.3 83.1 73.2 85.2 77.0 87.4
Hispanic 57.6 75.6 61.1 77.6 64.7 79.7 68.2 81.7 71.7 83.7 75.3 85.8 78.8 87.8
Multiracial 73.3 86.6 75.5 87.7 77.8 88.8 80.0 90.0 82.2 91.1 84.4 92.2 86.7 93.3
White 81.7 90.4 83.2 91.2 84.8 92.0 86.3 92.8 87.8 93.6 89.3 94.4 90.9 95.2
Economically 57.9 75.6 61.4 77.6 64.9 79.7 68.4 81.7 71.9 83.7 75.4 85.8 79.0 87.8
Disadvantaged
Limited English 37.2 32.1 42.4 37.8 47.7 43.4 52.9 49.1 58.1 54.7 63.4 60.4 68.6 66.1
proficient
Students With 39.5 45.9 44.5 50.4 49.6 54.9 54.6 59.4 59.7 63.9 64.7 68.4 69.8 73.0
Disabilities
23
Gap Closure
100.0
Reading ‐ 3 through 8
90.0
TOT
80.0
AMIN
Gap Closure over time.
70.0
60.0
ASIA
BLCK
HISP
MULT
WHTE
50.0
EDS
LEP
40.0
SWD
30.0
2011
2012
2013
2014
2015
2016
2017
24
Gap Closure
100.0
Reading ‐ 3 through 8
90.0
TOT
AMIN
80.0
ASIA
BLCK
70.0
HISP
60.0
MULT
WHTE
50.0
The furthest behind must improve at the fastest rate. 40.0
30.0
2011
2012
2013
2014
2015
2016
EDS
LEP
SWD
2017
25
Every Student READY
Implementing
School Performance
Grades
Scott Beaudry
Analysis and Reporting, NCDPI
26
School Performance Grades
Givens
1.
2.
3.
4.
5.
6.
7.
SBE Accountability System
Legislative Requirements
Meaningful Differentiation
ACT – 11th Grade
EVAAS Growth Values
Equity
Credibility
27
Recommendations
Recommendation 1:
Phase in Implementation with Full Reporting in
2013-14
Recommendation 2:
SBE Authority for Continuous Improvement of the
System
Recommendation 3:
Scale and Element Flexibility, per the law
28
Indicators
Performance Composite
(Elementary and High)
• Percent of proficient tests in a school
− All tests, subjects, and grade levels
− Uses the EOG/EOC test data
Algebra II/Integrated III
• Percent of graduates who take and pass Alg.
II or Int. Math III
− Excludes the 1% population
Graduation Rate
• Percent of students that graduate within 4
years
(4 year cohort graduation rate)
WorkKeys
• Percent of graduates who are CTE
concentrators who achieve a Silver
certificate, or better, on the WorkKeys
assessment
ACT
• Percent of college-ready benchmarks met
29
Indicators
High School
Performance Composite
Algebra II/Integrated III
{state
mean}
Graduation Rate
+ Growth
WorkKeys
{Schools that meet or
exceed growth are
raised one letter
grade}
ACT
Elementary and Middle
{state
mean}
Performance Composite
+ Growth
30
School Performance Grades
Guiding Principles for Work
1) Because we are using the state mean, raising
standards will not create undifferentiated
grades
2) Each indicator has a different mean and
scores are distributed around that mean
Example
Indicator A
46%
C
Example
Indicator B
87%
C
3) Set criteria against the state average
4) Monitor data for continuous improvement of
the accountability system
31
School Performance Grades
Guiding Principles for Work
5) Making or Exceeding EVAAS growth will
improve schools grade by one letter grade
6) Each met ACT Benchmark counts
(ex: 3 out of 5 counts more than 1 out of 5)
7) WorkKeys based on the state average % of
CTE graduates meeting silver level
32
School Performance Grades
Excellent Public Schools Act of 2013 (filed
March 19, 2013)
 Same basic structure outlined in 2012
• Indicators
• Grading scale (90,80,70,60)
• Reporting requirements
• Consequences for “F”s
33
School Performance Grades
Excellent Public Schools Act of 2013 (filed
March 19, 2013)
 Clarifies a couple of items
 ACT would require all 4 benchmarks to be
met (excluding writing test)
 Growth will be EVAAS
 Growth “shall not be used to alter or
adjust school performance scores or the
school performance grade awarded to a
school “
34
Evolving Online Delivery
Systems for Summative
Assessment
Past, Present, and Future
Jim Kroening
Test Development Lead, NCDPI
35
Brief History of Online Testing in
NC
• Spring 2005–Online Test of Computer
Skills
• Spring 2007–End-of-Course (EOC)
Online Tests (pdf)
Alg I, Alg II, Biology, Civics and
Economics, Eng I, Geometry, US
History
• 2007-08–EOC Tests of Physical
Science and Physics added (pdf)
Physics: No paper-and-pencil
version
36
New Generation NCTest v3 2012-13
• All End-of-Course (EOC) & NCEXTEND2
EOC
• EOG Science & NCEXTEND2 Science
• NCEXTEND2 ELA/Reading
• NCEXTEND2 Math
• Includes technology enhanced,
constructed response items and studentcontrolled audio
• Supports iPads meeting all technical
requirements
37
Chromebook Devices and
NCTest
• NCTest v3 (2012-13) does not
support Chromebook devices for
administering summative
assessments
• Piloting Chromebooks spring 2013
• Anticipating approval for 2013-14
meeting technical requirements
• Announcement expected late
spring
38
Number of Completed Online
Assessments Fall 2012
Assessment Name
Algebra I/Integrated I EOC
NCEXTEND2 Algebra I/Integrated I
Biology EOC
NCEXTEND2 Biology
English II EOC
NCEXTEND2 English II
Approximate Number
Completed between
12/03/12 and 02/14/13
14,571
231
26,798
668
31,789
687
• 72% of all English II EOC was administered online
• Tested over 14,000 students in one day
39
Growth in Online Administrations
Fact: When
surveyed in
October 2012,
85 LEAs were
committed to
administering at
least one EOC
online
assessment
during the fall
2012 testing
cycle.
40
Summative Online Assessment
Delivery: Current Year & Next Year
2012-13
2013-14
NCTest delivers:
NCTest plans to deliver:
 All EOC & NCEXTEND2 EOC
All EOC and NCEXTEND2 EOC
 Science EOG and NCEXTEND2 Science EOG and NCEXTEND2
Science Grades 5 & 8
Science Grades 5 & 8
 NCEXTEND2 ELA/Reading
Grades 3-8
 NCEXTEND2 Mathematics
Grades 3-8
NCEXTEND2 ELA/Reading
Grades 3-8
NCEXTEND2 Mathematics
Grades 3-8
ELA/Reading EOG Grades 3-8
Mathematics EOG Grades 3-8
41
NCTest 2013-14
The following are currently under
development:
Native iPad app
Native Android app
Locked down browser for
Windows and OS X (desktop
or laptop computers)
42
Summative Assessment Delivery
2014-15
• TestNav(Pearson) within HomeBase
Pearson is currently modifying the browser
lockdown feature in Home Base to support
tablet devices and Chromebooks and will
complete the modifications by August 2014
• Smarter Balanced Assessment Consortium
(computerized adaptive testing)
• More information forthcoming
43
2012-13 Technical Requirements for
NCTest V 3.0
• Periodically check the technical requirements for
updates at go.ncsu.edu/nct
• Also available at go.ncsu.edu/nct:
– NCTest Capacity Testing Architecture, Methodology,
and Results (ITNG)
– Designing and Building a Campus Wireless Network
(bandwidth footprint and best practices for wireless
networking)
– Functionality restrictions required for use of iPads for
online testing using NCTest
44
Online Assessments Best
Practices Guide
• Revised for 2012–13 and available at
http://www.ncpublicschools.org/acre/assessment/guide
• School case studies administering online
assessments school-wide
• Addresses issues of scheduling, financial
planning, and technical requirements and includes
first person voices from schools or districts about
the specific steps taken to plan for and administer
online assessments
45
Contact Information
Regional Accountability Coordinators (RACs)
http://www.ncpublicschools.org/docs/nceddirectory/
education.pdf
(page 42)
Hope Lung
Hope.lung@dpi.nc.gov
Jim Kroening
Jim.kroening@dpi.nc.gov
46
Smarter Balanced
Assessment Consortium
Dr. Nadine McBride
Test Development, NCDPI
47
What is Smarter Balanced?
• One of two assessment consortia for
general student population funded under
a RTTA USED grant
• State-led governance structure
– 21 states are members
– North Carolina has representation from K-12 and
Higher Education
48
System of Assessment
• Summative assessment to be given at
end of year
• Interim assessments to check student
progress throughout the year
49
System of Assessment
• Digital library of formative assessment
practices and professional learning
resources for teachers
• Secure, online reporting system
50
Smarter Balanced Summative
Assessment
• Aligned to Common Core
• Computer Adaptive
– Adapts to students level of achievement
– More precise measurement, particularly for high
or low performing students
51
Smarter Balanced Summative
Assessment
• Variety of Item Types
– Selected Response
– Technology Enhanced Items
– Constructed Response
– Performance Tasks
52
Smarter Balanced Summative
Assessment
• Given in multiple parts
– Computer Adaptive
– Performance Tasks
• Estimated time is between 3 and 4 hours
per subject
53
Smarter Balanced Interim
Assessment
• Non-secure bank of items
• Teacher and student can tap into item
bank unlimited times
• Items that are scored automatically or at
the local level
54
Smarter Balanced Digital
Library
• A library of research-based formative
assessment practices, including:
– Examples of instructional best practices
– Strategies for cross-classroom collaboration
– Professional development resources, such
as scoring rubrics for performance tasks
55
Recent Activities
• Initial Achievement Level Descriptors
Approved
• Sustainability Plan Approved
• College Content Readiness Policy under
revision
56
Recent Activities
• Pilot Test Underway!
– Over one million students signed up to test
– As of March 20, 90,977 pilot tests have been
completed
– Volunteer basis no longer available
57
Future Activities
• Practice Test
– Available starting May 29
• Field Testing
– Spring 2014
– May involve up to 50-60% of schools
58
Assessing Technology
Readiness
• 518,173 total devices entered for NC (only California and
Georgia have more)
• 45%* of the devices entered show as meeting the minimum
requirements
– Note that devices with missing or incomplete information are
automatically considered to not meet the requirements
• 43 LEAs or charter schools have not provided any device
information in the tech readiness tool
• The next data collection deadline is June 14, 2013
59
Resources
Smarter Balanced
can be found
online at:
SmarterBalanced.org
60
Item and Task Samples and Specifications
•
Sample assessment items and tasks are included in item/task
specifications: www.smarterbalanced.org/itemspecs
– ELA/literacy: http://www.smarterbalanced.org/wordpress/wp-
content/uploads/2012/05/TaskItemSpecifications/EnglishLanguageArtsLiteracy/ELAGeneralIte
mandTaskSpecifications.pdf
– Mathematics
–
•
Grades 3-5: http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2012/05/TaskItemSpecifications/Mathematics/MathematicsGeneralItemandTask
SpecificationsGrades3-5.pdf
•
Grades 6-8: http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2012/05/TaskItemSpecifications/Mathematics/MathematicsGeneralItemandTask
SpecificationsGrades6-8.pdf
•
High school: http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2012/05/TaskItemSpecifications/Mathematics/MathematicsGeneralItemandTask
SpecificationsHighSchool.pdf
Performance Tasks: http://www.smarterbalanced.org/wordpress/wp-
content/uploads/2012/05/TaskItemSpecifications/PerformanceTasks/PerformanceTasksSpecification
s.pdf
61
Item Writing Materials
• Training and samples available for
access by all teachers
– http://www.smarterbalanced.org/smarterbalanced-assessments/item-writing-andreview/
62
ALDs and CCR
• Draft Achievement Level Descriptors and
College Content Readiness Policy
– http://www.smarterbalanced.org/achievement
-level-descriptors-and-college-readiness/
63
Every Student READY
Questions