Transitioning to New
Assessments
Bilingual Coordinator’s Network
Transitioning to New Assessments
STAR and SBAC
September 20, 2012
Patrick Traynor, Ph.D.
Director, Assessment Development & Administration Division
CALIFORNIA DEPARTMENT OF EDUCATION
Tom Torlakson, State Superintendent of Public Instruction
California Current
Assessments
TOM TORLAKSON
State Superintendent
of Public Instruction
•
•
•
•
•
•
•
Standardized Testing and Reporting Program (STAR)
• California Standards Tests (CSTs)
• Early Assessment Program (EAP)
• California Alternate Performance Assessment (CAPA)
• California Modified Assessment (CMA)
• Standards-based Tests in Spanish (STS)
California High School Exit Exam (CAHSEE)
California English Language Development Test (CELDT)
General Educational Development Test (GED)
California High School Proficiency Exam (CHSPE)
National Assessment of Educational Progress (NAEP)
Physical Fitness Test (PFT)
Transitioning to New
Assessments
TOM TORLAKSON
State Superintendent
of Public Instruction
California Education Code Section 60604.5 requires the
State Superintendent of Public Instruction (SSPI) to:
• Develop recommendations addressing 16 areas
toward reauthorization of the assessment system
• Develop a plan for transitioning to new “high quality
assessments”
• Consult with specific stakeholders:
•
•
•
•
•
•
•
The State Board of Education
Public Schools Accountability Act (PSAA) Committee
Measurement experts from California’s private and public universities
Experts in assessing students with disabilities and English learners
Teachers, administrators, and governing boards from California’s local
educational agencies
Parents
Recommendations and Plan due to the Legislature fall 2012
3
“High-quality assessment”
TOM TORLAKSON
State Superintendent
of Public Instruction
• Means an assessment designed to
measure a pupil’s knowledge of,
understanding of, and ability to apply
critical concepts through the use of a
variety of item types and formats,
including, but not limited to, items that
allow for open-ended responses and
items that require the completion of
performance-based tasks
4
16 Areas of Consideration
TOM TORLAKSON
State Superintendent
of Public Instruction
The recommendations shall consider
including all of the following elements in
the reauthorized assessment system:
(1) Aligning assessments to standards
(2) Implementing common
assessments developed by state
collaborative
(3) Conform to ESEA reauthorization
5
16 Areas of Consideration (Cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
(4) Measurement of achievement at a
point in time and over time for
groups and subgroups of pupils and
for individual pupils.
(5) Allow for comparison from one
year to the next as a reflection of
growth over time
(6) Valid, reliable, and fair for all
students including English learners
(ELs) and students with disabilities
6
16 Areas of Consideration (Cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
(7) Assessments of ELs using primary
language assessments
(8) Ensure no bias with respect to race,
ethnicity, culture, religion, gender, or
sexual orientation
(9) Incorporate a variety of item types
including open-ended and performancebased tasks
7
16 Areas of Consideration (Cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
(10) Generating multiple measures of pupil
achievement, which, when combined with
other measures, can be used to determine
the effectiveness of instruction and the
extent of learning
(11) Assess science and history–social science
in all grade levels at or above grade 4
(12) Assess understanding and ability to use
technology necessary for success in the 21st
century classroom and workplace
8
16 Areas of Consideration (Cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
(13) Formative and interim assessments that
provide timely feedback for purposes of
continually adjusting instruction to improve
learning
(14) Use test administration and scoring
technologies that will allow the return of test
results to parents and teachers as soon as
possible
(15) Minimizing testing time
(16) Including options for diagnostic
assessments for pupils in grade 2
9
Types of Assessment
TOM TORLAKSON
State Superintendent
of Public Instruction
• Diagnostic — for learning
• Formative — for learning
• Interim — for learning and of
learning
• Summative — of learning
10
Diagnostic Assessment
TOM TORLAKSON
State Superintendent
of Public Instruction
• Ed Code 60603 defines diagnostic
assessment as:
“ . . . assessment of the current level of
achievement of a pupil that serves both of the
following purposes:
1) The identification of particular academic
standards or skills a pupil has or has not yet
achieved.
2) The identification of possible reasons that a
pupil has not yet achieved particular academic
standards or skills.”
11
Formative Assessment
TOM TORLAKSON
State Superintendent
of Public Instruction
Ed Code 60603 defines formative
assessment as:
“Assessment tools and processes that
are embedded in instruction and are
used by teachers and pupils to provide
timely feedback for the purposes of
adjusting instruction and to improve
learning.”
12
Interim Assessments
TOM TORLAKSON
State Superintendent
of Public Instruction
Ed Code 60603 defines interim
assessments as:
“ . . . an assessment that is given at regular and
specified intervals throughout the school year, is
designed to evaluate a pupil’s knowledge and skill
relative to a specific set of academic standards, and
produces results that can be aggregated by course,
grade level, school, or local education agency in order
to inform teachers and administrator at the pupil,
classroom, school, and local education agency levels.”
13
Summative Assessment
TOM TORLAKSON
State Superintendent
of Public Instruction
Ed Code 60603 defines “achievement
test” as:
“ . . . any standardized test that
measures the level of performance
that a pupil has achieved in the core
curriculum areas.”
14
Relationship of Assessments
Perie, Marion, Gong, Wurtzel, 2007
TOM TORLAKSON
State Superintendent
of Public Instruction
15
Sharing Assessment Information
TOM TORLAKSON
State Superintendent
of Public Instruction
• Summative Assessments
• Information for student, parent, teacher
• Information for school and district
• State and Federal Accountability
• Interim Assessments
• Information for student, parent, teacher
• Information for school and district
• Formative Processes and Tools
• Information for student, parent, teacher
16
Common Assessment Formats
TOM TORLAKSON
State Superintendent
of Public Instruction
• Paper and pencil testing
– Most common type of format
– Used for all assessments
• Computer based testing (CBT)
– Uses fixed form but is administered using a
computer
• Computer adaptive testing (CAT)
– Presents harder or easier questions, depending on
how the student performs as the test proceeds
– Can provide greater accuracy with fewer questions
17
Computer Adaptive Testing
TOM TORLAKSON
State Superintendent
of Public Instruction
• The system will adjust the difficulty of
items throughout the assessment
Correct or
Incorrect
Difficulty of
Next Item
Correct
More Difficult
Incorrect
Easier
18
Computer Adaptive Testing
20 Items
TOM TORLAKSON
State Superintendent
of Public Instruction
19
Six Item Types
TOM TORLAKSON
State Superintendent
of Public Instruction
•
•
•
•
•
•
Selected Response
Short Constructed Response
Extended Constructed Response
Performance Tasks
Technology-Enabled
Technology-Enhanced
20
TOM TORLAKSON
The Smarter Balanced
Assessment Consortium
(SBAC)
State Superintendent
of Public Instruction
• Area of Consideration 2 (of 16):
Implementing common assessments
developed by state collaborative
• Assessments for grades 3 - 8 and 11
aligned to the Common Core State
Standards (ELA and Math)
• Operational across Consortium states
in the 2014–15 school year
21
An Ever Changing Landscape
TOM TORLAKSON
State Superintendent
of Public Instruction
25 states
educating
approximat
ely 20
million of
the nation’s
public K to
12 students
21
governing,
4 advisory
states
22
SBAC Procurements*
TOM TORLAKSON
State Superintendent
of Public Instruction
RFP ID
RFP Name
RFP
Work
Released
Start Date
RFP-00 Project Management Partner
WestEd
06/04/2010 10/01/2010
RFP-01 Communications Services Provider
GMMB
02/25/2011 06/03/2011
RFP-02 Technology Readiness Tool
Pearson
05/20/2011 12/07/2011
RFP-03 IT Systems Architecture
Measured Progress
05/27/2011 09/15/2011
RFP-04 Item Specifications
Measured Progress/ETS 07/27/2011 12/01/2011
RFP-05 Psychometric Services
ETS
10/07/2011 02/06/2012
RFP-06 Accessibility and
Measured Progress
09/22/2011 01/04/2012
Accommodations Policy Guidelines
RFP-07 Item Authoring/Item Pool
Pacific Metrics
12/09/2011 04/03/2012
RFP-08 Item/Task Materials Development Measured Progress 10/14/2011 02/06/2012
RFP-09 Test and CAT Specifications
ETS
10/25/2011 02/06/2012
RFP-11/ Test Engine Development for Pilot
03/07/2012 08/20/2012
and Field Test
RFP-18/ and CAT Simulations
RFP-20
RFP-12 Initial Achievement Level CTB/McGraw-Hill
03/14/2012 07/13/012
Descriptors
*Status current as of 8/22/12
Contractor
SBAC Procurements (cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
RFP ID
RFP-13
RFP-14
RFP-15
RFP-16
RFP-17
RFP-19
RFP-21
RFP-22
RFP-23
RFP Name
Contractor
Translations
Item/Task Writing/Review — Pilot
CTB/McGraw-Hill
Report Development
Item/Task Development — Field Test
Scoring
Test Administration
Standard Setting
Sustainability Task Force
Formative Assessment Practices
and Professional Learning
RFP
Released
06/15/2012
12/13/2011
04/18/2012
Sept. 2012
Nov. 2012
04/13/2012
Jan. 2014
04/12/2012
Aug. 2012
Work
Start Date
07/15/2012
03/28/2012
Sept. 2012
Jan. 2013
Feb. 2013
Sept. 2012
Apr. 2014
Sept. 2012
Nov. 2012
Logical Components
TOM TORLAKSON
State Superintendent
of Public Instruction
English Language Arts and Mathematics, Grades 3–8 and High School
BEGINNING
OF YEAR
END
OF YEAR
Last 12 weeks of year*
DIGITAL CLEARINGHOUSE of formative tools, processes and exemplars; released items and tasks;
model curriculum units; educator training; professional development tools and resources; scorer
training modules; and teacher collaboration tools.
INTERIM ASSESSMENT
Computer Adaptive
Assessment and
Performance Tasks
INTERIM ASSESSMENT
Computer Adaptive
Assessment and
Performance Tasks
PERFORMANCE
TASKS
• Reading
• Writing
• Math
Scope, sequence, number, and timing of interim assessments locally
determined
Optional Interim
assessment system—
END OF YEAR
ADAPTIVE
ASSESSMENT
Re-take option
Summative assessment
for accountability
* Time windows may be adjusted based on results from the research agenda and
final implementation decisions.
26
Source: http://www.ets.org
School-level
Technology Issues
TOM TORLAKSON
State Superintendent
of Public Instruction
• Technology-enhanced test items will require
compatibility with latest browser technologies
• Test security will require device to run in “kiosk
mode”
• Devices must support text-to-speech software
• Test items with multimedia content will require
sufficient internal and Internet network bandwidth
27
Technology Readiness Tool
TOM TORLAKSON
State Superintendent
of Public Instruction
• Assesses current capacity and
compares to the technology that is
expected to be required for new SBAC
assessments
• Areas evaluated: devices, devices to
tester ratio, network infrastructure, and
staff and personnel
• Performs inventory, gap analysis and
recommended areas where LEAs will
need to upgrade technology
28
Technology Readiness Tool
Data Collection
TOM TORLAKSON
State Superintendent
of Public Instruction
• First data collection window was from
March 20, 2012 to June 30, 2012
• Subsequent data collections will occur
each spring and fall through 2014
• Input information for computers expected
to be available for online testing in 2014.
29
Technology Readiness Tool Stats (Cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
• 600,000 devices entered into system
• Operating systems inventory
– Windows XP: 52%
– Windows 7: 20%
– Mac OS X: 20%
– iOS 5.x: 3%
– Others (Vista, Linux, Windows – Other, iOS
4.x, Google Chrome, Android)
30
Technology Readiness Tool Stats (cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
• Hardware types
– Desktops: 67%
– Laptops: 19%
– Netbooks: 6%
– Tablets: 4%
– Thin Clients: 4%
31
Guidelines for Purchasing New
Hardware
TOM TORLAKSON
State Superintendent
of Public Instruction
• Inform current and future purchases
• Covering the vast majority of commercially
available computers and tablets
• Full guidelines:
http://www.smarterbalanced.org/smarterbalanced-assessments/technology
32
TOM TORLAKSON
State Superintendent
of Public Instruction
California Work with SBAC:
Opportunities for Teacher
Involvement
• Feedback on draft content and item
specifications
• Writing and reviewing of test items and tasks
(2012-13 pilot test; 2013-14 field test)
• Range-finding and score validation
• Collaborate on designing score reports and
Web tools for digital libraries (2013-14)
• Scoring of performance tasks (2014-15 and
beyond)
33
Current CDE Participation in
SBAC Work Groups
TOM TORLAKSON
State Superintendent
of Public Instruction
• SBAC Work Groups
– Assessment Design: Item Development
– Assessment Design: Performance Tasks
– Assessment Design: Test Design
– Accessibility and Accommodations
– Technology Approach
– Formative Assessment Practices and Professional
Learning
• Deputy Superintendent Deb Sigman executive
committee member
34
SBAC Resources
TOM TORLAKSON
State Superintendent
of Public Instruction
• California Department of Education/SBAC
presentations be found on the CDE SBAC
Web page at
http://www.cde.ca.gov/ta/tg/sa/smarterbalance
d.asp
• Join the SBAC CDE electronic mailing list by
sending a blank e-mail to:
subscribe-sbac@mlist.cde.ca.gov
35
Assessments to Consider (by content area and
grade level)
TOM TORLAKSON
State Superintendent
of Public Instruction
36
Transitioning to New
Assessments
TOM TORLAKSON
State Superintendent
of Public Instruction
California Education Code Section 60604.5 requires the
State Superintendent of Public Instruction (SSPI) to:
• Develop recommendations addressing 16 areas
toward reauthorization of the assessment system
• Develop a plan for transitioning to new “high quality
assessments”
• Consult with specific stakeholders:
•
•
•
•
•
•
•
The State Board of Education
Public Schools Accountability Act (PSAA) Committee
Measurement experts from California’s private and public universities
Experts in assessing students with disabilities and English learners
Teachers, administrators, and governing boards from California’s local
educational agencies
Parents
Recommendations and Plan due to the Legislature fall 2012
37
Statewide Assessment
Reauthorization Timeline
TOM TORLAKSON
State Superintendent
of Public Instruction
March 21–22:
Work Group
Meeting
May 22–23:
Work Group
Meeting
April
2012
Report writing
and review
June 12–14:
Work Group
Meeting
April 17–18:
Work Group
Meeting
March
2012
July 25–26:
Work Group
Meeting
May
2012
Regional Public Meetings:
5 meetings between
March and April
E-mail Account opened
June
2012
September 6:
Work Group
Meeting
July
2012
Launch of
Online Survey
August
2012
September
2012
State Board of
Education
Meeting
October
2012
November
2012
Online Survey
closes August
31
Focus Groups:
July September
Recommendations
to the Legislature in
fall 2012
38
Common Developing Themes
TOM TORLAKSON
State Superintendent
of Public Instruction
• Diagnostic assessments: ELA and Math in grade 2 and
grades 3-12 as needed.
• Formative Assessment Tools:
– SBAC: ELA and Math
– Explore the state providing formative tools and processes for science
and History/Social-Science
• Interim assessments:
– SBAC: ELA and Math Grades 3-8, 11
– ELA and Math for all students in grade 2, 9-10
– Science/HSS – state provided computerized interim assessments
similar to summative
• Summative:
– SBAC assessments in grades 3-8 and 11 (ELA and Math).
– Summative assessments for science and HSS in grades 3-12, for ELA
and math in grade 12, and writing in grades 3-12
39
Common Themes (Cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
• Matrix sampling in some grades/subjects, specifically
science and history–social science (HSS)
• Reduce linguistic complexity of assessments to more
accurately measure what students know and can do
• Items that evaluate critical thinking, problem solving,
communication, collaboration, creativity, and innovation
• Other measures: student engagement survey, parent
survey, graduation rates, attendance
• Multiple measures: within classroom, (e.g., tests,
quizzes, projects, essays, journaling, class work,
portfolios)
• Using results to satisfy the high school graduation
requirement
40
SBAC Timeline
Formative Processes,
Tools, and Practices
Development Begins
TOM TORLAKSON
State Superintendent
of Public Instruction
Technology
readiness tool
available
Teams of teachers
evaluate formative
assessment practices
and curriculum
resources
Writing and Review of
Pilot Items/Tasks
(including Cognitive
Labs and Small-Scale
Trials)
Content and Item
Specifications
Development
Writing and Review
Items/Tasks for Field
Testing
(throughout the
school year)
Field testing of
summative
assessment,
training school- and
district-level staff in
formative tools
Full
implementation
of assessment
system
Formative tools
available to
teachers
Pilot Testing of
Summative and
Interim
Items/Tasks
Conducted
41
Contact Information
TOM TORLAKSON
State Superintendent
of Public Instruction
• Patrick Traynor, Director
Assessment Development and Administration Division
– E-mail: ptraynor@cde.ca.gov
• Jessica Barr, Education Programs Consultant
Assessment Development and Administration Division
– Phone: 916-319-0364
– E-mail: jbarr@cde.ca.gov
• Kristen Brown, Education Programs Consultant
(SBAC Coordinator)
Assessment Development and Administration Division
– Phone: 916-319-0334
– E-mail: kbrown@cde.ca.gov
• Reauthorization E-Mail Account
– reauthorization@cde.ca.gov
42