TOM TORLAKSON
State Superintendent
of Public Instruction
Transitioning to New Assessments
Common Core Assessment Workshop
Los Angeles County Office of Education
March 22, 2012
Deborah V.H. Sigman
Deputy Superintendent of Public Instruction
District, School & Innovation Branch
California Department of Education
1
Smarter Balanced Assessment
Consortium (SBAC)
TOM TORLAKSON
State Superintendent
of Public Instruction
An Ever Changing Landscape
• 27 states
representing
43% of K-12
students
• 21 governing,
6 advisory
states
2
Opportunities and Challenges of
Transitioning to New Assessment
System
TOM TORLAKSON
State Superintendent
of Public Instruction
•
•
•
•
•
•
•
Richer assessment of and for learning
Use of technology as a tool
Adaptive testing
Universal access
National expertise
Preparation for 21st century skills
College and career ready students
3
Opportunities and Challenges of
Transitioning to New Assessment
System (cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
•
•
•
•
•
Change is difficult
Sequencing of activities
Technology infrastructure
Communication efforts
Fiscal climate
4
SBAC Balanced System
TOM TORLAKSON
State Superintendent
of Public Instruction
5
Timeline
Formative Processes, Tools,
and Practices Development
Begins
TOM TORLAKSON
State Superintendent
of Public Instruction
Summative Master
Work Plan Developed
and Work Groups
Launched
Procurement
Plan
Developed
Writing and Review of
Pilot Items/Tasks
(including Cognitive
Labs and Small-Scale
Trials)
Content and
Item
Specifications
Development
Writing and Review
of Field Test Items/Tasks
(throughout the
school year)
Pilot Testing of
Summative and
Interim Items/Tasks
Conducted
Common Core State
Standards Adopted by All
Member States
Field Testing of
Summative and
Interim Items/Tasks
Conducted
Preliminary Achievement
Standards (Summative)
Proposed and Other Policy
Definitions Adopted
Final Achievement
Standards (Summative)
Verified and Adopted
Operational
Summative
Assessment
Administered
6
Summative
Assessments Today
TOM TORLAKSON
State Superintendent
of Public Instruction
Each state procures its
own assessment system
• Each state bears the burden of test
development; no economies of scale
Measure proficiency against
state standards, not agreedupon standards
• Students often leave high school
unprepared to succeed in entry-level
college courses
Usually heavy reliance on
multiple choice questions
• Inadequate measures of demonstration
of skills and complex cognitive
performance
Results often delivered
months after tests are given
• Tests cannot be used to immediately
inform instruction or affect program
decisions
Accommodations for special
education and ELL students
vary
• Difficult to interpret meaning of scores;
concerns about access and fairness
Most administered on paper
• Costly, time consuming, and
challenging to maintain security
7
TOM TORLAKSON
Using Computer Adaptive
Technology for Summative
and Interim Assessments
State Superintendent
of Public Instruction
Faster results
• Turnaround in weeks compared to months
today
Shorter test length
• Fewer questions compared to fixed form tests
Increased precision
• Provides accurate measurements of student
growth over time
Tailored to
student ability
• Item difficulty based on student responses
Greater security
• Larger item banks mean that not all students
receive the same questions
Mature technology
• GMAT, GRE, COMPASS (ACT), Measures of
Academic Progress (MAP)
8
Assessment System
Components
TOM TORLAKSON
State Superintendent
of Public Instruction
Summative Assessment (Computer Adaptive)
•
•
•
Assesses the full range of Common Core in English
language arts and mathematics for students in grades 3–8
and 11 (interim assessments can be used in grades 9 and 10)
Measures current student achievement and growth
across time, showing progress toward college and career
readiness, allowing for growth models
Includes a variety of question types: selected response,
short constructed response, extended constructed response,
technology enhanced, and performance tasks
9
Assessment System
Components
Interim Assessment (Computer Adaptive)
TOM TORLAKSON
State Superintendent
of Public Instruction
• Optional comprehensive and content-cluster assessment to
•
•
•
•
•
help identify specific needs of each student
Can be administered throughout the year
Provides clear examples of expected performance on
Common Core standards
Includes a variety of question types: selected response,
short constructed response, extended constructed
response, technology enhanced, and performance tasks
Aligned to and reported on the same scale as the
summative assessments
Fully accessible for instruction and professional
development
10
Assessment System
Components
TOM TORLAKSON
Performance Tasks
State Superintendent
of Public Instruction
•
•
•
•
•
•
Extended projects demonstrate
real-world writing and analytical
skills
May include online research,
group projects, presentations
Require 1–2 class periods to
complete
Included in both interim and
summative assessments
Applicable in all grades being
assessed
Evaluated by teachers using
consistent scoring rubrics
“
The use of performance
measures has been found
to increase the
intellectual challenge in
classrooms and to
support higher-quality
teaching.
”
- Linda DarlingHammond and Frank
Adamson, Stanford
University
11
Assessment System
Components
TOM TORLAKSON
State Superintendent
of Public Instruction
Formative Assessment Practices
• Research-based, on-demand
•
•
tools and resources for
teachers
Aligned to Common Core,
focused on increasing student
learning and enabling
differentiation of instruction
Professional development
materials include model units of
instruction and publicly released
assessment items, formative
strategies
“
Few initiatives are
backed by evidence that
they raise achievement.
Formative assessment
is one of the few
approaches proven to
”
make a difference.
- Stephanie Hirsh,
Learning
Forward
12
Assessment System
Components
TOM TORLAKSON
Online Reporting
State Superintendent
of Public Instruction
•
•
•
•
•
Static and dynamic reports,
secure and public views
Individual states retain
jurisdiction over access and
appearance of online reports
Dashboard gives parents, students,
practitioners, and policymakers
access to assessment
information
Graphical display of learning
progression status (interim
assessment)
Feedback and evaluation
mechanism provides surveys, open
feedback, and vetting of materials
“
Data are only useful if
people are able to access,
understand and use
them… For information to
be useful, it must be
timely, readily available,
and easy to understand.
–
”
Data Quality Campaign
13
Support for
Special Populations
TOM TORLAKSON
State Superintendent
of Public Instruction
•
•
•
Accurate measures of
progress for students
with disabilities and
English Language
Learners
Accessibility and
Accommodations Work
Group engaged
throughout
development
Outreach and
collaboration with
relevant associations
CommonCore Tests
to Have Built-in
Accommodations
- June 8, 2011
14
How the System Fits
Together
TOM TORLAKSON
State Superintendent
of Public Instruction
ACCESSIBILITY AND ADMINISTRATION
TECHNOLOGY
ITEM AND TEST DESIGN
FORMATIVE PRACTICES, PROFESSIONAL LEARNING AND IMPLEMENTATION
15
Administration and Accessibility
TOM TORLAKSON
Enhance the Vision
State Superintendent
of Public Instruction
•
•
Support the Technology, Item Development and
Test Design work groups as they incorporate the
principles of accessibility and universal design
into the design of the Smarter Balanced system
Identify the variables, attributes and components of
tests that need to be dynamic to address the full
range of student needs
16
Accessibility and
Administration
Build Consensus and Systems
TOM TORLAKSON
State Superintendent
of Public Instruction
•
•
•
•
Facilitate consensus among member states regarding
common definitions of ELL and SWD, and common
accommodations for ELL and SWD
Draft manuals and materials to support state
implementation of pilot and field test as well as
operational test
Materials will be used as part of an iterative design, build
and revise approach to technology called agile
development
They will also be used to support the development of
professional learning modules and other formative tools
17
Technology
Identify Technology Needs
TOM TORLAKSON
State Superintendent
of Public Instruction
• Technology readiness
application available for
states, districts and schools to enter data regarding
hardware, software, bandwidth, staffing, electrical
systems and other infrastructure required for online
testing
– Data will be compared against minimum and
recommended requirements
– Application will support progress tracking
– Data useful for state and national policymakers
considering total cost of ownership of a highquality assessment system
18
Update – California Technology
Readiness Tool Rollout – Spring 2012
TOM TORLAKSON
State Superintendent
of Public Instruction
•
Technology Readiness Data Collection Schedule –
Spring 2012
– March 20 – April 9 ~ Pilot Districts Test Data Collection:
– April 9 – 16 ~ Enhance Communications for the
Technology Readiness Tool Survey:
– April 16 – June 14 ~ Begin Statewide LEAs Data
Collection
•
Technology Readiness Reporting – Spring 2012 the first
data collection window will be a technology readiness
inventory. Data collection summaries will be available April
9th. These reports will help improve data quality and to inform
the consortia regarding which types of hardware and
operating systems are currently being used in schools.
19
Technology
Build the System
TOM TORLAKSON
State Superintendent
of Public Instruction
• Provide the system based on the system
architecture
• Applications will include (subject to architecture):
– Item bank (based on Michigan Item Bank)
– Test delivery
– Reporting / hub
– Digital library with formative assessment
practices resources, curriculum resources and
interactive collaboration for Smarter Balanced
users
20
Item and Test Design
Write the Items
TOM TORLAKSON
State Superintendent
of Public Instruction
• Item and test specifications will be used to drive item
writing
– Item specifications: ensure items are accessible and
in the right form and format
– Test specifications: ensure the right number of items
will be written so the pool is sufficient
• Item writing led by vendors, states and Smarter Balanced
• Balance of item-writing burden will likely change from
short-term to the long-term
– Item writing in short-term needs to be aggressive to
build the initial pool; time and volume will be a driving
factor
– Long-term, other priorities can take precedence
21
Draft Assessment Claims for English
Language Arts/Literacy
TOM TORLAKSON
State Superintendent
of Public Instruction
22
Assessment Claims for English
Language Arts/Literacy Cont.’d
TOM TORLAKSON
State Superintendent
of Public Instruction
23
Draft Assessment Claims for
Mathematics
TOM TORLAKSON
State Superintendent
of Public Instruction
24
Draft Assessment Claims for
Mathematics (cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
25
Sample Items: Technology
Enhanced and Constructed Response
TOM TORLAKSON
State Superintendent
of Public Instruction
26
Test Question
Types & Designs
TOM TORLAKSON
State Superintendent
of Public Instruction
27
Sample Items (cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
28
Sample Items (cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
• Item a – Geometry – 7.4 – Know the formulas
for the area and circumference of a circle and
use them to solve problems; give an informal
derivation of the relationship between the
circumference and area of a circle.
• Item b – Ratios and Proportional
Relationships - 7.3 -Use proportional
relationships to solve multistep ratio and
percent problems.
29
Sample Performance Task
TOM TORLAKSON
State Superintendent
of Public Instruction
• Phase 1/Day 1: Directions: Write an imaginary or
real story telling about someone losing a tooth. Be
sure to have a beginning, middle, and ending.
• Phase 2/Day 2: Students will hear selected readaloud text. Directions: Using your notes, write two
to four paragraphs describing and comparing
different tooth traditions. Be sure to
– (1) tell some things that are the same about the
traditions,
– (2) tell some things that are different about the
traditions, and
– (3) explain something interesting that you learned
OR tell about your tooth tradition.
30
Sample Performance Task
(cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
•
Standards Assessed with this Task
•
Writing Standards:
–
–
–
–
•
Speaking and Listening Standards:
–
–
•
W.3.2. Write informative/explanatory pieces (a-d)
W.3.3. Write narratives (a-d)
W.3.5. With guidance and support from peers and adults, strengthen writing
as needed by revising and editing. (formative evidence)
W.3.6. With guidance and support from adults, use technology to produce
and publish writing.
SL.3.2. Identify the main ideas and supporting details of information
presented graphically, visually, orally, or multimodality.
SL.3.4. Report on a topic or recount stories or experiences with appropriate
facts and descriptive details.
Language Standards:
–
–
–
L.3.1. Observe conventions of grammar and usage.
L.3.2. Observe conventions of capitalization, punctuation, and spelling.
L.3.3. Make effective language choices. a. Use words for effect.
31
Formative Practices, Professional
Learning and Implementation
TOM TORLAKSON
State Superintendent
of Public Instruction
Understand the CCSS
• Teams of teachers from each state will:
– Participate in identifying formative assessment practices and
curriculum resources to put in Digital Library
– Participate on a committee to complete voluntary alignment
review of publishers’ materials to the content specifications
and develop a “Consumers Report” to upload to the Digital
Library
• National content experts to develop 54 (3 ELA and 3 math
per grade) formative assessment practices exemplar
modules that provide model products for Smarter Balanced
teachers (housed in Digital Library)
• Existing CCSS curriculum projects are adapted to align with
the Smarter Balanced content specifications (and uploaded
to the Digital Library)
32
Formative Practices, Professional
Learning and Implementation
TOM TORLAKSON
State Superintendent
of Public Instruction
Understand the Test
• Produce high-quality test manuals that include
administration guidelines and supports for teachers and
students
• Support administration of test consistent with its purpose
and intended use of data
• Offer trainings on how to administer the test, provide
accommodations, use reporting system and other
applications
• Enhance assessment literacy by providing well
articulated training on interpreting assessment results
• Support connections with pre-service teachers
33
Formative Practices, Professional
Learning and Implementation
TOM TORLAKSON
Improve Teaching and Learning
State Superintendent
of Public Instruction
•
•
•
•
Provide comprehensive support for formative
assessment, including instructional modules aligned
with CCSS
Training modules help teachers focus their instruction
on the CCSS and develop teaching practices that
support more in- depth learning
Enhance assessment literacy by training teachers to
use formative assessment tools and interim
assessment to determine next steps in instruction
Provide supports for students to manage their own
learning
34
Timeline
Formative Processes, Tools,
and Practices Development
Begins
TOM TORLAKSON
State Superintendent
of Public Instruction
Summative Master
Work Plan Developed
and Work Groups
Launched
Procurement
Plan
Developed
Writing and Review of
Pilot Items/Tasks
(including Cognitive
Labs and Small-Scale
Trials)
Content and
Item
Specifications
Development
Writing and Review
of Field Test Items/Tasks
(throughout the
school year)
Pilot Testing of
Summative and
Interim Items/Tasks
Conducted
Common Core State
Standards Adopted by All
Member States
Field Testing of
Summative and
Interim Items/Tasks
Conducted
Preliminary Achievement
Standards (Summative)
Proposed and Other Policy
Definitions Adopted
Final Achievement
Standards (Summative)
Verified and Adopted
Operational
Summative
Assessment
Administered
35
Progress to Date
Master Work Plan for
Summative Assessment
• Major tasks / scope of work
• Schedule and description of procurements
TOM TORLAKSON
State Superintendent
of Public Instruction
Content Specifications for
ELA/Literacy & Math
• Two rounds of public comment
• Review by Governing States
Item & Test Development
• RFPs issued for: item/task specifications,
item/task materials development; test and CAT
specifications; pilot test item/task development
IT Systems Architecture
• Phase 1 development complete
• Architecture review board established
Communications
Staffing
• Delivered presentations to more than 130
groups and organizations
• COO; Lead psychometrician; Director of higher
education collaboration; Director of strategic
communications
36
Next Six Months
Sustainability Task Force
TOM TORLAKSON
• Recommendations for procurement,
administration, and maintenance post 2015
State Superintendent
of Public Instruction
Item/Task Development
Master Work Plans for
Interim and Formative
Definition of College/
Career Readiness
Access and
Accommodations
Staffing
• Item/task specifications issued
• Rapid prototyping & small scale trials for pilot test
• Major tasks / scope of work
• Schedule and description of procurements
• Higher education collaboration; Research-based
alignment of CCSS and credit-bearing courses
• Common accessibility guidelines issued
• Convene advisory groups for ELL and SWD
• Math and ELA/literacy content specialists
37
Transitioning to New
Assessments - STAR
Reauthorization
TOM TORLAKSON
State Superintendent
of Public Instruction
• Assembly Bill (AB) 250 modifies
California Education Code (EC) to
address the development and
adoption of new curriculum
frameworks, instructional materials,
professional development practices,
and high-quality assessments.
38
Statutory Authorization
TOM TORLAKSON
State Superintendent
of Public Instruction
– Requires that the State
Superintendent of Public
Instruction (SSPI) develop
recommendations for the
reauthorization of the statewide
pupil assessment system, which
includes a plan for transitioning
to a system of “high-quality”
assessments as defined in EC
Section 60603.
39
High-Quality Assessments
TOM TORLAKSON
State Superintendent
of Public Instruction
• Means an assessment designed to
measure a pupil’s knowledge of,
understanding of, and ability to
apply critical concepts through the
use of a variety of item types and
formats, including, but not limited
to, items that allow for open-ended
responses and items that require
the completion of performancebased tasks.
40
High-Quality Assessments
TOM TORLAKSON
State Superintendent
of Public Instruction
• A high-quality assessment should have
the following characteristics:
– Enable measurement of pupil achievement
and pupil growth
– Be of high technical quality by being valid,
reliable, fair, and aligned to standards
– Incorporate technology where appropriate
– Include the assessment of pupils with
disabilities and English learners
– Use, to the extent feasible, universal design
principles, as defined in Section 3 of the
federal Assistive Technology Act of 1998
41
Requirements of Bill
– Requires that the SSPI consult with
TOM TORLAKSON
State Superintendent
of Public Instruction
• The State Board of Education
• Public Schools Accountability Act (PSSA)
Committee
• Measurement experts from California
private and public universities
• Individuals with expertise working with
students with disabilities and English
learners
• Teachers, administrators, and governing
boards from California’s local educational
agencies.
• Parents
42
Requirements of Bill (cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
– Requires that the SSPI’s
recommendations include a plan for
transition for and address sixteen
areas outlined in statute.
– The recommendations are to be
reported to the Legislature on or
before November 1, 2012.
43
16 Areas of Consideration
TOM TORLAKSON
State Superintendent
of Public Instruction
1.
2.
3.
4.
5.
6.
Aligning assessments to standards
Implementing common assessments
developed by state collaborative
Conform to ESEA reauthorization
Measurement of achievement at a point in
time and over time for groups and
subgroups of pupils and for individual pupils
Allow for comparison from one year to the
next as a reflection of growth over time
Valid, reliable and fair for all students
including English learners (EL) and students
with disabilities
44
16 Areas of Consideration (cont.)
7.
TOM TORLAKSON
State Superintendent
of Public Instruction
Assessment of ELs using primary language
assessments
8. Ensure no bias with respect to race,
ethnicity, culture, religion, gender, or sexual
orientation
9. Incorporate a variety of item types including
open-ended and performance-based tasks
10. Generate multiple measures of pupil
achievement, which, when combined with
other measures, can be used to determine
the effectiveness of instruction and the
extent of learning
11. Assess science and history-social science in
all grade levels at or above grade 4
45
16 Areas of Consideration (cont.)
TOM TORLAKSON
State Superintendent
of Public Instruction
12. Assess understanding and ability to use
technology necessary for success in the 21st
century classroom and workplace
13. Formative and interim assessments that
provide timely feedback for purposes of
continually adjusting instruction to improve
learning
14. Use test administration and scoring
technologies that will allow the return of test
results to parents and teachers as soon as is
possible
15. Minimize testing time
16. Options for diagnostic assessments for
pupils in grade 2
46
Reauthorization of Statewide
Assessment System CDE Contact Information
TOM TORLAKSON
State Superintendent
of Public Instruction
Diane Hernandez
Interim Division Director
Assessment Development & Administration Division
DHernand@cde.ca.gov
916-445-9449
Jessica Barr
AB 250 Lead
Assessment Development & Administration Division
jbarr@cde.ca.gov
916-319-0364
47
SBAC - CDE Contact Information
TOM TORLAKSON
State Superintendent
of Public Instruction
Diane Hernandez
Interim Division Director
Assessment Development & Administration Division
DHernand@cde.ca.gov
916-445-9449
Kristen Brown, PhD
SBAC Coordinator
Education and Research Evaluation Consultant
Assessment Development & Administration Division
kbrown@cde.ca.gov
916-319-0334
Join the SBAC CDE electronic mailing
list by sending a blank e-mail to:
48
subscribe-sbac@mlist.cde.ca.gov