Measuring the Power of Learning.™
California Assessment of Student
Performance and Progress (CAASPP)
2015–16 CAASPP
Pretest Administration
Workshop
January–February 2016
Presented by
Educational Testing Service
Measuring the Power of Learning.™
Purpose
By the end of this workshop, attendees will
understand:
−
−
−
−
−
−
CAASPP is part of an integrated system with the primary
purpose of improving teaching and learning
Appropriate procedures for administration of the CAASPP
summative assessments
Technology requirements for the local educational
agency (LEA)
Student test registration in the Test Operations
Management System (TOMS)
Identifying students that need designated supports and
accommodations
The Smarter Balanced Digital Library and Interim
Assessments
2015–16 CAASPP Pretest Administration Workshop
2
Measuring the Power of Learning.™
Agenda
Morning
Session
Topic
8:30–8:45
Welcome
8:45–8:55
Overview
8:55–9:00
Roles and Responsibilities
9:00–9:10
Technology Infrastructure
9:10–9:20
Practice Tests and Training Tests
9:20–9:50
Using the Test Operations Management System
9:50–10:00
Break
10:00–10:30
Test Administration
10:30–10:45
Test Security
10:45–11:00
Smarter Balanced Scoring and Reporting
11:00–11:15
Digital Library and Interim Assessments
11:15–12:00
Questions
2015–16 CAASPP Pretest Administration Workshop
3
Measuring the Power of Learning.™
Overview
Measuring the Power of Learning.™
A New System
With a New Purpose
Transition from STAR to CAASPP means more
than changing standards and testing technology:
•
•
THEN (STAR): Administered end of year with
accountability solely as the primary purpose
NOW (CAASPP): An integrated year-round system
with a primary purpose of improving teaching and
learning
2015–16 CAASPP Pretest Administration Workshop
5
Measuring the Power of Learning.™
Summative Assessment:
One Part of an Integrated System
2015–16 CAASPP Pretest Administration Workshop
6
Measuring the Power of Learning.™
A New Role for Assessment:
Supporting Teaching and Learning
• Rather than a result, test scores are starting
points to identify strengths and needs and help
set learning goals.
• The Smarter Balanced Digital Library and
Interim Assessments are designed to help
schools achieve goals for learning, monitor
progress, and adjust instruction throughout the
year.
2015–16 CAASPP Pretest Administration Workshop
7
Measuring the Power of Learning.™
Student Participation: General
• The Smarter Balanced Summative Assessments
consist of the following:
•
– English Language Arts/Literacy (ELA) and
mathematics tests
– Includes computer adaptive portion of the test, a
classroom activity, AND a performance task in each
content area
The California Alternate Assessments (CAAs)
consist of the following:
– English Language Arts and mathematics tests for
students with significant cognitive disabilities
2015–16 CAASPP Pretest Administration Workshop
8
Measuring the Power of Learning.™
Student Participation: General
• All students in grades 3–8 and grade 11 take
the Smarter Balanced Summative Assessments
with the following exceptions:
–
Students enrolled in or assigned to grades 3–8
and grade 11 whose IEP team designates the
use of the California Alternate Assessments
(CAAs) for ELA and mathematics
–
English learners who have been enrolled in a
school in the United States less than 12
months (ELA only)
2015–16 CAASPP Pretest Administration Workshop
9
Measuring the Power of Learning.™
Student Participation:
Early Assessment Program
•
•
•
The Early Assessment Program (EAP) uses the high school
Smarter Balanced Summative Assessments for ELA and
mathematics.
Students taking the Smarter Balanced Summative
Assessments in high school may choose to release their test
scores to the California State University (CSU) and
participating California Community Colleges at the end of the
computer adaptive tests.
Students who do not authorize the release of their test
scores to CSU and CCC at the time of testing may need to
provide a copy of the CAASPP score report to the campus,
upon request.
2015–16 CAASPP Pretest Administration Workshop
10
Measuring the Power of Learning.™
Importance of Informing Parents
and Guardians About Testing
Pursuant to California Code of Regulations, Title 5,
Section 852:
• Each year the local educational agencies (LEAs)
are required to notify parents or guardians of their
pupil's participation in the CAASPP tests.
• In addition, LEAs are required to inform parents of
their right to exempt their child from taking the
CAASPP tests.
– Parents must annually submit a written request
specifying the tests to be exempted.
2015–16 CAASPP Pretest Administration Workshop
11
Measuring the Power of Learning.™
Importance of Informing Parents
and Guardians About Testing
•
Inform parents and guardians about testing to:
– Seek input on designated supports that the student
may need during testing.
– Coordinate with the student’s IEP team to determine
the most appropriate test for the student.
– Provide an opportunity for the parent/guardian to
submit a testing exemption.
– Communicate any schedule changes to the normal
school schedule.
– Obtain updates to student demographic information
that may impact testing or score reporting.
2015–16 CAASPP Pretest Administration Workshop
12
Measuring the Power of Learning.™
Smarter Balanced
Summative Assessments
Computer adaptive test (CAT):
• Students complete the CAT portion of the
assessment for both ELA and mathematics.
• The CAT includes a wide variety of question types.
• As a student progresses, the difficulty of questions
is adjusted on the basis of the student’s responses.
– A student who answers a question correctly will
receive a more challenging question
– An incorrect answer will generate an easier
question
2015–16 CAASPP Pretest Administration Workshop
13
Measuring the Power of Learning.™
Smarter Balanced
Summative Assessments
Classroom Activity for ELA and mathematics:
• Educators from across the Smarter Balanced
Assessment Consortium conducted a review of the
classroom activity.
– To determine if the Smarter Balanced performance
tasks (PT) in English Language arts/literacy (ELA)
and mathematics contained sufficient explanation and
context to stand on their own without the
administration of a classroom activity.
2015–16 CAASPP Pretest Administration Workshop
14
Measuring the Power of Learning.™
Smarter Balanced
Summative Assessments
Classroom Activity for ELA and mathematics:
• Smarter Balanced determined that the vast majority of
performance tasks contain sufficient explanation and context to
be administered without requiring the classroom activity.
– Based on this result, Smarter Balanced provided states with
the flexibility to either continue requiring the administration of
the classroom activity, make the administration of the
classroom activity optional, or discontinue the administration
the classroom activity.
– Any Performance Task that was identified as not containing
sufficient explanation and context to be administered without
a classroom activity has been removed from the 2015–16
Smarter Balanced tests.
2015–16 CAASPP Pretest Administration Workshop
15
Measuring the Power of Learning.™
Smarter Balanced
Summative Assessments
Classroom Activity for ELA and mathematics:
• California will utilize the 2015–16 year as a
transition year and will continue to assign
classroom activities in ELA and mathematics.
– In 2015–16, failure to administer the classroom
activity is not considered a testing incident.
– The CDE will not require LEAs to report failure to
administer the classroom activity in the Security
and Test Administration Incident Reporting System
(STAIRS).
2015–16 CAASPP Pretest Administration Workshop
16
Measuring the Power of Learning.™
Smarter Balanced
Summative Assessments
Performance Task:
• Students complete a performance task for both
ELA and mathematics.
• The Performance Task is designed to provide
students with an opportunity to demonstrate
their ability to apply their knowledge and higherorder thinking skills to explore and analyze a
complex, real-world scenario.
2015–16 CAASPP Pretest Administration Workshop
17
Measuring the Power of Learning.™
What’s New for 2015–16
•
•
•
•
User roles
Secure browser used for administration of CAAs
Validation of selected test administration window
Validation rules for CAAs, IEP indicator, and
accommodations
• Process for requesting unlisted resources
(previously called “individualized aids”)
• Enrollment grade cutoff date
• No accountability consequence for classroom
activity not given to students
2015–16 CAASPP Pretest Administration Workshop
18
Measuring the Power of Learning.™
What’s New for 2015–16
• Security and Test Administration Incident
Reporting System (STAIRS)
• Deadline for condition codes and Appeals
• Demographic snapshot
• Assessment Target Reports
• Faster initial Student Score Reports
• Student Score Reports available in Spanish
• Notification e-mails for scoring and reporting
2015–16 CAASPP Pretest Administration Workshop
19
Measuring the Power of Learning.™
What’s New for 2015–16
Changes to http://www.caaspp.org
2015–16 CAASPP Pretest Administration Workshop
20
Measuring the Power of Learning.™
What’s New for 2015–16
Changes to http://www.caaspp.org
2015–16 CAASPP Pretest Administration Workshop
21
Measuring the Power of Learning.™
What’s New for 2015–16
Changes to http://www.caaspp.org
2015–16 CAASPP Pretest Administration Workshop
22
Measuring the Power of Learning.™
Roles and Responsibilities
Measuring the Power of Learning.™
2015–16 User Roles
Available User Roles:
LEA CAASPP Coordinator
Test Site Coordinator
Test Administrator
Test Examiner
Test Administrator & Test Examiner
New for 2015–16
Interim Assessment
Administrator Only
Educator—District
View-only roles
Educator—Site
Educator—
Roster
2015–16 CAASPP Pretest Administration Workshop
24
Measuring the Power of Learning.™
LEA CAASPP Coordinator
Responsibilities
•
•
•
LEA CAASPP coordinators must add all users to the Test Operations
Management System (TOMS) to enable the designated users to
administer, monitor, and manage the online summative and interim
assessments.
– Adhere to local policies for protecting student information when
assigning user roles.
– Once test site coordinators are added by their LEA CAASPP
coordinator and receive access to TOMS, they will have the
ability to add test administrators, test examiners, and other user
roles for their assigned site.
Ensure that each user assigned either a Test Examiner or Test
Administrator & Test Examiner role is a credentialed or licensed
employee.
Ensure that all staff receive necessary training.
2015–16 CAASPP Pretest Administration Workshop
25
Measuring the Power of Learning.™
Test Site Coordinator
Responsibilities
•
•
•
Identify test administrators and test examiners and ensure that
they have submitted signed CAASPP Test Security Affidavits.
Assign site staff test administrators and test examiners roles in
TOMS.
– Adhere to local policies for protecting student information
when assigning user roles.
Ensure that test administrators and test examiners are
properly trained so they can administer the assessments.
– Ensure that test examiners view the test administration
tutorial and collect the certificate of completion and/or
attend a training provided by the LEA.
2015–16 CAASPP Pretest Administration Workshop
26
Measuring the Power of Learning.™
Test Site Coordinator
Responsibilities
• Enter and/or verify student test assignments and test settings in
TOMS.
• Create or approve testing schedules and procedures for the
school site.
• Work with technology staff to ensure that necessary secure
browsers and supported operating systems are installed and
any other technical issues are resolved.
• Provide student login information on cards (i.e., login tickets) to
assist test administrators with student login.
– New: Login tickets can include any information as long as
they are consider secure test materials and securely
destroyed.
2015–16 CAASPP Pretest Administration Workshop
27
Measuring the Power of Learning.™
Test Site Coordinator
Responsibilities
• Monitor testing progress and ensure that all eligible students
participate.
• Address testing problems that arise at the school site.
• Mitigate and report all test security incidents to the LEA
CAASPP coordinator.
• Oversee CAASPP administration activities in the school.
• Collect any print-on-demand questions and other secure
testing materials (e.g., student login information and
Directions for Administration [DFAs] for CAAs) following
testing and securely destroy them.
2015–16 CAASPP Pretest Administration Workshop
28
Measuring the Power of Learning.™
Test Administrator
Responsibilities
•
•
•
•
•
Read and sign the CAASPP Test Security Affidavit.
Complete training and review all Smarter Balanced policy
and administration documents prior to administering any
Smarter Balanced assessments.
Review student information prior to testing to ensure that the
correct student receives the proper test with the appropriate
supports and test settings.
Administer the Smarter Balanced Summative Assessments.
Report all potential test security incidents to the test site
coordinator and LEA CAASPP coordinator in a manner
consistent with LEA and state policies.
2015–16 CAASPP Pretest Administration Workshop
29
Measuring the Power of Learning.™
Test Examiner
Responsibilities
•
•
•
Are trained by the LEA and/or view the test
administration tutorial, print the certificate, and submit it
to the LEA.
Read and sign a CAASPP Test Security Affidavit and
submit it to the test site coordinator.
Review and/or assign accessibility supports in TOMS
for each student being tested.
– Note: Test site coordinators may assign the accessibility
supports or the LEA CAASPP coordinator may provide
test examiners with temporary CAASPP test site
coordinator-level access to TOMS.
•
Download and read the DFAs before testing.
2015–16 CAASPP Pretest Administration Workshop
30
Measuring the Power of Learning.™
Other User
Responsibilities
• The “Test Examiner & Test Administrator” role
has the responsibilities of both individual roles.
• “Interim Assessment Only,” “Educator—District,”
“Educator—Site,” and “Educator—Roster” roles
also have general security responsibilities for
protecting student personally identifiable
information (PII).
– It is a local decision whether to require a
CAASPP Test Security Affidavit for these roles.
2015–16 CAASPP Pretest Administration Workshop
31
Measuring the Power of Learning.™
Technology Coordinator
Responsibilities
•
•
•
•
Assist LEAs in setting up and overseeing technology
infrastructure.
Support each school site.
Work closely with the LEA CAASPP coordinator to
ensure that network and testing devices are properly
prepared by installing the most current secure
browser based upon device operating system.
Help troubleshoot during testing.
2015–16 CAASPP Pretest Administration Workshop
32
Measuring the Power of Learning.™
CALPADS Administrator
Responsibilities
•
•
Work closely with the LEA CAASPP coordinator.
Ensure that student demographic, enrollment, and
program participation data are correct in CALPADS.
– New: Embedded and non-embedded accommodations
are available in TOMS only to students that have an
Education Program designation of “101” for a Section
504 plan or “144” for special education with a valid
primary disability category in CALPADS.
– New: Students cannot be registered for CAAs in TOMS
without an Education Program designation “144” for
special education with a valid primary disability category
in CALPADS.
2015–16 CAASPP Pretest Administration Workshop
33
Measuring the Power of Learning.™
User Resources
•
•
•
TOMS User Roles and Definitions
http://www.caaspp.org/administration/toms/tomsusers.html
Instructions for Adding Users
http://www.caaspp.org/administration/toms/tomsupload-users.html
2015–16 User Role Guidance
http://www.caaspp.org/administration/toms/index.html
2015–16 CAASPP Pretest Administration Workshop
34
Measuring the Power of Learning.™
User Resources
• Guides & Templates
– http://www.caaspp.org/administration/toms/index.
html
 2015–16 Adding and Managing Users Guide
 User template for batch upload
 User file specifications for batch upload
• TOMS Test Administration and User Setup
Webcast
– http://www.caaspp.org/rsc/videos/archivedwebcast_102015.html
2015–16 CAASPP Pretest Administration Workshop
35
Measuring the Power of Learning.™
Technology Infrastructure
Measuring the Power of Learning.™
Technology Infrastructure
Prior to test administration, LEA CAASPP
coordinators with technology coordinators should:
 Check for a stable, high-speed, Internet connection
(wired or wireless) because the Test Delivery System
(TDS) administers tests via the Internet.
 Check that testing devices are working.
 Ensure that secure browsers are properly installed a
minimum of two weeks prior to testing.
 Make sure the necessary hardware is available and
working (i.e., headphones, external keyboards, printers).
 Configure filters, firewalls, and proxy servers.
2015–16 CAASPP Pretest Administration Workshop
37
Measuring the Power of Learning.™
High-level Technology Overview
State-hosted
test delivery
servers
Student using
Smarter Balanced
California Secure
Browser software
application
Test administrator
using standard
Web browser
2015–16 CAASPP Pretest Administration Workshop
38
Measuring the Power of Learning.™
Secure Browsers
•
Secure browsers are specifically designed for
California to provide secure access.
• New: Secure browsers for all supported operating
systems now update annually; no need to reinstall
every year.
• Secure browsers prevent students from accessing
other hardware and software functions while testing.
• Secure Browser 8.0 must be installed on all
computers/devices used to administer CAASPP tests.
Note: Test administrator computers may
use a standard Web browser.
2015–16 CAASPP Pretest Administration Workshop
39
Measuring the Power of Learning.™
Secure Browsers
2015–16 CAASPP Pretest Administration Workshop
40
Measuring the Power of Learning.™
Secure Browsers
•
•
Students must use the secure browser to log on to
take any CAASPP online test, including summative
assessments, interim assessments, and the CAAs.
Prior to testing, test administrators and test
examiners should verify that:
− External user applications are closed.
 The secure browser will not work if the computer
detects that a forbidden application is running.
− Testing does not occur on computers with dual
monitors

Mirror mode can be used in the case of a read aloud but
cannot be used otherwise.
2015–16 CAASPP Pretest Administration Workshop
41
Measuring the Power of Learning.™
Operating Systems
New: iOS 9.2 Operating System
•
•
Users of iOS 8.3, 8.4, 9.0 and 9.1 will be required to
update to iOS 9.2 prior to administering Smarter
Balanced Summative Assessments or CAAs.
Once the summative assessments have been
launched, iOS 9.2 will be the only supported operating
system for accessing both interim and summative
assessments on iPad devices running on version 9.x.
2015–16 CAASPP Pretest Administration Workshop
42
Measuring the Power of Learning.™
Hardware Requirements
•
Device displays
–
10ʺ diagonal display or larger (iPads with 9.5ʺ screens
are acceptable.)
– 1024 x 768 display resolution or better
Note: Check monitor settings and brightness.
•
On-site printers
–
–
–
Print out test session information.
Print test stimuli or questions for students with the printon-demand accommodation.
Only the test administrator’s computer should have
access to a single local or network printer in the testing
room.
2015–16 CAASPP Pretest Administration Workshop
43
Measuring the Power of Learning.™
Hardware Requirements
•
Headphones are required for each Smarter
Balanced test station.
– English language arts/literacy (ELA) tests
– Text-to-speech accommodation
– Job Access with Speech (JAWS)
 Version 16 supported
– Supported headphones:
 Wired headphones with a 3.5 mm connector
 USB headphones with a wired or wireless connection
 Bluetooth headphones: Logitech H800, SoundBot
SB270, and ECO Sound ECO-V300
Note: Microphones are not required.
2015–16 CAASPP Pretest Administration Workshop
44
Measuring the Power of Learning.™
Hardware Requirements
• External keyboards are required for tablets.
– Mechanical
– Manual
– Bluetooth-based
Note: Avoid keyboards with “shortcut” buttons.
2015–16 CAASPP Pretest Administration Workshop
45
Measuring the Power of Learning.™
Technology Resources
•
CAASPP Secure Browsers
http://ca.browsers.airast.org/
•
Diagnostic testing tool to test school’s bandwidth
http://www.caaspp.org/about/testing/index.html
•
Secure Browser Installation Manual
http://www.caaspp.org/rsc/pdfs/CAASPP.secure-browsermanual.2016.pdf
•
Technology Specifications Manual
http://www.caaspp.org/rsc/pdfs/CAASPP.tech-specsmanual.2016.pdf
•
Preparing Technology for 2015–16 Webcast
http://www.caaspp.org/rsc/videos/archived-webcast_1117152.html
2015–16 CAASPP Pretest Administration Workshop
46
Measuring the Power of Learning.™
Practice Tests
and Training Tests
Measuring the Power of Learning.™
Preparing Students and Staff
There are multiple ways to prepare students and
staff for the summative assessments:
•
•
•
Smarter Balanced Practice Tests
Smarter Balanced Training Tests
New: CAA Training Tests, coming soon
Note: Practice tests for the CAAs are not available
for 2015–16.
2015–16 CAASPP Pretest Administration Workshop
48
Measuring the Power of Learning.™
Important Notes About
Practice and Training Tests
Practice Tests and Training Tests:
•
•
•
Do not assess all of the standards that will be in
the operational tests.
Do not predict or gauge a student’s success on
the operational assessments.
Do not produce overall scores.
2015–16 CAASPP Pretest Administration Workshop
49
Measuring the Power of Learning.™
Smarter Balanced
Practice Tests vs. Training Tests
Practice Tests
Purpose
Grade Levels
Number and Types of
Questions
Universal Tools,
Designated
Supports, and
Training Tests
Provide students with a grade
specific testing experience similar in
structure and format to the summative
assessments
Provide students with an opportunity to
quickly become familiar with the software
and interface features
Each grade:
• 3–8, 11
Grade bands:
• 3–5
• 6–8
• High school
Approximately 30 questions in ELA and 30
questions in mathematics per grade level;
includes 1 ELA Performance Task and 1
mathematics Performance Task per grade
level
Approximately 15 questions per grade
band (6 in ELA and 8–9 in mathematics);
Performance Tasks are not available
All
All
Accommodations
Scoring
Tests are not scored, however answer
keys and scoring rubrics are available
Tests are not scored
2015–16 CAASPP Pretest Administration Workshop
50
Measuring the Power of Learning.™
CAAs Training Tests
Training Tests for CAAs
Purpose
Provide students with an opportunity to quickly become familiar with
the test delivery system, question types, and interactions provided in the
DFAs
Grade Levels
One test each for ELA and mathematics
Number and
Types of
Questions
Approximately 5–10 questions for each content area
Universal Tools,
Designated
Supports, and
All that are available for CAAs
Accommodations
Scoring
Tests are not scored
2015–16 CAASPP Pretest Administration Workshop
51
Measuring the Power of Learning.™
Purpose of the Smarter Balanced
Practice Tests
Allow students and administrators to become
familiar with:
•
•
•
•
The experience of a grade-level assessment
A variety of grade-specific questions and
difficulty levels
Performance tasks
The format and structure of the operational
assessments
2015–16 CAASPP Pretest Administration Workshop
52
Measuring the Power of Learning.™
About the Smarter Balanced
Practice Tests
•
•
•
•
•
Practice Test are available in each grade level for grades 3–
8 and grade 11 for ELA and mathematics.
Each Practice Test contains approximately 30 questions.
Performance tasks are available for both ELA and
mathematics.
Scoring rubrics and answers are available on the Test
Administrators Resource Web page at
http://www.caaspp.org/ta-resources/index.html.
All universal tools, designated supports, and
accommodations are available.
–
–
Text-to-speech only available through secure browser.
Print-on-demand only available in proctored session.
2015–16 CAASPP Pretest Administration Workshop
53
Measuring the Power of Learning.™
Purpose of the Smarter Balanced
and CAAs Training Tests
•
•
•
Designed to try out the universal tools, designated
supports, and accommodations to establish the
most optimal test setting experience for the student
Allow students and administrators to become
familiar with the test delivery system and interface
features
Allow students to become familiar with:
– Accessing the tests (log on process)
– The testing software
– The navigational tools
– Embedded supports
2015–16 CAASPP Pretest Administration Workshop
54
Measuring the Power of Learning.™
About the Smarter Balanced
Training Tests
•
•
•
•
•
Training Tests are available by grade bands for grades 3–5,
grades 6–8, and high school.
Each Training Test contains approximately 14 or 15
questions:
– 6 ELA questions
– 8 or 9 mathematics questions
There are no performance tasks included in the Training
Tests.
There are no scoring rubrics or answer keys.
All universal tools, designated supports, and
accommodations are available.
–
–
Text-to-speech only available through secure browser.
Print-on-demand only available in proctored session.
2015–16 CAASPP Pretest Administration Workshop
55
Measuring the Power of Learning.™
Accessing the Practice
and Training Tests
•
•
There are two means for accessing the Practice and
Training Tests:
1. Through http://www.caaspp.org on a Web browser
2. Through the secure browser, which simulates
operational online testing environment.
Test administrators may create test sessions for the
Practice Tests or Training Tests on the Test
Administrator Practice and Training Site.
− Students may return to a Practice Test and/or
Training Test session.
2015–16 CAASPP Pretest Administration Workshop
56
Measuring the Power of Learning.™
Accessing the Practice
and Training Tests
1
2
Students and parents
who wish to access
the Practice Test
and/or Test via an
Internet browser will
select the button below
3
2015–16 CAASPP Pretest Administration Workshop
57
Measuring the Power of Learning.™
Practice and Training Tests
Resources
• Test Administrator (TA) Resources for the
Smarter Balanced Practice and Training Tests
http://www.caaspp.org/ta-resources/practicetraining.html
2015–16 CAASPP Pretest Administration Workshop
58
Measuring the Power of Learning.™
Using the Test Operations
Management System
(TOMS)
Measuring the Power of Learning.™
Tasks to Complete in TOMS
for the 2015–16 Administration
 Add and manage all users.
 Set up test administrations.
 Review student data.
−
Demographic data from CALPADS
 Assign students to non-default tests—California Modified
Assessment (CMA), CAAs/California Alternate
Performance Assessment (CAPA), and Standards-based
Tests in Spanish for Reading/ language arts (STS for RLA).
 Enter condition codes for Parent/Guardian Exemption
(PGE) and Not Tested due to Medical Emergency (NTE).
 Assign online test settings to students.
2015–16 CAASPP Pretest Administration Workshop
60
Measuring the Power of Learning.™
Add and Manage TOMS Users
•
•
•
LEA CAASPP coordinators were given access to TOMS
after the superintendent designation form and test
security agreement were submitted to the California
Technical Assistance Center (CalTAC).
LEA CAASPP coordinators received an e-mail with a
username and temporary password that allows access
to TOMS.
TOMS can be accessed from the CAASPP portal at
http://www.caaspp.org.
2015–16 CAASPP Pretest Administration Workshop
61
Measuring the Power of Learning.™
Add and Manage TOMS Users
•
•
•
LEA CAASPP coordinators are responsible for adding
additional users to TOMS.
Two methods for adding users to TOMS
– One-by-one through TOMS
– Template upload
 Blank Template
http://www.caaspp.org/rsc/pdfs/User_File_Upload_
Template_v3.xlsx
 File containing prior year users
Ensure that network firewalls include @ets.org,
@caaspp.org, and @airast.org as whitelisted domains.
2015–16 CAASPP Pretest Administration Workshop
62
Measuring the Power of Learning.™
Test Administration
Windows and Setup
Measuring the Power of Learning.™
Test Administration Windows
•
Testing windows for 2015–16 Smarter Balanced
Summative Assessments:
– Grades 3–8 for ELA and mathematics
 The day on which 66% of a school’s annual
instructional days have been completed
– Grades 11 for ELA and mathematics
 The day on which 80% of a school’s annual
instructional days have been completed
Note: Grade 11 scores used for Early Assessment
Program (EAP) purposes.
2015–16 CAASPP Pretest Administration Workshop
64
Measuring the Power of Learning.™
Test Administration Windows
•
Testing windows for 2015–16 Science
(CST/CMA/CAPA) and STS for RLA
– Testing window of 25 instructional days, which
includes 12 instructional days before and after
completion of 85% of the school’s, or track’s, annual
instructional days
•
Testing window for 2015–16 CAAs
–
–
April 11 through June 17, 2016, for all LEAs and schools
No administration window setup required in TOMS for
the CAAs
2015–16 CAASPP Pretest Administration Workshop
65
Measuring the Power of Learning.™
Test Eligibility
Smarter Balanced
ELA and
Mathematics
 Default test assignment for grades 3–8 and grade 11
 Not taking the CAAs
CST for Science
 Default science test assignment for grades 5, 8 and 10
 Not taking CAA (grades 5 and 8) or CAPA Science
CMA for Science
 Grade 5, 8, and 10
 Has IEP plan and IEP team determined CMA as most
appropriate test
 Not taking for CAA (grades 5 and 8), CST or CAPA Science
STS for RLA
 Voluntary tests for grades 2 through 11
 Determined by test site coordinator
 Not eligible for CAA/CAPA Science
CAA ELA and
Mathematics,
CAPA Science
 Must be designated as having severe cognitive disability
 Must have an IE and IEP team determined that CAA/CAPA
Science as the most appropriate tests
2015–16 CAASPP Pretest Administration Workshop
66
Measuring the Power of Learning.™
Test Administration Windows
•
Considerations for selecting your test
administration window:
– LEAs have the option to use the regulatory test
administration window or to select a shorter test
administration window.
– New: Selected testing window will be validated.
– New: Selected testing window must be at least 25
instructional days.
2015–16 CAASPP Pretest Administration Workshop
67
Measuring the Power of Learning.™
Test Administration Windows
•
Considerations for selecting your test
administration window:
– New: LEAs are not required to test students who
enroll after the close of their selected test
administration window.
– New: Condition codes must be set within the
selected test administration window.
– New: Appeals must be resolved within the selected
test administration window.
– New: Clock starts for paper final Student Score
Reports at the close of selected test administration
window.
2015–16 CAASPP Pretest Administration Workshop
68
Measuring the Power of Learning.™
Test Administration Setup
Resources
•
TOMS Test Administration Setup Guide
−
•
http://www.caaspp.org/rsc/pdfs/CAASPP.test_admin_set
up.2016.pdf
TOMS Test Administration and User Setup Webcast
−
http://www.caaspp.org/rsc/videos/archivedwebcast_102015.html
2015–16 CAASPP Pretest Administration Workshop
69
Measuring the Power of Learning.™
TOMS Student Profile:
Demographics
•
All demographic data
– Data are read-only; they must be modified in
CALPADS.
– New: A snapshot of the demographic data for
CAASPP reporting purposes will be pulled on a
selected date during spring 2016.
•
Students’ grade levels must be correct in TOMS
before the first date of testing window.
− New: Grade level is frozen on the first date of the
test administration window.
− Grade level is correctable only in CALPADS.
2015–16 CAASPP Pretest Administration Workshop
70
Measuring the Power of Learning.™
TOMS Student Profile:
Demographics
All
demographic
data is read
only
2015–16 CAASPP Pretest Administration Workshop
71
Measuring the Power of Learning.™
TOMS Student Profile:
Demographics
•
CALPADS Education Program designation for Special
Education (IEP) or Section 504 plan
– These data are read-only; they must be modified in
CALPADS.
– New: Embedded and non-embedded accommodations
are available only to students who have an Education
Program code of “101” for a Section 504 plan or “144”
for a Special Education indicator with a valid primary
disability type in CALPADS.
2015–16 CAASPP Pretest Administration Workshop
72
Measuring the Power of Learning.™
Special Education and Valid
Primary Disability Codes
• Determines access to:
– CAA, CAPA Science, CMA for Science
– Embedded and non-embedded
accommodations for any tests
• Must be updated in CALPADS at least 2
business days before testing the student.
Note: Removing a Special Education indicator from a
student’s CALPADS information will remove access to
any accommodations or test assignments to
CAA/CAPA Science or CMA for Science.
2015–16 CAASPP Pretest Administration Workshop
73
Measuring the Power of Learning.™
TOMS Student Profile:
Test Mode
•
•
Test Assignment
– All students in grades 3–8 and grade 11 are
automatically assigned the online Smarter Balanced
Summative Assessments.
– Students who are eligible for the CAAs must be
assigned in TOMS.
– New: Students cannot be assigned the CAAs in
TOMS without an Education Program code of “144”
for Special Education and with a valid primary
disability type in CALPADS.
LEAs should verify student test assignments a minimum
of 24 hours prior to student testing.
2015–16 CAASPP Pretest Administration Workshop
74
Measuring the Power of Learning.™
Student Profile in TOMS:
Test Mode
All students in grades
3–8 and 11 are
automatically enrolled in
the online Smarter
Balanced Summative
Assessments
2015–16 CAASPP Pretest Administration Workshop
75
Measuring the Power of Learning.™
Condition Codes
•
New: Only two codes can be uploaded directly to TOMS:
–
–
•
New: Batch upload of condition codes will be available.
–
•
•
Parent Guardian Exemption (PGE)
Not Tested due to Medical Emergency (NTE)
This applies only to the two codes that can be set in TOMS
as noted above.
New: The condition code deadline is last day of the
selected test administration window for the LEA; once the
testing window ends, the functionality to input condition
codes will no longer be available.
Only one condition code can be applied per content area
per student.
2015–16 CAASPP Pretest Administration Workshop
76
Measuring the Power of Learning.™
Indicating Parent/Guardian
Exemptions
• Set the parent/guardian exemptions in TOMS
prior to testing.
• The Parent/Guardian Exemption (PGE)
condition code can be set in
the Score Status
tab of the student’s
profile in TOMS.
2015–16 CAASPP Pretest Administration Workshop
77
Measuring the Power of Learning.™
Student Profile in TOMS:
Test Settings
•
Student online test settings must be configured at least
24 hours prior to testing.
–
•
Allow up to 24 hours from the time you configure student
test settings in TOMS to the time the changes appear in
the Test Administrator and Student Testing Interface.
LEA CAASPP coordinators and test site coordinators
are authorized to set accessibility supports for students
in TOMS using the following methods:
1. CAASPP Individual Student Assessment Accessibility Profile
(ISAAP) Tool file upload
2. Template upload
3. One-by-one through student profile
2015–16 CAASPP Pretest Administration Workshop
78
Measuring the Power of Learning.™
Student Profile in TOMS:
Test Settings
2015–16 CAASPP Pretest Administration Workshop
79
Measuring the Power of Learning.™
Student Online Test Settings
File Upload
2015–16 CAASPP Pretest Administration Workshop
80
Measuring the Power of Learning.™
Reports in TOMS
•
New: Student Move Report
–
•
Report includes a list of students who have moved within
the LEA and students who have moved into the LEA from
outside the LEA.
New: Request New Report functionality
–
Request New Report allows the user to leave the page
and receive an e-mail notification when the report is
available for download.
2015–16 CAASPP Pretest Administration Workshop
81
Measuring the Power of Learning.™
Reviewing Student Data
Resources
•
TOMS Online Student Test Settings User Guide
http://www.caaspp.org/rsc/pdfs/CAASPP.student-testsettings.2016.pdf
•
Configuring Online Student Test Settings in TOMS Webcast
http://www.caaspp.org/rsc/videos/archived-webcast_102815.html
2015–16 CAASPP Pretest Administration Workshop
82
Measuring the Power of Learning.™
Importance of CALPADS
•
CALPADS is the sole source for student demographic,
enrollment grade, and program participation data.
o
o
o
o
o
Statewide Student Identifier (SSID)
Student Name, gender, date of birth
LEA and school names and CDS codes
Grade
Primary language, LEP status and entry/exit dates, U.S.
school enrollment date, migrant status
o Ethnicity, race
o Education program code (Special Education, Section 504
status), primary disability code
o Parent education, education program participation, foster
youth indicator
2015–16 CAASPP Pretest Administration Workshop
83
Measuring the Power of Learning.™
Importance of CALPADS
• New: Grade level is frozen on the first date of
the test administration window.
− Grade level is correctable only in CALPADS.
• LEAs cannot update student demographic data
in TOMS.
• LEAs must keep data in CALPADS up to date.
• Processing time from CALPADS to TOMS is
24–48 hours.
• Student accessibility supports are uploaded and
maintained in TOMS, not in CALPADS.
2015–16 CAASPP Pretest Administration Workshop
84
Measuring the Power of Learning.™
CALPADS Resources
• CALPADS Integration with CAASPP Webcast
http://www.caaspp.org/rsc/videos/archived-webcast_111715.html
•
CALPADS System Documentation Web page
http://www.cde.ca.gov/ds/sp/cl/systemdocs.asp
2015–16 CAASPP Pretest Administration Workshop
85
Measuring the Power of Learning.™
Accessibility Supports
Measuring the Power of Learning.™
Accessibility Supports
• Universal Tools:
– Available to all; based on student preference and
selection (e.g., strikethrough, highlighter)
• Designated Supports:
– Available to all; need to be indicated by an educator
or guardian (e.g., color contrast, read aloud,
glossary, magnification)
• Accommodations:
− Based on IEPs or Section 504 plan (e.g., American
Sign Language, scribe)
2015–16 CAASPP Pretest Administration Workshop
87
Measuring the Power of Learning.™
Accessibility Supports
• The online assessments include embedded and
non-embedded accessibility supports:
– Embedded: part of the computer administration
(e.g., digital notepad, calculator)
– Non-embedded: provided outside the computer
administration (e.g., scratch paper, multiplication
table)
2015–16 CAASPP Pretest Administration Workshop
88
Measuring the Power of Learning.™
Accessibility Supports
2015–16 CAASPP Pretest Administration Workshop
89
Measuring the Power of Learning.™
Accessibility Supports
Embedded
Designated Support
Color contrast
Non-Embedded
Accommodation
Designated Support
Accommodation
American Sign Language (ELA
Bilingual dictionary (ELA
Abacus
listening and math questions) Performance Task full writes)
Masking
Braille
Color contrast
Text-to-speech (ELA and
math questions)
Translations (glossaries)
(math questions)
Translations (stacked) and
Translated test directions
( for math - Spanish)
Closed captioning (ELA
listening questions)
Color overlay
Streamlined interface
Magnification
Text-to-speech (ELA reading
passages)
Noise buffers
Alternate response options
Calculator (math—allowed
questions Grades 6–8, 11)
Multiplication table (math Grades 4–8, 11)
Print on demand**
Read aloud (ELA and math
questions)
Scribe (ELA non-writing and
math questions)
Separate setting
Read aloud in Spanish
Print size
Permissive mode
Turn off any universal tools*
*Must be activated in the test delivery
system.
**Print on demand cannot be activated in
TOMS. If a student needs print on demand
for questions and/or questions and stimuli,
contact the CalTAC.
Read aloud passages
(ELA)
Scribe (ELA Performance
Task full write)
Speech-to-text
Unlisted resources
Additional alternate
Translation (glossaries) (only
assessment instructional
paper pencil math tests)
supports (CAAs only)
Translated test directions
(for online ELA and math
questions)
2015–16 CAASPP Pretest Administration Workshop
90
Measuring the Power of Learning.™
Unlisted Resources
New: Process for requesting unlisted resources
(previously called “individualized aids”):
• Request review and approval for unlisted
resources by filling out the Web form in the
Unlisted Resources tab section in TOMS.
2015–16 CAASPP Pretest Administration Workshop
91
Measuring the Power of Learning.™
Online Fixed Braille Form
•
•
•
New: A Smarter Balanced Braille Fixed Online form for
mathematics will be available for the 2015–16 test
administration.
The test administrator will select the Braille Fixed Online
Form by selecting the language of braille and selecting the
fixed form option.
LEA CAASPP coordinators order the accompanying graphic
package by contacting CalTAC.
– A pre-embossed graphics package will be sent to LEA
prior to testing.
– The graphic package will be used in conjunction with the
Smarter Balanced Braille Fixed Online Form for
mathematics.
2015–16 CAASPP Pretest Administration Workshop
92
Measuring the Power of Learning.™
Accessibility Supports Resources
•
•
•
Smarter Balanced Usability, Accessibility, and
Accommodations Guidelines
http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2014/08/SmarterBalanced_Guidelines.pdf
Smarter Balanced Usability, Accessibility, and
Accommodations Implementation Guide
http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2014/03/Usability-Accessibility-andAccommodations-Implementation-Guide.pdf
Guidelines: Frequently Asked Questions
http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2013/12/SmarterBalanced_Guidelines_FAQ.
pdf
2015–16 CAASPP Pretest Administration Workshop
93
Measuring the Power of Learning.™
Accessibility Supports Resources
•
•
•
CAASPP Accessibility and Accommodations Webcast
http://www.caaspp.org/rsc/videos/archivedwebcast_121615.html
Matrix One
http://www.cde.ca.gov/ta/tg/ai/caasppmatrix1.asp
Using the CAASPP Individual Student Assessment
Accessibility Profile Tool video tutorial
http://www.caaspp.org/rsc/videos/archived-webcast_ISAAPoverview-and-demo.html
2015–16 CAASPP Pretest Administration Workshop
94
Measuring the Power of Learning.™
Test Administration
Measuring the Power of Learning.™
Systems and Processes for
Online Administration
1
2
3
TOMS
CALPADS
(California Longitudinal
Pupil Achievement Data
System)
•Manage student
demographic data
• Set up test administration
windows
• Verify student
demographic data
• Configure online test
settings (designated
supports and
accommodations)
Test Administrator
Interface & Student
Testing Site
• Administer online
Smarter Balanced
Assessments
• Administer CAAs
2015–16 CAASPP Pretest Administration Workshop
96
Measuring the Power of Learning.™
Scheduling Testing Time
•
Estimated testing times:
Computer
Adaptive Test
(CAT)
questions
Content Area
English
Language
Arts/Literacy
Mathematics
Both
Grades
hrs : mins
Performance
Task (PT)
Total
Classroom
Activity
(administered
prior to the
PT)*
hrs : mins
hrs : mins
hrs : mins
hrs : mins
Total
3–5
1:30
2:00
3:30
:30
4:00
6–8
1:30
2:00
3:30
:30
4:00
11
2:00
2:00
4:00
:30
4:30
3–5
1:30
1:00
2:30
:30
3:00
6–8
2:00
1:00
3:00
:30
3:30
11
2:00
1:30
3:30
:30
4:00
3–5
3:00
3:00
6:00
1:00
7:00
6–8
3:30
3:00
6:30
1:00
7:30
11
4:00
3:30
7:30
1:00
8:30
2015–16 CAASPP Pretest Administration Workshop
97
Measuring the Power of Learning.™
Scheduling Testing Time
•
•
The estimates do not account for any time needed to
start computers, load secure browsers, and log in
students.
To help estimate number of days and associated
network bandwidth required, schools should access the
Smarter Balanced Technology Readiness Calculator at
http://www3.cde.ca.gov/sbactechcalc/:
−
•
Enter the number of students testing, number of
computers available, computer hours available per day,
and Internet connection speed.
Review and account for test expiration rules when
scheduling testing time.
2015–16 CAASPP Pretest Administration Workshop
98
Measuring the Power of Learning.™
Scheduling Testing Time
•
•
Scheduling is a collaborative effort
Recommended: Schedule the CAT first and the
Performance Task (PT) second.
–
The classroom activity should be completed within 3 days
prior to administering a Performance Task.
 ELA PTs have 2 parts and can be completed over 2 days.
 Mathematics PTs have 1 part and can be completed in 1 day.
CAT
•
Classroom
Activity
Performance
Task
Recommended: PTs should not be administered on the
same day as the CAT.
2015–16 CAASPP Pretest Administration Workshop
99
Measuring the Power of Learning.™
Example Testing Schedules
•
•
Example 1:
Monday
Tuesday
Wednesday
Thursday
Friday
All math CAT
Math classroom
activity
Math Performance
Task
All ELA CAT
ELA classroom
activity
Next Monday
Next Tuesday
Next Wednesday
Next Thursday
Next Friday
ELA Performance
Task, Part 1
ELA Performance
Task, Part 2
Wednesday
Thursday
Friday
Math Performance
Task
ELA CAT, Session 1
ELA CAT, Session 2
ELA classroom
activity
Next Wednesday
Next Thursday
Next Friday
Example 2:
Monday
Tuesday
Math CAT, Session 2
Math CAT, Session 1
Math classroom
activity
Next Monday
Next Tuesday
ELA Performance
Task, Part 1
ELA Performance
Task, Part 2
2015–16 CAASPP Pretest Administration Workshop
100
Measuring the Power of Learning.™
Establishing Appropriate
Testing Conditions
•
Testing students in classroom-sized groups is
preferable:
–
–
•
•
Reduces test fear and anxiety
Facilitates monitoring and control for test administrators
Room should have good lighting and ventilation, and be
free of noise and interruptions.
Remove or cover displayed information (on bulletin
boards, chalkboards, and dry-erase boards, etc.) that
may be used by students to help answer questions:
–
Rubrics, vocabulary charts, student work, posters,
graphs, charts, etc.
2015–16 CAASPP Pretest Administration Workshop
101
Measuring the Power of Learning.™
Establishing Appropriate
Testing Conditions
•
Make appropriate seating arrangements prior to
test administration:
– Students should be seated so that they cannot view
the answers of others.
•
Establish procedures to maintain a quiet testing
environment:
– Remember that some students may finish more
quickly than others.
2015–16 CAASPP Pretest Administration Workshop
102
Measuring the Power of Learning.™
Establishing Appropriate
Testing Conditions
•
•
•
Cell phones or other electronic devices are not allowed
to be used during testing (e.g., they must be turned off
and put away).
Post an “Unauthorized Electronic Devices May Not
Be Used at Any Time During the Testing Session”
sign so that it is clearly visible to all students.
Place a “TESTING—DO NOT DISTURB” sign on the
door.
2015–16 CAASPP Pretest Administration Workshop
103
Measuring the Power of Learning.™
General Test Rules
•
•
•
•
•
Students must answer all test questions on a page (i.e.,
screen) before moving on to the next page.
Students may not return to a test segment once it has been
completed and submitted.
Students must answer all test questions before the test can
be submitted.
Students may not return to a test once it has been completed
and submitted.
Test administrators must follow the “SAY” script exactly each
time a test session is administered.
−
Refer to the “SAY” boxes of the Online Test Administration
Manual and/or Online Directions for Administration.
2015–16 CAASPP Pretest Administration Workshop
104
Measuring the Power of Learning.™
Administering a Test Session
1. The test administrator logs on to the Test
Administrator Interface from http://www.caaspp.org
on a standard Web browser.
2015–16 CAASPP Pretest Administration Workshop
105
Measuring the Power of Learning.™
Administering a Test Session
2. The test administrator selects the test category or
the grade level and content area.
2015–16 CAASPP Pretest Administration Workshop
106
Measuring the Power of Learning.™
Administering a Test Session
3. The test administrator selects the tests to administer.
–
Additional tests can be added to a session; you will
receive a confirmation message box.
4. The test administrator
starts a test session.
2015–16 CAASPP Pretest Administration Workshop
107
Measuring the Power of Learning.™
Administering a Test Session
5. The test administrator informs students of the test
Session ID.
–
–
Write or display it clearly in a place all students can see.
Session IDs are automatically generated upon selecting the
Start Session button.
Additional tests can be added
to a session; a confirmation
message box confirms adding
tests to a session.
2015–16 CAASPP Pretest Administration Workshop
108
Measuring the Power of Learning.™
Administering a Test Session
•
Test administrators need to distribute Student Logon
Information providing students with their Statewide
Student Identifier (SSID) and confirmation code (student
first name in CALPADS) for logon along with the
Session ID.
−
−
The information may be provided on a card or piece of
paper (i.e., login ticket).
Additional information can be included on the login ticket.
Important: Student personal information is secure and
must be collected and securely stored at the end of a test
session, then securely destroyed at the end of the test
administration.
2015–16 CAASPP Pretest Administration Workshop
109
Measuring the Power of Learning.™
Administering a Test Session
6. Students log on through the secure browser.
–
−
Recommended: Secure browsers are already launched on
testing devices before students sit down to test.
To log on, students need:
 First Name (Confirmation Code): Student’s legal first name as
spelled in CALPADS
 State-SSID: “CA” followed by a hyphen “-” and the student’s SSID
 Session ID: Test administrator
-generated Session ID
2015–16 CAASPP Pretest Administration Workshop
110
Measuring the Power of Learning.™
Administering a Test Session
7. Students verify their identity.
–
–
After logging on, students will see the “Is This You?” screen.
Students should select Yes to continue if the information on the
screen is correct.
2015–16 CAASPP Pretest Administration Workshop
111
Measuring the Power of Learning.™
Administering a Test Session
8. Student selects the test.
–
–
All grade-level tests that the student is eligible to take are
displayed.
Only the tests that the test administrator has selected for the
test session and those that have not been completed are
selectable by students.
Note: Only the tests
that have been
selected by the test
administrator will
appear active. Others
will be grayed out.
2015–16 CAASPP Pretest Administration Workshop
112
Measuring the Power of Learning.™
Administering a Test Session
9. Upon selecting a test, students wait for test administrator
approval.
10. As students select tests, the test administrator can see
approvals populate in the Approvals queue of the Test
Administrator Interface.
2015–16 CAASPP Pretest Administration Workshop
113
Measuring the Power of Learning.™
Administering a Test Session
11. The test administrator selects the Approvals (#) button to
access the Approvals and Student Test Settings screen.
2015–16 CAASPP Pretest Administration Workshop
114
Measuring the Power of Learning.™
Administering a Test Session
12. The test administrator verifies that the students have
selected the appropriate test and reviews the students’ test
settings by selecting See Details button next to each
student.
2015–16 CAASPP Pretest Administration Workshop
115
Measuring the Power of Learning.™
Administering a Test Session
2015–16 CAASPP Pretest Administration Workshop
116
Measuring the Power of Learning.™
Administering a Test Session
13. The test administrator selects Set to confirm the current test
settings and return to the list of students awaiting approval.
The test administrator still must approve the student for testing
OR
14. The test administrator selects Set & Approve to establish the
existing settings and approve the student for testing.
Note: If a student’s test settings are incorrect, test
administrators should deny the student, contact the LEA
CAASPP coordinator or test site coordinator to correct the
test settings in TOMS, and test the student at a later time.
2015–16 CAASPP Pretest Administration Workshop
117
Measuring the Power of Learning.™
Administering a Test Session
15. Upon reviewing test settings for all students, the test
administrator approves students to test on the Approvals and
Student Test Settings screen.
–
–
Select Approve to approve a specific student for testing.
Select Approve All Students to approve all students.
2015–16 CAASPP Pretest Administration Workshop
118
Measuring the Power of Learning.™
Administering a Test Session
16. After the test administrator has approved a student for
testing, the student verifies the test information and settings.
The student will select
Yes, Start My Test
to begin testing or
No if the settings
are incorrect.
2015–16 CAASPP Pretest Administration Workshop
119
Measuring the Power of Learning.™
Administering a Test Session
17. If a student has a text-to-speech accommodation or is
taking a test with ELA listening questions, he or she will
verify the audio checks through headphones.
Test with listening questions
audio check
Text-to-speech audio check
2015–16 CAASPP Pretest Administration Workshop
120
Measuring the Power of Learning.™
Administering a Test Session
18. Finally, students will see the Test Instructions and Help page
before they begin testing. Students will select Begin Test
Now to begin testing.
2015–16 CAASPP Pretest Administration Workshop
121
Measuring the Power of Learning.™
Monitoring Student Progress
•
Test administrators can monitor student testing
through the Students in Your Test Session table on
the Test Administrator Interface:
– Displays students who have logged in and been
approved for testing.
2015–16 CAASPP Pretest Administration Workshop
122
Measuring the Power of Learning.™
Pausing Tests
• Test administrators may pause an individual
student’s test in the Students in Your Test
Session table.
− This will not affect other students’ tests.
− There are different pausing rules for the CAAs.
2015–16 CAASPP Pretest Administration Workshop
123
Measuring the Power of Learning.™
General Pause Rules
•
•
•
•
If there is a technical issue (e.g., power outage or
network failure), students will be logged out, and
the tests will automatically be paused.
If a test is paused, the student must log in again to
resume testing.
Highlighted text is preserved during test sessions.
Student comments entered on a notepad are
preserved.
2015–16 CAASPP Pretest Administration Workshop
124
Measuring the Power of Learning.™
Smarter Balanced Pause Rules:
Computer Adaptive Test
•
If a CAT is paused for more than 20 minutes, the student is:
−
−
•
Presented with the test page (i.e., screen) containing the test
question he or she was last working on (if the page contains at
least 1 unanswered question) OR presented with the next test
page (if all questions on the previous test page were answered)
Not permitted to review or change any test questions on
previous test pages
If a CAT is paused for less than 20 minutes, the student is:
–
–
Presented with the test question or passage he or she was
working on when the test was paused
Permitted to answer previously shown questions within a
segment
2015–16 CAASPP Pretest Administration Workshop
125
Measuring the Power of Learning.™
Pause Rules: Performance Task
•
There are no pause rules for Performance Tasks.
− Even if a Performance Task is paused for more than
20 minutes, the student can return to the current
section and continue.
•
ELA Performance Tasks are divided into two parts.
− After a student completes the first part, he or she
cannot return to it.
2015–16 CAASPP Pretest Administration Workshop
126
Measuring the Power of Learning.™
Test Timeout Due to Inactivity
•
•
As a security measure, students are automatically logged out
of the test after 30 minutes of inactivity.
Activity means:
−
−
Selecting an answer
Using a navigation option in the test (e.g., selecting Next or
Back, using the Questions drop-down list)
Note: Moving the mouse or selecting an empty space on the
screen is not considered activity.
•
Before the system logs out, a warning message is displayed.
2015–16 CAASPP Pretest Administration Workshop
127
Measuring the Power of Learning.™
Stopping an Entire Test Session
• Test administrators can stop a test session (and
therefore, pause tests for all students in the session)
–
Select the Stop Session button in the upper-right corner
of the Test Administrator Interface.
–
An “Important!” box will appear, requesting verification to
end the session and log students out.
Select OK to continue or Cancel to keep the test
session open.
–
2015–16 CAASPP Pretest Administration Workshop
128
Measuring the Power of Learning.™
Reaching the End of the Test
1. After students answer the last question on the test,
the End Test button will appear in the upper-left
corner of the screen.
2015–16 CAASPP Pretest Administration Workshop
129
Measuring the Power of Learning.™
Reaching the End of the Test
2. Upon selecting the End Test button, an “Attention”
message will appear.
– Students will select [Yes] to proceed with reviewing
answers and submitting the test.
2015–16 CAASPP Pretest Administration Workshop
130
Measuring the Power of Learning.™
Reaching the End of the Test
3. Students may review answers before submitting a
test.
2015–16 CAASPP Pretest Administration Workshop
131
Measuring the Power of Learning.™
Reaching the End of the Test
4. When a student is ready to submit the test, he or
she will receive a confirmation screen.
2015–16 CAASPP Pretest Administration Workshop
132
Measuring the Power of Learning.™
Test Expiration Rules
•
•
•
•
•
A student’s CAT remains active until the student
completes and submits the test, the selected test
administration window ends, or 45 calendar days after
the student has begun the CAT.
A student’s Performance Task remains active for 10
calendar days after the student has begun it.
New: A student’s test expires at the end of the selected
test administration window.
New: An expired Performance Task may be extended an
additional 10 days.
New: Selected test administration window may be
extended an additional 10 days.
2015–16 CAASPP Pretest Administration Workshop
133
Measuring the Power of Learning.™
Classroom Activity
Administration Guidelines
•
•
•
All students will be assigned a classroom activity.
The classroom activity may be administered in a classroom
or any other appropriate space.
Designed to be given prior to administering the
Performance Task.
–
–
•
For ELA, 1 to 3 days prior to administration of the
Performance Task is recommended
For mathematics, administration on the day of the
Performance Task is recommended
No more than a 3-day lapse should occur between the
classroom activity and the Performance Task
administration.
2015–16 CAASPP Pretest Administration Workshop
134
Measuring the Power of Learning.™
Classroom Activity
Administration Guidelines
•
The classroom activity is designed to fit into a 30minute window.
– Time will vary according to complexity of topic and
individual student needs.
•
The site coordinator should download the assigned
classroom activity and provide to test administrators
at least 1–2 days prior.
– Any LEAs that do not have a classroom activity
assigned should administer the default classroom
activity.
2015–16 CAASPP Pretest Administration Workshop
135
Measuring the Power of Learning.™
Classroom Activity
Administration Guidelines
•
•
•
•
Computers, projectors, and other technology are
allowed but not required for the classroom activity.
The classroom activity involves the participation of all
students in an instructional task.
The Performance Task is to be completed by individual
students on the testing device.
Students may take notes during the classroom activity,
but the notes must be collected and securely destroyed.
before proceeding to the Performance Task.
2015–16 CAASPP Pretest Administration Workshop
136
Measuring the Power of Learning.™
Test Security
Measuring the Power of Learning.™
Test Security and Appeals
New: STAIRS (Security and Test Administration
Incident Reporting System) requires LEA CAASPP
coordinators and CAASPP test site coordinators to
submit an online form to report an incident.
If an appeal is required, an e-mail will be sent
instructing the coordinator to file an appeal in
TOMS.
2015–16 CAASPP Pretest Administration Workshop
138
Measuring the Power of Learning.™
Test Security
There are three different types of test security
incidents:
1. Impropriety
2. Irregularity
3. Breach
2015–16 CAASPP Pretest Administration Workshop
139
Measuring the Power of Learning.™
Test Impropriety
•
•
•
•
•
Unusual circumstance that has a low impact on the
testing individual or group of students
Low risk of affecting student performance, test security,
or test validity
Correctable and containable at local level
Must be reported to the LEA CAASPP coordinator or the
test site coordinator
– Coordinator must report the impropriety, using the
STAIRS form within 24 hours of the incident
May be reported in the appeals module should an
appeal be required
2015–16 CAASPP Pretest Administration Workshop
140
Measuring the Power of Learning.™
Test Irregularity
•
•
•
•
•
•
Unusual circumstance that impacts the testing individual or
group of students
May affect student performance, test security, or test validity
Correctable and containable at local level
New: LEAs are not required to report failure to administer the
classroom activity
Must be reported to the LEA CAASPP coordinator or the test
site coordinator
– Coordinator must report the irregularity, using the
STAIRS form within 24 hours of the incident
If needed, coordinator will be instructed to submit an appeal
in the TOMS Appeals module for resolution
2015–16 CAASPP Pretest Administration Workshop
141
Measuring the Power of Learning.™
Test Breach
•
•
•
Event that threatens test validity (e.g., release of secure
materials)
Example: Student or administrator takes a photograph
and posts to social media
External implications for the Consortium
Must be reported to the LEA CAASPP coordinator or the
test site coordinator, who must report the breach, using
the STAIRS form within 24 hours of the incident
2015–16 CAASPP Pretest Administration Workshop
142
Measuring the Power of Learning.™
Test Security Incidents
Type and Examples
Type of Security Incident
Example
Test Impropriety
Students talking during testing
Test Irregularity
Student cheating or providing answers to
one another
Test Irregularity
Student accessing or using electronic
equipment (e.g., cell phone)
Test Breach
Administrator or coordinator modifying
student responses or record at any time
Test Breach
Administrator allowing students to take
home test questions, passages, prompts,
or scratch paper
2015–16 CAASPP Pretest Administration Workshop
143
Measuring the Power of Learning.™
Test Security Resources
•
2015–16 CAASPP Test Security Webcast
Wednesday, February 24, 2016 from 1–2 p.m.
•
Smarter Balanced Online Test Administration
Manual
http://www.caaspp.org/rsc/pdfs/CAASPP.online_tam.20
16.pdf
2015–16 CAASPP Pretest Administration Workshop
144
Measuring the Power of Learning.™
Assessment Target
Reports
Measuring the Power of Learning.™
Assessment Target Reports
•
Assessment Targets:
–
–
–
•
Describe the expectations of what will be assessed by
the items and tasks within each claim.
Represent the prioritized content for summative
assessment at each grade level.
Have been used to develop more detailed item and task
descriptions through the item specification process.
Assessment Target Reports:
–
–
–
Support data-driven program evaluation.
Contribute to the investigation of a group of students’
strengths and needs.
Constitute one of many sources of evidence that should
be used in evaluating student performance.
2015–16 CAASPP Pretest Administration Workshop
146
Measuring the Power of Learning.™
Assessment Target Reports
•
•
Assessment Target Reports are reported at the
aggregate level by grade or roster but will not be
reported for individual students.
Assessment Target Reports include data from the
2014–15 test administration.
2015–16 CAASPP Pretest Administration Workshop
147
Measuring the Power of Learning.™
Assessment Target Reports
•
•
•
Target scores are reported for groups of students.
Caution should be used in interpreting results with fewer
than 50 students.
Only LEAs that complete online summative ELA and
mathematics assessments will receive target scores.
–
•
•
Assessment Target Reports are not available for LEAs
that administered paper-pencil Smarter Balanced ELA
and mathematics assessments.
Target scores are available for all four ELA claims (i.e.,
Reading, Writing, Listening, and Research/Inquiry).
Target scores are available for one mathematics claim
(i.e., Concepts and Procedures).
2015–16 CAASPP Pretest Administration Workshop
148
Measuring the Power of Learning.™
Assessment Target Reports
•
•
Assessment Target Reports will be available
through the Online Reporting System (ORS).
Users that have access to ORS will have access to
the Assessment Target Reports.
– Level of access is based on the user role assigned
to a particular user.
•
Users will be able to download Assessment Target
Reports.
2015–16 CAASPP Pretest Administration Workshop
149
Measuring the Power of Learning.™
Assessment Target Reports
• In preparation for the release of the 2015–16
Assessment Target Reports, the Assessment
Target Reports for the 2014–15 administration
will be available in February 2016.
• Additional resources that support the use of
Assessment Target Reports to inform teaching
and learning will be available in February 2016:
– Video tutorial
– Step-by-step instructions
2015–16 CAASPP Pretest Administration Workshop
150
Measuring the Power of Learning.™
Assessment Target Reports
2015–16 CAASPP Pretest Administration Workshop
151
Measuring the Power of Learning.™
Smarter Balanced
Scoring and Reporting
Measuring the Power of Learning.™
Completion Rules
• Completion of the Smarter Balanced online
assessments is dependent on the student
completing both parts of the assessment in each
content area:
1. CAT
2. Performance Task
Important: Students that do not complete both
parts will not receive an overall score.
2015–16 CAASPP Pretest Administration Workshop
153
Measuring the Power of Learning.™
Scoring Considerations:
Overall Score
•
If a student responds to fewer than 10 CAT questions
and fewer than one Performance Task question, the
student will be assigned the following for that content
area:
– Lowest Obtainable Scale Score (LOSS)
– Lowest Achievement Level (Standard Not Met–Level 1)
•
Students assigned the LOSS are counted in the
number enrolled and in the number tested, but they
are not included with the number of tests with valid
scores.
2015–16 CAASPP Pretest Administration Workshop
154
Measuring the Power of Learning.™
Scoring Considerations:
Claim Scores
• A student will receive a claim scale score if the
student responds to the majority of questions
that contribute to the claim.
2015–16 CAASPP Pretest Administration Workshop
155
Measuring the Power of Learning.™
Updates to
Scoring and Reporting
• New: Scale scores on the claims will be
included in the downloadable data files,
beginning with the 2015–16 test administration.
• Student Score Reports will only include
achievement levels for claims (i.e., Above
Standard, Near Standard, Below Standard).
2015–16 CAASPP Pretest Administration Workshop
156
Measuring the Power of Learning.™
Avoiding Scoring Delays
•
ETS may place a temporary hold on scoring a
student’s results to investigate and/or clarify
information provided; for example:
– SSID has multiple students’ test results.
– Student has more than one answer document or test
results for the same test.
– Student answer document was returned incomplete.
– Student record was not found in TOMS.
•
If a temporary hold is placed on a student score, it
will delay the reporting of results in ORS, Student
Score Reports, and data files.
2015–16 CAASPP Pretest Administration Workshop
157
Measuring the Power of Learning.™
Online Reporting System
•
•
•
•
New: Addition of CAAs scores in the fall 2016
New: Expanded to include additional demographic
fields (i.e., same fields that are included in CAASPP
Web reporting site) for aggregate reporting
New: Ability to create rosters at the school level for
aggregate reports and comparisons among groups
of students
New: Availability of aggregate Assessment Target
Reports for online summative ELA and
mathematics assessments as scores are loaded
into ORS
2015–16 CAASPP Pretest Administration Workshop
158
Measuring the Power of Learning.™
Online Reporting System
• Preliminary scores and aggregate reporting:
– The Online Reporting System (ORS) will include
scores for online summative assessments for ELA
and mathematics and paper-pencil scores for
Science (CST, CMA, CAPA) and STS for RLA.
– Online summative scores for a content area will be
in ORS within 3 weeks of a student having
completed the CAT and Performance Task for that
content area.
– Paper-pencil scores will be in ORS within 5 weeks
of a student’s answer document released for
scoring.
2015–16 CAASPP Pretest Administration Workshop
159
Measuring the Power of Learning.™
Score Reporting
• Scores for reporting will be released once all
accuracy checks are completed.
• Early testing LEAs likely will receive preliminary
scores later than the timelines described in the
previous slide.
2015–16 CAASPP Pretest Administration Workshop
160
Measuring the Power of Learning.™
2015–16 Student Score Reports
• A revised design of the Student Score Reports
will be reviewed and submitted for approval by
the State Board of Education in January.
• The proposed Student Score Report document is
available on the January 2016 CDE State Board
of Education as a Web document at
http://www.cde.ca.gov/be/ag/ag/yr16/documents/j
an16item04a1.pdf.
2015–16 CAASPP Pretest Administration Workshop
161
Measuring the Power of Learning.™
Sample Student Score Reports
2015–16 CAASPP Pretest Administration Workshop
162
Measuring the Power of Learning.™
Student Score Report Updates
•
•
•
The CAA and CAPA Student Score Reports will be
provided separately for the 2015–16 administration and
will be available in fall 2016.
New: Option to select Spanish Student Score Report at
the student level in TOMS:
– If selected, the student’s report will be available in
English and Spanish, except STS, which will be
provided in Spanish only.
– Both the initial and final versions will be created in both
English and Spanish and will be available in TOMS to
download and print locally.
If parents/guardians have exempted their child from all
2015–16 CAASPP tests, no Student Score Report will be
created.
2015–16 CAASPP Pretest Administration Workshop
163
Measuring the Power of Learning.™
Initial Student Score Reports
• New: Initial Student Score Reports will be
provided in downloadable PDF files in TOMS 1
week after the student’s last score is reported in
ORS.
• Initial Student Score Reports are created during
the test administration window as the student
completes the last assigned test.
2015–16 CAASPP Pretest Administration Workshop
164
Measuring the Power of Learning.™
Initial Student Score Reports
•
Initial Student Score Reports will not be created for
incomplete student records, such as:
–
–
–
–
–
Student has an open appeal.
Student who did not complete both parts (CAT +
Performance Task) for a content area.
Student has an expired test.
Student records receive a special scoring code by ETS
for reporting purposes (Not Tested, NEL-Student is
exempt due to English learner status, SCL-Student
completed a test and the grade level does not match the
grade of enrollment).
Student score is temporarily held by ETS for investigation
and/or clarification.
2015–16 CAASPP Pretest Administration Workshop
165
Measuring the Power of Learning.™
Final Student Score Reports
•
•
•
•
Final Student Score Reports will be delivered to LEAs in
printed format for distribution to parents/guardians.
Student Score Reports must be delivered to
parents/guardians within 20 days of receipt of the first
shipment (California Code of Regulations, Title 5,
Section 863).
Electronic packaging list will be available in TOMS.
Final Student Score Reports will be packed by school,
grade, and alphabetical order. Each group will be
separated by folders in the box.
2015–16 CAASPP Pretest Administration Workshop
166
Measuring the Power of Learning.™
Sample Reporting Timeline
2015–16 CAASPP Pretest Administration Workshop
167
Measuring the Power of Learning.™
Scoring Progress Dashboard
TOMS will have a dashboard displaying the progress of scoring for
your LEA’s administration
•
At 80% of student scores completed
for the LEA’s administration, the
LEA CAASPP coordinator will
receive a notification to carry out the
following:
– Complete testing for students
with processed appeals.
– Verify condition codes are and
enter any remaining condition
codes.
– Set Spanish Student Score
Report indicators.
– Select parent/guardian address
(if desired).
• At 90% of student scores
completed, the final
Student Score Reports will
be printed, packaged, and
delivered to LEAs.
• At the end of scoring, all
remaining final Student
Score Reports will be
delivered in a separate
shipment.
2015–16 CAASPP Pretest Administration Workshop
168
Measuring the Power of Learning.™
Public Web Reporting
• Summative ELA and mathematics Web site will
be expanded to include:
– Comparisons across entities—LEAs or schools
– Multi-school year data provided for 2014–15 and
2015–16 aggregate scores
– CAAs added to site and score data, to be made
available in fall 2016
2015–16 CAASPP Pretest Administration Workshop
169
Measuring the Power of Learning.™
Data Explorer
•
•
•
•
New analytical tool available fall 2016
Rollout to LEA CAASPP coordinators
Sign on through TOMS for secure access
Ability to run quick cross-tab aggregate reports
–
A cross-tab report shows the relationship
between two or more data elements (e.g., report
of ELA scale scores by gender and ethnicity).
• Ability to build custom reports or create graphs
• Comparisons across entities—schools, LEAs,
counties, and state-level
2015–16 CAASPP Pretest Administration Workshop
170
Measuring the Power of Learning.™
Digital Library
Measuring the Power of Learning.™
Digital Library Registration
•
•
•
LEA CAASPP coordinators are responsible for providing
access to LEA and nonpublic school staff who provide
services to their students.
LEA CAASPP coordinators and test site coordinators create
and manage users and manage LEAs on the Administration
and Registration Tools (ART) at
https://art.smarterbalanced.org
– Upload staff records individually or in large groups.
– Provide required information: first name, last name,
e-mail, role.
Questions? LEA CAASPP coordinators can contact the
CalTAC Digital Library and Interim Assessment Help Desk
at caltac@ets.org or 855-631-1510.
2015–16 CAASPP Pretest Administration Workshop
172
Measuring the Power of Learning.™
Resources for Promoting Use
of the Digital Library
The following resources are available on the CDE Digital
Library Web Page at http://www.cde.ca.gov/ta/tg/sa/diglib.asp.
• Videos
–
–
–
•
Introducing the Smarter Balanced Digital Library
Quality Criteria for the Digital Library
Exploring the Smarter Balanced Digital Library
Smarter Balanced Digital Library Spotlight Webinars and
Forums
–
Features Webinars presented by resource authors; followed by
forum discussions facilitated by authors
2015–16 CAASPP Pretest Administration Workshop
173
Measuring the Power of Learning.™
Resources for Promoting Use
of the Digital Library
These resources are available on the CDE Digital Library Web
Page at http://www.cde.ca.gov/ta/tg/sa/diglib.asp.
• Documents
–
–
–
–
The Formative Assessment Process (PDF)
Digital Library Fact Sheet (PDF)
Digital Library Overview and Sample Resources (PDF)
Digital Library Introduction for Teachers (PPT)
 Includes script
Another useful resource can be found on the CAASPP Portal
Web site at http://www.caaspp.org/
– Smarter Balanced Digital Library and Interim Assessment
Clinic Presentation (PDF)
2015–16 CAASPP Pretest Administration Workshop
174
Measuring the Power of Learning.™
Strategies to Encourage Use of
the Digital Library
•
•
•
•
Spend time working with site coordinators, district
staff, and teachers to familiarize them with the
Digital Library.
Dedicate time during staff meetings and/or
professional development time to explore the
Digital Library.
Dedicate time to review the resources provided on
the CDE Digital Library Web Page and materials
covered today.
Implement formative assessment processes and
practices for grade-level teams and school sites.
2015–16 CAASPP Pretest Administration Workshop
175
Measuring the Power of Learning.™
Interim Assessments
Measuring the Power of Learning.™
Overview of the Smarter
Balanced Interim Assessments
Two types of interim assessments:
• Interim Comprehensive Assessments (ICAs)
– ELA, mathematics, and Performance Tasks for each
content area
– Look and feel like the summative assessments
– Between 35 and 50 questions per computer-based fixed
form
• Interim Assessment Blocks (IABs)
– Shorter and more focused on specific content
– Designed to allow flexibility with individual pacing plans
– Between 5 and 20 questions per IAB
2015–16 CAASPP Pretest Administration Workshop
177
Measuring the Power of Learning.™
Benefits of Using
the Interim Assessments
• Evaluate progress and plan adjustments to
increase student learning.
• Experience content and questions reflective of the
summative assessments.
• Excellent opportunity to try out accessibility
supports with students and actual test questions.
• Build teacher comfort with the process of online
scoring.
2015–16 CAASPP Pretest Administration Workshop
178
Measuring the Power of Learning.™
Using Interim Assessment
Results
Interim assessment results may help inform:
• Progress on the California State Standards (when
combined with other information)
• Identification of specific areas of strength and
weakness
2015–16 CAASPP Pretest Administration Workshop
179
Measuring the Power of Learning.™
Using Interim
Assessment Results
Considerations in interpreting interim assessment
results:
•
Extent to which:
–
–
–
•
•
•
Administration was standardized.
Students had been exposed to questions.
Students, especially those with disabilities and English learners,
used appropriate accessibility supports during assessment(s).
Scores and achievement levels cannot be compared with data
from STAR.
Interim assessment results should be used only in combination
with other available information.
Interim assessment results should not be the sole basis for
making any high-stakes decisions.
2015–16 CAASPP Pretest Administration Workshop
180
Measuring the Power of Learning.™
Interim Assessment
Viewing System
2015–16 CAASPP Pretest Administration Workshop
181
Measuring the Power of Learning.™
Interim Assessment Viewing
System
2015–16 CAASPP Pretest Administration Workshop
182
Measuring the Power of Learning.™
Interim Assessment
Resources
•
•
•
•
Available on the CDE CAASPP Interim Assessments Web page at
http://www.cde.ca.gov/ta/tg/sa/sbacinterimassess.asp
– IAB blueprints
– Number of questions in ICAs and IABs that require hand scoring
– Estimated times for hand scoring questions
Available on CAASPP Web page
– Training Guides and Exemplars
http://www.caaspp.org/ta-resources/interim.html
– Archived Webcasts http://www.caaspp.org/training/caaspp
 Smarter Balanced Interim Assessments: Part I (January 2015)
 Smarter Balanced Interim Assessments: Part II (January 2015)
Interim Assessment Video Series Coming Soon!
Interim Assessment User Guide Coming Soon!
2015–16 CAASPP Pretest Administration Workshop
183
Measuring the Power of Learning.™
Interim Assessment
Resources
•
Available in TOMS
– Hand Scoring Videos and Presentation Slides
 Interim Assessment Hand Scoring Workshop—ELA
(October 2015)
 Interim Assessment Hand Scoring Workshop—Mathematics
(October 2015)
 Interim Assessment Hand Scoring System Webinar
(April 2015)
 Hand Scoring Demonstration—Grade 6 ELA (April 2015)
 Hand Scoring Demonstration—Grade 7 Mathematics
(April 2015)
2015–16 CAASPP Pretest Administration Workshop
184
Measuring the Power of Learning.™
Questions
Measuring the Power of Learning.™
Help Desk Support
The California Technical Assistance Center
(CalTAC) is there to support all LEA CAASPP
coordinators!
Monday–Friday from 7 a.m.–5 p.m.
E-mail: caltac@ets.org
Phone: 800-955-2954
Web site: http://www.caaspp.org
2015–16 CAASPP Pretest Administration Workshop
186