Test Chairpersons’
Orientation
Miami-Dade County Public Schools
September 27, 2012
Agenda
General Session (8:15 – 11:30 a.m.)
Welcome
Dr. Sally A. Shay, Student Assessment
Accountability: School Grades and Annual Yearly
Progress
Gisela Feild, and Vilma Rubiera, Assessment, Research,
and Data Analysis
2012-13 Testing Calendar
Updates to the Testing Programs
VPK Assessment
Dr. San Juanita de la Cruz, Early Childhood Education
Agenda
General Session (Continued)
Test Security, Reporting Violations, and Responding
to State Inquiries
District Investigations:
Detective Louis Nunez, Miami-Dade School Police
General Session (12:30 – 1:30 p.m.)
Introduction to File Download Manager
Tiffany Davis, American Senior High School and Betty
Viada, Miami Lakes Educational Center
Agenda
Breakout Sessions (1:45 – 3:30 p.m.)
Test Chair 101: Basics on preparing test materials
Zulema Lamazares, Henry Flagler Elementary
Edusoft: Testing Platform for Baseline and Interim
Assessments
Rudy Azcuy, Instructional Technology
On-line CELLA: Administration for ESOL Placement
Deland Innocent, Bilingual/World Languages
Accountability
Gisela Feild, Administrative Director,
Assessment, Research and Data Analysis
and Vilma Rubiera, Project Manager, Data Analysis
2012-13 Testing Calendar
http://oada.dadeschools.net/TestingCalendar/TestingCalendar.asp
2014 and Beyond
7
Program Updates:
Florida Comprehensive Assessment Test
(FCAT/FCAT 2.0)
and
End-of-Course (EOC) Assessments
Next Generation Sunshine State Standards
(NGSSS)
Maria C. Bruguera, Director I
Mara Ugando, Staff Specialist
FCAT 2.0 and EOC Updates
 Phasing in FCAT 2.0 Writing
 Phasing-in FCAT 2.0 Reading computer-based test (CBT) in
Grades 7 and 9
 Phasing in FCAT 2.0 Mathematics computer-based test (CBT)
in Grade 5
 FCAT (SSS) Reading Retake (graduation test) will not longer be
offered in the 2012-13 school year and beyond.
 Civics End-of-Course (EOC) will be field tested in selected
middle schools
EOC Requirements
 EOC Requirements based on cohort year:
Algebra 1, Biology 1, and Geometry EOC Assessments will be administered in
grades 6 -12, to enrolled students in an eligible courses in winter, spring, and
summer.
Grade 9 students in 2012-13 and younger: must pass the Algebra 1,
Biology 1, and Geometry EOC assessments to earn high school credit.
Grade 9 in 2011-12 (Grade 10 students): must pass the Algebra 1 EOC
to earn high school credit. In addition, the Biology 1 and Geometry EOC
scores will count as 30% of the final course grade.
Grade 9 in 2010-11 (Grade 11 students): the Algebra 1 EOC score will
count as 30% of the final course grade.
The US History EOC will count as 30% of the final course grade for ALL
students in Grades 9 -12 enrolled in an eligible course. This 30% rule will
remain in effect this school year and beyond.
Transition Schedule
2012–2013 Computer-Based Testing
System Minimum Specifications
General Specifications (Windows/PC and Apple/Macintosh)
Each computer must meet or exceed the following General Specifications in addition to
the applicable Operating System Specifications below:

500 MB available disk space

1024x768 screen resolution

Keyboard and mouse (or other pointing device)

High-speed internet connection
Operating System Specifications
Windows/PC
Windows XP
(Service Pack 3)
Windows (Service
Pack 1)

Pentium III
733 MHz
processor

512 MB RAM


Apple/Macintosh
Pentium III
1 GHz
processor
Windows 7

Pentium III
1 GHz processor

1 GB RAM
1 GB RAM
Mac OS X 10.4
Mac OS X 10.5
Mac OS X 10.6



Intel processor

1 GB RAM

G3, G4, G5 500
MHz processor
512 MB RAM

Intel, G4, G5
867 MHz
processor
512 MB RAM
Certification Tool
 The Certification Tool is to be completed by each school that will
offer computer-based assessment(s) in the 2012-13 school year.
(Please refer to Weekly Briefing #12320.)
 Six sections to complete online:
I. Computer Specifications
II. Student Scheduling Plan
III. Annual Checklist
IV. Administration – Specific Checklist
V. School Computer-Based Assessment Team
VI. Submit to District Office
 The Certification Tool Webcast Training is available at:
http://oada.dadeschools.net/Screencasts/2013EOC/201213CBTSCREENCAST.html.
Certification Tool, cont.
 The Certification Tool is to be completed twice in the 2012-13
school year.
One Certification Tool for FALL and WINTER assessments:
 Fall 2012 FCAT/FCAT 2.0 Retake: Reading and Mathematics
 Winter 2012 EOC Assessments: Algebra 1, Biology 1, and
Geometry
One Certification Tool for both SPRING assessments:
Spring 2013 FCAT /FCAT 2.0 Retake: Reading and Mathematics
Spring 2013 FCAT 2.0 Reading and Mathematics
Spring 2013 EOC Assessments: Algebra 1, Biology 1, Geometry,
US History, and Civics Field Test (selected schools only)
PearsonAccess
PearsonAccess – www.pearsonaccess.com/fl
A website used for almost all test preparation (e.g., PreID), setup,
administration, and reporting tasks.
Correct test administration must be selected each time you log in.
Requires a username and password.
Links to support materials:
www.FLAssessments.com/
PearsonAccess will timeout after 14 minutes of inactivity.
PearsonAccess, cont.
PearsonAccess Training Center
Accessed from the Training Center tab
on PearsonAccess Home page.
Provides an opportunity to practice
PearsonAccess tasks.
Manage the Infrastructure Trial in
preparation for testing.
Requires a username and password.
TestNav 6.9
Platform for Florida’s high-stakes
computer-based statewide
assessments.
Software application installed
either on a file server or on each
computer that will be used for
testing.
Secure, locked-down browser.
Runs on tablets (except for iPads)
but not recommended.
Note: This is the URL used for testing
this Spring; it is different from the URL
used for the Infrastructure Trial.
TestHear
 Platform used to deliver accommodated forms (e.g., large
print, zoom, color contrast, screen reader, assistive devices) for
Florida’s high-stakes computer-based statewide assessments.
Must be installed on all computers that will be used by
eligible students requiring accommodated CBT forms.
Local installation is required.
New updated version must be
installed.
Available for download at
www.FLAssessments.com/downloads.
Note: Audio card and headphones are required for the computers that will
run the Screen Reader accommodation.
Proctor Caching
Proctor Caching is a process of loading or “caching” test content locally
on a computer at the school level.
Does not require a separate caching server and can run on any
workstation on the network that meets minimum requirements.
Proctor Caching software is provided by Pearson.
Reduces test delays due to network congestion.
Provides students with a more seamless testing experience.
Required for all FCAT/FCAT 2.0/EOC computer-based testing in
Florida.
Infrastructure Trial
 The Infrastructure Trial simulates a computer-based test using
mock test items that will allow schools to test hardware and
network capacity prior to the CBT operational administration.
It should take approximately thirty minutes for users to log in,
supply random answers, and submit the test.
Please note that the content of the test used for the
Infrastructure Trial is NOT intended for student practice.
 Schools must set up the Trial to use all computers scheduled to be
used concurrently during the Fall / Winter / Spring CBT
administrations.
The Infrastructure Trial is managed in the Training Center.
 TestHear accommodated forms are available for the Infrastructure
Trial. These forms are larger (50 MB) than regular TestNav forms
(2 MB).
Practice Test (ePat)
 Electronic Practice Assessment Tool
Provides students an opportunity to practice using the
computer-based platform prior to testing.
Available for download at www.FLAssessments.com/ePATs.
Download ePAT launcher separately from test content.
Accommodated ePATs are also available. Launcher is bundled
with the practice test content for accommodated forms.
Scripts and instructions for downloading ePATs are under the
Resources tab.
Student Tutorials
www.FLAssessments.com/ePATs
 Resources tab
Provide additional exposure to the
format of computer-based
assessments
Viewing the tutorials does not
replace the required practice test
(ePAT) sessions
Student Tutorials, cont.
District Level Certification
Form
Current policy states that
individuals who handle secure
materials for statewide
assessments (FCAT/FCAT
2.0/EOC) must have state- or
district-level teaching
certifications.
If there are instances in
which a non-certified person
will assist the test chairperson
with preparing test materials,
the District-Level
Certification Form must be
completed and submitted for
approval.
*
Kathleen Sierra, Supervisor
26
RttT District Developed
Assessments
for
Hard
Measure
Content
 Funded
throughto
the Race
to the Top (RttT)
Grant – Three Areas
Year Project
Goals of the Initiative
Develop high quality, standards-based assessments in non-core content
areas
Provide Florida public and charter school districts with an extensive
bank of assessment items
High Quality, standards-based
Field tested
Vetted by Florida Educators
Once Item bank is fully operational, Florida Districts, schools, and teachers
will access to the bank to generate assessments to meet their needs.
RttT District Developed Assessments
for
Hard to Measure Content Areas (cont.)
How were courses chosen?
Through student enrollment and teacher assignments
Content Areas being developed:
Career and Technical Education
Health Education K-12
Physical Education K-12
Performing Arts
Visual Arts
World Languages
RttT District Developed Assessments
for
Hard to Measure Content Areas (cont.)
Timeline:
2011-12 and 2012-13 school years
Develop Item Specifications
Develop Test Items, Blueprints, and test forms
2013-14 school year
Field test
2014-15 school year
Item Bank fully operational for all Florida public and charter
school Districts
29
Interim Assessment Program
Formative Classroom Assessment
Felicia Mallory, Executive Director
Denetra Collins, Staff Specialist
Purpose of the Baseline
Benchmark Assessments
(BBA)
To determine students’ baseline knowledge of the Next Generation
Sunshine State Standards in Reading, Mathematics, Science and Social
Studies.
 To provide teachers with a quick snapshot of students’ initial
knowledge and abilities to focus instruction.
 To provide for state required progress reports in schools classified
under the Differentiated Accountability System as Focus, Focus DD,
Focus DDD, Prevent, and Priority.
 Program guide and instructions for downloading the assessments are
available on our website at http://oada.dadeschools.net/IAP/IAP.asp
Purpose of Interim
Assessment
 To monitor student progress on the Reading, Mathematics,
Tests
Science and Social Studies Next Generation Sunshine State
Standards (Fall, Winter, and Spring)
 To provide teachers with classroom assessment tools that will
provide reliable student-level benchmark information to inform
instruction
 To provide students with information on their progress on specific
benchmarks
 To provide schools with a reliable and valid assessment tool for
state progress reporting.
Overview of the Interim
Assessment Program
 Interim Assessment tests administered three times per year
 BBA, fall, winter, and spring
 Alignment with the Next Generation Sunshine State Standards
and District’s pacing guides
 Benchmark Assessment Item Banks for Reading and
Mathematics are available in ExamView
 Data analysis and disaggregation, score reports, and item
analysis are available in Edusoft
Interim Assessment Administration
Windows 2012-2013
Administration Windows
BBA
August 20, 2011 – September 21, 2012
Fall
October 29 – November 9, 2012
Winter
January 9 – January 20, 2012
Spring
March 18 – March 22, 2012
-
Subjects and Grade Levels
Reading : Grades 3-10,
Mathematics: Grades 3-8; Algebra 1 , Geometry,
Science: Grades 5, 8, and Biology 1
United States History
Civics
Reading : Grades 3-10
Mathematics: Grades 3-8; Algebra 1, Geometry,
Science: Grades 5, 8, and Biology 1
-
Reading : Grades 3-10,
Mathematics: Grades 3-8; Algebra 1, Geometry,
Science: Grades 5, 8, and Biology 1
United States History
-
United States History
Civics
Students to be Tested
 All eligible students* must take the Interim Assessment in
Reading, Mathematics, and Science.
Reading, Grades 3-10,*
Mathematics, Grades 3-8*
Science: Grades 5 and 8*
All students enrolled in:
Algebra I**
Geometry**
Biology I**
U.S History**
Civics**
*Students MUST take the test corresponding to the grade level in which they are
listed in ISIS (unless they are enrolled in an EOC course in same content area).
**Or an equivalent course (see Appendix D, page 42 of the CBT Supplemental
Program Guide)
Administration Format
Content- Area
Paper-Based Tests
Computer-Based Tests
Reading
Grades 3-5, and 8
Math
Grades 3, 4, 6-8
Grades 6, 7, 9, 10 , and
Retake
Grade 5
Algebra I
Geometry
Science
Grades 5, 8
Biology I
Social Studies
U.S. History*
Civics
U.S. History*
*Paper-based for Baseline only, computer-based in subsequent administrations.
Approximate Testing Times
Before the Test
Approximately 10
minutes to pass
out testing
materials
Administration Time
Reading*:
Approximately 70 minutes
Mathematics*:
Approximately 75 minutes
After the Test
Approximately 10
minutes to collect
testing materials
Science*:
Approximately 112 minutes
Civics
Approximately 60 minutes
U.S History*
Approximately 90 minutes
* Longer assessments may be administered over a two-day time period.
Performance Levels
The Performance Levels for grades 3-10 reading, 3-8 mathematics, and algebra are
outlined below.
Satisfactory Progress
Students who are
likely to score in
FCAT levels 3 and
above
Limited Progress
Students whose
performance was
not sufficient to
predict success
Insufficient Progress
Students who are
likely to score
below FCAT level
3
This student demonstrated a satisfactory level of achievement on the
content focus of the Next Generation Sunshine State Standards assessed
during this instructional period. To attain high levels of achievement in
this content area, the student must receive continued instruction on the
challenging content and skills across the benchmarks designated for this
grade level.
This student demonstrated a limited level of achievement on the content
focus of the Florida Sunshine State Standards assessed during this
instructional period. To attain high levels of achievement in this content
area, the student must receive targeted interventions and remediation in
the areas of concern, and continued instruction on the challenging content
and skills across the benchmarks designated for this grade level.
This student demonstrated an insufficient level of achievement on the
content focus of the Florida Sunshine State Standards assessed during this
instructional period. To attain high levels of achievement in this content
area, the student must receive intensive interventions and remediation in
the areas of concern, and continued instruction on the challenging content
and skills across the benchmarks designated for this grade level.
Score Reports
Useful reports that can be retrieved via Edusoft:
Class List Reports
Item Analysis Report
Item Response Report
Student Performance Report
Subgroup Report
Custom Group Report
Performance Band Report
Benchmark Assessment Item Bank
 To allow teachers to monitor student progress on the Reading and
Mathematics Next Generation Sunshine State Standards
 To provide teachers with classroom assessment tools that will provide
reliable student-level benchmark information to inform instruction
 To provide students with information on their progress on specific
benchmarks
Item Bank Resources:
Item Bank Overview
Item Bank Newsletter
Download Examview Test Generator
Comprehensive English Language
Learners (CELLA)
Felicia Mallory, Executive Director
Denetra Collins, Staff Specialist
Overview of CELLA
CELLA is a four-skill English language assessment measuring
student proficiency in listening, speaking, reading, and writing.
Students in grades 1 – 12 are administered the
Listening, Reading, and Writing sections in groups
Speaking on a one-on-one basis
Kindergarten students are assessed in all four areas on a one-to-one
basis.
Unlike the FCAT, CELLA is not a timed test.
Purposes of CELLA
The test results can provide:
Data useful for charting student progress over time,
Diagnostically useful information about students’ strengths
and weaknesses in English,
The language proficiency levels of individual students can
be used in making decisions regarding placement into, or exit
from English for Speakers of Other Languages (ESOL)
programs.
Identifying the proper ESOL level,
Determining whether the District and schools met Annual
Measureable Achievement Objectives (AMAOs)
Annual Measurable Achievement
Objectives (AMAOs)
AMAO 1:
Progressing in English language acquisition
annual increases in the number or percentage of students
making progress in learning English
AMAO 2:
Exiting or reaching English language proficiency
annual increases in the number or percentage of students
attaining English language proficiency by the end of each
school year
AMAO 3:
ELL-Adequate Yearly Progress (AYP)
AYP for the ELL subgroup in demonstrating proficiency on
grade-level in reading and mathematics on the FCAT
Important Dates for 2013
CELLA
Administration:
Individual Group: March 4 - April 5, 2013
National Assessment of
Educational Progress (NAEP)
Felicia Mallory, Executive Director
Denetra Collins, Staff Specialist
Features of NAEPTUDA
 Provides district-level data to participating urban districts
 Provide a common yardstick to measure student achievement in
comparison to
Other urban districts
States
Large Central Cities
National Public Schools
Census Regions
Purpose of the National
Assessment
of Educational Progress
 A common nationwide measure of student performance
(NAEP)/TUDA
 Reports changes over time.
 Provides results for the nation, states, and some urban districts.
 Secondary indicator for state progress and allows national
comparisons between states and large urban districts
 Used by the President, Congress, and state leaders to set
education policy
Overview of the NAEP
 A representative sample of schools and students are selected.
 Randomly selected students in grades 4,8, and12 are assessed
in reading, mathematics, economics, and writing.
 Assessments include multiple-choice and constructed
responses.
 Assessments are administered by NAEP field staff
 Assessments are 90 to 120 minutes, which include testing
and answering background questions
 Accommodations are offered for ELL and SWD students.
 Results are reported by scale scores, subscales, achievement
levels, subgroups, and geography
2013 Assessment Components
Operational
Components
Reading
Mathematics
Grades/
Ages
4, 8, 12
4, 8, 12
Pilots
Grades
Reading
4, 8
Mathematics
4, 8
Technology and
8
Engineering Literacy (TEL)
Special Research Studies
Varies
NAEP 2013 Assessment
Window
January 28,2013
to
March 8, 2013
Important Dates for NAEP
2013
Event
Date
Provide School Information October 11, 2012
Form registration
NAEP field staff will
contact schools to schedule
an assessment visit
Mid-December, 2012
Pre-Assessment packet
arrives in selected schools
January 2, 2013
Pre-Assessment visits
January 2 – February 15, 2013
NAEP Testing
Administration Window
January 28 – March 8, 2013
NAEP-TUDA 2013 Release
Date
Subject and Grade
Math – Grade 4 and 8
Reading – Grades 4 and 8
Release Date
November 2013
Program for International
Student Assessment (PISA)
International assessment sponsored by the National Center for
Education Statistics (NCES) of the United States Department
of Education.
Measures students’ mathematical, scientific, and reading
literacy and their preparedness for adult life.
Assesses students who are about age 15.
Student must be born between July 1, 1996 and June 30,
1997.
Forty-two students are sampled per selected school to take a
two-hour assessment of math, reading, and science questions.
Includes a student questionnaire.
Florida Competency on
Personal Fitness (FCEPF)
Denetra Collins, Staff Specialist
Stanford Achievement Test,
Tenth Edition (SAT-10)
Mayda Cabeza, Director I
SAT10
 What is the SAT-10?
 Who is administered the SAT-10?
Grades K - 2, district-wide
 Important dates to remember
April 9-12, 2013
SAT-10
 How do SAT-10 materials arrive to schools?
 Is the SAT-10 a secure test?
 SAT-10 Test Chairperson Training
 February 2013
 School level training of test administrators and
proctors
 Test Administration
 Untimed
SAT-10 Results
 School Listing of Results (June)
Must be picked up at ITS
 Individual Student Reports (August)
Delivered to schools
Results
Not available on the Parent Portal
Posted to SPI under the tab FCAT Scores
Uses
 Referral of students to the district’s gifted program
(Plan A or Plan B)
 Early identification of students who would be
considered at high risk for problems associated with
reading comprehension
Grade 3
Opportunities for Promotion
Grade 3
Opportunities for
Promotion
 Section 1008.25, Florida Statutes (F.S.), Eliminating Social Promotion.
Spring FCAT 2.0 Reading
Spring Grade 3 Reading Student Portfolio
Alternative Standardized Reading Assessment (ASRA)—ITBS,
Level 9, administered the last two weeks of school
Alternative Assessment for Grade 3 Promotion (AAGTP)—SAT10, Primary 3, administered in Summer School
Grade 3 Mid-Year Promotion (GTMYP)—ITBS, Level 10,
administered in November
Grade 3
Reading Student Portfolio
 Administered during the 2nd semester
 Participants: All grade 3 students complete a portfolio
 Scored at the school site
 Represents the 2nd opportunity for a student to be promoted to
grade 4
 Student must demonstrate mastery of the Next Generation
Sunshine State Standards (NGSSS)
 Training conducted via Screencast
Alternative Standardized
Reading Assessment (ASRA)
 Administered the last two weeks of schools.
 Participants:
Students scoring FCAT 2.0 Reading level 1 , students who did not
participate in the Spring FCAT, and
Students without a passing Reading Portfolio
 Represents the 3rd and last opportunity within the school year prior
to retention.
 This administration of the ITBS, Level 9 is scanned via Edusoft
therefore results are available immediately.
 Students must score at or above a given cut score on the ITBS to be
considered for promotion.
 Training conducted via Screencast
Alternative Assessment for
Grade 3 Promotion (AAGTP)
 Administered at the end of the summer Reading camp.
 Participants: Retained grade 3 students
 SAT-10, Primary 3 is scored locally through ITS
 Represents 4th opportunity for students to be promoted to grade 4
 Students must score at or above a given cut score on the SAT-10 to be
considered for promotion
Training conducted via Screencast
Grade 3
Mid-Year Promotion (GTMYP)
 Administered mid-November
 Participants: Students who “just missed” passing the AAGTP, or
other students at the principal’s discretion
 ITBS, Level 10 is scored at the school site via Edusoft
 Represents 5th opportunity for retained 3rd grade students to be
promoted to grade 4
 Students must have a passing score on the ITBS, representing 4th
grade work.
The Iowa Tests
Iowa Tests of Basic Skills (ITBS),
Grades K-8
and
Iowa Tests of Educational Development
(ITED), Grades 9-12
Iowa Tests
 Used for
Plan B Gifted Placement;
Extension of Services in the ESOL program;
Optional for Charter Schools Only, ESOL program placement
 Scanned and scored via Edusoft
 Download a copy of the Iowa Tests Guidelines at:
http://oada.dadeschools.net/TestChairInfo/InfoForTestChair.asp
HB1255 Section 1008.30, F.S.
 High schools must evaluate the college readiness of each student prior
to grade 12. Eligible grade 11 students with the following criteria
should test.
Student who score a level 2 or 3 on the reading portion of the
Grade 10 FCAT 2.0
Students who score Level 2, 3 or 4 on the Algebra I EOC in
Grade 10 (New).
 Results of the assessment will be used to advise students.
 (New) Spring 2013, FLDOE will conduct a special concordance study
between the Algebra I EOC and PERT Mathematics.
Select statewide sample in grades 6-10 (cohort required to pass)
Select students will take both assessments giving them
opportunities to earn a passing score once concordant scores are set.
What is the P.E.R.T.?
 Postsecondary Education Readiness Test
 Customized for Florida and aligned to Florida’s Postsecondary
Readiness Competencies
 Readiness Scores
Reading, 104
Writing, 99
Mathematics, 113
P.E.R.T. Components and
Design
 P.E.R.T. Placement Test (3 sub-tests) identifies appropriate course
placement in Reading, Writing and Mathematics
 Computer Adaptive
Larger font available for accommodation
 30 questions per subtest
25 operational items for placement score
5 field test items
P.E.R.T. Implementation in
High
Schools
 Testing window begins February 2013.
 Must test eligible grade 11 students. However, students may
demonstrate college-readiness via the ACT, SAT, or CPT.
 Students who do not demonstrate college readiness MUST
enroll in the appropriate postsecondary course in their senior
year.
 Students may not retake the PERT until the postsecondary
coursework is completed. Students have only 2 attempts to take
the PERT during their high school career.
 Schools who have an overage of test units may use them to test
select students for Dual Enrollment.
P.E.R.T. Score
Repository
 Roll out in the Fall 2012. Fully operational by November
 Will house all PERT scores
 Miami-Dade College (MDC) will be able to access PERT
test scores for students participating in Dual Enrollment
and incoming students interested in enrolling at MDC after
graduation.
 Miami-Dade County Public Schools will continue to
periodically upload PERT test scores on SPI for school
access.
Florida Voluntary
Prekindergarten (VPK)
Assessment
Office of Early Childhood Programs
Dr. San Juanita de la Cruz,
District Supervisor
(305) 995-7645
Test Security, Reporting
Violations, Responding to
State Inquiries, Caveon Data
Forensics
Dr. Sally A. Shay, District Director
Security
 Standards, Guidelines, and Procedures for
Test Administration and Test Security
available at
http://oada.dadeschools.net/TestChairInfo
/InfoForTestChair.asp
 Adopted by School Board
 General Guidelines
 Florida Test Security Statute and Rule
General Security Guidelines
DO
 Provide secure storage
 Restrict access
 3 or fewer keys
 Handle secure materials, including
affixing of labels in a limited-access area
 Secure visual barriers on computer
workstations
 Maintain documentation
 Chain of custody
 Book assignment to students
 Train test administrators and proctors
 Monitor students during lunch break or
extended breaks
 Visual barriers for computer-based
testing
DO NOT
 Allow unauthorized access to materials
or content
 Open secure materials prior to date
allowed
 Allow untrained or uncertified staff to
test students
 Coach or provide assistance during
test administration
 View, review, copy, or debrief test
content
Reporting Irregularities and
Security Violations
 All school staff are responsible for ensuring appropriate
administration procedures and reporting any irregularities.
 Reporting procedures
 Specified in Standards, Guidelines, and Procedures for Test
Administration and Test Security
 Test Chairperson / Principal
 Student Assessment and Educational Testing
 Regional Center or District Administrative Office
 Appropriate investigative unit
Caveon Data Forensics
 The FLDOE has contracted with Caveon Test Security to provide
its Caveon Data Forensics™ for all statewide assessments.
 Caveon will analyze data to identify highly unusual test results for
two primary groups:
 Students with extremely similar test responses; and
 Schools with improbable levels of similarity, gains and/or
erasures.
 Flagging only the most extreme results.
Troubling News
 Michigan Educator Survey*, July 2011
 34% felt pressure to change grades for the better
 29% felt pressure to cheat on standardized tests
 21% know of an educator that changed scores on student
tests
 8% admitted to changing students’ grades due to outside
pressure
*Detroit Free Press, July 26, 2011
Caveon Data Forensics
Process
Examples of “Unusual” Behavior
Very high similarity among pairs or groups of test takers (may
be across test groups)
Very unusual number of erasures, particularly wrong to right
Very substantial gains or losses from one occasion to another
Focused on impact on scores
Similarity
Most Powerful & “Credible” Statistic
Measures degree of similarity between two or more test
instances
Analyze each test instance against all other test instances in the
school
Possible causes of extremely high similarity:
Answer Copying
Test Coaching
Proxy Test Taking
Collusion
Erasures
Based on estimated answer changing rates from:
Wrong-to-Right
Anything-to-Wrong
Find answer sheets with unusual WtR answers
Extreme statistical outliers could involve tampering, “panic
cheating”, etc.
Important! No student–level score invalidations based on erasure
analysis; erasure analysis limited to school-level flagging for additional
review.
Unusual Gains/Losses
Predict score using prior year information.
Measure large score increases/decreases against predicted
score
Which score truly reflects the student’s actual ability or
competence?
Extreme Gains/Losses may result from:
Pre-knowledge, i.e., “Drill It and Kill It”
Coaching
Student development—visual acuity
Important! No student–level score invalidations based on gains analysis;
gains analysis limited to school-level flagging for additional review.
A Quick Discussion of Extreme
Results…
Chance of
Chance of
Chance of
Chance of
being hit by lightning = 1 in one million
winning the lottery = 1 in 10 million
DNA false-positive = 1 in 30 million
students being flagged for similarity, but doing
own work = 1 in a trillion
Student-Level Analysis
Similarity Analysis only
Most credible, strongest
No flagging for erasures or gains
 Invalidate test scores with Similarity Index ≥ 12
A Visual Representation
FCAT Spring 2011 Grade 8 (Reading)
N=196,866, Above 12=163 (0.08%)
Mean = 0.3
200000
180000
140000
120000
100000
80000
60000
40000
20000
Similarity Index
30.5
28.5
26.5
24.5
22.5
20.5
18.5
16.5
14.5
12.5
10.5
8.5
6.5
4.5
2.5
0
0.5
Frequency
160000
A Visual Representation
FCAT Spring 2011 Grade 8 (Reading)
With first bar removed
Index ≥ 12 is 15 standard deviations above the
mean of 0.3
7000
5000
4000
3000
2000
1000
Similarity Index Value
29.5
27.5
25.5
23.5
21.5
19.5
17.5
15.5
13.5
11.5
9.5
7.5
5.5
3.5
0
1.5
Frequency
6000
What does the mean look like?
Another Look at the Mean…
Index=0.25; Scores=303 & 309
2
1
0
0
5
10
No Match
15
20
25
Item Number
Same Correct
30
35
Same Incorrect
40
45
What does an index > 12 look
like?
A Comparison…
Index=0.25; Scores=303 & 309
2
1
0
0
5
10
15
No Match
20
25
Item Number
Same Correct
30
35
40
45
40
45
Same Incorrect
Index=15.7; Scores=303 & 309
2
1
0
0
5
10
15
No Match
20
25
Item Number
Same Correct
30
35
Same Incorrect
M-DCPS Student-Level Invalidations
Caveon Student Invalidation Outcomes, 2011-12
900
800
782
700
600
500
397
392
400
300
200
100
33
13
11
0
Retake
11
3
FCAT RMS
Invalidations
Appeals
EOC
Released
5
School-Level Analysis
Similarity, Gains, AND Erasures
Flagged schools will lead to school district review
Unsatisfactory reviews may lead to an inspection by the
Inspector General’s Office
Gains Rate
Index
286.20
306.49
300.62
310.77
376.53
292.00
280.79
344.23
299.73
303.05
265.98
Erasures Rate
Index
0.0
0.0
0.0
0.2
45.0
0.0
0.0
39.6
0.0
0.0
0.0
Mean Score
0.43
0.37
0.60
0.67
0.97
0.52
0.18
0.87
0.32
0.35
0.28
M4 Similarity
Rate Index
338
672
532
664
364
512
338
830
534
458
197
Pass Index
M
R
M
M
M
M
R
M
R
R
M
Pass Rate
Subject
xxxx
yyyy
zzzzz
aaaa
bbbb
cccc
dddd
eeee
ffff
gggg
hhhh
Number of Tests
District-School
Sample School-Level Analysis
34.9
24.3
21.3
17.5
0.0
14.8
13.5
0.0
13.1
1.5
12.5
0.6
0.1
0.3
0.1
0.0
0.1
0.3
0.0
0.0
12.7
2.4
9.1
0.1
4.5
3.5
6.1
5.3
1.0
2.2
0.2
0.0
10.3
Take Away
 This program is important and timely.
 It’s about valid test results.
M-DCPS
School Police
Detective Louis Nunez
District Resources
Assessment, Research, and Data Analysis Website:
http://oada.dadeschools.net/
Test Chairperson Website:
http://oada.dadeschools.net/TestChairInfo/InfoForTestChair.asp
Testing Calendar (check for updates regularly):
http://oada.dadeschools.net/TestingCalendar/TestingCalendar.asp
Guidelines and Tips for School Test Chairpersons:
http://oada.dadeschools.net/TestChairInfo/27GuidelinesandTipsforSchool
TestChairpersons.pdf
Standards, Guidelines, and Procedures for Test Administration and Test
Security:
http://oada.dadeschools.net/TestChairInfo/29StandardsGuidelinesand
ProceduresforTestAdministrationandSecurity.pdf
Contacts
 Student Assessment and Educational Testing:
 Dr. Sally A. Shay, District Director
 Florida Comprehensive Assessment Test (FCAT/FCAT 2.0) and End-of-Course Assessments
(EOC):
 Maria C. Bruguera, Director I
 Mara Ugando, Staff Specialist
 Stanford Achievement Test-Tenth Edition (SAT-10), Grade 3 Promotion Opportunities, and
Post Secondary Education Readiness Test (PERT):
 Mayda Cabeza, Director I
 Comprehensive English Language Learning Assessment (CELLA), Interim Assessment, and
National Assessment of Educational Progress (NAEP):
 Felicia Mallory, Executive Director
 Denetra Collins, Staff Specialist
 Race to the Top, Hard to Measure
 Kathleen Sierra, Supervisor
 Test Distribution Center
 Magaly Hernandez, Supervisor