Test Chairpersons’ Orientation
Miami-Dade County Public Schools
September 21, 2011
*
General Session
Accountability – Gisela Field, Vilma Rubiera
Assessment, Research and Data Analysis
Instructional Validity Study – Sally Shay, Student Assessment
2011-12 Testing Program – Student Assessment Staff
Caveon Data Forensics – Sally Shay, Student Assessment
FDM Test Chair Function – Kathy Sierra, Hialeah Gardens
Tiffany Davis, American
Beatriz Ambas, Miami Lakes
2
*
Afternoon Sessions
Test Chair 101 – Zulema Lamazares, Greenglade Elementary
Preparing for CBT – Maria Bruguera, Mara Ugando
Student Assessment
On-line CELLA – Deland Innocent, Caridad Perez
Bilingual/World Languages
3
*
Gisela Field
Vilma Rubiera
4
*
Sally Shay
5
*
•OPPORTUNITY TO LEARN
•First instructional validity study in Florida
•High School Competency Test
•1982 Debra P. vs. Turlington
•Again for FCAT
•Sunshine State Standards
•Now FCAT 2.0 and EOC
•Next Generation Sunshine State Standards
6
*
•Public Awareness
•Educator and Student Understanding
•Curricular Validity
•Instructional Validity
•Student Remediation
7
*
•District Level
•School Level
•Parent/Communty Meeting Notifications
•Letters/Memoranda to Parents, Staff, Students
•ConnectEd Messages
•Website Postings
•Professional Development Activities
•Tutoring/Remediation Activities
•Sample (Voluntary) Lesson Plans, Activities, Tests, Data Binders
•Interim Assessment Debriefings
8
2011-12 Testing Calendar
http://oada.dadeschools.net/TestingCalendar/TestingCalendar.asp
9
Program Updates:
10
FLORIDA COMPREHENSIVE
ASSESSMENT TEST (FCAT) /
FCAT 2.0 AND
END-OF-COURSE (EOC)
ASSESSMENTS
Next Generation Sunshine State
Standards (NGSSS)
Maria C. Bruguera, Director I
Mara Ugando, Staff Specialist
11
Transition to
FCAT 2.0 and EOC Assessments

Phasing-in FCAT 2.0 Science in grades 5 and 8 only
 Biology 1 EOC assessment will replace FCAT Science in high school

Phasing-in Florida End-of-Course (EOC) Assessments (Algebra 1, Biology 1, and
Geometry)
 United States History EOC will be field tested in selected senior high schools
only

Summer EOC Administration in 2012
 Beginning 2012-13 three EOC

administrations
FCAT Writing scoring standards will be increased in 2012
12
will
be
offered
Transition Schedule
13
Computer-based Testing (CBT) in 2011-12
The following FCAT / FCAT 2.0 assessments will be offered in CBT
format:

 FCAT Reading Retake (optional): untimed
 FCAT Mathematics Retake: untimed
 FCAT Reading 2.0 Retake: two ½ day sessions over two days
 FCAT Reading 2.0 (Grades 6 and 10): two 70-minute sessions over two days

Note: Paper-based accommodations for eligible students are available.
14
Computer-based Testing (CBT) in 2011-12, cont.

The following EOC assessments will be offered in CBT format:
 Algebra 1: one 160-minute session*
 Biology 1: one 160-minute session*
 Geometry: one 160-minute session*
 US History Field Test: TBD
Students may have additional time, if needed.

Note: Paper-based accommodations for eligible students are available.
15
Updates: 2011-12 School Year

High Schools
 Grade 9 students will be administered:
o FCAT 2.0 Reading
 Grade 10 students will be administered:
o FCAT 2.0 Reading graduation test on the computer
• New passing score (minimum scale score in
Achievement Level 3)
 No comprehensive FCAT Mathematics or Science assessment
will be administered in high schools
16
Updates: 2011-12 School Year, cont.

EOC Assessments will be on the computer

Algebra 1 EOC will be administered to students (in any grade)
enrolled in Algebra 1 or equivalent courses in December 2011
and/or May 2012, as applicable.
o Students
in middle schools and entering grade 9
students in 2011-12 must pass the Algebra 1 EOC
assessment to earn high school credit.
o Grade
10 students (entered grade 9 in 2010-11) will have
30% of the Algebra 1 EOC score calculated into the final
grade
17
Updates: 2011-12 School Year, cont.

EOC Assessments will be on the computer

Biology 1 and Geometry EOC will be administered to students
(in any grade) enrolled in Biology 1 and/or Geometry or
equivalent courses in May 2012.
o Students
entering grade 9 in 2011-12 will have 30% of the
Biology 1 and/or Geometry EOC scores calculated into
their final grade
o Any
other grade level must participate in the Biology 1
and/or Geometry EOC but will not be impacted by test
results
18
Certification Tool
The Certification Tool is to be completed by each school that will
offer computer-based assessment(s)in the 2011-12 school year.


Six sections to complete:
 I. Computer Specifications
 II. Student Scheduling Plan
 III. Annual Checklist
 IV. Administration – Specific Checklist
 V. School Computer-Based Assessment Team
 VI. Submit to District Office
19
Certification Tool, cont.

The tool must be completed for each test administration:
 Fall 2011 FCAT / FCAT 2.0 Retake
 December Algebra 1 EOC
 Spring 2012 FCAT / FCAT 2.0
 Spring 2012 Algebra 1, Geometry, and Biology 1 EOC Semester 2 and US
History EOC Field Test
A two-week window will provided for schools to complete the tool
for each test administration.

 Notifications to schools will be sent via Weekly Briefings.
20
TestHear
New platform used to deliver accommodated forms (e.g.,
large print, zoom, color contrast, screen reader, assistive
devices) for Florida’s high-stakes computer-based statewide
assessments.

 Must be installed on all computers that will be used by eligible
students requiring accommodated CBT forms (e.g., large print, zoom,
screen reader, color contrast, assistive devices).
 Local installation is required.
Note: Audio card and headphones are required for the computers that will
run the Screen Reader accommodation.
21
Infrastructure Trial
 The
Infrastructure Trial simulates a computer-based test using
mock test items that will allow schools to test hardware and
network capacity prior to the CBT operational administration.
 It should take approximately thirty minutes for users to log
in, supply random answers, and submit the test.
o Please note that the content of the test used for the Infrastructure
Trial is NOT intended for student practice.
 Schools
must set up the Trial to use all computers scheduled
to be used concurrently during the spring 2012 CBT
administrations.
 Test Hear accommodated form will be added this week to
Infrastructure Trial. These forms are larger (50 MB) than
regular TestNav forms (2 MB).
22
Practice Test (ePat)
 All
students MUST participate in a practice test at the school
prior to taking a CBT assessment.
 A practice session is required for each CBT assessment (i.e., FCAT
Retake, Algebra I EOC).
 Practice
test script must be used when conducting the
practice session .
 The script describes the presentation of items and computer tools
available to introduce students to the computer-based system.
 Students
are required to participate in a practice test using
the ePAT with the same accommodations that will be provided
to them on the operational assessment.
23
Interim Assessment Program
Formative Classroom Assessment
Felicia Mallory, Executive Director
Denetra Collins, Staff Specialist
24
Purpose of the Baseline Benchmark Assessments
(BBA)
 To determine students’ baseline knowledge of the Next Generation Sunshine
State Standards in Reading, Mathematics, and Science.
 To provide teachers with a quick snapshot of students’ initial knowledge and
abilities to focus instruction.
 To provide for state required progress reports in schools classified under the
Differentiated Accountability System as Prevent I or II, Correct I or II, or
Intervene.
 Program guide and instructions for downloading the assessments are available
on our website at http://oada.dadeschools.net/IAP/IAP.asp
25
Purpose of Interim Assessment Tests
 To monitor student progress on the Reading, Mathematics, and Science Next
Generation Sunshine State Standards (Fall and Winter)
 To provide teachers with classroom assessment tools that will provide reliable
student-level benchmark information to inform instruction
 To provide students with information on their progress on specific benchmarks
 To provide Differentiated Accountability schools with a reliable and valid
assessment tool for state progress reporting.
26
Overview of the Interim Assessment Program
 Interim Assessment tests administered three times per year: (BBA, fall, and
winter)
 Alignment with the Next Generation Sunshine State Standards and District’s
pacing guides
 Benchmark Assessment Item Banks for Reading and Mathematics are available
in ExamView
 Data analysis and disaggregation, score reports, and item analysis are available
in Edusoft
27
Interim Assessment Administration Windows
2011-2012
Administration Windows
BBA
-
August 22, 2011 – September 2, 2011
Fall
-
October 17 – October 31, 2011
Winter
-
January 9 – January 20, 2012
-
Subjects and Grade Levels
Reading : Grades 3-10, Retake NGSSS, Retake SSS
Mathematics: Grades 3-8; Algebra 1 , Geometry,
Retake SSS
Science: Grades 5, , 8, and Biology 1
Reading : Grades 3-10, Retake NGSSS, Retake SSS
Mathematics: Grades 3-8; Algebra 1, Geometry,
Retake SSS
Science: Grades 5, , 8, and Biology 1
Reading : Grades 3-10, Retake NGSSS, Retake SSS
Mathematics: Grades 3-8; Algebra 1, Geometry,
Retake SSS
Science: Grades 5, , 8, and Biology 1
*All Differentiated Accountability Intervene and Correct II D or F Schools must complete testing
during the first scheduled week.
28
Note. Fall and Winter Grades 6 and 10 Reading and Algebra 1 will be administered on computer .
Approximate Testing Times
Before the Test
Approximately 10
minutes to pass
out testing
materials
Administration Time
Reading*:
Approximately 70 minutes
After the Test
Approximately 10
minutes to collect
testing materials
Mathematics*:
Approximately 75 minutes
Science*:
Approximately 112 minutes
* It is suggested that assessments be administered over a two-day time period.
29
Students to Be Tested
All eligible students* must take the Interim Assessment in Reading,
Mathematics, and Science.
Reading, Grades 3-10,*
 Mathematics, Grades 3-8*
 Science: Grades 5 and 8*
 All students enrolled in:
Algebra I**
Geometry**
Biology I**

*Students MUST take the test corresponding to the grade level in which they are listed in
ISIS.
**Or an equivalent course
30
Grade Levels and Standards
by
Content Area
IA Test Form
Participating Students
Standards
Addressed
NGSSS
FCAT/EOC Form
Reading, Grades 3-10
Grades 3-10
Reading, Grade 10
First Time Retakers
NGSSS
FCAT 2.0
Reading Retake SSS
All Other Retakers
SSS
FCAT
Mathematics, Grades 3-8
Grades 3-8
NGSSS
FCAT 2.0
SSS
FCAT
Mathematics Retake (SSS) Retakers
FCAT 2.0
Science
Grades 5 and 8
NGSSS
FCAT 2.0
Algebra I
Enrolled students
NGSSS
Algebra I EOC
Geometry
Enrolled students
NGSSS
Geometry EOC
Biology I
Enrolled students
NGSSS
FCAT 2.0
31
Interim Assessment Computer-Based Testing
The following Interim Assessments (IAs) will be computer-based for the
Fall and Winter administrations:




Grade 6 Reading
Grade 10 Reading
Grade 10 Reading Retake NGSSS
Algebra 1
32
Interpreting Performance Levels for the NGSSS, 2011-12



The Next Generation Sunshine State Standards test forms will utilize
the Edusoft default of 70% for Overall and Question Group Bands.
• Proficient set at 70%
• Non Proficient set at 69% and below
Performance levels for Fall and Winter forms will be made available
in 2011-12 after the new FCAT 2.0 Scaled Scores and
corresponding Achievement Levels have been set by the FLDOE.
IA Science and Geometry performance levels will not be available
until the 2012-13 school year.
33
Performance Levels for Retake (SSS) Forms
The Performance Levels for grade 10 Reading and Mathematics tests based on the
original Sunshine State Standards are as outlined.
Satisfactory Progress
Students who are likely to
score in FCAT levels 3
and above
Limited Progress
Students whose
performance was not
sufficient to predict
success.
Insufficient Progress
Students who are likely to
score below FCAT level 3.
This student demonstrated a satisfactory level of achievement on the content focus of
the Florida Sunshine State Standards assessed during this instructional period. To
attain high levels of achievement in this content area, the student must receive
continued instruction on the challenging content and skills across the benchmarks
designated for this grade level.
This student demonstrated a limited level of achievement on the content focus of the
Florida Sunshine State Standards assessed during this instructional period. To attain
high levels of achievement in this content area, the student must receive targeted
interventions and remediation in the areas of concern, and continued instruction on
the challenging content and skills across the benchmarks designated for this grade
level.
This student demonstrated an insufficient level of achievement on the content focus
of the Florida Sunshine State Standards assessed during this instructional period. To
attain high levels of achievement in this content area, the student must receive
intensive interventions and remediation in the areas of concern, and continued
instruction on the challenging content and skills across the benchmarks designated for
this grade level.
34
Reading Forms – Grade 10
BASELINE
ASSESSMENT TEST
FORM
Grade 10 Reading
(NGSSS Standards)
ADMINSTERING
STUDENTS
EDUSOFT
EDUSOFT FOLDER
ANSWER SHEET
TITLE
First time 10th Grade
Reading Grade 2011 Computerstudents
10 (NGSSS)
Based Interim
Assessments
Reading Grade 2011 ComputerFirst time Retakers
10 Retake
Based Interim
(mostly 11th Grade students:
(NGSSS)
Assessments
those who failed to pass the
FCAT Reading 2.0 in March
2011 when they were in
grade 10)
Reading Retake (SSS)
All other Retakers
Reading Retake
(mostly 12th Grade students, (SSS)
or 11th Grade students who
took the retake last year)
35
2011 Interim
AssessmentsRetake folder
Score Reports
Useful reports that can be retrieved via Edusoft:

Class List Reports

Item Analysis Report

Item Response Report

Student Performance Report

Subgroup Report

Custom Group Report

Performance Band Report
36
Benchmark Assessment Item Bank
 To
allow teachers to monitor student progress on the Reading and
Mathematics Next Generation Sunshine State Standards
 To
provide teachers with classroom assessment tools that will provide reliable
student-level benchmark information to inform instruction
 To
provide students with information on their progress on specific benchmarks
Item Bank Resources:
Item Bank Overview
Item Bank Newsletter
Download Examview Test Generator
37
Comprehensive English Language
Learners (CELLA)
Felicia Mallory, Executive Director
Denetra Collins, Staff Specialist
38
Overview of CELLA




CELLA is a four-skill English language assessment measuring student
proficiency in listening, speaking, reading, and writing.
Students in grades 1 – 12 are administered the
Listening, Reading, and Writing sections in groups
Speaking on a one-on-one basis
Kindergarten students are assessed in all four areas on a one-to-one
basis.
Unlike the FCAT, CELLA is not a timed test.
39
Purposes of CELLA
The test results can provide:

Data useful for charting student progress over time,

Diagnostically useful information about students’ strengths and
weaknesses in English,

The language proficiency levels of individual students can be
used in making decisions regarding placement into, or exit from
English for Speakers of Other Languages (ESOL) programs.

Identifying the proper ESOL level,

Determining whether the District and schools met Annual
Measureable Achievement Objectives (AMAOs)
40
Annual Measurable Achievement Objectives (AMAOs)
AMAO 1:
Progressing in English language acquisition
annual increases in the number or percentage of
students making progress in learning English
AMAO 2:
Exiting or reaching English language proficiency
annual increases in the number or percentage of
students attaining English language proficiency by
the end of each school year
AMAO 3:
ELL-Adequate Yearly Progress (AYP)
AYP for the ELL subgroup in demonstrating
proficiency on grade-level in reading and
mathematics on the FCAT
41
Important Dates for 2012 CELLA
Administration:
Individual : March 5 – April 6, 2012
Group : March 5 – March 31, 2012
42
National Assessment of Educational
Progress (NAEP)
Felicia Mallory, Executive Director
Denetra Collins, Staff Specialist
43
Features of NAEP-TUDA
 Provides
district-level data to participating urban districts
 Provide
a common yardstick to measure student achievement in
comparison to
 Other urban districts
 States
 Large Central Cities
 National Public Schools
 Census Regions
44
Purpose of the National Assessment of Educational
Progress (NAEP)/TUDA

A common nationwide measure of student performance

Reports changes over time.

Provides results for the nation, states, and some urban districts.
Secondary indicator for state progress and allows national
comparisons between states and large urban districts

Used by the President, Congress, and state leaders to set education
policy

45
Overview of the NAEP

A representative sample of schools and students are selected
Randomly selected students in grades 4, 8, and 12 are assessed in
reading, writing, mathematics, economics, and writing.


Assessments includes multiple-choice and constructed responses.

Assessments are administered by NAEP field staff
Assessments are 90 to 120 minutes, which include testing and
answering background questions


Accommodations are offered for ELL and SWD students.
Results are reported by scale scores, subscales, achievement levels,
subgroups, and geography

46
2012 Assessments
Operational
Components
Grades/
Ages
Pilots
Grades
/Ages
Long-Term
Trend in
Reading
9, 13, 17
Reading LTT
9, 13, 17
Mathematics LTT
9, 13, 17
Long-Term
Trend in
Mathematics
9, 13, 17
Writing—ComputerBased
4
Reading
12
Economics
12
Mathematics
12
47
NAEP 2012 Assessment Dates
Type of Assessment
Assessment Dates
Age 13
October 10 - December 16, 2011
Age 9
January 9 – March 16, 2012
Age 17
March 19 – May 25, 2012
Grade 12
Economics, Reading,
Math
January 23 – March 2, 2012
Grade 4
Writing Pilot
January 23 – March 2, 2012
LTT
48
Important Dates for NAEP 2012
Event
Date
Provide School Information
Form registration
September 30, 2011 (Age 13 LTT)
NAEP field staff will contact
schools to schedule an
assessment visit
September (Age 13 LTT)
Pre-Assessment packet arrives
in selected schools
September (Age 13 LTT)
Pre-Assessment visits
September (Age 13 LTT)
NAEP Testing Administration
Window
Age 13 – October 10 – December 16, 2011
Age 9 – January 9 – March 16, 2012
Age 17 – March 19 – March 25, 2012
Grades 4 and 12 – January 23 – March 2, 2012
49
NAEP-TUDA 2011 Release Date
Subject and Grade
Release Date
Math – Grade 4 and 8
Reading – Grades 4 and 8
November 2011
50
Florida Competency on Personal Fitness
(FCEPF)
51
Denetra Collins, Staff
Specialist
Stanford Achievement Test,
Tenth Edition (SAT-10)
• Mayda Cabeza, Director I
52
SAT-10
 What
is the SAT-10?
 Who is administered the SAT-10?
o Grades 1 and 2, districtwide
o New for 2011 all kindergarten
 Important dates to remember
o April 10-13, 2012
53
SAT-10

How do SAT-10 materials arrive to schools?

Is the SAT-10 a secure test?

SAT-10 Test Chairperson Training
o (February 2011)

School level training of test administrators
and proctors

Test Administration
o Untimed
54
SAT-10 Results

School Listing of Results (June)
o Must be picked up at ITS

Individual Student Reports (August)
o Delivered to schools

Results
o Not available on the Parent Portal
o Posted to SPI under the tab FCAT Scores

Uses
o Referral of students to the district’s gifted program (Plan A or
Plan B)
o Early identification of students who would be considered at
high risk for problems associated with reading
comprehension
55
Grade 3 Opportunities for Promotion
56
Grade 3 Opportunities for Promotion

Section 1008.25, Florida Statutes (F.S.), Eliminating Social
Promotion.
o Spring FCAT 2.0 Reading
o Spring Grade 3 Reading Student Portfolio
o Alternative Standardized Reading Assessment (ASRA)—
ITBS, Level 9, administered the last two weeks of school
o Alternative Assessment for Grade 3 Promotion (AAGTP)—
SAT-10, Primary 3, administered in Summer School
o Grade 3 Mid-Year Promotion (GTMYP)—ITBS, Level 10,
administered in November
57
Grade 3 Reading Student Portfolio





Administered during the 2nd semester
Participants: All grade 3 students complete a portfolio
Scored at the school site
Represents the 2nd opportunity for a student to be
promoted to grade 4
Student must demonstrate mastery of the Next
Generation Sunshine State Standards (NGSSS)
58
Alternative Standardized Reading Assessment
(ASRA)
Administered the last two weeks of schools.
 Participants:

o Students scoring FCAT 2.0 Reading level 1 , students who did not
participate in the Spring FCAT, and
o students without a passing Reading Portfolio
Represents the 3rd and last opportunity within the school year prior to
retention.
 This administration of the ITBS, Level 9 is scanned via Edusoft
therefore results are available immediately.
 Students must score at or above a given cut score on the ITBS to be
considered for promotion.

59
Alternative Assessment for Grade 3
Promotion (AAGTP)
Administered at the end of the summer Reading camp.
 Participants: Retained grade 3 students
 SAT-10, Primary 3 is scored locally through ITS
 Represents 4th opportunity for students to be promoted to grade 4
 Students must score at or above a given cut score on the SAT-10 to be
considered for promotion

60
Grade 3 Mid-Year Promotion
(GTMYP)





Administered mid-November
Participants: Students who “just missed” passing the
AAGTP, or other students at the principal’s discretion
ITBS, Level 10 is scored at the school site via Edusoft
Represents 5th opportunity for retained 3rd grade
students to be promoted to grade 4
Students must have a passing score on the ITBS,
representing 4th grade work.
61
The Iowa Tests
Iowa Tests of Basic Skills (ITBS),
Grades K-8
and
Iowa Tests of Educational Development
(ITED), Grades 9-12
62
Iowa Tests

Used for
o Plan B Gifted Placement;
o Extension of Services in the ESOL program;
o Optional for Charter Schools Only, ESOL program placement
Scanned and scored via Edusoft
 Download a copy of the Iowa Tests Guidelines at
http://oada.dadeschools.net/TestChairInfo/138201011IowaTestGuidelines.pdf

63
HB1255 Section 1008.30, F.S.
High schools must evaluate the college readiness of each student prior
to grade 12.

o Students who score Level 2 or 3 on the Grade 10 Reading assessment.
o Students who score Levels 2, 3, or 4 on the Grade 10 Mathematics assessment.

Results of the assessment will be used to advise students.
64
What is the P.E.R.T?
PostSecondary
Florida’s
Readiness Test
new common placement exam
Customized for Florida and aligned to Florida’s
Postsecondary Readiness Competencies


Readiness Scores
oReading, 104
oWriting, 99
oMathematics, 113
65
P.E.R.T Components and Design
P.E.R.T Placement Test (3 sub-tests) identifies appropriate course
placement in Reading, Writing and Mathematics


Computer Adaptive (larger font available for accommodation)

30 questions per subtest
o 25 operational items for placement score
o 5 field test items
66
P.E.R.T Implementation in High
Schools

Department currently negotiating agreement with vendor

Expectation is for state to fund the cost of each test

Testing window in the fall 2011
67
*
Sally Shay
68
Security

Standards, Guidelines, and Procedures
for Test Administration and Test Security
available at
http://oada.dadeschools.net/TestChairIn
fo/InfoForTestChair.asp
– Adopted by School Board
– General Guidelines

Florida Test Security Statute and Rule
69
General Security Guidelines
DO
DO NOT
 Provide secure storage
 Restrict access
o 3 or fewer keys
 Allow unauthorized access to materials or
content
 Open secure materials prior to date allowed
 Allow untrained or uncertified staff to test
students
 Coach or provide assistance during test
administration
 View, review, copy, or debrief test content
 Handle secure materials, including affixing of
labels in a limited-access area
 Secure visual barriers on computer
workstations
 Maintain documentation
o Chain of custody
o Book assignment to students
 Train test administrators and proctors
 Monitor students during lunch break or
extended breaks
 Visual barriers for computer-based testing
70
Reporting Irregularities and Security
Violations

All school staff are responsible for ensuring appropriate administration
procedures and reporting any irregularities.

Reporting procedures
 Specified in Standards, Guidelines, and Procedures for Test
Administration and Test Security
o
o
o
o
Test Chairperson / Principal
Student Assessment and Educational Testing
Regional Center or District Administrative Office
Appropriate investigative unit
71
Caveon Data Forensics
* The FLDOE has contracted with Caveon Test Security to provide
its Caveon Data Forensics™ for all statewide assessments.
* Caveon will analyze data to identify highly unusual test results
for two primary groups:
* Students with extremely similar test responses; and
* Schools with improbable levels of similarity, gains and/or
erasures.
* Flagging only the most extreme results
72
*
Michigan Educator Survey*, July 2011
34% felt pressure to change grades for the better
29% felt pressure to cheat on standardized tests
21% know of an educator that changed scores on
student tests
8% admitted to changing students’ grades due to
outside pressure
*Detroit Free Press, July 26, 2011
*
*Examples of “Unusual” Behavior
*Very high similarity among pairs or groups of
test takers (may be across test groups)
*Very unusual number of erasures, particularly
wrong to right
*Very substantial gains or losses from one
occasion to another
*Focused on impact on scores
*
*Our Most Powerful & “Credible” Statistic
*Measures degree of similarity between two or more
test instances
*Analyze each test instance against all other test
instances in the school
*Possible causes of extremely high similarity:
*Answer Copying
*Test Coaching
*Proxy Test Taking
*Collusion
*
*Based on estimated answer changing rates from:
*Wrong-to-Right
*Anything-to-Wrong
*Find answer sheets with unusual WtR answers
*Extreme statistical outliers could involve tampering,
“panic cheating”, etc.
Important! No student–level score invalidations
based on erasure analysis; erasure analysis limited to
school-level flagging for additional review.
*
*Predict score using prior year information.
*Measure large score increases/decreases against
predicted score
*Which score truly reflects the student’s actual
ability or competence?
*Extreme Gains/Losses may result from:
*Pre-knowledge, i.e., “Drill It and Kill It”
*Coaching
*Student development—visual acuity
Important! No student–level score invalidations based on gains
analysis; gains analysis limited to school-level flagging for
additional review.
*
*Chance of being hit by lightning = 1 in one
million
*Chance of winning the lottery = 1 in 10 million
*Chance of DNA false-positive = 1 in 30 million
*Chance of students being flagged for similarity,
but doing own work = 1 in a trillion
*
*Similarity Analysis only
*Most credible, strongest
*No flagging for erasures or gains
* Invalidate test scores with Similarity Index ≥ 12
*
FCAT Spring 2011 Grade 8 (Reading)
N=196,866, Above 12=163 (0.08%)
Mean = 0.3
200000
180000
140000
120000
100000
80000
60000
40000
20000
Similarity Index
30.5
28.5
26.5
24.5
22.5
20.5
18.5
16.5
14.5
12.5
10.5
8.5
6.5
4.5
2.5
0
0.5
Frequency
160000
*
FCAT Spring 2011 Grade 8 (Reading)
With first bar removed
Index ≥ 12 is 15 standard deviations above the mean of
0.3
7000
5000
4000
3000
2000
1000
Similarity Index Value
29.5
27.5
25.5
23.5
21.5
19.5
17.5
15.5
13.5
11.5
9.5
7.5
5.5
3.5
0
1.5
Frequency
6000
*
*
Index=0.25; Scores=303 & 309
2
1
0
0
5
10
15
20
25
30
35
40
Item Number
No Match
Same Correct
Same Incorrect
45
*
*
Index=15.7; Scores=303 & 309
2
1
0
0
5
10
15
20
25
30
35
40
Item Number
No Match
Same Correct
Same Incorrect
45
*
Index=0.25; Scores=303 & 309
2
1
0
0
5
10
15
20
25
30
35
40
45
Item Number
No Match
Same Correct
Same Incorrect
Index=15.7; Scores=303 & 309
2
1
0
0
5
10
15
20
25
30
35
Item Number
No Match
Same Correct
Same Incorrect
40
45
*
*Similarity, Gains, AND Erasures
*Flagged schools will lead to school district
review
*Unsatisfactory reviews may lead to an
inspection by the Inspector General’s Office
WTR Erasures - Grade 4 - Math
0.818%
0.190%
0.049%
5.551%
93.392%
Low/Normal
Moderate
High
Very High
Extreme
Gains Rate
Index
286.20
306.49
300.62
310.77
376.53
292.00
280.79
344.23
299.73
303.05
265.98
Erasures Rate
Index
0.0
0.0
0.0
0.2
45.0
0.0
0.0
39.6
0.0
0.0
0.0
Mean Score
0.43
0.37
0.60
0.67
0.97
0.52
0.18
0.87
0.32
0.35
0.28
M4 Similarity
Rate Index
338
672
532
664
364
512
338
830
534
458
197
Pass Index
M
R
M
M
M
M
R
M
R
R
M
Pass Rate
Subject
xxxx
yyyy
zzzzz
aaaa
bbbb
cccc
dddd
eeee
ffff
gggg
hhhh
Number of Tests
District-School
Sample School-Level Analysis
34.9
24.3
21.3
17.5
0.0
14.8
13.5
0.0
13.1
1.5
12.5
0.6
0.1
0.3
0.1
0.0
0.1
0.3
0.0
0.0
12.7
2.4
9.1
0.1
4.5
3.5
6.1
5.3
1.0
2.2
0.2
0.0
10.3
*This program is important and timely
*It’s about valid test results
*
91
District Resources
 Assessment, Research, and Data Analysis Website:
http://oada.dadeschools.net/
 Test Chairperson Website:
http://oada.dadeschools.net/TestChairInfo/InfoForTestChair.asp
 Testing Calendar (check for updates regularly):
http://oada.dadeschools.net/TestingCalendar/TestingCalendar.asp
 Guidelines and Tips for School Test Chairpersons:
 http://oada.dadeschools.net/TestChairInfo/27GuidelinesandTipsforSchoolTe
stChairpersons.pdf
 Standards, Guidelines, and Procedures for Test Administration and Test
Security:
http://oada.dadeschools.net/TestChairInfo/29testsecuritymanual.pdf
92
Contacts
 Student Assessment and Educational Testing:

Dr. Sally A. Shay, District Director
 Florida Comprehensive Assessment Test (FCAT) / FCAT 2.0 and End-of-Course
Assessments (EOC):
 Maria C. Bruguera, Director I
 Mara Ugando, Staff Specialist
 Stanford Achievement Test-Tenth Edition (SAT-10), Grade 3 Promotion Opportunities,
and Post Secondary Education Readiness Test (PERT):
 Mayda Cabeza, Director I
 Comprehensive English Language Learning Assessment (CELLA), Interim Assessment,
and National Assessment of Educational Progress (NAEP):
 Felicia Mallory, Executive Director
 Denetra Collins, Staff Specialist
 Test Distribution Center
 Magaly Hernandez, Supervisor
93