ATI Data Analysis Protocol
Guidebook for Instructional Staff
Data Processes
2013-2014
This guidebook provides TUSD educators with instructions on the use and analysis of data generated by ATI Galileo assessments. Data analysis reports are
classified so that staff may make best use of the available reports and make decisions regarding instruction and intervention that will benefit students. The
guidebook also provides instructions of the background and use of ATI reports.
Section I: Analysis for Teacher Groups
Teachers have different data needs, based on their instructional position. This section outlines the different groups and their general data needs.
Core Teachers (CT)
A “core” teacher is an educator who is providing instruction in Math or English/Language Arts (ELA). ATI test results have a direct connection to the instruction
provided in these classrooms. Core teachers should conduct the most detailed analysis of ATI results to identify students at risk, design strategies to provide
interventions according to the district MTSS process, and to make decisions on the course of future instruction.
Non-Core Teachers (NC)
Within a learning community all educators carry responsibility for the learning of the students. While Core teachers carry out the most detailed analysis of ATI
test results, all teachers need to conduct analysis, so they know where their students stand relative to making progress towards AIMS success, to identify
students in need of additional assistance, and to support core staff with the implementation of interventions.
Administrators (AD)
School administrators carry the responsibility to analyze school level data, identify the trends that impact student learning and facilitate opportunities for
learning that best benefit students.
Section II: Data Analysis Process
The following steps make up the process of examining ATI data:
1. Identify what concepts have been taught during the period of instruction.
a. Identify the weight value of the concepts according to the AIMS Blueprint.
2. Review the ATI Test Blueprint to identify what concepts were included in the test.
1
3. Identify Data Questions that will be included in the test analysis.
4. Use the ATI website to generate reports that will answer your questions.
5. Document the data and how it answers your data questions.
6. Develop the plan of action for enrichment, intervention and/or instructional adjustments. Include assessments for the plan’s action steps.
7. Implement the action plan.
Section III: Data Questions and Reports
The following data questions will be included in the data analysis. The chart also prescribes which reports staff can use to collect the required data.
Data Question
What concepts were taught during the
period of instruction?
Responsible Group
Core Teachers (CT)
Administrators (AD)
Required Data Reports
Teacher Lesson Plans
District Pacing Calendars
Optional Data Reports
What concepts were included on the ATI
assessment?
Core Teachers (CT)
Administrators (AD)
ATI Test Blueprint Report
How did the students perform on the ATI
assessment?
Core Teachers (CT)
TUSD Stats Quarterly Benchmark
Report Builder
TUSD Stats ATI Parallel Report
Intervention Alert
Class Test Scores Report
Test Monitoring Report
Item Analysis Report
How did the students perform on the
concepts that were taught during the period
of instruction?
Core Teachers (CT)
Intervention Alert
Standards Mastery Report
Are the students in the class/grade
demonstrating growth?
Core Teachers (CT)
Non-Core Teachers (NC)
Administrators (AD)
Multi-Test Aggregate Report
Standards Mastery Report
Student Growth and Achievement
Report
Which students are at risk of not being
successful on the State assessment?
Core Teachers (CT)
Non-Core Teachers (NC)
Administrators (AD)
Benchmark Results Report
Benchmark Results by Group Report
Are the students in the class/grade/district
mastering concepts?
Core Teachers (CT)
Non-Core Teachers (NC)
Administrators (AD)
Standards Mastery Report
2
Section IV: Accessing ATI Data
This section contains information of the reports outlined in sections 2-3 and provides instructions on accessing each report.
Report Used By
What Data is Displayed?
Data Source(s)
Why is This Data Important?
What to Look For?
How to get this Data?
The reports shows the
standards assessed, the
percent of the test focused
on the standard, and the
numbers of the test items
focused on the standard.
This information allows the
teacher to see the construction
of the test and use that
information as further
information is examined.
Information on the
construction of the
test.
Click on the Reports tab → Click on
the Test Properties link → Click on
the Test Blueprint Link → Select the
grade level library for your report →
Select the test you would like to
examine → Click on Run Report.
How students
performed on
individual questions.
Click on the Dashboard link → Find
the test you would lime to examine →
Click on the Test Monitoring link for
that test.
Test Blueprint Report
CT, AD
Complete after test
release
Test Monitoring Report
CT
Complete after test
release
Data Source:
Shows an ‘at a glance’ table of
Raw and Percentage Scores. each question from the PM and
the how the students answered
Reported As:
them. This page is very visual
Percentage correct on one
and shows which questions are
test.
answered most often correct
(green)/ incorrect (red). Also
shows which students skipped
questions. The question numbers
are linked to the actual question
for easy reference. This page
can be shared with students (just
click the box: show student ids to
hide names)
Low performing
questions should be
analyzed. Can be
analyzed with
students
Online: Can monitor
for student progress
when taking the test.
3
Report Used By
What Data is Displayed?
Data Source(s)
Why is This Data Important?
What to Look For?
How to get this Data?
Data Source:
Percentage Scores.
This report provides teachers
with data related to how each
individual student performed on
the test and how the overall class
performed on the benchmark or
formative test.
Higher percentage
scores indicate
better overall
performance.
Click on the Reports tab → Click on
the Test Scores link → Select Class
Test Scores → Select the class and
library → Select one or more tests to
view → Click on Run Report.
The Intervention Alert report lists
all of the learning standards on a
given assessment and displays
the percentage of students who
have demonstrated mastery of
the learning standards. The
learning standards listed that do
not have 75 percent of students
mastering them, will be
highlighted in red. This allows the
users to easily identify what
standards interventions should
focus on.
Performance on
individual PO’s. Low
performing PO’s,
questions should be
examined.
Interventions may be
designed.
Class Test Scores Report
CT
Complete after test
release
Reported As:
Percentage correct on one
test
Can compare percent
score to anticipated
scale score.
Intervention Alert
CT
Complete after test
release
Data Source:
Percentage Scores.
Reported As:
Percentage of students
achieving mastery on one
test.
Can compare PO
performance with
school and district
results.
Develop plans to
address high and low
performing PO’s
Click on: Reports tab → click on
Standards Mastery tab → Select
level: School or Class → Select library
(e.g. for 7th grade click on 2012-13
TUSD 07 Gr. Progress Monitoring) →
Select a test (e.g. for 7th grade math
PM#2 click on 2012-13 TUSD PM
Math 07 Gr. #2) → Run Report → this
page shows your cohort selection as
compared to the other TUSD schools
in that grade by PO. Select 1 or more
PO’s to examine → Click on
Assignment button → the next page
provides mini assessments on specific
PO’s and shows the at-risk students.
4
Report Used By
What Data is Displayed?
Data Source(s)
Why is This Data Important?
What to Look For?
How to get this Data?
Data Source:
Percentage Scores.
Shows each question from the
PM and the distribution of
student answers. This page is
very concrete as it shows the
actual questions/ response rates.
This page can be shared with
students to re-teach for mastery
and to build test-taking skills
(what is a distracter, how to
narrow down answer choices
etc.) This page shows questions
that the more proficient students
missed as a gauge for what to reteach for all students. Click on
the linked percent and
corresponding student names
will appear. Click on the linked
question number/PO above each
table and the actual question will
be shown with the correct
answer displayed for re-teaching
opportunities.
Examine questions to
see how students
answered. Large
numbers of students
choosing a distractor
item may indicate
gaps in learning.
Click on Reports tab → click on Test
Properties → Item Analysis → Choose
student cohort: grade (all students), a
particular class and/or a particular
student → Select library (e.g. for 7th
grade click on 2012-13 TUSD 07 Gr.
Assessment Library) → Select test
(e.g. for 7th grade math PM#2 click on
2012-13 TUSD Math 07 Gr. #2) →
Click the box: show detailed analysis
→ Sort by: PO → Run Report. The
next page displays the proficiency
levels (percentile ranks) of students
and how they answered the question.
Item Analysis Report
CT
Complete after test
release
Reported As:
Percent of students who
chose each of the question
answer choices.
Follow up with
students to
determine why
distractor was
chosen.
Students in high
percentile groups
who choose incorrect
answers should be
followed up. Do gaps
in learning exist?
5
Report Used By
What Data is Displayed?
Data Source(s)
Why is This Data Important?
What to Look For?
How to get this Data?
Data Source:
Scale Scores.
Shows a bar graph of the
projected percent of students
who ‘master’ each concept. This
page is to monitor growth over
time of students from one
progress monitoring test to
another. The bars on the graph
should increase with each PM
and/or narrow the gap narrowed
between student scores and
targeted benchmark.
Students should
show progress in the
mastery of concepts.
Low levels of
progress or
regression from the
pre-test indicate a
need for further
examination of the
concept.
Instructional
adjustments may be
required.
Click on: Reports tab → click on
Standards Mastery tab → Standards
Mastery Progress → Select class
→Select library (e.g. 2012-13 TUSD
Assessment Planner Library) → Select
subject (e.g. for 7th grade reading,
choose, AZ -R07 2003: 07th Gr.
Reading or 4th grade math choose AZM0: 0 4th Gr. Math ) →which tests you
would like displayed→ click Print
Report.
Can monitor change over time at
the concept level (similar to
AIMS) Should see an upward
trend with each PM as the test is
more or less comprehensive. A
rule of thumb might be about 4060% mastery on PM1, 60-80%
mastery on PM2 etc. of identified
concepts.
The percentage of
students meeting the
standard should
increase with each
test application. Low
levels of progress or
regression from the
pre-test indicate a
need for further
examination of the
concept.
Instructional
adjustments may be
required.
Click on: Reports tab → click on
Standards Mastery tab →
Development Profile → Select library
(e.g. for 7th grade click on 2012-13
TUSD 07 Gr. Assessment Library) →
Select test (e.g. for 7th grade math
PM#2 click on 2012-13 TUSD Math 07
Gr. #2) → Select box: show concept
level data and unclick box: show
objective level data → Select box:
percentage → Run Report. To get
percent mastery scores, add the two
columns meets/exceeds together.
For Individual Profile, it is the same
process: select library and test and
run report.
Standards Mastery Report
CT, NC, AD
Complete after ATI
analysis.
Reported As:
Percent of students with
mastery probabilities above
the achievement threshold.
Development Profile
AD
Complete after ATI
analysis.
Data Source:
Scale Scores
Reported As:
Percent of students at each
level of the FAME scale by
standard.
6
Report Used By
What Data is Displayed?
Data Source(s)
Why is This Data Important?
What to Look For?
How to get this Data?
Data Source:
Scale Scores
Shows a snap shot of the average
percent mastery for all PM’s side
by side. This graph can be
filtered by grade, class or
individual student. Also, a dot
indicates the average score
needed for benchmark mastery
on AIMS (about 75%). The bars
on the graph should increase
with each PM and/or narrow the
gap narrowed between student
scores and targeted benchmark.
Student progress
should improve with
each test application.
Students need to
perform at or above
the cut score to have
a positive prediction
to AIMS
performance. Slow
progress or
regression indicates a
need for instructional
adjustments.
Click on: Reports tab → click on Test
Sets → Aggregate Multi-Test Report
→ Select library (e.g. for 7th grade
click on 2012-13 TUSD 07 Gr.
Assessment Library) → Select tests
(e.g. for 7th grade math PM#2 click on
2012-13 TUSD Math 07 Gr.: pre-test,
#1 and #2) → Run Report. The next
page shows the bar graph of each PM
mastery → Click on link: view detail to
see student names in each proficiency
category → click on link: view report
for printable version.
The Student Growth and
Achievement Report (SGA) crossclassifies student achievement
with student growth yielding new
and useful information to guide
instruction.
Ideally student
performance should
plot above the cut
score (green) line
and to the right. ATI
provides an
indication if proper
progress is being
maintained.
Insufficient progress
requires dialog
regarding instruction
and intervention.
Click on: Reports tab → click on Test
Sets → Scroll down to the bottom and
click on Student Growth and
Achievement. Select the classes for
the report → select the subject, start
test and end test → Run report.
Aggregate Multi-Test Report
CT, NC, AD
Complete after ATI
analysis.
Reported As:
Scale score plot of
achievement and growth
against targeted cut score.
Student Growth and Achievement
CT, AD
Complete after ATI
analysis.
Data Source:
Scale Scores.
Reported As:
Scale score plot of student
growth and achievement,
designating placement on a
progress grid.
7
Report Used By
What Data is Displayed?
Data Source(s)
Why is This Data Important?
What to Look For?
How to get this Data?
Data Source:
Scale Scores.
The Benchmark Results page
provides you with each student’s
results on each benchmark
assessment. The page will show
you the student’s risk level,
predicting success on AIMS.
Students listed as
Moderate or High
Risk must be
examined closely for
intervention. High
risk students will not
appear until the
post-test.
Click on the Dashboard link → In the
upper left of the Dashboard click on
Benchmark Results → Select the
2013-14 TUSD Assessment Planner
Library → Select the appropriate
subject. The report will generate
when the boxes are filled in.
The Benchmark Results by
Groups report lists each
benchmark assessment given in a
year and the percentage (and
count) of students who have
passed each benchmark
assessment, including an
instructional effectiveness (IE)
assessment. Additionally, this
report allows the user to see the
percentage of students in specific
subgroups that have “passed”
the benchmark assessments. The
user can create subgroups by
selecting student variables from
student information forms and
display this disaggregated data
side-by-side.
The percentage of
students passing the
assessment should
improve over time.
Use this report to
break out key groups
for more detailed
analysis. Flat
performance or
regression from the
pre-test indicates a
need for further
discussion on
instruction and
intervention.
Click on Reports tab → Standards
Mastery link → Scroll down and select
Benchmark Results by Groups. Select
the 2013-14 TUSD Assessment
Planner library → Select the subject
area. → Run the report.
Benchmark Results Report
CT, NC, AD
Complete after ATI
analysis.
Reported As:
Scale scores correlated to
cut scores. Used to
determine student risk
level.
Benchmark Results By Groups
AD
Complete after ATI
analysis.
Data Source:
Percentage Scores.
Reported As:
Percentage correct on one
test
8
Report Used By
What Data is Displayed?
Data Source(s)
Why is This Data Important?
What to Look For?
How to get this Data?
Data Source:
Percentage Scores.
The ATI Parallel report on TUSD
Stats provides detail information
on student performance on AIMS
concepts for one test. The report
allows disaggregation of scores
by groups, and shows the FAME
scale score for each concept
The performance of a
group or individual
student can be
assessed with this
report. Teachers can
also identify
important AIMS
strands by their WIP
value and identify
areas for
instructional
adjustment.
Log into TUSD Stats → Choose the
Assessment tab to the left → Choose
the AIMS data page → Choose the ATI
Parallel report and the complete the
information for the grade/subject that
will be analyzed
The quarterly Reading and Math
reports allow users to view the
number and percent of students
mastering a test at a grade level.
The percentage of
students passing the
assessment should
improve over time.
Flat performance or
regression from the
pre-test indicates a
need for further
discussion on
instruction and
intervention.
Log into TUSD Stats → Choose the
Assessment tab to the left → Choose
the Benchmark data page → Choose
the Quarterly Reading/Math Reports
and complete the information for the
grade/subject that will be analyzed
TUSD Stats – ATI Parallel Report
CT, AD
Complete after ATI
analysis.
Reported As:
Percentage correct on one
test
TUSD Stats – Quarterly Reading/Math Reports
CT, AD
Complete after ATI
analysis.
Data Source:
Scale Scores and Percent
Scores
Reported As:
The number and percent of
students achieving mastery
(based on scale scores).
While only one test can be
selected, by generating the test
for each test the scores for pretest and the PM December test
are comparable.
9