ATI Data Analysis Protocol
Guidebook for Instructional Staff
Assessment Foundations
2013-2014
This guidebook provides TUSD educators with guidance on the use and analysis of data generated by ATI Galileo assessments. The guidebook
provides background on the focus of ATI assessments and the composition of the TUSD Assessment Project.
Assessment Background
TUSD Assessments
Goals of Assessment: The August-December-March tests are comprehensive tests and are designed to identity growth in student performance,
with a goal of predicting success on the state assessment (2013 = AIMS). Students are being tested on their knowledge of standards representing
what they should know by March. Initial low pre-test scores are not a cause for alarm, as the students really have not been exposed to this body of
work. With each test the expectation is that students will record a higher score, representing their exposure and mastery of course material. As you
review your data from these assessments you are looking for growth.
Tests created by a teacher or school as a benchmark assessment will assess student mastery of standards that have been taught during an
identified time period, such as a unit, instructional cycle, quarter or semester.
For the 2013-14 school year the district is implementing tests that combine the complete AIMS blueprint, along with a mix of common core based
questions. Because many of the standards of AIMS and Common Core cross-walk to each other all of the assessments are valid measures of
potential success on AIMS.
Assessment Development: TUSD assessment development began last year. TUSD staff met with ATI and provided the stipulations of
assessment design requested by TUSD Directors and Assistant Superintendents. ATI produced the requested assessments for the district. Over
the summer TUSD convened committees of teachers, drawn from Principal recommendations to review the assessments. Committees reviewed
the assessments and, where necessary, replaced questions for better alignment to TUSD curriculums.
Ongoing Review: When a question is raised about a test question, the inquiry is first referred to ATI. The company may choose to remove the
question from the test results. If ATI feels that the test question is valid the district will then refer the question to the same teachers who reviewed
the test over the summer. Notes will be reviewed to see if any concerns regarding the question were raised
at that time. Based on feedback
from the teacher review team the district will then decide if the question should remain in the test or be removed.
Percentages and Scale Scores - Best use of the numbers
Percentage Scores: Most people refer to the student's percentage score on the test. The percentage score is a common number that educators,
students and parents use to measure growth. Percentage scores can be reviewed immediately after posting to ATI, so teachers can get useful
information from a percentage score. A percentage score is best used to examine how a student performed on one assessment. Percent scores
are often used to show how students performed on individual standards on one test. They are also used to show how students selected answers on
a test question. Because there are other factors that must be taken into account to compare one test with another, a percentage score from one
test cannot be used in comparison to another test.
To compare a student’s performance from one test to another test we need to use a different number, and for our work with ATI that number is the
scale score.
Scale Scores: After our scores are submitted to ATI they perform an analysis of our results. ATI calls their process "Item Response Theory." ATI
takes into account factors such as the difficulty level of the tests, item discrimination (analysis of a students success on an item compared to
success on the whole test), and guessing. The product of the ATI analysis is an equal rating system that allows a comparison of student
performance between tests. That rating system is called a “scale score”. ATI uses the term Developmental Level score. Since scale scores are
used in most standardized testing, such as AIMS, it is easier for us to call them scale scores.
Scale scores are reported as a 3-4-digit number. The scale cut score for proficiency on the 2nd grade Math test, for example, is 611. The numbers
represented by the scale score allow educators to compare results between tests and allow us to measure student growth.
Best Use of ATI Data: Analysis of ATI test scores is not simply a matter of looking at one score. An educator must determine what the data
question is prior to using reports. Some common data questions include:
1.
2.
3.
4.
5.
6.
7.
What did the ATI tests assess?
How did a class perform on one test?
How did my class perform on each PO?
Are the students in my class showing growth?
Which students are at risk of not passing AIMS?
Are the students in a class or grade level mastering concepts?
Which students are not mastering concepts?
To answer these questions and others the district recommends that staff members use a variety of reports to answer their data questions. Broadly,
staff members are assessing performance on one test, or gauging student growth across the school year, and level of anticipated success on AIMS.
The next two sections detail data reports that staff members should use when analyzing ATI data.
Assessment Technologies Incorporated (ATI)
Company Profile: ATI provides the Galileo K-12 Online Instructional Improvement and Instructional Effectiveness System. The system supports
an array of assessment, instruction, educator effectiveness, and reporting tools supporting implementation of Common Core State Standards. ATI is
a Tucson based company having been founded by graduates of the U of A. ATI currently implements the Galileo system across Arizona, California
and in school districts in the eastern United States.
History with TUSD: TUSD was an early adopter of the Galileo system. Records show the use of Galileo as far back as 2005. Prior to 2007 TUSD
implemented ATI assessments as benchmark assessments, built around an articulated curriculum. After 2007 the use of ATI was made optional as
schools were given wide discretion in the choice and implementation of curriculum and assessment. Title I schools continued to implement ATI as
part of their funding, and implemented ATI as a benchmark assessment.
In 2012 as part of the Unitary Status Plan (USP) the district was required to identify an assessment tool for most schools. ATI Galileo was chosen
as the assessment tool for the district because it was already in use in many schools and many staff was familiar with it's use. The district chose to
implement a comprehensive test built on the AIMS blueprint for participating schools. Given that an articulated curriculum had been de-emphasized
during the years of de-centralization, the comprehensive test provided the best way to administer a consistent test across the district’s schools.
Data results from the assessments are reported to the Federal courts as part of the desegregation process.