CENTER ON RESPONSE TO INTERVENTION AT AIR
STANDARD PROTOCOL
FOR EVALUATING RESPONSE TO
INTERVENTION TOOLS:
READING AND MATH SCREENING
August 2013
Screening involves brief assessments that are valid, reliable, and evidence-based. They are
conducted with all students or targeted groups of students to identify students who are at risk of
academic failure and, therefore, likely to need additional or alternative forms of instruction to
supplement the conventional general education approach.
1
Please Read and Complete Before You Start
In order to know if your tool qualifies for review, please answer the following four
questions regarding the research evidence for your tool:
(1) Can you provide direct evidence* on the effects of using
your tool?
YES_____
NO_____
YES_____
NO_____
YES_____
NO_____
YES_____
NO_____
YES_____
NO_____
YES_____
NO_____
(3) Is your outcome variable a reading (for reading
screeners) or math (for math screeners) measure?
YES_____
NO_____
(4) Are there at least three months between the screening
and your outcome measure?
YES_____
NO_____
*Direct evidence refers to data from a study that has been conducted
based on the tool submitted for evaluation. Studies that use data from
the use of another tool, even if it is similar, are considered indirect
evidence and will not be considered as adequate evidence for the
purposes of this review.
(2) Do you have the following classification data for your
tool?
a. Specificity
b. Sensitivity
c. Positive predictive power
d. Negative predictive power
e. Area Under the Curve (AUC) derived from a
Receiver Operating Characteristic (ROC) curve
analysis
If you cannot answer yes to all of these questions, we will not review your tool. If
you are able to answer YES to all of the above questions, then your tool qualifies
for review. Please proceed to the following page and begin filling out the protocol.
PLEASE NOTE: If your tool assesses more than one component skill of reading,
you MUST submit separate protocols for EACH sub-test that is used to identify
students.
The Center on Response to Intervention at AIR will review all of your submitted materials to
ensure that they adhere to the qualifications for review, stated above. If it is found that your
submission packet needs substantial amount of supplemental information or is missing critical
information, the entire packet will be returned to you. A revised protocol packet with additional
information may be re-submitted.
2
Results of the review will be posted on the Center’s website, in the Screening Tools Chart.
Once the review has begun, withdrawal of tools from the process will not be permitted.
If you are unfamiliar with any of the terms in the protocol or unsure of what information to
provide, Center staff will be available to answer your questions or to assist you completing the
protocol for submission. Please contact the Center on Response to Intervention at AIR:
Center on Response to Intervention at AIR
American Institutes for Research
1000 Thomas Jefferson Street, NW
Washington, DC 20007
Phone: 1-877-RTI-4-ALL (1-877-784-4255)
E-mail: rticenter@air.org
3
In order to be eligible for review, you must read and sign the following marketing
language agreement.
Marketing Language Agreement
By signing this agreement I have indicated my understanding of the intent and purpose of
the Center on Response to Intervention at AIR’s tools chart, and my agreement to use language
that is consistent with this purpose in any marketing materials that will be used to publicize my
product’s presence and ratings on the chart.
Specifically, I understand the following:
1. The Technical Review Committee (TRC) rated each submitted tool against
established criteria but did not compare it to other tools on the chart. The presence of
a particular tool on the chart does not constitute endorsement and should not be
viewed as a recommendation from either the TRC or the Center on Response to
Intervention at AIR.
2. All tools submitted for review are posted on the chart, regardless of results. The chart
represents all tools that were reviewed, not those that were “approved.”
When marketing my product, I will not use any language that is inconsistent with the above.
Examples of inappropriate marketing language include, but may not be limited to, the following:
(a) Reference to a “top-ranked” product in comparison to other products on the chart
(b) Reference to “approval” or “endorsement” of the product by the Center on RTI at AIR
If the Center becomes aware of any marketing material on my product that violates this
agreement, I understand that I risk removal of the product from the chart. I also understand that
I may draft language and submit to Center staff for review in advance of releasing it, in order to
ensure compliance with this agreement.
I have read and understand the terms and conditions of this Agreement. By signing
below, I signify my agreement to comply with all requirements contained herein.
Signature
Date
Print Name
Organization
4
SECTION I: BASIC INFORMATION
A. TOOL
Title:
Developer:
Publisher:
Contact Person:
Pub. Date:
Name
Telephone:
Email Address:
B. DESCRIPTIVE INFORMATION
Description of tool:
1. The tool is intended for use in grade(s)
(Check all that apply):
2. The tool is intended for use with ages(s)
(Check all that apply):
☐ Kindergarten
☐
Third Grade
☐ First Grade
☐
Fourth Grade
☐ Second Grade
☐
Fifth Grade
☐
Beyond Fifth Grade
☐
4
☐
7
☐
5
☐
8
☐
6
☐
9
☐
10+
3. What is the language of administration?
1
SECTION I: BASIC INFORMATION
4. What is area of skill screened? (Check all that apply)
Reading
☐ Subword processes:
☐ Phonological processing
☐ RAN
☐ Phonological Memory
☒ Phonological Awareness
☐ Letter sound correspondence
☐ Structural analysis
☐ Word ID:
☐ Accuracy
☐ Speed
☐ Nonword:
☐ Accuracy
☐ Speed
☐ Spelling
☐ Accuracy
☐ Speed
☐ Passage:
☐ Accuracy
☐ Accuracy
☐ Reading comprehension:
☐ Multiple choice questions
☐ Constructed Response
☐ Maze
☐ Other:
☐ Listening comprehension
☐ Multiple choice questions
☐ Constructed Response
☐ Sentence verification
☐ Vocabulary
☐ Expressive
☐ Phonics
☐ Other:
☐ Cloze
☐ Retell
☐ Sentence verification
☐ Cloze
☐ Retell
☐ Receptive
Mathematics
☐ Global Indicator of Mathematics Competence
☐ Accuracy
☐ Speed
☐ Multiple choice questions
☐ Constructed Response
☐ Early Numeracy
☐ Accuracy
☐ Speed
☐ Multiple choice questions
☐ Constructed Response
☐ Mathematics Concepts
☐ Accuracy
☐ Speed
2
SECTION I: BASIC INFORMATION
☐ Multiple choice questions
☐ Constructed Response
☐ Mathematics Computation
☐ Accuracy
☐ Speed
☐ Multiple choice questions
☐ Constructed Response
☐ Mathematic Application
☐ Accuracy
☐ Speed
☐ Multiple choice questions
☐ Constructed Response
☐ Fractions/Decimals
☐ Accuracy
☐ Speed
☐ Multiple choice questions
☐ Constructed Response
☐ Algebra
☐ Accuracy
☐ Speed
☐ Multiple choice questions
☐ Constructed Response
☐ Geometry
☐ Accuracy
☐ Speed
☐ Multiple choice questions
☐ Constructed Response
☐ Other (List specific skills or subtest)
3
SECTION I: BASIC INFORMATION CONTINUED
NOTES:
5. Acquisition Information:
Where to Obtain:
Address:
Phone #:
Web Site:
Cost per student for Year 1:
Including:
$
Complete Kit (describe contents)
$
Manuals and Test Materials
$
Directions for Administration
$
Test Forms
$
Technical Manuals
$
Protocol per Student
Other (Describe below):
$
$
$
$
4
SECTION I: BASIC INFORMATION CONTINUED
Cost per student for subsequent years:
Including:
$
Complete Kit (describe contents)
$
Manuals and Test Materials
$
Directions for Administration
$
Test Forms
$
Technical Manuals
$
Protocol per Student
Other (Describe below):
$
$
$
$
$
Additional Information on the tools:
Describe basic pricing plan and/or structure of the tools. Also, provide information on what is included in
the published tools, including information about special accommodations for students with disabilities.
5
SECTION II: EFFICIENCY
A. TIME, ADMINISTRATION, AND FREQUENCY
Assessment format (Check all that apply):
☐
Check lists
☐
☐
Direct (please specify):
Questionnaire
☐ Computerized
☐ One to one
☐ Group Administered
Administration time:
minutes per student
minutes per total group
Additional scoring time:
minutes per student
minutes per total group
Discontinue Rules:
☐ yes
☐ no
☐ basals
☐ ceilings
☐ other:
Alternate forms available:
☐ yes
Number per grade, test level, or units:
☐ no
Are norms available?
☐ yes
☐ no
Are benchmarks available?
☐ yes
☐ no
How many benchmarks per year?
What months during year?
B. TRAINING
Time required for training tester:
☐ less than 1 hour of training
☐ 1-4 hours of training
☐ 4-8 hours of training
☐ information not available
6
SECTION II: EFFICIENCY CONTINUED
Minimum qualifications of the examiner:
☐ professional
☐ paraprofessional
☐ information not available
Training manuals and materials available:
☐ yes
☐ no
Training manuals/materials field-tested:
Training manuals/materials included in cost of tools:
Sources for ongoing technical support available:
☐
yes
☐
no
☐
yes
☐
no
☐
yes
☐
no
C. SCORING
Types of scores available (Check all that apply):
☐ raw score
☐ standard score
☐ percentile score
☐ grade equivalents
☐ IRT-based score
☐ age equivalents
☐ stanines
☐ normal curve equivalents
☐ developmental benchmarks:
☐ cut points
☐ benchmarks
☐ equated
☐ probability
☐ lexile score
☐ subscale/subtest scores
☐ composite scores
☐ error analysis
☐ Other (specify):
Is a scoring key available?
Basis for calculating standard & percentile scores:
☐
yes
☐
no
☐ age norms
☐ stanines
☐ grade norms
☐ normal curve equivalents
7
SECTION II: EFFICIENCY CONTINUED
Scoring Structure (specify how raw scores are calculated and what comprises cluster/composite score):
Describe the tool’s approach to screening, behavior samples, test format, and/or scoring practices, including steps
taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with
disabilities.
Describe the tool’s theoretical justification.
8
SECTION III: TECHNICAL INFORMATION
A. CLASSIFICATION ACCURACY
1a. Please provide the following classification data1 for your tool:
Screen
At-Risk
Not At-Risk
RD
True Positive
a
Not RD
False Positive
b
Total
a+b
False Negative
c
True Negative
d
c+d
N = a+b+c+d
False Positive Rate [b/(b+d)] =
False Negative Rate [c/(a+c)] =
Sensitivity [a/(a+c)] =
Specificity [d/(b+d)] =
Positive Predictive Power [a/(a+b)] =
Negative Predictive Power [d/(c+d)] =
Overall Classification Rate [(a+d)/(a+b+c+d)] =
Area Under the Curve (AUC) = ____________________
What is the base rate in the sample [(a+c)/(a+b+c+d)] =
What were the cut-points?
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Please provide specificity for the following three sensitivity levels:
Sensitivity 90%: Specificity =_____________
Sensitivity 80%: Specificity =_____________
Sensitivity 70%: Specificity =_____________
1
For assistance in calculating these values, please use the following worksheet: Classification Data. For assistance
in conducting a ROC curve analysis and computing AUC, please see the following guidance document: Receiver
Operating Characteristic
9
SECTION III: TECHNICAL INFORMATION CONTINUED
1b. Do you provide, in your user’s manual, classification data that are disaggregated by race, ethnicity, or
language proficiency? If so, complete below for each race/ethnicity/LEP group for which you provide
disaggregated classification data. Copy additional pages as needed.
Race-ethnicity/LEP group: __________________________________
Screen
At-Risk
Not At-Risk
RD
True Positive
A
Not RD
False Positive
b
Total
a+b
False Negative
C
True Negative
d
c+d
N = a+b+c+d
False Positive Rate [b/(b+d)] =
False Negative Rate [c/(a+c)] =
Sensitivity [a/(a+c)] =
Specificity [d/(b+d)] =
Positive Predictive Power [a/(a+b)] =
Negative Predictive Power [d/(c+d)] =
Overall Classification Rate [(a+d)/(a+b+c+d)] =
Area Under the Curve (AUC) = ____________________
What is the base rate in the sample [(a+c)/(a+b+c+d)] =
What were the cut-points?
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Please provide specificity for the following three sensitivity levels:
Sensitivity 90%: Specificity =_____________
Sensitivity 80%: Specificity =_____________
Sensitivity 70%: Specificity =_____________
10
SECTION III: TECHNICAL INFORMATION CONTINUED
2. Please provide a description of the study sample on which the preceding data are based.
National representation:
☐
Yes
☐
No
Local representation:
☐
Yes
☐
No
Date:
Number of States:
Size:
Regions:
Gender (Percent):
Male
Female
Unknown
SES (Percent, measured as free or reduced-price lunch)
Eligible for free or reduced-price lunch
Other SES Indicators:
Race/Ethnicity (Percent):
White, Non-Hispanic
Black, Non-Hispanic
Hispanic
American Indian/Alaska Native
Asian/Pacific Islander
Other
Unknown
Disability classification (Please describe):
First language (Please describe):
Language proficiency status (Please describe):
3. In the aforementioned study, was a separate external reading or pre-reading measure with national
norms administered at the beginning of the study?
☐ Yes
☐ No
If yes, please describe:
If yes, what percentage of children in the sample scored:
___ below the 25th percentile
___ between the 25th and 75th percentile
___ above the 75th percentile
11
SECTION III: TECHNICAL INFORMATION CONTINUED
4. Were the children in the study involved in an intervention in addition to typical classroom instruction
between the screening measure and outcome assessment?
☐ Yes
☐ No
If yes, please describe the intervention, what children are receiving the intervention, and how they were
chosen:
5. Describe the criterion (outcome) measure(s) and how it is broken down into risk and no-risk categories.
6. Describe how the classification analyses were performed and cut-points determined.
12
SECTION III: TECHNICAL INFORMATION CONTINUED
B. CROSS-VALIDATION
1. Has a cross validation study been conducted?
☐
Yes
☐
No
Note: In a cross-validation study, previously identified classification rules are used to predict risk and non-risk in the
new sample.
If yes, provide the following classification data for the cross validation study:
Screen
At-Risk
Not At-Risk
RD
True Positive
a
Not RD
False Positive
b
Total
a+b
False Negative
c
True Negative
d
c+d
N = a+b+c+d
False Positive Rate [b/(b+d)] =
False Negative Rate [c/(a+c)] =
Sensitivity [a/(a+c)] =
Specificity [d/(b+d)] =
Positive Predictive Power [a/(a+b)] =
Negative Predictive Power [d/(c+d)] =
Overall Classification Rate [(a+d)/(a+b+c+d)] =
Area Under the Curve (AUC) = ____________________
What is the base rate in the sample [(a+c)/(a+b+c+d)] =
What were the cut-points?
___________________________________________________________________________________
___________________________________________________________________________________
___________________________________________________________________________________
Please provide specificity for the following three sensitivity levels:
Sensitivity 90%: Specificity =_____________
Sensitivity 80%: Specificity =_____________
Sensitivity 70%: Specificity =_____________
13
SECTION III: TECHNICAL INFORMATION CONTINUED
2. Please provide a description of the cross-validation study sample.
National representation:
☐
Yes
☐
Local representation:
☐
Yes
☐ No
Date:
No
Number of States:
Size:
Regions:
Gender (Percent):
Male
Female
Unknown
SES (Percent, measured as free or reduced-price lunch)
Eligible for free or reduced-price lunch
Other SES Indicators:
Race/Ethnicity (Percent):
White, Non-Hispanic
Black, Non-Hispanic
Hispanic
American Indian/Alaska Native
Asian/Pacific Islander
Other
Unknown
Disability classification (Please describe):
First language (Please describe):
Language proficiency status (Please describe):
3. In the cross-validation study, was a separate external reading or pre-reading measure with national
norms administered at the beginning of the study?
☐ Yes
☐ No
If yes, please describe:
If yes, what percentage of children in the sample scored:
___ below the 25th percentile
___ between the 25th and 75th percentile
___ above the 75th percentile
14
SECTION III: TECHNICAL INFORMATION CONTINUED
4. Were the children in the cross-validation study involved in an intervention in addition to typical
classroom instruction between the screening measure and outcome assessment?
☐ Yes
☐ No
If yes, please describe the intervention, what children are receiving the intervention, and how they were
chosen:
5. Describe the criterion measure(s) for the cross-validation study and how it is broken down into risk and
no-risk categories.
6. Describe how the classification analyses for the cross-validation study were performed and cut-points
determined.
15
SECTION III: TECHNICAL INFORMATION CONTINUED
C. RELIABILITY
1a. In the chart below, report reliability information (e.g., Cronbach’s alpha, test-retest, split-half, and/or
inter-rater reliability estimates).
If you include inter-rater reliability, provide formula:
Type of
Reliability
Age or
Grade
n
(range)
Coefficient
range median
Manual cites other published reliability studies:
*Attach manual.
*Attach published studies.
SEM
☐ yes
Information (including normative data)/Subjects
☐ no
16
SECTION III: TECHNICAL INFORMATION CONTINUED
1b. Do you provide, in your user’s manual, reliability data that are disaggregated by diverse populations
(e.g., race-ethnicity, ELL students)? If so, complete below for each group for which you have
disaggregated reliability data.
If you include inter-rater reliability, provide formula:
Type of
Reliability
Age or
Grade
n
(range)
Coefficient
range median
Manual cites other published reliability studies:
*Attach manual.
*Attach published studies.
SEM
☐ yes
Information (including normative data)/Subjects
☐ no
Copy additional pages as needed.
17
SECTION III: TECHNICAL INFORMATION CONTINUED
D. VALIDITY
1a. In the chart below, report information on predictive, content, and construct validity.2
Type of
Validity
Age or
Grade
Test or
Criterion
n
(range)
Coefficient (if
applicable)
range
Median
Information (including normative
data)/Subjects
Other forms of validity:
Manual cites other published validity studies:
*Attach manual.
*Attach published studies.
☐ yes
☐ no
2
For predictive validity, the TRC suggests predicting to an outcome between 3 months and 1 year after the screen. For
construct validity, the TRC suggests predicting to an outcome that is within 3 months of the screen.
18
SECTION III: TECHNICAL INFORMATION CONTINUED
1b. Do you provide, in your user’s manual, validity data that are disaggregated by diverse populations (e.g.,
race-ethnicity, ELL students)? If so, complete below for each group for which you have disaggregated
validity data.
Type of
Validity
Age or
Grade
Test or
Criterion
n
(range)
Coefficient (if
applicable)
range
median
Information (including normative
data)/Subjects
Other forms of validity:
Manual cites other published validity studies:
*Attach manual.
*Attach published studies.
☐ yes
☐ no
Copy additional pages as needed.
19