2010 STAR
Interpreting and Using
Results
August 11, 2010
Webcast
Webcast starts at 9:00 a.m.
Objectives
Workshop participants will be able to:
• Describe the purposes of STAR reports
• Interpret STAR results
• Explain key statistics
• Compare and contrast types of reports
• Identify proper uses of reports
2
August 2010
Post-Test Workshop
Agenda
•
•
•
•
•
•
•
3
What’s New?
Results and Statistical Analysis
Using Results
Summary and Internet Reports
Data CDs
Individual Student Reports
Early Assessment Program
August 2010
Post-Test Workshop
What’s New in 2010
Manual
• California Modified Assessment (CMA) (M) 2
reports
– Scale score, performance levels for
• Grades 6–8 English−Language Arts
(ELA)
• Grades 6–7 mathematics
• Grade 8 science
– Percent correct for
• ELA grade 9
• Life Science, grade 10
• Algebra I
4
August 2010
Post-Test Workshop
More What’s New
• Standards-based Test in Spanish (STS) reports
– Scale score, performance levels for
• Grades 5–7 Reading/Language Arts (RLA)
• Grades 5–7 mathematics
– Percent correct for
• Grades 8–11 RLA
• Grades 7–11 Algebra I
• Grades 8–11 Geometry
• Subgroup Summary: Ethnicity
• Grade 11 – Early Assessment Program (EAP) –
California Community Colleges added
5
August 2010
Post-Test Workshop
M2
Quiz Question 1
Which of these tests had scale scores
reported for the first time in 2010?
A. CST for World History
B. CAPA for Science
C. CMA for math grade 7 and science
grade 8
D. STS for math grade 7 and Algebra I
6
August 2010
Post-Test Workshop
Quiz Question 1
Which of these tests had scale scores
reported for the first time in 2010?
C. CMA for math grade 7 and science
grade 8
7
August 2010
Post-Test Workshop
Results:
Purposes of STAR Reports
• Report progress of students toward
proficiency on the state’s academic
content standards
• Notify where improvement needed
– To help students’ achievement
– To improve educational programs
• Provide data for state and federal
accountability programs
8
August 2010
Post-Test Workshop
M4
Results:
Performance Levels
• State Goal: All students score at
proficient or higher
• CAPA Proficient: 35 or higher scale
score
• 350 or higher scale score
– All CST
– STS, grades 2–7,
excluding Algebra I
– CMA, grades 3–8,
excluding Algebra I
9
August 2010
Post-Test Workshop
M 8−10
Results:
Other Performance Levels
• Advanced
• Basic cut score
– CAPA: 30
– CST: 300
– CMA, grades 3–8: 300
(excluding Algebra I)
– STS, grades 2–7: 300
(excluding Algebra I)
• Below basic
• Far below basic
• Cut points vary for
advanced and below basic by
– Subject
– Grade
10
August 2010
Post-Test Workshop
M 8−10;
Appendix B
Results: Scale Scores
M 8−10
• Scale scores allow same score to mean
same thing across test versions within
grade and content area
• Accounts for differences in difficulty
• Scale score ranges by program:
– CST, CMA (grades 3–8, excluding
Algebra I), STS (grades 2–7, excluding
Algebra I): 150–600 for each grade and
subject
– CAPA: 15–60 for each level and subject
11
August 2010
Post-Test Workshop
Results: Equating
• Psychometric procedure
• Adjusts for test difficulty
• Additional information in the CST
Technical Report on the CDE Web site
12
August 2010
Post-Test Workshop
M8
Results: Reporting Clusters
(Content Areas)
• Three to six clusters for each subject
M 8−12;
Appendix A
• May be useful as indicators of individual or group
strengths and weaknesses
• But . . . Reporting clusters
should be interpreted with
caution
13
August 2010
Post-Test Workshop
Results: Cluster Percent Correct
Available for 2010
M 8−12;
Appendix A
• CST: all
• CMA:
– Grades 3–8, excluding Algebra I
• STS: all
14
August 2010
Post-Test Workshop
Results: Reporting Cluster
Cautions
• Based on small numbers
of items; therefore, may not
be reliable or generalizable
• NOT equated from year to
year
• Should not compare
reporting cluster percent
correct from year to year
15
August 2010
Post-Test Workshop
M 8−12;
Appendix A
Interpreting Reporting Clusters or
Content Areas
M 8−12;
Appendix A
in the Same Year
• Compare to percent-correct range of
proficient students statewide
16
August 2010
Post-Test Workshop
2010 CST Reporting Clusters:
Number of Questions and
M Appendix A
Average Percent Correct
To be finalized with complete data in
2010 Post-Test Guide, after August 16.
17
August 2010
Post-Test Workshop
Examples—Interpreting Reporting
Clusters for the CST for Geometry
18
August 2010
Post-Test Workshop
M 11
Quiz Question 2
What is a scale score?
A. Percent correct of all questions
B. Mean percent correct of all questions
C. An adjustment of this year’s and last
year’s raw scores to show changes
D. An adjustment of the raw score to
account for differences in difficulty
19
August 2010
Post-Test Workshop
Quiz Question 2
What is a scale score?
D. An adjustment of the raw score to
account for differences in difficulty
20
August 2010
Post-Test Workshop
Using Results
M4
• For instructional decisions with other
data
• Used in Academic Performance Index
(API) calculations: CSTs, CAPA, CMA
(only grades 3–8, excluding Algebra I)
• Used in adequate yearly progress (AYP)
calculations, ELA and mathematics:
– Grades 2–8, CSTs
– Grades 3–8, CMA (excluding Algebra I)
– Grades 2–8 and 10, CAPA
21
August 2010
Post-Test Workshop
Year-to-Year Comparisons
M 13−16
Do Compare CSTs:
Same Grade and Same Content Area
• Mean scale score
– Same content and grade, varying
years
• Percent in each performance level
– Same content by grade across
years
• e.g., 2009 ELA grade 10 with 2010
ELA grade 10
22
August 2010
Post-Test Workshop
Year-to-Year Comparisons
Do Compare CSTs:
M 13−16
Percent Proficient and Advanced
• Percentage of students scoring at PROFICIENT
and above
– For a given grade and subject, e.g., Percent
proficient and above for grade 3 math in 2009
and 2010
– For a given subject and aggregated grades,
e.g., Percent proficient and above for grades
2– 6 mathematics in 2009 and 2010
– Across grades and a subject, e.g., Percent
proficient and above in all courses and all
grades
23
August 2010
Post-Test Workshop
Year-to-Year Comparisons
DON’T Compare
M 13−16
• Individual scale scores or statistics based on
scale scores for different grades or content
areas
– Subjects by grade are independently scaled
– Different content standards are measured in
different grades
• Cohorts across grades
• Across tests
• CMA grades 6−11, STS grades 5−11 to
previous years
• CAPA, CMA grades 3−5, STS grades 2−4 to
years before 2009
24
August 2010
Post-Test Workshop
Example—Using CST Results to
Compare Grade Results from Year to Year
2009 CST
for ELA
2010 CST
for ELA
% Prof or
Above
% Prof or
Above
Difference
Grade 2
31%
35%
4%
Grade 3
33%
33%
0%
Grade 4
29%
31%
2%
Grade 5
34%
32%
–2%
Grade 6
31%
32%
1%
All Grades
32%
33%
1%
Grade
25
M 14
August 2010
Post-Test Workshop
Quiz Question 3
What is the best comparison for CST scores of
students within a middle school?
A. 2009 mean scale scores for ELA of a cohort of
grade 7 students with 2010 scale scores for ELA
of the same students in grade 8
B. 2009 mean scale scores for ELA for grade 8
students with 2010 mean scale scores for ELA
for grade 8 students
C. 2009 mean percent correct scores for ELA with
2010 percent correct scores for ELA for the
same students in grade 8
D. 2009 mean percent correct scores for ELA for
grade 8 students with 2010 mean percent
correct for ELA for grade 8 students
26
August 2010
Post-Test Workshop
Quiz Question 3
What is the best comparison for CST scores of
students within a middle school?
B. 2009 mean scale scores for ELA for grade 8
students with 2010 mean scale scores for ELA
for grade 8 students
27
August 2010
Post-Test Workshop
Quiz Question 4
Which is the best comparison of cluster
scores for a single student? Compare . . .
A. To proficient students statewide
B. One cluster to another, same year
C. The same cluster to the same cluster,
different years
D. To the average percent correct of all
students in a class
28
August 2010
Post-Test Workshop
Quiz Question 4
Which is the best comparison of cluster
scores for a single student? Compare…
A. To proficient students statewide
29
August 2010
Post-Test Workshop
Aggregate (Summary) Reports
M 18−22
• What are they?
– Student Master List Summary
– Student Master List Summary End-ofCourse (EOC)
– Subgroup Summary
• Report Emphasis: CSTs
– Criterion-referenced tests
– Progress is measured in percent of
students scoring proficient and
advanced
30
August 2010
Post-Test Workshop
Student Master List Summary
•
•
•
•
•
M 19–20,
M 28−34
By grade
CSTs, CMA, CAPA, and STS
Lists subjects
% and # at each performance level
Mean scale score (CST, CAPA, CMA
grades 3–8 [not Algebra I], STS grades 2–7
[not Algebra I])
• Reporting cluster: mean percent correct
(CST, CMA grades 3–8 [not Algebra I], STS)
31
August 2010
Post-Test Workshop
Student Master List Summary
M 34
Grade 7 Example
32
August 2010
Post-Test Workshop
Student Master List Summary
Basic Statistics
M 28−34
33
August 2010
Post-Test Workshop
Who Counts?
Number Enrolled
M 28−34
• Total CST/CMA and CAPA multiplechoice answer documents submitted as
scorable
• Minus
- Documents marked as “Student
enrolled after the first day of testing and
was given this test”
34
August 2010
Post-Test Workshop
Who Counts? Number Tested
M 30−32
• All CST, CMA, CAPA, STS answer documents
with one or more answers
• Plus
– Z = Tested but marked no answers
• Not included
– A = Students absent
– E = Not tested due to significant medical
emergency
– P = Parent/guardian exemptions
– T = Enrolled first day, not tested, tested at
previous school
– Students with inconsistent grades
– Non–English learners who took the STS
35
August 2010
Post-Test Workshop
Who Counts? Number and
Percent Valid Scores
M 30−32
• Number Valid Scores
– For the subject, number of students tested at
grade level who received a score for the test
(that is, a scale score or percent correct).
• Percent Valid Scores
– For the subject, number of valid scores
divided by the number of students tested.
– Not included:
 Incomplete tests
 Modified tests
 Non–English learners who took the STS
 Unknown EOC mathematics (except grade 7
mathematics) or science tests
 Inconsistent grades
36
August 2010
Post-Test Workshop
Who Counts?
Number Tested with Scores
M 30−32
• All tests taken, including those taken
with modifications, that receive a
score
• Not included:
– Incomplete tests
– Non–English learners who took the
STS
– Unknown EOC mathematics (except
grade 7 mathematics) or science
tests
– Inconsistent grades
37
August 2010
Post-Test Workshop
Who Counts?
Performance Levels
•
•
•
•
M 30−32
All CSTs, CAPA
CMA Grades 3–8, excluding Algebra I
STS Grades 2–7, excluding Algebra I
Advanced, proficient, basic, below basic
– All valid scores falling in the performance level
• Far below basic
– All valid scores falling in the performance level
– CSTs taken with modifications (in aggregate
reporting and accountability only)
38
August 2010
Post-Test Workshop
Student Master List Summary
Performance Levels
39
August 2010
Post-Test Workshop
M 31
Student Master List Summary
Reporting Clusters
M 31−32
Compare to:
Average percent correct range for students
statewide who scored proficient on the total test
40
August 2010
Post-Test Workshop
Student Master List Summary:
Writing
B
C
I
L
41
=
=
=
=
August 2010
Blank
Copied prompt
Illegible
Language other
than English
M 32
R = Refusal
T = Off Topic
W = Wrong prompt
(prompt from an
earlier administration)
Post-Test Workshop
Student Master List Summary: M 35−40
End-of-Course (EOC) CST, CMA, and STS
• By subject
• Lists each grade eligible to take test
• Mathematics (Grades 7–11)
– CST General Math, Algebra I, Geometry, etc.
– CMA Algebra I
– STS Algebra I and Geometry
• CST Science (Grades 9–11)
– Earth Science, Biology, Chemistry, etc.
• CST History–Social Science (Grades 9–11)
– World History
• Same statistics as grade-level Student Master
List Summary
42
August 2010
Post-Test Workshop
Student Master List Summary
End-of-Course
CST Biology Example
43
August 2010
Post-Test Workshop
M 40
Subgroup Summary:
CSTs, CMA, CAPA, and STS
M 41−59
• Disability status
– Based on disability status for CST, CMA, STS
– CAPA: each disability type
• If missing, correct with demographic data
corrections
• Economic status
– Based on NSLP eligibility or parent education
level
• Gender
• English proficiency
• Ethnicity
• Ethnicity for Economic Status (only for CSTs,
CMA grades 3–8, and CAPA)
44
August 2010
Post-Test Workshop
Subgroup Summary: Gender
Grade 5 Example
45
August 2010
Post-Test Workshop
Subgroup Summary: Ethnicity
• From answer document sections 8 and 9:
– Hispanic or Latino?
– Race: grouped and worded differently
– May cause counts to differ from previous
year
– Federal Reporting Requirement
• Two or more races (instead of “unknown”)
46
August 2010
Post-Test Workshop
Subgroup Summary:
Ethnicity for Economic Status
Ethnicity for Economically Disadvantaged
CST Grade 6 Example
47
August 2010
Post-Test Workshop
M 64
Subgroup Summary:
Ethnicity for Economic Status M 60−64
Example: Economically
disadvantaged for each
ethnicity
48
August 2010
Post-Test Workshop
Subgroup Summary:
Ethnicity for Economically Disadvantaged
M 60−64
49
August 2010
Post-Test Workshop
Break — 10 minutes
Internet Reports
M 104−117
• Summaries based on same data as
paper reports: CSTs, CMA, CAPA, STS
• Available to the public online for
school, district, county, and state
• More subgroups than paper reports
– Parent education
– Special program participation
• Access from http://star.cde.ca.gov/
51
August 2010
Post-Test Workshop
Internet Reports:
Available Reports
• CSTs
– Mean scale scores
– Percents by
performance levels
• CST Summary
– Percent proficient
and advanced
• CMA
– Grades 3–8 (not
Algebra I): mean
scale scores,
percents by
performance levels
– Grade 9 ELA, Grade
10 Life Science,
Algebra I: average
percent correct
52
August 2010
M 104−117
• CAPA
– Mean scale scores
– Percents by
performance levels
• STS
– Grades 2–7 (not
Algebra I): mean
scale scores,
percents by
performance levels
– Grades 8–11:
average percent
correct
Post-Test Workshop
Internet Demonstration
53
August 2010
Post-Test Workshop
Internet Reports:
CST Sample
54
August 2010
Post-Test Workshop
M 108−109
Internet Reports:
CST Summary Sample
55
August 2010
Post-Test Workshop
M 109−110
Other Internet Reports
• CMA (M 111−112)
– Same as CST—CMA grades 3−8
– Percent correct—CMA grades 9–11, Algebra I
• CAPA (M 112−115)
– State level: same as CST; separate Level I
– County, district, school
• Mean scale score
• Percent proficient or above
• STS (M 116−117)
– Same as CST—STS grades 2−7
– Percent correct—STS grades 8−11
56
August 2010
Post-Test Workshop
Quiz Question 5
Which subgroup can only be accessed
from the Internet?
A. Parent Education Level
B. CAPA by individual disability status
C. Ethnicity for Economic Status
D. English Proficiency
57
August 2010
Post-Test Workshop
Quiz Question 5
Which subgroup can only be accessed
from the Internet?
A. Parent Education Level
58
August 2010
Post-Test Workshop
Data CDs
• What are they?
– Lists of information from answer documents
and scores of every student in district
– In .txt format: wraps
• What are they used for?
– Searching for specific data
– Creating unique reports
– Verifying paper reports
• What else is needed?
– Text editor
– or Desktop application
– or Student Information System
59
August 2010
Post-Test Workshop
View of Data
• As .txt, word wrap on
• With text editor, word wrap off
60
August 2010
Post-Test Workshop
Organization of Data
• Two files:
– Demographics, special conditions, and test
scores
– Accommodations, modifications, English
Learners, and irregularities
• Data Layout = guide to location of data
on files
– Position
– Number of characters
– Whether numeric or alpha
61
August 2010
Post-Test Workshop
Data Layout Sample
62
August 2010
Post-Test Workshop
Individual Reports
• STAR Student Record Label
M 23−26
– Adhesive label to affix to student’s permanent
school record
• STAR Student Master List
– Alphabetical list of students and their scores
– Tests listed in order within grade
• CSTs
• CMA
• CAPA
• STS
• STAR Student Report: individual’s scores
– 2 two-sided color copies for each test
– For parents/guardians, school
63
August 2010
Post-Test Workshop
Student Record Label Grade 10 Sample:
Student Name and Identification
M 65
64
August 2010
Post-Test Workshop
Student Record Label
M 65−66
CST/CMA Grade 10 Example
65
August 2010
Post-Test Workshop
Student Master List
CST/CMA Grade 3 Example
66
August 2010
Post-Test Workshop
M 69−71
Student Report
M 72; 73−82
CST Grade 11 Example
67
August 2010
Post-Test Workshop
Student Report
CST Grade 11 Example
68
August 2010
Post-Test Workshop
M 74−75
Student Report
CST Grade 10 Example
69
M 76−78
Student name
on back
♦ = Percent correct obtained by the student
on the reporting cluster/content area
▬ = Range of percent correct scores on the
reporting cluster for students statewide
who scored proficient on that test
August 2010
Post-Test Workshop
Student Report
CST Grade 11 Example, EAP Results
70
August 2010
Post-Test Workshop
M 78; 82
Explain to Parents
M 13
Scale Score  Average % correct cluster
score x 600
• Reporting Clusters not comparable
– Different difficulty
– Varying number of questions
– Average % correct of clusters  % correct of
total scale score
• Scale scores
– Use conversion tables and other statistical
techniques
– Equating allows scores to have similar
meaning
(e.g., 350 = lowest score for CSTs proficient)
71
August 2010
Post-Test Workshop
Other Student Reports
• CMA (M 82−90)
– Performance levels and scale scores reported
for grades 3–8
 Back, cluster reporting for grades 3–8
– Percent correct for entire test for grades 9–11
 Back, “About the CMA”
• CAPA (M 91−95)
– Back, “About the CAPA”
• STS (M 96−103)
– In Spanish
– Performance levels and scale scores, grades 2–7
 Back, cluster percent correct
– Percent correct for entire test, grades 8–11
 Back, cluster percent correct
 Back, how to use report
72
August 2010
Post-Test Workshop
Unmatched Reports Grade 7 Writing
• Unmatched Multiple-Choice Report
 CST multiple-choice score but no writing
score
 CMA multiple-choice score but no writing
score
• Unmatched Writing Report
 Writing score but no CST multiple-choice
score
 Writing score but no CMA multiple-choice
score
• Students receive 2 reports if writing score
not matched to multiple-choice score
73
August 2010
Post-Test Workshop
Unmatched Report
CMA Grades 8−11
•Student took at least 1 CMA test, but no CST
report matches
•All grades except grade 9 require a CST
Grade
8
9
10
11
Tests That Can be
Taken as CST or CMA
ELA, Algebra I, Science
ELA, Algebra I
Algebra I, Life Science
Algebra I
CST Required
(No CMA option)
History-Social Science
ELA
ELA, History-Social
Science
Table does not include end-of-course CSTs for
grades 8−11
74
August 2010
Post-Test Workshop
Unmatched Report CMA Grades 8−11
• Report for each grade
• Lists student data for those with only CMA
or demographics do not match CST
75
August 2010
Post-Test Workshop
Summary
• Do’s and Don’ts
– Do compare mean scale score, percent
at performance levels within same
grade, same content area
– Don’t compare mean scale scores
across grades, content area
• Summary and Internet reports
• Data CDs
• Individual reports
76
August 2010
Post-Test Workshop
For more information see:
•
•
•
•
http://www.cde.ca.gov/ta/tg/sr/
http://www.star.cde.ca.gov/
http://www.startest.org/
STAR Technical Assistance Center
– 800-955-2954
• CDE Accountability
– aau@cde.ca.gov
– 916-319-0863
77
August 2010
Post-Test Workshop
To Contact CDE
• CDE STAR office: 916-445-8765
– English–Language Arts tests, including
writing: Jamie Contreras, jcontrer@cde.ca.gov,
916-319-0353
– Mathematics tests: Jane Liang,
jliang@cde.ca.gov, 916-322-1854
– Science tests: Blessing Mupanduki,
bmupanduki@cde.ca.gov, 916-319-0969
– History–Social Science tests: Nicole
Jespersen, njespersen@cde.ca.gov, 916-3190364
– CAPA, CMA: Don Killmer,
dkillmer@cde.ca.gov, 916-319-0350
78
August 2010
Post-Test Workshop