2009 STAR
Interpreting and Using
Results
August 2009
Webcast
Objectives
Handout (HO) 1
Workshop participants will be able to:
• Describe the purposes of STAR reports
• Interpret STAR results
• Explain key statistics
• Compare and contrast types of reports
• Identify proper uses of reports
2
August 2009
Post-Test Workshop
Agenda
•
•
•
•
•
•
•
•
3
What’s New?
Results and Statistical Analysis
Using Results
Summary and Internet Reports
Data CDs
Individual Student Reports
Teacher Reports
Early Assessment Program
August 2009
Post-Test Workshop
HO 1
What’s New in 2009
• CMA reports
– Scale score, performance levels for
grades 3–5
– Percent correct for grades 6–8
– Writing scores for grades 4 and 7
• CAPA new cut scores for all including
scale scores for science
• STS reports
– Scale score, performance levels for
grades 2–4
– Percent correct for grades 5–11
• Data CD: two files only
4
August 2009
Post-Test Workshop
HO 2–3
Results:
Purposes of STAR Reports
• Report progress of students toward
proficiency on the state’s academic
content standards
• Notify where improvement needed
– To help students’ achievement
– To improve educational programs
• Provide data for state and federal
accountability programs
5
August 2009
Post-Test Workshop
HO 4
Results:
Performance Levels
• State Goal: All students score at
proficient or higher
• CAPA Proficient: 35 or higher scale
score
• 350 or higher scale score
– All CST
– STS, grades 2–4
– CMA, grades 3–5
6
August 2009
Post-Test Workshop
HO 4
Results:
Other Performance Levels
• Advanced
• Basic Cut Score
– CAPA: 30
– CST: 300
– CMA, grades 3–5: 300
– STS, grades 2–4: 300
• Below basic
• Far below basic
• Cut points vary for
advanced and below basic by
– Subject
– Grade
7
August 2009
Post-Test Workshop
HO 4
Results: Scale Scores
HO 5
• Scale scores allow same score to mean same
thing across test versions within grade and
content area
• Scale score ranges by program:
– CST, CMA (grades 3–5), STS (grades 2–4):
150–600 for each grade and subject
– CAPA: 15–60 for each level and subject
8
August 2009
Post-Test Workshop
Results: Equating
HO 5
• Psychometric procedure
• Adjusts for test difficulty
• Additional information in the CST
Technical Report on the CDE Web site
9
August 2009
Post-Test Workshop
Results: Reporting Clusters
(Content Areas)
HO 6–7
• Three to six clusters for each subject
• May be useful as indicators of individual or group
strengths and weaknesses
• But… Reporting clusters
should be interpreted with
caution
10
August 2009
Post-Test Workshop
Results: Cluster Percent Correct
Available for 2009
• CST: all
• CMA:
– Grades 3–5
– Grade 7: Writing only
• STS
– Grades 2–4
– Grades 5–7, but no comparison with
proficient students statewide
11
August 2009
Post-Test Workshop
Results: Reporting Cluster
Cautions
• Based on small numbers
of items; therefore, may not
be reliable or generalizable
• NOT equated from year to
year
• Should not compare
reporting cluster percent
correct from year to year
12
August 2009
Post-Test Workshop
HO 6–7
Interpreting Reporting Clusters or
HO 6–7
Content Areas
in the Same Year
• Compare to percent correct range of
proficient students statewide
13
August 2009
Post-Test Workshop
CSTs 2009 Reporting Clusters:
Number of Questions and
Average Percent Correct
To be finalized with complete data in
2009 post-test guide.
14
August 2009
Post-Test Workshop
HO 8
Examples—Interpreting Reporting
Clusters for CST for Geometry
15
August 2009
Post-Test Workshop
Using Results
• For instructional decisions with other data
• Used in Academic Performance Index (API)
calculations: CSTs, CAPA, CMA (grades 3–5
only)
• Used in adequate yearly progress (AYP)
calculations, ELA and mathematics:
– Grades 2–8 CSTs
– Grades 3–5 CMA
– Grades 2–8 and 10 CAPA
16
August 2009
Post-Test Workshop
Year-to-Year Comparisons
HO 9
Do Compare CSTs:
Same Grade and Same Content Area
• Mean scale score
– Same content and grade, varying years
• Percent in each performance level
– Same content by grade across years
• e.g., 2008 ELA grade 10 with 2009 ELA
grade 10
17
August 2009
Post-Test Workshop
Year-to-Year Comparisons
Do Compare CSTs:
Percent Proficient and Advanced
HO 9
• Percentage of students scoring at PROFICIENT
and above
– For a given grade and subject, e.g., Percent
proficient and above for grade 3 math in 2008
and 2009
– For a given subject and aggregated grades,
e.g., Percent proficient and above for grades
2– 6 mathematics in 2008 and 2009
– Across grades and a subject, e.g., Percent
proficient and above in all courses and all
grades
18
August 2009
Post-Test Workshop
Year-to-Year Comparisons
DON’T Compare
HO 9
• Individual scale scores or statistics based on
scale scores for different grades or content
areas
– Subjects by grade are independently scaled
– Different content standards are measured in
different grades
• Cohorts across grades
• Across tests
• CAPA, CMA, STS to previous years
19
August 2009
Post-Test Workshop
Example—Using CST Results to
Compare Grade Results from Year to Year
2008 CST
for ELA
2009 CST
for ELA
% Prof or
Above
% Prof or
Above
Difference
Grade 2
31%
35%
4%
Grade 3
33%
33%
0%
Grade 4
29%
31%
2%
Grade 5
34%
32%
-2%
Grade 6
31%
32%
1%
All Grades
32%
33%
1%
Grade
20
August 2009
Post-Test Workshop
Aggregate (Summary) Reports
• What are they?
– Student Master List Summary
– Student Master List Summary EOC
– Subgroup Summary
• Report Emphasis: California Standards Tests
(CSTs)
– Criterion-referenced tests
– Progress is measured in percent of students
scoring proficient and advanced
21
August 2009
Post-Test Workshop
Student Master List Summary
•
•
•
•
•
By grade
CSTs, CMA, CAPA, and STS
Lists subjects
% and # at each performance level
Mean scale score (CST, CAPA, CMA
grades 3–5, STS grades 2–4)
• Reporting cluster: mean percent
correct (CST, CMA grades 3–5, STS
grades 2–7)
22
August 2009
Post-Test Workshop
HO 10
Student Master List Summary
Grade 7 Sample
23
August 2009
Post-Test Workshop
HO 10
Student Master List Summary
Basic Statistics
24
August 2009
Post-Test Workshop
HO 10
Who Counts?
Number Enrolled
HO 10
• All CST, CMA, CAPA, and STS multiplechoice answer documents submitted as
scorable
• Minus
- Documents marked as “Student
enrolled after the first day of testing and
was given this test”
25
August 2009
Post-Test Workshop
Who Counts? Number Tested
HO 10
• All CST, CMA, CAPA, STS answer documents
with one or more answers
• Plus
– Z = Tested but marked no answers
• Not included
– A = Students absent
– E = Not tested due to significant medical
emergency
– P = Parent/guardian exemptions
– T = Enrolled first day, not tested, tested at
previous school
– Students with inconsistent grades
– Non English learners who took STS
26
August 2009
Post-Test Workshop
Who Counts? Number and
HO 10
Percent Valid Scores
• Number Valid Scores
– For the subject, number of students tested at
grade level who received a score for the test
(that is, a scale score or percent correct).
• Percent Valid Scores
– For the subject, number of valid scores
divided by the number of students tested.
– Not included:
 Incomplete tests
 Modified tests
 Non-English learners who took the STS
 Unknown EOC mathematics (except grade 7
math) or science tests
 Inconsistent grades
27
August 2009
Post-Test Workshop
Who Counts?
Number Tested with Scores
HO 10
• All tests taken, including those taken
with modifications, that receive a
score
• Not included:
– Incomplete tests
– Non-English learners who took the
STS
– Unknown EOC mathematics (except
grade 7) or science tests
– Inconsistent grades
28
August 2009
Post-Test Workshop
Who Counts?
Performance Levels
•
•
•
•
HO 10
All CSTs, CAPA
CMA Grades 3–5
STS Grades 2–4
Advanced, proficient, basic, below basic
– All valid scores falling in the performance level
• Far below basic
– All valid scores falling in the performance level
– CSTs taken with modifications (in aggregate
reporting and accountability only)
29
August 2009
Post-Test Workshop
Student Master List Summary
Performance Levels
30
August 2009
Post-Test Workshop
HO 10
Student Master List Summary
Reporting Clusters
Compare to:
Average percent correct range for students
statewide who scored proficient on the total
test
31
August 2009
Post-Test Workshop
HO 10
Student Master List Summary:
Writing
B
C
I
L
32
=
=
=
=
August 2009
Blank
Copied prompt
Illegible
Language other
than English
HO 10
R = Refusal
T = Off Topic
W = Wrong prompt
(Prompt from an
earlier administration)
Post-Test Workshop
Student Master List Summary:
End-of-Course (EOC) CST and STS
HO 11
• By subject
• Lists each grade eligible to take test
• Math (Grades 7–11)
– CST General Math, Algebra I, Geometry, etc.
– STS Algebra I and Geometry
• CST Science (Grades 9–11)
– Earth Science, Biology, Chemistry, etc.
• CST History–Social Science (Grades 9–11)
– World History
• Same statistics as grade-level Student Master
List Summary
33
August 2009
Post-Test Workshop
Student Master List Summary
End-of-Course, Biology
34
August 2009
Post-Test Workshop
HO 11
Subgroup Summary:
CSTs, CMA, CAPA, and STS
• Disability status
– Based on Disability Status for CST, CMA, STS
– CAPA: each disability code
• If missing, correct with demographic data
corrections
• Economic Status
– Based on NSLP eligibility and
parent education level
• Gender
• English proficiency
• Primary ethnicity
• Ethnicity for Economic Status (only for CSTs,
CMA grades 3–5, and CAPA)
35
August 2009
Post-Test Workshop
Subgroup Summary:
Ethnicity for Economic Status
36
August 2009
Post-Test Workshop
HO 12
Subgroup Summary:
Ethnicity for Economic Status
HO 12
Example: Economically
disadvantaged for each ethnicity
37
August 2009
Post-Test Workshop
Subgroup Summary:
HO 12
Ethnicity for Economically Disadvantaged
38
August 2009
Post-Test Workshop
Break — 10 minutes
Internet Reports
• Summaries based on same data as
paper reports: CSTs, CMA, CAPA, STS
• Available to the public online for
school, district, county, and state
• More subgroups than paper reports
– Parent education
– Special program participation
• Access from http://star.cde.ca.gov
40
August 2009
Post-Test Workshop
Internet Reports:
Available Reports
• CSTs
– Mean scale scores
– Percents by
performance levels
• CST Summary
– Percent proficient
and advanced
• CMA
– Grades 3–5: mean
scale scores,
percents by
performance levels
– Grades 6-8:
average percent
correct
41
August 2009
• CAPA
– Mean scale scores
– Percents by
performance levels
• STS
– Grades 2–4: mean
scale scores,
percents by
performance levels
– Grades 5-11:
average percent
correct
Post-Test Workshop
Internet Reports
Access Reports
42
August 2009
Post-Test Workshop
HO 13
Internet Reports:
CST Sample
43
August 2009
Post-Test Workshop
HO 13
Internet Reports:
CST Summary Sample
44
August 2009
Post-Test Workshop
HO 14
Internet Reports: CMA
45
August 2009
Post-Test Workshop
HO 14
Internet Reports: STS
HO 15
CAPA
on HO
16
46
August 2009
Post-Test Workshop
Data CDs
• What are they?
HO 18
– Lists of information from answer documents
and scores of every student in district
– In .txt format: wraps
• What are they used for?
– Searching for specific data
– Creating unique reports
– Verifying paper reports
• What else is needed?
– Text editor
– or Desktop application
– or Student Information System
47
August 2009
Post-Test Workshop
View of Data
• As .txt, word wrap on
• With text editor, word wrap off
48
August 2009
Post-Test Workshop
HO 18
Organization of Data
•Two files:
HO 18
– Demographics, special conditions, and test
scores
– Accommodations, modifications, English
Learners, and irregularities
•Data Layout = guide to location of data
on files
– Position
– Number of characters
– Whether numeric or alpha
49
August 2009
Post-Test Workshop
Data Layout Sample
50
August 2009
Post-Test Workshop
HO 18
Individual Reports
• STAR Student Record Label
– Adhesive label to affix to student’s permanent
school record
• STAR Student Master List
– Alphabetical list of students and their scores
– Tests listed in order within grade
• CSTs
• CMA
• CAPA
• STS
• STAR Student Report: individual’s scores
– 2 two-sided color copies for each test
– For parents/guardians
51
August 2009
Post-Test Workshop
Student Record Label Grade 10 Sample:
Student Name and Identification HO 19
52
August 2009
Post-Test Workshop
Student Record Label:
CST for Grade 10 Results
53
August 2009
Post-Test Workshop
HO 19
Student Master List:
CSTs/CMA for Grade 3 Sample
54
August 2009
Post-Test Workshop
HO 20
Student Master List:
CSTs/CMA for Grade 3 Sample
55
August 2009
Post-Test Workshop
HO 20
Student Report:
CST for Grade 10 Sample
56
August 2009
Post-Test Workshop
HO 21–22
Student Report:
CST for Grade 10 Sample
57
August 2009
Post-Test Workshop
HO 21
Student Report:
CSTs for Grade 10 Sample
HO 22
Student name
on back
58
♦ = Percent correct obtained by the student
on the reporting cluster/content area
▬▬▬ = Range of percent correct scores on the
reporting cluster for students statewide
who scored proficient on that test
August 2009
Post-Test Workshop
Student Report:
CST for Grade 11
with EAP Results Sample
Label the location of the EAP report on HO 23.
59
August 2009
Post-Test Workshop
HO 23
CMA Grade 7 Report Back
Label the location of the CMA Writing
Application results on HO 24.
60
August 2009
Post-Test Workshop
HO 24
Explain to Parents
Scale Score  Average % correct cluster
score x 600
• Reporting Clusters not comparable
– Different difficulty
– Varying number of questions
– Average % correct of clusters  % correct of
total scale score
• Scale scores
– Use conversion tables and other statistical
techniques
– Equating allows scores to have similar
meaning
(e.g., 350 = lowest score for CSTs proficient)
61
August 2009
Post-Test Workshop
Other Student Reports
HO 24–26
• CMA
– Performance levels grades 3–5
– Percent correct grades 6–8
– Back, “About the CMA,”
– Back, cluster reporting grades 3–5 and grade 7
writing
• CAPA
– Back, “About the CAPA”
• STS
– In Spanish
– Performance levels and scale scores reported
for grades 2–4
– Percent correct for entire test for grades 5–11
– Back, cluster percent correct for grades 2–7
– Back, how to use report
62
August 2009
Post-Test Workshop
Unmatched Report
HO 28
• Grades 4 and 7
– Unmatched Multiple-Choice Report
• CST multiple-choice score but no writing
score
• CMA Multiple-choice score but no writing
score
– Unmatched Writing Report
• Writing score but no CST multiple-choice
score
• Writing score but no CMA multiple-choice
score
– Students receive 2 reports if writing score
not matched to multiple-choice score
63
August 2009
Post-Test Workshop
Unmatched Report:
64
August 2009
Post-Test Workshop
HO 28
California Reports for Teachers
• Summary of results for previous year’s class(es)
• Two types of summary reports:
– By grade level (e.g., grade 11 ELA) or course
(e.g., Algebra I) for all schools
– By teacher name or group name (e.g., GATE) if
on Pre-ID or SGID (latter takes precedence)
• Separate reports for:
– CST ELA
– CST Mathematics
• Not available for:
– CST science or history–social science
– CMA
– CAPA
– STS
65
August 2009
Post-Test Workshop
California Reports for Teachers
• Individualized reports (teacher name or group
name)
– if name on SGID or Pre-ID
– SGID takes priority over Pre-ID
• Group reports (grade level or course): all ELA
and mathematics teachers should receive
• EOC Math
– Grades 7–11 by subject only
– Not grade-specific:
e.g., All Algebra I in a school
66
August 2009
Post-Test Workshop
Report for Teachers:
Grade 4 Sample
67
August 2009
Post-Test Workshop
HO 29–30
Report for Teachers:
2009 Performance
By Reporting Clusters
Reporting clusters defined and focus
suggested under this section.
68
August 2009
Post-Test Workshop
HO 29
California Report for Teachers:
HO 29
Cluster Reporting
• For each reporting cluster, the average
percent correct achieved by:
–
–
–
–
Statewide students
District students
Your students
Proficient range
• Definition of symbols
 = Average % correct for teacher’s students
▬▬▬ = Average % correct range for students
statewide who scored proficient on that test
69
August 2009
Post-Test Workshop
California Report for Teachers:
Back
• School % at each performance level by
subgroups
• % Proficient and above
– 2008 vs. 2009
– School, district, state
– By subgroup
• Resources
70
August 2009
Post-Test Workshop
HO 30
Summary
• Do’s and Don’ts
– Do compare mean scale score, percent at
performance levels within same grade, same
content area
– Don’t compare mean scale scores across
grades, content area
• Summary reports
• Data CDs
• Individual reports
• Teacher reports
• Quiz answers
• Evaluations
71
August 2009
Post-Test Workshop
For more information see:
•
•
•
•
http://www.cde.ca.gov/ta/tg/sr/
http://www.star.cde.ca.gov
http://www.startest.org/
STAR Technical Assistance Center
800-955-2954
• Data CD Webcast:
http://webcast.sdcoe.net/ets052709a/index.htm
• CDE Accountability
– aau@cde.ca.gov
– 916-319-0863
72
August 2009
Post-Test Workshop
To contact CDE
• CDE STAR office: 916-445-8765
– English–Language Arts tests, including
writing: Jamie Contreras, jcontrer@cde.ca.gov,
916-319-0353
– Mathematics tests: Jane Liang,
jliang@cde.ca.gov, 916-322-1854
– History–Social Science tests: Nicole
Jespersen, njespersen@cde.ca.gov, 916-3190364
– CAPA, CMA: Don Killmer,
dkillmer@cde.ca.gov, 916-319-0350
– STS: Dianna Gutierrez, dgutierrez@cde.ca.gov,
916-319-0337
73
August 2009
Post-Test Workshop