2013 STAR
Interpreting and Using
Results
August 7, 2013
Webcast
Webcast starts at 9 a.m.
Objectives
Workshop participants will be able to:
• Describe the purposes of STAR reports
• Interpret STAR results
• Explain key statistics
• Compare and contrast types of reports
• Identify proper uses of reports
2
August 2013
Post-Test Workshop
Agenda
• What’s New?
• Results and Statistical Analysis
• Using Results
• Summary and Internet Reports
• Data CDs
• Individual Student Reports
• Early Assessment Program
3
August 2013
Post-Test Workshop
What’s New in 2013
• Standards-based Tests in Spanish (STS)
performance levels are now also reported for
students in grades 8–11who took the grade-level
STS for RLA, STS for Algebra I, and STS for
Geometry.
• The score for the California Standards Test (CST)
for Writing is no longer doubled; possible scores
are 1, 2, 3, and 4.
4
August 2013
Post-Test Workshop
Manual
(M) 2 – 3
What’s New in 2013
• ELA scale scores for the grades 4 and 7 CSTs and
California Modified Assessment (CMA) will be
provided based on the multiple-choice items only
and the reporting cluster “Writing Applications”
is no longer part of the CST and CMA ELA
cluster groups; instead, the writing score is
provided as a standalone score called “Writing
Response Score.”
5
August 2013
Post-Test Workshop
M2–3
What’s New in 2013
• Enrollment and exit code data to determine which
students are counted as “continuously enrolled” for
accountability purposes, previously collected for
(CBEDS), were not collected on STAR answer
documents or in Pre-ID. Instead, these data were
extracted from the California Longitudinal Pupil
Achievement Data System (CALPADS).
• Please refer to pages 2 and 3 in the 2013 Post Test
Guide for a list of all changes.
6
August 2013
Post-Test Workshop
M2–3
Quiz Question 1
Which of these tests had scale scores reported for the
first time in 2013?
A.
B.
C.
D.
7
August 2013
CST for World History
STS for Science
CMA for Algebra I
STS for Algebra I
Post-Test Workshop
Quiz Question 1
Which of these tests had scale scores reported for the
first time in 2013?
A.
B.
C.
D.
8
August 2013
CST for World History
CAPA for Science
CMA for Algebra I
STS for Algebra I
Post-Test Workshop
Results:
Purposes of STAR Reports
• Report progress toward proficiency on the state’s
academic content standards
• Notify where improvement needed
– To help students’ achievement
– To improve educational programs
• Provide data for state and federal accountability
programs
9
August 2013
Post-Test Workshop
M4
Results: Performance Levels
• State goal: All students score at proficient or above
• 350 or higher scale score
– CST
– CMA
– STS
• CAPA proficient: 35 or higher
scale score
10
August 2013
Post-Test Workshop
M 8 – 11
Results:
Other Performance Levels
• Advanced
• Basic cut score
– CST: 300
– CMA: 300
– STS: 300
– CAPA: 30
• Below basic
• Far below basic
• For each testing program, cut points
vary for advanced and below basic by
– Subject
– Grade
11
August 2013
Post-Test Workshop
M 8 – 11;
Appendix B
Results: Scale Scores
• Scale scores allow the same score to mean the
same thing across test versions within grade and
content area.
• Scale scores account for differences in difficulty.
• Scale score ranges by program are:
– CST, CMA, STS: 150–600 for each grade and
subject
– CAPA: 15–60 for each level and subject
12
August 2013
Post-Test Workshop
M 8 – 11
Results: Equating
• Psychometric procedure
• Adjusts for test difficulty
• Additional information in the technical report on
the CDE Web site
13
August 2013
Post-Test Workshop
M8
Results: Reporting Clusters
(Content Area)
• Three to six clusters for each subject
• May be useful as indicators of individual or group
strengths and weaknesses
• But. . . reporting clusters should be interpreted with
caution
14
August 2013
Post-Test Workshop
M 9 – 11;
Appendix A
Results: Reporting Clusters
Cautions
• Cluster percent correct available for CSTs, CMA,
STS
• Based on small numbers of items; therefore, may
not be reliable or generalized
• NOT equated from year to year
• Should not compare reporting cluster percent
correct from year to year
15
August 2013
Post-Test Workshop
M 9 – 11;
Appendix A
Interpreting Reporting Clusters or
Content Areas in the Same Year
• Compare to percent-correct range of proficient
students statewide
16
August 2013
Post-Test Workshop
M 9 – 11;
Appendix A
2013 CST Reporting Clusters:
Number of Questions and
Average Percent Correct
2013 Post-Test Guide, Appendixes A and C as
posted on startest.org, will be finalized with
complete data on August 15.
17
August 2013
Post-Test Workshop
M Appendix A
Examples—Interpreting Reporting
Clusters for the CST for Geometry
My Class 2012
Just Proficient Students 2012
Just Advanced Students 2012
100%
Avg % Correct
80%
60%
40%
20%
0%
My Class 2013
Logic/Geom
Proofs
Volume & Area
Angle
Relationships
Trigonometry
Just Proficient Students 2013
Just Advanced Students 2013
100%
Avg % Correct
80%
60%
40%
20%
0%
Logic/Geom
Proofs
18
August 2013
Post-Test Workshop
Volume & Area
Angle
Relationships
Trigonometry
M 10
Quiz Question 2
What is a scale score?
A. Percent correct of all questions
B. Mean percent correct of all questions
C. An adjustment of this year’s and last year’s
raw scores to show changes
D. An adjustment of the raw score to account for
differences in difficulty
19
August 2013
Post-Test Workshop
Quiz Question 2
What is a scale score?
A. Percent correct of all questions
B. Mean percent correct of all questions
C. An adjustment of this year’s and last year’s
raw scores to show changes
D. An adjustment of the raw score to account for
differences in difficulty
20
August 2013
Post-Test Workshop
Using Results
• For instructional decisions in conjunction with
other data
• Used in Academic Performance Index (API)
calculations, all grades and subjects: CST, CMA,
CAPA
• Used in adequate yearly progress (AYP)
calculations, ELA and mathematics:
– CST — grades 2–8
– CMA — grades 3–8
– CAPA — grades 2–8, 10
21
August 2013
Post-Test Workshop
M4
Year-to-Year Comparisons
Do Compare CSTs: Same Grade
and Same Content Area
• Mean scale score
– Same content and grade, varying years
• Percent in each performance level
– Same content by grade across years
 e.g., 2012 grade 10 ELA with 2013 grade 10
ELA
22
August 2013
Post-Test Workshop
M 12 – 15
Year-to-Year Comparisons
Do Compare CSTs: Percent
Proficient and Advanced
• Percentage of students scoring at PROFICIENT
and above
– For a given grade and subject, e.g., Percent
proficient and above for grade 3 math in 2012 and
2013
– For a given subject and aggregated grades,
e.g., Percent proficient and above for grades 2–6
mathematics in 2012 and 2013
– Across grades and a subject, e.g., Percent
proficient and above in all courses and all grades
23
August 2013
Post-Test Workshop
M 12 – 15
Year-to-Year Comparisons
Don’t Compare
• Individual scale scores or statistics based on scale
scores for different grades or content areas
– Subjects by grade are independently scaled
– Different content standards are measured in
different grades
• Cohorts across grades
• Across tests
• Scale scores to percent correct scores
• CAPA to years prior to 2009, due to new standard
setting then
24
August 2013
Post-Test Workshop
M 12 – 15
Example—Using CST Results to
Compare Grade Results from Year to
Year
2012 CST for ELA
2013 CST for ELA
% Prof or Above
% Prof or Above
Difference
Grade 2
31%
35%
4%
Grade 3
33%
33%
0%
Grade 4
29%
31%
2%
Grade 5
34%
32%
–2%
Grade 6
31%
32%
1%
All Grades
32%
33%
1%
Grade
25
August 2013
Post-Test Workshop
M 14
Quiz Question 3
Which is the best comparison for CST scores of students within a
middle school?
A. 2012 mean scale scores for ELA of a cohort of grade 7
students with 2013 scale scores for ELA of the same
students in grade 8
B. 2012 mean scale scores for ELA for grade 8 students with
2013 mean scale scores for ELA for grade 8 students
C. 2012 mean percent correct scores for ELA with 2013
mean percent correct scores for ELA for the same students
D. 2012 mean percent correct scores for ELA for grade 8
students with 2013 mean percent correct for ELA for
grade 8 students
26
August 2013
Post-Test Workshop
Quiz Question 3
Which is the best comparison for CST scores of students within a
middle school?
A. 2012 mean scale scores for ELA of a cohort of grade 7
students with 2013 scale scores for ELA of the same
students in grade 8
B. 2012 mean scale scores for ELA for grade 8 students with
2013 mean scale scores for ELA for grade 8 students
C. 2012 mean percent correct scores for ELA with 2013
mean percent correct scores for ELA for the same students
D. 2012 mean percent correct scores for ELA for grade 8
students with 2013 mean percent correct for ELA for
grade 8 students
27
August 2013
Post-Test Workshop
Quiz Question 4
Which is the best comparison of cluster scores for a
single student? Compare. . .
A. To proficient students statewide
B. One cluster to another, same year
C. The same cluster to the same cluster, different
years
D. To the average percent correct of all students in
a class
28
August 2013
Post-Test Workshop
Quiz Question 4
Which is the best comparison of cluster scores for a
single student? Compare. . .
A. To proficient students statewide
B. One cluster to another, same year
C. The same cluster to the same cluster, different
years
D. To the average percent correct of all students in
a class
29
August 2013
Post-Test Workshop
Aggregate (Summary) Reports
• What are they?
– Student Master List Summary
– Student Master List Summary End-of-Course (EOC)
– Subgroup Summary
• Report emphasis: CSTs
– Criterion-referenced tests
– Progress is measured in percent of students scoring
proficient and advanced
• Note: Back of reports provides guide to
abbreviations, score codes
30
August 2013
Post-Test Workshop
M 18 – 22
Student Master List Summary
• By grade
• Results by program (CSTs, CMA, CAPA, and STS)
and subject
• # and % at each performance level
• Mean scale score
• Reporting cluster: mean percent correct (except
CAPA)
31
August 2013
Post-Test Workshop
M 18–19;
M 27 – 33
Student Master List Summary
End-of-Course
• By program (CSTs, CMA, and STS)
and subject
• Results for each grade, and for all grades
combined
• # and % at each performance level
• Mean scale score
• Reporting cluster: mean percent correct
32
August 2013
Post-Test Workshop
M 19–20;
M 34 – 39
Student Master List Summary
Grade 7 Example
33
August 2013
Post-Test Workshop
M 33
Student Master List Summary
Basic Statistics
34
August 2013
Post-Test Workshop
M 28 – 31
Who Counts? Number Enrolled
• Total CST/CMA and CAPA multiple-choice
answer documents submitted as scorable
• Minus
– Documents marked as “Student enrolled after the
first day of testing and was given this test”
35
August 2013
Post-Test Workshop
M 29
Who Counts? Number Tested
• All CST, CMA, CAPA, STS answer documents with one
or more answers
• Plus
– Z = Tested but marked no answers
• Not included
– A = Students absent
– E = Not tested due to significant medical emergency
– P = Parent/guardian exemptions
– T = Enrolled first day, not tested, tested at previous school
– Students with inconsistent grades
– Non–English learners who took the STS
36
August 2013
Post-Test Workshop
M 29
Who Counts? Number and
Percent Valid Scores
• Number Valid Scores
– For the subject, number of students tested at grade level who
received a score for the test.
– Not included:




Incomplete tests
Modified tests
Non–English learners who took the STS
Unknown EOC mathematics (except grade 7
mathematics) or science tests
 Inconsistent grades
• Percent Valid Scores
– For the subject, number of valid scores divided by the number of
students tested.
37
August 2013
Post-Test Workshop
M 29
Who Counts?
Number Tested with Scores
• All tests taken, including those taken with
modifications, that result in a score
• Not included:
– Incomplete tests
– Non–English learners who took the STS
– Unknown EOC mathematics or science tests
– Inconsistent grades
38
August 2013
Post-Test Workshop
M 29
Student Master List Summary
Performance Levels
39
August 2013
Post-Test Workshop
M 30
Who Counts?
Performance Levels
• All CSTs, CAPA, CMA, STS
• Advanced, proficient, basic, below basic
– All valid scores falling in the performance level
• Far below basic
– All valid scores falling in the performance level
– CSTs taken with modifications (in aggregate reporting
[CSTs and STS] and accountability [CSTs] only)
40
August 2013
Post-Test Workshop
M 30
Who Counts?
Mean Scale Scores
• Average of valid scale scores
• Can be used to compare results for the same
content/grade across years
41
August 2013
Post-Test Workshop
M 30
Student Master List Summary:
Reporting Clusters
Compare to:
Average percent correct range for students statewide who scored
proficient on the total test (See the Post-Test Guide, Appendix A.)
42
August 2013
Post-Test Workshop
M 30
Student Master List Summary:
Writing
B
C
I
L
43
August 2013
=
=
=
=
Blank
Copied prompt
Illegible
Language other
than English
R = Refusal
T = Off topic
W = Wrong prompt
(prompt from an
earlier administration)
Post-Test Workshop
M 30 – 31
Subgroup Summary:
CSTs, CMA, CAPA, and STS
• Disability status
– Based on disability status for CST, CMA, STS
– CAPA: each disability type
• Economic status
– Based on NSLP eligibility or parent education level
•
•
•
•
44
Gender
English proficiency
Ethnicity
Ethnicity for Economic Status (only for CSTs,
CMA, and CAPA)
August 2013
Post-Test Workshop
M 40 – 54
Subgroup Summary:
Ethnicity for Economic Status
45
August 2013
Post-Test Workshop
M 51 – 54
Subgroup Summary:
Ethnicity for Economic Status
Example: Economically
disadvantaged for each
ethnicity
46
August 2013
Post-Test Workshop
M 51 – 54
Subgroup Summary:
Ethnicity for Economic Status
47
August 2013
Post-Test Workshop
M 51 – 54
Break — 10 minutes
Internet Reports
• Summaries based on same data as paper reports:
CSTs, CMA, CAPA, STS
• Available to the public online for school, district,
county, and state
• “Students with Scores” = number tested with
scores
• CST summaries of % advanced and proficient
• More subgroups than paper reports
– Parent education
– Special program participation
• Access from http://star.cde.ca.gov/
49
August 2013
Post-Test Workshop
M 95 – 108
Internet Demonstration
50
August 2013
Post-Test Workshop
Internet Reports:
CST Sample
51
August 2013
Post-Test Workshop
M 95 – 106
Internet Reports:
CST Summary Sample
Only available for CSTs
52
August 2013
Post-Test Workshop
M 99 – 100
Other Internet Reports
• CST (M 98−100)
• CMA (M 100–101)
– Same as CST
• CAPA (M 101−104)
– State level: same as CST; separate Level I
– County, district, school
 Mean scale score
 Percent proficient or above
• STS (M 105−106)
– Same as CST
53
August 2013
Post-Test Workshop
Quiz Question 5
Which subgroup can only be accessed from the
Internet?
A. Parent Education Level
B. CAPA by individual disability status
C. Ethnicity for Economic Status
D. English Proficiency
54
August 2013
Post-Test Workshop
Quiz Question 5
Which subgroup can only be accessed from the
Internet?
A. Parent Education Level
B. CAPA by individual disability status
C. Ethnicity for Economic Status
D. English Proficiency
55
August 2013
Post-Test Workshop
Data CDs
• What are they?
– Lists of information from answer documents and scores
of every student in the district
– In .txt format
• What are they used for?
– Searching for specific data
– Creating unique reports
– Verifying paper reports
• What else is needed?
– Text editor
– or Desktop application
– or Student Information System
56
August 2013
Post-Test Workshop
View of Data
• As .txt, word wrap on
• With text editor, word wrap off
57
August 2013
Post-Test Workshop
Organization of Data
• Two files:
– Demographics, special conditions, and test scores
– Accommodations, modifications, English learners,
and irregularities
• Data Layout = guide to location of data on files
– Position
– Number of characters
– Whether numeric or alpha
58
August 2013
Post-Test Workshop
Data Layout Sample
59
August 2013
Post-Test Workshop
Individual Reports
• STAR Student Record Label
– Adhesive label to affix to student’s permanent school record
• STAR Student Master List
– Alphabetical list of students and their scores
– Tests listed in order within grade
 CSTs
 CMA
 CAPA
 STS
• STAR Student Report: individual’s scores
– 2 two-sided color copies for each test
– For parents/guardians, school
– Per regulations, district must forward within 20 business days
60
August 2013
Post-Test Workshop
M 55 – 93
Student Record Label Grade 10 Sample:
Student Name and Identification
61
August 2013
Post-Test Workshop
M 56
Student Record Label
CST/CMA Grade 10 Example
62
August 2013
Post-Test Workshop
M 56
Student Master List
CST/CMA Grade 3 Example
63
August 2013
Post-Test Workshop
M 57 – 60
Student Report
CST Grade 11 Example
64
August 2013
Post-Test Workshop
M 61 – 93;
71 – 72
Student Report
CST Grade 11 Example
65
August 2013
Post-Test Workshop
M 70
Student Report
CST Grade 11 Example
♦
=
▬ =
66
August 2013
Student’s name
on back
Percent correct obtained by the student on the
reporting cluster/content area
Average percent-correct range on the
reporting cluster for students statewide who
scored proficient on the total test scores
Post-Test Workshop
M 71
Explain to Parents
Scale Score  Average % correct
cluster score x 600
• Reporting clusters not comparable
– Different difficulty
– Varying number of questions
– Average % correct of clusters  % correct on total test (or
total number of test items)
• Scale scores
– Use conversion tables resulting from statistical procedures
– Equating allows scores to have similar meaning
(e.g., 350 = lowest score for CSTs proficient)
67
August 2013
Post-Test Workshop
M 8 – 11
Other Student Reports
• CMA (M 72−80)
– Front: Performance levels and scale scores
– Back: Cluster percent correct, About the CMA
• CAPA (M 81−85)
– Front: Performance levels and scale scores
– Back: About the CAPA
• STS (M 86−93)
– In Spanish
 Front: Performance levels and scale scores
 Back: Cluster percent correct, how to use the report
68
August 2013
Post-Test Workshop
Unmatched Reports
69
Writing CST, CMA—Grs 4, 7
CST, CMA—Grs 8, 11
• Only multiple-choice
• Only writing
• Students receive 2 reports if
writing score not matched to
multiple-choice score
• Only CMA and no CST
• Grades 8 and 11 require
CST for History–Social
Science
August 2013
Post-Test Workshop
Summary
• Do’s
– Do compare mean scale score, percent at performance
levels within same grade, same content area
– Do compare cluster scores to the corresponding
proficient range provided for this year
• Don’ts
– Don’t compare cluster scores to each other within test,
nor across years, grades, content areas
– Don’t compare mean scale scores across grades, content
areas
• Summary and Internet reports
• Data CDs
• Individual reports
70
August 2013
Post-Test Workshop
Upcoming Dates
Event
Post Test Guide posted on startest.org
State Superintendent's public announcement
Appendixes A & C of the Post-Test Guide
posted on startest.org
Web Reports available
Research files available
Post-Test Guide distributed
Data Corrections window
71
August 2013
Post-Test Workshop
Date
July 24
August 8
August 8
August 8
August 8
August 15
September 16 through October 25
2014 Setting Up Administration
1. Return the 2014 Superintendent’s Designation
form and Security Agreement.
2. Finalize instruction schedule.
3. Set up admin in STAR management system prior
to December 1.
4. Enter and approve orders prior to December 1.
After order approval, test administration dates are
NOT changeable.
72
August 2013
Post-Test Workshop
For more information see:
• http://www.cde.ca.gov/ta/tg/sr/
• http://star.cde.ca.gov/
• http://www.startest.org/
• STAR Technical Assistance Center
– 800-955-2954
• CDE Accountability
– aau@cde.ca.gov
– 916-319-0863
• CDE STAR office: 916-445-8765
73
August 2013
Post-Test Workshop
Next Webcasts
2013 Demographic Data Corrections
September 25, 2013, 9 a.m.
2014 STAR for Students with
Disabilities Webcast
September 25, 2013, 1 p.m.
74
August 2013
Post-Test Workshop
Early Assessment Program
Results
August 7, 2013
Connie Grueter
Early Assessment Program (EAP)
• EAP results are being accepted at participating
California Community Colleges (CCC).
– List of participating colleges is available on the CCC
website at www.CollegeEAP.org. Check the site often
for an updated list of campuses.
PowerPoint slides – www.CollegeEAP.org, Educators, About EAP.
76
August 2013
Post-Test Workshop
Early Assessment Program (EAP)
• An EAP informational web site has been
developed: www.CollegeEAP.org.
– The site provides EAP information for both
CSU and CCC
– It is a resource for Students, Parents, and
Educators
PowerPoint slides – www.CollegeEAP.org, Educators, About EAP.
77
August 2013
Post-Test Workshop
www.CollegeEAP.org
78
August 2013
Post-Test Workshop
Early Assessment Program (EAP)
• Reporting Results
– New this year – Conditional Exemptions for EAP English
– Ready for CSU and participating CCC college-level
English/mathematics courses – Conditional
 This status in conjunction with a grade of “C” or better in an
approved English/math course or a supervised e-learning
program can be exempt from the EPT or ELM for CSU or other
English or math placement tests at participating California
Community Colleges (CCC).
 A list of courses and e-learning programs can be found at:
www.csumathsuccess.org and www.csuenglishsuccess.org .
PowerPoint slides – www.CollegeEAP.org, Educators, About EAP.
79
August 2013
Post-Test Workshop
Early Assessment Program (EAP)
Sample EAP Box on the STAR Student Report
The EAP box can be found on page 67 of the Post-Test Guide
PowerPoint slides – www.CollegeEAP.org, Educators, About EAP.
80
August 2013
Post-Test Workshop
Early Assessment Program (EAP)
• Web Sites
– www.CollegeEAP.org
 Links to CSU and CCC informational sites for
Students, Parents, and Educators
– www.CalState.edu/EAP
 Provides a list of EAP Coordinators, EAP test
blueprints, and informational materials
– www.CCCCO.edu/EAP
 Provides a list of participating Community Colleges
PowerPoint slides – www.CollegeEAP.org, Educators, About EAP.
81
August 2013
Post-Test Workshop