2012 STAR
Interpreting and Using
Results
August 8, 2012
Webcast
Webcast starts at 9 a.m.
Objectives
Workshop participants will be able to:
• Describe the purposes of STAR reports
• Interpret STAR results
• Explain key statistics
• Compare and contrast types of reports
• Identify proper uses of reports
2
August 2012
Post-Test Workshop
Agenda
•
•
•
•
•
•
•
3
What’s New?
Results and Statistical Analysis
Using Results
Summary and Internet Reports
Data CDs
Individual Student Reports
Early Assessment Program
August 2012
Post-Test Workshop
What’s New in 2012
Manual
• California Modified Assessment (CMA) (M) 2
reports
– Scale score, performance levels for
 ELA, grades 10 and 11
 Geometry
4
August 2012
Post-Test Workshop
Quiz Question 1
Which of these tests had scale scores
reported for the first time in 2012?
A. CST for World History
B. CAPA for Science
C. CMA for Algebra I
D. CMA for Geometry
5
August 2012
Post-Test Workshop
Quiz Question 1
Which of these tests had scale scores
reported for the first time in 2012?
D. CMA for Geometry
6
August 2012
Post-Test Workshop
Results:
M3
Purposes of STAR Reports
• Report progress toward proficiency on
the state’s academic content standards
• Notify where improvement needed
– To help students’ achievement
– To improve educational programs
• Provide data for state and federal
accountability programs
7
August 2012
Post-Test Workshop
Results:
Performance Levels
• State goal: All students score at
proficient or higher
• 350 or higher scale score
– All CST
– All CMA
– STS for RLA and math in
grades 2–7, excluding Algebra I
• CAPA proficient: 35 or higher
scale score
8
August 2012
Post-Test Workshop
M 8−9
Results:
Other Performance Levels
• Advanced
• Basic cut score
– CST: 300
– CMA: 300
– STS: 300 for RLA and math in
grades 2–7 (excluding Algebra I)
– CAPA: 30
• Below basic
• Far below basic
• For each testing program, cut points
vary for advanced and below basic by
– Subject
– Grade
9
August 2012
Post-Test Workshop
M 8−9;
Appendix B
Results: Scale Scores
M 7−10
• Scale scores allow the same score to mean
the same thing across test versions within
grade and content area
• Accounts for differences in difficulty
• Scale score ranges by program:
– CST, CMA, STS (grades 2–7, excluding
Algebra I): 150–600
for each grade and subject
– CAPA: 15–60 for each level and subject
10
August 2012
Post-Test Workshop
Results: Equating
• Psychometric procedure
• Adjusts for test difficulty
• Additional information in the
technical report on the CDE Web site
11
August 2012
Post-Test Workshop
M7
Results: Reporting Clusters
M 8−10;
(Content Areas)
• Three to six clusters for each subject
Appendix A
• May be useful as indicators of individual or group
strengths and weaknesses
• But . . . Reporting clusters
should be interpreted with
caution
12
August 2012
Post-Test Workshop
•
•
•
•
13
Results: Reporting Cluster
Cautions
Cluster percent correct available
for all CST, CMA, STS
Based on small numbers
of items; therefore, may not
be reliable or generalizable
NOT equated from year to
year
Should not compare
reporting cluster percent
correct from year to year
August 2012
Post-Test Workshop
M 8−10;
Appendix A
Interpreting Reporting Clusters or
M 8−10;
Content Areas
Appendix A
in the Same Year
• Compare to percent-correct range of
proficient students statewide
14
August 2012
Post-Test Workshop
2012 CST Reporting Clusters:
Number of Questions and
Average Percent Correct
M Appendix A
2012 Post-Test Guide, Appendixes A and C
as posted on startest.org will be finalized
with complete data on August 31.
15
August 2012
Post-Test Workshop
Examples—Interpreting Reporting
Clusters for the CST for Geometry
My Class 2011
Just Advanced Students 2011
M9
Just Proficient Students 2011
100%
Avg % Correct
80%
60%
40%
20%
My Class 2012
Just Advanced Students 2012
100%
0%
Logic/Geom
Proofs
Volume & Area
Angle
Relationships
Trigonometry
Just Proficient Students 2012
Avg % Correct
80%
60%
40%
20%
0%
Logic/Geom
Proofs
16
August 2012
Post-Test Workshop
Volume & Area
Angle
Relationships
Trigonometry
Quiz Question 2
What is a scale score?
A. Percent correct of all questions
B. Mean percent correct of all questions
C. An adjustment of this year’s and last
year’s raw scores to show changes
D. An adjustment of the raw score to
account for differences in difficulty
17
August 2012
Post-Test Workshop
Quiz Question 2
What is a scale score?
D. An adjustment of the raw score to
account for differences in difficulty
18
August 2012
Post-Test Workshop
Using Results
M3
• For instructional decisions with other
data
• Used in Academic Performance Index
(API) calculations: all grades and
subjects; CST, CMA, CAPA
• Used in adequate yearly progress (AYP)
calculations, ELA and mathematics:
– CST – grades 2–8
– CMA – grades 3–8
– CAPA – grades 2–8, 10
19
August 2012
Post-Test Workshop
Year-to-Year Comparisons
M 11−14
Do Compare CSTs:
Same Grade and Same Content Area
• Mean scale score
– Same content and grade, varying
years
• Percent in each performance level
– Same content by grade across
years
 e.g., 2011 grade 10 ELA with 2012
grade 10 ELA
20
August 2012
Post-Test Workshop
Year-to-Year Comparisons
M 11−14
Do Compare CSTs:
Percent Proficient and Advanced
• Percentage of students scoring at PROFICIENT
and above
– For a given grade and subject, e.g., Percent
proficient and above for grade 3 math in 2011
and 2012
– For a given subject and aggregated grades,
e.g., Percent proficient and above for grades
2– 6 mathematics in 2011 and 2012
– Across grades and a subject, e.g., Percent
proficient and above in all courses and all
grades
21
August 2012
Post-Test Workshop
Year-to-Year Comparisons
DON’T Compare
M 11−14
• Individual scale scores or statistics based on
scale scores for different grades or content
areas
– Subjects by grade are independently scaled
– Different content standards are measured in
different grades
•
•
•
•
22
Cohorts across grades
Across tests
Scale scores to percent correct scores
CAPA to years prior to 2009, due to new
standard setting then
August 2012
Post-Test Workshop
Example—Using CST Results to
Compare Grade Results from Year to Year M 13
2011 CST
for ELA
2012 CST
for ELA
% Prof or
Above
% Prof or
Above
Difference
Grade 2
31%
35%
4%
Grade 3
33%
33%
0%
Grade 4
29%
31%
2%
Grade 5
34%
32%
–2%
Grade 6
31%
32%
1%
All Grades
32%
33%
1%
Grade
23
August 2012
Post-Test Workshop
Quiz Question 3
Which is the best comparison for CST scores of
students within a middle school?
A. 2011 mean scale scores for ELA of a cohort of
grade 7 students with 2012 scale scores for ELA
of the same students in grade 8
B. 2011 mean scale scores for ELA for grade 8
students with 2012 mean scale scores for ELA
for grade 8 students
C. 2011 mean percent correct scores for ELA with
2012 mean percent correct scores for ELA for
the same students
D. 2011 mean percent correct scores for ELA for
grade 8 students with 2012 mean percent
correct for ELA for grade 8 students
24
August 2012
Post-Test Workshop
Quiz Question 3
Which is the best comparison for CST scores of
students within a middle school?
B. 2011 mean scale scores for ELA for grade 8
students with 2012 mean scale scores for ELA
for grade 8 students
25
August 2012
Post-Test Workshop
Quiz Question 4
Which is the best comparison of cluster
scores for a single student? Compare . . .
A. To proficient students statewide
B. One cluster to another, same year
C. The same cluster to the same cluster,
different years
D. To the average percent correct of all
students in a class
26
August 2012
Post-Test Workshop
Quiz Question 4
Which is the best comparison of cluster
scores for a single student? Compare…
A. To proficient students statewide
27
August 2012
Post-Test Workshop
Aggregate (Summary) Reports
• What are they?
– Student Master List Summary
– Student Master List Summary End-ofCourse (EOC)
– Subgroup Summary
• Report emphasis: CSTs
– Criterion-referenced tests
– Progress is measured in percent of
students scoring proficient and
advanced
• Note: Back of reports provides guide to
abbreviations, score codes
28
August 2012
Post-Test Workshop
M 16−19
Student Master List Summary
M 16–17,
M 26−32
• By grade
• Results by program (CSTs, CMA, CAPA, and
STS) and subject
• # and % at each performance level (except
STS for: RLA grades 8−11, Algebra I, and
Geometry)
• Mean scale score (except STS for: RLA grades
8−11, Algebra I and Geometry)
• Reporting cluster: mean percent correct
(except CAPA)
29
August 2012
Post-Test Workshop
Student Master List Summary
End-of-Course
M 17,
M 33−38
• By program (CSTs, CMA, and STS)
and subject
• Results for each grade, and for all grades
combined
• # and % at each performance level (except
STS for Algebra I and Geometry)
• Mean scale score (except STS for Algebra I
and Geometry)
• Reporting cluster: mean percent correct
30
August 2012
Post-Test Workshop
Student Master List Summary
Grade 7 Example
31
August 2012
Post-Test Workshop
M 32
Student Master List Summary
Basic Statistics
32
August 2012
Post-Test Workshop
M 27−30
Who Counts?
M 28
Number Enrolled
• Total CST/CMA and CAPA multiplechoice answer documents submitted as
scorable
• Minus
– Documents marked as “Student
enrolled after the first day of testing and
was given this test”
33
August 2012
Post-Test Workshop
Who Counts? Number Tested
• All CST, CMA, CAPA, STS answer documents
with one or more answers
• Plus
– Z = Tested but marked no answers
• Not included
– A = Students absent
– E = Not tested due to significant medical
emergency
– P = Parent/guardian exemptions
– T = Enrolled first day, not tested, tested at
previous school
– Students with inconsistent grades
– Non–English learners who took the STS
34
August 2012
Post-Test Workshop
M 28
Who Counts? Number and
Percent Valid Scores
• Number Valid Scores
– For the subject, number of students tested at
grade level who received a score for the test
(that is, a scale score or percent correct)
– Not included:
 Incomplete tests
 Modified tests
 Non–English learners who took the STS
 Unknown EOC mathematics (except grade 7
mathematics) or science tests
– Inconsistent grades
• Percent Valid Scores
– For the subject, number of valid scores
divided by the number of students tested
35
August 2012
Post-Test Workshop
M 28
Who Counts?
M 28
Number Tested with Scores
• All tests taken, including those taken
with modifications, that result in a
score
• Not included:
– Incomplete tests
– Non–English learners who took the
STS
– Unknown EOC mathematics or
science tests
– Inconsistent grades
36
August 2012
Post-Test Workshop
Student Master List Summary
Performance Levels
37
August 2012
Post-Test Workshop
M 29
Who Counts?
Performance Levels
M 29
• All CSTs, CAPA, CMA
• STS grades 2–7 RLA and math (excluding Algebra I)
• Advanced, proficient, basic, below basic
– All valid scores falling in the performance level
• Far below basic
– All valid scores falling in the performance level
– CSTs taken with modifications (in aggregate
reporting and accountability only)
38
August 2012
Post-Test Workshop
Who Counts?
M 29
Mean Scale Scores
• Average of valid scale scores
• Can be used to compare results for the
same content/grade across years
39
August 2012
Post-Test Workshop
Student Master List Summary:
Reporting Clusters
Compare to:
Average percent correct range for students
statewide who scored proficient on the total test
(See the Post-Test Guide, Appendix A.)
40
August 2012
Post-Test Workshop
M 29
Student Master List Summary:
Writing
B
C
I
L
41
=
=
=
=
August 2012
Blank
Copied prompt
Illegible
Language other
than English
M 29
R = Refusal
T = Off Topic
W = Wrong prompt
(prompt from an
earlier administration)
Post-Test Workshop
Subgroup Summary:
CSTs, CMA, CAPA, and STS
M 39−53
• Disability status
– Based on disability status for CST, CMA, STS
– CAPA: each disability type
• Economic status
– Based on NSLP eligibility or parent education
level
• Gender
• English proficiency
• Ethnicity
• Ethnicity for Economic Status (only for CSTs,
CMA, and CAPA)
42
August 2012
Post-Test Workshop
Subgroup Summary:
Ethnicity for Economic Status
43
August 2012
Post-Test Workshop
M 50−53
Subgroup Summary:
Ethnicity for Economic Status
M 50−53
Example: Economically
disadvantaged for each
ethnicity
44
August 2012
Post-Test Workshop
Subgroup Summary:
Ethnicity for Economic Status M 50−53
45
August 2012
Post-Test Workshop
Break — 10 minutes
Internet Reports
M 93−106
• Summaries based on same data as paper
reports: CSTs, CMA, CAPA, STS
• Available to the public online for school,
district, county, and state
• “Students with Scores” = number tested
with scores
• CST Summaries of % Advanced and
Proficient
• More subgroups than paper reports
– Parent education
– Special program participation
• Access from http://star.cde.ca.gov/
47
August 2012
Post-Test Workshop
Internet Demonstration
48
August 2011
Post-Test Workshop
Internet Reports:
CST Sample
49
August 2012
Post-Test Workshop
M 97−106
Internet Reports:
CST Summary Sample
Only available for CSTs
50
August 2012
Post-Test Workshop
M 98−99
Other Internet Reports
• CMA (M 99−100) Same as CST
• CAPA (M 101−104)
– State level: same as CST; separate Level I
– County, district, school
 Mean scale score
 Percent proficient or above
• STS (M 105−106)
– Same as CST—STS grades 2−7 RLA and math
(except Algebra I)
– Average percent (%) correct—STS grades 8−11
RLA and Algebra I and Geometry
51
August 2012
Post-Test Workshop
Quiz Question 5
Which subgroup can only be accessed
from the Internet?
A. Parent Education Level
B. CAPA by individual disability status
C. Ethnicity for Economic Status
D. English Proficiency
52
August 2012
Post-Test Workshop
Quiz Question 5
Which subgroup can only be accessed
from the Internet?
A. Parent Education Level
53
August 2012
Post-Test Workshop
Data CDs
• What are they?
– Lists of information from answer documents
and scores of every student in the district
– In .txt format
• What are they used for?
– Searching for specific data
– Creating unique reports
– Verifying paper reports
• What else is needed?
– Text editor
– or Desktop application
– or Student Information System
54
August 2012
Post-Test Workshop
View of Data
• As .txt, word wrap on
• With text editor, word wrap off
55
August 2012
Post-Test Workshop
Organization of Data
• Two files:
– Demographics, special conditions, and test
scores
– Accommodations, modifications, English
learners, and irregularities
• Data Layout = guide to location of data
on files
– Position
– Number of characters
– Whether numeric or alpha
56
August 2012
Post-Test Workshop
Data Layout Sample
57
August 2012
Post-Test Workshop
Individual Reports
• STAR Student Record Label
M 20−24
– Adhesive label to affix to student’s permanent
school record
• STAR Student Master List
– Alphabetical list of students and their scores
– Tests listed in order within grade
 CSTs
 CMA
 CAPA
 STS
• STAR Student Report: individual’s scores
– 2 two-sided color copies for each test
– For parents/guardians, school
– Per regulations, district must forward within 20
business days
58
August 2012
Post-Test Workshop
Student Record Label Grade 10 Sample:
Student Name and Identification M 55−56
59
August 2012
Post-Test Workshop
Student Record Label
CST/CMA Grade 10 Example
60
August 2012
Post-Test Workshop
M 55−56
Student Master List
CST/CMA Grade 3 Example
61
August 2012
Post-Test Workshop
M 57−60
Student Report
CST Grade 11 Example
62
August 2012
Post-Test Workshop
M 61−92;
70 −71
Student Report
M 70
CST Grade 11 Example
63
August 2012
Post-Test Workshop
Student Report
CST Grade 11 Example
64
M 71
Student’s
name on back
♦ = Percent correct obtained by the student on
the reporting cluster/content area
▬ = Average percent-correct range on the
reporting cluster for students statewide who
scored proficient on the total test scores
August 2012
Post-Test Workshop
Explain to Parents
Scale Score  Average % correct M 8 −10
cluster score x 600
• Reporting clusters not comparable
– Different difficulty
– Varying number of questions
– Average % correct of clusters  % correct on
total test (or total number of test items)
• Scale scores
– Use conversion tables resulting from
statistical procedures
– Equating allows scores to have similar
meaning
(e.g., 350 = lowest score for CSTs proficient)
65
August 2012
Post-Test Workshop
Other Student Reports
• CMA (M 72−79)
– Performance levels and scale scores
 Back, cluster percent correct
• CAPA (M 80−84)
– Back, “About the CAPA”
• STS (M 85−92)
– In Spanish
– Performance levels and scale scores, grades
2–7 RLA and math (except for Algebra I)
 Back, cluster percent correct
– Percent correct for entire test, grades 8–11
RLA and Algebra I and Geometry
 Back, cluster percent correct
– Back, how to use report
66
August 2012
Post-Test Workshop
Unmatched Reports
Writing CST, CMA—Grs 4, 7
CST, CMA—Grs 8, 11
• Only multiple-choice
• Only CMA and no CST
• Only writing
• Grades 8 and 11
require CST for
• Students receive 2 reports if
History–Social Science
writing score not matched to
multiple-choice score
67
August 2012
Post-Test Workshop
Summary
• Do’s
– Do compare mean scale score, percent at
performance levels within same grade, same
content area
– Do compare cluster scores to the
corresponding proficient range provided for
this year
• Don'ts
68
– Do not compare cluster scores to each other
within test, nor across years, grades, content
areas
– Do not compare mean scale scores across
grades, content areas
• Summary and Internet reports
• Data CDs
• Individual reports Post-Test Workshop
August 2012
Upcoming Dates
69
Event
Post-Test Guide posted on startest.org
State Superintendent’s public
announcement
Appendixes A & C Post-Test Guide
posted on startest.org
Web Reports available
Research files available
Post-Test Guide distributed
Date
August 10
August 31
Data Corrections window
October 1–
November 9
August 2012
Post-Test Workshop
August 31
August 31
August 31
First week of
September
2013 Setting Up Administration
1. Determine if November election changes
number of instruction days.
2. Finalize instruction schedule.
3. Set up admin in STAR management
system prior to December 1.
4. Enter and approve orders prior to
December 1.
After order approval, test administration
dates NOT changeable.
70
August 2012
Post-Test Workshop
For more information see:
•
•
•
•
http://www.cde.ca.gov/ta/tg/sr/
http://star.cde.ca.gov/
http://www.startest.org/
STAR Technical Assistance Center
– 800-955-2954
• CDE Accountability
– aau@cde.ca.gov
– 916-319-0863
71
August 2012
Post-Test Workshop
To Contact the CDE
• CDE STAR office: 916-445-8765
– CST English–Language Arts, including writing:
Jamie Contreras, jcontrer@cde.ca.gov, 916319-0353
– CMA English–Language Arts, including
writing: Shobhana Rishi, srishi@cde.ca.gov,
916-391-0201
– CST and CMA Mathematics: Jane Liang,
jliang@cde.ca.gov, 916-322-1854
– CST and CMA Science: Blessing Mupanduki,
bmupanduki@cde.ca.gov, 916-319-0349
– CST History–Social Science, CAPA: Don
Killmer, dkillmer@cde.ca.gov, 916-319-0350
72
August 2012
Post-Test Workshop
To Contact the CDE
– STS Reading/Language Arts: Noelia
Ramirez, nramirez@cde.ca.gov, 916319-0337
– STS Mathematics: Amalia Chamorro,
achamorr@cde.ca.gov, 916-319-0192
– Directions for Administration,
Coordinator Manual: Mark McLean,
mmclean@cde.ca.gov, 916-319-0356
– Reports, Post-Test Guide, Sherry Shi,
sshi@cde.ca.gov, 916-319-0285
73
August 2012
Post-Test Workshop
Next Webcast
2013 STAR for Students with
Disabilities Webcast
September 19, 2012
74
August 2012
Post-Test Workshop
Early Assessment Program
Results
August 8, 2012
Connie Grueter
Early Assessment Program (EAP)
• EAP results are being accepted at
participating California Community Colleges
(CCC).
– List of participating colleges is available on the
CCC website at www.CollegeEAP.org. Check
the site often for an updated list of campuses.
PowerPoint slides - www.CollegeEap.org, Educators, About EAP.
76
August 2012
Post-Test Workshop
Early Assessment Program (EAP)
• An EAP informational web site has been
developed. www. CollegeEAP.org.
– The site provides EAP information for both
CSU and CCC
– It is a resource for Students, Parents, and
Educators
PowerPoint slides - www.CollegeEap.org, Educators, About EAP.
77
August 2012
Post-Test Workshop
www.collegeEAP.org
78
August 2012
Post-Test Workshop
Early Assessment Program (EAP)
• Reporting Results
– New this year – Conditional Exemptions for EAP English
– Ready for CSU and participating CCC college-level
English/mathematics courses – Conditional
• This status in conjunction with a grade of “C” or better in
an approved English/math course or a supervised elearning program can be exempt from the EPT or ELM for
CSU or other English or math placement tests at
participating California Community Colleges (CCC)
• A list of courses and e-learning programs can be found at
www.csumathsuccess.org
PowerPoint slides - www.CollegeEap.org, Educators, About EAP.
79
August 2012
Post-Test Workshop
Early Assessment Program (EAP)
– Sample EAP Box on the STAR Student Report
The EAP box can be found on page 70 of the Post-Test Guide
PowerPoint slides - www.CollegeEap.org, Educators, About EAP.
80
August 2012
Post-Test Workshop
Early Assessment Program (EAP)
• Web Sites
– www.CollegeEAP.org
 Links to CSU and CCC informational sites for
Students, Parents, and Educators
– www.CalState.edu/EAP
 Provides a list of EAP Coordinators, EAP test
blueprints, and informational materials
– www.CCCCO.edu/EAP
 Provides a list of participating Community
Colleges
PowerPoint slides - www.CollegeEap.org, Educators, About EAP.
81
August 2012
Post-Test Workshop