2014 CAASPP
Interpreting and Using
Results
September 2014
Webcast
Objectives
Workshop participants will be able to:
• Describe the purposes of CAASPP reports
• Interpret CAASPP results
• Explain key statistics
• Compare and contrast types of reports
• Identify proper uses of reports
2014 Post-Test Workshop
2
Agenda
•
•
•
•
•
•
•
What’s New?
Results and Statistical Analysis
Using Results
Summary and Internet Reports
Data CDs
Individual Student Reports
Early Assessment Program
2014 Post-Test Workshop
3
What’s New in 2014
• Results are available for only those California
Assessment of Student Performance and
Progress (CAASPP) content areas administered.
• Results for the Early Assessment Program (EAP)
for students in grade eleven who took these tests
are not reported by the CAASPP Program.
• Percent tested data are not reported for the
Standards-based Tests in Spanish (STS).
• Enrollment data—number and percentage—are
not reported for the STS.
Post-Test Guide
(PTG) 2–3
2014 Post-Test Workshop
4
What’s New in 2014
• Since there were no CAASPP end-of-course
(EOC) tests administered, there is no Student
Master List Summary EOC report.
• Electronic reporting, formerly Quick-turnaround
Reporting, is now offered for all remaining
paper-pencil tests. Files have been available
since August 8.
– EAP results are part of electronic reporting.
• Handscoring of tests is not offered in 2014.
PTG 2–3
2014 Post-Test Workshop
5
Results:
Purposes of CAASPP Reports
• Report progress toward proficiency on the
state’s academic content standards
• Notify where improvement needed
– To help students’ achievement
– To improve educational programs
PTG 4
2014 Post-Test Workshop
6
Results: Performance Levels
• State goal: All students score at proficient or
above
• 350 or higher scale score
– CST for Science
– CMA For Science
– STS for RLA
• CAPA proficient: 35 or higher
scale score
PTG 7–13
2014 Post-Test Workshop
7
Results:
Other Performance Levels
•
•
•
•
•
Advanced
Basic cut score
– CST for Science: 300
– CMA for Science: 300
– STS for RLA: 300
– CAPA: 30
Below basic
Far below basic
For each testing program, cut points
vary for advanced and below basic by
– Subject
– Grade
PTG 7 –13;
Appendix B
2014 Post-Test Workshop
8
Results: Scale Scores
• Scale scores allow the same score to mean the
same thing across test versions within grade
and content area.
• Scale scores account for differences in difficulty.
• Scale score ranges by program are:
– CST for Science, CMA for Science, STS for
RLA: 150–600 for each grade and subject
– CAPA: 15–60 for each level and subject
PTG 7–13
2014 Post-Test Workshop
9
Results: Equating
• Psychometric procedure
• Adjusts for test difficulty from year to year (form
to form)
• Additional information in the technical reports
on the CDE Technical Reports and Studies Web
page at
http://www.cde.ca.gov/ta/tg/sr/technicalrpts.asp
PTG 7
2014 Post-Test Workshop
10
Results: Reporting Clusters
(Content Area)
• Three to six clusters for each subject.
• May be useful as indicators of individual or
group strengths and weaknesses.
• But. . . reporting clusters should be interpreted
with caution.
PTG 8–13;
Appendix A
2014 Post-Test Workshop
11
Results: Reporting Clusters
Cautions
• Cluster percent correct available for CSTs for
Science, CMA for Science, STS for RLA.
• Clusters are based on small numbers of items;
therefore, may not be reliable or generalized.
• Clusters are NOT equated from year to year.
• You should not compare reporting cluster
percent correct from year to year.
PTG 8–13;
Appendix A
2014 Post-Test Workshop
12
Interpreting Reporting Clusters or
Content Areas in the Same Year
• Compare to percent-correct range of proficient
students statewide
PTG 8–13;
Appendix A
2014 Post-Test Workshop
13
2014 CST Reporting Clusters:
Number of Questions and
Average Percent Correct
PTG Appendix A
2014 Post-Test Workshop
14
Examples—Interpreting Reporting
Clusters for the CST for Science
PTG 9
2014 Post-Test Workshop
15
Using Results
• For instructional decisions in conjunction with
other data
• CAPA ELA and mathematics in grade ten used
in adequate yearly progress (AYP) calculations
PTG 2, 4
2014 Post-Test Workshop
16
Year-to-Year Comparisons
Do Compare CSTs: Same Grade
and Same Content Area
• Mean scale score
– Same content and grade, varying years
• Percent in each performance level
– Same content by grade across years
PTG 10–13
2014 Post-Test Workshop
17
Year-to-Year Comparisons
Do Compare CSTs: Percent
Proficient and Advanced
• Percentage of students scoring at PROFICIENT
and above
– For a given grade and subject, e.g., percent
proficient and above for grade 5 science in 2013
and 2014
– For a given subject and aggregated grades,
e.g., percent proficient and above for grades 5, 8,
and 10 science in 2013 and 2014
– Across grades and a subject, e.g., percent
proficient and above in all courses and all grades
PTG 10–13
2014 Post-Test Workshop
18
Year-to-Year Comparisons
Don’t Compare
• Individual scale scores or statistics based on
scale scores for different grades or content
areas
– Subjects by grade are independently scaled
– Different content standards are measured in
different grades
•
•
•
•
Cohorts across grades
Across tests
Scale scores to percent correct scores
CAPA to years prior to 2009, because of new
PTG 10–13
standard setting that year
2014 Post-Test Workshop 19
Example—Using CST Results to
Compare Grade Results from Year to
Year
PTG 10
2014 Post-Test Workshop
20
Aggregate (Summary) Reports
• What are they?
– Student Master List Summary
– Subgroup Summary
• Report emphasis: CSTs for Science
– Criterion-referenced tests
– Progress is measured in percent of students
scoring proficient and advanced
• Back of reports provides guide to abbreviations
and score codes
PTG 16–39
2014 Post-Test Workshop
21
Student Master List Summary
• Tests are as follows:
– CSTs, CMA for Science: Grades 5, 8, and 10
– CAPA for ELA and Mathematics: Grades 2–11;
Levels I–V
– CAPA for Science: Grades 5, 8, and 10; Level I,
III–V
– STS for RLA: Grades 2–11
• # and % at each performance level
• Mean scale score
• Reporting cluster: Mean percent correct (except
PTG 16–18;
CAPA)
PTG 22–25
2014 Post-Test Workshop
22
Student Master List Summary
Grade 5 Example
PTG 25
2014 Post-Test Workshop
23
Student Master List Summary
Basic Statistics
PTG 23–25
2014 Post-Test Workshop
24
Who Counts? Number Enrolled
• For the content area:
– Number of multiple-choice answer documents
submitted minus the number of answer
documents marked to indicate that the student
enrolled after the first day and was subsequently
tested
minus
– Documents marked as “Student enrolled after
the first day of testing and was given this test”
• Does not apply to the STS
PTG 22–25
2014 Post-Test Workshop
25
Who Counts? Number Tested
•
•
For the content area, number of students who
responded to any questions on the test or whose
answer documents were marked to indicate that the
student tested but marked no answers (special
condition Z).
Not included:
–
–
–
–
A = Students absent
E = Not tested due to significant medical emergency
P = Parent/guardian exemptions
T = Enrolled first day, not tested, tested at previous
school
– Students with inconsistent grades
– Non–English learners who took the STS
PTG 22–25
2014 Post-Test Workshop
26
Who Counts? Number and
Percent Valid Scores
•
Number Valid Scores
– For the subject, number of students tested at grade level
who received a score for the test.
– Not included:
 Incomplete tests
 Modified tests
 Non–English learners who took the STS
 Inconsistent grades
•
Percent Valid Scores
– For the subject, number of valid scores divided by the
number of students tested.
PTG 24–25
2014 Post-Test Workshop
27
Who Counts?
Number Tested with Scores
• All tests taken, including those taken with
modifications, that result in a score
• Not included:
– Incomplete tests
– Non–English learners who took the STS
– Inconsistent grades
PTG 24–25
2014 Post-Test Workshop
28
Student Master List Summary
Performance Levels
PTG 23–25
2014 Post-Test Workshop
29
Who Counts?
Performance Levels
•
•
All CSTs for Science; CMA for Science; CAPA for ELA,
Mathematics, and Science; STS for RLA
Advanced, proficient, basic, below basic
– All valid scores falling in the performance level
•
Far below basic
– All valid scores falling in the performance level
– CSTs for Science and STS for RLA taken with
modifications in aggregate reporting
PTG 22–39
2014 Post-Test Workshop
30
Who Counts?
Mean Scale Scores
• Average of valid scale scores
• Can be used to compare results for the same
content/grade across years
PTG 24–25
2014 Post-Test Workshop
31
Student Master List Summary:
Reporting Clusters
Compare to:
Average percent correct range for students statewide who scored
proficient on the total test (See the Post-Test Guide, Appendix A.)
PTG 25
2014 Post-Test Workshop
32
Subgroup Summary:
CSTs, CMA, CAPA, and STS
•
Disability status
– Based on disability status for CST, CMA, STS
– CAPA: each disability type
•
Economic status
– Based on NSLP eligibility or parent education level
•
•
•
•
Gender
English proficiency
Ethnicity
Ethnicity for Economic Status (only for CSTs, CMA,
and CAPA)
PTG 26–39
2014 Post-Test Workshop
33
Subgroup Summary:
Ethnicity for Economic Status
PTG 39
2014 Post-Test Workshop
34
Subgroup Summary:
Ethnicity for Economic Status
Example: Economically
disadvantaged for each
ethnicity
PTG 29
2014 Post-Test Workshop
35
Subgroup Summary:
Ethnicity for Economic Status
PTG 29
2014 Post-Test Workshop
36
Internet Reports
• Summaries based on same data as paper
reports: CSTs for Science; CMA for Science;
CAPA for ELA; Mathematics; and Science, STS
for RLA
• Available to the public online for school, district,
county, and state
• “Students with Scores” = number tested with
scores
• More subgroups than paper reports
– Parent education
– Special program participation
• Access from http://caaspp.cde.ca.gov/
PTG 65–73
2014 Post-Test Workshop
37
Internet Reports
• In 2014, neither total enrollment nor percent
tested data are included in Internet reports for
any test.
• Aggregate scores will be reported on the
Internet in mid-September 2014 after the
CALPADS data corrections window.
PTG 65–73
2014 Post-Test Workshop
38
Internet Demonstration
2014 Post-Test Workshop
39
Internet Reports:
CST Sample
PTG 65–73
2014 Post-Test Workshop
40
Other Internet Reports
•
•
CST (PTG 69)
CMA (PTG 69–70)
– Same as CST
•
CAPA (PTG 70−73)
– State level: same as CST; separate Level I
– County, district, school
 Mean scale score
 Percent proficient or above
•
STS (PTG 73)
– Same as CST
2014 Post-Test Workshop
41
Data CDs
•
What are they?
– Lists of information from answer documents and
scores of every student in the LEA
– In .txt format
•
What are they used for?
– Searching for specific data
– Creating unique reports
– Verifying paper reports
•
What else is needed?
– Text editor
– or Desktop application
– or Student Information System
2014 Post-Test Workshop
42
View of Data
• As .txt, word wrap on
• With text editor, word wrap off
2014 Post-Test Workshop
43
Organization of Data
• Two files:
– Demographics, special conditions, and test
scores
– Accommodations, modifications, English
learners, and irregularities
• Data Layout = guide to location of data on files
– Position
– Number of characters
– Whether numeric or alpha
2014 Post-Test Workshop
44
Data Layout Sample
2014 Post-Test Workshop
45
Individual Reports
•
CAASPP Student Record Label
–
•
CAASPP Student Master List
–
–
•
Adhesive label to affix to student’s permanent school
record
Alphabetical list of students and their scores
Tests listed in order within grade
 CSTs
 CMA
 CAPA
 STS
CAASPP Student Report: individual’s scores
–
–
–
Two 2-sided color copies for each test
For parents/guardians, school
Per California Code of Regulations, Title 5, Section 863,
LEA must forward one copy to parent/guardian within 20
business days
PTG 40–64
2014 Post-Test Workshop
46
Student Record Label Grade 5 Sample:
Student Name and Identification
PTG 40–41
2014 Post-Test Workshop
47
Student Record Label
CST Grade 5 Example
PTG 40–41
2014 Post-Test Workshop
48
Student Master List
CST/CMA Grade 10 Example
PTG 44
2014 Post-Test Workshop
49
Student Report
CST Grade 5 Example
PTG 46–50
2014 Post-Test Workshop
50
Student Report
CMA Grade 5 Example
PTG 51–55
2014 Post-Test Workshop
51
Student Report
STS Grade 2 Example
PTG 60–64
2014 Post-Test Workshop
52
Explain to Parents
Scale Score  Average % correct
cluster score x 600
•
Reporting clusters not comparable
–
–
–
•
Different difficulty
Varying number of questions
Average % correct of clusters  % correct on total test (or
total number of test items)
Scale scores
–
–
Use conversion tables resulting from statistical
procedures
Equating allows scores to have similar meaning
(e.g., 350 = lowest score for CSTs proficient)
PTG 7–13
2014 Post-Test Workshop
53
Student Report Summary
•
•
•
•
CST (PTG 46–50)
–
–
Front: Scale scores, performance levels, cluster percent correct
Back: Interpretation guide
–
–
Front: Scale scores, performance levels, cluster percent correct
Back: Interpretation guide
–
–
Front: Scales scores, performance levels
Back: Interpretation guide
CMA (PTG 51−55)
CAPA (PTG 56−59)
STS (PTG 60-64) (in Spanish)
–
–
Front: Scale scores, performance levels, cluster percent correct
Back: Interpretation guide
2014 Post-Test Workshop
54
Summary
•
Do’s
– Do compare mean scale score, percent at
performance levels within same grade, same
content area.
– Do compare cluster scores to the corresponding
proficient range provided for this year.
•
Don’ts
– Don’t compare cluster scores to each other within
test, nor across years, grades, content areas.
– Don’t compare mean scale scores across grades,
content areas.
•
•
•
Summary and Internet reports
Data CDs
Individual reports
2014 Post-Test Workshop
55
Upcoming Dates
Event
Post-Test Guide posted on http://caaspp.org
Appendixes A & C of the Post-Test Guide
posted on http://caaspp.org
Web Reports available on
http://caaspp.cde.ca.gov
Research files available
Post-Test Guide distributed
Summary Reports
Date
Mid August
September 12
September 17
September 17
Mid September
September 5 through October 10
2014 Post-Test Workshop
56
2015 Setting Up Test
Administrations
1. Return the 2015 CAASPP Superintendent’s
Designation form and Security Agreement.
2. Finalize instructional schedule.
3. Set up admin in CAASPP Test Operations
Management System prior to December 1.
4. Enter and approve orders prior to December 1.
After order approval, test administration dates
should NOT be changed.
2014 Post-Test Workshop
57
For more information see:
• CDE CAASPP office
– 916-445-8765
– CAASPP Web page: http://www.cde.ca.gov/ta/tg/ca/
– CAASPP Reporting Web site:
http://caaspp.cde.ca.gov/
• CDE Accountability
– aau@cde.ca.gov
– 916-319-0863
• California Technical Assistance Center
– 800-955-2954
– http://caaspp.org/
2014 Post-Test Workshop
58
Early Assessment Program
Results
August 2014
Connie Grueter
Early Assessment Program (EAP)
• 2014 EAP Student Report
– Looks different than 2007 – 2013
– Limited electronic data provided through Order
Management System
– Electronic data on CST Student Data CD-Rom
• 2014 Aggregate Data file
– Available mid-September
PowerPoint slides – www.CollegeEAP.org, Educators, About EAP.
2014 Post-Test Workshop
60
Early Assessment Program (EAP)
• An EAP informational web site has been
developed: www.CollegeEAP.org.
– The site provides EAP information for both
CSU and CCC
– It is a resource for Students, Parents, and
Educators
PowerPoint slides – www.CollegeEAP.org, Educators, About EAP.
2014 Post-Test Workshop
61
www.CollegeEAP.org
2014 Post-Test Workshop
62
Early Assessment Program (EAP)
• Reporting Results
– Ready for CSU and participating CCC college-level
English/Mathematics courses.
– Ready for CSU and participating CCC college-level
English/Mathematics courses – Conditional
 This status in conjunction with a grade of “C” or better in
an approved English/math course or a supervised elearning program can be exempt from the EPT or ELM for
CSU or other English or math placement tests at
participating California Community Colleges (CCC).
 A list of courses and e-learning programs can be found at:
www.csumathsuccess.org and www.csuenglishsuccess.org .
– Not yet demonstrating readiness for CSU and participating
CCC college-level English/Mathematics courses.
2014 Post-Test Workshop
63
Sample 2014 EAP Student Report
Early Assessment Program (EAP)
2014 Post-Test Workshop
64
Early Assessment Program (EAP)
•
Web Resources
– www.CollegeEAP.org
 Links to CSU and CCC informational sites for
Students, Parents, and Educators
– www.CalState.edu/EAP
 Provides a list of EAP Coordinators, EAP test
blueprints, and informational materials
– www.CCCCO.edu/EAP
 Provides a list of participating Community
Colleges
PowerPoint slides – www.CollegeEAP.org, Educators, About EAP.
2014 Post-Test Workshop
65