2010 STAR Interpreting and Using Results August 11, 2010 Webcast Webcast starts at 9:00 a.m. Objectives Workshop participants will be able to: • Describe the purposes of STAR reports • Interpret STAR results • Explain key statistics • Compare and contrast types of reports • Identify proper uses of reports 2 August 2010 Post-Test Workshop Agenda • • • • • • • 3 What’s New? Results and Statistical Analysis Using Results Summary and Internet Reports Data CDs Individual Student Reports Early Assessment Program August 2010 Post-Test Workshop What’s New in 2010 Manual • California Modified Assessment (CMA) (M) 2 reports – Scale score, performance levels for • Grades 6–8 English−Language Arts (ELA) • Grades 6–7 mathematics • Grade 8 science – Percent correct for • ELA grade 9 • Life Science, grade 10 • Algebra I 4 August 2010 Post-Test Workshop More What’s New • Standards-based Test in Spanish (STS) reports – Scale score, performance levels for • Grades 5–7 Reading/Language Arts (RLA) • Grades 5–7 mathematics – Percent correct for • Grades 8–11 RLA • Grades 7–11 Algebra I • Grades 8–11 Geometry • Subgroup Summary: Ethnicity • Grade 11 – Early Assessment Program (EAP) – California Community Colleges added 5 August 2010 Post-Test Workshop M2 Quiz Question 1 Which of these tests had scale scores reported for the first time in 2010? A. CST for World History B. CAPA for Science C. CMA for math grade 7 and science grade 8 D. STS for math grade 7 and Algebra I 6 August 2010 Post-Test Workshop Quiz Question 1 Which of these tests had scale scores reported for the first time in 2010? C. CMA for math grade 7 and science grade 8 7 August 2010 Post-Test Workshop Results: Purposes of STAR Reports • Report progress of students toward proficiency on the state’s academic content standards • Notify where improvement needed – To help students’ achievement – To improve educational programs • Provide data for state and federal accountability programs 8 August 2010 Post-Test Workshop M4 Results: Performance Levels • State Goal: All students score at proficient or higher • CAPA Proficient: 35 or higher scale score • 350 or higher scale score – All CST – STS, grades 2–7, excluding Algebra I – CMA, grades 3–8, excluding Algebra I 9 August 2010 Post-Test Workshop M 8−10 Results: Other Performance Levels • Advanced • Basic cut score – CAPA: 30 – CST: 300 – CMA, grades 3–8: 300 (excluding Algebra I) – STS, grades 2–7: 300 (excluding Algebra I) • Below basic • Far below basic • Cut points vary for advanced and below basic by – Subject – Grade 10 August 2010 Post-Test Workshop M 8−10; Appendix B Results: Scale Scores M 8−10 • Scale scores allow same score to mean same thing across test versions within grade and content area • Accounts for differences in difficulty • Scale score ranges by program: – CST, CMA (grades 3–8, excluding Algebra I), STS (grades 2–7, excluding Algebra I): 150–600 for each grade and subject – CAPA: 15–60 for each level and subject 11 August 2010 Post-Test Workshop Results: Equating • Psychometric procedure • Adjusts for test difficulty • Additional information in the CST Technical Report on the CDE Web site 12 August 2010 Post-Test Workshop M8 Results: Reporting Clusters (Content Areas) • Three to six clusters for each subject M 8−12; Appendix A • May be useful as indicators of individual or group strengths and weaknesses • But . . . Reporting clusters should be interpreted with caution 13 August 2010 Post-Test Workshop Results: Cluster Percent Correct Available for 2010 M 8−12; Appendix A • CST: all • CMA: – Grades 3–8, excluding Algebra I • STS: all 14 August 2010 Post-Test Workshop Results: Reporting Cluster Cautions • Based on small numbers of items; therefore, may not be reliable or generalizable • NOT equated from year to year • Should not compare reporting cluster percent correct from year to year 15 August 2010 Post-Test Workshop M 8−12; Appendix A Interpreting Reporting Clusters or Content Areas M 8−12; Appendix A in the Same Year • Compare to percent-correct range of proficient students statewide 16 August 2010 Post-Test Workshop 2010 CST Reporting Clusters: Number of Questions and M Appendix A Average Percent Correct To be finalized with complete data in 2010 Post-Test Guide, after August 16. 17 August 2010 Post-Test Workshop Examples—Interpreting Reporting Clusters for the CST for Geometry 18 August 2010 Post-Test Workshop M 11 Quiz Question 2 What is a scale score? A. Percent correct of all questions B. Mean percent correct of all questions C. An adjustment of this year’s and last year’s raw scores to show changes D. An adjustment of the raw score to account for differences in difficulty 19 August 2010 Post-Test Workshop Quiz Question 2 What is a scale score? D. An adjustment of the raw score to account for differences in difficulty 20 August 2010 Post-Test Workshop Using Results M4 • For instructional decisions with other data • Used in Academic Performance Index (API) calculations: CSTs, CAPA, CMA (only grades 3–8, excluding Algebra I) • Used in adequate yearly progress (AYP) calculations, ELA and mathematics: – Grades 2–8, CSTs – Grades 3–8, CMA (excluding Algebra I) – Grades 2–8 and 10, CAPA 21 August 2010 Post-Test Workshop Year-to-Year Comparisons M 13−16 Do Compare CSTs: Same Grade and Same Content Area • Mean scale score – Same content and grade, varying years • Percent in each performance level – Same content by grade across years • e.g., 2009 ELA grade 10 with 2010 ELA grade 10 22 August 2010 Post-Test Workshop Year-to-Year Comparisons Do Compare CSTs: M 13−16 Percent Proficient and Advanced • Percentage of students scoring at PROFICIENT and above – For a given grade and subject, e.g., Percent proficient and above for grade 3 math in 2009 and 2010 – For a given subject and aggregated grades, e.g., Percent proficient and above for grades 2– 6 mathematics in 2009 and 2010 – Across grades and a subject, e.g., Percent proficient and above in all courses and all grades 23 August 2010 Post-Test Workshop Year-to-Year Comparisons DON’T Compare M 13−16 • Individual scale scores or statistics based on scale scores for different grades or content areas – Subjects by grade are independently scaled – Different content standards are measured in different grades • Cohorts across grades • Across tests • CMA grades 6−11, STS grades 5−11 to previous years • CAPA, CMA grades 3−5, STS grades 2−4 to years before 2009 24 August 2010 Post-Test Workshop Example—Using CST Results to Compare Grade Results from Year to Year 2009 CST for ELA 2010 CST for ELA % Prof or Above % Prof or Above Difference Grade 2 31% 35% 4% Grade 3 33% 33% 0% Grade 4 29% 31% 2% Grade 5 34% 32% –2% Grade 6 31% 32% 1% All Grades 32% 33% 1% Grade 25 M 14 August 2010 Post-Test Workshop Quiz Question 3 What is the best comparison for CST scores of students within a middle school? A. 2009 mean scale scores for ELA of a cohort of grade 7 students with 2010 scale scores for ELA of the same students in grade 8 B. 2009 mean scale scores for ELA for grade 8 students with 2010 mean scale scores for ELA for grade 8 students C. 2009 mean percent correct scores for ELA with 2010 percent correct scores for ELA for the same students in grade 8 D. 2009 mean percent correct scores for ELA for grade 8 students with 2010 mean percent correct for ELA for grade 8 students 26 August 2010 Post-Test Workshop Quiz Question 3 What is the best comparison for CST scores of students within a middle school? B. 2009 mean scale scores for ELA for grade 8 students with 2010 mean scale scores for ELA for grade 8 students 27 August 2010 Post-Test Workshop Quiz Question 4 Which is the best comparison of cluster scores for a single student? Compare . . . A. To proficient students statewide B. One cluster to another, same year C. The same cluster to the same cluster, different years D. To the average percent correct of all students in a class 28 August 2010 Post-Test Workshop Quiz Question 4 Which is the best comparison of cluster scores for a single student? Compare… A. To proficient students statewide 29 August 2010 Post-Test Workshop Aggregate (Summary) Reports M 18−22 • What are they? – Student Master List Summary – Student Master List Summary End-ofCourse (EOC) – Subgroup Summary • Report Emphasis: CSTs – Criterion-referenced tests – Progress is measured in percent of students scoring proficient and advanced 30 August 2010 Post-Test Workshop Student Master List Summary • • • • • M 19–20, M 28−34 By grade CSTs, CMA, CAPA, and STS Lists subjects % and # at each performance level Mean scale score (CST, CAPA, CMA grades 3–8 [not Algebra I], STS grades 2–7 [not Algebra I]) • Reporting cluster: mean percent correct (CST, CMA grades 3–8 [not Algebra I], STS) 31 August 2010 Post-Test Workshop Student Master List Summary M 34 Grade 7 Example 32 August 2010 Post-Test Workshop Student Master List Summary Basic Statistics M 28−34 33 August 2010 Post-Test Workshop Who Counts? Number Enrolled M 28−34 • Total CST/CMA and CAPA multiplechoice answer documents submitted as scorable • Minus - Documents marked as “Student enrolled after the first day of testing and was given this test” 34 August 2010 Post-Test Workshop Who Counts? Number Tested M 30−32 • All CST, CMA, CAPA, STS answer documents with one or more answers • Plus – Z = Tested but marked no answers • Not included – A = Students absent – E = Not tested due to significant medical emergency – P = Parent/guardian exemptions – T = Enrolled first day, not tested, tested at previous school – Students with inconsistent grades – Non–English learners who took the STS 35 August 2010 Post-Test Workshop Who Counts? Number and Percent Valid Scores M 30−32 • Number Valid Scores – For the subject, number of students tested at grade level who received a score for the test (that is, a scale score or percent correct). • Percent Valid Scores – For the subject, number of valid scores divided by the number of students tested. – Not included: Incomplete tests Modified tests Non–English learners who took the STS Unknown EOC mathematics (except grade 7 mathematics) or science tests Inconsistent grades 36 August 2010 Post-Test Workshop Who Counts? Number Tested with Scores M 30−32 • All tests taken, including those taken with modifications, that receive a score • Not included: – Incomplete tests – Non–English learners who took the STS – Unknown EOC mathematics (except grade 7 mathematics) or science tests – Inconsistent grades 37 August 2010 Post-Test Workshop Who Counts? Performance Levels • • • • M 30−32 All CSTs, CAPA CMA Grades 3–8, excluding Algebra I STS Grades 2–7, excluding Algebra I Advanced, proficient, basic, below basic – All valid scores falling in the performance level • Far below basic – All valid scores falling in the performance level – CSTs taken with modifications (in aggregate reporting and accountability only) 38 August 2010 Post-Test Workshop Student Master List Summary Performance Levels 39 August 2010 Post-Test Workshop M 31 Student Master List Summary Reporting Clusters M 31−32 Compare to: Average percent correct range for students statewide who scored proficient on the total test 40 August 2010 Post-Test Workshop Student Master List Summary: Writing B C I L 41 = = = = August 2010 Blank Copied prompt Illegible Language other than English M 32 R = Refusal T = Off Topic W = Wrong prompt (prompt from an earlier administration) Post-Test Workshop Student Master List Summary: M 35−40 End-of-Course (EOC) CST, CMA, and STS • By subject • Lists each grade eligible to take test • Mathematics (Grades 7–11) – CST General Math, Algebra I, Geometry, etc. – CMA Algebra I – STS Algebra I and Geometry • CST Science (Grades 9–11) – Earth Science, Biology, Chemistry, etc. • CST History–Social Science (Grades 9–11) – World History • Same statistics as grade-level Student Master List Summary 42 August 2010 Post-Test Workshop Student Master List Summary End-of-Course CST Biology Example 43 August 2010 Post-Test Workshop M 40 Subgroup Summary: CSTs, CMA, CAPA, and STS M 41−59 • Disability status – Based on disability status for CST, CMA, STS – CAPA: each disability type • If missing, correct with demographic data corrections • Economic status – Based on NSLP eligibility or parent education level • Gender • English proficiency • Ethnicity • Ethnicity for Economic Status (only for CSTs, CMA grades 3–8, and CAPA) 44 August 2010 Post-Test Workshop Subgroup Summary: Gender Grade 5 Example 45 August 2010 Post-Test Workshop Subgroup Summary: Ethnicity • From answer document sections 8 and 9: – Hispanic or Latino? – Race: grouped and worded differently – May cause counts to differ from previous year – Federal Reporting Requirement • Two or more races (instead of “unknown”) 46 August 2010 Post-Test Workshop Subgroup Summary: Ethnicity for Economic Status Ethnicity for Economically Disadvantaged CST Grade 6 Example 47 August 2010 Post-Test Workshop M 64 Subgroup Summary: Ethnicity for Economic Status M 60−64 Example: Economically disadvantaged for each ethnicity 48 August 2010 Post-Test Workshop Subgroup Summary: Ethnicity for Economically Disadvantaged M 60−64 49 August 2010 Post-Test Workshop Break — 10 minutes Internet Reports M 104−117 • Summaries based on same data as paper reports: CSTs, CMA, CAPA, STS • Available to the public online for school, district, county, and state • More subgroups than paper reports – Parent education – Special program participation • Access from http://star.cde.ca.gov/ 51 August 2010 Post-Test Workshop Internet Reports: Available Reports • CSTs – Mean scale scores – Percents by performance levels • CST Summary – Percent proficient and advanced • CMA – Grades 3–8 (not Algebra I): mean scale scores, percents by performance levels – Grade 9 ELA, Grade 10 Life Science, Algebra I: average percent correct 52 August 2010 M 104−117 • CAPA – Mean scale scores – Percents by performance levels • STS – Grades 2–7 (not Algebra I): mean scale scores, percents by performance levels – Grades 8–11: average percent correct Post-Test Workshop Internet Demonstration 53 August 2010 Post-Test Workshop Internet Reports: CST Sample 54 August 2010 Post-Test Workshop M 108−109 Internet Reports: CST Summary Sample 55 August 2010 Post-Test Workshop M 109−110 Other Internet Reports • CMA (M 111−112) – Same as CST—CMA grades 3−8 – Percent correct—CMA grades 9–11, Algebra I • CAPA (M 112−115) – State level: same as CST; separate Level I – County, district, school • Mean scale score • Percent proficient or above • STS (M 116−117) – Same as CST—STS grades 2−7 – Percent correct—STS grades 8−11 56 August 2010 Post-Test Workshop Quiz Question 5 Which subgroup can only be accessed from the Internet? A. Parent Education Level B. CAPA by individual disability status C. Ethnicity for Economic Status D. English Proficiency 57 August 2010 Post-Test Workshop Quiz Question 5 Which subgroup can only be accessed from the Internet? A. Parent Education Level 58 August 2010 Post-Test Workshop Data CDs • What are they? – Lists of information from answer documents and scores of every student in district – In .txt format: wraps • What are they used for? – Searching for specific data – Creating unique reports – Verifying paper reports • What else is needed? – Text editor – or Desktop application – or Student Information System 59 August 2010 Post-Test Workshop View of Data • As .txt, word wrap on • With text editor, word wrap off 60 August 2010 Post-Test Workshop Organization of Data • Two files: – Demographics, special conditions, and test scores – Accommodations, modifications, English Learners, and irregularities • Data Layout = guide to location of data on files – Position – Number of characters – Whether numeric or alpha 61 August 2010 Post-Test Workshop Data Layout Sample 62 August 2010 Post-Test Workshop Individual Reports • STAR Student Record Label M 23−26 – Adhesive label to affix to student’s permanent school record • STAR Student Master List – Alphabetical list of students and their scores – Tests listed in order within grade • CSTs • CMA • CAPA • STS • STAR Student Report: individual’s scores – 2 two-sided color copies for each test – For parents/guardians, school 63 August 2010 Post-Test Workshop Student Record Label Grade 10 Sample: Student Name and Identification M 65 64 August 2010 Post-Test Workshop Student Record Label M 65−66 CST/CMA Grade 10 Example 65 August 2010 Post-Test Workshop Student Master List CST/CMA Grade 3 Example 66 August 2010 Post-Test Workshop M 69−71 Student Report M 72; 73−82 CST Grade 11 Example 67 August 2010 Post-Test Workshop Student Report CST Grade 11 Example 68 August 2010 Post-Test Workshop M 74−75 Student Report CST Grade 10 Example 69 M 76−78 Student name on back ♦ = Percent correct obtained by the student on the reporting cluster/content area ▬ = Range of percent correct scores on the reporting cluster for students statewide who scored proficient on that test August 2010 Post-Test Workshop Student Report CST Grade 11 Example, EAP Results 70 August 2010 Post-Test Workshop M 78; 82 Explain to Parents M 13 Scale Score Average % correct cluster score x 600 • Reporting Clusters not comparable – Different difficulty – Varying number of questions – Average % correct of clusters % correct of total scale score • Scale scores – Use conversion tables and other statistical techniques – Equating allows scores to have similar meaning (e.g., 350 = lowest score for CSTs proficient) 71 August 2010 Post-Test Workshop Other Student Reports • CMA (M 82−90) – Performance levels and scale scores reported for grades 3–8 Back, cluster reporting for grades 3–8 – Percent correct for entire test for grades 9–11 Back, “About the CMA” • CAPA (M 91−95) – Back, “About the CAPA” • STS (M 96−103) – In Spanish – Performance levels and scale scores, grades 2–7 Back, cluster percent correct – Percent correct for entire test, grades 8–11 Back, cluster percent correct Back, how to use report 72 August 2010 Post-Test Workshop Unmatched Reports Grade 7 Writing • Unmatched Multiple-Choice Report CST multiple-choice score but no writing score CMA multiple-choice score but no writing score • Unmatched Writing Report Writing score but no CST multiple-choice score Writing score but no CMA multiple-choice score • Students receive 2 reports if writing score not matched to multiple-choice score 73 August 2010 Post-Test Workshop Unmatched Report CMA Grades 8−11 •Student took at least 1 CMA test, but no CST report matches •All grades except grade 9 require a CST Grade 8 9 10 11 Tests That Can be Taken as CST or CMA ELA, Algebra I, Science ELA, Algebra I Algebra I, Life Science Algebra I CST Required (No CMA option) History-Social Science ELA ELA, History-Social Science Table does not include end-of-course CSTs for grades 8−11 74 August 2010 Post-Test Workshop Unmatched Report CMA Grades 8−11 • Report for each grade • Lists student data for those with only CMA or demographics do not match CST 75 August 2010 Post-Test Workshop Summary • Do’s and Don’ts – Do compare mean scale score, percent at performance levels within same grade, same content area – Don’t compare mean scale scores across grades, content area • Summary and Internet reports • Data CDs • Individual reports 76 August 2010 Post-Test Workshop For more information see: • • • • http://www.cde.ca.gov/ta/tg/sr/ http://www.star.cde.ca.gov/ http://www.startest.org/ STAR Technical Assistance Center – 800-955-2954 • CDE Accountability – aau@cde.ca.gov – 916-319-0863 77 August 2010 Post-Test Workshop To Contact CDE • CDE STAR office: 916-445-8765 – English–Language Arts tests, including writing: Jamie Contreras, jcontrer@cde.ca.gov, 916-319-0353 – Mathematics tests: Jane Liang, jliang@cde.ca.gov, 916-322-1854 – Science tests: Blessing Mupanduki, bmupanduki@cde.ca.gov, 916-319-0969 – History–Social Science tests: Nicole Jespersen, njespersen@cde.ca.gov, 916-3190364 – CAPA, CMA: Don Killmer, dkillmer@cde.ca.gov, 916-319-0350 78 August 2010 Post-Test Workshop