"
"
CRESST Senior Staff Meeting !
16 February 2012 !

"

"

"
College Ready for All is a BMGF funded initiative for college readiness with the goal of
!
80% of high-school students graduating
“college-ready” !

"
© Regents of the University of California

"
•   Common Core State Standards (CCSS) , are a set of academic standards in mathematics and literacy that establish what students at every grade level need to know in order to be prepared for success beyond high school.
!
•   Most states (all but five) have agreed to adopt the
CCSS providing a tremendous opportunity for the development and refinement of tools and materials to help support this transformation.
!
© Regents of the University of California

"
!
Part of the College for All initiative involves an investment in the development and implementation of formative assessment tools for middle and high school teachers that are aligned with the CCSS in
English/language arts and mathematics. !
© Regents of the University of California

"
"
"
"
"

© Regents of the University of California

!
"
Initial planning grant focused on: "
•   the development of a knowledge model for mathematics (ontology) to guide assessment development and validation !
•   developing a validation framework for performance-based assessments !
•   analysis within this framework of math items developed by other initiative partners .
!
© Regents of the University of California

!
!
"
Current grant extends that work to: "
•   develop ontologies for reading and writing as applied to core ELA, Social Studies, and Science content.
!
•   develop and validate tasks assessing college readiness in reading and writing !
© Regents of the University of California

"
Our work primarily supports the
Foundations ʼ FOCUS on success at a higher level : ontology frameworks and assessment tasks will support the definition, measurement, and implementation of “fewer, clearer, higher” standards for college readiness. !
Other tools developed, can also be seen as “innovations to support and engage students.” !
© Regents of the University of California

"
© Regents of the University of California

"
•   Specifically aligned with the Common Core
State Standards "
•   Connect to subject area content ontologies - organized disciplinary knowledge in History,
Social Studies, Literature, and Math !
•   Architectures defined by "
•   Explicit rules for stimulus materials and task constructions !
•   Explicit rules for scoring student responses !
© Regents of the University of California

"
!
!
!
!
!
!
•   Focus on fundamental topic/principle in given discipline "
•   Students read a variety of related texts "
   Appropriate text complexity, grade level !
   Text forms authentic to discipline !
•   Respond to reading comprehension questions of aligned to specific reading skills (CCSS) "
•   Respond to argument or expository prompt: establish and support claim "
© Regents of the University of California

"
•
Focus on two major CCSS writing standards spanning middle/high school grades: "
   Writing arguments to support claims in an analysis of substantive topics/texts, using valid reasoning and relevant or sufficient evidence !
   Write informative/explanatory texts to examine and convey complex ideas and information clearly and accurately through the effective selection, organization, and analysis of content.
!
•
All the standards in reading: "
   Key ideas & details !
   Craft & structure !
   Integration of knowledge & ideas !
© Regents of the University of California

"
Each assessment form contains: "
•   passages and/or illustrations (including primary source documents, fiction and/or non-fiction pieces) !
•   reading comprehension questions !
•   a writing prompt (either argument or explanation) !
•   teacher/student surveys !
© Regents of the University of California

Integrated Learning Assessments for ELA:
The assessments
© Regents of the University of California

Integrated Learning Assessments for ELA:
The assessments
© Regents of the University of California

"
Initial validation efforts focused on evidence from expert review and empirical pilot studies. "
Specific research questions address: "
•   Analyzing individual item quality across various forms !
•   The interrelation between individual items and overall assessment !
•   Impact of different prompt types on essay score !
•   Test reliability !
•   Evaluation of intended reading and writing skills !
© Regents of the University of California

"
Pilot testing included 39 teachers at 15 schools and we collected 3,401 essays: "
•
Life Science: 1,895 essays (6 topics) !
•
Social Studies: 1,088 essays (5 topics) !
•   Literature: 459 essays (3 topics) !
The student sample represented a range of ability level and relevant sub populations. "

Assessment Piloting: Preliminary Analysis "
Multiple-choice items: "
•
Preliminary data analysis (including descriptive statistics distracter analysis) was conducted on all reading comprehension items. !
•
This was followed by an IRT analysis using a 1PL Rasch
Model to investigate item difficulty.
!
•
A confirmatory factor analysis was conducted to verify the
CCSS reading standards classification.
!
•
Items revised (based on the above analyses) for field-testing.
!

Assessment Piloting: Preliminary Analysis "
Essays: "
•   The prompts elicited a range of scorable student responses !
•
All essays have been scored using the CRESST-developed rubric (5 dimensions—aligned to the CCSS) !
•   Analyses currently underway include: !
   determining students ʼ performance on essay items (mainly descriptive); !
   estimation of difficulty levels !
   comparison of student responses to different levels/types of essay prompts. !

Assessment Piloting: Preliminary Analysis "
Example Science Essays "
•
N= 1,438, 6 essay topics "
   7th grade, N = 955; 8th grade N = 483 !
   Argument prompt N = 687; Explanation N = 751 !
   On all rubric dimensions, 8 th grade students scored higher than 7 th grade students !
   On the organization dimension, the students with the argument prompt scored higher. All others were the same. !
   Analyses of these data are ongoing.
!

Next Generation Assessments for Math "
•   Specifically aligned with the Common Core State
Standards !
•   and the goals of the Shell Centre Formative
Assessment Lessons (FALs) !
•   Theoretically driven, empirically validated architectures for measuring different kinds of cognitive demands !
•   Initial drafts of 450 Shell Centre FAL-aligned items developed.
!
•   Small-scale pilot of a sub-set of items with students at the UCLA Community school and one KY classroom.
!
© Regents of the University of California

Field Testing "
© Regents of the University of California

Field Testing "
"
" " "
"
!
"
•   Lots of schools !
•   Lots of districts !
•   Lots of closed doors !
© Regents of the University of California

Field Testing ELA and Math "
•   2011-2012 work focuses on: "
   continued assessment development work and !
   field testing in districts involved in the MDC/LDC initiatives !
   Coordinate validation plan and implementation for CRESST assessments with other partners !
•   Primary goals are to collect more data on: "
   Generalizability !
   Relationship to other measures !
   Preliminary instructional sensitivity !
   Additional validity evidence !
© Regents of the University of California

Quasi-Experimental Study of LDC & MDC "
•   Combines CRESST capacities in assessment, evaluation, statistical design and analysis !
•   CRESST assessments to measure LDC and MDC outcomes !
•   Innovation in Fidelity of Implementation Measures !
•   Support intervention improvement !
•   Partners: RFA, WestEd, Measured Progress !