Surveys of Enacted Curriculum
Tools for Aligning Instruction, Standards,
& Assessments
SEC Project Planning & Leader Training
February 2005
SEC State Collaborative
Tampa, Florida
© 2005 CCSSO ALL RIGHTS RESERVED.
New Member Orientation
What? … are the Survey of Enacted
Curriculum tools?
 How?… are the collected, analyzed,
reported?
 Why?… are they useful to educators,
leaders, researchers?
 Now what?… do we do to implement and
apply to our programs and to schools?

© 2005 CCSSO ALL RIGHTS RESERVED.
Rationale
Standards
Assessment
Curriculum
© 2005 CCSSO ALL RIGHTS RESERVED.
What SEC offers

Core Components of SEC tools/services
1) Surveys with teachers – instruction for year
2) Alignment analysis – content code standards
and assessments
3) Data reports – pre-designed charts
4) Workshops – a) train for orientation
b) use of data
© 2005 CCSSO ALL RIGHTS RESERVED.
SEC Collaborative
15 States, 3 Districts
CCSSO – Project Management
Rolf Blank, Carlise Greenfield
WCER – Surveys, reporting, analysis
John Smithson, Alissa Minor, Eric Osthoff
Training & Technical Assistance
Learning Point Associates, TERC Regional Alliance, Consultants
Diana Nunnaley
Carolyn Karatzas
Jennifer Unger
Bryan Chumbley
Lani Seikaly
© 2005 CCSSO ALL RIGHTS RESERVED.
SEC Collaborative
Functions/ Assumptions (04-05)
 Training and assistance to state leaders
 State SEC budget to meet state objectives –
supported by SEC Collaborative
 Each state makes decisions
– on services & project plan

Multi-state benefits of collaborative
© 2005 CCSSO ALL RIGHTS RESERVED.
Multi-State Collaborative
Build Knowledge of leaders
 Share Strategies
 Produce joint products, tools

© 2005 CCSSO ALL RIGHTS RESERVED.
Key Question -- SEC Tools
Research into Practice

How can Educators obtain reliable, valid data
to determine Alignment of instruction with
required standards and assessments?
© 2005 CCSSO ALL RIGHTS RESERVED.
Survey Sections
• School & Class Description
• Instructional Activities
• General
• Problem Solving Activities
• Pairs & Small Group Work
• Use of Hands-on Materials
• Use of Calculators/Computers &
other Ed. Tech.
• Assessment Use
• Instructional Influences
© 2005 CCSSO ALL RIGHTS RESERVED.
• Instructional Readiness
• Teacher Opinions
• Professional Development
• Types
• Content Focus
• Active Learning
• Collegial Participation
• Coherence
• Time Span
• Teacher Characteristics
• Instructional Content
Steps in SEC Development
 R&D studies curriculum-- Early ‘90s







Models:TIMSS, NAEP, Analyze assessment
OTL interest of States, Porter/ Smithson research,
CCSSO Science Assessment project
State collaborative –Develop surveys-’98-’01 (NSF)
11-State field study, Reports, Alignment, USI
Use of data experiment – DEC (NSF/ROLE:‘01-04)
Evaluation: MSP RETA PD Study (‘02 – 05, NSF)
English/ Language Arts survey (‘03 – 04)
Current projects: 15 States, 4 Districts, 5 MSPs
© 2005 CCSSO ALL RIGHTS RESERVED.
Questions Addressed by SEC Data

How can in-depth data on content of instruction be
collected and reported–not topic checklists, to
analyze teaching content in relation to standards,
assessments, achievement?
 How can methods of teaching practices be
compared across classrooms, schools, districts,
and states?
 How can enacted curriculum data be reported in a
manner to encourage use by teachers to improve
instruction?
© 2005 CCSSO ALL RIGHTS RESERVED.
Key Education Questions
(cont’d)

How can we measure the effects of standardsbased initiatives on instructional practices and
curriculum in classrooms?
 How can we analyze effectiveness of professional
development on changes in teachers’ instructional
practices? (i.e. determine the quality of
professional development)
© 2005 CCSSO ALL RIGHTS RESERVED.
Applications
 Alignment analysis --instruction, standards,
assessments
 Instructional improvement in schools
 Needs assessment/ Evaluation
 Indicators – monitoring change over time
© 2005 CCSSO ALL RIGHTS RESERVED.
What if…
[for Intro of SEC to Educators]
You could use data on instructional quality and
content to guide professional development?
You
could have consistency across grade levels?
You
could know how well aligned your state
standards were to the state assessment?
You
could compare how you teach content
compared to how others across the nation teach?
You
could use anonymous teacher data to start a
powerful school discussion about what the teacher
needs are?
© 2005 CCSSO ALL RIGHTS RESERVED.
SEC Instructional Content
Uses a multi- dimensional la nguage for
describing instructional content
Topics
by
Cognitive Demand
(Expectations
for Student
Learning)
© 2005 CCSSO ALL RIGHTS RESERVED
.
Content Matrix
Categories of Cognitive Demand
Memorize
Topics
Perform
Procedures
Communicate Solve NonUnderstanding
Routine
Problems
Multiple
Step
Equations
Inequalities
Literal
Equations
Lines /
Slope and
Intercept
Operations
on
Polynomials
Quadratic
Equations
© 2005 CCSSO ALL RIGHTS RESERVED.
Conjecture
Generalize
Prove
Content Maps
State J Grade 8
Mathematics Instruction
Number Sense
Operations
Measurement
Algebraic Concepts
Geometric Concepts
Data Analysis
0.1
0.08
0.06
0.04
0.02
0
Instructional Technology
Memorize
Communicate
Connect
Perform
Conjecture
Memorize Communicate Connect
Perform
Conjecture
© 2005 CCSSO ALL RIGHTS RESERVED.
Surveys of Enacted Curriculum
The intended
curriculum:
State content
standards—
What students
should learn
A neutral content grid
The assessed
curriculum:
State (and other)
assessments—
tested learning
with cognitive demand
The learned
curriculum:
Student outcomes
based on school
learning
The enacted
curriculum:
What teachers
teach
© 2005 CCSSO ALL RIGHTS RESERVED.
Curriculum Analysis:
Enacted
Intended
What teachers teach
© 2005 CCSSO ALL RIGHTS RESERVED.
What standards require
Full Alignment Analysis
Intended—
standards
Enacted—
taught
Assessed—
state tests
GRADE 8 STATE MATH TEST
SCHOOL averages compared to
STATE averages
-10
-5
0
5
10
15
1
2
3
4
5
© 2005 CCSSO ALL RIGHTS RESERVED.
20
SEC Content Topics
Math, Science, ELA
The SEC provides a
neutral, researchbased language to
describe content of
English language
arts, mathematics,
and science.
NCLB anyone?
© 2005 CCSSO ALL RIGHTS RESERVED.
© 2005 CCSSO ALL RIGHTS RESERVED.
Time on Topic
Expectations for Students in M
K-8 Mathematics Topics
Data Analysis / Prob. / Statistics
Memorize Facts/
Definitions/
Formulas
601
Bar graph, histogram
€  ‚ ƒ
€  ‚ ƒ
602
Pie charts, circle graphs
€  ‚ ƒ
603
Pictographs
€  ‚ ƒ
604
Line graphs
€  ‚ ƒ
605
Stem and Leaf plots
€  ‚ ƒ
606
Scatter plots
€  ‚ ƒ
607
Box plots
€  ‚ ƒ
608
Mean, median, mode
€  ‚ ƒ
609
Line of best fit
€  ‚ ƒ
610
Quartiles, percentiles
€  ‚ ƒ
611
Sampling, Sample spaces
€  ‚ ƒ
612
Simple probability
€  ‚ ƒ
613
Compound probability
€  ‚ ƒ
614
Combinations and permutations
€  ‚ ƒ
615
Summarize data in a table or graph
<none>
6
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
© 2005 CCSSO ALL RIGHTS RESERVED.
€  ‚ ƒ
Perform
Procedures
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
Demonstrate
Understanding of
Mathematical
Ideas
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
€  ‚ ƒ
© 2005 CCSSO ALL RIGHTS RESERVED.
Expectations for Students in Mathematics
Memorize Facts/ Definitions/
Formulas
Recite basic mathematics facts
Recall mathematics terms & definitions
Recall formulas and computational
procedures
Perform Procedures
Use numbers to count, order, denote
Do computational procedures or
algorithms
Follow procedures/instructions
Solve equations/formulas/routine word
problems
Conjecture/ Generalize/ Prove
Determine the truth of a mathematical
pattern or proposition
Write formal or informal proofs
Recognize, generate or create patterns
Find a mathematical rule to generate a
pattern or number sequence
Make and investigate mathematical
conjectures
Identify faulty arguments or
misrepresentations of data
Reason inductively or deductively
Organize or display data
Read or produce graphs and tables
Execute geometric constructions
Demonstrate Understanding
of Mathematical Ideas
Communicate mathematical ideas
Use representations to model
mathematical ideas
Explain findings and results from data
analysis strategies
Develop/explain relationships between
concepts
Solve Non-routine Problems/
Make Connections
Apply and adapt a variety of appropriate
strategies to solve non-routine problems
Apply mathematics in contexts outside of
mathematics
Analyze data, recognize patterns
Synthesize content and ideas from several
sources
Show or explain relationships between
© 2005 CCSSO ALL RIGHTS RESERVED.
models, diagrams, and/or other
representations
None
10% or<
11-25%
26-50%
51-75%
>75%
25 Watch the teacher demonstrate how to do a procedure or
€

‚
ƒ


solve a problem.
26 Read about mathematics in books, magazines, or articles (not
textbooks).
€

‚
ƒ


€
€


‚
‚
ƒ
ƒ




€

‚
ƒ


€

‚
ƒ


€

‚
ƒ


€

‚
ƒ


€
€


‚
‚
ƒ
ƒ




€

‚
ƒ


€

‚
ƒ


How much of the total mathematics instructional time do
students in the target class:
27 Take notes from lectures or the textbook.
28 Complete computational exercises or procedures from a
textbook or a worksheet.
29 Present or demonstrates solutions to a math problem to the
whole class.
30 Use manipulatives (for example, geometric shapes or algebraic
tiles), measurement instruments (for example, rulers or
protractors), and data collection devices (for example, surveys
or probes).
31 Work individually on mathematics exercises, problems,
32
33
34
35
36
investigations, or tasks.
Work in pairs or small groups on math exercises, problems,
investigations, or tasks.
Do a mathematics activity with the class outside the
classroom.
Use computers, calculators, or other technology to learn
mathematics.
Maintain and reflect on a mathematics portfolio of their own
work.
Take a quiz or test. © 2005 CCSSO ALL RIGHTS RESERVED.
The Enacted Curriculum
© 2005 CCSSO ALL RIGHTS RESERVED.
© 2005 CCSSO ALL RIGHTS RESERVED.
Time on Topic
Middle School Science
Expectation
Nature of Science
Memorize
Facts/
Definitions/
Formulas
€  ‚ ƒ
101
Scientific Habits of Mind (e.g. reasoning,
evidence-based conclusions,
skepticism)
€  ‚ ƒ
€  ‚ ƒ
102
Scientific Method (e.g., observation,
experimentation, analysis, theory
development and reporting)
€  ‚ ƒ
€  ‚ ƒ
103
Issues of Diversity, Culture, Ethnicity,
Race, Gender in Science
€  ‚ ƒ
€  ‚ ƒ
104
History of Scientific Innovations
€  ‚ ƒ
€  ‚ ƒ
105
Ethical Issues in Science
€  ‚ ƒ
<none>
1
2
Science and Technology
€  ‚ ƒ
201
Design a Solution or Product, Implement
a Design
€  ‚ ƒ
€  ‚ ƒ
202
Relationship Between Scientific Inquiry
& Technological Design
€  ‚ ƒ
€  ‚ ƒ
203
Technological Benefits, Trade-offs &
© 2005 CCSSO ALL RIGHTS RESERVED.
Consequences
€  ‚ ƒ
<none>
© 2005 CCSSO ALL RIGHTS RESERVED.
None
10% or <
11-25%
26-50%
51-75%
> 75%
When students in the target class are engaged in
laboratory activities, investigations, or experiments as part
of science instruction, how much time do they:
37 Make educated guesses, predictions, or hypotheses.
€

‚
ƒ


38 Follow step-by-step directions.
€

‚
ƒ


39 Use science equipment or measuring tools.
€

‚
ƒ


40 Collect data.
€

‚
ƒ


41 Change a variable in an experiment to test a hypothesis.
€

‚
ƒ


42 Organize and display information in tables or graphs.
€

‚
ƒ


43 Analyze and interpret science data.
€

‚
ƒ


44 Design their own investigation or experiment to solve a
€

‚
ƒ


€

‚
ƒ


scientific question.
45 Make observations/classifications.
© 2005 CCSSO ALL RIGHTS RESERVED.
CLASSROOM INSTRUCTIONAL PREPARATION
For items 81-97, please indicate how well prepared you are to:
Not Well
Prepared
Somewhat
Prepared
Well
Prepared
Very Well
Prepared
81
Teach mathematics at your assigned level
€

‚
ƒ
82
Use/manage cooperative learning groups in
mathematics
€

‚
ƒ
83
Integrate mathematics with other subjects
€

‚
ƒ
84
Provide mathematics instruction that meets
mathematics standards (district, state, or
national)
€

‚
ƒ
85
Use a variety of assessment strategies
(including objective and open-ended formats)
€

‚
ƒ
86
Teach problem solving strategies
€

‚
ƒ
87
Teach mathematics with the use of
manipulative materials, such as counting
blocks, geometric shapes, and so on
€

‚
ƒ
€

‚
ƒ
€

‚
ƒ
88
Select and/or adapt instructional materials to
implement your written curriculum
89
Teach students with physical disabilities
90
Help students document and evaluate their
own mathematics work
€

‚
ƒ
91
Teach classes for students with diverse
abilities
€

‚
ƒ
92
Teach mathematics to students from a variety
of cultural backgrounds
€

‚
ƒ
93
Teach mathematics to students who have
limited English proficiency
€

‚
ƒ
94
Teach students who have a learning disability
which impacts mathematics learning
€

‚
ƒ
95
Encourage participation of females in
mathematics
€

‚
ƒ
© 2005 CCSSO ALL RIGHTS RESERVED.
© 2005 CCSSO ALL RIGHTS RESERVED.
Descriptive results can be reported using a
variety of graphic displays
© 2005 CCSSO ALL RIGHTS RESERVED.
SEC Collaborative Members
Current States participating:
•Alabama
•Idaho
•Indiana
•Iowa
•Kansas
•Maine
•Michigan
•Mississippi
•Montana
•New Hampshire
•North Carolina
•Ohio
•Oklahoma
•Oregon
•Wisconsin
•Districts
•Miami-Dade
•Milwaukee
•Winston-Salem
© 2005 CCSSO ALL RIGHTS RESERVED.
SEC Collaborative Services
 Surveys, Data Reports
 Alignment Analysis – content coding
 Planning, Training-- workshops
 Leader materials, e.g. Orientation
Education Leaders
Survey Administrators
Explaining SEC
 Using Data on Enacted Curriculum (Model)
© 2005 CCSSO ALL RIGHTS RESERVED.
© 2005 CCSSO ALL RIGHTS RESERVED.
Data on Enacted Curriculum PROJECT
DESIGN
5 Urban Districts—40 Middle Schools
Teacher & School
Standards
Data
Assessment
Reports
Teacher Data
Measure Change
Curriculum
Improved
Student
Achievement
Science/Math
Initiatives
Professional
Development
Assist Schools/
Professional
Development
© 2005 CCSSO ALL RIGHTS RESERVED.
Assist Schools/
Professional
Development
SEC Collaborating Organizations
•Council of Chief State School Officers
www.SECsurvey.org
•Wisconsin Center for Education
Research www.SEConline.org
•Learning Point Associates/NCREL
www.SECsupport.org
•TERC Regional Alliance DEC Project
www.ra.terc.edu/DEC
•American Institutes for Research —
MSP RETA Study – PD Evaluation
© 2005 CCSSO ALL RIGHTS RESERVED.
Empirical Study of Effects of PD on Instruction
Longitudinal study design
Baseline
BaselineData:
Data:
•Teacher
•TeacherSurvey
Survey
Instruction
InstructionPractices,
Practices,
Content,
Content,Alignment
Alignment
Preparation; PD
Preparation; PD
• Visits to 5 sites
PD
Teachers,
•PDLog
Logw/w/
Teachers,
Interviews-Interviews-Measure
PD
Quality
•Measure
PD
Quality
•• Type
Content
Type - Content
• Duration
• Duration
• Collective
•• Active
Collective
learning
•• Coherence
Active learning
•• Content
Coherence
focus
• Content focus
• Visits to 5 sites
Spring 2003
Professional
ProfessionalDev.
Dev.
[5
MSP
Sites]
[5 MSP Sites]
A)
A) MSP
MSPActivities
Activities
(treatment)
(treatment)
B) Other PD in sites
B) Other PD in sites
(control)
(control)
After
•AfterParticipation:
Participation:
Teacher
•TeacherSurvey
Survey
•• Compare
Comparepractices,
Content,
Alignment
practices,
Content,
• Measure change
Alignment
• Measure change
Spring 2005
2002-03
© 2005 CCSSO ALL RIGHTS RESERVED.
2003-04
2004-05
© 2005 CCSSO ALL RIGHTS RESERVED.
The teacher survey tool…

Collects data on what content is taught and how
it is taught.
 Produces reports indicating the degree of
alignment between the taught (enacted)
curriculum and state assessments or standards.
 Collects data on teacher beliefs, readiness to
teach the content and readiness to instruct
special groups of students.
 Provides a rich source of information to support
teachers analysis of student learning challenges.
© 2005 CCSSO ALL RIGHTS RESERVED.
The SEC can help your
schools…





Align curriculum, instruction, and assessment
including vertical alignment
Improve instruction within and across grade
levels,
Determine professional development needs
Focus teacher planning and development on
instructional practice and its effects on student
performance
Focus coaching and mentoring activity around
content and practice
© 2005 CCSSO ALL RIGHTS RESERVED.
The SEC can …

Help teachers monitor changes in practice as a
result of implementing new curricula.
 Help teachers and leaders determine areas of need
when planning for professional development.
 Help teachers see areas for deepening content
expertise and knowledge of how students learn
that content.
 Help teachers identify gaps/redundancies in
curriculum –weak materials to support instruction
© 2005 CCSSO ALL RIGHTS RESERVED.
SEC Survey Administration
Procedures for completing on-line surveys
Procedures for completing paper surveys
© 2005 CCSSO ALL RIGHTS RESERVED.
Online Survey Administration

Approximately 60-90 minutes to compete
 May be completed in multiple sittings
 Data is saved as each section is completed
© 2005 CCSSO ALL RIGHTS RESERVED.
SEC Online Registration
www.seconline.org
From the
Home Page
click on the
Registrar
button.
© 2005 CCSSO ALL RIGHTS RESERVED.
SEC Online Registration
Registrar Area
If not registered,
select your group
from the dropdown menu, then
click on Register
button.
If registered,
enter your
username and
Group ID#, then
click on Login
button.
.
Just Visiting? Click on Guest
button!
© 2005 CCSSO ALL RIGHTS RESERVED
Your data is saved each time you click on a Submit button.
© 2005 CCSSO ALL RIGHTS RESERVED.
You may log-off at any time by simply closing your browser.
© 2005 CCSSO ALL RIGHTS RESERVED.
Your progress will be noted on survey menu when you return.
© 2005 CCSSO ALL RIGHTS RESERVED.
SEC Contacts

Rolf Blank, CCSSO, SEC Collaborative
– RolfB@ccsso.org
– www.SECsurvey.org
Andrew Porter, Vanderbilt University
andyp@vanderbilt.edu
WCER, SEC Online Survey and Reports
www.SEConline.org
© 2005 CCSSO ALL RIGHTS RESERVED.