Students Learning in the IQWST & LP
Improving middle school students’
understanding of core science
ideas using coherent curriculum
Dec. 11, 2012, Sung-Youn Choi
Post-doc of CREATE 4 STEM Institute, Michigan State University
What will we do today?
• Examine why we should redesign curriculum materials
• Introduce the IQWST curriculum design; the next generation of
middle school science materials
• Introduce the test design; aligned with a LP approach
• Discuss key design principles and the CCD process
• Share some of our findings; using IRT analysis
• Discuss about improving middle school students’ understanding of
core science ideas using coherent curriculum
– Investigating and Questioning our World through Science and Technology
(IQWST)
– Learning Progression (LP)
– Construct-Centered Design Process (CCD)
– Item Response theory (IRT)
IQWST & LP
2
Why redesign curriculum?
• Inadequate Science Materials
–
–
–
–
–
Cover many topics at a superficial level
Focus on technical vocabulary
Fail to consider students’ prior knowledge
Lack coherent explanations of real-world phenomena
Provide students with few opportunities to develop explanations of
phenomena
– Materials lack coherence and as a result don’t focus on building
understanding over time
• Students need to be prepare for the world in which they live now and
will live as adults
– Deeper understanding of the core ideas becomes instrumental for
competitiveness in a global economy, as well as for the personal well being of
each citizen
– Students need deeper knowledge in order to use understanding to solve
problems
IQWST & LP
3
Design the Next Generation of Science Learning
Environments
• Solution
• The Investigating and Questioning our World through Science and
Technology (IQWST)
– Utilizing a coherent approach for building curriculum materials across the
school years
– Using learning goals as a focus
– Supporting students in developing understandings of the core ideas of science
and scientific practices
–
–
–
–
Applying what we know about student learning
Promoting literacy practices
Engaging students in complex tasks
Using a project-based approach
IQWST & LP
4
IQWST
Design Principles
IQWST & LP
5
Key Design Principles
• Curriculum Coherence
A. Inter-unit coherence
• Build links across units among the grades or content areas
• Provide opportunities to develop, reinforce, and use their understanding
during their three-year IQWST experiences
B. Intra-unit coherence
• Use driving-questions as support to link ideas together
• Create alignment by iteratively aligning learning goals with tasks
C. Learning-goals coherence
• Use big ideas
• Unpack standards from a learning perspective
• Create learning performances as a way to specify knowledge in use
IQWST & LP
6
A. Inter-unit Coherence: Structure of IQWST
• The IQWST is built on scientific core ideas.
Units
6th grade
7th grade
8th grade
Physics
Chemistry
Biology
Earth Science
Light & its
interaction with
matter
Particle nature
of matter, phase
change
Survival: From
organisms to
ecosystems
Water & rock
cycles
Nature of
chemical
reactions
Biological
organization &
development:
Cells to systems
Surface &
atmospheric
processes
behind weather
and climate
Heredity,
Natural selection
Large-scale
geological
processes on
earth & other
planets
Conservation &
transformation
of energy
Laws of motion
Photosynthesis
and respiration
(carbon cycling)
IQWST & LP
7
A. Inter-unit Coherence: Scope and Sequence
• The core ideas are coordinated across the grades and the contents.
6th grade
7th grade
8th grade
Physics
Chemistry
Biology
Earth Science
Light & its
interaction with
matter
Particle nature
of matter, phase
change
Survival: From
organisms to
ecosystems
Water & rock
cycles
Nature of
chemical
reactions
Biological
organization &
development:
Cells to systems
Surface &
atmospheric
processes
behind weather
and climate
Heredity,
Natural selection
Large-scale
geological
processes on
earth & other
planets
Conservation &
transformation
of energy
Laws of motion
Photosynthesis
and respiration
(carbon cycling)
IQWST & LP
8
B. Intra-Unit Coherence: Driving Questions
• Driving Question:
–
–
–
–
–
Links content in a context
Links tasks, investigations, and assessments to learning goals
Ties the unit together
Builds intra-unit coherence
Uses rich and open-ended questions with everyday language
IQWST & LP
9
C. Learning Goals Coherence
• Developing Learning Goals
• The construct-center design
approach (CCD)
– Step 1: Define the constructs
• Correcting ideas and
concepts to learn
• Identifying potential students
difficulties and alternative
ideas
– Step 2: Unpack the ideas and
concepts
• Creating a set of claims
• Specifying evidences
– Step 3: Design tasks and
materials
– Step 4: Review Products
IQWST & LP
10
IQWST
Example lesson
IQWST & LP
11
IQWST Smell Unit
• 8-week, project-based unit for 6th grade students
• Driving Question: How can I Smell Things from a Distance?
• Three Learning Sets, 15 Lessons
– Learning Set 1: Students construct models to help them understand the
particle nature of matter while focusing on the behavior of gases
– Learning Set 2: How models help to explain why different materials have
different properties
– Learning Set 3: Using the particle nature of matter model to explain phase
changes
 Ideas from all three learning sets are brought together through a culminating
final task in which students use their knowledge of the particle nature of
matter.
IQWST & LP
12
Driving question for Unit
Scientific Core Idea
Particle Nature of Matter
Properties of Matter
Phase Changes
IQWST & LP
13
Example (1) activity
• Learning Goal: Lesson 13
– Students will explain phase changes from gas to liquid to solid and from solid
to liquid gas at the molecular level.
• Scientific Principle
– Evaporation
IQWST & LP
14
Example (2) reading
•
Scientific Principle
– Condensation
•
Alternative Conceptions
– Condensation on the outside of a container is water that seeped through the
container itself (or sweated through the walls of the container).
– The coldness comes through the container and produces water.
– Condensation is when air turns into a liquid.
Driving question
Everyday
experiences
Prior knowledge
Question & writing
opportunity
Set purpose for
reading
IQWST & LP
15
Example test items
IQWST & LP
16
Example item (#1) PC02
• Contents area
– Phase change, Structure
• Claim
– Students can explain how condensation occurs (without a particulate model).
• Unpacking/evidence
– Level 1. Matter can exist as a solid, liquid or gas. Generally, the same pure
substance can exist in all three states. The state of matter depends on the
temperature.
IQWST & LP
17
Example item (#1) PC02
• Contents area
– Phase change, Structure
• Claim
PC02
Gas  Liquid
d = 0.958 (hard)
18.4%  29.0%
N1=1092, N2=399
– Students can explain how condensation occurs (without a particulate model).
• Unpacking/evidence
– Level 1. Matter can exist as a solid, liquid or gas. Generally, the same pure
substance can exist in all three states. The state of matter depends on the
temperature.
IQWST & LP
18
Example item (#2) PC18
• Contents area
– Phase change, Conservation
• Claim
– Students can explain the mass of matter are conserved in transforming from
solid to liquid or solid to gas (without a particulate model).
• Unpacking/evidence
– Level 1. Matter is not created or destroyed when substances change form or
change into other substances (e.g., crushing, dissolving, phase change,
chemical reaction).
IQWST & LP
19
Example item (#2) PC18
• Contents area
– Phase change, Conservation
• Claim
PC18
Solid  Gas
d = - 0.253
23.6%  34.9%
N1=368, N2=175
– Students can explain the mass of matter are conserved in transforming from
solid to liquid or solid to gas (without a particulate model).
• Unpacking/evidence
– Level 1. Matter is not created or destroyed when substances change form or
change into other substances (e.g., crushing, dissolving, phase change,
chemical reaction).
IQWST & LP
20
Example item (#3) PC19
• Contents area
– Phase change, Conservation
• Claim
PC19
Solid  Liquid
d = - 0.780 (easy)
35.8%  41.9%
N1= 358, N2=201
– Students can explain how condensation occurs (without a particulate model).
• Unpacking/evidence
– Level 1. Matter is not created or destroyed when substances change form or
change into other substances (e.g., crushing, dissolving, phase change,
chemical reaction).
IQWST & LP
21
Example item PC02, PC18, PC19
PC02
PC18
PC19
Phenomena
Phase Change
Contents area
Structure
Conservation
Conservation
Concept
Gas  Liquid
Solid  Gas
Solid  Liquid
Item difficulty
d = 0.958 (hard)
d = - 0.253
d = - 0.780 (easy)
% of correct answer
(pretest) (posttest)
18.4%  29.0%
23.6%  34.9%
35.8%  41.9%
# of students
(pretest), (posttest)
N1=1092, N2=399
N1=368, N2=175
N1= 358, N2=201
• Can we give same credit to each item?
• Do students need to take every item?
IQWST & LP
22
Item Response Theory
: IRT analysis vs. CTT (classical test theory)
Creating test forms
IQWST & LP
23
A. Scoring system
• CTT (raw score)
– O X O O O = 4 (total score)
– How many correct answers did you get?
– The original result obtained by a student on a test. (% of correct answer,
mean score, total score, or etc.)
• IRT (latent ability score/logit)
– ? ? ? ? ? = 1.3
– What is the probability of a correct response to an item?
– Using the IRT model (latent trait models), we can estimate overall latent ability
score/ logit.
– In three parameter logistic model (3PL)
• difficulty
• discrimination
• pseudo-guessing
IQWST & LP
24
Example: PC02 (ice cup, condense)
Observed
response
Latent trait
model
•
•
Item difficulty = 0.958
The item difficulty is determined by the latent trait logit of a person who has 50% of right
answer. In this graph, 0.5 probability meets the trait graph at 0.958.
The MNSQ fit is around 1. It has good fitness between observed responses and latent trait
model.
IQWST & LP
25
Example: PC18 (solid  gas)
Observed
response
Latent trait
model
•
•
Item difficulty = - 0.253
The item difficulty is determined by the latent trait logit of a person who has 50% of right
answer. In this graph, 0.5 probability meets the trait graph at -0.253.
MNSQ fit is around 1. It has good fitness between observed responses and latent trait
model.
IQWST & LP
26
B. Test form
• CTT
– (missing data) O X O O O _ _ = 4, or deleted
– The raw score, which is the number of items correct, does not present a broad
picture of test performance because it can be interpreted only in terms of a
particular set of test questions.
• IRT
– (missing data) ? ? ? ? ? _ _ = 1.7
– Linking item
– Latent student ability score allows direct comparisons of student performance
between specific sets of test items.
– IRT generally brings greater flexibility and provides more sophisticated
information.
IQWST & LP
27
B. Test form (LP test design)
• Learning Progressions
– LPs are research-based descriptions of how students build their knowledge
within and across disciplines (topics) over a broad span of time
– Explaining many important phenomena requires incorporating ideas from
many topic areas
– Focusing on the growth of sets of ideas instead of individual topic
Phase
Change
Chemical
Reaction
Others
Conservation
Topic areas
Structure of matter
Interactions
Phenomena
IQWST & LP
28
B. Test form (LP test design)
• Learning Progressions
– LPs are research-based descriptions of how students build their knowledge
within and across disciplines (topics) over a broad span of time
– Explaining many important phenomena requires incorporating ideas from
many topic areas
– Focusing on the growth of sets of ideas instead of individual topic
Phase
Change
Chemical
Reaction
Level 6
Others
Level 5
Conservation
Level 4
Topic areasLevel 3
Structure of matter
Level 2
Interactions
Level 1
Phenomena
IQWST & LP
29
B. Test form (LP test design)
• Test form design
– Total 62 items; 52 multiple-choice items and 10 open-ended or short-answer
items
• A-version Form (1A, 2A, 3A 4A)
– Each containing 14 items
– For 6th, 7th grade
• B-version From (1B, 2B, 3B, 4B)
– Each containing 15 items
– For 8th grade
– Three items were used to adjust the difficulty between A- and B-version
• Linking items
– Each test form contained three pairs of linking items
– Total six linking items were contained
– Linking items cover a range of contents areas along levels 1 to 3 on the LP
IQWST & LP
30
Research Questions &
Findings
IQWST & LP
31
Research Questions
• Do the chemistry IQWST units, which link content within and across
units, build understanding of targeted chemistry ideas?
– Comparing students’ performances and learning gains in Phase Change and
Chemical Reaction
– Comparing Students performances among Topic Areas
• Do student apply what they learned from IQWST Units to solve new
problems?
– Comparing between IQWST items and non-IQWST items in Chemical
Reaction
IQWST & LP
32
Data Collection: pre- and post-test
• Participants
– Total 1,519 students
• 481 students in 6th grade, 519 in 7th grade, 519 in 8th grade
– 7 teachers
– 6 schools
• Two of the schools represent middle- to upper-middle class
socioeconomic status
• Four of the schools were from neighborhoods that are lower-SES, with a
majority of students eligible for free and reduced lunch.
• Research period
– Schools used IQWST materials during national field trials (2007~2010)
– Pretest (fall, 2010), Posttest (Spring, 2011)
IQWST & LP
33
Test Instrument
• Hypothetical LP based test
– We developed test instruments based on a Learning Progression (LP) for the
transformation of matter
– The assessment instruments measured how students applied ideas within and
across topics to explain phenomena involving transformations of matter.
• Test forms
– total 62 items, 8 different forms, each form contained 3 pairs of linking items
for a total of 6 linking items.
• Validation of instrument
– Piloted with students from grade 6th -15th (N=792).
– Content validity: semi-structured interviews with a subset of students
supplemented the pilot test data.
– The LP research team is contributing to validate instruments. (Namsoo Shin,
Shawn Stevens)
IQWST & LP
34
Framework (1)
• Phenomenon
– Phase Change: 6th grade IQWST unit, 21 test items (20 with IQWST)
– Chemical Reaction: 7th grade IQWST unit, 16 test items (11 with IQWST)
Phase
Change
Chemical
Reaction
Others
Conservation
Topic areas
Structure of matter
Interactions
Phenomena
IQWST & LP
35
Framework (2)
• Topic areas
– Conservation
– Structure of matter; matter and materials, properties and periodicity, atomic
structure
– Interactions; kinetic energy, forces, energy, processes
Phase
Change
Chemical
Reaction
Others
Conservation
Topic areas
Structure of matter
Interactions
Phenomena
IQWST & LP
36
Item analysis
•
The purpose of item analysis is to provide how well assessments work, and how
well individual items on assessments work.
• Item Response Theory (IRT, Wu, et al., 2007)
– IRT allows us to evaluate students’ ability and to describe how well items on
the test are performing. In IRT, ability and item parameters are both estimated
based on students’’ response patterns on the test.
– Item parameters (difficulty, discrimination, pseudo-guessing)were used to
determine whether an item displayed sound psychometric properties
• General item analysis criteria
•
•
•
•
•
Items difficulty (estimate)
MNSQ fit (the weighted means square) ~ 1
T (pseudo-guessing) <3
Item correlation > 0.3
% of correct answer (total)
IQWST & LP
37
Analyzing data set
• Phase Change (20 IQWST items + 1 non-IQWST item)
– Conservation (4)
– Structure of matter (8+ 5 overlapped with interactions + 1 non-IQWST)
– Interactions (3+ 5 overlapped with structure)
• Chemical Reaction (11 IQWST items + 5 non-IQWST items)
– Conservation (6+ 2 overlapped+ 1 non-IQWST)
– Structure of matter (2+ 2 overlapped+ 5 non-IQWST)
– Interactions (2+ 1 non-IQWST)
IQWST & LP
38
Result from
Phase Change
IQWST & LP
39
Item analysis (PC)
• Accepted items range
•
•
•
•
- 0.988 < d <1.174
MNSQ fit <1.17
- 4.0 < T < 3.0
0.35 < r
Problematic item
IQWST & LP
40
Item difficulties & Latent distribution (PC)
conser
vation
Students distribution
conser
vation
struct
ure
Intera
ctions
struct
ure
Intera
ctions
Item diff distribution
IQWST & LP
41
Scoring
• Students latent ability logit
– The partial credit, multi dimensional model
– The expected a posteriori (EAP) parameter was standardized to
present student ability score
• Students ability score
– To facilitate the interpretation of their ability scores assigned to
students
– Normalization as t2 so that the mean and S.D of scores was
500 and 100
IQWST & LP
42
R1. Do students develop understanding of Phase
Change?
• Phase Change; Topic- 1. Conservation
7th grade
Pretest
Posttest
8th grade
Pretest
Posttest
Students’ ability score
6th grade
Pretest
Posttest
Distribution of Students
IQWST & LP
43
R1. Do students develop understanding of Phase
Change?
• Phase Change; Topic- 1. Conservation
% of variance
= 0.46 **
% of variance
= 0.45 **
% of variance
= 0.22
IQWST curriculum
IQWST & LP
44
R1. Do students develop understanding of Phase
Change?
• Phase Change; Topic- 2. Structure of Matter
7th grade
Pretest
Posttest
8th grade
Pretest
Posttest
Students’ ability score
6th grade
Pretest
Posttest
Distribution of Students
IQWST & LP
45
R1. Do students develop understanding of Phase
Change?
• Phase Change; Topic- 2. Structure of Matter
% of variance
= 0.63 **
% of variance
= 0.45 **
% of variance
= 0.28 **
IQWST & LP
46
R1. Do students develop understanding of Phase
Change?
• Phase Change; Topic- 3. Interactions
7th grade
Pretest
Posttest
8th grade
Pretest
Posttest
Students’ ability score
6th grade
Pretest
Posttest
Distribution of Students
IQWST & LP
47
R1. Do students develop understanding of Phase
Change?
• Phase Change; Topic- 3. Interactions
% of variance
= 0.48 **
% of variance
= 0.49 **
% of variance
= 0.37 **
IQWST & LP
48
Result from
Chemical Reaction
IQWST & LP
49
Item difficulties & Latent distribution: (2) CR
conser
vation
struct
ure
Intera
ctions
conser
vation
Students distribution
struct
ure
Intera
ctions
Item diff distribution
IQWST & LP
R2. Do Students apply their learning to solve new
problems?
• IQWST aligned items
• Non-IQWST
IQWST & LP
51
Students’ development for IQWST items and nonIQWST
% of variance
= 0.25 **
% of variance
= 0.12 **
IQWST & LP
52
R1. Do students develop understanding of Chemical
Reaction? (with IQWST items)
• D1. Conservation
% of variance
= 0.62 **
% of variance
= 0.40 **
% of variance
= - 0.08
IQWST curriculum
IQWST & LP
53
R1. Do students develop understanding of Chemical
Reaction? (with IQWST items)
• D2. Structure of Matter
% of variance
= 077 **
% of variance
= 0.45 **
% of variance
= - 0.05
IQWST & LP
54
R1. Do students develop understanding of Chemical
Reaction? (with IQWST items)
• D3. Interactions
% of variance
= 0.29 **
% of variance
= 0.41 **
% of variance
= 0.10
IQWST & LP
55
Interpretation of the finding
• Demonstrate that students’ latent ability changes across grade level,
supporting the hypothesis that coherent curriculum materials can
support students in developing integrated understanding.
• Provide support that curriculum coherence may increase student
achievement and build integrated understanding
• Suggest that curriculum coherence can help students develop
further understanding then when ideas are not linked
IQWST & LP
56
Acknowledgments
• Thanks to the IQWST and the LP research team
• Thanks to the participating schools, teachers, and students
• Email me; choisun9@msu.edu
IQWST & LP
57