Partnership to Improve Student Achievement in
Physical Science: Integrating STEM Approaches
(PISA2)
NSTA
April 5, 2014
A Mathematics and Science Partnership
Copyright © 2013 Stevens Institute of
Technology
PISA2
Leadership Team
o Arthur Camins, CIESE Director, Project
Director, Co-PI
o Kathy Kennedy, Professional
Development Specialist
o Tom Smith, Professional Development
Specialist
CIESE develops and supports
effective innovative curricula
and professional
development and conducts
research in order to inspire,
catalyze and strengthen
scientific, technological,
engineering and mathematics
literacy.
Stevens Institute of Technology The
Innovation University®, is a premier,
private research university situated
in Hoboken, N.J. overlooking the
Manhattan skyline. Founded in 1870,
technological innovation has been
the hallmark and legacy of Stevens’
education and research programs for
more than 140 years.
Mathematics and Science Targeted Partnerships
Develop new innovations in K-12 STEM education, linked
to a strong educational research agenda.
Critical Features
• Partnership Driven
• Teacher Quality, Quantity and Diversity
• Challenging Courses and Curriculum
• Evidence-based Design and Outcomes
• Institutional Change
Context: Constraining Factors
Teachers’ success in implementing new strategies is
mediated by school and district-based constraints.
o Ongoing Pressure to focus on language arts and
math;
o Current evaluation pressures constrain risk taking;
o Teachers are Isolated. While teachers may be
motivated to try new approaches, science and
engineering may not be their district’s priority;
o Limited by textbook-based curricula;
o Administrative changes create instability.
Today’s Discussion Questions
• How can we increase the likelihood of knowledge
transfer from professional development exemplars
to teacher practice across diverse curricula?
• Which professional development and one-one-one
coaching practices are most effective to target
individual teacher needs and support improved
instruction?
• How should we calibrate evidence of change in
teachers’ content knowledge, PCK and practices
given their initial state and our external role?
PISA2 School
District
Partners
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
Bayonne Board of Education
Brick Township Public Schools
Camden City Public Schools
Cherry Hill Public Schools
Hoboken Public Schools
Howell Township Public Schools
Jersey City Public Schools
Lakewood School District
Margate City School District
Morris School District
Mustard Seed School (private)
Princeton Regional Schools
Red Bank Borough Public Schools
Toms River Regional Schools
West New York School District
IHE & Other Partners




Stevens Institute of Technology
Columbia University/Teachers College
National Science Resources Center
Education Development Center
(Evaluator)
 St. Peter’s University
PISA2 Goals






To enhance teachers’ content knowledge in science & engineering (S&E) and
cultivate positive attitudes & beliefs towards teaching S&E
To increase students’ content knowledge and experiences in S&E
To promote students’ 21st century skills (Critical Thinking and Creativity)
To institutionalize new graduate programs in STEM education and impact
undergraduate teaching & learning
To increase the number of teachers with elementary endorsement in science
To build leadership and capacity in partner school districts
TEACHER LEARNING
(Content/Pedagogy)
STUDENT ACHIEVEMENT
(Science & Engineering)
21st CENTURY SKILL
ACQUISITION
PISA2 Improvement
Model
Graduate
Courses/ PD
Institutes
Science and
Engineering
Practices
Core Ideas
Full-Day PD
Workshops
Increase
students’
21st Century
skills
Increase
students’
science
content
knowledge
PCK
Classroom
Support Visits
Increase #
teachers with
science
endorsement
PISA2 Participant Enrollment
Year
1
2
3
4
5
6 (projected)
Scholar
Participants
41
Leader
Participants
28
22
20
16
47
102 (69 new)
120 (70 new)
Total
41
44
69
89
70
Teacher Scholars Program
Five Graduate Courses
•
•
•
•
•
Fundamental Principles of Physical
Science
Fundamental Principles of Earth
Science
Energy Production & Consumption
Understanding Global Climate Change
Engineering Solutions to the
Challenges of Energy & Global Climate
Change
Courses are designed to builds PISA2 teachers’ capacity and
confidence to master and teach more complex concepts.
Teacher Leader Program
Physical Science
Earth and Scace Science
ESS2: Earth Systems
(Weather)
Grades 3-5
PS2: Motion
PS1: Matter
and Stability:
and Its
Forces and PS3: Energy
Interactions
Interactions
Grades 6-8
CrossCutting
Concept
Science and
Engineering
Practices Foci:
PD
Dimensions
ESS2: Earth Systems
(Weather and Climate
Energy and Matter: Flows, Cycles, and Conservation
Models (Making thinking visible through discourse and writing)
Constructing Explanations and Designing Solutions
(Linking Explanation, Argumentation and Evidence)
K-12
Framework
for Science
Education
(Interwoven
Dimensions)
Teacher
Content
Knowledge
PCK
Science TopicExplanation
Linked
and Evidence Engineering
Dedesign
Professional Development Strategy:
Focus on Two High Leverage Practices + EDP
What Students Do:
Ask questions and define problems
Develop and use models
Plan and carry out investigations
Analyze and interpret data
Use mathematics and computational thinking
Construct explanations and design solutions
Engage in argument from evidence
Obtain, evaluate, and communicate information
PD Strategy: Focus on Two High Leverage Practices + EDP
Making Models
What Students Do:
What Teachers Do:
Make sketches, concept maps, graphs, and/or
Provide a context or questions for
charts or use words to illustrate concepts and
students to represent and articulate their
represent their models of the natural world or a models.
design solution.
Create and use models to predict an outcome;
Provide opportunities for students to
explain their ideas; express their understandings represent and articulate their models.
of the natural world (Science).
Create and use a model to explain their design
Engage students in productive discourse
and why the design will be successful (EDP).
to clarify their models; to explain and
defend their models.
Refine models based on evidence (Science).
Guide students to use their evidence;
facilitate discussion through questioning.
Test and revise models based on test results
(EDP).
Provide productive feedback to students
to clarify their models.
PD Strategy: Focus on High Leverage Practices + EDP
Claims, Evidence and Argumentation
What Students Do:
What Teachers Do:
Make an accurate and relevant claim.
Provide opportunities for students to
construct explanations and design
solutions.
Provide appropriate and sufficient evidence to
support claim.
Engage students to clarify and strengthen
their explanations by connecting evidence
to the model (Science).
Engage students to review their solution,
identify points of and reasons for failure
during the test; guide how to improve the
design (EDP).
Provide productive feedback to students
to clarify and strengthen their
explanations.
Provide reasoning that connects evidence to the
claim.
Include appropriate and sufficient scientific
principles to explain why the evidence supports
the claim.
Recognize alternative explanations and provide
appropriate and sufficient counter evidence and
reasoning when making rebuttals.
Components of the Coach Site Visits
Coaching goals for each visit
Planning
Observation
Co-Teaching/ model lesson
Instructional Support
Follow up or reflection meeting
CIESE PISA2 staff model and call attention
to Practices for Selected Science Topics…..
… that teachers try out in their classrooms
Teachers elicit students’
initial models about heat
transfer (e.g. metals are
inherently colder than
plastic). Students
investigate, collect
evidence and make
revised claims.
Teacher Directed Models
Activity: The teacher led an activity with play dough in
which the students created models of planets to
compare their sizes.
Teacher: Take the play dough out of the bag and mush it
into one big ball. Now we have to make our play dough
into like a bread loaf. You want to roll it out so the play
dough goes from 0-10 (teacher handed out sheets of
paper with 10 lines marked 0-10). Projector: Step 2:
Combine 6 parts together and put them in the Jupiter
section – put 3 parts into the Saturn section.
Modifying Lessons with Coach
• Teacher Modified GEMS lesson on models of Earth-SunMoon systems with the help of coach
• Students analyzed data from table to determine what
models were supported
• Modification – students explained why some models did
not match the data.
• Teacher needed to devote more time to activity
Evaluation Questions
• E1- Does a project which uses scientific inquiry and the
engineering design process (EDP) contribute to an increase in
teachers' content knowledge of science and engineering?
• E2- Does a project which uses scientific inquiry and EDP
contribute to an increase in students' content knowledge of
science and engineering?
• E3- Do students improve their 21st Century Skills as a result of
the program?
• E4- To what extent does participating in the program impact use
of teaching practices and and the amount of time spent on
science instruction?
• E5- To what extent did the program promote an increase in
collaboration and shared vision among partners? Partners
include University Faculty, District and Schools, Administrators,
Teachers, Students, and Parents.
PISA2 Outcome Measures
•
•
•
•
Teachers
Science and Engineering
Design Process (PrePost) Content
Practices and Time
Survey (Pre-Post)
Practices Observations
Interviews (including
limiting factors)
Students
• Science and Engineering
Design Process (PrePost) Content
• Critical Thinking and
Creativity in Science and
Engineering
Evidence from end-of-course assessments shows
increased content knowledge for Cohort 2 teachers in
each course.
Change
Pre-test
Post-test
(in percentage points)
Physical Science, selected response (Course 1)
45%
(SD=11)
69%
(SD=12)
24***
Earth Science , selected response (Course 2)
63%
(SD=14)
78%
(SD=7)
14***
40.7%
(SD=147.2)
62.1%
(SD=17)
21.4***
29.1%
(SD=13.4)
52.2%
(SD=16.7)
23***
Understanding Global Climate Change (Course 3)
Energy Production and Consumption (Course 4)
***p<.001
Student Score Increases on PISA2 Content Test
% Correct
(Pre-Test)
% Correct
(Post-Test)
Change
Elementary
student overall
performance
(N=143)
27.6%
(SD=8.4)
47.8%
(SD=14.3)
20.1***
Middle school
student overall
performance
(N=187)
44.1%
(SD=14.1)
49.9%
(SD=14.5)
5.9***
***p<.001
Topics tested are: Properties and Changes in Matter, Forces and Motion
(including mechanical energy), Weather and Climate, and EDP.
Composite Teacher Observation Measures
Traditional" Teacher Practices
Q5A: Teacher explains to whole class
Q5B: Sts read science/engineering text in class
Q5D: Sts work individually on assignment
Hands-on Science & Engineering
Q5E: Sts work in pairs/groups on assignment
Q5F: Sts do lab activity/investigation/experiment
Q5i: Sts collect and analyze data
Q5J: Sts use spreadsheets, databases, charts, etc. to summarize and display data
Explanation-Driven Science
Q5K: Sts maintain science/engineering notebook
Q5L: Sts explain understanding to teacher verbally
Q5M: Sts construct written explanation supported by evidence
Q5N: Sts engage in debate about how well claims are supported by evidence
Q5O: Sts reflect on strengths and weaknesses of their own thinking
Q5P: Sts reflect on strengths and weaknesses of others
Changes in Targeted Teaching Practices (After 1 Year)
Explanation-Driven Science
School
Level
Elementar
y (n=10)
Middle
(n=5)
Total
Time 1
Time 2
Change
2.13
2.80
0.67
2.83
2.37
-0.47
2.37
2.66
0.29
Traditional or Lecture & textbook
School
Level
Elementar
y (n=10)
Middle
(n=5)
Total
* p<.05
Time 1
Time 2
Change
3.37
2.83
-.53*
3.40
2.60
-0.80
3.38
2.76
-0.62*
Hands-on Science & Engineering
School
Level
Elementar
y (n=10)
Middle
(n=5)
Total
Time 1
Time 2
Change
2.50
2.68
0.18
2.60
2.60
0.00
2.53
2.65
0.12
Results from preliminary analysis
The analysis (repeated measures ANCOVA) on Cohort
2, Year 1 data did not find an association between
teachers’ content knowledge, time on topic nor
changes in teacher practice and student test gains.
The model includes: NJ ASK, grade as covariates; and
teacher content knowledge, changes in practices, and
time on topic as independent variables
Repeated Measures ANCOVA
Elementary School Model ( grades 4 & 5)
F*
Change in Scores
2.576
Middle School Model (Grade 5,6 & 7)
p val.
Partial Eta
Squared
.111
.021
Change in Score
Change in Scores * Grade
.118
.732
.001
Change in Scores * Grade 6
Change in Scores *
Teacher Content
Knowledge
Change in Scores * Time
on Topic
Change in Scores * NJ ASK
Math
Change in Scores * Change
in explanation driven
science practices
Change in Scores * Change
in hands-on science &
engineering practices
.620
.433
.005
Change in Scores * Grade 8
* Wilks’ Lambda
.078
.780
.001
3.143
.079
.025
4.743
.031
.038
1.666
.199
.014
p val.
F
Partial Eta
Squared
.@
.
.
.@
.
.
.@
.
.
Change in Scores * Teacher
.957
.330
.010
Content Knowledge
Change in Scores * Time on
1.395
.241
.015
Topic
Change in Scores * NJ ASK
.004
.947
.000
Math
Change in Scores * Change
in explanation driven science 1.454
.231
.015
practices
Change in Scores * Change
in hands-on science &
1.413
.238
.015
engineering practices
* Wilks’ Lambda; @ negative values or too near 0, reported as
missing
NOTE: The model treats grade differently because of the number of grades at each level
Institutes and PD
“The instructors are
outstanding. This workshop
has provided me more
options and resources to use
engineering design
problems in my
classroom. My kids love
EDPs. I have encouraged all
eligible educators in my
building to participate in
summer workshops.”
Impact on Struggling Learners
[In] our full-day inclusive classroom […] struggles with an
extraordinarily short attention span and has extreme
difficulty with retention. Her memory is among the
worst we've ever seen. It seems like the movie
Groundhog Day with her, we teach and reteach skills to
her, only to have her come in the next day like it is all
new material.
A traditional classroom would be a terrible fit for her.
She came to us with very low self-esteem after struggling
for years with her academics. Thanks to the ideology
Jaime and I share, coupled with our STEM classroom and
the training we received from PISA2, this student has
made considerable progress this year. She shines during
science experiments and engineering activities. While it
is a challenge for her to retain the step-by-step
methodology needed to solve long division problems,
she can brainstorm ways to problem-solve within a realworld context. Her strengths are her creativity and her
ingenuity and doing these projects affords her the
opportunities to excel when might would otherwise
flounder.
Teachers, most for the first time, are incorporating modeling
into instruction, but at a superficial level.
Students created an “amusement park ride” and were
asked to explain/defend where the three laws of
motion were present in their ride.
In the following dialogue, students presented and
explained their ride to their classmates, defended
where the targeted concept was present and answered
questions posed by the teacher and also other
students.
Dialogue:
would counteract the force of the ball being thrown.
T: Explain the game to me.
T: What would happen if the cans were knocked
S: We have 3 bottles in a pyramid form and you have down?
to knock them down with a ball.
S: Gravity would take its course and they would fall
down to the table, or the ground if the table wasn’t
T: Ok, so what’s the first law acting on it?
S: When you pick a ball and throw it the ball moves there.
T: Any questions for this group (teacher asked the
through the air.
T: What force is making the ball move through the rest of the class)?
T: How would we change the model for this game to
air?
make it harder?
S: The person throwing it.
T: If the ball is heavier would it move faster through S: We can change the form the ball by changing the
mass of the ball, and use like a baseball.
the air?
S: No, because it has more mass, so you would have T: What if we used a Ping-Pong ball?
S: It wouldn’t work because there is more mass in
to put more force on the ball to make it go faster.
the milk containers than in the ball.
S: The third law is in out project because when
you’re standing here and the bottles over there are T: What if I used a cannon to shoot the Ping-Pong
ball out? Would it knock down the bottles?
still the same shape and size.
S: Depends on the distance.
T: That’s the second law, right?
T: So you’re telling me distance matters?
S: Yes.
S: Yes.
T: Can you tell me about the third law?
S: We can put more weight in the cans; enough mass
Teachers, most for the first time, are incorporating the EDP into
instruction with explicit reference to “steps.”
Guiding Question: How can we use our knowledge of
the properties of play dough materials and the
Engineering Design Process to improve an existing play
dough process?
First, students investigate properties of flour, salt and
water and how they interact. Then they are given a
batch that is too grainy and sticky and asked to
improve it by altering proportions or the steps in the
process.
T: What’s the first stage?
S: Ask.
T: What are we asking?
What’s the challenge?
S: Play dough.
T: What’s the next stage?
S: Imagine.
What does that mean?
S: We imagine what it’s
going to look like.
T: Next step?
S: Plan.
T: What does that mean?
S: Draw a picture and
identify.
T: That’s right, we’re going
to plan what we’re going to
do and need. Then the most
fun part?
S: Create.
T: And then the last step?
S: Improve.
T: We’re going to create and
improve. We’re going to
work as chemical engineers.
More Dialogue
T: Ok, like if you think it’s too wet what could you do?
G: Add more salt.
T: What else?
S: Maybe squish it more and add more flour.
T: And [step] number 4, how will you test it?
B: we can play with it.
T: I guess that’s why they call it play dough, because we’re going to play with it. Do
you think the steps we’re going to follow are important?
Whole Class: Yes.
T: You want to be accurate with your steps and make sure you know what you’re
adding. You need to keep track of everything you put in so you know what you can do
to improve it.
T: Talk softly with your partner and figure out what you will do first, second, third etc,
but make sure you write it down. One of you can mix while the other keeps track of
what you’re putting in.
Questions to Observer
Did they reference prior investigation of the ingredients (flour, water, salt)?
She did not refer to prior activity to the students during the observation –
but in the post-lesson interview with Jaime she said she had covered the
materials previously. The only reference to prior knowledge she made to the
students was about states of matter.
Did they write and/or discuss and improvement plan?
They discussed it (in the brainstorm) but there was no moment of reporting
out their plan to the teacher or anyone.
Did you observe the actual improvement testing?
Yes. Some groups created whole new batches of playdough to test. Other
groups just kept adding ingredients to their "failed" batch. There was no
clarification about this from the teacher.
Response to Constraints
PISA2 Staff meet with teachers to plan and apply what
they have learned to their own curriculum, model
lessons, share teaching, coach and provide feedback.
Teachers are translating aspects of what they learned in
PISA2 in their classrooms.
• Applying improved science knowledge;
• Using the activities and resources from the courses,
institutes and PD workshops;
• Implementing pedagogical approaches to enrich
textbooks’ content (i.e. use of modeling and
explanation-driven science)
Teacher Learning Progression
Teachers have
limited
understanding
of practices and
EDP
Teachers gain
comfort with
and
understanding
of practices and
EDP during
institutes and
coaching
Teachers
struggle to
implement
practices and
EDP effectively
in the
classroom
In Class Support:
• Planning
• Observation/ Coteaching
• Reflection
Lessons Learned About Supporting Practice
• Model: Consistently use the targeted practices across all
courses, PD and coaching;
– explanation driven science;
– representing student thinking (models);
– Representing data and its use of data (lab notebooks, data
representations)
– Connections between EDP and science content
• Be Explicit: about practices, EDP process and the phases and
pedagogical purpose of each phase;
• Slow Down: Give teachers time to reflect and discuss new
practices;
• Clarify: the distinction between EDP and Inquiry Science;
• Help Plan: Discuss how to connect each EDP activity to
science content.
Today’s Discussion Questions
• How can we increase the likelihood of knowledge
transfer from professional development exemplars
to teacher practice across diverse curricula?
• Which one-one-one coaching practices are most
effective to target individual teacher needs and
support improved instruction?
• How should we calibrate evidence of change in
teachers’ content knowledge, PCK and practices
given their initial state and our external role?
Contact Us
Greg Benedis-Grab – gbenedis@stevens.edu
Arthur H. Camins – acamins@stevens.edu
PISA2- http://ciese.org/pisa2/
CIESE- http://ciese.org/
CIESE in the News- http://ciese.org/pub_news.html