WISE II Evaluation Presentation
2008-09
WO R K I N G T O G E T H E R T O I M PROV E S C I E N C E
E D U C AT I O N
PR E S E N T E D BY G I B S O N & A S S O C I AT E S
A C A L I FO R N I A M AT H A N D S C I E N C E
PA R T N E R S H I P R E S E A R C H G R A N T
Presentation Overview
 General information about the WISE II program
 Discussion of evaluation methods
 Challenges and how we addressed them
 Our preliminary data results.
WISE II Partners
San Francisco Unified School District
San Francisco State University
Cal Academy of Sciences
City College of San Francisco
UC Museum of Paleontology
WISE II Program Description
Based on the WISE I program, WISE II is a three-year
research grant for teachers in grades 3-5 to improve their
science content knowledge, pedagogical content
knowledge, and pedagogy.
Teachers attend a summer workshop, 3 release days, and a
Saturday session. Teachers receive in-classroom coaching
and lesson planning support 1 to 4 times a month and meet
collaboratively with their teams to plan curricula, discuss
student achievement, and develop lessons.
Challenging Courses and Curricula for Teachers and Students
CaMSP defines challenging courses and curricula as
aligned with state content standards and frameworks,
resulting in a greater number of students participating
and succeeding in advanced courses.
How do you evaluate challenging courses and
curricula for elementary students and teachers?
Teacher Data Collection Tools
Teacher Data Collection
Content Knowledge Assessment- Nationally Derived Tool
 Additional Content Assessments – Locally Derived Tools
 Pedagogical Questionnaires – Pre, Mid-Year, and Post
 Professional Development Feedback Forms - Ongoing
 Classroom Observations –twice a year
 Student Notebooks Collected- twice a year
 Use of a Control Group

Evaluation
Methods
Teacher Data
Collection
National Content
Knowledge
Assessments
Local Content
Assessments
Pedagogical
Questionnaires
Formative
Professional
Development
Feedback
Pre-Post Test content knowledge test
 Assessing Teacher Learning About
Science Teaching ATLAST by Horizons
Research.

Strength- piloted and tested and provides
useful information about teacher practice
and knowledge.

Challenge-not CA standards, not directly
aligned to PD, and it’s difficult – so teachers
do not enjoy taking it.
Evaluation
Methods
Teacher Data
Collection
National Content
Knowledge
Assessments
Local Content
Assessments
Pedagogical
Questionnaires
Formative
Professional
Development
Feedback
Complement ATLAST with other
content knowledge instruments:

Standards-based pre-post twice a year.

Faculty developed pre-post instruments
given the day of the treatment.
Teachers receive immediate feedback
on what they have learned that day.
Questionnaires
Evaluation
Methods
Teacher Data
Collection
National Content
Knowledge
Assessments
Local Content
Assessments
Pedagogical
Questionnaires
Formative
Professional
Development
Feedback
Pre-Post & Formative Evaluations of PD

Pedagogical questionnaires examine
frequency of practice & confidence in
knowledge and instruction. They look at
growth by individual teacher and area.
We compare it to our control group.

After every PD session, we provide
feedback forms that include questions
on relevancy to standards and to
instruction.
Control teachers for WISE II
Control Group
Not receive any similar
treatment
Willing to be in program
for 3 years
Willing to take
additional student
assessments
Their own confidence in
science teaching and
instruction
Stipend for participation
 Not receive any WISE II “treatment”
 Complete ATLAST pre-post tests.
 Complete pedagogical questionnaires.
 Two or more teachers at site
participating in the program.
 Students complete the pre-post tests.
 Teacher commits to 3 years.
 Likely to be in district and at grade
level for three or more years- and their
school willing to participate.
Control Group
Easy:
a. Have not
participated in
the program
Hard:
a. Have similar
characteristics
of those in the
program.
Challenge
Competing programs and reforms
Control group more experienced
Higher baseline of performance.
Resolution
View data in terms of growth.
Growth in student scores and
growth in competencies,
confidence, and practice.
Student Data Collection Tools
Student Data Collection
 Pre-Post
Standards Based Content Knowledge
Assessment
 Additional Content Assessments –used locally by
teachers
 California Standards Test – review both ELA and
Science
 Classroom Observations – conducted twice a year
 Student Notebooks Collected twice a year
 Control Group to compare growth
Evaluation
Methods
Student Data Collection
Pre-Post Tests
CST Data
Student
Notebooks
Classroom
Observations
STUDENT
 Pre-post tests in science.
 Look at growth in student
outcomes on California Standards
Test.
 Use rubric to analyze student
science notebooks content,
structure, and understanding.
 Observe engagement and learning
process through classroom
observations using a rubric.
 Biggest challenge- lack of 3rd, 4th
grade CST in science.
RESULTS
A SAMPLING OF PRELIMINARY
DATA FOR WISE II
Student Outcomes
CHANGES IN CST PERFORMANCE
CHANGES IN PRE -POST TEST
R EVIEWED IN CON T EXT OF GR AN T
WISE teachers
improved student
science outcomes
As compared to Control
Group and the district
classrooms, WISE II
teachers experienced
greater growth in science
than their peers between
2008 and 2009.
In particular, more
WISE II teachers had
fewer students in below
basic bands on the CST
and more students were
proficient than the
previous year.
Student outcomes
in Science on the
CST
WISE II teachers showed
an increase on the 5th
grade Science CST from
the baseline 2008 year to
2009 year.
Pre-Post Test
Results
Review of student
growth and average
growth in Earth, Life,
and Physical Science at
each grade level and as
compared to control
group.
Other student outcomes
 Treatment teachers had average
double digit gains on pre-post
science tests in every strand (earth,
life, physical).
 Student results on Pre-Post tests
analyzed by growth and by WISE
program offerings and curricular
practices.
Example of
Pre-Post Analysis
3rd grade areas of
greatest change on
the pre-post test of
WISE II students
 ES.4.b Students know the way in
Standards Learned
80% of 3rd grade WISE
II areas with greatest
growth – pre-to-post test
were addressed in WISE.
These topics were also
seen in student
observations and
drawings in student
notebooks.
which the Moon's appearance
changes during the four-week
lunar cycle.
 ES.4.d Students know that Earth is
one of several planets that orbit
the Sun and that the Moon orbits
Earth.
 PS.1.a Students know energy
comes from the Sun to Earth in the
form of light.
 PS.1.e Students know matter has
three forms: solid, liquid, and gas.
Teacher Changes
CO N T E N T K N OW L E D G E
CON F IDEN C E
PRACTICE
Teacher outcome
results
WISE II teachers
improved content
knowledge based on prepost assessments.
Control Group teachers
declined in content
knowledge.
WISE II Teacher
Content knowledge
WISE II teachers made
growth based on the
local –standards-based
assessment tool.
Treatment
teachers increased
quantity of
instruction.
93% of treatment
teachers reported
teaching a minimum of
46 minutes per week, as
opposed to 78% of the
same teachers the
previous year.
Treatment
teachers
increased use of
effective science
teaching
strategies.
Note-few changes for
control group in these
areas.
WISE II increased
teacher
confidence
Based on baseline and
post-treatment surveys,
WISE II teachers were
more confident in their
use of the following
instructional strategies:
Teaching
Comprehension
Vocabulary
Sketching and Drawing
Increased
confidence and
application
Confidence and Application
0.90
0.80
Build science vocabulary
0.70
Work in cooperative groups
0.60
Integrate real life science
examples
Build reading comprehension
WISE II teachers showed
an average growth in
frequency and
confidence of
application of effective
instructional strategies
in every area analyzed.
0.50
0.40
Make connections between
science and other subjects
Use sketching or drawing to
show understanding.
I am a science teacher leader at
my school site.
0.30
0.20
Lead students using
investigative strategies.
Record observations in a
science notebook.
Record predictions in a science
notebook.
0.10
0.00
Average Individual Growth
WISE II improved
more than control
group teachers.
On average, how many days per week did you teach science?
On average, how many minutes per day did you teach science?
Work in cooperative groups during a science lesson.
Record predictions in a science journal or notebook.
Record observations in a science journal or notebook.
Lead a class of students using investigative strategies.
Have students use sketching or drawing to record their understanding.
WISE II teachers
improved their
confidence and
competence at a greater
rate than the control
group teachers in nearly
every area.
Review student notebooks or journals to assess their science knowledge.
Make connections between science and other subject areas.
0.78
0.74
0.64
0.47
0.42
0.36
0.40
0.40
0.26
0.18
0.15
0.18
0.12
0.05
Control
0.07
0.13 0.13
0.13
Treatment
WISE II – Other Data Collected and Analyzed
Observation Data
Student Science Notebooks
Feedback forms from Evaluations
Teacher Comments
Faculty Input-Changes
WISE II – Challenging Courses and Curricula
For more information,
Contact Nada Djordjevich
Gibson & Associates
nada@gibsonandassociates.com
510-986-0990 ext. 207