ESEA Title IIA: Improving Teacher Quality Program
Foundation for a School - University Partnership for Science and Mathematics Reform:
Years 1 and 2
Adapted in May 2012 from Final Report submitted to Arizona Board of Regents in 2006.
Summary: Middle school and ninth grade science and math teachers in four suburban public
school districts in greater Phoenix participated in a three-week summer Physical Science with
Math Modeling Workshop that focused on matter and energy. Academic year follow-up focused
on force and motion. The workshop improved teachers’ content knowledge and science
students’ achievement, especially in conservation of volume, control of variables, graphing and
equations, and physical and geometric properties of matter. Math students’ achievement was less,
and math teachers continued to lecture frequently. Girls did much worse than boys on basic
proportional reasoning, and hispanics did worse than anglos on all scientific thinking skills
(conservation of mass and volume, basic proportional reasoning, and control of variables).
Greater student learning gains correlated with (resulted from?) greater degree of implementation
of Modeling Instruction, higher socio-economic status (SES), and greater experience of teacher.
Structural challenges at school and district level are barriers to full implementation. Barriers
include fragmented district curriculum, district quarterly benchmarks coupled with a culture of
“tell and practice” in math, lack of coordination between math and science, teacher isolation, lack
of administrative support.
1. Project Details
Project Title: Foundation for a School - University Partnership for Science and Mathematics
Reform: Years 1 and 2
Principal Investigator: Dr. Jane Jackson.
Co-PI: Dr. David Hestenes
Institution: Arizona State University, Department of Physics, Tempe, AZ
Project Period: Feb. 2004 to June 2006
(Note: This was actually two consecutive one-year related projects.)
Other Funds Leveraged with ITQ Funds: three of the four school districts provided pay for
a) high school physics and chemistry teachers who use Modeling Instruction to mentor
participants in their feeder schools, and
b) substitute teachers while participants were observing mentor teachers teach.
2. Project Activities
Background: the need
The first year project was done at the request of Linda Coyle, science specialist in Paradise
Valley Unified School District (PVUSD); she asked for the Modeling Workshop because she was
1
troubled by results of her district-wide survey that indicated poor preparation of many middle
school and early high school teachers in the physical sciences.
(Note: Paradise Valley USD is a misnomer, for it does NOT include the upscale town of Paradise
Valley. In fact, 11 of its 45 schools are Title I schools, and almost 9000 of its 34,000 students
(26%) qualify for free or reduced lunch subsidies. Sections of the school district have diverse
ethnic, mostly disadvantaged families.)
The second year project was done at the explicit request of most PVUSD teachers who
participated in the first Modeling Workshop in summer 2004. They found it so valuable that they
wanted to recruit their colleagues to take it. I invited Deer Valley USD to participate because it
is a large neighboring district. Recruitment by PVUSD and Deer Valley staff was sporadic, so I
later invited the other neighboring districts, Cave Creek and Scottsdale, to ensure that the
workshop filled.
Summer workshops:
Fifty middle school and ninth grade teachers participated in one of two 3-week Modeling
Workshops held in June 2004 and 2005 at Paradise Valley High School. Each workshop was in
Douglas Rickard's chemistry and physics classroom and was taught by Douglas Rickard, the lead
teacher in science, and mathematics co-leader Alicia Collins in 2004 and Stella Ollarsaba in
2005. Douglas Rickard is a Presidential Science Awardee. Alicia taught math at Paradise Valley
High School and Stella at Dobson High School in Mesa Public Schools. Stella had prior
experience co-leading this workshop in Mesa.
Teachers met daily for 4 hours and had three to four hours of daily homework. Contact time,
including academic year follow-up, was about 80 hours, plus individual work (readings, written
reflections, learning technology, adapting instructional materials for their courses), totaling about
135 hours of effort.
Prior to the second workshop, Mr. Rickard and two expert science and mathematics teachers met
for three half-days during spring break 2005 to refine “force & Newton’s laws” workshop
materials to align with new Arizona eighth grade science performance objectives.
Topics of instruction in each workshop and academic year follow-up meetings were thematic
strands in scientific modeling, structure of matter, energy, force and motion, and use of
computers as scientific tools. Mathematics instruction was integrated seamlessly throughout the
entire course by a systematic development of mathematical models – alternating between
analyzing the mathematical structure of a model and its application to make sense of real
phenomena and data.
Instructional methods and materials: The Modeling Workshop is a Methods of Physical Science
Teaching course that addresses many aspects of teaching, including integration of teaching
methods with course content as it should be done in the classroom. The workshop incorporates
up-to-date results of science and mathematics education research, exemplary instructional
materials, use of technology, and experience in collaborative learning and guidance.
2
Participants are introduced to the Modeling Method as a systematic approach to design of
curriculum and instruction. The name Modeling Instruction expresses an emphasis on making
and using conceptual models of physical phenomena as central to learning and doing science.
Adoption of “models and modeling” as a unifying theme for science and mathematics education
is recommended by both NSES and NCTM Standards as well as AAAS Project 2061. However,
to our knowledge, no other program has implemented it so thoroughly.
Content of an entire semester course is reorganized around basic models to increase its
structural coherence. Participants are supplied with a complete set of course materials. The
course includes these models and modeling activities.
1. Modeling geometric properties of matter: length, area and volume
2. Modeling physical properties of matter: mass and density
3. Model of a point particle with constant velocity
4. An atomic (small particle) model of solids, liquids and gases
5. Modeling transfer of energy and its relation to states of matter.
6. Modeling forces (and introducing Newton’s three laws).
Participants alternate between student mode, in which they work through key lessons in the
units, and teacher mode, during which they discuss pedagogical issues surrounding the design
and implementation of the course, as well as become familiar with necessary classroom
technology. The course syllabus/calendar was updated to align with the new Arizona grade-level
science standards for grade 8.
Student activities are organized into modeling cycles, which engage students systematically in
all aspects of modeling. A modeling cycle is described at http://modeling.asu.edu. The teacher
guides students unobtrusively through each modeling cycle, with an eye to improving the quality
of student discourse by insisting on accurate use of scientific terms, on clarity and cogency of
expressed ideas and arguments. Instruction with the modeling cycle repairs a common deficiency
in methods of collaborative inquiry by showing precisely how to conduct scientific inquiry
systematically. After a few cycles, students know how to proceed with an investigation without
prompting from the teacher. The main job of the teacher is then to supply them with more
powerful modeling tools. Lecturing is restricted to scaffolding new concepts and principles on a
need basis. Documents that describe the Modeling Method are posted at http://modeling.asu.edu.
Teachers had a high-tech and a low-tech option for most lab activities. They learned to use the
Graphical Analysis software by Vernier Software & Technology, and the research-based
MathWorlds software for computers. MathWorlds is a powerful junior high modeling software
tool developed in Jim Kaput's NSF SimCalc project.
The course draws upon exemplary research-based materials and resources, notably
Introductory Physical Science, by Uri Haber-Schaim. Workshop design is by David Hestenes.
Course instructional materials were developed by experienced high school physics teachers and
Modeling Workshop leaders, Larry Dukerich and Jeff Hengesbach, with input from Dr. Hestenes.
(See Modeling Instruction for STEM Education Reform, a major proposal by David Hestenes in
2009, pages 10 and 11; downloadable at http://modeling.asu.edu.)
Follow-up activities:
3
Teachers were asked to participate in three days of follow-up, which included a choice of these
day-long activities:
a) three Saturday workshops during fall and winter,
b) adapt lessons using the modeling method (one day; lesson plans were submitted to P.I.),
c) a one-day visit to the classroom of an expert teacher, with a report submitted to the P.I.
Most teachers did participate fully.
The focus of follow-up workshops is exemplified by these notes by Doug Rickard, workshop
leader, regarding the first follow-up on Nov. 6, 2004, attended by 18 teachers. Doug wrote:
"During this session we whiteboarded results of lessons that participants had been
able to implement since last summer's workshop that involved techniques/information
learned this past summer. There was a lot of positive feedback during the discussion
regarding the whiteboards. A good number of participants do not start units on
physics until the spring, but just about everybody had starting using whiteboards with
excellent reports. We then spent the rest of the morning doing labs that develop
Newton's Second Law."
3. Goals and Objectives.
Original goals. Participating teachers will:
 master content in structure/properties of matter, force and motion, energy, scientific thinking
skills, and related skills in each of the Arizona Mathematics Standards,
 improve their instructional pedagogy by incorporating the modeling cycle, inquiry methods,
critical and creative thinking, cooperative learning, and effective use of classroom
technology,
 strengthen coordination and articulation between mathematics and the physical/earth
sciences.
Measurable objectives are:
 increased content knowledge of teachers in properties of matter, energy, force and motion,
graphing, & related math skills (particularly graphical and algebraic representations of
models),
 better instructional strategies, including effective classroom discourse management, use of
standardized evaluation instruments, and improved content organization,
 improved student understanding in structure of matter, energy, motion and force, graphing,
and related mathematics and reasoning skills such as measurement, conservation of volume,
and relation between graphs and equations.
4. Project accomplishments.
Teachers' accomplishments.
4
Results for both years are similar, so I discuss accomplishments of the 23 teachers in the 2005
workshop.
Participants: Thirteen taught science and twelve taught math. Three taught both subjects. One
teacher's classes consisted entirely of middle school students who have behavioral problems, and
another teacher taught only special education. Fourteen taught eighth grade, two taught seventh
grade mathematics, four taught ninth grade algebra, and two taught 10th grade earth science.
The four chief evaluation instruments (see Appendices) were:
 Math Concepts Inventory and the Physical Science Concepts Inventory (given to teachers
and their students as pretests and posttests) (available from Jane.Jackson@asu.edu)
 "Participant Experiences Survey" given between March and October 2006 to most of the
15 teachers who submitted matched pretest and posttest student data, and
 final survey answered by 22 teachers between June and September 2006.
a) All teachers improved their content knowledge in mathematics and the physical sciences.
This is evidenced in two ways:
i. Gains in pretest - posttest scores on the Math Concepts Inventory and the Physical Science
Concepts Inventory (see four graphs in Section 6). Pretests and posttests of both instruments are
given to all teachers on the first and last workshop day in summer.
Half to two-thirds of the teachers started out deficient in content understanding in eighth grade
level math and science. Most improved considerably by the end of the workshop. Our experience
shows that most teachers continue to learn during the academic year if they teach the workshop
content using Modeling Instruction.
ii. Teachers' self-reports to this survey question in the next summer after their Modeling
Workshop:
"Overall, to what extent (on a scale of 1 to 5, with 1 'not at all' and 5 'a lot') has the
Modeling Workshop enhanced your teaching, in improving your content knowledge in
physical science and/or mathematics?"
Of the 22 teachers who took the survey in summer 2006, two-thirds (14) of the teachers wrote
"4" or "5", meaning that they learned much content. The two earth science teachers wrote that
they'd already mastered the content relevant to earth science, but they learned much pedagogy.
The other six teachers answered "3" (i.e., somewhat).
b) All teachers improved their instructional pedagogy, according to their self-reports on this
survey question:
"Overall, to what extent (on a scale of 1 to 5) has the Modeling Workshop enhanced your
teaching, in its pedagogy?"
Three-fourths of the 22 teachers wrote "4" or "5" (much enhancement in pedagogy), and the
other one-fourth wrote "3" (somewhat).
c) Coordination with fellow teachers did not occur for most participants. Applicants for each
workshop were given priority if they applied in teams. Few teams applied, unfortunately.
5
Anecdotes indicate extensive coordination in a few schools, but in most schools the teacher was
isolated, apparently. A survey question was:
"To what extent did you coordinate your math or science course with your colleague(s)?
(so that the courses enhance each other, and thus students learn more)"
Half the teachers replied "not at all" or "little", whereas three reported "a lot". None of those
three had a teammate who participated in the workshop, so apparently they influenced nonparticipants.
d. Teachers' implementation of modeling pedagogy was high and content was medium.
Background: Modeling Instruction is a complex innovation that has several components. Our
experience with thousands of high school physics teachers is that most teachers' implementation
improves for three or four years, as measured by their students' posttest scores on concept
inventories and by teachers' self-reports. Thus we don't expect full implementation in the first
year, nor do we expect high student gains on concept inventory posttests.
In middle school and early high school, big obstacles to full implementation of content and
pedagogy are school district curricula and quarterly benchmarks (in math), which mandate a
particular order of content and a specified pace. (See Section 5.)
With that as background, our overall implementation results for the 22 teachers in 2005-2006
follow.
i. Science teachers' self-reported implementation of all four content units was medium (i.e., "3",
"somewhat"; about halfway to full implementation), on average.
ii. Math teachers' implementation of content in Unit 1 (models of measurement: geometric
properties of matter, motion) was medium, and some math teachers implemented Unit 4 (force,
Newton's laws) at medium extent, on average. Most math teachers did not implement Units 2 and
3 (modeling physical properties of matter; density; atomic model of matter; phases & energy
transfer). If teams had participated in the workshop, math teachers' classroom instruction could
have benefited from these units, for the units use math in science contexts.
iii. Implementation of modeling pedagogy.
 Whiteboarding is the easiest component to implement. Whiteboards (2’ x 2.5’ kitchen-bath
tileboard) are used by small groups of students to develop scientific models via lab activities
and later to adapt those models in other contexts (i.e., to solve problems). Three-fourths of
the teachers rated their implementation as 'high' or 'very high'. (A teacher wrote, "When
I was observed last year, whiteboarding was a++ from my administrator's evaluation!”)
 Socratic questioning is done using whiteboards after every lab activity and in model
deployment (qualitative and quantitative problem-solving). Small groups of students present
their model to the class. 60% of the teachers reported 'high' or 'very high'
implementation.
 Circle whiteboarding (board meeting) is an alternative to Socratic dialogue: the entire class
sits in a circle and discusses a lab activity or problems, using whiteboards. Two-thirds of the
teachers reported 'high' or 'very high' implementation.
6

Cooperative groups. Two-thirds of the teachers reported 'high' or 'very high'
implementation.
iv. Implementation of classroom technology.
 Computers: Three-fourths of the teachers had no student-used computers or only one. We
offered to provide up to 10 refurbished computers per classroom from Arizona StRUT, and
half of the teachers said that they have a strong interest in having a classroom set of
computers, but no teachers returned the StRUT application. Some teachers said that their
classroom was too small for computers. In the workshop, they learned to use Go!Motion
detectors and Go!Temp detectors. Each teacher received one of each.
 Graphical Analysis software was used by half of the teachers for these endeavors: students
used it to mimic graphs and to gather data in labs; and teachers used it with a smart board for
time/motion experiment (in Unit 1), to analyze student data, and to introduce the line of best
fit and scatter plots. Some teachers used it extensively, but some reported that they didn't
have access to computers so couldn't use it.
 SimCalc MathWorlds software was used for 5 days by two eighth grade teachers (one
science, one math), but half of the teachers didn't use it at all. (This research-based math
software was provided free to the teachers by the originator of the product, and teachers
practiced student exercises to use it.)
e. Teachers value the workshop, and half of them want another Modeling Workshop.
Teachers summed up the workshop as follows:
 This response by a second-career science teacher is typical of most science
teachers: "I have learned something important during each day of the
training and during each follow up session. The curriculum is extremely
relevant with excellent teaching tips given to us all. The presenters are very
knowledgeable and give valuable input on what is important to teach in
each topic. The program has integrated math and science together
seamlessly!!!"
 Middle school math teachers of lower-level students wrote that the math was too advanced
for their students and that the workshop focused more on science than on math.
Teachers place high value upon modeling pedagogy! This is indicated by their response to this
survey question:
For which type of students, if any, is your Modeling Workshop learning especially
suitable? ELL? gifted? girls? any other groups?
Almost half of the teachers wrote, "all groups", "any group", "all; it is the best way to
teach", "any and all". Several teachers specified ELL, gifted, and girls; and teachers added
"mainstream special ed", "helped with language issues", "alternative education".
Half of the 22 teachers wrote that they want another Modeling Workshop. Several specified
particular content (e.g., chemistry, mathematics) and level (e.g., lower than eighth grade, and at a
slower pace); their needs vary. A few indicated that they are not ready for another one because
they need to implement more from their first workshop.
7
f. Summary of teachers’ Modeling Workshop accomplishments:
 High implementation in use of whiteboards.
 High implementation of cooperative groups.
 High implementation of student discourse (Socratic questioning).
 High implementation of circle whiteboarding.
 Medium implementation of appropriate content units.
 High enhancement of their content knowledge.
 High enhancement of their pedagogy.
 Little coordination of courses with colleagues.
 Little improvement in classroom technology.
 The degree of encouragement from school administration to implement the workshop
learning was either very low or very high.
 Most science teachers seldom or sometimes lectured; most math teachers lectured frequently.
(Note: I find this persistence of lecturing in math classes troubling.)
 Favorable students' overall reaction to the modeling cycle.
 Fair or good overall implementation of the modeling cycle.
 Most teachers followed modeling cycle phases of model development and model deployment
sometimes; a few followed them frequently or regularly.
Students' accomplishments.
The 23 teachers in the June 2005 workshop gave the Math Concepts Inventory or the Physical
Science Concepts Inventory to one typical class as a baseline posttest in May 2005. Teachers
gave the same inventory as a pretest and a posttest to one class in the 2005-2006 academic year.
This is a goldmine of data! Some are discussed here. Raw data tables are in Section 6.
In the 2005 - 2006 year, 15 teachers submitted usable pretest and posttest student data. Seven
taught eighth grade science, four taught 8th grade math (two algebra, two pre-algebra), and four
taught high school algebra (two regular, one honors, one 2nd year algebra).
(Recruitment for the June 2004 modeling workshop occurred too late to get baseline posttest data
from that group of 27 teachers, unfortunately, due to circumstances beyond our control. Thus,
although that group of teachers submitted student pretest and posttest data, those data are not
discussed here because they are of limited value.)
Baseline posttest data are important aids to help distinguish gains in student achievement that are
due to the Modeling Workshop from ordinary, non-workshop-related, student achievement. In
essence, the teachers act as their own control group.
Background facts of importance: first, gathering and analyzing student data are extremely timeconsuming and labor-intensive activities; for that reason we collected data from only one class of
each teacher. We did not have money nor staff to process more data. Second, some teachers are
careless about having students fill in identifiers; in such cases, even though teachers submit
8
pretest and posttest data sheets, the data can't be matched. Third, in a few cases (e.g., classes for
students who have behavioral problems), student enrollment is unstable and therefore teachers
can't submit matched pretest and posttest data. Finally, a few teachers don't attend follow-up
sessions, nor do they submit data; and no incentive can cause them to contribute; it's beyond our
control.
The instruments:
The Mathematics Concepts Inventory (MCI) was developed by the Physics Underpinnings
Action Research team at Arizona State University (ASU) in June 2000 and revised six times in
the next three years. Version 7 has been used since August, 2003. It has 23 questions and is
intended for 8th and 9th grade students of teachers who take this Modeling Workshop.
The Physical Sciences Concept Inventory (PSCI) was likewise developed by Action Research
teams in the same years, for the same clientele. It was last revised in August, 2003 (version 8).
The first eight questions on the MCI and PSCI are identical; they are paired questions on
scientific thinking skills (conservation of mass and volume, proportional reasoning, control of
variables). They were recommended by Professor Anton "Tony" Lawson, ASU School of Life
Sciences, from his Classroom Test of Scientific Reasoning, a widely-used research-informed
instrument. Conservation of mass and volume, and control of variables are explicit focuses in
Unit 1. Proportional reasoning was addressed only indirectly in the workshop. (Since we found
that some teachers and most students do not reason proportionally, the workshop design should
be revised to explicitly focus on proportional reasoning.)
Other MCI questions are released TIMSS, AIMS, and other standardized test questions: they are
on graphing skills (#10-12,19-23); relating linear equations to other representations (#9,15,18);
estimating area (#13) and volume (#14); measurement (#16) and mean value (#17). Twenty of
the questions are addressed in Unit 1 (models of measurement: geometric properties of matter),
and the other three in Unit 2 (modeling physical properties of matter; density).
The Physical Science Concepts Inventory's 25 questions include 11 that are addressed in Unit 1,
six in Unit 2, and eight in Unit 3. Unit 4 (force and Newton's laws) is not included on the
inventory because it is taught in follow-up meetings.
EVALUATION QUESTIONS:
We address the following questions about overall student achievement in science classes and in
math classes. We address questions about student achievement in individual classes, too,
although numbers are too small to make many inferences.
1. Questions about the entire group of students in science classes, and in math classes.
In each of these two groups:
a. How did students improve during the year in each of
i) the four "Piagetian" scientific thinking skills?
ii) the four science or math content areas (topics), respectively?
9
b. Which improvements during the year were likely to be due to the teachers' Modeling
Workshop? Indicators of this include:
i) baseline posttest scores; those that are much lower than the 2005-06 posttest score
may be due to the teachers' poor content knowledge and/or ineffective pedagogy
before their Modeling Workshop,
ii) teachers' answers to the "Survey of Participant Experiences" (given from March
2006 to Oct. 2006) and the final survey (given from June 2006 to Oct. 2006),
iii) our previous research findings on pretest and posttest scores of students in similar
courses.
c. Did girls and boys score differently from each other in any subcategory?
i) pretest scores?
ii) gains?
iii) posttest scores?
iv) baseline posttest scores?
d. Did hispanics and non-hispanic white (called anglos) score differently? (The number
of blacks and American Indians were too small to contribute meaningful data on both
inventories.)
2. Questions about individual teachers' classes.
a. The four subcategories of Piagetian scientific thinking skills were given to all students.
In which thinking skills, if any, did students of most teachers improve a lot?
b. In each of the two groups (science teachers and math teachers), in what science or math
content, respectively, did students of most teachers improve a lot?
PSCI AND MCI DATA ANALYSIS:
Method: one can look at the data in two ways: mean scores; and percentage of students who
understand the topic, as evidenced by answering most questions on the topic correctly. The
teachers' goals are student understanding of concepts and student development of scientific
thinking skills, so it makes sense to ascertain the percentage of a teachers' students who start and
end the course with good understanding. Thus we focus mostly on the second method.
Overall results:
 On the PSCI, the mean class score increased by 12 percentage points, from 40% to
52% (N=154). Since the baseline posttest was 39%, this indicates a sizable positive
effect of the Modeling Workshop!
 On the MCI, the mean class score rose by 8 percentage points, from 50% to 58%
(N=173). The baseline posttest mean score was 52%, so the Modeling Workshop appears
to have had a small positive effect. The MCI reliability estimate (Cronbach's alpha
coefficient) for the 2007 posttest is 0.83 (ref. Sharon Osborn Popp, Ph.D.). When
drawing inferences from group level data (as in much educational research), a reliability
estimate over 0.80 is often considered sufficient.
Disaggregating the data into individual topics and thinking skills sharpens these tentative
conclusions. Question #1a is addressed in the next two tables. (Raw data are in Section 6.)
10
PSCI
THINKING SKILL
conservation of mass
conserv. of volume
proportional reasoning
control of variables
posttest - pretest
9%
19%
7%
24%
posttest- baseline posttest
7%
11%
7%
11%
Is gain due to workshop?
probably
yes
probably in some classes
yes
Table 1: Gain in percentage of science students (N=154) who understand each scientific thinking
skill. Understanding is defined as "student got both paired questions correct in that thinking
skill."
MCI
THINKING SKILL
conservation of mass
conserv. of volume
proportional reasoning
control of variables
posttest - pretest
1%
6%
13%
13%
posttest- baseline posttest
1%
2%
10%
15%
Is gain due to workshop?
no gain
no
probably in some classes
? (very low baseline)
Table 2: Gain in percentage of math students (N=173) who understand each scientific thinking
skill. Understanding is defined as "student got both paired questions correct in that thinking
skill."
In math classes, the scientific thinking skills of conservation of mass and volume, and control of
variables aren't usually discussed. Most students in the eight math classes did not take science in
a class taught by a Modeling Workshop participant. Thus, to some extent, the math classes are a
control group for these three thinking skills.
[After the final report was submitted to the Arizona Board of Regents, I discovered errors in the
graduate student’s analysis of science and math topics, so I removed all percentages in Tables 3
and 4 below, and the corresponding two pages of raw data in Section 6. I won’t re-do the
percentages unless someone convinces me that it is important, because it is time-consuming.]
My overall qualitative conclusions are:
1. For the science teachers, the Modeling Workshop contributed to large gains in their
student understanding in at least six of the eight thinking skills and science topics.
2. For the math teachers, the Modeling Workshop probably contributed to gains in their
student understanding in three of the eight thinking skills and math topics.
-------------------------------------------------------------------------------PSCI
SCIENCE TOPIC
posttest - pretest posttest- baseline posttest
graphing & equations
geom/phys properties
atomic
nature
of
Is gain due to workshop?
yes
yes
yes, where it was taught
11
matter
energy, states of matter
yes, where it was taught
Table 3: Gain in percentage of science students (N=154) who understand each science topic.
Understanding is defined as "student got at least 2/3 of the questions in each topic correct."
[Percentages are removed from original report, due to probable errors by graduate student.]
To arrive at the judgment of the last column in the above two tables for the PSCI, the PSCI
student data tables in Section 6 were used. In particular, the fifth data table [in original report]
gives evidence for a correlation between PSCI student scores of individual teacher and that
teacher's characteristics, including their self-reported degree of implementation, baseline
posttest mean score in spring 2005, content knowledge, and years' science teaching experience.
The relationship could perhaps be quantified, though only approximately. A factor not included
in the data table and that could be important is the teacher's comment on whether or not their
students in the two successive years were similar in ability, motivation, and life experiences.
MCI
MATH TOPIC
graphing
equations
measurement/geometry
statistics/data analysis
posttest - pretest posttest- baseline posttest
Gains (qualitative)
No apparent gain
No apparent gain
Gain in a few classes
Gain in some classes
Table 4: Gain in percentage of math students (N=173) who understand each math topic.
Understanding is defined as "student got at least 2/3 of the questions in each topic correct."
[Percentages are removed from original report, due to possible errors by graduate student.]
----------------------------------------------------------------------Gender effects (question 1c): The largest gender effect was in proportional reasoning. On all
three tests (baseline posttest, pretest, and posttest), the percentage of girls who got both of the
paired questions in proportional reasoning correct was about half that of boys. I find this
very disturbing. (I could disaggregate our previous years' PSCI and MCI data for 1000 junior
high students to see if this gender effect is widespread!) (Raw data are in Section 6.)
Ethnicity effects (question 1d): On the PSCI, hispanics scored much lower than anglos on all
scientific thinking skills, both on pretest and posttest. The disparity was less on science topics
(but please disregard those numbers, for they are suspect). (Note: only 6 math students were
hispanics, so their MCI data are meaningless. Raw data are in the last table in Section 6.)
Individual teachers' classes (question 2):
Thinking skills: By far, the largest gains in student understanding occurred for control of
variables: in 13 of the 15 classes, at least 10 percent more students understood control of
variables at the end of the school year than at the beginning (i.e., 10% gain from pretest to
posttest). Two science classes were especially effective: approximately 50% more students
understood it! Proportional reasoning was next highest in gain. In 3/4 of the math classes and
12
in four of the seven science classes, at least 10 percent more students understood proportional
reasoning at the end of the school year than at the beginning (i.e., 10% gain from pretest to
posttest). In fact, in seven of these classes, 15% to 20% more students understood it at the end of
the school year.
Science topics, math topics: [Comments are removed, due to probable errors by graduate
student.]
DISCUSSION AND SIGNIFICANCE of some important results:
Piagetian reasoning skills (scientific thinking skills).
Conservation of weight (or mass) and volume are fundamental concrete operational
scientific/mathematical reasoning abilities; and eighth grade students should do well on them.
Conservation of weight (or mass) was tested on the first 2 questions of the MCI and the PSCI.
About 80% of 8th and 9th graders answered these 2 questions correctly at the beginning of
the course, and a gain of 9 percentage points occurred in science classes but no gain in
math classes. So most 8th graders understand that when you flatten a piece of clay, its weight
doesn't change. Anton Lawson has found that 10 year old children typically understand this
concept (private communication, and Lawson and Bealer, 1984-2). Students in math classes
were somewhat deficient in understanding.
Conservation of volume was tested in questions #3 and 4 on each test. The question is this:
given two identical cylinders filled to the same height with water. If you drop 2 marbles, one of
glass and the other of steel, both the same size, into the water, compare the new heights of the
water. Of course, since the marbles have the same volume, the water rises by the same height in
each cylinder. At the beginning of the year, about 50% of our students exhibited
understanding; and for science students about 60% understood it at the end of the school
year (math students didn't improve much). This concept is typically mastered by 11 or 12 year
olds, according to Anton Lawson (private communication, and Lawson, 1990).
Anton Lawson reports that more than 3000 7th, 8th, and 9th graders in North Carolina were
given these same two questions on conservation of volume; their mean score (success rate) was
about 60% (Lawson, 1990). Japanese students of the same age group did even better, scoring
about 70%. The test in the1990 paper was not in multiple choice format, yet scores are consistent
with the multiple choice format, Anton Lawson assured me.
Control of variables: The same published paper includes results showing that our science
students did well in the simple formal operational reasoning skill called "control of variables"
(questions #7 and 8). The two questions concern three strings hanging from a bar, with weights
attached to their ends; decide which strings should be used to find out if length or weight affects
the time it takes to swing back and forth. In our study, science students started at 36%
success, higher than North Carolina students, and they made good gains during the year,
ending up at 58%, comparable to the Japanese students.
13
Proportional reasoning (questions #5 and 6) is a complex formal operational thinking skill that
is crucial for the physical sciences. School instruction and life experiences both contribute to
development of this ability, as was reported in a research study by Prof. Anton Lawson of
Arizona State University (Lawson and Bealer, 1984-1). A crucial finding of previous large-scale
international studies is that American 7th, 8th, and 9th grade students' understanding of
proportional reasoning is much poorer than that of Japanese students.
Question #5 states that a given quantity of water occupies 4 units in a wide cylindrical
container and 6 units when poured into a narrow one. Students are asked to predict how high a
given quantity of water that occupies 6 units in the wide container would rise if poured into the
narrow container, and in question #6 they are asked to choose among five reasons.
In our study, at the beginning of the year, about 12% of students answered both
questions correctly. At the end of the school year, 22% of students answered both
questions correctly. By random guessing, 4% are predicted to answer both questions
correctly; thus it is likely that only 8% of students reasoned proportionally at the
beginning of the year, and 18% at the end of the year.
These percentages are comparable to or higher than those of 104 suburban Arizona 8th
graders and 110 8th graders in rural Arizona in January 1982 (Lawson, & Bealer, 1984-1): the
percentage of correct responses was about 8% for both groups on the original pencil and paper
version of the test. For 104 eighth graders in the San Francisco Bay region, in a middle class
suburban area of cultural diversity, in spring 1978 the percentage of correct responses was about
20% (Lawson & Bealer, 1984-1), comparable to our students' posttest scores.
In other studies, the percentage of correct responses for 3291 randomly selected 7th, 8th,
th
and 9 grade North Carolina students was 17%, and for 4397 randomly selected 7th, 8th, and 9th
grade Japanese students was 53% (Lawson, 1990). A problem is that in none of the above studies
was pre-test and post-testing done. The studies indicate that American students' understanding is
much poorer than that of Japanese students in the same grade band.
In our study, about twice as many boys as girls answered both questions correctly
on all three tests. We aren't aware of any other study that examined gender effects. A much
smaller percentage of hispanic students understood; but the numbers of students (29) are low, so
this only indicates a possible trend.
REFERENCES ON SCIENTIFIC THINKING SKILLS:
Lawson, A.E. (1978). The development and validation of a classroom test of formal reasoning.
Journal of Research in Science Teaching 51(1): 11-24.
Lawson , A. E. (1990). Science Education in Japan and the United States: Are the Japanese
beating us at our own game? Science Education 74(4): 495-501.
Lawson, A. E. and Bealer, J. M. (1984-1). Cultural Diversity and Differences in Formal
Reasoning Ability, Journal of Research in Science Teaching 21, 735-743.
Lawson, A. E. and Bealer, J. M. (1984-2). The acquisition of basic quantitative reasoning skills
during adolescence: Learning or development? Journal of Research in Science Teaching
21(4): 417-423.
Science topics:
14
The weakest topic for student and teacher understanding, as revealed by the PSCI, was energy &
states of matter (pretest scores 0%; no gains in 5 of the 7 classes). This is alarming, for energy
and matter are the most fundamental concepts in all sciences.
Four research-based conceptual questions about energy were included, one of which is this:
As water in an ice cube tray freezes,
a. it absorbs energy from its surroundings.
b. its surroundings absorb energy from it.
c. it absorbs coldness from and releases energy to its surroundings.
d. it only absorbs the coldness from its surroundings.
e. it neither absorbs nor releases energy, because its temperature stays constant.
The five choices are expressed in the language of children who were interviewed.
Our results are that in most eighth grade science classes, students don't BEGIN to understand
energy. Scores were BELOW random, indicating the powerful alternative conceptions and also
the fact that the teachers' final surveys revealed that most teachers did not implement Unit 3,
even though it aligns with eighth grade science performance objectives. Perhaps most teachers
didn't feel comfortable with energy and states of matter; in other words, they need more
workshop experience. On the other hand, perhaps they ran out of time. Or maybe the AIMS
science test doesn't include many questions on these topics, so they didn't think it worth doing.
One exception occurred: in an eighth grade science class in which most students were
concurrently taking geometry and the teacher had a decade of science teaching experience, by the
end of the year 70% of students answered at least three of the four questions on that topic
correctly.
15
5. Project Challenges
a) Teacher turnover at Greenway Middle School, a Title I school in PVUSD that failed to meet
academic benchmarks for AYP under Arizona's guidelines for NCLB for the spring of 2003 and
2004. Seven Greenway teachers participated in the Modeling Workshop of summer 2004, but all
of them are gone from that school now. Greenway wanted to participate again; I met with their
principal in December 2004 for an hour, and he submitted a letter with the Year 2 proposal,
describing their need. However, he did not follow through with recruitment, and thus only one
Greenway teacher participated in 2005. That teacher is an enthusiastic second career teacher.
The PVUSD science coordinator and I devised a strategy to teach modeling instruction to new
teachers there. However, it did not receive administrative support and was not implemented.
b) Obstacles to implementation, due to a district-mandated fragmented science curriculum, as
exemplified by this email from an eighth grade science teacher (Feb. 28, 2006):
"We no longer teach a subject based curriculum because of a fear that students
will forget concepts taught in the sixth grade long before they reach it on an
AIMS test in high school. Hence, we have this chopped up curriculum often
requiring us to teach concepts without the necessary pre-requisite background. A
prime example is my eighth grade curriculum. I am expected to teach the basic
structure of the periodic table without teaching any real background into atomic
structure?! Stunning and insulting to those of us with secondary degrees in our
subject area."
c) Difficulties in implementation due to district-mandated quarterly benchmarks that are aligned
with a traditional classroom and textbook (which does not always order content in a logical
sequence that facilitates integration with science), as exemplifed by this email from a math
teacher (Feb. 2006):
"It seems that it will be difficult to align the benchmarks which are given quarterly, to
classrooms which focus on concepts in a different order." "The mis-alignment of the
modeling curriculum and benchmarks is a concern which will need to be addressed
given the emphasis on testing being developed in our district."
A teacher wrote: "Even if the time was set aside [each month], the push to testing benchmarks
dictates the math curriculum in a way that makes coordination difficult."
d) Lack of structure in schools to promote math and science coordination.
 Few teams of teachers applied, even though teams were given priority.
 A question on the survey in summer 2006 was:
"How valuable would it be, if you had an hour every month set aside, to coordinate your
math and science courses with teachers in your school? "
Half of the teachers replied, "EXTREMELY or VERY valuable". "But it probably will never
happen", added one teacher. Another wrote, "It would require MORE time to be effective".
Another wrote, "Valuable if math had a more flexible schedule". A science teacher wrote,
"Testing math benchmarks make coordination hard."
16
e) Lack of support, especially financial support, by school administration.
Teachers' answers to these questions reveal this set of problems.
"To what extent have you received encouragement (from your school administration) to
implement the new materials/techniques in your classroom? "
"What obstacles exist, to implementing your workshop learning in your classroom?"
Half the teachers reported little or no administrative support, and that a major obstacle is
"administration won't spend money for supplies". On the other hand, one-fourth of the teachers
wrote that their school principal is "very encouraging".
f) Teachers cited these other obstacles that relate to school and school district structures (or lack
thereof: see part d above): "time and experience", "time and energy", "takes longer to teach a
concept", "too little time, too much information to cover", "outside requirements", "testing",
"science over my head". Also: "Teachers have become bombarded with other issues they are
required to address that the ability to concentrate on improved instruction and collaboration is
diminished."
g) Last, but incredibly important: Science and mathematics content are in a bad state. For one
thing, mathematics is divorced from science. The fundamental concept in mathematics is
quantity; a quantity is a number and a unit. But mathematicians have neglected unit and focused
only on number, to the detriment of students' real understanding of mathematics. Modeling
Instruction improves instruction by focusing on quantity and mathematical modeling. Much
evidence exists of its success.
Another huge problem is the poor way that energy is taught. Energy is the fundamental
unifying concept of all sciences, but middle school textbooks do a poor job of treating it.
Modeling Instruction clarifies the concept of energy, and evidence exists of consequent improved
student learning. Unfortunately, our proposal to the U.S. Dept of Education to develop an 8th-9th
grade Modeling Workshop that focuses on energy was rejected twice.
17
6. Summary of Evaluation Data
A. Evaluation Data for Teachers
Teachers' workshop evaluations, measurement of teacher learning:
Workshop rating (10=excellent, 1=poor): 2004: 9.4 – excellent!
An experienced teacher sent me this e-mail, just after the summer 2004 workshop ended:
"I was extremely impressed with the course, the materials, and the instructors. I
can honestly say that this is the best course that I have taken in my journey to become
a teacher. Too often we are given theory that isn't compatible with the reality of the
classroom. This course gave us tools/methods that will be easy to incorporate, and by
having experienced the procedures from the student point of view we have
experienced first-hand how to do it.
From the comments made in the classroom and the overwhelming desire for a
follow-on course, I'd say my attitude is in line with the rest of the teachers that took
the course."
On the next pages are four graphs that show that most weakly prepared teachers improved
considerably in content knowledge! The graphs for the PSCI and MCI look similar in that
teachers improved in both content areas. (Notice that 100% is NOT at the top of the 2004 graphs;
numerous teachers’ posttests scores are at or near 100%.)
It is worrisome that half to two-thirds of the teachers start out deficient in content understanding
in eighth grade math and science. It is heartening that most improve considerably in the
workshop (although improvement was greater in 2004 than in 2005). As noted above, our
experience shows that most teachers continue to learn during the academic year if they teach the
workshop content using modeling instruction.
We feel strongly that most teachers would benefit from a second three-week modeling workshop.
18
Math Concepts Inventory: Summer 2004. Teacher Pre-tests and Posttests
PSCI Summer 2004 Individual Teacher data: Pre-tests and Posttests
19
20
21
B. Evaluation Data for Students
Timeline:
* April & May 2005: science students took Physical Science Concept Inventory (PSCI), and
math students took Math Concepts Inventory (MCI) to get baseline data. (Recruitment in Year 1
occurred too late to get baseline data.)
* August 2004 and 2005: science students took PSCI pretest, math students took MCI pretest.
* March to May 2005 and 2006 (or earlier, if appropriate): students took PSCI or MCI posttest.
Data tables of student results on the PSCI and MCI are on the next three pages. On the third
page, science teachers' characteristics are correlated with their students' scores.
[Note: two pages of data have been removed from the original final report submitted to the
Arizona Board of Regents in 2006, because mistakes were later found in the graduate students’
data analysis in some of these science topics: graphing & equations, geometric/physical
properties, atomic nature of matter; energy, states of matter. Possibly also in some math topics:
graphing, equations, measurement/geometry, statistics/data analysis.]
22
PSCI student data: baseline posttest; pretest, posttest 2005-2006
OVERALL TEST SCORES: all seven science teachers' classes. (Pre-test and posttest data are matched.)
Test
N # Qs
PSCI grade 8 students 154
25 Overall score
MEAN CLASS SCORE
pretest posttest
baseline
40%
52%
39%
(Gains of 10 percentage points or more from pretest to posttest are in boldface. Gains of 5% or higher from baseline to posttest in italics.)
SCIENTIFIC THINKING SKILLS:
% students who got BOTH questions RIGHT:
MEAN CLASS SCORE
Test
N # Qs thinking skill
pretest posttest
baseline baseline N
pretest posttest
baseline
PSCI grade 8 students 154
2 Mass Conservation
81%
90%
83%
167
85%
93%
87%
PSCI grade 8 students 154
2 Volume Conservation
49%
60%
49%
167
51%
62%
50%
PSCI grade 8 students 154
2 Proportional Reasoning
16%
23%
16%
167
25%
28%
20%
PSCI grade 8 students 154
2 Control of Variables
34%
58%
47%
167
43%
64%
55%
Test
PSCI
PSCI
PSCI
PSCI
PSCI
PSCI
PSCI
PSCI
% MALES who got BOTH questions RIGHT:
N # Qs thinking skill
grade 8 males
69
2 Mass Conservation
grade 8 males
69
2 Volume Conservation
grade 8 males
69
2 Proportional Reasoning
grade 8 males
69
2 Control of Variables
% FEMALES who got BOTH questions RIGHT:
grade 8 females
85
2 Mass Conservation
grade 8 females
85
2 Volume Conservation
grade 8 females
85
2 Proportional Reasoning
grade 8 females
85
2 Control of Variables
pretest posttest
83%
94%
55%
67%
22%
30%
39%
61%
80%
44%
11%
29%
87%
54%
17%
55%
INDIVIDUAL TEACHERS' STUDENT SCORES
# students in each class who got BOTH questions CORRECT:
Test
teacher
N # Qs thinking skill
pretest posttest
PSCI grade 8
4103 24
2 Mass Conservation
83%
100%
PSCI grade 8
4103 24
2 Volume Conservation
54%
88%
PSCI grade 8
4103 24
2 Proportional Reasoning
17%
38%
PSCI grade 8
4103 24
2 Control of Variables
38%
67%
baseline baseline N
80%
83
54%
83
21%
83
46%
83
86%
44%
11%
49%
84
84
84
84
baseline baseline N
100%
21
57%
21
24%
21
71%
21
MEAN CLASS SCORE
pretest posttest
baseline
88%
96%
83%
55%
67%
56%
29%
33%
25%
48%
67%
53%
MEAN CLASS SCORE
82%
90%
90%
47%
58%
44%
22%
23%
15%
39%
61%
57%
MEAN CLASS SCORE
pretest posttest
baseline
83%
100%
100%
58%
94%
57%
25%
54%
26%
46%
77%
76%
PSCI
PSCI
PSCI
PSCI
grade 8
grade 8
grade 8
grade 8
5101
5101
5101
5101
15
15
15
15
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
60%
47%
13%
13%
73%
33%
13%
53%
59%
24%
6%
18%
17
17
17
17
67%
53%
30%
23%
73%
37%
17%
60%
71%
24%
15%
26%
PSCI
PSCI
PSCI
PSCI
grade 8
grade 8
grade 8
grade 8
5104
5104
5104
5104
26
26
26
26
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
77%
27%
15%
27%
89%
42%
12%
31%
81%
62%
12%
46%
26
26
26
26
85%
31%
25%
35%
90%
44%
17%
40%
83%
63%
17%
58%
PSCI
PSCI
PSCI
PSCI
grade 8
grade 8
grade 8
grade 8
5114
5114
5114
5114
18
18
18
18
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
94%
44%
22%
39%
83%
44%
11%
56%
86%
54%
18%
43%
28
28
28
28
97%
44%
25%
50%
92%
44%
11%
58%
89%
54%
21%
54%
PSCI
PSCI
PSCI
PSCI
grade 8
grade 8
grade 8
grade 8
5119
5119
5119
5119
26
26
26
26
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
85%
62%
15%
54%
96%
73%
31%
65%
83%
65%
17%
70%
23
23
23
23
88%
62%
25%
58%
98%
75%
35%
67%
87%
65%
22%
70%
PSCI
PSCI
PSCI
PSCI
grade 8
grade 8
grade 8
grade 8
5121
5121
5121
5121
24
24
24
24
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
83%
46%
8%
29%
83%
54%
21%
54%
87%
65%
35%
52%
23
23
23
23
85%
46%
25%
44%
88%
56%
23%
65%
91%
65%
37%
59%
PSCI
PSCI
PSCI
PSCI
grade 8
grade 8
grade 8
grade 8
5122
5122
5122
5122
21
21
21
21
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
81%
62%
19%
29%
100%
71%
29%
81%
29
29
29
29
86%
62%
21%
38%
100%
71%
29%
81%
83%
21%
3%
41%
79%
17%
0%
31%
23
Math Concepts Inventory: student data. Baseline posttest; pretest, posttest 2005-06
24
OVERALL TEST SCORES OF ALL EIGHT TEACHERS' CLASSES. (Pretest & posttest data are matched.)
N
# Qs
MCI grade 8, 9
students 173
23 OVERALL SCORE
MEAN CLASS SCORE
pretest posttest
50%
58%
baseline
52%
(Gains of 10 percentage points or more from pretest to posttest are in boldface. Gains of 5% or higher from baseline to posttest are in italics.)
SCIENTIFIC THINKING SKILLS: % students who got BOTH questions RIGHT:
MEAN CLASS SCORE
N
# Qs thinking skill
pretest posttest
baseline
baseline N
pretest posttest
MCI grade 8, 9
students 173
2 Mass Conservation
82%
83%
82%
200
83%
84%
MCI grade 8, 9
students 173
2 Volume Conservation
46%
52%
50%
200
47%
53%
MCI grade 8, 9
students 173
2 Proportional Reasoning
8%
21%
11%
200
12%
24%
MCI grade 8, 9
students 173
2 Control of Variables
36%
49%
34%
200
43%
55%
baseline
86%
53%
16%
41%
% MALES who got BOTH questions RIGHT*:
N
# Qs thinking skill
MCI grade 8, 9
males
82
2 Mass Conservation
MCI grade 8, 9
males
82
2 Volume Conservation
MCI grade 8, 9
males
82
2 Proportional Reasoning
MCI grade 8, 9
males
82
2 Control of Variables
% FEMALES who got BOTH questions RIGHT*:
MCI grade 8, 9
females
81
2 Mass Conservation
MCI grade 8, 9
females
81
2 Volume Conservation
MCI grade 8, 9
females
81
2 Proportional Reasoning
MCI grade 8, 9
females
81
2 Control of Variables
* For teacher #5106, baseline data can't be disaggregated by gender.
INDIVIDUAL TEACHERS' STUDENT SCORES
% students in each class who got BOTH questions CORRECT:
teacher N
# Qs thinking skill
MCI grade 9 algebra
4105
23
2 Mass Conservation
MCI grade 9 algebra
4105
23
2 Volume Conservation
MCI grade 9 algebra
4105
23
2 Proportional Reasoning
MCI grade 9 algebra
4105
23
2 Control of Variables
21
21
21
21
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
85%
32%
4%
36%
83%
42%
12%
44%
pretest posttest
74%
65%
30%
44%
4%
9%
4%
22%
100%
67%
0%
57%
86%
71%
14%
67%
baseline
baseline N
78%
92
59%
92
12%
92
25%
92
83%
40%
9%
39%
92
92
92
92
baseline
baseline N
76%
29
38%
29
3%
29
17%
29
71%
32%
4%
29%
MEAN CLASS SCORE
pretest posttest
77%
84%
59%
65%
18%
32%
41%
55%
MEAN CLASS SCORE
88%
85%
33%
44%
7%
17%
41%
53%
MEAN CLASS SCORE
pretest posttest
74%
67%
30%
43%
13%
15%
15%
30%
86%
71%
17%
76%
baseline
85%
64%
19%
36%
85%
41%
11%
45%
baseline
83%
41%
5%
26%
MCI
MCI
MCI
MCI
grade 8 algebra
grade 8 algebra
grade 8 algebra
grade 8 algebra
28
28
28
28
100%
67%
12%
62%
MCI
MCI
MCI
MCI
gr 8,9 honors algebra
5106
13
2 Mass Conservation
92%
85%
100%
15
gr 8,9 honors algebra
5106
13
2 Volume Conservation
54%
46%
60%
15
gr 8,9 honors algebra
5106
13
2 Proportional Reasoning
0%
15%
20%
15
gr 8,9 honors algebra
5106
13
2 Control of Variables
62%
77%
53%
15
* Teacher #5106 reports that the baseline class (spring 2005) was of much higher ability than the 2005-2006 class.
96%
54%
0%
65%
85% *
46%
15%
81%
MCI
MCI
MCI
MCI
grade 8 prealgebra
grade 8 prealgebra
grade 8 prealgebra
grade 8 prealgebra
5109
5109
5109
5109
20
20
20
20
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
80%
40%
10%
45%
85%
60%
25%
60%
84%
32%
4%
32%
25
25
25
25
80%
40%
18%
53%
88%
63%
25%
70%
88%
38%
6%
44%
MCI
MCI
MCI
MCI
grade 11 algebra II
grade 11 algebra II
grade 11 algebra II
grade 11 algebra II
5112
5112
5112
5112
18
18
18
18
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
83%
72%
6%
44%
89%
72%
39%
61%
80%
52%
16%
40%
25
25
25
25
86%
72%
6%
50%
89%
75%
47%
64%
86%
54%
18%
48%
MCI
MCI
MCI
MCI
grade 8 prealgebra
grade 8 prealgebra
grade 8 prealgebra
grade 8 prealgebra
5115
5115
5115
5115
29
29
29
29
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
83%
31%
7%
28%
79%
38%
31%
38%
72%
50%
11%
11%
18
18
18
18
83%
33%
7%
34%
81%
38%
31%
45%
78%
53%
14%
14%
MCI
MCI
MCI
MCI
grade 8 algebra
grade 8 algebra
grade 8 algebra
grade 8 algebra
5118
5118
5118
5118
24
24
24
24
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
79%
58%
17%
50%
96%
63%
29%
67%
86%
66%
21%
52%
29
29
29
29
79%
58%
21%
54%
98%
63%
33%
73%
88%
67%
26%
57%
MCI
MCI
MCI
MCI
grade 9 reg
grade 9 reg
grade 9 reg
grade 9 reg
5120
5120
5120
5120
25
25
25
25
2
2
2
2
Mass Conservation
Volume Conservation
Proportional Reasoning
Control of Variables
68%
28%
12%
20%
80%
32%
4%
20%
90%
73%
13%
37%
30
30
30
30
76%
32%
16%
28%
84%
38%
8%
26%
93%
75%
18%
48%
algebra
algebra
algebra
algebra
5103
5103
5103
5103
pretest posttest
77%
83%
59%
63%
12%
29%
33%
49%
77%
38%
16%
38%
100%
60%
27%
53%
25
Overall PSCI mean scores and overall scores by gender.
Also, the correlation between PSCI student scores of the individual teacher and that
teacher's characteristics (including the teacher's self-reported degree of implementation,
baseline posttest mean score in spring 2005, content knowledge, and years' science teaching
experience.)
OVERALL PSCI RESULTS IN GRADE 8 (baseline posttest 2005; 2005-'0 6)
Test
pre/post
total N Mean (%)
PSCI PRETEST
154
40%
PSCI POSTTEST
154
52%
baseline baseline N
39%
167
40%
83
39%
84
OVERALL PSCI RESULTS, BY GENDER
pre/post
PSCI PRETEST
gender
total N Mean (%)
male
69
42%
PSCI POSTTEST male
69
54%
pre/post
PSCI PRETEST
gender
total N Mean (%)
female
85
39%
PSCI POSTTEST female
85
50%
OVERALL RESULTS of CLASS OF INDIVIDUAL TEACHER
pre/post
PSCI PRETEST
TEACHER'S PSCI (June '0 5)
teacher total N Mean (%) implementation
4103
24
47%
PSCI POSTTEST 4103
24
75%
PSCI PRETEST
5101
15
35%
PSCI POSTTEST 5101
15
43%
PSCI PRETEST
5104
26
38%
PSCI POSTTEST 5104
26
41%
PSCI PRETEST
5114
18
42%
PSCI POSTTEST 5114
18
45%
PSCI PRETEST
5119
26
40%
PSCI POSTTEST 5119
26
52%
PSCI PRETEST
5121
24
41%
PSCI POSTTEST 5121
24
47%
PSCI PRETEST
5122
21
39%
PSCI POSTTEST 5122
21
57%
pretest
posttest
yrs taught
high
44%
21
(didn't take)
100%
12
low-medium
31%
17
84%
100%
1
low-medium
40%
26
68%
84%
1
medium
41%
28
76%
92%
8
medium-high
41%
23
88%
92%
4
high-very high
45%
23
96%
92%
1
high
32%
29
96%
100%
5
26
PSCI student scores for hispanic and anglo 8th graders: 2005-2006.
OVERALL TEST SCORES: all seven teachers' classes. (Pre-test and posttest data are matched.)
MEAN CLASS SCORE
Test
pretest posttest
N
PSCI grade 8
students
154
# Qs
25 Overall score
40%
baseline
52%
39%
(Gains of 10 percentage points or more from pretest to posttest are in boldface. No baseline ethnicity data are available.)
SCIENTIFIC THINKING SKILLS:
% HISPANIC who got BOTH questions RIGHT:
Test
N # Qs thinking skill
MEAN CLASS SCORE
pretest
posttest
pretest posttest
PSCI grade 8
hispanic
29
2 Mass Conservation
69%
76%
78%
79%
PSCI grade 8
hispanic
29
2 Volume Conservation
31%
35%
34%
36%
PSCI grade 8
hispanic
29
2 Proportional Reasoning
7%
35%
34%
36%
PSCI grade 8
hispanic
29
2 Control of Variables
10%
41%
24%
50%
% ANGLO who got BOTH questions RIGHT:
MEAN CLASS SCORE
PSCI grade 8
anglo
116
2 Mass Conservation
84%
94%
89%
96%
PSCI grade 8
anglo
116
2 Volume Conservation
53%
66%
55%
69%
PSCI grade 8
anglo
116
2 Proportional Reasoning
18%
66%
55%
69%
PSCI grade 8
anglo
116
2 Control of Variables
39%
60%
47%
66%
SCIENCE TOPICS:
% HISPANIC who got 2/3 or more questions in topic RIGHT:
Test
N # Qs science content
MEAN CLASS SCORE
pretest
posttest
pretest posttest
PSCI grade 8
hispanic
29
4 Graphing Skills/Equations
35%
52%
48%
60%
PSCI grade 8
hispanic
29
3 Geometric & Phys Properties of Matter
48%
59%
49%
59%
PSCI grade 8
hispanic
29
6 Atomic Nature of Matter
7%
24%
28%
37%
PSCI grade 8
hispanic
29
4 Energy and States of Matter
0%
10%
5%
17%
% ANGLO who got 2/3 or more questions in topic RIGHT:
MEAN CLASS SCORE
PSCI grade 8
anglo
116
4 Graphing Skills/Equations
45%
66%
61%
69%
PSCI grade 8
anglo
116
3 Geometric & Phys Properties of Matter
54%
67%
51%
60%
PSCI grade 8
anglo
116
6 Atomic Nature of Matter
10%
30%
31%
45%
PSCI grade 8
anglo
116
4 Energy and States of Matter
2%
14%
8%
23%
27
Workshop in Physical Science & Math Modeling
Survey of Participant Experiences
Academic Year 05-06
Name: ______________________________________
School: ______________________________________
This survey is intended to assist in the assessment and evaluation of the workshop.
Answer the survey for the course you're implementing Modeling Instruction in most fully, this
year.
For which course are you answering this survey? ________________________________
For what grade are you answering the survey? ____________
 Please answer all questions based on your experience during this academic year.
 Write N/A next to any question that doesn't apply.
Your cooperation is greatly appreciated.
1. How often do you ask students to work in
groups in your class?
1
2
3
Regularly Frequently Sometimes
2. How often do you ask different groups to discuss
their ideas in class?
Regularly Frequently Sometimes
3. How often do you use (student-size, portable)
whiteboards in conjunction with group work?
Regularly Frequently Sometimes
4. How often do you lecture (for more than onequarter of the period)?
Regularly Frequently Sometimes
5. How often do you use a standard textbook?
1
1
1
1
2
2
2
2
3
3
3
3
Regularly Frequently Sometimes
18. How often do you follow the modeling cycle
phases of model development and deployment?
1
2
3
Regularly Frequently Somettimes
4
5
Seldom
Never
4
5
Seldom
Never
4
5
Seldom
Never
4
5
Seldom
Never
4
5
Seldom
Never
4
5
Seldom
Never
19. How do you rate your own understanding of the
instructional modeling cycle?
1
2
3
4
5
Very good
Good
Fair
Poor
Nil
20. Which of the modeling cycle components are most
helpful for your students (check all that apply)?
1
2
3
4
Modeling
phases
Group
work
21. What is your students’ overall reaction to the
modeling cycle?
22. How do you rate your overall implementation of
the modeling cycle?
Whiteboards Handouts
1
2
3
Very
favorable
Favorable
Neutral
4
5
Other
5
Not that Not favorable
favorable
at all
1
2
3
4
5
Very good
Good
Fair
Poor
Nil
28
23. Are you on block schedule? If so, what type of block?_____________________________
24. How many Go!motion detectors or calculator based rangers (CBRs) do you have easy access
to? ___ How many student-used computers? ___Are the computers in your classroom?
29
FOLLOW-UP REPORT (sent by e-mail to all teachers in June 2006):
your name: _________________________________
Grade in which you gave PSCI or MCI: ___________
Course & level in which you gave PSCI or MCI
(
pre-alg?
alternative
alg?
phys
sci?
for
lower-level?
gifted?)______________________
# times you've taught that course: ____________
# years you've taught middle school or grade 9: _________
-------------------------Answer on scale of 1 to 5 (1= not at all or insignificant, 3 = somewhat,
5 = fully or substantial, n/a = not applicable)
CONTENT:
To what extent did you implement the modeling workshop units this year?
Unit 1: models of measurement: geometric properties of matter, motion ________
Unit 2: modeling physical properties of matter; density _________
Unit 3: atomic model of matter; phases & energy transfer _________
Unit 4: force, Newton's laws _________
honors?
METHODS:
To what extent did you use this year:
1) whiteboarding? ____
2) Socratic questioning? ____
3) circle whiteboarding? ____
4) cooperative groups? _____
COORDINATION OF MATH AND SCIENCE:
To what extent did you coordinate your math or science course with your colleague(s)? (so that
the courses enhance each other, and thus students learn more)_________
SUMMARY: Overall, to what extent (on a scale of 1 to 5) has the Modeling Workshop enhanced
your teaching?
a) in its pedagogy? ______
b) in improving your content knowledge in physical science and/or mathematics? _______
Please comment.
For which type of students, if any, is your Modeling Workshop learning especially suitable?
ELL?___ gifted? ____ girls? ____ boys? _____ any other groups?______________________
TECHNOLOGY:
a) For how many class periods have you used SimCalc MathWorlds software? __________
b) Briefly describe your use of Graphical Analysis software. (How often? Where did you use it:
in your classroom or in a computer lab? How effective was it for student learning?)
c) On a scale of 1 = 'little or no interest' to 5 = 'willing to do what it takes to make use of this
resource', describe your interest in having a classroom set of 8 to 10 computers. ___
30
How valuable would it be, if you had an hour every month set aside, to coordinate your math
and science courses with teachers in your school? ________ Please comment.
To what extent have you received encouragement (from your school administration) to
implement the new materials/techniques in your classroom? ________ Please comment.
What obstacles exist, to implementing your workshop learning in your classroom?
Do you need/want another Modeling Workshop? (say, for 2 weeks) ____ Why or why not?
31
ATTACHMENT A
Part I – Grant Outcomes
1. List K-12 School Districts in Project
List
Paradise Valley Unified School District (PVUSD)
Deer Valley Unified School District (DV)
Cave Creek Unified School District (CCUSD)
Scottsdale Unified School District (SUSD)
2. List High Need K-12 School
Districts Served
High need school: Greenway Middle School
High Schools:
A. in PVUSD: Horizon (1), North Canyon (1), Paradise
Valley (7)
B. in Deer Valley: Deer Valley (1), Mountain Ridge (1),
Sandra Day O'Connor (1)
C. in Cave Creek: Cactus Shadows (1)
3. List Schools Served by School
Level [identify underperforming
schools with an asterisk (*)]
Middle schools:
A. In PVUSD:
Desert Shadows (4)
Explorer (4)
* Greenway (7)
Mountain Trail (4)
Prospect (a school for troubled students) (1)
Roadrunner School (2 middle, 2 high school teachers)
Shea (2)
Sunrise (1)
Vista Verde (1)
B. In Deer Valley USD:
Anthem School (1)
Westwind (1)
C. In Cave Creek:
Desert Arroyo (2)
D. In Scottsdale:
Cocopah (2)
Desert Canyon (1)
Ingleside (2)
4. Number of K-12 Teacher
Participants Served by School Level
Count
Elementary: 0
Middle School: 35
32
5. Number of Administrator
Participants (e.g., principals) by School
Level
High School: 15
Elementary: 0
Middle School: 0
High School: 0
6. Number of K-12 Students Impacted
(e.g., for a teacher, the number of
students in his or her class; for an
administrator, the number of students in
his or her school)
Conservatively estimated as 7000 students each year.
(A typical middle school or high school teacher in these
districts teaches 160 students each year.)
7. Number of Higher Education
Faculty Working with a Grant Funded
Program or Project
Three
Part II – Contact Hours
Average Contact Hours Per Participant
80
Time Period Over Which Contact
Hours Took Place
9 months
Part III – Courses
List Courses for Credit Taught
List Courses for Professional
Development
List
PHS 534: Methods of Physical Science Teaching
cross-listed with
MTE 598: Physical Science with Math Modeling
Workshop
(same: teachers could opt out of ASU grad credit)
23
33