NC Modeling Project Mathematics and Science Partnership
Final External Evaluation Report
July, 2011
Michael N. Howard, External Evaluator
I. Introduction
The NC Modeling Project began its operation under NC DPI Math/Science Partnership funding in
spring, 2008. This final external evaluation report summarizes evidence gathered and results of evaluation
activities through the end of the project period, May 31, 2011. In keeping with the evaluation plan for the
project, formative and summative evaluation activities focused on documenting the quality and outcomes
of the project’s professional development and support activities, and gathering evidence of the degree to
which the project attained its goals and objectives. Specific evaluation questions addressed the following
areas:
1. To what extent did the project develop and deliver effective programs of professional development
and support aligned with project goals?
2. To what extent did participating teachers improve their knowledge of disciplinary content and
instructional strategies, leading to effective, standards-based instruction?
3. To what extent did science performance improve for students of participating teachers?
4. To what extent did the project developing quality partnerships to support and sustain ongoing
improvements in science teaching and learning?
In accordance with the evaluation plan, evaluation activities conducted during the three-year project
collected a variety of data through a mixture of quantitative and qualitative methods:
•
Pre/post participant content assessments to gather information on participants’ content knowledge in
science. These are “concept inventory” assessments developed by the national Modeling Project and
aligned with the content of the respective institutes.
•
Student assessments to monitor changes in student performance. Course-specific pre/post student
assessments track student performance in the specific content targeted by the project; North Carolina
EOC scores track overall student performance in the relevant science courses (however, changes in
state assessment system make these less useful in the evaluation).
•
Pre/post participant questionnaires to gather information on attitudes, perceptions, and practices
relative to teaching.
•
Participant feedback forms to document participant perceptions of the quality and effectiveness of
project activities (summer institutes, followups, coaching, etc.).
•
Observations of a sample of classrooms, professional development sessions, and other project
activities to provide additional data on implementation.
2011 Modeling Project Final External Evaluation Report, page 1
•
Focus group and individual interviews to gather feedback from a participants, project personnel, and
project partners regarding project activities and impact.
•
Examination of project-collected data and artifacts, including participant products, course materials,
staff activity log summaries and other records of project work.
The project evaluation plan specifies a non-matched quasi-experimental design, in which outcomes
for project participants are compared with outcomes reported in national research on the Modeling
Project. The national research has established the effectiveness of the Modeling Project design; the
evaluation is to gauge the extent to which this project’s implementation yields comparable results.
Statistical analysis compares the nature and magnitude of pre/post changes in the participant group to
those reported in the national literature. This report summarizes the data collected and analyzed for the
three cohorts of participants, discussing the short-term outcomes observed. Additional information is
reported about classroom implementation and results noted by early participants who have continued to
use Modeling in their instructional practice.
Summary of Results
Evaluation findings are summarized below and discussed in more detail in the remainder of this
report. Also included below is a summary of the closing section of the report.
Evaluation Findings: Modeling Project Professional Development and Support
1) The project’s professional development activities – Modeling institutes and follow-up sessions – were
well designed and implemented.
2) The project was successful in connecting participating teachers with knowledgeable facilitators and
other Modeling teachers.
3) In-school support by the Project Coaches was valued by participating teachers, who felt that it
positively influenced their practice.
4) Overall, participants reported that their professional development experiences in the Modeling Project
were effective in enhancing their knowledge, skills, and teaching practice.
Evaluation Findings: Impact on Participating Teachers
5) All groups of participating teachers significantly enhanced their knowledge of the science content
targeted in their respective institutes. Magnitude of the changes is comparable to values reported in
the national literature for seven of the eight groups over the three years of the project.
6) Participating teachers feel more comfortable with their understanding of the instructional strategies
promoted by the Modeling project.
7) Participating teachers began the project typically using “traditional” instructional strategies in their
science teaching, with some use of “standards-based” strategies. They report significant changes as a
result of Modeling professional development and support.
2011 Modeling Project Final External Evaluation Report, page 2
8) Participating teachers vary in their implementation of the Modeling Approach. Nature and degree of
implementation are related to participants’ time in the project and their comfort with the strategies.
The majority of participants intend to continue their implementation in the coming school year.
Evaluation Findings: Impact on Student Performance
9)
Results of pre/post student testing show significant growth in student understanding of the targeted
concepts. Two thirds of students in targeted courses in Year 2 demonstrated significant pre/post
gains on the assessments. Three fourths of students in targeted courses in Year 3 demonstrated
significant pre/post gains.
Evaluation Findings: Operation of the Modeling Project Partnership
10) Project partners worked well together and carried out their roles effectively.
Summary of Closing Observations
A. Summary Observations about the Modeling Project
Overall, the Modeling Project’s professional development and support activities over its three-year
period were effective in contributing to progress toward the project goals. The majority of participating
teachers were positive about their experiences in the project. Over one fourth of the participants returned
to take a second (or even third) Modeling institute. Several of the teachers are continuing their direct
participation in Modeling professional development by enrolling in another Modeling summer institute in
2011 through a newly-funded Title IIB project.
Participants have demonstrably grown in their knowledge and skills, and are putting their learning
into practice (to varying degrees). As noted earlier, the great majority of teachers expressed their intention
to continue implementing the Modeling Approach in their teaching in the coming school year. While they
may not yet have reached a level of fluent implementation desired by the project, nevertheless they have
made progress and are confident that they will continue to do so.
In summary, the evidence gathered in the Modeling Project external evaluation indicates that:

Project activities were well-designed and implemented, and participants valued their experiences.

Teacher participants demonstrated significant growth in their knowledge of the targeted science
content.

Teacher participants report enhancing their pedagogical knowledge and skills, and have begun to
implement the Modeling Approach in their science teaching.

Student assessment results and anecdotal evidence indicate a positive contribution of the
Modeling Project to student learning of the targeted content.
2011 Modeling Project Final External Evaluation Report, page 3
B. Evaluation Successes
The greatest success with respect to the Modeling Project evaluation is the quality of the data
gathered, and this is due in large measure to the support of the project staff. Project personnel were
committed to ensuring the maximum possible response rate on the various instruments and protocols used
in the evaluation, and were of great assistance in prompting participants to respond. The result was an
86% response rate overall. In addition, project personnel encouraged participants to take the evaluation
seriously as an assessment of the project, resulting in more thoughtful (and, we assume, more accurate)
responses.
C. Evaluation Caveats and Constraints
There were no major challenges in carrying out the evaluation of the Modeling Project. However, the
evaluation’s ability to report project outcomes and impacts confidently was constrained by several issues.
For the most part, these were known from the outset of the project; while they did not impact conducting
the evaluation, they must be considered in interpreting the results and forming conclusions.
1) Relatively small number of participants
2) Changes in participants’ teaching assignments
3) Determining a useable comparison group.
4) Determining degree of implementation.
5) Attributing student performance changes.
6) Relating project impact to state assessment results.
The substance of this report is presented in the six sections that follow:
Section
page
II.
Program Overview
5
III.
Evaluation Findings: Modeling project Professional Development and Support
6
IV.
Evaluation Findings: Impact on Participating Teachers
17
V.
Evaluation Findings: Impact on Student Performance
35
VI.
Evaluation Findings: Operation of the Modeling Project Partnership
38
VII.
Closing Observations
39
2011 Modeling Project Final External Evaluation Report, page 4
II. Program Overview
This section provides a brief description of the Modeling Project strategies and activities, highlighting
key components of the project. Note that this is not intended as a complete treatment of the project design
and implementation, but rather to give background important for examining results of the evaluation. The
project focuses on providing content and pedagogy enhancement to grade 8-12 teachers of Physics,
Chemistry, and Physical Science. The Modeling Project included the following components:

Three-week summer institutes. These 14-day institutes are designed by experienced Modeling
facilitators based on materials from the national project based at Arizona State University. The
institutes focus on deepening teachers’ understanding of the targeted content (physics, physical
science, or chemistry) as well as introducing them to the Modeling Approach for designing and
implementing instruction. In project year 1, teachers applied for either the physics or physical
science cohort, with a target of 20 participants in each. In project year 2, offerings were expanded
to include physics, physical science, and chemistry, targeting 20, 20, and 30 participants,
respectively. In project year 3, the project offered the physical science and chemistry institutes
along with a second level of physics, focusing on electricity and magnetism. Enrollment in the
physics II institute was limited to teachers who had completed the physics I institute.

Academic year follow-up workshops. Three two-day follow-up workshops, led by project
facilitators, provided assistance and reinforcement for implementing Modeling strategies
addressed in the summer institute, as well as attention to additional content not addressed in the
summer.

Project Coaches. The project contracted with two experienced Modeling teachers to serve as
project coaches. They provided school-based support to assist participating teachers in translating
their new knowledge and skills into practice. The coaches scheduled times to visit participants’
schools and worked with the teachers during school hours. They also provided support to
facilitators and participants during the summer institutes. The coaches focused their efforts on
current-year participants, but also provided assistance to previous participants upon request and
as their time and schedules allowed.

Modeling teachers community. Participants received access to information, resources, and
professional interaction through two web-based portals: the national Modeling Project at Arizona
State University and the American Modeling Teachers Association. These sites provided ongoing
places from which teachers could communicate with and receive lesson ideas and support from
other teachers implementing the Modeling Approach, as well as the project facilitators.
Participating teachers received numerous materials through their project activities, including curricular
resources. They also were compensated for time attending project activities outside of normal working
hours.
2011 Modeling Project Final External Evaluation Report, page 5
III. Evaluation Findings: Modeling Project Professional Development and
Support
1) The project’s professional development activities – Modeling institutes and follow-up sessions –
were well designed and implemented.
As described in the Project Overview, the Modeling project professional development consists of a
three-week summer institute, with three follow-up sessions during the academic year. Review of institute
materials and observation of institute and follow-up sessions by the external evaluator indicate that the
project’s professional development reflected characteristics of effective adult learning. Participant
responses on feedback forms also support the quality of the professional development design and
implementation. The participants’ positive perceptions are similar across all three years, indicating
consistency in the implementation of project activities.
From participant feedback forms:
The sessions involved participants in an active manner.
Session facilitators were knowledgeable about the
content addressed.
Session facilitators made the topics interesting and
understandable.
Session facilitators answered questions in ways that
were relevant to classroom situations.
Session facilitators modeled effective strategies that I
can use in teaching my students.
Session facilitators worked well together as a team in
conducting the institute activities and discussions.
The content addressed was relevant to the topics I’m
supposed to cover in the courses I teach.
The sessions built from participants’ existing
knowledge of the topics addressed.
The sessions provided opportunities to reflect on how
the information learned applies to my own situation.
The sessions provided opportunities to share ideas with
and learn from other participants as well as from the
facilitators.
2009
Mean Very
(4 pt
True
scale)
(4)
3.9
90%
3.9
85%
2010
Mean Very
(4 pt
True
scale)
(4)
3.8
81%
3.9
91%
2011
Mean Very
(4 pt
True
scale)
(4)
3.7
78%
3.9
92%
3.9
85%
3.8
86%
3.7
81%
3.7
75%
3.7
74%
3.6
71%
3.6
63%
3.8
84%
3.7
70%
–
–
3.5
64%
3.7
78%
3.8
80%
3.5
60%
3.5
62%
–
–
3.7
70%
3.5
65%
–
–
3.6
63%
3.4
54%
3.6
74%
3.7
75%
3.7
75%
Participants had few concerns about the institute design. The concerns that were voiced – particularly in
Physical Science – typically involved addressing all the topics specified in the NC Standard Course of
Study and having a clear understanding of how the institute content aligns with the SCOS.
The participating teachers were very positive about the structure and components of the summer
institutes, as seen in the following table. Again, their ratings were consistent from year to year.
2011 Modeling Project Final External Evaluation Report, page 6
2009
2010
2011
(From participant feedback forms) How much
did each component listed below contribute to
Mean Major Mean Major Mean Major
the overall effectiveness of the Summer
(4 pt contribu- (4 pt contribu- (4 pt contribuInstitute?
scale) tion
scale)
tion
scale)
tion
Activities/discussion focused on deepening your
3.5
58%
3.4
57%
3.5
60%
understanding of the chemistry/physics/physical
science content addressed.
Discussion focused on building your understanding 3.9
90%
3.8
82%
3.7
81%
of the Modeling approach for teaching and learning
science.
Modeling of instructional strategies by session
–
–
3.8
83%
3.6
68%
facilitators to illustrate their use.
Having participants lead Modeling activities with
3.5
55%
3.4
55%
3.4
56%
the group and receive feedback from session
facilitators and other participants.
Materials and resources received at the institute.
3.6
65%
3.7
70%
3.6
67%
Interacting with session facilitators.
3.5
65%
3.7
76%
3.7
79%
Interacting with and sharing ideas with other
3.9
90%
3.6
67%
3.7
73%
institute participants
According to participants’ responses, the institutes were successful in meeting their stated objectives.
Specific outcomes relative to the teachers’ content knowledge and instructional practice are discussed in
subsequent sections in this report. Overall, participants felt that the institutes had impacted their
knowledge and skills in ways pertinent to their teaching, and valued their participation:
From participant feedback forms:
There was little or
no change in my
understanding of
the content
addressed.
2009
2010
2011
–
1%
2%
I received a good
I learned some new
refresher on the
things about the
content addressed, content addressed; my
but little new
understanding is now
understanding.
a little deeper.
–
13%
3%
From participant feedback forms:
I am now more knowledgeable about the content
addressed in the institute.
I understand how the topics we addressed relate to
the Standard Course of Study.
I am now more knowledgeable about the Modeling
approach for teaching science.
I learned a lot of new
things about the content
addressed; my
understanding is much
deeper than before.
–
39%
48%
2009
Mean
Very
(4 pt
True
scale)
(4)
3.5
65%
–
46%
48%
2010
Mean Very
(4 pt
True
scale)
(4)
3.3
52%
2011
Mean Very
(4 pt
True
scale)
(4)
3.4
57%
3.0
30%
3.5
57%
3.1
30%
3.7
70%
3.7
79%
3.6
63%
2011 Modeling Project Final External Evaluation Report, page 7
I am comfortable that I can effectively implement
the Modeling approach in my classes.
I am comfortable that I can effectively use a variety
of technology in my science teaching.
I received valuable curricular materials and
resources in this institute.
I intend to use the Modeling approach strategies in
my teaching this school year.
I intend to use the Modeling curriculum units in my
teaching this school year.
I expect that my science teaching will be more
effective as a result of what I learned in this institute.
This Modeling Project Summer Institute was well
organized.
The physical arrangements for the institute were
adequate.
I feel my time was well spent in this Modeling
Project Summer Institute.
I am looking forward to participating in Modeling
Project activities next school year.
I would recommend this institute to other science
teachers.
2.9
32%
3.2
28%
3.0
26%
3.4
55%
3.2
34%
3.3
43%
3.7
79%
3.7
75%
3.7
77%
3.6
79%
3.8
83%
3.8
83%
–
–
3.6
65%
3.6
71%
–
–
3.8
78%
3.8
79%
3.7
75%
3.9
87%
3.9
87%
3.9
90%
3.8
82%
3.8
83%
3.9
85%
3.9
88%
3.8
83%
3.6
70%
3.9
88%
3.8
87%
–
–
4.0
96%
3.8
87%
From end-of-year feedback forms:
I am using things I learned at the summer institute as a
regular part of my science teaching.
The summer institute had a significant influence on my
science teaching.
Overall, I feel my time was well spent in the summer
institute.
I received valuable teaching ideas from the follow-up
sessions.
I regularly use things I learned at the follow-up sessions.
Participating in the follow-up sessions enhanced the
effectiveness of my teaching.
Overall, I feel my time in the follow-up sessions was well
spent.
2009
Mean Very
(4 pt True
scale)
(4)
3.5
64%
2010
2011
Mean Very Mean Very
(4 pt True (4 pt True
scale) (4) scale) (4)
3.6
3.7
76%
68%
3.6
76%
3.6
73%
3.8
84%
3.7
84%
3.8
87%
3.9
94%
3.5
59%
3.7
73%
3.6
69%
3.3
3.3
50%
50%
3.3
3.6
49%
65%
3.3
3.6
48%
69%
3.5
68%
3.7
77%
3.6
71%
The following are illustrative comments from participants about the institutes and follow-up sessions:
By having an intensive, residential type program I was immersed in the material. I talked about
the material over lunches, dinners, and during free time, and of course all day, every day, for the 3
2011 Modeling Project Final External Evaluation Report, page 8
weeks of class. It sparked a new passion and interest for me in a topic I was very uncomfortable
with. The facilitators were wonderful- they answered all questions thoughtfully and completely.
They guided us and taught us in a way that makes our teaching during the school year much more
effective. I encountered professionals like myself- individuals who truly desire to learn more about
the subjects we teach.
We were able to work through the modeling units in "real-time" during the summer workshop and
during follow-up sessions, I felt very comfortable immediately implementing the modeling
curriculum this school year. Initial instruction was good, but the follow-up sessions gave us an
opportunity to ask questions from experienced modelers and discuss what did and didn't work in
other classrooms, giving us even more ideas.
The discussions we had were very analytical. I tend to need to get into those discussions to really
understand conceptually. I am not one to just memorize and gain an understanding in the process.
I need the discuss piece. Also, facilitators shared ideas for creating or using attainable equipment,
especially for schools that are just building technology.
The follow-up sessions served to refresh my memory about concepts from the summer institute,
which was helpful since I did not start teaching chemistry until January. The follow-up sessions
were also great for learning what struggles and successes other teachers were having as new
modelers. I really looked forward to seeing all the professionals and friends I had connected with
during the summer institute. It gave me the opportunity to ask questions and compare student
reactions to other professionals.
The program seemed to recognize that an extended period of time is required for teachers to so
dramatically change long held teaching methods. That time was provided in two ways: 3 weeks
initially and during the course of a teaching year. I really felt both components were crucial and
well implemented in terms of physical support (housing, substitutes, stipends) and intellectual
support (sessions staffed by knowledgeable Modeling professionals excited to extend their expertise
to others.)
I received beneficial instruction and experience in content-specific instructional practices. By
placing the participants in the role of a student, it forces a teacher to re-examine misconceptions
and how to get students to think through them to resolve these misconceptions. One cannot be an
effective instructional leader unless they have experienced themselves the situations they are
placing their students in and understand these situations.
From the perspective of the participants, as well as from observations by the external evaluator, the
Modeling project summer institutes and follow-up sessions were well-designed and implemented.
Specific outcomes of the activities are discussed in the section on participant impact later in this report.
2011 Modeling Project Final External Evaluation Report, page 9
2) The project was successful in connecting participating teachers with knowledgeable facilitators
and other Modeling teachers.
A key feature of the Math Science Partnership program is developing closer relationships among
content experts and classroom teachers. Experienced Modeling teachers designed and facilitated the
Modeling institutes and led the sessions. Participants were very positive about their instructors:
From participant feedback forms:
Session facilitators were knowledgeable about the
content addressed.
Session facilitators made the topics interesting and
understandable.
Session facilitators answered questions in ways that
were relevant to classroom situations.
Session facilitators modeled effective strategies that I
can use in teaching my students.
Session facilitators worked well together as a team in
conducting the institute activities and discussions.
2009
Mean Very
(4 pt
True
scale)
(4)
3.9
85%
2010
Mean Very
(4 pt
True
scale)
(4)
3.9
91%
2011
Mean Very
(4 pt
True
scale)
(4)
3.9
92%
3.9
85%
3.8
86%
3.7
81%
3.7
75%
3.7
74%
3.6
71%
–
–
3.8
84%
3.7
70%
–
–
3.5
64%
3.7
78%
Participating teachers report greater familiarity with resource persons after their experience in the
Modeling project. The project activities were also valuable opportunities for teachers from across the state
to get to know each other as sources of lesson ideas, resources and implementation support. According to
their responses on the participant feedback forms, both objectives were achieved.
2009
From participant feedback forms: Please share your
opinion of the value of the following components to
the quality of your overall experience in the project
this year:
Working with experienced Modeling teachers who
led project activities
Working and sharing ideas with other participating
teachers in the project
2010
2011
Mean
(4 pt
scale)
3.7
Great
value
(4)
79%
Mean
(4 pt
scale)
3.7
Great
value
(4)
71%
Mean
(4 pt
scale)
3.7
Great
value
(4)
78%
3.6
67%
3.7
71%
3.7
80%
2011 Modeling Project Final External Evaluation Report, page 10
2009
From participant feedback forms: Please give your
opinion of the following statements about your
overall experience in the Modeling Project:
Overall, the Modeling Project has fostered a
valuable relationship with experienced resource
persons that I can continue to use to help my
teaching.
Overall, the Modeling Project has fostered a
valuable relationship with other physics/ physical
science teachers that I can continue to use to help
my teaching.
2010
2011
Mean
(4 pt
scale)
3.4
Very
True
(4)
56%
Mean
(4 pt
scale)
3.5
Very
True
(4)
65%
Mean
(4 pt
scale)
3.6
Very
True
(4)
74%
3.4
56%
3.4
51%
3.7
76%
Narrative comments from participants include:
The instructors and leaders for the Modeling Project have been the most accessible I have ever
had. Communications about follow-up meetings, etc have been excellent. The instructors provided
the support I needed. Each session was focused, well planned and rewarding. Their knowledge
and energy stands as one of the project's most positive characteristics.
The Modeling instructors were well prepared to deal with a group of teachers that had varying
amounts of knowledge of the topic. I cannot imagine how anyone could prepare to pull off the task
of herding a group of willing and mostly eager teachers in the direction where we all would learn
more than we thought was possible.
Our instructors were enthusiastic about the subject. They were experienced using the modeling
approach and were able to advise us of issues/pitfalls/misconceptions. They gave us practical
information that we could take DIRECTLY back to the room and put in action. We discussed the
theory and the research but we left with concrete materials to use with our students.
My instructors and modeling coaches were always and without fail available to help when I needed
them. As far as the program is concerned, it was presented in a manner that reinforced the
modeling techniques. I was able to begin developing modeling instruction skills and confidence
because of this.
It is not often that I would see other physics teachers within my school system because we only
have one per high school. I was able to learn and relearn content while interacting with teachers
from the same discipline. I gained a lot of resources and strategies that I can take directly to my
classroom. I also became part of a network of science teachers that I stay connected to and share
experiences with.
Having a summer to spend with the same participants helped to form a solid group of teachers that
I feel I could contact for help. The opportunity to share with other physics teachers was
tremendous. I got so many great ideas from other teachers.
The relationships that I built during the year with the participants were fulfilling and helped me
grow as a science educator. Talking with other teachers who were also implementing the modeling
approach in their classes allow for comparison of how we felt things were going. Discussions
occurred that involved the pacing of instruction, the resistance from administration and/or other
teachers, how students were responding, etc. These conversations allowed us to compare "notes"
and to discuss what was good and what was bad concerning modeling as we were using the
2011 Modeling Project Final External Evaluation Report, page 11
approach in our classes.
The project was successful in connecting participants to other persons who could serve as resources –
facilitators and other teacher participants. On an individual basis, these connections can continue to
provide benefits after the project is concluded.
An ongoing area of concern in the project was the lack of substantive involvement of science faculty
from NCSU or other higher education institutions. Normally, an MSP project focuses on connecting
participating teachers with such IHE faculty as resources for subject area expertise. In this particular case,
however, the expertise in Modeling lies not with IHE faculty in NC, but with experienced teachers in the
state who have undergone extensive development through the national project at Arizona State
University. These individuals are highly qualified – two hold doctorates – so their use in leadership roles
is appropriate. One NCSU faculty member assisted with some follow-up sessions for the chemistry
institute, but was not considered a formal instructor for the project. While the project sought to develop
opportunities to engage faculty in a substantive manner, for the most part faculty involvement in the
project was minimal. The quality of project activities and outcomes did not appear to be impacted by this
situation, but it should be noted as part of the project context.
3) In-school support by the Project Coaches was valued by participating teachers, who felt that it
positively influenced their practice.
The Project Coaches were a critical component of the Modeling project. The coaches provided
regularly scheduled in-school support to the participating teachers during each academic year. The project
plan was for the coaches to visit all participants at some level, concentrating on those requesting or
needing their assistance and support. This plan was subsequently modified in the school year following
the summer institutes when some participants were given teaching assignments that did not include the
course targeted by the Modeling institute they attended. Reassignment of teachers to different classes was
an issue each year of the project, but especially in the first year. Coaching support was available to these
teachers, but often of a different nature than the coaching focused on using the materials from the summer
institute and follow-up sessions.
On the end-of-year feedback survey, most participants indicated that they saw their coach an
appropriate amount. Overall, about one fourth wished for more contact.
From participant feedback forms: Mark the statement below that best describes your opinion about the
frequency of meeting with the project coach:
Not enough – I wish we
About right – we met
More than enough – we
could have worked
often enough to do what
could have had fewer
together more often
was needed
meetings
2009
30%
70%
0%
2010
24%
73%
3%
2011
19%
79%
2%
2011 Modeling Project Final External Evaluation Report, page 12
The coaches worked on a variety of issues with the teachers, according to their interests or needs. As
seen in the table below, the nature of the coaches’ interactions was fairly consistent across the year,
except for an increase in attention to using technology.
2009
From participant feedback forms: Different teachers
worked on different things with the project coach. How
much time and attention did you and the coach give to
each of the following areas during your work together?
Learning more about the Modeling Approach
Working on particular teaching strategies (inquiry,
questioning, grouping, whiteboards, etc.)
Using technology during instruction
Planning units or lessons that use the Modeling
Approach
General assistance with science teaching issues and
questions
2010
2011
Mean
(3 pt
scale)
2.2
2.4
A
lot
(3)
30%
43%
Mean
(3 pt
scale)
2.2
2.4
A
lot
(3)
36%
48%
Mean
(3 pt
scale)
2.0
2.3
A
lot
(3)
22%
41%
1.8
1.8
22%
13%
2.0
1.9
28%
25%
2.2
1.8
39%
14%
2.7
70%
2.4
57%
2.4
47%
Overall, participants were extremely positive about the value of their work with their project coach, as
seen in the following table and narrative comments:
From participant feedback forms
I received valuable teaching ideas from the project coach.
I regularly use things I learned in working with the
project coach.
The support from the project coach enhanced the
effectiveness of my teaching.
Overall, I feel my time working with the project coach
has been well spent.
2009
Mean Very
(4 pt True
scale)
(4)
3.6
70%
3.3
52%
2010
2011
Mean Very Mean Very
(4 pt True (4 pt True
scale) (4) scale) (4)
3.6
70%
3.6
69%
3.4
53%
3.4
55%
3.6
65%
3.5
62%
3.5
63%
3.7
74%
3.7
73%
3.6
73%
Examples of participants’ narrative comments include the following:
________ made herself very available to me during the year. I really connected with her, and I felt
like she was only an email or phone call away. Just knowing that help is available is a huge
reassurance. When she visited my classroom she helped explain some things to my students that I
was struggling with, and that was wonderful seeing her teach. It was very helpful for me to see how
she handled the student's responses and what she asked next.
_________ was extremely supportive and available whenever I needed her. She has such an indepth knowledge of both the subject and the modeling approach that she could and did jump in to
turn the discussion more to a quality discussion for the students, thereby demonstrating to me how
to model physics for my students. She provided valuable ideas about calculator use when computers
were not available. She also circulated frequently throughout the class period and assisted students
2011 Modeling Project Final External Evaluation Report, page 13
in need. Her guidance has been invaluable to me.
_________ tailored her assistance to my specific needs and circumstances. She became familiar
with my particular teaching constraints (e.g., class length, room space, available apparatus) and
helped me to make the most of teaching within these restrictions. She spent considerable time
listening and observing before she offered advice. She joined me in working directly with the
students, such that I was able to observe in action one who is proficient with the modeling process.
She made herself as available to me as I desired - was always here when I needed her.
My coach was great about offering ideas to improve my Modeling instruction. She often jumped in
and helped students, allowing me to shadow her and watch a master at work. I found that most
helpful. The workshops are one thing, but seeing a good Modeling teacher interact with real
students was extremely beneficial. Simply stated, my modeling coach provided me with the support,
encouragement and inspiration to keep going. ________'s visits along with well timed follow-up
sessions gave me the strength to keep pushing forward especially as the school year progressed and
energy levels waned.
The project coach was very helpful. I really appreciated having someone knowledgeable about
Modeling observe my classes and make sure I was on the right track and then give me suggestions if
I was not. When students worked, she would circulate and help them. Her enthusiasm, energy, and
knowledge affected the students. I think it made them see how much time and effort was put into
developing and implementing the Modeling Approach. I convinced another teacher at my school to
teach chemistry using the Modeling Approach and she helped him out as well. He is hoping to
attend the Modeling Workshop this summer. Whenever I needed resources or made requests, my
project coach got them to me very quickly. Her encouragement was also very valuable. If I was
having difficulty with a topic, she helped me figure it out with enthusiasm.
Teachers were appreciative of the coaches’ knowledge and experience, indicating that giving
implementation support, sharing good advice and ideas, and serving as a “sounding board” and
“supportive ear” were especially helpful.
4) Overall, participants reported that their professional development experiences in the Modeling
Project were effective in enhancing their knowledge, skills, and teaching practice.
The project’s institutes, follow-up sessions, and school-based coaching were designed to address
multiple objectives, including enhancement of content and pedagogy, familiarity with resources, and
ability to design and implement effective science lessons using the Modeling Approach. Responses on the
participant feedback questionnaire indicate that participating teachers valued their experiences in the
project and felt the project had an impact in each of its targeted areas. As the table below shows, the
perceived impact of the project generally increased from the first year to the third. In part, this may be due
to the presence of teachers in years 2 and 3 who were attending their second (or even third) Modeling
institute.
2011 Modeling Project Final External Evaluation Report, page 14
From end-of-year participant feedback forms:
Overall, the Modeling Project has had a significant
influence on my science teaching.
Overall, the Modeling Project has had a significant
influence on my understanding of science content.
Overall, the Modeling Project has had a significant
influence on my use of technology in my science
teaching.
Overall, the Modeling Project has had a significant
influence on my ability to plan and implement
effective science units and lessons.
Overall, the Modeling Project has had a significant
influence on my awareness of quality curricular
resources to use in my science teaching.
Overall, the Modeling Project has fostered a
valuable relationship with experienced resource
persons that I can continue to use to help my
teaching.
Overall, the Modeling Project has fostered a
valuable relationship with other physics/ physical
science/chemistry teachers that I can continue to
use to help my teaching.
2009
Mean
Very
(4 pt
True
scale)
(4)
3.5
64%
2010
Mean Very
(4 pt
True
scale)
(4)
3.7
76%
2011
Mean Very
(4 pt True
scale)
(4)
3.9
90%
3.0
42%
3.1
44%
3.5
70%
2.5
16%
3.0
30%
3.3
56%
3.0
32%
3.3
41%
3.5
66%
3.2
48%
3.3
54%
3.7
76%
3.4
56%
3.5
65%
3.6
74%
3.4
56%
3.4
51%
3.7
76%
Overall, teachers were very positive about the value of their experience in the Modeling project
overall, as well as its various components:
2009
2010
2011
Very satisfied
76%
81%
86%
Satisfied
16%
19%
12%
Mixed feelings
8%
0%
2%
Dissatisfied
0%
0%
0%
2009
2010
From end-of-year participant feedback forms:
Please share your opinion of the value of the
following components to the quality of your overall
experience in the project this year:
3-week Summer Institute
Two-day follow-up workshops during the school
year
In-school work with the project coach
Materials and resources received through the
project
Using the project website and online resources to
interact with others and gather ideas
Very dissatisfied
0%
0%
0%
2011
Mean
(4 pt
scale)
3.8
3.4
Great
value
(4)
88%
63%
Mean
(4 pt
scale)
4.0
3.6
Great
value
(4)
95%
66%
Mean
(4 pt
scale)
4.0
3.6
Great
value
(4)
96%
74%
3.4
3.5
57%
65%
3.5
3.8
66%
82%
3.4
3.9
64%
90%
3.1
42%
3.2
35%
3.2
40%
2011 Modeling Project Final External Evaluation Report, page 15
From end-of-year participant feedback forms:
I feel my time has been well spent as a Modeling
Project participant.
I would recommend the Modeling Project to other
science teachers.
2009
Mean
Very
(4 pt
True
scale)
(4)
3.8
84%
3.8
88%
2010
Mean Very
(4 pt
True
scale)
(4)
3.9
89%
3.9
92%
2011
Mean Very
(4 pt True
scale)
(4)
3.9
96%
3.9
96%
Participants’ narrative comments support their positive view of their Modeling Project experience, as
seen in the following illustrative examples:
The professional development for the Modeling Project was well-organized and professional. It was
the first workshop I went to that made me feel like a professional teacher. And I have been teaching
for 10 years. The support after the initial workshop is excellent. If I ever needed a answer to a quick
question, the answer was an email away. If I needed someone here to help, they came quickly and
helped one-on-one. And the modeling listserv keeps me up-to-date on current issues concerning
other modelers.
This is the only professional development I've had that not only is useful but has fundamentally
changed the way I teach as well as my view of the science classroom. I think the Modeling Project
did a great job of providing support and professional development throughout my first year using
modeling. It was wonderful to go back and talk with modelers from around the state. The coaches
and length of the program assisted in implementation.
The modeling workshop was a great experience. I learned a lot not only about Modeling but about
Chemistry. There were many times during the initial 3 week meeting and the three 2-day meetings
when I would say "Why didn't I think of it that way?" or "How did I never see that before?” My
students showed a much greater understanding of things like density or gas laws through the use of
things I learned from the course. The labs and use of LabQuests has brought my class up a notch in
the use of technology. I and one of my co-workers, have talked another member of our department
in to attending this summer and I really think it will be good for him, both in learning how to teach
using modeling as well as the professional lift you get from being around great people who are as
excited to be learning as yourself.
There is no doubt in my mind that this Modeling workshop has been the best and most useful
workshop experience I have ever had! The ongoing level of support has been fantastic. If I feel that I
need help - either materials, data, or any kind of feedback- the modeling staff has been there for all of
us. I feel a very strong bond of communal support from my classmates as well. I have never yet failed
to get a rapid and timely response from everyone concerned with the modeling project to emails or
through the Moodle site. Thanks for the help!
The project did an excellent job of introducing me to the modeling program, teaching us how to
implement modeling instruction, allowing us to practice instructing in the modeling techniques, and
providing us with coaches to help us through direct classroom observation, e-mail communication,
and follow-up weekends.
2011 Modeling Project Final External Evaluation Report, page 16
IV. Evaluation Findings: Impact on Participating Teachers
5) All groups of participating teachers significantly enhanced their knowledge of the science
content targeted in their respective institutes. Magnitude of the changes is comparable to values
reported in the national literature for seven of the eight groups over the three years of the
project.
Participants’ growth in understanding of the targeted science content was measured using nationally
developed and validated “concept inventories,” developed by the national Modeling Project at Arizona
State University and American Chemical Society. The Force Concept Inventory (physics), Conceptual
Survey of Electricity and Magnetism (physics II), Physical Science Concept Inventory (physical science),
and Assessment of Basic Chemistry Concepts (chemistry) are aligned with the content of their respective
institutes. A pre-assessment was administered on the first day of each summer institute. The postassessment was administered during the final follow-up session the following spring. The tables below
show the results of the pre/post assessments:
Physics (Force Concept Inventory)
# (%)
pre
post
with both
mean
mean
pre and
% correct % correct
post data
(sd)
(sd)
200816 of 20
70.4%
80.0%
2009
(80%)
(.274)
(.187)
200917 of 19
69.6%
79.4%
2010
(89%)
(.287)
(.294)
mean
prepost
gain
19.6%
p value
(paired
t-test)
0.0305
9.8%
0.0010
Physics II (Conceptual Survey of Electricity and Magnetism)
# (%)
pre
post
mean
with both
mean
mean
prep value
pre and
% correct % correct
post
(paired
post data
(sd)
(sd)
gain
t-test)
201020 of 23
58.8%
70.0%
11.2% 0.0004
2011
(87%)
(.280)
(.219)
Physical Science (Physical Science Concept Inventory)
# (%)
pre
post
mean
with both
mean
mean
prepre and
% correct % correct
post
post data
(sd)
(sd)
gain
200814 of 18
73.1%
79.1%
6.0%
2009
(78%)
(.179)
(.198)
200917 of 19
76.5%
85.4%
8.9%
2010
(89%)
(.179)
(.122)
201015 of 21
83.5%
89.6%
6.1%
2011
(71%)
(.176)
(.115)
p value
(paired
t-test)
0.1156
normalized
average
Effect
gain
size
<g>
.42
.32
.34
.32
# (%) of teachers
with significant
pre/post gain
(> 1/3 sd)
7
(44%)
9
(53%)
normalized
average
Effect
gain
size
<g>
.45
.27
# (%) of teachers
with significant
pre/post gain
(> 1/3 sd)
11
(55%)
normalized
average
Effect
gain
size
<g>
.52
.22
# (%) of teachers
with significant
pre/post gain
(> 1/3 sd)
7
(50%)
10
(59%)
7
(47%)
0.0011
.54
.41
0.0290
.42
.37
2011 Modeling Project Final External Evaluation Report, page 17
Chemistry (Assessment of Basic Chemistry Concepts)
# (%)
pre
post
mean
with both
mean
mean
prep value
pre and
% correct % correct
post
(paired
post data
(sd)
(sd)
gain
t-test)
200927 of 30
72.9%
84.9%
12.0% <0.0001
2010
(90%)
(.171)
(.145)
201014 of 22
79.1%
89.5%
10.4% 0.0160
2011
(64%)
(.184)
(.115)
normalized
average
Effect
gain
size
<g>
.70
.44
.71
.49
# (%) of teachers
with significant
pre/post gain
(> 1/3 sd)
16
(59%)
6
(43%)
Overall, the pre/post results above support the following conclusions:

Using the USDE standard for significance – paired t-test p<0.15 – all eight groups of teachers
demonstrated statistically significant pre/post mean gains. Moreover, effect sizes indicate the
pre/post gains reflect meaningful learning in each of the institutes, with the largest effect sizes in
the chemistry institutes.

Over the three years of the project, complete pre/post assessment data were collected for 81.4%
of the total participants (140 of 172). Using the USDE standard for individual gains – gain greater
than one-third standard deviation – 52.1% of these teachers (73 out of 140) posted a significant
increase in their performance. It should also be noted that the pre-test performance of 33 of the
140 teachers (23.6%) was high enough to make it impossible to meet the criterion for significant
gain, even with a perfect score on the post-test. Taking this factor into account, the proportion of
teachers making significant individual gains rises to 68.2% (73 out of 107).

The normalized average gain is considered by developers of the Modeling program as a more
valid calculation than a raw pre/post difference. National research on the Modeling Physics
program showed typical normalized gains in the range of .3 to .6. The normalized gains shown by
most of the eight institute groups fall into this range, and are therefore consistent with other sites.
An exception is the 2008-09 physical science institute, which showed improvement in the
subsequent two years.
The majority of teachers participating in the project perceived that their content knowledge had
grown as a result, as indicated by their responses on the end-of-year feedback survey:
(From end-of-year participant feedback forms) Project objective: Understand key physics/physical
science /chemistry concepts at a depth needed to effectively teach the content specified in the Standard
Course of Study.
2009
2010
2011
From end-of-year participant feedback forms:
Mean
Mean
Mean
(5 pt
%
(5 pt
%
(5 pt
%
scale)
4-5
scale)
4-5
scale)
4-5
Status before participating in the Modeling project
3.5
56%
3.8
59%
3.6
56%
Status as a result of participating in the Modeling
4.6*
100%
4.3* 81%
4.5*
96%
project
* Difference significant at p<0.05
2011 Modeling Project Final External Evaluation Report, page 18
From end-of-year participant feedback forms:
Overall, the Modeling Project has had a significant
influence on my understanding of science content.
2009
Mean
Very
(4 pt
True
scale)
(4)
3.0
42%
2010
Mean Very
(4 pt
True
scale)
(4)
3.1
44%
2011
Mean Very
(4 pt True
scale)
(4)
3.5
70%
Participant comments also indicate that most feel their content knowledge was enhanced by their
participation in the project, although in different ways for different participants:
Upon completion of the Modeling workshops, my physics knowledge increased greatly. Before I
participated in these workshops my knowledge was very basic; I only knew the bare-minimum in
order to teach my students. I have now internalized the concepts in a way that when I teach the
topics I am confident that I can answer most questions my students have.
Fortunately, my Chemistry content for teaching high school chemistry was mostly solid. However,
there were many areas that I knew the "what" but not the "why". Through the modeling workshop I
have a better grasp of the "why" which allows me answer student questions better. It also helps me
to scaffold and question better in my classes. When I can ask better questions and get students
thinking at a more in depth level, student understanding improves and that is exactly what I can do
because of the modeling workshops. As I practice, it should be become better yet.
Through the Modeling Project my understanding of why things happen has become greater. I
believed prior to the course that I had a notable grasp of the concepts and I was again regularly
excited and disappointed to find that my understanding was far too superficial. In short, the
training had a profound impact on my level of content knowledge and depth of conceptualization.
As one who initially studied Physics by attending lectures and memorizing laws and equations, I
knew many of the basics of this science but my understanding was rather shallow. I also thought
that I "knew Physics", at least to the extent of the principles with which I was comfortable; I did not
know what I did not know. Over the last two years, as I participated in the Modeling Project, I
discovered, first, that my understanding of even the most basic Physics concepts was limited. After
that, as I determined that I still had much to learn, the process of the Modeling Project helped to
strengthen and deepen my understanding of Physics concepts in every area we studied: mechanics,
energy, electrical charge. Now, even though I realize that my greater content knowledge is still
limited, I am determined to continue studying Physics, with the hope that, as my own understanding
improves, I'll be better equipped to lead my students to good understanding of Physics, too.
I was not an education major in college. I spent most of my time in the science lab and in the library
memorizing genus and species names of plants. That being said I thought I knew a lot about science
coming into my classroom two years ago as a lateral entry teacher fresh out of college. However,
not until I went to the modeling institute did I gain a deeper understanding of the Chemistry content.
I have taken 4 years of chemistry and I finally begin to understand concepts at a deeper level this
past summer. This depth of understanding translated into higher achievement levels with my kids
this past year.
Physics is a class I took in college just to complete the requirement. The unit was placed back in the
7th grade Standard Course of Study a few years ago and I have struggled to get up to speed;
however the Modeling Project made the concepts clear and meaningful for the teacher. This
enabled me to teach with more confidence. I feel that my own eyes have been opened up due to the
way the content was presented in the workshop. Rather than just knowing the content, I feel that I
2011 Modeling Project Final External Evaluation Report, page 19
have a deeper understanding of the how and why of various things involving the curriculum.
I had a degree in Chemistry before I started teaching, so the content knowledge didn't change a lot.
I felt confident with the chemistry content before the workshop; however, I do appreciate the way
the modeling method has caused me to think about that content and how I present it differently. The
modeling project has encouraged me to think about content presentation in a more connected and
narrative manner.
I knew my content very well before taking the institute, so I have not learned very much additional
content. The Modeling approach has done more in terms of recognizing students' misconceptions
than increasing my depth of content knowledge. I would say that it has changed the nature of my
instruction more than anything.
Overall, the evidence indicates that the Modeling project has been successful in enhancing the content
knowledge of the majority of its participating teachers.
6) Participating teachers feel more comfortable with their understanding of the instructional
strategies promoted by the Modeling project.
Implementation of the instructional strategies that comprise the Modeling approach is dependent, in
part, on teachers’ perceptions of being well prepared to use the desired strategies. The results from the
teacher pre/post questionnaires indicate that – other than strategies generally associated with process
skills in laboratory activities – most participants did not feel particularly well prepared to do so prior to
the Modeling project. Their perceived preparation had improved significantly at the end of the project
year. The table below gives values of “composite” indicators, derived from aggregating groups of
questionnaire items. Mean values are on a 100-point scale; a value of at least 70 is generally seen as a
desirable level. Significant improvement is seen in all four composite indicators. In addition, linear
regression modeling revealed that, controlling for the pre-questionnaire value, participating in the project
for more than one year made a significant contribution to the post-value of three composite indicators.
That is, participating for multiple years added from 5.4 to 7.4 points to the predicted value of the
indicator’s post-value.
Pre-Questionnaire
From pre/post questionnaires:
Composite indicators
Mean % at or
(sd) above 70
Perceived preparation to support
65.93
43%
development of science process skills
(20.28)
Perceived preparation to use a variety of 54.99
26%
standards-based instructional strategies (19.86)
Perceived preparation to plan effective
42.47
13%
lessons aligned with Modeling approach (20.51)
Confidence regarding aspects of
58.25
24%
investigative science instruction
(15.80)
Post-Questionnaire
Mean % at or
(sd) above 70
79.09
68%
(17.63)
72.98
60%
(16.45)
69.20
50%
(18.69)
67.31
45%
(15.44)
Multiple
institute
p*
contrib.
< 0.0001
n.s.
< 0.0001
5.36
< 0.0001
7.40
< 0.0001
6.90
* p values computed by paired sample t-test (two-tailed), with n=114
“Multiple institute contribution” = increase in post-score (controlling for pre-score) for participating in the project
for more than one year; computed by linear regression.
2011 Modeling Project Final External Evaluation Report, page 20
The table below gives pre/post results for the teachers’ perceived preparation to use particular strategies
listed in the questionnaire. For most of the strategies, their perceived preparation significantly improved
after their project participation.
Pre-Questionnaire
From pre/post questionnaires: Many science teachers feel
better prepared to guide and help students develop some
science processes than others. How well prepared do you feel
to provide guidance in the following?
Generating testable hypotheses and designing appropriate
scientific experiments to investigate them
Describing, graphing, and interpreting data
Analyzing and evaluating information using scientific
reasoning
Using empirical data to validate models or test hypotheses
Representing scientific ideas or relationships in multiple
formats (mathematical, graphical, diagrams, etc.)
Communicating scientific ideas both orally and in writing
Post-Questionnaire
Mean % “well
(4 pt prepared”
scale)
(4)
2.89
22%
Mean % “well
(4 pt prepared”
scale)
(4)
3.28*
40%
3.19
3.10
34%
26%
3.54*
3.50*
59%
58%
2.79
2.94
17%
26%
3.27*
3.28*
46%
40%
2.96
26%
3.37*
46%
2.61
14%
3.33*
45%
3.05
33%
3.43*
51%
2.89
2.90
27%
23%
3.36*
3.25*
44%
41%
2.90
2.43
22%
11%
3.27*
2.94*
46%
28%
2.56
11%
3.24*
38%
2.48
2.67
2.41
14%
19%
14%
3.09*
3.19*
3.02*
32%
39%
32%
2.24
14%
2.96*
33%
2.47
14%
3.01*
28%
2.23
11%
2.93*
25%
From pre/post questionnaires: Many science teachers feel
better prepared to use some teaching strategies than others.
How well-prepared do you feel to do each of the following
things in the science courses you teach?
Lead a class of students using investigative, inquiry-oriented
activities.
Manage students and materials to create an orderly
environment for lab activities
Have students work effectively in cooperative learning groups.
Use questioning to elicit students’ thinking or probe for
misconceptions.
Use questioning to assess students’ conceptual understanding.
Use writing tasks as a tool to foster students’ scientific
thinking.
Use student presentations as a means to monitor what they are
learning.
Use performance tasks for assessment purposes.
Use Internet resources to teach science concepts.
Display and analyze data using computers or graphing
calculators.
Collect data using scientific probes connected to computers or
calculators.
From pre/post questionnaires: Many science teachers feel
better prepared to engage in some planning/ preparation
activities than others. How well prepared do you feel to do
each of the following things?
Develop science units around key “big ideas” rather than
traditional topics
Organize lessons in a unit as a “learning cycle” – using
2011 Modeling Project Final External Evaluation Report, page 21
investigative activities to develop, elaborate, and apply a
targeted concept.
Identify misconceptions that students are likely to have about
a science concept.
Design science lessons that promote student exploration or
investigation.
Build specific formative assessments into appropriate places in
a science lesson.
2.63
12%
3.04*
30%
2.47
15%
3.16*
36%
2.48
11%
3.09*
32%
* Difference significant at p<0.01
The questionnaire results suggest that the teachers needed and received additional knowledge about
and experience with the strategies underlying the Modeling Approach. Furthermore, responses to the endof-year feedback survey indicate that participants feel their pedagogical repertoire has grown as a result of
their project experiences thus far. The teachers were asked to use a 1 (low) – 5 (high) scale to describe
their status prior to the project and their current status. The table below contains mean responses for
participants with one year and two or more years experience in the project.
From end-of-year participant feedback forms:
Understand characteristics of the Modeling approach, including
how it is similar to and different from “inquiry” science
teaching.
Understand current research on student thinking and student
learning of science concepts, and the implications of the
research for effective science instruction.
Be familiar and comfortable with instructional strategies used in
the Modeling approach, including how teacher and student roles
change at different parts of the Modeling Cycle.
Be familiar and comfortable with the implications of the
Modeling approach for how science units and lessons should be
designed.
Be familiar and comfortable with using technology to collect,
organize, display, and analyze data in laboratory activities.
Be familiar and comfortable with using appropriate computer
software and the Internet to illustrate, explore, and apply science
concepts.
Learn common student misconceptions about physics/physical
science/chemistry content and how to identify them in my
students.
Learn strategies to promote student discourse that probes their
thinking and monitors their understanding.
Learn strategies to foster collaborative student work and to
assess student learning in collaborative groups.
1 year exp.
Mean Mean
Prior Current
2.1
4.1*
2+ years exp.
Mean Mean
Prior Current
3.3
4.4*
2.8
3.9*
3.4
4.1*
2.1
4.0*
3.1
4.4*
2.0
4.0*
3.1
4.3*
3.1
4.0*
3.2
4.3*
3.3
4.0*
3.2
4.1*
2.8
4.0*
3.1
4.2*
2.6
3.9*
3.1
4.2*
3.0
4.1*
3.3
4.3*
* Difference significant at p<0.01
2011 Modeling Project Final External Evaluation Report, page 22
As might be expected, the prior status ratings on items related to the Modeling Approach were higher for
“veteran” participants. Both groups reported significant mean growth on all items.
Overall, participants left the project activities with greater perceived understanding of the strategies
utilized, and greater confidence that they can apply them effectively in their teaching. Thus, they would
be expected to be more likely to try implementing the strategies with their students. As discussed in the
next section, this was indeed the case.
7) Participating teachers began the project typically using “traditional” instructional strategies in
their science teaching, with some use of “standards-based” strategies. They report significant
changes as a result of Modeling professional development and support.
Participants were asked about their teaching practices on the pre/post teacher questionnaires. They
reported their frequency of using a variety of instructional strategies – some associated with a
“traditional” approach to instruction, some associated with a “standards-based” approach – as well as
frequency of assessment and technology strategies. As seen in the table below, the composite indicators
show a preponderance of using “traditional” teaching strategies, with infrequent use of standards-based
strategies or technology. (A level of 70 is indicative of regular use.) There was significant change on the
post-questionnaire. Teachers significantly decreased their frequency of using traditional strategies and
significantly increased their frequency of using more investigative and technology-based strategies. In
addition, linear regression modeling revealed that, controlling for the pre-questionnaire value,
participating in the project for more than one year made a significant contribution to the post-value of
three composite indicators. That is, participating for multiple years added from -4.17 to 6.30 points to the
predicted value of the indicator’s post-value.
Pre-Questionnaire
From pre/post questionnaires:
Composite indicators
Mean % at or
(sd) above 70
Frequency of using “traditional”
65.24
49%
teaching strategies
(14.40)
Frequency of using standards-based
55.24
20%
teaching strategies
(14.06)
Frequency of using strategies for
62.59
34%
formative assessment during instruction (16.20)
Frequency of using technology in
51.21
15%
science teaching
(18.72)
Post-Questionnaire
Mean % at or
(sd) above 70
57.13
21%
(13.40)
67.28
46%
(13.16)
70.57
51%
(16.36)
60.98
25%
(15.57)
Multiple
institute
p*
contrib.
< 0.0001
-4.17
< 0.0001
4.94
< 0.0001
n.s.
< 0.0001
6.30
* p values computed by paired sample t-test (two-tailed), with n=113
“Multiple institute contribution” = increase in post-score (controlling for pre-score) for participating in the project
for more than one year; computed by linear regression.
Examining pre/post results for individual questionnaire items reveals that participants report significant
reduction in their use of most strategies associated with “traditional” instruction:
2011 Modeling Project Final External Evaluation Report, page 23
“TRADITIONAL” STRATEGIES
From pre/post questionnaires: In your classes,
how often do you . . .
Introduce content through formal presentations.
Demonstrate a science-related principle or
procedure.
Use short answer tests (multiple choice, fill in the
blank, problem sets, etc.).
From pre/post questionnaires: In your classes,
how often do students . . .
Read from the textbook or other informational
materials in class.
Answer questions or problems from the textbook
or a worksheet.
Review or check homework or worksheet
assignments in class.
Engage in a laboratory activity after a teacher
presentation or class discussion, to illustrate or
apply the concept being studied.
Use a calculator to assist with computations.
Pre-Questionnaire
Weekly
Mean
basis
(5 pt scale)
(4-5)
3.82
72%
3.74
67%
Post-Questionnaire
Weekly
Mean
basis
(5 pt scale)
(4-5)
3.20*
38%
3.62*
61%
3.89
65%
3.60*
54%
2.89
33%
2.54*
19%
3.67
67%
3.41*
50%
3.59
61%
3.60
61%
3.67
62%
3.03*
34%
4.20
83%
4.23
86%
* Difference significant at p<0.05
With the exception of cooperative grouping, real-world connections, requiring evidence, and working
with data, teachers initially reported infrequent use of specific strategies associated with standards-based
instruction and the Modeling approach. All items but one showed significant pre/post change in reported
frequency:
“STANDARDS-BASED” STRATEGIES
From pre/post questionnaires: In your classes,
how often do you . . .
Organize students into collaborative groups to
discuss ideas or to work on an activity or
assignment.
Pose questions to students that are open-ended,
allowing multiple responses.
Use real-world contexts to introduce/develop
science concepts.
Require students to supply evidence to support
their claims.
Encourage students to explain concepts to one
another.
Encourage students to come up with alternative
explanations.
Pre-Questionnaire
Weekly
Mean
basis
(5 pt scale)
(4-5)
3.95
77%
Post-Questionnaire
Weekly
Mean
basis
(5 pt scale)
(4-5)
4.42*
94%
3.53
53%
4.22*
83%
4.04
77%
4.27*
88%
3.71
64%
4.25*
86%
3.70
66%
4.36*
94%
3.29
41%
3.93*
74%
2011 Modeling Project Final External Evaluation Report, page 24
From pre/post questionnaires: In your classes,
how often do students . . .
Engage in a laboratory activity to explore a
concept before discussing it in a formal manner.
Design their own scientific investigation to
answer a question or solve a problem.
Record, represent, and/or analyze data.
Make a presentation or demonstration for the
class.
Prepare a written lab report or other description of
what they found in a science investigation.
Write reflections about their science learning in a
notebook or journal
2.84
29%
3.71*
65%
2.23
5%
2.92*
27%
3.73
2.82
62%
25%
4.01*
3.65*
77%
58%
3.11
34%
3.25
41%
2.27
18%
2.80*
32%
* Difference significant at p<0.01
Similarly, participants showed significant pre/post change on their reported use of strategies
associated with assessment and use of technology:
ASSESSMENT STRATEGIES
From pre/post questionnaires: In your classes,
how often do you . . .
Use informal questioning to find out what
students know before a unit.
Use pre-planned questions or formative
assessment tasks to identify student
misconceptions prior to addressing a concept.
Use informal questioning to check student
understanding during a lesson.
Use pre-planned questions to check student
understanding during a lesson.
Use specific tasks or assignments in a lesson as
“check points” to monitor student understanding
during the lesson.
Embed assessment into regular class activities.
Use short answer tests (multiple choice, fill in the
blank, problem sets, etc.).
Use tests requiring open-ended responses
(description, justification, etc.).
Use a performance task or laboratory activity as a
formal assessment.
Use student notebooks or journals to assess
student learning.
Pre-Questionnaire
Weekly
Mean
basis
(5 pt scale)
(4-5)
3.50
52%
Post-Questionnaire
Weekly
Mean
basis
(5 pt scale)
(4-5)
3.85*
68%
2.78
22%
3.25*
40%
4.13
79%
4.34*
87%
3.43
50%
3.73*
60%
3.63
58%
3.93*
76%
3.56
3.89
57%
65%
3.85*
3.60*
68%
54%
3.25
42%
3.56*
55%
3.15
40%
3.18
40%
2.65
27%
2.96*
33%
* Difference significant at p<0.05
2011 Modeling Project Final External Evaluation Report, page 25
TECHNOLOGY STRATEGIES
From pre/post questionnaires: In your classes,
how often do students . . .
Use a calculator to assist with computations.
Use a computer or graphing calculator to display
or analyze data.
Gather data using scientific probes attached to a
computer or graphing calculator.
Access Internet resources for information,
activities, simulations, etc. about a science
concept.
Pre-Questionnaire
Weekly
Mean
basis
(5 pt scale)
(4-5)
4.20
83%
2.75
33%
Post-Questionnaire
Weekly
Mean
basis
(5 pt scale)
(4-5)
4.23
86%
3.31*
46%
2.15
14%
2.81*
25%
3.09
33%
3.41*
48%
* Difference significant at p<0.01
Furthermore, when asked on the questionnaires about their use of various types of lesson designs, the
results indicate an initial inclination toward using activities that confirm a previously presented idea.
Reported baseline use of the most inquiry-oriented kinds of lessons was infrequent. On the postquestionnaire, there was significant decrease in reported use of confirmatory lessons and increase in use
of inquiry-oriented lessons.
From pre/post questionnaires: For each description below, indicate
the extent to which it is similar to the kinds of science lessons you
typically use in your classroom (the lesson activities described may
extend over more than one class session).
% responding
“My lessons
are rarely if
ever like
this”
Pre
Post
% responding
“My lessons
are frequently
similar to
this”
Pre
Post
*(Textbook-centered) Students read a section of the text, then work
individually or together on a set of assigned questions or problems. The
teacher goes over the questions, either explaining the correct answers or
calling on students to explain.
62%
76%
5%
4%
*(Confirming activity) The teacher presents information on a concept.
Students then do an activity that illustrates the concept, followed by work on
questions to reinforce their learning. The teacher goes over the questions,
either explaining the correct answers or calling on students to explain.
7%
28%
35%
13%
(Teacher-guided activity) Students do an activity, following directions in the
text or other given material. The teacher then leads a discussion of the
activity, highlighting the important science concepts to be learned. Students
apply the concepts, either in written questions or following directions for a
related activity.
32%
38%
9%
7%
*(Inquiry-oriented activity) The teacher introduces a problem or question to
the class. Students work in groups to explore the question and make
conjectures about how to investigate it. After trying out their ideas, students
present their findings in a class discussion. The teacher uses data collected
by the students to further clarify the concepts. Students write in a journal to
explain what they found and to reflect on their learning.
38%
10%
13%
49%
* Distribution difference significant at p<0.05
The data consistently indicate that prior to the project, inquiry-oriented teaching was being attempted
by some participants, but was not yet frequent or consistent in its implementation. The post-questionnaire
2011 Modeling Project Final External Evaluation Report, page 26
data indicate significant increases in use of the desired strategies. On feedback forms the majority of
participants attribute their changes to participating in the Modeling Project. Teachers with multiple years
in the project gave significantly higher ratings to the project’s influence on their technology use and
ability and plan and implement effective lessons.
From end-of-year participant feedback forms:
Overall, the Modeling Project has had a significant influence on my
science teaching.
Overall, the Modeling Project has had a significant influence on my
use of technology in my science teaching.
Overall, the Modeling Project has had a significant influence on my
ability to plan and implement effective science units and lessons.
I am using things I learned at the summer institute as a regular part
of my science teaching.
1 year exp.
Mean Very
(4 pt.
true
scale)
(4)
3.8
86%
2+ years exp.
Mean Very
(4 pt.
true
scale)
(4)
3.9
95%
3.0
43%
3.6*
73%
3.3
54%
3.7*
82%
3.7
75%
3.7
77%
* Difference significant at p<0.05
When asked to describe the changes they have made in their teaching, participant responses included
the following illustrative examples:
My teaching is much more inquiry-based, constructivist-oriented, student-centered. I use more
student-led problem-solving with fewer questions but more time spent on each question. I am
teaching less total content, but the content is being taught more deeply. I am doing more questioning
and differentiated scaffolding. I have almost eliminated stand and deliver from daily lessons. I focus
on identifying big ideas for the content and misconceptions that students have associated with those
big ideas.
Before the Modeling Project my lesson were very routine (i.e. lecture, worksheets, demo, lab); and
they were teacher-centered. The Modeling Project has given me the tools I need to completely
change my approach to incorporate a inquiry-based student-centered team oriented classroom. This
approach has allowed me to take all the research based strategies that I have read about and
implement them in my classroom successfully. I now use Modeling not only in physics, but I've
adapted basic principles in other courses as well. I'm using a textbook less and I've carried my
pedagogical approach more deeply into the usage of whiteboarding and Socratic questioning.
The Modeling Project has really changed everything about the way I teach science. I used to do a lot
of direct instruction with cookbook labs and then wonder why the students didn't "get it". After all, I
had "covered" all the material! Now, I work hard at making the students do all the work. They need
to figure out what's going on and what it all means. I have worked hard at developing Socratic
questioning techniques and forcing students to THINK. I am doing much less rote memorization
tasks, instead focusing on showing students about processes and how things fit together into systems.
I completed my National Board certification last year, and certified this year, and much of what I
did, video taped, and wrote about was based on the Modeling curriculum. I honestly don't know if I
would have certified if I hadn't done the Modeling workshops. They have certainly made me a better
teacher. I focus more now on student learning and understanding.
2011 Modeling Project Final External Evaluation Report, page 27
Participating in the Modeling Project has completely changed my teaching, and it continues to do so.
Prior to working with the Modeling curriculum, I considered myself an apt and highly-qualified
science teacher. Nevertheless, I still struggled with the way the curriculum was structured and found
it difficult to genuinely foster critical thinking skills. Of course, I was big on inquiry and included
many demonstrations. Now I do considerably less demonstrations. I coach students rather than just
lecture and guide them. The classroom is more interactive and I finally discovered an effective way to
employ Socratic dialog in the science classroom. For the first time, I use assessments such as lab
practicals and informal questioning other than just tests and quizzes to evaluate the level of a
student's understanding and skill. This has really helped me to finally realize the difference between
formative and summative assessments. As I continue improving my skill in employing the Modeling
Method, each year, each week, and even each day prompts me to be more-and-more reflective and
more willing to try something new. I'm not as rigid and the excitement of not knowing what will
happen with each class each day keeps me engaged. Above all, the most fulfilling part is that I
experience many, many more of those "aha" student moments, when they really and finally "get it."
My instruction has changed dramatically because of my participation in the Modeling Project. I was
already teaching using inquiry, but I often felt like my lessons lacked time for sense making. Using
modeling gives the structure for sense making and allows students to construct their own knowledge,
which leads to a deeper level of learning.
I have been surprised at how my attitude toward instructing my students has changed since
participating in the Modeling Project and implementing the strategies into my classroom. I am
considering the MAIN IDEAS that thread through the course as anchors to the information I present
in lecture. I am doing much more assesment of my students' prior knowledge by allowing them to
voice their ideas before I simply give them facts. I am doing less telling and more listening. I am
asking more guided questions instead of authoritative preaching of facts. I find the students are
more engaged when I give them a situation where they can invest their thoughts and ideas.
After my participation in the Modeling Project I have been forced to think more about what I am
teaching it, why I am teaching, and how my students will learn the material. Since the use of
modeling in the classroom, my students focus less on memorization of many 'separate' concepts and
focus more on mastery and application of a few concepts that apply to other areas of the course. This
enables students to spend more time thinking and doing and less time being 'busy' with things like
definitions. Its given me new ways to evaluate students, and this is extremely valuable because some
students who know the material, and know it well, may not do well on written exam but can illustrate
mastery of concepts through a practicum type setting.
8) Participating teachers vary in their implementation of the Modeling Approach. Nature and
degree of implementation are related to participants’ time in the project and their comfort with
the strategies. The majority of participants intend to continue their implementation in the
coming school year.
Data gathered from participating teachers – both in the current cohorts and previous participants –
indicate that most are attempting to use what they learned in the Modeling Project in their classrooms.
2011 Modeling Project Final External Evaluation Report, page 28
Only about 5% of the 104 current and former participants responding to the end-of-year
implementation survey indicated that they did not use the Modeling Approach at all in their teaching
during the 2010-11 school year. The typical reason was not being assigned to teach the course
corresponding to the Modeling institute they attended (a factor outside the control of both the teacher and
the project). For these teachers, failure to use Modeling was due to lack of opportunity, not of intent; they
did not feel comfortable trying to apply the strategies in the courses they were assigned. Only two
respondents indicated that they were not using Modeling because they did not agree with the approach.
Of the 95% of respondents reporting that they used Modeling in their teaching, teachers vary in the
degree to which they have adopted the full Modeling approach, as well as their comfort with their
implementation to-date. As seen in the following tables from the end-of-year implementation survey,
results differ for teachers who were only in the project for one year and those who elected to participate
for multiple years. About half in both groups are using Modeling as their main approach at least some of
the time, but those with greater project participation feel more comfortable in their implementation and
use the Approach more frequently in their overall teaching.
1
year
33%
2+
years
55%
27%
18%
13%
12%
19%
9%
9%
6%
14%
24%
47%
55%
33%
21%
3%
0%
3%
0%
Mark the statement below that most closely represents how you currently use the
Modeling Approach.
The Modeling Approach is the core of my instructional approach; I use it in lessons
wherever possible.
I use the Modeling Approach as my main approach when teaching lessons supplied
through the project and sometimes when teaching other lessons as well.
I use the Modeling Approach as my main approach, but only when teaching lessons
supplied through the project (not when teaching other lessons).
I use the Modeling Approach as a supplement to my regular teaching, to enhance or
enrich some lessons.
I do not use the full Modeling Approach in my teaching, but I incorporate particular
strategies from the Approach into my typical teaching.
bout your
I am comfortable using the Modeling Approach and feel I do an effective job with the
lessons.
I am comfortable using the Modeling Approach, but feel I can do a better job
implementing the lessons.
I don’t feel I am very effective using the Modeling Approach sometimes, but I feel good
about the attempts I am making and know I will do better with more experience.
I feel very mechanical when using the Modeling Approach; I’m not comfortable with it
yet.
I am very frustrated with the Modeling Approach; I don’t feel good about how things
have gone so far.
Participants were asked for their perceptions of aspects of their science instruction on the pre/post
teacher questionnaires. They assessed the degree to which particular factors represented an issue or
barrier to their use of the desired strategies. They also reported the degree of success or challenge they felt
2011 Modeling Project Final External Evaluation Report, page 29
with their implementation of particular strategies. As seen in the table below, the composite indicators
show that participants perceived relatively few barriers to using the Modeling Approach, and felt
moderately successful in using particular strategies. (A level of 70 is typically the target level.) There
were significant changes in both indicators on the post-questionnaire. Teachers significantly decreased
their perception of barriers and significantly increased their perceived success with the strategies. In
addition, linear regression modeling revealed that, controlling for the pre-questionnaire value,
participating in the project for more than one year made a significant contribution to the post-value of
three composite indicators. That is, participating for multiple years reduced the predicted value of the
“issues” indicator by 6.68 points and increased the success indicator by 7.72 points.
From pre/post questionnaires:
Composite indicators
Pre-Questionnaire
Post-Questionnaire
Mean % at or
(sd) above 70
31.12
6%
(21.59)
Mean % at or
(sd) above 70
24.55
2%
(17.28)
Frequency of reporting a variety of
factors as issues limiting use of the
Modeling Approach.
Frequency of reporting success with
59.28
strategies associated with implementing (15.51)
the Modeling Approach.
21%
68.92
(17.32)
52%
Multiple
institute
p*
contrib.
< 0.0056
-6.68
< 0.0001
7.72
* p values computed by paired sample t-test (two-tailed), with n=113
“Multiple institute contribution” = increase in post-score (controlling for pre-score) for participating in the project
for more than one year; computed by linear regression.
In general, the most common challenges to implementation on the pre-questionnaire involved time,
materials, comfort with the strategies, and concerns about students. Most (except the student concerns)
showed a significant decrease on the post-questionnaire.
Pre-Questionnaire
From pre/post questionnaires: The items listed below
are factors that other teachers have cited as
challenges to using an inquiry approach in their
science teaching. For each one, please rate the extent
to which it is an issue for you.
Too much preparation time is needed to get materials
ready.
I do not have the proper equipment and materials
needed for the activities.
I do not feel well enough prepared with the strategies;
I don’t feel comfortable using them.
I am not convinced that an inquiry approach is better
than what I currently do.
It takes too long to address the topics; I need to cover
material more quickly to keep up with the pacing
guide.
It is too difficult to assess learning or assign grades
using this approach.
Post-Questionnaire
Mean
(3 pt
scale)
1.89
This is not
an issue
for me
(1)
31%
Mean
(3 pt
scale)
1.53*
This is not
an issue
for me
(1)
54%
2.21
23%
1.93*
32%
1.87
38%
1.50*
54%
1.21
82%
1.17
85%
1.72
44%
1.95*
30%
1.59
52%
1.32*
72%
2011 Modeling Project Final External Evaluation Report, page 30
My students don’t have the kind of background to be
successful with this kind of teaching; it’s too
challenging for them.
My students’ behavior won’t allow me to use this kind
of approach.
My school administration discourages using this kind
of approach to teaching.
My district administration discourages using this kind
of approach to teaching.
1.71
50%
1.63
50%
1.65
53%
1.54
54%
1.14
89%
1.18
86%
1.21
85%
1.15
89%
* Difference significant at p<0.01
The post-questionnaire contained additional issues that were not included on the pre-questionnaire, since
they presume some familiarity with the Modeling Approach. Lack of curriculum materials and
preparation for district or state assessments were areas of moderate concern.
From participant implementation survey: The items listed below are factors that
other teachers have cited as challenges to using the Modeling Approach. For
each one, please rate the extent to which it is an issue for you.
I am not teaching the course(s) for which the Modeling Approach was developed.
I do not have appropriate curriculum materials that follow the Modeling Approach
My background in the content is not deep enough to use the Modeling Approach
effectively.
My class periods are not long enough to use the Modeling Approach effectively.
I am concerned that I won’t be able to answer all the questions that students might
ask when using this approach.
I am concerned that students won’t be adequately prepared for assessments given
by the district and/or state.
Mean
(3 pt
scale)
1.24
1.51
1.33
This is not
an issue
for me
(1)
83%
59%
71%
1.21
1.30
84%
73%
1.52
61%
When asked about their degree of success in using particular strategies associated with Modeling
instruction, participants reported a mixture of success and challenges. Initially, they felt successful with
use of groups, questioning, multiple representations, and having student explain their ideas orally; the
greatest challenges were presented by shifting away from more “traditional” strategies – allowing
students to design their own approach, stepping back from a “telling” role, delaying introduction of
formal vocabulary, and using writing for student explanations. The table below shows significant
increases in perceived success with all the listed strategies:
2011 Modeling Project Final External Evaluation Report, page 31
Pre-Questionnaire
From pre/post questionnaires: Listed below are
instructional elements that are particularly emphasized in inquiry-based instruction. For each one that
you currently do in your science teaching, indicate the
degree to which you feel you are successful or find it
challenging to do.
Having students work effectively in collaborative
groups.
Having students design their own approach to
investigate a question or hypothesis.
Prompting students to justify their ideas or solutions,
whether correct or incorrect.
Stepping back from a “telling” or “explaining” role.
Using effective questions to probe deeper for student
understanding beyond their initial response.
Having students represent ideas or relationships in
mathematical, graphic, and pictoral forms.
Having students explain their ideas, results, or
explanations orally.
Having students explain their ideas, results, or
explanations in writing.
Testing ideas (or models) using data from lab
activities.
Using tasks, projects, presentations, and other different
ways of assessing student understanding
Delaying introduction of formal vocabulary or
terminology until needed for effective classroom
discourse.
Post-Questionnaire
Mean
(5 pt
scale)
3.73
% responding
“success”
(4-5)
62%
Mean
(5 pt
scale)
4.21*
% responding
“success”
(4-5)
83%
2.73
24%
3.32*
46%
3.43
53%
3.93*
72%
3.13
3.55
42%
58%
3.48*
3.83*
53%
68%
3.44
55%
3.96*
73%
3.55
57%
3.89*
69%
3.25
39%
3.65*
60%
3.19
41%
3.64*
62%
3.48
53%
3.81*
69%
3.26
44%
3.67*
63%
* Difference significant at p<0.01
The post-questionnaire contained additional strategies that were not included on the pre-questionnaire,
since they presume some familiarity with the Modeling Approach. Teachers reported great success with
whiteboards, moderate success with using technology, and less success with fostering student critique.
From participant implementation survey: Listed below are instructional elements
that are particularly emphasized in the Modeling approach. For each one that you
currently do in your science teaching, indicate the degree to which you feel you
are successful or find it challenging to do.
Using whiteboards as a tool for students or groups to display their thinking for
others to see.
Helping students learn to question each other and to critique each others’ ideas in a
positive manner.
Using technology myself to collect, analyze, and display data from lesson
activities.
Having students use technology to collect, analyze, and display data from lesson
activities.
%
Mean responding
(5 pt “success”
scale)
(4-5)
4.43
92%
3.32
46%
4.00
74%
3.98
76%
2011 Modeling Project Final External Evaluation Report, page 32
Participants were asked to elaborate about challenges they were facing in using the Modeling
Approach. Examples of their responses include the following:
I've had to adapt away from the modeling curriculum in order to meet the state’s required SCOS
within the time limits provided. I look forward to another opportunity to model this year.
I feel comfortable with the modeling approach, but find myself having to work with limited
resources, and I do feel much in the way of time constraints, so that I implement facets of modeling
when and where I feel that it will work out to the students' advantage, and allow me a base for them
to build upon. I would love to use it more, but also have limitations imposed by my classrooms,
such as largely "dry" environments where most chemicals and some demonstrations are just not
possible. In short, I try to use it where I can, but recognize my limitations and work within them as
well as possible.
I have unsupportive administrators and a difficult co-teacher to work with. The other person is very
anti-modeling but what we are doing in class is to be he same. There is no room for teacher
individually.
A key characteristic of the modeling approach is getting the students used to talking with each other
and discussing each other's theories and results. This is the most difficult step. Getting them to talk
to each other about their work is not easy, but once they get past this, modeling takes off.
I am concerned about how long it takes to cover the material. My chemistry classes are on the 4x4
block schedule and I know that I haven't covered all the standard course of study's goals and
objectives as determined by the state DPI.
Overall, the great majority of participants – particularly those taking part for multiple years –
expressed their intention to continue their implementation efforts in the next school year.
From participant feedback forms: Do you plan to continue trying to implement the Modeling Approach in
your science teaching next school year?
1 year
2+ years
Definitely yes
74%
94%
Probably yes
17%
3%
Not sure
3%
0%
Probably no
4%
3%
Definitely no
1%
0%
Participant comments illustrate their generally positive view of Modeling as they continue to work on
their implementation:
I will not forget what _____________ expressed to us about becoming an effective modeler and
questioner. They stated that it could take many years to become fluent. I tried to keep this in mind
as I felt ineffective at times. I do believe that modeling is a better way to teach and I refuse not to
give it my best effort. My students need and deserve that much from me. I look forward to more
practice, assistance and follow-ups from fellow modelers.
At first it was difficult to step out of my comfort zone of teaching. I was used to doing all the work
for the student. I was not letting the students be accountable for their work. I started out slowly
letting the students work in small groups. I invested in some white boards for my class. I am still a
little uneasy about teaching this way, but overall my well-being is less stressed. I still have days
2011 Modeling Project Final External Evaluation Report, page 33
when I question myself about a concept that was presented, but this is still a work in progress for
me.
I will continue to use the techniques, whether they are pre-made lessons or whether I implement
them on what I already teach. This year, being new with it, I was a little shaky/ hestitant in going
full-out on it because I was new to it and there were state EOCs at the end of the semester that made
me iffy on using it all the way. However, next year, with a little experience under my belt and no
EOC to worry about, I feel that I can definitely let loose on it more.
I am not comfortable yet, but I believe the Modeling Method provides a greater understanding of the
physical science concepts than my current teaching. I want my students to know the concepts longer
than the end of the semester. I also believe that modeling will teach the students to become more
critical thinkers and problem solvers. I understand that my improvement in using the models will
only continue to improve with practice so I will not give up on trying to use it. This teaching method
takes a while to feel comfortable doing but it is worth learning.
I'm not particularly adept at using the modeling approach yet. However, I am firmly convinced that
this is the best way I know to help students learn Physics, and I intend to employ it extensively. I see
the impact of the modeling approach to students' learning. And I would enjoy trying to implement it
every class that I have, but I don't feel confident and efficient yet. I am confident that I'll improve my
implementation of the modeling approach from year to year.
The Modeling Approach definitely takes practice, but I feel that this year has been more successful
than last year, and I have some great ideas for next year. My colleagues have also been learning
about the Modeling Approach, so we are able to better support one another. There are a few units
that we teach that are not explicitly in the modeling curriculum, so we've had to modify those "on
the fly" and aren't yet as able to "model" that material as we'd like, although I think that that, too,
will come with time and practice.
I can't wait to use this teaching method again. I know that I struggled at times with the method,
asking the most effective questions; however, I noticed growth in this area with myself and through
reflection already have ideas on how to improve. I also saw a greater understanding with my
students in the concepts despite my struggles and can't wait to see how much more the students gain
as I improve.
2011 Modeling Project Final External Evaluation Report, page 34
V. Evaluation Findings: Impact on Student Performance
9)
Results of pre/post student testing show significant growth in student understanding of the
targeted concepts. Two thirds of students in targeted courses in Year 2 demonstrated
significant pre/post gains on the assessments. Three fourths of students in targeted courses in
Year 3 demonstrated significant pre/post gains.
Ultimately, The Modeling project’s success will be gauged by its impact on student learning. The
project’s evaluation plan calls for administration of a pre/post assessment to students that is aligned to the
focus of the project’s professional development. The same assessments that teachers completed (Force
Concept Inventory, Physical Science Concept Inventory, and Assessment of Basic Chemistry Concepts)
were given to students of those teachers at the beginning and end of the pertinent course.
2008-09
The project had a significant issue during its first year with teachers attending a Modeling institute with
the assurance they would be teaching the relevant course, then being reassigned to different courses in the
next school year. Consequently, these teachers were unable to administer the pre/post assessments to
students. In addition, communication issues within the project resulted in several participants not
collecting both pre- and post-test data from their students. Altogether, complete pre/post data were
received from only 5 participants, representing 85 students in 7 classes. This was deemed insufficient for
conducting an analysis that would give evidence about the project overall.
2009-10
During the 2009-10 school year, 51 of the 69 teacher participants taught at least one section of the course
corresponding to the institute they attended. Of these 51 teachers, 32 (63%) submitted complete pre/post
student assessment data, showing results for a total of 1,270 students. The table below summarizes the
results:
2009-10 Pre/post Student Assessment Results
#
teachers
reporting
class data
#
students
tested
Mean
pretest
score
Mean
posttest
score
Signif.
level –
paired
t-test
Eff.
size
Mean
<g>
Students
with
signif.
gain
Teachers
with
signif.
class gain
Physics
11 of 15
(73%)
335
23.1%
(.089)
43.3%
(.199)
<0.0001
1.01
.263
248
(74%)
10
(91%)
Physical
Science
4 of 10
(40%)
196
43.0%
(.149)
56.1%
(.196)
<0.0001
.67
.229
127
(65%)
2
(50%)
Chemistry
17 of 26
(65%)
739
32.8%
(.142)
44.4%
(.180)
<0.0001
.64
.173
464
(63%)
13
(76%)
TOTAL
32 of 51
(63%)
1270
31.8%
(.267)
45.9%
(.192)
<0.0001
.74
.200
839
(66%)
25
(78%)
2010-11
During the 2010-11 school year, 57 of the 66 teacher participants taught at least one section of the course
corresponding to the institute they attended. Of these 57 teachers, 42 (74%) submitted complete pre/post
2011 Modeling Project Final External Evaluation Report, page 35
student assessment data, showing results for a total of 2,210 students. The table below summarizes the
results:
2010-11 Pre/post Student Assessment Results
# (%)
teachers
reporting
class data
#
students
tested
Mean
pre-test
score
(sd)
Mean
post-test
score
(sd)
Signif.
level –
paired
t-test
Eff.
size
Mean
<g>
Students
with
signif.
gain
Teachers
with
signif.
class gain
Physics
16 of 20
(80%)
718
24.9%
(.212)
49.9%
(.124)
<0.0001
1.47
.333
622
(87%)
16
(100%)
Physical
Science
8 of 16
(50%)
437
37.3%
(.140)
44.3%
(.168)
<0.0001
.46
.112
230
(53%)
6
(75%)
Chemistry
18 of 21
(86%)
1055
30.3%
(.176)
42.9%
(.133)
<0.0001
.82
.181
798
(76%)
16
(89%)
TOTAL
42 of 57
(74%)
2210
29.9%
(.139)
45.5%
(.190)
<0.0001
.94
.222
1650
(75%)
38
(90%)
Noteworthy results contained in the tables include:

Overall, the mean scores both years increased from about 30% correct on the pre-test to 45%
correct on the post-test. Using paired t-tests, the differences were significant at p<0.0001.
Furthermore, each individual course showed significant pre/post gain, and over three fourths of
the teachers demonstrated significant gains in pre/post class means.

The effect sizes in the table indicate that the pre/post gains are evidence of meaningful
enhancement of student understanding, particularly in the physics classes.

The average normalized gain <g>, a measure used by the National Modeling Project, ranges from
0.11 to 0.33 (0.20 and 0.22 overall, in years 2 and 3 respectively). This is within the range
reported for Modeling classes in the national literature.

In year 2, two-thirds of the students posted a significant pre/post gain, defined by USDE as a gain
of at least one-third standard deviation from their pre-assessment score; in year 3 three-fourths of
the students posted a significant gain.
The results indicate that students in these Modeling classes did make significant gains in their learning.
Furthermore, from a more qualitative perspective, participating teachers, for the most part, are strongly
positive in their anecdotal reflections about the effect of their Modeling instruction on their students, as
seen in this sample of comments:
After students get past the frustration and reluctance of a new learning method, they enjoy more the
involvement and go much deeper (by their own admission) in their understanding of the concepts
considered.
My students that are repeaters (taught the old way) have found great success using modeling (most
have a B or better) and have expressed feelings of confidence in the material they never felt before.
It is more clear when students truly understand the material as opposed to just being able to pass a
test. I am very pleased with the results as are my students and my administration!
2011 Modeling Project Final External Evaluation Report, page 36
The level of participation among my students has increased dramatically. The students take more
responsibility for their learning, and are more actively engaged on a daily basis. And with this being
my first year teaching this way, my test scores have remained constant. So next year, I feel I will see
great improvement as I improve on my implementation.
The understanding students achieve through the modeling process is deeper and longer lasting than
that received by more traditional sequence of lecture, demo, lab and assessment. It is also more fun
for both me and the students.
It works. My students are learning. It allows the confident student a positive platform and the
weaker students have a voice as they usually shine in the construction or implementing stage.
Students learn Physics at a much deeper level. No other approach I have used or seen pushes
students past their misconceptions as this approach does.
I didn't realize how much my students didn't understand the concepts until this year. I assumed that
because they were testing well they knew the material, Since I have backed away and let the students
take part in their learning (by modeling) student understanding has increased. Students develop the
"ideas" instead of memorization. My students have a deeper understanding of chemistry in
comparison to last year when I did not use modeling. Students are learning more, thinking more, and
enjoying it.
My students are doing a lot of white boarding with their labs and activities becoming self assessing
as well as helping others see errors of flaws in their thinking or calculations. I have been impressed
with their ability to take criticism and re-evaluate their stance. This does not always happen and
some get very frustrated but few seem to be willing to quit altogether. That has been a great
improvement over the past years where a little negative feedback would cause them to shut down. My
students are learning to question and connect rather than take each subject as a single idea that must
be memorized.
The state End-of-Course (EOC) assessments are another measure of student learning important to
teachers and administrators. Unfortunately, the state EOC assessments are not a good measure for the
impact of the Modeling Project on students for multiple reasons. The first set of reasons involves a
question of the data’s representativeness:

In 2009, the state discontinued the EOC assessments in chemistry and physics due to budgetary
constraints. This means that EOC data were not available for four of the six teacher cohorts after
Year 1, containing over 70% of the participants in those years.

An EOC assessment was given in the physical science course in Years 2 and 3. Unfortunately,
due to scheduling factors outside the project’s control, only about half the teachers in the physical
science cohorts were assigned to teach a physical science class.
The combination of these factors means that state EOC assessment results would available for only 25 of
the 135 teachers participating in the project in Years 2 and 3 (18%), representing only 1 of the 3 courses
targeted. Such data cannot be assumed to be representative of the project as a whole.
Additional factors suggest caution in making specific attribution of EOC results to the Modeling
Project:
2011 Modeling Project Final External Evaluation Report, page 37

No “EOC”-equivalent pretest is administered at the beginning of the courses, so actual change in
student learning cannot be determined. Comparing the results from one year to prior years would
involve tenuous assumptions about the comparability of the student groups involved.

The science content addressed represents only a portion of the total curriculum for the targeted
courses. In the absence of detailed disaggregation, it is unclear how the overall results reflect
performance in areas addressed by the Modeling curriculum materials.

97 of the 133 teachers (73%) who participated in the project had just one year of professional
development and support for implementing the significant instructional changes promoted by the
project. It would be expected that most would not yet be fluent in their use of the strategies.
About one fourth of the teachers participated in multiple years of the project and would be
expected to be more effective in their implementation.
For these reasons, the state EOC assessments are not judged to be a good indicator of Modeling Project
impact.
VI. Evaluation Findings: Operation of the Modeling Project Partnership
10) Project partners worked well together and carried out their roles effectively.
The Modeling project was a partnership of five county school districts (Buncombe, Durham, Martin,
Caldwell and Watauga) and the Science House at North Carolina State University. The national Modeling
program at Arizona State University ws also linked to the partnership, providing access to expert
personnel and quality resources. The partnership was expanded beginning in project year 2, adding the
New Schools Project as a partner and providing specific opportunities for participation by teachers in its
member schools. The project recruited participants statewide, with first priority given to qualified
applicants from the partner districts. Its formative evaluation component included providing feedback on
the operation of the partnership, for use in planning and decision-making.
Because it was funded as a “mathematics and science partnership”, it was important that the entities
involved in the Modeling project – five school districts, one IHE, and the New Schools Project – actually
acted as partners, collaborating toward achievement of mutually-held goals. Several indications suggest
that this was the case:

Project partners carried out their defined roles in operating the project. NCSU provided the
Project Director, as well as facilities on campus for holding summer institutes and follow-up
sessions. Watauga Co. served as fiscal agent for the project and provided key individuals who
served as the project coach and an institute facilitator. All five district partners assisted in
recruiting eligible teachers, who comprised 24 of the 133 (18%) participants over the three
project years. The New Schools Project provided funds to expand project offerings, as well as
recruiting eligible teachers from member schools.
2011 Modeling Project Final External Evaluation Report, page 38

Project planners had a good working relationship and engaged in collaborative planning and
decision-making regarding project activities. Concerns or disagreements were handled
professionally and in a spirit of seeking consensus on the best course of action. In large measure,
this was likely due to the pre-existing relationship among the planners, which enabled them to
work together effectively from the earliest stages of the project.

The partnership was responsive to needs or concerns expressed by participants, particularly
regarding scheduling of sessions, maintaining communication and adjusting the level and pace of
activities.

The partnership demonstrated a commitment to gather and use formative feedback to make
improvements in the nature and quality of its activities. Project staff gathered feedback from the
project experiences and are addressing the concerns identified in planning for subsequent
activities.

The partnership made effective use of existing capacity in the state to lead the physics and
physical science institutes, and developed capacity for chemistry by involving one of the
accomplished participants as a facilitator during Year 3. These individuals will continue to be
available to lead and support Modeling institute and other activities in future years.
The Modeling project involved the appropriate partners for its purpose, and operated effectively in pursuit
of its goals.
VII. Closing Observations
A. Summary Observations about the Modeling Project
Overall, the Modeling Project’s professional development and support activities over its three-year
period were effective in contributing to progress toward the project goals. The majority of participating
teachers were positive about their experiences in the project. Over one fourth of the participants returned
to take a second (or even third) Modeling institute. Several of the teachers are continuing their direct
participation in Modeling professional development by enrolling in another Modeling summer institute in
2011 through a newly-funded Title IIB project.
Participants have demonstrably grown in their knowledge and skills, and are putting their learning
into practice (to varying degrees). As noted earlier, the great majority of teachers expressed their intention
to continue implementing the Modeling Approach in their teaching in the coming school year. While they
may not yet have reached a level of fluent implementation desired by the project, nevertheless they have
made progress and are confident that they will continue to do so.
In summary, the evidence gathered in the Modeling Project external evaluation indicates that:

Project activities were well-designed and implemented, and participants valued their experiences.

Teacher participants demonstrated significant growth in their knowledge of the targeted science
content.
2011 Modeling Project Final External Evaluation Report, page 39

Teacher participants report enhancing their pedagogical knowledge and skills, and have begun to
implement the Modeling Approach in their science teaching.

Student assessment results and anecdotal evidence indicate a positive contribution of the
Modeling Project to student learning of the targeted content.
The following samples of participant reflections describe their view of their experiences in the
project:
It was an amazing opportunity to be able to participate in both of the institutes. I learned so much
about physics and about teaching science I have been able to apply these methods to other areas of
my teaching.
I absolutely love Modeling. It has changed me forever; I am on a mission to get more teachers
trained in Modeling. Teachers who attend this program will be guaranteed to improve both their
understanding of the content and also their methodology for presenting the material to the
students. The ability of the students to learn in a classroom will also be greatly improved if a
teacher has attended a modeling workshop. There is no way a teacher can participate and not
gain from the experience.
I did not fully realize where this workshop would take me when I applied. I thought it would give me
some new labs to add to my curriculum. Instead of tools added to my well-developed toolbox, I had
to find a whole new box and the tools were not like the old tools. These were transformative tools
(not tinkering tools) and preconceived ideas were challenged, disregarded, or rearranged. The
evidence drove the curriculum development, not the textbook.
The Modeling project has changed my teaching significantly. For a long time, I have felt like
something was missing in my instruction and have even redesigned my curriculum to address this. I
became convinced that learning needed to be centered around big ideas and be sequenced to flow
more naturally. Until this workshop I had no resources to pull from; I found a whole group of
people that felt like I did and were willing to change the way science was taught in their classroom
to a more authentic form of learning. I have always used inquiry, questioning, and collaboration, I
just needed one more step that modeling has given me.
Everyone was so supportive. Nobody hesitated to e-mail, give contact information, or encourage us
to do anything. The teachers and facilitators were so full of good ideas, both in and outside the
classroom. Everything was very well-organized, and the people in charge knew what they were
doing. Communication was excellent, and we were allowed to give input at any time. They listened
to us and took our ideas into account. The environment was one of encouragement and mutual
respect at all times. Everybody learned more than we thought we could, and everyone had great
ideas for how to help our students. I'd recommend this to anyone.
The design and implementation of the Modeling Project showed numerous strengths that contributed
to the effects observed, including:

The project was based on the well-researched and documented program developed at Arizona
State University. The instructional materials used in the institutes were developed and vetted
through the national program, and thus provided a strong framework to support classroom
implementation (participants did not have to try to “adapt” materials to the Modeling Approach).
The current project was not a “cookie cutter” replication of the original project, however. Project
2011 Modeling Project Final External Evaluation Report, page 40
leaders made strategic adaptations to the professional development design to fit the current North
Carolina context, making it more meaningful and relevant to participants.

Participants were deeply immersed in Modeling from the beginning of their three week summer
institute experience. This extended experience enabled the teachers to engage with the Modeling
Approach, experience it as a student, reexamine their own content understanding, and reflect on
the implications for their teaching. The follow-up sessions and coaching visits during the
academic year extended the in-depth work on content and pedagogy, providing support as
participants’ learning built over time. The project structure also fostered strong relationships
among participants and between participants and facilitators, which in turn created a supportive
context to push participants’ thinking and learning.

The experienced Modeling teachers who designed and facilitated the institute, follow-up, and
coaching sessions were knowledgeable about the science content, research-based pedagogy, and
effective professional learning. The facilitators each brought individual strengths that
complemented each other well. As current or former classroom teachers with extensive practical
experience using Modeling, they had great credibility with the teacher participants.

The facilitators showed flexibility in planning project activities, while still adhering to the overall
goals of the project. The project’s willingness to adjust its activities showed responsiveness to
changing conditions and enhanced its credibility with the participants and the partner districts.

The project’s management at The Science House provided leadership that made it easier for
issues to be resolved and needed tasks to be completed in a timely manner. The relationships
enabled the Modeling Project to function as a partnership, rather than simply a provider/client
project.
That said, some issues and concerns were identified in the evaluation, including:

The bulk of project contact with participants was limited to a one-year period (summer and the
following academic year). The next summer, project activities focused on a fresh group of
participants and implementation support for previous participants was limited. To a large
measure, this was a consequence of the project budget and project planners’ desire to involve as
many teachers as possible. As a result, there was great variability in continued implementation
after the “participation year.” Project coaches made an effort to stay in touch with previous
participants and to provide support where possible; and the project website remained available for
previous participants to engage in the online community. Given the demands of shifting
instructional practice to the Modeling Approach, however, implementation support for multiple
years would likely produce more consistent use of the Approach as envisioned by the project.
Participants recognized this, as seen in some of their comments, and in the fact that one fourth of
them participated for multiple years. This enabled them to extend their Modeling into other
subject areas, but also provided additional professional development and support for their
Modeling implementation in general. Finding the appropriate balance between breadth of
participation and depth of involvement is a persistent issue for any project of this type.

Several participants noted the important role that their principal plays in supporting
implementation of the Modeling Approach. Principals committed to do this as part of the teacher
application to participate, but were uneven in their follow-through. Attempts were made to
engage principals, to help them learn about Modeling, but this was not a major effort of the
project.
2011 Modeling Project Final External Evaluation Report, page 41

The project had a strong rationale for its choice of facilitators to lead the institutes and followups. However, the MSP program’s emphasis on involving higher education science faculty made
finding an appropriate role for university physics and chemistry faculty a persistent issue that the
project did not address in a meaningful way.
B. Evaluation Successes
The greatest success with respect to the Modeling Project evaluation is the quality of the data
gathered, and this is due in large measure to the support of the project staff. Project personnel were
committed to ensuring the maximum possible response rate on the various instruments and protocols used
in the evaluation, and were of great assistance in prompting participants to respond. The result was an
86% response rate overall. In addition, project personnel encouraged participants to take the evaluation
seriously as an assessment of the project, resulting in more thoughtful (and, we assume, more accurate)
responses.
C. Evaluation Caveats and Constraints
There were no major challenges in carrying out the evaluation of the Modeling Project. However, the
evaluation’s ability to report project outcomes and impacts confidently was constrained by several issues.
For the most part, these were known from the outset of the project; while they did not impact conducting
the evaluation, they must be considered in interpreting the results and forming conclusions.
1) Relatively small number of participants
The eight cohorts of teachers over the three years of the Modeling Project reflect the numbers
proposed by the partners and supportable by the level of funding provided. Aggregating the data from
the cohorts yielded 133 total participants (unduplicated count), which is an acceptable number for
analysis. Finer-grained examination of the data – by subject area or participation level – involved
subgroups with smaller numbers, limiting the power of the analysis. This could not be avoided, but the
limitation must be recognized in reporting results of the project.
2) Changes in participants’ teaching assignments
Overall, 24% of participants – 16 out of 62 (26%) in the physics group, 23 out of 59 (39%) in the
physical science group, 3 out of 51 (6%) in the chemistry group – did not teach any classes of their
targeted course during their participation year. In most cases, their teaching assignment was changed
to accommodate changes in student enrollment and scheduling at their schools, a factor beyond the
control of the project. The effect, however, was that teachers were unable to apply their learning
directly in the course context intended, making their initial implementation of the Modeling strategies
more challenging. Furthermore, they were unable to take part in pre/post student testing, which was
designed for the targeted courses only, thereby limiting the data on changes in student learning and
introducing a question as to its representativeness of the group as a whole.
3) Determining a useable comparison group.
The quasi-experimental design for the summative evaluation entailed having a suitable comparison
group available against which to compare the pre and post performance of participating teachers. The
original intent was to use results for teachers in the evaluation of the national Modeling Project as the
comparison group. For physics (using the Force Concept Inventory) national results are available in
the existing literature. Such is not the case for physical science and chemistry.
4) Determining degree of implementation.
2011 Modeling Project Final External Evaluation Report, page 42
The evaluation design contained multiple ways to examine participants’ implementation of the
Modeling Approach – observations, surveys, narrative reflections, coaches’ assessments. Of these,
classroom observations using the RTOP protocol would provide the most objective and quantifiable
data. The project intended for coaches to use the RTOP when visiting participants, to collect data over
time about their instructional characteristics. However, it became apparent that coaches needed to be
more actively involved in participants’ classrooms during their visits, so opportunities to gather data
using the RTOP were sporadic and generated insufficient data for analysis. The remaining
implementation data were perceptual and self-reported. While participants were generally quite frank
in acknowledging their implementation challenges and successes, reliance on their self-reports alone
introduced some uncertainty into any assignment of implementation level.
5) Attributing student performance changes.
While the long-term goal of the Modeling Project is to impact students’ achievement in their science
courses, documenting attributable changes is a significant challenge in the evaluation. The project
gathered pre and post concept inventory data (FCI, PSCI, or ABCC) for the students of participating
teachers. However, questions about fidelity of implementation and opportunity-to-learn in the
teachers’ first attempts to use the Modeling approach made issues of attribution of pre/post changes
problematic.
6) Relating project impact to state assessment results.
The Annual Performance Report called for the project to report performance on the pertinent state
assessment for students of participating teachers. This was problematic for the Modeling Project for
multiple reasons, including: a) The state no longer administers End-of-Course assessments in physics
and chemistry, so the data called for in the APR do not exist in these subjects; b) The areas of the
curriculum addressed by the project are only a portion of what is contained in the Standard Course of
Study; c) Several participants in the physical science institute did not teach the course in the following
school year, and so would have no student EOC data; and d) Attributing broad student performance
changes during a teacher’s first year of implementing a challenging new approach is a difficult
proposition.
2011 Modeling Project Final External Evaluation Report, page 43