Modeling Physics Instruction in South Dakota Part 2: 2011 and 2012
Evaluation Report June 2012
I.
Cover Page
II.
Evaluation Overview
III.
Evaluation Summary
IV.
Project Timeline
V.
Surveys Designed
VI.
Evaluation Findings
VII.
Lessons Learned and Implications
VIII.
Summer Followup
IX.
Overall Conclusion
-1-
II.
Evaluation Overview
Contained within the following pages is the 2010-2011 evaluation results for the Physics Modeling
workshop, which were collected during and after sessions held from June 6, 2011, through June 17, 2011.
Based upon tested and effective methods developed at Arizona State University, the Black Hills State
University Physics Modeling Part 2 workshop aimed to improve teachers’ understanding of physics, to
influence their classroom teaching to become more student-learning centered, and to extend existing
collaborations between math and science teachers who may have attended as teams during the 2009 or
2010 workshops. Prior attendance at the Part 1 workshops in 2009 or 2010 was not essential to attend
the Part 2 workshop in 2011, but most participants had attended one of the prior years. Previous
research and workshop experiences at Arizona State University had shown that participants must attend
at least three weeks of the modeling workshop to "get it" and reliably change their teaching practices.
Thus the Part 2 workshop on which this report focuses should be regarded as an essential completion of
the Modeling training for teachers.
The information collected by a representative of Technology and Innovation in Education (TIE) within this
evaluation provides:
• The extent to which the project activities facilitate the achievement of grant outcomes;
• The level of satisfaction of received services; and,
• The impact project activities had on the target audience.
III.
Evaluation Summary
Evaluation Summary
To determine whether the project has achieved its established objectives and successfully impacted its
targeted audiences, this evaluation collected evidence (data) from those involved in the project (using
surveys, interviews, observations, and/or focus groups) to obtain a documented, shared understanding of
the project’s effectiveness. It answers the question: Is this project accomplishing its stated goals?
Evaluation of the Modeling Physics Instruction within South Dakota Program
During this reporting period, evaluation activities included direct observation by a TIE representative of
Physics Modeling sessions, discussions with participants, sets of surveys given to participants, and data
gathered via Physics Modeling instructors who functioned as facilitators and coordinators for the project.
IV.
Project Timeline
March 2011
April-May 2011
June 2011
June 6-17, 2011
June 17-30, 2011
May 2012
June 17-22, 2012
June 29, 2012
Funding announced
Course and Evaluation Planning
Development of Survey Instruments and Planning
Weeks 1 &2 of Physics Modeling Sessions
Participant survey data collection
Receipt and Analysis of Evaluation Data and Information
One-week followup session for participants
Sharing and Discussion of Report with Dr. Andy Johnson
-2-
V.
Surveys Designed
Surveys were designed by Dr. Johnson—who asked questions based on course goals and key aspects of
conceptual understanding. Additionally, Dr. Johnson devised questions that would challenge teachers to
ask themselves: Are these new approaches a good option for their students? Some questions required
“open-ended” responses which were compiled and used to improve upon course design. Other questions
were designed to be scaled on a rating system that allowed answers to be numerically calculated and are
presented below. The extensive Modeling follow up session conducted June 17 - 22, 2012 is not included
in these data.
VI.
Evaluation Findings
Content Learning
The first measure involves content learning. Nine of the
participants took the Force Concept Inventory as both a
pretest and posttest for this course. The pretest average of
71.9% reflects a relatively high level of understanding of
forces and motion from previous years of Modeling
Workshop attendance. The posttest scores showed some
improvement with an average of 78.9%. Thus the majority
of the teachers in attendance understood the majority of
the basic ideas about forces and motion and showed some
improvement during the course. These scores are much
higher than those of typical college students who may
average around 50% or 60%.
Use and Appreciation of Modeling
Based on the answers received from the 13 participants of the Modeling Workshop, the following graphs
and narratives provide a detailed examination in determining if the expectations of the workshop were
met.
The charts/graphs and narrative responses represent the survey questions designed by Dr. Andy Johnson,
and were asked on SurveyMonkey:
-3-
Question #1:
To what extent have you used the overall modeling method? (Consider the class where you use
Modeling the most.)
Based on the information provided,
all 13 participants reported the use of
modeling in some format. Twelve of the 13
participants (92%) have engaged in the use of
the modeling technique more than “almost
never.” Furthermore, the use of modeling by
12 of the participants is greatly enhanced
when viewed at the level of at least “about
half the time.” Of the 12 participants, seven
(58%) reported that they used the modeling
method at least half of the time in their
classrooms. Reasons for why the six
participants who used the modeling only
“sometimes” or “almost never” were not
stated.
Question #2:
In which class(es) have you used modeling techniques to some extent?
Twelve of the 13 participants responded to this question. Although most stated that they used
the modeling technique for physic classes, responses also included: physical science, chemistry, biology,
geometry, statistics, and/or algebra.
Question #3:
How frequently have you used the following aspects of modeling:
The Whiteboarding technique is arguably the most important component of the Modeling
pedagogy. It involves students in organizing and presenting their experiment findings or ideas on a
particular science topic to the rest of the class. In order to present their whiteboard to their peers,
-4-
students must become clear on their own thinking and results. When students are presenting results, the
Modeling teacher asks probing questions that lead students to reveal their understandings and also cause
students to think more deeply about their findings. Whiteboarding discussions provide some of the most
fruitful moments in physics courses because of the preparation - students use the whiteboard to
construct coherent understandings - and because of learning opportunities during discussions. The
teacher asks students to go another step beyond what they currently know.
Participants' adoption of Modeling Whiteboarding techniques is overwhelming. All 13
participants reported using student whiteboard presentations sometime during the past 365 day period.
Ten participants (77%) used whiteboarding more than once a week. Twelve of the 13 participants (92%)
used whiteboarding at least once a month. From this data, one can conclude that teachers have largely
adopted a key student-focused learning strategy from the Modeling workshops.
Open ended labs are a way for students to develop their own hypotheses and to garner more
student individual thought. Instead of giving explicit instructions on what a student should or should not
accomplish, look for, or ascertain, these labs encourage students to test their own leveling of
understanding and thinking. This question was answered by all 13 participants. Two participants (15%)
answered that they used these types of labs at least “once a quarter.” The remaining participants (85%)
reported they used these labs at least “once a month” (39%) and occasionally as often as “more than
once a week” (46%). Although it was never ascertained how comfortable teachers felt after creating and
implementing a lab of this nature, the high percentage of teachers who routinely use this type of lab
suggests a higher than normal level of comfort engaging in this type of activity. The reason why the two
participants only used this type of lab “once a quarter” was never stated.
-5-
The Modeling laboratory introduction scheme is an inviting invitation to consider a new topic
from a scientific perspective. A carefully selected phenomenon—representative of the topic—is shown
to the students, and the instructor then asks the following questions one at a time:
 What do you observe?
 What could you change?
 What could you measure?
All 13 participants used this type of modeling at least once in the past year. Ten participants (77%)
reported that they use this type of modeling “more than once a week.” This high percentage suggests
the participants regard this as a powerful way to introduce students to new topics of study. The
remaining three participants (23%) used this type of modeling at “once a quarter.”
All 13 participants answered this question. Almost half of the participants (46%) reported that
they use a worksheet from the Modeling manual at least once a week, suggesting that the manual is a
tested and proven resource to help instruction. Three other participants (23%) reported that they used a
-6-
worksheet from the manual at least once in the “past 365 days.” Of the four participants (31%) that
reported not using the manual, no reason was given for this decision.
Basic questioning strategies involve enabling and obligating groups to answer for themselves
questions that they can answer through experiments and/or reasoning. During these questioning
strategies, the instructor asks appropriate questions to:
 Clarify or complete the groups’ presentations;
 To push the groups to think more deeply or make connections they are close to making;
 To establish a shared understanding in the class about the topic and phenomena they are
studying.
All 13 participants responded to this question. The questioning strategies obviously are important to the
participants! A plurality of the 13 participants (77%) reported that they use questioning strategies “at
least once a week.” Furthermore, all 13 participants reported using this modeling technique at least once
in the past 365 days. This data, compared to the data of laboratory introduction scheme, suggests that
these two modeling techniques support each other. When one of these two modeling techniques is
used, the other is an excellent way to supplement further knowledge about a subject.
-7-
Besides the abovementioned modeling techniques (whiteboard, open-ended labs, laboratory
introduction scheme, worksheets, and questioning strategies) teachers reported that they also use other
modeling strategies. What other modeling strategies these teachers use was never stated. Only two
participants (15%) reported that they do not use outside modeling strategies. Five participants (38%)
reported that they use other modeling strategies than the ones list above at least “more than once a
week”; likewise, five participants (38%) reported that they use different modeling strategies at least
“once a month.” Only one participant (8%) reported using a different modeling strategy at least “once a
quarter.”
This information can be read in two different manners: 1) of the 13 participants, five of them
(38%) feel they need to explore additional methods than the ones stated above to help augment their
explaining of a given topic; and, 2) eight of the participants (62%) feel comfortable using the
abovementioned methods and feel they do not need to explore different options on a regular basis.
However, this evidence does not give any proof as to what group has better success explaining a topic of
interest to their students. If anything, the evidence suggests that the abovementioned techniques are
useful and practical to the teaching of math and science, and can be augmented with different strategies
to help explain new material to students.
Question #4:
If you’ve used questioning method strategies, please say something about what you’ve tried and/or
your experience.
Three of the participants (23%) did not respond to this question; and no reason was given as to
why they did not respond. Of the remaining ten participants (76%) the answers varied. Some of the
participants used questioning strategies to elicit responses from students that would “make [students]
explain and ask questions” that would then “get [students] to figure out” answers or solutions.
Sometimes this strategy was successful; sometimes it was not. Other participants used questioning
strategies to “introduce new concepts/chapters” and found that this method of modeling was
successfully stimulating “greater discussions” and an elevated willingness to “jump into a subject.” One
participant reported that they always used a questioning strategy to start a new topic. They believe that
this type of modeling has kept students in an “inquiry mode,” encouraging questioning and sensemaking.
This participant also stated that they “used lecturing as the main mode of teaching until being exposed to
Modeling Physics” and since this exposure, they “have worked hard to make a 180 degree change” in
-8-
their mode of teaching. Another participant reported that they have learned to use questioning
strategies with Bloom’s Taxonomy, which has increased the level of insight and inquiry in their students.
Overall, none of the participants who reported using questioning method strategies reported a dislike of
this modeling technique.
Question #5:
If you have in mind “other strategies” not listed in question three, what were they?
Eight participants did not respond to this question; no reason was given for not responding. The
remaining five participants gave the following examples of other modeling techniques and strategies
used to help explain different areas of study: have students design their own experiments; develop
questioning habits in students; try to get students to recognize and see proportional relationships within
same and/or different subjects; and, classroom collaboration before starting a new area: this
collaboration was used to see what information students retained from previous classes, and gave them
the opportunity to identify as a group what information may have been forgotten.
Question #6:
Which of the above (or other) aspects of modeling have been particularly valuable?
Three participants (23%) did not respond; no reason was given. Although the remaining ten
participants (76%) responded, an overwhelming number of them reported that the techniques of
whiteboard modeling and questioning method strategies have been the most valuable. One participant
in particular stated: “The key to teaching is to inspire students to think. Students have to think when
teachers use modeling…Many of them are really great at regurgitating memorized information [but
modeling makes students think].” This participant also adds, “I get to observe them grow stronger and
more confidant as problem solvers over time.” Another participant adds: “Students are not used to
thinking or articulating learning” and modeling strategies force students to think. One of the most
valuable comments stated declares: Physics Modeling has “open[ed] a door to [students’] minds instead
of shoveling in facts and formulas.”
Question #7:
If you have difficulty or reluctance to use the modeling, what are they?
Three participants (23%) did not respond; no reason was given. The remaining ten participants
gave varied answers. Some difficulties arouse from “getting [students] to understand something they
have never done before” to the participant admitting that they “need more experience in using modeling
techniques.” Additionally, “lack of resources” and “funding issues” has hindered the full use of modeling
techniques in the classroom. In some cases, the class using the modeling technique found the techniques
difficult because they “could not actually do exploratory experiments.” Even with these stated
difficulties, most of the participants reported that they will continue to use modeling techniques in their
classrooms.
Question #8:
Have students used MBL measurements in your class? What kind of use was it?
Two participants (15%) did not respond to this question; no reason was given. The remaining 11
participants reported that six participants (55%) used MBL measurements; two participants (18%)
reported that they did not use MBL measurements. Of the six participants that used MBL measurements,
five participants (83%) reported that they used Logger Pro.
-9-
Question #9:
Have students in your class used the Graphical Analysis software? If so, what kind of use?
All 13 participants responded to this question. Of the 13 participants, eight participants (62%)
reported that they have used Graphical Analysis software in their classrooms; five participants (38%)
reported they did not use Graphical Analysis software. Of the eight participants that used this software,
they reported they used this software to graph information, collect information, modify information,
correlate data, and/or used it to discern mathematical relationships.
Question #10:
Have your students tried and/or learned the linearization technique of discovering mathematical
relationships?
The linearization technique is powerful because it enables the user to infer mathematical
relationships between measured variables. However, it requires understanding of algebraic relationships
which sometimes are not understood by students. In effect, students must perform algebraic
transformations of their variables which is not a trivial topic. All 13 participants responded to this
question. Five participants (38%) reported that they have tried and/or learned the linearization
technique; eight participants (62%) responded that they have not tried and/or learned the linearization
technique, with most stating that their students “seem to be unable to conceptualize mathematical
relationships” using this technique.
This is an important area in which mathematics and science courses could support each other.
Question #11:
Have you looked on or followed these types of Modeling:
Eleven of the 13 participants (85%) responded to this question regarding the Modeling Website
at ASU. The remaining two participants (15%) did not state as to why they had never used and/or
followed the ASU website. Of the 11 that reported using ASU’s website, three participants (27%)
reported that they used the site “a lot.” Six participants (55%) reported that they used the ASU site
“sometimes.” Overall, of the 11 participants that answered this question, evidence suggests that the ASU
website is utilized. The two participants (18%) that reported not using the ASU site did not state a reason
as to why.
- 10 -
Ten of the 13 participants (76%) responded to this question regarding Schober’s modeling site.
The remaining three participants (23%) never stated why they had not followed and/or used Schober’s
modeling site. Of the ten participants that reported using Schober’s site, two participants (20%) stated
they used this site “a lot.” Five participants (50%) reported using the site “sometimes.” Of the ten
participants reporting using Schober’s site, seven participants (70%) stated that they used the site on a
regular basis. The three participants (30%) that reported never using Schober’s site did not state as to
why they did not use the site.
Twelve of the 13 participants (92%) responded to this question regarding the usage of other
modeling listservs. Six participants (50%) reported using other listservs “a lot.” Five participants (42%)
reported that they used other listservs at least “sometimes.” One participant (8%) reported not using
other modeling listservs. Overall, 11 participants (92%) used other listservs, suggesting that participants
want to learn more about modeling techniques.
- 11 -
Question #12:
If you used online resources, did you get anything useful from any of them? Why or why not?
Two of the 13 participants (15%) did not respond to this question; no reason was given. Overall,
most participants reported that using online resources aided them in their instruction. One participant
reported that online resources helped “explain topics” and offered different worksheets. Additionally,
these different online resources “gave a different way to introduce material” or “teach a concept.” Some
of the participants found the online resources useful in supplanting “overall strategies and lesson plans.”
Furthermore, participants reported that online resources helped to gain “general background
knowledge.” One participant reported that they used online resources from the ASU modeling site with
“videos that were incorporated throughout the school year.” Of the negative reflections of using online
resources, participants expressed complaints of sources being too limited, as in “topics [being] specific to
physics.” Other complaints were participants were “limited on time” and could not vet all the sources.
Question #13:
Have ideas from the Modeling workshop influenced your work as a teacher?
All 13 participants responded to this question. One participant reported that they are “more
sure in their teaching” and the way they “teach will continue to change” as they learn more about
modeling. Another participant reported that because of the modeling their students had “more ‘a-ha’
moments” and that “modeling makes the concepts clearer to them and their students.” Additionally, one
participant notes that modeling has “revamped and energized their teaching,” stating that “whiteboard,
lab investigations, and independent problem solving are now the cornerstones” to their teaching.
Besides reporting a change in teaching confidence, participants reported that “students have shifted the
responsibility of learning onto their own shoulders” and are eager to engage in more “exploring and
learning.” One participant states that: “[They] wish they had years of experience to make modeling even
more productive” and that the “whole modeling idea is invaluable.” Another participant conveyed the
importance that modeling had in their classroom, stating that modeling helps “students see real world
problems” and “[helps] students to see the relevance of these problems in their everyday lives.” Overall,
the overwhelming praise and positive feedback from the participants suggests the modeling workshop
has had a significant influence on teachers' day-to-day classroom practices. The teachers themselves
report that their changes in practice are causing students to be more powerful scientists.
Question #14:
To what extent have these ideas changed or influenced your teaching?
All 13 participants answered this question. Although a few of the participants were hesitant
answering this question, stating that they “still need time to fully implement what they have learned,”
most of the participants responded that they had been positively influenced by the modeling workshop.
One participant in particular stated that they: “have encouraged other science teachers to enroll in any
available modeling workshops.” Other participants conveyed strong feelings that the workshop helped
them in their own teaching, stating that the workshop “gave much needed confidence” and “helped
transform their own way of teaching.” Participants reported that classroom preparation, lesson plans,
and assignments now have “modeling strategies integrated.” One participant went as far as to say that
they “look at students differently” and “no longer spoon-feed answers” to their students, making their
own teaching “new and fresh and exciting.”
- 12 -
Question #15:
Do you have any evidence that things are changing for your students as a result of your taking the
modeling workshop?
All 13 participants responded to this question. One participant noted that “ACT scores are rising
across the board in [the] sciences” and that “more students seem willing to take the science classes after
they have seen what modeling classes are doing.” Another participant reported that “students who were
normally low C or high D average students” in other science classes were now “earning B’s in Physics.”
One participant stated that they “enjoy[ed] teaching physics more” because of the modeling workshop.
Numerous participants reported that they saw “students more willing to tackle unfamiliar topic[s] now
than in previous years,” attributing this to modeling techniques. Additionally, participants have seen
more confidence in students, a higher level of eagerness to learn, and better retention levels.
Furthermore, student writing skills has developed and is now showing “deeper thinking” and “better
conclusions.”
Question #16:
Is there anything else you would like to tell us about your use of modeling in your classroom?
All 13 participants responded to this question. Most of the responses reiterated responses from
questions 13, 14, and 15; however, a few participants did have suggestions. One participant suggested
that they would like to “see a listserv of the modeling participants so they could stay connected and ask
questions.” Another participant responded that they are encouraged in seeing their “students
developing more sophisticated thought processes and better communication skills.” The only negative
responses from participants reflected the wish to “have modeling incorporated into other subject areas
like biology” and for “districts to allow more time to truly learn [courses] through modeling.”
VII.
Lessons Learned and Implications
The 2011 Modeling Part 2 workshop provided teachers with additional training and experiences
in using the Modeling Materials. The majority of teachers came to the training with strong understanding
of Newtonian mechanics and with prior experiences in modeling. The participants clearly valued and
welcomed the modeling approach taught during the three years. The majority (around 75%) used crucial
aspects of the modeling approach in their classrooms such as whiteboarding and the laboratory
introduction scheme. Consistent with claims made by national directors of the Modeling Project, South
Dakota teachers often did need more than two weeks to fully "get it" and choose to implement the
modeling materials in sufficient completeness to transform their teaching. Questions 13 and 14 on the
evaluation reveal substantial self-reports of teacher transformations towards more inquiry-based and
student-centered teaching. Participant responses to question 15 show that these changes have
qualitatively improved student learning in science - students are more inquisitive, more competent at
asking and answering questions, they enjoy physics class more and more are taking physics. Also, ACT
scores have risen due to teachers adopting the modeling approach.
The time required for the full modeling workshop (a minimum of three weeks for teacher
transformation) has provided logistical and fiscal challenges in South Dakota. Traditionally, teacher
workshops have only lasted one week and most teachers - who often find additional sources of income
during the summer - are reluctant to commit to a workshop that will last two or three weeks. Also, Title
II funding limitations restricted workshops to a maximum of two weeks. The project leaders' compromise
- to split the Modeling workshop in half and schedule two two-week workshops over multiple summers seems to have worked for those teachers who were able to attend both the Part 1 and Part 2 workshops.
Those who attended for one of the two-week sessions did gain useful knowledge but it appears that the
Part 2 session has led to greater use of the modeling approach by participants.
- 13 -
Not surprisingly, the majority of attendees were from western South Dakota. The project
directors - Johnson and Emanuel - hope to offer modeling courses in the more populous eastern half of
South Dakota in the summer of 2013 and beyond.
VIII.
Summer 2012 Followup
The summer 2011 workshop was followed up in the summer of 2012 with an extensive week of
additional training on inquiry based approaches to teaching light and color. Participants enthusiastically
embraced the content, research-based materials, and inquiry-based method presented in the followup
session.
The Light and Optics Conceptual Evaluation (LOCE) was used as an indicator of content learning. Not all
of the content of the LOCE was taught in the followup session, but participant scores improved in the
relevant categories as shown below.
Participants were asked to write daily journals during the followup session. The teachers were clearly
thinking about their roles as teachers and how they could use the approaches they were experiencing in
their own classrooms. One teacher wrote:
There is tremendous value in being wrong, if students are taught to embrace the value in it,
they could then realize how much more exciting it is observe what actually does happen and
not only that but to try to explain why. To figure out what is actually happening and know
why.
This teacher was noting problems with the traditional approach to presenting information, which is that
students cannot recognize the value or importance of much of the science presented to them unless they
work out some understandings themselves.
IX.
Overall Conclusion
The 2011 Modeling Physics Part 2 workshop and 2012 followup were conducted by experienced, skilled,
and committed leaders. Content evaluations show that the participants gained important knowledge in
physics content - additional topics in mechanics and light and color. The opinion survey showed that the
participants also developed greater understanding of and appreciation for learner-focused approaches to
teaching. The teachers valued the experience - at least three thirds have applied important aspects of the
Modeling pedagogy to transform their classrooms. Students in these classrooms will encounter deeper
and more satisfying experiences with physics and other sciences, and are quite likely to emerge with
better understandings than they would have otherwise.
- 14 -