Wisconsin Teaching Fellow Final Report – John Colton, UW-La Crosse
1. Explain the question, problem, or issue related to teaching and learning you investigated.
In Physics, as in other areas, exams can be given in many different formats, or “exam styles”:
 students sometimes are prohibited from using any notes, while taking the exam
 students are sometimes given key formulas on the exam by the professor
 students are sometimes allowed to prepare a sheet of notes which they may use on the exam
 students are sometimes allowed to use their class notes and/or the textbook
 students are sometimes given take-home exams
The most common method that I personally have used, is to allow the students to prepare a one-page “notesheet” for themselves, typically containing key formulas, which they may use for assistance during the exam.
This is also perhaps the most common method of giving exams among other physics professors I have asked.
The general feeling is that this will allow students to focus on “what is really important”, rather than “wasting
time” memorizing formulas. The thought is that because the students see the fundamental concepts and
equations so many times and places during their undergraduate career, they will end up having the most
important things memorized even without having needed to memorize them for individual exams along the
way.
However, as I was interviewing students after they took the ETS “Major Field Test” given by our department
for the first time in 2006 (completely closed book & notes), I found that many of the students had not known
some very basic physics equations. Things that I considered fundamental definitions and concepts—which
must be surely known by every physicist—were viewed by some of the students as superfluous items which
they couldn’t answer because they “didn’t have the formulas memorized”.
The broad issue I wished to explore, set up by this experience, is: are we doing more harm than good by not
requiring the students to memorize equations and take exams closed-notes? More specifically, in this research
project I wanted to get at issues such as the effect of exam style on students’ study habits and attitudes.
2. Explain the methods or approach you used to investigate the question, problem or issue.
During the Fall 2006 semester, in an PHY 335 (Electronics), I gave various styles of exams throughout the
semester (one with a note-sheet permitted, one closed book, one open book/notes, and one take-home). At the
end of the semester I interviewed the students in focus groups to see what they thought about the various exam
styles. These focus groups indicated that exam style was perceived by students as having various effects. Some
effects mentioned by multiple students included:
 accuracy of assessment of students’ ability
 stress level
 amount of learning that occurred
 amount of studying that occurred
 type of studying that occurred
 retention of material
Analysis of the focus groups’ comments allowed me to write a survey to be given the following semester,
which would probe at some of the most important potential effects of exam style on students—see the attached
document, “coltonsurvey.doc”
During the Spring 2007 semester, in PHY334 (Circuits), I conducted a controlled experiment: I split the class
randomly into two groups. Group 1 took the first exam closed-notes, then the second exam with a note-sheet
allowed. Group 2 took the first exam with a note-sheet, and the second exam closed-notes. (I.e. each student
took one closed-note exam and one note-sheet exam). The experiment was performed in this fashion to
1
eliminate apparent differences that could arise from e.g. the second exam being more difficult than the first
exam.
After each exam, both groups were surveyed to find out how they had studied, how they felt they had done on
the exam (and why), their anxiety level prior to the exam, anticipated retention of the material, and
attentiveness during lectures. Differences between closed-notes and note-sheet groups were analyzed via chisquare tests to see which if any survey responses were statistically different between the two groups.
Additionally, students were allowed to make open-ended comments on their experiences and opinions.
The groups’ exam scores were also analyzed to determine whether the note-sheet group did better than the
closed-notes group, or vice versa.
3. Summarize the results even if they are preliminary.
The results were quite interesting—both to see where differences existed between the two groups, and where
they did not.
Areas where exam style caused a clear difference
Study time: There was a very significant difference in time spent studying for the exams: 77% of the students
said that they studied more for the closed notes exam than they would have studied for a typical exam. When
asked how much they had studied, 60% said that they had spent more than 4 hours studying by themselves for
the closed note exam; only 45% said that they had spent more than 4 hours studying by themselves for the
open notes exam.
Exam study habits: There was a very significant difference in how likely students were to prepare formula
sheets to help them study for the exams: 74% of the students did not prepare a formula sheet to help study for
their closed-note exam. By contrast, all but one student prepared a formula sheet to help study (or to help use
during) their note-sheet exam.
Areas where exam style may have caused a difference
Homework time: The amount of time spent doing homework may have been significantly different: students
spent more time doing homework prior to their closed-note exam. Specifically, 37% of the students spent 4+
hours per week doing homework by themselves leading up to the closed note exam; by contrast, only 14% of
the students studied 4+ hours per week by themselves leading up to the open note exam.
The chi-square analysis of this question yielded a p-value of 0.27. This indicates that there was likely some
actual effect here, because there was only a 27% chance that the differences in responses could have been due
to chance alone. However, this falls short of the traditional statistical standard (of p = 0.05) to indicate a
difference beyond doubt. Thus this is an area where further investigation is warranted.
Attention in class: The amount of attention paid in class may have been significantly different: students
indicated they paid more attention in class prior to their closed-note exam. Specifically, 74% of students said
that they paid a “large amount of attention” in class prior to the closed-note exam; only 59% of students
reported paying a large amount of attention prior to the open-note exam.
Chi-square analysis of this question yielded a p-value of 0.18; that is, there is only an 18% chance that the
difference in responses could have been due to chance alone. As with the “homework time” question, this is
enough to indicate a likely effect which warrants further investigation, but not enough to state conclusively that
the effect could not have been due to chance.
Areas where exam style caused no difference
2
Exam score: Exam style turned out to not significantly affect the students’ exam scores: almost exactly half the
class did better on their closed-note exam, and half did better on their open-note exam (relative to the class
average). On the first exam, the note-sheet group scored highest; on the second exam the closed-notes group
scored highest. That is, the same physical group of students scored highest on each exam. This contradicts the
students’ impressions that they will score higher on note-sheet exams (see student comments below).
Perceptions of validity of exam score: After each exam style, the vast majority of the students (75%) felt that
their exam score did not accurately reflect their ability, and was lower than they deserved. There was no
significant difference between the two exam styles, however—students after the closed-notes exam felt like the
exam did not adequately assess their true ability in nearly the exact same way that students after the note-sheet
exam did.
However, a large difference occurred in why the students felt like their exam score was invalid: 89% of the
students in the closed-notes group listed absence of note sheet as a factor for why their grade was lower than
they deserved, with 52% of those indicating that it was a moderately significant factor or larger. Since the
overall level of complaint was no different than in the note-sheet exam, however, it seems that the lack of notesheet had no real effect, but merely served as a focal point for the students’ complaints. (Students in the open
note group attributed the inadequacy of the exam’s assessment to other sources.)
Anxiety: Although 48% of students felt like anxiety had lowered their exam score, this was nearly identical
for students talking about the closed-note exam vs. the note-sheet exam. Thus students did not seem to feel that
the “no note-sheet” contribution towards their anxiety level affected them beyond their typical anxiety.
Exam study habits: All of the students worked sample exam problems to help them prepare for both exams.
77% of students re-worked homework problems to prepare for exams, regardless of exam style. 98% of
students reviewed class notes to prepare for exams, regardless of exam style.
27% of students did not read the textbook to help prepare for the exams, regardless of exam style. 62% of
students did not take any notes from the textbook to prepare for the exams, also not depending on exam style.
Thus, the only study preparation item which was affected by exam style was the preparation of the note-sheet
itself, as mentioned above.
Estimated Retention: Students were asked to estimate how much material they would remember three months
from the exam. They did not answer differently for the closed-note exam as compared to the note-sheet exam:
in each case, 25% said they would remember the material “slightly”, 62% said they would remember it
“moderately”, 10% said they would remember a “large amount”, and 3% said they would remember “nearly
all”.
Therefore, students apparently did not feel that they themselves would remember any better for the closed-note
exam than they did for the open-note exam. This contradicts what several students answered in the open-ended
question regarding advantages to a closed-note exam, where they said a closed book exam ought to help
students remember the material better.
Student comments in open–ended responses
Also quite illuminating were many of the students’ comments in the open-ended response section. Some
interesting and/or representative comments are given, sorted by common themes:
Benefit of a “closed-note” exam
Causes students to study more
• “[offers] a larger incentive to spend more time studying”
3
• “You are forced to study even more.”
• “…you don't have a false sense of security, like you do with a note sheet.”
Aids learning
• “You work harder and remember more”
• “[puts things in] long term storage”
Better assessment of student learning
• “more accurately reflects student knowledge/learning - material must be known better”
• “Making sure students really understand, and aren't just winging it based on a notes sheet.”
• “you rely on your knowledge and your understanding rather than just a note sheet”
No benefit to closed-book exams (many)
• “Memorizing formulas you can easily look up if needed is completely worthless”
• “No real point, formulas are typically forgotten soon after the end of the semester anyone.”
•
“…my personal belief is that if an equation is important/used enough that it should be memorized,
you will either eventually just remember it from looking it up so much, or get sick of looking it up and
spend the time to memorize it” (Same attitude as most Physics professors—but is it true?)
Benefit of a “note-sheet” exam
Aids learning
• “can focus more on the concepts than just the equations and math” (many)
• “It's a very good study guide when studying for the exam.” (True, in my opinion! So why didn’t they
do that to study for the closed note exam?)
• “As a professor once said to me, the physics is not in the formulas, but knowing how they are used”
(Common Physics professor attitude)
More real-world
• “More practical in the real world”
Relieves stress (many)
• “You don't have to memorize simple equation which can allow for less stress.”
• “I personally was a lot more calm going into the [note-sheet] test.”
• “[provides] a security blanket for a little more confidence”
Better assessment of student learning
• “...a note card allows you to double check the equation you wrote down. Your brain can work in
strange ways when it is under pressure and you can forget the most basic info and the note card just
gives you that safety blanket”
Higher scores
• “You can get a higher score while knowing less physics”
• “Less deducted points for simple memory lapses”
No benefit to note-sheet exams
• “There are none [if the professor gives the hardest formulas on the exam]”
4. What are the implications of your work for your teaching and student learning and more broadly
for the practice of teaching in your field?
Exam style is obviously a complicated topic, as evidenced e.g. by the students’ open-ended comments: some
students felt closed-note exams more accurately assessed student learning; others felt the opposite. Some
students felt closed-note exams more aided learning; others felt the opposite. So students’ perceptions of exam
styles—and professors’ as well, undoubtedly—demonstrate a wide range. The complicated nature of
“perception vs. reality” was also demonstrated in the students’ overall unhappiness about how well the
exams—both exam styles!—accurately assessed their ability.
A few items were clear from this project: students did study more for the closed-note exam, and students
taking the exam closed-notes did not perform any worse than students allowed the note-sheet. Thus any
disadvantage in the taking of the exam without a note-sheet was offset by additional learning gained by
studying longer for the exam.
4
Some items were not as clear: in preparation for the closed-note exam, students may have invested more time
doing homework, and they may have paid more attention during class. But these effects were not as large, and
were potentially ambiguous. Also, many students felt that in general more information would be retained in a
closed-note exam, but when probed for details in how much information they themselves would retain, there
was no difference between the closed-note and the note-sheet exams.
Overall, I would say that this project has yielded many interesting results which can serve as a springboard for
further research in this area. There have been many prior papers published on the concept of open-note exams,
but overwhelmingly they have been full of opinion and/or anecdotal evidence, and empty of concrete data.
Hopefully the data I was able to obtain will thus add to the overall body of research in this area.
5. Describe how you have presented and/or will present the project to others (e.g., presentations on
campus or at professional meetings, publications, open classroom, etc).
So far, the only formal presentation has been my poster at the OPID Spring 2007 Conference. Informally,
however, I have had some discussions of the results with colleagues (see question 8, below, for one example).
I am considering submitting these results for publication in an education research journal, but have not yet
taken serious steps along that path.
6. Describe any products such as papers, publications, course portfolio, web-based materials, etc. you
have produced related to the project.
None, yet.
7. Describe the next phase of your work on the project. If the project is not yet completed, what
remains to be done? If it is completed, describe any plans to continue working on questions, problems,
or issues related to teaching and learning.
This particular project is completed, but as noted above some things warrant further study—specifically the
time spent on homework, and the attention paid during class. Those could be topics for a follow-up study.
Also, the retention question is interesting—what the actual retention is like, rather than just the students’ guess
of their retention.
8. Describe the impact of your work on your colleagues, departments and institutions. Will others use
your methods or findings? Can you describe any changes that seem to be in process?
I am currently teaching a section of a large introductory class at BYU, in combination with another professor’s
large section (~600 total students). As a result of this research, we have decided that for this semester we will
try a hybrid approach, where we will give some equations on the exams (the more complicated ones), but
require students to memorize others (the more fundamental ones). The hope is that this will catch some of the
best features of the different styles.
9. Explain how you spent your $500.00 OPID S&E Grant during the year.
I used the grant to purchase equipment to aid in transcribing the interviews—a digital voice recorder, speechto-text computer software, and a soundcard which was required by the software. For the benefit of others’
future projects: I found the voice recorder to be very helpful, but the software did not work nearly as well as I
had hoped.
5
“Part 4” – My experience in the Teaching Fellows program
1.
How has the Teaching Fellows/Scholars program contributed to your development as a teacher-scholar?
I think this is best answered via my replies to question 2, below.
2.
What were the most beneficial or rewarding aspects of the program?
a. Associating with other fellows - This was the most immediate reward that I discovered from the first get-together.
It was great to meet fellow faculty members that had many things in common with me, even though their disciplines
were very different. They were a great group of people who were bright, dedicated to “normal” scholarship, but also
dedicated to teaching, and the scholarship of teaching & learning. This contributed to my development by giving me
incentive to live up to their example, and to “keep my chin up” when things seemed at times overwhelming.
b. Learning the basics of how to do a SoTL project – I had heard of SoTL, and had a vague idea of what SoTL
projects entailed—but I was very clueless about the actual mechanics of how to do a SoTL project. I had no idea
about student consent forms, IRB approval, what kind of data can be obtained, how to analyze the data, etc.
Learning all these things obviously will impact my ability to be a teacher-scholar in the future by a tremendous
amount. If I do more SoTL projects, it will only be because of the foundation this program laid for me.
c. Learning other helpful teaching techniques – By being “forced” to attend various conferences as part of the
Teaching Fellows program, I was able to hear many interesting talks about teaching techniques that were not
necessarily even related to SoTL. I picked up many interesting ideas of things to try in the classroom, to make me a
better teacher. That was a definite benefit of the program.
3.
Open microphone—please comment on anything else about the program.
Probably the thing I appreciated the most was the flexibility of the program. It was great that the program leaders
were willing to let us explore a bit: do our own thing, go at our own rates, and not be pressured into doing things in
an assembly-line way. I very much liked how informal our meetings and get-togethers were. Those aspects very
much contributed to the overall “enjoyability” of my experience.
6