ASSESSING HIGHER-ORDER THINKING IN LARGE INTRODUCTORY SCIENCE
CLASSES
RICHARD F. YURETICH
Department of Geosciences
University of Massachusetts
611 N. Pleasant St.
Amherst, MA 01003-9297
yuretich@geo.umass.edu
Engaging students sufficiently in the subject matter to stimulate critical thinking
and higher-order reasoning is a challenge in many teaching situations, but the
difficulties are often most acute in introductory-level science courses. The large
classes in lecture halls often found in colleges and universities complicate the
effort even further. Active-learning methods, such as cooperative in-class
activities, and on-line quizzes or homework with rapid evaluation and feedback,
help to promote higher-level reasoning. In addition, multiple-choice exams can be
modified to include questions involving analysis, synthesis and evaluation of
diagrams, situations and data. Exams that include cooperative components help
align the assessments with the educational strategies, and these also provide
opportunities to exercise critical thinking. Data from student performance,
surveys, and interviews, confirm the efficacy of these techniques.
Introduction
Many science and mathematics classes at large universities are taught as lectures in
auditoriums to large numbers of students, often more than 200. Even at smaller institutions,
introductory-level classes of 50 to 100 students are not unusual. Such classes evolved from a
perception that information transfer from professor to student is essentially independent of the
setting in which it occurs. Large lectures are viewed favorably by administrators as efficient
instructional vehicles, and departments find that such classes help boost the FTE count in their
programs. In many cases, these large lectures are divided into smaller discussion or laboratory
sessions that are supposed to promote greater engagement on the part of the student. However,
laboratories are often decoupled from the lecture, so that the connections don’t carry across. In
addition, many colleges and universities are scaling back the laboratory requirement in science to
help address budgetary difficulties and the only opportunity for supervised learning may be
during class time. We may deplore these trends based on educational principles, but the reality of
modern higher education is that the large lecture is here to stay.
At the same time, many faculty members will assert that one of their goals in teaching is
to encourage students to “reason like scientists” or “think critically,” qualities which require
some nurturing, encouragement, personal guidance, and careful assessment. Clearly large classes
pose special challenges to teaching and learning at higher levels of "critical thinking." Is it
possible to adjust our instructional strategies in such a way that students in large lecture classes
can move beyond learning “just the facts?”
Bloom [1] has organized the types of learning behaviors into a hierarchical classification
(Table 1). Lectures tend to focus on the first (information) and second (comprehension) levels,
which are relatively easy to attain in this mode. Assessment of the students’ abilities at these
cognitive levels can be achieved easily through traditional examinations. If the instructional
goals include learning at higher levels, which encompass the generic term of “critical thinking,”
then assessment methods need to be aligned with these goals.
Table 1. Bloom’s taxonomy of learning levels and some skills demonstrated at each level.
Competence
Knowledge (Information)
Comprehension
Application
Analysis
Synthesis
Evaluation
Skills Demonstrated
list, define, tell, describe, identify, show, label, collect, examine, tabulate, quote,
name, who, when, where, etc.
summarize, describe, interpret, contrast, predict, associate, distinguish, estimate,
differentiate, discuss, extend
apply, demonstrate, calculate, complete, illustrate, show, solve, examine, modify,
relate, change, classify, experiment, discover
analyze, separate, order, explain, connect, classify, arrange, divide, compare, select,
explain, infer
combine, integrate, modify, rearrange, substitute, plan, create, design, invent, what
if?, compose, formulate, prepare, generalize, rewrite
assess, decide, rank, grade, test, measure, recommend, convince, select, judge,
explain, discriminate, support, conclude, compare, summarize
The results and experiences I summarize
here are based on more than five years of concerted
experimentation with various teaching methods in a
large-enrollment course in introductory
oceanography at the University of Massachusetts
[2}. Each semester there are more than 600
students enrolled in two lecture sections taught by
the same instructor. Each section meets for 75
minutes twice weekly, and the two classes are
taught back-to-back in the schedule. There is
usually one teaching assistant for each section, and
there are no laboratory or discussion sessions. This
is a general-education course that is primarily for
first- or second-year students who are not majoring
in science or math. In fact, this may be the only
course in the physical sciences that these students
take in their college careers. The challenge has
been to make this course an effective learning
instrument for the majority of students enrolled,
who come from a wide variety of backgrounds and
preparation in science, and to engage them in the
type of careful reasoning that characterizes
scientific investigation.
The Gulf Stream
The Gulf Stream is a western
boundary Current that originates in the
equatorial Atlantic, flows up the eastern
coast of North America and crosses the
North Atlantic towards Ireland and England.
The water returns to the equatorial region
by way of the Canary Current along the
western coast of Africa.
1).
What causes the Gulf Stream to flow?
2).
What are the major physical
characteristics of the Gulf Stream?
What distinguishes it from the
surrounding water?
3).
Is productivity high or low within the
Gulf Stream? Why?
4).
Where is the Sargasso Sea? How did it
get that name, and why is it there?
5).
What are meanders in the Gulf Stream,
and how do they form?
6).
How does the Gulf Stream affect the
ocean ecology off the coasts of the
northeastern United States and
eastern Canada?
7).
How does the Gulf Stream affect the
climate of Europe?
Fig. 1. An example of an oceanography
in-class exercise that promotes critical
thinking [4].
Methods
IN-CLASS EXERCISES. Lecturing alone does not usually promote higher-level
information processing, and, consequently, other active-learning strategies during class are
encouraged [3]. In-class exercises are one way that critical thinking skills can be introduced into
large classes. Questions can be designed so that through discussion and cooperative-learning
methods, students are forced to process the information before reaching a conclusion. Figure 1 is
an example of an exercise used in the oceanography course, which requires students to
synthesize and evaluate information that they have gathered from the readings and lectures [4].
In the oceanography class, students do these exercises as “think-pair-share”, where they
contemplate the answers, discuss them with their neighbors, and then the entire class reviews the
answers together. In this particular exercise, the students have watched a short video segment
about the Gulf Stream, so these questions also serve as a way to focus their attention on the
substantive parts of the video.
ON-LINE INTERACTIVE QUIZZES. There are many supported platforms now
available for students to complete on-line homework assignments or quizzes. These systems
have the flexibility to ask more involved questions than just multiple-choice, and to have these
graded with feedback to the students. The ability to ask more questions, give feedback, and then
have the students repeat the quiz or homework, provides excellent opportunities for critical
thinking, in-depth analysis, or higher-order processing of data. In the example provided (Fig. 2)
students use a variety of critical-thinking skills. Questions 1 and 2 emphasize comprehension,
where students must interpret the information on the graph. Question 4 also requires graphical
interpretation, but they must relate and connect the diagram to the process of tidal cycling.
Accordingly, this tests their ability at analysis and application. Questions 3 and 5 are questions
Fig. 2. An example of an on-line quiz used in the oceanography course.
involving quantitative reasons. In addition, rather than have a list of choices, the student must
enter the correct words or phrases in the answer box. If the answers are not correct, the program
will give some guidance on possible answers. Calculation questions are especially valuable online because the numerical values change with each succeeding attempt. If a student gets an
incorrect answer the first time, the feedback will display the correct answer. When she or he then
does the quiz again, the numerical values in the question will change. For example in question 3
of Fig. 2, the wavelength (L) will be a different value during a subsequent iteration. The student
must be able to solve a formula to obtain the desired answer. Question 5 involves more than just
“plug and chug” into an existing formula. The student must be able to understand the basic
principle of the tidal cycle and the different kinds of tides in order to obtain the correct answer.
Again, if the wrong answer is given, the re-take will come back with a different time for the high
tide. Here they are integrating ideas and calculations in order to formulate an answer, skills that
match the “Synthesis” level of Bloom’s taxonomy (Table 1).
RE-INVENTING THE TRADITIONAL EXAMINATION. Multiple-choice
examinations are often called “objective” tests, with the implication that they are only useful for
assessing lower-order or fact-based knowledge. In large classes, machine-scored exams may be
the only practical solution to routine assessment, so that if we are truly devoted to engaging
students in higher-order learning, then the multiple-choice exam needs to be adapted to these
goals. There are two strategies that work:
1. Writing questions that require application, analysis, synthesis or evaluation. These are
not as difficult to compose as they might seem, as illustrated in the following
examples:
On the following “continent”, choose the letter corresponding to the place where coastal
upwelling will most likely to occur:
A
subtropical
gyre
(continent)
North
B
C
subtropical
gyre
D
This question tests the students’ abilities to apply and analyze. They must have a basic
knowledge of the facts about ocean circulation, the Coriolis effect and how these interact in the
coastal region. Then they need to analyze the patterns and apply them to this abstract situation,
which they have not seen before.
A)
B)
C)
D)
In the situation illustrated in the diagram in Fig. 3, what will happen over the course of time?
Sand will accumulate at locations 1 and 2.
Sand will erode from locations 1 and 2.
Sand will accumulate at location 1 and erode at location 2.
Sand will erode at location 1 and accumulate at location 2.
sw
ell
2. Changing the nature of the multiple-choice exam so that
students can learn from the examination process
coast
In this question, the students need to synthesize and evaluate.
They must first interpret the diagram as a representation of a coastline
with groins, and then determine the prevailing direction of longshore
drift. Then they must evaluate the impact of the groins on the
movement of beach sand, and decide what the likely outcome will be.
1
This may seem like a tall order, but it can occur with the
strategy known as the pyramid exam [5], which we have adapted
successfully for use in our large introductory oceanography course.
The essence of the pyramid exam is that students re-take an exam one
or more times, working in successively more collaborative settings to
complete the test. In the original design, a very difficult exam is
2
repeated several times throughout the course until the entire class
works together to solve the final most challenging problems. We use an
Fig. 3. Sketch map
adaptation called the two-stage cooperative exam, in which students do a
of a coastal region
multiple-choice exam twice . The first go-round is a traditional test, and
students fill out optical scanning forms with their answers. However, after they hand in the
answers, they are given new answer sheets and they re-take the test, discussing their answers
with other students. These two parts are done during the same class period. For grading
purposes, we take 75% of their individual scores and add them to 25% of the cooperative scores
to arrive at a grade. The cooperative exam raises the class average between 3 and 5 points, but
the most important part is that it turns the much-maligned multiple-choice exam into a learning
experience. Because students discuss the questions, answers, and logic or principles behind the
questions, they are analyzing, synthesizing, and evaluating the topics, and they are thereby
employing higher-order learning skills.
Evaluating the Assessments
The critical question remains, as always, "How do we know that these methods are
effective?" The evidence in this case comes from multiple measures: analysis of student
performance, surveys and interviews.
EXAM PERFORMANCE. The class as a whole has been improving in the numerical
scores on exams. The most recent class outperformed its predecessors by a wide margin on all
in-class examinations save the first one (Table 2). The differences among the exams are
significant at the 99% level, owing to the large sample population and ensuing degrees of
freedom. Cooperative exams were administered in 1998 and 2002, and the chart compares the
results only on the solo portions of the exam. There is a progressive increase in scores during the
entire period that the active-learning techniques were being introduced, but the incremental
increase in exam scores for the most recent semester is the largest. Although the University
contends that the student body is more capable now than in the past, I can’t say that this is
obvious from my own experiences in the classroom. I therefore conclude that the modifications
to the teaching of the course have had an impact on student learning.
Table 2. Comparison of examination results from several years in the oceanography
course. 1998 and 2001 data are from the solo portion only of the collaborative exam; 1993 and
1996 are traditional exams. Highest grades are in bold type.
Exam 1
Exam 2
Exam 3
Exam 4
Final Exam
Overall
2001
70.4 ± 15.8
79.9 ± 11.6
74.5 ± 13.0
80.4 ± 12.0
80.9 ± 11.5
77.1 ± 13.6
1998
71.1 ± 13.8
77.8 ± 12.5
70.1 ± 12.8
75.8 ± 13.0
77.9 ± 12.4
74.5 ± 13.3
1996
73.1 ± 13.2
71.5 ± 14.5
75.8 ± 12.6
---71.9 ± 12.4
73.0 ± 13.3
1993
71.5 ± 14.6
68.5 ± 13.4
75.0 ± 11.9
---74.8 ± 12.1
71.6 ± 15.3
Syn
I
I
I
I
I
C
C
C
C
App
App
% Change in Correct Response
Details of comparable final exam questions show that the greatest improvement occurred
when active-learning strategies were incorporated for the first time in 1998 (Fig. 4). Although
there was improvement in students’ abilities to answer all questions, of particular note are the
positive changes in Comprehension (C), Application (App) and Synthesis (S) questions. A prior
analysis of the data showed that there was equal improvement in the class’ answers to questions
from earlier in the
semester as well as
30%
more recent topical
25%
material, indicating that
their ability to retain the
20%
subject matter had been
15%
Average Change
enhanced [2]. The
1996-98
10%
2001 class showed
5%
incremental
improvement over
0%
Average Change
1998, although there are
1998-01
-5%
no obvious trends or
-10%
preferences related to
1996-8
-15%
the type of question as
1998-01
classified according to
-20%
Bloom’s Taxonomy.
There are differences
Bloom's Taxonomy Category
among the individual
questions, but the
teaching strategies have
Fig. 4 Comparisons of responses to specific questions on final
obviously matured to the
exams in 1996, 1998 and 2001.
point where large
changes would not be expected.
The exam results don’t allow us to factor out the relative importance of in-class exercises,
on-line quizzes and cooperative exams in influencing the performance changes. There is some
correlation between the number of in-class-exercises completed and the performance on the solo
portion of the exams (Fig. 5); a best fit line gives an R2 value of 0.44. There is no correlation
between the total number of on-line quizzes completed and exam performance.
120.0
y = 2.6693x + 34.668
2
R = 0.442
Average Grade on Solo Exams
100.0
80.0
a) Rate the usefulness of outside
assignments in helping you learn.
b) Rate the usefulness of lectures in helping
you learn.
c) Rate the usefulness of in-class activities
in helping you learn.
60.0
40.0
STUDENT SURVEYS. Exam
performance is only one way of measuring
learning; student surveys give another
perspective on the issue. In 2001, we asked
several additional questions about the
students’ experience in the course. Three
questions relate directly to the learning
process:
Exercises
Linear (Exercises)
20.0
Student responses to these questions (Table
3) were based on a five-point Likert scale:
5 = Almost always useful
0.0
4 = Usually useful
0.0
5.0
10.0
15.0
20.0
3 = Sometimes useful
Total Score for Exercises
2 = Rarely useful
Fig. 5. Relationship between semester score for
1 = Almost Never useful
in-class exercises and solo exams.
The results indicate that both the
assignments (on-line quizzes) and the inclass activities were perceived as very useful, on a par with the perceived value of the lectures.
An earlier survey done during the course in 1997 (Fig. 6) showed a similar proportion of students
who felt that the in-class exercises increased their learning. In addition, an even greater
proportion agreed strongly that the two-stage cooperative exams (pyramid exam) had an impact
on their understanding of the subject
Table 3. Student answers to survey questions about aspects of course that helped in learning.
Question
Assignments
Lectures
In-Class Activities
5
28%
39%
26%
4
37%
41%
36%
3
25%
15%
26%
2
7%
3%
9%
1
3%
2%
3%
N
317
317
317
S.D.
1.03
0.89
1.01
Mean
3.80
4.13
3.74
INTERVIEWS. Students were interviewed in focus groups by external evaluators when
the in-class exercises and cooperative exams were instituted in 1997 [6]. Comments during these
interviews provide evidence of a positive impact on critical thinking, mostly resulting from the
opportunity to collaborate on questions and problems during class:
“If you really did get involved with the group you’re in…you can understand things
better because people learn differently and they can teach you the way they see it and you can
better understand what is going on rather than just getting one perspective on how things are
done.”
“you’re taking your knowledge
and you are applying it to something,
so that helps a lot on the exams, to
better understand.”
160
In-class activities increased learning
140
In-class activities increased interest in course
120
Number of responses
“the co-op groups offer a chance
to get other ideas from students; in a way
to get a new perspective, either on a
subject or a new way of thinking.”
Pyramid exams increased learning
100
80
60
40
The cooperative exams also elicited
several comments about learning:
20
0
“I learned more from the pyramid
exams than from any other kind..”
Strongly
agree
Generally
agree
Neutral
Generally
disagree
Strongly
disagree
Fig. 6. Results of earlier student survey (1997) concerning
effectiveness of various instructional methods [6]
“I’m more likely to
remember [what a peer explains
during a cooperative exam] than what the professor said.”
“If I get help [from a peer during an exam], it might contradict an answer and it will
make me think about it more.”
Although written comments on the end-of-semester student surveys from the most recent
semester revealed several strong negative impressions about the on-line quizzes, most of these
related to the logistics of the system rather than the learning experience. For example, some
students had protracted difficulty logging in and completing assignments on time. Positive
comments outnumbered these negatives, and learning improvements were specifically mentioned
in a few instances.
Discussion and Conclusions
Higher-order thinking can be a part of a large-enrollment class, but it requires moving
beyond the traditional lecture and exam mode to encourage it. Active-learning methods offer the
best solution. When students really ponder a question, discuss it in groups, or explain their
answers to others, they are more likely to use skills at the more advanced levels of Bloom’s
Taxonomy. It doesn’t take a revolution to incorporate these techniques even into classes of
hundreds of students. Having the opportunity to pause, reflect, analyze, and discuss processes
and concepts is the real key. In the oceanography class, the in-class exercises are the major
devices for this effort. We typically do one every class, and the exercises are often multi-part, so
that the instructor can take 5 minutes for the class to work on a question or problem. Additional
lecturing can follow discussion of the answers, which will lead into the next set of questions for
the exercise. The in-class exercises can also be built around video segments, live demonstrations
or a hands-on/participatory experience for the class. Logistics may be a consideration for the
latter. In oceanography, for example, I’ve found that “doing a wave” in the lecture hall is an ideal
introduction to the difference between the movement of wave and the motion of the individual
water particles. Student surveys and interviews emphasize that such activities develop better
thinking and processing skills.
Internet or on-line activities and quizzes can be an effective means of developing higherorder thinking skills outside of class. Many on-line instructional platforms have quizzing
available that allows students to receive constructive feedback on their answers. It is also
possible to ask more open-ended questions requiring a short-answer response so that the students
don’t simply choose from a list of choices. Quantitative reasoning can also be emphasized more
robustly. Nevertheless, the evidence from the oceanography class indicates that completion of
the on-line quizzes has no influence upon the students’ performance in subsequent exams. There
are also issues about the technology that detracts from the experience of completing the quizzes.
These include problems students have in logging on to the system, having the program crash in
the middle of a session, and not having sufficient technological support. My most recent
experience suggests that these have become minor issues. Only five or six students out of 600 in
the Fall, 2001 chose not to participate in the on-line component.
Examinations are opportunities to promote higher-order learning that are not frequently
exploited in large classes. Multiple-choice questions can be written that do more than just ask
for information, although learning the basics is usually a goal for most introductory classes.
Interpreting graphs or diagrams, using data to analyze a situation, and applying the results of an
experiment to a new situation are the kinds of “objective” questions that entail higher-order
thinking. In addition, incorporating a collaborative component into the exam is an effective way
of increasing student learning. Students have an immediate opportunity to review their answers,
discuss the methods of solution, and analyze the implications behind their decisions. These are
all skills that go beyond the basic acquisition of information. In courses where active learning
has become a significant instructional method, collaborative examinations align the assessment
with the teaching, a goal that all teachers should have.
There is no sure-fire way of guaranteeing that all students in a large class will be equally
engaged and intellectually challenged in ways that will encourage critical thinking. Yet the
evidence from this large class in oceanography shows that active-learning strategies,
opportunities for problem solving with rapid feedback, and modifications of existing exam
formats, all contribute to improving students' capabilities in higher levels of reasoning.
Acknowledgments
The improvements in the teaching of oceanography have been supported by STEMTEC,
the Massachusetts Collaborative for Excellence in Teacher Preparation (NSF DUE9653966).
Additional research in 2001 was enabled by a grant from the Hewlett Foundation. Mary Deane
Sorcinelli and Martha Stassen were the Principal Investigators of this latter grant, and I thank
them for the support they provided. My colleagues Mark Leckie, Laurie Brown and Julie
Brigham-Grette helped implement many of the teaching and learning improvements in the
oceanography course.
Bio
Richard Yuretich is a professor in the Department of Geosciences at the University of
Massachusetts. His research specialties include geochemistry and sedimentology as well as
aspects of science education; in the latter capacity he is currently a Co-Principal Investigator of
STEMTEC. He has been involved in the teaching of the oceanography course in question for
nearly 20 years.
References
[1]
B. S. Bloom, Taxonomy of Educational Objectives: The Classification of Educational
Goals, by a Committee of College and University Examiners, Longmans & Green, New
York, 1964
[2]
R.F. Yuretich, S. A. Khan, R. M. Leckie, and J.J. Clement, “Active-learning methods to
improve student performance and scientific interest in a large introductory oceanography
course,” Journal of Geoscience Education, 49 (2001), 11-119.
[3]
G. E. Uno, Handbook on Teaching Undergraduate Science Courses, A Survival Training
Manual, Saunders, Fort Worth, 1999.
[4]
R.M. Leckie and R.F. Yuretich. Investigating the Ocean: An Interactive Guide to the
Science of Oceanography, McGraw-Hill, New York, 2000.
[5]
D. Cohen and J. Henle, “The pyramid exam,” UME Trends, July 1995, 2,15.
[6]
S. Khan and J. Clement, “Case study of innovations in an oceanography course,”
STEMTEC Annual Report (1998), E1-E14.