Assessing Student Understanding with Technology: The Use of IMMEX Problems
in the Science Classroom
Charles T. Cox Jr.1, Joni Jordan2, Melanie M. Cooper1, Ron Stevens3
1Clemson
2SC
University, Department of Chemistry, Clemson, SC 29634
Governor’s School for the Arts and Humanities, Greenvillle, SC 29601
3The
IMMEX Project, 5601 W. Slauson Ave., Suite 255, Culver City, CA 90230
Abstract
The development of problem solving skills is a central theme in science
classrooms, particularly with regard to national standards. Students’ problem solving
strategies often cannot be studied easily, but with a relatively new software package
known as IMMEX, this has become a feasible task. IMMEX problems focus on
important concepts across the science curriculum and can be easily incorporated as an
assessment tool in preparation for state or school mandated exams.
Introduction
Without a doubt, most science teachers have been amazed when grading tests
and quizzes, often wondering how and why students have reached a particular
conclusion, particularly when students fail to provide a detailed account of their logic.
Attempting to understand students’ thought processes can be time-consuming and
challenging, even when a written explanation is provided. Ideally the variety of
assessments should be used to identify alternate conceptions or gaps in understanding,
particularly when preparing students for state mandated exams. With this in mind, a
relatively new internet-based software package known as IMMEX (for interactive
multimedia exercises) was developed at UCLA. As a result of the success at improving
1
problem solving skills with medical school students, IMMEX rapidly and easily expanded
to K – 12 classrooms with high school science being well represented.
IMMEX problems are case – based problems, designed to provide students with
real – life scenarios to which they solve using science content learned in class. The
problems begin with a prolog statement designed to grab the students’ attention. Table
1 provides examples of problem scenarios for high school science problems.
Table 1: Problem Scenarios for some common high school science IMMEX problems.
Problem
Name
Hazmat
Gas Law
Scenario
An earthquake just hit your
school. An unmarked container
was damaged and the contents
are spilling out. Can you
identify the chemical that has
spilled so that you can help
dispose of it properly before it
becomes a hazard to your
school.
Help secret agents deliver a
balloon holding top secret
material to an in-flight plane by
determining the mass it can lift
if filled with different gasses
under various environmental
conditions
True
Roots
Help Leucine determine who
her true biological parents are.
Blast to
the Past
Can you choose the best
vehicle to transform into a time
machine?
Concept/Topic
Subject
Qualitative Analysis
Litmus Paper Testing
Chemical Reactions
Flame Tests, Solubility
Tests
Conductivity Tests
Chemistry
Combined Gas Law
Charles’ Law, Boyle’s
Law
Density/Buoyancy
Percent Composition
Chemistry,
Introduction
to Physical
Science
Mendelian Genetics
Punnett Squares
RFLP Analysis, DNA
Analysis
Fingerprint Analysis
Blood Typing, Pedigree
Analysis
Blood Type Inheritance
Kinematics, Power
Newton’s Laws,
Weight
Kinetic Energy
English:Metric
Conversions
Biology
Physics
2
Virus, Bacteria, Protozoa
Can you determine the cause,
Fungus, Cultures, Smears
mode of transmission, and
Latex Agglutination
Creeping
place of origin of the illness that Ecology, Parasitism
Biology
Crud
is afflicting a group of
Commensalism
travelers?
Anthropod Vectors
Agar, Antibodies
Figures 1 – 3 provide screenshots of the student view of the prolog statements.
The prolog statements are often very colorful and students often comment on the way
they focus attention on the problem.
Figure 1: Creeping Crud Prolog
3
Figure 3: Gas Law Prolog
Figure 4: Blast from the Past Prolog
4
Use for the Curriculum
Table 1 is a brief summary of IMMEX problems that are available (free of
charge). A more extensive list can be found on the IMMEX website
(www.immex.ucla.edu). The wide variety of science subjects available with IMMEX
problems ensure a good fit into the curriculum. For instance, the True Roots problem
can be implemented following the unit on genetics (Palacio-Cayetano 1999) and the
Creeping Crud problem can be implemented following a unit on bacteria, viruses, and
cell cultures.
Multiple cases (or clones) exist for each problem, ranging between 5 and 60
cases per problem (Underdahl 2001). Students can work each problem multiple times
to test their conceptual understanding of a given topic. Unlike traditional assessments,
the number and variety of IMMEX clones per problem prevents students from
memorizing content subject matter and gives teachers an opportunity to truly assess
student understanding. Furthermore, students and educators alike are given instant
feedback; students are immediately informed whether their answers are correct, and
they are allowed to rethink the problem and re-approach it a different way. Because of
the immediate feedback given during the problem solving process, the problems are
excellent study tools in preparation for end of course examinations mandated by the No
Child Left Behind Act. These examinations are often designed to test students’
synthesis and application skills to situations they have not encountered previously.
IMMEX problems allow students the opportunity to practice and review before tests.
The feedback provides teachers with information to make necessary instructional
changes before students take tests.
5
Another unique aspect of the problems is that each clone has an associated
difficulty. This information is readily available for each problem and educators can
choose the level of difficulty appropriate for each student. It is possible to use easier
clones for the first attempt and more difficult clones for subsequent attempts, so that
educators can tailor the level of difficulty appropriately based on individual student
needs.
Problem Design
IMMEX problems always begin with a prolog statement such as those described
in Table 1. After reading the prolog, students are given freedom to navigate throughout
the problem space which is defined as all of the available information for use in solving
the problem including all tests, the prolog, and the library. For example, the problem
space includes chemical and physical tests with Hazmat, pedigrees and blood types
with True Roots, or environmental factors with Gas Law. Most IMMEX problems are
equipped with a library of information that will aid students as needed. An example of a
problem space for True Roots is shown in Figure 4. The student must ascertain what is
relevant for solving the problem. Although they may choose any menu item, there are
costs associated with each choice (usually a loss of points or time) to discourage
students from choosing every item. IMMEX software is equipped with a tracking device
that will identify the items students’ view, the order in which they are viewed, the amount
of time they are viewed, and whether they are viewed more than once. This information
is presented graphically using “search path maps” that are readily available to students
and teachers after the answer is submitted. Figure 5 provides an illustration of a search
path map.
6
Figure 4: Problem space for True Roots. Different types of information are segregated
using different color schemes (the blue items are karyotypes, the yellow items are birth
certificates, the orange items are pedigrees, the red are blood types, etc.).
The search path maps can be used specifically to identify how students solved
the problems. Unlike written assessments, the search path maps can be very
descriptive and informative since students cannot leave out details that are necessary in
identifying problems with understanding. Search path maps trace and identify every
decision a students makes during the problem solving process.
7
Figure 5: Search Path Map (Hazmat). The search path map provides a “map” of a
students’ strategy using transition lines which move from the left hand corner of an item
box to the center of a second item. The search path map also provides a toolbar at the
top and bottom which provides useful information. The top toolbar provides basic
information such as student ID, whether the problem was solved or completed, the
number of attempt, and the total amount of time spent on the problem. The bottom
toolbar represents the relative amount of time spent on each type of item.
The following activities may be used to directly involve students in understanding
the process of problem solving.

Metacognitive activities in which students describe why they chose a
particular path and explain how they could improve their strategy in the future.
Students can answer questions concerning why a particular strategy was
either successful or unsuccessful, and can work on developing a better
strategy for future problems. From this activity students could be required to
develop a strategy and implement it for a new case.

Group assignments in which successful and unsuccessful students are paired
to discuss how their strategies differ.
8

Classroom discussions in which students collectively describe their reasoning
when solving a problem. The discussion can include problem space items
that students felt were relevant or irrelevant, the logical progression of how to
move throughout the problem space, and how strategies would differ for
different cases.
The activities described above could serve as an excellent review for the state
mandated exams connected with the No Child Left Behind Act. The nature of these
problems allows them to be easily integrated in the existing curriculum.
Discussion
Research has shown when students are first presented with an IMMEX problem,
they often view most or all of the items in the problem space. They are essentially
“framing” the problem to determine what information is relevant and irrelevant. After
students work one case, their strategies often change – in IMMEX terminology, they
undergo a “transition.” Such transitions are often observed when students work the
second or third case of a problem. However, usually after the third or fourth case,
students will reach a point of stabilization. This point of stabilization can be of particular
interest to educators because we have found that 80% of students continue using such
strategies, whether they are successful or unsuccessful. Since we can predict how a
student will perform on subsequent cases of a problem we can tailor interventions to the
student’s particular problem solving strategy. When appropriate, this information can
also be used to prompt educators to review particular instructional strategies or request
a student-teacher conference for students who are falling behind. Such interventions
can also improve student performance on state – mandated exams.
9
The IMMEX staff provides a free analysis of search path maps to make
interventions easier. A scaled score and detailed summary explaining how to interpret
this score is provided. The time to analyze the results and pinpoint problems will likely
take a fraction of the time needed for a traditional assessment. Table 2 provides an
illustration of these student ratings based upon the analysis of the search path map.
Table 2: State descriptions for the Hazmat problem. The state is described by three
factors: solve rate, transitions, and items represented. The solve rate identifies the
probability that students in a particular state will solve the problem correctly. The
transitions describe how students will behave at a particular state (i.e. will they move to
another state). For instance, states 1, 4, and 5 are stable indicating that once a student
arrives at these states they have a high probability (greater than 80%) of staying at
these states while states 2 and 3 are in transit indicating that students at these states
have a high probability of moving to another state.
State
Solve Rate
Transition
State Description
1
55%
Stable
2
3
60%
45%
In Transit
In Transit
4
54%
Stable
5
70%
Stable
Mostly use of test items with little use of the
library or other available information
Equal use of tests and library
Extensive testing with most test items viewed
Mostly test items with little use of the library or
other available information
Effective strategies
In addition to providing a description for each student, the IMMEX laboratories will
also provide an analysis for the class as a whole. Figure 6 provides a graphical
representation for the state description for the class. This representation will allow
teachers to identify the proportion of their class performing at each state after given a
number of performances and allows them to readily identify whether students are
improving after practice or even interventions. Overall, the state descriptions provide an
account of students’ progression and how their strategies have collectively changed.
10
Figure 6: The state descriptions for the class.
Currently, the state descriptions for students and the class are automated for the
Hazmat problem only; however, this information can be requested from the IMMEX
staff. Other problems will be automated in the near future.
Conclusion
IMMEX is a useful tool in assessing problem solving skills and can provide
pertinent answers to questions concerning why students arrive at a particular result.
Science teachers find themselves frequently asking how students arrive at a particular
understanding of subject matter. IMMEX allows teachers to determine how students
solve problems, and even pinpoint where students are developing misconceptions.
11
Knowing this information and implementing appropriate intervention, allows for more
effective teaching and greater student achievement. The No Child Left Behind Act has
led to state mandated testing and IMMEX problems give students practice with
conceptual understanding, unlike traditional assessments in which students often simply
memorize steps for solving a particular problem. With IMMEX students can work up to
60 “new” problems so memorization is not an option. Also IMMEX problems can be
implemented within the existing curriculum. Educators can use IMMEX problems
instead of traditional homework or other assessment methods to assess understanding
of concepts such as qualitative inorganic analysis, genetics, bacteria, kinematics,
viruses, gas laws, Lewis structures, stoichiometry, and many other topics related to
chemistry, biology, earth science, and physics. Overall, these problems provide
students with practice that can be pivotal in yielding a successful performance on state
mandated exams.
References
IMMEX Website. Available at: www.immex.ucla.edu
Palacio-Cayetano, J., Allen, R.D., Stevens, R.H. 1999. Computer – Assisted Evaluation
– The Next Generation, The American Biology Teacher, 61 (7), 514.
Underdahl, J., Palacio-Cayetano, J., Stevens, R. 2001. Practice Makes Perfect:
Assessing and Enhancing Knowledge and Problem-Solving Skills with IMMEX
Software, Learning and Leading with Technology, 28 (7), 26.
12