Assessment of Students’ Analytical Reasoning of an Unknown Lewis Structure Using
Multi-Layered Assessment IMMEX Technology
Abstract
Lewis structures are first introduced in general chemistry as a method of identifying how
the electrons are organized in a molecule or compound. In organic chemistry, Lewis structures
are often reviewed during the first lecture again as a way of describing electron location as well
as molecular structure of various functional groups. Because of the central role of Lewis
structures in both organic and general chemistry, an IMMEX (Interactive Multimedia Exercise)
problem was developed requiring students to interpret elemental analysis, mass spectrometry,
and physical data to identify an organic unknown Lewis structure. The goal of this paper is to
describe the problem features in detail the assessment layers of IMMEX, and present some data
for Lewis structure from general and organic chemistry students as a way of describing how
students have matured from one semester to the next and to identify whether differences exist in
student states (which describe their strategies) as a function of gender and academic status.
Introduction
The development of problem solving skills is a central goal in science classes, and the
purpose of traditional assessments such as labs, homework, and exams is for educators to
develop a better understanding of their students’ problem solving abilities. However, the use of
traditional assessments for such purposes can be quite time consuming and unambiguous results
are not always possible, even when students provide a detailed account of their logic. With
regard to assessment of problem solving, it is desirable to extrapolate the following information:
 What is the strategic sophistication of students at a particular point in time (a
performance measure)?
 How did students arrive at a particular strategy (a progress measure)?
 How will students likely progress with more practice/experience (a prediction measure)?
 How long will students maintain this level of strategic sophistication (a retention
measure)?
 What learning/instructional interventions will most effectively accelerate each student’s
learning?
 How effectively do skills transfer to other problem solving situations (a transfer
measure)?
Such information will provide a theoretical basis for development of teaching methods designed
to improve student understanding and problem solving abilities and improve overall student
retention.
An internet-based software package known as Interactive Multimedia Exercises,
IMMEX, was developed in response to the difficulty of thoroughly assessing the elements of
problem solving described above (www.immex.ucla.edu). This software package originated in
the UCLA medical school; however, because of the rich layers of assessment information
provided, it quickly expanded into K – 12 and college classrooms with science being well
represented. Thorough assessment of problem solving is achieved using an HTML tracking
feature in conjunction with artificial intelligence modeling software. The four fundamental
aspects of problem solving as described by Herron (1995) therefore are mapped using the
1
software: identification of the goals and objectives of the problem, problem representation,
strategy development, and verification.
IMMEX problems are case – based problems similar to what is commonly used in
medical (Wilkerson & Feletti, 1989) and business schools (Christensen & Hansen, 1987). Each
problem begins with a problem scenario (a prolog statement) which serves to identify the goals
and objectives of the problem. The following is an example of a prolog statement from
Lewis Structure:
Your team of forensic scientists has found an unlabeled vial at the scene of a crime.
It is your job to identify the substance by performing the appropriate tests and
observations of the compound. Once you think you have identified the compound
you should choose the Lewis structure that corresponds most closely with the data
that you have collected.
This problem scenario is job related, but other scenarios exist that are family related (True Roots
in which one must determine who are Leucine’s true parents), disaster related (Hazmat in which
an earthquake as lead to the spill of an unknown compound that may be hazardous), and lab
related (Separation in which students must develop a separation scheme to purify a particular
compound). These problems often present scenarios that students may encounter in real – life,
on the job, or in graduate school
After reading the prolog, students can then navigate throughout the problem space which
is defined as all of the items students can elect to view ranging from chemical to physical tests
for chemistry problems, object descriptors such as mass and velocity for physics problems, and
pedigrees to medical records for biology problems. In order to make the problems complete and
help discourage use of external sources, IMMEX problems have a library containing information
pertaining to all of the concepts presented in the problem. The problem space serves to provide
problem representation, and the term problem space in cognitive psychology describes the
information one associates with a particular problem (Sternberg, 1994). A sample problem space
is shown in Figure 1.
The HTML tracking ability is the unique feature that separates IMMEX problems from
other software packages because this feature allows for mapping student strategies and
subsequent modeling of strategies. This tracking features identifies the items viewed in the
problem space, the order in which they are viewed, whether they are viewed more than once, and
the amount of time they are viewed. In essence, each students’ cognitive “search tree” is mapped
(Anderson, 1980). This information is presented graphically to students and educators alike
using a search path map which provides a list of all of the problem space items but uses lines to
indicate strategy development. A sample search path map is provided in Figure 2.
IMMEX problems are equipped with dialog boxes asking students to verify their decision
to view problem space items; therefore, the verification aspect of problem solving can also be
observed by tracking whether students review items more than once. A point system is utilized,
and students either gain or lose points for every item they opt to view in the problem space and
for each incorrect response, thereby making students more aware of their actions (PalacioCayetano et. al, 1999).
Instant feedback is provided for students and educators alike. Upon submitting their
answers, students are informed whether their answers are correct or incorrect. If students are
2
incorrect, most problems allow them to review the problem space and submit another answer.
On the solve page a complete list of items that were viewed is provided which promotes
metacognitive activities in which students reflect back on their methods and work toward
improving them (Rickey & Stacy, 2000). Immediate feedback also serves to allow students to be
aware of their own thinking and acquired strategies which are the two key components of
metacognition (Gredler, 2001). A sample feedback screen is provided in Figure 3. The purpose
of such immediate feedback is to provide students with either the assurance they may need or
perhaps motivate students to review certain concepts or even seek help from either peers or the
instructor. For the instructor, this feedback will identify students who are having difficulty to
gauge whether to adjust the pace of the course if possible or implement various intervention
activities. It should be stressed that using immediate feedback protocols has been found to foster
development of student expertise (Lajoie, 2003).
Prolog
Statement
Library
and
Solubility
Information
Solve
Menu
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Combustion
and Mass
Spectrometry
Data
Lewis
Structures
Physical Properties
Figure 1: The problem space for Lewis Structure consists of various items including analytical
data such as combustion data, the physical properties (such as melting point, boiling point, and
solubility data) of the unknown, and the actual Lewis structures for the possible unknown
compounds. The different colors represent different types of problem space items.
3
Basic
Info
Strategy
Depiction
Relative amount
of time spent on
each type of item.
Figure 2: A representative search path map for Lewis Structure. The lines indicate movements
in the problem space. Such transitions are represented by lines from the left hand side of a
problem space item to the center of another problem space item.
4
Identifies
whether the
solution is
correct or
incorrect
A complete list
of the items
viewed.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Figure 3: The immediate feedback screen for the Lewis structure problem. Students are not
only informed whether their solution was correct or incorrect, but a complete list of items viewed
and their solution is provided. This will give students an opportunity to reflect back on their
methods before attempting the problem again—most problems allow for two submissions.
Artificial Intelligence Models
In order to provide rapid feedback concerning problem solving various models have been
incorporated for modeling the results of the HTML tracking. These models will serve to answer
the questions described in the introduction.
5
Artificial Neural Networks (ANNs)
Artificial neural networks (ANNs) are used for pattern recognition, and for IMMEX, this
software will group strategies according to their similarities. Students’ selection of problem
space items are collectively combined and analyzed for similarities to provide a set number of
output nodes describing problem solving strategies (Stevens & Najafi, 1993, Stevens et. al.,
1996). The number of nodes is an arbitrary value that can be set to any number, but we have
found that 36 is most adequate. The nodes are plots of probability vs. problem space item;
therefore identifying the probability that a given problem space item will be selected for each
strategy, is modeled by a particular node. ANN data can be used to provide a snapshot of how a
student is solving a problem at a particular point in time. We can compare the node information
with the success rate, and we can compare the strategy types with how students are performing
on classical assessments such as tests or quizzes. Figure 4 provides a description of the ANN
output.
While ANN data can be very useful for identifying strategy types at a given time, the
method is limited. This data provides information concerning only one performance – it does not
describe the progression of student strategies. Furthermore, with IMMEX problems, we have
found that students undergo various transitions when solving problems. When students first
encounter IMMEX problems, their strategies are mostly prolific described by the use of most
problem space items because students are exploring the available information to ascertain what
information is relevant and what is irrelevant. However, after solving one problem, most
students’ strategies will change because they will evaluate and reform their strategies according
to their prior performance. The transition phase, in which students are continually re-evaluating
their strategies, continues until students have worked four or five problem cases and at that time
students’ strategies have been observed to stabilize (Stevens et. al., 2004). Therefore, with
regard to ANN data we must consider whether students’ strategies have stabilized because
otherwise the information may not be relevant. However, this can be a useful tool to look for
patterns in problem solving after the students have indeed stabilized.
6
Node
1–6
7 - 12
13 - 18
19 - 24
25 - 30
31 - 36
A
B
Figure 4: Sample Neural Network Nodal Analysis. A. This analysis plots the selection
frequency of each item for the performances at a particular node (here, node 15). General
categories of these tests are identified by the associated labels. This representation is useful for
determining the characteristics of the performances at a particular node, and the relation of these
performances to those of neighboring neurons. B. This figure shows the item selection
frequencies for all 36 nodes following training with 5284 student performances. These plots
describe the relative probability of a given set of problem space being selected (Stevens, 2004).
Considering each IMMEX problem set has multiple cases (or clones) it is feasible that
students can work several problems with at least five cases and as many 60 cases for select
problem sets to ensure stabilization (Underdahl, 2002). The IMMEX problem space is nonprescriptive indicating that every case cannot be solved using a single strategy – therefore,
students must have a deep rooted understanding of the underlying concepts to successfully solve
the problems because problem solving by analogy or algorithms will not suffice (Stevens, 2003).
Hidden Markov Models
As an extension of ANNs, Hidden Markov Models (HMMs), provide information
concerning student progression. A simplistic description of HMMs is that they provide a sub
grouping of ANN data to yield a more manageable description of strategies. Like nodes, the
number of HMM states are determined by the programmer, and for our studies we have decided
to use five states. HMM data is defined in terms of probabilities which describe how students’
strategies change with time – how they progress. These models are used to describe stochastic
processes such as problem solving (Soller & Lesgold, 2003).
Prior probabilities describe where students end up after their first performance. So there
will be five probabilities for five states. The transition probabilities represent describe the
likelihood of moving from one state to another (i.e. from state 1 to state 3, state 2 to state 5, etc.).
Such movements are often observed during the framing and transitioning process as illustrated
by Figure 5. Based upon this data, we can determine whether a state transition is stable from a
7
high probability that the student will remain at the state or in transit from a low probability that a
student will remain at the state. This information is very important because it will enable us to
identify students who are stabilizing at ineffective state particularly those which have stable
transition probabilities. Such information can be very important for deciding when to develop an
intervention method such as group work (Case, 2004), a metacognitive activity requiring
students to think about their strategies (Rickey & Stacy, 2000), or even a one-on-one instructorstudent intervention. The latter will not be feasible at many college institutions, but group work
or other types of activities are feasible.
State 3
State 4
State 5
State 2
State 1
Figure 5: This describes the framing, transitioning, and stabilization aspects of problem solving.
Emission probabilities are used to describe the probability that a given node will be associated
with a particular state. Therefore, this probability is used to effectively correlate states and
nodes. Figure 6 provides a graphical illustration of the transition and emission probabilities.
From Figure 6, we can determine that states 1, 4, and 5 are relatively stable indicating students
who transition to these states will likely remain at these states while there is a high probability
that students will move away from state 2 and a higher probability for state 3. It should be
emphasized that for each state, we can determine the type of strategy (i.e. prolific, efficient, or
limited) that is most indicative of the state, as well as, the overall probability of a correct
performance. For the in transit states, we can also determine the probability that students will
move to a more effective or even a less effective state or perhaps move to another in transit state.
8
Figure 6: This represents the emission probabilities which correlates that nodes with the states
and the transition probabilities which describes how students progress when working problems.
Overall, the HMM data can effectively describe how students progress with regard to
solving a particular type of problem, and these can be used to predict future performances with
90% accuracy or higher (Stevens et. al, 2004). This data provides much more detailed
information concerning problem solving than any written assignment because we can
immediately identify students who will continue to use ineffective strategies. A traditional
assessment can be used as a measurement of student understanding, but these assignments say
little about how students would approach the problem in subsequent attempts.
Item Response Theory
IMMEX software uses item response theory (IRT) to determine student ability and item
difficulty which are treated as a consequence of one another with CTT and are not considered
individually (Hambleton et. al, 1991). The use of IRT enables testing on multiple groups (Kim
et. al., 1995) and parallel testing (Hambleton et. al., 1991). With regard to IMMEX, this ensures
that we can reliably compare student performance. Item difficulty and student abilities are
provided with the ANN and HMM data for each IMMEX output. Therefore, it is possible to
correlate ability (the greater the ability, the greater the probability of a correct performance) with
state and node data. This is one method to further identify which states or nodes are more
effective. IRT item difficulty values can be used by professors as a means of crafting
assignments with the desired level of difficulty – as is possible with traditional assignments.
9
Results and Discussion
The strategies were modeled using ANNs and HMMs, and the ANN diagram for general
and organic chemistry is shown in Figure 7.
1
2
3
4
5
53
6
53
7
8
9
10
11
12
13
14
15
16
19
25
31
20
26
32
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
21
27
33
22
28
34
17
18
23
24
29
30
35
36
Figure 7: Artificial Neural Network Output for Lewis Structure. As expected, the topology
indicated an array of strategies ranging from extensive testing (prolific) as shaded in red to very
limited testing as shaded in blue.
The nodes are completely summarized in table 1 including the solved rate for organic and
both organic and general chemistry, as well as, a description of the strategy depicted by the node.
10
Table 1: Complete Node Description for Lewis Structure.
Node
1
2
3
Number of
Overall
O.C.
O.C.
Number of
Percent
Observations Observations Correct
144
253
47
4
5
25
178
320
51
Overall
Percent
Correct
39
20
39
4
29
55
38
35
5
47
77
43
31
6
83
119
32
32
7
26
45
35
31
8
3
8
67
38
9
34
77
30
22
10
67
106
42
32
11
31
46
49
43
12
68
113
34
28
13
44
58
55
47
14
15
25
40
48
15
26
47
35
30
Strategy Description
Use of physical tests & elemental analysis
Use of physical tests & mass spectrometry
Use of physical tests, elemental analysis,
and mass spectrometry.
Same as node 3, but significant noise was
observed for the Lewis structures and
solubility information in the library.
Same as node 3, but increased use of
Lewis structures and library (melting &
boiling point) and solubility.
Significant use of all problem space
materials but the Lewis structures.
Use of elemental analysis, state, melting
and boiling point, and solubility tests.
High probability for all problem space
items except the Lewis structures, melting
point, and library materials.
High probability for all tests with
significant noise in the Lewis structure
region and limited use of the library.
Same as node 9 with different
probabilities.
High probability for all tests with a 70%
probability for the melting point,
solubility, and elemental analysis library
data.
Same as node 12, but a probability of 80%
or higher was observed for all library
materials.
High probability for everything but color,
state, the library materials, and the Lewis
structures.
Same as node 13 but there was some noise
in the Lewis structure region.
High probability for all tests except color
(p < 0.3) and melting point (p < 0.6) and
little or no use of the library materials or
Lewis structures.
11
16
11
21
82
53
High probability for all tests except: state
(p ~ 0.75), solubility in water (p~0.75),
solubility in hexane (p~0.5), and solubility
in HCl (p~0.5). The same pattern was
observed for the Lewis structures and the
library as with node 15.
17
5
6
40
34
18
17
22
47
41
19
17
43
71
47
20
20
46
35
26
21
13
17
69
59
22
50
80
54
48
23
45
53
38
38
24
37
62
52
48
25
33
41
58
59
26
3
7
67
57
High probability for everything with
increased noise in the library region.
A probability of 100% for elemental
analysis, state, color, melting and boiling
points. A probability of 80% for
solubility in water and mass spectrometry.
A probability of 50% or greater was
observed for most library items.
High probability for elemental analysis
and solubility (except in HCl).
Extensive use of mass spectrometry and
solubility data with some use of the
elemental analysis data (p~0.6).
A probability of 100% for mass
spectrometry, elemental analysis, and
solubility in hexane. High probabilities
(less than 100%) were observed for
solubility in water and sodium hydroxide.
Significant probabilities (greater than
50%) was observed for state and melting
point.
A probability of 100% for every test but
solubility tests. The solubility in water
was 80% and the other solubility tests
were not used significantly.
A probability of greater than 90% was
observed was observed for elemental
analysis, state, color, melting and boiling
points, and solubility in water.
A probability of 100% was observed for
state, color, and melting and boiling
points.
A probability of 100% was observed for
elemental analysis and solubility in water.
A probability of 60% for solubility in
hexane.
A probability of 100% for elemental
analysis and solubility in sodium
hydroxide.
12
27
14
29
65
52
28
26
43
50
40
29
12
20
59
65
30
38
41
50
47
31
117
259
58
48
32
28
76
15
11
33
65
124
62
53
34
9
10
89
80
35
12
32
75
63
36
89
116
64
62
A probability of 100% for elemental
analysis, mass spectrometry, and
solubility in water.
A probability of 100% for elemental
analysis and mass spectrometry with
probabilities greater than 50% for melting
and boiling point data.
A probability of 100% for elemental
analysis, mass spectrometry, and melting
and boiling point data. A probability of
approximately 50% was observed for state
and solubility in water.
A probability of 100% for melting and
boiling point data, 80% for solubility in
water, and 50% for elemental analysis
data.
A probability of 100% for elemental
analysis and essentially 0% for other tests.
Sporadic random testing in which no test
item was used with significant probability.
A probability of 100% for mass
spectrometry and 80% for elemental
analysis.
A probability of 100% for mass
spectrometry, elemental analysis, and
boiling point.
A probability of 100% for boiling point
and 60% for solubility in water.
A probability
Hidden Markov Models
The observation or emission probabilities are described in Figure 8. The state
descriptions are provided in table 2 with the percent correct for general, organic, and both
groups.
13
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Figure 8: Emission probabilities for Lewis Structure. The emission probabilities describe the
types of strategies represented by each state.
Table 2: The state description.
State Number of
Total
Observations Number of
for Organic
Observations
1
271
460
Percent
Correct
for
Organic
48
Percent
Correct
for
General
28
Percent Description
Correct
Overall
2
629
1045
48
30
41
3
47
88
30
24
28
4
258
551
47
28
37
5
255
338
56
41
52
40
Use of mainly elemental
analysis, mass spectrometry,
with use of solubility data.
Use of most items except the
library and Lewis structures.
Use of all items including the
library and Lewis structures.
Gaming strategy with use of
only one or at most two items
in the problem space.
Use of greater analytical
reasoning with greater
selectivity of the tests/
14
The states and problem cases were compared using a crosstabulation procedure. The
results indicated that there was not a statistical difference in the state as a function of the problem
case (p = 0.62). Therefore, we can assume that the strategies are not affected by the nature of
problem case, thus, students will continue to use comparable strategies regardless of the nature of
the case. However, there was a statistical difference in performance and case identity (Pearson
Chi-Square = 162.4, df = 9, p-value < 0.000). The problem was designed such that the cases or
clones are structural isomers, therefore, the elemental analysis and mass spectrometry data
cannot always specifically identify the unknown. Furthermore, unlike most IMMEX problems,
Lewis Structure allows only one attempt in order to prevent students from randomly guessing.
Ethyl acetate, dioxane, and butanoic acid are examples of the isomers used for the problem, and
consequently these have the lowest solved rates. Students stabilizing at state 4 are more likely to
be less successful for the constitutional isomers; however, it is reasonable to assert that students
who have a thorough understanding of the underlying concepts can make reasonable conclusions
for select cases with little information. Students using state 5 were using more logical strategies
because they were tailoring their strategies to the nature of the problem case; therefore, these
students have an adequate understanding of the underlying concepts and the tests which provide
the most pertinent information.
The melting and boiling point information can encourage students to game or guess at the
correct structure. However, most students who used strategies relying heavily upon melting and
boiling point data (examples nodes 5-8, 13-16, 23,-24, 29-30, 34 – 36) were not very successful
and the number of students using these strategies was not very significant. Furthermore, students
were not likely just “gaming” by using literature melting or boiling point data because additional
tests were viewed such as solubility, mass spectrometry, and elemental analysis data. Some of
the physical information may not always be very useful but differentiating between the isomers
can be readily achieved using solubility information. States 1 and 5 support this hypothesis
because they had higher solved rates in comparison to states 3 and 4. Furthermore, table 1
supports this claim as well.
From the data, we can first conclude that students either fail to realize the differences in
the physical properties of the Lewis structure isomers particularly with regard to solubility.
Conversely, students may being trying to “game” the problem and guess the correct solution.
With regard to this problem, prior emphasis on physical properties should be emphasized or
intervention activities stressing these properties should be implemented. An example of an
intervention activity would require students to compare the physical properties of various
functional groups such as acids, ethers, esters, ketones and amines. Another possibility is that
students are not using the Lewis structures and may have incorrect depictions for the unknowns.
For example, students may not realize that dioxane is an ether or even has a cyclic structure.
This could particularly be the case for general chemistry students; however, organic students
should have a greater understanding of these structures, but this problem was assigned during the
first week of organic chemistry. Thus, another intervention method should emphasize using
Lewis structures. Perhaps, a metacognitive activity in which students reflect upon their methods
and why they were unsuccessful may be useful in improving student performance. Another
associated activity would require students to draw out several sets of Lewis structures for various
functional groups.
A crosstabulation procedure was completed to compare the strategies of students enrolled
in general and organic chemistry. This procedure revealed with 99% confidence (Pearson ChiSquare = 48.4, df = 4, p < 0.000) that general and organic students use different strategies.
15
Likewise, we compared performance and academic status, and as expected, we found that the
performance was statistically different based upon academic status (Pearson Chi-Square = 79.4,
df = 1, p < 0.000). Figure 9 illustrates students’ framing, transitioning, and stabilization patterns.
The prolific strategies (state 3 and state 2) were more profound for the organic chemistry
students during the framing stage; however, these were less persistent with organic than general
chemistry students. Furthermore, the “gaming” strategies associated with state 4 were present in
greater proportions for general than organic chemistry students. The more analytical logical
strategies (state 5) became more common with both students, but these were present in greater
proportion for organic students. These latter observations indicate that organic chemistry
students have conceptually matured as we would expect. The organic students more readily
identify that they cannot merely differentiate among the constitutional isomers using a single
test. Overall, there is a clear difference in the strategic trajectories for general and organic
students—the organic students demonstrate a greater maturity with regard to their strategies and
logic.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Figure 9: The stabilization behavior for the Lewis structure problem.
16
Some possible explanations for the differences in strategies include:




Students enrolled in organic chemistry are enrolled in curricula involving greater science
rigor, therefore, they have a greater inkling for science material.
The introductory lecture on functional groups and Lewis structure during the first week of
organic chemistry does provide these students with an advantage.
Students enrolled in organic are more academically mature and are willing to put forth a
greater effort.
The problems were assigned in different environments: For organic students the problems
were assigned in a traditional lecture environment, and for general chemistry students the
problems were assigned in a laboratory environment. Students typically do not view lecture
and laboratory similarly, therefore, general chemistry students are less likely to have taken
the problem as seriously.
It is not feasible to expect general and organic students to perform similarly, however,
both groups of students can benefit from interventions stressing both Lewis structures and
physical properties.
Gender, Strategy, and Performance
The initial study indicated that males and females did not perform statistically different
nor did they different strategies. As an extension of this study we also incorporated lecture
professor and teaching style as a function of gender and strategy. Figure 10 illustrates the types
of strategies and methods employed by the two organic professors who used the problems in the
study.
Professor 2
Professor 1
In – Class
Group Work
Lecture
ConcepTests
Occasional
Questions
Lecture
Cooperative
Learning
Figure 10: The types of strategies used by the two professors in the study.
The significance of this study is to determine the effects of the relative teaching style on
the types of strategies students develop. The first professor in this study used primarily active
17
learning teaching methods such as ConcepTests and cooperative learning while the second
professor used primarily passive learning methods. The question arises as to whether this affects
the development of strategies for students. With an active learning environment, students are
more responsible for their learning—they must do more than simply take notes and listen—they
must get involved in the class!
A crosstabulation procedure was employed to determine the relative distribution of state
as a function of gender and professor. The results indicated a statistical difference in the strategy
for females as a function of professor (Pearson Chi-Square = 10.9, df = 4, p – value = 0.02), but
there was not a statistical difference for males (Pearson Chi-Square = 3.7, df = 4, p-value =
0.498). This correlates with observed results from the literature which support that females have
a greater retention and generally perform better in an active learning environment (cite
references here). The females in the active environment used states 3, 4, and 5 more
significantly than females in the passive learning environment. The fact that females used states
4 and 5 more significantly indicates that the active learning has positively influenced their
strategy. These were the most successful states. State 3 was also used more significantly;
however, this could be a characteristics of students’ learning style in which they are developing
and re-evaluating their strategies. State 2 was most commonly used in high rates for females in
the passive learning environment which indicates that students viewed most of the items in the
problem space except the Lewis structures and the library materials. This is a prolific type
strategy – not as significantly as state 3 – but still more test items are viewed than is absolutely
necessary.
Conclusion
IMMEX offers a multi-layer assessment and will readily identify how students are performing at
a given point in time, how they are progressing over an interval of time, and their overall student
ability. This information goes beyond traditional assessments that just provide information on
how students have performed on a given test – which may or may not be indicative of how they
would perform on a second test. With IMMEX, the internet – based problems are designed to fit
nicely into the curriculum, and adjustments in syllabi should be minimal if any. Educators can
truly obtain multiple pieces of assessment information in only a fraction of the time needed to
craft and grade traditional assessments. A summary of the layers of assessment is provided in
Table 3.
Table 3: A summary of the layers of assessment.
Trait
Approach
Performance
Model sequences of actions
Progress
Prediction
Intervention
Transfer
Tools
Artificial Neural Network
Clustering
Cumulative response modeling of student Item difficulty modeling using
performance
IRT
Prediction from existing HMM models
Hidden Markov Modeling
Experience-based matching of learning Perturbation of predicted
trajectory and interventions
performance models
Mixed Methods
Multidimensional
scaling
using IRT, ANN, and HMM
18
The above measures relating to this assessment model can be reported in real time and linked to
other student achievement measures.
Our findings with the Lewis structure problem indicate that students’ strategies are
dependent upon their academic level, as well as, their lecture professor’s teaching style. General
chemistry students were found to use less efficient strategies with a lower overall performance,
and the strategy trajectories for general and organic students differed. Organic students moved
toward more efficient states while prolific states remained more prominent with general
chemistry students.
The gender finding is a interesting result that complements literature stating that females
perform better in an active learning environment. There will be an extension of this study to
determine which teaching styles are unique for both lecture professors as a means of identifying
the styles most likely responsible for the differences in performance for females.
One final extension of this research will involve the use of metacognitive type activities
or instructor-developed activities as a means of stressing the importance of using the Lewis
structure menus in the problem space, as well as, the physical properties of the unknown
compounds. These interventions are aimed at improving the student problem solving behavior.
19
Bibliography:
Anderson, J.D., (1980). Cognitive Psychology and Its Implications. San Francisco: W.H.
Freeman.
Gredler, M. E. (2001). Learning and instruction: Theory into practice. Upper Saddle River:
NJ: Prentice Hall.
Halpern, D.F. (1997) Sex Differences in Intelligence, American Psychologist, 52(10), 1091 –
1192.
Hambleton, R.K., Swaminathan, H., and Rogers, H.J. (Author) (1991), Fundamentals of Item
Response Theory (Measurement Methods for the Social Science), SAGE Publications
Herron, D. and Eubanks, I.D. (1995). The Chemistry Classroom: Formulas for Successful
Teaching, American Chemical Society, Washington, D.C.
Kim, S.H., Cohen, A.S., & Park, T.H. (1995). Detection of differential item functioning in
multiple groups. Journal of Educational Measurement, 32, 261 - 276.
Lajoie, S.P. (2003). Transitions and Trajectories for Studies of Expertise. Educational
Researcher, 32: 21 – 25.
Rickey, D., Stacy, A.M. The Role of Metacognition in Learning Chemistry, Journal of Chemical
Education, 77, 915 – 920.
Soller, A., and Lesgold, A. (2003). A Computational Approach to Analyzing Online Knowledge
Sharing Interaction. Proceedings of Artificial Intelligence in Education, 2003. Australia,
253 – 260.
Stevens, R.H., Ikeda, J., Casillas, A., Palacio-Cayetano, J., and Clyman, S., (1999) Artificial
Neural Network-Based Performance Assessments, Computers in Human Behavior, 15:
295 – 314.
Stevens, R.H., and Najafi K. (1993). Artificial Neural Networks as Adjuncts for Assessing
Medical Students' Problem-Solving Performances on Computer-Based Simulations.
Computers and Biomedical Research 26(2), 172-187.
Stevens, R. and Palacio-Cayetano, J., (2003) Design and Performance Frameworks for
Constructing Problem-Solving Simulations, Cell Biology Education, 2: 162 – 179.
Stevens, R., Soller, A., Cooper, M., and Sprang, M., (2004) Modeling the Development of
Problem-Solving Skills in Chemistry with a Web-Based Tutor, Computer Science
Editorial III, Springer-Verlag, Heidelberg Germany (in press).
Stevens, R., Wang, P., Lopo, A. (1996). Artificial Neural Networks Can Distinguish Novice and
Expert Strategies During Complex Problem-Solving. JAMIA vol. 3 Number 2 p 131-138.
Underdahl, J., Palacio-Cayetano, J., and Stevens, R. (2002). Practice Makes Perfect: Assessing
Student Knowledge and Problem-Solving Skills with IMMEX Software, Learning &
Leading with Technology, 28 (7): 26 – 31.
20