ChemPrep : Self-Paced OWL Preparation for First-Semester General and Organic
Chemistry
Beatrice Botch, Roberta Day, William Vining, Stephen Hixson, Peter Samal, Barbara Stewart
Department of Chemistry, University of Massachusetts Amherst, Amherst, MA 01003
Kenneth Rath, Alan Peterfreund
Peterfreund Associates, Amherst, MA 01003
David Hart
Department of Computer Science, University of Massachusetts Amherst, Amherst, MA 01003
Abstract
Two self-paced chemistry courses were written to help improve student preparedness for first
semester General and Organic Chemistry. These courses are delivered over the web using the
OWL (Online Web-based Learning) system developed at the University of Massachusetts. The
courses are designed to take ten to twenty hours to complete and are offered prior to the start of
the semester. ChemPrep/General Chemistry covers topics such as matter, algebra, significant
figures, nomenclature and dimensional analysis. ChemPrep/Organic Chemistry covers concepts
such as Lewis structures, formal charge, geometry, hybridization and acid/base theory.
Evaluation studies of the Spring 04 and Fall 04 semesters show that for those students who
completed more than half of the ChemPrep units, course grades and retention rates in the
subsequent chemistry courses were higher. The content, implementation, and evaluation of
the ChemPrep courses is discussed.
ChemPrep Demo Site
Introduction
General chemistry is often one of the first really challenging college courses faced by students. We have
found that students frequently overestimate their "readiness" to do well in this course. Organic
chemistry, on the other hand, is a course that many students approach with trepidation, either because
of its reputation for being “tough” or because of the need for a very good grade. To help students with
a diverse range of needs and backgrounds succeed in general and organic chemistry we have designed a
pair of online short courses1.
ChemPrep/General Chemistry was developed as a self-paced preparatory course to help students
review concepts, fill in content gaps, build confidence, and strengthen mathematical
skills. ChemPrep/Organic Chemistry was designed to help students review the basic concepts of general
chemistry that are used extensively in organic chemistry. The curriculum for each course is listed in
Table 1.
Table 1: ChemPrep Curricula
General Chemistry
Organic Chemistry
1. Intro to OWL
1. Intro to OWL
2. Structure of Matter
2. Electronic Structure of Atoms
3. Nomenclature
3. Lewis Structures
4. Dimensional analysis, metric units, significant figures,
exponents and roots
4. Geometry and Polarity
5. Mole/mass conversions
5. Bonding and Resonance
6. Chemical Reactions
6. Structural Formulas
7. Algebra, Temperature, Density, Graphs and Logs
7. Alkanes & Functional Groups
8. Acids and Bases
Each topic contains short information pages to introduce a concept or activity, interactive tutors and
simulations to help develop the ideas more concretely, and homework questions that provide the
opportunity to test understanding and see detailed, well-formulated solutions for each problem.
The ChemPrep courses are delivered over the internet using the OWL (Online Web-based Learning)
system developed at the University of Massachusetts2-9. They are offered before the beginning of each
semester (in January and July) and may be accessed from any location, 24 – 7, with a web browser. Each
course is designed to be complete, and no textbook is required or recommended. Students work
independently at their own pace, but are encouraged to interact with the course instructors via the OWL
message system. The average student takes approximately 20 hours to complete the general chemistry
prep course and 10 hours to complete the organic prep course. No formal grades are assigned and
there is no penalty for non-participation or partial completion of the work.
The OWL System
The OWL electronic learning system has been used in the Chemistry Department at the University of
Massachusetts for over eight years10. Extensive databases of homework problems and interactive
simulations, exercises, and tutors, have been developed by our faculty for general and organic
chemistry. These materials have been correlated to a number of textbooks and are licensed by Thomson
Learning11.
The flexible authoring and course management tools in OWL allow for the creation and delivery of
instructional materials via the web. One of the most important pedagogical features within OWL is the
support for mastery learning. Students may work until they master a concept, and each time they
attempt a problem a similar, but not identical, question is given. This is accomplished in OWL by the
powerful feature that allows authors to parameterize questions both chemically and numerically.
Once a student submits an answer for evaluation OWL gives a detailed solution specifically
parameterized to the problem. Students may try the question again (similar but not identical) or move
on to new material. Contrast this to hand-graded homework where a student may turn in incomplete or
even incorrect work, and, by the time they receive feedback, they are no longer engaged with the
question. Contrast this again to traditional end-of-chapter homework questions where it is difficult for
students to test their understanding using a new instance of the question, and it is easy for them to
memorize solutions to particular problems without seeing the general idea behind each.
Instant, answer-based feedback has been shown to help students build confidence with course material
in a non-threatening (and non-testing) environment12. The combination of parameterized questions
with detailed, immediate feedback, and the requirement that a student correctly complete a question
before credit is given, makes OWL a very powerful learning system.
Additional features in OWL help to enhance the interaction between instructors and students. The
message system allows students to send questions directly to instructors about specific problems. A
detailed log of student activity makes it possible to see the particular instance of each question that a
student has been assigned, and the responses that were entered. Instructors can retrace a student's
steps and help students diagnose where they are having difficulty. Finally, instructors can collect data
about completion of each assignment and use this to guide further in-class activities.
For ChemPrep students are particularly encouraged to use the OWL message system to ask questions or
submit comments as they proceed through the material because they do not have direct contact with an
instructor. These messages are answered by an instructor, usually within 24 hours of submission. Our
students report that the combination of detailed content pages, interactive simulations, tutors and
exercises, parameterized homework where they can retry problems until they master a concept, and the
ability to send messages to the instructor, permits them to work at their own pace without the need for
a textbook or lecture.
ChemPrep Study
The Spring 2004, off-sequence sections of first semester general chemistry and first semester organic
chemistry were chosen to be our pilot groups. The performance of participating students was compared
to that of non-participants in the subsequent courses. Based upon the encouraging results obtained
with these groups, ChemPrep has been offered to all students enrolled in any of our introductory
general and organic chemistry courses since that time. These courses are referred to as: Nursing,
General, General Honors, Organic and Organic Honors in the discussion that
follows. The ChemPrep curricula were identical for all three general chemistry courses and similarly for
both of the organic courses13.
Students pre-registered in one of these courses were invited via email to work on ChemPrep over the
January break or during the summer. They had until the first day of classes to complete the work (about
5 weeks) and their progress was recorded in the OWL system. Online surveys were given at the end
ofChemPrep and at the end of the subsequent course.
Details of the pilot study for general chemistry have been presented at the Spring 2005 Meeting of the
American Chemical Society14 and can be found in the preprint of a paper that has been accepted for
publication in the Journal of Chemical Education15. Details of the pilot study for organic chemistry are
being prepared for submission.
Completion Rate
The participation rate of students in each of the courses is summarized in Table 2, where "users" are
defined as those students who completed more than 50% of the ChemPrep modules, and "non-users" as
those who did not sign up for ChemPrep. Students who signed up but completed less than half of the
work in the prep courses were not included in the analysis. Note the higher participation rate overall by
organic students and by honors students.
Table 2: Completion Rate
Gender Information
Women were more likely than men to participate in ChemPrep. Data corresponding to the general
chemistry courses are reported in Table 3.
Table 3: Percent Female Students by course
%Female
%Female
Total
Users
GenChem S04
41
66
GenChem F04
53
72
Nursing F04
80
94
Honors F04
51
69
Overall
53
73
Success Rates
We defined success in a course as a grade of "C-" or better. The success rates for students in the Fall 04
courses are summarized below. In all cases users out-performed non-users.
Average Grades
Average grades based on a four-point scale were higher for users, as shown below.
Discussion
Because the prep courses were entirely voluntary we sought to determine if the higher grades were due
toChemPrep or if they were simply a case of the better students choosing to take advantage of this
resource to help them excel.
We used math and verbal SAT I scores, and the algebra and trigonometry scores on the University’s
Math Placement exam, as proxy measures for the students’ prior achievement and ability. We then
determined the effect size for each of these measures as a standardized method of comparison. Effect
size (ES) is defined as the difference between the average score of user and non-user groups, divided by
the standard deviation of the total class.
Effect Size = (Avg Score User - Avg Score Non-user)
(Standard Deviation Class)
ES = 0, No difference between groups
ES < 0, Users weaker than Non-users
ES > 0, Users stronger than Non-users
Effect sizes with absolute values around 0.2 are considered small, 0.5 moderate, and 0.8 large by social
science researchers16,17. Small effect sizes indicate that there is little difference in aptitude between
user and non-user groups.
Effect Size - General Chemistry
For every course, the effect size for the grade was higher than those of the prior achievement/ability
measures. While the students in GenChem and Nursing who elected to do ChemPrep tended to be
stronger than their peers (ES > 0 for SAT and Math Placement), they also appeared to benefit beyond
what would be predicted by their entry skills. As shown in the figure below, the effect sizes of all of the
grades are positive and they range from moderate to large, indicating that there are meaningful grade
differences between users and non-users in all of the courses.
In the Honors course, there does not appear to be a significant difference between users and non-users
based on predictors of prior achievement/ability, but there is a large difference in grade
performance. Here, there seems to be a substantial positive relationship between ChemPrep usage and
course grades. This was a surprising result because the Honors group was stronger overall and we had
expected there to be, at best, a small increase in course grades for users. We were also surprised by the
fact that a much larger percentage of Honors students completed ChemPrep (23% vs. 9% for GenChem
and Nursing). If we assume that this group has a larger proportion of motivated students then this result
is reasonable.
Effect Size - Organic Chemistry
In the standard organic chemistry courses it appears that ChemPrep had a greater effect than would be
expected from the prior achievement/ability differences between groups. However, in the honors
course the grade difference is what one would expect from these measures.
Questions to Address
• How can we get students who really need a preparatory course to complete it ?
• What is best way to reach high school, transfer, and returning students ?
• Will we see similar results at other schools ?
• Does the curriculum meet the needs of potential users ?
Summary
On average, students who completed ChemPrep had higher grades in the subsequent general and
organic chemistry courses, with a greater percentage achieving a grade of C- or higher. Participation
inChemPrep was voluntary, and more women than men responded. Students in the Honors and Organic
courses enrolled in ChemPrep in higher percentages than students in GenChem and Nursing. SAT and
math placement exam scores were used as proxy measures of prior achievement/ability. Based on
these, users in Honors GenChem were on par with their peers but performed better in the course than
non-users. Honors OChem users performed as would be expected from the predictor variables. In
GenChem and Nursing, ChemPrep helped students of high prior achievement/ability perform better
than their achievement scores would predict. Weaker/less motivated students did not respond to the
voluntary offerings of ChemPrep in the same numbers as stronger/more motivated students, and we are
seeking alternate ways to reach this population.
References and Notes
1. Funding for this project was provided by the Professional Development Grant Program of
the University of Massachusetts Information Technology Council.
2. Stewart, Barbara N.; Vining, William J.. Analysis of student learning using online tutorials. In 229th
ACS National Meeting, San Diego, CA, March 13-17, 2005. AN 2005:188556.
3. Dufresne, R.; Mestre, J.; Hart, D.; Rath, K. The Effect of Web-based homework on Test
Performance in Large Introductory Physics Classes. Journal of Computers in Mathematics and
Science Teaching. 2002 , 21(3), 229-251.
4. Vining, W. J.; Botch, B.; Day, R. O.; Hart, D.; Woolf, B. OWL web-based homework system:
Improved student learning. In 221st ACS National Meeting, San Diego, CA, April 1-5, 2001. AN
2001:199085.
5. Vining, W. J.; Day, R. O.; Botch, B.; Woolf, B. Intelligent tutoring systems for general
chemistry. In 219th ACS National Meeting, San Francisco, CA, March 26-30, 2000. AN
2000:329052.
6. Woolf, B.; Hart, D.; Day, R.; Botch, B.; Vining, W. Improving Instruction and Reducing Costs with a
Web-Based Learning Environment. Proceedings of the International Conference on
Mathematics/Science Education & Technology, M/SET 2000. Pp. 410-415.
7. Hart, D.; Woolf, B.; Day, R.; Botch, B.; Vining, W. OWL: an Integrated Web-Based Learning
Environment. Proceedings of the International Conference on Math/Science Education &
Technology, San Antonio, TX, March, 1999. M/SET 99 Pp. 106–112.
8. Botch, B.; Day, R.; Vining, W.OWL: Online web-based learning. In 216th ACS National Meeting,
Boston, August 23-27, 1998. AN 1998:525480.
9. Day, R. O.; Botch, B. H.; Vining, W. J.; Woolf, B. P.; Hart, David M. OWL: Web-based interactive
multimedia discovery and intelligent tutoring environments for general chemistry. In215th ACS
National Meeting, Dallas, March 29-April 2 1998. AN 1998:141481.
10. OWL is currently used by over 20 departments on the UMass campus in disciplines ranging from
physics and chemistry to resource economics and art history. Examples of the OWL courses
delivered at UMass Amherst can be found at http://owl.oit.umass.edu (accessed July, 2005).
11. OWL: General Chemistry and OWL: Organic Chemistry are licensed and distributed by Thomson
Learning - Brooks/Cole Publishing and are used on over 150 college and university campuses
nationwide. A demonstration of the OWL system delivered through Thomson Learning can be
found at http://owl.thomsonlearning.com/demo (accessed July, 2005).
12. Cole, Renée S.; Todd, John B. J. Chem. Educ. 2003 80 1338-1343.
13. We were actually a bit surprised that the same curriculum worked for such diverse groups of
students. We attribute the general usefulness of these materials to the mastery learning feature
in OWL. Students who are comfortable with a topic can move through it quickly; those who are
less familiar can cycle through as many questions as needed to solidify their understanding.
14. Botch, Beatrice; Day, Roberta O.; Vining, William J.; Hixson, Steven; Samal, Peter; Hart, David;
Peterfreund, Alan; Rath, Kenneth A. ChemPrep: Self-paced OWL preparation for General
Chemistry and Organic Chemistry. In 229th ACS National Meeting, San Diego, CA, March 13-17,
2005. AN 2005:187164.
15. Botch, Beatrice; Day, Roberta O.; Vining, William J.; Stewart, Barbara; Hart, David; Peterfreund,
Alan; Rath, Kenneth A. “ChemPrep: A Self-Paced, Online Preparatory Course for General
Chemistry” J. Chem. Educ, accepted.
16. Cohen, J. Statistical Power Analysis for the Behavioral Sciences (Revised Edition). New
York: Academic Press. 1977.
17. Slavin, R. E. Educational Researcher 1990 19(3); 30-34,44.