Evaluation of On-Line Learning
on Campus
Harry R. Matthews
Professor and Director
Instructional Technology & Digital Media Center
University of California at Davis
http://moby.ucdavis.edu/HRM/IEEE00.htm
http://moby.ucdavis.edu/HRM/IEEE00.ppt
Hypothesis
Delivery of course content on-line can
• be cost-effective
• improve learning
• be acceptable to students
• be acceptable to faculty.
The Strategy

Create on-line content for
–
–
–
–
10
large
undergraduate
general education courses

Students choose the on-line or in-person content
 Compare
– costs
– learning
– acceptability.
Course names










Food Science & Mythology
Agricultural Labor
General Biology
Environmental Law
Introduction to Computing
Computers in Agriculture
Anthropology
Statistics
Introduction to Wine-making
Asian Art History.
Some Variables
•
Students choose on-line course
• Appropriate for our question
•
Variable use of on-line/in-person content
• Treat as a continuous variable
•
Variable on-line presentation
• From mainly text to rich multiple media
•
Costs depend on prior materials.
Results from the pilot studies
•
5 year study of BCM 410A:
• lectures offered in-person in years 1, 2, and 5
• on-line in years 3 and 4
•
Aim was to improve higher-order thinking
skills:
• shift instructor contact-time from lecture to
discussion sections.
•
Improved grades in on-line classes
• 21%–26% A grades during in-person lecture years
• 35%–55% A grades in on-line years.
70
Number of students
60
50
1995
40
1996
1997
30
1998
20
1999
10
0
A
B
C
Grade
D
F
•
Student views of on-line content
• textual analysis of a free-response questionnaire.
What did you like about the virtual
lectures?
Number of Students
contol pace
visuals
back up/repeat
flexible schedule
home use
glossary
transcript
skip ahead
audio narration
w eekly quizzes
MoBy study aid
interaction
emall questions
1
0
10
20
30
40
50
60
Value for studying: virtual lectures
Student Evaluations (1997 + 1998)
extremely valuable
62
very valuable
47
valuable
33
somewhat valuable
22
not valuable
11
didn't use this method
8
0
10
20
30
40
# of students
50
60
70
Value for studying: textbook
Student Evaluations (1997 + 1998)
extremely valuable
7
very valuable
13
valuable
13
somewhat valuable
35
not valuable
28
didn't use this method
87
0
20
40
60
# of students
80
100
Results from the pilot studies
•
BIS 10 – General Biology
– Exam data showed no significant difference in
student learning between the in-person and on-line
course
•
•
•
•
BIS 10 was mainly text-based – no change in
learning
BCM410A was multiple media with integrated
tools – improvement in learning
Is difference in learning, if significant, due to
the nature of the on-line content?
What about the quality of the learning?
Evaluation of Learning

Quantitative
– test scores

Qualitative
– Student perception
– Analysis of tests

assign questions to
Bloom’s taxonomy of
levels of learning
– Determine learning
style

Kolb form.
Cost Analysis
•
•
Pre-requisite for scalability
Differential Cost Analysis
• Faculty time
• Assistance for faculty
• Infrastructure (buildings, network)
• Student support
• Costs associated with impacted courses.
Cost Analysis
•
Major component of cost is instructor time
– Direct time: e.g., planning and delivering the
course
– Indirect time: e.g. attending seminars.
Instructor time sheet
Web form completed daily
Email reminder
http://moby.ucdavis.edu/Mellon/timesheet.cfm
?Name=Falk
Conclusion
• “The Jury is IN. On-line learning works!
— David Brown, Jan. 2000
• Does the use of multiple media types affect
learning? In what situations?
• Is on-line learning cost-effective? Under
what circumstances?
• Where and to what extent should UC Davis
use on-line content delivery?
Evaluation of On-Line Learning on
Campus
http://moby.ucdavis.edu/HRM/IEEE00.htm
http://moby.ucdavis.edu/HRM/IEEE00.ppt