STUDENT SATISFACTION WITH
WEB-BASED EDUCATIONAL
RESOURCES FOR COMPUTER &
INFORMATION SCIENCE: Teaching
with Merlot Collection
Emrah Orhun, Ph.D.
Troy State University Montgomery
Alabama, USA
Outline
Introduction
 Learning and Teaching Materials in The
Merlot Collection
 Students’ Satisfaction with Merlot Materials
in CIS Classes
 Conclusions

Introduction

Difficulties in locating relevant and
affordable materials of high quality
– World Lecture Hall at the University of Texas
» 83 categories to browse and search
utilities to locate materials
– WebCT (2002)
» 1,250 publisher course content packs in more
than 50 subject areas
» customizable materials around which to build
courses combined with the functionality of WebCT's
course management software.
Quality issue

World Lecture Hall materials
– courses offered at accredited colleges and
universities around the world

WebCT packs
– editorial standards of the publisher and
author of the text
MERLOT (Multimedia Educational Resource
for Learning and Online Teaching)
Web-based open resource designed
primarily for faculty and students in higher
education
 emphasizes the quality issue

MERLOT repository
– Over 8,000 free learning materials and
support materials that can be integrated
within the context of a larger course
Searching and Browsing in MERLOT
Figure 1. Entries in the Applications sub/subcategory of Information Technology subcategory under
Figure 2. Search results for TECHtionary
Support resources
– sample assignments
– evaluations of the learning materials
»discipline panel peer reviews
»individual user reviews
Peer Review Process
Discipline communities
 Biology, business, chemistry, engineering,
health sciences, history, information
technology, mathematics, music, physics,
psychology, teacher education, and world
languages.

 Discipline
teams
(“MERLOT Editorial Boards”)
– selected by the MERLOT partner organizations
in accordance with the MERLOT guidelines.
– develop evaluation standards,
– conduct peer reviews of learning materials
within the discipline, and
– make connections with their professional
organizations
Peer review process


Evaluated by at least two team members who
have relevant expertise.
Review process stages
– triage review to identify worthy candidates
– review of candidates
– composite review report sent back to the authors
for feedback and permission to post the review on
MERLOT.
– approved review is posted to the MERLOT Web
site by the discipline leader
 The
evaluation criteria:
–quality of content,
–potential effectiveness as a
teaching tool,
–and ease of use.
Each dimension evaluated separately
(on a scale of 1-5 stars, with 5 being the
highest), and an overall rating is
computed
 A review must average three stars in
order to be posted to the site.

Figure 3(a). Peer Review of TECHtionary (Part a)
Figure 3(b). Peer Review of TECHtionary (Part b)
Figure 3(c). Peer Review of TECHtionary (Part c)
Figure 4. Member Comments for TECHtionary
Current state of MERLOT
Table 1 Growth of MERLOT materials,
members, and peer reviews (May 28, 2002)
Total
Added
Added
during last during last
365 days
30 days
1991
229
Materials
6863
Members
8613
4142
600
Peer
Reviews
479
361
20
Table 1 Growth of MERLOT materials,
members, and peer reviews
(February 27, 2003)
Total
Added
during last
365 days
Added
during last
30 days
Materials
8331
2338
284
Members
12637
5527
712
Peer Reviews
778
86
2
Member
Comments
Assignments
1790
305
18
322
59
1
Table 1 Growth of MERLOT materials,
members, and peer reviews
for IT Category
Total
Added
Added
during last 365 during last 30
days
days
Materials
559
274
21
Members
524
259
30
Peer Reviews
36
3
0
Member
Comments
Assignments
218
99
6
11
9
0
Ratio of Peer Reviews, Member Comments, and
Assignments to Materials
May 25, 2002
September
26, 2002
February 27,
2003
Materials
6863
7395
8331
% Peer
Reviews
7.0
8.5
9.3
% Member
Comments
21.2
20.8
21.4
%
Assignments
3.4
3.8
3.9
Problem

Percentage of reviewed materials low
Cause

Editorial peer review process too slow
Potential Solution

Peer review by community of learners
STUDENTS’ SATISFACTION WITH
MERLOT MATERIALS IN CIS CLASSES
Goal of Study

To find out to what extent MERLOT
provides relevant and quality materials for
the CIS courses.
Questions




Is the quality and quantity of learning materials in
the MERLOT collection sufficient to satisfy CIS
students’ knowledge/learning needs in specific
courses?
Is the MERLOT collection ‘usable’ by the
students?
Does the MERLOT collection induce the students
to become members of learning communities?
Can peer review by students substitute for peer
review by editorial boards?
Questions re-formulated
How do students rate their satisfaction with
MERLOT collection?
 How do students rate the learning materials
as reviewers? Do students’ ratings of the
learning materials differ from the ratings
given by Discipline Panel members?

Teaching with the MERLOT
Collection in CIS Courses

Summer-Fall 2002 semesters in the following
CIS classes at Troy State University
Montgomery:
–
–
–
–

CIS 5547 CAR
CIS 5547 CBR
CIS 3332
CIS 3357
Applied Systems Analysis
Applied Systems Analysis
Software Engineering I
Logical Structures of
Computer Design
Overall, 41 students participated in the evaluation.
A total of 252 evaluations were made.
Student Assignment
Evaluation of web-based learning materials
. 1. Visit MERLOT (www.merlot.org) site and register as a member.
. 2.
For each chapter covered in the course:
. a.
Find a relevant learning material in the IT subcategory.
. b. Review the material using the guidelines provided by MERLOT.
In the table provided below, state what learning outcomes are supported
by the material and post your review as a Member Comment on
MERLOT.
. c.
Rate your satisfaction with MERLOT collection on a scale 1-5,
with 5 implying that you were fully satisfied with the quality and quantity
of learning materials you found in connection with your task, and 1
implying that you were not satisfied at all.
. d. Find a relevant online learning material not contained in
MERLOT, post it to MERLOT and add a Member Comment.
Example of Relevant MERLOT Material
Chapter
Topic
Title
URL
Learning
outcome
Satisfaction
with MERLOT
collection for this
task 5= very good
1=very poor
1
Players in the
System Game
Enterprise
Resource
Planning
Supersite
http://www.erp
supersite.com
Provides
analysis on
ERP trends
3
2
Information
System Building
Blocks
Interface
Design for
Educational
Multimedia
http://www.cdl.
edu.rachel/PD
F/intdesmm.pd
f
Describes the
principle of
interface
design
4
3
Information
System
Development
Risk analysis
and
management
for Projects
(RAMP)
http://www.ra
mprisk.com/fro
ntindex.htm
Comprehensive
outline of
RAMP
5
Example of New MERLOT material posted
Chapter
Topic
Title
URL
Learning
outcome
1
Players in the
System Game
New
www.informatio
responsibilities
nweek.com
Fall Heavy on IT
Workers
Talks about the
new job
responsibilities
of current and
upcoming IT
employees.
2
Information
System Building
Blocks
TP Middleware
keeping up with
Mainframe
Revival
www.adtmag.co
m
Explains the
functionality of
the product TP
middleware on
mainframe
systems.
3
Information
System
Development
Tips for a
successful
Software
Distribution
Project.
www.computerw
orld.com
Shares other
companies
success stories of
in relation to
Software
Distribution
Project.
How do students rate their satisfaction with
MERLOT collection?
Students’ mean satisfaction ratings for MERLOT collection
Numbe
r of
Studen
ts
Numbe
r of
Topics
Numbe
r of
Evaluat
ions
Averag
e
sdev
CIS3357 Logical Structures of Computer Design
8
7
49
3.8
1.3
CIS5547 (A+B) Applied Systems Analysis
22
14
165
3.8
0.8
CIS3332 Software Engineering I
11
4
38
3.8
1.0
COURSE
CIS3357 Students’ satisfaction with MERLOT collection
C
h
Topic
N
Averag
e
sdev
Min
Max
Percent
rated 4-5
1
Binary Numbers & Codes
8
4.7
1.4
3
5
87.5
2
Digital Circuits
8
4.6
1.5
3
5
75.0
3
Combinational Circuits
8
4.3
1.9
1
5
62.5
4
Sequential Logic Systems
8
4.2
1.9
1
5
62.5
5
Registers & Counters
8
3.8
2.1
1
5
50.0
6
Memory & Programmable Logic
5
3.2
1.5
1
5
40.0
7
Register Transfer & Computer
Operations
4
3.8
1.9
1
5
75.0
CIS3332 Students’ satisfaction with MERLOT collection
C
h
Topic
N
Averag
e
sdev
Min
Max
Percent
rated 4-5
1
Process
13
4.2
0.8
1
5
76.9
2
Project Management
12
4.2
0.4
2
5
91.7
3
Requirements Analysis I
11
3.8
1.2
1
5
72.7
4
Requirements Analysis II
10
3.6
1.2
1
5
70.0
5
Software Architecture
13
4.2
0.8
1
5
76.9
6
Detailed Design
12
4.2
0.4
2
5
91.7
CIS5547 Students’ satisfaction with MERLOT collection
C
h
Topic
N
Avera
ge
sdev
Min
Max
1
Players in the System Game
22
3.6
0.8
2
5
Percent
rated
4-5
59.1
2
Information System Building
Blocks
20
3.7
0.8
2
5
55.0
3
Information System Development
19
3.6
1.2
1
5
63.2
4
Project Management
10
4.1
0.7
3
5
80.0
5
Systems Analysis
12
3.8
0.6
3
5
75.0
6
Requirements Discovery
12
3.9
0.8
3
5
66.7
1
1
Application Architecture and
Modeling
10
3.9
0.7
3
5
70.0
CIS5547 Students’ satisfaction with MERLOT collection
(contd.)
A
Object-Oriented Analysis and Modeling
9
3.7
0.7
3
5
55.6
B
Object-Oriented Design and Modeling
9
3.9
0.8
3
5
66.7
1
3
Output Design and Prototyping
9
4.0
0.5
3
5
88.9
1
4
Input Design and Prototyping
9
4.2
0.4
4
5
100.0
1
5
User Interface Design
9
3.9
0.8
3
5
66.7
1
6
Systems Construction and
Implementation
7
4.1
0.9
3
5
71.4
1
7
Systems Operations and Support
8
4.1
0.8
3
5
75.0
How different are students’
evaluations from Editorial Peer
Reviews?
Distribution of materials in IT
Type
***
Tutorial
File System
Design for an
NFS File Server
Appliance
Cascading Style
Sheets, Level1
****
*****
Electronic
Commerce
Tutorial
Learning
Microsoft Word
2000
Data
Communication
s
FTP Tutorial
Web-Database
Integration
ADSL
Computer
Knowledge
Virus Tutorial
Mulder's
Cascading Style
Sheets Tutorial
Sharky's
Netscape
Frames Tutorial
Tje ABCs of
Building a Web
Site
Study Guides
and Strategies
Reference
Material
Animation
TECHtionary
Warriors of the
net
Tips for Using
Blackboard
Sample of Materials from IT Category
Material
Type
***
****
*****
Tutorial
ADSL
Web-Database
Integration
Reference
Material
Animation
TECHtionary
Distribution of materials according to type and rating
(excluding materials with no user comments with 3 categories)
Sample of Students


CIS3332 SOFTWARE ENGINEERING (16)
CIS5547 SYSTEMS ANALYSIS (27)
Students’ Ratings
Case
TECHtionary
Web-Database
Integration
Editorial
Board
Rating
Member
Rating
3.0
*****
Students’
Rating
(mean)
3.75
3.67
***
3.82
Students’ ratings for TECHtionary
Mean
Std. Dev.
Std. Error
Variance
Coef.Var.
3.754
.421
.086
.177
11.218 24
Minimum Maximum Range
Sum
Sum
Squared
2.5
90.1
342.33 0
4.4
1.9
Editorial Board Rating: 3.00
Count
# Missing
Students’ ratings for Web-Database Integration
Mean
Std. Dev.
Std. Error
Variance
Coef. Var. Count
3.829
.545
.119
.297
14.238 21
Minimum Maximum Range
Sum
Sum
Squared
2.8
80.4
313.76 0
4.9
2.1
Editorial Board Rating: 3.67
# Missing
Is the mean student rating different from the rating
given by the Editorial Board?

TECHtionary
– Different (one-group t-value = 8.773, p=0.0001)

Web-Database Integration
– Insufficient evidence (one-group t-value = 1.333,
p=0.1975)
Conclusions
Students are satisfied with the quality and
quantity of learning materials they found in
the IT category.
 Current data suggest that students rate
materials differently than editorial boards.
 The present limited study does not provide
evidence for the notion that peer review by
students can substitute for peer review by
editorial boards.
