Developing a Metric for Evaluating
Discussion Boards
Dr. Robin Kay
University of Ontario Institute of Technology
2 November 2004

Overview
Background
Data Collected
12 Areas of Evaluation
Sample Results
Summary

Background




Discussion board use has grown extensively
(e.g., Cooper, 2001)
Some say tool is revolutionary
( e.g., Hara et. al., 1998, Li, 2003)
Others say our understanding of discussion boards is minimal
(e.g., Blignaut & Trollip, 2003)
One problem: The metric used to evaluate effectiveness of discussion boards

Previous Metrics
 Most studies examine one or two aspects of online discussion
 A few researchers have attempted more complete analyses
(e.g., Hara et. al., 1998; Zhu, 1998)
 Metrics are rarely theory-driven

Metric Needs



Comprehensive
Theory-Based
Consistent

Data Collected (1 of 3)

Data Collected (2 of 3)

Data Collected (3 of 3)
 Subject Pool
 secondary, higher education
 Purpose of Discussion Board
 debate, posting resources, solving problems
 Individual Differences
 gender, typing speed, access
 Type of Course
 Online Only vs. Mixed

12 Areas of Evaluation






Social Learning
Cognitive Processing
Discussion Quality
Initial Question
Role of Educator
Navigation






Challenges
Types of Users
Attitudes Toward DB
Reponses Time
Time of Learning
Performance

Social Learning
Research
 Vygotsky (1978) and Slavin (1995)
 Number of researchers have reported that true social interaction is rare in DB
Data Used
 Length of discussion thread
 Number of messages read
 Primary focus of message
 2 or more times in the same thread

Cognitive Processing
Research
 Rare to see theoretical taxonomy

Revised Bloom’s Taxonomy
(Anderson & Krathwohl, 2001)
Data Used
 Knowledge Type
 Processing Level

Discussion Quality
Research
 Researcher have looked at tone, reasoning, degree of controversy, and content
Data Used
 Message clarity & quality
 New Knowledge Added
 Reference to Course Knowledge
 External Resources Used

Initial Question
Research
 Some research supports clear, provocative questions that promote higher level thinking
 Other research notes that it is hard to find clear patterns
Data Used
 Clarity, quality, knowledge and processing type
 Number of times question was read
 Length of discussion

Role of Educator
Research

Some researchers say instructor’s role is critical for raising the level of discussion
 Others claim students need to construct their own knowledge; instructor stifles discussion
Data Used
 Student vs. Instructor
 Number of times message was read
 Length of message
 Response Time

Navigation
Research
 Problems reported with respect to message length, number of entries, unclear subject lines, lack of organization
Data Used
 Subject line clarity & location
 How often message was read
 Response time
 Interview data

Challenges for Participants
Research
 Ability to participate, pace slower, time taken to participate, being grader
Data Used
 Interview data
 Open ended question about use

Types of Users
Research
 Participants assume different roles based on participation, degree of reflection, and mediation skills
Data Used
 Average number of message read
 Average response time
 Number of words
 Message quality
 Number of message posted

Attitudes Toward Discussion Board
Research
 Little systematic research done in this area
Data Used
 Interview
 Perceived usefulness (consumer and provider)

Response Time
Research
 No systematic research, but speculation that delays in response time could decrease value of discussion
Data Used
 Message location and response time
 Response time corrected with how often messages are read

Time of Learning
Research
 No research on how much discussion goes on outside school
Data Used
 Inside vs. Outside School
 Number of message posted
 Clarity and quality

Learning Performance
Research
 Has yet to be formally tested
 Do discussion boards improve learning?
Data Used
 Final Test and Project Grade correlated with
 number of visits
 number of days visited
 number of message posted

Sample Results
Sample






Overview
Social Learning
Cognitive Processing
Discussion Quality
Initial Question
Role of Educator
Navigation






Challenges
Types of Users
Attitudes Toward DB
Response Time
Time of Learning
Performance

Sample








45 secondary students (2 classes)
13-15 years old
Introductory computer science course
HTML and Programming
Private school
All boys
HTML (24 days)
Programming (36 days)

Overview of Results
Measure
Length of Thread
Words
Clear subject line
Message quality
Number of times read
Response Time
Content
Non-Academic
Average
3.5 msg. (1 to 11)
48.3 (1 to 263)
1.68 (0 to 3)
2.3 (0 to 4)
11.3 (2 to 77)
2.5 days (1 m. to 34 d.)
86% course or beyond
6%

Social Learning
Measure Results
Threads with 5 or more messages
Average number of times a message was read
Ask questions
47%
29.5
66%
Participating twice in same discussion
37%

Cognitive Processing
Measure
Procedural Knowledge
Conceptual Knowledge
Understanding
Remembering
Applying
Results
57%
21%
35%
27%
22%

Discussion Quality
Measure
Clear Messages
Message quality good
Results
67%
41%
New Knowledge Added 67%
Content (course or more) 86%
Non-Academic 6%

Effect of Initial Question
Results Measure
Easily Answered
Subject line clarity
Message quality
Knowledge Type
Processing Level
No impact on number of times a message was read or length of discussion

Role of Educator
Measure
Student
Teacher
Results
No difference with respect to number of times read, length of message, or response time

Navigation
Measure Results
Correlation between clear subject line & number of times read
Correlation between clear subject line & response time
Correlation between message number & number of times read
Correlation between message number & response time
Navigation a problem
Not Significant
Not Significant r = .26; p <.001
Not significant
54% of the time

Challenges
Measure
Too Slow
Results
38%
Technical of software problems 25%
Trusting peers answers 22%
Difference in Learning Style
Lack of ability
Inhibited by grades
12%
12%
11%

Types of Users (5+ messages)
Measure
Average message read
Average response time
Number of words used
Message quality
Number of message posted
Subject line clarity, question difficulty, knowledge and processing type
Results p <.001
p <.001
p <.001
p <.001
1 to 17
No significant difference

Attitude Toward Discussion Board
Measure
Effective Learning Tool
Used Frequently
Results
37%
38%
Received useful information 65%
Provided helpful information 39%

Response Time
Measure
Average Response Time
Results
25.3 hours
Jump from 3 rd to 4 th message 19.7 to 31 hours
Correlation between response time and number of times message read r = -.254; p < .01

Time of Learning
Measure
Message posted outside of school hours
Difficulty of question
Subject line, quality of message, response time, number words
Results
55%
Outside class (p <.05)
No significant difference between inside and outside class

Performance

Summary
 Previous research uses limited metric
 12 areas covered offer a relatively rich analysis of discussion
 If we want to understand the use if discussion boards we need metric that is


Comprehensive (details are everything)
Theory-Based

Contact Information
 Robin Kay
 robin.kay@uoit.ca
 Website for Paper & Presentation
 faculty.uoit.ca/kay/elearn2004/elearn2004.htm