Usability evaluation in a multiphase,
exploratory design-based research study of
an online community for the practice of
special education in Bulgaria
Rob Peterson, PhD
University of Wollongong
Faculty of Education
robpeterson@gmx.com
www.robertrpeterson.com
Ian Olney, EdD
University of Western Sydney
Teaching Development Unit
i.olney@uws.edu.au
June 2009, ED-MEDIA, Hawaii
2
Overview
Background of study
Website development cycle
Types of usability evaluations
Participant selection
Usability tasks and scenarios
Data collection
Equipment
Data analysis
Results
June 2009
ED-MEDIA
Hawaii
3
Background
Establishment of the Special Education
Bulgaria online community




3-year PhD project
Design-based research
Exploratory research design
Design principles for virtual communities of
practice
 Moodle and Mahara
 www.SpecialEducationBulgaria.com
June 2009
ED-MEDIA
Hawaii
4
Data Sources
Research Participants…………………..76





Personal interviews
19
Usability interviews...……………………….18
Expert consultations
4
Web-based questionnaires
18
Email-based questionnaires
17
Website Participants…………………….242
 Registered website users
 Discussion forum and other posts
Website Usage Logs
June 2009
ED-MEDIA
Hawaii
Prototype
Website
Development
Cycle by
Research
Phase
1. User and
usage needs
analysis
Phase 1: Needs assessment
Exploratory usability evaluations
2. Specification
of requirements
Alpha
3. Preliminary
design
Phase 2: Formative evaluation
Assessment usability evaluations
4. Detailed
design
Аdapted from “Product
development life cycle”
(Rubin,1994)
Beta
5. Product
build
Phase 3: Effectiveness evaluation
Validation usability evaluations
Final
6. Product
release
Website Versions
Prototype
Alpha
Beta/Final
7
Usability Evaluations
Inquiries, inspections, and tests
Exploratory, assessment, and validation
usability evaluations
Phase 1, Phase 2, Phase 3
Qualitative
Quantitative
June 2009
ED-MEDIA
Hawaii
8
Participant Selection
Purposeful-quota sampling
Purposeful-intensity sampling
June 2009
ED-MEDIA
Hawaii
9
Tasks and Scenarios
Task: Find a definition for the word dyslexia.
Task: Send a message to the website
administrator.
Scenario 1: Registration
Scenario 2: Chat
Scenario 3: Discussion forums
Scenario 4: Lost password and username
Scenario 5: Upload
Post-Test Questions
June 2009
ED-MEDIA
Hawaii
10
Data Collection
Observation Sheet
Design Element
Importance (A-C)
Repairability (1-3)
June 2009
ED-MEDIA
Hawaii
Problem /
Recommendation
Setup and Equipment for Usability
Evaluation at Remote Location
Problem/Solution Matrix for
Analysis of Usability Data
A
g
e
S
e
x
E
x
p
Version
Task (1-22)
Scene (A-E)
#
yyyy-mm-dd
hh:mm:ss
1
2005-09-15
00:50:38
Name
40+
F
H
Proto
6
Navigation bar
2
2005-09-15
00:48:23
Name
23-30
F
M
Proto
9
Registration
3
2006-05-15
00:10:17
Name
17-22
M
L
Alpha
C
Forums
4
2006-05-15
00:22:14
Name
40+
M
H
Alpha
E
Internet resources
Participant
Imp. (A-C)
Rep. (1-3)
Problem /
Recommendation
Solution
Design Element
See
Also
Re
test
Fixed
/ or X
B/3
- Problem description
- Recommendation
- Solution enacted
2
Y
/
A/2
- Problem description
- Recommendation
- Solution enacted
1
Y
N
X
C/2
- Problem description
- Recommendation
A/1
- Problem description
- Recommendation
N
X
- Solution enacted
Usability Findings by
Importance (A-C) and Repairability (1-3)
30
Total number of
findings: 142
25
Number of
findings
repaired: 108
20
15
10
5
0
A1
B1
C1
A2
B2
C2
A3
B3
C3
14
Common Findings (Repaired)
Superfluous or confusing data required in
registration process.
Confusion regarding the type of characters
to use, Cyrillic or Latin.
Cannot see other participant’s full profile
information such as email address or city.
June 2009
ED-MEDIA
Hawaii
15
Common Findings (Not Repaired)
Cyrillic characters not supported by Moodle,
e.g., Cyrillic characters do not alphabetize
correctly in lists.
Only 50% of the RSS feed’s Bulgarian news
items returned actually had to do with
education and Bulgaria.
June 2009
ED-MEDIA
Hawaii
16
Key Points
Special Education Bulgaria project
Exploratory and design-based research
Usability inquires versus tests
Usability evaluations at remote locations
Data analysis in terms of importance and
repairability
Challenges and solutions … see the paper
June 2009
ED-MEDIA
Hawaii
17
Authors
Rob Peterson, PhD
University of Wollongong
Faculty of Education
www.robertrpeterson.com
Ian Olney, EdD
University of Western Sydney
Teaching Development Unit
i.olney@uws.edu.au
Supervisors
Prof Jan Herrington
Murdoch University
School of Education
A/Prof Mira Tzvetkova-Arsova
Sofia University, Faculty of Primary
and Preschool Education
A/Prof Deslea Konza
Edith Cowan University
Faculty Education and Arts
A/Prof Krassen Stefanov
Sofia University, Faculty of
Mathematics and Informatics
Ian Olney, PhD
University of Western Sydney
Teaching Development Unit
Building the European
Network for Lifelong
Competence Development
18
Key References
Barnum, C. M. (2002). Usability testing and research. New York, NY: Longman
Publishers.
Barnum, C. M., Mukherjee, A., Mielke, N., Porter, M., Schumann, M., Speaks,
B., et al. (1999). Hotmail usability test plan. Retrieved January 12, 2006, from
http://www.ablongman.com/barnum
Creswell, J. W., & Clark, V. L. P. (2007). Designing and conducting mixed
methods research. Thousand Oaks, CA: Sage Publications.
Reeves, T. C. (2006). Design research from a technology perspective. In J. van
den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational
design research (pp. 52-66). London: Routledge.
Rubin, J. (1994). Handbook of usability testing: How to plan, design, and
conduct effective tests. New York, NY: John Wiley & Sons, Inc.
Rubin, J., Chisnell, D., & Spool, J. (2008). Handbook of usability testing: How
to plan, design, and conduct effective tests (2nd ed.). Indianapolis, IN: Wiley
Publishing, Inc.
June 2009
ED-MEDIA
Hawaii
Addendum
Multiphase
Design
Based
Research
Adapted from “Predictive
and design research
approaches in educational
technology research”
(Reeves, 2006)
Stage 1
Analysis of practical
problems by
researchers in
collaboration
Stage 2
Development of
solutions informed by
existing design
principles and
technological
innovations
Stage 3
Iterative cycles of
testing and refinement
of solutions in practice
Stage 4
Reflection to produce
“design principles” and
enhance solution
implementation
Stage 1
Analysis of
practical problems
by researchers in
collaboration
Stage 2
Development of
solutions informed
by existing design
principles and
technological
innovations
Stage 3
Reflection to
produce “design
principles” and
enhance solution
implementation
Phase 2
Formative
Evaluation
Stage 1
Iterative cycle of
testing and
refinement of
solutions in
practice
Stage 2
Development of
solutions informed
by existing design
principles and
technological
innovations
Stage 3
Reflection to
produce “design
principles” and
enhance solution
implementation
Phase 3
Effectiveness
Evaluation
Stage 1
Iterative cycle of
testing and
refinement of
solutions in
practice
Stage 2
Development of
solutions informed
by existing design
principles and
technological
innovations
Stage 3
Reflection to
produce “design
principles” and
enhance solution
implementation
Phase 1
Needs
Assessment
Future impact
and maintenance
evaluations