Evaluating Adaptive Authoring of
AH
Maurice Hendrix
IAS seminar, The University of Warwick, 09/02/2007
maurice@dcs.warwick.ac.uk
http://www.dcs.warwick.ac.uk/~maurice
Outline
• Why automatic authoring
• System overview
• Semantic Desktop
• Adding resources
• Evaluation
How to do it, in general, for
AH?
• We want personalization, thus multiple paths
• Simple idea:
– Say we have a basic course on a topic
– Adding papers (articles) on the same topic, would
make it an advanced course
– So we would have 2 versions: one for beginners,
one for advanced learners => thus AH material
– How to do this automatically?
Screenshot beginner version
Screenshot advanced version
Why automatic authoring
• Make authoring task easier
• Manual annotation is bottleneck
• By integrating authoring environment
into semantic desktop
System overview
RDF
Database
MOT
Enricher
Lucene
Beagle++
CAF
CAF
Sesame
MOT hierarchy structure
Concept maps and lessons are hierarchies:
Adaptive Hypermedia
Hypermedia
AH
From AH to Adaptive Web
Semantic Desktop
• Desktop where everything is stored with
extra metadata
• We uses RDF as storage format
• Example RDF (also has an XML representation):
Conference
published At
Publication
Adding Resources
• MOT goal/domain maps are hierarchies
with tree structure, siblings are concepts
at the same level
• The Semantic Desktop can be searched
for resources. They are ranked by 2
formulae
Ranking
• Concept oriented
• Article Oriented
| k(c)  k(a) |
rank(a,c) =
| k(a) |
| k(c)  k(a) |
rank(a,c) =
| k(c) |
where:
rank(a,c) is the rank of article a with respect to the current domain concept c;
k(c) is the set of keywords belonging to the current domain concept c;
k(a) is the set of keywords belonging to the current article a;
|S| = the cardinality of the set S, for a given set S.
Selection of ranking method snapshot
Equal ranks
Allow duplicates among
siblings
• We call concepts in MOT at the same
depth in the hierarchy Siblings
• The author has to make a choice.
• Adding to all siblings can mean students
get the link multiple times
• Choosing one of the siblings can mean
students don’t always get the link when
relevant.
Selection of duplicates/none
snapshot
Add meta-data as separate
concepts
• The retrieved resources might have
attributes themselves
• If resources have further attributes,
these can be added as domain
attributes in MOT
• The resource can also be made into a
domain concept with its own separate
domain attributes
Add metadata as attributes
Add metadata as Separate concepts
Separate concepts/ attributes
snapshot
Compute resource keywords
as set
• The number of times a keyword occurs
might indicate the relevance of the
keyword.
• The ranking formulae can be computed
on sets of keywords or multisets.
Set/ multiset snapshot
Before MOT hierarchy snapshot
After MOT hierarchy snapshot
Evaluation
• intensive two-week course on AH &
Semantic Web
• 33 out of 61 students selected: 4th year
Engineering & 2nd year MsC in CS
• After week: theoretical exam (for selecting) at
the end: practical exam & 5 questionaires
• 3 systems: OLD MOT, NEW MOT &
Sesame2MOT
• 3 SUS 2 more specific
Hypotheses
1. The respondents enjoyed working as
authors in the system.
2. The respondents understood the
system.
3. The respondents considered that
theory and practice match.
4. The respondents considered the
general idea of Adaptive Authoring
useful
Questionnaires
• Constructed direct questions: enjoy,
understand in line with theory,
preference etc
• Based upon division of main
hypotheses
• Mostly multiple choice
• Some open questions for later analyses
Hypotheses results
1. The respondents enjoyed working as
authors in the system.
2. The respondents understood the
system.
3. The respondents considered that
theory and practice match.
4. The respondents considered the
general idea of Adaptive Authoring
useful
SUS
• System Usability Scale
• Measure for comparing systems.
• 10 questions, 5 positive 5 negative to make
respondents think.
• Score 1-5
• Normalised score: for positive questions:
score-1 for negative questions 5-score
• Total score: Sum of scores * 2.5
Extended hypotheses
1.
2.
3.
4.
5.
6.
7.
8.
9.
Respondents enjoyed the 3 systems
NEW MOT preferable to the OLD MOT in terms of
usability
sesame2MOT preferable to OLD/NEW MOT
NEW MOT is easier to work with then OLD MOT
Sesame2MOT is easier to work with then
OLD/NEW MOT
NEW MOT is more enjoyable then the OLD MOT
Sesame2MOT is more enjoyable then OLD/NEW
MOT
NEW MOT is easier to learn then OLD MOT
Sesame2MOT is easier to learn then OLD/NEW
MOT
SUS Results
Sesame2MOT
OLD MOT
NEW MOT
use frequently
4
need to learn a lot to use
complex
3
confident to use
2
easy
1
cumbersome
need support
learn quickly
well integrated
inconsistency
Extended hypotheses
1.
2.
3.
4.
5.
6.
7.
8.
9.
Respondents enjoyed the 3 systems
NEW MOT preferable to the OLD MOT in terms of
usability
sesame2MOT preferable to OLD/NEW MOT
NEW MOT is easier to work with then OLD MOT
Sesame2MOT is easier to work with then
OLD/NEW MOT
NEW MOT is more enjoyable then the OLD MOT
Sesame2MOT is more enjoyable then OLD/NEW
MOT
NEW MOT is easier to learn then OLD MOT
Sesame2MOT is easier to learn then OLD/NEW
MOT
SUS results correlation
• The scores of the SUS questionnaires
are all significantly correlated
• These correlations are all significant
• From reactions we suspect respondents
did not notice the difference between
the SUS questionnaires.