School of Technology
Current status of
Formative Evaluation
Paul Rudman and Mary Zajicek
AtGentive First Project Meeting, Oxford, 22 May 2006
1
Thought experiments
Practical experiments
School of Technology
AtGentive First Project Meeting, Oxford, 22 May 2006
2
Formative Evaluation of Conceptual Framework
Software and PowerPoint presentation developed to visualise the
conceptual framework
• elaborates conceptual framework document
• facilitates presentation to potential AtGentive users
• clarifies framework for AtGentive partners
User profiling
• “Yes / Not now / Never” rather than “Yes / No” to facilitate collecting
user preference
• automatic inference of Workgroups from previous collaborations
• Student modelling by teachers
Collaborative document to develop scenario seven (re-attracting an idleuser attention) for Ontdeknet
Discussion of relationship between conceptual framework and design
• to what extent design should be constrained by conceptual
framework or framework expanded to incorporate new areas
School of Technology
AtGentive First Project Meeting, Oxford, 22 May 2006
3
Formative Evaluation of Design
Practical methodology for implementing the scenarios with AtGentNet
using “Temporal Context”
• relevant with AtGentNet as platform displays one document at any
one time
Discussion about application independence for the AtGentive module
• relationship between AtGentive module and application(s)—which
knowledge managed by application and which knowledge
managed by AtGentive module?
• should AtGentive module comprise multiple agents?
• extent to which AtGentive module should interface with “general”
utilities (e.g. email, Microsoft™ messenger, Skype™, Word™,
PDF™ documents etc.) General view is AtGentive will not initially
incorporate these
Discussion on avatar emoting and personality.
• utility of emoting?
School of Technology
AtGentive First Project Meeting, Oxford, 22 May 2006
4
Formative Evaluation of Avatars
Special meeting for Formative Evaluation at AUP 23rd Feb 2006. Mainly
discussed the evaluation of avatars.
• 2:40 pm
WP4
Questionnaire results (UTA)
School of Technology
AtGentive First Project Meeting, Oxford, 22 May 2006
5
Formative Evaluation for AtGentNet
Experiment asks participants to answer
questions on herbal remedies, using
a stack of Herb information cards
and indexes
For example, “You are going on a long
journey. Briefly describe the flowers
of a plant that could help”
Investigates the potential of scenarios one (restore task context) and two
(propose alternative task)
Measures are questionnaire and interviews
• views on the experience
• feelings when interrupted and usefulness of information offered
School of Technology
AtGentive First Project Meeting, Oxford, 22 May 2006
6
Formative Evaluation for AtGentNet
Investigates the potential for:
•
scenario one—restore task context (on resuming a task)
• participants interrupted and a note taken of the cards in use
• later asked to continue, with or without being told which cards used
previously
•
scenario two—propose alternative task (current task too long)
• participants given a choice of two questions. After a short time, told
them that the other question is easier to complete in the time
available
School of Technology
AtGentive First Project Meeting, Oxford, 22 May 2006
7
Formative Evaluation for AtGentNet
Pilot with ourselves (11:10 am tomorrow)
Focus group-style discussion afterwards will inform the development of a
questionnaire and contribute to the refinement of Key Indicators.
Interested in:
• views on the experience
• feelings when interrupted
• feelings about being offered “help”
• usefulness of information offered
School of Technology
AtGentive First Project Meeting, Oxford, 22 May 2006
8
Formative Evaluation for AtGentSchool
In March 2006 Mary visited Ontdeknet for a very productive meeting.
We discussed:
• The concepts and working of Ontdeknet
• Scenarios for attention
• Key Indicators for AtGentSchool
• Wizard of Oz AtGentSchool – provides rapid feedback on design
decisions. All intervention decisions normally taken by the AtGentive
module are taken by a human (the experimenter) who sits (logically)
between the participant and the available application responses.
School of Technology
AtGentive First Project Meeting, Oxford, 22 May 2006
9
Wizard of Oz (WOZ) and AtGentSchool
Res
Start task
pon
se 1
Response
Res
2
pon
se 3
1. Set the user or heuristic evaluator a task (e.g start to introduce yourself)
2. User responds at the interface
3. Wizard types short code for appropriate Onty script + new interface
(e.g. interruption from teacher etc.)
4. User responds at the interface
5. Repeat from Step 3 until the task is completed or has gone irretrievably wrong
School of Technology
AtGentive First Project Meeting, Oxford, 22 May 2006
10
Formative Evaluation using WOZ
(Ontdeknet) PILOT
Implement scenario in
AtGentSchool
Design meets criteria
(OBU + Ontdeknet)
Gather new design
information from the
evaluation
Redesign
(OBU + Ontdeknet)
Evaluate effectiveness
of intervention
according to Key
Indicators
(Ontdeknet)
Design
implementation of
scenario according to
function and
feasibility
School of Technology
(Ontdeknet)
Implement WOZ
scenario
(OBU)
Select Key
Indicators and
evaluation methods
AtGentive First Project Meeting, Oxford, 22 May 2006
11
School of Technology
Current status of
Formative Evaluation
Paul Rudman and Mary Zajicek
AtGentive First Project Meeting, Oxford, 22 May 2006
12