Text-Free User Interfaces
Kentaro Toyama
Assistant Managing Director
Microsoft Research India
Based on work with Indrani Medhi
TCS Excellence in Computer Science
January 10, 2008 – Pune, India
People
Lead Researcher
– Indrani Medhi
Collaborators
–
–
–
–
–
–
Kentaro Toyama
Archana Prasad
Vibhore Goyal
Shibili G
Nimmi Rangaswamy
Sean Blagsvedt
Interns
– Aman Sagar (Adobe)
– Bharathi Pitti (CMU)
– Chandima Patabandhige (Univ. of
Moratuwa, Sri Lanka)
– Renee Kuriyan (UC Berkeley)
Photo: Indrani Medhi
Indrani (holding baby) and some interns
visiting a family in Bangalore
Outline: Text-Free User Interfaces
Introduction
Initial ethnography and iterative rapid prototyping
Initial evaluation
Optimal audio-visual representation
Full-context video
Methodology
Discussion
Outline: Text-Free User Interfaces
Introduction
Initial ethnography and iterative rapid prototyping
Initial evaluation
Optimal audio-visual representation
Full-context video
Methodology
Discussion
Illiteracy
1-2 billion illiterate
population in the world.
98% live in developing
countries.
India’s rate of literacy
estimated at ~60%.
Photo: Indrani Medhi
Two residents of a slum in Jayanagar, Bangalore
Many labeled “literate” are
semi-literate with low
functional literacy.
Target Users
Women from Bangalore slums
Informal sector domestic
workers
Income range: INR 800-2500
(US$20-60) per month
Illiterate or semi-literate
Most have never seen a PC
(those who have seen have
never touched)
Photo: Indrani Medhi
initial explorations before text-free UI work
NGO Partner
Stree Jagruti Samiti
– Head: Geeta Menon
– Small NGO
– Works with several
Bangalore slums
– Focus on children’s rights,
women’s empowerment,
fair wages for domestic
laborers
Photo: Indrani Medhi
Geeta Menon leading a Saturday
Stree Jagruti Samiti meeting
The Problem
How can an application UI be converted into one
that is usable by non-literate users?
Outline: Text-Free User Interfaces
Introduction
Initial ethnography and iterative rapid prototyping
Initial evaluation
Optimal audio-visual representation
Full-context video
Methodology
Discussion
Initial Ethnography
TVs, radios, CD players
common; some mobile phones;
no PCs
Established slums – some with
30-year history in city; concrete
housing
Subjects most concerned
about…
Photo: Indrani Medhi
Nimmi interviewing a subject
– Jobs
– Healthcare
– Education
Text-Free UI?
Design principles:
– Pen or touch interface
– Liberal use of imagery
• No text
• Semi-abstracted
cartoons
– Voice annotation
– Aggressive use of mouseover functionality
Naukri.com for domestic labourers?
– Consistent help icon
Text-Free UI Details
Nouns vs. Verbs
Goal is to achieve exact
association between image and
word.
Without action cues drawings
tend to be interpreted as nouns.
“Kitchen sink” or “washing dishes”?
With action cues, drawings tend
to be interpreted as verbs.
Phenomenon well-understood by
cartoonists and diagram
designers.
“Pot” or “cooking”?
Text-Free UI Details
Cultural Differences
Goal is to achieve as global a
representation as possible.
Abstracted representations are very
dependent on culture.
Minor misrepresentations can result
in major confusion.
Testing, testing, testing!
An urban family of three?
Text-Free UI Details
Religious Differences
Goal is still to achieve as global a
representation as possible.
From when to when?
Religious differences can be the
strongest of cultural differences.
There is no “neutral” culture or
religion.
Testing, testing, testing!
Increasingly general representations
for time indication
Text-Free UI Details
Quirks of Non-Literacy
Various degrees of literacy
–
–
–
–
–
–
Total illiteracy
Numbers, but not alphabets
Alphabets, but not words
Words, but not sentences
Semi-literate, but not fluent
Etc.
Many who are non-literate can
read numbers
– Accustomed to currency
– Able to do basic arithmetic
– Not necessarily 100% accurate
Testing, testing, testing!
Illiteracy doesn’t necessarily mean
inability to read numbers.
Text-Free UI?
Original design
Revised design
Text-Free UI
Design principles:
– Pen or touch interface
– Liberal use of imagery
• No text (but numbers OK)
• Semi-abstracted cartoons
– Aggressive use of mouseover functionality
– Voice feedback
Screenshot of text-free job search
– Consistent help icon
Other Domains
Principles transfer to other applications:
Text-free maps
Text-free healthcare information
Outline: Text-Free User Interfaces
Introduction
Initial ethnography and iterative rapid prototyping
Initial evaluation
Optimal audio-visual representation
Full-context video
Methodology
Discussion
Initial Evaluation
Questions
1. Can non-literate users use traditional textbased user interfaces at all?
2. Do the proposed design principles for textfree user interfaces allow non-literate users
to use computers, and to what extent?
3. Which of the design principles make the
most difference for a text-free UI?
Initial Evaluation
Experimental Set-Up
Three versions:
– Text-based
– Text-free without help icon
– Text-free with help icon
Subjects:
– Illiterate or semi-literate
• (no literate subjects)
– No PC experience
Subject grouping:
– Five-women group x 2
– Individual x 4
Total time up to one hour per
session
Task:
For a friend who is
unemployed, find the bestpaying job in her
neighborhood.
Measured:
– Task completion (yes/no)
– Time required (second)
– Number of prompts required
(n)
All comments recorded; some
trials video-recorded.
Initial Evaluation
Quantitative Results
Text-based UIs were
completely unusable.
– Obvious, but first formal
such test in literature
With text-free UI, 30% were
able to complete task.
(However, not enough
participants for statistical
significance.)
Tabulation of initial results
Initial Evaluation
Qualitative Results
Strong preference for text-free
principles, in particular…
– Voice annotation
– Help icon
Collaborative use more
successful
Subjects eager to engage,
once comfortable with set up
Various barriers to technology:
fear, lack of awareness, lack of
cognitive model
Indrani and a subject during
initial evaluation
Initial Evaluation
Questions
1. Can non-literate users use traditional textbased user interfaces at all?
2. Do the proposed design principles for textfree user interfaces allow non-literate users
to use computers, and to what extent?
3. Which of the design principles make the
most difference for a text-free UI?
NO
Yes, but only
partially; more
work required
Hypothesis:
- Voice feedback
- Imagery
- Help
Outline: Text-Free User Interfaces
Introduction
Initial ethnography and iterative rapid prototyping
Initial evaluation
Optimal audio-visual representation
Full-context video
Methodology
Discussion
From initial round of research…
Text-Free UI
Design principles:
– Pen or touch interface
– Liberal use of imagery
• No text (but numbers OK)
• Semi-abstracted cartoons
– Voice feedback
– Aggressive use of mouseover functionality
Screenshot of text-free job search
– Consistent help icon
Goal of Study
Determine what types of icon representations are
best for UIs for non-literate users.
– Audio:
• With voice annotation
• Without voice annotation
– Visual:
•
•
•
•
•
Photorealistic video
Hand-drawn animation
Photograph
Hand-drawn image
Text
Domain of Testing
Desired characteristics of domain:
– Admits to visual representations,
– Consists of information which subjects are not exposed to on a daily
basis,
– Has universally recognized meaning,
– Offers a range of cognitive complexity,
– Offers a range of visual complexity, and
– Can be applied to a practical purpose.
Healthcare / medical imagery
– 13 medical symptoms: multimodal in nature, involving elements of color,
temperature, proprioception, and some requiring motion spanning some temporal
duration
Creation of Representations
Ensure consistency across representations…
Video
Photograph
Animation
Drawing
Tested Representations
Video
Photograph
Animation
Drawing
Text
Each with and without voice annotation (total 10 representations),
for 13 medical symptoms
Experimental Set-Up
Representations:
– One of 10 types
– Evenly distributed among
subjects
– All 13 symptoms per subject
Subjects:
– Illiterate or semi-literate
Threefold task:
•
•
•
Speak phrase that first comes to
mind when shown a
representation.
Explain symptom representation is
intended to convey
Enact or depict the intended
symptom (as explained by tester),
for analysis
• (no literate subjects)
– No PC experience
Size:
– Individual; n = 200
– 20 for each representation
Measured:
– Accuracy (correct, close,
wrong)
– Response time (seconds)
All comments recorded
Total time up to one hour per
session
Quantitative Results
Voice annotation critical, with
dominant effect
•
30% quicker response time
with audio
ACCURATE RESPONSES (out of 20)
12
9.75
10
Among cases with audio, handdrawn representations (both
animation and static imagery)
most accurately recognized
•
But, effect not dramatic
9.65
9.3
7.35
8
5.95
5.9
5.45
6
4
2.65
2
0
STATIC
PHOTO
ANIMATION
AVERAGE RESPONSE Without AUDIO
Among cases without audio,
animation most accurately
identified
•
23% more accurate than
static drawing
9.45
9.3
VIDEO
TEXT
AVERAGE RESPONSE With AUDIO
Number of accurate responses (out of 20)
for various representations
Qualitative Results
Test / text anxiety of subjects
– Subjects uncomfortable with test-like environment of studies
Short-term category conditioning
– Priming for “aches” over other types of symptoms when they appeared
in sequence
Richer information is not necessarily better understood overall
– Photorealism contains clutter; associated with actual instance
Context-laying activity provides additional in dynamic imagery
Socio-economic conditions correlated with cognition
– Those with less formal education had difficulty processing visuals and
audio at the same time; confused by multi-modal interface
Outline: Text-Free User Interfaces
Introduction
Initial ethnography and iterative rapid prototyping
Initial evaluation
Optimal audio-visual representation
Full-context video
Methodology
Discussion
From initial evaluation…
Quantitative Results
Text-based UIs were
completely unusable.
– Obvious, but first formal
such test
only!!!
Help icon
With text-free UI, 30% were
able to complete task.
However, not enough
participants for statistical
significance.
Tabulation
of initial video
results
UI instruction
Insufficient for end goal!
From initial evaluation…
Qualitative Results
Strong preference for text-free
principles, in particular…
– Voice annotation
– Help icon
Collaborative use more
successful
Subjects eager to engage,
once comfortable with set up
Various barriers to technology:
fear, lack of awareness, lack of
cognitive model
From post-experiment
interviews:
– Afraid to break PC
– Intimidated by use of
complex technology
– Confusion as to value of
PC for job search
– Lack of understanding
about how PC contains
relevant information
– Task better accomplished
Indrani
a subject
during
byand
asking
people
initial evaluation
– Verbal
explanations by
tester not convincing
New Problem!
Original question:
How can an application UI be converted into one
that is usable by non-literate users?
ILLITERACY
FEAR OF TECHNOLOGY
LACK OF TRUST IN TECHNOLOGY
LACK OF AWARENESS OF
WHAT TECHNOLOGY CAN DELIVER
New question:
Can a UI be developed to allow an illiterate, firsttime PC user to access information he/she
needs without any assistance or prompting?
What’s the Underlying Problem?
If you were told that,
if you put anything in this box and spelled
the object’s name in reverse
100 times out loud, it would come to life…
“Any sufficiently advanced
technology is indistinguishable
from magic.”
– Arthur C. Clarke
would you bother to do it?
Solution?
Observations about TV:
– Many households have TVs
– Soap operas a common topic of
discussion
– Even households without TVs are
familiar with TV
Observations about subject’s cognitive
styles:
Photo: Indrani Medhi
– Subjects engage in long linear
narratives
– General ideas conveyed through
specific instances
– Stories included real people, specific
objects, or actual instances of events
(Based on intuition developed
through prolonged engagement with
subjects!)
Photo: Indrani Medhi
Televisions common in households
Full-Context
Video
Solution:
Full-Context Video
A full-context video explains the
broader context of the application
and how it works, in addition to
instructional material about how to
use the application.
Full-Context Video
Experimental Set-Up
Two versions:
– Original text-free (without fullcontext video)
– Text-free with full-context
video
Subjects:
– Illiterate or semi-literate
• (no literate subjects)
– No PC experience
Task (the same):
For a friend who is
unemployed, find the bestpaying job in her
neighborhood.
Measured:
– Task completion (yes/no)
– Time required (second)
– Number of prompts required
(n)
Subject grouping:
– Individuals x 35
Total time up to one hour per
session
All comments recorded; some
trials video-recorded.
User Studies
Full-context video has
clear value:
Without video first, only one out
of 17 (6%) was able to complete
the task at all, taking 11 prompts
and 8.2 minutes
With video first, 18 out of 18
completed the task, with an
average of 4.7 prompts and 6.5
minutes
Without
Video (A)
With Video
(B)
Total
Task Completed (out of 35)
8
35
Prompts reqd for completion
9.8
5.2
9.01
4.59
Avg completion time (min)
Sequence -
AB (17 total)
1
17
Prompts reqd for completion
11
5.9
Avg completion time (min)
8.2
8.6
Task Completed - AB
Sequence -
BA (18 total)
Task completed – BA
7
18
Prompts reqd for completion
6
4.7
10.8
6.5
Avg completion time (min)
Qualitative Results
Other observations:
Round-two subjects were incredulous
that round-one subjects didn’t
understand the application.
Impact of video not permanent for most
subjects. Many wanted to see the fullcontext video each time, even after
seeing it before.
Full-context video appears to increase
motivation, as well as performance.
Photo: Indrani Medhi
Those who saw full-context video were
interested in providing feedback on the
specifics of the UI.
Text-Free UI
Design principles:
– Pen or touch interface
– Liberal use of imagery
• No text (but numbers OK)
• Semi-abstracted cartoons
– Aggressive use of mouseover functionality
– Voice feedback
– Consistent help icon
– Full-context video
Full-context video playing at start
of application
Outline: Text-Free User Interfaces
Introduction
Initial ethnography and iterative rapid prototyping
Initial evaluation
Optimal audio-visual representation
Full-context video
Methodology
Discussion
Methodology
Initial Ethnography
Open-ended interviews
– Various combinations: focus
groups and individuals
– Topic: everything and anything
– Follow leads around hardships
Structured interviews
– Focus on themes from openended interviews
– Prepare questions
– Be willing to deviate
Photo: Indrani Medhi
An open-ended interview in a subject’s home
Goal: Information, intuition, rapport
Time and attention critical!
Methodology
Iterative Prototyping
Build initial prototype, based on
intuition from interviews.
Iterate prototype and test.
–
–
Maintain informality
Work with small groups
•
–
–
Prompt whenever necessary
Record every comment
•
Photo: Indrani Medhi
–
–
2-4 people
Video-record, if possible (but, high procedural
overhead)
Listen to every comment
Modify prototype
Testing a prototype
Refine until steady state.
Methodology
Participative Design
Modify standard methods for
non-literate partners.
Encourage storytelling.
Photo: Indrani Medhi
Use aids that can be
physically manipulated.
Eliciting descriptions of medical symptoms
with physical design aids
Be prepared for long,
counseling-like sessions.
Subject enacting medical symptoms
Methodology
Formal Evaluation
Base on techniques from courses,
textbooks
–
Plan according to standard practices
Avoid direct counterfactuals
–
–
–
Place task in a story
Use friend or neighbor as
protagonist, but not subject
Chavan’s “Bollywood method”
Be flexible on…
–
–
–
Location (avoid labs)
Formal instructions
Etc.
Video: Archana Prasad
Scene from a full-context video:
hypothetical user and NGO worker
Outline: Text-Free User Interfaces
Introduction
Initial ethnography and iterative rapid prototyping
Initial evaluation
Optimal audio-visual representation
Full-context video
Methodology
Discussion
Related Work
Interfaces for non-literate users
– Huenerfauth [2002]
– Chand [2002]
– Parikh et al. [2003]
Visual representations
– Modley et al. [1976]
– Horton [1996]
– Tversky et al. [2002]
Video tutorials
– DeBloois et al. [1984]
– Rickman et al. [1986]
– Bransford et al. [1992]
Much more related work in these
categories (see papers)
Text-Free UI novelty:
– Focus on non-literate users
– Rigorous evaluation
– Full-context video
Publications
Medhi, I., Pitti B. and Toyama K. Text-Free UI for Employment Search. Asian Applied Computing
Conference (AACC2005). Nepal, December 2005.
Medhi, I., Sagar A., and Toyama K. Text-Free User Interfaces for Illiterate and Semi-Literate Users.
International Conference on Information and Communication Technologies and Development
(ICTD2006). Berkeley, USA, May 2006.
Medhi, I., Prasad, A. and Toyama K. Optimal Audio-Visual Representations for Illiterate Users.
International World Wide Web Conference (WWW2007). Banff, Canada, May 2007
Medhi, I. and Kuriyan R. Text-Free UI: Prospects for Social Inclusion. International Conference on
Social Implications of Computers in Developing Countries (IFIP 2007). Brazil, May 2007
Medhi, I. and K. Toyama. CHI Workshop position paper on User-Centered Design and International
Development, San Jose, USA, April 2007
Medhi, I., K. Toyama. Full-Context Videos for First-Time, Illiterate Users. alt.chi forum at ACM CHI
'07, San Jose, USA, April 2007
Medhi, I. User-Centered Design for Development. ACM Interactions. COLUMN: Forum: under
development Volume 14 , Issue 4 (2007), pp. 12 – 14.
Medhi, I., C. R. Patabandhige, K. Toyama. Cartoon Generation for Text-Free User Interfaces. 1st
International Workshop on Computer Vision Applications for Developing Regions in Conjunction
with Eleventh IEEE International Conference on Computer Vision (ICCV 2007 Workshop), Rio de
Janeiro, Brazil, October 2007
Medhi, I., Toyama, K. Full-context video for. International Conference on Information and
Communication Technologies and Development (ICTD2007). Bangalore, India, December 2007.
Text-Free UI History
Text-Free
healthcare
information
Practical
Systems
Online site
UI
Research
(babajob.com)
Initial
principles
Optimal AV
representation
UI
authoring
tool
Text-Free UI
for mobiles
Full-context
video
Semiautomated
cartooning
Other
Research
Key:
Completed
Ongoing
Future
Psychology
of non-literate
users
Summary: Text-Free User Interfaces
Introduction
Initial ethnography and iterative rapid prototyping
Initial evaluation
Optimal audio-visual representation
Full-context video
Methodology
Discussion
Photo: Indrani Medhi
Thank you!
http://research.microsoft.com/research/tem
{indranim,kentoy}@microsoft.com