Interactive sonification of geographical maps:
a behavioural study with blind subjects
Marta Olivetti Belardinelli 1,2, Franco Delogu 1,2,
Massimiliano Palmiero 1, Stefano Federici 1,3, H. Zhao4,
Catherine Plaisant 4
1 Department
of Psychology, University of Rome “La Sapienza”, Italy
E.CO.N.A. Interuniversity Center for Research on Cognitive Processing in Natural and Artificial Systems
3 Department of Human and Educational Science, University of Perugia, Italy
4 Human Computer Interaction Laboratory, University of Maryland, USA
2
ESCOP 2007 - Marseille
1
Blindness and spatial cognition:
an open debate
DOES EARLY VISUAL EXPERIENCE DEFINETELY AFFECT SPATIAL PROCESSING?
Controversial Results:
Visual experience is necessary
to the acquisition of spatial information
Spatial representations generated
by different sensory modalities
are equivalent to visuo-spatial representation
(Lahav & Mioduser, 2003)
(Bryant, 1992)
Blind people have difficulties
in forming mental representations of space
Blind people acquire spatial information
using tactile, auditory, kinaesthetic information
(Vecchi, 1998)
(Alfonso et al. , 2005; Zhao et al., 2004)
ESCOP 2007 - Marseille
2
Non- visual display solutions
Tactile graphics
- Cumbersome to produce
- Limited in the amount of information
- Psychophysics limitations
Screen Readers
- Strictly sequential
- Reading single values
can overcome the WM capacity
- lack of spatial reference
Audio-Based Systems: can these difficulties be bypassed by
SONIFICATION?
ESCOP 2007 - Marseille
3
Sonification of spatial information
Sonification is the use of nonspeech audio to convey
information (Kramer et al. 1997)
• Displays several parameters simultaneously
(loudness, pitch, timbre, rhythm)
• Easy use of analogies: rising volume easy interpreted as something
increasing
• Sonificated information may result in augmented tactile maps
(Parente, P., and Bishop, G., 2003)
• Easy to transmit geographical and environmental data (Zhao et al., 2004)
AIM
Is it possible to effectively transmit
the features of large-scale geopolitical maps
using sonification?
ESCOP 2007 - Marseille
4
APPARATUS
iSONIC
(developed by the Human Computer Interaction Laboratory
of the University of Maryland
for facilitating blind users to acquire geographical information)
E.G.
Unemployment rates
(here in three different colors:
High
Medinum
Low
ESCOP 2007 - Marseille
5
APPARATUS - Sounds
NON VERBAL SOUNDS
VIOLIN PITCH : LEVEL OF EMPLOYMENT
value
DURING
EXPLORATION
HIGH
MEDIUM
LOW
STATE BORDER: CLICK
END OF HORIZONTAL SWEEP = BELL
BACKGROUND / SEA = GUITAR FRET
HORIZONTAL POSITION = STEREO PANNING
VERBAL SOUNDS
ON SUBJECTS DEMANDS BY
PRESSING THE SPACE-BAR
STATE NAMES
ESCOP 2007 - Marseille
6
APPARATUS - Navigation Methods
STEREOPHONIC HEADPHONE
Stereo-panning effect
KEYBOARD
TOUCH-PAD
ESCOP 2007 - Marseille
7
APPARATUS - Keyboard Interface
DISCRETE STEPS
SPACE BAR: DETAILS ON DEMAND (STATE NAMES)
GIST: 0 KEY
Automatic spatial sweep
ARROW KEYS
Relative navigation
UP
LEFT
RIGHT
NUMBER-PAD FROM 1 TO 9
Absolute navigation
7
8
9
4
5
6
1
2
3
DOWN
ESCOP 2007 - Marseille
8
APPARATUS - Touchpad Interface
CONTINUOUS EXPLORATION
Integrated audio-tactile exploration
SPACE BAR: DETAILS ON DEMAND (STATE NAMES)
GIST: 0 KEY
Automatic spatial sweep
FINGER
ESCOP 2007 - Marseille
9
THE EXPERIMENT - Hypothesis
iSONIC effectively allows users
to correcly recognize the geographical maps
CONGENITALLY ACQUIRED BLIND
BLIND SUBJECTS
SUBJECTS
BLIND
FOLDED
SUBJECTS
As spatial representations are not necessarily
linked to the visual modality, the three groups
do not differ on the tactile recognition of the
maps
Blind subjects use different strategies
with respect to sighted people,
using more specifically sound information
TOUCH-PAD
KEYBOARD
The two interfaces do not show any
differences on discriminating the target from
the distractors
The two methods of exploration allow users to
acquire the same information by means of
different strategies
ESCOP 2007 - Marseille
10
THE EXPERIMENT - Design
BETWEEN SUBJECTS DESIGN 2x3
THE INFLUENCE OF:
TOUCH-PAD
KEYBOARD
TYPE OF INTERFACE
CONGENITALLY BLIND
SUBJECTS
ACQUIRED BLIND
SUBJECTS
BLIND FOLDED
SUBJECTS
VISUAL IMPAIRMENT
ON:
MAP RECOGNITION
X
SPATIAL REPRESENTATION (Global shape of the map, Shape and Size of the states, Estimated number of states)
MAP NAVIGATION STRATEGIES (Gist, Step by Step, 3x3 Quadrant)
SELF REPORTS, QUESTIONNAIRE ANSWER (Self-Confidence, Understanding, Satisfaction)
ESCOP 2007 - Marseille
11
THE EXPERIMENT - Subjects
35 subjects
16 females and 19 males, participated in the study
[mean age = 32,46; SD= 6,73]
with no auditory sensory deficits.
15
blindfolded sighted
10
acquired blind
10
congenitally blind
(8 femals and 7 males)
(4 females and 6 males)
(4 females and 6 males)
ESCOP 2007 - Marseille
12
THE EXPERIMENT – Materials/1
MAPS
4 iSONIC Software Maps
(TARGETS)
Violin Pitch Level of Unemployment
a
HIGH
MEDIUM
LOW
b
16 Tactile Paper Maps
(TARGETS PLUS DISTRACTORS)
Textures type of Unemployment
DOTTED
HERRINGBONE
STRIPED
c
d
PLUS BOUNDARIES (BORDER LINE
WITHOUT TEXTURES) OF IDAHO MAP
ESCOP 2007 - Marseille
13
THE EXPERIMENT – Materials/2
4 QUESTIONNAIRES
SELF REPORT LEVEL OF ACCURACY OF USA 9 QUADRANTS
1
For the first three tasks
EXAMPLE: FIRST QUADRANT
a) high; b) Medium; c) Low; d) Background
X
7
4
8
5
9
6
1
2
3
SELF-REPORT LEVEL OF ACCURACY OF:
2
3
SPACE ORIENTATION, GEOMETRICAL SHAPE OF IDAHO MAP, NUMBER OF IDAHO REGIONS
(The regions are the counties of the Idaho state)
SELF EFFICACY/CONFIDENCE SCALE IN RECOGNITION OF TACTILE MAPS
1
2
3
4
5
6
X7
8
9
10
SELF-REPORT EVALUATION OF SUBJECTS UNDERSTANDING AND SATISFACTION
4
EXAMPLE: INTERFACE WAS EASY TO USE
STRONGLY
DISAGREE
1
2
3
4
ESCOP 2007 - Marseille
5
X6
STRONGLY
AGREE
14
Pre-Experimental phase
THE EXPERIMENT – Procedure
Psychoacustics Trials:
KB
TS
Training
Exploration
Experimental phase
Sound Localization and Pitch Discrimination
(3 minutes)
Four Tasks
Questionnaires
SELF REPORT LEVEL OF ACCURACY OF: Usa 9 Quadrants for Task 1 – 2 - 3
and space orientation, geometrical shape and number of regions forTask 4
First task
Tactile Recognition Tasks
Four Tasks
Final Questionnaire
X
SELF-REPORT EVALUATION OF
SUBJECTS UNDERSTANDING AND SATISFACTION
Post-Experimental Phase
ESCOP 2007 - Marseille
15
THE EXPERIMENT – Results/1
Tactile Map Recognition
Post-hoc analysis shows that Mean recognition confidence was significantly higher for target tactile maps
than for distractors in all the four tasks (Post-Hoc LSD: p<0.05)
MAP - TASK 1
Current Effect: F(3, 102)=19,338, p=,00000
MAP - TASK 2
Current Effect: F(3, 102)=14,088, p=,00000
9
6
8
7
6
5
5
4
4
3
3
1
2
task1a
task1b
task1c
task1d
MAP
MAP - TASK 3
Current Effect: F(3, 102)=12,408, p=,00000
9
6
task2a
task2b
task2c
task2d
MAP
MAP - TASK 4
Current Effect: F(3, 102)=25,039, p=,00000
9
8
7
6
TARGET
7
1
10
TARGET
8
MEAN RECOGNITION CONFIDENCE
MEAN RECOGNITION CONFINDENCE
2
TARGET
7
TARGET
8
9
5
5
4
4
3
2
task3a
task3b
task3c
MAP
task3d
MEAN RECOGNITION CONFIDENCE
MEAN RECOGNITION CONFIDENCE
3
2
1
task4a
ESCOP 2007 - Marseille
task4b
task4c
task4d
MAP
16
THE EXPERIMENT – Results/2
Tactile Map Recognition – EFFECT OF BLINDNESS
In all of four tasks, congenitally blind, acquired blind and
sighted subjects did not differ in discriminating the target from
distractors:
GROUP - TASK 1
Current Effect: F(2, 32)=,36125, p=,69961
10,5
Task 1 [F(2, 32)=0,36, p > .05]
9,5
8,5
Task 2 [F(2, 32)=1,11, p > .05]
7,5
6,5
Task 3 [F(2, 32)=1,77, p > .05]
5,5
MEAN RECOGNITION CONFIDENCE
Task 4 [F(2, 32)=2,70, p > .05]
4,5
SIGHTED
ESCOP 2007 - Marseille
ACQUIRED BLIND
CONGENITALLY BLIND
GROUP
17
THE EXPERIMENT – Results/3
Tactile Map Recognition – EFFECT OF INTERFACE
In all of the four tasks, the use of Key-Board and Touch-Pad
during the exploration did not lead to significant differences in
discriminating the target from distractors :
INTERFACE - TASK 1
Current Effect: F(1, 29)=,46883, p=,49896
10
Task 1 [F(1, 29)=,46, p > .05]
9
Task 2 [F(1, 29)=,21, p > .05]
8
7
Task 3 [F(1, 29)=,24, p > .05]
6
MEAN RECOGNITION CONFIDENCE
Task 4 [F(1, 29)=,30, p > .05]
5
TOUCH-PAD
ESCOP 2007 - Marseille
KEYBOARD
INTERFACE
18
THE EXPERIMENT – Results/4
Questionnaire - SELF REPORT LEVEL OF ACCURACY OF USA 9 QUADRANTS
Which level of the value (low, medium, high unemployment, or sea)
is more represented into each quadrant?
Participants showed a better accuracy in the first task, the easiest,
than in the second and third task, more complex (Post-Hoc LSD: p<.05).
[F(2, 16)=8,3451, p=,00329].
0,9
Accuracy (percentage)
0,8
0,7
0,6
0,5
0,4
0,3
Task 1
Task 2
Task 3
Task
ESCOP 2007 - Marseille
19
THE EXPERIMENT – Results/5
Map Navigation Strategies
Number of Actions during the exploration
Touch-Pad users moved more rapidly from region to region than Keyboard users.
600
545
500
400
300
200
96
100
0
Touch-Pad
ESCOP 2007 - Marseille
Keyboard
20
THE EXPERIMENT – Results/7
Final Questionnaire – STEREO PANNING EFFECT
How useful for the orientation
was the stereo panning?
Both groups of blind subjects more than sighted subjects reported to be helped by the stereo
panning (Post-Hoc LSD: p<0.05)
6,0
5,5
5,0
4,5
4,0
3,5
3,0
2,5
2,0
1,5
1,0
0,5
0,0
Sighted
Acquired Blind
Congenitally Blind
Group
ESCOP 2007 - Marseille
21
THE EXPERIMENT – Results/8
Final Questionnaire – SUBJECTIVE EVALUATION OF INTERFACE
Both groups of blind subjects more than sighted ones rated the interface
to be easy to use.
How easy to use was the
interface?
6,5
6,0
5,5
5,0
4,5
4,0
3,5
3,0
2,5
sighted blindfolded
acquired blind
congenitally blind
Group
ESCOP 2007 - Marseille
22
GENERAL DISCUSSION
1.
The software allows correct recognition of geographical maps.
2.
Sonification allows to transmit complex spatial information like large-scale
geopolitical maps, even though difficulties in recognizing specific details of the maps were
found.
3.
No substantial differences between blind and sighted users, nor between early and
late blind subjects were found, plausibly indicating that multi-sensory spatial representations are
possible. The a-modal theory of spatial representation should be considered
(Avraamides, Loomis, Klatzky, & Golledge 2004; Bryant, 1992).
4.
Blind subjects, more than sighted subjects, reported to be helped by the stereopanning effect.
5.
With respect to sighted subjects blind subjects use different strategies based on a
greater attention and experience to sound information.
6.
No significant differences between interfaces were found.
ESCOP 2007 - Marseille
23
CONCLUSION
1.
Considering the limitations of tactile maps sonification
used to present geo-political information.
can be effectively
2.
These findings have relevant empirical implications in order to allow the
blind community an easy access to geopolitical data and more in
general to complex spatial information.
3.
Our data support the convincement that visual
4.
Further studies are necessary to examine the effectiveness of sonification in other
application contexts, and its integration in multi-modal contexts. A dynamic
combination of non-speech sounds and haptics could represent a new frontier in
the implementation of effective non-visual displays. A multimodal approach
allows users to access spatial information in more flexible ways.
experience is not
necessary for an efficient spatial cognition.
ESCOP 2007 - Marseille
24
THANKS FOR YOUR ATTENTION
REFERENCES:
•
•
•
•
•
•
Avraamides, M., Loomis, J. Klatzky, R. L., & Golledge, R. G. (2004). Functional equivalence of spatial representations
derived from vision and language: Evidence from allocentric judgments. Journal of Experimental Psychology: Human Learning,
Memory & Cognition, 30, 801-814
Bryant, D. J. (1992) A Spatial Representation System in Humans. Psycholoquy 3(16)
Jacobson, R.D. (1998) Navigating maps with little or no sight: A novel audio-tactile approach. Proceedings of Content
Visualization and Intermedia Representations. August 15, University of Montreal, Montreal.
Parente P. and Bishop G. (2003). “BATS: the blind audio tactile mapping system”, in Proc. ACM Southeast Regional Conf. 2003
Tinti C., Adenzato M. & Tamietto M., Cornoldi C. (2006). Visual experience is not necessary for efficient survey
spatial cognition: Evidence from blindness. The quarterly journal of experimental psychology, 59 (7), 1306–1328
Zhao H., Plaisant C., Shneiderman B., and Duraiswami R. (2004). Sonification of geo-referenced data for auditory
information seeking: design principle and pilot study, Proc. ICAD 2004.
ESCOP 2007 - Marseille
25