Multimodality: an
introduction
New ways of reading, new ways of writing
‘literacy teaching and learning needs to change
because the world is changing’
Cope and Kalantzis. 2000: 41
Multimodality
Multimodality involves the complex
interweaving of word, image, gesture and
movement, and sound, including speech.
These can be combined in different ways and
presented through a range of media.
Bearne, E. And Wolstencroft, H. 2007
Multimodality
Mode – signs: sound, graphic material, print
Media – the manner of dissemination
A traffic sign as the medium
of communication and the red
border and the image inside
it as the mode.
Kress, G.1997
Medium of communication
• The computer: internet information and
software presentations
• Paper-based texts: picture books, magazines,
novels, information books
• Sound and visual media: radio, television,
videos, CDs and DVDs
Modes of communication
• Writing or print, including typographical
elements of font type, size and shape
• Images: moving and still, diagrammatic or
representational
• Sound: spoken words and music
• Gesture and movement
• Main alternative
1. Modalities of perception and production
a. Visual - gestures
b. Auditory - sounds, speech
c. Tactile - touch
d. Olfactory - smell (molecules)
e. Gustatory - taste (molecules)
Sub modalities: color, shape & movement
Other alternatives
2. Medium = physical carrier Light waves, sound
wave, molecules, electromagnetic, waves etc.
3. Type of representation = index, icon, symbol
4. Communication aids writing, radio, TV etc.
Multimodal Communication -is coactivation,
sharing and coconstruction of information
simultaneously and sequentially through several
modes of perception & production. Multimodal
communication often becomes also:
• Multimedial (always)
• Multirepesentational
• Using several aids of communication
WHY STUDY MULTIMODAL COMMUNICATION?
1. Interesting in itself.
2. What we are built for and are used to. - very
much is still unknown - need for multimodal
communication corpora - provides a background
for what can be simulated or made use of in
technical systems
3. Source for design of computer supported
applications - gives redundancy, robustness,
synergy - more natural and efficient HMI - more
efficient and enjoyable Human-Human
Communication
Examples: New interactive search machines
(internet & local DB) VR applications (games,
education, trade) Other educational support
(Tutoring etc.) Public information , Commercial
systems
• Multimodality needs to be combined with
multiculturality and multifunctionality.
Why multimodality matters
• The future of reading and writing are closely interwoven
with the future of digital technologies.
• Children already know much about multimodal texts
from their home experiences. As teachers it is our
responsibility to build on these experiences and the
children’s knowledge of multimodality in the classroom.
This means we need to recognise the relationships
between different modes: text and image, sound and
gesture and use this in our teaching.
• Supporting children with reading, analysing and using
modes will enable them to develop literacy skills for
today and the future.