Designing Multimodal Discourses
Bai Hong’ai
College of Foreign Languages, YBU
monomodality :
Language as the central and only full
means for representation and communication
multimodality
means of expression
(material and other
technologies)
When humans converse with one another, we
utilize a wide array of media to interact,
including spoken language, gestures, and
drawings.
We exploit multiple human sensory systems
or modes of communication, including vision,
audition, and taction.
the manner of
expression
Some media and modes of communication
are more efficient or effective than others
for certain tasks, users, or contexts
e.g. the use of speech ---
control devices in hand and eyesbusy contexts,
the use of maps ---
convey terrain and
cartographic
information
Communication is defined as a process in
which a semiotic product or event is both
articulated or produced and interpreted or
used, and a semiotic product or event may
involve multiple modes and media.
The design process of this multimodality
semiotic product or event, is to be
explored in the present paper, i.e. the
generation of multimodal discourses
I. What is multimodal discourse
Multimodal discourse is defined as “the use of
several semiotic modes in the design of a
semiotic product or event, together with
the particular way in which these modes
are combined” – they may for instance
reinforce each other (‘say the same thing in
different ways’), fulfil complementary roles, or
be hierarchically ordered, as in action films,
where action is dominant, with music adding a
touch of emotive colour and sync sound a
touch of realistic ‘presence’ (Kress G. & Van
Leeuwen, T, 1996, 2001; Batesman et al.,
2002)
This suggests that we need to expand our
conception of mode to embrace
multimodal discourse analysis (hereafter
MDA).
MDA draws upon recent work in the field
of social semiotics in an attempt to theorise
how discourses are generated in a variety
of modes through a variety of media, and
to investigate empirically the design
decisions employed and how these have
been distributed across the presentation
modes available.
II. How to design a multimodal discourse
STEP 1: The Process of Designing a multimodal discourse
Content planning – determining what information should
be presented;
Technique selection – selecting appropriate media or
modalities which will be used for realization, and
coordinating the selections for specific information
content;
Presentation design – determining how media or
modalities will be used to communicate their
selected information content;
Coordination – composing, organizing, resolving
conflicts and maintaining presentation consistency.
Figure 1: Feedback and Multiple Knowledge Sources involved in the Processes
Content
Planning
Presentation
Design
Technique
Selection
Coordination
Data
Task
Characteristics Characteristics
Context
Discourse
State
STEP 2: Determine what and how media or modes to be used
Myers (1999: 91)
Different media convey different information
Different media go with different practices
Spoken and written words can limit, link
and extend images
Words (including typography) define voice
and audience
STEP 3: Establishing the right modality: Naturalistic modality,
Abstract modality and Sensory modality
Naturalistic modality: produced with sound, images,
audio-visual texts or the combination of all these.
Abstract modality: with regard to images
Sensory modality: based on the sensory pleasure
or displeasure represented by an audio-visual text
‘larger-than-life’ or “more-than-real” images
or sounds
III. Advertisements:
typical multimodal discourses
Medium with sound only--- Radio
MAN: I… I… I love you.
WOMAN: Awwww.
MAN: No, I mean … I … I mean. I don’t … just …
WOMAN: What (GIGGLES), go ahead …
MAN: I just … don’t think I’d be anything without you …
WOMAN: Aw … that’s so nice …
MAN: I, uh, made these for you …
(Baking pan and foil sounds.)
WOMAN: Aw … you didn’t have to do this …
MAN: They’re just brownies.
WOMAN: Ooh, and they’re still warm too.
MAN: I was gonna get you something good.
WOMAN: Aww, don’t be silly, could I have one?
MAN: (Laughs.) Well, yeah sure … go ahead …
WOMAN: (Mouth full.) You know … ?
MAN: What … you don’t like brownies?
WOMAN: No … no … (Slight gag.) miggkkk …
MAN: I’m sorry. I can’t understand you.
WOMAN: Neet … dmiggkk.
MAN: You need Mick?
WOMAN: Miccliggkk!
MAN: A guy named Mick! I can’t believe this …
WOMAN: Pleeeease miiiggggggllkk
MAN: Yeah, yeah, okay you don’t have to scream his name in ecstasy. I
get the message. All right I got it, I’m gone.
WOMAN: No, umbhum, no, miiiiiiggglllkk!
MAN: Look, I gotta go, I’m sorry I’m gonna need my CDs back … and
WOMAN: No, no, miiiggllk.
MAN: … and I’m sorry for this whole thing …
WOMAN: … No, miiigggllkk …
MAN: No, it’s Scott remember ?
WOMAN: No, miigggkkkffff …
ANNOUNCER: True love means never having to say, “Got Milk?”
Medium with image, sound, colour, words, etc.
--- internet banner ads, TV ads
V. Implications to Language teaching
There is a need to re-conceptualise the term
‘literacy’ as a multimodal process. As language is
beginning to be de-centred and image and other
modes are increasingly foregrounded, the
traditional conceptions of learning and ‘literacy’
as entirely language based are inadequate.
Multimodal discourse with language de-centred
Literacy, as a complex process, is
multimodally mediated, in which the
modes of speech and writing are only
ever one part of the multimodal modes,
just the same as image, colour, sound and
animated movement on screen.
We need to pay attention to the substance of learning
including new learning materials. In recent years,
educationalists have focused upon learning
environments as social and personal settings and
forms of interaction. It is time to take note of the fact
that learning is about something and that “something”
is communicated through a range of media – from
books and photocopies, through audiovisual media to
computer media and in the near future perhaps also
mobile media. Only with the advent of a range of new
media it is perhaps made clear to us that learning and
media were always intimately increasingly intertwined.
Thank you for your attention!