Published in O. Garcìa, N. Flores and M. Spotti (2016)
7 Oxford Handbook of Language and
Society. Oxford University Press. pp. 451-472
Chapter 22
Mu ltimoda l i t y
Elisabetta Adami
Introducing Multimodality
Multimodality is a concept introduced and developed in the last two decades to
account for the different resources used in communication to express meaning. The
term is used both to describe a phenomenon of human communication and to identify
a diversified and growing field of research. As a phenomenon of communication, multimodality defines the combination of different semiotic resources, or modes, in texts and
communicative events, such as still and moving image, speech, writing, layout, gesture,
and/or proxemics. As a field of inquiry, research in multimodality is concerned with
developing theories, analytical tools, and descriptions that approach the study of representation and communication considering modes as an organizing principle.
As a phenomenon of communication, the term is used not only by multimodal analysts, but also, and increasingly so, by works in disciplines concerned with texts and
meaning, such as linguistics and communication studies, all of which, however, tend to
devote their analytical focus on language.
Within the field of “multimodal studies” (O’Halloran and Smith, 2011), the phenomenon of multimodality is approached through different theoretical perspectives (Jewitt,
2009a; O’Halloran, 2011), all hinging on four key assumptions (Jewitt, 2014a), namely
that (1) all communication is multimodal; (2) analyses focused solely or primarily on
language cannot adequately account for meaning; (3) each mode has specific affordances
arising from its materiality and from its social histories, which shape its resources to fulfill given communicative needs; and (4) modes concur together, each with a specialized
role, to meaning-making; hence relations among modes are key to understand every
instance of communication.
Multimodality as a field of research conceives of representation and communication as relying on a multiplicity of modes, all of which have been socially developed as resources to make meaning. Modes such as gesture, sound, image, color, or
layout, for example, are conceived as sets of organized resources that societies have
452
Elisabetta Adami
developed—each to a greater or lesser level of articulation in different social groups—to
make meaning and to express and shape values, ideologies, and power relations. When
in combination with speech and/or writing, they are not a mere accompaniment of, or
support to verbal language, as labels such as para-/extra-linguistic or non-verbal might
suggest; rather, each concur with a specific functional load to the meaning made by the
overall text—and as such they deserve attention.
All communication is, and has always been, multimodal (Kress and van Leeuwen,
1996). Be it either face to face or distant, synchronous or asynchronous, every instance
of communication relies on more than one mode to make meaning. This might sound
today like a commonplace; yet historically, the dominant role attributed to verbal
language, and the mode of writing especially, has overshadowed the multiplicity of
resources shaped socially to communicate. This has meant not only that societies have
developed the resources of speech and writing at a particularly high level of articulation,
but also that research and education have focused their almost exclusive attention to the
development of descriptions and the teaching of prescriptions and conventions for the
use of language. As a result, the investigation of other modes has been restricted to specialized fields, such as musicology, the arts, and so on.
In recent years, the social impact of digital technologies for text production, among
other factors, has made more visible the fact that texts are multimodal and hence that
language alone cannot suffice to explain meaning made through them. Digital technologies have reduced costs for the production of printed images and the use of color.
Their (market-led) widespread use has made available—to an unprecedented number of
sign-makers—forms of text production that afford modes other than speech and writing. Online environments have provided sign-makers with platforms and easy-to-use
interfaces for publishing their multimodal texts and distributing them to diversified
audiences, thus making the phenomenon of multimodality visible to an unprecedented
extent.
The digital texts we daily engage with make meaning through the combined use of
color, writing, sound, images, and layout, at least. It is not only the case of texts that
we encounter on the web, but also of “texts” that we interact with daily, to fulfill ordinary tasks in our offline environments, such as the interfaces displayed on the screens
of ATM machines or those for purchasing a train ticket, for instance. This holds also for
the texts that we produce; everyday communication in digital environments faces signmakers with a wide range of modal options. The multimodal character of digital texts is
also redefining the use of the resources of language (van Leeuwen, 2008); writing itself is
changing its functions, as lexis integrated in visual ensembles/syntagms (van Leeuwen,
2004), or as something to be acted upon rather than read, as in the case of URLs used
as hyperlinks (Adami, 2015); writing is also increasingly developing resources for
meaning-making, like those of font (van Leeuwen, 2005b, 2006), which are generally
disregarded in linguistic studies. While speech is changing its functional load in the
online homologue of face-to-face interaction (i.e., video-chats; for the phenomenon of
“mode-switching” in video-chats, see Sindoni, 2013), the mode of image is being used
for new interactive functions, as in Facebook comments, for example. Such a changed
Multimodality
453
semiotic landscape contributes in essential ways to the visibility of multimodality as a
phenomenon of contemporary communication, and to its usefulness as a notion that
can account for contemporary meaning-making.
Over the last two decades, disciplines concerned with text, discourse, and meaning
have increasingly devoted attention to “nonverbal” resources. Yet the point of reference
and focus of analysis has traditionally hinged on speech, with other modes considered
as playing an accessory function, and under-analyzed as to their specific resources and
potentials in meaning-making:
While modes of communication other than language are, to varying degrees,being
attended to in social linguistic work, its central units of analysis are usually linguistic units (e.g. “intonation unit”) or units defined in linguistic terms (e.g. a “turn” is
defined in terms of “who is speaking”). (Bezemer and Jewitt, 2010:183).
The advent of digital technologies has contributed to changes in the perception of
what constitutes data in many text-based disciplines. Digital technologies provide
analysts with multimodal means of recording, coding, and transcribing data, such as
videos and video annotation systems. When analyzing a video-recorded rather than a
tape-recorded face-to-face interaction, the multimodal character of the communicative
event becomes more immediately manifest, and what could be regarded as “context” or
“contextual information” in earlier tape recordings (something that the researcher could
neither see nor handle from tape-recorded data) is now visible as meanings expressed by
participants through gestures, movement, and face expressions, or through 3D objects.
In this regard, Goodwin’s (2001) work has opened a tradition of studies in conversation
analysis that are now approaching multimodality as a means to account in detail for
meaning made through actions and their relations to speech (for a recent output of the
related body of work, see Streeck et al., 2011).
Studies in corpus linguistics, which have developed tools and have compiled,
tagged, and parsed corpora of (predominantly) written and (to a lesser extent) spoken
language—yet transcribed in written form—are now increasingly advocating the need
to compile multimodal corpora (Adolphs and Carter, 2007; Allwood, 2008; Haugh,
2009). However, these tend to assume a central role of speech and writing, with other
resources functioning as an accompaniment to language. Also studies in computermediated communication acknowledge the multimodal nature of digital environments,
like Herring (2010: 244), who argues that “the interpretation of visual content can benefit from methods drawn from iconography and semiotics.” Yet, in her review and development of methodologies for the analysis of “web content” (2010: 233), the multimodal
nature of web texts is referred to only in terms of the presence of images, while the main
reference point and concern are still on language and language-based interaction, as if
language and hyperlinks were the defining resource for the understanding of web-based
phenomena like the blogosphere (see also, more recently, Herring, 2013, in which the
multimodality of web 2.0 texts is addressed more explicitly, and hypotheses are made
regarding whether it should or could be included/integrated as a further level of analysis
454
Elisabetta Adami
within computer-mediated discourse analysis, while nevertheless maintaining that
“text remains the predominant channel of communication among web users”; 2013: 9).
In sum, studies in linguistics and communication have increasingly acknowledged the
multimodal nature of texts, yet, with the notable exception of Goodwin’s tradition, their
main focus often remains on spoken and written language. In contrast, studies in multimodality assume that any analysis today can no longer rely only or mainly on language,
if it aims at interpreting the meanings of a text or communicative event, rather than
merely the use of (selected aspects of) speech or writing within them.
Given the increasingly manifest multimodal character of communication vis-àvis the attention paid historically to developing analytical labels and tools mainly to
describe language, multimodality as a field of research attends different tasks. It aims to
investigate the meaning potentials of each mode (including speech and writing, differently conceived of, through a multimodal lens), and to provide an account of how each
mode has been shaped historically in different cultures and societies to fulfill particular
tasks. It also aims to find common labels that can describe meaning made in all modes,
to be able to treat all modal resources in a unifying and coherent account. Finally, it aims
to describe and explain meaning made through the relation among modes in multimodal ensembles, given that the meaning expressed by each modal resource influences
the other in a text.
The next section traces the origins and developments of multimodal analysis, while
briefly reviewing different approaches and mentioning current work relevant to disciplines concerned with texts and language. Then, the subsequent section discusses key
notions of a social semiotic approach to multimodal analysis, as a means of both looking at social phenomena through representation and communication, and looking at
representation and communication as socially shaped phenomena. The final section
mentions the potentials and limitations of the approach, opening to future research in
the field.
Historical Perspectives
on Multimodal Analysis
Multimodality finds its origins in the adaptation of Halliday’s framework to modes
other than speech and writing. Kress and van Leeuwen’s (1996) seminal work Reading
Images: The Grammar of Visual Design adapted Halliday’s (1978) ideational, interpersonal, and textual meta-functions for the description of meaning made by images and
their combined use with writing. They defined and described the resources through
which visual texts can (a) represent something about the world, (b) represent something about their authors and addressees, and (c) shape cohesion, information structure, and different truth-values toward what is represented. Earlier, O’Toole (1994) had
applied the three meta-functions to the analysis of visual art, while, in later works, the
Multimodality
455
three metafunctions have been mapped onto the resources of speech, sound, and music
(van Leeuwen, 1999), gesture and movement (Martinec, 2000), color (Kress and van
Leeuwen, 2002), the moving image, or kineikonic mode (Burn, 2013; Burn and Parker,
2003), and layout (Kress, 2010). In more recent work, Bezemer and Kress (2014) try to
map the three meta-functions onto the meaning potential of signs made through touch,
as to determine the defining criterion of touch as a mode vis-à-vis touch as a sense.
Since the first years of the 2000s, the notions of mode and multimodality have
become a growing focus of interest. In Multimodal Discourse: The Modes and Media of
Contemporary Communication (2001), Kress and van Leeuwen draw the notion of mode
from Halliday’s (1978) distinction between speech and writing in language and extend
it to all resources for representation. Always using examples of texts mainly combining
images and writing, but abandoning frames and terminology tightly bound to linguistic
traditions, in Multimodal Discourse, Kress and van Leeuwen
aim to explore the common principles behind multimodal communication. We
move away from the idea that the different modes in multimodal texts have strictly
bounded and framed specialist tasks […]. Instead we move towards a view of multimodality in which common semiotic principles operate in and across different
modes, and in which it is therefore quite possible for music to encode action, or
images to encode emotion. […] we want to create a theory of semiotics appropriate
to contemporary semiotic practice. (Kress and van Leeuwen, 2001: 2)
At that time, Kress and van Leeuwen still referred to a “multimodal theory of communication” (2001: 111). However, in more recent years, the increased interest in multimodality has seen researchers approaching and developing multimodal analysis through
different theoretical perspectives. Two of these stem from Halliday’s theories, one drawing from his social semiotic take (i.e., on his idea that language is a resource shaped to
express and establish social roles and values; see the discussion in the next section), the
other from his systemic functional grammar framework (i.e., on his idea that language
is a network of systems that offer options to perform socially driven functions).
As to the former, van Leeuwen (2005a) and Kress (2010), each with a distinctive
focus, have further elaborated on Hodge and Kress’s (1988) social semiotic theory.
Social semiotics conceives of sign-making as the expression of social processes; through
a fine-grained qualitative analysis of usually small samples of texts, social semiotics is
interested in unveiling ideologies, social values, power roles, and identities as expressed
in texts, together with how individuals actively maintain, reinforce, contest, and challenge them through their sign-making choices.
As to the latter, the works by O’Halloran (2008) and Baldry and Thibault (2006),
among others, apply and develop Halliday’s Systemic-Functional Grammar to multimodal texts, with a special interest and focus on modes as systems for meaning-making
rather—or more—than as the sign-maker’s expressions of social processes. Using
(slightly) wider corpora, systemic functional multimodal discourse analysis—defined
by O’Halloran (2011: 4) as a “grammatical approach” versus social semiotics’ “contextual
456
Elisabetta Adami
approach”—aims at developing frameworks, analytical tools, and descriptions of the
regularities in the functioning of each mode and the relations between modes.
As an example of the different takes of the two approaches, given news reporting as
the object of analysis, social semiotic multimodal analysis could focus on a small sample
of texts reporting on the same news and would be interested in unveiling ideologies and
discourses as differently represented through the combined use of images and writing.
This take could reveal different media outlets’ interests and positioning toward power
and the parties involved in the news event (see, for example, the analysis of news representation of the Palestinian conflict in van Leeuwen and Jaworski, 2002). Instead, a
systemic-functional analysis would be more concerned with mapping regularities in
the functional use of images versus writing in a—usually larger—news data set, thus
investigating their structural relation (e.g., theme or focus) in shaping discursive functions (see, for example, the analysis of the changing functions of images as thumbnails in
news homepages in Knox, 2009).
Not only can these perspectives be seen as complementing each other (O’Halloran,
2011), but also, as the field develops, boundaries between approaches become less clearcut, while different takes arise in-between. So, critical multimodal discourse analysis
(Machin, 2007, 2014; Machin and Mayr, 2012) combines critical discourse analysis (specifically, Fairclough, 1989; Kress, 1985; van Dijk, 1991; Wodak, 1989) and social semiotics for the investigation of naturalized ideologies as expressed through the combined
used of modes (especially in printed texts), whereas geosemiotics (Scollon and Wong
Scollon, 2003) combines Goffman’s interaction order with visual grammar and place
semiotics for the analysis of the interaction between space, signs, and situated action,
and multimodal interactional analysis (Norris, 2004) focuses on how interactants make
meaning through action in different modes in face-to-face encounters. Others introduce new theoretical perspectives, such as Forceville and Urios-Apparisi’s (2009) work,
which uses Lakoff and Johnson’s (1980) cognitive linguistic framework (rather than
Hallidayan functional linguistics) to explore metaphors as represented through images
or through the combined use of image and writing.
At the same time, multimodal analysis is being approached by scholars in other
fields. In 2013, a special issue in the Journal of Pragmatics has been entirely devoted to
“Conversation Analytic Studies of Multimodal Interaction” (Deppermann, 2013), marking a very promising dialogue between Goodwin’s tradition in conversation analysis and
multimodality, while, in a useful interdisciplinary effort, a special issue of Qualitative
Research (Dicks et al., 2011) has discussed the potentials of the combined uses of multimodality and ethnography.
Irrespective of the theoretical approaches, multimodal analysis is increasingly being
applied to the investigation of several domains, with fields of applications spanning
from museum exhibition designs to surgeon training. As to areas strictly related to language, multimodal communication in second-language contexts and learning has been
studied in Royce (2007), Romero and Arévalo (2010), and Pinnow (2011). Translation
studies are devoting growing attention to the challenges that multimodal texts pose
Multimodality
457
to translators, especially, but not exclusively, in audio-visual translation; see, in this
regard, the special issue on “Translating Multimodalities” in the Journal of Specialized
Translation (O’Sullivan and Jeffcote, 2013), the work by Taylor (2004) on subtitling,
and Borodo (2015) on the translation of comics. The multimodality of corporate and
business discourse has been investigated in Maier (2008, 2011), Garzone (2009),
and Campagna and Boggio (2009), among others. An edited volume by Page (2010)
explores the relations between multimodality and narrative. Within the context of education and writing studies, Lemke’s (1998) special issue on multimodality of Linguistics
and Education has initiated an increasingly rich strand of application of multimodality in the field (e.g., Unsworth, 2008), which has been further developed in works on
literacy and communication in the classroom, as in Jewitt (2005, 2006, 2008), on academic literacy (Archer, 2006), and on writing (Archer and Breuer, 2015); more recently,
Bezemer and Kress (2016) explore the changing multimodality of learning practices.
Cross-cultural issues in (mainly non-Western) multimodal genres have been examined in Bowcher (2012), while the relation between genre and multimodality has
been explored in Bateman (2008), Bateman, Delin, and Henschel (2007), and Prior
(2009). Multimodal works on digital texts are particularly numerous; among these,
Lemke (2002) provides a framework for the analysis of hypermodality, Adami (2015)
develops tools for the analysis of web interactivity, and a special issue on multimodality in Text & Talk (Adami et al., 2014) discusses the redefining notion of text in digital
environments.
This brief and necessarily selective review cannot aim to provide a comprehensive account of the increasingly numerous and diversified works in multimodality.
Multimodality is an admittedly fluid field of investigation, and so are its key notions
and working definitions (Jewitt, 2014a). As a concept, it has attracted growing attention
from different disciplines concerned with meaning, text, and communication. As a field
of research, it is achieving the multifaceted shape of a growing and diversified community, gathering scholars from increasingly different backgrounds, adding and intertwining new (inter-)disciplinary perspectives to the originally predominant linguistic take.
These different views, unavoidably, bring an increased complexity to the geopolitics of
the field and highlight the need for shared terms and agreed-upon definitions that can
set the ground for dialogue, debate, and exchange of ideas and findings. Along with a
series of handbooks and edited volumes bringing together different perspectives (Jewitt,
2009b, 2014b; Jewitt et al., 2016; Klug and Stöckl, 2014; Norris and Maier, 2014; Ventola
and Guijarro, 2009), a biannual International Conference on Multimodality (ICOM)
has reached its eighth edition in 2016 (8ICOM, Cape Town, December 7–9); the websites of the past conferences can serve as a further reference to grasp the increasingly
wider spectrum of studies in the field.
The next section examines key concepts and notions of a social semiotic perspective to multimodal analysis, which, in seeing sign-making as inherently social, is particularly concerned with the entexting of social relations and offers a lens for looking at
social phenomena through multimodal representation.
458
Elisabetta Adami
A Social Semiotic View
of Multimodality
Social semiotics has been developed as a theory of multimodal sign-making in the
works of Hodge and Kress (1988), van Leeuwen (2005a), and Kress (2010), who have
extended Halliday’s socially framed view of language to all semiotic resources. Social
semiotics draws on Halliday’s assumption that language is a product of social processes
and that the resources of a language are shaped by the functions that it has developed
to satisfy the needs of people’s lives. Through their everyday acts of sign-making, while
exchanging meanings, speakers express social structure, affirm their social roles, and
transmit their systems of values and knowledge. Grammar, as much as vocabulary, is
a resource, rather than a set of predefined rules, that speakers use creatively by making choices. Through choice, speakers produce variation; variation expresses (affiliation
and conflict with) social structures and roles, along with systems of knowledge and values (i.e., power). Language in Halliday—and all semiotic resources in Hodge and Kress
(1988)—is a social process in two senses. In expressing social values and structures, language reveals them and, at the same time, it constructs them, thus establishing social
relations and systems of knowledge and values every time it is used. Through choice at
all levels (or strata), language expresses larger relations of power existing within society
and constructs power roles in the specific event.
Hodge and Kress follow Halliday in assuming “the primacy of the social dimension in
understanding language structures and processes” (1988: vii), yet they see
the limitation to verbal language […] as a major inconvenience […] Meaning
resides so strongly and pervasively in other systems of meaning, in a multiplicity of
visual, aural, behavioural and other codes, that a concentration on words alone is not
enough. (1988: vii)
Therefore, “no single code can be successfully studied or fully understood in isolation” (1988: vii), and thus social semiotics is conceived as “a theory of all sign systems as
socially constituted, and treated as social practices” (1988: vii–viii).
While “ ‘mainstream semiotics’ emphasizes structures and codes, at the expense
of functions and social uses of semiotic systems” (1988: 1), social semiotics focuses
on “speakers and writers or other participants in semiotic activity as connected and
interacting in a variety of ways in concrete social contexts” (1988: 1). It uses modes as
analytical tools to investigate the ways in which societies have shaped their semiotic
resources, and the social meanings made by sign-makers’ specific use of modes in
multimodal texts.
Rather than describing semiotic modes as though they have intrinsic characteristics and inherent systematicities or “laws,” social semiotics focuses on how people
Multimodality
459
regulate the use of semiotic resources—again, in the context of specific social practices and institutions, and in different ways and to different degrees. (van Leeuwen,
2005b: xi)
Social semiotic multimodal analysis draws on a series of key concepts; the following
subsections examine four of them: mode as socially shaped, the motivated sign, the
meaning potential of semiotic resources, and genre as the entexting of social relations.
Mode as Socially Shaped
Social semiotics’ approach to modes is socially specific. What constitutes a mode
depends on the social group that uses it and the range of meanings that the group
can express through its resources. For wine tasters, wine is a fully articulated mode,
with color, aroma, and taste as its modal resources, which can express all three metafunctions; these resources of wine can represent something about the world (the type of
soil where the vine was grown, the level of maturation of the grapes, or any defects in the
wine-making process, such as oxidation, for example); they can tell something about the
participants (wine preferences in aroma and flavors are associated with identity features,
e.g., adjectives such as “feminine” or “masculine,” “gentle” or “harsh,” in the vocabulary
of wine tasting); they can construct cohesion and vary information structure in their
combined use (if aroma is long but taste is short, wine is considered unbalanced, the
same if aroma is fruity while taste is markedly mineral, for example). One could argue
that color, aroma, and taste in wine are rather indexes, in the sense that their presence is
given a certain meaning by the interpreter, while no sign-maker has intentionally produced them; however, in the wine business, wine-making experts increasingly coordinate all phases of the vine growth, grape harvest, and making of the wine to achieve
the designed color/taste/aroma. Furthermore, given that in mass-production societies, sign-makers increasingly make meaning through selection rather than production
“from scratch” (as when designing the style of their homes through ready-made pieces
of furniture, for example), the meaning potentials of wine are fully in force as signs when
a wine connoisseur selects a given wine for his or her guests. When I am offered a fruity
and flowery glass of wine by a (usually male) wine bartender, I cannot help but interpret his choice as a gendered sign that addresses me as a woman through (stereo)typical
“feminine” aroma/taste preferences for wine. In this case, as with all other modes, larger
relations of power are always at work in individual choices of meaning-making—not
only in the bartender’s choice, but also in my response to his stereotypically gendered
suggestion. If I followed his suggestion, that would reinforce power role distributions
in terms of gender preferences as expressed in the mode of wine tasting. If, instead,
I opted for a particularly dry or markedly mineral (rather than flowery) white wine,
or a tobacco-and-leather smelling (rather than fruity) red wine, these choices would
reveal my interest in disassociating myself (and my identity) with a dominant distribution in gender roles. Also in this second option, although contesting my belonging
460
Elisabetta Adami
to a certain gendered stereotype, I cannot escape the expression of power in my signmaking through wine choices. If my choice conflicts with the bartender’s idea of me
(and my preferences) as a woman, it simultaneously reinforces the authority of “masculine” aroma/taste as the center of reference/power in the social and identity values
expressed through wine tasting as mode (i.e., “although I am a woman, I am a wine
connoisseur because my preferences align with males’ rather than women’s”). In this
sense, a social semiotic analysis of the resources of color, aroma, and taste in wine (as
for the resources of any other mode) can reveal (a) how societies have shaped them to
express power, (b) how individuals position themselves toward that established system
of values, and (c) how systems of values might be changing as affected by—and revealed
through—wider changes in the individuals’ modal choices.
As a modal resource of writing, font has had a wide range of meaning potentials in
typography since the advent of print. Now, with the advent of digital technologies, it
is increasingly widening its meanings for all sign-makers, at a point where it might be
considered a mode (rather than a resource of writing) among increasingly numerous
social groups. Font also has social meaning potentials; font types can shape a text as
addressing children or adults, as designed to look professional or amateur, traditional or
“high-tech,” and so on. In so doing, in each context in which a resource of font is used, it
expresses, affirms, or contests broader power roles within society.
In contemporary representation, sign-makers usually need to combine different
modes in the same text, as when students format a paper including layout, font, writing,
and graphs—or when setting up a blog on their personal and/or professional interests.
Three related consequences motivate a socially situated study of modes and multimodal
relations:
1. Multimodality is increasingly the normal state of communication (Kress and van
Leeuwen, 2001) and, hence, language-based tools for text analysis are less adequate to its description;
2. The potentials of modes and relations between modes are being shaped in new
ways by everyday acts of sign-making, through the increasing number and diversity of so-called user-generated content;
3. Awareness of the potentials of modes and their intertwined use is increasingly
needed for meaning-makers to interpret and unveil social meanings in texts, and
for sign-makers to be effective “rhetors” (Kress, 2010) when producing their texts,
that is, to be able to assess which resources in each mode are most apt to express
their meaning to their addressed audience in each communicative situation.
The Motivated Sign
Social semiotics analyzes texts as expressions of the sign-maker’s interest, conceived as
the momentary focusing of his or her social history and position (Kress, 2010). Signs are
Multimodality
461
socially shaped resources that are newly made every time they are used. In this regard,
social semiotics’ take on sign-making is influenced by Kress’s (1993, 1997) concept of the
motivated sign. Against a Saussurean’s view of signs as an arbitrary association between
a form (signifier) and a meaning (signified), Kress’s motivated sign stresses the motivation that can be traced in the relation between a sign-maker’s selection of a given form
and its expression of a given meaning.
In the Saussurean structuralist tradition (rather or more than in Saussure’s original
elaboration), the positing of an arbitrary relation between signifier and signified has
meant a focus on language as a system (langue) and its driving forces, thus disregarding
how individuals and social groups shape signs (through individual acts of parole)—and
hence which systems of values and power drive their choices in doing so. In contrast, in
tracing the motivation between signifier and signified, multimodal analysts can achieve
insights into the sign-maker’s social, cultural, and material context at the time of producing the sign. The motivated association existing between a form and a meaning in a
sign is crucial to interpretation, and provides empirical grounds to multimodal analysis;
indeed,
[i]f the “shape” of the signifier aptly suggests the “shape” of the signified […], it
allows an analyst—whether in everyday interaction or in research—to hypothesize
about the features which the maker of the sign regarded as criterial about the object
which she or he represented. Positing that relation between “sign” and “world” is crucial [and] can lead to an understanding of the sign-maker’s position in their world at
the moment of the making of the sign. Such a hypothesis is of fundamental importance in all communication. (Kress, 2010: 65)
In this sense, social semiotic multimodal analysis sees signs in a text as the “material
residues” of the sign-maker’s interest and social position at the time of his or her making
of the sign.
As an example, the photos featured on the BBC news website in the feature titled
Gaza-Israel Conflict: Why Are Civilians on the Front Lines?, dated July 15, 2014,1 deploy
long-shots when portraying explosions, destroyed houses, and Palestinians, shot as a
crowd, affected by the Israeli air strikes in Gaza. In turn, they deploy a closer shot of
individual persons when portraying citizens in Israel witnessing a rocket attack coming from Gaza. In a social semiotic perspective, distance of shot is a motivated signifier
for social distance (Kress and van Leeuwen, 1996) between the reader/viewer and the
represented participants, thus inviting to a greater or lesser identification with them;
the representation of individuals versus groups encodes individualization versus collectivization, with a humanizing versus anonymizing effect, respectively (Machin,
2007: 118–119). Through the resources of shot and number of represented participants,
these photos shape differently the relation with the bipartisan “civilians” noted in the
news header, humanizing and inviting readers/viewers’ identification with some, while
anonymizing and presenting others as a more distant reality. The motivated association between the signifier and the signified in these signs reveals the news provider’s
462
Elisabetta Adami
standpoint toward that specific event, in line with other cases of media representation of
the conflict in the region (cf. the findings in van Leeuwen and Jaworski, 2002).
Hence a detailed analysis of relations between motivated signs in images and writing
(and any other mode) can offer deeper insights on the meanings produced by a text, on
the relations they shape with viewers, and on the social positioning of its producer.
Meaning Potential of Semiotic Resources
Semiotic resources have meaning potentials deriving from their materiality and the history of their uses in a given society. When a semiotic resource is used in representation,
a sign is newly made. Every time it is used, it undergoes a certain degree of transformation. Two principles drive transformation: provenance and experiential meaning potential. Provenance, closely related to Barthes’s (1977) notions of “myth” and “connotation,”
defines “where signs come from.”
The idea here is that we constantly “import” signs from other contexts (another era,
social group, culture) into the context in which we are now making a new sign, in
order to signify ideas and values which are associated with that other context by
those who import the sign. (Kress and van Leeuwen, 2001: 10–11)
As a banal example, the meaning of ketchup (the entity as a semiotic object, and, consequently, the word naming it) in the Italian context (the national context of the author)
is endowed with the meaning component “American,” with all related values associated
to “American” by the Italian culture, generally speaking, and those of specific social
groups within it, which might well differ in terms of affect. This component is instead
absent in the meaning of ketchup for a US-based sign-maker. Whenever sign-makers
use a semiotic resource to create a sign, they transform it by endowing it with the meanings associated to its provenance by their social group.
Experiential meaning potential is instead akin to Lakoff and Johnson’s (1980) view of
metaphor and it condenses
the idea that signifiers have a meaning potential deriving from what it is we do when
we produce them, and from our ability to turn action into knowledge, to extend our
practical experience metaphorically, and to grasp similar extensions made by others.
(Kress and van Leeuwen, 2001: 10–11)
As an example of experiential meaning potential, the “sepia” color effect used, for example, when editing images with software tools available on Instagram, has come to have
the meaning of “past/old” (and by extension “vintage” or “nostalgia”) through association with the experience we have of the particular (dis-)coloring process that printed
photographs undergo through time.
These two concepts—provenance and experiential meaning potential—are used in
multimodal analysis to derive meaning potentials of resources used in texts, by tracing
Multimodality
463
the associated meanings given to their uses in other contexts. This helps reveal manipulative uses of resources through “borrowing” from previous uses in other contexts.
For example, in videos, a rapidly moving frame and an unstable focus are generally
associated—through experiential meaning potential—to amateur production; professional video-makers are increasingly making use of these resources in TV commercials
to give a sense of authenticity to their advertisement and, metonymically, to the promoted
features of the product. The same can be said for the changing dress code (along with gesturing, language, and so on) by political figures. Contemporary (Western) political dress
code, by borrowing resources from informal and everyday fashion, through provenance,
is increasingly shaping politicians as peer-citizens and laypersons, in the attempt to shape
a more informal, familial, and closer relation with voters (for the use of provenance and
experiential meaning potential, see Adami 2014’s framework for the analysis of the aesthetic meaning potentials of layout, font, color, images, and writing in digital texts).
Genre as the “Entexting” of Social Roles
Being socially situated, signs and sign-complexes embody power relations that are
entexted in genres and generic forms. Through genres, signs and sign-complexes project social positioning and identity values onto those who design and produce them and
onto those addressed by them. As an example, the selfie (i.e., a self-portrait picture taken
through a mobile device and shared online) is a recently born digital genre arisen from
technological affordances of mobile devices (their front camera and online connectivity
feature). It has received attention in the media to an extent that celebrities are increasingly shooting selfies (as a particularly famous instance, see the selfie that a group of
celebrities have collectively taken during the 2014 Oscars ceremony).2 Started as a practice by “lay” sign-makers online as the digital and online-shared form of old self-portrait
photographs, when made by a celebrity, the selfie communicates the identity values of
“everyday person,” who shoots his or her own photos of him- or herself and shares them
online with his or her friends, rather than of a “celebrity” whose pictures are taken by
professional photographers and addressed to fans. Hence the celebrity selfie practice
can be seen as an indication of the increased social value attributed to informality and
horizontal power relations in the entertainment industry in particular and in Western
societies in general (relevant cases of selfies involve politicians and other social elites).
Revealing the social meaning potential of a genre can offer insights onto broader social
dynamics at force in society and can provide sign- and meaning-makers with tools for
critical interpretation. This includes the understanding that identity features and social
relations are designed and projected by the genre, rather than “lived or real” ones, as
an analysis of the environment where the selfie was taken can show. In this sense, the
act of taking selfies by celebrities can be seen as a performance of peer-identity features
enacted in front of the media, as in the example of Eva Longoria and Melanie Griffith
taking selfies at Taormina Film Festival in 2014, surrounded by photographers and an
audience taking photos of the selfie-event.3
464
Elisabetta Adami
Here again, a social semiotic take on genre contrasts (or can integrate) the focus of
structuralist traditions. In a social semiotic perspective, genre is never stable; rather, it is
an ever-changing frame of reference and orientation that enables sign-makers to shape
and make meaning of social roles in a given communicative event/text—social roles
that are themselves also always subject to change through agentivity.
Potentials, Limitations,
and Future Perspectives
As one of the theoretical perspectives in multimodal analysis, social semiotics uses
modes as units of analysis to trace social values, positioning, and identity features projected by a text onto its author and addressed audience. Given the multimodality of contemporary communication, it is a useful framework to account for the social meanings
of texts, providing a wider and more in-depth picture than traditional discourse analysis focused solely on language. In focusing on how the meaning potentials of modal
resources are combined together in texts, and in tracing the sign-maker’s interests in
their motivated making of signs, it provides tools that can reveal naturalized discourses,
values, and ideologies in the current use of all modal resources.
Because of its unit of analysis and specific focus, the approach has certain methodological limitations, along with lines of investigations that are still unexplored. Analysis
is necessarily carried out qualitatively on small samples of texts; it is fine-grained,
informed by the research question, and can be time-consuming (for details on methods
and steps of analysis, cf. Bezemer and Jewitt, 2010). Generalizations are often difficult to
make and some (e.g., Bateman et al., 2004) have argued for the need to develop methods
to approach larger corpora.
As to visual texts, extant research so far has paid predominant attention to the
resources of image and on the relation between image and writing. In this regard, van
Leeuwen (2008) advocates
an integrated multimodal approach to visual communication in which the analysis
of images becomes less central than the analysis of semiotic resources such as composition, movement and colour, which are common to a range of semiotic modes
including images, graphics, typography, fashion, product design, exhibition design
and architecture. (van Leeuwen, 2008: 130)
As to its scope, analysis of texts “can reveal the ‘how’ certain meanings are produced;
it cannot say how readers will interpret them nor the real intentions of producers”
(Machin and Mayr, 2012: 10). In this, the use of methodologies drawn from other disciplines, such as ethnographic research, or studies in readers’ and designers’ perceptions,
can fruitfully complement the approach.
Multimodality
465
Interdisciplinary work is increasingly sought after in social semiotic multimodal
research. Its perspective can offer other social sciences a fine-grained and empirically
based methodology for the analysis of social meanings in multimodal texts; at the same
time, it can draw from other social sciences broader frames for the interpretations of
larger social dynamics underlying the production of these meanings or deriving from
their interpretation.
Multimodal analysis is well equipped to investigate texts, yet further work is needed
to approach text-making processes, as advocated by Iedema (2003):
Often oriented to finished and finite texts, multimodal analysis considers the complexity of texts or representations as they are, and less frequently how it is that such
constructs come about, or how it is that they transmogrify as (part of larger) dynamic
processes. (Iedema, 2003: 30)
The focus on text can be limiting in another respect. In contemporary sign-making,
texts and signs are selected and recontextualized, reused, repurposed, and disseminated
in different semiotic spaces; looking at single texts might offer a limited point of observation. As Lemke suggests, “we need to extend the usual repertory of analytical tools for
critical multimedia analysis from those which look at single works to those which look
across transmedia clusters” (2009: 140). Transmedia text production and dissemination
are often driven by corporations; hence Lemke advocates a move from analyses “which
focus on the formal features of the media themselves, to ones which place the experience of media within political economy and cultural ecology of identities, markets, and
values” (2009: 140).
Van Leeuwen (2008) also stresses the need to focus on the technology, and the power,
restrictions, and ideological frames that it imposes on sign-making, especially in light of
the increased use of predesigned software tools for text production offering preset templates and preferred options for sign-making. In his view, there is a need for
a new emphasis on the discourses, practices and technologies that regulate the use
of semiotic resources, and on studying the take-up of semiotic resources by users
in relation to these regulatory discourses, practices and technology. (van Leeuwen,
2008: 130).
The increasingly multimodal nature of communication, combined with a wider
availability of technologies for public dissemination, can certainly be seen as a trend
towards a democratization of resources available to everyday sign-makers; however,
the current multimodal landscape does not escape broader social dynamics of power.
Not only is technological development—and what it affords as preferred/dispreferred
modal choices—driven by the (huge) interests of corporations operating in the field,
but also access to and awareness of the meaning potential of modal resources are
differently distributed within societies, where broader power dynamics are always
in place. In this sense, multimodal analysis could combine interdisciplinarily with
466
Elisabetta Adami
theories and approaches in the social sciences to explore further the issue of access
and provide broader socially based frames for a critical engagement with multimodal
texts/events.
Critical interpretation is not the only concern of social semiotic multimodal analysis; in this regard, Kress (2000, 2010) has long stressed the need for a move from
critique to design. Analogously to what critical linguistics and critical discourse
analysis have analyzed for language, multimodal critical discourse analysis (Machin
and Mayr, 2012) intends to reveal naturalized ideologies, social values, power interests, and manipulative uses of all modal resources, in texts combining more than one
mode. Social semiotics aims at going one step further. In Kress’s view, while critique
was needed in a social semiotic landscape that was stable and needed change, a fastpaced changing media landscape like today’s foregrounds design choices and options.
In a time when social relations (and their semiotic counterpart, i.e., genres) are fluid
and texts are increasingly multimodal, when conventions are no longer fixed and signand meaning-makers are everyday faced with a wide range of choices for representation, a theory aimed to describe sign-making as a social practice needs to focus on the
ways in which sign-makers design their texts and meaning-makers design their forms
of engagement with them. When representation is conceived as a record of society as
well as contributing to shape it, the agency of sign-makers is foregrounded not only in
their creative use of resources to express meaning, but also in the potentials of these for
(social) change. Design is hence a key aspect for future research in social semiotic multimodal analysis.
Concluding Remarks
and Review of Key Points
The chapter has defined the concept of multimodality as a phenomenon of communication. It has discussed the reasons of its increased use in linguistics and disciplines
interested in meaning and text, which, however, do not necessarily use methods of multimodal analysis. It has then reviewed the growing field of multimodal studies, which
adopt different theoretical perspectives for the analysis of modes, and their intertwined
use in texts and communicative events. A social semiotic take to multimodal analyses
has then been presented in detail, by introducing selected key notions, before mentioning the potentials and limitations of the approach, together with some directions for
future research in the field.
Social semiotics is a theoretical approach to multimodal analysis; it informs the way
in which multimodal analysis is conceived and carried out. It sees human communication as the expression of social processes, and it sees this expression as intrinsically
multimodal. With the assumption that the social is prior to the semiotic, social semiotics frames the interpretation of multimodal representation and communication with
Multimodality
467
a special focus on sign- and meaning-makers. It conceives of signs as socially shaped
resources that are newly made every time by sign-makers who, according to their interests, associate in a motivated way selected criterial aspects of a form (the signifier) to
selected criterial aspects of the meaning (the signified) that they want to express. Every
resource has potentials to make meaning, derived from its materiality and the history of
its social uses. Analogously, every mode has affordances, deriving from its materiality
and social histories. Being socially situated, signs and sign-complexes embody power
relations that are entexted in genres and generic forms. Through genres, signs and signcomplexes project social positioning and identities values onto those who design and
produce them as well as those addressed by them.
A social semiotic multimodal analysis of a text asks questions such as the following: Which modes are at work here? What is their relative functional load? What is
the motivated association of a given form to a given meaning? Whose interests does it
reveal? What identity features are projected on the text’s author and addressees? Who is
given power/freedom? (e.g., readers/addressees, in designing their own reading path,
or the author?) And what does this all indicate in terms of social relations, values, and
ideologies?
The use of a certain color and color palette or of a font type, like the selection of different modalities in images (e.g., as photo-realistic or abstract), carries certain meanings that are socially shaped and vary across cultures. That is, the use of all modal
resources is principled, and modal resources have meaning potentials that are given
by the history of their past uses. Even if not expressed explicitly, as has long been done
for speech and writing in linguistic traditions, genre- context- society- and culturespecific conventions do exist for the use of all modes. These are naturalized conventions, which stem from regularities and variations in the past and present uses of a
given modal resource.
From the overall multimodal orchestration of a webpage, its use of colour, layout of
elements, fonts, images, and writing, we can intuitively tell whether it is designed to
look professional or amateur, whether it addresses children or adults, whether we are
addressed as experts or as general public, or as belonging to a specific social group, in
terms of gender, age, education, profession, and lifestyle. Yet precisely because conventions of modal resources other than language are naturalized, as interpreters of these
texts, we have a lack of awareness of the social values of their meaning potentials. Hence
investigating the meaning potential of modal resources, together with developing analytical tools that make these conventions explicit, can empower meaning- and signmakers in their everyday activity of interpreting, critiquing, and designing texts that can
effectively fulfill their rhetorical aims.
A social semiotic multimodal approach, then, always combines a twofold focus
on texts; it investigates texts and representational practices as socially and culturally
shaped; and it uses the investigation of texts and representational practices as a means to
achieve insights into society and social groups, into the ways in which they shape power
relations and their cultural values.
468
Elisabetta Adami
Notes
1. http://www.bbc.com/news/world-middle-east-28252155, accessed July 15, 2014.
2. It can be viewed online on The Guardian, among other websites: http://www.theguardian.
com/media/2014/mar/07/oscars-selfie-most-retweeted-ever, accessed July 15, 2014.
3. Photos of the event can be viewed on the website of the local newspaper La Sicilia:
http:// www.lasicilia.it/ gallery/ melanie- griffith- e- eva- longoria- raffica- di- %E2%80%
9Cselfie%E2%80%9D-al-teatro-antico, accessed July 15, 2014.
References
Adami, E. (2014). “Aesthetics in Digital Texts beyond Writing and Image: A Social Semiotic
Multimodal Framework.” In A. Archer and E. Breuer (eds.), Multimodality in Writing: The
State of the Art in Theory, Methodology and Pedagogy (pp. 43–62). Leiden: Brill.
Adami, E. (2015). “What’s in a Click: “A Social Semiotic Framework for the Multimodal
Analysis of Website Interactivity.” Visual Communication 14(2): 1–21.
Adami, E., R. Facchinetti, and G. Kress (eds.). (2014). “Multimodality, meaning-making and
the issue of ‘text.’” Text & Talk 34(3, special issue).
Adolphs, S., and R. Carter. (2007). “Beyond the Word: New Challenges in Analysing Corpora
of Spoken English.” European Journal of English Studies 11(2): 133–146.
Allwood, J. (2008). “Multimodal Corpora.” In A. Lüdeling and M. Kytö (eds.), Corpus
Linguistics: An International Handbook (pp. 207–225). Berlin: Mouton De Gruyter.
Archer, A. (2006). “A Multimodal Approach to Academic ‘Literacies’: Problematising the
Visual/Verbal Divide.” Language and Education 20(6): 449–462.
Archer, A., and E. Breuer (eds.). (2015). Multimodality in Writing. The State of the Art in Theory,
Methodology and Pedagogy. Leiden: Brill.
Baldry, A., and P. J. Thibault. (2006). Multimodal Transcription and Text Analysis: A Multimedia
Toolkit and Coursebook. London; Oakville: Equinox.
Barthes, R. (1977). Image-Music-Text. London: Fontana.
Bateman, J. (2008). Multimodality and Genre: A Foundation for the Systematic Analysis of
Multimodal Documents. New York: Palgrave Macmillan.
Bateman, J., J. Delin, and R. Henschel. (2004). “Multimodality and Empiricism: Preparing
for a Corpus-Based Approach to the Study of Multimodal Meaning-Making.” In
E. Ventola, C. Charles, and M. Kaltenbacher (eds.), Perspectives on Multimodality (pp. 65–
87). Amsterdam: John Benjamins.
Bateman, J., J. Delin, and R. Henschel. (2007). “Mapping the Multimodal Genres of Traditional
and Electronic Newspapers.” In T. D. Royce and W. Bowcher (eds.), New Directions in the
Analysis of Multimodal Discourse (pp. 147–172). Mahwah, NJ: Lawrence Erlbaum.
Bezemer, J., and C. Jewitt. (2010). “Multimodal Analysis: Key Issues.” In L. Litosseliti (ed.),
Research Methods in Linguistics (pp. 180–197). London: Continuum.
Bezemer, J., and G. Kress. (2016). Multimodality, Learning and Communication: A Social
Semiotic Frame. London: Routledge.
Bezemer, J., and G. Kress. (2014). “Touch: A Resource for Making Meaning.” Australian Journal
of Language and Literacy 37(2): 77–85.
Multimodality
469
Borodo, M. (2015). “Multimodality, Translation and Comics.” Perspectives: Studies in
Translatology 23(1): 22–41.
Bowcher, W. (ed.). (2012). Multimodal Texts from Around the World: Cultural and Linguistic
Insights. Basingstoke, UK: Palgrave Macmillan.
Burn, A. (2013). The Kineikonic Mode: Towards a Multimodal Approach to Moving Image Media.
London: NCRM. http://eprints.ncrm.ac.uk/3085/.
Burn, A., and D. Parker. (2003). Analysing Media Texts. London: Continuum.
Campagna, S., and C. Boggio. (2009). Multimodal Business and Economics. Milano: LED.
Deppermann, A. (ed.). (2013). “Conversation Analytic Studies of Multimodal Interaction.”
Journal of Pragmatics 46 (special issue).
Dicks, B., R. Flewitt, L. Lancaster, and K. Pahl (eds.). (2011). “Multimodality and Ethnography:
Working at the Intersection.” Qualitative Research 11(3, special issue).
Fairclough, N. (1989). Language and Power. London: Longman.
Forceville, C., and E. Urios-Aparisi (eds.). (2009). Multimodal Metaphor. Berlin: Mouton De
Gruyter.
Garzone, G. (2009). “Multimodal Analysis.” In F. Bargiela-Chiappini (ed.), The Handbook of
Business Discourse (pp. 155–165). Edinburgh: Edinburgh University Press.
Goodwin, C. (2001). “Practices of Seeing Visual Analysis: An Ethnomethodological Approach.”
In T. van Leeuwen and C. Jewitt (eds.), The Routledge Handbook of Visual Analysis
(pp. 157–182). London: Routledge.
Halliday, M. A. K. (1978). Language as Social Semiotic: The Social Interpretation of Language
and Meaning. London: Arnold.
Haugh, M. (2009). “Designing a Multimodal Spoken Component of the Australian National
Corpus.” In M. Haugh and et al. (eds.), Selected Proceedings of the 2008 HCSNet Workshop on
Designing the Australian National Corpus (pp. 74–86). Somerville: Cascadilla Proceedings
Project.
Herring, S. C. (2010). “Web Content Analysis: Expanding the Paradigm.” In J. Hunsinger,
M. Allen, and L. Klastrup (eds.), The International Handbook of Internet Research (pp. 233–
249). New York: Springer.
Herring, S. C. (2013). “Discourse in Web 2.0: Familiar, Reconfigured, and Emergent.” In
D. Tannen and A. M. Tester (eds.), Georgetown University Round Table on Language
and Linguistics 2011: Discourse 2.0: Language and New Media. Washington, DC: Georgetown
University Press (pp. 1–25). http://ella.slis.indiana.edu/~herring/GURT.2011.prepub.pdf.
Hodge, R., and G. Kress. (1988). Social Semiotics. Cambridge: Polity.
Iedema, R. (2003). “Multimodality, Resemiotization: Extending the Analysis of Discourse as
Multi-Semiotic Practice.” Visual Communication 2(1): 29–57.
Jewitt, C. (2005). “Multimodality, ‘Reading,’ and ‘Writing’ for the 21st Century.” Discourse:
Studies in the Cultural Politics of Education 26(3): 315–331.
Jewitt, C. (2006). Technology, Literacy and Learning: A Multimodal Approach. London:
Routledge.
Jewitt, C. (2008). “Multimodality and Literacy in School Classrooms.” Review of Research in
Education 32(1): 241–267.
Jewitt, C. (2009a). “Different Approaches to Multimodality.” In C. Jewitt (ed.), The Routledge
Handbook of Multimodal Analysis (1st ed., pp. 28–39). London: Routledge.
Jewitt, C. (ed.). (2009b). The Routledge Handbook of Multimodal Analysis, 1st ed. London:
Routledge.
470
Elisabetta Adami
Jewitt, C. (2014a). “An Introduction to Multimodality.” In C. Jewitt (ed.), The Routledge
Handbook of Multimodal Analysis, 2nd ed. London: Routledge.
Jewitt, C. (ed.). (2014b). The Routledge Handbook of Multimodal Analysis, 2nd ed. London:
Routledge.
Jewitt, C., J. Bezemer, and K. O’Halloran. (2016). Introducing Multimodality. London:
Routledge.
Klug, N. M., and H. Stöckl (eds.). (2014). Language in Multimodal Contexts. Berlin: Mouton De
Gruyter.
Knox, J. (2009). “Punctuating the Home Page: Image as Language in an Online Newspaper.”
Discourse & Communication 3(2): 145–172.
Kress, G. (1985). “Ideological Structures in Discourse.” In T. A. van Dijk (ed.), Handbook of
Discourse Study (pp. 27–42). London; New York: Academic Press.
Kress, G. (1993). “Against Arbitrariness: The Social Production of the Sign as a Foundational
Issue in Critical Discourse Analysis.” Discourse and Society 4(2): 169–191.
Kress, G. (1997). Before Writing: Rethinking the Paths to Literacy. London: Routledge.
Kress, G. (2000). “Design and Transformation: New Theories of Meaning.” In B. Cope and
M. Kalantzis (eds.), Multiliteracies (pp. 153–161). London: Routledge.
Kress, G. (2010). Multimodality: A Social Semiotic Approach to Contemporary Communication.
London: Routledge.
Kress, G., and T. van Leeuwen. (1996). Reading Images: The Grammar of Visual Design.
London: Routledge.
Kress, G., and T. van Leeuwen. (2001). Multimodal Discourse: The Modes and Media of
Contemporary Communication. London: Arnold.
Kress, G., and T. van Leeuwen. (2002). “Colour as a Semiotic Mode: Notes for a Grammar of
Colour.” Visual Communication 1(3): 343–369.
Lakoff, G., and M. Johnson. (1980). Metaphors We Live By. Chicago: University of Chicago Press.
Lemke, J. (ed.). (1998). “Language and Other Semiotic Systems in Education.” Linguistics and
Education 10(3, special issue).
Lemke, J. (2002). “Travels in Hypermodality.” Visual Communication 1(3): 299–325.
Lemke, J. (2009). “Multimodality, Identity, and Time.” In C. Jewitt (ed.), The Routledge
Handbook of Multimodal Analysis (pp. 140–150). London: Routledge.
Machin, D. (2007). Introduction to Multimodal Analysis. London: Hodder Arnold.
Machin, D. (2014). “What Is Multimodal Critical Discourse Studies?” Critical Discourse Studies
10(4): 347–355.
Machin, D., and A. Mayr. (2012). How to Do Critical Discourse Analysis. London: Sage.
Maier, C. D. (2008). “Multimodal Communication of Specialized Knowledge in Interactive
Corporate Materials.” International Journal of Learning 14(12): 109–116.
Maier, C. D. (2011). “Knowledge Communication in Green Corporate Marketing:
A Multimodal Discourse Analysis of an Ecomagination Video.” In K. O’Halloran and
B. Smith (eds.), Multimodal Studies: Exploring Issues and Domains (pp. 153–169). London:
Routledge.
Martinec, R. (2000). “Types of Processes in Action.” Semiotica 130(3–4): 243–268.
Norris, S. (2004). Analyzing Multimodal Interaction: A Methodological Framework.
New York: Routledge.
Norris, S., and C. D. Maier (eds.). (2014). Interactions, Images and Texts: A Reader in
Multimodality. Berlin: Mouton De Gruyter.
Multimodality
471
O’Halloran, K. (2008). “Systemic Functional-Multimodal Discourse Analysis (SFMDA): Constructing Ideational Meaning Using Language and Visual Imagery.” Visual
Communication 7(4): 443–475.
O’Halloran, K. (2011). “Multimodal Discourse Analysis.” In K. Hyland and B. Paltridge (eds.),
Companion to Discourse (pp. 120–137). London: Continuum.
O’Halloran, K., and B. A. Smith. (2011). “Multimodal Studies.” In K. O’Halloran and
B. A. Smith (eds.), Multimodal Studies: Exploring Issues and Domains (pp. 1–13). London:
Routledge.
O’Sullivan, C., and C. Jeffcote (eds.). (2013). “Translating Multimodalities.” Journal of
Specialized Translation 20 (special issue).
O’Toole, M. (1994). The Language of Displayed Art. London: Leicester University Press.
Page, R. (ed.). (2010). New Perspectives on Narrative and Multimodality. London;
New York: Routledge.
Pinnow, R. J. (2011). “‘I’ve got an idea’: A Social Semiotic Perspective on Agency in the Second
Language Classroom.” Linguistics and Education 22(4): 383–392.
Prior, P. (2009). “From Speech Genres to Mediated Multimodal Genre Systems: Bakhtin,
Voloshinov, and the Question of Writing.” In C. Bazerman, A. Bonini, and D. Figueiredo
(eds.), Genre in a Changing World (pp. 17–34). West Lafayette, IN: Parlor Press and The WAC
Clearing House.
Romero, E. D., and C. M. Arévalo. (2010). “Multimodality and Listening Comprehension:
Testing and Implementing Classroom Material.” Language Value 2(1): 100–139.
Royce, T. D. (2007). “Multimodal Communicative Competence in Second-Language
Contexts.” In T. D. Royce and W. Bowcher (eds.), New Directions in the Analysis of
Multimodal Discourse (pp. 361–390). Mahwah, NJ: Lawrence Erlbaum.
Scollon, R., and S. Wong Scollon. (2003). Discourse in Place: Language in the Material World.
London: Routledge.
Sindoni, M. (2013). Spoken and Written Discourse in Online Interactions: A Multimodal
Approach. London: Routledge.
Streeck, J., C. Goodwin, and C. D. LeBaron (eds.). (2011). Embodied Interaction: Language and
Body in the Material World. Cambridge; New York: Cambridge University Press.
Taylor, C. (2004). “Multimodal Text Analysis and Subtitling.” In E. Ventola, C. Cassily, and
M. Kaltenbacher (eds.), Perspectives on Multimodality (pp. 153–172). Amsterdam: John
Benjamins.
Unsworth, L. (2008). Multimodal Semiotics: Functional Analysis in Contexts of Education.
London: Continuum.
Van Dijk, T. A. (1991). Racism and the Press. London: Routledge.
van Leeuwen, T. (1999). Speech, Music, Sound. London: Macmillan.
van Leeuwen, T. (2004). “Ten Reasons Why Linguists Should Pay Attention to Visual
Communication.” In P. Levine and R. Scollon (eds.), Discourse and Technology. Multimodal
Discourse Analysis (pp. 7–19). Washington, DC: Georgetown University Press.
van Leeuwen, T. (2005a). Introducing Social Semiotics. London: Routledge.
van Leeuwen, T. (2005b). “Typographic Meaning.” Visual Communication 4(2): 137–143.
van Leeuwen, T. (2006). “Towards a Semiotics of Typography.” Information Design Journal
14(2): (139–155).
van Leeuwen, T. (2008). “New Forms of Writing, New Visual Competencies.” Visual Studies
23(2): 130–135.
472
Elisabetta Adami
van Leeuwen, T., and A. Jaworski. (2002). “The Discourses of War Photography:
Photojournalistic Representations of the Palestinian-Israeli War.” Journal of Language and
Politics 1(2): 255–275.
Ventola, E., and A. J. M. Guijarro (eds.). (2009). The World Told and The World Shown:
Multisemiotic Issues. Basingstoke: Palgrave Macmillan.
Wodak, R. (1989). Language, Power and Ideology: Studies in Political Discourse Amsterdam:
John Benjamins.