DOI 10.1515/text-2014-0007
Text&Talk 2014; 34(3): 231 – 237
Elisabetta Adami and Gunther Kress
Introduction: multimodality, meaning
making, and the issue of “text”
Elisabetta Adami: Dipartimento di Lingue, Letterature e Culture Moderne, Università G.
D’Annunzio Chieti-Pescara, Pescara, Italy. E-mail: e.adami@unich.it
Gunther Kress: Department of Culture, Communication and Media, Institute of Education,
University of London, UK. E-mail: g.kress@ioe.ac.uk
Something may be an entire commonplace and yet barely be understood. The contemporary media of communication are an instance: used everywhere, by nearly
everyone (even if only in the most rudimentary fashion – as in automated checkouts at the supermarket), and yet they remain just about opaque in terms of their
present social effects and future consequences. In terms of the thematic remit of
this journal, the increasingly insistent presence of multimodal texts is another
such instance: at a different level, though with related profound consequences.
This special issue will not delve into the more far-reaching, more profound,
social consequences; it will confine itself to exploring lesser questions, such as
“how do we think about ‘text’ in a world where multimodal semiotic entities are
beginning to dominate the semiotic landscape?” For a journal that has, enshrined
in its title, the aim to expand the notion of “text” to include “talk,” what developments, threats, challenges does the appearance of the notion of “multimodality”
foreshadow for its field of enquiry?
Here we make a quick list of some of the issues which go to the core of work
represented in this journal; we discuss each one briefly, in turn. The list is meant
neither as a definitive nor as an exhaustive list, but to prompt debate and possibly
further work.
Most of all, the phenomenon of multimodality shifts the center of gravity from
linguistic to semiotic concerns. When all the resources which matter in meaning
are at issue, then the tools of one discipline – linguistics and its satellite (sub-)
disciplines – can no longer be sufficient to provide satisfactory accounts of the
materials to hand and the questions they pose.
To make this somewhat concrete, consider the screenshot shown in Figure 1.
Assume just for the sake of argument that we wanted to give some account
of this (now entirely usual kind of) semiotic entity – we might want to say something about its genre; maybe about the audience and the kinds of imagined subjectivities of its members; or maybe describe its style; etc. The kinds of tools which
(different kinds of) linguistics offers will not allow us even to begin making any
Authenticated | e.adami@unich.it author's copy
Download Date | 5/21/14 10:55 AM
232
Elisabetta Adami and Gunther Kress
Fig. 1: (Part of) Homepage of the website www.visitlondon.com (accessed 15 December 2013)
inroads. Two alternatives offer themselves at once: we might invent ad hoc categories, which we bolt onto the linguistics of our choice, and see where we get to;
or we might say: “this is not the sort of thing we have an interest in; it is not (a)
text.”
Members of a wide and disparate audience who come to this website do make
meaning from and with the resources offered here. They have no difficulty noting
– and interpreting in various ways – the difference between this web page and the
one shown in the screenshot of Figure 2.
This would introduce a further problem: some web pages, such as the one in
Figure 2, seem amenable, at least in part, to a linguistic analysis of some kind;
while others, like the one in Figure 1, are not. Clearly there are problems here;
turning away from them may not be the perfect response. We can say that the example in Figure 1 requires us to take account of the fact that there are images; that
the images – unlike the one in Figure 2 – are “in” color; that is, color as well as
image seems to be a resource drawn on in the construction of that semiotic entity.
And of course there is writing. We might need to comment on the varying proportions of writing and image (and color) in the two cases; and we might wish to
hypothesize what might be “motivating” that differential use of the three modes.
We might further wish to comment on the startlingly different forms of layout of
the two sites – a relatively “traditional” linearity in the one case (Figure 2) and
a relatively strong modularity in the other (Figure 1) – and speculate about the
origins and meanings of these different principles of composition: are they effects
Authenticated | e.adami@unich.it author's copy
Download Date | 5/21/14 10:55 AM
Introduction
233
Fig. 2: (Part of) Web page of the Mayor of London, www.london.gov.uk/who-runs-london/mayor
(accessed 17 August 2013)
of different social environments, or of the “digital” environment, or of both? We
might ask about resources and principles for producing cohesion and coherence
in texts which are based on modular composition and are multimodal in character – and what follows for conceptions of coherence, socially and semiotically.
Beyond the co-presence of different modes, there is the matter of their different uses and functions: writing is central in one case (Figure 2) and marginal
in the other (Figure 1). If our interest were to remain settled on the two linguistic
modes of speech and writing, we would nevertheless feel a need to reflect about
the changes in function of writing which seem indicated here, and speculate
about the future of the mode of writing in a broad way.
That route takes us beyond description alone, and into exploring connections between social environments and the respective uses (and functions) of a
mode, or of different modes. Even the two examples used here suggest that we
cannot answer those kinds of questions without considering all the resources
which occur in what is now a larger frame than that of speech and writing alone,
namely of all the modes which are available to members of one social/cultural
group. We have, in taking that step, moved decisively from the one discipline
– linguistics – concerned with one mode (or the two modes of speech and writing) to another discipline – (social) semiotics – and its focus on all resources for
making meaning available for members of social groups as “their” cultural resource. The insights of the various forms of linguistic theorizing and work would
Authenticated | e.adami@unich.it author's copy
Download Date | 5/21/14 10:55 AM
234
Elisabetta Adami and Gunther Kress
remain as a resource to draw on, though they might, given the social semiotic
framing, need to be modified and extended.
As one seemingly slight example, take the realization of “URLs,” as in the
case here (taken at random from a “thread” on an article on Syria in The Guardian
newspaper:
http://sabbah.biz/mt/archives/2013/12/15/sy-hershs-chemical-misfire/#utm_source=
feedburner&utm_medium=feed&utm_campaign=Feed%3A+SabbahsBlog+
%28Sabbah+Report%29
We are unable to say whether this is writing in any conventional sense. We are
able to say – as any of our readers will agree – that such kinds of elements are
entirely usual in contemporary semiotic entities. In an up-to-date theory of text
making it ought to be possible to account for any semiotic entity which the ordinary man and woman in the street encounters constantly, and treats as usual.
There are two further issues which are distinct and closely connected. The
first is that multimodality is an approach is not a theory. It “stakes out” major
aspects of the domain in which meaning is made. Different theories pose different questions within this domain, questions which are central to that theory. The
theory invoked here is social semiotics (Hodge and Kress 1988; Kress 2010; Van
Leeuwen 2005) and, generally speaking, it is used by all the contributors in this
volume. It asks questions, broadly about all aspects of meaning and meaning
making. We are aware of the limits of applicability of that theory; and where we
come to its margins, we are happy to enlist the services of other theories. The
second matter is that “meaning” is brought into being in particular sites, by those
who make meanings. Some of those sites are those of the “new media,” so called.
In using two websites as our examples, we have made some slight reference to
such sites already. Yet multimodality as such does not concern itself with “sites
of appearance” – whether these are the material sites of everyday social interaction, or the differently material sites of digitally produced and instantiated
media. Quite clearly the potentials and limitations of sites – in their material,
technological, social aspects – cannot simply be ignored when we think about
meaning. For us, it is social semiotics which supplies hypotheses, descriptions,
frames, and principles here. Multimodal composition as an issue is at once independent of sites, and, at the same time, entirely shaped by the characteristics of
the sites.
Above all, we feel there is a strong claim to be made that the issues raised
here and developed in particular ways in the rest of this special volume are issues
which are now central to the domain of work of this journal.
Authenticated | e.adami@unich.it author's copy
Download Date | 5/21/14 10:55 AM
Introduction
235
Each of the papers that follow examines a different kind of multimodal
“text.” These encompass digitally produced texts and transformations of traditional texts in mass media productions: online videos (Adami) and urban youth’s
video-based music production (Domingo); “transmediated” TV series (Moschini)
and new forms of filmmaking (Rowsell); video chats (Sindoni) and PowerPoint
presentations (Zhao, Djonov, and Van Leeuwen).
Focusing on video-based asynchronous communication, Adami discusses interaction practices on YouTube. The paper analyses relatedness in videos linked
as responses to other videos, identifying a wide range of variation. Interactions
include not only remotely related and highly implicit exchanges (the result of
practices of recontextualizing), but also relations hinging on form-as-meaning.
The notion of intertextual “relatedness” established through “form” opens to unexpected uses of the multimodal possibilities of the medium, thus extending the
notion of dialogue.
Discussing another form of digital video production, Domingo’s ethnographic study examines hip hop music videos produced by a group of UK-based
Filipino young people. It investigates how participants use multimodal resources
to express cultural remix, affiliation, and belonging across different linguistic
and discourse communities. In the participants’ use, modes embody social and
cultural significance. Combining both online and offline production, their sign
making extends notions of text to include fluid configurations, as is increasingly
the case in contemporary representation.
That issue is discussed further by Moschini in her analysis of recent transformations in TV series, encompassing multimodal intertextuality, “webridized”
sign making, and transmedial production. These features can hardly be described
in any traditional textual analysis. Such transmedial processes problematize
what constitutes “a text,” extending sign making across different spaces and assigning co-production to different agencies, including fans.
Rowsell observes filmmakers’ selection of modal resources in designing documentaries and 3D animations. In her discussion of their use of modes to evoke
emotions in viewers’ experiencing of texts, the author raises questions concerning the role of embodiment and perception in meaning making. These questions
inevitably become more compelling as forms of engagement increasingly involve
immersive digital environments and the use of mobile and wearable technologies.
Focusing on a different kind of video production, Sindoni’s paper examines
a new type of synchronous communication, i.e., video chats, where verbal language itself is multimodally orchestrated, with functional loads shifting between
speech and writing and with other modal resources being significantly shaped by
the medium’s affordances. The paper discusses the issues faced by multimodal
Authenticated | e.adami@unich.it author's copy
Download Date | 5/21/14 10:55 AM
236
Elisabetta Adami and Gunther Kress
analysts in transcribing and analyzing such a multimodally dense interactional
environment.
As a final example of time-based multimodal communication, Zhao, Djonov,
and Van Leeuwen’s paper introduces a multimodal social semiotic approach to
studying PowerPoint presentations. These are seen as a multidimensional semiotic practice combining the software’s design, the multimodal composition of
slideshows, joined with their spoken performance/presentation in the communicative event. The paper problematizes the notion of text in each dimension; it discusses the limitations of logocentric approaches to their analysis while proposing
a holistic and adaptive multimodal approach, crucial for the understanding of
time-based multimodal productions.
While the papers raise a series of questions on what is to be considered as
“text” in contemporary sign making, this special issue does not aim to provide
definitive answers, nor does it intend to provide an exhaustive catalogue of contemporary forms of multimodal representation. Both tasks are impossible. First,
new uses of multimodal resources appear every day, thanks to the socially driven
pace of technological change and the sign makers’ creative uses of technologies
for sign making. Second, and maybe more profound, digitally produced and mediated materials are likely to be “mere” changed instances – if at a somewhat
general level – of older phenomena. So, by adopting a social semiotic perspective
to text, sign making in digital environments might be making more tangible for
researchers and theorists – and bring to a point – what had “always” been there
in all multimodal forms of communication.
This special issue intends to prompt debate, and further work. We feel certain
that with the many questions raised about our understanding of text and meaning, it will offer a perspective that enables a look at texts (and, more broadly, at
meaning making) in a relevant and aptly enlarged frame.
References
Hodge, R. & G. Kress. 1988. Social semiotics. Cambridge: Polity.
Kress, G. 2010. Multimodality. A social semiotic approach to contemporary communication.
London: Routledge.
Van Leeuwen, T. 2005. Introducing social semiotics. London: Routledge.
Authenticated | e.adami@unich.it author's copy
Download Date | 5/21/14 10:55 AM
Introduction
237
Bionotes
Elisabetta Adami is a researcher in English Language and Translation at the
Department of Modern Languages, Literature and Cultures at the University G.
D’Annunzio of Chieti-Pescara, Italy. Her research focuses on language, multimodal representation and communication in digital environments. Her recent
publications use a social semiotic framework for the analysis of text production
in social media, the affordances of mobile phones (with G. Kress), and the use of
copy and paste.
Gunther Kress is Professor of Semiotics and Education at the Institute of Education, University of London. His interests are in meaning (making) and communication in contemporary environments, with two broad aims: to continue
developing a social semiotic theory of multimodal communication; and, in that,
to develop an apt theory of learning and apt means of “valuation of learning.”
Some of the books along the road are Language as Ideology; Social Semiotics
(both with Bob Hodge); Before Writing: Rethinking the Paths to Literacy; Reading
Images: The Grammar of Graphic Design; Multimodal Discourse: The Modes and
Media of Contemporary Communication (both with Theo van Leeuwen); Literacy
in the New Media Age; Multimodality: A Social Semiotic Approach to Contemporary
Communication.
Authenticated | e.adami@unich.it author's copy
Download Date | 5/21/14 10:55 AM