Using eye-gaze visual technologies to compare consumer response in
real and 3D virtual worlds: an exploratory application to retail
Abstract
This paper reports on the development and findings of a research investigation using
visual technologies to investigate comparisons into consumer responses of store-based
and virtual retail environments. The visual technologies used were mobile headmounted and desk-bound eye-gaze tracking tools. Desk-bound tracking technologies
have for some time been used in understanding consumer response in retail and
marketing settings whereas mobile head-mounted eye-gaze technologies have only
recently become available for use in these contexts. Whilst the visual technologies
presented some limitations in their application, the exploratory and qualitative
methodology adopted using content analysis triangulated against pre- and posttracking questionnaires and retrospective interviews with participants, enabled a
comparative analysis that suggests similarities in consumer response patterns to both
the real and virtual (3D) retail environments. Implications of the research approaches
and future applications are discussed in the context of retail and marketing.
Keywords
Mobile eye-gaze tracking, 3D virtual worlds, retail, visual technologies, qualitative
research, content analysis
Introduction
With innovations in non-invasive eye movement detection technologies since the
1970s, research into what respondents observe and how they physically view things
has been undertaken in diverse research fields such as cognitive science, psychology,
psycholinguistics, human-computer interaction, neurology and marketing, specifically
in advertising and communications research, packaging design and website
development. Such technologies have until recently, however, been cumbersome
when affixed to a respondent or have otherwise been desk-bound. A new class of
head-mounted eye-gaze tracking technology has now become available. This
tetherless and mobile technology is worn much like a pair of standard eye glasses.
Field research involving eye-gaze tracking can therefore for the first time take place
beyond laboratory, simulated and desk-based investigations. There are, however,
numerous challenges with such research, primarily because of the complexity of real
world environments (for example, the number of distractors within the setting) and
our ability as researchers to interpret findings and compare studies to previous
investigations using fixed base technologies. The aim of this paper is to report on the
development and findings of a qualitative study comparing application of mobile and
desk-bound visual technologies within similar environments. The paper evaluates
consumer responses in a marketing context to a retail environment and a virtual (3D
computer model) representation of the same retail environment. It begins with a brief
review of the eye-gaze research and technologies, a review of extant literature in eyegaze research in marketing that informed the design of the study and subsequently
discusses the methodology and limitations of the technologies, before concluding with
an evaluation of the findings and implications of the research.
Literature review
Eye movement research has been undertaken using electro-oculography, which
electrically measures differences around the eyes, scleral contact lenses, which
measures eye movements relative to head position with a wired lens, and video-based
infrared oculography (Wedel and Pieters, 2008a; Duchowski, 2007). This research
uses the latter technology, which has received greatest attention in marketing research
and is less obtrusive than others identified. With this approach, infrared light is
reflected (‘Purkingje’ reflections) on to the front and back layers of the eye ie.,
cornea, lens, retina, to identify the precise ‘point of regard’. Of interest to researchers
are the eye fixations indicating response to some stimulus, and saccades between the
fixations ie., the eye movements or ‘scan path’, calculated using velocity and
dispersion algorithms (Salvucci and Goldberg, 2000; Duchowski, 2007). Fixations
represent the overt attention of the respondent whereas peripheral vision or covert
attention, whilst not directly captured by the technologies, is thought to be preceded
by a change in overt attention and is hence captured through subsequent fixations.
Eye movement data, which may be accurate to less than .5 centimetre, is however
only a partial representation of response to stimuli (Langeman, 2005) and whilst
technologies may record visual behaviour other research methods are needed in order
to assess a broader range of cognitive and emotional responses. Within marketing,
most previous published investigations have applied desk-bound and computermounted technologies, which are suited to communications and web-based studies ie.,
screen-based or screen viewed. Mobile trackers enable naturalistic studies of human
visual behaviour, say in retail environments, albeit that no academic research of its
application has been published to date in the domain of marketing to the best
knowledge of the authors. The small body of literature in marketing calls for further
research across a range of different contexts, such as retail and online where
interactive design of experiences is now prevalent (Wedel and Pieters, 2008a). Thus
far, visual technologies have been used to understand consumer cognitive and
emotional response to advertising communications focussing on impact of branding,
images and text (Pieters et al, 2007; Wedel and Pieters, 2008b) in media such as print
and feature advertisements (eg., Aribarg et al, 2010; Zhang et al, 2009), billboards
(eg., Dreze and Hussherr, 2003), product labelling (eg., Fox et al, 1998), TV
commercials (eg., Janiszewski, 1998). Some research has also been undertaken on
shelves in retail stores, typically supermarkets (eg., Chandon et al, 2007; Van der
Lans et al, 2008). Researchers have found correlations between visual attention (eye
gaze) in the number and length of fixations and product preferences (Maughan et al,
2007) where positive attention results in more and longer fixations, albeit this may be
a function of gender, age, personality (Rosler et al, 2005; Isaacowitz, 2005) and
familiarity with brands (Russo and Leclerc, 1994). Of interest to marketers has been
how well respondents remember what they have seen, for example, firms invest
heavily in differentiating brands predicated upon their distinctive and memorable
features. Previous research using visual technologies has suggested, however, that
even when attention is recorded, recall and memory do not always correlate, such
phenomenon is known as ‘inattentional blindness’ (Memmert, 2006). One of the main
challenges is in understanding attention in complex environments where high levels
of visual clutter (‘distractors’) vie for consumer attention, such as where feature
advertisements compete on the page (Swartz, 2004) and naturalistic environments
such as retail environments. A theory of visual attention has been conceptualized by
Wedel and Pieters (2008a) who suggest it is both a function of consumer goals that
inform what and where to look (top-down factors) and saliency of marketing stimuli
ie., prominence of objects discerned by consumers within the scene (bottom-up
factors). These two elements combine to produce attentional priorities and
simultaneously suppress non-target perceptual features.
Research proposition
The research aims to explore and compare consumers’ visual attention using mobile
head-mounted eye-gaze tracking in a real store and desk-bound technology in a 3D
virtual marketing context used to simulate the real world. Both contexts are visually
comparable – virtual environments can now be created with very high levels of
photorealism to the real world such that they enable naturalistic behaviour to occur
albeit undertaken on screen within a computer-based model (Istance et al, 2008).
Wedel and Pieters (2008a) conceptualisation of visual attention is used to evaluate the
data. The research will assess consumer behaviour response patterns in the real world
and use findings to design and develop participant tasks within the virtual
environment so as to constrain the range of variables examined. Relevant intrinsic
top-down factors are the purchase goals and intentions of consumers in the retail
context, specified within the virtual environment to simulate a real world task,
coupled with the familiarity (previous experience) with the store layout. An
understanding of these factors will enable the researchers to identify primary search
behaviour and attention within a complex store environment, for example pre-planned
and impulse purchase and browsing behaviours. Extrinsic bottom-up factors
considered include the actual store layout (space and product), navigational and
section signage, promotional merchandising, product display, in-store offers, sales
assistants/store staff and roles played by other consumers (people) that facilitate
search and goal related behaviour. These are likely to be the aspects that consumers
fixate upon within the retail environment in order to achieve their goals and contribute
to complexity of the scene.
Methodology
A UK-based leisure, car maintenance and enhancement retailer was selected and full
permission including access to store layout plans was granted for the study. This type
of retailer is seen within the sector as a ‘destination’ store, where consumers visit with
pre-determined actions in mind, reinforced by store location in out-of-town retail
parks necessitating planned visits. The virtual 3D store was created for the research
by skinning a model of the real store within a games environment. Store layout,
gondola positions, product placement on shelves and signage using images from the
retail context produced a 3D navigable photorealistic virtual store comparable to the
real environment. Characters were precluded, however, the 3D virtual environment
was felt to accurately simulate the real environment including physical dimensions,
lighting and store layout. A dominant-less dominant mixed method approach was
used (Wedel and Pieters, 2008a), including eye-gaze tracking, pre- and postquestionnaires and retrospective interviews. Market Research Society ethics were
adopted for the conduct of the study. Eye-gaze tracking was undertaken using Tobii’s
Mobile Glasses for the real store (tracking distance 60-250cm ideal, accuracy 0.5
degrees, velocity 30Hz) and XL120 desk-bound system for the virtual store (tracking
distance 60cm, accuracy 0.5 degrees, velocity 60Hz). Both technologies were used in
conjunction with Tobii Studio processing software to produce statistical analyses of
the data collected. Findings from previous research suggest proprietary analyses show
high levels of reliability in 3D research contexts (Duchowski, 2007). Content analysis
was used as the dominant method to classify and analyse the frequency of visual
attention (fixation threshold 35Hz) of participants in both the real and virtual stores.
From initial review of the data collected, mutually exclusive categories were
developed in a qualitative mode (Haney et al, 1998) and subsequently data was
categorised by two coders in a quantitative mode (Berelson, 1952; Weber, 1990;
Kripendorff, 2004). Some 10 categories were identified for the real store 8 of which
applied to the virtual store. Cohen’s (1960) Kappa coefficient was used to test interand intra-coder reliability of content analysis. Data from the real store was used as the
basis for analysing the reliability of categorisation.
Within the retail environment, eye-gaze data was collected at the store location using
visiting consumers who were asked not to change their intended shopping behaviour
during the recruitment process. Data was collected in a lab-based environment for the
virtual store. Participants were selected on the basis that they owned or had access to
a motorized vehicle and had previously visited the type of store used for the study.
They were trained to use the navigation aids before commencing a series of tasks that
emulated the consumer stated behaviours in the real world context. Similar pre- and
post- eye-gaze data collection questionnaires were used. Pre-questionnaires identified
the extent of familiarity with the store brand and, for the real store, the primary
purpose of the visit. Post-questionnaires used for the virtual store noted the outcome
of the search tasks and demographic features and were followed by a retrospective
interview asking participants to recall their experience (Eger et al, 2007). Physical
constraints in the real environment precluded the use of this final data collection
stage, thus, post-questionnaires were used to establish the outcome of the visit and
evaluate the shopping experience. The preliminary recruitment phase included
calibration of the equipment to the participant and, with the virtual store, instructions
for three search tasks that consumers might undertake, based on data collected within
the real environment. Triangulation of methods and investigators were primary modes
of validation (Shapiro and Markoff, 1997).
Findings and discussion
Twenty-four participants took part in the real environment and 16 in the virtual
environment. Appendix 1 summarises the eye-gaze data collected, showing the
duration of data captured is lower in the virtual environment (tasks took relatively less
time to complete than in the real environment) albeit the amount of visual attention
data captured was higher. Unsurprisingly, this indicates the relative stability of deskbound eye-gaze equipment where participants move less compared to head mounted
equipment where consumers are physically interacting with the environment, ie.,
walking. Nonetheless, the amount of data captured enabled comparative content
analysis of participants’ visual attention in both environments with categorisation of a
similar number of behaviours for each context (virtual n=1843, real n=1889).
Findings from categorisation are shown at Appendix 2. Using the real environment as
the basis for reliability analysis, Cohen’s Kappa coefficient (>.9) indicates a high
level of similarity in the content analysis of the two data sets. This suggests validity of
the tasks relative to real world behaviour. It also suggests participants perceived a
high level of similarity between the two environments, evidenced in the visual
response patterns explored further below. Notwithstanding this, observed variations
suggest participants’ task completion rates enabled them to visually explore the
virtual store in more detail than observed in the real world.
Content analyses for real and virtual stores were reviewed by familiarity with store
brand, determined by recency of previous visit(s). Visual behaviour suggests store
visits involve an element of learning about products and their location within the store
that may facilitate future visits and purchase, since many consumers did not make
unplanned purchases. Data shows that consumers focussed a third of their visual
attention on products bought, whilst visual attention in relation to other products in
the store appears to increase if the consumer is not familiar with the shopping
environment. Indeed, views of other products is the behaviour observed most
frequently, and together with views of section and product signage, and engaging with
staff about other products, suggests participants are actively browsing the store. When
consumers are less familiar with the store environment their visual behaviour suggests
they are exploring the store, using more section and navigational signage and less
staff assistance than those familiar with the store. Conversely those familiar with the
store use product signage (positioned within close proximity to products on gondolas
and merchandising units) to facilitate browsing behaviour. Pre- and postquestionnaires established that most participants visited to make a pre-planned
purchase and made relatively infrequent visits eg., once or twice a year. Within the
virtual store whilst there is some variation in findings of content analysis explainable
by design of the research the overall patterns of visual attention is broadly similar.
Overall, the data suggests that participants engaged in greater visual exploration than
observed in the real environment. Variation between participants familiar with the
store brand and those that are unfamiliar is less clear, however, this may be a
consequence of participant selection process (familiar with store ‘type’). Content data
shows a quarter of visual behaviour observed is task related. A similar pattern is
observed to the real environment in relation to attention on other products and signage
and the task (product) itself.
Conclusions
Comparative use of the different eye-gaze technologies in the current study suggests
there are differences in their stability in use albeit reported in the literature that both
technologies are within acceptable tolerances for marketing research (Duchowski,
2007). This has implications for the design of research investigations that may, for
example, seek to understand consumer behaviour using eye-gaze tracking. In the
current design, mobile and desk-bound technologies were used to compare visual data
captured in two similar environments best suited for each respective technology (real
and 3D virtual). Evidently, consumer behaviour within the virtual and real space is
different insofar as visual attention is a function of four dimensions (depth, width,
height and time) as well as ‘interface’ design, where the real world is about
interaction between the human-physical space and the virtual world is humancomputer. Furthermore, underlying motivation in engaging with the environments is
different – in the real store consumers were engaged in actual shopping behaviour
whereas virtual store tasks were simulated. These considerations will undoubtedly
have impacted on the research findings. Despite these differences, and the limitations
of the research (sample size, experimental and qualitative design), findings suggest
both environments result in a similar pattern of visual attention and may therefore
underpin more quantitative investigations within different retail contexts. Importantly
and intuitively, similarities afforded by the 3D photorealistic virtual environment such
as developed for this study appear to provide opportunities to explore consumers’
visual behaviour. In this scenario, mobile tracking technologies may provide a useful
cross-referencing tool to compare consumer behaviour in the real world.
Content analysis of the data for the real store in the current study is interesting in
terms of understanding how consumers set goals for subsequent store visits and learn
about the store brand. Previous research (Pieters and Wedel, 2004) suggests visual
attention to brand decreases with familiarity whereas attention to detail of an object
increases, albeit the context of this research was communications and detail was text
of an advert. The implication for the type of ‘destination’ retailer from our study is
how to manage consumer expectations for future visits, made especially challenging
since greatest attention is given to browsing rather than buying during infrequent
visits. Evidence suggests that retailers may need to enhance product signage and
information on the shelf, such that it stimulates consumers to buy or visit more
frequently. This is supported by previous research that found that whilst goals
(intrinsic ‘top-down’ factors) of the consumer may account for around a third of
visual attention, the remainder of attention is due to salience of stimuli in store (Van
der Lans et al, 2008); where more facings on shelves may increase attention by as
much as 25% (Chandon et al, 2008); and, price and promotion may also positively
increase attention (Pieters et al, 2007). Little is known about the relationship between
frequency and recency of store visits and actual buying behaviour (versus stated
intentions to buy and theory of planned behaviour eg., Foxall, 2005; de Canniere et al,
2009), and as such this is an area for future investigation using eye-gaze technologies.
Findings for the virtual store suggest that visual attention is more likely to be a
function of the complexity of visual stimuli (extrinsic ‘bottom-up’ factors) whilst the
relatively lower visual attention given to task (product) signage and other product
signage is likely to be a function of a difference in motivation (intrinsic ‘top-down’
factors). This is also consistent with findings of previous research, for example Pieters
et al (2008b; 2010) found a relationship between complexity and informativeness in
feature adverts.
References
Aribarg, A.R., Pieters, R. & Wedel, M. (2010). Raising the BAR: Bias Adjustment of
Recogntion tests in advertising. Journal of Marketing Research, 47(3), 387-400.
Berelson, B. (1952). Content Analysis in Communication Research. Glencoe, Ill: Free
Press.
Chandon, P.J., Hutchinson, W., Bradlow, E.T. & Young, S. (2007). Measuring the
value of point-of-purchase marketing with commercial eye-tracking data, In Wedel,
M. & Pieters, R. eds. Visual marketing: from attention to action (pp. 225-258),
Mahwah, NJ: Lawrence Erlbaum Associates.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and
Psychological Measurement, 20, pp. 37- 46.
De Cannière, M.H., De Pelsmacker, P. & Geuens, M. (2009). Relationship Quality
and the Theory of Planned Behavior models of behavioral intentions and purchase
behavior. Journal of Business Research, 62(1), 82-92.
Dreze, X. & Hussherr, F. (2003). Internet advertising: is anybody watching? Journal
of Interactive Marketing, 17(4), 8-23.
Duchowski, A. (2007). Eye Tracking Methodology: Theory and Practice. Berlin:
Springer.
Eger, N., Ball, L.J., Stevens, R. & Dodd, J. (2007). Cueing retrospective verbal
reports in usability testing through eye-movement replay. People and Computers XXI
– HCI... but not as we know it. Proceedings of HCI, British Computing Society.
Fox, R.J., Krugman, D.M., Fletcher, J.E. & Fischer, P.M. (1998). Adolescents’
attention to beer and cigarette print ads and associated product warnings. Journal of
Advertising, 27(3), 57-68.
Foxall, G.R. (2005). Understanding Consumer Choice. Basingstoke: Palgrave
MacMillan.
Haney, W., Russell, M., Gulek, C., & Fierros, E. (1998). Drawing on education:
Using student drawings to promote middle school improvement. Schools in the
Middle, 7(3), 38- 43.
Isaacowitz, D. M. (2005). The gaze of the optimist. Personality and Social
Psychology Bulletin, 31(3) 407–415.
Istance, H., Bates, R., Hyrskykari, A. & Vickers, S. (2008). Snap clutch, a moded
approach to solving the Midas Touch problem. ETRA, March, 26–28, Savannah:
Georgia.
Janiszewski, C. (1998). The influence of display characteristics on visual exploratory
search behavior. Journal of Consumer Research, 25, 290-301.
Komogortsev, O.V., Jayarathna, S., Koh, D.H. & Gowda, S.M. (2009). Qualitative
and Quantitative Scoring and Evaluation of the Eye Movement Classification
Algorithms. Report Number TXSTATE-CS-TR-2009-16, Department of Computer
Science San Marcos, TX. Available online
http://www.diigo.com/list/tobiieyetracking/eye-tracking-development-comparisonusage, accessed 19 Jan 2011.
Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd
ed.). Thousand Oaks, CA: Sage.
Langeman, M. (2005). A Review of Eye Movement Tracking Research (25 April).
Available online http://www.langeman.net/SYDE_740_paper.html, accessed 19 Jan
2011.
Maughan, L., Gutnikov, S. & Stevens, R. (2007). Like more, look more. Look more,
like more: The evidence from eye-tracking. Journal of Brand Management, 14(4),
335-342.
Memmert, D. (2006). The effects of eye movements, age, and expertise on
inattentional blindness. Conscious Cognition, 15(3) 620–627.
Pieters, R. & Wedel, M. (2004). Attention capture and transfer in advertising: brand,
pictorial and text-size effects, Journal of Marketing, 68, 36-50.
Pieters, R., Wedel, M. & Batra, R. (2010). The Stopping Power of Advertising:
Measures and Effects of Visual Complexity. Journal of Marketing, 74(5), 48-60.
Pieters, R., Wedel, M. & Zhang, J. (2007). Optimal feature advertising design under
competitive clutter. Management Science, 53(11), 1815-1828.
Rosler, A., Ulrich, C., Billino, J., Sterzer, P., Weidauer, S., Bernhardt, T., Steinmetz,
H., Frolich, L. & Kleinschmidt, A. (2005). Effects of arousing emotional scenes on
the distribution of visuo-spatial attention: Changes with aging and early subcortical
vascular dementia. Journal of Neurological Science, 229–230, 109–116.
Russo, J.E. & Leclerk, F. (1994). An eye-fixation analysis of choice processes for
consumer nondurables. Journal of Consumer Research, 21, 274-290.
Salvucci, D.D., & Goldberg, J. H. (2000). Identifying fixations and saccades in eyetracking protocols. Proceedings of the Eye Tracking Research and Applications
Symposium (pp. 71-78), New York: ACM Press.
Schwatz, B. (2004). The paradox of choice: why more is less, New York:
HarperCollins Publishers.
Shapiro, G., & Markoff, J. (1997). A Matter of Definition. In C.W. Roberts (Ed). Text
Analysis for the Social Sciences: Methods for Drawing Statistical Inferences from
Texts and Transcripts. Mahwah, NJ: Lawrence Erlbaum Associates.
Van der Lans, R., Pieters, R, & Wedel, M. (2008). Competitive brand salience.
Marketing Science, 27(5), 922-931.
Weber, R.P. (1990). Basic Content Analysis (2nd ed). Newbury Park, CA: Sage.
Wedel, M. & Pieters, R. (2008a). Eye tracking for visual marketing. Foundations and
Trends in Marketing, 1, 4, 231-320, 2006.
Wedel, M. & Pieters, R. (2008b). A review of eye-tracking research in marketing, In
Malhotra, N.K. (Ed.). Review of Marketing Research, 4 (pp. 123-147), Armonk, NY:
M.E. Sharpe.
Zhang, J., Wedel, M & Pieters, R. (2009). Sales effects of attention to feature
advertisements: a Bayesian mediation analysis. Journal of Marketing Research, 46
(October), 669-681.
Appendix 1
Eye-gaze data
Eye-gaze data (seconds)
Number / avg samples by participant
Tobii Studio version 2.2.3 (2010)
Real Store
10,790
644 / 26.8
Virtual Store
4,231
1215 / 75.9
Appendix 2
All behaviours
(n=3732)
Virtual store - familiar
Total visual behaviours
(n=1843)
Virtual store
Unfamiliar (n=1181)
Virtual store
Familiar (n=660)
Real store
Total visual behaviours
(n=1889)
Real store
Unfamiliar (n=652)
Visual behaviour categories
(%)
Real store
Familiar (n=1237)
Content analysis by store brand familiarity
Product/s bought (task)
13.1 8.4
7.3
8.4
11.5
8.1
9.8
Other product/s (browsing)
36.9 43.9
34.4 29.3
39.4
31.1
35.3
Product signage relating to
6.3
5.2
5.8
8.7
5.9
7.7
6.8
purchase
Product signage relating to
16.2 11.7
8.9
9.9
14.6
9.5
12.1
other products
Specific in-store offer
2.4
2.0
2.8
3.8
2.3
3.5
2.9
Section (incl navigational)
7.8 11.2
6.3
3.6
8.9
5.9
7.4
signage relating to purchase
Section (incl navigational)
2.9
3.4
19.7 21.3
3.1
20.7
11.8
signage relating to other
products
Sales assistance relating to
3.7
1.7
3.0
3.0
purchase
Sales assistance relating to
4.0
2.8
3.6
3.6
other products
Other (eg., looking at floor,
6.7
9.7
14.7
13
7.7
13.6
10.6
ceiling)
Total visual attention related to 38.7 37.7
25.7 24.3
38.2
27.6
34.4
product purchased
Total visual attention related to 61.3 62.3
74.3 75.7
61.8
72.4
65.4
other products in store
Kappa coefficient: inter-coder reliability >.96; intra-coder reliability >.9 (acceptable
level >.85, Cohen, 1960)