Measuring Internet Advertising Effectiveness*
Lars Bergkvist, Stockholm School of Economics
Jonas Melander, CyberQuest AB
Paper presented at Internet Research 1.0: The State of the Interdiscipline (Association of
Internet Researchers), University of Kansas, Lawrence, KS, USA, September 14-17 2000.
Abstract
Much of the focus in connection with Internet advertising has been on click-through or other
behavioral responses, rather than on ad- or brand-related responses, which are used by many
advertisers to evaluate advertising in traditional media. In this paper results from four
empirical studies in which traditional ad- and brand-related responses were measured for
Internet advertising campaigns are presented. Results indicate that web advertising in some
cases cause ad- and brand-related responses and that these can be measured via the World
Wide Web.
*
The authors wish to express their gratitude towards DoubleClick Scandinavia who have contributed data as
well as other important input to this paper.
1 Introduction
It has been argued that the Internet is a fundamentally different marketing environment
compared to traditional media and that traditional advertising models are not applicable in this
context (e.g., Hoffman and Novak, 1996). The Internet is supposed to be different because it
makes many-to-many communication possible, i.e., consumers can interact with advertisers
and each other, as opposed to the one-to-many communication of traditional advertising.
Notwithstanding this, there are many forms of advertising on the Internet (or, rather, on the
World Wide Web) that are similar to traditional advertising and which appear to be used by
advertisers in a manner similar to how traditional advertising is used.
These forms of web-advertising include, for example, banner ads and pop-ups (see, e.g.,
http://www.doubleclick.net/), which are inserted into web-sites in the same manner as
traditional ads are inserted into newspapers, magazines or TV programs. As a further step
towards increased similarity with traditional advertising, the technological possibilities of web
advertising have developed quickly and it is today possible to include various types of rich
media (e.g., Shockwave or Real Audio) in some types of web advertising.
In some of the work on the effectiveness of Internet advertising it has been argued that the
effectiveness of Internet ads should be evaluated by their ability to generate click-through,
i.e., to make people click on the ad and, thereby, open a web-page related to the advertiser, or
some other behavioral response, such as sales or interaction on a web-site (e.g., Hofacker and
Murphy, 1998; Hoffman and Novak, 2000). This approach is similar to the one used for direct
response advertising (Goodwin, 1999), but different from how the effectiveness of traditional
advertising is evaluated, where the focus to a large extent is on various ad- or brand-related
responses (e.g., Bergkvist, 2000; Rossiter and Percy, 1997).
The question is whether measuring click-through is an appropriate method for evaluating
Internet advertising. The answer naturally depends on the goals of the advertiser. If the goal is
to attract visitors to a web-site, or some other direct response, then the effectiveness should be
evaluated by measuring the ability to generate the desired response. If the goal, on the other
hand, is to build brand awareness or brand attitude, then it is more reasonable to use measures
of ad-related and brand-related responses to measure the effectiveness of the advertising. It is,
of course, possible that an advertiser’s goals are a combination of the two types of responses
and in that case the effectiveness of the advertising should be evaluated with both types of
measures.
For Internet advertisers that have brand-related responses as one of their goals there are two
questions. First, can Internet advertising generate brand-related responses in the same way as
traditional advertising? Can web-ads be used for, for example, building brand awareness or
strengthening brand attitude? Second, can these responses be measured in a satisfactorily
manner? Traditional advertising is often evaluated with various types of surveys that measure
ad- and brand-related responses and an important question is whether this can be done for
web-advertising as well?
There is research indicating that both the first and the second of these two questions could be
answered in the affirmative. An early (in the Internet world) study indicated that banner ads
affect people’s attitudes in the same manner as traditional ads (Briggs and Hollis, 1997), and a
recent study indicated that web-advertising works according to the same hierarchy-of-effects
as traditional advertising (Bruner and Kumar, 2000). In both of these studies, advertising
1
effects were measured with traditional measures, but the surveys were administered through
web pages.
In this paper, results from five empirical studies related to Internet advertising effectiveness
will be presented. The results from these studies are in line with the earlier studies (Briggs
and Hollis, 1997; Bruner and Kumar, 2000) and, thus, support that banner ads can affect
people in a way similar to that of traditional advertising and that this can be measured.
The structure of the paper is the following. First, there is a brief theoretical background with a
review of theories of how advertising works and advertising effectiveness measurement. This
is followed by a description of the method of the empirical studies, which, in turn, is followed
by the results from the empirical studies. The final section in the paper offers a discussion and
some concluding comments.
2 Theoretical background
2.1 Theories of how advertising works
Measurement of advertising effectiveness should be based on some idea of how advertising
works. A theoretical understanding of the topic of interest is of great help during all stages of
applied marketing research (as is generally the case in research). It has been suggested that
theory should guide marketing research with respect to identification and operationalization
of key variables, selection of research design and sample, as well as analysis and
interpretation of results (Malhotra, 1996).
General theories of how advertising works tend to come in the form of hierarchy-of-effects
models. These models vary with respect to the variables that are included and the order in
which effects take place, but in all models each effect is assumed to be dependent on the
preceding effect, i.e., the included variables are related in a specific, hierarchical order (cf.
Grønhaug, Kvitastein and Grønmo, 1991; Tellis, 1998). In the well-known hierarchy-ofeffects model DAGMAR, for example, it is assumed that advertising works in the sequence
awareness, comprehension, conviction and action (cf. Scholten, 1996) and in another early
model proposed by Lavidge and Steiner (1961) the sequence was awareness, liking,
preference, conviction and purchase, which was summarized as cognition, affect and
conation.
Hierarchy-of-effects models have been criticized. According to Scholten (1996) there have
been two major criticisms. The first criticism is that the hierarchy-of-effects models have
disregarded the marketing situation in which the advertising works; the second criticism is
that effective advertising, according to the models, requires complex attitude changes in
consumers. Various alternatives to the early hierarchy-of-effects models, in which the order of
effects was cognition, affect and conation, have been suggested, e.g., that the order of effects
is cognition, conation and affect and that low-involvement advertising works without the
affect effect (cf. Smith and Swinyard, 1982). The debate about hierarchy-of-effects models is
ongoing and there is no agreed-upon model among advertising researchers or even agreement
that advertising works in a hierarchy of effects (cf. Vakratsas and Ambler, 1999).
Despite the criticism directed against them, hierarchy-of-effects models should be useful in
the context of advertising effectiveness measurement, e.g., for identifying key variables and
interpreting results, but caution is warranted. Hierarchy-of-effects models are useful as
frameworks, but a certain sequence of effects, e.g., awareness, comprehension, conviction and
2
action, should not be applied dogmatically to all situations since the sequence might vary
between different product categories, brands and situations. Further, not all effects are
applicable in all situations.
In the present context, a hierarchy-of-effects model from Rossiter and Percy (1997), the sixstep effects sequence, will be used as theoretical framework. This model is more general than
most hierarchy-of-effects models and it includes both individual and advertiser variables.
According to the six-step effects sequence, advertising works through five prior effects before
affecting the profit of a company. The steps, or effects, are the following:
1.
2.
3.
4.
5.
6.
Exposure
Processing
Communication effects and brand position
Target audience action
Sales or market share and brand equity
Profit
The first four steps are called the buyer response steps, which are the effects advertising has
on individuals. People are exposed to, or have the opportunity to see, advertising in different
media. People might pay attention to ads, or parts of them, thereby processing them to a lesser
or greater extent. As a result of processing an ad, there will be communication effects on some
people, e.g., some people might learn a brand name and others might form an opinion about a
brand. The communication effects, in turn, might lead to target audience action in the form of
some people purchasing the product. The final two steps in the six-step effects sequence are
aggregate effects on the market or company levels.
Each of the six steps is associated with different advertising effectiveness measures. Exposure
is associated with media measures, processing is mainly associated with ad-related measures,
communication effects with brand-related measures and target audience action with measures
of purchase or other behavior. The final two steps are associated with sales and market
measures and profit measures. The focus in this paper will be on measures of processing and
communication effects, which are often used to evaluate the effectiveness of advertising (cf.
Bergkvist, 2000).
2.2 Advertising effectiveness measures and tracking studies
Broadly speaking, there are two alternatives when it comes to measuring the effectiveness of
advertising campaigns. They can be evaluated by measuring certain variables before and after
a campaign or by some kind of continuous, e.g., weekly or monthly, measurement that is
carried out irrespective of any advertising campaigns. The latter is generally referred to as
tracking or brand tracking and is probably the more common choice among advertisers.
The choice of measures to use can be made more or less independent of whether measurement
will be made pre-/post campaign or in a tracking study. Both types of studies generally
include a number of intermediate effectiveness measures, e.g., advertising recall, brand recall
and brand attitude, and sometimes also media measures and sales measures. For overviews of
tracking studies and advertising effectiveness measurement see, for example, Bergkvist
(2000), Rossiter and Percy (1997) or Sutherland and Sylvester (2000).
3
In the empirical studies presented in this paper, the methodology used in the evaluation of
traditional advertising was applied to Internet advertising. The methodology of the studies
will be discussed in the next section.
3 Method
In this paper, results from five empirical studies will be presented. The Swedish marketing
research company CyberQuest carried out these studies for the banner advertising network
DoubleClick Scandinavia. Four of the studies were carried out to evaluate banner advertising
campaigns for clients of DoubleClick Scandinavia, while the fifth was made for a different
purpose, i.e., to measure brand awareness of a number of Swedish Internet brands.
In all five studies, DoubleClick’s banner ad administration system was used to draw random
samples of Internet users who visited one of the web-sites in DoubleClick's Swedish network.
Questionnaires were shown to respondents in a new window that opened automatically in the
browsers for those Internet users who were selected for the study. When respondents had
answered the questionnaire and submitted the answers the extra window closed and they were
back on the web page that they were originally surfing to. Those who did not wish to
participate in the studies had the option to close the extra window and thereby get back to the
original web page. The response rates in the studies were between 20 and 30 percent, except
for one part of the brand awareness study (the brand recall questionnaire) in which it was
lower, around 15 percent. The number of respondents varied between 166 and 236 in the
campaign evaluation studies, while it was 575 for one of the questionnaires in the brand
awareness study and 146 for the other. The respondents in four of the studies were not offered
any reward for their participation, while the respondents in one of the campaign evaluation
studies were offered the chance of winning a Red Cross lottery ticket.
In the brand awareness study two different questionnaires were used. In the first questionnaire
respondents were shown a list with brand names, in an ordinary typeface, under headings of
different product categories. For each brand name they were asked to indicate whether they
recognized it or not. Thus, the first questionnaire measured brand recognition. In the second
questionnaire subjects were asked, for the same product categories as in the first
questionnaire, to write down the brand names which came to their minds when they thought
of the product category. Six text fields for entering the brand names were provided for each
product category. Thus, the second questionnaire measured brand recall.
All campaign evaluation questionnaires contained questions on brand recognition, brandprompted ad recall and brand attitude. One of the studies also contained measures of brand
benefit beliefs (Rossiter and Percy, 1997). Brand recognition was measured in the same way
as in the brand awareness study, while brand-prompted ad recall was measured by showing a
list of brand names, under a product category heading, and asking which of the brands the
respondent had seen advertised on the Internet lately. Brand attitude was measured with a
four-point scale, following a format suggested in Rossiter and Percy (1997). Brand benefit
beliefs were measured by asking to what extent the brand in the study had certain benefits.
Responses were measured on a 7-point scale.
For a detailed description of the measurement procedure in the studies, applied in another
context, see Bergkvist (2000).
4
4 Results
4.1 Awareness of internet brands
There were between three and five brands in each product category in the brand recognition
questionnaire. With three exceptions these brands were mentioned by at least a few of those
respondents who answered the brand recall questionnaire. The share of respondents who
recognized or recalled the brands included in the brand recognition study is shown in Table 1.
Table 1
Brand Recognition and Brand Recall of Internet Brands
CD, Video and Similar
Boxman
Amazon
Ginza
CDNow
Books and Similar
Amazon
Bokus
Bokex
Internet Bookshop
Shopping/Co-shopping
Letsbuyit
BlueMarx
Sumo
Yatack
Haburi
Portals
Passagen
Spray
Torget
MSN
Financial Services
Avanza
E*Trade
HQ.SE
NordNet
Price Comparisons
Goodguy
Shopsmart
Jahaya
n
Recognition
Recall
81%
58%
79%
49%
55%
8%
41%
8%
61%
49%
8%
16%
34%
27%
1%
0%
84%
35%
42%
50%
18%
21%
6%
6%
3%
0%
96%
93%
93%
73%
42%
45%
16%
10%
39%
41%
17%
24%
10%
8%
3%
2%
31%
26%
13%
4%
1%
0%
575
146
The results in Table 1 show that there is a large difference between the share of respondents
who could recognize a brand name compared to the share that could recall it. This is natural
since recognition and recall are two different memory processes, albeit related (cf. Bagozzi
and Silk, 1983; Strong, 1912). Among other things, recall requires a greater mental effort than
recognition and it is generally expected that recall values should be lower than recognition
values. Another noteworthy feature of the results in Table 1 is that the brand awareness was
high for many of the brands, considering that they have existed for only a few years.
5
The results in Table 1 say little about whether Internet advertising influence people in the
same way as traditional advertising, since probably most of the brands in the table have used
traditional media in their advertising campaigns. The results indicate, however, that surveys
administered via the Internet can be used to measure brand awareness, both brand recall and
brand recognition, for Internet brands and, most likely, for other brands as well.
The results also raise the issue of which type of brand awareness is the more important for
Internet brands. This depends on how people find their way to a web-site when they
experience a need relevant for a certain category of web-sites. If Internet users decide which
web-site to visit on the basis of sites they can recall from their memory, then brand recall is
the relevant brand awareness measure. If Internet users, on the other hand, decide by choosing
a web-site from a list, e.g., at a portal or from their bookmarks, then brand recognition is the
relevant brand awareness measure. Since advertising should be designed differently
depending on which type of brand awareness it seeks to build, web-sites should seek to
determine whether consumers in general base their choice of sites in their category on recall
or recognition, or both.
4.2 Web-advertising effectiveness
Two of the brands in the campaign evaluation studies (the “Telecom web-site” and the
“Travel Company”) belong to well-known companies, which both use their company name in
the product brand name, while the other two are Internet brands, i.e., web-sites, with a short
history. All four brands had comparatively large budgets for their web-advertising campaigns,
in excess of SEK300, 000 (about USD31, 500), which is a large budget on the (small)
Swedish market. An uncertainty in the present context is that the extent of other advertising is
not known, except for the health site, which did not advertise in any other media than the
Internet. For the other brands, it cannot be ruled out that other advertising may have
influenced some of the results.
Table 2 shows the pre- and post-campaign results for the questions included in all the studies.
There were small differences between pre- and post-campaign for the “Travel company” (an
exception was brand attitude) and the “Recruitment web-site” brands. Moreover, some of the
differences were negative. (The results for the “Recruitment web-site” were probably
influenced by the fact that the post-campaign measurement was carried out during the
Swedish holiday season.) None of the differences for these two campaigns was statistically
significant.
The pre-/post-campaign differences were fairly large, and statistically significant, for the
“Telecom web-site” brand and the relative difference between pre- and post-campaign for the
“Health web-site” brand was fairly large, even if the absolute levels and differences were
small. (Statistical significance could not be calculated for the “Health web-site” brand, since
the raw data was not available at the time of writing.)
6
Table 2
Results from three Web-Campaign Evaluation Studies
Pre-Campaign
Post-Campaign
χ2
p
16%
99%
54%
3%
35%
96%
50%
4%
19.37
n.a.1
.46
–
.00
1
n.a.
.50
–
9%
40%
34%
1%
24%
39%
26%
3%
14.44
.06
1.93
–
.00
.81
.17
–
3%
3%
1%
1%
8%
6%
1%
1%
4.53
1.38
.27
–
.03
.24
.61
–
11%
42%
17%
0%
23%
44%
16%
2%
9.02
.25
.09
–
.00
.62
.78
–
211
184
148
161
171
232
159
215
Brand Recognition
Telecom web-site
Travel company
Recruitment web-site
Health web-site
Brand-Prompted Ad Recall
Telecom web-site
Travel company
Recruitment web-site
Health web-site
Brand Attitude (The best
service/product/site)
Telecom web-site
Travel company
Recruitment web-site
Health web-site
Brand Attitude (One of several good
services/products/sites)
Telecom web-site
Travel company
Recruitment web-site
Health web-site
n
Telecom web-site
Travel company
Recruitment web-site
Health web-site
1
More than one cell had an expected count < 5.
The results in Table 2 indicate that there was some positive pre-/post-campaign differences
for the brands that were evaluated. The results, however, do not in themselves indicate, except
for the “Health web-site” brand, that the Internet advertising was partly or entirely responsible
for these differences. Advertising in other media could have caused them. To probe deeper
into this issue, the scores on the brand attitude questions were tabulated for those who recalled
seeing the brand advertised on the Internet and those who did not. This was done for the
“Telecom web-site,” the “Travel Company” and “Recruitment web-site” brands.
Table 3 shows frequencies for the different scale-steps on the brand attitude question for
recallers and non-recallers for the “Telecom web-site,” the “Travel Company” and
“Recruitment web-site brands. For all three brands the share of respondents was higher on the
second and third scale-steps for recallers than non-recallers, while the share of respondents
who were unfamiliar with the brand was higher for non-recallers than recallers. For the fourth
scale-step (“…a below-average brand”), the share was higher for recallers than non-recallers
for the “Telecom web-site” and “Recruitment web-site” brands, while the situation was
reverse for the “Travel company” brand.
7
Table 3
Brand Attitudes for Recallers and non-Recallers of Internet Advertising
Recall of Internet Advertising
…the single best site/brand
…one of several top sites/brands
…an average site/brand
…a below-average site/brand
Don’t know the site/brand
Telecom web-site
Yes
No
18%
3%
37%
12%
12%
6%
3%
2%
30%
77%
n
2
χ
p
1
More than one cell had an expected count < 5.
Travel company
Recruitment web-site
Yes
No
Yes
No
9%
4%
2%
1%
50%
43%
24%
10%
34%
23%
16%
11%
1%
4%
0%
2%
7%
25%
59%
75%
137
1
n.a.
1
n.a.
250
1
n.a.
1
n.a.
157
1
n.a.
1
n.a.
The results in Table 3 could indicate that Internet advertising in some cases makes people
aware of a brand and increase the share of people with a positive attitude. Some caution is
warranted, however, as the causal relationship could for some individuals be the reverse, i.e.,
knowing of and liking a brand causes a person to notice or think that they have noticed
advertising for that brand (cf. Alba and Hutchinson, 1987, pp. 413-414).
To take the analysis one step further, the brand benefit beliefs measured in the campaign
evaluation for the “Recruitment web-site” brand were analyzed in the same way as the brand
attitudes. Table 4 shows the share of respondents who indicated one of the three top scalesteps on the seven-point scale used to measure brand benefit beliefs for the five beliefs that
were included in the questionnaire.
The results were similar to those obtained in the analysis of brand attitude. The shares of
positive responses were higher for all benefits, even if only one of the differences was
statistically significant.
Table 4
Brand Benefit Beliefs for the Recruitment Web-Site for Recallers and nonRecallers of Internet Advertising
Recall of Internet Advertising
Benefit 1
Benefit 2
Benefit 3
Benefit 4
Benefit 5
Yes
34%
36%
32%
32%
38%
No
33%
34%
27%
15%
34%
χ
.01
.05
.27
3.59
.18
2
p
.94
.83
.61
.06
.67
n=93-95
5 Discussion and concluding remarks
The results presented in this paper indicate that web advertising influence people in the same
way as advertising in traditional media and that this can be measured. This means that Internet
advertising could get less focused on behavioral measures, e.g., click-through, and include
brand-related goals in campaign planning.
An increased focus on brand effects is likely to expand the total Internet advertising market.
Many advertisers are more concerned with brand effects than with obtaining any kind of
direct response and these would be more likely to spend their advertising budgets on the
Internet if the medium works with brand advertising. This means that web-sites, or banner
8
advertising networks, that expect to earn their revenue from advertising, should try to
demonstrate, e.g., with advertising effectiveness research, to advertisers that web advertising
can, in fact, have brand effects. This should entail less focus on click-through in the
evaluation of campaigns.
There are limitations to the studies presented in this paper. They were made as commercial
research projects and did not have as their main purpose to investigate the effectiveness of
Internet advertising in a broader sense. The main limitation is that the effects of other
advertising could not be properly controlled for. It would also have been desirable to control
for the effects of brands and product categories, which tend to have different responsiveness
to advertising (cf. Lodish, Abraham, Kalmenson, Livelsberger, Lubetkin, Richardson and
Stevens, 1995). The results from the studies should thus be read with some caution.
Given the brief history of Internet advertising and the limited amount of published research, it
would be desirable with more research in this area. One possible step would be to carry out
more studies similar to those presented in this paper in order to get more data, which would
allow for more thorough analysis. Another approach would be to conduct laboratory
experiments in which the effects of product category, brand name, advertising in other media,
and so forth, were controlled for. This would allow for rigorous tests of when and to what
extent web ads influence brand perceptions.
References
Alba, Joseph W. and J. Wesley Hutchinson (1987), “Dimensions of Consumer Expertise,”
Journal of Consumer Research, (4), 411-454.
Bagozzi, Richard P. and Alvin J. Silk (1983), “Recall, Recognition, and the Measurement of
Memory for Print Advertisements,” Marketing Science, (2), 95-134.
Bergkvist, Lars (2000), Advertising Effectiveness Measurement: Intermediate Constructs and
Measures. Stockholm: EFI, The Economic Research Institute. (Published doctoral
thesis.).
Briggs, Rex and Nigel S. Hollis (1997), “Advertising on the Web: Is There Response before
Click-Through?,” Journal of Advertising Research, (2), 33-45.
Bruner, Gordon C. and Anand Kumar (2000), “Web Commercials and Advertising Hierarchyof-Effects,” Journal of Advertising Research, (1/2), 35-42.
Goodwin, Tom (1999), “Measuring the effectiveness of online marketing,” Journal of the
Market Research Society, (4), 403-406.
Grønhaug, Kjell, Olav Kvitastein and Sigmund Grønmo (1991), “Factors Moderating
Advertising Effectiveness as Reflected in 333 Tested Advertisements,” Journal of
Advertising Research, (5), 42-50.
Hofacker, Charles F. and Jamie Murphy (1998), “World Wide Web banner advertisement
copy testing,” European Journal of Marketing, (7/8), 703-712.
Hoffman, Donna L. and Thomas P. Novak (1996), “Marketing in Hypermedia ComputerMediated Environments: Conceptual Foundations,” Journal of Marketing, (3), 50-68.
Hoffman, Donna L. and Thomas P. Novak (2000), “How to Acquire Customers on the Web,”
Harvard Business Review, (3), 179-188.
Lavidge, Robert J. and Gary A. Steiner (1961), “A Model for Predictive Measurements of
Advertising Effectiveness,” Journal of Marketing, (6), 59-62.
Lodish, Leonard M., Magid Abraham, Stuart Kalmenson, Jeanne Livelsberger, Beth
Lubetkin, Bruce Richardson and Mary Ellen Stevens (1995), “How T.V. Advertising
9
Works: A Meta-Analysis of 389 Real World Split Cable T.V. Advertising
Experiments,” Journal of Marketing Research, 125-139.
Malhotra, Naresh K. (1996), Marketing Research. An Applied Orientation. Upper Saddle
River, New Jersey: Prentice Hall Inc.
Rossiter, John R. and Larry Percy (1997), Advertising Communications & Promotion
Management. International Edition: McGraw-Hill.
Scholten, Marc (1996), “Lost and Found: The Information-Processing Model of Advertising
Effectiveness,” Journal of Business Research, (2), 97-104.
Smith, Robert E. and William R. Swinyard (1982), “Information Response Models: An
Integrated Approach,” Journal of Marketing, (1), 81-93.
Strong, Edward K. , Jr. (1912), “The Effect of Length of Series Upon Recognition Memory,”
Psychological Review, (1), 447-462.
Sutherland, Max and Alice K. Sylvester (2000), Advertising and the Mind of the Consumer.
London: Kogan Page Limited.
Tellis, Gerard J. (1998), Advertising and Sales Promotion Strategy. Reading, MA: Addison
Wesley Longman, Inc.
Vakratsas, Demetrios and Tim Ambler (1999), “How Advertising Works: What Do We
Really Know?,” Journal of Marketing, (1), 26-43.
10