A
A Categorization and Competitive Analysis of Web-Based Financial Information
Aggregators
By
Gregory C. Harman
Submitted to the
Department of Electrical Engineering and Computer Science
in Partial Fulfillment of the Requirements for the Degree of
Master of Engineering in Electrical Engineering and Computer Science
at the Massachusetts Institute of Technology
February 16, 2001
Copyright @ 2001 Gregory C. Harman. All rights reserved.
The author hereby grants to M.I.T. permission to reproduce and
distribute publicly paper and electronic copies of this thesis
and to grant others the right to do so.
Author
Department of Electrical Engineering and Computer Science
February 16, 2001
Certified
by
Stuart Madnick
Thesis Supervisor
Accepted
BARKER
by
-
Arthi-C. Smith
Chairman, Department Committee on Graduate Theses
MASSACHUSETTS INSITUTE
OF TECrFhr)jWi-,V
JUL I
1001
A Categorization and Competitive Analysis of Web-Based Financial Information
Aggregators
By
Gregory C. Harman
Submitted to the
Department of Electrical Engineering and Computer Science
February 16, 2001
In Partial Fulfillment of the Requirements for the Degree of
Master of Engineering in Electrical Engineering and Computer Science
Abstract
Web-based aggregators use several business and technology models to operate. The business models are
identified as: brokerages, merchants, advertisers, infomediaries, affiliates, and subscription-based services.
The technology models are identified as: real-time agents, spiders, and manual information entry. The
legality of unauthorized aggregation on the web is currently being decided by the American judicial system
in a variety of ongoing cases. A sampling of major web-based financial aggregators is analyzed and the
business and technology models being used are identified.
Thesis Supervisor: Stuart Madnick
Title: John Norris Maguire Professor of Information Technology & Leaders for Manufacturing Professor of
Management Science
2
Table of Contents
A b strac t ........................................................................................................................................................... 2
Table of Contents ............................................................................................................................................ 3
T ab le o f F ig u re s .............................................................................................................................................. 6
1 . In tro d u ctio n ............................................................................................................................................ 7
2.
Business M odels ..................................................................................................................................... 9
2 .1 C riteria .................................................................................................................................................. 9
2 .1.1
Pro fitab ility ........................................................................................................................... 10
2.1.2
Partnerships ........................................................................................................................... 10
2.1.3
Availability of Technology ................................................................................................... 10
2.1.4
Sustainability ........................................................................................................................ 10
2.1.5
Profitability Potential ............................................................................................................ 10
2.1.6
Competition .......................................................................................................................... 11
2 .2 B rok erag e ............................................................................................................................................ 1 1
2.2.1
Profitability ........................................................................................................................... 12
2.2.2
Partnerships ........................................................................................................................... 12
2.2.3
Availability of Technology ................................................................................................... 12
2.2.4
Sustainability ........................................................................................................................ 12
2.2.5
Profitability Potential ............................................................................................................ 13
2.2.6
Competition .......................................................................................................................... 13
2.2.7
Sum m ary of Criteria ............................................................................................................. 13
2 .3 A d v ertisers .......................................................................................................................................... 14
2 .3 .1
P ro fitab ility ........................................................................................................................... 14
2.3.2
Partnerships ........................................................................................................................... 14
2.3.3
Availability of Technology ................................................................................................... 15
2.3.4
Sustainability ........................................................................................................................ 15
2.3.5
Profitability Potential ............................................................................................................ 15
2.3.6
Competition .......................................................................................................................... 15
2.3.7
Sum m ary of Criteria ............................................................................................................. 15
2.4 Infornediary ........................................................................................................................................ 16
2.4.1
Profitability ........................................................................................................................... 16
2.4.2
Partnerships ........................................................................................................................... 17
2.4.3
Availability of Technology ................................................................................................... 17
2.4.4
Sustainability ........................................................................................................................ 17
2.4.5
Profitability Potential ............................................................................................................ 17
2.4.6
Competition .......................................................................................................................... 17
2.4.7
Sum mary of Criteria ............................................................................................................. 18
2 .5 M erch an t ............................................................................................................................................. 18
2.5.1
Profitability ........................................................................................................................... 18
2.5.2
Partnerships ........................................................................................................................... 19
2.5.3
Availability of Technology ................................................................................................... 19
2.5.4
Sustainability ........................................................................................................................ 19
2.5.5
Profitability Potential ............................................................................................................ 19
2.5.6
Competition .......................................................................................................................... 19
2.5.7
Sum m ary of Criteria ............................................................................................................. 20
2 .6 A ffiliate ............................................................................................................................................... 2 0
2.6.1
Profitability ........................................................................................................................... 20
2.6.2
Partnerships ........................................................................................................................... 21
2.6.3
Availability of Technology ................................................................................................... 21
2.6.4
Sustainability ........................................................................................................................ 21
2.6.5
Profitability Potential ............................................................................................................ 21
2.6.6
Competition .......................................................................................................................... 21
2.6.7
Sum mary of Criteria ............................................................................................................. 22
2.7 Subscription ........................................................................................................................................
2.7.1
Profitability ...........................................................................................................................
2.7.2
Partnerships...........................................................................................................................
A vailability of Technology ..............................................................................................
2.7.3
Sustainability ........................................................................................................................
2.7.4
2.7.5
Profitability Potential............................................................................................................
Com petition ..........................................................................................................................
2.7.6
2.7.7
Sum m ary of Criteria .............................................................................................................
3.
Aggregation Technologies....................................................................................................................
3 .1 C rite ria ................................................................................................................................................
3.1.1
Ease of Implem entation .....................................................................................................
3.1 .2
Durability ..............................................................................................................................
Resistance to Blockage .....................................................................................................
3.1.3
Accuracy of Inform ation...................................................................................................
3.1.4
3.1.5
Quantity of Inform ation .....................................................................................................
Target Friendliness ...............................................................................................................
3.1.6
3.2 Real-Tim e A gent ................................................................................................................................
3.2.1
Ease of Im plem entation .....................................................................................................
26
26
26
26
26
27
27
3.2.2
Durability..............................................................................................................................
27
3.2.3
3.2.4
3.2.5
3.2.6
3.2.7
3.2.8
Resistance to Blockage .....................................................................................................
Accuracy of Inform ation...................................................................................................
Quantity of Inform ation .....................................................................................................
Target Friendliness ...............................................................................................................
Sum m ary of Criteria .............................................................................................................
Real-W orld Exam ples...........................................................................................................
28
28
28
29
29
30
3 .3 S p id er..................................................................................................................................................
4.
22
22
23
23
23
23
24
24
25
25
25
31
3.3.1
3.3.2
3.3.3
3.3.4
3.3.5
3.3.6
3.3.7
Ease of Im plem entation .....................................................................................................
Durability ..............................................................................................................................
Resistance to Blockage .....................................................................................................
Accuracy of Inform ation...................................................................................................
Quantity of Inform ation .....................................................................................................
Friendliness to Target .......................................................................................................
Sum m ary of Criteria .............................................................................................................
33
33
34
34
34
34
35
3.3.8
Real-W orld Exam ples...........................................................................................................
35
3.4 M anual Entry ......................................................................................................................................
3.4.1
Ease of Im plem entation .....................................................................................................
37
38
3.4.2
Durability ..............................................................................................................................
38
3.4.3
3.4.4
3.4.5
3.4.6
3.4.7
Resistance to Blockage .....................................................................................................
Accuracy of Inform ation...................................................................................................
Quantity of Inform ation .....................................................................................................
Friendliness to Target .......................................................................................................
Sum m ary of Criteria .............................................................................................................
38
38
38
39
39
3.4.8
Real-W orld Exam ples...........................................................................................................
39
Com m ercial Applications .....................................................................................................................
4.1 Loan Interest Rates .............................................................................................................................
4.1.1
Business M odel Usage..........................................................................................................
4.1.2
Technology M odel Usage .................................................................................................
4.2 Insurance Rates...................................................................................................................................
4.2.1
Business M odel Usage..........................................................................................................
4.2.2
Technology M odel Usage .................................................................................................
4.3 Investm ent Rates.................................................................................................................................
4.3.1
Business M odel Usage..........................................................................................................
41
42
42
43
44
44
44
45
45
4.3.2
Technology M odel Usage .................................................................................................
46
4.4 Consum er Product Inform ation .......................................................................................................
4.4.1
Business M odel Usage..........................................................................................................
46
46
4.4.2
Technology M odel Usage.................................................................................................
47
4
4 .5 A irfa re.................................................................................................................................................
4.5.1
Business M odel Usage..........................................................................................................
4.5.2
Technology M odel Usage .................................................................................................
4 .6 A u ctio n s ..............................................................................................................................................
4.6.1
Business M odel Usage..........................................................................................................
Technology M odel Usage .................................................................................................
4.6.2
4.7 Frequent Flier M iles ...........................................................................................................................
4.7.1
Business M odel Usage ..........................................................................................................
4.7.2
Technology M odel Usage .................................................................................................
4.8 Account Consolidation .......................................................................................................................
4.8.1
Business M odel Usage..........................................................................................................
4.8.2
Technology M odel Usage .................................................................................................
4.9 Auction Consolidation ........................................................................................................................
4.9.1
Business M odel Usage ..........................................................................................................
4.9.2
Technology M odel Usage .................................................................................................
5.
Legal Issues Surrounding Aggregation...............................................................................................
5.1 The Issues ...........................................................................................................................................
5.1.1
Ownership of Inform ation ................................................................................................
5.1.2
Copyright Infringem ent .....................................................................................................
48
48
48
49
49
50
50
50
51
52
52
53
53
53
54
55
55
55
55
5.1.3
Im plied Contracts..................................................................................................................
5.1.4
Advertiser Contracts .............................................................................................................
5.2 Solutions .............................................................................................................................................
5.2.1
Four Law s of W eb Robotics ..............................................................................................
5.2.2
The Six Com m andm ents for Robot Operators....................................................................
56
56
56
56
58
5.3 Legal Rulings......................................................................................................................................
5.3.1
Ebay v. Bidder's Edge ..........................................................................................................
59
59
5.3.2
N ew s Index v. The Sunday Tim es .....................................................................................
5.3.3
R.I.A .A. v. N apster ...............................................................................................................
5.3.4
Ticketm aster v. Tickets.com ..............................................................................................
5.4 Future D irection..................................................................................................................................
6.
Conclusion ............................................................................................................................................
Appendix A : Results of Com pany Categorization Grouped by M odel .....................................................
Technology M odels ..................................................................................................................................
Business M odels .......................................................................................................................................
Appendix B: Categorization of M ajor Online Aggregators......................................................................
References ....................................................................................................................................................
59
60
60
61
62
63
63
65
67
95
5
Table of Figures
Figure
1 Rating of business models vs. specific criteria............................................................................
9
Figure 2 Summary of Broker criteria............................................................................................................
Figure 3 Summary of Advertiser criteria...................................................................................................
Figure 4 Summary of Infomediary criteria .................................................................................................
Figure 5 Summary of M erchant criteria ...................................................................................................
Figure 6 Summary of Affiliate criteria .....................................................................................................
Figure 7 Summary of Subscription criteria...............................................................................................
Figure 8 Rating of technology models vs. specific criteria........................................................................
Figure 9 Summary of Real-Time Agent criteria........................................................................................
Figure 10 Example state-set for search discussion. ..................................................................................
Figure 11 Summary of Spider criteria .......................................................................................................
Figure 12 Summary of M anual Entry criteria............................................................................................
Figure 13 Current Aggregation M arkets...................................................................................................
Figure 14 Loan Interest Rates - Business M odel Usage .........................................................................
Figure 15 Loan Interest Rates - Technology Usage M odels ...................................................................
Figure 16 Insurance Rates - Business M odels .........................................................................................
Figure 17 Insurance Rates - Technology M odel Usage ............................................................................
Figure 18 Investment Rates - Business M odel Usage..............................................................................
Figure 19 Investment Rates - Technology M odel Usage ..........................................................................
Figure 20 Consumer Product Information - Business M odel Usage........................................................
Figure 21 Consumer Product Information - Technology M odel Usage....................................................
Figure 22 Airfare - Business M odel Usage ..............................................................................................
Figure 23 Airfare - Technology M odel Usage .........................................................................................
Figure 24 Auctions - Business M odel Usage ............................................................................................
Figure 25 Auctions - Technology M odel Usage .......................................................................................
Figure 26 Frequent Flier M iles - Business M odel Usage..........................................................................
Figure 27 Frequent Flier M iles - Technology M odel Usage.....................................................................
Figure 28 Account Consolidation - Business M odel Usage......................................................................
Figure 29 Account Consolidation - Technology M odel Usage.................................................................
Figure 30 Auction Consolidation - Business M odel Usage .....................................................................
Figure 31 Auction Consolidation - Technology M odel Usage ................................................................
Figure 32 Real-Time Agent Usage ...............................................................................................................
13
16
18
20
22
24
25
30
32
35
39
42
42
43
44
44
45
46
46
47
48
48
49
50
50
51
52
53
53
54
63
Figure 33 Spider Usage ................................................................................................................................
63
Figure
Figure
Figure
Figure
Figure
Figure
Figure
64
65
65
65
66
66
66
34
35
36
37
38
39
40
M anual Entry Usage.....................................................................................................................
Brokerage Usage ..........................................................................................................................
Advertiser Usage ..........................................................................................................................
Infomediary Usage .......................................................................................................................
M erchant Usage............................................................................................................................
Affiliate Usage .............................................................................................................................
Subscription Usage.......................................................................................................................
6
1. Introduction
The advent of the Internet has created a deluge of information on almost every
topic. Since the (commercial) Internet was largely developed under strong capitalism, a
great deal of this information deals with commercial enterprises. Much of this
information is related to direct sales, or to the research and ownership of equities relating
to these sales. Information about competing companies can be obtained. Some websites,
known as aggregators, make a business of compiling this information and presenting the
comparisons to their users.
Historically, financial aggregation services were available only to the rich by
investment advisers, private bankers, etc.' some mass-marketed software has been
available since the advent of personal computers, bringing aggregation abilities to the
general public, but this software was difficult to use. Only with the advent of the web has
financial aggregation been a viable option for the majority of consumers.
This paper is a critical analysis of those aggregators that compile financial
information on the web. It will attempt to define and then analyze the merits of several
different business and technological models used by aggregators today. It will also
provide an overview of the specific commercial markets in which aggregators are
competing and the current state of the legalities that concern aggregation. Finally, it will
contain an appendix listing most of the major financial aggregators in business today, and
describing the business and technological models to which they subscribe.
In this paper we will define the term aggregator with the following, taken from
Madnick et al. 2:
An aggregator is an entity that transparentlycollects and analyzes
information from different data sources. In the process, the aggregator
resolves the semantic and contextual differences in the information.
As the definition suggests, there are two characteristics specific to an
aggregator.
I.
Transparency - There are two aspects to transparency. First, the data
sources cannot differentiate an aggregator from a normal user who is
accessing the information. Consequently, data sources cannot deny
7
access to an aggregator. Second, the aggregator resolves the contextual
differences to allow for comparison of equals.
2.
Analysis - Instead of simply presenting the data as-is, the aggregator
synthesizes value-added information based on post-aggregation
analysis.
It is important to note that, under our definition, search engines, such as
Lycos and Alta Vista, and personalized Web portals, such as My
Netscape or My Yahoo, are not aggregators. Similarly, Web-based
malls, category e-store, or community-based Web sites also do not fit
under this category. Although these Web sites amass different
information, little is done to integrate and analyze that information.
This paper will focus specifically on aggregators that gather and display their
information primarily through the World Wide Web, and specifically on those that are
concerned with financial information. We will, however, take a loose definition of what
information qualifies as "financial." In addition to that information which pertains to the
traditional definition: information about markets, equities, interest rates, etc., we will also
include information concerning insurance rates, auctions, online relationship aggregation,
airfare, and the price of consumer products.
8
2. Business Models
2.1 Criteria
Every (successful) company makes money. The way in which they do this is
referred to as their business model. We will define business model as "a company's
value proposition for making money." 3 In this section, we will define and discuss
different business models that aggregation companies can apply to derive profit from
their business.
First, a set of criteria will be established to provide an effective way to evaluate
and compare these different business models. Next, several possible business models
will be defined and then analyzed, based on the established criteria. These models have
been adapted from "Business Models on the Web" by Professor Michael Rappa of NC
.4
State University .
The results of this analysis are summarized below in Figure 1. Note that while
this data presents certain models as superior to others in the various criteria that each
model is still best-suited for certain markets due to their fundamental differences. These
differences are discussed in this section, and the best applications for each model are
discussed later in this paper.
Profitability
Partnerships
Availability
of
Technology
Sustainability
Brokerage
+
Infomediary
+
+
+
+
Affiliate
Competition
+
Advertiser
Merchant
Profitability
Potential
+
+
+
Subscription
Figure 1 Rating of business models vs. specific criteria
9
2.1.1
Profitability
The first criterion will be time to profitability. No matter how much initial
funding a company receives, it is still a fixed amount that will eventually be used up.
The company must eventually turn a profit and support itself. And, ultimately, that is the
purpose of operating the company! We must also take into account the start-up cost of
the business. The more in-debt a company is and the greater the start-up cost, the longer
it will take to become profitable.
2.1.2 Partnerships
In order to most efficiently tap all of its target markets, a company may require
the services of another company that has access to those markets. These partnerships can
be a great boon to both companies, but a partnership is not completely under the control
of either company; therefore, by entering into a partnership, a company introduces an
element of risk. The number of (and difficulty to acquire) required partnerships will be
our second criterion.
2.1.3 Availability of Technology
Our third criterion will be the availability of necessary technology. In order to
build its product or service, a company will need to use certain technologies.
Technologies able to achieve the given task may not yet be developed sufficiently for
commercial application, or may require licensing fees that outweigh the profit to be made
by their application.
2.1.4 Sustainability
The fourth criterion will be sustainability. In order for a company to truly provide
value, its business model must be sustainable over a long period of time. With the
current pace of technological innovation, an unsustainable business model may become
outdated and no longer work before a company has even turned a profit!
2.1.5 Profitability Potential
The fifth criterion will be profitability potential. A company's business model
must be capable of providing an adequate return on the capital investment. Certainly, this
10
return must be more than that same capital could return in a less risky investment; but
since a start-up company is so risky (only 1 in 10 new companies survive to
profitability),5 the return must be that much greater.
2.1.6 Competition
The final criterion will be the level of competition. Every market is of limited
size, and several companies will compete to capture as much of the given market as
possible. 6 In fact, there is generally only room for two or three companies to be
successful in any given market.7 In addition, the "first mover advantage" is often
considered to be a benefit for companies. For example, Yahoo, the first major portal has
been extremely successful, and is still the top portal site on the web. However, being the
first mover certainly does not guarantee success. Juno, for example, was the first
company to offer free email, but Hotmail is by far the leading provider of free email
today. In conclusion, the timing of a company's entry to market can be quite important,
but being the first is not always a guarantee for success.
2.2 Brokerage
Our first business model is the Brokerage Model. Brokers connect buyers and
sellers and facilitate transactions. The business models suggested by the buzzwordabbreviations "B2B" (Business to Business), "B2C" (Business to Consumer), "C2C"
(Consumer to Consumer), "B2E" (Business to Enterprise), etc. are Brokers. Rappa lists
nearly dozen subcategories of the Brokerage Model. The Buyer Aggregator, Auction
Broker, and Search Agent subcategories are especially relevant to the topic of financial
information aggregators .
A Buyer Aggregator, such as MySimon.com, allows users to find the best buy on
a particular item regardless of the seller. It scans the major Internet retailers and presents
the user with a price comparison. The user can then connect to the dealer that offers them
the best value. An Auction Broker, such as AuctionWatch or Bidder's Edge, provides the
same service across online auctions instead of online retailers.
11
2.2.1
Profitability
A Broker provides value in the form of arranged partnerships and acts as a
medium for communication between companies, between consumers, or between
companies and consumers. Before it can begin to derive profit from these partnerships,
the Broker must first accumulate sufficient quantity of users within its site to provide the
best partnerships possible, and to foster a sufficient community to maintain the creation
of these relationships. The speed at which these communities can be formed is critical. It
is fairly straightforward that a broker cannot bring in revenue until there are users on its
site. If the broker cannot bring in revenue before it runs out of funding, then it is out of
business.
2.2.2 Partnerships
While brokers are not completely dependent on partnerships to further their
business, they can certainly use them to their advantage. In the case of a Buyer
Aggregator, for example, the aggregator may not need to partner with a seller to rate its
information, but may find it much easier to do so. If the seller freely provides
information about the its products, then the Buyer Aggregator won't need to use an
aggregating agent to retrieve this information. In addition, this partnership will alleviate
any possible legal concerns (see later discussion of legal issues).
2.2.3 Availability of Technology
The technology necessary for a brokerage spans the entire range of depth. While
some brokers can be little more than electronic storefronts, others, such as automatic
aggregators, can push the limits of technological feasibility. In order to retrieve its
information, an aggregator may have to employ various forms of spidering technology,
which will be discussed later.
2.2.4 Sustainability
Brokers connect buyers and sellers and, as such, are "middle-men." The
Brokerage model seems sustainable in the short-run, but as middlemen, Brokers must
12
continue to provide value or it is conceivable that they could be driven out of the
economy leaving only direct relationships between buyers and sellers. Today, Brokers
add cost to the supply chain while providing the service of connecting sellers and buyers.
Technology could eventually supply this need more readily, forcing the Brokers to find
new ways to add value to the supply chain. For example, recent versions of Apple's
MAC OS, discussed later, provide an automatic Buyer Aggregator.
2.2.5 Profitability Potential
A Broker wants to be in the middle of every transaction. This in itself can be
enormous, but a Broker also has the potential to apply other business models as well,
creating many sizable sources of income. For example, almost every website, including
Brokers, advertise to their users. Many of these advertisements take the form of clickthrough banners, providing a means for affiliate partnerships and marketing information
sales (the Infomediary model).
2.2.6
Competition
There is a great deal of competition for Brokers. Because of the high profitability
potential, everyone wants to be in this position. The partnerships that a brokerage has
greatly define its worth, and if one broker has an exclusive partnership in a given market,
then all competitors will not have access.
2.2.7 Summary of Criteria
Brokers face heavy start-up costs and intense competition. They are in some
danger of being replaced by new technology if they do not continue to provide a value
that the technology cannot. However, the potential profit for a Broker is enormous.
Profitability
Partnerships Availability
of
Sustainability
Profitability
Potential
Competition
Technology
Figure 2 Summary of Broker criteria
13
2.3 Advertisers
The next business model is the Advertising Model. This model follows that of
traditional media broadcasting9 . The website offers content - usually for free - and thus
attracts a large user base. Advertisers then pay for ads viewed by these users. Portals of
all types (generalized, personalized, and vertical), incentive sites, and free sites that give
a service, such as Internet access or hardware, subscribe to the Advertising Model10 .
Yahoo is perhaps the most prominent advertiser, as most of its revenue is derived from
advertisement.
2.3.1
Profitability
Advertisers must accumulate a significant user base before they have sufficient
hits to attract major clients. Once they have these clients, they must wait for the revenues
to trickle in, as the per-view fees are very small. Advertisers are generally completely
internet-based, and so have few physical expenses; most of their initial funding will go to
advertisement for themselves. The faster they can attract a large user base, the faster they
can bring in significant revenue, which will shorten their time to profitability. Also, the
startup costs can differ greatly among different companies, depending on their chosen
method of attracting a user base.
For example, a free Internet service, such as Free! or
NetZero, must support the expenses of an ISP without the immediate reimbursement by
the consumers that traditional ISPs enjoy. An alumni website, however, requires very
little cost to maintain -just the production and hosting of the web site, and comes with an
instant user base.
2.3.2 Partnerships
Since their sole source of income is advertisements paid for by clients, advertisers
are completely dependent upon their partnerships with these clients for profit. Luckily,
there is an overwhelming demand for targeted advertisement, which in 1998 had
revenues of $2 billion' 2 and is growing at 200% per year.' 3
14
2.3.3 Availability of Technology
Advertising requires very little unique technology. Most online advertising takes
the form of banner advertisements that are displayed to the side of a screen containing
content. The number of page views may be measured with a simple counter. Though the
methods of advertising may change with new technology, the need for companies to
inform potential customers of their value will not change.
2.3.4 Sustainability
The need for advertisement depends only on the strength of the economy,
specifically the number of businesses with products and services to sell. The more
businesses there are, the more advertising that will be necessary; the need for advertising
for existing businesses will not change. Unfortunately for advertisers, many of them
have fallen upon hard times since the NASDAQ crash in early 2000. Very few
advertisers are successfully turning a profit, and many analysts think that the future is
dim for all but the largest pure advertisers.
2.3.5 Profitability Potential
High-profile advertisers with many hits to their websites can make a good profit,
since the fees are generally based on the number of views an advertisement receives, but
ultimately there is a defined limit to the number of advertisements that a site can display.
2.3.6 Competition
Almost every commercial website utilizes advertisement, but companies are
willing to advertise in multiple locations. In fact it is to the client company's advantage
to do so, so the only limitation is the budget that the company has allocated to
advertising. However, the low barriers to entry (very little startup-cost and easily
available technology) have created a huge number of advertisers. In addition to the pure
advertisers, most other online businesses still derive some profit from advertisement.
2.3.7 Summary of Criteria
Advertising businesses are technologically simple to enact, and are likely to be
sustainable well into the foreseeable future, dependent on the markets in general.
15
Competition is heavy because of the low barriers to entry, however these companies may
have significant start-up costs to repay on an initially slow income.
Profitability
Partnerships Availability
of
Sustainability
Profitability
Potential
Competition
Technology
Figure 3 Summary of Advertiser criteria
2.4 Infomediary
The third business model is the Infomediary Model. Infomediaries record data
about their customers' browsing and spending habits. This information is then used in
focused marketing campaigns, and often sold to other firms. Sites in which users rate and
recommend products, services, and companies, and content-based sites that are free but
require registration fall under the Infomediary Model'.
DoubleClick is a well-known Infomediary. Privacy issues surrounding the
Infomediary model and focused specifically on DoubleClick have come to light recently.
What is and is not acceptable use of the marketing information has come into question.
DoubleClick used cookies to track users' online activities, sometimes without the explicit
consent of those users.' 5 Privacy advocates claim that this is a deceptive practice that
invades the privacy of the users. DoubleClick says that it provides an opt-out system that
gives fair warning to the users; again, this is criticized by the privacy advocates as being
confusing and therefore not providing "genuine informed consent."
2.4.1
Profitability
Before an Infomediary can sell its market data, it must first collect that data. This
can be a time-consuming process, as it waits for users to contribute (regardless of
whether or not it is voluntary) their information. Once this information is collected, it
may be analyzed to provide more value. An Infomediary does not have significant
physical start-up costs, and does not generally spend large amounts on consumer
advertising; therefore the start-up costs are relatively low.
16
2.4.2 Partnerships
If it operates its own site, an Infomediary generally does not require any specific
partnerships beyond the standard buyer/seller relationship that it has with its clients.
However, some Infomediaries enable other content-oriented sites to collect consumer
data, in which case the partnerships become all-important. Therefore an Infomediary
spans the entire range of liability in regard to partnership requirements.
2.4.3 Availability of Technology
Infomediaries must keep track of customer profiles and track their movements on
the Internet. This is not particularly difficult from a technological standpoint - users are
logged into the website, and their choices are recorded. However, this can result in an
enormous mass of information that must be analyzed to be useful. Luckily for the
Infomediaries, statistical analysis software is readily available.
2.4.4 Sustainability
Though technologies and markets change, companies always need to know and to
re-assess whom their customers are. The technologies utilized by Infomediaries will
certainly change, but the basic need for their services will not.
2.4.5 Profitability Potential
Infomediaries derive profit by selling relevant market information to companies.
Companies always need this information, but there are only so many companies to go
around, so the market for this business model is limited in a similar way to Advertisers.
2.4.6 Competition
There is no shortage of data to collect, but Infomediaries must compete with each
other to sell their information to clients as client companies can afford only so much
market data. This data can only be as varied as the number of markets available, and
additional data simply hones the accuracy of the statistical analysis. There is a point at
which more accuracy is not useful, therefore the space is defined by the number of
markets, rather than the amount of data available. Competition for these limited spaces is
not light.
17
2.4.7 Summary of Criteria
Infomediaries fill a real need of most other businesses, and this need will not
disappear, leaving a very sustainable future. The profit can come slowly, but the start-up
costs are also relatively small.
Profitability
Partnerships Availability
of
Technology
Sustainability
Profitability
Potential
Competition
Figure 4 Summary of Infomediary criteria
2.5 Merchant
Our next business model is the Merchant Model. Merchants on the Internet
function just as offline merchants, or "brick-and-mortar" companies, do. Many larger
retailers have online presences, but Internet-only businesses, "e-tailers," also compete
with them. Merchants in every offline market have moved to compete online, and new
exclusively online companies have sprung up to compete with them. Perhaps the most
notable competition has been between the online bookseller Amazon.com and its "brickand-mortar" counterpart, Barnes and Noble, which has an online presence at bn.com.
2.5.1
Profitability
A merchant sells a product, and therefore makes a profit on each sale
immediately, regardless of the size of that profit. However, a merchant has significant
startup costs to address. In addition to the costs of building and maintaining a web site,
selling a physical product requires either a warehouse or manufacturer, and distribution
must also be taken into account. This very significant distribution cost is one that is not
incurred by Infomediaries or Advertisers. This cost includes supply chain expenses and
extra customer service staff.16 Since merchants must rely on customers to find them,
advertising costs are also quite significant.
18
2.5.2 Partnerships
A Merchant may advertise, create an affiliate program, or provide content to draw
customers, but, just like the traditional offline merchant, its primary method of doing
business is to sell products directly to consumers. In general, the only partnerships a
merchant, either online or off, is likely to have are those with their distributors and
manufacturers, although some merchants may benefit from working together to present
product packages to their customers.
2.5.3 Availability of Technology
Merchants have fairly straightforward technical operations from a purchase
standpoint. Customers select items, which are stored in a "shopping cart," and are then
"checked out" by placing a charge on the customer's credit card. However, once the
order is placed, the items must be appropriately retrieved and shipped to them from the
company's warehouse. This process can be an operational nightmare, and has been the
weak spot for many companies following this model. For example, one of the biggest
problems for Amazon.com is its product distribution, and it is considering outsourcing
the entire operation.' 7
2.5.4 Sustainability
The role of a Merchant, selling goods and services to consumers, is quite stable.
As long as currency exists, people will need to exchange their currency for goods and
services.
2.5.5 Profitability Potential
The profitability potential of a merchant online is similar to that of an offline
merchant, with the added advantage of having direct access to a non-localized customer
base. However, distribution costs can eat into profits, as can the increased amount of
advertising necessary.
2.5.6 Competition
Online merchants must compete with each other to provide the lowest prices and
the greatest values, as do their offline counterparts. Online, these merchants compete
19
against every other merchant in their market segment regardless of location, as location
no longer separates markets. In addition, the entry costs can be lower for an online
merchant than for a standard "bricks-and-mortar" merchant, and as a result many upstart
online-only merchants have sprung up to compete as well.
2.5.7 Summary of Criteria
Merchants have significant start-up costs, but also have a significant revenue
stream. They supply their own products, and therefore do not rely upon partnerships with
other companies out of their direct supply chain. The profitability potential is large, and
the need that merchants fill is not likely to disappear. Because of these positive traits,
there is extremely heavy competition among merchants.
Profitability
Partnerships
Availability
of
Sustainabiity
Profitability
Potential
Competition
Technology
Figure 5 Summary of Merchant criteria
2.6 Affiliate
Next, we have the Affiliate Model. Similar to the Advertising Model, Affiliates
derive their income through advertisements, generally in the form of banners. Affiliates,
however, do not get paid simply because one of their banners are viewed. If a user
connects to a client site through a banner and makes a purchase, the affiliate receives a
percentage of the sale. Affiliation has been especially successful in the online
pornography industry.' 8 This industry was one of the first to successfully adopt an
Affiliate model and its example was followed by the more mainstream websites, such as
Amazon.com and Bn.com.
2.6.1
Profitability
Affiliates take in profit on each referral starting with the first one, however
these incremental profits are quite small. An affiliate has no physical start-up costs, and
may only need a moderate amount of advertising, since it makes the same profit-perreferral regardless of volume.
20
2.6.2 Partnerships
By definition, Affiliates are dependent on having companies with whom to
affiliate. An Affiliate's entire business proposition is based around partnerships.
2.6.3 Availability of Technology
Affiliates require similar technology to advertisers. Banner advertisements are
displayed on the page, with a link pointing to the advertiser. These links contain
information identifying the affiliate so that they can be credited for the referral.
2.6.4 Sustainability
Affiliates are also middlemen. The service they provide is the referral of
customers to sellers. In this sense, they are similar to Brokers. The difference lies
primarily in the method of referral. An affiliate could be said to provide a passive
referral, whereas a Broker provides an active one. That is, while a Broker provides a
marketplace for transactions to occur, an Affiliate gathers buyers and brings them to a
marketplace provided by a seller.
2.6.5 Profitability Potential
Affiliates extract a small percentage of the profits from any referred purchase.
While affiliates have much lower operating costs, not selling an actual product, they also
make far less from each transaction than the merchants with which they are affiliated.
For example, affiliates of Amazon.com, which runs the largest affiliate network on the
Internet, earn between 5% and 15% of the revenues on an item that they sell. This leaves
85%-95% of that revenue for Amazon.com.
2.6.6 Competition
The merchant sites that partner with Affiliates want to receive as many referrals
as possible, so there is no competition among Affiliates to obtain these affiliate
partnerships. The only obstacle they face is to get the end user to pass through their site.
21
2.6.7 Summary of Criteria
Affiliates must rely heavily on partnerships with other companies. Their lifetime
may be limited, and the profitability potential during this lifetime seems to be meager
compared with other business models. However, an affiliate site is technologically simple
to enable, and competition is very light. Profits will be seen relatively soon. Because the
profit per transaction is so slow, an affiliate needs a significant volume. This is a case of
quantity over quality of transactions.
Profitability
Partnerships Availability
of
Technology
Sustainability
Profitability
Potential
Competition
Figure 6 Summary of Affiliate criteria
2.7 Subscription
The final model we will examine is the Subscription Model. Traditional print
magazines follow the Subscription Model; customers pay a fee to view content. The fee
can be paid per time period, or per byte viewed. This concept has not proved successful
on the Internet for traditional news content - users simply will not pay for it, since so
many free sources are available - but there are other forms of information for which this
model can work. Many major print publications have online versions, for example,
Hoover's Online. Games have also proven to be very attractive content.
2.7.1
Profitability
A Subscription service sells content to viewers. It needs a significant user base in
order to derive profit since generating the content has a fixed cost, regardless of the size
of the user base. Once this fixed cost is covered, however, any additional customers are
pure profit. Similar to an Advertiser, there are no significant physical expenses involved,
but they must still advertise in order to attract a user base.
22
2.7.2 Partnerships
A basic Subscription service does not require any partnerships to operate their
business, as it simply creates and sells content. Of course it can outsource part of the
content generation, and all businesses need to advertise, but this is certainly not a
requirement of the basic model.
2.7.3 Availability of Technology
Subscriptions require a site to record user accounts in a database and to allow the
content to be viewed only by owners of those accounts. Issues such as a browser's back
button and bookmarks must be addressed, both to protect the site from unauthorized
views and to enable the customer access to his purchased content.
2.7.4 Sustainability
The value of providing content to people will not change with technology; only
the means of providing that content will change. To illustrate, we can examine the
changing methods for managing stocks. Traditionally, stock information was obtained
through daily newspapers, and more up-to-date information, as well as transactions, was
available from a broker. With the advent of the Internet, this information became
available from a desktop PC. Obtaining this information was now far easier than before,
and a greater variety of this information was available. Today, this information is all
available on the wireless web. In this case, three major methods of information
acquisition via different technologies have been used in the past 10 years, but the value of
stock information has remained unchanged.
2.7.5 Profitability Potential
A Subscription service has the profitability advantage of frequent repeat
customers, as content can be updated, as in a daily newspaper or monthly magazine.
Repeat customers have no acquisition cost, unlike new customers. However, the
customer base is limited to the number of people who fall into the target segment, and
that is generally a fixed number. Heavy advertisement is required to expand that target
segment.' 9
23
2.7.6 Competition
Subscription services must compete for their market segments exactly as standard
print publications do. There are only so many readers in a given market, and while these
customers may subscribe to multiple subscriptions, there is a limit to the amount of
information they can use. With the added ease of publication, these sites must compete
not only with print publications, and other similar sites, but also with a growing number
of free sites that provide similar information. On the other hand, subscription services
tend to be targeted toward a niche consumer group, and since this consumer group will
generally be limited, a significant number of competitors cannot be sustained.
2.7.7 Summary of Criteria
A Subscription model business provides a service that will be indefinitely in
demand. It does so with relatively simple technology and independence from outside
partnerships. The major drawback is stiff competition due to a limited market.
Profitability
Partnerships Availability
of
Sustainabiity
Profitability
Potential
Competition
Technology
Figure 7 Summary of Subscription criteria
24
3. Aggregation Technologies
3.1 Criteria
In this section, we will discuss the different technologies that drive web-based
financial aggregators. First, a set of criteria will be established to provide an effective
way to evaluate and compare these different technologies. Then several general
categories of technologies will be established and further broken down into specific
technology models. Each of these models will be defined and then evaluated by the
rating criteria previously established. Current implementations, both commercial and
non-profit, will be discussed in detail.
The results of this analysis are summarized below in Figure 8. Note that while
this data presents certain models as superior to others in the various criteria that each
model is still best suited for certain applications due to their fundamental differences.
These differences are discussed in this section, and the best applications for each model
are discussed later in this paper.
Ease of
Implementation
Durability
Resistance
to
Blockage
Accuracy of
Information
Quantity of
Information
Target
Friendliness
RealTime
Agent
Spider
+
Manual
Entry
Figure 8 Rating of technology models vs. specific criteria
3.1.1
Ease of Implementation
The first criterion will be the ease of implementation for the technology. In this
age of accelerated obsolescence, it is crucial that a technology be quickly applied. Timeto-market for an Internet venture is quite short and new technologies are developed all the
time. In addition, a complex implementation requires a larger technical staff, increasing
the costs of the company.
25
3.1.2
Durability
The second criterion will be durability. I will define durability as scalability, and
longevity. If an application is not sufficiently scalable or will be quickly obsolete and
unable to perform the necessary services, then it will have to be replaced faster, resulting
in the cost of a second implementation before it would otherwise be necessary.
3.1.3 Resistance to Blockage
The third criterion will be resistance to blockage. Because many aggregations
may not be welcome by the aggregatees, a resistance to blockage will be essential to
maintaining a reliable aggregation technology. In addition to intentional blockages, an
aggregator needs to be able to circumvent network problems to provide a reliable source
of information despite outages or other problems.
3.1.4 Accuracy of Information
The fourth criterion will be accuracy of information. In any transition of data,
there is the possibility of data loss or corruption. Obviously, this is a detrimental factor to
any data transfer and should be minimized.
3.1.5 Quantity of Information
The fifth criterion will be quantity of information. Using an aggregation
technology has a cost in hardware and processing time. The more information a
technology can retrieve, the smaller an implementation can be used. Every technology
uses a set amount of system resources (memory, processing time, etc.). A more efficient
technology will retrieve the same information with fewer resources; this corresponds to a
lesser expense to operate that technology and obtain the information.
3.1.6 Target Friendliness
The sixth criterion will be friendliness to the target site. A technology that adds a
significant load to the target's servers will not be welcome on that site.
26
3.2 Real-Time Agent
The first major type of aggregation technology is active autonomous searching.
Technologies of this type gather information from other public sources, primarily other
web pages, and index or store this information on the local machine. They are sometimes
known as "bots," because they act like an electronic person, browsing different web
pages, and writing down what they find. There are two major methods of active
autonomous searching: agents and spiders (also known as robots or wanderers).
Agents in general can be defined as "personal software assistants with authority
delegated from their users.20 More specifically, agents as we will refer to them in this
paper, are pieces of software that intelligently surf the web in real-time in response to a
specific request from a user or another agent. They attempt to process the web pages to
which they are assigned, retrieve only data that is useful and relevant to their assignment,
and navigate the web accordingly. A financial aggregator might have an agent
automatically collect stock quote data from other leading sites, or fill out loan application
forms with given parameters and report the resulting rates.
3.2.1
Ease of Implementation
Agents are not particularly easy to implement. Until recently, agents existed only
in controlled laboratory environments. These agents were often run on very proprietary
networks, and with extremely simple tasks to carry out relative to our concerns. Generic
real-time agents have recently become commercially available, (for example, SWI
Systemware, Inc. created and supports real-time agent applications on AuctionWatch,
among others) but they are expensive and difficult to configure. On the other hand, an
aggregation company could build agents in-house, but that would involve a concerted
long-term effort by a skilled engineering team.
3.2.2
Durability
A well-designed agent should be adaptable to changing situations, resulting in a
longer useful lifespan. An agent's power can be increased with increasing hardware, and
given the standard format of the web, an agent can be expected to operate as long as that
format is intact.
27
3.2.3 Resistance to Blockage
A real-time web agent must gather its information when called. If there is an
obstacle obstructing that agent over repeated attempts, then it cannot complete its task,
and will fail. Agents do exhibit intelligent behavior, and may have a limited ability to
attempt to circumvent a problem. When the problem is beyond the abilities of the agent,
then it will return to its operator and ask for help. For example, an agent designed to
automatically log in to a web site may first try to directly view the desired URL. If this is
not successful, then it may start at the site's home page, and navigate through a login
script. If this also fails, then the agent will inform the user that it was not successful, and
request further "training." Many account aggregators, such as Clickmarks, work this
way.
3.2.4 Accuracy of Information
An aggregating agent will directly view other web data, as would a human user.
However, the agent is still limited in its ability to interpret that data by the strength of its
algorithm, and some degree of error is probable. In addition, the agent, if it is free to
select the sites it views, cannot determine whether that site is likely to have valid
information.
3.2.5 Quantity of Information
An agent is limited in the amount of information it returns by the efficiency of its
algorithm, the power of its processor, the speed of its network connection, and the
amount of time given to complete the task. We will assume that the agent has a
reasonable algorithm and has been trained to its specific task. Processor speed is
generally not a limiting factor, except that it has a certain time period in which to
complete the given task. Most computer applications today do not make full use of the
processor2, which Moore's Law tells us is doubling in power every 18 to 24 months. In
addition, multiple processors can be used together in order to increase speed as might be
necessary. High-speed networks and connections to the Internet are readily available for
all companies. Financial information, unlike video and audio files, is fairly small in size,
so network speed is also not a limiting factor. For comparison, an average Mpeg-3
28
format audio file is 3-4 megabytes in size, while a collection of information about the
exchange rates of the major currencies of the world averages around 10 kilobytes.2 2 A
real-time agent must return its results to a user within seconds; the average user will give
up and move to a different site if it does not load within eight seconds. 2 3 This limited
search time can definitely be a limiting factor.
3.2.6 Target Friendliness
An agent must accomplish as much as possible in a short period of time. In order
to meet this goal, a web agent may send as many requests as its network will hold.
However, this automated probing can use up the targets' precious bandwidth.24 At the
beginning of this year, for example, many major websites were brought down by denialof-service attacks. This sort of attack simply sends a rapid stream of requests to a web
server, bogging down its resources until it can no longer adequately respond to legitimate
users.
There are standards in place to address this problem (see section 4), but these will
limit the effectiveness of the agent, creating a fundamental trade-off between agent
friendliness and the amount of information an agent can return. Additionally, an agent
may have to masquerade as a human in order to gather its information, filling the target
database with fictitious data. For example, the agent behind an auction aggregator, such
as AuctionWatch, will simply navigate through an online auction, such as Ebay, using the
same steps a user would. It logs on under a fictitious account, follows the same links, and
fills out the same forms that human user would encounter. Because it follows the same
path, it is indistinguishable from a human user to the server in terms of online behavior.
3.2.7 Summary of Criteria
To summarize, the current real-time agent technology is lacking in several areas.
It is extremely difficult to implement. While in operation, it is susceptible to blockage,
cannot be relied upon to always return accurate data, and can be a burden on the target
servers. On the other hand, real-time agents can return a reasonable amount of
information, and can be quite durable over the long run.
29
Ease of
Implementation
Durability
Resistance
to
Blockage
Accuracy of
Information
Quantity of
Information
Target
Friendliness
Figure 9 Summary of Real-Time Agent criteria
3.2.8
Real-World Examples
There are many web-based aggregation businesses that make extensive use of
real-time agents. Agents are a necessary technology when real-time data is needed, and
that data is too customized or complex to be obtained via a standard news feed (discussed
later).
One such company is AuctionWatch. AuctionWatch uses real-time agents to
conduct searches across many auction sites. Since auctions are timed, and constantly
updated, a real-time solution is needed. AuctionWatch consolidates information made
publicly available on these sites with automated agents; there is no other method
available, aside from an actual human conducting the task. An agent's shortcomings are
not a problem in this case, because the number of auction sites is set and relatively few,
and these sites do not often change their format; the agent can quickly learn all it needs to
about its task, and this can be directly programmed or taught by human users.
AuctionWatch also makes use of real-time agents to help users place items up for bid on
these auction sites. These agents need not conduct a search of information displayed on
the auction sites, but rather keep track only of the format of product submission forms.
They provide one common interface to users placing an item up for bid, then
automatically translates that information into a format understood by the auction sites and
posts the items.
Another company that extensively utilizes agents is Clickmarks. Clickmarks
allows its users to create a custom habitat consisting of any web content, including news
headlines, email accounts, online calendars, etc. All of this information is gathered and
kept current by way of sophisticated agents. A user can view the information in their
habitat and expect it to be kept current with the original source. If it is not current, the
30
user is notified, and has the option to deploy the agents at any time to retrieve the most
current information.
3.3 Spider
Often, information does not change so fast that a real-time aggregation is
necessary. In this case, conducting real-time searches results in many repetitive results,
and therefore unnecessarily consumes resources. A better solution would be to determine
how often the data is likely to change, and then conduct a single search after each change
and store the results in a local database. These results can then be searched on an
aggregator's local machines, resulting in a far more efficient search without the
complication of using up the computational resources of the aggregatee. For example,
frequent flier information is updated with a delay of up to several days. A daily update
by an aggregator is sufficient. This is known as indexing, and the software solution,
defined in this paper as a timed agent, is known as a "spider."
There are two methods that an individual spider can use to search the web;
multiple spiders working in tandem can also be used, and will be discussed later. Let us
examine a sample web space of two hyperlinks, each referred page containing another
two links [shown in figure 1]. We are given the goal that, starting on page A, we want to
visit every page. There are two systematic approaches to this problem. The first is to
follow each series of links as far as they will take us, then retrace the minimal amount
and follow the next, resulting in the following path: A B D B E B A C F C G (previously
unviewed pages are denoted in bold face). This is known as a depth-first search. The
other approach, known as a breadth-first search, will complete its categorization of each
level before continuing down another link. The resulting path from this type of search in
our example would be A B A C A B D B E B A C F C G.
31
A
D
EFG
Figure 10 Example state-set for search discussion.
In this example, the depth-first search actually resulted in a shorter path, and
therefore it was more efficient. However, the web differs from the simple example of
figure 3-1 in that while there are only a finite number of links on each page (a finite
breadth), there is essentially an infinite chain of links that could be followed from any
page. The practical result of this observation is that, given the page you are looking for is
within a few links of the starting page, a depth-first search would be enormously
inefficient, and may never find the page. If you wished to find page F, you would never
arrive unless all links below page B - again, essentially infinite in the web - were
followed. A solution to this would be a nondeterministic search, which provides a good
compromise between the breadth-first and depth-first searches 25 . This solution, in
essence, switches randomly between breadth-first and depth-first search at each node,
resulting in a search somewhere in between.
The nondeterministic search is random in nature, and a more focused search
would be ideal. A best-first search assigns scores to each node in its immediate path
based on various relevance criteria, and the highest scored node is followed, regardless of
level. In the example of figure 1, C would have a higher score than B, so the spider
would start at C instead of B, resulting in a very short search.
32
A good example of a best-first search on the web is Lycos. Lycos ranks its pages
by the number of links (in its database) that refer to the given page. The reasoning is that
pages more often linked to are more popular, and therefore more likely to be the desired
result 26
Another example of a best-first search is Direct Hit. Here the results of a specific
query are stored in a database, as well as which of the results returned are viewed by the
user. Pages more frequently viewed are judged as being more likely to be the desired
page for a new query.
When a general collection of data is the goal, or when it is not possible to
accurately assign scores to potential links, then it is important to limit the search, as the
web is essentially an infinite search space. Since the depth factor of the web is larger
than the branching factor, a breadth-first search is the better choice2 7 . As illustrated in
the example in figure 1, where the goal was simply to visit (and presumably index) every
page, the breadth-first search was the superior method.
A good example of a breadth-first search on the web is Webcrawler. Webcrawler
uses this strategy to explore as many servers as possible, resulting in a broad index and
not placing a heavy load on any target web server28
3.3.1
Ease of Implementation
Because real-time response is not as crucial for spiders as for real-time agents,
they do not need to be built as efficiently. Less efficient software equates to easier
implementation. Indeed, several spiders of lesser power can work in tandem to provide
better, faster results. For example, the Web Ants project has created a system of spiders
that communicate with each other to avoid duplicating each others' results29 . Many
implementations of tandem spiders are in use currently, and several generic spider
"packages" are available for purchase or free download.
3.3.2
Durability
As with a real-time agent, a well-designed spider should be adaptable to changing
situations, resulting in a longer useful lifespan. A spider's power can be increased with
increasing hardware or by running several implementations in parallel, and given the
33
standard format of the web, a spider can be expected to operate as long as that format is
intact.
3.3.3 Resistance to Blockage
A spider is limited by the same factors that limit a human browsing the web; that
is, it will be vulnerable to any network blockages or security actions taken by a target
site's administrator. A spider does have the ability to make a large number of repeated
attempts to view a page, and since it does not need to immediately interpret what it finds,
but only store it for later analysis, it can continue to do this over a longer period of time.
3.3.4 Accuracy of Information
A spider will directly view other web data, as would a human user. The spider
does not need to interpret this information in real-time, except to make the decision about
what links to follow. Since the actual analysis and interpretation can occur locally and
over an extended time, it will be much more accurate than a real-time analysis. In
addition, the spider, if it is free to select the sites it views, cannot determine whether that
site is likely to have valid information.
3.3.5 Quantity of Information
Since spiders do not need to operate in real-time, and multiple spiders can work in
tandem, they can return an enormous amount of information. Most major search engines
use spiders as their primary means of information acquisition. The average major search
engine has over a hundred million web sites catalogued. 30
3.3.6 Friendliness to Target
Spiders have a reputation for being inefficient and hogging resources.31 However,
most spiders now follow the standard robot exclusion protocols (discussed in section 4),
and are often timed so that they make page requests no faster than one per second. These
concessions, while slowing a spider's progress, make it less aggravating to the target
sites.
34
3.3.7 Summary of Criteria
While spiders still have the fundamental flaws of an autonomous informationgathering system, they are able to minimize the effects by not operating in real-time.
They capitalize on the advantages of the autonomous system.
Ease of
Implementation
Durability
Resistance
to
Blockage
Accuracy of
Information
Quantity of
Information
Target
Friendliness
Figure 11 Summary of Spider criteria
3.3.8 Real-World Examples
Many sites only need very specific information from a few pre-determined sites.
This information, however, may change very often. Stock quotes, for example, will only
come from a few sources, but are constantly changing. In this case, an automated search
may not be the ideal method. A real-time agent consumes an enormous amount of
resources to obtain information, the location of which is already known. A spider, on the
other hand, operates far too slowly to be of any use in this situation. The solution, then,
is to create a direct pipeline of information from the source to the destination site. This
pipeline will probably be purchased from the source, and a destination that wishes to
aggregate several sources can simply purchase a pipeline from each of them. Although an
information pipeline as described does not strictly fit our definition of an aggregator (it
does not transparently collect information), we will discuss this technology briefly, as it
is a viable source of information, and one against which other aggregating technologies
must compete.
An information pipeline is not overly difficult to implement. A server needs to
send the data to the client site - but that implementation is relatively straightforward.
From the client's point of view, they simply need to install software to receive the
information stream from the server, and then filter that data and display it as appropriate.
Since the server maintains a direct pipeline, durability is not really an issue. As
long as information flows in the same format, any filtering software installed on a client
site will not care how the information gets there. Given a standard modular
35
implementation, the method in which the information is sent can be changed without
affecting the processing of that information.
Under normal network operation, a direct pipeline will never be blocked, since it
is authorized by the information-providing site. However, if a network outage somehow
does block a pipeline, there is no way to avoid the obstacle except repeated attempts to
transmit the information.
The information provided by a direct pipeline comes from a known and
presumably reputable source. There are no content decisions made completely
autonomously; this information can be considered completely reliable.
An information pipeline can be a high-speed connection which funnels a great
deal of data, but the information provided is limited to a very specific subscription, and
this information all comes from one source.
Again, an information pipeline is actually supported by the host, and the host is
under complete control of these pipelines, so friendliness to the host is a non-issue.
If a specific type of information is needed, and the breadth of sources is not
important, a direct information pipeline may be a very good move. These pipelines are
durable, and provide accurate information. The only drawbacks are the quantity of
information provided, and the resistance to blockage under unusual network
circumstances.
Major online brokerages and financial portals generally use a direct information
pipeline (such as that provided to Yahoo Finance by Tibco Software) to retrieve market
data. This data can be displayed directly to the user, or analyzed in a multitude of ways.
Ameritrade, for example, allows users to compare their choice of several equity attributes
against the current market, or against other equities. This data can be displayed in tabular
format or graphed.
A recent experiment I did illustrates some of the potential of this analysis. I
received a tip that the earnings of Applied Materials (AMAT) are cyclical. While sales
are regular, and of large amounts (a very few large sales make up the bulk of this
company's profit), these sales are infrequent; the time between sales is often considerably
longer than one financial quarter.3 2 This, theoretically, would result in unexpectedly low
or high earnings for the quarter, depending on whether or not a large sale was made
36
during that quarter, and the share price would rise or fall accordingly. When viewing the
graph of the AMAT share price over a timeline of five years, any cycles are lost in the
noise of the general market strength. The share price of AMAT tends to follow the
NASDAQ index as a whole. I used the analysis tools in Ameritrade to normalize the
AMAT price against the NASDAQ index, essentially showing only the difference in
percentage gain or loss for a particular day. Unfortunately, this did conclusively prove
that the theorized cycle does not, in fact, exist.
Online brokerages are not the only financial aggregators that use pipelined
information. Popular personal financial managers, such as Microsoft Money and
Quicken, gather investment information in much the same way as do classic online
brokerages, and provide their own custom sets of analytical tools.
3.4 Manual Entry
Sometimes the required information requires a dissemination ability beyond that
of which a computer is capable, or perhaps it is not readily available online. Other times,
the information volume is small enough that a computer is not necessary. In these cases,
manual information entry can be used.
One method is for company employees to take the role of spiders and manually
find and enter the information. The results will be far more accurate, as the employees
can determine what information is important, and can more accurately determine where
that information is likely to be. This is the primary method that Yahoo uses to create its
online index. However, this requires a lot of manpower, and cannot create a large
amount of information, relative to that produced by an autonomous spider. Alta Vista, for
example, which uses an autonomous spider, has indexed 100 times the number of web
pages that Yahoo has.3 3
Another variation of manual information entry is for consumers to input
information. Each consumer may only input their personal information, but a
compilation of all this information can provide valuable data. Almost every commercial
website now keeps a database of its customers. All information in these databases was
entered manually by the consumers. These databases are very valuable marketing tools
37
both to the companies, and to many others to whom the companies may sell their
customer data.
3.4.1
Ease of Implementation
Manual information entry is by far the easiest form of aggregation to implement.
No new technology or special coding is required. Employees simply search the web with
a standard web browser and existing search tools, and input the information into the
database. Consumer-based entry requires very little if any research on the consumer's
part, and the information is entered directly into the local database through a web
interface.
3.4.2 Durability
This kind of model is also extremely durable, as a human is certainly more
adaptable to changing situations than today's software. Given that the standard websearch tools are kept up-to-date by their operators, this model is extremely adaptable.
3.4.3 Resistance to Blockage
Again, humans are quite adaptable, compared to today's software, and can
improvise methods to avoid blockages. Humans, however, cannot send rapid-fire
requests, as can autonomous searches, so may not be able to break through a blockage
using "brute force."
3.4.4 Accuracy of Information
Compared to autonomous software, humans, in general, are not as likely to be
fooled by similar-looking, but unrelated data. Humans also have the ability to assess
whether a site is likely to be an authoritative source of information, for example making
the separation between the Wall Street Journal Online, and Joe's Stock Picks. However,
unlike autonomous software, humans grow fatigued, and can make clerical errors.
3.4.5 Quantity of Information
Humans are at a decisive disadvantage when it comes to the quantity of
information that they can process. Computers are always better at performing rote tasks
38
in volume. Again, we can examine the case of Yahoo vs. Alta Vista. As mentioned
before, Alta Vista, which uses automated technology, has catalogued over one hundred
times the amount of information that Yahoo has compiled using its manual indexing. 34
3.4.6 Friendliness to Target
Humans cannot send requests quickly enough to seriously slow a host server.
However, humans are adept at entering false information (i.e., creating fictitious
accounts) in order to access information.
3.4.7 Summary of Criteria
Humans in general are far more adaptable than their software counterparts, and
can disseminate important information from useless information. However, they are
limited in the quantity of information that they can process.
Ease of
Implementation
Durability
Resistance
to
Blockage
Accuracy of
Information
Quantity of
Information
Target
Friendliness
Figure 12 Summary of Manual Entry criteria
3.4.8 Real-World Examples
One successful implementation of manually entered aggregated information is
Ebay. Ebay, the largest online auction site, has over 10 million users, and 4 million
auctions on any given day. 35 Each user may participate in many of these auctions. Every
time a user enters an auction, either as a buyer or a seller, they give Ebay personal
information. When originally signing up for the service, Ebay receives their contact
information. With each auction in which a user participates, Ebay gains another piece of
knowledge both about that user's spending habits and interests, and about the general
market value of the item up for auction. This information is stored in Ebay's databases
and could very easily be analyzed and used as valuable marketing information.
Sometimes data can be collected without the knowledge or direct consent of the
consumers entering the information. Companies such as DoubleClick and Engage
Technologies record the transactions enacted by users on many other sites. Much of the
39
time, it is not apparent to the users that one of these companies are logging their
transactions. Like Ebay, this gives the information-collection company a store of
marketing information that can be analyzed and used or sold. The specific uses of this
information have come under fire, as questions have arisen as to whether the use of this
information is an invasion of privacy.
All four of these technological models have merit in the proper applications.
Autonomous real-time agents are a superior choice when instant information is needed
and the information changes quickly and is spread out among many sources that may not
be willing to sell a direct pipeline of information at an acceptable price. When a vast
amount of information is important, but that information changes relatively slowly, then a
spider may be the best choice. If a stream of rapidly changing information is important,
but that information is contained within a few affordable sites, then a direct pipeline is the
best solution. On the other hand, if accuracy and adaptability is important, or if the
information requires complex processing, but rapid changes and large volumes are not
essential, then a manual solution may be the ideal choice.
40
4. Commercial Applications
Aggregation is widespread in the "dot com" world, and many commercial
applications have been developed. In the consumer sector, aggregators are most notably
used for comparison shopping, consolidation of financial relationships, and compilation
of related information from varied sources.
A widespread commercial use of aggregators is comparison-shopping, both for
products and for various services. Aggregators are used to compare consumer products,
airfare, auction prices, and financial service rates, including insurance rates and loan
interest rates. Often this is done simply by using a robot to automatically view and
compile rates and prices published by the major purveyors of such products and services.
The aggregator will gather all the published prices on a particular item, service, interest
rate, etc., and compare them to give the user the best buy.
Aggregators can also be used to help people manage their online accounts.
Frequent flier miles, auctions, and many online accounts can be consolidated into one
place. Not only does this provide a convenience for the user, but also it allows the
aggregator to be a middleman, collect valuable market data, and possibly collect a
transaction fee on some of the user's financial interactions.
Finally, aggregators are being used to compile general financial information.
Equities, market data, and general company research can be compiled to aid investors in
making the best decisions when investing their money.
The following figure will help to categorize some of the commercial applications
to which aggregators are being applied today.
41
1.
Comparison Shopping
1.1. Financial Service Rates
1.1.1. Loan Interest Rates
1.1.2. Insurance Rates
1.1.3. Investment Rates
1.2. Consumer Product Information
1.3. Airfare
1.4. Auctions
2.
Relationship Aggregation
2.1. Frequent Flier Miles
2.2. Account Consolidation
2.3. Auction Consolidation
Figure 13 Current Aggregation Markets
Over fifty real-world companies have been analyzed, and their business and
technology models have been determined. A complete and detailed list of the results is
available in Appendix B. and some of the results are analyzed below. Please note that
some companies use more than one business and technology model and may therefore be
counted more than once, resulting in a graph that totals more than 100%.
4.1 Loan Interest Rates
4.1.1
Business Model Usage
Loan Rate Business Models
100%
-
80%
Percent Use
60%
-
40%----
---
20%
Brokerage
Advertiser
Infomediary
Merchant
Affiliate
Subscription
Business Model
Figure 14 Loan Interest Rates - Business Model Usage
42
All loan aggregators analyzed use the Brokerage model, extracting a fee for each
loan referred. Quicken Loans also functioned as a Merchant, selling their financial
software. Bankrate.com sells a subscription newsletter containing their financial
information, as well as their standard Brokerage.
4.1.2 Technology Model Usage
Loan Rate Technology Models
60%
50%
40%
Percent Use 30%
20%
10%
0%
Real-Time
Agent
Spider
Manual Entry
Technology Model
Figure 15 Loan Interest Rates - Technology Usage Models
Both Real-Time Agents and Manual Entry are used frequently for loan rate
aggregators. Manual Entry would generally be sufficient for a loan aggregator, as the
number of different rates is generally relatively small, is often determined in part by the
institution offering the rates, and changes infrequently. Many loan rate aggregators do
offer real-time tools in an effort to expand their service and provide a financial tool that
can do more than simply search loan rates.
43
4.2 Insurance Rates
4.2.1
Business Model Usage
Insurance Rate Business Models
100%
80%
Percent Use
-----
40%
20%25
Brokerage
Advertiser
Infomediary
Merchant
Affiliate
Subscription
Business Model
Figure 16 Insurance Rates - Business Models
All insurance rate aggregators analyzed used a brokerage model, taking a fee for
each new customer referred. One aggregator, InsureChoice, allows customers to
subscribe for a membership to receive additional online features and information.
4.2.2 Technology Model Usage
Insurance Rate Technology Models
100%
80%
Technology
Model
60%
40%
20%
0%'
Real-Time
Agent
Spider
Manual Entry
Percent Use
Figure 17 Insurance Rates - Technology Model Usage
All insurance rate aggregators analyzed use Manual Entry; as with loan rates, this
is all that is necessary to run a basic aggregator.
44
4.3 Investment Rates
4.3.1
Business Model Usage
Investment Rate Business Models
7 0%--
-
- - - - - -- -
--
- -
60%
Percent Use
50%
40%/----
40%/
20%
10%
7
A17
0%
Brokerage
Advertiser
Infomediary
Merchant
Affiliate
Subscription
Business Model
Figure 18 Investment Rates - Business Model Usage
Most investment rate aggregators used a brokerage model, extracting a fee for
each customer referred. BankCD.com instead uses a merchant model, selling the name of
the offering bank to the customer after they have selected a set of CD terms.
Moneyrates.com provides investment rate information for free, instead drawing revenue
from advertisement. In addition to the standard Brokerage, Bankrate.com also offers a
paid membership (Subscription) that provides additional services and information to
consumers beyond the basic rate search.
45
4.3.2 Technology Model Usage
Investment Rate Technology Models
100%1"
80%
Percent Use
60%
40%
20%
0%
Real-Time
Agent
Spider
Manual Entry
Technology Models
Figure 19 Investment Rates - Technology Model Usage
Most investment rate aggregators use Manual Entry to obtain their information.
One company, Banx.com, advertises real-time rate information.
4.4 Consumer Product Information
4.4.1
Business Model Usage
Consumer Product Business Models
30%
A
25%20%--
2
Percent Use 15%
27
10%
Brokerage
Advertiser
Infomediary
Merchant
Affiliate
Subscription
Business Model
Figure 20 Consumer Product Information - Business Model Usage
46
All business models except Subscriptions are used to extract revenue from
consumer product aggregation. There does not seem to be a clear indication of the
superiority of any one business model in this market space.
4.4.2 Technology Model Usage
Consumer Product Technology Models
60%
50%
40%
Percent Use 30%
20%
10%
0%
Real-Time
Agent
Spider
Manual Entry
Technology Models
Figure 21 Consumer Product Information - Technology Model Usage
Manual Entry is the most heavily used technology model, although the others are
used as well. In these cases, consumers or merchants simply enter the product
information by hand. A few services employ automated software to retrieve product
information from merchant web sites.
47
4.5 Airfare
4.5.1
Business Model Usage
Airfare Business Models
100%
80%
---------
60%
------
Percent Use
40%
20%
Brokerage
Advertiser
Infomediary
Merchant
Affiliate
Subscription
Business Model
Figure 22 Airfare - Business Model Usage
Almost all airfare aggregators use a Brokerage model as their primary source of
revenue, extracting a fee for each ticket sold. Many of them also derive profit from
advertisement, having a very specific user group captive. One company, Frequent Flier
Points, actually purchases and re-sells tickets.
4.5.2 Technology Model Usage
Airfare Technology Models
100%
80%
Percent Use
%
40%20%
0%
Real-Time
Agent
Spider
Manual Entry
Technology Models
Figure 23 Airfare - Technology Model Usage
48
All airfare aggregators use Manual Entry as their primary method of information
collection. In fact, many of them simply subscribe to a rate database, such as Sabre, and
directly query it. In this case, the rates are put into the database by hand (by airline
employees).
4.6 Auctions
4.6.1
Business Model Usage
Auction Business Models
100%
80%
60%
Percent Use
- --
40%
20%
Brokerage
Advertiser
Infomediary
Merchant
Affiliate
Subscription
Business Model
Figure 24 Auctions - Business Model Usage
All auction aggregators analyzed used a brokerage as their primary source of
revenue, generally extracting a fee from the seller. Often this fee is based on the selling
price of the item being auctioned.
49
4.6.2 Technology Model Usage
Auction T
yModePs
100%
80%
Percent Use
60%
40%
20%
0%
Real-Time
Agent
Spider
Manual Entry
Technology Models
Figure 25 Auctions - Technology Model Usage
All auction aggregators analyzed also used a Real-Time Agent as their technology
model. This is essential, as auction prices must always be current within a fraction of a
minute in order to be useful.
4.7 Frequent Flier Miles
4.7.1
Business Model Usage
Frequent Flier Business Models
70%
60%
50%
40%
Percent Use 3___--
--
20%-
10%
0%
Brokerage
Advertiser
Infomediary
Merchant
Affiliate
Subscription
Business Model
Figure 26 Frequent Flier Miles - Business Model Usage
50
,
Most frequent flier aggregators analyzed used a subscription-based service,
allowing users to create and use an account for an initial trial period before charging them
a subscription fee. MilePoint instead aggregates users' miles, takes them, and allows
users to use the points to purchase online goods (minus a brokerage fee). In addition to
providing a standard subscription-based service to general consumers, TotalMiles.com
will sell its aggregation engine to companies to run internally on their own servers.
4.7.2 Technology Model Usage
Frequent Flier Technology Models
100%
80%
Percent Use
60%
1000
40%
20%
0%
Real-Time
Agent
Spider
Manual Entry
Technology Models
Figure 27 Frequent Flier Miles - Technology Model Usage
All frequent flier miles aggregators analyzed used a Spider to retrieve their
information. This is an ideal technology, as daily updates are more than sufficient, and
the number of airlines to aggregate is relatively small.
51
4.8 Account Consolidation
4.8.1
Business Model Usage
Account Consolidation Business Models
50%
40%
30%
450
Percent Use
20%
2%-
18%
10%
Brokerage
Advertiser
Infomediary
Merchant
Affiliate
Subscription
Business Model
Figure 28 Account Consolidation - Business Model Usage
Account consolidation aggregators use almost every model in this analysis, but
especially tend toward Brokerages and Advertisers. Having specific information about
users, provided by themselves, makes this market especially good for Advertisers (or
Infomediaries). Those companies that provide "frictionless transaction" capabilities
(automatic entry of personal information when ordering products online) act as Brokers,
taking a fee for each transaction, usually from the seller.
52
4.8.2 Technology Model Usage
Account Consolidation Technology Model
70%
60%
50%
40%
Percent Use
30%
20%
10%
0%
Real-Time
Agent
Spider
Manual Entry
Technology Model
Figure 29 Account Consolidation - Technology Model Usage
Account consolidation aggregators tend to use Manual Entry, as most of the
account information is collected directly from the users. Some aggregators will then use
this user information to go online and automatically retrieve information.
4.9 Auction Consolidation
4.9.1
Business Model Usage
Auction Consolidation Business Models
100%
80%
Percent Use
40%20%
Brokerage
Advertiser
Infomediary
Merchant
Affiliate
Subscription
Business Model
Figure 30 Auction Consolidation - Business Model Usage
53
All auction consolidators analyzed use a Brokerage model, extracting a fee for
each transaction (generally by the seller). Advertisement revenue is definitely possible,
and one auction consolidator (Bidcrawler) derives additional revenue by selling its
aggregation engine.
4.9.2 Technology Model Usage
Auction Technology Models
100%
80%
Percent Use
60%
40%
20%
0%
Real-Time
Agent
Spider
Manual Entry
Technology Models
Figure 31 Auction Consolidation - Technology Model Usage
All auction consolidators analyzed used a Real-Time agent to gather information,
as up-to-the-minute is necessary when bidding on a timed auction.
54
5. Legal Issues Surrounding Aggregation
5.1 The Issues
5.1.1
Ownership of Information
One of the central issues surrounding the rise of aggregators and e-commerce in
general, has been the ownership of information. The web is aptly named, as information
is inter-linked and inter-twined with other information. Who may use certain information
and how they may use it has become a hot issue both in and out of the courtroom.
Some believe that the web was founded on principles of sharing open
information. Any information put on the web should be regarded as public; any
information that a company or individual wishes to keep private should not be placed on
the web. Hyper-links are the infrastructure of the web, and disabling them disables the
entire web.
Others believe that the rules of information ownership should remain the same,
regardless of the medium. If it is illegal to use copyrighted work or private information
in more standard mediums, then the Internet should be no exception. Owners of
information should have the right to regulate how and by whom their information - their
property - is used.
Transposing traditional property rights to Internet terms proves to be a challenge.
For example, it is not hard to picture the act of unauthorized hacking into a private
computer system as one of trespassing. However, when that computer system has been
opened to the general public, as in the case of an e-commerce website, it becomes a hazy
issue as to what is trespassing and what is simply viewing information put in the public
domain.
5.1.2 Copyright Infringement
Another burgeoning issue is that of copyright infringement. While in theory the
rules of copyright should be easily transposed across different mediums of presentation,
the ease of duplication and display of information provided by the web enables this to be
55
a widespread problem. In addition, the question arises as to whether a copyright is
infringed if the work in question remains unaltered in its original location, but is hyperlinked by an unauthorized source.
5.1.3 Implied Contracts
In order to address these first two issues, many websites will post a list of terms
and conditions for access to the information contained therein. It remains to be seen if
these lists are legally binding, and if so, to what extent. Also, these terms and conditions
must be placed in the website in such a way that they are obvious to a visitor of the site.
Adapting standard business methods to this new medium is a work still in
progress, and many new practices are labeled as unfair by the competition. It is very easy
for a company to misrepresent its relationship with a competitor; the distinction between
one company's website and another can become very hazy.
5.1.4 Advertiser Contracts
Since many websites are funded partially or in whole by advertisers, the display
of advertising banners is of utmost importance to them. Since information is so easily
copied, altered, and re-displayed, there is a worry that aggregation is impairing the
advertisers' relationships with the original source of the information. These advertisers
generally pay a set amount per a certain number of users viewing their banner, and by
having the website's content displayed elsewhere without these banners, the advertisers
lose those customer views. Alternatively, since they are paying per view, the advertisers
pay every time a robot indexes the page. If the page is frequently visited, the advertisers
are paying for an inhuman audience.
5.2 Solutions
5.2.1
Four Laws of Web Robotics
In order to address these concerns, the Internet community as a whole is working
toward a set of guidelines to which all sites should adhere. The most notable and
widespread convention for spiders is The StandardforRobot Exclusion36 proposed by
Martijn Koster in 1994. This standard requires that any webmasters who want to restrict
56
robot access to their site create a file called robots.txt. This file contains information
regarding which robots may visit the site. Any robot visiting a site must first check for
this file, and may not continue to index that site if it is barred. More recently, a "robots
37
meta tag" is available to block robots from specific pages instead of entire sites
In order for future robots to be accepted by webmasters, they should adhere to a
basic code of robot "etiquette." Fah-Chun Cheong has summarized the advice given by
the web community in his Four Laws of Web Robotics. 38 These laws are:
I.
A web robot must show identifications.
II.
A web robot must obey exclusion standard.
III.
A web robot must not hog resources.
IV.
A web robot must report errors.
Similar sets of guidelines have been developed by Oren Etzioni and Daniel Weld of the
University of Washington39 , and by David Eichmann of the University of Houston40 .
Webmasters need to know the identity and source of any robots accessing their
site, so they can contact the owner if necessary. In addition, many wish to know how the
robot found their site. By supplying the path by which it traveled to the site in question,
the robot can perform a service for the webmaster, and "earn its keep."
It goes without saying that a robot must follow the aforementioned exclusion
standard. However, this can be in the robot's best interest as well as the webmaster's, as
the site may be off-limits to protect the robot from become trapped in an infinite virtual
space.
In order to avoid the ire of webmasters, a robot should be unnoticeable. If the
robot becomes a burden on the server, then it will surely be excluded. Cheong lists
several suggestions for reducing a robot's impact on a server, such as retrieving the
minimal information needed, checking for errors in the pages, retrieving at low-traffic
times, specifically checking to avoid looping, and, of course, retrieving information in
moderation.
Finally, as a service to sites it indexes, a robot should report any errors it finds to
the webmaster. This is simply being a good citizen.
57
5.2.2 The Six Commandments for Robot Operators
Cheong also lists some guidelines that robot operators should adhere to in order to
ensure that their robot is a good citizen. These have been named The Six
Commandments for Robot Operators 41
I.
Thou Shalt Announce thy Robot.
II.
Thou Shalt Test, Test, and Test thy Robot Locally.
III.
Thou Shalt Keep thy Robot Under Control.
IV.
Thou Shalt Stay in Contact with the World.
V.
Thou Shalt Respect the Wishes of Webmasters.
VI.
Thou Shalt Share Results with thy Neighbors.
When a new robot has been created, it is best to notify the sites that will be
visited, or the web community at large if applicable. Although in some instances this
may seem to be detrimental to competitive advantage, keeping a robot's visits secret is
not realistic, and will result in a less positive response from the webmaster.
The reasons for testing a robot locally should be apparent. An untested robot can
not be guaranteed to adhere to the Four Laws of Web Robotics, and may quickly become
unwelcome, as well as not correctly performing its intended task.
If something does go wrong with a robot, the owner should be easily reached.
The owner may well be the only person who can fix any problems that the robot causes
or encounters.
In order to maintain the goodwill of webmasters, one should adhere to their
wishes. If they request or undertake actions to bar a robot from their site, their wishes
should be obeyed.
Web robots can use up significant resources in their operations. By sharing this
information with the world, other robots can be prevented from needing to perform the
same task, saving resources. The raw data can be made available, or refined results can
be published.
Often in business, it is to one company's advantage to use a robot on another's
site, and this advantage outweighs any problems caused by an antagonized webmaster.
In some cases, the companies are in direct competition. It is important to remember that
the aforementioned guidelines are only guidelines, and are not law. These guidelines,
58
while potentially applicable to corporate robots, were originally designed without the
factor of competition. The issues involved in with information ownership on the internet
have all come up in actual conflicts between companies, and have been addressed in
court. There have been definitive rulings in some cases, and out-of-court settlements in
others.
5.3 Legal Rulings
5.3.1
Ebay v. Bidder's Edge
Ebay has raised the issue of trespassing in its lawsuit against Bidder's Edge. In
December of 1999, Ebay, the largest auction site on the internet sued Bidder's Edge, an
aggregator of auction sites for trespassing, specifically for "deep-linking," the practice of
including hyperlinks to pages deep within the target site, and bypassing any higher-level
pages 42 . Ebay claims that the spiders slow its service - they access Ebay's site about
100,000 times per day 43 - and that the all information on its site is its intellectual
property. Bidder's Edge claims that the information on the site belongs to the individuals
auctioning their goods, and that Ebay has no claim to this information.
Ebay does not have a robots.txt file, but Bidder's Edge says they would not
observe it if there were one because they do not feel that Ebay owns the information44 . If
Ebay were to implement a robots meta tag, each user could prevent Bidder's Edge from
accessing their individual auction if they wished to do so.
On May 24, 2000, Judge Ronald M. Whyte of the United States District Court in
San Jose issued a decision in favor of Ebay. Bidder's Edge has been banned from
accessing any information on Ebay's site by automated means without written permission
from Ebay until the end of the full trial in March, 2001 unless it is overturned on appeal.
If this ruling stands, a consequence will be that many comparison sites will go out of
business, or be forced to pay access fees to those sites that they compare 45
5.3.2 News Index v. The Sunday Times
The issue of copyrights on the web is still largely unanswered. In 1997, a dispute
arose between News Index and the Sunday Times over News Index's spidering of
Sunday Times content. The issue of deep linking was involved, but more notably, the
59
Sunday Times was concerned that its copyrights were being violated. If this issue went
to court and were resolved in favor of the Sunday Times, it could be disastrous for search
engines, as they also index content in a similar way, although since they only link to
content and do not store it, it may not apply 46 . This dispute was settled out-of-court, but a
future ruling on a similar case could have the aforementioned results.
5.3.3 R.I.A.A. v. Napster
Even sites that do not specifically target other commercial sites, but are capable of
linking copyrighted material have come under fire. The Recording Industry Association
of America has sued Napster Inc. over software that allows Napster users to trade
copyrighted music files. While these users are clearly breaking copyright laws, these
laws are unenforceable at the user level; as a result Napster is the defendant in this case.
This case is similar to that of Sony Corp v. Universal Studios Inc. in 1984. The message
sent by this case is that that "if you sell a product with 'significant non-infringing uses,'
then you are home free. 47" The Napster case will specifically test this decision in the
Internet medium. If Napster prevails, then the door will be opened for software tools that
gather information, regardless of that information's ownership status.
5.3.4 Ticketmaster v. Tickets.com
Ticketmaster Corp. sued Tickets.com Inc. over a deep-linking issue, and a ruling
was issued on March 27, 2000 by Judge Harry L. Hupp of the Federal District Court in
Los Angeles. One of Ticketmaster's claims was that Tickets.com had violated the "terms
and conditions" presented by Ticketmaster for use of its site. While the judge dismissed
this claim as written, he did allow Ticketmaster to refile the complaint given that they
amend it to show that these "terms and conditions" create an enforceable contract, seen
and agreed to by Tickets.com. This ruling seems to suggest that a site may be able to
prevent unwanted visitors by phrasing its "terms and conditions" in the form of an
enforceable contract, and by ensuring that a visitor must view and acknowledge this
contract in order to access the site.48
In the lawsuit vs. Tickets.com, Ticketmaster also made claims that Tickets.com
was guilty of unfair forms of competition: "passing off' and "reverse passing off," 49 as
60
well as false advertising.
0
Consumers could incorrectly be led to believe that the two
companies were connected in certain ways detrimental to Tickets.com and beneficial to
Ticketmaster.'" Judge Hupp allowed these claims because Tickets.com had built their
website such that it was unclear which portions were their own, and which belonged to
Ticketmaster.
Ticketmaster also successfully made a claim that Tickets.com was interfering
with the economic relationships it had with its advertisers. These advertisers pay a hefty
fee to display an advertisement on Ticketmaster's site. If users are able to view
Ticketmaster's inner content without seeing these advertisers - i.e., by accessing it
through Tickets.com - then the value of those advertisers' investments is compromised.
5.4 Future Direction
In many respects, the advent of the Internet has challenged the rules of
information ownership. While there are cases of copyright infringement pending, for
example RIAA v. Napster, technology is making a copyright unenforceable. Once
Napster came under legal fire, several clones appeared, many with technological
improvements to thwart any attempt to shut them down, legal or otherwise. Companies
will have to derive new methods of extracting profit from their information.
On the other hand, the preliminary results of cases pending, such as Ticketmaster
vs. Tickets.com and Ebay vs. Bidder's edge seem to suggest that common sense will
prevail. If a company is using the work of another without permission, it is not likely to
be in the legal right. This is significant for aggregators, many of which it seems will have
to negotiate usage licenses with the sites they aggregate.
61
6. Conclusion
Aggregation ultimately provides value to the end-user, the consumer. Those
middlemen who stand to profit by the inefficiencies in the value-chain are fighting this
technology, but it can be avoided for only so long.
Legal battles surrounding the application of aggregation technology to the music
industry have hit the forefront of the media, as in the case of the Recording Industry
Association of America v. Napster, Inc. The RIAA, which arguably, does not provide
value to the consumer or small-time artist who, again arguably, contributes more to the
advancement of the art of music than a big commercial star. The RIAA may well halt
Napster's business, but unaffiliated, de-localized clones, such as FreeNet and Gnutella,
have already appeared, and cannot be likewise shut down with legal action. Because this
benefits the consumer, the market will be forced to conform.
Aggregation, financial and otherwise, ultimately benefits the consumer, and
therefore it will become embedded into the tools with which we buy and sell. There is
evidence of this happening everywhere - comparison-shopping tools and the like are
freely available, and are becoming more and more integrated and invisible with time.
Apple's MAC OS (v8.x and higher) has taken this a step further, building automatic
aggregation directly into the user's operating system. It will no longer be a choice or
separate task the user performs; every user query will aggregate the information from all
available sources.
Aggregation is the future of online commerce, financial and otherwise, and in
many cases is already the most beneficial method of commerce for consumers.
Companies are successfully applying several business and technology models to create
successful aggregation services, and these will only improve with the pace of technology.
62
Appendix A: Results of Company Categorization Grouped by Model
Please note that the inter-graph totals of some markets may exceed 100% as some
companies use more than one model, and have therefore been counted more than once.
TechnologyModels
Real-Time Agent Usage
120%
100%
80%
Percent Use 60%
40%
43
43
-27%-
0%
20%
0%
a)
a)
CD
(D
E) CA
a)
Cu
CA
a)
Cu
:3
0
0-
"0
0
CL
CAC
a)
0)
C
E
0
-j
0%
F-1
C:
02
i3
0
~
C
0
C.)
2
=o
2
LL'
t
< 7
LL
<
0O
t
< C
0
Aggregation Market
Figure 32 Real-Time Agent Usage
Spider Usage
a)
-fl
,.,
a)
C.)
a)
120%
100%
80%
60%
40%
20%
0%
a)
cc
Mu
0
a
-0
a.
EC
Q=uo
(nM
>
C
)
0
<a
t3
0
<
C
-
C
aCT
U
0
.
3Z 0
C
0
0)
0
<
0
U
< 5
34)0
Aggregation Market
Figure 33 Spider Usage
63
0
9
=
9
CD
0)
0
Ca
Ca
CD
Ca
0)
Auction
Consolidation
Consolidation
Web Account
Flier
Aggregation
Frequent
Auctions
Airfare
Pricing
Consumer
Product
Investment
Rates
Rates
Insurance
Loan Rates
c
*
10
10
$
.-,1-0
0 0- 1 .-,10
N)1O)O$O0)
0000000
Percent Usage
C"
0)
(0
CD
C
'ii
0
9
0
-4
9
C)
a)
CO
CO
CD
CO
Consolidation
Auction
Web Account
Consolidation
Flier
Aggregation
Frequent
Auctions
00
Th
CD
0
a)
0
110
CD
Airfare
Pricing
C
Auction
Consolidation
Web Account
Consolidation
Frequent
Flier
Aggregation
Auctions
Airfare
Product
Pricing
0
0
as
CO
CO
CD
CO
Consumer
Product
-
Consumer
-4
Investment
Rates
Rates
Investment
Loan Rates
Insurance
Rates
9o
(IQ
9t
Insurance
Rates
Loan Rates
0(3101001
Percent Use
0000000
Percent Use
CD
Auction
Consolidation
Consolidation
Web Account
Aggregation
Flier
Frequent
Auctions
0
0
CO
CD
C
Airfare
CD
0
CO
CD
0
U,
-I
15
0
Consumer
Product
Pricing
Investment
Rates
Rates
Insurance
Loan Rates
KN0kO)D0N)
0000000O
Percent Use
0)
C
0.
as
as
9
0
9
9
0
(4
QS
0
0
(4
(0
(0
w
rio
9
=
C,)
9
=
(D
0
Auction
Consolidation
Web Account
Consolidation
Aggregation
Flier
Frequent
Auctions
Airfare,
c
a)
c
c
0
QI
C
CD
0
C0p
(0
Airfare
Auction
Consolidation
Consolidation
Auction
Consolidation
Web Account
CD
C
C
Consumer
Product
Pricing
Investment
Rates
Rates
Insurance
Loan Rates
Web Account
Consolidation
00
9
Frequent
Flier
Aggregation
0
Frequent
Flier
Aggregation
(
0=
Auctions
0)
-0
0
0 (10Cn
Percent Use
Auctions
Airfare
Consumer
Product
Pricing
Consumer
Product
Pricing
CA
Investment
Rates
Loan Rates
Investment
Rates
ft
9T
Insurance
Rates
=%X04
il
Insurance
Rates
Loan Rates
0
Percent Use
I
Percent Use
1
D
0D
ii
Appendix B: Categorization of Major Online Aggregators
Note: The information provided about these companies is for a large part derived from
speculation and experimenting with the public site, as detailed operational information is
seldom obtainable. In addition, company descriptions may be taken in part or whole
from descriptions published by the companies themselves or from other publicly
available research tools (notably, Hoover's Online).
Company Name
AmazingRates, Inc.
Domain Name
www.amazingrates.com
Stock Ticker Symbol
n/a
Basic Market
Investment Rates
Business Model
Brokerage
Technology Model
Manual Entry
General Description
Amazing rates stores CD rates information from over
20,000 institutions and provides users with the best 3 to 5
quotes within their geographic region.
Other
Company Name
Auction Watch
Domain Name
www.auctionwatch.com
Stock Ticker Symbol
private
Basic Market
Auction Consolidation
Business Model
Broker
Technology Model
Real-Time Agent
General Description
AuctionWatch is a leading online auction aggregator. They
provide tools for sellers to customize and manage auctions
across several online auction sites. In addition, they provide a
tool to allow buyers to search and bid at auctions across these
same sites.
Other
67
Company Name
bankcd.com
Domain Name
bankcd.com
Stock Ticker Symbol
n/a
Basic Market
Investment Rates
Business Model
Merchant
Technology Model
Manual Entry
General Description
This site compares CD rates from different banks across the
nation. It presents a list of terms, and the customer pays to view
the name of the bank offering that package.
Other
Company Name
Bankrate, Inc.
Domain Name
www.bankrate.com
Stock Ticker Symbol
RATE
Basic Market
Loan Rates, Investment Rates
Business Model
Broker, Subscription
Technology Model
Manual Entry
General Description
[Bankrate] offers content on banking, financial planning, and
investing, and serves as the gateway to a string of other
financial Web sites. Among the company's print publications
are Green Magazine, several financial newsletters, and a
consumer mortgage guide. (Hoover's Online)
Other
68
Company Name
BANX.com, Inc.
Domain Name
www.banxquote.com
Stock Ticker Symbol
private
Basic Market
Loan Rates, Investment Rates
Business Model
Broker
Technology Model
Real-Time Agent
General Description
BanxQuote@ operates a network of Web sites providing realtime market data from leading financial service providers. It
also features state-by-state, regional and national composite
benchmarks, as well as useful links and snapshot profiles of its
financial service providers. (Banxquote website)
Other
Company Name
BidCrawler.com
Domain Name
www.bidcrawler.com
Stock Ticker Symbol
OTCBB: BCWOF
Basic Market
Auction Consolidation
Business Model
Merchant
Technology Model
Real-Time Agent
General Description
BidCrawler provides a search tool to help businesses and
consumers search across many online auctions to find the best
deals available on the desired products.
Other
69
Company Name
Bidder's Edge
Domain Name
www.biddersedge.com
Stock Ticker Symbol
private
Basic Market
Auction Consolidation
Business Model
Broker
Technology Model
Real-Time Agent
General Description
The tools and services offered by Bidder's Edge, Inc. help
online auction users easily and automatically find great
bargains from many online auctions, all from a single "portallike" web site. Users can bid smarter and be more informed
consumers, thanks to Bidder's Edge. (Bidder's Edge website)
Other
Will cease operations on Feb. 21, 2001.
Company Name
BizRate
Domain Name
www.bizrate.com
Stock Ticker Symbol
N/A
Basic Market
Consumer Products
Business Model
Infomediary
Technology Model
Manual Entry
General Description
The online rating service collects feedback from consumers
shopping at some 4,600 e-retailers that offer merchandise
including clothing, sporting goods, travel reservations,
auctions, and other items. BizRate.com receives no money
from the retailers, and takes in revenue by selling market
research and from promotions. BizRate.com also makes its
ratings available to Consumer Reports (online and magazine
versions) and to portals such as MSN, AltaVista, and CNET.
(Hoover's Online)
Other
70
Company Name
Clickmarks.com, Inc.
Domain Name
www.clickmarks.com
Stock Ticker Symbol
n/a
Basic Market
Account Consolidation
Business Model
Merchant
Technology Model
Real-Time Agent, Spider
General Description
Clickmarks allows users to consolidate any web-based content
into one custom portal, and the view that content on any
device. Information contained within a password-protected
account is still accessibly with a Clickmarks personal habitat.
Other
Company Name
ConsumerREVIEW, Inc.
Domain Name
www.consumerreview.com
Stock Ticker Symbol
n/a
Basic Market
Consumer Products
Business Model
Affiliate, Merchant
Technology Model
Manual Entry
General Description
Consumer Review provides reviews of consumer products.
These reviews are entered by consumers and include a
summary about the item in question, as well as an overall
rating and a value rating.
Other
71
Company Name
DealTime.com Ltd.
Domain Name
www.dealtime.com
Stock Ticker Symbol
DEAL (proposed, Nasdaq)
Basic Market
Consumer Products
Business Model
Affiliate
Technology Model
Real-Time Agent
General Description
Dealtime provides users the ability to compare products before
they shop. They have sites targeting the U.S., the U.K. and
Germany.
Other
Company Name
eBay.com Inc.
Domain Name
www.ebay.com
Stock Ticker Symbol
EBAY
Basic Market
Auctions
Business Model
Broker
Technology Model
Real-Time Agent
General Description
eBay is the largest online auction service. They are the
standard by which other online auctions are measured.
Other
It recently acquired half.com, a marketplace combining
traditional auction style trading and fixed-price trading.
72
Company Name
ebix.com, Inc.
Domain Name
www.ebix.com
Stock Ticker Symbol
EBIX
Basic Market
Insurance Rates
Business Model
Broker
Technology Model
Manual Entry
General Description
Ebix.com (formerly Delphi) provides insurance quote
aggregation in the form of an online auction. Consumers bid
on rates posted by insurance companies.
Other
Company Name
E-Loan
Domain Name
www.e-loan.com
Stock Ticker Symbol
EELN
Basic Market
Loan Rates
Business Model
Broker
Technology Model
Manual Entry
General Description
E-LOAN offers a full range of mortgages, auto loans, credit
cards and small business loans and is committed to
continuously adding new ones. Customers can search for the
best loan from thousands of products with a single click. A
comprehensive set of tools and information gives customers
everything they need to select the right product -- all in one
place. (E-Loan webpage)
Other
73
Company Name
Epinions.com
Domain Name
www.epinions.com
Stock Ticker Symbol
n/a
Basic Market
Consumer Products
Business Model
Broker, Advertiser
Technology Model
Manual Entry
General Description
Epinions collects consumer reviews on over 150,000 products
and makes these reviews available to the public. Currently,
they have over 1 million reviews on these products.
Other
Company Name
FiNet
Domain Name
www.finet.com
Stock Ticker Symbol
FNCMC
Basic Market
Loan Rates
Business Model
Broker
Technology Model
Real-Time Agent
General Description
FiNet uses a real-time agent, iQualify, to find out for which
loans a user qualifies. They have a parnership with
Homeseekers.com to help users find their dream homes, and
with AskJeeves and CoxInteractive Media to provide loan
prospects.
Other
FiNet is a trade name for Monument Mortgage, Inc.
74
Company Name
Flycheap.com
Domain Name
www.flycheap.com
Stock Ticker Symbol
n/a
Basic Market
Airfare
Business Model
Broker
Technology Model
Manual Entry
General Description
Flycheap helps users find economical airfare, displaying only
flights that are actually available. They have realtionships with
most (but not all) major airlines.
Other
Company Name
Frequent Flier Points
Domain Name
frequentflierpoints.com
Stock Ticker Symbol
n/a
Basic Market
Airfare
Business Model
Merchant
Technology Model
Manual Entry
General Description
Frequent Flier Points helps consumers with extra frequent flier
miles receive cash for their miles, and uses these miles to
provide discounted business and first class airfare around the
world.
Other
Affiliate of Travel Management Systems.
75
Company Name
Frugalflyer.com
Domain Name
www.frugalflyer.com
Stock Ticker Symbol
n/a
Basic Market
Airfare
Business Model
Broker
Technology Model
Manual Entry
General Description
Frugalflyer.com is a provider of discount tickets and services
for leisure travel. In addition to providing their own listing of
discount tickets, they provide a gateway to Travelocity and
Priceline.com.
Other
Company Name
Gator.com
Domain Name
www.gator.com
Stock Ticker Symbol
n/a
Basic Market
Account Consolidation
Business Model
Broker, Infomediary
Technology Model
Manual Entry
General Description
Gator is a smart online companion that fills out forms and
remembers passwords. Gator comes with the
OfferCompanionSM application, the premier application for
saving money on the web. Yet Gator and OfferCompanion are
extremely polite, staying out of sight, popping up only when
they can help you. (Gator website)
Other
76
Company Name
InsureChoice
Domain Name
http://www.infochoice.com.au/insurance/health/
Stock Ticker Symbol
Australian Stock Exchange: ICH
Basic Market
Insurance Rates
Business Model
Brokerage, Subscription
Technology Model
Manual Entry
General Description
InsureChoice, a subsidiary of InfoChoice, provides unbiased
insurance quotes throughout Australia.
Other
Company Name
InsWeb Corporation
Domain Name
www.insweb.com
Stock Ticker Symbol
INSW
Basic Market
Insurance Rates
Business Model
Broker
Technology Model
Manual Entry
General Description
Online insurance marketplace InsWeb offers quotes on such
insurance products as auto, term life, homeowners, renters, and
individual health insurance from its Web site, as well as
through links from other online partners. (Hoover's Online)
Other
77
Company Name
iOwn
Domain Name
www.iown.com
Stock Ticker Symbol
private
Basic Market
Mortgage Rates
Business Model
Broker
Technology Model
Manual Entry
General Description
iOwn.com is a web portal for housing purchase. They offer
online help for all of the "three-part housing purchase cycle."
In addition, they offer a mortgage rate aggregation service and
an online mortgage application.
Other
Company Name
Lending Tree
Domain Name
www.lendingtree.com
Stock Ticker Symbol
TREE
Basic Market
Loan Rates
Business Model
Broker
Technology Model
Real-Time Agent
General Description
Lending Tree bills itself as a loan market; consumers complete
one loan application that is shopped around to many lenders.
Other
Lending tree provides Lend-X, software to help automate the
lending process in real time.
78
Company Name
Lowestfare.com
Domain Name
www.lowestfare.com
Stock Ticker Symbol
FARE proposed, Nasdaq
Basic Market
Airfare
Business Model
Broker, Advertiser
Technology Model
Manual Entry
General Description
Lowestfare.com sells discounted airfare from over 400 airlines
online through their website and over 10,000 travel agents.
Other
It also has 1800 number that is available 24x7. The company,
owned by financier Carl Icahn, recently acquired Jetset Tours,
an international airline ticket consolidator and tour operator,
and Maupintour, a luxury escorted tour provider.
Company Name
MaxMiles
Domain Name
www.maxmiles.com
Stock Ticker Symbol
NCNT
Basic Market
Frequent Flier Miles
Business Model
Subscription
Technology Model
Spider
General Description
MaxMiles automatically gathers all of your frequent flyer
balances and account information from airlines, hotels, and
credit cards, analyzes the most current mileage offers, searches
for missing mileage credits, and provides a personalized report.
Other
Owned by Netcentives.
79
Company Name
MaxRate.com
Domain Name
www.maxrate.com
Stock Ticker Symbol
private
Basic Market
Investment rates
Business Model
Broker
Technology Model
Manual Entry
General Description
MaxRate.com is registered with the FDIC as a CD broker.
MaxRate is a market intermediary between the banks and CD
investors but is not a bank.
Other
Not operational as of January 31, 2001.
Company Name
Microsoft Expedia
Domain Name
www.expedia.com
Stock Ticker Symbol
MSFT
Basic Market
Airfare
Business Model
Broker, Advertiser
Technology Model
Manual Entry
General Description
Microsoft-owned online travel agent.
Other
---------------------------------------------------------------
80
Company Name
Microsoft Passport
Domain Name
www.passport.com
Stock Ticker Symbol
MSFT
Basic Market
Account Consolidation
Business Model
Broker
Technology Model
Manual Entry
General Description
Microsoft's consolidation site that collects password and credit
card information targeted for ease of shopping online. Use
ONE sign-in name and password at all Passport sites. Sign in at
any participating Passport site, and then sign in to other
participating sites with just a single click. Store information in
your Passport "wallet" that will help you make faster, safer
online purchases at any Passport express purchase site.
Other
Company Name
Milepoint.com
Domain Name
www.milepoint.com
Stock Ticker Symbol
n/a
Basic Market
Frequent Flier Miles
Business Model
Broker
Technology Model
Spider
General Description
MilePoint.com is an Internet exchange site that allows
members to convert their frequent flyer miles and points into a
new online currency, MilePoint Money, and immediately go to
any participating MilePoint Merchant's online site to begin
shopping. Online merchandise or gift certificates that are
currently offered on their site can be purchased using
MilePoint Money as partial payment. (MilePoint webpage)
Other
81
Company Name
money-rates.com
Domain Name
www.money-rates.com
Stock Ticker Symbol
private
Basic Market
Investment Rates
Business Model
Advertiser
Technology Model
Manual Entry
General Description
Money-rates.com provides free comparisons of financial
account rates. These rates are updated manually by the bank in
question.
Other
Company Name
Mypassword.net
Domain Name
www.mypassword.net
Stock Ticker Symbol
n/a
Basic Market
Account Consolidation
Business Model
Advertiser
Technology Model
Manual Entry
General Description
MyPassword.net provides a password storage and
bookmarking service. They will automatically fill in
passwords for their users.
Other
82
Company Name
mySimon
Domain Name
www.mysimon.com
Stock Ticker Symbol
n/a
Basic Market
Consumer Products
Business Model
Advertiser
Technology Model
Spider
General Description
mySimon is the largest comparison shopping site on the Web,
aggregating over 2,000 merchants, and providing information
about products such as price, shipping details, and availability.
Other
Company Name
Obongo, Inc.
Domain Name
www.obongo.com
Stock Ticker Symbol
n/a
Basic Market
Consumer Products
Business Model
Broker
Technology Model
Real-Time Agent
General Description
Obongo provides, among other products, a consumer product
aggregation tool that combines price comparisons from
mySimon with consumer reviews from epinions.com.
Other
83
Company Name
OnMoney.com Financial Services Corp
Domain Name
www.onmoney.com
Stock Ticker Symbol
n/a
Basic Market
Account Consolidation
Business Model
Advertiser, Affiliate
Technology Model
Real-Time Agent, Spider
General Description
OnMoney allows users to maintain their entire financial
portfolio in one place. It allows them to retrieve automatic
updates and to interface with many web-accessible financial
accounts.
Other
OnMoney was originally funded by Ameritrade Holding Corp.
Company Name
OverBID.com
Domain Name
www.overbid.com
Stock Ticker Symbol
n/a
Basic Market
Auction Consolidation, Consumer Product Pricing
Business Model
Broker
Technology Model
Real-Time
General Description
Overbid has combined both retail and auction sites into a single
virtual storefront, providing a complete list of products in a
single search.
Other
84
Company Name
Passwordsafe.com
Domain Name
www.passwordsafe.com
Stock Ticker Symbol
n/a
Basic Market
Account Consolidation
Business Model
Advertiser
Technology Model
Manual Entry
General Description
Password Safe is an online tool to help users remember and
manage the passwords to their online accounts.
Other
Company Name
Planetfeedback.com
Domain Name
www.planetfeedback.com
Stock Ticker Symbol
n/a
Basic Market
Consumer Products
Business Model
Infomediary
Technology Model
Manual Entry
General Description
PlanetFeedback is the premier online destination and utility for
consumer feedback. They connect consumers with companies
and make it easy to send feedback of all kinds quickly, easily
and for free. It guides consumers through a process of writing
and sending compliments, complaints, questions and
suggestions to any entity or organization specified by the
consumer. (Planetfeedback.com website)
Other
85
Company Name
PriceGraber.com, LLC
Domain Name
www.pricegrabber.com
Stock Ticker Symbol
n/a
Basic Market
Consumer Products
Business Model
Advertiser
Technology Model
Spider
General Description
PriceGrabber is a provider of online shopping comparison
services including searching and tracking of products with
MyPriceGrabber, a free service that makes online shopping
easier for consumers.
Other
Company Name
Priceline.com
Domain Name
www.priceline.com
Stock Ticker Symbol
PCLN
Basic Market
Airfare
Business Model
Broker, Advertiser
Technology Model
Manual Entry
General Description
Using a simple and compelling consumer proposition--"name
your price," Priceline collects consumer demand (in the form
of individual customer offers guaranteed by a credit card) for a
particular product or service at a price set by the customer and
communicate that demand directly to participating sellers or to
their private databases. Consumers agree to hold their offers
open for a specified period of time to enable priceline.com to
fulfill their offers from inventory provided by participating
sellers. Once fulfilled, offers generally cannot be canceled. By
requiring consumers to be flexible with respect to brands,
sellers and/or product features, they enable sellers to generate
incremental revenue without disrupting their existing
distribution channels or retail pricing structures. (Hoover's
Online)
Other
86
Company Name
Q*Wallet.com
Domain Name
www.qwallet.com
Stock Ticker Symbol
n/a
Basic Market
Account Consolidation
Business Model
Advertiser
Technology Model
Manual Entry
General Description
Q*Wallet is a small, quick application that provides you with
instant access to personal information when you really need it.
Credit card numbers, site logins and passwords, and frequent
flyer numbers are all securely stored on your computer's hard
drive by Q*Wallet. When you need to use a credit card, just
click on the Q*Wallet icon in the system tray, and drag and
drop your credit card number and expiration date directly onto
the web page. (Q*Wallet website)
Other
Company Name
Quicken Loans
Domain Name
quickenloans.quicken.com
Stock Ticker Symbol
INTU
Basic Market
Loan Rates
Business Model
Broker, Merchant
Technology Model
Manual Entry
General Description
Quicken Loans makes sure you get the right loan, the right
way. It offers residential mortgages in all 50 states and
provides a wide variety of mortgage products including
conventional, alternative, home equity, jumbo, and government
loans. (From Quicken Loans website)
Other
Subsidiary of Intuit. In addition to standard loan comparison,
Quicken Loans has a joint venture with Michigan National
Bank to provide loans for customers.
87
Company Name
Quotesmith.com, Inc.
Domain Name
www.quotesmith.com
Stock Ticker Symbol
QUOT
Basic Market
Insurance Rates
Business Model
Broker
Technology Model
Manual Entry
General Description
Quotesmith.com gives Web surfers instant quotes from more
than 300 insurance companies for such coverage as whole and
term life; dental; family, individual, and small group medical;
Medicare supplements; and fixed annuity. (It also provides
auto quotes through a "click-through" arrangement with
Progressive.) The free service processes applications for the
insurers and even offers users a $500 cash guarantee on the
accuracy of its quotes. Quotesmith.com earns fees from
insurance policies sold through its site. (Hoover's Online)
Other
Company Name
Rateitall.com
Domain Name
www.rateitall.com
Stock Ticker Symbol
n/a
Basic Market
Consumer Products
Business Model
Merchant
Technology Model
Manual Entry
General Description
RateItAll provides consumer opinions on everything. Relevant
to this paper are their ratings on consumer products.
Other
88
Company Name
Totalmiles.com
Domain Name
www.totalmiles.com
Stock Ticker Symbol
n/a
Basic Market
Frequent Flier Miles
Business Model
Subscription, Merchant
Technology Model
Spider
General Description
Total Miles allows users to keep track of all of their frequent
flier accounts. Commercial users can host an internal version
on their own servers.
Other
Company Name
Travel Services International, Inc.
Domain Name
www.mytravelco.com
Stock Ticker Symbol
n/a
Basic Market
Airfare
Business Model
Broker
Technology Model
Manual Entry
General Description
This is an online airfare site and specializes in cruise vacations.
Other
A wholly owned subsidiary of Airtours plc.
89
Company Name
Travelocity
Domain Name
www.travelocity.com
Stock Ticker Symbol
TVLY
Basic Market
Airfare
Business Model
Broker, Advertiser
Technology Model
Manual Entry
General Description
The leading online travel website and the third largest ecommerce site, Travelocity.com provides reservations
capabilities for 95 percent of all airline seats sold, more than
49,000 hotels, more than 50 car rental companies and more
than 5,000 vacation and cruise packages. (Hoover's Online)
Other
Company Name
TRIP.com
Domain Name
www.trip.com
Stock Ticker Symbol
GLC
Basic Market
Airfare
Business Model
Broker, Advertiser
Technology Model
Manual Entry
General Description
TRIP.com is the premier one-stop online travel service and
technology provider devoted to the mobile professional market,
whether traveling for business or pleasure. It provides
registered users with a host of tools and services including
round-the-clock reservation capabilities, city, restaurant and
hotel information and 24-hour personalized customer service.
(TRIP.com website)
Other
TRIP.com is a subsidiary of Galileo International, Inc.
90
Company Name
ubid.com
Domain Name
www.ubid.com
Stock Ticker Symbol
CMGI
Basic Market
Auctions
Business Model
Broker
Technology Model
Real-Time Agent
General Description
uBid.com is a leading e-commerce auction site offering
consumers and small to medium-sized businesses the
opportunity to "set the price" on a wide range of brand name
merchandise through 24-hour live-action bidding using
sophisticated auction technology. uBid.com features a rotating
selection of more than 6,700 brand products a day. (uBid
website)
Other
Company Name
ValueStar
Domain Name
www.valuestar.com
Stock Ticker Symbol
VLST
Basic Market
Consumer Products
Business Model
Infomediary
Technology Model
Manual Entry
General Description
ValueStar uses a four-step rating system to certify local service
companies in industries including auto and home repair as well
as professional services and health care, then lists the best ones
on its Web site. Companies pay a fee for the certification
process, which surveys customer satisfaction and examines
whether the firms have the proper licenses and insurance, and
whether any complaints have been lodged against them.
(ValueStar website)
Other
91
Company Name
VerticalOne Coporation
Domain Name
www.verticalone.com
Stock Ticker Symbol
SONE
Basic Market
Account Consolidation
Business Model
Broker
Technology Model
Real-Time Agent
General Description
VerticalOne enables the next generation of Internet content
personalization services by consolidating, organizing and
presenting Internet users' personal account information. One
master password provides consumers with a single, easy to
access interface for such personal account information as bank
and brokerage statements, credit card balances, voice mails, Emails, household bills and travel award programs. (VerticalOne
website)
Other
VerticalOne Corporation is a wholly owned subsidiary of S 1
Corporation. It joined forces with Yodlee in 12/2000.
Company Name
Walletonline.com
Domain Name
www.walletonline.com
Stock Ticker Symbol
n/a
Basic Market
Account Consolidation.
Business Model
Advertiser
Technology Model
Manual Entry
General Description
Walletonline lets you manage your passwords, access your
websites with just one click, and manage your personal
information in a hassle free environment.
Other
92
Company Name
Yahoo Auctions
Domain Name
auctions.yahoo.com
Stock Ticker Symbol
YHOO
Basic Market
Auctions
Business Model
Broker
Technology Model
Real-Time Agent
General Description
Yahoo is a general-purpose portal and search engine on the
web. It is arguably the best known, and is the leading online
advertiser. Yahoo Auctions is an auction site maintained by
Yahoo, and is the second largest on the web, after Ebay.
Other
Company Name
Yahoo! Inc. Wallet
Domain Name
wallet.yahoo.com
Stock Ticker Symbol
YHOO
Basic Market
Account Consolidation
Business Model
Broker, Advertiser
Technology Model
Manual Entry
General Description
Yahoo's site that saves and stores shipping, billing, and credit
card information for future purchases. It provides "easy check
out at 1 000s of merchants" and can be used for purchases on
Yahoo as well.
Other
93
Company Name
Yodlee. Inc.
Domain Name
www.yodlee.com
Stock Ticker Symbol
n/a
Basic Market
Account Consolidation
Business Model
Merchant
Technology Model
Real-Time Agent
General Description
Yodlee's e-Personalization Platform enables e-businesses to
offer their customers a convenient way to consolidate and
manage all of their personal online information. Customers get
one-click access to a dynamic summary of the personal
information that matters most - bank balances, travel
reservations, investments, email, shopping, bills, and calendars,
etc. Users can access their information from a PC, PDA, or
web-enabled phone. (Yodlee website)
Other
Partner with VerticalOne.
94
References
Tabb, Larry. "SIA 2000 Special Report: Redefining Convergence Via the Internet." Securities Industry
News 6/19/2000
2 Madnick, Stuart; Siegel, Michael; Frontiti, Mary Alice; Khemka, Saraubh; Chan, Steven; Pan, Howard.
"Aggregator: The New Business of the World Wide Web." p. 5
3 Resto, Christopher, Consultant, Cap Gemini Ernst & Young. Personal Interview 6/23/2000
4 Rappa, Michael. "Managing the Digital Enterprise: Business Models on the Web." NC State University.
http://ecommerce.ncsu.edu/business models.html
5 Rhodes, Brahm. MIT MITES program EntrePrep course lecture. July 1999
6 Tufail, Rizwan. V.P. Marketing, Clickmarks.com. Personal Interview 6/20/2000
' Rhodes
8Rappa
9 Rappa
10 Rappa
" Chan, MeiLin. U.C. Berkeley. Personal Interview 7/11/2000
1 Jupiter Research report cited by Cyberatlas. 2/1997
http://www.cvberatlas.com/segments/advertising/m,'arket forecast.html.
13 Joseph, Patrick. "On-line Advertising Goes One-On-One." Scientific American 12/1997
http://www.sciam.com/1297issue/I297cyber.html
14 Rappa
1
Jacobus, Patricia; Macavinta, Courtney; Hansen, Evan. "Is DoubleClick Privacy
Plan Adequate?" CNET
2/14/2000
16 Chan
17 Lund, Brian. "Amazon Entertains Future Distribution Spin-off' The Motley Fool 4/4/2000
http://www.fool.com/news/foolplate/2000/foolplate000404.htm
18 Hartigan, Patti; "Web Porn Biz Finds Niche
In Suburbia." Boston Globe 6/2/2000
19 Chan
20 Cheong, Fah-Chun. Internet Agents: Spiders, Wanderers,
Brokers, and Bots. Indianapolis: New Riders
Publishing, 1996.
Lindquist, Christopher. "Supersonic Chips: How Fast Can Processors Get?" CNET
http://computers.cnet.com//lardware/0-1017-7-270843.html
22 Stock Smart Currencies. AvantGo https://avantgo.coin/channels/deatil.html?cha
id=1247&cat id=23
3 Zona Research April 2000
24 Clark, David. "Shopbots Become Agents
for Business Change." Computer 2/2000: 20.
21
25
Winston, Patrick. Artificial Intelligence. Addison-Wesley,
1992.
26
27
Cheong
Winston
Cheong
Cheong
28
29
30
31
InFoPeopleSearch Tools Chart http://www.infopeople.org/src/chart.html
Cheong
Anonymous Employee, Applied Materials Corp. Personal
Interview. 10/2000
33 InFoPeople Search Tools Chart
34 InFoPeople Search Tools Chart
3 eBay, Inc. http://pages.ebay.com/community/aboutebay/overview/benchmarks
html
36 Koster, Martijn. "The Standard for Robot Exclusion." 1994
http://info.webcrawler.com/mak/projects/robots/norobots.html
37 Sullivan, Danny, "Auction Search Case Awaits Ruling." The Search Engine Report.
5/3/2000
http:/searchenginewatch.
com/sereport/00/05-ebav.html
38 Cheong
32
95
39
Cheong
40 Cheong
41 Cheong
Kaplan, Carl. "Auction Dispute Centers on Question of Control Over Data." The New York Times on the
Web Cyber Law Journal.7/4/2000 http://www.nytimes.com/library/tech/00/04/cyber/cyberlaw/14law.html
43 Kaplan, Carl. "Judge Says a Spider Is Trespassing on Ebay." The New York Times on the Web Cyber Law
Journal.5/26/2000 http://www.nytimes.com/library/tech/00/05/cyber/cyberlaw/26law.html
44 Sullivan 5/3/2000
45 Kaplan 5/26/2000
46 Sullivan, Danny. "News Robot Leads To Linking, Indexing Dispute." The Search Engine Report.
1/9/1998 http://searchenginewatch.com/sereport/98/01 -newsindex.html
47 Kaplan, Carl. "Napster May Have Weak Defense in Fight With Music Industry." The New York Times on
the Web Cyber Law Journal.5/12/2000
http://www.nytimes.com/library/tech/00/05/cvber/cyberlaw/12law.html
48 Kaplan, Carl. "Legality of 'Deep Linking' Remains Deeply Complicated." The New York Times on the
Web Cyber Law Journal.4/7/2000 http://www.nytimes.com/library/tech/00/05/cyber/cyberlaw/07law.html
49Kaplan, 4/7/2000
50 "Ticketmaster Corp., et al. v. Tickets.Com, Inc." U.S. DistrictCourt, CentralDistrict of California.
3/27/2000
5 Kaplan, 4/7/2000
42
96