Customer Case Study
Costco.com
“
Spirent’s Avalanche is unique because it has enabled
us to test the capacity and performance of our Web
infrastructure and applications while verifying functionality
Highlights
at very high loads.
Spirent
Communications
26750 Agoura Road
Calabasas Hills, CA
91302 USA
e: enterprise@spirentcom.com
www.spirentcom.com/enterprise
Company.
”
–Don Allen, E-commerce Infrastructure Manager,
Costco Wholesale Corporation
Costco Wholesale Corporation
Industry/Market.
Enterprise/Retail
Key Business Issues.
Ensure that Web site can withstand
high-volume loads—up to 500,000
pages views per hour in times of
peak demand.
■ Support e-commerce activities
such as browsing, login and
purchasing without negatively
affecting performance.
■ Evaluate changes to application
code before releasing them
to production.
■ Assess the capacity of a major
infrastructure update that includes
new hardware and software.
■
Results.
The Avalanche system allows
Costco to replace racks of client
workstations with a single
integrated appliance, saving
time and expense.
■ Costco’s QA team uses the
Avalanche to test the effect
of new code releases under
realistic, real-world conditions.
■ Flexible scripting capabilities allow
QA to record and play back
transactions, enabling the QA
team to quickly set up tests
without writing new code.
■ With its ability to simulate a
virtually unlimited number of
users, the Avalanche allows Costco
to accurately evaluate the stability of its new infrastructure and
troubleshoot problems under highvolume loads—before the new site
goes live.
■
Wholesaler puts new
Web infrastructure to
the test with Avalanche.
In today’s competitive ecommerce
environment, enterprises need
to know that their Web sites can
withstand the purchasing activities
of thousands—even millions—of
shoppers. But testing Web site
and infrastructure performance
can be expensive, requiring large
investments of equipment and
staff. And no company is more
aware of this than wholesaler
Costco, which has built its business
on providing members with the
best possible value for their money.
To meet this challenge, Costco
Wholesale Corporation recently
turned to the Avalanche™ capacity
assessment appliance from
Spirent Communications. The
Avalanche helps Costco’s QA team
test the functionality of its Web
site, Costco.com, on an ongoing
basis, allowing Costco to replace
racks of client workstations with
a single, easy-to-use appliance
and save time, effort and expense.
Costco is also using the Avalanche
to evaluate the scalability and
reliability of a major infrastructure
update to Costco.com—ensuring
that when the new site goes live,
Costco will be ready.
Shortened, more
efficient QA test cycles.
Costco Wholesale Corporation
operates an international chain of
membership warehouses that carry
quality, brand-name merchandise
at substantially lower prices than
are typically found at conventional
wholesale or retail sources. With
nearly 20 million members, the
company operates 418 warehouses
and an e-commerce Web site,
Costco.com.
Costco.com is a crucial part of
the company’s online business
strategy. Currently, the Web site
must be able to withstand traffic
loads that can reach 5 million page
views every day—500,000 page
views an hour at times of peak
demand—and support customer
activities such as logins, searching
and purchases.
Costco’s Web site is developed
and maintained internally. To
test the performance of the site
under various loads, Costco’s
QA department previously used
a software solution based on
Microsoft’s Web Application Stress
(WAS) Tool running on eight client
workstations. After running the
tool on the clients, the QA team
then ran the same tests in the
production environment.
“It was very cumbersome,” says
Don Allen, manager of e-commerce
infrastructure for Costco.com.
“We had to physically move the
machines, change the IP addresses
and spend more time getting them
set up—a process that could take
as long as full day—before we could
even start testing.”
After reading about Avalanche in
an industry magazine and meeting
with Spirent, Costco purchased
an Avalanche system to help with
Costco.com
capacity testing of the Costco.com
site. The Avalanche helps Costco
reduce the QA cycle by allowing QA
engineers to set up tests much more
quickly than they could previously.
For example, according to Tom Jones,
Costco QA specialist, he was able
to set up the Avalanche system and
begin running tests the same day.
“When you compare that with
how long it takes to set up eight to 10
load testing clients, it’s a substantial
savings of both time and money,”
Jones says. “Managing the Avalanche
is much easier than supporting
multiple desktops or servers.”
Reliable evaluation
of new code.
Costco’s QA process also benefits from
Avalanche’s unique ability to record and
play back transactions, allowing the QA
group to create scripts that will perform
not only basic browsing functions
but more complicated e-commerce
activities such as login, checkout and
purchasing, as well as SSL transactions.
“Once I start the record feature, I
can go to our test site and navigate
as if I was a user, and the script
recorder records each step that I
take,” says Jones. “When I have
the script recorded, I can set up a
test that will run that script as if
thousands of users were navigating
the site. It saves an incredible
amount of time, because we don’t
have to write new code every time
we want to run a test.”
For example, as Costco
entered the holiday season, the
development team used Avalanche
to test the effect of a change to the
order tracking page.
Spirent
Communications
26750 Agoura Road
Calabasas Hills, CA
91302 USA
e: enterprise@spirentcom.com
www.spirentcom.com/enterprise
“Our development team wanted to
evaluate the impact of the change on
site performance,” says Allen. “They
came to us to run a load test using
Avalanche, and we were able to give
them realistic, objective results.”
2
Testing a new Web
infrastructure.
The Avalanche system is also
playing a key role in a major project
to update the Costco.com site and
supporting infrastructure. Costco is
currently migrating from Microsoft
SiteServer to the Commerce Server
2002 platform, incorporating
Microsoft’s .NET framework.
Meanwhile, to handle projected
growth, Costco is also upgrading
the hardware with new switches,
load balancers and firewalls. The
Avalanche will play a key role
in helping to test and configure
the new infrastructure to run at
optimum levels.
“Having undertaken this project
to modernize our Web site, we
need to accurately simulate both
network load and the experience of
real users,” says Allen. “Spirent’s
Avalanche is unique because it
has enabled us to test the capacity
and performance of our Web
infrastructure and applications
while verifying functionality at very
high loads.”
Avalanche can initiate and
maintain more than a million
concurrent connections, each
appearing to come from different
IP address. This gives Costco more
than enough headroom to conduct
realistic and accurate capacity
assessment of the new devices in
the infrastructure, as well as identify
problem areas before the site is
launched. Costco has configured
its Avalanche system with two NIC
cards and two GigE interfaces on
each card.
“The Avalanche system can
scale to simulate an almost infinite
amount of load,” says Allen. “If
we need to, we can add another
interface card that will enable us
to throw more load at the site than
it will ever have to take. Avalanche
allows you to check the stability
of the site but also find out what
its limitations are, and take action
before users show up. It allows us
to stay ahead of the growth.”
As Costco develops the new
site infrastructure, the QA team will
use the Avalanche system to test
capacity and conduct further testing
on content and applications before
releasing to the production site.
“Nothing’s going to be used on
our site without going through some
form of load testing. In most cases,
this means Avalanche,” says Allen.
Comprehensive reporting.
For Costco, another benefit of
the Avalanche is its extensive,
comprehensive reporting capabilities,
which provide an integrated picture
of all relevant statistics in a single
report. The ability to save reports
in various formats simplifies the
distribution process.
“Using the HTML editor, I can add
comments about certain areas so
that people can better understand
what they’re seeing,” Jones says. “I
can also post reports on an internal
website so that people who need
to see them can do so at any time
without having to contact me or
someone else to send them. Most
people have Web browsers, so they
can easily review the reports.”
The test reports produced
by Avalanche allow Costco’s
development and QA teams to
document the results of changes to
application code and present them
in an easily understandable format.
“The standard reports you get
with Avalanche Analyzer™ are so
extensive that I can always find
the exact report I need, instead of
having to create one myself,” says
Allen. “If there’s a problem on a
page, I can show someone exactly
where it is. Avalanche’s reporting
capability has made our jobs in QA
a lot easier.”
©2004 Spirent Communications. All rights reserved. Spirent Communications, the Spirent Communications logo,
Avalanche and Reflector are trademarks of Spirent Communications.