why
teradata
Caveat
emptor
Beware the unreasonable
claims of some data
warehouse vendors.
by Dan Higgins
E
ver hear advertisements like these?
Drink Hyper-Trim and you will lose 10
pounds in one week without exercising! … With
the new Wing from Golf Master you will hit the
ball straighter and 50 yards farther! … Make
$5,000 a week from the comfort of your own
home—just use the plan from my new book!
It is nearly impossible to avoid sales and
marketing claims that grossly overstate the
value of a product or service. They imply
that an extreme case is the norm. Ultimately,
this strategy risks having the opposite effect
with its prospective customers: Because most
people realize these advertisements are only
borderline truth, they simply dismiss them.
PAGE 1 | Teradata Magazine | December 2009 | ©2009 Teradata Corporation | AR-6036
loaded system measure only a single query
at a time on a system optimized for that
particular scenario.
Misleading comparisons are especially possible when you consider that
the technology for a production system
is most likely many years older than
the brand-new technology used in the
laboratory tests. For instance, after a new
Teradata software version is released,
performance improvements with certain
queries are not uncommon, even on an
older system. With the implementation
of Teradata 13, a 30% overall performance improvement is achieved, with
some workloads performing as much
as 20 times faster. The point is, performance comparisons are nearly always
misleading because the user would see
improvements on any system, regardless
of the age of the technology.
Take my word for it
Misleading statements are not limited to
performance or cost. Vendors also commonly bend industry terminology in an
effort to manufacture a perception about
their technology. Too often, “scalability”
is claimed when it applies only to storing
Photography by iStockphoto
My dad can beat your dad
Over the past few years, a number of data
warehousing technology vendors have
made similar outlandish claims, exaggerations and misleading statements. They tell
prospective and current customers:
Our technology is 10 to 50 times as fast
as other technologies! … Our products are
half the cost of our competitors’! … Our
platform set a world record in transactions
per second! … In a recent customer test, our
product was six times faster than the one
currently in use!
It has become a game of one-upmanship
as companies attempt to top one another
with hyperbolic marketing. First, the
system was 10 times as fast. Then 50. Then
100, and then 300.
Just watch: In a couple of years some
vendor will be announcing technology that
will enable the total population of a major
country to have real-time analytics access
to hundreds of exabytes of data. And you
won’t need any IT staff, because it will all be
handled in the cloud where the proverbial
“miracle” occurs.
Of course, not all advertising claims are
outlandish, nor are they all false. Some
might even state a bit of truth. It is possible
for a query to run 10, 50 or even 100 times
as fast on one system as it does on another.
In fact, it is not unheard-of for queries
to run at these faster speeds on the same
system. All it may require is a simple change
to a line of code or a slightly different
optimizer plan.
Furthermore, it should come as no
surprise that performance on a lightly
loaded laboratory system with a few users is
much faster than a heavily loaded production system with hundreds of users. Often,
the performance tests done on the lightly
more data, rather than supporting more
concurrent applications or users, providing
more extensive data models, etc.
A few years ago, a vendor touted linear
scalability because queries that accessed
only one month’s data would not run
longer when additional months of data
were added. In reality, the data was
partitioned, so the queries would not run
longer anyway. This is not linear scalability as Teradata customers have come
to understand.
Similarly, some vendors claim they
can do “enterprise data warehousing”
because their system promotes leveraging multiple platforms throughout the
organization. But managing multiple
platforms defeats the purpose of having
an enterprise data warehouse.
And while simplicity can certainly be a
system attribute, some merchants use this
term to disguise woefully lacking functionality. Once this “simple” system is installed,
however, the data warehousing customers typically end up placing considerable
burden on IT and end users to either
develop that functionality in-house or suffer without it.
Distinguish apples from oranges
Technologies are not all the same. Most
products in the market have substantial
differences depending on the job, with
advantages and disadvantages, strengths
and weaknesses. In some benchmarks,
a certain technology might excel at one
type of query or workload and struggle
with another.
Because most vendors don’t provide
the full story in their marketing claims,
customers can become victims if they don’t
do their own research. Prospective customers need to know the answers to questions
such as:
>W
hat is the truth and what are the
details surrounding a specific claim?
> How are vendors using industry
terminology? To gain the advantage,
Eyes wide open
Y
ou need to be thorough when considering
alternative technologies. These pointers
will help uncover the facts about a product
before you make an investment:
1.
>U
se a wide variety of queries, including
>V
endor assurance that you will realize similar
problem queries and anticipated and
benefits in a production environment within
unanticipated workloads.
a similar and reasonable timeframe; it may
>M
ake sure the testing process places the
highest priority on production levels of query
Keep these points in mind when using a
concurrency, mix and variety, load scenarios,
proof of concept (POC) or a benchmark:
and so on.
>B
e cautious of POCs and benchmarks that
do not adequately reflect the characteristics
of a production workload, as well as current
>T
est reliability, resiliency and failover with
production workloads and data volumes.
>R
equire full disclosure of test scripts, system
and future demands. The results can be
settings and query results. Information such
dangerously misleading if only a small
as system resource utilization (e.g., CPU
portion of a workload is run or if not all
and I/O) will uncover whether the system
aspects are included. Customers who rely
is balanced or whether bottlenecks might
on these tests will find that their system
create problems. A vendor might be unwilling
behaves much differently in production.
to provide this type of information, but doing
>A
void “lowest common denominator” tests,
which eliminate tests that one or more
vendors are unable to functionally complete.
so could expose a deficiency in the system’s
management tools and capabilities.
>U
se third-party auditors that understand
Benchmarks and POCs should reflect your
your requirements and have experience
current and future technology requirements,
with benchmarks but refrain from making
not the capabilities of specific vendors.
>U
se a disciplined benchmark process
and monitor all activities. Tests that are
incorrectly implemented can provide
misleading results. A query execution
may be amazingly fast if it returns only
10% of the rows because of an improperly
loaded table.
> Insist on testing realistic data and data volumes
that are appropriate for the system size.
recommendations.
>C
onsider benchmark results in the context of
other evaluations, such as reference inquiries.
2.
be appropriate to demand guarantees, which
could require a consensus of what is tested
and how the tests are conducted
>A
list of the vendor’s customers that are
experiencing these marketing claims in
production environments
3.
Investigate at least three customer
references to verify vendor claims, asking:
>H
ow long has your organization had the
system, and is it in production?
>W
hat vendor configuration(s) do you have
in production?
>H
ow much user data do you have, and how
complex is the data model?
>W
hat is the nature of the workload as far
as number and variety of queries?
>W
hat degrees of concurrent (i.e., in-flight)
queries and load or update processes do
you run?
>H
ow many different applications or groups
of users, or business units or functional areas
Always challenge the vendor or at least
question marketing claims to get:
>D
etails surrounding the claims, such as
system size, data volume, nature of the
queries and workload, any background
workload, degree of concurrency, etc.
PAGE 2 | Teradata Magazine | December 2009 | ©2009 Teradata Corporation | AR-6036
concurrently and actively share the same
platform and data?
>H
ow reliable and available is the product?
>W
hat type of applications and solutions
directly access the single database platform?
—D.H.
why
teradata
do they beef up the meaning behind
words and phrases like “scalability,”
“simultaneous versus concurrent
users,” “user data space,” “simplicity,”
“ad hoc” and so on?
> O
n what basis are different platforms
compared? What are the details?
>W
hat are the system configurations
and product generations and releases?
>W
hat are the tests? How are they run?
How are they measured?
>W
hat are all of the results, not just the
ones that look impressive?
Ultimately, it falls to the customers
to dig into the facts before establishing a technology strategy or making an
investment that might prove unwise. It
is far too difficult to develop industrystandard measures and comparisons that
can approximate even the simplest data
warehousing workloads.
The Transaction Processing Performance Council, a consortium of vendors of which Teradata is a member, has
attempted to establish these standards, but
for the sake of repeatability and to control
costs, the tests have been simplified to the
point of irrelevance.
Uncovering the truth is not easy,
and it is far more difficult with new,
unproven products.
What I meant was …
Why is it necessary to thoroughly investigate any vendor’s technology claim?
Because the cost of an ill-chosen product
can far outweigh its purchase price or total
cost of ownership.
The old adage “too good to be true” is
often correct. Many people have purchased
golf clubs or dietary supplements or got
hooked into get-rich-quick schemes only
to be disappointed.
Likewise, based on vendors’ claims
and promises, many companies invest
in data warehousing technology only to
find they must spend significant time and
money—often years and millions of dollars—trying to make the technology work.
And after suffering through one unwise decision, many organizations are gun-shy about
investing in what would be a far wiser choice.
Instead, they continue to struggle with their
original technology, or they abandon their
vision for a successful data warehouse and
associated business solutions.
Thorough evaluation of alternative technologies can minimize or eliminate the high
risk and considerable cost of an ill-informed
decision. Even better, by asking probing
questions and applying that knowledge on
new product purchases, the likelihood of success in data warehousing is greatly increased.
So don’t just take every claim you hear
as truth. Dig deep, ask questions and
choose wisely. T
Dan Higgins has been in the IT industry
for 35 years with experience in IT systems
architecture design, technology evaluation,
and data warehouse design and implementation. He leads a Teradata sales support team
of data warehousing subject matter experts.
Why Teradata for customer satisfaction?
I
n the midst of recent unpredictability—with vendors making skewed product claims that end
up disappointing the client, and with companies starting up and quickly going under—Teradata
online
has remained steady. Since 1984, Teradata has expanded and enhanced the potential of the data
warehouse with tangible evidence. The success and satisfaction of its customers is proof that
Teradata delivers on its claims:
> Every year, Teradata customers win a variety of indepen-
> The Teradata PARTNERS
Visit Teradata.com for more
information on customer awards
and the PARTNERS User Group.
dent, industry-leading awards that exemplify how the
User Group is an
Teradata product and solutions work to help bring the
independent, non-profit corporation made up of thousands
customer success.
of Teradata customers active in data warehousing and
> Peer Advantage—the Teradata customer reference program—
its associated technologies and products. It exists to
facilitates access to more than 200 member companies
optimize the business relationships between Teradata
worldwide, providing powerful evidence that Teradata is the
and its customers to the benefit of each. Members come
global technology leader in enterprise data warehousing,
together annually to share their experiences, successes
analytic applications and data warehousing services. A Teradata
and challenges at the Teradata PARTNERS User Group
account executive can arrange a conversation between you and
Conference & Expo.
one of these customers to discuss their use of Teradata.
PAGE 3 | Teradata Magazine | December 2009 | ©2009 Teradata Corporation | AR-6036
—D.H.