8 Common Myths About Performance Testing in Production
8 Common Myths About
Performance Testing in
Production
Don’t Let These Misconceptions Keep
You From Production Testing
White Paper
Table of Contents
Myth #1: Production testing = live testing ................................................................................... 1
Myth #2: It’s too risky ..................................................................................................................... 2
Myth #3: Production testing is all about breaking the site .......................................................... 2
Myth #4: It takes too long – we’re lucky to have time for QA testing ......................................... 3
0\WK3URGXFWLRQWHVWLQJLVWRRGLIðFXOW – we don’t have the expertise ................................ 3
Myth #6: Production testing isn’t necessary if we do thorough testing in the lab ..................... 3
Myth #7: I’ll never get management buy-in ................................................................................. 4
Myth #8: We don’t have the budget – it costs too much ............................................................ 4
Summary: ........................................................................................................................................ 4
About Soasta, Inc.: ........................................................................................................................ 5
SOASTA, Inc.
TESTING IN PRODUCTION MYTHS WHITE PAPER
In today’s web environment, performance matters – a lot. And one of the best ways to ensure that your application will perform as expected, under production loads, is to do production performance testing. This entails testing the application with production-level load in the actual environment in which it operates.
ACCORDING TO GOOGLE,
AN EXTRA 500MS OF
LATENCY COSTS THEM
20% OF THEIR SEARCH
TRAFFIC.
Testing your application in the production environment uncovers performance issues at scale that simply don’t exist in more limited lab tests. It calls on all of the components of your application ecosystem, including outside factors such as third-party services or content delivery networks (CDN).
In addition, performance testing in the production environment uncovers problems outside the application itself but within your application infrastructure, such as:
5L[^VYR and bandwidth limits
-PYL^HSSJVUÄN\YH[PVUZ
3VHKIHSHUJLYJVUÄN\YH[PVU
+5:YV\[PUN
4VIPSLUL[^VYRWLYMVYTHUJL
If you don’t do performance testing in the production environment, then you’re de facto outsourcing that testing to your production users – which is probably not a wise decision.
Despite its clear advantages, a surprising number of companies fail to performance test their applications in the production environment today. In part, this is due to misconceptions or myths about production testing that keep companies from even exploring the possibilities.
This paper aims to dispel those myths and give you a better idea of what is possible with today’s
WLYMVYTHUJL[LZ[PUNJHWHIPSP[PLZ>LOVWL`V\»SSNL[[OLJVUÄKLUJLHUKIHJRNYV\UK`V\ULLK[V
KL]LSVW`V\YV^UWYVK\J[PVU[LZ[PUNZ[YH[LNPLZ-PYZ[SL[»ZHKKYLZZ[OLT`[OZOLHKVU .
Myth #1: Production testing = live testing
;OLÄYZ[HUKTVZ[JVTTVUTPZJVUJLW[PVUPZ[OH[WYVK\J[PVU[LZ[PUNHS^H`ZYPZRZHɈLJ[PUNSP]L users. Certainly, some companies regularly do production performance testing on their applications while real users are present, but there are many other ways to test on the production infrastructure, without risk to live data or actual users.
Test during maintenance windows: If you have the luxury of scheduled downtime, test during
this window. You need to work with the responsible teams to carefully plan and ramp the test
[VÄ[^P[OPU[OL^PUKV^I\[`V\^PSSH[SLHZ[IL[LZ[PUN[OLLU[PYLWYVK\J[PVUPUMYHZ[Y\J[\YL
Test before release: Test an application in the production environment before it’s announced
HUKVɉJPHSS`YLSLHZLK(ZWHY[VM[OLYLSLHZLTHUHNLTLU[WSHUTHRLWYVK\J[PVU[LZ[PUN
here a requirement. Involve all core team members across development, test and operations
in these test events and soon the value of production testing will be clear to your entire
organization.
SOASTA, Inc.
TESTING IN PRODUCTION MYTHS WHITE PAPER | 1
Test on a slice of the infrastructure or on your redundant system: 7HY[P[PVUVɈH¸Z[HNPUN¹
section of the production infrastructure that contains a representation of all of the components
VM[OLM\SSZ`Z[LT;OLU[LZ[[VHMYHJ[PVUVMM\SSZJHSLYLWYLZLU[LKI`[OL¸ZSPJL¹>OPSLUV[H
M\SSZJHSL[LZ[[OPZ^PSSZ[PSSLUHISL`V\[VHKQ\Z[JVUÄN\YH[PVUZHUKYVV[V\[PZZ\LZ[OH[^VU»[
appear in a lab. Alternatively – or additionally – if you have a fully redundant system, you can
YLYV\[LWYVK\J[PVU[YHɉJ[V[OH[Z`Z[LT[V[LZ[HUK[\UL[OLWYVK\J[PVUZ`Z[LT;OPZ
technique is used by Amazon and Google and serves the dual purpose of also testing your
MHPSV]LYVYKPZHZ[LYYLJV]LY`Z`Z[LTZ(UKPM`V\JHUUV[ÄUK[OLZLUVUSP]L^H`Z[V[LZ[PU
production, keep reading.
Myth #2: It’s too risky
Production testing simply sounds risky. But if you think about it, it is much less risky to run a
JVU[YVSSLK[LZ[[OHU[VSL[\ZLYZYL]LU\LHUKYLW\[H[PVUZ\ɈLY^OLUHZ`Z[LTMHPSZ\UKLYJVUKP[PVUZ that are easily replicated and preventable.
The risks of performance testing in production can be mitigated by:
*HYLM\SS`JOVVZPUN`V\Y[LZ[PUNVIQLJ[P]LZ
(KVW[PUN[LJOUPX\LZHUKHWWYVHJOLZMVYUVUSP]LWYVK\J[PVU[LZ[PUNZLL[OLWVPU[HIV]L
4HRPUNZ\YL`V\OH]LYLHS[PTLPUZPNO[PU[V^OH[»ZOHWWLUPUNH[HSS[PTLZK\YPUN[OL[LZ[
/H]PUNM\SSJVU[YVSK\YPUN[LZ[ZZV`V\JHUYLZWVUKX\PJRS`PMWLYMVYTHUJLKLNYHKLZ
A real-time dashboard gives you insight to application infrastructure performance.
Myth #3: Production testing is all about breaking the site
When you think of performance testing, do you think of testers trying to break the site? While stress and failure testing are clearly performance testing strategies, other types of performance testing are more common and less destructive:
Baseline testing: Establish the level of load that the application can handle while meeting
`V\YJYP[LYPHVMZ\JJLZZZ\JOHZYLZWVUZL[PTL^P[OPUHKLÄULK[OYLZOVSK
Spike: Simulate high, fast variance from normal load staying well under expected full
capacity.
Endurance: ;YHJRHWWSPJH[PVUZ[HIPSP[`V]LY[PTLH[]HY`PUNSL]LSZVM\ZLY[YHɉJ
Diagnostic testing: ;YV\ISLZOVV[HZWLJPÄJPZZ\LVYZLL[OLYLZ\S[ZVMZWLJPÄJJOHUNLZ
to infrastructure.
SOASTA, Inc.
TESTING IN PRODUCTION MYTHS WHITE PAPER | 2
SOASTA, Inc.
Within production testing, you can measure things like:
,_WLYPLUJL!/V^MHZ[PZHWHNLYLZWVUKPUNMYVTKPɈLYLU[SVJHSLZHUKSVHK&
Transaction: How long does it take to complete a transaction under varying circumstances?
Code-level performance: How fast are individual pieces of UI or Server code?
Is there contention?
Mobile performance impact: Does having many transactions over slower Edge or 4G
networks result in more open connections than your application can handle?
Site failure would be an unexpected outcome from any of the above tests. Performance testing is not all about breaking the site.
Myth #4: It takes too long – we’re lucky to have time for QA testing
This excuse might be valid if you’re only planning to do production infrastructure testing before the application is live. But, as we saw when discussing Myth #1, it’s possible to extend the test window and do performance testing when the application is in production, using maintenance windows, slices or silos of the infrastructure, or even low-usage times.
-\Y[OLYTVYL[OLWYVJLZZVMJYLH[PUNHUKL_LJ\[PUNHWLYMVYTHUJL[LZ[H[ZJHSLPU[OLWYVK\J[PVU environment, does not need to be time-consuming and complex. New cloud-based testing tools let you ramp up a global community of virtual users in a matter of minutes. And creating the actual tests can be as simple as running some commands and/or interacting with a mobile device and capturing the generated HTTP to create a test script. In fact, running smaller iterative tests with incremental objectives can allow more testing in the same or less time.
4`[O!7YVK\J[PVU[LZ[PUNPZ[VVKPɉJ\S[¶^LKVU»[OH]L[OLL_WLY[PZL
Once this may have been true. But the technology available for performance testing has changed.
New cloud-based testing applications provide you full control and visibility that make it simple to create the tests you need, spin up thousands of virtual users, and launch and closely monitor real-time results during tests as they run.
But that’s just the technology end. Any kind of performance testing requires planning, and production performance testing requires more than others. You should involve several key teams in your organization to plan and monitor the tests.
One strategy is to start with those kinds of test that are easier to execute while still delivering a lot
VMNVVKPUMVYTH[PVU3VVRMVY[LZ[Z[OH[[V\JOHIYVHKHTV\U[VMPUMYHZ[Y\J[\YL[VZ[HY[HUKKLSP]LY
[OVZLÄYZ[LHZ`^PUZ
-VYL_HTWSL[LZ[PUNYLHKVUS`ZJLUHYPVZKVLZUV[YLX\PYLHSV[VMIHJRLUK^VYR`L[JHUYL]LHS quite a bit about your site performance under load. Testing transactional scenarios requires more work to develop the test data and back out or otherwise negate testing transactions. As a result, transaction testing might wait until you have some success with other types of performance testing.
Once your entire organization begins seeing the results from production testing, it’s easier to gain buy-in for testing further, and often leads to modifying your applications and operations to make production performance testing easier and a core operational activity.
Myth #6: Production testing isn’t necessary if we do thorough testing in the lab
As we’ve already mentioned, testing in a lab environment is great for testing the application itself,
I\[KVLZU»[[LZ[[OLIYVHKLYPUMYHZ[Y\J[\YL-YLX\LU[S`ZLYPV\ZHUK\UL_WLJ[LKWYVISLTZHYPZL
V\[ZPKLVM[OLHWWSPJH[PVUP[ZLSM-VYL_HTWSLTVYL[OHUVUJL^L»]LZLLUHWWSPJH[PVUZIYV\NO[
KV^U\UKLYOLH]`SVHKI`TPZJVUÄN\YLKSVHKIHSHUJLYZ
Testing in production does not replace lab testing or testing in a staged environment but it can uncover problems that other tests cannot, including:
-PYL^HSSJHWHJP[`PZZ\LZ
Unbalanced web servers
3VHKIHSHUJLYJVUÄN\YH[PVU
*VUÅPJ[Z^P[OV[OLYHWWSPJH[PVUZ
DNS routing problems
Network bandwidth issues
3H[LUJ`IL[^LLUZ`Z[LTZ
*+5ÄSLWSHJLTLU[
Mobile network performance issues
TESTING IN PRODUCTION MYTHS WHITE PAPER | 3
AMAZON LOSES 1% OF
ITS SALES FROM EVERY
100MS OF LATENCY.
Worried about those third-party application providers if you do a performance test? Choose testing approaches that enable proper load distribution, good monitoring and tight control over test execution to do this, and then check in with them ahead of time. They may well be happy to participate in the performance test and have ways to handle the testing on their end. Because ultimately, no one wants to be surprised by unexpected glitches in production.
Myth #7: I’ll never get management buy-in
There are two key strategies for getting management buy-in for production performance testing:
1. Wait until there’s a major problem, then suggest production testing.
7YLZLU[HJSLHY[LZ[PUNZ[YH[LN`[OH[I\PSKZJVUÄKLUJL]PHZTHSSP[LYH[P]L[LZ[Z^P[OYPZR
assessment and mitigation strategies, and get all of the relevant teams on board.
:[YH[LN`PZKPɉJ\S[[V[PTLHUKVM[LUWHPUM\SMVY[OLI\ZPULZZ0UMHJ[P[»Z^OH[ you’re trying to avoid. So, let’s get right to #2.
>OLUWYLZLU[LK^P[OJYLKPISLL_HTWSLZH[LZ[PUNZ[YH[LN`HUKH\UPÄLK[LHTTVZ[THUHNLTLU[ teams will embrace the value of performance testing in production. Production performance testing is in fact a wise risk mitigation strategy for the business as a whole, and should be presented as such.
Myth #8: We don’t have the budget – it costs too much
With traditional testing techniques, this is true. Building a test lab to support the application under
[LZ[^P[OHSS[OLJVTWSL_P[`NSVIHSKPZ[YPI\[PVUHUKYKWHY[`ZLY]PJLZZPTWS`PZU»[ÄUHUJPHSS`MLHZPISL
To generate user loads common to most consumer sites with open source or legacy test tools requires hundreds or thousands of load servers –
again, an immense cost. Add to that the cost of test software licenses ample to drive a test of even moderate scale, plus the people to build and analyze the tests, and you quickly reach millions of dollars to test for just thousands of users.
Again, existing tools that recognize the need to test the latest web and mobile technologies to realistic usage levels are reshaping the game when it comes to production testing. Using cloud platforms like Amazon EC2, IBM Smart Cloud, Rackspace Cloud, or Windows Azure, you can pay for only the capacity you need to run test platforms around the globe and at scale. To test mobile
HJJLZZ`V\JHUKLWSV`H]PY[\HSWYP]H[LKL]PJLJSV\KHUK[LZ[UL[^VYR[OH[YLÅLJ[Z[OLZPNUPÄJHU[ and changing diversity of the mobile device market.
But of course there are other costs as well, in addition to the tools. If you need
ZVTLOLSW^P[OJVZ[Q\Z[PÄJH[PVUOLYLHYLHML^WVPU[Z!
7LYMVYTHUJL[LZ[PUNPUWYVK\J[PVUJHUOLSW`V\NL[[OLILZ[WLYMVYTHUJLV\[VM[OL
infrastructure you have – and prevent you from investing in the wrong places.
If problems arise outside of testing, your team will spend more time trouble- shooting and
Ä_PUN[OLTVUHJYPZPZIHZPZ[OHU[OL`^V\SK^P[OWSHUULK[LZ[PUN
Performance testing helps you hold third-party applications accountable for their promised
service levels.
Minor changes in performance can have a major impact on your bottom line.
Summary
;OPZWHWLYZOV\SKNP]L`V\[OLJVUÄKLUJL[V[HRLHUL^SVVRH[`V\YWLYMVYTHUJL[LZ[PUNHUK hopefully the ammunition to move forward with production testing plans.
You’ll be joining the rapidly growing number of companies that are running performance tests in a production environment. At SOASTA, we work with thousands of customers in the e-commerce,
LSLHYUPUNTLKPHHUKLU[LY[HPUTLU[ÄUHUJPHSZLY]PJLZOPNO[LJOUVSVN`HUKZVJPHSTLKPHPUK\Z[YPLZ to implement performance test plans including production testing. With cloud testing services,
JVTWHUPLZVMHSS[`WLZHUKZPaLZJHUUV^[Y\S`[LZ[PUWYVK\J[PVUH[ZJHSLPUHJVZ[HɈVYKHISLHUK low-risk manner.
SOASTA, Inc.
TESTING IN PRODUCTION MYTHS WHITE PAPER | 4
About SOASTA, Inc.
SOASTA is the leader in cloud testing. Its web and mobile app test automation solution, the
CloudTest® Platform, enables developers, QA professionals and IT operations teams to test with unprecedented speed, scale and precision. The innovative product set streamlines test creation, automates provisioning and execution and distills analytics to deliver actionable intelligence faster.
>P[O:6(:;(JVTWHUPLZJHUOH]LJVUÄKLUJL[OH[[OLPYHWWZ^PSSWLYMVYTHZKLZPNULKL]LUPU
WLHR[YHɉJ:6(:;(»ZJ\Z[VTLYZPUJS\KLTHU`VM[VKH`»ZTVZ[Z\JJLZZM\SIYHUKZPUJS\KPUN
(TLYPJHU.PYS*OLNN.PS[.YV\WL/HSSTHYR0U[\P[4PJYVZVM[HUK5L[ÅP_:6(:;(PZWYP]H[LS` held and headquartered in Mountain View, Calfornia.
-VYTVYLPUMVYTH[PVUHIV\[:6(:;(WSLHZL]PZP[ www.soasta.com
Connect with us:
Contact us:
866-344-8766
444 Castro Street
Mountain View, CA 94041
To learn more visit: soasta.com
or email us at info@soasta.com
©2012 SOASTA. All rights reserved. SOASTA, the SOASTA logo, and SOASTA CloudTest are trademarks of SOASTA. All other product or company names may be trademarks and/or registered trademarks of their respective owners.
EST-1003-1-0512
TESTING IN PRODUCTION MYTHS WHITE PAPER | 5
