Planning Test Automation
(Wisely)
Introductions
Avner Uzan
Test Automation Analyst
Thomas Messerschmidt
Test Automation Specialist
Now @ Paramount Pictures
Planning Test Automation
2
Disclaimer
The views expressed in this presentation are those
of the presenters & do not necessarily reflect the
views of their current or past employers; their
subsidiaries or affiliated companies!
Planning Test Automation
3
Planning Automation
Wisely




Premise
Set the hook
Basic Rules
Methodology:





Keyword-Driven Spreadsheet-Test Cases
Framework Construction
Scope:


(Does Test automation Find Bugs?)
Smoke, Functional, System, Performance
Prioritizing Test Construction
Best Kept Secrets (Trackers & Analysis Masters)
Test Automation Best Practices
Planning Test Automation
4
Premise:
Create maintainable automated test suites
Create a testing framework that will use
easy-to-read spreadsheets to enumerate
the actual test cases
 Create & maintain generic & application
specific functions & subroutines to support
the automation

Planning Test Automation
5
Does Test Automation Find Bugs?
Automation Vs Manual
160
140
120
100
80
60
40
20
0
Test Automation VS. Manual
100%
80%
134
60%
Bugs
Testers
76
Resources
20%
1
0%
Manual
Defects
40%
16
Automation
134
16
76
1
Resources
Defects
Basic Rules







Plan your automation process thoroughly
Establish & follow naming conventions
Establish & persist a directory structure *
Build automation for platform
independence
Solicit development’s cooperation to
develop for testability
Expand your knowledge with other SMEs
MEASURE OR DIE
Planning Test Automation
7
Methodology

Smoke Testing – Keyword-driven Spreadsheets

Functional Testing – Keyword-driven Spreadsheets

System Testing – Functional Decomposition

Performance Testing – Weighted Functionality
Planning Test Automation
8
Zambelich Method
Planning Test Automation
9
Spreadsheet-Test Cases
Planning Test Automation
10
Generic Functions
Planning Test Automation
11
App Specific Functions
Planning Test Automation
12
Sample Keyword Action
Planning Test Automation
13
Parameter & Data File
Planning Test Automation
14
Smoke Test
Objectives


Primarily validate builds
Open and close as many GUIs as possible
 Some windows are acct data dependant
 Manually test GUIs that could not be automated
Smoke Test Execution Metrics
 Covers 115/120 L1 GUIs - 105/110 L2 GUIs
 Execution time: 1:25 / L1, 1:10 L2
 This was run 200+ times on the WR version
 This has runs 200+ times on the QTP version
 Currently demonstrates 100% GUI Pass rate
Planning Test Automation
15
GUI Functional
GUI Functional Objectives
 Validates the functionality of Each GUI






Verify screen functionality as per spec, NOT business
processes
Validates objects exist on UI per spec
Verifies object operations, calculations & navigations
Verifies data formats ($-Amts, SSN, Phone #, etc.)
Verifies field/object data comparisons across screens
Verifies all error processing (invalid values, pop-ups, etc.)
GUI Functional Scope

Targeted two languages
Planning Test Automation
16
System Test
Use Functional Decomposition
System Test Objectives



Focus on core functionality – align w/ project team &/or customer
Regression test regular release & special projects
Future plans include converting our system test to BPT & auto.
Scope
Execution Metrics


200 scripts of which about 175 are active - take about 4-5 days
50 full suite runs, 10,000 tests run over 5 year period
Planning Test Automation
17
Performance Test
Planning Test Automation
18
Prioritize Test Construction

Target low hanging fruit to get early ROI
 Prioritize

& build common functions first
Create tests for most frequented GUIs
Planning Test Automation
19
Optimizing Automation





Optimize automation for maximum coverage
Increase number of executing machines
The downside
Brake automation into “byte-size” pieces
If one function can do the work of two
Planning Test Automation
20
Other Considerations







Number of test platform / environments
Availability of test platform (hours/day)
Comparability of test environment to prod.
Environment refresh
Test data availability / harvesting (method of)
Test data accuracy
Product or application stability
Planning Test Automation
21
Trackers (Measure or Die)
Planning Test Automation
22
Analysis Masters

Demo Analysis Masters
Planning Test Automation
23
Performance Analysis Master
Planning Test Automation
24
Automation Best Practices









Setup directory structure & use it consistently
Define & establish filing conventions
Publish changes to subroutines & functions.
Keep test assets under version control
Choose automation tools carefully based on compatibility to your
development platform, resource knowledge, budget.
Include resource, time & process allowances for test asset
maintenance.
Keep an inventory list of functions & routines easily accessible.
If possible cross-reference use of functions & subroutines in scripts to
more predicatively understand the impact changes to these assets
might have.
Prioritize tests to be automated carefully to avoid marginalizing ROI
Planning Test Automation
25
Automation Best Practices2





Get management buy-in & commitment early
Set realistic expectations – automation is not just the push of a
button
Identify a set of relevant metrics & measure regularly & consistently
Include regular progress reviews to ensure that the automation team
is in fact focused on the business or customer needs
Share the wealth – the more people use it the richer you’ll be!
Additional References:
TOTALLY DATA-DRIVEN AUTOMATED TESTING – Keith Zambelich
http://automation.org.uk/downloads/documentation/white_papers-totally_data_driven_automated_testing.doc
Keys to Functional Test Automation Success, by Thomas Murphy / The Gartner Group - Thanks to Microfocus
an HP Partner for the link
http://www.gartner.com/technology/media-products/reprints/microfocus/vol4/article4/article4.html
LoadRunner Test Analysis Master – A. Uzan http://www.youtube.com/watch?v=m860IurgHM8
Planning Test Automation
26
Automation Demo

Demo Automation
(time permitting)
Planning Test Automation
27
Contact us:
Avner Uzan: qaanalysis@ca.rr.com
Thomas Messerschmidt: qa4quality@gmail.com
Planning Test Automation
28
Parting Thought On Automation

Why Automate?
Automation, fundamentally, is the mechanization of a manual task. Love, education &
human interaction should not be automated.
However, there is no evil in taking a manual process & allowing a machine to do that
which would take a person longer, work a person harder or take the pleasure out of
doing a task over & over. However, if by automating something, the existence of life,
the values of morality, the sensibilities of one’s existence or the livelihood of those
who would otherwise perform the task manually is usurped, than automation has
defeated it’s very purpose.
Automation should be carefully conceived so that future generations are not
burdened with undoing the damage of unwitting or well intended but misguided
endeavors.
Planning Test Automation
29