Test Automation Success:
Choosing the Right People & Process
Kiran Pyneni, Automation Manager
Aetna, Inc.
Agenda
Company Background
Automation Team
Automation Objectives
Our Journey…
Strategy Team
Delivery Team
Benefits of Teams Strategy
Automation Lifecycle
Benefits of Process Integration
Metrics Of Our Success
Automation Coverage, Efforts Automated, Utilization & Cost Avoidance
Questions
Aetna Background
•
Aetna is one of the nation's leading diversified health care benefits
companies, serving members with information and resources to help them
make better informed decisions about their health care.
•
Products and management Programs:
–
–
–
–
Aetna offers a broad range of insurance and employee benefits products.
The first national, full-service health insurer to offer a consumer-directed health
plan, Aetna continues to lead the way with its Aetna HealthFund2 line of products,
including HSA, HRA and RRA options.
Aetna offers a wide array of programs and services that help control rising
employee benefits costs while striving to improve the quality of health care, such as
case management; disease and patient safety programs; integrated medical,
dental, pharmaceutical, behavioral health and disability information.
Aetna provides members with access to convenient tools and easy-to-understand
information that can help them make better-informed decisions about their health
and financial wellbeing.
3
Automation Team
Improves efficiencies and quality at our organization by automating
consistent and repeatable manual tasks using industry standard
technologies / tools / frameworks.
Standards &
Best Practices
Reports & Metrics
Automation
Team
Automation
Development &
Maintenance
Assessments &
Designs
Automation Objectives
Automation
Objectives
Automating Right Tasks
Utilization of existing
automation
Sustaining and Growing
Automation Suite
Our Journey…
Decentralized Approach
•
Benefits
–
–
•
Centralized Approach
•
–
Easy identification of
automation opportunities
Higher collaboration between
application & automation teams
–
–
Challenges
–
–
–
Automation development has
become expensive
Usage of different tools &
solutions
Standards & Best practices are
not shared across applications
Benefits
•
Standardization of automation
tool set at enterprise level
Sharing of coding best practices,
templates and standards
Cost of automation development
is lowered
Challenges
–
–
–
Slower automation coverage
progression
Lower automated scripts
utilization
Funding Challenges
Our Journey Cont’d. –
Challenges & Solutions
Challenges
•
Identification of automation opportunities
•
Prioritization of automation projects
based on ROI
•
Cost of automation
•
Implementation of optimized designs
Challenges
•
Early completion of maintenance for
automation scripts
•
Funding for automation projects
•
Collaboration between automation &
business teams
Solution
•
Structuring automation as strategy &
delivery team
Solution
•
Create automation life cycle process and
integrate with company's overall software
development life cycle
Automation Team Structure
Automation Team
Automation
Strategy Team
Automation
Delivery Team
Automation Strategy Team
•
Definition:
– To provide automation strategic direction and identify
potential new automation opportunities
•
Characteristics:
–
–
–
–
–
•
Project management experience
Various automation frameworks/tools knowledge
Ability to manage multiple projects in parallel
Very strong written and verbal communication skills
Very strong leadership and analytical skills
Activities:
–
–
–
–
–
–
–
Process & Best Practices
Assessment & Strategies
Design Solutions & Techniques
Estimations & Planning
ROI Cost/Benefit Analysis
Reports & Metrics
Tool Technical Support & Analysis
Automation Delivery Team
•
Definition:
– To develop, deliver, and maintain automation solutions
•
Characteristics:
– Understanding of programming principles and best
practices
– Knowledge of various automation tools
– Knowledge of multiple programming languages
– Strong communication skills
•
Activities:
–
–
–
–
–
Scripts Development
Scripts Maintenance
Coding Standards
Scripts Planning & Execution
Day to Day interactions w/End User
Benefits of Teams Strategy
Benefits
•
•
•
•
•
•
•
Easy identification of automation opportunities
Prioritization of automation projects based on ROI
Improved automation design
Optimized the cost of automation
Clearly defined roles & responsibilities
Better communication process
Optimized automation project planning
Phases
Automation Lifecycle
Automation Evaluation
Execution & Acceptance
• Create Business Process
Test (BPT) components
• Understand testing
processes
• Understand the
requirements with the help
application demos.
• Parameterize test data in
BPT components
• Unit testing of
automation
• Understand target
environments
• Create automation
framework / design
• Review components for
quality and correctness
• Deliver automation to
application team
• Understand test data
details
• Identify timelines and
milestones
• String the BPT
components together to
create a test script
• Execute automation by
application team
• Verify automation tool
compatibility
• Training required for
automation tools
• Develop automation scripts
for BPT components
• Identify Automation
opportunities
• Tool installation if required
• Identify test data to
conduct unit & acceptance
testing
• Access & Environment
Setup
G2
G1
Deliverables
Automation Generation
• Identify the resources
needed.
• Understand functionality of
the project / application
Activities
Automation Preparation
• Preliminary Estimations
• Automation Test Strategy
• Automation Project Plan
• Business Case - ROI
• Framework documentation –
Re-usable components,
Portability, test data grid
design, etc.
• Selected automation tool
• Detailed Estimations
• Go, No Go
• Business Process Test
(BPT) components
• Automation Scripts
• Test Data
• Validation of ROI
• Retrospection
• Sign off
G3
• Tested automation
scripts
• Automated test
execution results
• Lessons Learned
• Sign Off
Team Collaboration
Automation
Strategy Team
Automation
Delivery Team
Automation
Automation
Automation
Execution
Evaluation
Preparation
Generation
& Acceptance
Automation ROI Calculator
Automation ROI Calculator
Integration of Automation Lifecycle &
Software Development Lifecycle
Automation
Evaluation
Feasibility/Analysis
Automation
Preparation
Design
Automation
Generation
Implementation
Execution &
Acceptance
Testing &
Maintenance
Benefits of Process Integration
Benefits
• Improved understanding on automation process across
the entire organization
• Better collaboration between business and automation
teams
• Funding challenges are removed
• Improved on-time delivery of automation scripts
• Helped to promote the automation across entire
organization
• Formal escalation process developed
• Initialized the creation of automation reports & metrics
Automation Coverage –
Regression Testing
Year-2008
Q1-2009
Q2-2009
Q3-2009
Q4-2009
35%
42%
45%
47%
52%
Manual Testing Efforts Automated
13,803 hrs
11,750 hrs
10,200 hrs
8,570 hrs
7,047 hrs
Year-2008
Q1-2009
Q2-2009
Q3-2009
Q4-2009
Year
As of 2008
As of 2009
2009 only efforts
Total Efforts Automated
7047 hrs
13803 hrs
6756 hrs
Utilization & Cost Avoidance
Utilization
150%
134%
130%
Q2-2009
Q3-2009
Q4-2009
87%
45%
Year-2008
Q1-2009
Cost Avoidance
10,810 hrs
9,618 hrs
9,867 hrs
7,200 hrs
Q1-2009
Q2-2009
Q3-2009
Q4-2009
Savings Vs. Cost Avoidance
• Cost Savings
–
–
–
–
Efforts difference between manual and automation testing
Manual (Not automation) execution frequency
Produced by automation strategy team after assessment
Automation ROI Calculator
• Cost Avoidance
–
–
–
–
Test coverage difference between manual & automation testing
Efforts difference between manual and automation testing
Automation execution frequency
Generated from Quality Center after release implementation
Helped to improve the credibility of automation metrics significantly
Automation Metrics – Best Practices
Automation is a way of Testing and process followed for manual testing
can be applied
• Best Practices followed in Test Management Tool – Quality Center
– Define automation work flow
• Regression Testing identifier
• Automation Assessment status
• Automation Development status
– Store automation inventory
– Manage automated test scripts execution
Questions