A Structured Approach to
Enterprise Test Automation
Venkatesh Kovvuri
Head of Innovation & Automation Practice
Overview
»
With increase in the complexity of application software and enterprise infrastructure, the
quality guidelines & standards have also mounted. Effective testing of such apps can’t be
assured by manual testing alone. By reducing human intervention during test processes
and automating them can result in an effective and efficient methodology to meet those
standards.
»
A Structured Approach will provide opportunity to deal with complex application software
and enterprise infrastructure.
•
Extends the usage of automation to entire software testing lifecycle in addition to conventional automated test
execution (design, data, tasks, infra etc)
•
Addresses challenges around selection, prioritization and execution of automation projects in large enterprises
•
Encompasses all key components essential for executing complex automation projects
•
Provides an asset ‘toolkit’ for Test Automation - Process, Technology, Integration and Quality
•
Supports integration with industry leading open source and commercial software testing tools
•
Provides visibility into the aspects of Quality Governance on the application portfolio
© 2012 Computer Sciences Corporation - Proprietary
1
Enterprise Application Landscape
Legacy
systems
Packaged
software
•ERP
•Sales
•CRM
•HRMS
•Supply chain
•BI
•CMS
•Accounting
Maintenance
Migration
Quality Assurance &Testing
Custom
applications
• Thick/thin custom
clients
•Custom web
applications
•Custom reports
•OLTP Databases
•Data warehouses
New
development
External
interfaces
•Integration APIs
(partner-specific)
•EDI
•Web Services
(SOAP/REST)
• Handheld interfaces
© 2012 Computer Sciences Corporation - Proprietary
Integration
Functionality, Performance and Security
•Transaction
processing
(Mainframes and
AS/400s)
•Old databases
•Obsolete
technologies
2
Tool support for Enterprise Testing
Test Management Tools
Test Automation Tools
Test Design Tools
Proprietary Test Runners
Data Preparation Tools
Custom Test Harnesses
(and Drivers)
Static Analysis Tools
Dynamic Analysis Tools
Comparators
Review/Inspection Tools
Security Testing Tools
Performance Testing Tools
Lack of Unification into a
Singular Platform
© 2012 Computer Sciences Corporation - Proprietary
3
Enterprise Test Automation
Business Challenges
Challenge
Description
Candidate
Selection/Prioritization
Enterprise IT stack contains a large number (dozens to hundreds
to thousands) of applications and often pose the difficulty to
select and prioritize from automation perspective.
Tool/technology Proliferation
Automation tools operate as vendor-locked and tightly coupled
solutions with the technologies without giving the business a way
to control them in a unified and standardized fashion.
Change Management
Alignment
Change is the continuous mantra (maintenance, migration,
integration, retirement etc) in the IT stack and often times the
strategy to align automation remains out of control.
Success Criteria Definition
and Measurement
While ROI remains a known metric, continuous measurement of
the value delivered by automation requires a rich definition of the
success criteria and continuous measurement (with corrective
actions)
Future-proofing automation
assets
The percentage of assets that can be used beyond the scope of
the planned cycles remains an unknown and often times events
like tool migration etc make some of the old assets vendorlocked and void.
© 2012 Computer Sciences Corporation - Proprietary
4
Enterprise Test Automation
Technical Challenges
1.
2.
3.
4.
5.
6.
7.
Multiple testing tools for heterogeneous technology stacks
Multiple interfaces(GUI, Web, API, EDI, Mobile) with varied automation needs
Test coverage involving emerging versus legacy applications
Test scenarios involving complexly integrated partner systems
Test suite adequacy for controlled and emergency releases
Capability of automated regression test automation to find new bugs in old
functionality
Changing test priorities
1.
2.
3.
Compliance vs. Business functionality
Manual bug hunts vs. Automation efforts
Functionality vs. Performance vs. Security
8. Lack of rich test data (variety &volume) that simulates the live enterprise
9. Impact of Migration on Test Automation assets
10. Multi-environment deployments (repetitive testing)
1.
2.
Synchronization
Data population
© 2012 Computer Sciences Corporation - Proprietary
5
A Structured Approach
Executive
Executive
Quality
Dashboard
Control
Dashboard
Test
Infrastructure
Toolkit
Test
Test
Planning
Planning
Toolkit
Toolkit
Reusable
Automation
Assets
Packaged
Application
Accelerators
Test
Data
Toolkit
Internal Test
Automation Tools
Automation
Approach
Test
Design
Toolkit
Enterprise
Test
Automation
Approach
Test
Management
Toolkit
Market Test
Automation Tools
Automation Asset
Library
Automation Methods
Test
Execution
Toolkit
© 2012 Computer Sciences Corporation - Proprietary
6
Test Planning Toolkit
Select, Prioritize and Plan for test automation
ROI Calculator
Feasibility Analyzer
» Web enabled application to forecast returns
and thereby prioritize automation projects
» Ease decision making process based upon
qualitative and quantitative ROI information
» Logical, quick and efficient way to identify
automation candidates from the application
landscape
© 2012 Computer Sciences Corporation - Proprietary
7
Reusable Automation Assets
Reusable assets and Process assets to reduce time to market
» Over a period of time CSC has developed lots of reusable assets and Process assets
» CSC ACOE made those assets generic and can be used for any projects.
» Category wise reusable assets repository has set of assets that enables to start and execute an
automation project with minimum expendable effort
» Process assets has Checklists, Guidelines, Templates and process documents
Process assets
Reusable assets
Category
Number of functions
Check lists and Guidelines
300
133
250
Notification Utility
25
200
Test Management Tools Integration
26
150
Test Logging & Reporting
Ready-made Plug-In for Global Utility,
Component/Reusable
Functions,Events,Business Processes.
Exception handling & Diagnosis
Information
Folder Management
Hand Book-ATM - Test Automation
Automated Test Script Review Checklist
Test Automation Requirements Elicitation Checklist
Test Automation Framework Guidelines
Test Automation Guidelines
Test Automation Metrics Operational Definition
100
Test Automation Tool Evaluation Guidelines
Test Automation Engineering Process
66
14
50
Series1
0
Test Automation Framework Selection Guidelines
Templates
Test Automation Cost Calculator Template
5
Test Automation Execution Report Template
File Management
Web Application's Control
Technology specific
(Linux+Sieb_GlobalFuncs+Siebel_DP_
Functions)
38
Test Automation Maintenance User Guide Template
263
Test Automation Plan Template
Test Automation ROI Calculator Template
Test Automation Scope and Traceability Template
54
Test Automation Suite Release Notes
Automation Test Tool Feasibility Report Template
Total number of reusable functions
624
Test Automation Tool Evaluation Template
© 2012 Computer Sciences Corporation - Proprietary
8
Test Design Toolkit
Optimal and efficient test case generation
Test Case Optimizer
»
»
Proprietary implementation of Pair
wise/Orthogonal and other scientific
techniques that reduce the exploded test
combinations into an optimized/smaller
number
Capability to deal with large Test Input
Models
Detects up to 80% of the total defects with
optimized test cases
Test
Coverage
Configuration
Journey mode: Return trip, One way
Preferred time: Morning, Afternoon,
Evening, Late night
Leaving from:
Type1_City,Type2_City,Type3_City
Going to:
Type1_City,Type2_City,Type3_City
Booking type: Economy, First class
Customer type: Registered, New
Adults: 1, 2, 3, 4, 5, 6
Children: 0, 1, 2, 3, 4
Test Coverage
Configuration
Test Input Model
Test Input
Model
» Provides automated generation of test cases
specifically for SOA testing
» Covers both positive and negative test cases
ensuring 100% test coverage
» Reduces test case authoring effort up to 90%
Optimized
Test Cases
2-way coverage
strength
Optimized Test
Cases
»
Web Services Test Case Generator
#Total Test
Configurations for
100% coverage :
8640
#Total (reduced)
Test
Configurations
: 31
© 2012 Computer Sciences Corporation - Proprietary
9
Test Data Toolkit
Reduce cycle time by quickly generating large volumes of test data
Field Data Validator
Test Data Generator
»
Field set
Metadata
<?xml version="1.0"?>
<TestFields>
<Groups groupName = "sample"
groupNumber ="1">
<TestField fieldName ="person name" id
= "42">
• <PersonName>
• <Country>US</Country>
• <Gender>Female</Gender>
• <Format>FirstName</Format>
• </PersonName>
</TestField>
…
» Automates negative and security test data
generation for use by developers to test code
under development
» Provides both command line and GUI interface
options
CSV file with
real-looking
data
Volume
configuration
1000000
records
Real-looking
data
»
Volume
Configuration
»
Large volumes of real-looking test data is
generated.
Support for 50+ data types for US & UK (like
real names, addresses, SSNs, custom-syntax
and so on)
Support for multiple groups of fields with
dependencies like referential integrity
No restriction on volume of data generated
(speed: 1 million records per minute)
Field set
Metadata
»
person
name,PhoneNumber,SSN,Compa
nyName
TASHA,+91(0)52522-86434
ext.6072,117-93-2658,Diamond of
California
JULEE,+91(0)532-3213402
ext.5136,628-625959,MerriamWebster Inc
PENNY,+91(0)8343-878361
ext.5427,622-23-8243,Komag Inc
BRYNN,+91(0)2462-078412
ext.7113,174-80-4445,Sweetheart
Cup Company Inc
© 2012 Computer Sciences Corporation - Proprietary
10
Test Execution Toolkit
Abstract test execution and simulate third party integration
Unified Interface Automation Framework
» CSC keyword driven framework for test case development
that abstracts the process enabling test case development to
be performed by SMEs & BAs
» CSC Hybrid framework is the combination of Functional
Decomposition & Data Driver approach.
» Tool and Technology agnostic
» Integrated with QC and TEMA, Customized report
generation
Other Automation Frameworks
» GSTAR – Transaction based framework for Insurance
domain
» FLAME – Accelerated framework for Finance Domain
» Hybrid (Functional and Data Driven) Framework
» QTP ,Selenium and Silk Test
» Wiki Based framework
» Finesse + Selenium
© 2012 Computer Sciences Corporation - Proprietary
11
Test Management Toolkit
Manage test lifecycle intuitively
TEMA (Test Management)
» Integrated QA suite with an intuitive webbased interface for effectively managing and
monitoring the Quality Assurance processes
» Offers full product life cycle support and can
be used to manage multiple test cycles
» Integrated with CSC customized version of the
popular BugZilla bug tracking application
which is further enhanced for mandatory
authorization, limited privilege access and
custom reporting features
© 2012 Computer Sciences Corporation - Proprietary
12
Packaged Application Accelerators
Accelerate test automation of complex systems
» Integrates market, third party and homegrown solutions to accelerate key testing activities on packaged software
» Provides accelerators to readily automate plain vanilla installations like
Module
Manual Automation
» Key SAP processes
Test Cases Test Scripts
» Key Siebel processes
127
112
Oracle-GL
138
122
Oracle-AP
» Key PeopleSoft processes
80
74
Oracle-AR
» Key Oracle E-business processes
Oracle-FA
Oracle-CM
Oracle-Iexpense
Oracle- Purchasing
Oracle-I procurement
Oracle-OM
Oracle-Inv
Total
SAP-GL
SAP-AP
SAP-AR
SAP-AA
SAP-Co
SAP-BA
SAP-MM
SAP-SD
SAP-IS Utilities
Total
PeopleSoft-ESS
PeopleSoft-MSS
PeopleSoft- Payroll
PeopleSoft-WFA
Total
150
100
50
0
© 2012 Computer Sciences Corporation - Proprietary
67
30
13
90
27
159
124
855
42
32
17
33
22
5
92
152
78
473
21
23
17
66
127
60
29
13
90
27
155
118
800
40
30
17
33
22
5
92
152
39
430
14
20
17
55
106
13
Test Infrastructure Toolkit
Manage test infrastructure remotely and efficiently
AIM (CSC Infrastructure Management)
» Multi-tenant inventory and infrastructure management web application
» Interfaces with Test Automation tools, providing an environment for
management of un-attended test execution and results monitoring
© 2012 Computer Sciences Corporation - Proprietary
14
Executive Dashboard
Facilitates Go/No-Go decisions for production releases
A centralized dashboard for management teams to view quality & productivity data and reports:
»
»
»
»
»
Enables benchmark comparison across enterprise testing projects
Provides just in time statistics
Relieves QA team from routine capture and computation processes
Eliminates errors caused by manual processes and wipes out data manipulation possibilities
Helps take informed decisions on project performance
© 2012 Computer Sciences Corporation - Proprietary
15
Contact:
Venkatesh Kovvuri
Mobile: 09885579911
Email: vkovvuri4@csc.com
© 2012 Computer Sciences Corporation - Proprietary
16
© 2011 Computer Sciences Corporation
17