Building an automation framework
around open source technologies
Agenda
•
•
•
•
•
•
•
•
Challenges
Choosing an automation framework
Generic Architecture
How does it work?
Features of an ideal test automation framework
Additions to Selenium framework
Other components of test automation framework
Beyond test automation
12/11/2010
Datamatics Confidential for Limited Circulation Only
2
Challenges today
ROI
Technology
Automation
Tools
12/11/2010
Datamatics Confidential for Limited Circulation Only
3
ROI Facts
ROI
Technology
Automation
Frequency of usage
Tools
Coverage of Tests
Reusability in the system
Creation efforts
Maintenance efforts
12/11/2010
Datamatics Confidential for Limited Circulation Only
4
Technology Facts
ROI
Technology
Automation
• Over 73% of applications developed today are web enabled
Tools
applications
• Prevalence of Firefox, IE, Safari and Chrome as the major
browsers, with multiple versions
• Incompatibilities between these browsers primarily due to
javascript implementation
• Testing needs for Web Applications is multiplied due to
permutations of these challenges
12/11/2010
Datamatics Confidential for Limited Circulation Only
5
Test Automation Facts
ROI
Technology
•
New requirements are introduced on a daily basis
Automation
Tools
•
Testing window is getting smaller day by day
•
Test automation can never replace human testers
•
Not all test can / should be automated
•
Test automation is a supplement to help / boost overall testing
•
Use test automation for area that are least prone to changes
and yet require frequent tests
12/11/2010
Datamatics Confidential for Limited Circulation Only
6
Test Automation Tool Facts
ROI
Technology
•
Only people that claim 100% automation
•
Learning curve is very higher with every variant
•
Product support for emerging technologies
•
Extendibility
12/11/2010
Datamatics Confidential for Limited Circulation Only
Automation
Tools
7
Myth#1: Test automation is a goal never reached
• Automation is a project on top of our goal list
• Many testers intimidated with test automation
• Developers don’t want to learn test automation
• GUI automation is hard so lets not do automation
• Automate everything and fire the testers
• For some - panacea and others - waste of time
Automation testing is an investment – will take time and effort to bring to fruition
12/11/2010
Datamatics Confidential for Limited Circulation Only
8
Myth#2: Record / Playback
• Main feature recommended by any test automation
tool vendors
– Capture user actions and later on play them back for
regression testing.
• Simple steps to follow
–
–
–
–
–
Design test case in test automation tool
Use capture feature to record user actions
Enhance script with verification points
Add delay and wait state points for synchronization
Playback and observe results in log file
12/11/2010
Datamatics Confidential for Limited Circulation Only
9
Record / Playback Myth
• Learning’s
–
–
–
–
–
–
Scripts resulting in hard-coded values
Costs associated with maintenance is astronomical
Scripts are often not reliable (pop-up, messages)
If tester makes a mistake – re-record the script
If application changes – re-record the script
Scripts cannot be recorded until software works
Nothing is ever tested by this approach
12/11/2010
Datamatics Confidential for Limited Circulation Only
10
Test Automation using Open Source Framework
Automation
Approach
Technology
Layer
Automation
Tool
12/11/2010
Datamatics Confidential for Limited Circulation Only
11
Choosing a test automation approach
• Test script modularity
• Test library architecture
• Keyword or table driven testing
• Data driven testing
• Hybrid-Keyword & Data driven both.
12/11/2010
Datamatics Confidential for Limited Circulation Only
12
Choosing a technology layer
•
•
•
•
•
•
•
•
Platform Independence
Execute unattended runs
Distributed test execution
Extensible and Customizable
Ease of debugging and logging
Status reporting
Repeat tests as needed (CI)
Version control
12/11/2010
Datamatics Confidential for Limited Circulation Only
13
Generic Architecture
Data Manager
Test Manager
Automation Tool
Version Control
Test Artifacts
Application Under Tests (AUT)
12/11/2010
Datamatics Confidential for Limited Circulation Only
Integration (CI)
14
Choosing a Test Automation Tools
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Abbot Java GUI
Test Framework
Anteater
Apodora
Arbiter
Autonet
AutoTestFlash
Avignon
Blerby Test Runner
Canoo WebTest
Capedit
Concordion
Crosscheck
csvdiff
CubicTest
Cucumber
DBFeeder
DbFit
12/11/2010
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Eclipse TPTP
•
EMOS Framework •
Enterprise Web Test •
Expect
•
expect-lite
•
Feed4TestNG
•
Frankenstein
GITAK
•
Harness
•
httest
•
IdMUnit
•
Imprimatur
•
ItIN - Infopath
•
testing in .Net
•
ITP
•
ItsNat, Natural
AJAX
•
WebDriver
•
WebInject
WebTst
ivalidator
•
Jacobie
•
Jameleon
•
jDiffChaser
•
Jemmy
•
JFunc: JUnit
Functional Testing•
Extension
•
JSystem
•
Latka
•
Linux Test Project •
LogiTest
•
LReport
•
MActor
•
Marathon
•
Maveryx
MaxQ
•
miTester for SIP •
•
Datamatics Confidential for Limited Circulation Only
SIP Testing
Tool
Mockito
MozUnit
OLVER
Om
org.tigris.mbt
PAMIE
Pounder
pywinauto
QAliber
QAT
QMTest
Rasta
Robot
Framework
Sahi
Samie
Sauce Labs
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Selenium
SharpRobo
soapui
STAF
Solex
SWAT
SWTBot
Systin
T-Plan
Robot
tclwebtest
TestPlan
TextTest
Tomato
Toster
Watij
WatiN
Watir
WET
15
Test automation tool
12/11/2010
Datamatics Confidential for Limited Circulation Only
16
How does it work?
12/11/2010
Datamatics Confidential for Limited Circulation Only
17
Why Selenium?
•
•
•
•
•
•
•
•
Support for Windows, Linux and Mac OS
Supports browsers – Explorer, Firefox, Safire
Supports AJAX calls
Non-developers can develop test cases
Graphical Selenium IDE
Export test cases as JUnit test cases
When test fails, Selenium can send an email w/ screenshot
Selenium Grid enabled test execution on multiple machines
in parallel reducing overall execution time
12/11/2010
Datamatics Confidential for Limited Circulation Only
18
Features of an ideal test automation framework
Scripting
• Script Recording
• Parameterization
• Control
Statements
• Database
Integration
• Modal Dialog
Support
• Keyword Support
12/11/2010
Execution
• Across Multiple
OS and Browsers
• Across Multiple
machines in
parallel
• Ability to recover
testing from
failures
Analysis
• Detailed
Reporting for
failures
• Snapshots on
failures
• Multilayer
Reporting
Datamatics Confidential for Limited Circulation Only
19
Features Missing within the selenium suite
Scripting
• Script Recording
• Parameterization
• Control
Statements
• Database
Integration
• Modal Dialog
Support
• Keyword Support
12/11/2010
Execution
• Across Multiple
OS and Browsers
• Across Multiple
machines in
parallel
• Ability to recover
testing from
failures
Analysis
• Detailed
Reporting for
failures
• Snapshots on
failures
• Multilayer
Reporting
Datamatics Confidential for Limited Circulation Only
20
Additions to Selenium framework
Keywords
Datasheets
Business Components
Reusable Framework
Utility / Generic
Functions
Reporting Functions
Recovery Functions
SELENIUM
Application Under Test (AUT)
12/11/2010
Datamatics Confidential for Limited Circulation Only
21
Features of an ideal test automation framework
Features
Best found in
Scripting Phase
Script recording
Selenium IDE
Parameterization
Datasheets
Control Statements
Implementation specific
Database Integration
Utility / Generic Functions
Modal Dialog Support
Utility / Generic Functions
Keyword Support
Keywords
12/11/2010
Datamatics Confidential for Limited Circulation Only
22
Features of an ideal test automation framework
Features
Best found in
Execution Phase
Across Multiple OS and Browsers
Selenium RC
Across Multiple machines in Parallel
Selenium Grid
Ability to recover testing from failures
Recovery Function
Analysis Phase
Detailed Reporting for Failures
Selenium
Snapshots on failures
Selenium
Multilayer Reporting
Reporting Function
12/11/2010
Datamatics Confidential for Limited Circulation Only
23
Generic Architecture
Data Manager
Test Manager
Automation Tool
Version Control
Test Artifacts
Application Under Tests (AUT)
12/11/2010
Datamatics Confidential for Limited Circulation Only
Integration (CI)
24
Continuous Integration
•
•
•
•
•
•
•
•
Apache Continuum
Apache Gump
Beebox
Bitten
Buildbot
Cabie
Cerberus
Control Tier
12/11/2010
•
•
•
•
•
•
•
•
Cruise Control
cDash
Draco
Hudson
Jhbuild
LuntBuild
Sin
Tinderbox
Datamatics Confidential for Limited Circulation Only
25
Version Control
•
•
•
•
•
•
•
•
•
Aegis
ArX
GNU arch
Bazaar
Bonsai
Cervisia
Codeville
Cogito
CVSNT
12/11/2010
•
•
•
•
•
•
•
•
•
Darcs
Fossil
Git
Subversion
Libre Source
Mercurial
Monotone
Repo
CVS
•
•
•
•
•
•
•
•
OpenCVS
OpenGrok
Revision Control System
Apache Subversion
SVN
Tortoise
Vesta
DCVS
Datamatics Confidential for Limited Circulation Only
26
Looking beyond automation testing
• Test reports are automatically prepared
• Defects are automatically captured in defect tracking system
• Test metrics are updated automatically
• Traceability matrix is updated automatically
• Select test cases automatically based defect status updates
• Calculating code and test coverage automatically
• Suggesting improvements in test coverage
12/11/2010
Datamatics Confidential for Limited Circulation Only
27
Thank You!!!