Automated Software Testing using Open Source
Testing Tools
By Elfriede Dustin
Agenda
•
•
•
•
•
•
Who is IDT
State of Software Testing
What is Automated Test and Re-test (ATRT)
Approach to ATRT
Advantage of open source
ATRT and open source – Our IDE and
solution
• ATRT Challenges
2
http://www.idtus.com
Who is IDT?
• IDT specializes in the design, development, and
implementation of Automated Software Testing and
Re-Test (ATRT) and Quality Assurance (QA)
solutions
– Deliver turn key automated test suite
• Automated test strategy and identification of highest payoff areas to
apply automation for your project
• Selection of best automation tools for your project
• Using your existing test cases or develop test cases for you
• Test results documented in requirements traceability matrix
– Training and pilot project implementation with companies
3
http://www.idtus.com
Who is IDT?
• IDT specializes in the design, development, and
implementation of Automated Software Testing and
Re-Test (ATRT) and Quality Assurance (QA)
solutions
•We are currently hiring Java Developers
– Deliver turn key automated test suite
•Plus we’d
like toand
hire
2 or 3 grad
students
• Automated
test strategy
identification
of highest
payoff to
areas to
apply
automation
your
project
work
for us for
part
time
( i.e. 20 hours week )
• Selection of best automation tools for your project
•must be US citizen
• Using your existing test cases or develop test cases for you
• Test results documented in requirements traceability matrix
– Training and pilot project implementation with companies
4
http://www.idtus.com
State of Software Testing Why ATRT?
?
5
http://www.idtus.com
State of Software Testing Why ATRT?
Size & Complexity of SW Baseline
50% or More of Overall Development Cost is
Typically Spent on Testing
Utilize Automated Testing Strategies and
Technology to Improve Productivity and Quality
Spiral / Release 4
Reduction in
Test Days
Spiral / Release 3
Test
Days
Spiral / Release 2
Increased
Test
Coverage
Test
Days
Spiral / Release 1
Test
Days
Test
Days
Manual vs. Automated
IDT Provides Automated Software Testing Solutions
http://www.idtus.com
6
What is ATRT?
• Our definition of ATRT is:
Application and implementation of software
technology throughout the entire Software
Testing Life Cycle (STL) and QA lifecycle;
with the goal to improve STL efficiencies and
effectiveness
7
http://www.idtus.com
ATRT Spans the Software Development
Life Cycle – Visual Modeling
Desig
n
Build
Assembl
e
Round-Trip Engineering
Development Tools
Use
cases

Test
Components
Modeling and test case generation
8
http://www.idtus.com
ATRT Spans the Software Development
Life Cycle – Requirements
Requirements Management and
Process Automation
Visual Modeling
Development Tools
Components
 Organizes, tracks, & controls requirements
 Requirements Traceability Matrix (RTM)
9
http://www.idtus.com
ATRT Spans the Software Development
Life Cycle – Automated Test Tools
Requirements Management and
Process Automation
Visual Modeling
Development Tools
Development
Management
Components
Execution
AutomatedSoftware
Testing Testing
Tools
Automated
http://www.idtus.com
 Automates test cases using vendor-provided,
open-source tools or in-house development
10
ATRT Spans the Software Development
Life Cycle – Integrated Suite of Tools
Additionally:
 Middleware
 Infrastructure
 Defect Tracking
 Configuration
Management
 Memory Leak
Detectors
 Performance
Testing Tools
 Documentation
Tools
 others
Requirements Management and
Process Automation
Visual Modeling
Infrastructur
e
Development
Tools
Components
Middleware
Automated Testing Tools
Software Configuration Management
11
http://www.idtus.com
Defect Tracking
Types of Software Test
Life-cycle Support Tools
• Which tools are you familiar with?
12
http://www.idtus.com
Types of Automated Test
Tools
Life-Cycle
Phase
Business
Analysis
Phase
Type of Tool
Tool Description
Business Modeling
Records definitions of user needs and automates rapid
construction of
flexible, graphical, client-server applications
Configuration
Management
Defect Tracking
Technical Review
Management
Requirements
Definition
Phase
Manages system life-cycle defects
Facilitates communication, while automating the
technical review/inspection process
Documentation
Generators
Automate document generation
Requirements
Management
Manages and organizes requirements; allows for test
procedure design and test progress reporting
Requirements
Verifiers
Verify syntax, semantics, and testability
Use Case
Generators
http://www.idtus.com
Baselines important data repositories
Create use cases
13
Types of Tools (Cont’d)
Life-Cycle
Phase
Analysis and
Design Phase
Type of Tool
Database Design
Structure Charts,
Flowcharts, and
Sequence Diagrams
Test Procedure
Generators
Programming
Phase
Metrics Tools
Develops second generation enterprise client-server systems
Manage processes
Generate test procedures from requirements, design, or data and object
models
Syntax Checkers/
Debuggers
Perform syntax checking and have debugging capability; usually available
with built-in programming language compiler
Memory Leak and
Runtime Error
Detection
Detects runtime errors and memory leaks
Source Code Testing
Verifies maintainability, portability, complexity, and standards compliance
Static and Dynamic
Analyzers
Depict quality and structure of code
Code (Test) Coverage
Analyzers or Code
Instrumentors
Usability Measurements
http://www.idtus.com
Tool Description
Identify untested code and support dynamic testing
Provide usability testing as conducted in usability labs
14
Types of Tools (Cont’d)
Life-Cycle
Phase
Other Testing
Life-Cycle
Support Tools
http://www.idtus.com
Type of Tool
Tool Description
Data Extraction Tool
Extract Data from various formats into various formats
Test Data Generators
Generate test data
File Compare Utilities
Find discrepancies between files that should be identical in content
Simulation
Simulates application to measure for scalability, among other tasks
Test Management
Tests management
Network Testing
Monitors, measures, tests, and diagnoses performance across the
entire network
GUI Testing
(Capture/Playback)
Conducts automated GUI tests; capture/playback tools record user
interactions with online systems so they may be replayed
automatically
Load/Performance Testing
Conducts load/performance and stress testing
Security Testing
Performs security testing and vulnerability scanning at the
application or network level; plus debuggers will allow to check for
security coding errors (source code checkers)
15
Deployment Approach –
Adapted ATLM
Populate Test Manager
with Test Cases and
Requirements
Develop Test Scripts
from Test Procedures
Test Results:
• Pass/Fail by Test Case
• Pass/Fail Summary
• Test Time Required
Identify or Develop
& Pilot
• Test Manager
• Test Scripts
• Data Comparators
• Performance Test Tools
• GUI Record/Playback
• Storing/Retrieving Results
Collect/Create:
• Requirements
• Test Cases
• Test Procedures
• Expected Results
• Interface Specifications
• System/Component
Configuration Description
16
http://www.idtus.com
Why Open Source
?
17
http://www.idtus.com
Why Open Source?
• Advantage of Open Source:
– no licensing issues; i.e. license cost; maintenance;
etc.
– easily modifiable
– adaptable
– lightweight
– flexibility - not tied to one vendor
18
http://www.idtus.com
ATRTs IDE - Eclipse
Eclipse – this site lists all the plugins
http://www.eclipseplugins.info/eclipse/plugins.jsp?category=SCM&pager.offset=0&firstItem=1
Eclipse/TPTP – Open Source - Developed by IBM
–
Eclipse/Test and Performance Tools Platform (TPTP)
offers a common extensible framework for the following
functions:
•
•
•
•
http://www.idtus.com
Requirements Management http://sourceforge.net/projects/osrmt/
Configuration Management - Subversion
Testing Tools (http://www.nabble.com/Eclipse-TPTP---TestingTools-f2262.html)
Profiling: default Java applications, but can be extended to other
apps
19
ATRTs IDE - Eclipse
Eclipse/Test and Performance Tools Platform (TPTP)
offers a common extensible framework for the
following functions (cont):
•
•
•
•
•
http://www.idtus.com
Static Analysis: default for C++ and Java, but can be
extended to other apps
Application monitoring and log analysis
Build tool – ant
Xml input and output - using an open source Java
package http://jakarta.apache.org/ecs/ , which is a Java
library that has an API for constructing and outputting
XML.
Test Manager – STAF/STAX
20
Current Test Tool Implementation –
Example and Challenges
Vendor-provided (Capture/Playback) Tool:
Automated test tools mimic actions of the test engineer.
During testing, the engineer uses the keyboard and mouse to perform
some type of test or action.
Testing tool captures all keystrokes and subsequent results, which are
baselined in an automated test script.
During test playback, scripts compare latest outputs with previous
baseline.
Testing tools have built-in, reusable test functions.
Most test tools provide for non-intrusive testing; i.e., they interact with
the “application-under-test” as if the test tool was not involved.
21
http://www.idtus.com
Current Testing Tools – Challenges
(cont)
In-House Software Development required:
Capture/Playback tools generate hard-coded values; test
scripts are not reusable, nor do they implement software
development best practices right out of the box; scripts need
to be modified.
Capture/Playback tools don’t necessarily provide all testing
features required; code enhancements are often required to
meet testing needs.
Capture/Playback tools are not necessarily compatible with
system engineering environment, and software testing scripts
need to be developed in-house.
Developed testing scripts are not cross-vendor compatible,
even if the same scripting language is used
22
http://www.idtus.com
Linux Capture/Playback Tools
IDT Findings
Replay Xcessory
http://www.scl.com/products/ics/motif/testing/replay/
www.ics.com
QF-Test
http://www.qfs.de/en/qftestJUI/index.html
Froglogic
http://www.froglogic.com/pg?id=Google&category=squishjava
Redstone Software
Eggplant
KD Executor
http://www.kdab.net/?page=products&sub=kdexecutor
Jameleon
http://sourceforge.net/projects/jameleon/
ShUnit2
for shell scripts testing
GNU Linux Desktop Project
http://ldtp.freedesktop.org/wiki/Home
Cantana
http://www.ipl.com/pdf/p0003.uk.pdf
crontab
Part of Unix
crontab, cruisecontrol, www.adminschoice.com/docs/crontab.htm
Dogtail
http://people.redhat.com/zcerza/dogtail/about.html
http://www.cyberciti.biz/tips/linux-automated-gui-testing-with-python-baseddogtail-tool.html
Expect
http://expect.nist.gov/
23
http://www.idtus.com
Automated Test and Re-Test
(ATRT)
• ATRT includes a common framework comprised of
re-usable software services
– Open Source Software Test Automation Framework
(STAF)
• Provides automated test procedures, test inputs, and
test results for SW components
– Suitable for use by 3rd parties not associated with
development of the component
24
http://www.idtus.com
What is STAX?
STAX is an automation system
With STAX you can automate and monitor your entire test environment
ƒ
ƒ
ƒ
ƒ
System/product setup
Testcase distribution
Testcase execution
Testcase results collection and analysis
STAX consists of
ƒ A programming language designed for automation
–Programs written in this language are called jobs
ƒ An execution engine (similar to an interpreter for other programming/scripting
languages) which manages the execution and runtime behavior of the job
ƒ A GUI application, called the STAX Monitor, which provides a dynamically
updated view of your executing jobs
ƒ A tool (STAXDoc) that is used to generate documentation for your STAX jobs
25
http://www.idtus.com
STAF can run on:
* Windows 95
* Windows 98 (and 98 SE)
* Windows Millenium Edition
* Windows NT Version 4.0
* Windows 2000
* Windows XP
* Windows Server 2003 (IA32, IA64, AMD64)
* Windows Vista (IA32, AMD64)
* Linux (Intel32/64, AMD64, PPC32/64)
* Linux on zSeries (31-bit, 64-bit)
* AIX Version 4.3.3.0 or higher (32-bit, 64-bit)
* Solaris (Sparc) 2.6 and higher
* Solaris AMD Opteron 64-bit
* HP-UX 11.00 and higher (PA-RISC, IA64 32-bit and 64-bit)
* OS/400 V5R2 or higher
* z/OS UNIX V1.4 and higher
26
http://www.idtus.com
STAF Internal Services
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
DIAG Provides diagnostics services Internal ("DIAG")
DELAY Provides a means to sleep a specified amount of time Internal ("DELAY")
ECHO Echos back a supplied message Internal ("ECHO")
FILE SYSTEM Allows you to get and copy files across the network Internal ("FS")
HANDLE Provides information about existing STAF handles Internal ("HANDLE")
HELP Provides Help on STAF error codes Internal ("HELP")
MISC Handles miscellaneous commands such as displaying the version of STAF
that is currently running Internal ("MISC")
PING Provides a simple is-alive message Internal ("PING")
PROCESS Allows you to start, stop, and query processes Internal ("PROCESS")
QUEUE Provides a network-enabled IPC mechanism for STAF Programs Internal ("QUEUE")
SEMAPHORE Provides network-enabled named event and mutex semaphores Internal ("SEM")
SERVICE Allows you to list services available on a machine and to examine the
Requests that have been submitted on a machineInternal ("SERVICE")
SHUTDOWN Provides a means to shutdown STAF and register for shutdown
Notifications Internal ("SHUTDOWN")
TRACE Provides tracing information for STAF services Internal ("TRACE")
TRUST Interfaces with STAF's security Internal ("TRUST")
VARIABLEProvides a method for maintaining configuration and runtime data
(variables) Internal ("VAR")
http://www.idtus.com
27
STAF External Services
The executable code for external STAF services resides outside of STAFProc, for
example in a Java jar file, a C++ DLL file, or a Rexx script file.
•
•
•
•
•
•
•
•
•
•
CRON Calls into STAF services at a specified time interval External (Java)
EMAIL Allows you to send email messages External (Java)
EVENT Provides a publish/subscribe notification system External (Java)
EVENTMANAGER Allows you to call STAF services when a specified Event
occurs External (Java)
HTTP Allows you to make HTTP requests which can be grouped together in a
session External (Java)
LOG Provides a full-featured logging facility External (C++)
MONITOR Allows a testcase to publish its current running execution status for
others to read External (C++)
RESOURCE POOL Allows you to manage exclusive access to pools of elements,
e.g. VM UserIDs or Software Licenses External (C++)
STAX Provides an XML-based execution engine External (Java)
ZIP Provides a means to zip/unzip/list/delete PKZip/WinZip compatible archives
External (C++)
28
http://www.idtus.com
Technologies used in STAX
STAF
ƒ STAF provides the infrastructure on which STAX builds. The full power of STAF
and its services is exposed for use within STAX jobs.
XML
ƒ The STAX programming language is based on XML. This provides built-in
structure to your jobs, as well as providing a set of existing tools for use in
constructing your jobs, such as XML (aware) editors and XSLT.
Python
ƒ The STAX programming language builds on Python to provide a rich and
accessible data model
ƒ STAX's Python integration also allows you to access the wealth of existing
Python libraries
Java
ƒ The STAX programming language allows you to access existing Java
classes/libraries, providing another source for reuse
http://www.idtus.com
29
Automation Tasks
Automation Startup
System Setup
Testcase Execution
Execution
Monitoring
Synchronization
Testcase Output Analysis
Resource Management
Testcase Cleanup
Results Notification
Automation Completion
30
http://www.idtus.com
End-to-End Automation with STAF and
STAX
Automation Startup
System Setup
Testcase Execution
Testcase Output Analysis
Testcase Cleanup
Results Notification
Automation Completion
Event, EventManager, Cron
FS, Process
Process, Monitor, Log, Variable,
Queue, ResPool
Log, Process
Process
Email, HTTP
S
T
A
X
J
o
b
<function>
<parallel>
<sequence>
<process>
<stafcmd>
<testcase>
<timer>
<block>
<loop>
<message>
<log>
<import>
<job>
<iterate>
<paralleliterate>
<hold>
<release>
<terminate>
31
http://www.idtus.com
Key ATRT Architecture
Requirements
• Support applications running on multiple computers
• Support applications developed in different languages
• Support applications running on different types of OS’s
• Support applications which have GUI and those which do not (for
example Interface testing)
• Support applications which use different types of network
protocols such as TCP/IP, DDS, etc
• Support integration of multiple commercial testing tools from
different vendors ( allowing as new or better products emerge
in the market they can be utilized )
• Support testing w/o having to install ATRT on the same
computers as the application under test and be able to be
32
distributed across computers
http://www.idtus.com
Prototype – Proof of Concept
Display Application
Analysis & Display
Application
Display Application
System Under Test
Windows
Based
Simulation of Application
Processing Messages
Eggplant
R/P
VNC Robot
R/P
Simulation of Application
Processing Messages
ATRT
Test Manager
RTM
Bugzilla
http://www.idtus.com
STAF/STAX
Linux Based
33
ATRT Prototyping Effort
• Able to successfully execute ATRT for 48 hours without operator
intervention
• During the tests:
 VNCRobot - Approximately 25,000 operator key strokes were
replayed
 STAF/STAX - More than 1,000,000 messages processed and
verified by ATRT
34
http://www.idtus.com
Additional Requirement
• Pristine test environment required – automated
testing tool cannot be installed on ApplicationUnder-Test PC
 Eggplant – allows for remote capture/playback via
VNCserver and KVM switch
 VNCRobot
35
http://www.idtus.com
Automated Software
Testing
• Software testing paradox
• What is it?
36
http://www.idtus.com
Effective Automated
Software Testing
Current State:
Developing software in order to test software
Automate the test support development: automated IDL
to code generation; test data generation (using FireEye
from NIST); compare utilities; log file analyzers;
Our Automated Software Testing Goal:
• Minimize Human interaction
– Possibly get on par with Hardware Automated Testing?
37
http://www.idtus.com
Hardware Testing: current
state
• Many automated test generators for
manufacturing/ logic devices, circuit testing,
etc., available i.e. SEI/ATG, uMaster, etc.
• Automated testing without human interaction
is the norm for hardware testing
38
http://www.idtus.com
Hardware vs. Software
AT
How is hardware testing different?
• Software Challenge:
–
–
–
–
Changing requirements
GUI interfaces
Code access/availability
Defects uncovered earlier in the software
development lifecycle are cheaper to fix
• Solution flexibility required – problem
dependent
39
http://www.idtus.com
Proposed solutions
Focus on Reusable Components
– Test Data / Expected Results
– Interfaces
– Middleware
Assuming code and models are available:
• Model based testing; automated test case generators based on models
• Eclipse – effective ATRT IDE
• Probes – see Aprobe – Code instrumentation
• Building self-testable components
• Using standard language MOF to Text to generate automate test cases
GUI Testing tool ideas
New tools - other
• Working with NIST to produce additional tools
40
http://www.idtus.com
GUI Testing
• Jemmy - java library to test UI
• http://jemmy.netbeans.org/
• http://abbot.sourceforge.net/doc/overview.shtml
• Perl modules
• run via VNCServer?
41
http://www.idtus.com
Questions
Email edustin@idtus.com with any follow-up questions
42
http://www.idtus.com
IDT
We are hiring…….
Please come see me now if interested or send
email edustin@idtus.com with any follow-up
43
http://www.idtus.com