Tools for Automating Verification and
Validation
Validation: checking: Are we building the right system?
Verification: checking: Are we building the system right?
•
•
•
Model dependent
size dependent
platform dependent
Purpose: improve the S/W development process
•
•
•
•
•
•
•
- avoid delays, reworks
less error-prone
more standard
better defined
more predictable
faster (time-to-market)
better managed process
assist in better understanding product (helping to manage
complexity of product)
Automating V & V in the S/W
Development Process
Must be at SEI CMM Level 3 or higher.
(- process is defined)
[pre-condition for use of tools]
e.g.
Reqts.
High Level
Design
BUT only automate
where necessary
Detailed
Design
Release &
Support
Codes Inspections
& Unit Test
Regression
Performance
...
Build
& System
Test
Level 4 CMM
- metrics to help in
process management
 taking the process view for V & V tool assessment,
1) Where in the process does the tool apply?
2) What is the entry criteria to the use of the tool(set)?
3) What is the exit criteria (for stopping the use of the
tool?)
Ed Kit’s Tool Classifications
Tools for reviews and inspections
- assist in reviews, walkthroughs and inspections of
requirements, functional design, internal design, and code.
- Complexity analysis
- “20% of code causes 80%of the problems” (Pare to
principle) [not good predicates of bad code, easy to collect $]
- Code comprehension
- help to understand unfamiliar code (take an unbiased look)
- Syntax and semantic analysis
- extensive error checking to find errors a compiler would
miss
- flag potential defects
Tools for Test Planning
- define the scope, approach, resources and schedule of testing
activities
- tools required are:
- templates for test plan documentation
- test schedule and staffing estimates
- complexity analyzer
# of
issues
R8 R3 R1 R2 R4 R10 R5 R6 R7 R9
Kit’s Tool Classifications (2)
Tools for Test Design and Development
• detailing the overall test approach specified in the test plan for
software features or combinations of features, and identifying
and prioritizing the associated test cases.
• Not much tool support
• types of tools required:
–
–
–
–
test data generator
requirements-based test design tool
capture/playback
coverage analysis
Test Execution and Evaluation Tools
• process of executing test cases and evaluating the results
– includes selecting test cases for execution, setting up the
environment, running the selected tests, recording the execution
activities, analyzing potential product failures, and measuring the
effectiveness of the effort.
•
Types of tools required:
–
–
–
–
–
capture/playback
coverage analysis
memory testing
simulators and performance
regression testing - necessary
Kit’s Tool Classifications (3)
Software Testing Support Tools
• tools that lend overall support to the overall test process
• when these tools are of poor quality or non-existent the
professional tester suffers
• types of tools required:
– Problem management
• defect tracking tools, etc., used to record, track, and
assist with the management of defects and
enhancements throughout the lifecycle
• Configuration management
– manages, controls and coordinates changes to
documents and other artifacts
Questions before tool acquisition
•
•
•
•
How do the tools fit into and support our test process?
Do we know how to plan and design tests?
Who will be responsible for making sure we get the proper
training on our new tool?
Who will promote and support tool use within the organization
on an ongoing basis?
Beizer’s Test Tool Classifications
• Fundamental Tools
–
–
–
–
coverage certifiers
flowgraph generators and analyzers (e.g. MSC generator)
metrics support
instrumentation support (probe insertion & monitoring)
• Test Execution Automation
– capture/replay
•
•
•
•
•
test editor
smart comparator
local and distributed
single and multi-thread
exception reports
– stubs and drivers
•
•
•
•
•
•
•
test case initialization
input simulation
outcome comparison
path instrumentation and verification
reporting
passage to next test
debugging support
– test environments
• support for testing in a CASE environment
Beizer’s Tool Classifications (2)
• Test Design Automation
–
–
–
–
–
–
structural generators
data-flow test generators
functional generators
FSM (EFXM) based generators
random test data generators
Test Result Analysis Automation
Load and Performance Testing
Current Listings
Benchmark Factory
Chariot
CYRANO Test
ITF
Load Runner Product Family
Pegasus
PLATINUM Final Exam Internet Load Test
pre Vue-C/S
pre Vue-X
QALoad
SQA LoadTest
Teleprocessing Network Simulator
TestMaster
WebLoad
Silk Performer
Non-GUI Drivers and Test Suite Managers
Current Listings
Acqua
QES/EZ
AcquaNavio
SNA Development Test Facility
AdaTEST
SQA Manager
ANVL
SQA Suite
ARTT
STAR
BenchWorks
STEPMaster
Cantata
STW/Regression
CobRetest
STW/Regression/MSW
CTB
TALC2000
DejaGnu
TBGEN
ESIM Testing Environment
TCS
for Embedded Software
TestDirector
Internetworking Test Facility TestExpert
MYNAH
TestMate
OTF - An Object Testing Framework TestPlan
Pilot
TMS
Pre Vue
TMX
QA Director
TETware
QES/Architect
VPRO-G
Test Evaluation Tools
Current Listings
Branch validator
C-Cover
CodeTEST
CoverTest
CTC++
DeepCover for Java
DeepCover for C/C++
GCT
Hindsight/TCA
Hindsight/TPA
Insure++
LDRA Testbed
Logiscope
McCabe Visual Testing Toolset
ObjectCoverage
Panorama-C/C++
pf77
PureCoverage
safec
SofInst
STW/Coverage
TCMON
TCAT for Java
TestCenter
VisionSoft/TEST
GUI Test Drivers
Current Listings
ATF (Automated Test Facility)
B-Tree Validor Family
CAPBAK/X, CAPBAK/MSW
CYRANO TEST
Evaluator
Ferret
QARun
Panorama-2
PLATINUM’s Final Exam C/S-Test
PLATINUM Final Exam Internet Test
QA Partner
QC/Replay
QES/EZ for GUI
ReplayXt
RSW Web Analyst
SET (Solution Evaluation Tool)
Smalltalk Test Mentor
Vermont High Test Plus
SQA Robot
Visual Test 4.0
STW/Web
WinRunner
Test Edge
Win Vue
Test Now
WITT for OS/2, WITT/Windows, X/WITT
TestRunner
xrc - X Remote Control
TestWeb
XRunner
Marick’s Corner
Site Hosted by ST Labs
http://www.stlabs.com/marick/faqs/t-design.htm
CALIBER: www.tbi.com