Verification and Validation

Assuring that a software system
meets a user's needs
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 1
Verification vs validation


Verification:
"Are we building the product right"
Validation:
"Are we building the right product"
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 4
The V & V process


Is a whole life-cycle process - V & V must be
applied at each stage in the software process.
Has three principal objectives
•
•
•
Ensuring that the requirements are correct
The discovery of defects in a system
The assessment of whether or not the system is usable in
an operational situation.
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 5
Static and dynamic verification





Software inspections (static verification and
validation)
Software testing (dynamic verification)
STATIC VALIDATION Should be done during requirements
specification, and probably to some extent during high level
design.
STATIC VERIFICATION should be done during design and
programming.
DYNAMIC VERIFICATION should be done on prototypes and
on programs
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 6
Program testing






Can reveal the presence of errors, NOT their
absence
A successful test is a test which discovers one
or more errors
It is important to go out of your way to create hard test
cases, not to be mean – but to catch errors!
Is the only validation technique for non-functional
requirements
Should be used in conjunction with static
verification and validation to provide full V&V coverage
Defect Testing is covered in detail in Chapter 20
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 8
V& V goals



Verification and validation should establish
confidence that the software is fit for its
purpose
This does NOT mean completely free of defects
Must be good enough for its intended use
•
•
•
Software function
User expectation
Marketing environment
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 10
The debugging process
Test
results
Locate
error
©Ian Sommerville 2000
Test
cases
Specification
Design
error repair
Repair
error
Software Engineering, 6th edition. Chapter 19
Re-test
program
Slide 13
19.1 V & V planning




Careful planning is required to get the most out of
testing and inspection processes
Planning should start early in the development
process
The plan should identify the balance between
static verification and testing
Planning should define standards for the testing
process
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 14
The structure of a software test plan







The testing process
Requirements traceability
Tested items
Testing schedule
Test recording procedures
Hardware and software requirements
Constraints
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 16
19.2 Software inspections




Involve people examining the source
representation with the aim of discovering
anomalies and defects
Do not require execution of a system so may be
used before implementation
May be applied to any representation of the
system (requirements, design, test data, etc.)
Very effective technique for discovering errors
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 17
Inspection success


Many different defects may be discovered in a
single inspection. In testing, one defect ,may
mask another so several executions are required
They reuse domain and programming knowledge
- reviewers are likely to have seen the types of
error that commonly arise and look out for them
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 18
Inspections and testing




Inspections and testing are complementary and not
opposing verification techniques
Inspections can check conformance with a
specification but not conformance with the customer’s
real requirements
Inspections cannot test integration of subsystems
Inspections cannot check non-functional
characteristics such as performance, usability, etc.
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 19
Program inspections



Group meeting to carefully, line-by-line review code
Intended explicitly for defect DETECTION (not
correction)
Defects may be logical errors, or non-compliance
with standards
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 20
Inspection teams






Made up of at least 4 members
Author of the code being inspected
Inspector who finds errors, omissions and
inconsistencies
Reader who reads the code to the team
Moderator who chairs the meeting and notes
discovered errors
Other roles are Scribe and Chief moderator
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 21
Inspection pre-conditions






A precise specification must be available
Team members must be familiar with the
organisation standards
Syntactically correct code must be available
An error checklist should be prepared
Management must accept that inspection will
increase costs early in the software process
Management must not use inspections for staff
appraisal
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 22
The inspection process
Planning
Overview
Follow-up
Individual
preparation
©Ian Sommerville 2000
Rework
Inspection
meeting
Software Engineering, 6th edition. Chapter 19
Slide 23
Inspection checklists




Checklist of common errors should be used to
drive the inspection
Error checklist is programming language
dependent
The 'weaker' the type checking, the larger the
checklist ***
Examples: Initialization, Constant naming, loop
termination, array bounds, etc.
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 25
Fault clas s
Data faults
Ins pection check
Are all program variab les in itialis ed befo re th eir values
are u sed?
Hav e all con stants been n amed ?
Sh ould the lower bou nd o f arrays be 0, 1, or something
else?
Sh ould the upp er b ound of array s b eequal to the size of
the array or Size -1?
If character s tring s are us ed , is a delimiter explicitly
ass igned?
Con trol faults
Fo r each co nditional s tatement, is the con ditio n correct?
Is each loo p certain to termin ate?
Are compou nd s tatements correctly bracketed ?
In case statements, are all p oss ible cas es accounted fo r?
Inp ut/outp ut faults
Are all inpu t variab les us ed ?
Are all outpu t variab les ass igned a value before they are
ou tput?
Interface faults
Do all fu nction and procedu re calls hav e th e correct
nu mb er o f parameters?
Do fo rmal and actual parameter typ es match?
Are th e p arameters in the righ t order?
If compon ents acces s s hared memory, do th ey hav e the
s ame mo del of th e s hared memory s tructure?
Sto rage man agement If a linked s tructure is mo dified,
have all links been
faults
correctly reass igned?
If d ynamic s torag e is u sed, h as sp ace b een allocated
correctly?
Is sp ace explicitly de-allocated after it
is no lon ger
required?
Exception
Hav e all pos sib le error con ditio ns been taken
in to
man agement faults
account?
Inspection checks
Inspection rate





500 statements/hour during overview
125 source statement/hour during individual
preparation
90-125 statements/hour can be inspected
Inspection is therefore an expensive process
Inspecting 500 lines costs about 40 person-hours
effort
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 27
19.3 Automated static analysis



Static analysers are software tools for source text
processing
They parse the program text and try to discover
potentially erroneous conditions and bring these
to the attention of the V & V team *
Very effective as an aid to inspections. A
supplement to but not a replacement for
inspections *
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 28
Static analysis checks
Fault clas s
Data faults
Static analys is check
Variables us ed before initialisation
Variables declared but never used
Variables ass igned twice but never us ed
between ass ignments
Poss ible array bound violations
Undeclared variables
Control faults
Unreachable code
Unconditional branches into loops
Input/output faults
Variables output twice with no intervening
ass ignment
Interface faults
Parameter type mismatches
Parameter number mismatches
Non-usage of the results of functions
Uncalled functions and procedures
Storage
management Unass igned pointers
faults
Pointer arithmetic
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 29
Use of static analysis


Particularly valuable when a language such as C
is used which has weak typing and hence many
errors are undetected by the compiler ***
Less cost-effective for languages like Java that
have strong type checking and can therefore
detect many errors during compilation
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 33
19.4 Cleanroom software development


The name is derived from the 'Cleanroom'
process in semiconductor fabrication. The
philosophy is defect avoidance rather than
defect removal
Software development process based on:
• Formal specification.
• Incremental development
• Structured programming
• Static verification using correctness arguments
• Statistical testing to determine program reliability.
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 34
Cleanroom process evaluation




Results in IBM have been very impressive with
few discovered faults in delivered systems
Independent assessment shows that the
process is no more expensive than other
approaches
Fewer errors than in a 'traditional' development
process
Not clear how this approach can be transferred
to an environment with less skilled or less
highly motivated engineers
©Ian Sommerville 2000
Software Engineering, 6th edition. Chapter 19
Slide 40