ECE 453 – CS 447 – SE 465
Software Testing &
Quality Assurance
Instructor
Paulo alencar
1
Overview
 Software Quality
 Software Quality and Quality Assurance
 Non-Execution Based Testing
 Reviews and inspections
2
The State of Software Development
2000 Defense Science Board Study:
• 53% of projects were late and over budget, 16%
were on time, 31% were canceled before
completion.
• There is tremendous growth in software content in
both manned and unmanned systems.
• Software requirements now amount to the bulk of
the overall specification requirements (65% for
the B-2, 80% for the F-22)
3
Software Quality
• The American Heritage Dictionary defines quality
as
– “a characteristic or attribute of something.”
• For software, some kinds of quality may be
encountered:
– Quality of design encompasses requirements,
specifications, and the design of the system.
– Quality of conformance is an issue focused
primarily on implementation.
– user satisfaction: compliant product + delivery
within budget and schedule, etc.
4
Software Quality
• Quality, simplistically, means that a product should
meet its specification.
• This is problematical for software systems
– There is a tension between customer quality
requirements (efficiency, reliability, etc.) and developer
quality requirements (maintainability, reusability, etc.);
– Some quality requirements are difficult to specify in an
unambiguous way;
– Software specifications are usually incomplete and
often inconsistent.
5
Cost of Quality
• Prevention costs include
– quality planning
– formal technical reviews
– test equipment
– Training
• Internal failure costs include
– rework
– repair
– failure mode analysis
• External failure costs are
– complaint resolution
– product return and replacement
– help line support
– warranty work
6
Software Quality Assurance
Process
Definition &
Standards
Formal
Technical
Reviews
Analysis
&
Reporting
Measurement
Test
Planning
& Review
7
Software Quality Team Size
=>4%
SAMPLE OF 135 ORGANISAT IONS (1983)
=<4%
Software Quality St aff / Development Staff
=< 3%
=< 1%
8
Role of a Software Quality Team
Review
Applications
Provide
T echnical Advice
Review
and Build a
Quality Environment
Develop Standards
and Guidelines
Analyse Development Errors
9
Tasks of a Software Quality Team
ROLE
CHALLENGE
TASKS
Review
Applications
When to abort a project
Executive management ignorance
User ignorance
Audit requirements
Evaluate systems in all phases
Provide management with technical assessment
Ascertain user requirements are met
Ascertain audit requirements are met
Provide
Technical
Advice
Changing technology
Use of consultants
Ability to keep current technically
Complexity of systems
Know current technology
Act as internal consultant
Act as technical consultant
Know many systems
Review and
Build a
Quality
Environment
How to evaluate software products
Build a quality environment
Evaluate software products
Counsel management
Develop
Standards
and
Guidelines
Few systems and programming standards
Professionalism
Help set standards
Evaluate quality of work
Analyse
Development
Errors
Know type of problems
Know cost of problems
Know magnitude of problems
Quantify problems
Identify problems
Determine cost of problems
10
Role of the SQA Group
• Prepares an SQA plan for a project.
– The plan identifies
• evaluations to be performed
• audits and reviews to be performed
• standards that are applicable to the project
• procedures for error reporting and tracking
• documents to be produced by the SQA group
• amount of feedback provided to the software project team
• Participates in the development of the project’s software process
description.
– The SQA group reviews the process description for compliance with
organizational policy, internal software standards, externally imposed
standards (e.g., ISO-9001), and other parts of the software project plan.
11
Role of the SQA Group
• Reviews software engineering activities to verify compliance
with the defined software process.
– identifies, documents, and tracks deviations from the process and
verifies that corrections have been made.
• Audits designated software work products to verify
compliance with those defined as part of the software process.
– reviews selected work products; identifies, documents, and tracks
deviations; verifies that corrections have been made
– periodically reports the results of its work to the project manager.
• Ensures that deviations in software work and work products
are documented and handled according to a documented
procedure.
• Records any noncompliance and reports to senior
management.
– Noncompliance items are tracked until they are resolved.
12
Analyzing Defects
Product
& Process
Collect information on all defects
Find the causes of the defects
Move to provide fixes for the process
measurement
... an understanding of how
to improve quality ...
13
Analyzing Defects
Remember, high quality products
 Must meet all user requirements
 Must be free of defects
 In some processes, the main measure of software
product quality is total defect density
 Total defect density
 is measured as defects/KLOC (number of defects
removed in development per thousand LOC)
14
Example Quality Measurement
Defect Removal Efficiency
DRE = E /(E + D)
E is the number of errors found before delivery of
the software to the user
D is the number of defects found after delivery
15
Why SQA Activities Pay Off?
cost to find
and fix a defect
100
60.00-100.00
log
scale
10
1
10.00
0.75
Req.
1.00
1.50
Design
3.00
test
system
code
test
field
use
16
Software Quality Team
The major role of the Software Quality Team is to review
applications.
Reviewing Applications
–
–
Includes verification and validation
It also includes the following:
1.
2.
3.
4.
5.
Software reviews and inspections
Traceability of software deliverables
Testing
Auditing of selected key software items
Standards and Guidelines
17
1. Software Reviews and Inspections
A software review is an evaluation of a software
element to ascertain discrepancies from planned
results and to recommend improvements
– e.g. Design Review, Code Review
– Different types, e.g.:
• Walkthrough
• Software Inspection
• Technical Review
18
2. Traceability
– Forwards Traceability
» each input to a phase must be
matched to an output of the same
phase to show completeness
– Backwards Traceability
» each output is traceable to an input of
a phase
19
3. Testing
Testing is the evaluation of a system or component to:
• confirm that it satisfies requirements
• identify differences between actual and expected
results
Testing may be 50% of project costs;
so use cheaper methods before testing starts,
i.e., walkthroughs and inspections
20
4. Auditing
Audits are independent reviews that assess compliance
with specifications, standards and procedures
Perform an audit before software release:
• Physical audit
check that all items identified as part of the
configuration are present
• Functional audit
check that unit, integration and acceptance tests
have been carried out and that records show their
success or failure
21
5. Software Quality Standards
Standards and Guidelines
‘A standard is an approved, documented, and available set of
criteria used to determine the adequacy of an action (process
standard) or object (product standard).’
‘A guideline is a well-defined and documented set of criteria that
guides an activity or task.’ (Dorfman & Thayer, 1990)
• differ from standards - allows for judgement and flexibility
Categories of Standards and Assessment
– Product Standards
– Calibration and Measurement Standards
– Quality Management Systems Standards
22
Verification and Validation
23
Verification and Validation
• Validation:
Are we building the right system?
• Verification:
Are we building the system right?
Quality Assurance ensures that verification and
validation takes place.
24
Software Reviews and Inspections
A way of using the diversity of a group to:
• uncover errors in function, logic, or
implementation
• verify that software under review meets
requirements
• ensure that the software meets predefined
standards
• achieve uniformity of development
• make projects more manageable
25
The Review Players
review
leader
standards bearer (SQA)
producer
maintenance
oracle
reviewer
recorder
user rep
26
Software Reviews
• A meeting conducted by technical
people for technical people
• A technical assessment of a work
product created during the software
engineering process
• A software quality assurance
mechanism
• A training ground
27
Types of Review
• There are a number of types of review
ranging in formality and effect. These
include:
– Buddy Checking
• having a person other than the author informally
review a piece of work.
• generally does not require collection of data
• difficult to put under managerial control
• generally does not involve the use of checklists to
guide inspection and is therefore not repeatable.
28
Types of Review
– Walkthroughs
• generally involve the author of an artifact presenting
that document or program to an audience of their
peers
• The audience asks questions and makes comments on
the artifact in an attempt to identify defects
• often break down into arguments about an issue
• usually involve no prior preparation on behalf of the
audience
• usually involve minimal documentation of the process
and of the issues found
• process improvement and defect tracking are therefore
29
not easy
Types of Review
– Review by Circulation
• similar in concept to a walkthrough
• artifact to be reviewed is circulated to a group of the
author(s) peers for comment
• avoids potential arguments over issues, however it
also avoids the benefits of discussion
• reviewer may be able to spend longer reviewing the
artifact
• there is documentation of the issues found, enabling
defect tracking
• usually minimal data collection
30
Types of Review
– Inspection
• formally structured and managed peer review
processes
• involve a review team with clearly defined roles
• specific data is collected during inspections
• inspections have quantitative goals set
• reviewers check an artifact against an unambiguous
set of inspection criteria for that type of artifact
• The required data collection promotes process
improvement, and subsequent improvements in
quality.
31
Technical Reviews
The Review Meeting
» between 3 to 5 people (reviewers) should be
involved
» advance preparation should occur (< 2 hours work)
» meeting should be < 2 hours
» should be for a small part of the overall software
» is attended by the producer, review leader and
reviewers
» must have a follow-up procedure to ensure any
32
corrections are completed
Technical Reviews
The Review Meeting
» At the end of the review the attendees must
decide whether to:
• accept the work product without
modification
• reject the work product due to severe
errors (once corrected another review is
performed)
• accept the work product provisionally
(minor errors to be corrected but no further
33
review)
Conducting a Review
1. be prepared—evaluate
product before the review
2. review the product, not
the producer
3. keep your tone mild, ask
questions instead of
making accusations
4. stick to the review agenda
5. raise issues, don't resolve them
6. avoid discussions of style—stick to technical
correctness
7. schedule reviews as project tasks
8. record and report all review results
34
Technical Reviews
The Review Report should contain:
- Type of Review, Team Members, Date, Time, Place
- Item being reviewed
- Agenda
- Checklist
- Comments per Checklist item
or per Issues list item (Pressman) with problems
mentioned
Action per Comment - No action, Fix error, Reconsider
- Decision for the item being reviewed
35
- Accept, Reject (severe errors), Accept (minor errors)
Technical Reviews
Review Guidelines
» Review the product not the producer
» Set an agenda and maintain it
» Limit debate and rebuttal
» Enunciate problem areas, but don’t attempt to solve every
problem
» Take written notes
» Limit the number of participants and insist on advance
preparation
» Develop a checklist for each work product
» Allocate resources and time schedule
» Conduct meaningful training for all reviewers
36
» Review earlier reviews
Software Inspection
• The inspection process comprises three broad
stages:
– preparation
– collection
– repair
• Gilb and Graham [GilbGraham93] expand this
three stage process into the inspection steps; Entry,
Planning, Kickoff Meeting, Individual Checking,
Logging Meeting, Root Cause Analysis Edit,
Follow Up, Exit.
37
Benefits of Software Inspection
•
•
•
•
30% to 100% net productivity increases;
Overall project time saving of 10% to 30%;
5 to 10 times reduction in test execution costs and time;
Reduction in maintenance costs of up to one order of
magnitude;
• Improvement in consequent product quality;
• Minimal defect correction backlash at systems integration
time.
• In addition to these tangible benefits, less tangible benefits
such as a training effect for inspectors are also evident.
38
Disadvantages
• Up front costs (although far outweighed by
benefit):
– Training
– Implementation Support
– Ongoing allocation of staff resources
• Not strictly repeatable
39