CTIS 494
Software Quality Assurance
by
Dr. Ozlem Albayrak
2007 Fall
by Ozlem Albayrak
menu
CTIS 494 Outline (1)
Week
Subject
17.09 Introduction to Software Quality Assurance
24.09 Software Inspections and Testing
01.10 Capability Maturity Model
08.10 ISO9000 and SPICE (1 hour missing)
15.10 Software Quality Metrics
22.10 Seven Basic (B7) Tools of Quality
29.10 The Seven MP Tools
05.11 MIDTERM (30%)
by Ozlem Albayrak
CTIS 494 Outline (2)
12.11 5S for Intelligent Housekeeping in Software Development
19.11 Software QFD and Voice of the Customer
26.11 Risk Assessment and Failure Modes and Effects Analysis
03.12 Quality Measures and Statistical Methods for Trustworthy Software
10.12 Verification and Validation
(PROJECT REPORTs Submission 15%)
17.12 Testing and Evaluation (1 hour missing)
24.12 Integration, Extension and Maintenance for Trustworthiness
(PROJECT PRESENTATIONs 20%)
31.12 Review (2 hours missing)
7.12 FINAL (35 %)
by Ozlem Albayrak
menu
Today ...
Week
Subject
17.09 Introduction
Introduction to Software
QualityQuality
AssuranceAssurance
to Software
24.09 Software
Inspections
and Testing
Software Inspections
and Testing
01.10 Capability Maturity Model
08.10 ISO9000 and SPICE (1 hour missing)
15.10 Software Quality Metrics
22.10 Seven Basic (B7) Tools of Quality
29.10 The Seven MP Tools
05.11 MIDTERM (30%)
by Ozlem Albayrak
Review



Introduction to course elements
What is quality?
Differences between service and product
by Ozlem Albayrak
CTIS494 PROJECT DETAILS



35% of the total grade is from the projects
(same weight as final)
The course project is a group project each of
which will be developed by 4 students.
Project subjects will be announced by the
instructor. Project requires groups to submit a
report and deliver a in-class presentation.
Reports will be submitted 2 weeks before the
presentations.
by Ozlem Albayrak
Project Report specifications




Project reports will be prepared as web pages that can
be accessed by all the students and the instructor.
The groups are free to design their web pages format,
but the generated pages should be readable and
understandable.
Navigation among the pages should be carefully
organized.
The quality of the web pages will be determined by is
content, format as well as its timely distribution. Late
submissions will be heavily penalized.
by Ozlem Albayrak
Project Presentations





Project study will be presented in classroom.
Following the presentations there will be a question-answer
session during which the instructor and the other students are
allowed to ask questions.
Those students who prepare good questions will be awarded by
the instructor.
Fluency of the presentation is as important as its content, thus the
groups are urged to prepare their presentations in advance to
deliver a high quality presentation.
Duration of the presentations will be announced by the instructor.
Using time effectively is an important element.
by Ozlem Albayrak
Project Evaluation


20% of the project grade is allocated for project
presentation and 15% is for the project report.
15% includes evaluation of the following items:
–
–
–

Content
Format and ease and fluency of navigation
Timely delivery
20% includes evaluation of the following items:
–
–
–
–
Content
Fluency of the presentation
Obedience to time limits
Answers to questions
by Ozlem Albayrak
Quality exercise






Define high quality toilette paper?
Single (5 min)
In groups of two (5 min)
Reviews.......
Hw # : form a group of four
One address for the class and registration to
the group.... Do not wait for invitation register
by Ozlem Albayrak
What is quality?
by Ozlem Albayrak
Different Perspectives on Quality

There are many different definitions and
dimensions of Quality
–
–
–
–
–
Transcendent
Product based
User based
Manufacturing based
Value based
by Ozlem Albayrak
Garvin’s 8 quality dimensions
Performance
Durability
Serviceability
Features
Reliability
Aesthetics
Conformance
Percieved
quality
by Ozlem Albayrak
Performance

Efficiency with which a product achieves its
intended purpose
by Ozlem Albayrak
Features

Attributes of a product that supplement the
product basic performance
by Ozlem Albayrak
Reliability

Propensity (eğilim) for a product to perform
consistently over its useful design life
by Ozlem Albayrak
Conformance

Is a particular dimension of a product is within
the allowable range of tolerance of the
specification, the product conforms.
by Ozlem Albayrak
Durability

The degree to which a product tolerates stress
or trauma without failing
by Ozlem Albayrak
Serviceability

Ease of repair for a product
by Ozlem Albayrak
Aesthetics

Subjective sensory characteristics such as
taste, feel, sound, look and smell.
by Ozlem Albayrak
Perceived Quality

As the customer perceives the quality.
QUALITY
by Ozlem Albayrak
Service Quality dimensions




More difficult to define than to define product
quality
Tangibles
Service reliability
Assurance
by Ozlem Albayrak
Differing Functional Perspectives
on Quality






Engineering
Operations
Strategic Management
Marketing
Financial
Human Resources
by Ozlem Albayrak
Engineering

Engineering is an applied science.
–
–

Product Design Engineering
Concurrent Engineering (simultaneous
performance)
Statistics + technical orientation
by Ozlem Albayrak
Design Life Cycle
Idea Generation
Preliminary Design
Iterations
Prototype Development
Final Definition
Product Design Evaluation
Implementation
by Ozlem Albayrak
Operations


Focus was initially on statistical process control
Uses system view underlying modern quality
management thinking
by Ozlem Albayrak
Strategic Management



Refers to the planning processes
Long term orientation
Goal: to achieve sustainable competititve
advantage
by Ozlem Albayrak
Generic
Strategic Planning Process (1)
Firm Mission and Goals
External analysis
Strategic Options
Business level strategy
Corporate level strategy
Operational Subplans
by Ozlem Albayrak
Internal analysis
Generic
Strategic Planning Process (2)
Operational Subplans
Organizational
Design
Conflict Politics and
Change
Strategic Alignment between
Structure and Goals
by Ozlem Albayrak
Organizational
Reward System
Marketing

Customer satisfaction
by Ozlem Albayrak
Financial

Risk of investment vs. Potential rewards
by Ozlem Albayrak
Deming Value Chain
Improve quality
Cost decrease
Productivity Improves
Capture the market
Stay in business
Provide Jobs and more jobs
by Ozlem Albayrak
Human Resources



Employee empowerment
Job analysis
Total quality human resources management
by Ozlem Albayrak
The Three Spheres of Quality
Quality
Management
Quality
Assurance
Quality
Control
by Ozlem Albayrak
Quality
Control
Quality Control

Scientific method includes:
–
–
–
Analysis ( a process is broken into fundamental
pieces)
Relation (relationships between parts)
Generalization (how interrelationships apply to the
larger phenomenon of quality being studied)
by Ozlem Albayrak
Quality Control Activities






Monitoring process capability and stability
Measuring process performance
Reducing process variability
Optimizing processes to nominal measures
Performing acceptance sampling
Developing and maintaining control charts
by Ozlem Albayrak
Quality
Assurance
Quality Assurance

Activities associated with guaranteering the
quality of a product or service.
by Ozlem Albayrak
Quality Assurance Activities







Failure mode and effect analysis
Concurrent Engineering
Experimental Design
Process improvement
Design team formation management
Off-line experimentation
Reliability/durability product testing
by Ozlem Albayrak
Quality
Management
Quality Management

Management processes that overarch and tie
toegther the control and assurance activities
make up quality management.
by Ozlem Albayrak
Quality Management Activities






Planning for quality improvement
Creating a quality organizational culture
Providing leadership and support
Providing training and re-training
Designing a system to reinforce quality ideals
Facilitating organizational communication
by Ozlem Albayrak
Other perspectives


Value Added Perspective
Cultural Perspectives
by Ozlem Albayrak
Contingency Theory

There is no theory or method for operating a
business that can be applied in all instances.
This provides flexibility to managers in pursuing
quality.
by Ozlem Albayrak
Summary


Different perspectives on quality management
Quality sphere is composed of:
–
–
–
Quality Control
Quality Assurance
Quality Management
by Ozlem Albayrak
Quality Example





Define high quality toilette paper?
Single (5 min)
In groups of two (5 min)
Reviews.......
One address for the class and registration to
the group.... Do not wait for invitation register
by Ozlem Albayrak
CTIS 494
Software Quality Assurance
by
Dr. Ozlem Albayrak
2007 Fall
by Ozlem Albayrak
A Practical Approach to Software
Quality
Book by
G. O’Regan
Slides by
Dr. Ozlem Albayrak
2006 Spring
by Ozlem Albayrak
Outline






Software Engineering Challenge
History of Software Failures
Background to Software Quality
History of Quality
Software Engineering
Modern Software Quality Management
by Ozlem Albayrak
Next Week Assignment...

Your group will
–
Write a C/C++ code to do addition....
–
Please refer to your first assignments
by Ozlem Albayrak
Software Engineering Challenge

Challenge is to deliver:
High quality software on time to the customer

Focus on customer satisfaction
by Ozlem Albayrak
Software Engineering Challenge




Overestimation
How to estimate cost and schedule accurately
is a big challenge
Risk management is key
Example: Taurus project at London Stock
Exchange was 13,200% late and costed a lot,
and abondened
by Ozlem Albayrak
Software Engineering Challenge



Changing requirements
In civil engineering there are also problems
Software eng. less mature than civil eng.
by Ozlem Albayrak
History of Software Failures

Y2K problem
–
–
–

Find and analyze all code with a Y2K impact
Plan and make necessary changes
Cost to UK is $32 billion!!!
Software quality needs to be a key driving
factor
by Ozlem Albayrak
Introduction to Software Quality(2)

History of Quality
–
–
–
–
–
Shewart
Deming
Juran
Crosby
Other Quality Gurus
by Ozlem Albayrak
Background to Software Quality



Customers have high expectations
In the market there are many high quality
companies
Late delivery leads to extra cost
by Ozlem Albayrak
Background to Software Quality (2)

What is software quality
–
“Conformance to the requirements” Crosby
–
“Fitness for use” Juran
–
ISO 9126 standard for information technology
by Ozlem Albayrak
ISO 9126 – Quality Characteristics






Functionality: extent to which functions are avaliable
Reliability: extent to which software reliable
Usability: users say easy to use
Efficiency
Maintainability: easy to maintain
Portability: easy to transfer do different environments
by Ozlem Albayrak
Total Quality Management

Modern approach to quality management is
TQM
–
–
–
Customer focus
Process improvement
Development of quality culture
by Ozlem Albayrak
Software Quality Control





Extensive inspections and testing
Formal reviews
Aim is to identify defects
White box and black box tests
Unit, function, subsystem, system,
performance, integration, acceptance tests
by Ozlem Albayrak
History of Quality

Shewhart
–
–
Statistian at AT&T, 1920s
PDCA Model




Plan
Do
Check
Act
by Ozlem Albayrak
History of Quality (2)

Deming
–
–
–
1945s, Japan
14 step programme
Consistent purpose and direction in the organization
is more important then everyone doing his best
alone
by Ozlem Albayrak
History of Quality (3)

Juran
–
–
–

Another giant in the quality movement
Fitness for use
10 step programme
Crosby
–
Has influenced CMM (Capability Maturity Model)
by Ozlem Albayrak
Other Quality Gurus



Taguchi
Ishikawa
Feigenbaum
by Ozlem Albayrak
Introduction to Software Quality(3)

Software Engineering
–
–
–
Requirements Analysis
Specification
Design




–
Architectural
Functional
Object-oriented reuse
Verification of Design
Implementation
by Ozlem Albayrak
Introduction to Software Quality(4)
–
Testing






–
Unit
Integration
Sub-system
System
Performance
Acceptance
Maintenance
by Ozlem Albayrak
Maintenance


1st Malcom Baldridge Quality award to Motorola
Steps of Six Sigma:
–
–
–
–
–
–
Identify product or service
Identify customer and requirements
Identify needs to satisfy customer
Define process of doing work
Eliminate waste
Ensure continious improvement by:

Measuring, analyzing and controlling improved process
by Ozlem Albayrak
Modern
Software Quality Management




Software Inspections
Software Testing
Software Quality Assurance
Problem Solving
–
–
–
Fishbone diagrams
Histograms
Pareto Chart
- Trend Graph
- Scatter Graphs
- Failure Mode Effect Analysis
by Ozlem Albayrak
Modern
Software Quality Management(2)






Modeling
Cost of Quality
Metrics
Customer Satisfation
Assessments
Total Quality Management
by Ozlem Albayrak
Metrics

Measurement is important part of science and
engineering

Goal Question Metric by Basili

Data, Charts, Analysis, Trends, Action
by Ozlem Albayrak
Customer Satisfaction
Customer Saticfaction
Surveys
Customer Saticfaction
Measurement
Improvement
Planning
Execute Plan
by Ozlem Albayrak
Formal Methods

Advantage:
–
–
Utilization of mathematical models are being used
Lead to more robust(gürbüz) systems
by Ozlem Albayrak
Modern
Software Quality Management




Future of Software Quality
Organizational Culture and Change
Formal Methods
Quality and the Web
by Ozlem Albayrak
Quality and the WEB
Accesss from everywhere in the world
 May different browsers
 Usability, look and feel important
 Performance as key concept
 Security
 Many transactions at any time
 Avaliability (24x365)
 Relaibility
JAD/RAD with spiral development suggested

by Ozlem Albayrak
End of Musical notes

Check this web site:
http://sw-assurance.gsfc.nasa.gov
by Ozlem Albayrak
TO DO by Next Week:


Review
Next class:
Search web for software quality assurance
Come to class to discuss
by Ozlem Albayrak
Software Quality Assurance
S. Thomas Foster, Jr.
Boise State University
PowerPoint
prepared by
Dave Magee
University of Kentucky
Lexington Community College
modified by O. A.
Stephen McIlroy
BT Retail
Senior Systems Architect
Software Quality Assurance

What is it?

Why do we need it?

How do we achieve it?
by Ozlem Albayrak
What is Quality Assurance ?

Quality =
–
–
–


...meeting the customer’s requirements,
...at the agreed cost,
...within the agreed timescales.
Quality = “Fitness for purpose”
Quality = Customer satisfaction !
Quality Assurance is what we do to achieve this
by Ozlem Albayrak
“Success” v “Failure”
Quality Assurance helps us avoid failure !
Failure
Failure
Success
Content
Failure
by Ozlem Albayrak
Software failures
some examples

Taurus (London Stock Exchange)
–

London Ambulance Service
–

faults led to reversion to manual processes
Ariane 5
–

11 years late & 13,200% over budget
coding defect in module reused from Ariane 4
AT&T switching system
–
coding defect in three-line “fix”
by Ozlem Albayrak
Software failures
the statistics
Cancelled
31%
Successful
16%
Challenged
53%
by Ozlem Albayrak
Source: The Standish Group
Software failures
cost overruns
Under 20%
21 - 50%
51 - 100%
101 - 200%
201 - 400%
Over 400%

15.5%
31.5%
29.6%
10.2%
8.8%
4.4%
Average cost for “failed” projects is 189% of
the original estimate, i.e. almost double.
by Ozlem Albayrak
Source: The Standish Group
Software failures
time overruns
Under 20%
21 - 50%
51 - 100%
101 - 200%
201 - 400%
Over 400%

13.9%
18.3%
20.0%
35.5%
11.2%
1.1%
Average time for “failed” projects is 222% of
original estimate, i.e. more than double.
by Ozlem Albayrak
Source: The Standish Group
Software failures
content deficiencies
Under 25%
25 – 49%
50 – 74%
75 – 99%
100%

4.6%
27.2%
21.8%
39.1%
7.3%
On average “challenged” projects deliver 61%
of specified functions
by Ozlem Albayrak
Source: The Standish Group
Software failures
are things getting any better?
Significantly more failures
Somewhat more failures
No change
Somewhat fewer failures
Significantly fewer failures

27%
21%
11%
19%
22%
Only 41% of IT managers think there are fewer
failures today than 5 years ago
by Ozlem Albayrak
Source: The Standish Group
Why is it so hard to succeed?
by Ozlem Albayrak
Why is it so hard to succeed?
Software engineering is difficult
–
–
–
–
–
problems are complex & difficult to define
solutions are complex & difficult to specify
high degree of precision often required
progress is difficult to monitor
rapid pace of technological and business change
by Ozlem Albayrak
Why is it so important to succeed?





Huge demand for software
Shortage of skilled people
Limited financial resources
The need for correctness
Limited windows of opportunity
We can’t afford the cost of failure
by Ozlem Albayrak
How do we achieve QA?


By adopting a disciplined, professional
approach - engineering is predictable and
repeatable
Quality Management System
–
–

Documented processes & Best Practice
ISO 9001 & TickIT registration
Continuous Improvement
–
–
New methods (e.g. DSDM, UML) & tools
Better & more focussed Training
by Ozlem Albayrak
The Quality Management System

Defines the way we work
–
–
–

Procedures and guidelines
–

focuses on what rather than how
flexible
broadly applicable
mandatory elements and recommendations
Mix of best practice & common sense
by Ozlem Albayrak
Project control







Project initiation
Project closure
Estimation
Risks & issues
Project & Quality plans
Schedule plans
Progress monitoring & reporting
by Ozlem Albayrak
Quality control




Traceability
Quality Control Reviews
Approval process
Validation, Verification & Testing
by Ozlem Albayrak
Summary




What is Quality Assurance?
Why is it so difficult to succeed?
Why is it so important to succeed?
How do we achieve QA?
by Ozlem Albayrak

The goal of software inspection and test is to
reduce the expected cost of software failure
over the life of a product.
by Ozlem Albayrak
Popular Definitions of
Software Quality








Software that is developed right the first time
Minimization of fixes between releases
Measured loss imported to society after delivery
User satisfaction
Fitness for use
Performance/ expectations
Measurable properties of software
Conformance to requirements
by Ozlem Albayrak
Quantitative Definitions






Availability
Reliability
Defect rate
Problems per month
Mean time between failure
Loss to society (Taguchi)
by Ozlem Albayrak
What is a good software?





Written according to a coding standard
Design must be high quality
Succint and useful documentation
Comprehensive suite of regression tests must
accompany the software
Ref: H. Erdogmus, IEEE Software, 2007
by Ozlem Albayrak
Software Inspections and Testing



Inspection plays a key role in building software
Testing plays a key role in verifiying software
There are different ways to software
inspections:
–
–
Formal
Informal
by Ozlem Albayrak
Formal Inspections


Fagan inspection methodology
Gilb methodology
Roles
Author
Inspector
Tester
Moderator
by Ozlem Albayrak
Informal Inspection


Walkthrough of document or code by individual
other than author
Deliverable is reviewed informally
by Ozlem Albayrak
Fagan Inspections







Planning
Overview
Preparation
Inspection
Process improvement
Re-work
Follow-up
by Ozlem Albayrak
Economic Benefits of Sofware
Inspections

Positive impacts on:
–
Productivity
Quality
Time to market
Customer satisfaction
–
IBM Space Shuttle %85 inspections, %15 testing
–
–
–
by Ozlem Albayrak
Benefits of Software Inspections



Good for educating people about standards
and procedures
Cost of defect increases as software is being
developed
Cost of quality estimation varies between 20%40%
by Ozlem Albayrak
Software Inspection Methodology


Fagan, Gilb
Typical Roles
Author
– Inspector
Also
– Moderator
– Tester
– Reader
–
by Ozlem Albayrak
E-mail/Fax reviews


No physical meeting reviews
Depending on the organization this type of
reviews may work
by Ozlem Albayrak
Semi Formal Review Meeting



Actual review meeting
If review leader is other than author may work well
Phases are:
–
–
–
–
–
–
Planning
Distribution
Informal meeting (optional)
Preparation
Review meeting
Post review (corrections are verified)
by Ozlem Albayrak
Meeting Minutes
Item Under Review__________ Review Reference_________
Date____________________
Document Name __________ Version No_______________
Author__________________ No of Previous Reviews____
Reviewers_________________________________________
__________________________________________________
NOTES___________________________________________
_______________________________________________
by Ozlem Albayrak
Actions from the review
Issue
No
Raised
by
Page/
DSection/Line
by Ozlem Albayrak
Description
Action
Summary block in inspections





# major defects discovered
# minor defects discovered
Author estimate of rework time
# hours preparation
Amount of reviewed (pages/LOC)
by Ozlem Albayrak
Fagan Inspection Methodology

Objective of inspection is
–
–
to identify as many defects as possible and
to confirm the correctness of a particular
deliverable.
by Ozlem Albayrak
Fagan Inspection Process







Moderator – Planning/overview
Moderator, author – Overview
Inspectiors – Preparation
Inspectors – Inspection
Inspectors – Analysis, improvement
Author – Rework
Moderator, author – Follow-up
by Ozlem Albayrak
Fagan Inspection Guidelines
Activity
Area
Amount/hr
Max/hr
Preparation
Requirements
Design
Code
Test Plans
4 pages
4 pages
100 LOC
4 pages
6 pages
6 pages
125 LOC
6 pages
Inspection time
Requirements
Design
Code
Test Plans
4 pages
4 pages
100 LOC
4 pages
6 pages
6 pages
125 LOC
6 pages
by Ozlem Albayrak
Relaxed Fagan Inspection
Guidelines
Activity
Area
Amount/hr
Max/hr
Preparation
Requirements
Design
Code
Test Plans
4-8 pages
4-8 pages
100-130 LOC
4-8 pages
6 pages
6 pages
125 LOC
6 pages
Inspection time
Requirements
Design
Code
Test Plans
by Ozlem Albayrak
Fagan Inspection Methodology





Roles
Entry Criteria
Preparation
Inspection Meeting
Exit Criteria
by Ozlem Albayrak
Inspection Exit Criteria (1)
Inspection
Type
Mandatory/
Optional
Exit Criteria
Requirements
M
1.Requirements satisfy the customers needs
2.All requirements defects are corrected
Design Ins.
M
1.Design satisfies the requirements
2.All defects in the inspection are corrected
3.Design satisfies the design standards
Code Ins.
M
1.Code satisfies design and requirements
2.Code follows coding standards
3.Code compiles correctly
4.All defects from inspection removed
by Ozlem Albayrak
Inspection Exit Criteria (2)
Inspection
Type
Mandatory/
Optional
Exit Criteria
Test Plan
M
1.Test plan sufficient to test requirements
2.Test plan follows test standards
3.All defects from inspection removed
by Ozlem Albayrak
Issue Severity




Major
Minor
Process Improvement
Investigate
by Ozlem Albayrak
Defect Types
Code
Design
Requirements
Logic
Design
Requirements
Maintainability
Interface
Data Usage
Performance
Standards
Code Comments
Usability
Requirements
Logic
System Interface
Portability
Reliability
Maintainability
Error handling
Other
Product objectives
Documentation
Hardware interface
Competition Analysis
Function
Software Interface
Performance
Reliability
Spelling
by Ozlem Albayrak
Software Testing

Objective is
–
–

to verify the correctness of the software system and
to identify defects that are present.
Testing effort is often complicated by real world
issues, such as late delivery of software.
by Ozlem Albayrak
Test Planning











Identify scope of testing to be done
Estimates of time, resources, people, hardware, software and
tools
Provide resources needed
Provide test environment
Assign people to tasks
Define schedule
Identify risks and contingency plan
Track progress and take corrective action
Provide regular test status of passed, blocked,failed tests
Re-plan if scope of the project changes
Conduct post mortem to learn any lessons
by Ozlem Albayrak
Sample Test Plan
Activity
Resource
Name
Start
End
Comments
Review
requirements
Test Team
15.02.2007
05.03.2007
Complete
by Ozlem Albayrak
Steps in Test Plan












Review requirements
Overall test plan and review
System test plan and review
Performance test plan and review
Usability test plan and review
Regression plan and review
Test environment setup
System testing and re-test defects
Performance testing and re-test defects
Usability testing
Regression testing
Test reporting
by Ozlem Albayrak
Test Process

The quality of testing is dependent on the maturity of
the test process and a good test process include:
•Test
planning and risk
management
•Dedicated test environment and
test tools
•Test case definition
•Test automation
•Formality in handover to test
department
by Ozlem Albayrak
•Test
execution
•Test result analysis
•Test reporting
•Measurements of test
effectiveness
•Post mortem and test process
improvement
Test Case

A Test Case should include:
–
–
–
–
–
Purpose of test case
Setup required to execute the test case
Inputs to test case
The test procedure
Expected outputs or results
by Ozlem Albayrak
Test Management Tools
Open source Test Management Tools:
 http://www.opensourcetesting.org/testmgt.php

http://www.technologyevaluation.com/request/t
esttool_softeval.asp?se=google_testtools&gcli
d=CJvgv8Ca3o4CFQxaXgodIy-FPA
by Ozlem Albayrak
Next week:

Bring your first assignment to class
–

Demo’s will be done
Come to class prepared:
–
–
What is CMM/CMMI?
List of Turkish software companies having CMM(I)
by Ozlem Albayrak