Reusability
Interoperability
Survivability
Safety
Manageability
Supportability
Replaceability
Functionality
Reliability
Efficiency
Usability
Integrity
Maintainability
Flexibility
Testability
Security
Reliability
Availability
Maintainability
Safety
Security
What
Whathorror
horrorstories
stories(true
(true
or
orfalse)
false)do
doyou
youknow
knowof?
of?
and needs experience
Kristian Sandahl, IDA
krisa@ida.liu.se
Critical systems makes expensive methods worthwhile
Kristian Sandahl, IDA
krisa@ida.liu.se
lead to damage
Incident – dangerous system behaviour which
does not result in an accident
Hazard – a state that can result in an incident, but
is hindered by other mechanisms
Accident – unplanned sequence of events that
When failed a critical system can cause injury of people,
damage the environment or loose immense sum of
money.
Criticality is often expressed in terms of:
Terminology
Kristian Sandahl, IDA
krisa@ida.liu.se
Critical systems
Many
⇒Sandahl, IDA
Manyopinions
opinions
Kristian⇒
krisa@ida.liu.se
Statistical
Statisticaltechniques
techniques
requirements
Value-based – market sets the value
Manufacturing-based – conformance to
Usage-based – in the eyes of the beholder
Price?
Portability
Correctness
Transcendent – something we learn to recognize
Product-based – measurable variable
Quality factors
Perspectives of quality
is built on the principles:
Principle 1 Customer focus
Principle 2 Leadership
Principle 3 Involvement of people
Principle 4 Process approach
Principle 5 System approach to management
Principle 6 Continual improvement
Principle 7 Factual approach to decision making
Principle 8 Mutually beneficial supplier relationships
Kristian Sandahl, IDA
krisa@ida.liu.se
Another approximation: λ = (1-R)/t
[failures / hours of execution time]
Easier to manage: Failure intensity,
Approximation: MTTF/(1+MTTF)
failures during a specified time interval
Kristian Sandahl, IDA
krisa@ida.liu.se
The probability that the software executes with no
Reliability
A guideline to apply ISO 9001 to software industry, which
ISO 9000-3
10-1
1
$10 cost
$1 cost
Kristian Sandahl, IDA
krisa@ida.liu.se
Operation,
input state,
direct and indirect variables
A test case is defined as a run with named direct variables and values
A test case is independent of the operational mode
Thus a test case can generate multiple runs
Problem: how to select test cases?
Maximise variable distance in input space
behaviour
Spread input variables on levels, with similar failure
Accelerate rare, critical operation
Assign at least one case per operation
Distribute test cases proportionally to operations
Sanity check: #new operations + 100
Determine the total number of test cases.
Feature test – check that each operation work properly
Load test – mimic filed usage
Tests are executed in runs comprising:
1h
100 h
10 h
10-2
Select test cases
Kristian Sandahl, IDA
krisa@ida.liu.se
6 weeks
10-3
$100 cost
114 years
$1000 cost
10-6
Kristian Sandahl, IDA
krisa@ida.liu.se
Kristian Sandahl, IDA
krisa@ida.liu.se
114 000 years
10-9
Hundreds of deaths,
$109 cost
1-2 deaths, $106 cost
Time btwn failures
Failure intensity
Impact
Failure intensity guideline
Prepare for test
Apply data to decisions
Execute test
Plan tests
Develop operational profile
Define target failure intensity
Software reliability engineering
Kristian Sandahl, IDA
krisa@ida.liu.se
Don’t count multiple failures due to single fault
Run performance tests simultaneously
criteria
Decisions made according to reliability growth
fault (don’t count twice)
Kristian Sandahl, IDA
krisa@ida.liu.se
Run test cases
When a failure is found, log the time, correct the
Execute tests
duration which returns control to the system when
complete and whose processing is substantially different
from other operations.
Operational modes (e.g. day, night)
Initiators (human, systems)
Representation (tabular, graphical)
Ignore non-critical with p < 1-Rtarget
How do you find and asses probabilities of operations?
Find operations and their relative probability
An operation is a major system logical task of short
Develop operational profiles
Automatic logging
Heuristic evaluation 50% hit rate
User review with expert users
Attitude
Learnability
Usability engineering:
Normal users
Minor problem
Medium problem
Annoying
Kristian Sandahl, IDA
krisa@ida.liu.se
http://www.tickit.org/
n
tio
ra
st
illu
ly
on
Kristian Sandahl, IDA
krisa@ida.liu.se
Kristian Sandahl, IDA
krisa@ida.liu.se
Kristian Sandahl, IDA
krisa@ida.liu.se
standard deviation from the mean = 3.4 faults per million
Define
Measure
Variant of the Shewhart cycle
Analyze
Improve
Control
Statistical process control
The variance of approved products should lie +/- 3
certification auditors,
a standardized training course for certification auditors,
a registration scheme for approved certification auditors,
a system for accrediting certification bodies for conducting TickIT
certifications,
a logotype to be used on certificates to show TickIT certification.
a standard set of requirements on the competence and behavior of
An interpretation of ISO 9001 for software,
Missing function
Task failure
Six sigma
Input: artefact, test tasks (realistic, closed)
Participants: test user, facilitator, log keeper
Explain purpose
Background and daily matters
Task
Observe
If necessary: give hints
Debriefing
Output: List of problems
TickIT
Kristian Sandahl, IDA
krisa@ida.liu.se
Testing process
Usability problem severity classes
Kristian Sandahl, IDA
krisa@ida.liu.se
Observation test
Efficiency
Avoid trade-union representatives and IT-experts
Think-aloud test
Relevance
Task analysis
Prototyping (HI-FI, LO-FI), 3 rounds?
Usability testing
Usability
6.
5.
4.
3.
2.
1.
4: Managed
Kristian Sandahl, IDA
krisa@ida.liu.se
Kristian Sandahl, IDA
krisa@ida.liu.se
5: Optimising
Each level has process areas.
Set quantifiable goals
Select processes
Run processes
Measure objectives
Analyse measurements
Package experience
QIP
1: Initial
3: Defined
2: Repeatable
CMMI
Experience factory
Configuration Management
Process and Product Quality Assurance
Measurement and Analysis
Supplier Agreement Management
Project Monitoring and Control
Project Planning
Requirements Management
PA CMMI2
Kristian Sandahl, IDA
krisa@ida.liu.se
Kristian Sandahl, IDA
krisa@ida.liu.se
QFD
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Risk Management
Decision Analysis and Resolution
Organizational Environment for Integration
PA CMMI3
Kristian Sandahl, IDA
krisa@ida.liu.se
Kristian Sandahl, IDA
krisa@ida.liu.se
Inspection
data
Analysis
Coding
Inspection
Sandahl, IDA
data Kristiankrisa@ida.liu.se
Test-cases
Inspection Inspection Inspection
data
data
data
Design
Improvement: reduce variation, increase precision
Control – adjust the process
Assurance – prediction of defects
Appraisal – defect detection
Inspections in quality assurance
What’s get measured gets done
Importance of feed-back
Non-personal software
Creating a passion for quality
Live as you learn
Incentive system
Involve customers
Set prioritized goals
Quality is everybody’s responsibility
Document how you will work with quality
Improve continuously
Management
Kristian Sandahl, IDA
krisa@ida.liu.se
=TQM
communication
Wisdom
Kristian Sandahl, IDA
krisa@ida.liu.se
performance
infrastructure