Measuring Assurance Case
Confidence using
Baconian Probabilities
Charles B. Weinstock
John B. Goodenough
Ari Z. Klein
May 19, 2013
© 2013 Carnegie Mellon University
Copyright 2013
Carnegie Mellon University and IEEE
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with
Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development
center.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS
FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER
EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR
PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE
MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT,
TRADEMARK, OR COPYRIGHT INFRINGEMENT.
This material has been approved for public release and unlimited distribution.
This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without
requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software
Engineering Institute at permission@sei.cmu.edu.
DM-0000387
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
2
The Problem
C1
The system is safe
C2
C3
Hazard A has
been eliminated
Hazard B has
been eliminated
Ev1
Ev2
Ev3
Evidence
Evidence
Evidence
How confident in C1? Why?
What does it mean to have confidence?
What could be done to improve confidence? Why?
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
3
How is Confidence in a Hypothesis Increased?
A classic philosophical problem:
• Determining the basis for belief in a hypothesis when it is impossible to
examine every possible circumstance covered by the hypothesis
Use Induction
• Enumerative: number of confirming instances (Pascalian)
• Eliminative: variety of reasons for doubt (Baconian)
• Measuring support for a hypothesis/claim
– Enumerative: support increases with number of confirmations
– Eliminative: support increases with the number of excluded alternative
explanations, i.e., by eliminating reasons for doubting the claim
• Defeaters: reasons for doubting a claim
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
4
A Baconian Theory of AC Confidence
Confidence is the degree of belief
We grade “degree of belief” in terms of the number of eliminated defeaters
(reasons for doubt)
•
•
•
•
i out of n reasons for doubt (i/n)
0/n – no confidence
n/n – no remaining reasons for doubt
8/10 – residual doubt
Fundamental principle: Build confidence by eliminating reasons to doubt the
validity of:
• Claims (look for counter-examples and why they can’t occur)
• Evidence (look for reasons the evidence might be invalid and show those
conditions do not hold)
• Inference rules (look for conditions under which the rule is not valid and why those
conditions do not hold)
As reasons for doubt are eliminated, confidence grows (eliminative
induction)
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
5
Citigroup Tower – New York
C1.1
Citigroup Tower
is safe
R2.1
Unless there
is a design
error
IR2.2
If the design of the
system is shown
to be safe, then
the system is safe
Ev3.1
Wind tunnel
tests
showing safe
design
UM4.1
•
•
•
•
The design accounted
passed all reviews
for straight
on wind, not quartering wind
The wind tunnel (and other) tests
werehad
Still
successful
adequate safety margins
All parties
But
construction
involved
replaced
were welded
convinced
joints
with bolted
the building
jointswas safe
• The
But…
margin of safety was lost
But the model of the
building tested does
not match the actual
building
UC3.2
Unless salient attributes
of the design were
overlooked
UM4.2
But not all
real-world wind
conditions were
simulated
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
7
Concerns with the Baconian Approach
•
•
•
•
What if a relevant defeater has not been identified?
What if a defeater cannot be completely eliminated?
Surely not all defeaters are of equal importance. How is this handled?
Baconian probability seems rather weak in contrast to Bayesian or Pascalian
probability. What is being gained or lost?
• The potential number of defeaters seems incredibly large for a real system. Is
this approach practical?
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
8
Unidentified Defeaters
Issue: Does the failure to identify a relevant defeater lead to an inflated
sense of confidence?
• The inability to identify some defeaters is itself a reason for doubt that needs
to be recognized in a case.
• An assessment process for reviewing a case will need to take the possibility
of missing defeaters into account.
• A similar problem exists with the use of assurance cases. How can you be
sure that your hazard analysis has covered all potential hazards?
• We always admit the possibility that additional defeaters will be identified in
the future (this is what defeasible reasoning is all about).
– When we say that we have “complete” confidence in a claim (i.e., that the
claim has Baconian probability n|n) we understand that this only reflects
what we knew at a particular point in time.
The concepts of eliminative induction and defeasible reasoning help in
developing sound and complete arguments.
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
9
Partially Eliminated Defeaters
Issue: How are defeaters that cannot be completely eliminated dealt
with?
• An argument should state a claim that, if true, serves to completely eliminate
an associated defeater.
• There may be residual doubts about the truth of such a claim, in which case,
the same doubts apply to the defeater’s elimination
– as an argument is refined, one eventually develops defeaters that are
“obviously” eliminated by associated evidence
• If it is impossible to eliminate some lowest level defeaters, then the
associated doubt leads to incomplete elimination of a higher level defeater
(and the claim that it defeats.)
A goal in developing a convincing argument is to formulate low level
defeaters that can be eliminated by appropriate evidence.
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
10
Defeaters of Different (Relative) Importance
Issue: It seems reasonable that eliminating some defeaters would lead
to higher levels of confidence than eliminating others (of lesser
importance).
• Independent of relative defeater importance, 1|2 (some doubts eliminated)
always represents more confidence than 0|2 (no doubts eliminated), and 2|2
(all doubts eliminated) always represents complete confidence
• Even though the hazards that are represented in a safety case have different
likelihoods and impacts relative to one another, they all must be demonstrably
mitigated in order to establish sufficient confidence that the system is safe.
– Assessing the relative importance of hazards is not practically profitable.
• A system developer would not use the elimination of low impact defeaters to
justify an increase in confidence any more than he would represent minimally
unlikely and impactful safety hazards in a safety case.
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
11
Why Baconian Probability?
Issue: What is being gained and lost (over Pascalian or Bayesian
probability) with this approach?
• With eliminative induction we learn something concrete about why a system
works, and with enumerative induction, we may learn something statistical
about the system.
• The Baconian approach allows us to articulate both the reasons why a
system can fail and the reasons why the argument can be defective
– Uneliminated defeaters serve to focus additional assurance efforts.
• the Baconian approach allows evaluation of a system prior to its operational
use
• The Baconian approach avoids confirmation bias
The Baconian, Pascalian, and Bayesian approaches inform each other.
• Make Pascalian claims about system reliability and then elucidate the
possibilities that would make you doubt the validity of the claim.
• Take repeated samples to understand the operational reliability of the system.
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
12
Scalability and Practicality of the Approach
Issue: is this approach practical?
• The number of defeaters relevant to an argument is quite large for a real
system.
– But the amount of relevant argument and evidence for a real system is
inherently quite large.
• We believe that the Baconian approach allows one to develop a more
thorough and cost-effective basis for developing confidence in system
behavior than current methods.
• We hope that this approach will lead to more effective and focused assurance
efforts.
This all remains to be seen. Our initial interactions with systems
developers have been promising.
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
13
Summary
The identification of defeaters and how they are eliminated provides
framework for assessing confidence
The framework provides useful ways of thinking about assurance case
deficiencies
• Identification of deficiencies is the first step towards their mitigation
• The end result is a stronger assurance case
The approach is still under development. We welcome your ideas
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
14
Contact Information
Charles B. Weinstock
Senior Member of the Technical
Staff
Software Solutions Division
Telephone: +1 412-268-7719
Email: weinstock@sei.cmu.edu
U.S. Mail
Software Engineering Institute
4500 Fifth Avenue
Pittsburgh, PA 15213-2612
USA
John B. Goodenough
SEI Fellow (retired)
Software Solutions Division
Telephone: +1 412-268-6391
Email: jbg@sei.cmu.edu
Ari Z. Klein
Ph.D. Candidate – Rhetoric
Software Solutions Division
Telephone: +1 412-268-7700
Email: azklein@sei.cmu.edu
Measuring Assurance Case Confidence
Weinstock/Goodenough/Klein, May
2013
© 2013 Carnegie Mellon University
15