Risk Theme
DIRC Research Conference
16 March 2005
Summary





Scope of the theme
Issues for risk in computer-based systems.
Exemplars of the Risk Theme approach
Outputs from the Risk Theme
Plans for the Risk Theme
2
Scope


The risk literature is huge and spans many disciplines
– need to focus.
Focus on computer based systems with a bias
towards “everyday risk”:
– How to improve the way we deal with risk (Ho, Mackie &
Martin).
– Develop approaches to new classes of risk (Baxter et al,
Wherton).
– Consciously take a cross-disciplinary approach – engaging
with relevant disciplines.
3
Issues

Risk Management:
– Assessment of current techniques using social,
psychological, economic, statistical viewpoints.
– Strengthen current approaches to address a wider range of
risks.
– Ways to use the approaches with real systems.

New Kinds of Computer-Based Risk:
– Trust failures,
– Emergent Risk,
– Adversarial Risk, … (e.g. LTCM studies)

Dissemination: guidance, standards
4
Exemplars



Risk perception: strong sociological literature –
important in policy arena.
Risk in organisational structure: how does the
development of new organisational structure affect
risk.
Trust and Risk: increasingly human trust
relationships are computer mediated – what affect
does this have on risk?
5
Exemplar: Risk and Trust



Human trust is increasingly machine mediated –
source of potential failures (e.g. Bed management
system).
Trust is a natural link between Risk and
Responsibility: A is responsible to B for W often
means that B trusts A to do W in the current
environment.
B usually can’t justify all the trust in A so there is
Risk associated with trust.
12
Risk and Trust: Bed Management


Two main actors, M: the hospital bed manager, W:
ward level staff.
Different perception of danger:
– M mostly worries about closing admission (admin concern)
– W mostly worries about disrupting the “normal” work
(professional concern).



Likelihood is dynamic and depends on current state.
Things are fine if risk is low (prompt accurate
reporting)
If close to capacity – should W report a bed coming
free:
– Decreases likelihood of M’s worries (reduces risk)
– Increases likelihood of W’s worries (increased risk)
13
Risk and Trust: Dimensions





Taxonomy (meanings of trust) – affective,
competence, honesty, … dealing with polymorphic
aspect of trust.
Compositional accounts of trust – provenance,
authentication, …
Diversity/Form of information sources (issues about
market mechanisms, fault tolerance) (Ryan et al).
Bounded rationality (Gigerenzer, Tversky): Fast and
Frugal Heuristics (Aagaard)
Formal models of trust (Peacock)
14
Risk and Trust: Issues




Can we reduce “big”, Trust to smaller more
acceptable trust in the presence of relevant
evidence?
E.g. could we show that in an automated system we
can reduce seemingly unacceptable levels of trust to
those that are acceptable in non-automated systems?
Does the automated system support fast and frugal
heuristics?
How do formal models of trust failure help with
system design?
15
Risk theme outputs


A “Risk Reader” – DIRC outputs with narrative
commentary.
Case studies:
– Risk in PiMS, Bed Management
– SUN, QuinetQ, NATS?



Augmentation of existing Risk standards – NIST SP
800-30, EN 14971.
Short guide to applying DIRC techniques to Risk
analysis of computer-based systems.
Possibly web-based body of knowledge.
16
Plans





Risk reader complete mid 2005.
Complete work on analysing Risk aspects of case
studies.
Develop stronger linkages to other themes – via risk
perception, to Timing and Structure
Trust and Risk seems to be a good point of contact
with Responsibility,
Articulate linkage to Diversity from all the areas of
work – particularly market/regulated diversity.
17
The End
18
Responsibility and dependability

Most of us can think of failures that occurred
because of problems with responsibility
– For example, files were lost because A believed than B
had responsibility for all backups whereas B believed
that all users had responsibility to backup their own
files.

Failure of omission
– Misunderstood responsibilities meant that something
wasn’t done

Failure of commission
– Misunderstood responsibilities can mean that
something is done in an ‘incorrect’ or unintended way.
19
The notion of responsibility

Responsibility is a slippery concept because it can
mean so many different things:
– Responsibility for enacting a ‘workflow’
• X is responsible for the daily backup cycle
– Responsibility for a ‘state of affairs’
• Y is responsible for system management
– Responsibility for an ‘enterprise’
• Z is responsible for IT in the organisation
– Responsibility for a ‘goal’
• The Home Office is responsible for national security
20
Problems with responsibility

Responsibility and authority
– A is given the responsibility for B but not the authority to
do things that allow them to fulfil that responsibility.

Responsibility and blame
– If A is responsible for B and there is a failure of B then A
may be blamed for that failure. A may therefore disclaim
the responsibility.

Collective responsibility
– The ‘responsible’ may not be a single, clearly identified role
or person.
21
Responsibility ‘failures’





The ‘responsible’ who is assigned a responsibility is
inappropriate.
The ‘responsible’ does not have adequate resources
to discharge a responsibility.
The ‘success criteria’ for the responsibility and
undefined or poorly defined.
The ‘responsible’ and/or the assigner of the
responsibility misunderstand the nature of the
responsibility.
‘Responsibles’ have conflicting responsibilities.
22
Objectives of the theme



To explore the nature of responsibility and its
relationship with system dependability.
To propose notations, and techniques that will help
system procurers, designers and owners make
responsibilities explicit and reason about these
responsibilities.
To investigate how knowledge of responsibilities may
be used to reduce vulnerabilities in operational sociotechnical systems.
23
Theme outcomes





A body of work documenting our understanding of
the nature of responsibility.
A number of case studies illustrating the significance
of responsibilities in different systems.
A notation or a number of notations for modelling
responsibility.
Preliminary guidance on how knowledge of
responsibilities can be used to inform systems design
and operation.
These MAY be delivered as a book, a web site, a
collection of papers, etc.
24
Progress and plans


Progress
– On the nature of responsibility. Initial paper discussing
responsibility included in PA2 book.
– Case studies. Analysis of the Ladbroke Grove rail accident
from a responsibility perspective completed.
– Responsibility modelling. Experiments with a notation for
relating responsibilities to roles (documented in PA2 book).
Initial proposals for a notation that allows reasoning about
responsibility assignment and delegation.
Plans
– Focus on development of responsibility models and case
studies. Further case study and proposals for modelling
notation and modelling experiments by mid-2005.
25
Issues



Is responsibility purely a human notion or can
automated systems be considered to be ‘responsible’
for something?
Can we have a unified notation that allows causal (who
or what did something) responsibility and
consequential responsibility (who takes the blame)?
How do we integrate this work with work on the risk
theme?
26