27 ISMOR
Learning from Experience (LFE) - MCA,
SWIFT & HAZOP
Chris Tilney, Atkins Ltd
2 September 2010
Structure of Presentation
●
Multi Criteria Analysis (MCA)
– Lessons learned from decision support applications in defence
●
Structured What If Technique (SWIFT) & Hazard & Operability (HAZOP)
– Brief introduction to the techniques
– Lessons learned from recent nuclear safety applications
– Potential non safety related applications in defence
Multi Criteria Analysis
(MCA)
MCA – LFE (1)
●
Context for lessons learned
– Primarily focused on application of MCA to UK MoD procurement
competitions i.e. down select decisions on Pre Qualification
Questionnaire (PQQ) responses and Tenders
– Personal opinions of presenter, not necessarily shared by other
Atkins staff or MoD personnel!
MCA – LFE (2)
●
Structure of Assessment Hierarchies
●
Derivation of Weighting Factors
●
Preparing the Evaluation Team
●
Scoring Criteria
●
Aggregation
●
Audit Trail/Toolset
●
Presentation of Results
MCA – LFE (3)
Structure of Assessment Hierarchies
Tower 1
Tower 2
Tower 3
Merits of
solution
offered
Capability to
deliver
solution
Commercial
Aspects
●
Senior Decision
Makers/Scrutiny favour:
– Towers not weighted relative to
each other
– No single, overall Figure Of Merit
(FOM) score
Performance
People
Price
System
Requirements
Document
(SRD)
Compliance
Processes
Terms &
Conditions
(T&Cs)
Tools
Plans
Delivery
Schedule
●
Small set of criteria (15 – 20)
works best
MCA – LFE (4)
●
Derivation of Weighting Factors
– Pairwise comparison works best. Three ‘rounds’ to derive weights:
●
1st round: initial pairwise scores followed by explanation of outliers
●
2nd round: score again then calculate weights based on consensus scores
●
3rd round: review weights, adjust if necessary (with rationale)
– Group dynamics, watch out for/control:
●
●
Dominant personalities
●
Deferral to rank (if military Subject Matter Experts (SMEs))
●
Non assertive gurus
Preparing the Evaluation Team
– Evaluation Guide: helps enforce coherency, transparency, robustness of
scoring process
– Kick-off group brief: ditto
– 1-2-1s on toolset: reduce risk of failing to keep to evaluation schedule
MCA – LFE (5)
●
Scoring Criteria
– Scoring scale
●
Absolute
●
0 to 10 point scale
●
3 reference points: 10, 7 & 4 with guidance for assessors
●
No ‘extra’ marks for exceeding requirement
– Confidence adjusted scores (demonstrable/evidence based capability)
– Minimum of 2 Assessors: Lead & Shadow(s)
– Moderators: resolve disagreements, holistic view/input to assessors
●
Aggregation
– Automate at lower levels of hierarchy
– “Man-in-loop” at higher levels of hierarchy
– Trigger points/capability thresholds for all criteria that are scored
MCA – LFE (6)
●
Audit Trail/Toolset - AWARD
– Simultaneous, distributed input (over Restricted LAN Interconnect (RLI))
– Strong functionality for:
●
●
Capturing assessor rationales, issues & concerns
●
Monitoring progress of evaluation
●
Reports/interrogation
Presentation of Results
– KISS principle to avoid information overload for Assessment Panel Members
– Executive Summaries of what is being offered
– Bar graphs for Towers 1 & 2
– ‘Horse blanket’ for Tower 3 T&Cs
– Risk adjusted Whole Life Cost (WLC) plots for price
– Combined Operational Effectiveness & Investment Appraisal (COEIA) plots
for Value for Money (VfM)
– Slides on major concerns, issues and ‘red flags’
MCA – LFE (7)
●
Commercial Issues
– Central finding of recent court judgements (e.g. case “Lettings International
Ltd v London Borough of Newham”):
Any aspect of the Authority’s requirements or the way that it will evaluate tenders that could affect
the way a bidder prepares its bid must be fully disclosed before tenders are returned.
– Implications may include, for example:
●
Assessment hierarchies – fully weighted, automatic roll up of weighted scores
●
Disclosure of assessor scoring guidance to tenderers
●
Predefined, prescriptive COEIA methodology (i.e. can’t be tailored after tenders
received). Dstl study is investigating this issue.
– Are we entering dangerous waters?
i.e. OA/Decision Support Analysis no longer informing decision makers but
effectively making the decision (by option scores and option ranking) by slavish
adherence to a detailed, predefined process
MCA – LFE (8)
●
Overall LFE findings
– Well understood/accepted by Senior Decision Makers
– Works well with careful preparation and checks & balances on aggregation
– BUT
– Can be very resource intensive
– Commercial issues may prevent/limit future application of some of the LFE
and undermine central tenet of OA to inform decision makers
Structured What If Technique
(SWIFT) & Hazard & Operability
(HAZOP) – Intro to techniques
SWIFT – Intro to Technique (1)
●
Structured brainstorming method for analysing process or system
●
Usually applied to systems or processes not deemed to be safety critical
but which do have safety related failure modes
●
Approach
– Select system, subsystem or process
– Chair poses pre-planned ‘What if’ questions, identified via
●
Task analysis
●
Basis of Design
●
Generic Checklists
●
Process Description
●
Standards, regulations & guidelines
●
Past incidents & accidents
– Multi disciplinary team (design, operation & maintenance) of SMEs answer ‘What if’
questions
– Results captured in log sheet by scribe
SWIFT – Intro to Technique (2)
Example SWIFT Log sheet
Project: Aircraft Hangar
Page No
Description: Aircraft movements inside and outside the Hangar space
Question
Category
What If
Consequences
Safeguards
Recor
d No
Recommendations
External
Factors and
Influences
High Winds
Potential collision
between Vehicle and
Hanger side walls
during operation
Handling crew properly
trained. Vehicle only
operated under strict
weather conditions.
Weather reports
regularly made during
operations
RN1
Handling crew training to
include regular practice of
scenario
Operating
errors and
other human
factors
Handling
operator
hitches up
vehicle
incorrectly
Vehicle breaks free of
handling crew control
during operation.
Collision could result in
injury to personnel and
damage to vehicle /
superstructure
Handling crew properly
trained. Vehicle parking
brakes manned during
operation
RN2
Include double checks
when handling vehicle
Handling
operator
directs
vehicle into
obstacle
Vehicle and or
superstructure
damaged, personnel
injured through
collision, crushing
Handling Crew Properly
trained. Operator fully
supported by crew when
handling the vehicle
Step 1
Step 2
RN3
Pre operation brief to be
given to Handling crew.
Additional trained
personnel to be made
available during operation.
HAZOP – Intro to Technique (1)
●
Considers the operation (operability) of a system or subsystem
●
Systematically identify deviations from the design intent which could
lead to Hazards or operability problems
●
Multi-disciplinary SME team activity
●
Makes recommendations which could influence the design
●
Approach
– Use Guide Word / attribute combination to suggest possible deviation(s)
– Discuss and agree all credible causes
– Identify all consequences, identifying Hazards
– List existing safeguards
– Discuss and agree further actions
– Record
– Repeat until complete
HAZOP – Intro to Technique (2)
Example HAZOP Log sheet
Project Vehicle Fuel System
Page No
Description The fuel tank, engine inlet/outlet pipes and hoses, fuel pump, fuel filter
Guideword
Causes
Consequences
Safeguards
Record
number
Recommendation
s
More Flow
Fuel pump out of
calibration
Engine floods
and stalls, fuel
leaks from
system and
pools under
vehicle with the
potential to ignite
upon contact
with hot surfaces
Operators trained to
check and maintain
fuel pump at regular
intervals
RN1
Fit fire
extinguishers in
vehicle cabins.
Provide training to
check for leaks
when vehicle stalls
Reverse Flow
Fuel pump
incorrectly set up
during fit
Major spillage of
fuel with
potential to ignite
and cause major
fire
Replacement fuel
pumps are pre use
checked before
dispatch from
manufacturers
RN2
Fit Non return valve
in fuel system
Backflow
experience
during a
pressure drop
caused by fuel
pump failure
Engine cuts out.
Small fuel
spillage may be
experienced with
potential to ignite
Fuel pumps are
checked and
maintained by
operators and
maintainers at set
intervals
RN3
Fit Non return valve
in fuel system
SWIFT & HAZOP - LFE
SWIFT & HAZOP – LFE (1)
●
Preparation
– Key to effective workshop
– Don’t circulate ‘what if’ Qs in advance to help avoid preformed views
– Ensure reference material to-hand during workshop e.g. large-scale design drawings
●
Attendees
– Number: No more than 8, including Chair & Scribe
– Expertise: Full coverage required (tension with number!)
– Chair:
●
Strong/assertive
●
Independent of the project
●
Ideally knows attendees to draw them out, but not too well (bias/favouritism)
●
Have sufficient technical knowledge to understand the discussion and be able to apply
judgement when directing the study, e.g. distinguishing between superficial/trivial issues and
important issues
– Scribe: Understands domain and technical jargon
SWIFT & HAZOP – LFE (2)
●
SWIFT vs HAZOP
– SWIFT
●
Don’t have to have finished design i.e. can apply to concepts
●
Quick and easy to implement (cf HAZOP)
– HAZOP
●
Mature design needed
●
Exhaustive analysis, completeness of analysis
●
Time consuming/resource intensive
SWIFT & HAZOP – Potential
Non Safety Applications
SWIFT & HAZOP – Potential Non
Safety Applications
●
Explore/define CONEMP
●
Develop OA scenarios & vignettes
●
Explore and assess merits of business change programme
●
General risk identification & analysis