Advanced Engineering
Risk Audit Instruments
SYRAS & QMAS
Bob Bea
National Science Foundation
1
RESIN PROJECT
Define & Characterize
System, Performance Attributes & Failures
Operators, Organizations, Environments
Hardware, Structures, Procedures, Interfaces
QMAS
Audit team, Audit training, Audit scheduling
Screening assessments, Factors of Concern
Gradings, Consequences of 'Failure', Remediations
SYRAS
Normal rates of malfunctions, Performance Shaping Factors
Probabilities of proper Detection, Analysis, Correction
Probabilities of Failure for all System Components, Correlations
Probabilities & Consequences of Failure As Is
Mitigation Measures
Probabilities & Consequences of Failure As Could Be
Implementation Programs
2
Managing Pfi
Pfi
Concept
Definition
Design
Construction
Maintenance
Decommissioning
Reduce Likelihoods
Present
Future
Reduce Effects
Increase Detection
& Correction
Increase
Margins of Quality
3
Managing Cfi
Cfi
Rescue
Repair
Recovery
Rehabilitation
Direct
Immediate
Delayed
Indirect
On-Site
Off-Site
4
Resilience - time & cost
to regain Functionality
Good resililence
100 %
Functionality
Poor resililence
rehabilitation
rescue
recovery
repair
Time
5
Benefits / Costs
Sustainability - Utility
during life cycle
High Sustainability
Improvement
Utility
Initiation
Operation
Low Sustainability
Decay
Time
6
PROACTIVE
to prepare
increase detection &
correction
reduce likelihoods
reduce consequences
REACTIVE
to recover
INTERACTIVE
to respond
7
Analytical methods
QUALITATIVE
'soft' - linguistic v ariables
'subj ectiv e' integration
Moderate
Low
High
QUANTITATIVE
'hard' - numerical v ariables
analytical integration
0. 00
5. 00
10.00
MIXED QUALITATIVE & QUANTITATIVE
linguistic v ariables translated to numerical v ariables
analytical integration
Low
0
High
Moderate
2. 5
5. 0
7. 5
10
8
System analysis approach
Step #1: PERFORM A SYSTEM ANALYSIS
defi ne system hardware, software, environments,
organi zati ons and i ndivi dual s
Step #2: PERFORM A PROCESS ANALYSIS
defi ne how thi ngs are done: how the system hardware,
software, envi ronments, organi zati ons, and indi viduals
work
Step #3: PERFORM ANALYSES OF THE SYSTEM &
PROCESS AS PRESENTLY CONFIGURED
defi ne the l ikel ihood and "quali ty" achieved
Step #4: PERFORM ANALYSES OF THE
RECONFIGURED SYSTEM & PROCESS
defi ne the l ikel ihood and additi onal "qual ity"
that could be achi eved
9
Screening analysis strategy
Develop Coarse
Qualitative Model
of Existing
"System"
Evaluate
High Risk
Components
Identify High Likelihood
System Problem and
Consequence Components
and Potential Mitigation Measures
Revise Models and
Evaluate Risk Reductions
of Mitigation Measures
Develop Detailed
Quantitative Models
of High Risk Components
Evaluate Costs and
Benefits of Mitigation
Measures and Impliment
"Most Effective"
10
Objectives of analyses
•
•
•
•
•
•
NOT prediction
NOT numbers or analytical elegance
Detect potential hazards and flaws
Evaluate effects of mitigations
Disciplined framework for thinking
Empower those who have daily
responsibilities for quality and reliability
11
12
Example QRA/PRA
13
Proactive - Robustness
= damage tolerance
defect in tolerant
defect tolerant
Intensity of damage or defects
14
Quality
Attributes
SYRAS©
Durability
Safety
QA / QC
Design
Phase
QA / QC
Construction
Phase
Compatibility
Serviceability
Lifecycle
Task Organization
Operations
Phase
Task Organization
QA / QC
Maintanence
Phase
Task Organization
Task 1
Task 2
Task 3
Task 1
Task 2
Task 3
Base Rates
Base Rates
Base Rates
Base Rates
Base Rates
Base Rates
Influencing
Factors
Influencing
Factors
Influencing
Factors
Influencing
Factors
Influencing
Factors
Influencing
Factors
15
Independent
&
Dependent
Errors
Detection
Analysis
Failures of
QA/QC
•lack of expertise
•resource deficiencies
•excessive authority gradients
•communication malfunctions
•management breakdowns
•rejection of information
•violations
Errors
Correction
16
P[F] = P[FI|E] P[E]+P[FI|E]P[E]+P[FE|E]P[E]
due to intrinsic
causes with no error
due to intrinsic
causes with error
due to extrinsic
causes with error
17
18
19
20
21
22
SMAS Grade
1
Minor negative influence
Moderate negative influence
Extreme negative influence
1
1E1
1E2
1E3
4
5
6
7
Below average
Poor
Very poor
does not meet
any standards
or requirement s
Moderate positive influence
Extreme positive influence
Negligible influence
3
1E- 1
Good, average
exceeds all standards
and requirements
1E- 2
Minor posit ive influence
2
Very good
1E- 3
Excellent
Outstanding
exceeds all standards
and requirements
SYRAS PSF
23
QMAS©
•
•
•
•
Assessment instrument
Assessment protocol
Assessment team development
Follow-up
24
SMAS Main Menu
This is the starting point of the SMAS Data Management Tool.
Pre-Assessment Data - This way to Pre-Assessment Data Input Menu
Phase 1 Data - This way to Phase 1 analysis Data Input Menu
Phase 2 Data - This way to Phase 2 analysis Data Input Menu
FOCÕs / Scenarios - This way to Factors of Concern / ScenarioÕs Menu
Reports - This way to Reports
Utilities - This way to Import, Export, Delete Utilities
System Exit - This way to Exit SMAS and close MS Access.
25
QMAS components
Operating
Teams
Factors
Structural
Factors
Organizational
Factors
Interfaces
Factors
Environmental
Factors
Hardware
Factors
Procedural
Factors
26
QMAS components, factors, attributes
27
QMAS factors
Operating Teams
Process Aud iti ng
Safety Culture
Ris k Perception
Emergen cy Preparednes s
Command & Controls
Training
Communications
Requisit e Variety
Equip ment
Design gu idelines and
specification s
Materials
Demand systems
Power systems
Configurations
Control systems
Operators & other
Organ izations & other
Orga nizational
Process Aud iti ng
Safety Culture
Ris k Perception
Emergen cy Preparednes s
Command & Controls
Training
Communications
Resources
Structure
Design gu idelines & specs.
Materials
Loadings
Structure configura tion
Computer programs
Research, development and
testing background
Interfaces
Procedur es & other
Envi ronmental & other
Proce dures
Operating
Maintenance
QA/QC
Contractor selection
Pre-start up review
Emergen cy response
Manage ment of change
Validations
Environmental
Externa l ( Weather)
Internal
Social External (Regu latory, Society)_
Social (Internal)
(Within organ ization and
operating team)
Equip me nt & other
Structure & other
28
29
QMAS gradings
30
Components
QMAS mean results
Interfaces
Environments
Highest mitigation priorities
Structure
Equipment
Procedures
Organizations
Operators
0
1
2
3
4
Grades
5
6
7
31
QMAS protocol
Stage #1
ONSHORE
general information
system specific
interviews - operating,
management
Stage #2
OFFSHORE
walk downs
operations observations
maintenance observations
emergency drills
interviews
anonymous discussions
Stage #3
ONSHORE
additional information
additional interviews
32
QMAS - coarse qualitative
33
QMAS detailed qualitative
34
35
36
QMAS assessors
• QMAS counselor
• QMAS assessors
– Facility operators
– Facility management
• QMAS training program
– HOF background
– QMAS operations
– QMAS applications
37
QMAS assessors: THE MOST
IMPORTANT PART!
• Experienced
– Facility
– Process auditing
•
•
•
•
•
Respected
Integrity
Motivated
Observant
Thoughtful, insightful
38
QMAS
applications
39