Registration Management Committee (RMC)
How to Audit Risk Management
Atlanta, GA
July 22 & 23, 2010
Kimberly Maggie
Ron Tarach
QUAL-TECH, INC.
Company Confidential
Auditor Workshop
Atlanta, GA
July 22-23, 2010
1
Registration Management Committee (RMC)
Agenda
• What is Risk?
• Risk Management Process
• Examples Risk Management Criteria
• Auditor perceptions of Risk Management
• Risk Management Tools
– Auditor knowledge of tools and actions
Atlanta, GA
July 22-23, 2010
2
Registration Management Committee (RMC)
Agenda (continued)
• Audit Planning
– Audit Planning Tools
• Activity 1 - Brainstorming session using
Audit Planning Tool
• Conducting the Audit of Risk Management
Process
– Examples of areas to evaluate
• Activity 2 - Brainstorming session using Case
Study and Failure Modes and Effects Analysis
(FMEA)
Atlanta, GA
July 22-23, 2010
3
Registration Management Committee (RMC)
Ice Breaker!
Atlanta, GA
July 22-23, 2010
4
Registration Management Committee (RMC)
What is Risk?
An undesirable situation or circumstance that
has both a likelihood of occurring and a
potentially negative consequence.
AS9100:2009, clause 3.1
Atlanta, GA
July 22-23, 2010
5
Registration Management Committee (RMC)
“Risk is inherent in all processes. Unfortunately, we don’t
see the results of ineffective risk management methods
until later”.
Atlanta, GA
July 22-23, 2010
6
Registration Management Committee (RMC)
Risk Management Process
– Most organizations spend a great deal of time and
manpower trying to document “Risks” but many
times this data is decentralized and not easily
accessible to the functions that need this
information.
– Process manufacturing can be so complex that
“Risks” can be very subtle and if there is not a
structured “Risk Management Process” that takes
advantage of corporate knowledge, lessons learned
an organization’s exposure to “Risk” can remain
high.
Atlanta, GA
July 22-23, 2010
7
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010
8
Registration Management Committee (RMC)
Examples of Risk Management Criteria
» Understanding the types of risk that could come
into a company. They could be related to
• Employees
• Process
• Design
• Manufacturing
• Equipment
• Environment
• Project
• Security
Atlanta, GA
July 22-23, 2010
9
Registration Management Committee (RMC)
Examples of Risk Management Criteria
» Understanding the types of risk that could come
into a company cont.
• External
• Contractor
Atlanta, GA
July 22-23, 2010
10
Registration Management Committee (RMC)
Examples of Risk Management Criteria
(continued)
– Employees – the organizations need to
ensure the safety, training, and
qualifications of employees.
– Process – managing process variation.
– Design – building quality into the product
design from the start, including it’s affect
on planning.
– Manufacturing – ensuring that
manufacturing is more efficient with
streamlined quality planning.
Atlanta, GA
July 22-23, 2010
11
Registration Management Committee (RMC)
Criteria for Risk Management Process
(continued)
– Equipment – ensuring that equipment can
meet capabilities, current and future.
– Environment – ensuring that the
operations are not compromising the
environment (adequate lighting,
temperature control, noise, cleanliness,
etc).
– Security – managing the security needed
by the facility.
– Project – ensuring project risks are
evaluated before beginning.
Atlanta, GA
July 22-23, 2010
12
Registration Management Committee (RMC)
Criteria for Risk Management Process
(continued)
– External – developing plans to address the
potential impact of weather, issues with
transportation companies, city
infrastructure (relating to construction,
road closures).
– Contractor – ensuring impact is considered
for contractors working on the building,
equipment, or with employees.
Atlanta, GA
July 22-23, 2010
13
Registration Management Committee (RMC)
Auditor Perceptions of Risk Management
• That’s the way we identified and handled risk
when I worked at Aviation Anywhere, Inc.
• When I audited a Original Equipment
Manufacturer (OEM) last month they were
using FMEAs.
• This little company only uses tool XYZ – they
can’t be managing risk properly.
Atlanta, GA
July 22-23, 2010
14
Registration Management Committee (RMC)
Auditor Perceptions of Risk Management
(continued)
“Remember, the design and implementation
of an organization’s aerospace quality
management system is influenced by varying
needs, particular objectives, the products
provided, the processes employed and the
size and structure of the organization.”
AS9100:2009 General
Atlanta, GA
July 22-23, 2010
15
Registration Management Committee (RMC)
Auditor Perceptions of Risk Management
(continued)
• Organizational application of Risk can vary
based on situation, customer, product line.
• Audit approach & interviewing will need to be
appropriate to the organization.
• Remember, what is “Appropriate” to the
organization.
Atlanta, GA
July 22-23, 2010
16
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010
17
Registration Management Committee (RMC)
Risk Management Tools
– FMEAs e.g. dFMEA, pFMEA, etc.
– Fault Tree Analysis (FTA)
– Probabilistic Risk Assessment (PRA)
– Event Tree Analysis (ETA)
– Event Sequence Diagram (ESD)
– Master Logic Diagrams (MLD)
– Reliability Block Diagram (RBD)
Atlanta, GA
July 22-23, 2010
18
Registration Management Committee (RMC)
Risk Management Tools (continued)
– Risk Assessment Matrix
– Likeliness/Consequence Table
– SWOT (Strength Weakness Opportunity
Threat)
– Business Continuity/Current Capability
Matrix
– Risk Map and Control Scale
Atlanta, GA
July 22-23, 2010
19
Registration Management Committee (RMC)
Risk Management Tools (continued)
– Auditor knowledge of tools and actions
» No one auditor has experience with all the tools
available in the industry and how they are used.
» Familiarize your self with the various Risk
Management Tools (self study).
Atlanta, GA
July 22-23, 2010
20
Registration Management Committee (RMC)
Risk controlled – or “Oh No”?
Atlanta, GA
July 22-23, 2010
21
Registration Management Committee (RMC)
Risk Management Tools (FMEA)
Atlanta, GA
July 22-23, 2010
22
Registration Management Committee (RMC)
Risk Management Tools (Influencer Analysis)
Atlanta, GA
July 22-23, 2010
23
Registration Management Committee (RMC)
Risk Management Tools (Risk Consequence)
Atlanta, GA
July 22-23, 2010
24
Registration Management Committee (RMC)
Risk Management Tools
Atlanta, GA
July 22-23, 2010
25
Registration Management Committee (RMC)
Audit Planning
– Selecting the right audit tool.
– Identifying your audit criteria and any
reference documents.
– Identifying your audit scope, including
identification of the organizational and
functional units and processes to be
audited.
– Identifying an appropriate audit scope.
Atlanta, GA
July 22-23, 2010
26
Registration Management Committee (RMC)
Audit Planning Tools
– Process (Turtle) Tool
– Process Map Tool
– Supplier Input Process Output Customer
(SIPOC) Form
– Process Based Management (PBM) Process
Flow
Atlanta, GA
July 22-23, 2010
27
Registration Management Committee (RMC)
Process (Turtle) Tool
With Who?
With What
(Comp./Skills/Training)
(Materials, Equipment, Facilities)
Inputs (information and
Process
Outputs (information
material from other
and Material to other
processes)
processes
How?
How Effective/Efficient?
(Methods/Procedures/Techniques
(Measurable Objective)
Atlanta, GA
July 22-23, 2010
28
Registration Management Committee (RMC)
Process Map
Atlanta, GA
July 22-23, 2010
29
Registration Management Committee (RMC)
Supplier Input Process Output Customer
(SIPOC) Form
Atlanta, GA
July 22-23, 2010
30
Registration Management Committee (RMC)
Process Based Management (PBM) Process
Flow
Atlanta, GA
July 22-23, 2010
31
Registration Management Committee (RMC)
Activity 1 - Brainstorming session using
Audit Planning Tool
Atlanta, GA
July 22-23, 2010
32
Registration Management Committee (RMC)
Process (Turtle) Tool (Design)
With What
With Who?
Risk Management Software
Sales
Forms
Engineering
Documents
Production
Quality
Inputs
Customer, Internal Organization,
Regulatory, Statutory
Special Requirements (e.g. product or
process complexity)
Outputs
Outputs
Process
Drawing/Spec
Design
Contract Review
Travelers
Planning
- Risk Management
Routers
Production
Work Orders
Purchasing
Inspection Reports
Suppliers
Critical Items (functions, parts, software,
characteristics, processes)
How?
Shipping
AS9100, AS9110 and AS9120 Standards
How Effective/Efficient?
Quality Manual
Customer complaints
Standard Operating Procedure for Contracts
In process/final rejection
FMEA
Design verification/validation
Risk Assessment Matrix
Atlanta, GA
July 22-23, 2010
33
Registration Management Committee (RMC)
Process (Turtle) Tool (Design Excluded)
With What
With Who?
Risk Management Software
Sales
Forms
Engineering
Documents
Production
Quality
Inputs
Customer, Internal Organization,
Regulatory, Statutory
Special Requirements (e.g. product or
process complexity)
Outputs
Outputs
Process
Travelers
Planning
Contract Review
Routers
Production
- Risk Management
Work Orders
Purchasing
Inspection Reports
Suppliers
Critical Items (functions, parts, software,
characteristics, processes)
Shipping
How?
AS9100, AS9110 and AS9120 Standards
How Effective/Efficient?
Quality Manual
Customer complaints
Standard Operating Procedure for Contracts
In process rejection
FMEA
Final rejection
Risk Assessment Matrix
Atlanta, GA
July 22-23, 2010
34
Registration Management Committee (RMC)
Conducting the Audit of Risk Management
Process
– Examples of areas to evaluate
» Are all “Risk” identified during the RFQ and Contract
Review Process e.g. special requirements, critical
requirements.
» Ensure Top management clearly understands what
“Risks” they have and what they are doing to ensure
they are mitigating those “Risk”.
» Evaluate the selected Risk Management Tool for
effectiveness.
» How are “Risks” communicated and managed
throughout the organization e.g. Design, Planning,
Purchasing, Suppliers, Manufacturing, Inspection,
Delivery and Post Delivery.
» Design inputs, Design FMEAs, Design Verification and
Validation.
Atlanta, GA
July 22-23, 2010
35
Registration Management Committee (RMC)
Conducting the Audit of Risk Management
Process
– Examples of areas to evaluate continued
» Critical characteristics across the quality lifecycle,
ensuring the Process FMEAs and Control Plans are
linked.
» Processes in place for capturing leading and lagging
indicators related to Design Quality Performance.
» Evaluate whether the organization has closed loop
Continual Improvement Processes that captures and
sustains Product and Process Quality.
» Organization is using Lessons Learned and Best
Practices.
Atlanta, GA
July 22-23, 2010
36
Registration Management Committee (RMC)
Conducting the Audit of Risk Management
Process
– Examples of areas to evaluate continued
» Ensure organization’s Change Management Process
involves the right people at the right time with the
right process.
» Ensure integration of Change Management with
assessments to ensure correct consideration of “Risk”.
» Ensure “Risk Assessment” tracked, recommended
controls to completion and ensured that “Risk” were
mitigated as prescribed.
» Ensure controls are in place for “Risk” that still remain
after mitigation actions.
Atlanta, GA
July 22-23, 2010
37
Registration Management Committee (RMC)
Activity 2 - Brainstorming session using
Case Study and FMEA
Atlanta, GA
July 22-23, 2010
38
Registration Management Committee (RMC)
Closing!
Atlanta, GA
July 22-23, 2010
39
Registration Management Committee (RMC)
Questions!
Atlanta, GA
July 22-23, 2010
40
Registration Management Committee (RMC)
References
1.AS9100:2009
2.ISO 19011
3.FAA Risk Management Handbook 2009
4.NASA
Atlanta, GA
July 22-23, 2010
41