Management System
Auditing
Assessment of auditing methods
© David Hoyle 2012
Objective

© David Hoyle 2012
To assess whether current auditing
methods will hold back or stimulate
developments to ISO 9001
Audit methods
Clause based
 Department based
 Contract based
 Process based
 Behaviour based

© David Hoyle 2012
Determine
Select a what
you
are going
to
method
that
AlwaysAT
know
LOOK
and
fulfils
your
whatyou
youfind
areit
when
objectives
and
attempting
to
what
you
are
matches
the
establish
going to LOOK
information
and
relative
to
your
FOR
resources
objective
available. (NAO)
Approach
Examine the method
 Identify assumptions
 Evaluate the method

© David Hoyle 2012
Clause based approach
ISO 9001
Select clause of the standard
Choose department
?
Objective
To determine
the extent to
which the QMS
conforms to
requirements of
ISO 9001
© David Hoyle 2012
Assess conformity with
requirements by


Audit
Report
Asking searching questions of each
person
Revealing evidence of conformity with
clauses of standard
Produce report of nonconformities
against clauses
Assumptions





© David Hoyle 2012
Departments are exclusively responsible
for meeting certain clauses
Evidence of conformity in one Dept is
indicative of conformity in other Depts
Conformity with clauses is evidence of
capability
Correcting non-conformity will improve
system effectiveness
People operate from the same causality as
nature and machines
Evaluation
1.
2.
3.
4.
5.
6.
7.
8.
9.
© David Hoyle 2012
Conformity?
Inherent risks?
Performance?
Capability?
Efficiency?
Effectiveness?
Improvements?
Confidence?
Use of resource?
Score on a
scale of 0-5
Clause based approach
ISO 9001
Select clause of the standard
Choose department
?
Objective
To determine
the extent to
which the QMS
conforms to
requirements of
ISO 9001
© David Hoyle 2012
Assess conformity
with requirements by


Audit
Report
Asking searching questions of each
person
Revealing evidence of conformity with
clauses of standard
Produce report of nonconformities
against clauses
Department based approach
Choose department
ISO 9001
Objective
To determine
the extent to
which the QMS
conforms to
requirements of
ISO 9001
© David Hoyle 2012
Select clauses of the standard
that apply
Assess conformity
?
with requirements by
Asking searching questions in each area
 Following a trail through each
department
 Revealing evidence of conformity with
clauses of standard

Audit
Report
Produce report of nonconformities
against clauses
Assumptions
Departments
exclusively
Conformityare
with
clauses responsible
is evidence
forof
meeting
certain clauses
capability
Evidence
of conformity
in one Dept
Correcting
non-conformity
willis
indicative of conformity in other Depts
improve system effectiveness
 People operate from the same
causality as nature and machines
© David Hoyle 2012
Evaluation
1.
2.
3.
4.
5.
6.
7.
8.
9.
© David Hoyle 2012
Conformity?
Inherent risks?
Performance?
Capability?
Efficiency?
Effectiveness?
Improvements?
Confidence?
Use of resource?
Score on a
scale of 0-5
Department based approach
Choose department
ISO 9001
Select clause of the standard
?
Objective
To determine
the extent to
which the QMS
conforms to
requirements of
ISO 9001
© David Hoyle 2012
Assess conformity
with requirements by
Asking searching questions in each area
 Following a trail through each
department
 Revealing evidence of conformity with
clauses of standard

Audit
Report
Produce report of nonconformities
against clauses
Contract based approach
Contract/
Order
Select random sample of contracts/orders and
critical characteristics
Obtain or produce flow chart of activities
from contract/order to fulfilment
?
Objective
© David Hoyle 2012
Asking searching questions at each stage
 Following a trail through each process
from input to output
 Revealing evidence of conformity with
customer requirements and with clauses
of standard

ISO
9001
To determine
whether the organization
has the ability to
consistently provide
product that meets
customer requirements
Assess conformity with
contracts and clauses by
Audit
Report
Produce report showing the degree
of conformity with requirements
Assumptions




© David Hoyle 2012
Conformity with clauses is evidence of
capability
Conformity with ISO 9001 will enable
consistent provision of conforming
products
Correcting non-conformity will improve
system effectiveness
People operate from the same causality as
nature and machines
Evaluation
1.
2.
3.
4.
5.
6.
7.
8.
9.
© David Hoyle 2012
Conformity?
Inherent risks?
Performance?
Capability?
Efficiency?
Effectiveness?
Improvements?
Confidence?
Use of resource?
Score on a
scale of 0-5
Contract based approach
Contract/
Order
Select random sample of contracts/orders and
critical characteristics
Obtain or produce flow chart of activities
from contract/order to fulfilment
?
Objective
© David Hoyle 2012
Asking searching questions at each stage
 Following a trail through each process
from input to output
 Revealing evidence of conformity with
customer requirements and with clauses
of standard

ISO
9001
To determine
whether the organization
has the ability to
consistently provide
product that meets
customer requirements
Assess conformity with
contracts and clauses by
Audit
Report
Produce report showing the degree
of conformity with requirements
So what’s the problem?
People don’t operate from the same
causality as nature and machines
 We think we can design organizations
in the same way we design machines
 A management system is not a set of
documents

© David Hoyle 2012
The circles of influence that
create management systems
Outputs are the results of processes
Satisfied
Dissatisfied
Stakeholders
Stakeholders
Demands
Business
environment
Produce
Influence
System of effectively
ineffectively
Organization
managedprocesses
processes
managed
Undesirable
Outputs
Desired
Outputs
Delivers
© David Hoyle 2012
Organization
Organization
open
isisaan
closed
system
system
Reality check





© David Hoyle 2012
Conformity is only part of the picture
People make choices
People are influenced by images, events and
their interaction with others
Many invisible risks are hidden in behaviour
The impact of behaviour is what effects
outcomes and capability
Alternative Approaches
Process management audit
 Behaviour assessment

© David Hoyle 2012
Process based approach
Mission,
Objectives
Measures
Identify organization’s mission, objectives and
success measures
Identify processes and sub-processes that
achieve these objectives
?
Assess process effectiveness by
Asking searching questions at each
stage
 Revealing how processes are being
managed
 Revealing evidence of capability
against objectives and measures

Objective
To determine whether
the organization’s
processes are being
managed effectively
© David Hoyle 2012
ISO
9001
Audit
Report
Produce report showing the
effectiveness of the system of
processes
Demands
Inputs
A Process Audit
Plan
Production
Produce
Product
Deliver
Product
Resources
S
R
Q
Support
Product
T
Satisfied
Outputs
How do you
Demands
How do you
How do you
Process
Process
Improved
Improved
Improved
know
you
are
How do you
know
know
you
areit happen?
What are
you trying
make
objectives & efficiency
monitoring
performance
effectiveness
doing
the
right
you are doing it right?
to do?
Measures
thing?
in the best
way?
&
What will success
Process review
look like?
© David Hoyle 2012
Process
improvement
Assumptions
The system as revealed through
systematic enquiry is the system that is
producing the results
 The information presented by the
organization is legitimate and has not been
fabricated

© David Hoyle 2012
Evaluation
1.
2.
3.
4.
5.
6.
7.
8.
9.
© David Hoyle 2012
Conformity?
Inherent risks?
Performance?
Capability?
Efficiency?
Effectiveness?
Improvements?
Confidence?
Use of resource?
Score on a
scale of 0-5
Process based approach
Mission,
Objectives
Measures
Identify organization’s mission, objectives and
success measures
Identify processes and sub-processes that
achieve these objectives
?
Assess process effectiveness by
Asking searching questions at each
stage
 Revealing how processes are being
managed
 Revealing evidence of capability
against objectives and measures

Objective
To determine whether
the organization’s
processes are being
managed effectively
© David Hoyle 2012
ISO
9001
Audit
Report
Produce report showing the
effectiveness of the system of
processes
Behaviour based approach
Goals,
Objectives
Drivers
ISO
9001
Identify the system objectives and
performance drivers
Identify a range of behaviours as outcomes
Maturity
Grid
that reflect levels of maturities and cross ref
clauses and apply weighting
Identify people who experience what is
happening
Objective
To measure the
inherent level of risk
to optimum business
outcomes and
governance issues
caused by the impact
of everyday patterns
of behaviour
© David Hoyle 2012
Mainframe
Audit
Report
Invite on-line 360° participation
based on the part they play –
confidential
Run analysis engine to compute results
Produce report showing scored risks to
the achievement of objectives and
drivers
Fulfilment process results (part)
#
Outcome
Av Score
1
We know the objectives that have to be achieved and how our
performance will be measured
39.5
2
We know the activities that need to be carried out to achieve the
objectives
31.7
3
The necessary physical and human resources are provided when
needed
28.2
4
Provisions made to minimize risk are successful
28.5
5
Work commences on time
35.7
6
Work is executed in accordance with policies and plans
30.5
7
Work flows without unplanned interruption
41.6
8
When things go wrong we put them right
39.3
9
No work is released until found in conformity with all requirements
43.4
10
Outputs delivered on time
26.0
© David Hoyle 2012
Maturity Grid for outcome 8
Outcome Which statement matches your experience the closest?
When
things go
wrong we
put them
right.
When things
go wrong we
put them
right as best
we can and
get on with
the job
We normally
correct
mistakes as
they occur
following our
procedures
but we
wouldn’t
record these
Records
show our
procedures
have been
applied to
correct
mistakes
and
prevent
them
happening
again.
We usually
review all
mistakes
formally in
line with our
procedure
and take
effective
action to
prevent
them
recurring
Records of
all mistakes
show that
it is rare
for the
same
mistake to
happen
twice
Level
0.00
1.00
2.00
3.00
5.00
Clauses
linked
8.5.2 Corrective action
4.2.4 Control of records
© David Hoyle 2012
Graphical Representation
By stakeholder
By clause
By business process
4
1
1
5
2
4
20
2
3
1 Process owner
2 Process workers
3 Process supplier
4 Process customer
© David Hoyle 2012
4
20
40
40
8
20
6
40
7
3
1 Mission management
2 Resource management
3 Demand creation
4 Demand fulfilment
4 Quality management system
5 Management responsibility
6 Resource management
7 Product realization
8 Measurement, analysis &
improvement
Assumptions




© David Hoyle 2012
The system as revealed through systematic
enquiry is the system that is producing the
results
Patterns of behaviour are lead indicators
of risk
Correlation between clauses and business
drivers relative to performance
The statements don’t produce wildly
different interpretations
Evaluation
1.
2.
3.
4.
5.
6.
7.
8.
9.
© David Hoyle 2012
Conformity?
Inherent risks?
Performance?
Capability?
Efficiency?
Effectiveness?
Improvements?
Confidence?
Use of resource?
Score on a
scale of 0-5
Behaviour based approach
Goals,
Objectives
Drivers
ISO
9001
Identify the system objectives and
performance drivers
Identify a range of behaviours as outcomes
Maturity
Grid
that reflect levels of maturities and cross ref
clauses and apply weighting
Identify people who experience what is
happening
Objective
To measure the
inherent level of risk
to optimum business
outcomes and
governance issues
caused by the impact
of everyday patterns
of behaviour
© David Hoyle 2012
Mainframe
Audit
Report
Invite on-line 360° participation
based on the part they play –
confidential
Run analysis engine to compute results
Produce report showing scored risks to
the achievement of objectives and
drivers
Confidence level
Clause based
Not for 3rd Party Audits
Department based Not for 3rd Party Audits
Contract based
Process based
Behaviour based
© David Hoyle 2012
Acceptable for 2nd Party
Audits
Acceptable for 3rd Party
Audits
Acceptable when combined
with other audit methods
Conclusion
Which methods will hold back or
stimulate developments to ISO 9001?
Clause based
Hold back
Department based Hold back
Contract based
Hold back
Process based
Stimulate
Behaviour based
Stimulate
© David Hoyle 2012
For Further Details:
Behaviour Assessment contact
ian.rosam@the-hpo.com
Process Audit contact
hoyle@transition-support.com
Thank you and have a safe journey home
© David Hoyle 2012