Start
1.0 – Identify
Information Required
2.0 – Gather
Information
3.0 – Assign Ratings
in the SMS
Evaluation Tables
4.0 – Record
Evidence to Support
Ratings
5.0 – Identify
Underlying SSDs
6.0 –Implement
Improvement
Initiatives
Safety Management System
1-SO-06
March 31, 2012
Self-Evaluation Guide
Page 1 of 12
1.0 - Identify information required
 Examine SMS Evaluation tables and identify the
information required to evaluate each sub-function
 Information comes from documents, interviews and
observation
2.0 – Gather information
 Conduct interviews, review documents
 Obtain information from multiple sources to ensure
accuracy and to eliminate bias
3.0 – Assign ratings in the SMS evaluation tables
 Rate performance in each sub-function on scale of 1 – 7
 Ratings from 1 – 4 relate to soundness
 Ratings from 4 – 6 relate to appropriateness
 Ratings from 6 – 7 relate to effectiveness
4.0 – Record evidence to support ratings
 Ratings are subjective and so must be supported by
evidence
 Do not identify people by name in the evidence column
5.0 - Identify underlying system safety deficiencies
 System Safety Deficiencies (SSDs) prevent a company
from achieving effective performance
 Identify underlying conditions that contribute to the
issues identified in the evaluation
6.0 – Implement improvement initiatives
 Implement activities to improve performance
 Prioritise improvement activities based on risk
 Record activities in the QA or SMS database to ensure
actions are completed and their effectiveness is
assessed
Finish
Figure 1 – Safety Management System Self-Evaluation Process
Safety Management System
1-SO-06
March 31, 2012
Self-Evaluation Guide
Page 2 of 12
Introduction
This user guide provides Certificate Holders with instructions on completing a safety
management system self-evaluation, and provides a template with which to document a
completed self-evaluation, for both internal use and for submission to BDCA if requested.
Background
One of the key’s to sound safety management is monitoring safety performance. If an
organization has been operating an approved SMS for several years, using a systematic
process to understand current safety-management performance is essential.
When completing a self-evaluation of safety management performance, it is important to
understand the concepts of hazard, risk, system safety deficiency (SSD) and mitigation. Refer to
document 1-SO-02 Key Definitions for BDCA’s definitions of these ideas.
Purpose
A safety management self-evaluation allows a Certificate Holder already operating an SMS to
determine the degree to which their SMS is reducing safety-risks to ALARP, and to identify
areas of strong and weak safety-risk management so that improvement initiatives can be
implemented.
Recommended Approach
For best results, assign one or two people to complete the self-evaluation. The personnel
chosen to complete the evaluation should be independent from the processes being evaluated.
To do a thorough evaluation, they should have a critical and analytical mindset: personnel with
some experience as auditors or conducting occurrence/incident investigations are good
candidates.
The safety management self-evaluation process is performance-based. Therefore, while
auditing skills (such as interviewing, identifying sources of information for review, etc.) are useful
skills to the evaluation process, the safety management self-evaluation is not an audit. Some
elements of the self-evaluation may require the evaluator to compare the company’s processes
against those required by regulation, though the majority of the areas examined in the selfevaluation relate to performance (i.e., the outputs of existing company processes), not to
regulated standards.
BDCA has identified five Functions that together constitute effective safety-risk management.
For each function, BDCA has created a Statement of Ideal Functionality. This is the
performance standard against which each Function will be rated. The Statements of Ideal
Functionality can be seen at the top of each page in Appendix A. These are the same tables
used by BDCA to evaluate the effectiveness of certificate holders’ SMS.
Safety Management System
1-SO-06
March 31, 2012
Self-Evaluation Guide
Page 3 of 12
There are three indicators of safety management performance:

Soundness - Does the company’s SMS have a foundation in documentation, process
and management support necessary to manage safety-risks to a level as low as
reasonably practicable?

Appropriateness – Does the SMS consistently target hazards specific to the certificate
holder? Does the SMS ensure that the organization understands and manages
operational, human and organizational factors with a view to improving safety
management?

Effectiveness – Are processes being employed to continuously improve proactive
safety management and safety performance? Does a positive safety culture exist
throughout the organization? Does the SMS ensure integrated management of risk and
the achievement of ALARP as a way of doing business?
Assigning ratings in terms of soundness, appropriateness and effectiveness makes it easier to
communicate what type of actions may be needed to address a given deficiency in
performance. For example, issues related to soundness may require additional training,
documentation, or coaching. Issues related to effectiveness may require new management
behaviours or updated rewards and recognition systems.
It is not unusual for a single organization to have ratings that range from barely ‘sound’ to highly
‘effective’, depending on the part of the company that is being evaluated, and the function being
assessed.
Scores from 1 to 7 are applied to indicate the level of soundness, appropriateness or
effectiveness of an aspect of the company’s SMS. This achieves two results: first, it allows the
company to quickly determine its areas of greatest weakness; second, it allows the company
and BDCA to track the company’s performance through time.
Safety Management System
1-SO-06
March 31, 2012
Self-Evaluation Guide
Page 4 of 12
Steps to Complete a Safety Management Self-Evaluation
Figure 1 depicts BDCA’s recommended process for completing a safety management selfevaluation, described in further detail in the paragraphs that follow.
1.0 – Identify Information Required
The evaluation will examine different types of information, including: the documentation and
training that underpins the SMS and provides guidance to managers and staff; day-to-day
safety-risk management activities (such as hazard and event reporting, mitigating actions, daily
management activities, and the integration of operational and technical activities with SMSrelated activities); and evidence that risk management and SMS-related activities influence long
term planning, allocation of resources, etc.
Activities in most SMS self-evaluations will consist of:

The review of policies, procedures and guidelines related to the SMS;

The examination of training records and course materials related to the SMS and
technical functions;

A review of activities resulting from internal and/or external audits or evaluations;

The review of reports of hazards and events, and the follow-up actions;

Interviews with a sample of managers and staff; and

The observation of daily activities related to the company’s day-to-day operations.
It is important to reduce bias when evaluating an SMS. People close to the SMS will be good
sources of information, but it is possible that they may be blind to issues. In order to minimize
and hopefully eliminate bias and to get different perspectives on the effectiveness of the SMS,
interview middle managers, supervisors and line employees about their experiences with the
SMS, and their thoughts on how it is performing.
2.0 - Gather Information
Conduct interviews and examine documents to find evidence of SMS performance. Refer to the
statements of ideal functionality and the sub-function descriptions located in the SMS Evaluation
Tables (Appendix A) to help develop questions. Multiple lines of evidence for each area will
result in more accurate measures of performance.
Safety Management System
1-SO-06
March 31, 2012
Self-Evaluation Guide
Page 5 of 12
Interview Behaviours
DO
DON’T
Start and end on time
Intimidate the interviewee
Have open-ended questions prepared when Ask too many closed questions: they can lead
you go into the interview.
the interviewee to the answer, and bias the
interview
Introduce yourself and the interview purpose Interrupt or talk over the interviewee
to put the person at ease
Listen attentively
Bring more than two interviewers to an
interview
Interview in pairs when possible
Conduct an interview with someone’s
supervisor in the room.
Take notes
Get distracted (e.g., don’t check emails)
Answer questions about the evaluation Ignore the interviewee’s answers
process.
Take some time to establish rapport with the Stick purely to prepared questions: adapt your
interviewee.
interview as needed to get the best
information.
Ask follow-up questions to gain deeper
understanding.
Turn off your cell-phone or blackberry
3.0 - Assign Ratings in the SMS Evaluation Table
The SMS Evaluation Tables (Appendix A) are the tools used to communicate a Certificate
Holder’s rating in each of the five functions of an effective SMS.
When assigning scores using the guidance below, refer to the statements of “Ideal
Functionality” and to the different sub-functions on the template to determine what constitutes
ideal functionality. This will help you to determine where the company being assessed falls on
the spectrum of performance.
Ratings of 1 to 4
Ratings of 1 to 4 relate to Soundness, which generally applies to the foundation of the
service provider’s safety management – the structure that enables it to consistently
manage safety risks. Ratings of 1 to 4 are to be expected in any organization that has
only recently implemented an SMS.
A score of ‘1’ in any sub-function indicates that the functionality is absent and that the
service provider is unaware of the need for the functionality or is not planning to develop
the specified functionality.
Safety Management System
1-SO-06
March 31, 2012
Self-Evaluation Guide
Page 6 of 12
A score of ‘2’ is applied if the functionality is absent or nascent, but the service provider
is aware of the need for such functionality and plans to address the shortfall.
A rating of ‘3’ is normally the minimum required for an assessment of ‘sound’ safety
management. Scores less than 3 indicate the function is not sound – that it lacks
documentation, knowledge and/or commitment to function.
Ratings of 2 to 4 in a mature SMS may indicate problems in the company’s safety
culture, where the processes have become ineffective or are not consistently followed,
or top-management commitment to proactive safety management has waned or been
eroded by complacency.
Ratings of 4 to 6
Ratings of 4 to 6 generally relate to the Appropriateness of the service provider’s SMS.
In other words, the company is conducting the “right activities for the right reasons”.
If a company’s SMS is Appropriate, priorities within the company are risk-based. The
organization demonstrates that it is well on the way to being proactive. Hazards and
risks are identified before changes are initiated, and the mitigating actions are taken as
necessary. Staff have (and are employing) the necessary skills to manage risk. Safety
management is integrated with other management systems, and with business
processes. Measures of safety performance are sought and used.
Ratings of 6 to 7
Ratings of 6 to 7 indicate Effective functionality, and normally co-relate with a positive
safety culture.
In a company with an Effective SMS, proactive safety management is consistently
conducted, and there are signs of constant “correction” in how the organisation manages
safety risks – at the individual level, and as a collective entity. The organization is
resilient: able to identify and recover from abnormal conditions quickly and with minimal
loss. The company at all levels has a healthy sense of unease which helps to combat
complacency.
4.0 – Record Evidence to Support Ratings
The last column in the SMS Evaluation Tables provides a space for the evaluator to record
evidence to support the findings in each of the sub-functions of an effective SMS. Concrete
evidence is crucial to the accuracy and credibility of an SMS self-evaluation, since the ratings
assigned are subjective. It is very important that people reading the evaluation report
understand why a given rating was assigned.
Safety Management System
1-SO-06
March 31, 2012
Self-Evaluation Guide
Page 7 of 12
5.0 - Identify Underlying System Safety Deficiencies
Sometimes issues identified with the way an SMS is performing are related to an underlying
condition known as a System Safety Deficiency (SSD), which is a condition or circumstance that
allows hazards of a like nature to exist.
Examples of SSDs include:
-
Lack of management support for non-punitive reporting or the development of a justculture;
Lack of documented key-processes, such as change management or risk assessment;
Inadequate QA processes related to work or to documentation;
Less than adequate personnel resources or training in safety-critical areas;
Etc.
If underlying SSDs identified through self-evaluation are not addressed, the company can
expect to see repeat problems and will be exposed to a greater degree of safety risk.
6.0 –Implement Improvement Initiatives
Improvement initiatives should be developed based on the ratings assigned in the tables and
the SSDs identified. Activities should be prioritised according to the effect they will have on
aviation safety (i.e., they should be prioritised based on risk). Activities should be tracked in the
company’s QA or SMS database to ensure that they are tracked to completion and evaluated
for effectiveness.
Conclusion
Self-evaluation of safety management is an excellent way to build understanding of the
performance of an SMS, and to identify areas for improvement.
The BDCA is available to review any SMS self-evaluation report and will provide feedback on
the application of the process, the conclusions reached and the actions taken.
1-SO-06
March 31, 2012
Page 8 of 12
APPENDIX A
SMS Evaluation Tables
Function 1 - Proactive & Comprehensive Safety-risk Management.
All components of the organisation are actively engaged in or support the proactive management of safety-risks (e.g. operational, technical,
financial, HR components, etc.). Safety-risk management is top-down, and integrated with strategic, business and HR planning and performance
measurement. The SMS is integrated with other management systems (e.g. financial, HR, quality, etc.). Safety-risk management by contracted
service providers, partners and stakeholders is aligned with and contributes to the SMS.
Function
Proactive & comprehensive safety
management
a. Performance-based safety policy (e.g. strategic
safety goal) and principles (e.g. sharing and using
safety-related information) guide the organisation’s
direction, and its staff’s behaviours
b. Safety goals & objectives are explicitly set (safety
planning) and measured, and progress reported
(accountability), to continuously improve safety
management and performance.
c. Safety planning is integrated with project, business
and strategic planning
d. Department-wide safety management policies &
procedures are applied (e.g. committees; reporting
programs); and integrated with other company
policies/procedures (operational/organizational)
e. Policies and processes for proactive safety-risk
management are employed when introducing
change (operational, technical, organizational)
f. Safety management activities of sub-contracted
service providers (aviation and non-aviation) are
compatible and effectively integrate with the
Certificate Holder’s safety management.
Not
Present
Exists in
Part
Exists
Completely
1
2
3
4
5
6
7
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
Observations / Evidence (cite sources)
Attachment Page 8 of 5
1-SO-06
March 31, 2012
Page 9 of 12
APPENDIX A
SMS Evaluation Tables
Function 2 - Individual and Organizational Reliability
The Accountable Manager is responsible for safety performance, and formally delegates responsibility for specific functions and tasks, which are
clearly articulated in up-to-date job descriptions. There are neither gaps nor overlaps, and spheres of influence are respected. Managers and staff are
qualified and proficient. Procedures are documented, up-to-date and employed by all. Processes to continuously improve organizational performance
are embraced and valued by all.
Function
Individual and organizational reliability
a.
b.
c.
d.
e.
f.
g.
There are explicit, unambiguous, aligned roles
and responsibilities within the organization which
are known, employed and respected.
There are explicit and unambiguous roles and
responsibilities between the organization,
contracted service providers and other
stakeholders.
Qualified staff demonstrate safety management &
technical competence (e.g. training, certified,
performance evaluation).
All components of the organisation demonstrate
consistent adherence with:
- regulations & standards;
- operational, technical & safety management
policies/processes.
Documents are complete, controlled, up to date
and employed.
Organisational reliability is periodically measured
(e.g. risk-based audit program) and the results
employed.
Processes are actively employed to continuously
improve organisational performance.
Not
Present
Exists in
Part
Exists
Completely
1
2
3
4
5
6
7
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
Observations / Evidence (cite sources)
Attachment Page 9 of 5
1-SO-06
March 31, 2012
Page 10 of 12
APPENDIX A
SMS Evaluation Tables
Function 3 - Explicit Safety Risk Management
Mechanisms are in place to identify, analyse, assess and manage hazards and risks, with a focus on their organisational “roots” - system safety
deficiencies. Safety-risk management is conducted in all components of the organisation. Hazard analyses are employed proactively during the
development and deployment of technical systems, and during operational and organisational changes. Hazards and SSDs are identified from
numerous sources, including reporting programs, audits and inspections, committees, evaluations and surveys, etc. Information is employed in
day-to-day decisions, and longer-terms plans to improve or sustain safety performance.
Function
Explicit safety-risk management
a.
b.
c.
d.
e.
f.
g.
h.
i.
j.
Safety policy & principles “drive” safety
management decisions and staff activities
throughout the organisation
Information from an up-to-date safety-risk profile
shapes short- and long-term decisions throughout
the organisation
The occurrence (event) reporting program is
trusted and understood, and consistently
employed (input, analysis, output)
The hazard reporting program is trusted and
understood, and consistently employed (input,
analysis, output)
Explicit hazard analyses and risk assessments
are consistently and effectively employed to
manage future and ongoing safety-risks
System safety deficiencies are periodically
identified by analysing safety-related information
Company-wide safety management data is
collected, analysed and employed in tactical and
strategic decisions
Where applicable, fatigue is consistently and
effectively managed
Abnormal operations are anticipated, planned for
and managed
Up-to-date emergency response is planned,
coordinated and tested/measured
Not
Present
1
2
Exists in
Part
3
4
5
Exists
Completely
6
7
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
Observations / Evidence (cite sources)
Attachment Page 10 of 5
1-SO-06
March 31, 2012
Page 11 of 12
APPENDIX A
SMS Evaluation Tables
Function 4 - Safety Culture
Everyone in the company understands their role and is committed to sharing information so that the risks related to organizational, human and
operational factors are actively managed. Proactive safety management is embraced throughout the organization. There is a free flow of safetyrelated information vertically and laterally, within and outside the organization. The positive safety culture provides organisational resiliency, which
enables the organisation’s SMS to remain appropriate and effective during times of change.
Function
Not
Present
Exists in
Part
Exists
Completely
1
2
3
4
5
6
7
Safety information is valued and used at all levels
of the organisation. It is obtained from consistent,
unfettered and active (3-way) communication
throughout the organisation.
☐
☐
☐
☐
☐
☐
☐
Safety information is actively sought and
communicated with stakeholders, including BDCA.
☐
☐
☐
☐
☐
☐
☐
c. Staff have confidence in non-punitive reporting.
☐
☐
☐
☐
☐
☐
☐
d. Managers trust and employ validated, safetyrelated information from all sources.
☐
☐
☐
☐
☐
☐
☐
e. Everyone from all levels “walks the talk” and are
actively engaged in proactive safety management.
☐
☐
☐
☐
☐
☐
☐
Safety Culture
a.
b.
Observations / Evidence (cite sources)
Attachment Page 11 of 5
1-SO-06
March 31, 2012
Page 12 of 12
APPENDIX A
SMS Evaluation Tables
Function 5 - Safety Measurement
The organisation seeks and employs feedback regarding the effectiveness of its safety management activities (e.g., internal audits, selfassessments, meeting debriefs). This information is used to improve the SMS, the company’s other management systems, and the company’s
safety performance. Measures are used to modify day-to-day activities and priorities; and influence strategic initiatives.
Function
Safety measurement
The strategic safety performance of the
organisation is consistently and periodically
measured, and the results employed to continue
or change ongoing initiatives to manage safety
risks to a level as low as reasonably practicable.
(This inquiry aligns with 1 b.)
b. Safety-related information from across the
company is systematically and consistently
collected, analysed and employed to identify areas
of strong or weak safety performance (and SMS
performance).
c. System safety deficiencies (SSDs) are assessed in
terms of safety significance to guide priorities to
improve safety management/performance
d. Risk-based audit and evaluation plans are
employed to obtain independent measures of
safety performance (internal or third party),
including measures of soundness (i.e. adherence
to processes) and of appropriateness and
effectiveness (i.e. reduction in safety risks or
improvements to safety management
performance). (Measures of soundness align with
lines of inquiry from section 2, and particularly 2 f.
Measures of appropriateness and effectiveness
align with those in sections 3 and 4).
Not
Present
Exists in
Part
Exists
Completely
1
2
3
4
5
6
7
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
Observations / Evidence (cite sources)
a.
Attachment Page 12 of 5