Unit 1, Huguenot Place
17a Heneage Street
London E1 5LJ
United Kingdom
www.isealalliance.org
Evaluation of Auditor Competence
R074 Internal Version 1 - September, 2007
1. Introduction .......................................................................................................................... 2
2. Defining What is Being Evaluated ...................................................................................... 2
3. Setting up Auditor Evaluation ............................................................................................. 3
4. Validity and Reliability ......................................................................................................... 4
5. Evaluation Tools .................................................................................................................. 6
5.1 Initial Contact .................................................................................................................. 8
5.1.1 Self-Evaluation ......................................................................................................... 8
5.2 Training and Evaluation ................................................................................................. 9
5.2.1 Training Courses.....................................................................................................10
5.2.2 Assessing Personal Attributes ..............................................................................11
5.3 Audit and Post-audit Review ........................................................................................13
5.3.1 Workplace Assessments ........................................................................................13
5.3.2 Annual Review and Professional Development ....................................................14
6. Conclusion ..........................................................................................................................14
Annex 1: Attributes, Skills and Knowledge ...........................................................................15
Annex 2: Bloom’s Taxonomy .................................................................................................19
Annex 3: Self Assessment Example ......................................................................................20
Annex 4: Auditor Witnessing Report Example .....................................................................21
Annex 5: Auditor Evaluation Form Example .........................................................................24
Resources ...............................................................................................................................26
Evaluation of Auditor Competence
R074 Internal Version 1
1 of 26
September, 2007
1. Introduction
In a background research paper (R066) on auditor competence that ISEAL prepared in 2006,
the case was made that assessing the competence of auditors has evolved from a passive
qualifications-based approach (completion of training courses, # of audits, etc.) to a more
engaged and active assessment of an individual’s personal attributes, skills and knowledge.
The ISEAL Member workshop on auditor competence confirmed that the best auditors are not
always the ones who have the most qualifications but those who also possess the right
attributes and skills.
While ISEAL members are relatively content with how they have defined the competencies
required of their respective auditors, they are less clear on the options available for evaluating
their auditors against these competencies. This is likely a shared sentiment among much of the
conformity assessment world, as tools for evaluating the less tangible skills and attributes
required of auditors are only starting to be applied. This background paper focuses on providing
some examples of the ways in which the competence of auditors can be evaluated. In doing so,
it also touches on the definition of auditor competencies and the training required to instill the
knowledge and skills desired in potential auditors.
It should be stressed that current good practice for a certification or accreditation body, in terms
of how it demonstrates the competence of its auditors, is that “the basis for approving its
auditors' competency, their certification and the justification for awarding it, should be
[demonstrated]. A [certification or accreditation body] should be able to prove that its auditors
have been evaluated, tested and that their competence has been properly demonstrated.”
(AAPG Guidance, 2005) ISEAL members will need to assess whether their respective
definitions of auditor competencies and evaluation tools are sufficient to meet the understanding
of good practice.
2. Defining What is Being Evaluated
When looking at the various tools for evaluating the competence of auditors, it quickly becomes
clear that evaluation cannot be separated from the definition of competencies. How those
competencies are defined or structured will influence how easily compliance with them can be
assessed. In the background paper from the International Federation of Accountants (IFAC),
‘Towards Competent Professional Accountants’, this stepwise approach to achieving effective
evaluation tools is presented as follows:
“As long as it suits individual environments, the recommended way forward is a staged
approach by which educators and professional bodies:
 identify expected performance outcomes, that is, what accountants actually do at work
and at what standard;
 state what has to be demonstrated at work in terms of those outcomes;
 specify the capabilities required to demonstrate competence (knowledge, skills,
professional values and attitudes);
 decide on the most appropriate mix of assessment mechanisms for the environments
concerned. Assessments may include examinations of one sort or another as well as a
consideration of performance at work or workplace simulations.” (IFAC, 2003: p30)
While ISO 19011:2002, Guidelines for quality and/or environmental management system
auditing suffers from a number of commonly cited shortcomings, such as the potential for
Evaluation of Auditor Competence
R074 Internal Version 1
2 of 26
September, 2007
varying interpretations of its requirements, it remains the international reference document for
defining and evaluating auditor competence. Similar to the IFAC paper, ISO 19011 applies a
stepwise framework for evaluation. In clause 7.6.2, four main steps in the evaluation process
are defined:
 Step 1 - Identify the personal attributes, and the knowledge and skills to meet the needs
of the audit programme
 Step 2 - Set the evaluation criteria
 Step 3 - Select the appropriate evaluation method
 Step 4 - Conduct the evaluation
Both of these examples highlight the importance of framing auditor competencies in concrete
and measurable terms. The International Accreditation Forum (IAF) has developed guidance
(Draft 1) on the application of ISO 19011, in which they state in G7.6.2.1. that “Step 2 should
avoid the use of quantitative criteria in favour of qualitative competency based criteria that is
definite and measurable.” In other words, quantitative criteria such as years of experience or
courses completed are no longer the primary measure of competence. Rather, the focus should
now be on demonstrated attributes, skills and knowledge. Similarly, clause G.7.1.2. states that
“Competence requirements should be expressed in definitive and measurable terms in order
that they may be objectively examined.”
In the IFAC paper, they speak in terms of understanding performance outcomes - what is it that
an auditor is required to achieve - as a basis for defining competencies. These competencies
are identified primarily through an examination of the actual activities that auditors are required
to perform.
“A wide variety of methods have been used to identify, as distinct from assess, job
competences. These include:
 interviews, usually with employers and employees;
 surveys, for example, using postal questionnaires;
 workshops of various kinds, including group interaction; and
 various other techniques, including direct observation.” (IFAC, 2003: p22)
Organisations can also use existing documentation, such as training records, job descriptions
and experience records to inform the definition of competences.
While it is not the focus of this paper to go into detail on the definition of desired auditor
attributes, skills and knowledge, there is a relatively high level of consensus among leading
practitioners about the types of competencies required. A list of these competencies, drawn
from various key sources is provided in Annex 1 and ISEAL members are encouraged to assess
their own auditor competencies against this list, both for completeness and applicability.
3. Setting up Auditor Evaluation
It is useful to go back to ISO 19011 for a summary of how to develop an evaluation process.
Auditor evaluation is not a once-off proposition but occurs at a number of stages throughout the
duration of an auditor’s career.
“The evaluation of auditors occurs at the following different stages:
 the initial evaluation of persons who wish to become auditors;
 the evaluation of the auditors as part of the audit team selection process; and
Evaluation of Auditor Competence
R074 Internal Version 1
3 of 26
September, 2007

the continual evaluation of auditor performance to identify needs for maintenance
and improvement of knowledge and skills.” (ISO 19011, 2002: Clause 7.6.1)
Clause 7.6.2 is reproduced here in full as a clear summary of the steps required to develop an
evaluation process:
“Step 1 - Identify the personal attributes, and the knowledge and skills to meet the needs
of the audit programme
In deciding the appropriate knowledge and skills, the following should be considered:
 the size, nature and complexity of the organization to be audited;
 the objectives and extent of the audit programme;
 certification/registration and accreditation requirements;
 the role of the audit process in the management of the organization to be audited;
 the level of confidence required in the audit programme;
 the complexity of the management system to be audited.
Step 2 - Set the evaluation criteria
The criteria may be quantitative (such as the years of work experience and education,
number of audits conducted, hours of audit training) or qualitative (such as having
demonstrated personal attributes, knowledge or the performance of the skills, in training or
in the workplace).
Step 3 - Select the appropriate evaluation method
Evaluation should be undertaken by a person or a panel using one or more of the methods
selected from those in Table 2. In using Table 2, the following should be noted:
 the methods outlined represent a range of options and may not apply in all situations;
 the various methods outlined can differ in their reliability;
 typically, a combination of methods should be used to ensure an outcome that is
objective, consistent, fair and reliable.
Step 4 - Conduct the evaluation
In this step the information collected about the person is compared against the criteria set in
Step 2. Where a person does not meet the criteria, additional training, work and/or audit
experience are required, following which there should be a re-evaluation.”
4. Validity and Reliability
The primary consideration in choosing or developing evaluation tools is that they provide a
meaningful and accurate assessment of auditor competence. The very first clause of ISO
19011 dealing with auditor evaluations (7.6.1) states that:
“The evaluation of auditors and audit team leaders should be planned, implemented and
recorded in accordance with audit programme procedures to provide an outcome that is
objective, consistent, fair and reliable.” (emphasis added)
Psychometric testing is concerned with assessing the validity and reliability of a test.
“Classical test theory states that a test is reliable:
a. to the degree that it is free from error and provides information about examinees’ "true"
test scores, and
Evaluation of Auditor Competence
R074 Internal Version 1
4 of 26
September, 2007
b. to the degree that it provides repeatable, consistent results.
“Any factor which reduces score variability or increases measurement error will also reduce
the reliability coefficient. For e.g., all other things being equal, short tests are less reliable
than long ones, very easy and very difficult tests are less reliable than moderately difficult
tests, and tests where examinees’ scores are affected by guessing (e.g. true-false) have
lowered reliability coefficients.
“A test has content validity if it measures knowledge of the content domain of which it was
designed to measure knowledge. Expert judgement (not statistics) is the primary method
used to determine whether a test has content validity. Nevertheless, the test should have a
high correlation with other tests that purport to sample the same content domain.
“The process of using a test score as a sample of behavior in order to draw conclusions
about a larger domain of behaviors is characteristic of most educational and psychological
tests. Responsible test developers and publishers must be able to demonstrate that it is
possible to use the sample of behaviors measured by a test to make valid inferences about
an examinee's ability to perform tasks that represent the larger domain of interest.” (ACES,
Validity Handbook)
The American National Standards Institute (ANSI) has a paper on psychometric testing that
highlights the relationship between these terms:
“Fairness is considered to be the sum of validity and reliability; that is, if a psychometric
procedure promotes validity and reliability, it intrinsically promotes fairness.” (ANSI, 2004)
Reliability is a prerequisite for validity and refers to the ability of a test to measure a particular
trait or skill consistently. However, tests can be highly reliable and still not be valid for a
particular purpose.
“If a test is unreliable, it cannot be valid.
For a test to be valid, it must reliable.
However, just because a test is reliable does not mean it will be valid.
Reliability is a necessary but not sufficient condition for validity!”
(ACES, Validity Handbook)
For more information on the types of validity evidence, you are encouraged to visit:
http://www.collegeboard.com/highered/apr/aces/vhandbook/evidence.html
As a concrete example, ANSI assesses personnel certification programs for compliance with
ISO 17024. This process consists of an assessment of the folllowing elements:
 the validity of the study to determine legitimate competencies;
 whether the exam tools adequately fit the competencies to be evaluated;
 whether the tools are fair, valid and reliable;
 that they have a recertification system; and
 that the system has disciplinary procedures in order to be able to pull certified auditors
for unethical behaviour or incompetence.
While this level of detail may not be necessary for all evaluation tools, it does provide a clear
indication that tests must be validated and the results replicable.
Evaluation of Auditor Competence
R074 Internal Version 1
5 of 26
September, 2007
The IFAC paper on competence states that “Reliability depends on whether different assessors
can reproduce the test results in new tests. Although reliability should be a feature of
assessment of competence, consistency is often a problem. Assessors may interpret the
definition of competence provided in the official standard differently and judge the sufficiency
and acceptability of the evidence pertaining to the competency differently.” (IFAC, 2003: p 28)
A recommended strategy to improve the validity and reliability of competence evaluation as a
whole is to increase the number and type of evaluation tools, thus strengthening the likelihood
of discerning an accurate picture of auditor competence. “The key is for all parties to try to
understand how practical considerations affect the validity, reliability and relevance of
assessment outcomes and to use a variety of assessment methods appropriate to the type of
competence being assessed.” (IFAC, 2003: p27)
5. Evaluation Tools
The basic range of auditor evaluation tools is presented in ISO 19011. In an ideal world, the
evaluation of auditors would combine many of these approaches as they each provide evidence
of a different aspect of an auditor’s competence. In combination, these evaluation techniques
would provide a good overall assessment of auditor competence. However, it is more likely that
a certification or accreditation body will emphasize or employ a sampling of these methods. It is
important to keep in mind that the range of tools employed needs to provide a comprehensive
overall picture.
Evaluation method
Objectives
Examples
Review of records
To verify the background of the auditor
Analysis of records of education,
training, employment and audit
experience
Positive and negative
feedback
To provide information about how the
performance of the auditor is perceived
Surveys, questionnaires, personal
references, testimonials, complaints,
performance evaluation, peer review
Interview
To evaluate personal attributes and
communication skills, to verify information
and test knowledge and to acquire
additional information
Face-to-face and telephone interviews
Observation
To evaluate personal attributes and the
ability to apply knowledge and skills
Role playing, witnessed audits, on- thejob performance
Testing
To evaluate personal attributes and
knowledge and skills and their application
Oral and written exams, psychometric
testing
Post-audit review
To provide information where direct
observation may not be possible or
appropriate
Review of the audit report and
discussion with the audit client, auditee,
colleagues and with the auditor
ISO 19011 Table 2 — Evaluation methods
The IFAC paper on auditor competence also provides a good basic list of auditor assessment
methods:
“There are several established means of assessing competences and capabilities, some of
which are more appropriate for some types of competences than others, as shown below.
Evaluation of Auditor Competence
R074 Internal Version 1
6 of 26
September, 2007
Assessment Methods
Evidence from prior learning
Multiple choice tests
Written tests
Oral tests
Project reports
Assignments
Log books, portfolios
Self-assessment
Simulations
Direct observation of work activities
Indirect observation
Supervisor assessment/ratings
Knowledge
X
X
X
X
X
X
X
Skills
Attitudes
X
X
X
X
X
X
X
X
X
X
X
X
X
(Derived from work done by Gonczi A., et al. “Establishing competency-based standards in
the professions,” and “The development of competency-based assessment strategies for the
professions.” Research Papers Nos. 1 and 8. Canberra, Australian National Office of
Overseas Skills Recognition, Department of Employment, Education and Training,
December and June 1993; also Eraut, M., “Developing professional knowledge and
competence.” London: The Falmer Press, 1994).” (IFAC, 2003, p27)
Another way to look at these lists of potential evaluation tools is by the stage at which they
would be applied in the auditor hiring and review process. The steps in auditor engagement
(and the respective tools at each stage) may include the following:
Initial Contact
 analysis of records (education, training, experience)
 analysis of audit experience
 self-assessment
 interviews (face-to-face and telephone)
 personal references and testimonials
Training and Evaluation
 oral exams
 written exams
 role playing and simulations
 personality testing
 psychological analysis
Audit and Post-audit Review
 witness audits
 performance evaluation
 peer / supervisor review
 review of audit reports
 complaints
 discussion with audit client
Evaluation of Auditor Competence
R074 Internal Version 1
7 of 26
September, 2007
5.1 Initial Contact
Traditional auditor assessment has focused on qualifying auditors based on their education and
work experience. Qualifications for ISO 9001 auditor and lead auditor status still often require
that individuals have a minimum educational background and years of experience or number of
audits completed. Records of education, training and experience serve as a good, basic
minimum determinant of eligibility of new auditors. If a potential auditor does not have a
combination of the basic knowledge and experience then it may not be worth investing the
resources required to train that individual. However, the flipside is that significant experience
does not necessarily make for a good auditor.
5.1.1 Self-Evaluation
The self-evaluation process is an opportunity for prospective auditors to assess their own
strengths and weaknesses in relation to the required competencies. Self-evaluation often takes
the form of a questionnaire that can either be for the private benefit of the individual or can be
shared with the evaluator / conformity assessment body during the initial contact phase of the
program.
The most important consideration in developing a self-evaluation tool is that it should focus on
an assessment against the defined skills, knowledge and attributes required of the auditing
position. Auditors should be provided with a scale against which to evaluate their competence.
One useful option would be Bloom’s Taxonomy (included in Annex 2) which categorizes the
levels of proficiency or cognition. If the required knowledge, skills and attributes are also
classified according to this taxonomy, then the self-assessment can serve as a basis for
identifying relative strengths and shortcomings. Bloom’s Taxonomy includes, from least
complex to most complex, the ability to:
 recognise;
 understand;
 apply;
 analyse;
 synthesise; and
 judge.
A self-assessment form can also be used as a professional development tool by existing
auditors. This tool can be used by the auditor to identify areas of potential improvement and
can also be circulated to managers, colleagues and clients to get a well-rounded picture of how
others assess an auditor’s competencies.
The American Institute of Certified Public Accountants (AICPA) has an online Competency SelfAssessment Tool (CAT) for CPAs that can be purchased relatively inexpensively for US$49
(free for AICPA members). www.cpa2biz.com/CPEConferences/CAT.htm The tool provides a
useful model for adaptation but can also usefully apply as is for assessments of basic auditing
competencies. (excerpt in Annex 3)
The AICPA Competency Assessment Tool contains many models and position profiles,
including junior and senior auditors for different size firms. Each model includes a description of
basic, intermediate and advanced levels of proficiency for each of 36 competencies, grouped
under the following four categories:
Evaluation of Auditor Competence
R074 Internal Version 1
8 of 26
September, 2007
Personal Attributes
The characteristics, which enable the professional to attract others to well-reasoned and
logical points of view, to communicate effectively and to relate to others (e.g. integrity and
ethics; professionalism; etc.).
Leadership Qualities
The skills that allow the professional to assume a position of influence by assembling and
leveraging a variety of resources that address problems and opportunities throughout the
organization (e.g. negotiating; teamwork; persuading; etc.).
Broad Business Perspective
The body of knowledge that encompasses an understanding of the organization, its industry,
and management accounting practices and their applicability to the organization (e.g.
organisational systems and processes; relationship management; etc.).
Functional Specialty
The traditional technical skills that CPAs possess and which form the basis for their unique
ability to understand an organization from a perspective that others cannot (e.g.
performance standards; managing audit risk; etc.).
5.2 Training and Evaluation
The evaluation of auditor competence by ISEAL members takes place predominantly in relation
to training programs in which those auditors participate. Many ISEAL members have mandatory
training requirements for their auditors. These can be delivered either by the organisation itself
or can take the form of generic auditor training programs such as ISO 9000 Lead Auditor
training. A significant component of a training program is the evaluation techniques that take
place within it. These range from formal oral or written examinations to role playing and
simulations.
Broadly speaking, the competencies that are most often acquired and tested through training
programs fall into two categories: generic auditing knowledge and skills, and technical systems
knowledge related to the product or process being certified. ISEAL members are more likely to
focus their training on the latter system-specific technical knowledge and understanding of how
their standard is applied. The AAPG Guidance provides some advice on what is required for
these two categories of training.
“Generic knowledge and skills
As far as audit principles, procedures and techniques are concerned, the relevant
knowledge may result from successful completion of formal training courses and /or a CB's
specific training programme (e.g. skills may be developed from role play or from audits
carried out under guidance or supervision). There should be evidence that the CB's auditors
have acquired knowledge of these competence components effectively, and that their
performance has been demonstrated, tested and accepted, rather than just depending on
records of completion or mere attendance on a training course.
“Processes and products
Evaluation of Auditor Competence
R074 Internal Version 1
9 of 26
September, 2007
Competence in processes and products is the most difficult set of knowledge and skills for
auditors to acquire (and the most troublesome for CBs to apply). Experience has shown
that outdated or time expired workplace experience is often of limited value (and especially
claims of having gained product and process knowledge through operating as a consultant;
such claims need to be treated carefully).” (AAPG, 2005)
5.2.1 Training Courses
The development of a training course curriculum should happen in parallel with the development
of the evaluation tools for that training course. “If examinations form part of the assessment
process, a curriculum stated in terms of learning outcomes will be needed.” (IFAC, 2003: p30)
Learning outcomes are what an auditor is required to know subsequent to the training. They
should mirror the previously defined auditor competencies. Learning outcomes are well suited
to assessment using objective testing and simulations.
While the traditional end of course examination, if properly constructed (valid and reliable), will
yield an understanding of auditors’ knowledge of the subject areas, it is largely the interactions
through classroom activities and mock audit exercises that provide the complementary insights
into auditor skills and personal attributes.
“One of the strengths of written examinations is that they are well suited to testing
intellectual skills such as knowledge and understanding. Generally, skills in practical
applications are difficult to assess realistically in higher education, and this is a weakness of
traditional examinations. Nor are written examinations free of technical problems such as
consistency of marking and subjective judgments as to standards.” (IFAC, 2003: pg25)
In understanding that the evaluation of auditors is so much more than reviewing their test
performances, evaluation is clearly less effective if delivery of the training courses is outsourced
to a third party, as in the case of the delivery of ISO 9000 Lead Auditor training.
In terms of good practice, the International Personnel Certification (IPC) Association has defined
a number of criteria for the structure of an auditor training course in Quality Management
Systems:
“5.2.2 The training course shall include both knowledge-based sessions (to facilitate
understanding of concepts) and skill-based sessions (application of knowledge and skills in
practical activities) and each student shall be subjected to realistic quality system audit
practices and conditions.
5.2.3 Knowledge-based sessions may be instructor led, but shall allow for some interaction
with students, enabling instructors to test learning and students to clarify their
understanding, as required.
5.2.4 Skills-based sessions may be supported by instructor input to address the relevant
requirements and techniques such as for managing meetings and interviews.
5.2.5 Methods for validating student achievement of the learning objectives and for providing
timely feedback shall be included in the course.
Evaluation of Auditor Competence
R074 Internal Version 1
10 of 26
September, 2007
5.2.6 Each student shall be required to participate in practical skills-based activities:
workshops, case studies, auditor role playing or actual quality system audit situations. At
least 50% of course time shall be used for such activities.” (IPC, 2002)
5.2.2 Assessing Personal Attributes
Personal attributes, such as perceptiveness or decisiveness, are probably the most difficult
aspects to evaluate, and yet are often what differentiates a great auditor from a good one. The
methods suggested to assess an auditor’s personal attributes include:
 self-assessment;
 simple observations;
 personality and aptitude exams;
 psychological analysis.
A number of new, competence-based evaluation methodologies build on the requirements laid
out in ISO 17024 for Personnel Certification. Personnel certification bodies, such as IRCA in
the UK and RABQSA in the US focus on testing the personal attributes of individuals through
personality tests that are tailored to the requirements of a specific industry (such as the auditing
industry). As an example, IRCA has a new Corporate Auditor qualification program.
“The Corporate Auditor grade differs fundamentally from the Principal, Lead and other
grades in that it is ‘competence-based’ and complies with ISO 17024. This means that to be
certified to the Corporate Auditor grade, auditors must demonstrate they are able to apply
the skills knowledge and personal attributes at a one-day Assessment Centre which
employs a combination of examination methods.
 An occupational personality questionnaire
 A written in-tray task
 A collaborative group task
 A competence-based interview”
(www.irca.org/certification/certification_13_3.html)
The following information is provided about the personality questionnaire:
“This is a multiple choice questionnaire, usually completed ‘online’, the output from which
will help us gauge the extent to which an individual’s preferred style or way of behaving (i.e.
attributes) is likely to impact on their potential performance against the auditor
competencies. It is important to understand that this is not a pass / fail ‘test’. Instead the
summary report produced from the questionnaire will guide the examiners during the CA
Assessment Centre.”
While personality tests may seem like an attractive way of quantifying whether an auditor has
the desired personality traits, there are some distinct shortcomings, not least of which is that
there is very little hard evidence of the validity of the attributes that have been chosen as
desirable. The RABQSA Personal Attribute Assessment System (PAAS Master) test consists of
85 questions that evaluate the following 11 attributes that it claims will separate more effective
and efficient auditors from less effective and efficient ones:
 proactive and organized;
 systematic;
 logical;
 decisive;
 observant;
Evaluation of Auditor Competence
R074 Internal Version 1
11 of 26
September, 2007






diplomatic;
flexible;
process preference;
people sensitive;
adaptable and resourceful; and
confident.
The first thing to note is that these attributes are not the same as those listed in ISO 19011,
though they are similar. As noted in an article by Thea Dunmire of ENLAR Compliance
Services in the US, “It is not clear that there is any overall consensus, at least in the United
States, on the personality traits required for management system auditors. The attributes listed
in ISO 19011 were not selected because of any scientific evaluation – they were selected
because they sounded right.”
All of the attributes listed in ISO 19011, PAAS Master and other personality attribute lists for
auditors sound like they should be a useful basis on which to assess auditor competence. And
yet there is little evidence whether this is, in fact, the case and, if so, which of these attributes is
most important to consider. Ms. Dunmire continues:
“For a scientific evaluation of personal attributes, we can look for guidance to the studies
that have been conducted of financial auditors. These studies have been going on for many
years and the results have been reported in several articles published in professional
accounting journals. One recent study evaluated 32 different auditor attributes as a follow-up
to a study conducted originally in the early 1990’s (An Investigation of the Attributes of Top
Industry Audit Specialists, M.J. Abdolmohammadi, D.G. Searfoss, J. Shanteau, Behavioral
Research in Accounting vol. 16, no. 1 1-17, 2004). Of the attributes identified, only one,
current knowledge, was ranked as “extremely important.”
“Ten more attributes were ranked as “very important” including problem solver, experience,
perceptive, communicates expertise, self confidence, adaptability, intelligence, knows what
is relevant, assumes responsibility and quick thinker.
“Although all of the personal attributes identified sound similar, there is very little direct
correlation between the auditor attributes identified in these studies and either those listed in
ISO 19011 or those tested for by PAAS Master®. In addition, it is difficult to determine what
difference, if any, there may be between traits such as versatile and adaptable. Even more
difficult is the resolution of personal attributes that seem to conflict - for example, process
preference and adaptable.” (ENLAR, 2005)
By bringing together these various lists of desirable personal attributes, it is possible to get a
sense of the overall picture of what attributes a good auditor will possess (see Annex 1).
However, the overall message is that these are intuited guidelines, based on the attributes
commonly seen or valued in good auditors. Investing in expensive personality tests may not be
the most effective use of funds for auditor evaluation. Simple observation of auditors, keeping
in mind the proposed desirable attributes may also be effective.
The steps to be taken by a conformity assessment body are to define the list of personal
attributes that they feel are most relevant, and to determine the level of formality that is needed
in the evaluation of these attributes. Informal observation during training courses, witness
audits or performance evaluations will yield a subjective picture of an auditor’s skills and
Evaluation of Auditor Competence
R074 Internal Version 1
12 of 26
September, 2007
attributes. If a more formal (and potentially less subjective) assessment is desired, then looking
into personnel certification models or psychological testing may be appropriate.
5.3 Audit and Post-audit Review
Evaluation of auditors is an ongoing process throughout the lifetime of the relationship between
an auditor and conformity assessment body. There are a range of regular materials, such as
audit reports, performance evaluations and complaints forms that form the basis for continuing
evaluation. These are supplemented by the results of any continuing professional development
and the skills, knowledge and attributes acquired therein.
5.3.1 Workplace Assessments
While the assessment of acquired knowledge lends itself to examinations, assessing an
auditor’s skills is more easily accomplished through practical exercises. IAF Guidance on ISO
19011, clause G7.3.3 states that, “The demonstration of skill should be examined on-site under
auditing conditions, or a simulation of on-site audit conditions that has been validated as
effective.”
One of the most effective forms of evaluation is to be able to witness an auditor in the course of
her or his work. This can take a variety of forms, ranging in the level of formality. The least
intrusive witnessing is when audit team peers or the team leader provide written or oral
feedback after the audit about an auditor’s performance. Auditor feedback forms are applicable
here and a sample is provided in Annex 4 (RABQSA).
Requiring slightly more intervention, it is also useful to get a perspective from the company or
enterprise being audited about the performance of the auditor and audit team. This requires
that the company is willing to complete an evaluation form. A sample is provided in Annex 5
(NADCAP).
A potentially more invasive, yet valuable form of workplace assessment takes place through
witness audits, in which the auditor is witnessed carrying out an assessment. It is often the
case that before an auditor can become qualified, he or she is witnessed participating in two or
three audits. However, the major shortcoming of witness audits or of peer or client evaluation is
that the resulting assessments are subjective. In some instances, such as where witness audits
are considered culturally inappropriate or intrusive, the conformity assessment body may opt for
workplace simulation exercises, which may, themselves, have some limitations.
“The problem with the performance outcomes approach is that practical applications are
usually well defined but are difficult to test satisfactorily. For instance, it is difficult to ensure
comparability of workplace assessments over a wide number and variety of employers.
“To compensate for problems such as lack of expertise, inconsistency of assessment and
the complexity of contributing factors when assessing work outcomes, simulations of
workplace environments at places of learning may offer acceptable alternatives. On the
other hand, certain capabilities, such as group skills, might be better tested in the work
environment as they can present some challenges in academia, not least because of the
artificial environment in which they are being tested.” (IFAC, 2003: pg25)
Evaluation of Auditor Competence
R074 Internal Version 1
13 of 26
September, 2007
5.3.2 Annual Review and Professional Development
Once an auditor has qualified and is working for a conformity assessment body, there is usually
a requirement for continuous professional development. This often takes the form of training
courses or seminars, as well as completing a minimum number of audits annually. In terms of
evaluating auditor competence, the skills and attributes that were assessed in the initial
evaluation process are likely to be retained by an auditor. There is some need to maintain a
level of comfort with basic auditing skills, which is the reason for requiring regular participation in
audits. However, the primary area in which continuing profession development is important is
for auditors to stay on top of technical developments in their auditing field.
Much like the acquisition of initial competencies, the focus in evaluating professional
development should be on demonstration of skills, knowledge and attributes. IAF Guidance to
ISO 19011, states in clause G.7.5.1.1. that, “Continual professional development activities
should relate directly to the maintenance and improvement of competence. The recognition of
professional development should be based on evidence of attained competence.” Many of the
tools already described are appropriate for further assessment of auditor competence, such as
self-assessment questionnaires and witness audits.
Self-assessment questionnaires can also be used as the basis for an auditor to prioritise and
attain additional skills and knowledge that relate to the defined competencies. When taken as a
joint exercise with the conformity assessment body as part of the annual review of an auditor’s
performance, the questionnaires can serve as the framework for setting out the plan for
continuing professional development.
Other tools that can inform the personnel evaluation include any complaints forms pertaining to
audits on which the auditor participated, audit team assessments by the clients being audited
and peer and lead auditor evaluations.
6. Conclusion
The evaluation of auditor competence is part of an overall process of ensuring that auditors are
qualified for the work they are assigned. Evaluation assumes that accurate competencies are
first defined, for the knowledge, skills and attributes required of an auditor. Training strengthens
the competence of auditors both prior to engagement and in building up areas of weakness
identified through regular evaluation.
The evaluation of auditor competence is, therefore, a continuous process that should involve the
application of a variety of evaluation tools. Good practice in evaluating auditors ensures that the
auditor’s attributes, skills and knowledge are all demonstrated in an active evaluation process,
whether this is simulated during training programs, assessed through various testing methods,
or comes as the result of review of an auditor in action.
Much of the basic information about the tools used for evaluation is adequately noted in ISO
19011. While there are a relatively well-defined (though not definitive) set of management
system auditor competencies, these can be evaluated to varying levels of thoroughness. It is up
to the conformity assessment body to determine what level of detail is required to ensure that
their auditors are competent to carry out their assigned work, as well as to identify weaknesses
that can be strengthened.
Evaluation of Auditor Competence
R074 Internal Version 1
14 of 26
September, 2007
Annex 1: Attributes, Skills and Knowledge
Personal Attributes
ISO 19011
RABQSA PAAS Master®
Financial auditor study
(Quoted in ENLAR)
Ethical
Proactive and organized
Current knowledge
Open-minded
Systematic
Problem solver
Diplomatic
Logical
Experience
Observant
Decisive
Perceptive
Perceptive
Observant
Communicates expertise
Versatile
Diplomatic
Self-confidence
Tenacious
Flexible
Adaptability
Decisive
Process preference
Intelligence
Self-reliant
People sensitive
Knows what is relevant
Professional (added from
17021 Part II draft)
Adaptable and resourceful
Assumes responsibility
Moral courage (added from
17021 Part II draft)
Confident
Quick thinker
ISO 19011 - 7.2 Personal attributes (in more detail)
a) ethical, i.e. fair, truthful, sincere, honest and discreet;
b) open-minded, i.e. willing to consider alternative ideas or points of view;
c) diplomatic, i.e. tactful in dealing with people;
d) observant, i.e. actively aware of physical surroundings and activities;
e) perceptive, i.e. instinctively aware of and able to understand situations;
f)
versatile, i.e. adjusts readily to different situations;
g) tenacious, i.e. persistent, focused on achieving objectives;
h) decisive, i.e. reaches timely conclusions based on logical reasoning and analysis; and
i)
self-reliant, i.e. acts and functions independently while interacting effectively with others.
Evaluation of Auditor Competence
R074 Internal Version 1
15 of 26
September, 2007
Skills
ISO 19011, Clause 7.3.1
plan and organize the work
effectively
conduct the audit within the
agreed time schedule
prioritize and focus on matters
of significance
collect information through
effective interviewing
understand sampling
techniques for auditing
verify the accuracy of
collected information
confirm the sufficiency and
appropriateness of audit
evidence to support audit
findings and conclusions
assess those factors that can
affect the reliability of the audit
findings and conclusions
use work documents to record
audit activities
prepare audit reports
ASQ Body of Knowledge
(Auditor Competencies)
ISO 17021 Part 2 (Draft)
Resource management
reading
Conflict resolution
writing
Communication techniques
listening
Interviewing techniques
oral presentation
Team membership,
leadership and facilitation
organization
Presentation techniques
planning
Verification and validation
note taking
intuitive sampling
proactive approach
managing conflict
maintain the confidentiality
and security of information
communicate effectively
time management
command presence
using language appropriate to
all levels
staying focused
team player
effective use of translators or
experts
Team Leader - Planning
Team Leader - facilitation
Team Leader - public
speaking
Evaluation of Auditor Competence
R074 Internal Version 1
16 of 26
September, 2007
ASQ Body of Knowledge
III. AUDITOR COMPETENCIES (in more detail)
A. Auditor characteristics
Identify characteristics that make auditors effective, such as interpersonal skills, problem-solving
skills, close attention to detail, cultural sensitivity, ability to work independently and in a group or
on a team. (Apply - refers to Bloom’s Taxonomy)
B. Resource management
Identify and apply techniques for scheduling people, events, logistics, and audit related
activities. (Apply)
C. Conflict resolution
Identify typical conflict situations (disagreements, auditee delaying tactics, and interruptions)
and determine the techniques for resolving them, such as negotiation and cool-down periods.
(Analyze)
D. Communication techniques
Select and use written and oral communication techniques in various applications, such as
technical reports, active listening, empathy, and paraphrasing. (Analyze)
E. Interviewing techniques
Define and apply appropriate interviewing techniques (e.g., when to use open-ended and closed
question types, determining the significance of pauses and their length, and when and how to
prompt a response), based on various factors, such as when supervisors are present, when
interviewing a group of workers, and when using a translator. (Apply)
F. Team membership, leadership, and facilitation
Define and use various techniques to support team-building efforts and to help maintain group
focus, both as a participant and as a team leader. Recognize and apply the classic stages of
team development (forming, storming, norming, and performing). Identify various team member
roles and apply coaching, guidance, and other facilitation techniques necessary to effective
team functioning. (Analyze)
G. Presentation techniques
Define and apply various tools and techniques such as graphs, charts, diagrams, and
multimedia aids for written and oral presentations made at opening, closing, and other
meetings. (Apply)
H. Verification and validation
Define, distinguish between, and apply various methods of verifying and validating processes.
(Analyze)
Evaluation of Auditor Competence
R074 Internal Version 1
17 of 26
September, 2007
Knowledge
ISO 19011, Clause 7.3.1: Generic knowledge and skills of quality management system
and environmental management system auditors
b) Management system and reference documents: to enable the auditor to comprehend the
scope of the audit and apply audit criteria. Knowledge and skills in this area should cover
 the application of management systems to different organizations,
 interaction between the components of the management system,
 quality or environmental management system standards, applicable procedures or other
management system documents used as audit criteria,
 recognizing differences between and priority of the reference documents,
 application of the reference documents to different audit situations, and
 information systems and technology for, authorization, security, distribution and control
of documents, data and records.
c) Organizational situations: to enable the auditor to comprehend the organization's operational
context. Knowledge and skills in this area should cover
 organizational size, structure, functions and relationships,
 general business processes and related terminology, and
 cultural and social customs of the auditee.
d) Applicable laws, regulations and other requirements relevant to the discipline: to enable the
auditor to work within, and be aware of, the requirements that apply to the organization being
audited. Knowledge and skills in this area should cover
 local, regional and national codes, laws and regulations,
 contracts and agreements,
 international treaties and conventions, and
 other requirements to which the organization subscribes.
7.3.3 Specific knowledge and skills of quality management system auditors
Quality management system auditors should have knowledge and skills in the following areas.
a) Quality-related methods and techniques: to enable the auditor to examine quality
management systems and to generate appropriate audit findings and conclusions. Knowledge
and skills in this area should cover
 quality terminology,
 quality management principles and their application, and
 quality management tools and their application (for example statistical process control,
failure mode and effect analysis, etc.).
b) Processes and products, including services: to enable the auditor to comprehend the
technological context in which the audit is being conducted. Knowledge and skills in this area
should cover
 sector-specific terminology,
 technical characteristics of processes and products, including services, and
 sector-specific processes and practices.
Evaluation of Auditor Competence
R074 Internal Version 1
18 of 26
September, 2007
Annex 2: Bloom’s Taxonomy
The level of proficiency to demonstrate for the knowledge and skills is described in this standard
is based on the "Levels of Cognition" (from Bloom's Taxonomy) and are presented below in rank
order, from least complex to most complex:
a) recognize:
able to remember, recall or recognize situations based on terminology, definitions, facts,
ideas, materials, patterns, sequences, methodologies, or principles
b) understand
able to understand documentation, information and data and situations (e.g., descriptions,
ideas, procedures, methods, formulas, principles, theories, communications, reports, tables,
diagrams, directions, regulations)
c) apply
able to apply information and data (e.g., descriptions, ideas, procedures, methods,
formulas, principles, theories, communications, reports, tables, diagrams, directions,
regulations)
d) analyze
able to break down information into its constituent parts and recognize the parts’
relationship to one another and how they are organized; identify sublevel factors or salient
data from a complex scenario
e) synthesis
able to put parts or elements together in such a way as to show a pattern or structure not
clearly there before; identify which data or information from a complex set is appropriate to
examine further or from which supported conclusions can be drawn
f)
judge
able to make well-reasoned decisions and conclusions
Evaluation of Auditor Competence
R074 Internal Version 1
19 of 26
September, 2007
Annex 3: Self Assessment Example
Personal Attributes
Integrity and Ethics
COMPETENCY DESCRIPTION:
Is honest and ethical when addressing all issues and interactions with others.
Basic
Intermediate
Advanced
Demonstrates
understanding of and
appreciates professional
ethics, and applies those
ethics to day-to- day
business activities.
Identifies ethical issues
and discusses them in a
constructive manner.
Sets the tone for ethical
behavior. Will not
tolerate a compromise
of ethics in others;
serves as an example to
others.
Exercises unyielding
integrity; will not
compromise firm values
to achieve short-term
objectives.
Builds trust with key
personnel at all levels
of the firm by
consistently
demonstrating direct,
honest behavior.
Applies judgment and
counsels personnel on
appropriate behavior in
gray (undefined) or
highly judgmental
areas.
Demonstrates personal
integrity by providing
accurate financial data
and straightforward
interpretation of data.
Demonstrates an indepth understanding of
more complex and
judgmental
areas/issues.
Would never
subordinate his or her
responsibility of serving
the public's interest and
honoring the public's
trust to client demands
Evaluation of Auditor Competence
R074 Internal Version 1
20 of 26
September, 2007
Annex 4: Auditor Witnessing Report Example
RABQSA International
P.O. Box 602
Milwaukee, WI 53201-0602
USA
Phone: +1 888 722 2440
Fax: +1 414 765 8661
Website: www.rabqsa.com
RABQSA International
P.O. Box 347
Penrith, BC NSW 2751
AUSTRALIA
Phone: +61 2 4728 4600
Fax: +61 2 4731 6466
Email:info@rabqsa.com
OFFICE USE ONLY
Customer
No:…………….…..
REVIEW OF AUDITOR PERFORMANCE REPORT
Applicant Information
Family Name
Given Name(s)
Customer Number (if applicable)
Date certified in current grade
Note: If transferring from another certification body, list equivalent current certification information.
Applicant’s Signature
Witnessing Auditor’s Information
Witnessing Auditor’s Family Name
Given Name(s)
RABQSA Customer Number
AS9100 auditor grade
Witnessing auditor’s employer
Employer’s Address
City
Telephone (include country code and area code)
State
Country
Fax
(include country code and area code)
ZIP or Postal Code
Email
If the candidate performed acceptably on the audit described in this report, please sign on the line
below. The witnessing auditor’s signature is required for the application to proceed.
I attest to the applicant’s ability to perform audits as an AS9100 Aerospace Experience Auditor (AEA) in
accordance with RABQSA AS9100 auditor certification requirements.
Witnessing auditor’s signature
Evaluation of Auditor Competence
R074 Internal Version 1
21 of 26
September, 2007
The following sections are to be completed by the witnessing auditor. The witnessing auditor should refer to the
RABQSA Criteria for AS9100 Auditor Certification.
Audit Information
1st date of site visit
Number of days:
Applicant’s role in the audit:
leader
Witnessing auditor’s role in the audit:
leader
Type of Audit:
on-site
off-site
Name of facility audited
□
Auditor team member
□
Solo auditor
□
Audit team
□
Witness only
□
Audit team member
□
Audit team
□
Complete
□
Partial
□
Internal
• Clauses of the standard audited by the applicant:
• Was the applicant involved in making a judgment on the effectiveness of the complete AS9100 management system for this
audit? If not, please describe the scope of the applicant’s involvement.
Audit Evaluation
Please complete the following evaluation as it pertains to the applicant’s ability to perform as an AS9100 Aerospace
Experience Auditor. Please note that some of the information does not apply to all applicants, for example not all
auditors will be performing team leader functions.
Criteria
Evaluation of Auditor Performance
(including strengths and/or Opportunities for
Improvement)
1. Personal skills:

Open-minded and mature.

Possess sound judgment, analytical skills and tenacity.

The ability to understand complex operations from a
broad perspective.

Maintain ethical behavior.
Evaluation of Auditor Competence
R074 Internal Version 1
22 of 26
September, 2007
Auditor Evaluation (continued)
Criteria
Evaluation of Auditor Performance
(including strengths and/or Opportunities for
Improvement)
2. Audit skills and techniques:

Obtain and assess objective evidence fairly.

Remain true to the purpose of the audit.

Evaluate the effects of observations and personal
interactions

Remain attentive to the audit process without
deviating due to distractions.

Reach acceptable conclusions based on objective
evidence and remain true to conclusion despite
pressure to change.
3. Communication skills; oral and written.
4. Effective planning, preparation, and briefing.
5. Participation in team meetings (including opening/closing
meeting).
6. Reporting clear and concise.
7. Relationship with team.
8. Audit management capabilities:

Coordinate and control meetings.

Conduct and audit and delegate responsibility.

Manage and audit, keep the audit team within the
scope.

Control conflict.

Make decisions in an intelligent, decisive, and
authoritative manner.

Assess situations and take appropriate actions.

Evaluate performance of audit team members.
9. Performance of document review.
10. Understands the application of each clause of the standard.
Evaluation of Auditor Competence
R074 Internal Version 1
23 of 26
September, 2007
Annex 5: Auditor Evaluation Form Example
NADCAP Post-Audit Questionnaire
(National Aerospace& Defense Contractors Accreditation Program)
Rating System:
5 – Excellent
Please circle the number on each question that represents your opinion:
4 – Good
3 – Average
2 – Below Average
1 – Poor
1. Did the auditor(s) conduct themselves in a positive and professional manner? Yes or No
Comments: _________________________________________________________
___________________________________________________________________ 5
___________________________________________________________________
4 3 2 1
2. How would you judge the auditor’s technical knowledge and proficiency?
Comments: _________________________________________________________
___________________________________________________________________
___________________________________________________________________ 5
4 3 2 1
3. Was the auditor(s) objective explained during your audit?
Comments: _________________________________________________________
___________________________________________________________________
___________________________________________________________________ 5
4 3 2 1
4. Did the auditor(s) have enough time to effectively meet with all personnel involved?
Comments: _________________________________________________________
___________________________________________________________________ 5
___________________________________________________________________
4 3 2 1
5. Did the auditor(s) efficiently use their time at your facility?
Comments: _________________________________________________________
___________________________________________________________________
___________________________________________________________________ 5
4 3 2 1
6. Did the auditor(s) witness all testing or processes being performed?
Comments: _________________________________________________________
___________________________________________________________________
___________________________________________________________________ 5
4 3 2 1
7. Were all nonconformances clearly explained?
Comments: _________________________________________________________
___________________________________________________________________ 5
___________________________________________________________________
4 3 2 1
– Continued on reverse side –
Evaluation of Auditor Competence
R074 Internal Version 1
24 of 26
September, 2007
NADCAP Post Audit Questionnaire
Page 2
-
8. Did you feel that you had adequate time to prepare for the audit?
Comments: _________________________________________________________
___________________________________________________________________
___________________________________________________________________ 5
4 3 2 1
9. What value added process did you realize from this audit?
Comments: _________________________________________________________
___________________________________________________________________
___________________________________________________________________
10. Do you have any constructive input for improving the process?
Comments: _________________________________________________________
___________________________________________________________________
___________________________________________________________________
11. What was your purpose for obtaining NADCAP accreditation?
______ Mandate
______ Improved Quality ______ Other
Comments: _________________________________________________________
___________________________________________________________________
___________________________________________________________________
Please indicate which audit(s) you have completed:
Month / Year of Audit ___________________________
The following information is optional:
Company Name ___________________________________________________________________
Address _________________________________________________________________________
Auditor Name(s) __________________________________________________________________
Thank you for your time in completing this questionnaire. Your comments are important to us.
Evaluation of Auditor Competence
R074 Internal Version 1
25 of 26
September, 2007
Resources
Accreditation Auditing Practices Group (AAPG). 2005. Auditing the Competence of Quality Management
System Certification/Registration Body Auditors and Audit Teams.
isotc.iso.org/livelink/livelink/4300106/AAPGCRBAuditorCompetence.doc?func=doc.Fetch&nodeid=4300106
Admitted Class Evaluation Service (ACES). Validity Handbook. Accessed from
www.collegeboard.com/highered/apr/aces/vhandbook/testvalid.html
American Institute of Certified Public Accountants (AICPA). Competency Self-Assessment Tool.
Subscription available from www.cpa2biz.com/CPEConferences/CAT.htm
American National Standards Institute (ANSI). Dec. 2004. ANSI-PCAC-GI-502 Guidance on Psychometric
Requirements for ANSI Accreditation. Available from
http://ansi.org/conformity_assessment/personnel_certification/apply.aspx?menuid=4
American Society for Quality (ASQ). 2004. Quality Auditor Certification Body of Knowledge.
http://www.asq.org/certification/quality-auditor/bok.html
International Federation of Accountants (IFAC). April, 2003. Towards Competent Professional
Accountants.
http://www.ifac.org/store/Details.tmpl?SID=10512128061312989
Dunmire, Thea. ENLAR Compliance Services, Inc. 2005. Auditor Personality Test not without
Controversy. Available from http://iso14000expert.com/freepubs.html
International Personnel Certification Association (IPC). Jan. 2002. Criteria for QMS Auditor Training
Courses. Accessed from http://www.iatca.com/
ISEAL Alliance. November, 2006. Strengthening Auditor Competence. Available from ISEAL Secretariat.
ISO/IEC 17024:2003 Conformity assessment -- General requirements for bodies operating certification of
persons. Available from the ISO Store
http://www.iso.org/iso/en/prods-services/ISOstore/store.html
ISO 19011:2002 Guidelines for quality and/or environmental management systems auditing.
Available from the ISO Store http://www.iso.org/iso/en/prods-services/ISOstore/store.html
National Aerospace& Defense Contractors Accreditation Program (NADCAP). 2007. NADCAP Post-Audit
Questionnaire. Available from
forums.sae.org/access/dispatch.cgi/x_chem_process_pri_pf/folderFrame/100012/0/def/1582
RABQSA. 2007. PCF-85 - AS9100 Review of Auditor Performance Report
www.rabqsa.com/docs/downloads/PCF85.doc
Evaluation of Auditor Competence
R074 Internal Version 1
26 of 26
September, 2007