Audit Issues
Ian Paterson,
Vice-Chairman, Scottish Registration Board
The SER Audit
I expect that by now most of you will have been audited –
and that experience can be expected to include a wide range
of results and part of my task this afternoon is to allay some
of the myths, legends and misunderstandings surrounding
the audit process
But if I could begin by quoting some words of wisdom, which
in their original context had nothing whatsoever to do with
audits but none the less are quite appropriate to this
afternoon’s topic:
“Nothing in Life is to be feared.
It is only to be understood”.
Marie Curie
Topics Covered
• Why do we audit ?
• What is an audit ?
• What does the audit cover ?
• What is the audit process ?
• Recent changes to the audit procedures
and their implementation.
Why do We Audit ?
• It is a requirement of SER’s appointment by the
Government that the activities of all certifiers are
monitored to confirm compliance with the
requirements of the Statutory Regulations and the
agreed Scheme.
• The audit is the principal mechanism by which the
government / public can have confidence in the
performance/practice of certifiers without
independent scrutiny of each of their projects.
• Certifiers are not alone in being audited under the
legislation as BSD regularly audit the performance
of SER and the Verifiers.
Certifiers Appointment
• Before looking at the audit process it is worth
reminding ourselves of the basis of the appointment of
certifiers.
• Certifier status is awarded to professional individuals
who have demonstrated the requisite experience in
the design of (some types of) building structures –
generally “on trust” and on the basis of statements
made in their application form.
• They are expected to understand the relevant
regulatory framework, the requirements of scheme
membership, and to carry out their work in a diligent,
responsible and professional manner (certifying within
their competence).
Certifier’s Appointment
• There are no categories of certifier’s but it is
incumbent on all certifiers to ask themselves if
their knowledge is sufficient to enable them to
certify each project before doing so. We
experience more problems with certifiers
“trading down” to small domestic projects outwith
their experience as we do with those “trading up”
to larger more complex projects.
• The audit is generally the first real test of their
understanding of, and compliance with, these
requirements.
What is an Audit ?
Wikipedia:
• Audits are performed to ascertain the validity and
reliability of information; also to provide an assessment
of a system's internal control. The goal of an audit is to
express an opinion of the person / organization /
system (etc.) in question, under evaluation based on
work done on a test basis.
• An audit must adhere to generally accepted standards
established by governing bodies. These standards
assure third parties or external users that they can rely
upon the auditor's opinion on the fairness of financial
statements, or other subjects on which the auditor
expresses an opinion.
What is an SER Audit ?
At a subjective level that depends very
much on your point of view.
What is an SER Audit ?
We recognise that whilst larger practices with QA
accreditation are used to being scrutinised at audit for
smaller practices / sole practitioners this is unlikely to
be the case.
So let us be clear that the purpose of the audit is to
seek evidence of compliance in the form of an “audit
trail” of contemporaneous actions and this is what the
certifiers will be asked to demonstrate at audit.
What is an SER Audit ?
• Our experience suggests that your opinion of the
audit will be formed more by your mindset going into
it than on the process or outcome of the audit itself.
• But to paraphrase Madam Curie The audit should be
understood – then it should hold no fears.
• There should be no unknowns as the audit process,
and the scheme requirements are well documented
within the various publications produced by SER.
• So lets start by trying to come up with a definition of
the audit ……..
What is an SER Audit ?
• An adversarial meeting where the auditors try their
hardest to catch the auditee out on trivial issues.
– NO. What we seek, and hope for, from the audit
is a demonstration of the certifier’s compliance.
The best outcome for SER is to be able to show
BSD that the certification scheme is working to a
high standard – not to have high levels of non
conformances or suspensions which will call the
effectiveness or reliability of the scheme into
question.
What is an SER Audit ?
• A necessary, (or unnecessary), evil ?.
Let’s try another definition which I think is more
appropriate:
• An objective review of the activities of the
certifiers/bodies against a pre-published set of
requirements/criteria to confirm compliance with
the scheme criteria.
Or, to put a more positive spin on it …..
What is an SER Audit ?
• An opportunity for certifiers to demonstrate
that their professional practice is of such a
high standard it does not require to be
subject to external checking/scrutiny.
• The principal face to face contact between
certifier’s and SER.
• An opportunity to discuss the processes
being used, share best practice and suggest
improvements within the context of the work
undertaken.
What is an SER Audit ?
• But perhaps the definition which best puts the
audit into the appropriate context, and
perhaps makes it more palatable is …
• The alternative to verifier /
external engineering scrutiny of
the design on EVERY project
you undertake.
Objectivity – Consistency of Audit
• To a certain extent the audit process has
evolved over the 3-4 years during which the
audits have been carried out – largely in
response to the experiences and situations
encountered.
• SER make every effort to ensure the outcome of
the audits are as fair and consistent as possible.
• Audit teams comprise two auditors - so the
outcome/report should be a balanced view and
not based on the opinions of a single individual.
Objectivity – Consistency of Audit
• Audits are carried out and scored against a fixed set
of criteria (known both to the auditors and auditees
in advance).
• The auditors are briefed to carry out the audit and
scoring “to the book” – not to “take a view” or make
a judgement which would skew the scoring to affect
the outcome.
• EVERY audit is discussed in detail by SRB and the
auditors opinions are frequently challenged –
sometimes resulting in a change of score.
• That discussion is the stage at which any other
influences, extenuating circumstances, and the
auditees response are taken into consideration.
What Does the Audit Cover ?
• A number of projects selected to cover the range
and scope of certification being undertaken in terms
of project size, complexity, type and date of the
certificates generated.
• A review of the available records for the project
– we can only audit on the information presented to
us if the audit process is to have any integrity. It is
the auditee’s responsibility to provide access to
contemporaneous records demonstrating
compliance.
• An audit of the process and level of scrutiny
undertaken before signing the certificate.
What does the audit cover ?
• The audit is very much a review of process:
• Has the certifier identified all of the building elements
which the certificate will be deemed to have covered ?
– does he know what he is certifying ?.
• You must remember we are not certifying “the
structure” but are certifying “the building” in respect of
the structural requirements of the Regulations.
• So it is not just your won drawings you are certifying
but all of the drawings on which the Warrant is
granted.
• Failure to recognise this often gives rise to issues at
audit.
What does the audit cover ?
• Has the certifier produced, or obtained, the
necessary level of information for warrant (Blue
Book) ?
• Have they specified CDE’s in sufficient detail
(TB1) ?
• Have they obtained/scrutinised the information
accompanying the warrant application ?.
• Has the design been checked to the required
standard ?
• Was the design completed before the certificate
was signed ?
One Size Fits All ?
• We have frequently been asked why we can’t have
a simpler audit process for smaller projects.
• We did initially try a smaller projects check list on
the early audits – but it proved inconsistent and
ineffective.
• In reality the use of the same check list on both
small and large projects invariably means that there
is more to scrutinise in larger projects because of
their content and complexity.
• There are no “simpler regulations” for smaller
projects – every building is expected to comply with
the requirements of the regulations.
What is Required for the Audit ?
• The certifier is required to demonstrate compliance with
the scheme requirements through reference to
contemporaneous project records (audit trail).
• All information used or consulted in the certification
process.
• Certification plan.
• Drawings submitted with the warrant application
(architect’s and engineering).
• Calculations (own or suppliers).
• SI reports, survey reports, specifications etc.
• Hard copy, or accessible electronic copy – onus is on
certifier to provide access to the relevant information.
What does the Audit NOT cover ?
• We DO NOT carry out a detailed engineering or
arithmetic check of the drawings or calculations
– we look for evidence that that has been carried
out by others (as required under the scheme
guidance) - though on some occasions
deficiencies in the design may become apparent
during review of the project documentation.
• We DO NOT look into commercial arrangements
and appointments (other than Management of
Risk and Protection of Certifiers – GN6).
Audit Process
• Notification and arrangement of audit, including
projects to be audited.
• Opening meeting to explain audit procedure.
• Audit carried out by 2 experienced engineers,
able to relate their own experience to “real life”
certification situations, against a pre-published
set of check lists.
• Closing meeting to discuss findings and advise
on remainder of process.
• Preparation of audit report
Audit Process
• Certifier (or Certification Coordinator) responds to
audit report – opportunity to dispute or ask for
clarification if necessary but we look for recognition
of issues involved and statement as to how these
will be avoided or dealt with on ongoing
certification activities.
BUT
• There is no point in disputing the facts of the audit
at this stage (e.g. availability of info) as the issues
will all have been discussed as part of the close out
meeting.
Audit Process
• Full report, including response, discussed in
detail at SRB (again for consistency) at which
stage circumstances of audit will be taken into
consideration before arriving at a final
recommendation.
• SER Board consider SRB recommendations and
make final determination of audit outcome.
• So it is quite a cumbersome process but has
been made deliberately so to maximise the
standardisation and protect the certifiers.
Jekyll and Hyde ?
• The role of the certifier is independent of, and
different to, that of the engineer – even if
undertaken by the same individual – and that
requires a change in mindset.
• As and engineer your scope / remit can be set or
amended through you appointment.
• As a certifier you are responsible for confirming
the structural adequacy/performance of the entire
building – not just “the structure”.
• When acting as a certifier YOU are Building
Control and you must make adequate enquiry to
determine/demonstrate compliance of elements
you have not designed.
Common Issues from Audit
• Inadequate scoping of certificate coverage.
• Poor management of process/project to reflect
certification requirements.
• Poor records.
• Do you know what you are certifying ? – and
have you a record of this ?.
• Inadequate review – or evidence of review (GN
3 certification options 3 and 4).
• Inadequate detail on warrant plans.
• Further details in the audit feedback reports
available on the website.
Audit Procedures
• Many certifiers don’t recognise
the full significance of this
document.
• Only read it when they know
they are going to be audited
(if even then).
• Document sets in detail what
we are looking for/at in an
audit.
• Excellent source of guidance
on what you should be
producing or looking for before
signing a certificate.
Revised Audit Procedures
• Revised Audit procedures published on 24/10/11.
• Intended to reflect/ address experience over first 3
or so years of auditing, in particular where some
aspects of the scheme requirements were not being
interpreted correctly by certifiers.
• Very few material “changes” in the requirements.
• More clarification or further guidance.
• Brings together and formalises practice already
discussed at SRB and implemented during audits
(how situations should be recorded/scored).
Revised Audit Procedures
• As only minor or process changes revised
procedures will be implemented immediately on
all future audits.
• Auditors will consider any issues where
retrospective application would impact on audit
outcome and these will be discussed at SRB
before a final decision is made.
Principal Changes - Timescales
• Affect auditors and auditees.
• Audits must be undertaken within 8 weeks of
receipt of notification.
• Projects to be audited will be confirmed after
audit has been arranged – 10 working days prior
to audit.
• Auditors to upload reports within 5 days of audit.
• Auditee to respond within 10 working days.
• Warning issued then if no response within a
further 20 working days automatic temporary
suspension.
Principal Changes - Bodies
• Grade of body to reflect performance of certifiers
(large bodies expected to support certifiers and
promote common practices and standards).
• Body obligation to provide training (certifiers
CPD outcomes reflected in body scores).
• Protection of Certifiers emphasised.
Principal Changes – Project Audits
• Evidence standards clarified – and harmonised
with BSD “Blue Book” and other recently issued
or revised documents (TB1 & 6, GN 3 and 9).
• Certifier and Project Audits merged creating new
category P9A to check certifiers acting within
limits of competence.
• Some revisions to categorisation of non
conformances.
• Scoring system amended to reflect changes
(scoring is a measure of performance but is not
the only determinant of the audit outcome).
In Conclusion
• Hopefully you now have a greater understanding
of not just of what is involved in the audit – but
perhaps more importantly also why.
• Hopefully that new understanding will serve to
allay some of the misconceptions and fears
surrounding the audit process.
Please remember:
• The audit is the alternative to verifier /
external engineering scrutiny of the design
on EVERY project you undertake.