QUALITY OF EVIDENCE
FRCC Compliance Workshop
September/October 2008
Evidence is essential
The Registered Entity has the burden to
prove compliance through presentation of
quality evidence to a standard or
requirement.
“Provide Quality Evidence ”
2
Objective
Objective – What is quality evidence?
–
–
–
–
Define evidence and the types of evidence.
Describe what constitutes quality evidence.
Define appropriate and sufficient evidence.
Provide a better understanding of evidence and its
role in compliance monitoring activities.
3
Definition of Evidence
Evidence is:
– Data or information on which factual
statements can be based.
– A collection of relevant and sufficient
information offered to verify a presented fact.
– Obtained through examining and analyzing
records, observing events, or conducting
interviews.
4
Presenting Your Evidence
• Auditors gather evidence to draw conclusions on findings in a
logical method based upon factual evidence.
• Facts are to be valid and relevant for the point needed to be
proven.
– A fact is something that can be objectively verified, is real
and does exists.
• Facts must be verifiable.
– A verifiable fact is one whose data or information can be
substantiated by comparison, investigation or confirmation.
• Entities should consider presenting and submitting their
evidence in the same manner.
5
Types of Evidence
Evidence can be provided as:
– Physical evidence
– Direct observations
– Corroborating evidence
• Corroborating evidence is evidence that tends to support a
proposition that is already supported by some evidence.
(Wikipedia 9/11/2008)
• To strengthen or support with other evidence; make more certain
(the Free dictionary 9/11/2008)
6
Types of Evidence
Physical Evidence is direct inspection of
activities of people, property, or events:
– Some examples include:
•
•
•
•
•
•
•
•
Schedules
Charts
Maps
Graphs
One-lines
Manuals
Procedures and Policies
Plans
7
Types of Evidence
Documentation:
– Physical forms of evidence which already exists
– Can be both external or internal
– Some examples include:
•
•
•
•
•
•
•
•
Letters
Contracts
Assessments
Agreements
Spreadsheets
Database Extracts
E-mails
Studies
8
Types of Evidence
Direct Observations:
– What an auditor sees during your review.
– Strengthened through corroboration with other
evidence.
– By itself is almost always unreliable.
– Is achieved by monitoring of personnel, facilities,
equipment, and tools.
9
Types of Evidence
Corroborating evidence is:
– Additional information from interviews, logs, etc provides clarity
to make the appropriate compliance decision.
– Used to establish the appropriateness of evidence.
– Tests the reliability of evidence and provides confirmation.
– Some examples include:
•
•
•
•
Reliability Coordinator or Neighboring questionnaires
Voice recordings
Logs
Interviews
10
An Example of Corroborating Evidence
The auditor requests evidence that a Registered Entity
has followed all directives from the Reliability
Coordinator
– The Registered Entity states it has not received any directives:
• Response is neither sufficient, nor adequate
– Testimonial
– Low quality
– May not withstand scrutiny
Corroborating evidence could be an e-mail from the
Reliability Coordinator stating the Reliability Coordinator
did:
– Not send any directives to the Registered Entity during the
specified time frame, or
– Did send a directive and it was followed
11
Quality Evidence
Quality evidence is evaluated by its:
– Appropriateness
• relevant, valid and reliable to support findings.
– Sufficiency
• enough information to lead another person to the same
conclusion.
– Quality
• high quality evidence to be used as proof of findings.
12
Appropriateness of Evidence
Quality evidence must be appropriate:
– Relevant to the reliability standard requirements:
• Is there enough to persuade the auditor his findings are
reasonable?
• Is it logically related to the requirement?
– Valid as a document:
• Is the evidence based upon sound reasoning and accurate
information?
– Reliable information:
• Can the evidence be substantiated when tested?
13
Example of Appropriate Evidence
An agreement is offered as evidence.
The auditor will seek to determine:
– Is this the original document?
– Does the agreement contain the Registered Entity
letterhead, approval signature and date, revision
date, etc?
– Does the agreement address the subject of the
requirement?
– Is this version the current version?
14
Sufficient Evidence
Quality evidence must be sufficient:
– Is there enough evidence to lead an auditor to the same findings
that you have reached.
– Strong evidence may be better than a large volume of weak
evidence.
15
Example of Sufficient Evidence
• The auditor is provided three documents to determine if
a procedure exists and is coordinated with neighboring
entities:
– A procedure
– E-mails
– Previous version of procedure
• The above evidence is sufficient as:
– A valid procedure is provided which addresses all subrequirements even if they do not apply to the audited entity.
– E-mails provide evidence the procedure was sent, received and
confirmed as coordinated.
– Previous version of procedure shows the procedure existed for
the audit period.
16
Quality Evidence
• The evidence must be of the highest
quality to be used as proof of findings.
• Quality evidence should contain:
–
–
–
–
–
–
Documents title
Purpose
Date approved
Revision level
Effective date
Authorizing signatures
17
Testimonial Evidence
Written or oral statements in response to
inquiries or interviews.
– Examples are:
• Reliability Coordinator or Neighboring questionnaires
• Interviews
• Letters sent for confirmation
18
Computer Processed Evidence
Computer evidence can be provided if:
– The evidence can be determined to be sufficient and
appropriate.
• Is the information considered quality evidence?
• Can the evidence be validated from electronic logs to confirm
creation?
• Some examples include:
–
–
–
–
Printouts (titled, dated)
Screen shots
Logs (titled, dated)
Study results (titled, dated)
19
Professional Judgment
All evidence reviewed requires some level of
professional judgment.
“Professional judgment of the auditors is based on their:
–
–
–
–
–
Knowledge
Skills
Experiences
Good Faith Application
Integrity
20
Final Thoughts
Some things to remember:
– Who you are trying to convince, the auditors.
– Compliance is based upon the evidence provided.
– Would you objectively make a determination of
compliance or possible violation?
21
Conclusion
At the end of the review, enough quality
evidence must have been provided to:
Soundly substantiate the teams findings
and
Support your expected findings
22
“Provide Quality Evidence ”
23
Questions ?
24