CIP Spot Check Process
Gary Campbell
Manager of Compliance Audits
ReliabilityFirst Corporation
August, 2009
Presentation Goals
The audience should be :
 Aware of the ReliabilityFirst CIP Spot Check Process to be used
for review of the thirteen requirements for Table 1 entities or CIP
Spot Checks in general
 Cognizant of differences between an audit and spot check
processes
 Have an understanding of the auditors perspective in
performance of the audits/spot check
3/23/2016
2
Compliance Audits
ReliabilityFirst performs compliance audits:
 Once every three years for BA, TOP, RC, TO/LCC
 Once every six years on all other functional designations starting
from 2008




Proper notice as per standard or CMEP
Unscheduled as required to monitor compliance
Can be on-site or off-site
CIP standards audit intervals have not been determined at this
time
 At this time , assume a three /six year interval for applicable functions
 Public and Non-Public Reports sent to NERC, Registered
Entities, FERC and maintained on file at ReliabilityFirst
3/23/2016
3
Spot Checks
RFC performs spot checks
 Proper notice as per standard or CMEP
 Performed as discussed in CMEP
 Can be triggered by an event, concern, trend, NERC or
FERC request, etc.
 Verify/confirm self certification, self reporting, data submittals
 Any functional designations or registered entities can be
subject to spot check
 Report maintained on file at ReliabilityFirst
 Registered Entity receives copy
 NERC does not receive a copy, at this time
3/23/2016
4
ReliabilityFirst Audit & Spot Check
Goals
 To be Performed:
 To the highest standard
 Government auditing standards. CMEP, NERC RoP
 Professionally
 Consistently
 Auditor tools – QRSAWs, Surveys, RFI’s
 Regional agreed upon practices
 Credibly
 With reasonable assurance, sufficient and appropriate
evidence to substantiate the findings
3/23/2016
5
Audit Team Member Goals
The audit team will strive to be:




Consistent and fair
Cooperative
Professional
Substantiate their findings
 Providing credibility for their findings
 Findings which can withstand scrutiny of review
 Develop a complete record of its findings
• Documentation
• Notes
3/23/2016
6
The Audited Entity
The audited entity should present Just the Facts by
providing the evidence through documentation to meet
the requirements of a standard as :
 A complete record and understanding demonstrating
compliance to a standard
 Evidence that is valid
 Evidence that can be substantiated?
 And evidence which can withstand the scrutiny of the auditor
and the public
3/23/2016
7
Compliance Advice
The ReliabilityFirst staff and audit teams can not :
 Tell an entity how to be compliant
 Specify which practice, process to implement
 Provide assurance of being compliant outside of the audit
process
The staff or audit team can:
 Listen and provide guidance
 Direct registered regional entities to seek the assistance of a
consultant if the staff cannot direct the person to available
documentation addressing the question
3/23/2016
8
Confidentiality Agreements
Audit Team members are:
 Bound by their Code of Conducts or applicable Confidentiality
Agreements
 provided to the Audited Entity
 NERC staff falls under the statement of NERC's obligation on the
ROP (Section 1500) and code of conduct
 FERC is bound by its agreements
 Regional staff fall under their Code of Conduct and
confidentiality statement per our delegation agreement
 Contractors and industry volunteers will sign regional
confidentiality agreements
 Regional staff shall not sign an entity specific confidentiality
agreement
3/23/2016
9
Team Member Review of Information
The team will:
 Have a conference call with the entity 85 days before the spot
check review
 Clear up an items of concern or understanding in the process
 Have a team meeting to discuss the audit teams review of
submitted information approximately 2 weeks before the review
date
 Request additional information for clarification or understanding
 Discuss preliminary requirement findings
 This effort allow auditors to focus on those areas of importance, lacking
information or understanding at the review.
3/23/2016
10
CIP Spot Check Scope
 The current CIP Spot Check Scope:
 For Table 1 entities - 13 requirements identified for review by
NERC for the period xxxxxxxxxxxxxxxxxxxxxxxxx
 After July 1, 2010 – Table 1 and 2 entities – 41 requirements Not
yet determined to be a spot check/audit
3/23/2016
11
CIPS Compliance Review Team
Consist of:
 Usually at least 3 – 4 members with experience with
CIPS, IT and Operations
 Lead (RFC Compliance Staff)
 NERC observer or participant (@ NERC’s discretion)
 FERC participant (@ FERC’s discretion)
3/23/2016
12
Audit Team Members Roles
Team Members:
 Utilize technical experience
 Exercise professional judgment
 Gather data and information
 Perform Interviews
 Determine validity of the evidence
 Substantiate the evidence
3/23/2016
13
Objection to a Team Member
A Registered Entity can object to an team member
 On the grounds of conflict of interest, or the existence of other
circumstances that could interfere with the teams impartial
performance of their duties
 Objection must be in writing to the Compliance Enforcement
Authority no later than 15 days prior to the start of the audit or
spot check
 ReliabilityFirst will make the final determination if the member
can participate in the audit or spot check
 NERC and FERC staff can not be limited in their participation on
an audit or spot check
3/23/2016
14
The Spot Check
Process
The Spot Check Process consists of





Initial Notification and Request for information
Conference Call with entitiy
Spot Check Team Review of Information
Spot Check Review on site
Preparation of Spot Check Assessment and
Report
 Distribution of Sport Check Report
3/23/2016
15
Initial Notification
Initial Notifications will be:
 For the 13 requirements, will be sent at least 90 days before the
scheduled the scheduled review date of a spot check or audit.
 CMEP requirement is 20 days for a Spot Check and 60 days for an
audit.
 Contains
 Notification Letter
•
•
•
•
Request for information
Background info on the process
Audit Preparation Guidelines
Audit Team Bios, Confidentiality, and COIs
 An agenda
 Spot Check Worksheet
 Questionnaires/Reliability Standard Audit Worksheets
 Pre-Audit Questionnaires
3/23/2016
16
Audit Agenda
ReliabilityFirst will provide an agenda which:

Covers the expected days to complete the audit

Provide Audit sub-teams if appropriate

Schedule for standards to be audited and time
allotted for presentations

Interview and group meeting schedules
3/23/2016
17
Spot Check Worksheet
The worksheet will:
 Provide listing of all standards to be
addressed in the spot check
For your use to track progress on standards
3/23/2016
18
Questionnaires/Reliability Standard
Auditor (QRSAWs)
QRSAWs:
 Must be completed and returned 30 days before your
audit your scheduled review date
 Provides guidelines concerning the requirements
 Does not add additional requirements
 Posted on NERC Website
 Could be used by internal compliance programs
3/23/2016
19
Pre-Audit Questionnaires
The Pre-Audit Questionnaires request:
Entity Profile
Logistical Information Request
• Hotel, airport, and travel information
Security Considerations
• Identification Requirements
• Restrictions
• Escorts
3/23/2016
20
The On-site Review
and
Post Monitoring Reporting
3/23/2016
21
Typical Audit
The audit consists of:
 Opening Briefing
 Review of requirements with SMEs and entity
personnel
 Any site visits as necessary
 Exit Briefing
The CIP Spot Check will consist of the same basic
steps
3/23/2016
22
Opening Briefing
Opening Briefing with management and
participants of the review process:
 For audits and spot checks combined the 693 and CIPs topics will be
discussed together
 Allows audit team to:






State Objective and Scope
Explain process of the audit
Discuss Confidentiality and COI
Set the tone for the audit
Provide the roles of the audit team and audited entity
Opportunity to seek clarification on issues from RSAWs and any other
preliminary information submitted.
 Allows registered entity to:
 Provide overview of the their system and operations
 To provide logistic and security information
 Seek clarifications on scope of the audit
3/23/2016
23
The Review
 The Compliance Review of evidence to the
requirements is completed:
 According to the Agenda
 With entity personnel as they designate
 SME, PCC, other personnel
 With an opportunity for the team to additional
information, clarification and obtain an understanding
of the entities evidence and approach
 Should lead to a team finding on compliance
3/23/2016
24
Exit Breifing
Exit Breifing with management and all participants
of the audit to:
 Will perform with similar organization of the opening
briefing
 Provide the preliminary findings
 Review the scope of the audit
 Provide the findings and the team’s basis for the findings
 Discuss Confidentiality
 Discuss the report process and timeline
 Request completion of feedback forms
3/23/2016
25
Reports
 CIP Spot Checks will
 Have an assessment and report created ( Audits do not have a
documented assessment)
 Assessment is the compilation of information contained in the
completed QRSAWs, not sent to the entity.
 Spot Check Reports are a condensed version of the audit report
containing:
• Executive Summary
• Scope
• Requirement Findings
 Draft report will sent to the entity for comments
 Final Spot Check Reports will be sent to the entity and kept on file
at ReliabilityFirst.
• Will not be sent to NERC at this time
3/23/2016
26
Audit/Spot Check Report Timeline
The Audit Team
Lead transmits
the report for
audit team
review
The Audit Team
Lead receives
comments from
the Audit team
20 business
days
10 business
days
5 business
days
The Audit Team
Lead develops a
draft report
Audit Team
provides
comments
Audit Team
Lead revises
the draft
compliance
report
The Audit Team
conducts an
exit briefing
with the
Registered
Entity with
preliminary
findings
Audit Team Lead
sends the draft
report to the Audit
Team for their
review and
comments
20 Business days
5 business
days
5 business
days
Registered Entity
reviews and
provide comments
Revision of
the draft
report
Audit Team
provides
comments
The Audit Team
Lead sends the
draft report to the
Registered Entity
for their review
and comments
The draft
report is edited
upon receipt of
Registered
Entity
comments
5 business
days
Audit Team
Lead
completes
final
compliance
report
Audit Team
Lead
revises the
report upon
receipt of
Audit
Team’s
comments
Final report sent
to RFC VP and
Director of
Compliance,
Registered
Entity, NERC &
FERC as
applicable
Questions ?
Gary Campbell
ReliabilityFirst Corporation
Senior Consultant – Compliance
3/23/2016
28