C L A R I T Y ▪ A S S U R A N C E ▪ R E S U LT S
MIDWEST
R E L I AB I L I T Y
O R G AN I Z AT I O N
Reliability Assurance
Initiative
Thomas P. Tierney
Director of Compliance, MRO
SPP Compliance Forum
May 23, 2013
Improving RELIABILITY and mitigating
RISKS to the Bulk Power System
Common Mission
Improve the Reliability
of the Bulk Power System
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
2
Improvement Is the Goal
We have very reliable systems within
MRO/SPP, but we can still improve by
identifying problems and fixing them – no
weak links
There is always opportunity for improvement
within the design criteria of an
interconnected system
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
3
Demystifying Internal
Controls
No, Really… What Is an Internal Control?
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
4
Nothing New
Registered Entities have been managing
reliability for decades – they have
management practices (i.e. controls) around
reliability
Existing practices have been translated into
the Reliability Standards and documented –
“operationalizing compliance”
Don’t overthink “internal controls”
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
5
Definitions
Risk
• Possibility that something undesirable will happen
• Measured as a combination of likelihood and impact
Control/Control Activity
• Policy, procedure, checklist, etc. designed to minimize
the opportunity for a risk to be realized
Internal Control
• Control activity performed internally, not by a third
party
• Management practices that include control activities
performed internally (“self monitoring”)
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
6
Types of Risk
Inherent Risk
• Risks “built-in” to a given entity, based on geography, what
facilities it operates, “interconnectedness,” etc.
• Reliability Standards are designed to mitigate inherent risk in a
broad sense
Control Risk
• Risk that management practices or control activities are not
achieving their reliability or compliance objectives
Detection Risk
• Risk that possible violations are going unnoticed
Residual Risk
• Risk that remains after application of a control and other
mitigating factors
• Difficult and expensive to eliminate 100% of risk – we must live
with some risk
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
7
Types of Controls
Preventive
• Controls designed to stop something from occurring
Detective
• Controls designed to identify when a possible violation
has occurred and facilitate timely remediation
• Also known as “Monitoring” controls
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
8
Control Hierarchies
Multiple, complementary controls that work
together to reduce risk (“Defense in depth”)
• Primary
• Secondary
• Tertiary
Secondary and Tertiary controls serve as a
“safety net” in case the Primary control does
not function as expected
Each subsequent tier of controls further
reduces residual risk
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
9
Examples
Protection System Maintenance and Testing
• Relay technicians complete work orders according to a
pre-defined checklist to prevent steps being skipped or
performed incorrectly
• Supervisors review and approve completed work
orders to verify technicians’ proper use of the checklist
• A sample of work orders is reviewed by Internal Audit
to verify accuracy and completeness
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
10
Examples
Procedure/
Process
Program
Documents
(Procedures)
Control Activity
Control
Checklist followed and
completed, exceptions
noted, follow-up notes
signed
Standard
Work Order
Review for completeness and
accuracy, follow-up actions
closed or scheduled to be
completed, signed
Supervisory
Review
Management
Oversight
May 23, 2013
CL A RI T Y
Periodic sampling of work
orders to determine
program is being
completed and properly
reviewed
▪ A S S URA NCE ▪
RE S ULT S
Control Type
Primary
Control
Secondary
Control
Tertiary Control
11
Examples
Training
• Management establishes training objectives and
reviews training materials to confirm objectives are met
• Individuals are tested after completion of training to
ensure effectiveness of delivery
• Supervisors conduct performance observations to
verify past training has been effective and to identify
additional training needs
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
12
Examples
Procedure/
Process
Program
Documents
(Procedures)
Control Activity
Control
Management establishes
training objectives and
reviews training materials to
confirm objectives are met
Training
Objectives
Individuals are tested after
completion of training to
ensure effectiveness of
delivery
Training
Evaluation
Performance
Observations
May 23, 2013
CL A RI T Y
Supervisors conduct
performance observations to
verify past training has been
effective and to identify
additional training needs
▪ A S S URA NCE ▪
RE S ULT S
Control Type
Primary
Control
Secondary
Control
Tertiary Control
13
Examples
Cybersecurity
• Systems are configured to require passwords to
prevent unauthorized access
• All changes to systems are reviewed, approved, and
tested to ensure that unauthorized changes do not
occur
• Periodic reviews are conducted to ensure that
password controls adhere to corporate security policies
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
14
Examples
Procedure/
Process
Security
Policies
Configuration
Management
Procedures
Control Activity
Control
Systems are configured to
require passwords to
prevent unauthorized
access
Password
Controls
Configuration
Management
Security
Assessments
May 23, 2013
CL A RI T Y
All changes to systems are
reviewed, approved, and
tested to ensure that
unauthorized changes do not
occur
Periodic reviews are
conducted to ensure that
password controls adhere
to corporate security
policies
▪ A S S URA NCE ▪
RE S ULT S
Control Type
Primary
Control
Secondary
Control
Tertiary Control
15
Reliability Assurance
Initiative
Focusing on Risk
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
16
Current State
“One size fits all” compliance model
• NERC Actively Monitored Standards do not change based
on regional differences, entity size, etc.
• No consideration of management practices (i.e. controls)
around reliability standards
Zero-defect approach to enforcement is
burdensome
• Every violation requires a regulatory filing regardless of
severity
• Self-reports require significant effort
Administrative Citation Process (ACP) & Find,
Fix, Track (FFT) are not sufficient
• Expediting enforcement won’t solve the problem
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
17
Key Elements of RAI
Shape compliance monitoring and mitigation
based on risk
Reserve enforcement for most significant
risks
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
18
Scoping of Work
Assessment of each entity’s inherent risk
Some factors influencing assessment
•
•
•
•
•
•
•
•
May 23, 2013
Facilities
Special Protection Systems
IROLs
Geographic location
Functions performed
Connectivity (physical and cyber)
EMS/SCADA system
Compliance history
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
19
Scoping of Work
What does a risk assessment look like?
• Not a letter grade or single rating
• Entities will not be compared and ranked
Assessment will look more like a matrix
• Certain families of standards may be higher risk for
one entity, less risky for another
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
20
Scoping of Work
Internal controls established by each entity must
be identified
Evaluation of select controls to determine
effectiveness
• Design – Is the control, as documented, adequate to
address the risk?
• Operational – Is the control implemented as designed?
Effective controls reduce residual risk to an
acceptable level
MRO staff can rely on effective controls
• Regulatory scope can be adjusted – less auditing and
testing (or none) where strong controls exist
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
21
Scoping of Work
Risk assessments and internal controls will
be leveraged across all compliance
monitoring activities
Internal emphasis should shift over time
toward maintaining effective controls around
Reliability Standards
• Continue to identify and correct issues in a timely
fashion
• Focus on reliable operations first
• Compliance should be a natural outcome of strong
operations
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
22
Compliance Exceptions
“Compliance Exceptions” represent lower
risk violations
•
•
•
•
Do not represent significant risk to the BES
Identified by an entity itself or by regional staff
Initially tracked at the regional level
No enforcement proceedings, no penalties
Mitigation will always be important
• What was done to address the problem itself?
• What is being done to prevent recurrence?
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
23
Compliance Exceptions
Enforcement will focus on most significant or
high-risk issues
• Violation poses significant risk to reliable operation of
the BES, e.g. cause or contributing factor in a
cascading event
• Multiple smaller issues may aggregate into a bigger
problem or are indicative of a poor control environment
• Willful misconduct
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
24
MRO Pilots
Compliance audit
• Tools being developed with input from industry, the
Regions, and NERC
• Currently developing risk assessment
• Internal controls evaluation to occur during June & July
• Scope will reflect risk and presence of effective
controls
• Audit completion in Q4 of 2013
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
25
MRO Pilots
Self-certification
• Transition from blanket, “check the box” approach to
narrowly focused self-certifications
• Scope limited to FAC-008-3 R6 based on problems
identified on recent audits
• Focus on self-assessment process and on controls to
identify and correct problems
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
26
Contact Information
Thomas P. Tierney, Director of Compliance
Midwest Reliability Organization
tp.tierney@midwestreliability.org
(651) 855-1745
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
27
Questions?
May 23, 2013
CL A RI T Y
▪ A S S URA NCE ▪
RE S ULT S
28