OPERATING INCIDENT MITIGATION - Human Error Reduction For Protection Testing
The intent of this guideline is to help employees avoid making mistakes or equipment mis-operating
which might not even be the employee’s fault. As professionals, we realize that we can always do
better and must self-improve. Inadvertent operations not only cause damage, interrupt power but
impact both the employee and the company financially. Jobs have been shut down or not awarded
based on poor performance.
Please review this document periodically, especially just before a critical operation.
Please keep this document alive and pertinent by submitting your own challenges and solutions.
Your suggestions will be added in future revisions and may help someone else!
Fundamental Guiding Principles
•
Any time critical work is taking place, we should be structuring our tasks such that two
employees have to both make a mistake or both fail to prevent an irreversible action.
•
The work plan should be structured such that at least two mistakes must occur before an
irreversible action results.
•
Add or create additional steps in the work plan to prevent undesirable outcomes.
•
Use a checklist for tasks such as work planning, trip checks, etc.
•
Use a wire lift log or other document to indicate wires lifted or test switches open.
Preferred Practice – Operating Incident Mitigation Rev. 7 07/21/2017
Resolving Conflicts and Priorities
Time pressure, stress, understaffing a project or lack of skills / proficiencies available may conflict
with the goals and proper execution of the OIM guidelines. AET has set OIM as a priority should
conflicts arise. If employees are issued conflicting goals such as OIM guidelines must be followed
but the job has to be finished too quickly to use all of the OIM practices, the job should be stopped
and the concern should be elevated to AET management. Employees can become stressed out
when issued conflicting goals or resort to unusually long work days which pose additional error
hazards due to fatigue. Employees should reflect stress or problems up to management.
Sometimes, additional support can be used or the schedule can be re-adjusted to effect reasonable
practices.
If a project presents unusual risks or additional challenges, both the host utility and AET
management needs to be communicated with. If something were to go wrong, the responsibility
can be properly attributed to all parties. AET employees should not be put in a position of having to
take on too much responsibility or risk.
There is an expectation that if an employee is stressed out, fatigued, over worked, is traveling long
hours to and from a job, does not have sufficient skills, etc., that employee should contact their
management and express a concern. The employee should also be self critical, recognize that their
performance may not be up to the challenge and take additional action to maintain the quality of
work.
AET management embraces the OIM principles strongly, wants to support its employees and
wishes to acknowledge success when performing the work correctly. AET also encourages sharing
near misses and other errors as an opportunity to learn and prevent similar recurrences. Employees
should feel free to share both their successes and failures. AET sees that as a sign of honesty,
responsibility and strength.
Preferred Practice – Operating Incident Mitigation
2
Preparation
•
The pre-job brief should include references to this guideline and other preferred practices.
Solicit input from all workers, encourage questions and suggestions. Each worker should have a
clear understanding of what is to be done, how, by whom and what to do if something goes
wrong. Identify specific challenges, critical steps where peer checks are required, and what
barriers / safeguards are to be used. Refer to practicing perfection handbook.
•
Review the LOTO for the task planned with all workers. Verify LOTO is adequate for the work to
be performed.
•
Use write-ups and perform thorough analysis of protection scheme alterations – Compare
schemes with wiring drawings and actual panel wiring to identify discrepancies and poorly
marked equipment. Drawings might not be up to date so pay particular attention to extra or
missing wires found in the wiring to panel comparison.
•
Write a step by step commissioning test plan with expected results. In a post incident
investigation, you may be asked, “where was your written plan?” See appendix for an example
of a recently completed plan.
•
Protect equipment and panels using warning tape to block access to equipment not part of the
test sequence.
•
Communicate with all employees on the job the criticality of the operation, its potential impacts
and solicit input / support from all interested parties. Eliminate distractions in the work place.
•
Use three part communication and phonetic alphabet. Avoid use of confusing numbers, i.e.,
sixty sounds like sixteen so use individual numbers like six zero for sixty or one six for sixteen.
•
Value the Questioning Attitude
Preferred Practice – Operating Incident Mitigation
3
Preparation (cont.)
•
Review with all employees on the job the sequence of the testing and what devices are
expected to operate. Explain what will be done in the case of an error.
•
Walk down and inspect all devices in a “dry run” with the mindset of improving the process or
eliminating more traps.
•
Use blue painter’s tape to provide temporary relay or panel markings. Use magic marker or
sharpie on the tape.
•
Use blue painter’s tape to block terminal blocks not to be touched.
•
Apply tape to block the terminal above and below the terminal that will have the jumper
applied, in an effort to avoid accidentally mis-counting. Double verify.
•
Use tape to protect open test fingers on test switches from accidentally being contacted or
operated. Sometimes moveable blades or fingers are energized.
•
Pay special attention to unusual conditions. Protection does not operate the same as expected
when a second contingency presents itself or you create a second contingency.
Example: One transmission line is out of service and the whole station with its customer
load is hanging on the other line. Take extra steps or use dual isolation (two series opens) to
make sure testing will not cause the sole remaining line to trip.
•
Use “what if mentality” and plan accordingly. Equipment may not have operated in decades or
contacts may be badly corroded. Breakers might open too slowly.
Preferred Practice – Operating Incident Mitigation
4
Preparation (cont.)
•
Realize that if a device doesn’t operate as expected that breaker failure may wipe out an entire
station.
•
Do not allow disconnects to open under load – block operation until sure all breakers have
opened first.
•
Sometimes when scrutinizing the plan, we fail to step back and look at the big picture and miss
something. We are so detail specific that a basic issue not even thought about is ignored.
•
If you don’t completely understand the protection scheme, how to test it, or the relay, seek
assistance from one of the SME’s (internal subject matter experts always willing to help). Asking
for help, knowledge or a better understanding is a sign of strength. It is ok to say, “how would
you test this?”
•
Check annunciators and battery ground detectors when entering (starting the workday) or
leaving the station (end of the work day). Check for abnormal conditions, i.e., a breaker open
that wasn’t the workday before.
•
Additional problems encountered, drawing issues, manpower shortages, time pressure, and
equipment problems can cause stress on the job. It is important that such problems be raised to
the project management where conflicting goals (schedule verses quality) can be addressed.
STOP the job when necessary and obtain additional support.
Preferred Practice – Operating Incident Mitigation
5
Use a Checklist
•
Reminds one of possible missing steps
•
Encourages one to practice and think it out ahead of time
•
Allows one to concentrate on the big picture
•
If done in order it becomes a procedure
•
Makes it easier to communicate our plan to others
•
Reduces procrastination
•
Creates a sense of accomplishment as tasks are done
•
If something goes wrong, someone may ask you, “Where is your written plan?”
•
A signed off checklist becomes a record or document of work performed
See appendix of a checklist which became a procedure for Quarry Substation
Preferred Practice – Operating Incident Mitigation
6
Protection Isolation Planning
The protection isolation plan should be documented on a separate record such as a lift log or
alteration log. The entries into the log should be in the correct order that the protection is to be
isolated with. Failure to follow the correct order in isolating or restoring protection to service could
cause a trip in itself. Before isolating protection, make sure that the clearance issued reflects the
protection that will be out of service and the local authority / dispatcher knows what protection is being
removed and its impacts. Verify what protection will operate should a fault occur while the protection is
out of service. Wires lifted should be taped and tagged with an OIM tag. Allow a co-worker an
opportunity to independently review the isolation plan and then compare notes. Use a co-worker to
peer check while the isolation is being performed.
Isolating line distance protection: Open breaker failure outputs first, regular trips second,
currents third, voltage fourth and other logic inputs last. Note, extra steps are required if the line
protection is part of a line differential or phase comparison scheme where protection must be defeated
at both ends simultaneously before opening current inputs. Extra steps are also required when the line
protection is part of a communication aided or supervised scheme such as directional comparison
blocking or permissive overreaching transfer trip.
Isolating transformer / bus protection: Open breaker failure outputs first, regular trips to the
lockout relay(s) second, currents third, voltage fourth and other logic inputs last. Since a lockout is
being used, also open the output fingers of the lockout as well to accommodate safe restoration.
Isolating generation protection: Most generators do not have redundant protection packages
and cannot be run without protection. Verify that the generator is off line and will not be restarted
prior to isolating generator protection. However, an offline generator does not assure that the
protection is always safe to remove from service. An offline generator does not assure that the trips are
not still in service. For example, the generator may have a unit differential that protects portions of bus
still energized as only the generator isolation disconnect is open while the generator is out of service.
Open breaker failure outputs first, regular trips second, currents third, voltage fourth and other logic
inputs last.
Preferred Practice – Operating Incident Mitigation
7
Protection Isolation Planning (cont.)
Isolating a lockout relay: Before isolating, verify if there are any other relay coils in parallel with
the lockout relay coil. Some utilities use a redundant breaker failure relay in parallel with the lockout
relay that must also be isolated. Opening the coil circuit without opening the trip outputs of the relay in
itself is not sufficient because it leaves no safe way to close the coil circuit back in. It may be necessary
to operate the lockout relay or test a specific contact on the relay. However, the coil circuit can be
verified with a DC voltmeter prior to closing.
Isolating an over current or under voltage relay: Open the trip fingers first if a Westinghouse
design or pull GE paddle slowly (trip fingers are shorter and separate first). Caution: some GE relays
have automatic shorting bars in parallel with output contacts. Review the Instruction Booklet for GE
under voltage relays.
Preferred Practice – Operating Incident Mitigation
8
Execution
•
Work together on every step – avoid the “you do that and I’ll do this” mindset. Remember – if
something goes wrong, all on the job will be held responsible, so, work together.
•
Point to a device before operating or applying a jumper and ask co-worker – “agreed?” Wait for
co-worker to respond “agreed!” Then execute.
•
Any hesitancy or pinch feeling – STOP, re-review and repeat from beginning – STAR / AESOP /
Practicing Perfection Workbook
•
OOPS – Out of procedure – STOP
•
Read the step or report actions aloud so all can hear. Sometimes your ears will pick up a conflict
or a hesitancy. (use measure twice – cut once philosophy)
Example: “ Am I in panel 10C?”, wait for response from other employee to state aloud or if
alone state aloud, “ yes, you are (I am) in panel 10C”.
•
Experience has shown that a typical precursor to an unwanted event is a last minute change to
the plan. During the procedure, something does not operate as intended and the employees
suddenly realize they omitted some condition or find an unanticipated response. Be wary of a
quick fix. STOP, pause for a considerable time period and re-work the plan thoroughly.
•
Value unanticipated responses or obstacles as warning signs that the plan needs more work.
•
Avoid “Missionitis”. Set aside all time constraints and concerns like “We have to get this done
by 2:00 PM because I have to be somewhere by 4:00 PM”. A year from now, no one will
remember if you were a day late bringing in a job but they will remember that you inadvertently
tripped a line, transformer, bus or feeder. If you make a mistake, you won’t be where you
wanted to be at 4:00 PM anyway.
•
DO NOT RUSH
Preferred Practice – Operating Incident Mitigation
9
Execution (cont.)
•
Before applying a jumper or ohm - meter, check for expected or un-expected voltage first. Use
two digital volt meters when using ground as an extension lead i.e., ohm checks to equipment in
the yard from the relay house. Turn on the audible of the digital voltmeter on one end of the
conductor being checked and verify the 7 volts DC from the digital voltmeter in the ohm mode
at the other end of the conductor before applying the short at the remote end.
•
Use switch in place of jumper – connect switch in the open position and have a co-worker verify
it is connected to the correct terminals before closing.
•
Used fused jumpers or switch where circuits are under-protected (large size circuit breakers or
fuse slugs instead of fuses).
•
Look for sneak circuit paths or contacts that may not be in the anticipated state.
•
Should an unexpected event or operation occur, STOP the job, and notify all workers on site.
Call the local dispatching jurisdiction as soon as possible. Notify management / oversight of
both the host utility and AETCO. Review the drawings, targets and conditions to assist in
restoration but do not operate any devices until told to do so by the dispatchers. Experience has
shown that the best policy is to be open and honest when an event occurs, despite the
conditions or causes.
Preferred Practice – Operating Incident Mitigation
10
Protection Restoration
Restoring line distance protection: Close logic inputs first, close voltage inputs second, close
currents third, if the line is in service verify normal current and voltage on the relay and all targets are
reset, close regular trips fourth using a voltage check on each test switch prior to closing each test
switch. Close breaker failure trips last using a voltage check on each test switch prior to closing each
test switch. When performing a voltage check on test fingers; check for no voltage across a test switch
finger, close that finger only, then test voltage across the second test switch finger and then close that
finger only and so on. Use peer checking during restoration and verify that voltmeter is in the voltage
mode, not in ohms. Note, extra steps are required if the line protection is part of a line differential or
phase comparison scheme where protection must be defeated at both ends simultaneously before
opening current inputs. Extra steps are also required when the line protection is part of a
communication aided or supervised scheme such as directional comparison blocking or permissive
overreaching transfer trip.
Restoring transformer / bus protection: Close logic inputs first, close voltage inputs second,
close current inputs third, verify relay is in restraint and all targets are reset. If a lockout relay was
isolated, verify its output fingers are still open, close the coil circuit only and verify no trip of the lockout
relay and then close regular trips fourth using a voltage check on each test switch prior to closing each
test switch. Close breaker failure trips last using a voltage check on each test switch prior to closing
each test switch. When performing a voltage check on test fingers; check for no voltage across a test
switch finger, close that finger only, then test voltage across the second test switch finger and then close
that finger only and so on. Use peer checking during restoration and verify that voltmeter is in the
voltage mode, not in ohms.
Restoring generation protection: Close logic inputs first, close voltage inputs second, close
current inputs third and verify relay is not calling for a trip and targets are reset, close regular trips
fourth using voltage check on each test switch prior to closing each test switch. Close breaker failure
trips last using a voltage check on each test switch prior to closing each test switch. When performing a
voltage check on test fingers; check for no voltage across a test switch finger, close that finger only, then
test voltage across the second test switch finger and then close that finger only and so on. Use peer
checking during restoration and verify that voltmeter is in the voltage mode, not in ohms.
Preferred Practice – Operating Incident Mitigation
11
Protection Restoration (cont.)
Restoring a lockout relay: Before restoring, verify if there are any other relay coils in parallel
with the lockout relay coil and the lockout relay is reset. Some utilities use a redundant breaker failure
relay in parallel with the lockout relay that must also be isolated. Verify the lockout relay trip outputs
(and breaker failure relay if used) are open and then close the coil circuit back in first. Verify the lockout
relay did not trip (and breaker failure relay is not picked up if used), close regular trips second using
voltage check on each test switch prior to closing each test switch. Close breaker failure trips last using
voltage check on each test switch prior to closing each test switch. When performing a voltage check on
test fingers; check for no voltage across a test switch finger, close that finger only, then test voltage
across the second test switch finger and then close that finger only and so on. Use peer checking during
restoration and verify that voltmeter is in the voltage mode, not in ohms.
Restoring an over current or under voltage relay: Close the voltage or current inputs first, verify
the relay is not calling for a trip and the target is reset if equipped. Close the trip fingers last if a
Westinghouse design or install the GE paddle slowly (trip fingers are shorter and connect last). Caution:
some GE relays have automatic shorting bars in parallel with output contacts. When restoring GE relays,
the contacts may take some time to open when installing the paddle and the trip may need to be
blocked to allow time for the relay to reset. Review the Instruction Booklet for GE under voltage relays.
Post Execution
•
Restore protection in correct sequence.
•
Identify any near misses and what could have been done better.
•
If something needs to be repeated, be sure that initial conditions have not changed and
repeated steps are started and done in the correct order.
•
Share with other employees what worked and what could have been improved. You might
prevent someone else from making a mistake.
Preferred Practice – Operating Incident Mitigation
12
OIM Standing Procedure
•
Use a standing procedure to protect equipment for multiple days of construction.
•
Post the standing procedure in a conspicuous place such as on the inside of a commonly used
door or on the end of a panel.
•
•
Periodically update the standing procedure as conditions change.
Make sure all employees and crafts on the job understand the procedure.
Example: In the attached OIM SP used in 2009, the OIM SP color was blue because that is
the color of painter’s tape. Painter’s tape was applied to CT blocks in breakers that were
still connected to live protection. Painter’s tape was applied to the ends of string tags to
indicate the tag was part of an OIM SP. All employees knew, when they saw blue – take
extra precautions.
OIM Standing Procedure Used in a Nuclear Substation:
Preferred Practice – Operating Incident Mitigation
13
OIM Standing Procedure Examples
Preferred Practice – Operating Incident Mitigation
14
Preferred Practice – Operating Incident Mitigation
15
Preferred Practice – Operating Incident Mitigation
16
Specific Challenge – Identifying which CT to short out, i.e., with nearby CT’s still in service
An assignment included shorting a live CT circuit in a closed breaker. The associated protection
package was defeated and the challenge was, “ how do we know for sure which CT block to
short other than the drawings?”
Solution: At the relay panel, the CT wire grounding the CT circuit for the defeated protection
was temporarily lifted. The neutral wires on the protection package at the panel and all four live
CT circuits in the breaker were first checked for voltage to ground and then checked for ohms to
ground to confirm the correct CT circuit was ungrounded thereby confirming the drawings.
Specific Challenge – How to verify which contact to short out during trip testing
A jumper had to be applied to a relay contact to trip a breaker but terminal numbering was not
certain. The relay also had other live breakers that could not be tripped without dumping load.
Solution: Before applying the jumper, a DC voltmeter was applied, the DC was temporarily shut
off to the intended breaker’s trip coil. The loss of DC on the relay contact proved that the
correct contact was selected for applying the jumper. If possible, leave the voltmeter leads
connected using alligator clips and apply the fused jumper to the voltmeter ends of the lead
(using stacking terminals) to prevent losing track of the terminals to apply the jumper. Use the
fused pushbutton switch if possible.
Another variation was to prove a contact into a SCADA RTU.
Solution: Before shorting the contact at the lockout relay, the voltmeter was connected and
while the meter was still connected, the SCADA point was first jumped at the SCADA RTU which
collapsed the voltage on the lockout contact which verified that the voltmeter and jumper
would be applied to the correct contact.
Preferred Practice – Operating Incident Mitigation
17
Specific Challenge – How to safely remove a cover from a live HEA lockout relay
Although undesirable, occasionally an HEA lockout cover must be removed while in service.
1.) Use two hands and carefully squeeze the panel end of the cover top to bottom causing the
sides to bulge out slightly while pulling off to keep the cover from dragging on the “a” contacts.
2.) Install a short control switch cover over the back of the HEA to protect the hinged armature
from accidental physical actuation.
3.) When re-installing, remove the screw completely by unthreading it from the back plate,
squeeze the cover again as in 1.) above, carefully slide on, align the hole in the back with the
threads showing and re-install the screw.
Preferred Practice – Operating Incident Mitigation
18
Specific Challenge – How to avoid operating the wrong control switch in a control panel
A multiple page trip and breaker function test required operating the breaker control switch
numerous times through out the procedure. Adjacent to the breaker control switch, were
several other breaker and motor operated disconnect controls that could not be operated.
Solution: 8.5 x 11 inch paper was hung over the controls that could not be operated by hanging
with one piece of tape attached to the panel blocking the controls. Or, a temporary
construction tag could have been hung on the device to be operated. Use point first, confirm by
co-worker and double verify before operating in either event.
Specific Challenge – How to avoid applying a short to the incorrect terminals
Red tape was used to protect terminals not to be touched during trip testing and to identify
correct relays to apply jumper. Note holes cut in red tape which expose the correct terminals to
apply jumper.
Red tape was used to cover terminals leaving exposed only the screws that a jumper had to be
applied to. This allows two workers to verify beforehand that the correct relay and terminals
will have the jumper applied to.
Preferred Practice – Operating Incident Mitigation
19
When applying a jumper to a terminal block or device or lifting / landing a wire, apply tape to
the terminal above and below the terminal where the work is to be performed upon. This
leaves only the intended terminal exposed. Use peer checking and have another crew member
verify that the correct terminal has been left exposed including the correct panel, terminal
block, etc.
Preferred Practice – Operating Incident Mitigation
20
Other OIM Guidelines
•
Find ways to identify the intended panel or gear to be worked in. Close all doors that should not
be entered or identify where the correct work should take place. Be especially cautious of look
alike devices, breakers, relay panels, etc. Many events include those that have just walked into
the wrong cabinet.
•
Use caution tape secured with electrical or painter’s tape to block panels or devices to stay away
from.
•
Look for unusual conditions, devices in an abnormal state or indications in the station before
each testing segment. Check annunciator at beginning and end of work shift or day
•
Check battery ground detector at beginning and end of work shift or day
•
Constantly be self vigilant. If tired, distracted or not up to the task, do not proceed, seek
assistance.
•
Ask yourself and your co-worker, “are we green - good to go, yellow ragged edge or red – got to
stop?” Be a good listener.
•
When using the ABB FT multiple test plug, verify that no wires are installed into the plug will it is
being installed or removed. If leads are attached, the CT circuit can be grounded through the
test set and cause a trip of other live protection connected to the same CT circuit.
Attach orange label:
Preferred Practice – Operating Incident Mitigation
21
•
Before landing or removing wires, check for no potential and no current.
•
Before removing wires, check for no AC or DC current using mini clamp-on ammeter.
•
Before opening test switches assumed to be dead or having no current, check for no AC or DC
current using mini clamp-on ammeter. Use class 0 gloves or higher with leather protectors
when working on or in the vicinity of CT circuits.
Preferred Practice – Operating Incident Mitigation
22
OIM Appendix – Examples of barriers
Preferred Practice – Operating Incident Mitigation
23
Preferred Practice – Operating Incident Mitigation
24
Preferred Practice – Operating Incident Mitigation
25
Preferred Practice – Operating Incident Mitigation
26
Preferred Practice – Operating Incident Mitigation
27
Preferred Practice – Operating Incident Mitigation
28
OIM Appendix – Executed Trip Test
Preferred Practice – Operating Incident Mitigation
29
Preferred Practice – Operating Incident Mitigation
30
Preferred Practice – Operating Incident Mitigation
31