System Vulnerability Review Report
Hope Creek Feedwater
December 2011
Station:
Hope Creek
System Engineer:
Walter Bischoff / Pete Pino
Date Completed:
12/09/2011
Station Challenge:
TBD
CAP Tracking #:
80105125
1
Table of Contents
1
Executive Summary………………………………………………………………
2
Purpose…………………………………………………………………………… 4
3
Scope……………………………………………………………………………....4
4
Team Members…………………………………………………………………....5
5
Methodology..……………………………………………………………………..5
6.0
Fault Tree Analysis/Single Point Vulnerability Findings…………………………6
6.1 Feedpump suction side vulnerabilities and enhancements………………………..7
6.2 Feedpump and discharge vulnerabilities and enhancements…...…………………8
6.3 Feed pump turbine vulnerabilities ………………………………………………14
6.4 Feed pump and turbine control and bearing oil package………………………...18
6.5 Number 6 feedwater heaters……………………………………………………..25
6.6 Operations Feedback…………………………………………………………….32
6.7 Maintenance Feedback…………………………………………………………..36
6.8 External operating experience
6.9 Simulator results…………………………………………………………………47
6.10 Corrective Action Program Items………………………………………………. 49
6.11 System Walkdown…………………………………………………………….…53
7
Scheduling Priority………………………………………………………………53
8
Vulnerability Elimination or Mitigating Strategies…………………………….. 54
9
Review of SPV Initiatives from Other Sites……………………………………..63
10
References………………………………………………………………………..65
11
Attachments
2
1.0
Executive Summary
The feedwater system vulnerability review team identified several latent vulnerabilities based on
deep and broader reviews of system design drawings, industry operating experience (OPEX),
EPRI topical reports, and initiatives, both previous and current, related to equipment reliability
(2005 scram derate, critical parts). The major system components included in this review were
the Reactor Feed Pumps (RFP), Reactor Feed Pump Turbines (RFPT), RFPT steam control
valves, Control and Lubricating oil auxiliaries, and the high pressure feedwater heaters.
Significant findings by the team consisted of the single point vulnerabilities associated with the
feedpump suction valve position trip, feedpump turbine moisture drain valve position trip, 2-outof-3 sensing trip logic that uses a single sensing header, the feedpump turbine thrust bearing wear
trip, the loss of feedpump power supply trip solenoid, Control and Lube Oil vulnerability for oil
pump auto swap failures, and oil pump discharge check valves.
The team recommendation is to eliminate most of the significant equipment vulnerabilities by use
of plant design change modifications. Examples of major modifications is to install additional
sensing lines for the 2-out-of-3 trip switches connected to a single sensing header line, remove the
position trip logics for the feedpump suction valve and feedpump turbine moisture drain valve,
install larger accumulators and faster redundant oil pump start modifications, and changing the
failsafe auto trip logic on a loss of power trip.
The identified deadlines have been evaluated per ER-AA-2004 and implementation should be
commensurate with the assigned risk ranking. . Also, strong consideration for augmented quality
assurance or additional supervisor oversight for SPV work during outages would be highly
beneficial for achieving breaker to breaker runs.
The team also identified other vulnerabilities in the BWROG report for Scram Frequency
Reductions. Such improvements will not eliminate a trip but will provide added margin and
defense in depth. The RFP suction trips are 2/3 logic and a have time delays, however the trip set
points for all three trains (A,B, and C) are all the same. Staggering the trip set points will prevent
a trip of all 3 pumps. If a low suction pressure transient develops, tripping one pump sooner will
prevent all 3 from tripping.
3
Some of the single point vulnerabilities (SPVs) identified by the team will not be eliminated and
will remain with new mitigating strategies (preventive maintenance) proposed to ensure high
reliability and risk management to plant operations. The cost benefit and feasibility of
elimination was balanced by the team in its final recommendations.
The team’s recommendation is for eight (8) new modifications, one (1) CM work order, thirty
(30) new PCR tasks, and one (1) procedure change activity to be implemented based on the
schedule risk category suggested by the team. The final recommendations will then be
incorporated into the CAP process and LTA Manager for management oversight.
2.0
Purpose
Hope Creek has experienced a declining trend in unplanned major power losses and will continue
to strive for excellence and intolerance for unexpected equipment failures. Industry experience
has shown that some scrams are initiators to more significant events and challenge both
equipment and operators.
Consistent with the Principles for a Strong Nuclear Safety Culture, A Questioning Attitude is
Cultivated, this action supports the attribute that anomalies are recognized, thoroughly
investigated, promptly mitigated, and periodically analyzed in the aggregate.
3.0
Scope
The scope for this vulnerability review included all major components and instrumentation
associated with the feedwater portion of the main condensate system.
The boundary included all components from the suction valves of the main feedwater pumps up
to and including the feedwater dual isolation check valves. The listing of major components is as
follows:

Reactor Feed Pump (RFP)

Reactor Feed Pump Turbine (RFPT)

RFP and RFPT lube oil system

RFP Recirc valves

RFPT control steam control valves

RFP, RFPT, and lube oil trip logic
4

RFP and RFPT vibration monitoring system

#6 Feedwater heaters (FWH) and associated level controllers and control valves

#6 FWH bypass Start up level control valve (SULCV)

Applicable portions of support systems:

Gland sealing steam

Instrument air

Fire protection deluge
Major components or systems excluded from this review were: digital feedwater control (DFCS)
logic and associated electronics (for example Field bus modules and computer processors). DFCS
is a double redundant system and there are no known fault modes that could provide a single
point of vulnerability. Condensate pumps and 1-5 FWHs were excluded from this review. FWHs
1-5 will be reviewed in main condensate system vulnerability review team. The other systems
were not considered based on not meeting the threshold of problematic systems during the
selection of systems for the ER HIT Team charter.
5
4.0
Team Members
Team Leader:
Richard Cummins
System Engineer:
Walter Bischoff / Pete Pino
ER HIT Team Lead
Rudy Chan
Senior Reactor Operator:
Michael Cline, Mariaz Davis
Operations Simulator Instructor:
Gary Schmelz
Maintenance:
Richard Chuck
5.0
Methodology
The approach used by the team involved several steps listed as follows to probe design,
operations, training, maintenance, and parts for potential vulnerabilities that could be latent to the
organization:

Identify single point vulnerabilities (SPV) components by reviewing system P&IDs,
instrumentation and controls schematics and logic drawings.

Review of simulator scenarios to understand the type of equipment faults used to
simulate plant transients and trips was performed to determine other single point
vulnerabilities not readily apparent looking at drawings. In addition, use of the simulator
to inject other component failures was used to verify or refute potential impacts to the
plant.

Reviewed operations procedures such as abnormal procedures, system operating
procedures, overhead alarm response, and local alarm response type procedures. The
review focused on identifying actions that the operating crew will take based on a single
indication that would result in a plant trip, derate, or shutdown. These single indications
represent single point vulnerabilities when operator action is taken instead of an
equipment type failure. The second focus of the procedure reviews is to identify
potential human induced errors when operating the equipment with lack of or deficiencies
in the level of detail with guidance or instructions.
6

Reviewed sample of maintenance work orders and procedures to determine the level of
detail contained in these documents that could affect the ability of the worker to properly
complete the tasks without relying on knowledge skills

Reviewed internal and external operating experience (OPEX), for at least the past five
years, to learn from the industry and past experiences onsite to determine if appropriate
actions have been taken to eliminate or mitigate the threat to plant operations.

Reviewed previous scram derate initiative reports to determine if previous actions or
strategies to eliminate or mitigate have been taken and to identify if the components we
properly identified as SPV.

A team field walkdown was conducted. The purpose of this field walkdown was to
visually look at the operating and environmental conditions of the operating system for
other potential vulnerabilities not previously identified or known to station personnel.
For example, the team will look for vibrations on equipment, water, oil, or steam leaks on
components, physical position vulnerabilities, and visual materiel degradation.

At the completion of the vulnerability reviews, the team will determine the best strategy
to eliminate the threat or mitigate by use of maintenance strategies (for example,
preventive maintenance), performance monitoring (for example, PdM), or procedure
changes. The philosophy for the team is to eliminate the threat to the greatest extent
possible and then mitigate as appropriate.
The team also referred to the guidance from the system vulnerability review process in
accordance with ER-AA-2004. In addition, the team also developed enhanced guidelines to
supplement this procedure.
6.0
Fault Tree Analysis/Single Point Vulnerability Findings
The team focused on logic and components that posed a vulnerable threat to the station and
generation. A vulnerability was identified as a single failure that would result in a derate of
>20%, scram, or plant trip. The review only focused on normal 100% power operations. Pump
shutdowns and start ups were not included for this review. OE was used to help confirm a
7
vulnerability and to provide supporting documentation for the corrective action. The review
considered vulnerabilities for elimination or mitigating strategies. Elimination was the preferred
method but some components are required to protect the equipment and have no alternate
methods for maintaining the protective functions. Vulnerabilities of this nature can be mitigated
with PMs or additional margin in procedures or improved guidance. Some items already have
PMs and can not be eliminated and are mitigated to the full extent possible. These items will be
mentioned but may not include additional corrective actions.
6.1
RFP suction side vulnerabilities and enhancements
The investigation for the suction side of the RFP identified several vulnerabilities and
enhancements that can be implemented to improve reliability. Areas for improvement include the
RFP suction valve trip function and the RFP low suction pressure trip. These improvements
included recommendations from the SCRAM frequency reduction report in addition to polling the
industry for feedback.
6.1.1
RFP suction valve position 1/1 logic (vulnerability)
Each RFP is equipped with a suction valve H1AD –AD-HV-1781A (B,C). The valves are
operated from the control room with a push button. These valves are located in the turbine
building mezzanine on 153ft elevation upstream of the RFPs. These valves function to isolate the
RFPs during shutdown. The valve is a 20 inch motor operated gate valve and manufactured by
Pacific Valve Manufacturing. The valve is normally operated in the 100% open position. When
the valve is not 100% open, logic interlocks will trip the RFPT.
The team reviewed the valve components and determined the SPV threat was related to failures
of the switch contact, breaker control, power fuse, bailey module failure, or the breaker opening.
The switch logic is only a single 1/1 permissive with no time delay function. Therefore this trip
function will be considered a single point of vulnerability and evaluated for elimination.
The team considered the possibility of the valve drifting and actually changing position.
However, the valve is a motor operated valve and the team has found nothing to support failures
of this nature.
The team determined this trip function was intended to protect the pump from a low suction
pressure condition and pump cavitations. If the suction valve was closed via a pushbutton, there
8
would be a low suction pressure condition and it would damage the pump internals. However,
this valve is a motor operated gate valve and incidental closure was determined to be unlikely.
Each RFP is already equipped with a low suction pressure trip. The team determined the actual
low suction pressure trip provides more equipment protection than the suction valve not 100%
open trip. The team determined that the RFP suction valve not 100% open trip can be removed
and, replaced by an alarm.
6.1.2
A,B,C RFP low suction pressure trip (enhancement)
The team considered input from the BWROG SCRAM Frequency Reduction committee to
stagger the low suction pressure trips. The SFRC recommendation #16 is to stagger the low RFP
suction pressure trips. The action can be to either stagger the suction pressure set points or to
stagger the time delays until the trip actuates. Hope Creek has time delays of 10 seconds for each
RFP suction pressure trip; however the setpoints for all three pumps are set at 230PSIG. The
recommendation directs installation of time delays and either stagger the low suction pressure trip
set points or stagger the time delays.
However, the intent of this recommendation “is to avoid simultaneous trip of all feed pumps
when a single pump trip, or no trip at all, would suffice.”
With the pump tripping logic scheme described in Hope Creek’s response and review of Lesson
Plan NOH01MNCONDC-05 ,Condensate System, the Team has determined that the intent of this
recommendation is met because the designed intent of this logic is meant to keep as many RFPs
running as the condensate system can support.
Staggering these trips to prevent all 3 RFPs from tripping simultaneously should be implemented
at Hope Creek. The recommendation to change the time delay should not be implemented. In the
event a low suction pressure condition actually exists, the RFP could experience this damaging
condition for an extended period of time. The protective function of this trip is to stop the RFP in
the event a low suction pressure condition exists. This function can not be performed as it was
intended if the time delay is increased too long. If the trip set points are changed to a lesser time
delay, the margin of the time delay is reduced and more likely to trip under a false pretense.
Staggering the trip setpoints would be more effective. The current set point is 230 psig and the
feed pumps normally operate 350 psig. The suction pressures can be raised by 10 and 20 psig and
9
leaving the third RFP suction pressure as-is. This would satisfy the SFRC recommendation to
stagger the suction pressure trips to prevent a simultaneous trip of all 3 RFPs. Margin would only
be slightly reduced and the protective feature would not be challenged or lost.
6.2
RFP and discharge vulnerabilities and enhancements
This portion of the vulnerability review documented vulnerabilities found related to the pump.
The vulnerabilities for this section involved the vibration monitoring equipment
6.2.1
Vibration monitoring equipment (Vulnerability)
The RFP vibration monitoring equipment used to monitor RFP vibrations is a General Electric
Bentley Nevada 3500 model. On the pump side of the feedwater system, the vibration data
provides alarms and indication only. There are no vulnerabilities due to logic trip outputs from
the monitoring equipment.
Vulnerability exists due to the potential for Ops alarm response to cause an inadvertent reduction
in RFP speed to reduce vibrations. Ops procedures HC.OP-AR.ZZ-0028 and HC.OP-AR.ZZ0022 include guidance to response to axial vibration experience on the RFP and journal bearings.
10
11
The procedures instruct operators to reduce RFP speeds to maintain vibrations below the danger
levels. This practice is supported to protect RFP internal components. If the vibration data is false
indication, then operators will reduce RFP speed inadvertently. In several occurrences a probe has
failed resulting in a spike in vibration indication. The chart below shows the data that operators
observed shortly after a probe failed.
12
The team determined that an enhancement is required for operations to verify vibrations before
taking action to remove the RFP. The attached graph above was one example of how a vibration
probe can suddenly fail and show a sudden increase in vibrations. The actions should be a quick
verification check that can be done immediately. A lengthy evaluation can result in damage to the
pump internals if there is a real vibration. If the vibrations are due to false indication, operations
could derate the plant when it is not necessary.
Bearing temperature is a close indication of bearing vibrations. Sudden changes and rises in
bearing temperature are likely due to increased vibrations. Bearing temperature can be verified
quickly in the control room using CRIDS. If there is a sudden change in bearing temperature or a
rising trend in bearing temperature, the operators should remove/reduce the respective RFP to
prevent excessive damage. However if the bearing temperatures appear unchanged and unaffected
by the vibrations, then the RFP should stay in service at the respective speed.
6.2.2
RFP discharge Check valves (No vulnerability)
The team considered the discharge check valves for potential vulnerability. However the only
vulnerability that these valves present is during shut down and start up. A review of maintenance
procedures indicates an adequate preventative maintenance plan is in place. No vulnerability
exists for these valves within the scope of this investigation.
13
6.2.3
RFP minflow recirc line to the condenser (No vulnerability)
The minflow recirc valve was originally considered to be vulnerability. The minflow valve fails
open on a loss of air and a single failed air line would result in the minflow valve failing open.
The theory was if the valve failed open it would cause the full 5500 gpm flow to suddenly flow to
the condenser from the reactor. The sudden impact to operations would cause a low level in the
reactor forcing a derate to restore level. If level could not be maintained by a reactor derate, the
feed pumps would speed up and eventually trip on overspeed forcing a derate greater than 20%.
This failure was long thought to be a vulnerability and would cause a plant trip similar to INPO
document IER 11-10.IER 11-10, which referenced a minflow valve failing open at Palo Verde
plant while at 100% power. Palo Verde and Hope Creek differ since Palo Verde is a two
feedpump plant while Hope Creek has three feedpumps. In the IER, SONGS minflow valve failed
open diverting flow from the reactor. Inventory dropped in the reactor causing the speed
controller for the two pumps to speed up. As flow speeds up the pump with the failed minflow
valve tripped on low suction pressure, and the remaining pump could not maintain level.
The simulator refuted this theory. In the simulator, the minflow valve was failed open while at
100% reactor power. Level drops and the two other pumps speed up to compensate for the lost
flow and falling level. During this test, reactor level dropped to level 4 with a minimum of 25
inches with no scram, power reduction, or runbacks. Manual Operator action to SCRAM would
not be taken since a reactor level of 25 inches is above the procedural direction to SCRAM at 15
inches. The final outcome is all pump speeds increased to 5450 rpm (14.9 Kgpm) to compensate
for loss of flow to minflow line. The affected pump then only supplies 8.4 Kgpm.
6.2.4
RFP high-high discharge pressures (No vulnerability)
It was brought up during the study that there might be a vulnerability on the high-high discharge
pressure sensors, due to the fact that they are all connected to a single header, which could cause
all 3 to get a false reading from their common header, but this was dismissed due to the fact that a
shearing or disturbance of the header would cause a lower pressure reading than true, and would
not cause a trip.
14
The purpose for this trip was to prevent over pressurization in the event the Start Up Level
Control Valve (SULCV) fails closed during unit start up. All three of the pumps receive a trip
since any pump of the three may be in service during SULCV operation.
6.2.5
Feed pump discharge piping and bypass lines (Enhancement / No
vulnerability)
The team reviewed discharge piping of the RFPs up to the #6 Feedwater heaters. Specific
components and vulnerabilities reviewed in this section include the RFP minflow recirc line that
redirects water to the main condensers the start up level control valve (SULCV), and the RFP
high-high discharge pressure system. The team was able to utilize the field walkdown results for
further review of physical plant conditions, and the simulator to verify plant response to
transients.
The SULCV fails closed on a loss of air so in the event an air line fails the valve will remain
closed and have no impact on 100% power operations. Therefore the SULCV will not be
considered a SPV. During past outages the SULCV challenged plant shutdown and start up when
the air line failed requiring emergent work. The valve should be enhanced to prevent future
complications.
6.2.5.1 Start up level control valve (Reliability Enhancement)
The SULCV is a 12 inch air operated drag valve manufactured by “Control Components Inc”.
The valve uses a series of steel airlines and boosters to move the valve as required. The valve
fails closed on a loss of air. A demand signal is generated via a digital positioner manufactured by
Fisher. The valve is rated for up to 14000 gpm of water at 370 degrees F, and 1180 psig. The
valve is also equipped with an isolation valve that allows for maintenance and performing a
function check activity.
The valve is normally only used during shut down and start up. The SULCV function is to bypass
feedwater around the #6 FWH and control level in the reactor. The valve’s demand signal is
based solely on level upsets from the level set point. The valve normally controls level in
automatic control during power start up from depressurized up to approximately 15% reactor
power. During shut down the SULCV is called to service during plant cooldown. A valve failure
15
would result in challenges to an outage or planned shutdown, but not to 100% normal power
operations.
The team identified a reliability issue during the system walkdown. The air lines are steel tubing
and were vibrating due to vibrations resonating from nearby structures. A full operating cycle
with the connections and tubing vibrating puts stress on the tubing and fittings which could result
in an air line break. The air line break would have no impact on generation during normal modes
of operation. However during shut down, the valve would be unavailable for plant shutdown. On
start up, the valve would not function and would delay the start up.
Internal Operating experience has shown metal tubing is subject to failure if proper measures are
not taken to address the vibrations. In some instances, maintenance practices are to form a spiral
pattern with the tubing so the vibrating tubing won’t put stress on the connections and fittings.
Another approach has been to install flex fit tubing. Both approaches have been successfully
implemented throughout the feedwater system for the minflow lines and the seal water injection
lines.
16
The steel tubing should be replaced with flex fit tubing. This item was written up as part of the
walkdown but will not be considered an SPV. The SULCV will fail closed on a loss of air and has
no impact to full power operations. Since this valve is not in service during normal power
operations it was not considered a SCRAM or derate single point of vulnerability.
6.3
Feed pump turbine vulnerabilities:
The team reviewed the RFP Turbines (RFPTs) for any potential vulnerability. The team
determined this portion of the Feedwater system would benefit from additional PMs and design
changes. Some of the components do not have PMs and are subject to age related failures. The
vibration monitoring equipment would benefit from a design change to address a single point of
vulnerability due to a single sensor failure leading to a trip.
17
6.3.1
Thrust bearing probe and wear detector (vulnerability):
The team reviewed the vibration monitoring installation. It is a GE Bentley Nevada 3500
installation and was installed during RF15 in spring of 2009. The thrust bearing probes monitor
the turbines’ axial position. The shaft’s position is fed into the ‘thrust bearing wear’ detector
which alarms when the thrust exceeds the set point. If the shaft thrusts much further it will reach
the danger set point. At this time the RFPT is automatically tripped.
The thrust bearing installation is configured such that the thrust collar will thrust into the thrust
shoes and achieve a medium and will maintain a thrust -5 to -10 mils from the 0 mils position.
The trip is set for -30 mils. After a refuel the turbine is thrust against the shoes and is considered
‘zero’ed’.
The purpose for this logic is to provide protection to the RFPT rotating to station subcomponents.
Vendor information has determined that the thrust required for rotating to station subcomponent
interaction is far less for turbine components than pump side components. This is why the RFP
does not have a thrust bearing wear trip. The turbine rotating to stationary parts is approximately
65 mils to 75 mils. For the pump thrust, the clearance is 250 to 350 mils. The turbine rotating to
18
stationary parts interaction will occur in the turbine before it does on the pump. This is why the
thrust trip is on the turbine thrust and only alarms for the pump.
Vulnerability exists for this component since there is a single thrust probe installed for this trip
logic. A probe failure or shift will actuate a trip of the RFPT on ‘thrust bearing wear detector
Danger limits’. Internal operating experience has shown the vibration probes have failed before.
The probes that have failed before had no dependent trip logic but were for alarm and indications
only. The RFPT will trip if the thrust bearing probe were to fail.
The best strategy to address this vulnerability is a design change for elimination. PMs are
performed to mitigate the vulnerability; however the risk is still present and should be addressed
with a design change. The change should install a second thrust probe and modify the trip logic.
The trip should be converted to a 2/2 logic so both probes would have to fail to actuate the trip.
This will eliminate the 1/1 only logic.
Reference (498-950124-1) for a single thrust probe failure that resulted in a unit trip. The design
was similar to Hope Creek since there was a single probe providing indication. The false
indication drove Operators to reduce feedpump speed and trip the pump.
6.3.2
Feed turbine trip on a loss of trips system power (Vulnerability)
The team review of the RFPT trip logic determined that the RFPT will trip on a loss of power to
the trip system. The trip is integrated in the oil delivery system. On a loss of power, the 2
solenoids will open and redirect control oil from the control valve to the reservoir, closing the
RFPT control valves and tripping the turbine.
These solenoids are closed while energized, with 2 fuses supplying power. If either of these
fuses, or the power source is lost, the RFPT trip solenoid valves will fail open and dump oil back
to the reservoir tripping the RFPT. A single solenoid valve failure will also result in a RFPT trip.
The vulnerability exists on a loss of power or a fuse failure. Without power to the trip system the
pump will automatically trip and may not be the desirable outcome. Also if the power to the trip
solenoids is interrupted momentarily, they will lift resulting in a RFPT trip on low control oil
pressure. There are two solenoid valves and both should be removed while installing an alarm
that will alert the control room if power is lost instead of automatically tripping the turbine.
19
Manual trip capability is still available locally, giving operations the ability to trip the turbine
when necessary
The Following Drawing shows the normal flow path for hydraulic control oil and the loss of
power trip solenoid valves.
20
6.3.3
Include the rupture disc on the RFPT overhaul
The team reviewed other RFPT components and indentified the RFPT steam rupture disc as a
single point of vulnerability. Exhaust steam from each RFPT is directed to the main condenser
via a 60-inch header. These headers are each equipped with a breakable diaphragm which
ruptures at 5 psig pressure. These discs were supplied by the Delaval manufacturer. The purpose
for these diaphragms is to protect the expansion joints from over pressurizing.
21
The plant is vulnerable to a contaminated steam release into the turbine building if these rupture
disks fail. Failure modes of these rupture diaphragms include, but are not limited to; Cycle
fatigue, excessive pressure application of sealing steam to the RFPT, and filling of the RFPT
exhaust line isolation valve water seal. These discs can not be eliminated from the plant because
they provide a protective function to the RFPT expansion joints.
A review of the RFPT overhaul activity determined that there are no actions to inspect or to
replace the rupture disc. There is a note in the procedure that directs maintenance technicians to
inspect the rupture disc for any signs of wear or fatigue. An inspection of the surface will not
detect failure or wear from the inside of the steam header. Since elimination can not be performed
via a design change, the best strategy is to mitigate the vulnerability and replace it as part of the
RFPT overhaul. Replacing this component will prevent age related failures from occurring.
Reference (390-960428-1): An automatic reactor scram occurred at Watts Bar following loss of
both main feedwater pumps (MFP). A leaking valve on the B MFP train caused a MFP turbine
condenser rupture disk actuation. The actuation caused the loss of vacuum which tripped the A
pump and resulted in the SCRAM. The turbine automatically tripped, followed by a reactor trip.
The OE did not reference if the steam leak was directly responsible for the rupture disc actuation.
At Hope Creek there is no action to replace these discs. After reviewing this OE, it is apparent
that a rupture disc failure will cause a loss of condenser vacuum and will force a derate. These
components should be added to the Hope Creek RFPT overhaul.
6.3.4
Feed pump turbine steam exhaust bellow (vulnerability)
The team determined the expansion joints on the RFPT exhaust were a vulnerability. Each main
feedpump turbine is equipped with exhaust bellows to the main condenser.
Industry OPEX suggests that bellows type arrangements in steam exhaust or bleed systems have
failed prematurely. For example reference OE27323: During power operation rising dissolved
oxygen levels and declining main condenser vacuum were detected. A Steam Generator Feed
Pump Turbine (SGFPT) exhaust bellows developed leakage that required a downpower to repair.
The cause was determined to be age, vibration and high cycle fatigue related.
The team determined PMs were required to prevent a sudden failure such as the one referenced in
the OE. The PM strategy will mitigate the vulnerability but can not eliminate the vulnerability.
22
6.3.5
Feed pump turbine first stage moisture removal drain valve (vulnerability)
The team reviewed the RFPT first stage moisture removal drain valve, equipped on each RFPT.
Each valve is a Velan manufactured 6-inch motor-operated gate valve that drains to the main
condenser. The valves are operated from Control Room panel 10C651A, using the
OPEN/CLOSE, momentary contact push-button provided for each valve. These provide isolation
for the RFPT moisture drain line.
These valves are vulnerable due to trip logic when not fully open. The trip is a 1/1 contact switch
engaged when the valve is not 100% open. The team reviewed the valve components and
determined the SPV threat was related to failures of the switch contact, bailey module, power
fuse, or the breaker control failure. The logic is only a single 1/1 permissive with no time delay
function. Therefore this trip function will be considered a single point of vulnerability and
evaluated for elimination.
The basis for this trip is to prevent the drain valve from closing and allowing moisture to fill the
drain line and damage the RFPT blades. According to Salem station and other industry peers, this
trip is nonexistent and not required. Indication for valve position is maintained to ensure it does
not go closed, isolating the drain line. The team polled the industry peers and determined this trip
is not required. None of the responding plants with turbine driven feedpumps have this trip.
Without a strong basis for this trip’s function and purpose, the team determined the trip should be
eliminated. The trip is also a 1/1 logic and makes the station vulnerable. The team determined the
most favorable strategy is to eliminate the trip logic entirely. If the logic provided a more
reasonable protective function, the team would have considered a different strategy. Since there is
minimal basis and industry disposition for this trip to remain, a design change will be presented to
remove the trip.
6.4
Feed pump and turbine control and bearing oil package
The Lube oil system and delivery was reviewed by the team for vulnerabilities. The system is
comprised of 3 independent trains responsible for delivering lubricating oil to the pump and
turbine bearings. The oil is also used as hydraulic control oil for the steam inlet valves. Each train
is comprised of 2 AC driven oil pumps and an emergency DC driven oil pump. The emergency
oil pump provides oil directly to the bearings and only the bearings.
23
6.4.1
SSPV- Oil system design and electrical lineup (vulnerability)
The station’s oil system has a legacy design issue is due to a combination oil pump power supply
lineup and the design of the oil pump delivery system. Hope Creek has been subject to 2
SCRAMs in 2003 and 2007 due to the RFPT design issue that occurs when automatically
swapping oil pumps and the electrical configuration with 2 of the oil pumps on the same 480V
power supply. The cause of the RFPT trips is that the oil system design is not adequate to assure
that the standby lube oil pump will start and achieve operating pressure on loss of the operating
oil pump. This is an original equipment manufacturer design deficiency related to system margin.
Other contributing factors are due to the oil system being placed on and elevation 17 ft below the
turbine further reducing pressure margin.
6.4.1.1
SSPV oil pump electrical line up (vulnerability)
During the 2 SCRAMs an unexpected slow transfer of a 4 kV Class 1-E bus from the normal to
alternate source occurred during monthly relay testing. The slow transfer and subsequent loss of a
non-safety related motor control center resulted in the loss of an MCC set with 2 I/S oil pump
power supplies. The power supplies are configured as follows:
Pump
480V bus
4160V Bus
A1
10-B-323
“B” channel 4160 1E
A2
10-B-272
“C” Channel 4160 1E
B1
10-B-323
“B” channel 4160 1E
B2
10-B-313
“A” Channel 4160 1E
C1
10-B-272
“C” Channel 4160 1E
C2
10-B-323
“B” channel 4160 1E
The following line up must be used:
 A1P124, B2P124 and C1P124
 A2P124, B2P124 and C2P124
Procedure HC.OP-SO.AE-0001 contains the following operator work around to prevent making
the plant vulnerable to a Single SCRAM Point of Vulnerability (SSPV):
24
NOTE:
To prevent a single 4KV bus failure causing more than one RFPT to trip due to
loss of an oil pump, the preferred lineup for the pumps should be:
A1P124, B2P124 and C1P124 in service
OR
A2P124, B2P124 and C2P124 in service
If this lined up is not used, the station is at risk of a single 4KV bus failure resulting in a
SCRAM. Both of the 2003 and 2007 SCRAMs occurred while the oil pumps were not in the
listed lineup. A slow power transfer occurred resulting in a 4KV bus trip. Two oil pumps tripped
due to the loss of power. After the 2007 SCRAM, the station implemented the corrective action to
operate with the listed pump lineup. This eliminated the SSPV for a SCRAM but did not
eliminate the SPV.
6.4.1.2
Design SPV for the Oil system swaps:
The station is still vulnerable to a design issue for the oil system. The oil system is designed to
allow, in the event of an oil pump trip, a transfer of pumps without a drop in oil pressure through
the implementation of an accumulator. After the 2007 SCRAM the oil accumulator was found
dry of oil after an inability to maintain oil pressure. Therefore, when an oil pump tripped, the
back up oil pump could not start fast enough to maintain header pressure. When pressure dropped
below the trip set point, the RFPT tripped on low control oil pressure. The loss of a feed pump
resulted in a unit derate. The vulnerability was determined to be due to undersized oil
accumulators and the auto start logic being pressure based and not instantaneous.
RFPT Control Oil Accumulators
The existing control oil accumulators are 10 gallon bladder accumulators that provide a reserve
supply of control oil to mitigate the effects of any hydraulic transients such as a sudden increase
in control oil pressure or a sudden decrease in control oil pressure. The accumulator is designed
only for a momentary change in pressure. During the past plant events when an oil pump trips,
the accumulator does not have a large enough reserve to compensate for the momentary loss of
oil pressure. After both events in 2003 and 2007, pressure dropped below the RFPT low control
oil pressure trip setpoint. The accumulators were found empty after the event. In RF17 a DCP
25
will be implanted to remove the 10 gallon accumulators and install two 40 gallon accumulators.
This will allow for maintenance and provide additional margin before the accumulator is fully
drained.
Oil pump auto start vulnerability:
The existing oil pump logic is configured such that the back up oil pump will only start
automatically when a low oil pressure condition is sensed. In the event that the operating pump
trips, header pressure will decay to a point where the standby oil pump will start on low oil
pressure. This auto start feature is not effective for addressing a sudden oil pump trip. A review of
header pressure data
26
- Item 1 is the control oil pressure header designed to start the back up oil pump at 100 psig and
trip the RFPT when pressure falls below 60 psig.
- Item 2 is bearing oil pressure and is designed to start the back up oil pump at 8 psig and trip the
RFPT when pressure falls below 5 psig. Both trips are 2/3 logics and the bearing oil trip has a 3
second time delay.
- Item 3 is the Main oil pump breaker going in the open position
- Item 4 is the back up oil pump breaker going in the closed position.
This data confirms that when the oil pump tripped there was a period that both oil pump breakers
are open allowing header pressure to further decay. The delay was not due to the breaker but
because oil header pressure decay is not linear and remains above the set point not clearing the
permissive for the standby oil pump to start. When pressure decayed to the start set point, oil
pressure decay rate had significantly increased, and the back up pump was not able to make up
for the lost oil pressure before reaching the RFPT trip set point.
6.4.1.3
DCP (80102874) to eliminate SSPV and oil pump swap SPV
A DCP has been issued for installation refueling outage RF17 in spring of 2012. Installation of
this DCP will eliminate the vulnerability caused by undersized oil accumulators and auto start
logic:
RFPT Oil Pump Auto Start
The Hope Creek Reactor Feed Pump Turbines (RFPTs) each have two oil pumps, one normally
operating (main) and one on standby (auxiliary), that provide high pressure oil to the control oil
system and low pressure oil to the lube oil system. On low lube oil or low control oil pressure,
the standby oil pump will start if in the automatic mode. This auto start will not prevent an RFPT
trip if the lube oil pressure falls below 5 psig. This (DCP) modifies the auto start feature of the
RFPT main and auxiliary oil pumps so that the standby pump auto starts on a trip of the operating
pump or on low lube/control oil pressure.
RFPT Control Oil Accumulators
The existing 10 gallon RFPT control oil accumulators do not provide a large enough reserve
volume of pressurized control oil to provide adequate protection in the event of low control oil
conditions. This DCP replaces each existing 10 gallon RFPT control oil accumulator with two 40
27
gallon accumulators to provide a larger reserve of control oil which increases the margin for
recovering from hydraulic transients with no adverse impact to the control oil system.
6.4.2
Lube oil pumps
The RFPT main and auxiliary oil pumps are DeLaval, 25 hp, positive displacement, submerged
suction pumps that provide high pressure oil to the RFPT hydraulic controls and low pressure oil
to the RFP bearings and the RFPT bearings. Both of these pumps are rotary vertical pump. The
flow path of the oil starts at the reservoir and is discharged through a discharge check valve,
where it combines with the oil discharged from the other pump (main or aux). The oil then goes
through a duplex filter and dual oil coolers used to regulate oil temperatures.
The normal control oil pressure supplied by these pumps is 125 psig. The normal bearing pressure
discharge is between 15-20 psig. Normally, one pump is in service (main) with the second pump
in stand by (Aux). The pumps are equipped with an auto start feature that auto starts the Aux oil
pump if the control oil header pressure drops below 100psig or the bearing pressure drops below
8 psig. The emergency oil pump will start when pressure drops below 7 psig. The purpose of the
emergency oil pump is to protect the bearings from a loss of oil.
During the review, the team identified that the oil pumps had no preventative maintenance
activities scheduled. The pumps are classified as non-critical and should have a preventative
maintenance strategy. Vibrations are taken on these oil pumps quarterly and have shown a
degrading trend. Recently, the B oil pump was taken to the ‘restricted use’ category due to
elevated vibration trends. The aux oil pump was placed in service until the following refuel
outage when the pump is scheduled to be rebuilt. With the one pump in a degraded condition, the
RFPT is at risk. If the running oil pump stops, the station will be forced to operate with a
degraded pump.
A preventative maintenance strategy will be implemented to rebuild these oil pumps. The pumps
can be rebuilt while online using a spare pump and ordering new parts as required. This activity
can be performed while online. Then the pumps can be changed during the next outage.
6.4.2.1 Oil pump discharge relief valves:
The team reviewed the oil pump discharge relief valves. These valves are installed on both the
main and the aux oil pumps. The relief valves are 4 inch Fulflo pilot valves, designed to lift when
28
pressure reaches 170 psig. Their function is to prevent over pressurizing the oil pump discharge
line and dead heading the oil pump.
During the OE review, the team identified instances where relief valves lifted and resulted in a
sudden drop in header pressure. Relief valves’ internal components such as the spring can
degrade to the point that the relief valve will lift at a lower pressure than expected. If the valve
goes without inspection and verification that the valve will lift at the required pressure, the
internals could be degrading without any indication until it actually starts to prematurely lift.
The team determined that these relief valves require a preventative maintenance strategy to
mitigate the vulnerability. Elimination of these relief valves is not a favorable strategy since they
provide a protective feature for the oil pumps. A mitigation strategy is preferred to verify these
relief valves and internals will function as they are expected to. The springs must be verified that
they will lift at the required set point and not at a lower set point.
6.4.2.2
Oil pump discharge check valves (Vulnerability):
The team reviewed the drawings of the oil system and determined there were check valves on
both of the oil pumps discharge piping. These check valves were installed to prevent back flow of
the oil through the out of service oil pump, which would result in a loss of oil pressure and a
RFPT trip.
Industry OE has identified several failure modes for check valves. Check valves stick open and
the spring fails to the close the valve resulting in oil flowing back to the reservoir and tripping on
low oil pressure. Another failure mode is if the valve internals connection fails and the valve disc
breaks away. A return line would open and direct oil back to the reservoir.
Reference OE (341-020519-1 and SER 27-87) for events that caused a loss of oil pressure due to
oil check valves failing to close when the pumps were swapped. This caused a sudden drop in oil
header pressure resulting in the feedpump tripping on low oil pressure.
During the review, the team determined that the check valves for the oil pumps do not have
functional locations or preventative maintenance strategies. Immediate action was taken and a
notification was generated to create functional locations for the listed check valves. Once these
29
valves have functional locations they will be classified as ‘critical mild environment and mild
duty’. Actions from this review will also include implementing a preventative maintenance
strategy to perform check valve rebuilds and inspections.
6.4.2.3
Oil reservoir vapor extractor (Vulnerability)
The oil reservoirs have been equipped with a vapor extractor. This device functions to create a
vacuum within the reservoir cavity by applying suction to the reservoir. Oil fumes are extracted
through this device and out of the turbine building through a vent. This is a safety function that
must be maintained to sustain a habitable reservoir room. The extractor also provides the driving
force for draining oil back to the reservoir by applying a vacuum to the reservoir. This vacuum
force is intended to keep oil from leaking from the joints and seals.
Internal experience with the vapor extractor has proven it to be vulnerable to failure. In May of
2009, the vapor extractor structurally collapsed and failed. Shortly after the extractor failed,, the
harmful vapors accumulated in the oil room, creating a thin could of oil vapor in the air that
presented a fire hazard. Oil leaks from the feed pump and turbine became additional effects of the
extractor failure. Other OE expressed how oil leaks, if left not mitigated can accumulate. If the oil
accumulation is near any fire source, the oil can ignite.
To mitigate the May 2009 vapor extractor failure, maintenance technicians had to configure
blower and hose to exhaust harmful fumes to the turbine building truck bay. The extractor was
replaced but needs scheduled action to prevent the extractor from failing again. The risk was
rather low since there was no challenge to generation. Therefore a preventative maintenance
strategy would be the best strategy.
Reference OE 31000 and the Salem SPV review for how a vapor extractor can reduce oil leaks
and how oil leaks can result in a fire. OE 31000 confirms the risk of fire for leaking oil and is
important for the vapor extractor to perform without allowing degraded vacuum. In OE 31000 a
fire started due to oil leak ignition. The station allowed oil to drip and accumulate to the point
where it contacted a hot surface. The oil atomized and ignited burning the surrounded spilt oil.
The fire caused a SCRAM. The vapor extractor is designed to maintain vacuum on the reservoir
and be the driving force to direct oil back to the reservoir. If the vapor extractor fails or can not
perform its function the oil will be more likely to leak. Salem recently worked on their vapor
30
extraction line and found the exhaust pipe plugged with debris from the environment. Hope Creek
will add a similar action for a vapor extractor and exhaust line.
6.4.3
Low oil pressure trips with 2/3 logic with 1 sensing line (Vulnerability)
The low oil pressure trips for the oil system are for low bearing oil pressure and low control oil
pressure. Both trips are 2/3 logic and the low bearing oil pressure trip has a time delay to provide
additional margin. The logic and switches are not vulnerabilities, however all three switches use a
single sensing line. A loss of a fitting or connection for these switches could cause a trip. If a
connection or tubing fails, the oil pressure will drop and exceed the low pressure set point.
Changing these pressure trips will be difficult because the cabinet these switches are in is small
and there is not enough room to provide the switches with individual sensing lines.
Reference OE24195. A feedwater Pump tripped on low suction flow. The low suction flow trip
signal was caused by clogging of the common impulse line for three transmitters.
At Hope Creek the oil in this system is filtered and reduces the possibility plugging the oil line
with a piece of FME. However if the line is subject to sludge accumulation and build up similar
to OE33423, the station could be subject to a trip on low oil pressure. OE33423 was for oil sludge
build up in the oil line that resulted in elevated bearing temperatures because line blockage was
preventing oil from reaching the bearing. The blockage was determined to be oil sludge build up
from years without flushing.
The station will evaluate if the switches can support a different configuration and will implement
a PM to flush the system out to prevent sludge build up.
6.4.4
Oil delivery lines and sludge accumulation
The team reviewed the oil delivery system and piping and determined a vulnerability is possible
due to sludge accumulation. Oil sludge build up is an aging issue and can not be precluded with
FME controls. If oil sludge accumulation occurs in a smaller section of piping it can cause a
derate or degradation of a component that a derate is required to address the issue.
Reference OE33423 was for oil sludge build up in the oil line that resulted in elevated bearing
temperatures because line blockage was preventing oil from reaching the bearing. The blockage
was determined to be oil sludge build up from years without flushing. During the OE, the oil
31
accumulated in an oil line that fed a pump bearing. As oil sludge accumulated over the years the
bearing temperature slowly rose. However it was not an immediate concern because all
parameters were well within the required limits and the trend was very subtle. The bearing
temperature was addressed during a planned outage for an unrelated issue. During the bearing
inspection blockage was found in the oil line. This blockage was determined to be sludge and
residue accumulation. There was no FME in the reservoir or amongst the residue. Flushing the oil
system is required to maintain proper component lubrication so oil can flow freely through the
system.
6.5
Number 6 feedwater heaters
The 6A, 6B, and 6C feedwater heaters are two zone, horizontal, shell, U-tube heat exchangers.
These high pressure feedwater heaters provide the final stage of feedwater heating before
injection into the reactor vessel. Extraction steam from the 4th stage of the high-pressure turbine
is admitted to the shell side of the heaters where it is condensed as it supplies the heat for the
feedwater. The #6 heaters have shell side relief valves for overpressure protection, and internal
drain coolers. These drain coolers can be bypassed to dump directly into the condenser to
minimize the possibility of heater flooding.
Wide Range Level Transmitters
The wide range level transmitters provide FW Heater level (shell side) signals to electronic bistable alarm cards and wide range indication function to the main control room, local panels
10A/B/C-C102 and CRIDS.
Narrow Range Level Transmitters
Feedwater heaters 6 (A, B and C) each have two narrow range (10”) level transmitters. One of
these transmitters provides input to the normal drain valve level controller and the other provides
input to the emergency/dump level controller for the heater.
6.5.1
Feedwater heater level control (Vulnerability)
The team investigated the 6 FWHs and their ability to control level within the heater shells. Each
Heater is equipped with four level transmitters which perform the following functions:
 One level transmitter/level indicating controller is used to position the normal heater
drain valve.
32
 A second level transmitter/level indicating controller positions the alternate drain valve
to the main condenser.
 The third and fourth level transmitters provide local (panel 1A/B/C C102) and control
room (10C650A) indications, control room alarms, and heater trip functions. The local
heater level indication can be selected to either of these two transmitters.
Reference the following diagram for the level control span of the 6 FWHTR:
 Level is normally controlled using the drain valves up to 8 inches.
 Above 8 inches, the dump valve will being to lift to permit additional drainage directly to the
condenser. The dump valve will continue to lift as level rises until it reaches full open.
33
 High level at 22.5 inches will cause the dump valve to fail fully open allowing maximum
drainage to the condenser.
 High High level trip is at 29 inches. At this level the dump valve is fully open and the heater
will trip. There is a 10 second time delay for this heater that will prevent a momentary false trip
actuation signal from isolating the heater string.
For level control, level is maintained at a set point for optimal performance. If shell level is
higher than required the excess water will reduce the tube surface area exposed to the steam,
which will reduce heat transfer and result in cooler feedwater. Potential water intrusion into the
Main Turbine if Hi Hi level isolations do not occur automatically
Insufficient heater level could result in inadequate subcooling of the condensate which would
lead to flashing as it enters the lower pressure heater shell. This results in erosion of tubes and
other components in the heater and /or the drain lines if the normal level or dump valves are
malfunctioning.
If shell levels start to increase to abnormal levels, the associated alternate drain (dump) valve will
start to open to restore appropriate levels. The dump valve is modulated as a function of the high
level in its respective heater. Dump valve drainage is directed to the main condenser shell instead
of the 5th Feedwater heaters. If heater level reaches the "Hi" setpoint, the alternate drain valve
will open. The level control band for the dump valve is set higher than the level control band for
the normal drain valve. The normal control and high control bands do not overlap. The Hi level
setpoint is above (or near) the top of the high control band.
If the "Hi-Hi" setpoint is reached, the respective train 3-6 heaters will experience the following:
 Extraction steam to that heater is isolated.
 Cascading drain flow from the upstream heater is isolated (level control valve fails
closed
 For the 1 and 2 feedwater heaters there is a 10 second time delay before the isolation to
occur.
Vulnerability exists for the level controllers and transmitters due to location. They are locked in a
high rad area and can only be accessed while the respective feedwater heater is tagged out of
service and cooled. This requires a derate of approximately 20% when planned. If a heater trips
34
unexpectedly the derate will be more than 20%. The transmitters will be placed outside of the
locked high-radiation area during RF17
The team also determined vulnerability exists for the level control transmitters that provide a
signal to modulate the drain valves. Internal OE of the normal level controller failing is
referenced in the timeline as follows:
4/19/05- The normal drain valve for 6A FWH AF-LV-1506A fails to control level. 6A FWH
level is being controlled by dump valve 1505A. Positioner replaced with same model. Failed
Positioner sent to Exelon Power Labs for failure analysis.
5/2/505- Exelon Power Labs concludes the 4/05 Positioner failed from vibration and side loading
of the valve spool by wear on the aluminum bell crank arm.
7/17/05- 1AFLV-1506A fails to control 6A FWH level and dump valve slowly starts to open.
LV-1506A slowly fails closed until fully closed. 6A FWH level maintained by dump valve (LV1505A). LV-1506A Positioner failed due to high vibration per System Engineering inspection
and comparison to 4/05 failure and Exelon Power Labs report. Positioner replaced on 7/23/05
with vibration resistant model.
8/16/05 - Control air tubing to LV-1506A Positioner pulls out of Positioner causing LV-1506A
to fail close. 6A FWH level controlled by dump valve (1AFLV-10505A).
8/21/05 - 6A FWH cascading drain line to 5A FWH observed to be vibrating at the piping run in
the 3/4/5A FWH room on remote video cameras 10. Piping vibration visually measured and
engineering calculation determined displacement is within fatigue limits.
8/22/05 - The normal drain lines for the 6B to 5B FWH & 6C to 5C FWH walked downs and
verified not to be vibrating. Walkdown performed in the 3/4/5-B/C FWH rooms.
8/24/05 - Troubleshooting performed to determine if cycling of LV-1506A is a cause of
vibration. LIC-1506A placed in manual for approx. 5 mins. Piping vibrations did not change.
8/25/05 - 6A FWH level increased by 1.2" to determine if increased sub cooling affects piping
vibration. Increased level maintained for approx 16 hours. Piping vibrations did not change.
35
8/28/05 Forced Outage - DCP added additional pipe support (hanger H06) to the 6A to 5A FWH
drain line to reduce piping vibration observed on 8/21/05.
9/6/05 - Engineering verified 6A FWH drain line is not vibrating as part of the retest for the DCP
that added hanger H06.
9/25/05 - Greater than expected noise was heard from the 3/4/5A FWH room. The remote video
camera showed greater than expected movement of the 6A to 5A FWH normal drain line, and
debris and water on the floor in the location of the 6A to 5A FWH drain line. An entry into the
room confirmed the noise was from hanger H05 on the 6A to 5A FWH drain line, the drain line
had excessive motion, the debris was pipe insulation from the same drain line, and the water was
due to leaks at the MSDT level control valves at the 5A FWH. NOTE The MSDT valves are not
connected to the 6A to 5A FWH drain line.
9/26/05 - System Engineering started the Complex Trouble Shooting process to identify all
possible failure modes and causes for excessive drain piping vibration.
10/4/05 - While at 100% reactor thermal power, the control room crew observed the 6A to 5A
Feedwater Heater (FWH) drain line MOV isolation valve (1AFHV-1508A) OPEN indication
signals were failing. A walk down was performed by operations and maintenance personnel in the
6A FWH room to measure drain piping movement. The piping movement measurements obtained
was approximately 1/8" steady state horizontal and 3/8" occasional horizontal peaks.
10/5/05 - An operational decision was made to remove the 6A FWH from service. After the 6A
FWH was removed from service, an inspection revealed the 6A FWH drain line MOV isolation
valve (1AFHV-1508A) operator hand wheel was found on floor. This MOV internal limit
switches were very loose and rotors damaged. The valve MOV hand wheel and internals were
repaired and returned to service. Vibration instrumentation (accelerometers) was installed on the
piping and valve in 6A FWH room.
10/7/05 - 6A FWH was placed into service using an operational evolution plan to raise overall
heater levels to approximately 17" for piping vibration reduction. The visual indications of
vibration were observed to decrease slightly. The FWH level was returned too normal operating
36
level band. Prior to placing the 6A FWH in service Adverse Condition Monitor (ACM) criteria
with established for piping displacement limits developed by PSEG and an independently
consulting firm on piping integrity analysis. The potential for two-phase flow to cause internal
piping erosion was evaluated. An assessment of NDE results during refueling outage 12 (RF12)
indicates no internal piping damage has occurred. The 6A FWH remained in service, piping
vibration were acceptable IAW ACM.
10/9/05 - The 6A FWH was removed from service due to visual observation of piping vibration
and conservative management decision-making.
10/13/05 - While at 98.6% reactor thermal power an Infrequently Performed Test & Evolution
(IPTE) for placing the 6A FWH in service. The IPTE was performed successfully to raise FWH
levels by 5" increments up to 25" dependent upon overall piping vibration reduction. This test had
vibration-monitoring instrumentation (accelerometers, acoustics, etc.) installed on the piping to
determine the source of the vibration (i.e. FWH, flow in drain, degradation in the FWH drain
cooler, etc.). A 6A FWH extended service decision flowchart was developed to determine the
next actions based on the results of the testing.
10/15/05 - Design Change Package (DCP) was developed and issued to increase the normal water
level into the existing dump valve level region (15" versus normal level of 5") by using the dump
level transmitter LT-1505A to provide the level input signal to both the normal LIC-1506A and
dump LIC-1505A level indicating controllers. The dump level controller setpoint was also
adjusted to actuate slightly higher than the new normal control level. The existing high-level trip
at 22.5" for the solenoid trip of the dump valve and the high FWH isolation at 29" were not
changed. The increased operating levels were also supported by written concurrence by the
vendor (YUBA). Prior to placing the heating into service an Operational and Technical Decision
Making (OTDM) and Adverse Condition Monitoring (ACM) plan were completed with
established FWH removal criteria. The DCP process was used for the level control modification
instead of the temporary modification process to provide higher level of process rigor.
10/18/05 - The 6A FWH was placed back to service at 98.6% reactor thermal power with higher
operating heater levels IAW the DCP. Visual observations of piping IAW ACM vibration levels
were acceptable at this power level. When reactor power was returned to 100%, the visual piping
37
vibration levels in the 6A FWH room were less than the ACM limits. Visual observations
indicated increased vibration in the 3/4/5 "A" room consequent with the power increase.
10/19/05 - The raw accelerometer piping data was analyzed for actual piping displacement. The
"Y" axis data peaked periodically exceeding the current ACM displacement criteria (0.038"
actual vs. ACM limit 0.036") and the "Z" axis data was not present on the tape.
10/20/05 - Data was collected and evaluated. The "Y" axis exceeded the acceptable ACM limit
of 0.036" (actual 0.038" - 0.042") and verified the "Z" axis accelerometer had failed. The 6A
FWH was removed from service IAW approved ACM displacement criteria and failure of a
monitoring method.
While 6A Feedwater heater was removed from service reaching 100% rated power was limited by
current operating procedures.
11/24/05 - Subsequent analysis performed by Westinghouse to allow application of Crossflow
Correction Factor with the 6A FWH OOS (GE Sub case 20051101-0308-1) with the following
stipulations:
•
Crossflow can only be “Applied” in approved configurations of the extraction steam,
heater vents and drains system IAW HC.RE-RA.ZZ-0011.
•
When the configuration of the extraction steam, heater vents and drains system changes,
the Crossflow Correction Factor should be closely monitored and HC.RE-RA.ZZ-0011
referenced to ensure that the configuration is approved.
•
Reactor Engineering should be notified of any Crossflow alarms or significant changes in
the Crossflow Correction Factor (> 0.0050).
Reference OE 395-010501-1 and several internal events. A failed transmitter has resulted in the
level in the FWH rising to the point where the dump valve opens. In the OE referenced, a
transmitter failed and caused a slug of cool water being injected to the core and caused an
unexpected rise in power above 100%. Due to several internal events with failed transmitters
forcing down powers to repair, Hope Creek has set actions in place to upgrade the transmitters
and their configuration. This installation is to be worked under DCP 80103819
38
Past Hope Creek DCPs installed in RF 13 spring 2006
A past DCP restored the normal level transmitter LT-1506A input signal to normal level
controller LIC-1506A, returning the 6A FWH normal level control back to the original band of 010. This DCP also removed accelerometers and cables from the 6A FWH piping on pipe
installed previously..
As a result of the troubleshooting associated with these issues the station has decided to replace
the level control valves in the 6A, 6B and 6C heater drain lines.
•
Flashing in the valve internals of the 6A FWH LCV (1AFLV-1506A) is believed to be
the cause of the unacceptable 6A piping vibrations.
•
The valves will be replaced with a CCI DRAG valves. CCI DRAG valves are designed to
solve this type of problem.
•
The past plan was to replace H1AF –1AFLV-1506A and to remove hanger 1-P-AF-075-
H006 in the future refueling outage RF13. Valves 1 H1AF –1AFLV-1506B & C were replaced
during the following refueling outage RF14.
In 2011, Engineering evaluated a method for placing the level controllers outside of the FWH
rooms so if they began to degrade, they could be repaired without having to derate the plant and
removing the heaters from service to gain access to the heater room. DCP 80103819 has already
been issued and is scheduled for the R17 refueling outage in spring 2012. The team determined
the only required actions for this DCP is to verify it is installed as scheduled.
6.5.1.1 DCP 80103819 to reconfigure the feedwater heater level transmitters
The team reviewed DCP 80103819. This DCP replaces the current Feedwater Heater Masoneilan
Torque Tube Level Transmitters with Rosemount Differential Pressure Level Transmitters for all
four of the 6 FWH level transmitters. The replacement of the Masoneilan transmitters with
Rosemount differential pressure transmitters improves the reliability of the system and has been
implemented previously on the #2 FWH wide range level transmitters. To implement this change,
remote mounted diaphragm seals for the Rosemount Transmitters are bolted to flanges on the
sensing lines. The #6 FWH Rosemount Transmitters will be mounted on the lower process piping
outside of the FWH rooms at Elevation 137’ of the Turbine Building. The piping for the #6 FWH
process lines are routed from inside the rooms through a series of new core bores on the east wall
of the FWH rooms for mounting of the diaphragm seals. The level transmitters are mounted on
39
the exterior of the east wall of the FWH rooms to flanges attached to the process piping. New
valves are included for flushing of the lower sensing lines and venting the process flanges. This
provides access to the #6 FWH Level Transmitters without entry into high radiation areas if
needed. The basic functions of the feedwater heaters and the level transmitters are not changed by
this modification. This DCP only replaces the level transmitters, elements, other associated
components, and their location.
The following table provides the performance for the Masoneilon transmitters:
Table 4.1.5-1 Existing Masoneilon
12127AB Transmitter
Accuracy
Response
Time
Radiation
0.6%
Not Listed
Not Listed
The following table provides the performance for the Rosemount transmitters per
VTD 135818:
Table 4.1.5-2 Replacement
Transmitter Characteristics
Rosemount
3051CD2
(#6 FWH Narrow
and Wide Range)
Accuracy
Response
Time
Radiation
0.1%
100 ms**
None
The team has determined that this DCP will improve reliability of the level control function as
well as allow access to the level transmitters while online. This DCP will eliminate the
40
vulnerability that requires the plant be derated to gain access to address a failing controller. The
dump valve is still available to provide protection high-high level trips. Therefore the single point
of vulnerability for the 6 FWH will be eliminated.
6.5.1.2
LTAM item H-11-0057 Upgrade FWH WR Level Trip and Indication Circuits
The team reviewed additional initiatives to improve FWH level control. This initiative is
currently in the LTAM database and does not have a DCP. The initiative has been approved by
PHC to perform a conceptual study and is targeted to start in refueling outage RF18. The design
change will replace the Westinghouse 7500 electronic signal condition cards and alarm cards for
the Feedwater heater control Panels 1A-C-102, 1B-C-102 and 1B-C-102. The replacement will
consist of 72 cards, 24 cards per panel. It was proposed to utilize OTEK Model HQ-114 Digital
Programmable Intelligent Controllers (IC) or Foxboro 762 digital controllers. The intent is to
have each IC provide power, trip, alarm and indication for each wide range level transmitters.
The conceptual design for this change is currently in progress.
Reference OE26586, False Control Room Annunciation from OTEK HI-Q2000 Instruments. St.
Lucie Unit 1 and 2 utilizes over 300 OTEK indicators for measurement and display of plant
parameters. Several months after installation false alarms were occurring on the plant
Annunciator system. The Annunciator system uses 125vdc that interfaces with the OTEK output
alarm relay contact. Late in 2007, 3 more failures occurred with the OTEK meters output relays,
associated with circuits to small Agastat relays. The failure was increased circuit open contact
resistance with the OTEK alarm relays made by HANDOUK. The station corrective action
required them to replace the indicator output relays with a more suitable relay. The proposed
design uses Otek modules with dry relay contacts rated for 10A at 30VDC/240VAC. Otek rating
meets and exceeds the 2A rating for current 7500 card relays. Suitability with in panel repeater
relays will be checked in the DCP process.
Hope Creek’s design change proposal will address aging and obsolescence and mitigate the
vulnerability. This item still requires the concept study and PRC approval and is intended to be
implemented in RF18. The team determined the only action is to ensure the issue is approved by
PRC and to track it to implementation.
41
6.6
Operations feedback
The team considered the input from SROs for vulnerabilities they encounter while operating the
plant. No additional items have been added from Operation input. These items are required to be
reviewed quarterly for system health report inclusion. All of these issues are already being track
and have action for mitigation and elimination via the normal procedure. Reference procedure
ER-AA-2002 for system health reporting. The following excerpt was provided by operations for
this evaluation:

Components Off-Normal Report (includes interfacing systems)
o
Reviewed off-normal and off-normal tagged reports (run date 09/16/11) with the
following components off-normal or tagged:

Off-normal not tagged NONE

Off-normal tagged  The following components are tagged per HC.OPIO.ZZ-0003


H1AE -52-252073 MOV AE-HVF011A FDWTR INL S/O

H1AE -52-264062 MOV AE-HVF011B FDWTR INL S/O

H1AE -HS-AE-F011A FW SPLY LN A HV-F011A C/SW

H1AE -HS-AE-F011B FW SPLY LN A HV-F011B C/SW
Operator Burdens Assessment Quarterly Report
For operator burdens, the issue is tracked in the system health report until completion. The
issue can not be removed until Operator screening clears it from the burden report. None of
the issues on the operator burden assessment are a threat to the feedwater system and
continued 100% reactor power operations.
o
Reviewed quarterly operator burden assessment report for 2nd quarter 2011:

The AE and AD systems were both categorized as LOW and showed no
change from the previous quarter.

Items identified as a burden that have an impact on the AE system were:

20483568 SULCV PDS Communication Failures (When in
MANUAL, a SULCV PDS communication failure will result in
closure of the SULCV (ODTM HC-2010-0011). MANUAL
operation of the SULCV will be minimized and training has been
42
provided to the operators. This condition will be corrected in
RF17).
a) Relation to AE system – SULCV is the primary
flowpath of feedwater to the vessel when power is <20%

20468349 Low Condenser Vacuum margin during hot weather
(During hot weather, increased monitoring of condenser
backpressure, CDI temperature, and SJAE performance may
result in the need to de-rate to ensure the heat input to the
condenser does not exceed the capacity of the cooling tower.
Operating procedures contain enhanced monitoring guidance.
The Condenser Backpressure alarm has been re-evaluated and
raised to 5.7” HgA which has greatly reduced the required power
maneuvers. This item is on the Engineering margin management
list.)
a) Relation to AE - RFP’s trip at 10”HgA.

20510973 Service Air compressor discharge Check valves
(Service Air Compressor discharge check valve failures have
caused Air Header transients and required entry into Abnormal
Procedure HC.OP-AB.COMP-0001. EQACE 70124136 was
completed 06/16/11 and was presented to CARB 07/11/2011.
Field order 600973714 is scheduled 11/14/2012
a) Relation to AE – RFP min flow valves fail open on loss
of air. SULCV fails closed on loss of air.

20461800 Excess CRIDS alarms (Continuously flashing CRIDS
alarms have potential to make identification of a real alarm more
difficult. This will be corrected by the CRIDS upgrade project –
due to complete 10/31/11
a) Relation to AE – As states above, continuous alarm
mask real issues.

TEMP LOGS – Reviewed the current list of temp logs. The following temp logs have
an impact on the AE system:
o
Temp log 11-069 1BD483 / 1DD483 Inverters
43

CRIDS pt D4980 locked in alarm due to 1BD483 Load on Alt Source
from 1B1D473 on equalize

Loss of BD483 will result in a trip of the B RFP, as well as a loss of all
RFP PDS’s.

Aux Bldg operator is verifying no new alarms are present on BD483
2x/shift.

BD483 is currently on it’s alternate source 20525290, it appears it shifted
to the alternate after 1B1D473 battery was placed on equalize.

CRAB (Control Room Alarm Bypass) – The following alarm bypass will have an
impact on the AE system:
o
10-007 A Reactor Feed Pump Eccentricity 60093247

H1FW -1FWVY-3769A RFPT A SHAFT VIBR CONVERTER is
providing indications of vibrations when the pump is not inservice. This
vibration proximity probe was replaced in RF15.

Relation to AE – Currently there is no guidance in HC.OP-AR.ZZ-0020
for CRIDS points A2334 or A2331 which would be driven from FWVY3801A as fed from FWVY-3769A (reference M-31-1 G6). This could
lead to the incorrect removal of a RFP due to an erratic vibration point
 ACM (Adverse Condition Monitoring)
o
HC10-011 #4 TCV anti-rotation bearing 20483804

Ability of the #4 Turbine Control Valve (TCV) actuator spring can antirotation bearing to maintain radial alignment of the Upper Guide (Item
#24 of GE Drawing 823E884 OR GEK figure 8-13). This Guide provides
axial alignment of the turbine control valve Linear Variable Differential
Transformers (LVDT). There are 3 LVDTs on the #4 TCV providing a
valve position signal to the Mark VI controller. At least two-out-of-three
LVDTs must provide a valid position signal to the Mark VI control
system. Failure of two-out-of-three LVDTs will result automatic slow
closure of the #4 TCV.

During implementation of WO 60087893 for repairing the anti-rotation
slot on the #1 TCV, the slots on the remaining 3 TCVs were examined
and the slot on the #4 TCV was noted as having similar wear to the #1
44
TCV 20483804. Engineering performed a follow up inspection and
concluded the following:

The slot for the #4 TCV has been previously weld repaired and is
moderately worn.

The anti rotation bearing is heavily worn and cannot be turned by
hand.

There is a large gap on one side of the bearing which is indicative of
the opposing side worn slot and worn/ flat area of the bearing.
o
A Scope Change Request Form (SCRF) was processed to
perform the repairs in R16. The Outage Scoping Panel rejected
the SCRF on 11/2/10 due to the amount of hours/resources
involved to perform the repair

Relation to AE – The slow closure of #4 TCV could cause a Rx scram
due to high APRM flux thus challenging the operator to maintain level in
band
o
HC11-014 C CW discharge valve drifting 20487634

The “C” Circ Water Pump Discharge Valve H1DA –DA-HV-2152C,
periodically drifts from the OPEN FULL toward the Open / Closed Mid
position resulting in unexpected Open/ Closed Mid position indication.
The alarm condition is occurring with about ¼ inch of valve drift in the
closed direction. Two conditions are believed to be contributing to the
issue: 1) The HV-2152C positioner actuator is leaking internally
allowing HV-2152C to drift toward Open/ Closed Mid position. 2) The
Open/ Closed Mid alarm / CRIDS indication is occurring early, most
likely due to degradation of the HV-2152C ZS-1 (Not 100% Open) snap
lock limit switch. The “C” Circ Water Pump Discharge Valve H1DA –
DA-HV-2152C drifting to Open / Closed Mid position could adversely
impacting Main Condenser Vacuum as a result of lowering total cooling
water flow. The intent of this ACM is to minimize challenges to the
Hydraulic Control Unit (HCU) for HV-2152C by minimizing the number
of times a OPEN FULL signal is given.

Relation to AE – Full closure of the HV-2152C from 100% power could
degrade vacuum to point where the 6.5” HgA Retainment Override of
45
HC.OP-AB.BOP-0006 Main Condenser Vacuum would require a Rx
Scram if Immediate Operator actions to reduce power are not taken
promptly enough.
o
HC11-012 A Moisture Separator LIC-1039A output drift 20515490

The A moisture separator dump level controller, controls LV-1039A in
the event the normal range controller is unable to maintain level or stops
functioning. Level controller LIC-1039A is currently indicating 8.6% of
the dump range. This would mean that the level in the tank is above the
normal range transmitter and can be monitored in CRIDS with point
HC.A2622. However, the normal range is set to control LV-1364A (B,
C) at 30% of the normal drain range, and is fluctuating between 20 –
40% with occasional spikes; this can be followed with CRIDS point
A2624.

Relation to AE – High level in the Moisture separators is a trip signal
(2/3 input) to the Main TB, thus a Rx Scram. From 100% power the
scram recovery and subsequent Level 2 (-38”), HPCI/RCIC response and
potential overfeed to +54” (RFP trip) is a challenge to the RFP’s to
respond.
 OTDM
o
HC10-011- DFCS Startup Level Control PDS 20483568
When in MANUAL, a SULCV PDS communication failure will result in
closure of the SULCV (ODTM HC-2010-0011). MANUAL operation of
the SULCV will be minimized and training has been provided to the
operators. This condition will be corrected in RF17).

Relation to AE system – SULCV is the primary flowpath of feedwater to
the vessel when power is <20%

The decision made was to continue to operate the DFCS with the SULC
PDS in automatic during plant startup and shutdown. A DCP is required
to replace the termination assembly and upgrade the FBM224 firmware
at the next system outage in RF17

The contingencies required were When the SULC PDS is in manual
during plant startup and shutdown an operator should keep the PDS
under observation and respond promptly to DFCS system alarms and
46
adjust valve position as required if the demand goes to zero. Specific
Operator Training was provided.
o
HC11-008 10-A-104 Bus bus under voltage relays B-C tripped 20515932

This OTDM will document the risks associated with continued operation
of Hope Creek Unit 1 with tripped 4.16 KV AC Non-1E (NB) \ H1NB 10-A-104 Bus bus under voltage (UV) relays until permanent repairs can
be implemented.

Relation to AE system – 10A104 bus undervoltage condition could
potential trip the C SCP thus challenging the abiltity of the RFP’s to
respond.

To minimize the risk of equipment damage or an unnecessary plant
transient the following actions were implemented:

A Temporary Configuration Change Package (TCCP) to defeat
the degraded bus UV trip logic for the C SCP.

Schedule fuse inspection and replacement, and or
troubleshooting of the UV Relay scheme for B-C Phases during
a bus outage in R17.

Continue to protect (via the clearance and tagging process) the
bus voltmeter selector switch as it is a potential cause of PT fuse
blowing (if the switch fails during use).

OPEVAL – No current OPEVAL’s have any affect on the AE system.

POD –
o
II D. OPERATIONS CONCERNS – 10A104

Attempted to swap 10A104 in-feeds during last down power. When
depressed 10A104-AUTO CLOSE BLOCK would not extinguish.

Relation to AE – C SCP is powered from 10A104 bus, loss of C SCP
would challenge the RFP to respond.
o II G. NUISANCE (N) OR ILLUMINATED (L) ALARMS

120 VAC UPS TROUBLE – BD483 on Backup power
o II I. OTDM’s / ACM’s / Op Determinations

HC-2010-11 DFCS PDS loss of communications
47


HC-2011-08 10A-104 Bus Undervoltage relay de-energized (1 of 2)

H10-11 #4 TCV anti-rotation bearing

H11-012 A Moisture Separator Dump VLV

H11-014 C CW disch vlv drift
TCCP’s
o
4HT-10-046 Splice 1AP102 PCP “C” phase power supply cable to PCP motor
lead

A PCP pump tripped on start. Upon investigation it was discovered
that the ‘C’ phase power supply cable to the PCP motor lead was
damaged. "C" Phase Motor lead was terminated with a bolted
connection.

Relation to AE – Loss of A PCP will challenge the ability of the RFP’s
to respond
o
4HT-11-013 Defeat “C” SCP 1CP137 Under Voltage Trip

A Temporary Configuration Change Package (TCCP) to defeat the
degraded bus UV trip logic for the C SCP.

Scheduled fuse inspection and replacement, and or troubleshooting of
the UV Relay scheme for B-C Phases during a bus outage in R17.

Relation to AE – Loss of A SCP will challenge the ability of the RFP’s
to respond
6.7
Maintenance feedback
Maintenance feedback from turbine services was provided as follows:
 Salem found their oil system vapor extractor lines were fouled (rust and beetles), which
impacted the operation of the system, and led to oil leaks. They created a PM for periodic
testing and cleaning. HCGS doesn’t have this.
One of the actions the team is going to implement is to perform a PM for the vapor extractors.
The PM will address this issue since it will inspect and repair the extractor as needed. At Hope
Creek there are no PM’s scheduled to perform this task and it will be added to the new lube oil
system PM.
48
 Taking oil pressures at the turbine with it running can help determine how well the extractors
are functioning.
This task can be performed by installing a temporary gage on the RFP casing. The gage will
measure vacuum from the oil reservoir and perform this task. However this is not an SPV and
will not be included in the corrective actions.
 HCGS has no spare rotating elements for the feed pump
 HCGS has spare parts inventory issues with the turbine stop valves
 Thermocouple reliability is questionable. There are indications that operations will remove a
pump if the limit is exceeded. A false indication could result in removal of a pump under false
pretenses.
This action was reviewed with maintenance and determined that maintenance practices may not
be thorough enough. When machine overhauls are performed that remove thermocouples, they
are not replaced and the seals are not replaced. Maintenance stated that the oil leaks in the turbine
room are probable causes of thermocouples and vibration probes that do not have proper seals
installed. Oil then drips out of the thermocouple port and accumulates over time.
The elements are placed back in the pump or turbine without being replaced. Several events have
resulted in bad indications from these points. Certain points can result in operations taking action
to mitigate the condition, including reducing RFP speeds.
Overhaul activities need to include actions to replace aged sensing elements and seals.
Maintenance activities will be changed to include new probes and seals to prevent probe failures
and other mitigate small oil leaks.
There are also actions from this report to include a procedure change that will prevent operators
from removing the pump from service. Normally when a thermocouple fails, the temperature
drops and would not require Operators to remove the pump from service. In some instances a
thermocouple has failed high where there is a momentary spike in temperature to the maximum
49
possible indicated value. The team determined that enhancements during maintenance overhauls
are required to ensure degraded equipment is not overlooked. Outage Services is currently
reviewing Hope Creek feedpump maintenance procedures for enhancements based on lessons
learned from Salem. Feedback has been provided to the Outage Services group for improved
maintenance practices.
6.8
OPEX review
The team reviewed the original OE review from the 2005 SPV review. The 2005 review has been
attached. The action items were captured under WO 80081141. The majority of the items were
alignment with the Exelon PCM templates. The team reviewed order 80081141 to verify that the
actions had been completed or were in progress. The report from 2005 has been included in this
report since. The original OE list was reviewed again to verify that no entries were overlooked
and that the actions generated from the review will mitigate or eliminate those vulnerabilities.
The team also reviewed OE dating after 2005 up to the most recent entries. In addition to the OE
reviewed before 2005, the team created a new OE review matrix that will also be included with
this review.
6.8.1 INPO - Industry Operating Experience (OPEX)
The following INPO documents were reviewed (SOER, SER, SEN, and topical reports) for
identifying vulnerabilities not found from review of design drawings and field walkdowns.
There were over 400 industry type OPEX reviewed by the team. The team reviewed the previous
OPEX list from the 2005 scram/derate initiative and included any new OPEX up until November
2010. For the most part, the OPEX reviewed provided validation of suspected SPV components
or vulnerabilities that the team identified during Phases 1 and 2 of the project and captured in the
vulnerability matrix included in this report. There were no noteworthy OPEX identifying new
SPVs. The team captured their review in an OE matrix included as part of this report. Each
OPEX found applicable was cross referenced to the most applicable component found in the
vulnerability matrix. Each of the vulnerabilities identified below have been captured in the
vulnerability matrix with appropriate elimination/mitigation strategies.
50
The team reviewed OE documents for applicability and then reviewed programs and procedures
implemented at the station to determine if a vulnerability existed. The following is a list of INPO
SEN’s
DocID
Title / Issue
HC
HC Comments
Applicable
SEN 271
Failures in redundant auctioneered
No
The backup power
power supplies for digital FW
supply failed without
control (DFWC) system resulted in
indications to the
an invalid high reactor water level
operating crew. Of
trip signal. This caused main turbine
interest is that the
and FW turbines to trip and an auto
redundant power supply
scram
degraded without the
plant being aware of it.
Hope Creek has
indication if the back up
and main power supply
fails.
No Vulnerability
SEN 248
On January 23, 2004, Calvert Cliffs
yes
This SEN identifies
Unit 2 automatically scrammed from
several equipment
100 percent power on low steam
issues. The cause was
generator level when one of two
the result of a spurious
turbine-driven main feedwater
electronic overspeed
pumps inadvertently tripped.
signal because of a fuse
holder developed high
resistance due to local
corrosion. Hope Creek
requires two signals for
trip actuation.
51
No Vulnerability
SEN 199
On May 14, 1999, with Point Beach
yes
Of interest in this SEN
Unit 1 operating at 100 percent
is the shell rupture to
reactor power, the shell of feedwater
feedwater heater due to
heater 4B ruptured, blowing
wall thinning. The sites
insulation and debris into the turbine
FAC program precludes
building. Control room operators,
this issue.
alerted by the noise and a 5megawatt decrease in
SEN 174
On September 6, 1997, both
No Vulnerability
Yes
In this SEN a backup
McGuire units automatically
power supply breaker
scrammed from 100 percent power
failed that resulted in
when the alternate supply breaker to
the main feedpump to
nonsafety-related 120-volt AC
trip due to loss of
instrument and control power bus
power. This may be
KXA opened, stripping control
similar vulnerability
power to several important plant
with the power to the
components
trip system.
Vulnerability exists in
the ‘loss of power trip to
the FW trip system’
Reference 80105125
0200 for vulnerability
elimination
SEN 155
On January 26, 1997, Indian Point 2
No
This SEN is not
was being shut down because of
applicable because
level control problems on one steam
Hope Creek does not
generator. During the power
have FRVs.
52
reduction, the main feedwater
regulating valve for the steam
No Vulnerability
generator appeared to bind.,
SEN 19
On September 6, 1987, with Davis-
No
HC digital feed will fail
Besse operating at full power, a
to the back up signal.
feedwater flow transmitter failed
There are 2 transmitters
causing the integrated control system
that feed a flow signal.
to increase feedwater flow to the
Also Hope Creek
steam generators. This resulted in
DFWC will adjust RFP
cooldown of the reactor coolant
speed with respect to Rx
system.
level and there are 3
level transmitters.
No Vulnerability
SEN 4
On December 9, 1986, Surry Unit 2
No
Pipe failure due to wall
was operating at 97% power when
thinning should be
the main steam isolation valve to the
precluded from FAC
"C" steam generator closed for
program
unknown reasons.
implementation.
No Vulnerability
The team reviewed OE documents for applicability and then reviewed programs and procedures
implemented at the station to determine if a vulnerability existed. The following is a list of INPO
SER’s
DocID
Title / Issue
HC
HC Comments
Applicable
SER 5-06
All reactor types have experienced
Yes
This SER focus on
FAC events in piping systems. Even
industry failures with
with mature FAC programs, events
FW and MSR heater
continue to occur. In addition, as
shells and piping due to
plants age, and many increase power
wall thinning. The site
output, secondary plant conditions
FAC program should
53
and piping wear rates may change.
govern and provide the
necessary oversight to
prevent premature
failure.
No Vulnerability
SER 6-91
Several recent plant transients have
Yes
This SER focuses on
been caused by instrument air
instrumentation failures
system fitting failures. The following
of fittiings and resultant
contributed to these events:
plant transients.
Problems related to
design deficiencies and
inadequate PMT
followiing maintenance.
Perform Walkdown For
vulnerability:
The AOVs have been
walked down to identify
any air supplies that
were not fitted with
Flex Fit tubing or the
‘spiral’ tubing
configuration. The
SULCV does not have
any of these
configurations.
54
SER 24-
This event is significant because it
No
The spring valves were
88
demonstrates that during loss of
returned to service with
offsite power, the failure of either
excessive spring force
stop valve on the main feedwater
and could not close and
pump turbine to fully close can
isolate the turbine. Also
result in the destruction of this
the steam valves failed
turbine.
as is. HCs steam control
valves would fail closed
on a loss of oil. Oil
controls our valve
positions. This SER
focuses on damage to
the main feedpump
turbine when
stop/governor valves
fail to close following
loss of offsite power.
The SER stresses the
need to test turbine
valves.
No Vulnerability
SER 4-88
Following a transient initiated by a
No
This failure was the
feedwater flow transmitter failure
result of a single FW
the reactor tripped due to a control
flow transmitter failure
rod misoperation. Subsequent
and caused the event. A
recovery of the plant was
single transmitter failure
complicated by failure of a steam
results in DFCS failing
safety valve to reseal, failure of a
to the remaining
13.8 kV supply break
transmitter for
indication and control.
There were several
issues identified in this
SER but the FW flow is
55
not a trip initiator.
No Vulnerability since
the DFCS adjusts RFP
speed with respect to
the Rx level before the
Feed flow.
SER 27-
A reactor trip occurred on low steam
Yes
This failure was due to
87
generator level due to failure of the
maintenance. The valve
train “A” main feedwater pump
was reassembled with
discharge check valve with the “B”
the wrong clearences
main feedwater pump running. The
causing FW to back
“B” feedwater pump discharge
flow through a tripped
pressure decreased to about 700 psig
feed pump. HC has
due to back flow
similar valves, Though
there are also discharge
isolatoion valves for
feedwater pump
discharge, the check
valves must maintained.
Also this failure was
stated a lack of vendor
oversight resulted in this
failure.
Vulnerability exists in
the maintenance
procedures and
maintenance practices.
56
SER 1-87
Following a reactor trip from full
No
FAC issue and there are
power, an elbow in the 18- inch
programs in place for
suction pipe to the "A" main
this issue. In 1986,
feedwater pump ruptured, injuring 8
Slurry did not have a
workers. Four of the workers
FAC that monitored
subsequently died, and two were
water but only steam
hospitalized. Inspection of the
and 2 phase. Their
ruptured elbow revealed severe
design did not allow
turbulence to disperse
before changing
direction. This SER
focuses on the trend of
high pressure pipe
failures on the suction
of main feedpumps due
to wall thining from
erosion/corrosion
issues. The siets FAC
program should provide
the oversight and
program strategy.
No Vulnerability
SER 2-86
During a power reduction from 40
Yes
Various equipment
percent power to perform a main
failures resulted in this
turbine overspeed trip test,
SCRAM. Most were
secondary system instabilities and
PWR related and
multiple secondary system
beyond the scope of this
equipment failures resulted in a
review (one event lead
steam leak, loss of feedwater and
to several subsequent
reactor trip.
events) However a
steam relief valve on the
FWH could pose
vulnerability.
57
No Vulnerability since
Hope Creek has a PM
for this valve and
checks the
The team reviewed OE documents for applicability and then reviewed programs and procedures
implemented at the station to determine if a vulnerability existed. The following is a list of INPO
SOER’s
DocID
Title / Issue
HC
HC Comments
Applicable
SOER
Instrument air systems are typically
Yes
OE does not apply to
88-1
classified as non-safety-related
the extent this review is
systems. However, both safety-
investigating and should
related and non-safety-related
be assigned to the
systems use instrument air and have
station air system
been adversely affected by air
manager. This SOER
system failures.
focuses on the
importance of air
qualitiy and reliability
of the instrument air
system and the affects
to plant transients and
trips.
58
Perform Walkdown For
vulnerability:
The AOVs have been
walked down to identify
any air supplies that
were not fitted with
Flex Fit tubing or the
‘spiral’ tubing
configuration. The
SULCV does not have
any of these
configurations.
SOER
Reactor trips and plant transients
No
This SOER focuses on
84-4
initiated by main feedwater system
plant trips related to
control problems are common
control system problems
events. Reducing the frequency and
associated with FW
severity of these trips would improve
pumps and regualting
plant availability and reduce the
valves.
number of challenges to plant
protection systems
No Vulnerability. Hope
Creek does not have
FRVs
The team reviewed OE documents for applicability and then reviewed programs and procedures
implemented at the station to determine if a vulnerability existed. The following is a list of INPO
Topical Reports.
59
TR6-55
Review of Large-Pump-Related
No
No Vulnerability –
Events Resulting in Scrams, S/D and
issues are captured in
Outage Ext. Review of Large Pump
this report.
Related Events Resulting in Scrams,
Shutdowns, and Outage Extensions,
November 2006”
TR4-41
INPO TOPICAL REPORT TR4-41
Yes
About 36 percent of the
Review of Main Feedwater System
events were attributed to
Related Events, November 2004”
problems associated
with preventive
maintenance (PM). The
most common problems
noted included
inappropriate PM
frequencies, insufficient
actions identified in the
PM, inadequate
implementation of a PM,
or a PM activity did not
exist. Operation 0210
will be to verify the
results of the RF17
RFP/T overhauls to
determine the impacts
EPU had on the B RFP
and C RFPT including
wear and aging. Using
the as found conditions,
determine if the
overhaul frequency
should be extended or
shortened.
TR4-34
INPO TOPICAL REPORT TR4-34
Review of Feedwater System
No
There are no ultrasonic
flow meters at Hope
60
Ultrasonic Flowmeter Problems,
Creek. However, Hope
March 2004”
Creek. continuously
monitors the correction
factor and provides an
alarm if a deviation is
noted. The station's
operators are provided
with Crossflow alarm
and procedures to
respond
once a Crossflow alarm
is received.
6.8.2
NRC IN reviews
The team reviewed OE documents for applicability and then reviewed programs and procedures
implemented at the station to determine if a vulnerability existed. The following is a list of NRC
Information Notices (IN):
DocID
Title / Issue
HC
HC Comments
Applicable
IN 2008-
Turbine-driven Auxiliary Feedwater
09
Pump Bearing Issues
No
These issues were for
Aux feedwater pumps
and not the main Feed
pumps.
No vulnerability. Hope
Creek does not use Aux
feedwater
IN 2008-
Main Feedwater System Issues and
13
Related 2007 Reactor Trip Data
Yes
Loss of power to the
trip system will cause
trip solenoid valves to
open tripping the RFPT.
61
Vulnerability will be
addressed under
80105125 0200
IN 2010-
The lessons learned from these
Yes
Plants identified in this
20
events may apply to turbine driven
IN did not properly test
pumps in other systems such as
the control and trip
reactor core isolation cooling and
functions of the system
high-pressure coolant
during plant start up
injection systems.
such as over speed and
control valve stroking.
No Vulnerability
because Hope Creek
strokes the control
valves on start up to
verify full valve travel.
Hope Creek also
performs electric
overspeed tests on start
up.
6.8.3 BWROG SCRAM frequency reduction report:
The team reviewed the SCRAM frequency reduction report and determined that the only
applicable item was recommendation 16 ‘install a time delay on the RFP low suction pressure trip
and stagger the RFP low suction pressure trip set points OR the respective time delays’. The basis
for this recommendation is that the staggered trip actuation will prevent all three RFPs from
tripping on low suction pressure. In the event that a low suction pressure condition actually
occurs, the staggered trips will shed a single feedpump at a time instead of all 3 at once.
Hope Creek already has a 2/3 trip with a 10 second time delay. However all three switches for all
three RFPs will actuate at 230psig. The suction pressure trips should be staggered to prevent a
sudden simultaneous loss of all three RFPs. Increasing the time delay could cause damage to the
RFP if the low suction pressure exists. The pump would be subject to pump cavitations. Reducing
62
the time delay would significantly reduce margin for the RFP trip time delay. Increasing the
suction pressure trip pressure set point would also reduce margin. Reducing the suction pressure
trip set point would allow the respective pump to operate at a lower suction pressure and increase
margin but make it more vulnerable to cavitations. The best mitigating strategy will be to stagger
the pressure set points. Increase one to 250 psig, increase the second to 240 psig, and leave the
third one as is. This was chosen because the team knew that damage was possible if the suction
pressure set points were lowered or the time delay was increased, and there was little margin to
decrease the time delays. Increasing pressure trip setpoints did not increase damage risk, yet left
enough operating margin for uninterrupted operation. This action will be captured under
80105125 operation 0150.
6.8.4 EPRI documents review:
The team reviewed EPRI documents for applicability and then reviewed programs and procedures
implemented at the station to determine if a vulnerability existed. The following is a list of EPRI
documents and dispositions.
EPRI
Document provides a guide for
Yes
Hope Creek has had
1884-10
proper pump warm up before being
complications in the past
placed in service. The document
while starting up and
included a section for reducing
seizing a RFP internals.
thermal shock and water hammer.
The breakdown bushing
would seize to the shaft.
The vendor documents for
the HC RFPs gives a limit
of 40 degrees F for delta
temperature across the top
and bottom of the case.
The report referenced a
failure to properly warm
the RFP resulting in
thermal shocks.
No vulnerability. Hope
Creek has implemented
63
procedure changes to
prevent inappropriate
warm up of a cold
feedpump.
EPRI
Mechanical hydraulic controls
Yes
Hope Creek has
1003094
(MHC) used on feedwater pump
determined that the
turbines (FWPTs) as installed in
Lubricating oil system is
nuclear power plants are a high
in need of additional PMs
contributor to plant capacity derates.
for components that are
Specifically, the controls and
not normally service. For
associated lubrication systems have
instance the team has
produced chronic problems in the
already identified check
FWPT applications. The majority of
valves that do not receive
FWPT trips have resulted in tripping
preventative maintenance.
the plant off-line. The mechanical
hydraulic controls are also widely
Vulnerabilities were
used in large capacity fossil power
captured under the
plants as the control for the prime
80105125 order has
mover on the boiler feedwater
operations to improve the
pumps.
lube oil preventative
maintenance program for
this system.
64
EPRI
Mechanical hydraulic controls
Yes
Review determine that the
1021066
(MHCs) in nuclear plant feedwater
majority of applicable
pump turbines (FPTs) are a high
failures were attributed to
contributor to plant capacity derates.
the following
The purpose of this study was to
Age—failures attributed
evaluate MHCs’ degradation over
to normal wear,
time and life-limiting properties in
insufficient PM, the
order to provide guidance about their
unsuitability of a
expected life span and the point at
component
which major refurbishment or
for its application, valve
replacement should be considered.
leak by, and other
Insights from this study should
mechanical issues
provide readers with life-cycle
• Sticking/binding—
guidance that helps in the
failures related to the
management of their facilities’ long-
sticking and/or binding of
term operating strategies.
linkages and valves due
to issues like
misalignment and wear
• Oil quality—failures
attributed to
contamination in the
hydraulic/lubricating oil
due to dirt
and contamination and
buildup/deposits/corrosion
on components
Vulnerabilities were
captured under the lube
oil PM program to be
implemented under
80105125
65
EPRI
This guide provides basic
Yes
This EPRI document
105933
information on the design ,
provided general
construction and maintenance of the
information for feed
main feedwater pump equipment
pumps and oil delivery
supplied by the six pump
systems throughout the
manufacturers to the domestic
industry. This review
United States nuclear power
included material for all
generation industry. This guide is
models and styles of
intended to provide useful
feedpumps including
information to all disciplines and
motor and turbine driven
skills associated with the
feed pumps. Most of the
maintenance of main feedwater
information in this review
pump equipment, the planning of its
was applicable to Hope
maintenance, and the monitoring and
Creek but had already
evaluation of its performance.
been captured under other
SPV entries.
No vulnerability
6.8.5
Industry Operating Experience
There were over 400 industry type OPEX reviewed by the team. The OEs reviewed dated as far
back as January 1986 up to November 2011. During the OE review, the team determined that
OEs containing Feedwater Regulating Valves (FRV), Auxiliary Feedwater, and motor driven
pumps were not considered applicable to Hope Creek. The team found 74 OEs applicable to Hope
Creek. The failures were categorized by failure to include electrical, I&C, mechanical, and others
related to human error. Failures are as follows:
- There were 5 failures that were related to electrical issues
 1 connection failure
 4 fuse failures
- There were 23 failures related to I&C issues
 2 circuit card failures
 11 controller failures
 4 power supply failures
 5 switch failures
66
 1 transducer failure
- There were 32 failures related to mechanical issues
 5 corrosion and erosion failures
 4 oil leak failures
 1 steam leak large enough to force a derate
 1 water leak large enough to force a derate
 2 linkage failures
 1 impeller failure
 1 shaft failure
 11 valve failures (includes AOV, MOV, isolation, and control valves)
- There were 14 failures categorized as ‘other’ for failures not directly related to component
failures
 3 design failures
 4 FME induced failures
 5 human Error failures
 1 maintenance induced failure
 1 procedure induced failure
For the most part, the OPEX reviewed provided validation of suspected SPV components or
vulnerabilities that the team identified during Phases 1 and 2 of the project and captured in the
vulnerability matrix included in this report.
During this review, the team noticed a subtle trend in OE failures and their complexity. The OEs
reviewed included failures dating back to as early as the late 1980’s. The OEs that caused
SCRAMs early on were related to single failures such as a false switch actuation, fuse failure, or
connection failure usually caused by lack of PMs and procedural guidance. The more recent
failures are the result of a chain of events that force the derate. For instance OE33423 was the
result of oil sludge accumulation in the supply piping. Sludge build up caused bearing
temperatures to rise and forced the feed pump to be removed for repairs. Or in some instances the
failure was an overlooked subcomponent that was overcome by aging. A PM was being
performed on the component, but there might have been a subcomponent that was not replaced or
configured incorrectly. The subcomponent fails, the component fails, a transient is experienced in
the system flow/level control and results in a derate.
67
6.9
Feedwater Simulator Results:
The simulator was used by the team as additional input in determining whether the component
should be considered a single point vulnerability. It is understood by the team that limitations in
simulator modeling in the secondary systems may not provide sufficient response to clearly
discern an SPV.
All simulator faulted scenarios associated with feedwater induced transients or plant trips were
reviewed. The conclusion from this review did not identify any new or different equipment
failures not already considered on the vulnerability matrix. However the simulator actually
eliminated the failure of the minflow recirc valve. This valve was originally thought to be an SPV
but is not after verifying simulator data.
6.9.1
Trip of 6A FWH from 100%
The team simulated a loss of the 6A FWH at 100% Rx power.
20 degree reduction in feedwater temperature. Requires Ops to reduce power by 20% per
HC.OP-AB.BOP-0001. Reactor power rise without operator action was 3% to 3967 MWth.
6.9.2
Trip of in-service RFP lube oil pump
Swap occurs to standby pump per design. Station vulnerability is not modeled. The station
vulnerability would cause a trip of the RFP since the back up oil pump can not restore oil pressure
before the trip set point is breached.
6.9.3
Loss of speed input to Woodward governor (SE-3749)
Sensed as control signal failure. RFP control auto-transfer to manual. No impact on operation.
Several plants have had experienced a loss of speed input to the controller which has resulted in
the loss of a feedpump, however this is not a vulnerability at Hope Creek.
6.9.4
Min flow valve fails open
No scram. No power reduction. Level lowers to level 4 at 25 inches which is above the 15 inch
Operator procedure limit to manually SCRAM. There were no runbacks since this is not a trip.
All 3 pumps feed pumps speed up to 5450 RPM to compensate. Those pump flows go to 14.9
68
Kgpm. Affected pump flow lowers to 8.4 Kgpm. Simulator models flow being restricted to minflow line capacity of 5000 gpm.
6.9.5
Fail open start-up level control valve
Power rises to 100.6%. No level transient.
6.9.6
RFPT Trip
Level 4 – recirc runback. Power lowers to 71%. Lowest level seen was 27”. Operators manually
trip the unit if level drops to 15 inches.
6.9.7
Transmitter failures – could not simulate
Drawing review with instructor.
FT-N011 – only gives flow indication at DFCS. No impact to system
FT-1755A – indication to CRIDS only. No impact
FT1800A – input for min flow controller, also low flow trip vs. speed. Min flow valve would
open on low failure.
FT1770A – min flow controller setpoint input to controller. No impact.
6.9.8
Electrical – loss of 10-B-313 or 10-B-323
In the accepted proceduralized lineup- RFP lube oil pumps swap as designed. This lineup
maintains availability of power to standby pumps.
Power supply to feed pump lube oil pumps:
1A1P124 – 10-B-323 (“B” 4160 1E source)
1A2P124 - 10-B-272 (“C” 4160 1E source)
1B1P124 - 10-B-323 (“B” 4160 1E source)
1B2P124 – 10-B-313 (“A” 4160 1E source)
1C1P124 – 10-B-272 (“C” 4160 1E source)
1C2P124 – 10-B-323 (“B” 4160 1E source)
Loss of 10-B-323 will trip three pumps, if running.
Loss of 10-B-272 will trip two pumps, if running.
69
Allowable lineups prevent loss of one 4KV bus from tripping two pumps (one pump from each
1E bus).
A1P124, B2P124 and C1P124 in service
Or
A2P124, B2P124 and C2P124 in service
Note that B2 pump always runs. PM is being generated to eliminate the vulnerability of having
the B2 pump constantly operating. Order 70123865 has action to create a PM that will swap the
RFP/T lube oil pumps quarterly to prevent excessive pump use.
6.9.9
Loss of 1B-D-483 inverter:
“B” RFP Trip
Loss of power to control PDS’s. and control
Manual control available (no procedure) at 10-C-612 Woodward station.
Level dropped to 15” before recovering. Operators would have manually scrammed. Recirc
runback due to pump trip and level 4. Possible single element control due to loss of steam flow
signal – which gave slower response for two remaining feed pumps.
6.9.10 Loss of 1A / B-D-318 125VDC Busses:
Respective RFP trip on loss of trip circuit power. Single Recirc runback. Final power 82%.
Loss of main power transformer cooling for loss of 1A-D-318 will likely result in taking unit off
line if alternate power is not selected.
Simulator cannot run loss of 1C-D-318. This would cause turbine trip, due to loss of DEHC.
6.10
Review of past internal CAP items
The team performed a review of past corrective actions, corrective maintenance activities that
have not been performed, and open CAP items. The purpose for this review was to verify there
were no items that were overlooked that could challenge that station if not mitigated. A review of
current open evaluation activities was performed to verify all tasks have been assigned and the
70
evaluations are the appropriate level and are not aged beyond 100 days. Past evaluations were
also reviewed for accuracy and timely implementation of corrective actions.
No additional actions were generated from this review. All maintenance items are being properly
scheduled and planned for implementation for earliest opportunity to repair. All the evaluation
activities’ age are under 100 days which is the requirement for excellence. The past evaluations
were closed out and the mitigating actions are being performed to mitigate degraded conditions.
6.10.1
Review of open Maintenance activities
The team reviewed open Maintenance activities. The majority of the feedwater corrective
maintenance actions have been scheduled for the next possible outage. Scheduling considers the
‘risk’ that could be sustained by the plant. Risk is considered ‘impact’ and ‘possibility’. If an
activity is too high of risk to the plant it will be scheduled for the refuel outage.
Most of the scheduled maintenance activities must be performed during R17 due to high radiation
or the requirement to tag a power supply or a pump out of service to gain access. Most of these
items were generated from failing indication trends or equipment that was found to start showing
signs of degradation. There are no major equipment failures on this list that caused a derate or
challenge to the station. The team was unable to produce any additional items for the system.
This list was generated from SAP:
Order
Functional location
Description
60060327
H1AE
CONTCY-1A/B-S-105- JUMPER ZERO SPEED SW
60077333
H1CJ -1CJTE3181C1
60081975
C RFPT THRUST BRG TEMP IND FAILED
H1AE -AE-HVF032A
(CTGY) AE-HV-F032A: CLEAN/INSP VLV(MOV)
60090512
H1AE -1AELV-1785
R16 80101747 INSTALL LOW DRAG PLUG SEAL
60091396
H1AE -AE-HVF074A
60095662
F074A - CONTINGENCY VALVE REPAIR.
H1AE -1AEFIR603A-C32
RPLC -H1AE -1AEFI-R603A-C32 PER 80098173
71
60095663
H1AE -1AEFIR603C-C32
60093692
H1AE -1AEVT7908B6
60099210
RPLC -H1AE -1AEFI-R603C-C32 PER 80098173
REPLACE H1AE -1AEVT/VE-7908B6
H1FW -1FWYISBC653081005
DIGITAL POINT D2005 COMING IN/OUT
60099310
H1AE
80100455 / 1SBPISH-N652D REVISE SETPOINT
60097044
H1AE -1C-E-106
H1AE -1C-E-106 REPLACE FAC PIPING
60094237
H1AE -1A-P-101
INSULATION FALLING OFF FEEDPUMP PIPING
60099159
H1AE -1AELV-1785
RPLC- 1AELV-1785 SULC LOWER BOOSTER
60096939
H1AE -1AETV1796A
60096941
A-RFP SEAL WATER VALVE DEGRADED
H1AE -1AETV1796B
60089122
dn
B-RFP SEAL WATER INJ VALVE DEGRADED
H1CJ -1CJHS3178C2
TS&R H1CJ -1CJ-HS3178C2
60094805
H1AE -1B-P-101
TROUBLESHOOT B RFP OSCILLATIONS
60092954
H1AE -1AETIC1796C
1AETIC-1796C/UNABLE TO BRING INTO CAL
60096240
H1FW -1-FW-V063
1FW-V063 LEAKING BY SEAT
60097899
H1CJ -1CJTI-3160C
REPLACE H1CJ -1CJTI-3160C
60091631
H1AE -1AEZT1783A
A RFT RECIRC VLV POS IND ON COMP POINT
60097807
H1CJ -1CJTI-3157B
RPLC- 1CJTI-3157B TEMPERATURE IND
60097808
H1CJ -1CJTE3180A1
A-RFPT/CRIDS A2340: RPLC 1CJTE-3180A1/A2
60097898
H1CJ -1CJTI-3153B
REPLACE H1CJ -1CJTI-3153B
60093691
H1AE -1AEVT7908A6
1A-P-101 REPLACE H1AE -1AEVT/VE-7908A6
60095660
H1AE
DFCS FBM UPGRADE DCP 80103280
60095910
H1CJ
'A' RFPT CNTRL OIL ACCUM 80102874
60097743
H1CJ
'B' RFPT CNTRL OIL ACCUM 80102874
60097744
H1CJ
'C' RFPT CNTRL OIL ACCUM 80102874
60099307
H1AE
80100455 / 1SBPISH-N652A REVISE SETPOINT
72
60085831
H1FW -1FWZTS1794B
60093247
H1FW -1FWZTS-1794B FAILING INDICATION
H1FW -1FWVY3769A
TS&R 1FWVY-3769A AND 1FWVE-3769A
60097912
H1AE -10-C-612
RP- 10-C-612 CARD RACK RAILS
60099155
H1FW -1FWZY1794C
C RFPT CV POS IND DEGRADING A2315
60083138
H1AE -52-113023
H1AE -52-113023/REPAIR STAB BLOCK
60095698
H1CJ -1CJHS3177B2
60097734
RP-1CJHS-3177B2-"B" RFP OIL PMP CNTRL SW
H1FW -FW-HV1760C
60097820
dc
HV-1760C DRAIN VALVE LEAK BY
H1AE -1AETIC1796B
TIC-1796B/TIC-1780B PERFORM TUNING
60099127
H1CJ -1B2P-124
RADIAL VIBRATION INCREASING TREND
60097706
H1AE -AE-HVF032A
RF17 - BORESCOPE MOTOR OF 1AEHV-F032A
60097902
H1FW
R17 RPLC BAILEY MODULES FOR RFPT A
60097903
H1FW
R17 RPLC BAILEY MODULES FOR RFPT B
60097904
H1FW
R17 RPLC BAILEY MODULES FOR RFPT C
60097043
H1FW -1C-S-105
Replace C RFPT Drain Lines H1R17
60097864
H1AE -1-AE-V092
1-AE-V092 LEAKING FROM PIPE CAP
60097888
H1AE -1AEXI651LCD
60097889
DFCS Operator Monitor Burn In
H1AE -1AEXY651WP1
Operator WS Keyboard Difficult to Access
60077090
H1AE -52-242115
DCP 80095620 RPLC MCC 52-242115 MOV F039
60077091
H1AE -52-242161
DCP 80095620 RPLC MCC 52-242161 HV-F039
60086973
H1AE -52-222102
DCP 80098425 RPLC MCC 52-222102 MOV F032
60096000
H1AE
80103166: On-Line Noble Chem FW Tie-ins
60098460
H1AE -1A-E-106
DCP 80103819/FWH 6A XMTRS UPGRADE
60098482
H1AE -1B-E-106
DCP 80103819/FWH 6B XMTRS UPGRADE
60098483
H1AE -1C-E-106
DCP 80103819/FWH 6C XMTRS UPGRADE
73
60099309
H1AE
80100455 / 1SBPISH-N652C REVISE SETPOINT
60099308
H1AE
80100455 / 1SBPISH-N652B REVISE SETPOINT
60096958
H1AE -1-AE-V189
H1AE -1-AE-V189 LEAKBY WITH RWCU I/S
60086917
H1AE -52-212102
DCP 80098304-Replace MCC H1AE -52-212102
60087204
H1AE -52-232054
DCP 80098424 Replace MCC H1AE -52-232054
60087205
H1AE -52-232171
DCP 80098424 Replace MCC H1AE -52-232171
60079815
EXTEND REAR DOOR XFLOW COMP PANELH1AE -10-Z-370A
6.10.2
CAP
Review of Engineering items
The team reviewed open Engineering activities. The majority of the feedwater corrective
Engineering actions are for work scheduled for the next possible outage. The evaluations under
7000 series wor orders have been performed. The engineering goal is to evaluate and resolve
issues within 100 days of creation.
Functional
Order
Description
location
PM DEFERRAL (PMDR) EVALUATION
30142986 GUIDELINES
H1AE -1C-P-101
30142986 NUPM DEFERRAL SUPERVISOR REVIEW
H1AE -1C-P-101
ENGINEERING SUPPORT EDDY CURRENT
30159696 TESTING
H1AE -1C-E-106
30159696 1C-E-106: PERFORM INTERNAL INSPECTION
H1AE -1C-E-106
30159696 1C-E-106: SYS ENGR. CLOSEOUT INSPECTION
H1AE -1C-E-106
30206023 First Call Risk Review
H1AE -1C-E-106
30190564 DETERMINE HOLE LOCATIONS / SIZE
H1AE -1B-E-106
30190564 DATA EVALUATION
H1AE -1B-E-106
ENGINEERING SUPPORT EDDY CURRENT
30201131 TESTING
H1AE -1B-E-106
30201131 1B-E-106: SYS ENGR. CLOSEOUT INSPECTION
H1AE -1B-E-106
ENGINEERING SUPPORT EDDY CURRENT
30180550 TESTING
H1AE -1A-E-106
70117252 Validate procedure and order changes
H1AE
70124662 RFP FLOW AND SPEED DIFFERENCES
H1AE
74
6.10.3
Review of past Engineering evaluations
The team reviewed past engineering evaluations for the feedwater system. These issues have been
evaluated and presented to MRC. Corrective actions came from these evaluations. Evaluations are
reviewed by a peer, then the supervisor and director if the evaluation is an apparent cause
evaluation or root cause evaluation. Evaluations are presented at MRC where they must have a
corrective action to mitigate or eliminate the degraded condition identified. About three quarter of
the evaluations were performed by the current system manager but the earlier quarter were
performed by past system managers.
Order
Functional loc.
Description
REACTOR FEEDPUMP VIBRATION ALARM70046614 H1AE -1C-P-101
A2010
70057113 H1CJ -1CJPI-3150A
A" RFPT CONTROL OIL PRESS HOOS
70057305 H1FW -1B-S-105
During O/S testing W/U did not work.
70057360 H1AE
Startup Level Control Oscillations
70057405 H1AF
BLEEDER CHECK VALVES ISSUES
70057567 H1FW
reactor feed pump turning gear speed SW
70075109 H1AE -1AELV-1785
R14 BOTH STARTUP LCVS ARE STUCK OPEN
70075773 H1AF -1AFLV-1506C
R14 Blocking point missed LSLD
70076610 H1AE
R14 DFCS ISSUES DURING RF14 STARTUP
70077303 H1AE -1A-P-101
'A' Rx Feed Pump Seized
70077472 H1AE -10-Z-370C
XFLOW CPU HARD DRIVE FAILED
6B FEEDHTR NORM DRN VLV FAILED X OP
70077812 H1AF -1AFLV-1506B
ON
70091658 H1AF -1AFLV-1532B
POSITIONER BELL CRANK DEGRADED
70094704 H1AE -10-Z-370C
CROSS FLOW CABLING DISCREPANCY
'B' MOISTURE SEP DRAIN TNK LVL
70097600 H1AF -1AFLIC-1040B
CHALLENGE
70097677 H1AE -1AELV-1785
Incorrect S/U Lvl software config loaded
70097936 H1AF -HS-1506B
5B AND 5C FWH TRIP ON SCRAM
70097937 H1AE
RFP / RFPT AXIAL POSITION CONTROL
70099018 H1AE -1AEFV-1783B
B RFP MIN FLOW IND FLUCTUATING
75
SULCV POSTIONS NOT AT EXPECTED FOR 3
70099542 H1AE -1AELV-1785
PO
H1FW -1FWVY70099906 3769A
CRI A-RFP OUTBD BRNG VIBE FAILED
70101641 H1AF -1AFLV-1363A
'B' MOIS SEP LOW LEVEL, DRAIN VALVE
70103635 H1FW
CRIDS POINT A2352 IS FAILING
70105948 H1AF -1AFLV-1363A
'B' MOIS SEP LOW LEVEL, DRAIN VALVE
70106135 H1AF -1AFLT-1559B
FWH 5B Trip
70106403 H1AE -1AELV-1785
SULCV did not control Rx Level during 1/
70106475 H1AF -1AFLT-1559B
PROBLEMS RESTORING LT-1559B FOR FWH 5B
70106478 H1FW -1C-S-105
7366475Inboard Bearing Oil Leak
70107495 H1AE -1C-P-101
Delays in warming C RFPT
70111059 H1AE -1B-P-101
FEEDWATER FLOW OSCILLATIONS
70111105 H1AF -1AFLV-1464A
EXCESSIVE USE OF NEOLUBE PIPE SEALANT
70112065 H1AE -AE-HV-F032B
CLOCK RESET FOR RF16 MOV DCP
70113195 H1AE
Let him fix the RFP Minflow oscillations
70114760 H1AE
5 R16 PMS NOT REZEROED AFTER R15
70115348 H1AE
DFCS System Alarm
70115467 H1AE -AE-HV-F032B
DCP steps not incorporated in work order
Order
Functional loc.
70117252 H1AE
Description
Heat Exch. workflow process gap - RF16
F.O./2A FWH DRN VLV LIC OUTPUT NEAR
70118359 H1AF -1AFLV-1464A
100%
70128893 H1AE -10-D-497
Water dripping in lower relay room.
6.11
System Walkdown
Part of the Hope Creek Single Point of Vulnerability investigation is to perform a walkdown of
the system. The SPV activity required the system be investigated to find any field vulnerabilities
that could challenge this system and are not identified on drawings or procedures. This walkdown
does not include the Feed Pump Turbine rooms due to high radiation conditions. This walkdown
was also performed with the Conduct of Plant Engineering template for walkdowns as found in
ER-AA-2030 Attachment 4. The attachment is a great tool for tracking items that should be
monitored while on a walkdown. Other general inspections checked for leaks or oil from
76
equipment, coatings and insulation, area lighting, scaffolding installed per procedure and no
seismic ll/l issues., unusual noises, smells, leaking fluids and general housekeeping.
All the vibration trends and local panels were checked. The RFPs’ thrust has trended in the
positive direction and is not a concern at this time. The B RFP IB bearing vibration peaked in the
3rd qtr at 3.0 mils and has since trended down to 2.6 mils this quarter. No actions are required at
this time. The B RFP IB and OB bearing temperatures have been diverging indicating a possible
undistributed load or misalignment. This can only be addressed in the R17 outage when the B
RFP is overhauled.
An airline for the SULCV was found vibrating. The line was a copper tube and linear with a 90
degree elbow. The better practice is to install these lines with a spiral to remove any uneven load
and stress from the joints and fittings. The other alternative is to install a flex fit tube which
moves and absorbs the stress instead of the joints and fittings. A notification will be generated to
install a flex fit copper tube for these lines.
A small oil leak was noticed on the H1AE -AE-HV-1753A 6A FWH outlet valve. This leak was
too small to quantify but was documented with a notification.
The 2/3 trip logic with a single sensing line is located in the high radiation feedpump turbine
rooms and could not be checked at this time. During the refueling outage, engineering will
investigate this panel for a possible elimination to the single sensing line vulnerability.
Will continue to monitor vibration trends and initiate notifications if the trends regress in a
degraded pattern. Notification 20529257 has been generated to address the bearing metal trend
divergence. This notification has been rolled to the B RFP overhaul in R17 in WO 30142985.
Notifications 20530352 and 20530357 have been generated to address the shaking SULCV airline
and the 6A FWH outlet valve oil leak.
7.0
Scheduling Priority
It’s been deemed that any SPV represents a high consequence to a plant trip or derate and as such
warranted additional guidance to help the senior leadership team make informed decisions as the
scheduling priority of proposed SPV strategies.
77
Discussed here was the team proposed approach to providing input to scheduling priority based
on a low, medium, and high likelihood of occurrence. Each component in the vulnerability
matrix will be assigned a scheduling priority risk based on the below table.
Consequence
HIGH
MEDIUM
LOW
> 100,000 MW-hrs lost
10,000 - 100,000 MW-hrs
< 10,000 MW-hrs lost
lost
Probability
HIGH
HIGH
HIGH
MEDIUM
A degraded condition or negative
Implement within
Implement within
Implement within 2-5
trend exists
2 years or next fuel cycle 2 years or next fuel cycle
years or 2-3 fuel cycles
- OR Component has failed previously
Fast Track
Fast Track
Normal implementation
HIGH
MEDIUM
LOW
at PSEG
MEDIUM
A neutral trend exists that if not
Implement within
Implement within 2-5
Implement within > 5
improved could lead to a
2 years or next fuel cycle
years or 2-3 fuel cycles
years or >3 fuel cycles
Fast Track
Normal implementation
Long Range Plan
MEDIUM
LOW
LOW
degrading trend or failure
- OR Component has never failed at
PSEG, but industry OE exists
regarding failures
LOW
No degraded condition or trend
Implement within 2-5
Implement within >5
Implement within >5
exists
years or 2-3 fuel cycles
years or >3 fuel cycles
years or >3 fuel cycles
Normal implementation
Long Range Plan
Long Range Plan
- AND No PSEG failures
- AND No industry OE failures
Some of the vulnerabilities identified were procedure related resulting in Ops actions in response
to alarms. Other vulnerabilities were a lack of preventative maintenance strategies. These
vulnerabilities can be implemented sooner than the design changes strategies.
The following table lists vulnerabilities and their associated risk levels and their current mitigat
bring immediate management attention. The following SPV threats were assessed high risk and
recommended for priority at the next available refueling or forced outage opportunity. The bases
for the high risk is that these components have known failure or degradation based on historical
78
search of SAP and/or industry OPEX that could result in the high consequence of plan transient
or trip:
8.0
Vulnerability Elimination or Mitigating Strategies
The team evaluated different mitigation and elimination strategies for the identified
vulnerabilities. Elimination strategies consisted mostly of design changes to augment the margin
before the station would succumb to the vulnerability. Several elimination strategies have already
been approved by PHC and PRC and will be implemented in the following refueling outage in
Spring 2012. Other design changes will be scheduled for R18 in fall of 2013 or R19 in spring of
2015.
Some vulnerabilities were components that impact generation if failed but can not be removed
since they provide a protective function or are required for continued operation. Therefore
mitigation was the proposed solution. Mitigation strategies involved implementing new PM
strategies. Most of the new PM strategies were proposed to compensate for a lack of PM strategy
for the components that have a potential to impact generation.
Vulnerability: “RFP suction valve not 100% open” Trip (80105125 0020)
Recommendation: Modify Bailey logic to eliminate the RFPT trip actuation, but keep the
indication, for when the RFP suction valve is not 100% open. Install an alarm to preclude an
inadvertent valve closing
Basis: The RFP suction valve not 100% open trip is not necessary and can be removed. The
plant is also vulnerable to the open limit switch failure, control power fuse failure, breaker failing
open, or a bailey logic failure. The trip function is to protect the RFP in the event that the RFP
suction valve inadvertently closes resulting in low suction pressure. However, the RFPs are
already equipped with a low suction pressure trip to protect the RFPs from damage caused by
pump cavitations. The pumps are protected from a low suction pressure condition by a different
trip. In the event that the suction valve did inadvertently close the low suction pressure trip would
protect the pump. Install an alarm to preclude an inadvertent valve closing
Review of Salem station report and other industry reports have determined that all single switch
trips should be eliminated from the plant. The other alternative is to change the trip logic to a 2/2
or 2/3 logic. This issue was asked amongst industry peers and none of the other stations have this
79
trip function. The low suction pressure trip sufficiently protects the feedpumps from low suction
pressure without a suction valve close trip.
The contacts and other components should remain since they illuminate or extinguish the valve’s
position indication light. The drawing below expresses where the logic should be broken to
preserve indication while removing the unnecessary trip function. The only PMs for this
vulnerability are a 12Y limitorque PM. Mitigating actions are to process DCP presentations to
PHC.
Vulnerability: Lube oil vapor extractor failure PCR and enhancements
Recommendation: Generate a reliability PM to inspect and repair the lube oil components.
(80105125 0030, 0040, 0050, 0130, and 0140)
Basis: The action is to mitigate the vulnerability. After performing a search for existing PMs on
the lube oil reservoir, the search determined there were no PMs in place for the oil pump
discharge relief valves, the pump discharge check valves, the vapor extractor or to rebuild the oil
pumps. This PCR will address this gap and prevent age related failures for these components. The
PM will align the components and stagger the implementation to prevent multiple trains being
worked in one outage. There are 6 oil pump discharge relief valves, 6 oil pump discharge check
valves with 6 oil pumps, 3 flow orifices, and 6 pressure control valves. The PCM templates
80
recommend performing pump refurbishments ‘as required’. The check valves are 8 years, the
relief valves are required to be tested every 10 years, and the vapor extractor is a RTF component.
The recommendation is to perform work on each train once every 12 years.
The vapor extractor function is to remove toxic fumes from the lube oil rooms and to provide
drainage for the return portion of the lube oil system. A design change will not eliminate the
vulnerability and the vapor extractor can not be removed. The vulnerability will not trip the unit
or result in a derate. The only suitable strategy is to mitigate the vulnerability. There are no PMs
for these components. The action is to process a PCR and generate a PM that will inspect and
repair the oil vapor extractor. This action should be performed once every 12 years. Mitigating
actions at this time will be to process the PCR and to verify spares are onsite to implement the
PM. This action should include checking the vapor extractor discharge line and clearing it of
debris and residue. If the exhaust line clogs it will prevent the extractor from maintaining a
negative pressure on the reservoir. Reference (OE 31000) for a fire starting due to oil leak
ignition. The station allowed oil to drip and accumulate to the point where it contacted a hot
surface, atomized, and ignited burning the surrounded spilt oil.
The oil check valves’ function is to prevent oil from flowing backwards to the reservoir. This
could cause a trip of the RFPT on low bearing and control oil pressure. The vulnerability would
exist with any check valve and is not warranted. There are no other means to eliminate these
check valves. Therefore SPV elimination is not possible but it can be mitigated by generating new
PMs. These PMs can be performed in conjunction with the respective proposed oil pump
refurbishment PM. The PM scope should disassemble the check valve and replace the internal
subcomponents as required. Post maintenance testing should be performed to verify the check
valves will close as required. Mitigating actions at this time will be to process the PCR and to
verify spares are onsite to implement the PM. Additional actions are to verify the FLOCs get
created for these check valves. Reference OE (341-020519-1 and SER 27-87) for events that
caused a loss of oil pressure due to oil check valves failing to close when the pumps were
swapped.
The oil pump function is to provide adequate control oil to the RFPT steam control valves and
allow for steam valve actuator to respond to a demand change as required. Another oil pump
function is to provide lubricating oil to the bearings. This ensures that the bearings and the shaft
will continue to operate smoothly without excessive heat and vibrations. There is an Aux oil
81
pump and the function is to be able to auto start as required without perturbation to bearing or
control oil header pressures. The vulnerability can not be eliminated since these pumps are
required to be in service to maintain RFPT operation. A pump upgrade would not eliminate the
vulnerability or guarantee internal degradation would not cause the pump to fail. By performing a
pump overhaul, the station can identify degraded parts and replace as needed and prevent an
equipment failure from occurring. The recommendation from the PCM template is to rebuild as
required. The installation will have to be performed when the RFPT is offline in the refueling
outage. However, these pumps can be rebuilt while online and staged for installation once the
refueling outage starts. This will help alleviate outage resources demand. The rebuild frequency
should be once every 12 years, per pump. Mitigating actions at this time will be to process the
PCR and to verify spares are onsite to implement the PM.
The oil pumps’ discharge piping is equipped with a pressure relief valve on the pump discharge
piping. The relief valves prevent over pressurizing the oil pump discharge line and deadheading
the pump. The oil pumps are not equipped with a high discharge pressure trip function and the
pumps would dead head if a block developed. The vulnerability is if the valve falsely lifts, it
would cause a pressure transient that would trip a RFPT on low control oil pressure or bearing oil.
The relief valves can not be removed since there would be no means to protect the pump from an
over pressurized condition. No PMs were identified while performing this evaluation. Mitigation
is the best strategy for addressing this vulnerability. A PM to inspect the relieve valve internals
such as the spring, fittings, valve body, and that the valve will lift at the required set points would
be sufficient. Verifying the valve internals are satisfactory and the valve will lift when required
will mitigate the vulnerability. These actions should align with the respective oil pump for
optimization. Immediate mitigation actions are to process the PCR and verify spares are onsite to
implement the PM. Reference (OE Event 341-020519-1 and OE15380) for details on how a
discharge relief valve failed open and resulted in a sudden loss of header pressures. The loss of
header pressures caused the feed pump to trip on oil pressures causing a SCRAM.
The oil pressure control valves’ function is to control oil pressure to the bearing header and the
control oil header. The pressure control valves maintain pressure across the headers. The control
oil pressure valve ports excess oil to the bearing oil line. The bearing oil header control valve
ports excess oil to the reservoir. The vulnerability for this component is if the control oil header
valve failed, it would fail open and port excessive amounts of oil to the bearing header line. The
bearing header would respond and port the excess oil to the reservoir. This could result in a RFPT
82
trip on low header pressure. The bearing oil header valve is designed to fail closed and would
preserve oil flow to the bearings. Removing these pressure control valves would require a major
design change to the oil system and is not recommended. The most effective strategy is to
mitigate the SPV with a new PM. These valves currently do not have PMs and need to be
addressed. The two valves should be worked during the same outage so they may be calibrated
together. Calibrating the two valves together would verify the proper pressures are being
maintained for the headers. The control header pressures valve impacts the bearing oil pressure
control valve. If the control valve is porting excess oil to the bearing oil line, the bearing oil
pressure valve will attempt to maintain oil as close to the set point as possible. Immediate
mitigation actions are to process the PCR and verify spares are onsite to implement the PM.
A PM was recently approved to flush the lube oil system. The flush will address the vulnerability
for the oil flow orifices becoming fouled due to sludge and oil breakdown. The PM will be
scheduled on a 6Y frequency and can line up with each oil system PM. No actions are required at
this time.
Vulnerability: RFPT oil pressure SPV (80105125 0060)
Recommendation: Implement DCP for oil SPV
Basis: The action is to eliminate the lube oil low pressure SPV. Hope Creek approved a design
change to install 2 larger accumulators on each RFPT oil train and modify the bailey logic to auto
start the backup oil pump instantaneously in the event running oil pump trips. The original low oil
pressure auto start feature will remain and continue to start an oil pump if header pressure drops
to the start set point. The previous auto start logic was based only on pressure and would start the
back up oil pump when a low oil pressure condition was sensed. The back up oil pump breaker
remained open while pressure decayed, costing valuable time to restore oil pressure. The
approved DCP will decrease the time between the main oil pump trip and the aux oil pump start.
. The new logic will allow the breaker to close immediately following the main oil pump trip,
instead of after the pressure decays to the lower auto start set point. The remaining mitigating
actions for this SPV are to verify the new, larger accumulators are installed and the bailey logic is
modified to the required specifications in the next refueling outage.
A considered mitigation strategy was to change the pump auto start set points. The mitigation
would have raised RFPT oil pump auto start pressure so the pump could start sooner. This was
83
refuted because there was too little margin in between operating pressure and the auto start set
point. The back up oil pump would auto start when not required resulting in 2 oil pumps in
service. The back up oil pump would need to be secured which resulted in pressure perturbations
to maintain oil pressure on the header. This strategy was refuted since it created more risk than
benefit while addressing the vulnerability.
Vulnerability: RFPT moisture drain valve not 100% trip (80105125 0070)
Recommendation: Perform DCP to eliminate this trip function. Install an alarm to preclude an
inadvertent valve closing
Basis: The action is to eliminate the trip function on the moisture drain valve. If the drain valve
is not 100% the respective RFPT will trip. This elimination strategy is similar to the one listed in
section 8.1. The trip function has no margin to prevent an instantaneous trip actuation since there
is a single contact switch with no time delay buffer. The station is vulnerable to the open limit
switch failure, control power fuse failure, breaker failing open, or a bailey logic failure. The
station should not attempt to harden the trip since it is not necessary to remove moisture from the
turbine case. Industry peers have confirmed that this trip function is not required and is not
implemented at other stations. There is indication of the valve’s position on the process
monitoring computer. Operators can identify if the valve is or is not full open.
The trip function should be removed from the drain valve while keeping all the other indication
and feed to the process computer. An alarm should be installed to preclude an inadvertent valve
closing. A DCP needs to be processed to perform the elimination strategy. Refer to the attached
logic drawing below for the scope of the proposed change. The next mitigating actions are to
process the DCP request presentations and present to PHC subcommittee and PHC. This issue
was polled to the industry and asked which stations have a similar trip configuration. None of the
stations polled indicated they have this trip.
84
Vulnerability: SPV - RFPT thrust bearing wear detector trip (80105125 0090)
Recommendation:
1. Install second redundant thrust probe. The thrust probe will be of the same make and
model as the current thrust probe.
2. Modify the bailey logic to actuate the RFPT trip when both high thrust signals are
received. An ‘And’ module will have to be added to the bailey card for both thrust signals to
output a single trip signal to the trip circuits.
Basis: The actions are to harden the thrust bearing wear detector trip. The current logic has a
single probe with no time delay. There is no margin for this trip function and could
instantaneously trip the respective RFPT. The trip provides a protective function to the RFPT. If
the turbine is able to thrust excessively the possibility of stationary to rotating component contact
is possible. Review of the RFPT clearance data confirms that the thrust bearing clearances have
the least margin of all the ‘rotating to stationary’ parts tolerances. This trip function can not be
eliminated since there is no redundancy to protect the equipment from excessive thrust and
‘stationary to rotating parts’ vulnerability. Additional margin can be installed to make the trip
function less susceptible to a false actuation. Adding the second probe and modifying the bailey
logic will eliminate the single point of vulnerability that will cause a pump trip. The thrust
bearing wear detector will still provide its intended protective feature. Mitigating actions include
a PM to perform calibrations every 36 months. Additional mitigating actions are to present to
PHC subcommittee and PHC to process a DCP. Installation should occur during the refueling
85
outage in Fall of 2013. Reference (498-950124-1) for a single thrust probe failure that resulted in
a unit trip. The design was similar to Hope Creek since there was a single probe providing
indication. The false indication drove Operators to reduce feedpump speed and trip the pump.
Vulnerability: SULCV instrument air tubing enhancements. (80105125 0110)
Recommendation: Install flex fit bronze tubing on components exposed to vibration.
Considerations should be given to plant equipment lines whose failure could result in plant
derates. The SULCV tubing would benefit most from flex tubing.
Basis: The action is to implement the enhancement to improve the equipment air lines. Currently
the steel and copper tubing installed is subject to light vibrations. The system walkdown observed
the equipment lines vibrating. A line break in the air supply would result in the SULCV failing
closed on a loss of air. This would not cause a derate during full power operations but would
challenge the plant on shutdown and start up. The steel and copper tubing ends should be fitted
with flex fit tubing. The tubing will not change plant configuration or functionality of the
SULCV. Current mitigating actions are to isolate the SULCV bypass line and stroke the SULCV
one month before an outage. This verifies if the SULCV is capable of performing its function and
will detect degraded components. The next mitigating action is to add flex fit tubing to the lines.
Risk exists while isolating the SULCV line while at 100% power, if the SULCV is suddenly
needed to stroke and the respective bypass line is unavailable. The tubing should be installed in
October 2013 outage.
Vulnerability: Revise Ops procedure for vibration response (80105125 0120)
Recommendation: Change Ops procedure HC.OP-AR.ZZ-0028 and HC.OP-AR.ZZ-0022 to
include guidance for response to axial vibration experience on the RFP and RFPT journal
bearings. Vibrations should be confirmed before reducing pump speeds. Bearing temperatures
should be monitored for an increase and/or unexpected fluctuations to verify high vibrations. If
bearing temperature rises or shows an unexpected or erratic change then reduce RFP speeds.
Basis: The action is to eliminate the vulnerability. Operations concern was that the referenced
procedures direct operations to reduce RFP speed to maintain vibrations below the danger set
point, which could be done unnecessarily without ruling out a false signal. This is the appropriate
action to address actual vibration increases and should remain in the procedure. However, if the
vibrations are the result of false indication, operations would inadvertently reduce RFP speed due
to false indications. Bearing temperatures are expected to rise if the bearings are experiencing
86
excessive vibrations. Before reducing RFP speeds, operations must verify the bearing
temperatures have not shown a change that would result from vibrations. A false vibration due to
a degraded instrument would show no temperature indication. Vulnerability mitigating PMs exist
to minimize the possibility of false indication. The vibration monitoring equipment is calibrated
every 36 months. To eliminate the vulnerability Ops procedure should be changed to prevent Ops
from reducing RFP speed without verifying the respective bearing temperature has not changed.
Vulnerability: Stagger RFP suction pressure trip set points (80105125 0150)
Recommendation: Modify the trip set points for the RFP suction pressure to trip at staggered
pressures. Currently all 3 RFP’s have 2/3 logic and a 5 second time delay that actuates when RFP
suction pressure falls to 230 psig. For 1 RFP change the suction pressure trip set point to 240
psig. For a different RFP change the suction pressure trip set point to 250 psig. Leave the third
RFP suction set point trip as is.
Basis: This action was suggested by the SFRC. The suggestion was to stagger 2/3 trip pressure
set point to prevent a simultaneous trip of all 3 RFPs. A low pressure condition may exist that
would drop low enough to actuate the trips for all 3 RFPs and would take the unit offline. If a
pressure transient does occur, tripping one pump before the others would restore pressure and
prevent the other 2 from tripping. The change in set point is for a higher pressure and a more
conservative change.
The SFRC other method to address this vulnerability was to stagger the time delays. Staggering
the time delays would reduce margin and trip the RFP sooner. A change in the opposite direction
would make the pumps more vulnerable to damage in the event an actual low suction pressure
existed. The pump would be subject to cavitations for a longer period of time. The decision to
stagger the pump time delay trips was refuted for this review.
Mitigating actions already calibrate the sensing instruments and switches every 18 months. The
next action is to process the set point change request and implement the change. The change
should be implemented during the R18 outage in October 2013. This is a set point change and
does not require a DCP but will require an update to the ICD cards.
87
Vulnerability: Create PM to replace the RFPT expansion joints (80105125 0160)
Recommendation: Create a PM to replace the expansion joints on the RFPT. The expansion
joint replacement should be performed every 18 years and align with the respective RFPT turbine
overhaul.
Basis: These expansion joints do not have PMs for replacement. Aging will eventually result in
failure forcing a loss in generation. The extent of the loss depends on the remaining time until the
following refueling outage. If a failure results early in the operating cycle the loss the station will
be forced to operate at a derated level longer than if the failure occurs closer to the end of the
operating cycle. If the losses endured while operating in a derated state will exceed the cost of a
maintenance outage, the station would have to plan a maintenance outage to repair the leak.
An 18 year PM is the most effective duration to replace the expansion joints. To gain access to
the expansion joints, the respective RFPT turbine must be disassembled. It would be effective to
schedule a replacement while the LP turbine is disassembled. Each respective RFPT is
overhauled every 6 years. OE states that these expansion joints begin to fail after 20 years. An 18
year PM will meet the life expected limits and not require addition work be performed in the
refueling outage. The mitigating action is to process the PCR to implement the suggested PM.
Reference (OE17885 and 19832) For the impacts an expansion bellow failure can have on plant
operations.
Vulnerability: Upgrade the A,B,C 6FWH level transmitters (80105125 0170 & 80103819)
Recommendation: Ensure the DCP is installed in R17. The DCP has already been issued and
is currently scheduled for R17.
Basis: The strategy to eliminate the vulnerability has already been evaluated and is scheduled for
the next refueling outage. The current configuration for the narrow and wide range transmitters
has them all within the #6 FWH rooms. These rooms are inaccessible with the heaters online due
to heat and radiation levels. This DCP will install the level control loops outside of the FWH
rooms so they may be calibrated and repaired while online. Several events have occurred at Hope
Creek that involved the level control function of the heaters to fail or degrade. Failure of level
control function has resulted in level rising or falling causing the dump valve to lift. In some
instances the heater tripped resulting in a forced derate. Mitigating actions have already been
planned for the next refueling outage in Spring of 2012. The only remaining actions are to verify
the design change is installed and the new level control loop functions as expected. Reference
88
(EPIX 77 and 395-010501-1) for instances FWH dump and drain valves can fail to open and will
result in the trip of a FWH.
Vulnerability: Upgrade the A,B,C 6FWH Upgrade FWH wide range level trip and indicating
Circuits (80105125 0180 & LTAM H-11-0057)
Recommendation: Ensure the recommendation is approved by PRC and is installed in R18.
The recommendation has been approved by PHC for a conceptual design study.
Basis: The FWH level Control Panel Wide Range level circuits uses Westinghouse 7500
electronic Signal Conditioning Cards (SCC) & Alarms Cards (AC). The SCCs and ACs are
obsolete and require periodic refurbishment of electronic components to maintain basic
reliability. These controllers have been subject to age related failures since 2002. Actions have
been presented to PHC and approved. The team determined these controllers should be upgraded
no later than the refueling outage in Fall 2013. The proposed work consists of the replacement of
72 cards, 24 cards per panel. It is proposed to utilize OTEK Model HQ-114 Digital
Programmable Intelligent Controllers (IC) or Foxboro 762 digital controllers. The intent is to
have each IC provide power, trip, alarm and indication for each wide range level transmitters.
The proposed solution have 12 new controllers/panel to replace the 24 signal conditioners. This
action item will be used for tracking purposes and to verify the corrective action is performed on
time.
Vulnerability: RFPT rupture disc failure (80105125 0190)
Recommendation: RFPT steam rupture discs require a replacement PM. Add the rupture disc
replacement to the RFPT overhaul.
Basis: The action is to implement a PM change and add the rupture disc to the RFPT overhaul
PM. The overhaul procedure has a note to inspect for signs of steam leaks however this will not
prevent a sudden failure of the disc during an operating cycle. The integrity of the disc can not be
evaluated with a visual inspection and will age until failure occurs. The rupture disc is designed
to protect the exhaust line expansion joints in the event that the line over pressurizes. The disc can
not be removed, eliminating the vulnerability. Mitigation is the preferred method for addressing
the vulnerability. Including action to replace the disc during the RFPT overhaul is the most
effective strategy to address this vulnerability. The next mitigating action is to process a PCR to
add the rupture disc to the overhaul PM.
89
Vulnerability: RFPT loss of power trip solenoid (80105125 0200)
Recommendation: Eliminate the trip solenoid and trip that occurs due to a loss of power to the
trip system. Install an alarm that will annunciate due a loss of power to trip system so the pumps
will not trip on a loss of power to the trip system.
Basis: The strategy is to eliminate the vulnerability with a design change to the trip system. The
current purpose of this trip is to stop the turbine if there is a loss of power to the trip system.
These solenoid valves will dump oil back to the reservoir in the event that the trip system loses
power. If they are not energized they will lift and trip the RFPT. Therefore a momentary loss of
power to the trip system would trip the RFPT. These solenoid valves should be removed and an
alarm should be installed that will annunciate in the event trip power is lost. Removing these
solenoids will keep the RFPT online and eliminate the possibility of a momentary loss of power
that would result in a trip.
In the event that power to 1A(B,C)-D-318 panel is lost, or failure of either normally charged
solenoid coils will result in a feedpump trip. The trip solenoids are subject to a fuse failure as well
which also results in a RFPT trip. In the event that power is lost to the trip system, the manual
push buttons would still be available and will shut down the RFPT is Operations determines it is
necessary to shut them down.
Currently there are PM tasks that verify the function of the solenoid valves to open when not
energized. DCP should be implemented in R19. This DCP will require more engineering
compared to the other design changes of this review. The following actions for this item are to
process the PHC sub committee and PHC presentations.
9.0
Review of System Vulnerability Initiatives from Other Sites
The team reviewed other SPV reports from other stations in the fleet to compare corrective
actions with. This review will confirm the proper actions are being implemented as well as other
issues that may have been overlooked. Hope Creek is implanting actions as part of the SPV
review and actions not considered SPVs but are enhancements for improving equipment
reliability. Some of the major trends noted across the industry are as follows:
90
 Common themes for the industry is to upgrade of single 1/1 trip logic to 2/3 trip logic. Hope
Creek verified that all of the 1/1 logic was removed or upgraded to remove the SPV. The single
contacts for the pump suction valve and Turbine drain valves not being 100% open will be
removed. The excessive thrust bearing wear detector will be upgraded so a single switch is not
responsible for actuating a trip.
 Expansion bellows are being overlooked as “passive” components and do not get replaced.
Several sites identified the expansion bellows as an SPV that is subject to aging that requires a
PM to mitigate. The team determined that Hope Creek replaced their expansion bellows after
20-23 years of service and has action to implement a PM program.
 There were some differences between the stations when performing the report and evaluation.
Some stations defined a Single Point of Vulnerability as a derate greater than 5%. Some
stations used the 20% or greater criteria to identify an SPV. Some Exelon and Entergy plants
were more involved as a fleet than as individuals when performing the report. It seemed like
when these reviews were done, it was more beneficial to share ideas directly while writing the
report.
9.1
Grand Gulf station SPV report review
Company – Entergy
Station – Grand Gulf
Findings - The team reviewed the 2010 reactor feed system vulnerability report from Grand
Gulf Station. The review was focused on the reactor feedpump and condensate system with
several of the single point vulnerabilities similar to the ones identified in the Hope Creek
vulnerability report. The following are noteworthy observations:

Feedpump trip logic uses 2/2 or 2/3 logic to eliminate SPV threat. For example, the low
suction and high discharge trips are 2/3 coincidence. In contrast, Hope Creek has
vulnerability with 1/1 logic for the RFP suction valve position not 100% open, RFPT
drain valve position not 100% open, and the thrust bearing wear detector. Other trips
were verified to be 2/3 or 2/2.

Many of the instrumentation switches are off a common instrument line. Hope Creek
shares the same vulnerability, the low bearing and low control oil pressure trip switches
2/3 logic but are all off a common sensing line.
91

Grand Gulf identified the expansion bellows as a passive component that does not have a
PM but is subject to age related failures. Failure of the expansion bellows results in a loss
of condenser vacuum and requires a unit outage to repair.
9.2
Salem Station SPV report review
Company – PSEG
Station – Salem
Findings - The team reviewed the 2010 reactor feed system vulnerability report from Salem
Station. The review was focused on the reactor feedpump and condensate system with several of
the single point vulnerabilities similar to the ones identified in the Hope Creek vulnerability
report. Salem and Hope Creek have similar feedpumps, turbines, and oil delivery systems. The
following are noteworthy observations are as follows:
 Salem noted their turbine loss of power trip solenoids as an SPV as well as Hope Creek.
In contrast, Salem recommended the trip solenoid valves design should be changed
from de-energize to actuate (trip valve closed) to energize actuate. This would
eliminate the possibility of failure on a momentary loss of power or a fuse failure. At
Hope Creek, the stations is going to change the design and remove the loss of power
trip solenoids from the plant.
 The rigid stainless steel (SS) instrument air supply line from the instrument tray to the
air supply regulator should be changed to high pressure SS flex hose. Also, other rigid
SS tubing connecting the accessories on the actuator should be given consideration to
the use of flex hose to the extent possible. Hope Creek has performed a similar change
for the feedwater minflow valves and will implement the change as an enhancement.
This change applies to copper tubing in addition to SS.
 Several changes are being made to the trip system. At Salem there are several trips that
use 1/1 logic without time delay. Their actions from the SPV review are to change the
1/1 logic to 2/2 or 2/3 logic to prevent a momentary false switch actuation forcing a
feedpump to trip.
 There are several action for improving the lube oil delivery system. Most of these
actions are similar to the ones Hope Creek identified including orifice inspection,
92
check valves, and relief valves. In contrast, there are no actions to flush the oil system
as suggested in the Hope Creek review.
Company – Entergy
Station – Indian Point
Findings - The team reviewed the 2010 reactor feed system vulnerability report from Indian
Point Station. The review was focused on the reactor feedpump and condensate system with
several of the single point vulnerabilities similar to the ones identified in the Hope Creek
vulnerability report. The following are noteworthy observations:
 Indian Point identified the expansion bellows as a passive component that does not have
a PM but is subject to age related failures. Failure of the expansion bellows results in a
loss of condenser vacuum and requires a unit outage to repair. There were no
documented failures at Indian point but they did have the original bellows still installed
and were perusing action to replace them.
 1/1 logic was being replaced at Indian Point with 2/2 or 2/3 logic. This action is being
implemented at Hope Creek as well.
 Lube Oil Leaks/System Issues (AC motor swap over without trip). Chronic leaks at
Flanged and threaded joints. Need consistent design/test approach to ensure reliable
AC motor swap over on loss of running pump and proper Lube oil cooler equalizing
and vent lines for cooler swaps. Hope Creek has a similar issue but Indian Point stated
an action was to improve check valve location and allow for a bumpless swap of the oil
coolers. This is not the same issue that Hope Creek has.
Attachment 1 – Vulnerability Matrix
Attachment 2 – OE Vulnerability Matrix
Attachment 3 – Cooper Reactor Feed System Vulnerability Report 2007
Attachment 4 – Exelon OPCC list for Feedwater System 2010
Attachment 5 – ANO Feed and Condensate System Vulnerability Report 2010
Attachment 6 – Indian Point Feedwater System Vulnerability Report 2010
Attachment 7 – Salem Feedwater System Vulnerability Report 2010
93