Advisory
Value
o
Quality
o
|
Excellence
Excellence
o
|
Assurance
Relationship
o
Integrity
o
Dependability
Advisory
|
Excellence
|
Assurance
Independent Incident
Investigation
Lake City Power House GenSet 1 Accident - 15 Nov 2022
Project No: E 0202001a LKC
Advisory
|
Excellence
|
Assurance
1. History
GenSet Particulars
Origin
Engine
Used marine engine, tag BAREBONE I; Re-purposed for on-shore power
generation
Make: MAN
Year of Manufacture: 2015
Model: 9L21/31S
Country of Manufacture: India
RPM: 1000
Serial No: E 403340251
Rated Capacity: 1980 kW
Alternator
Make: TOYO Deiki
Year of Manufacture: 2014
Model: TD 110
Country of Manufacture: India
Capacity: 2351.5 kVA
Serial No: T 02738
Voltage: 6.6 kV
Frequency: 50 Hz
Rated Capacity: 1980 kW
PF: 0.8
GenSet Particulars
Make: MAN
Turbocharger
Model: TCR 18/42
Year of Manufacture: 2014
NS Max: 40,400 rpm
NC Max 39,600 rpm
T Max: 650 oC
Country of Manufacture: USA
Serial No: T 02738
History
Mid-2022: Lake City Holdings, the Owner of Lake City Power Plant concluded its contract with IMS and
1
took over the Power Plant assets.
Running Hours (since last overhaul): well overdue.
Owner self-managed operations and maintenance initiated. No proper handing-over - taking-over
15 Sep 2022: First Accident A. GenSet shutdown.
Running Hours (since last overhaul): 33,906
2
Reported Reason: Jacket water leakage - Leakage from HT/Exhaust multi duct separating wall; seepage of
HT water through exhaust ports into the cylinder units. Position 7.
Root Cause Analysis: Report not available; Asset Condition Assessment: Report not available
13 Nov 2022: GenSet Partial Overhaul - with partly new and partly ‘existing used / retained’ Parts ; Own
3
team, Own supervision
Parts Used: Partly new (HNH, TriTech), partly existing, used / retained parts in store.
Overhaul Report: Not available
History
15 Nov 2022: First Accident A. GenSet Breakdown; Fire; Asset Damage
4
Running Hours (since partial overhaul): approx. 28
Reported Reason: Engine Overspeed
Root Cause Analysis: Independent Investigation undertaken by EE; parallel investigation by MAN.
Heavy Noise reported from the Engine
5
Fire Break out seen requiring fire fighting to be undertaken.
HMI reported to be out of order; Reportedly (i) giving false data, (ii) no possibility of ‘power-up, (iii) data
not accessible.
Immediate Sequence of Events
Sequence
13 Nov 2022 - Engine Started and put on Load up to 1600 kW within 1 hour and put on Load Sharing
15 Nov 2022 17: 38; Frequency starts to decrease
1
15 Nov 2022 17: 46; Circuit Breaker opens; Engine continues to run
15 Nov 2022 17: 53; Safety Chain Tripped; Engine still Running
15 Nov 2022 17: 58: 37; Emergency Stop Applied
15 Nov 2022 17: 58: 52; Emergency Shutdown
Engine Running Normal
Frequency starts to decrease
Circuit Breaker Opens
20 mins
Immediate Sequence of Events
Safety Chain Tripped; Engine
still running
Emergency Stop applied
Spare Parts Used
Reported use of Spare Parts on Position 5 & Position 6
Cylinder Heads: Existing Cylinder Heads were reused for both positions. Those obtained from HNH and
TriTech were not used on other positions.
Valve Seats: Likely those supplied by HNH
1
Cylinder Liners: Likely those supplied by HNH
Kits: Likely those supplied by HNH. Note: Kits have been partially used
Piston Rings; Likely those supplied by HNH
Big End Bearings; Likely those used by HNH
2
Spare Parts Lot 1 supplied by HNH (OEM in China) inspected in Plant Stores on 8 Nov 2022; Some queries raised.
Response to the queries from HNH’s OEM were awaited as at the date of the Overhaul.
Spare Parts Lot 2 pertaining to Fuel System were awaiting delivery in China as at the date of the Overhaul.
3
Additional Spare Parts supplied by TriTech (OEM in China) were also received, inspected and admitted in Plant
Stores.
4
Further, Spare Parts from previous overhauls, left-overs were kept in Plant Stores (IMS ?, Sources not clear)
Advisory
|
Excellence
|
Assurance
2. Findings
Immediate Cause of Failure
Position No 5 - Hydraulic Lock
All physical, symptomatic evidence found on the GenSet 1 pointed towards a ‘hydraulic lock’. This is
confirmed by the following;
1
(1) Presence of water in the Position 5 chamber
(2) ‘Pinched’ improperly installed ‘O’ Ring Seal between; This allowed water ingress into the firing
chamber
(3) Piston found stuck beyond TDC Position
Position No 6 - Disbalance owing to stress caused by ‘Hydraulic Lock’ in Position No 5.
2
(1) Piston found completely broken, approaching full compression / TDC position.
(2) Physically Position 6 is the very next one to Position 5.
(3) Position 6 was already moving towards TDC
Root Cause - Diagrammatic
Route of Jacket Water ingress into
combustion chamber; between
lifted face of Head and Liner.
Head
pinched ‘O Ring’
Seal
Water Passage
Head
Liner
Piston
Liner
Water
Position No 6
Root Cause of Failure … 1
Poor Workmanship - Maintenance
Incorrect Installation of ‘O’ Ring Seal; Not carefully put in position. Caused Pinching of ‘O’ Ring Seal.
No care while lowering of Head on to Liner. No evidence of use of ‘LOCTITE’ or similar material per MAN
recommendations.
Uneven or Inadequate Head Tightening Torque; Uneven metal contact between Head and
1
Liner. No record kept of the ‘tightening torque’; The hydraulic machine for torque-ing doesn’t work and is unreliable
(leaks).
Position 6 Injector Bad Fitting; On opening, one hold-down nut was found missing, the other was found
loose in position. Power Plant Safety Compromised
No Pre-Maintenance Health Checks: No ‘benchmark’ information kept on record for the first
Accident nor ‘benchmark’ data kept for the ‘partial’ Overhaul GenSet Overhaul Compromised.
Root Cause of Failure … 2
Poor Post Overhaul Workmanship - Maintenance
No Jacket Water Pressure Test; No record or evidence of this crucial standard requirement having been
performed. Could have been first indication of water ingress (if ‘O’ Ring started passing immediately).
No Compression Test; No evidence or record of Peak Pressure Tests undertaken on each Head per standard
2
procedure. No test equipment at Site. Such test could have given early indication of ‘O’ Ring damage (if it started
passing at the outset).
No Valve Operation Sequence Test; No evidence or record of such testing. Such test could have
released any water ingress from the combustion chambers.
No Slow Speed Running Test; No evidence or record of such testing. Such test could have helped
identify residual problems in overhaul.
Root Cause of Failure … 3
Poor Operations
Incorrect Emergency Shutdown Response; Instead rushed to start the other GenSet; Electrical
breaker was manually cut-off. Swift Engine shut-down / fuel cut-off could have saved severe damage.
Poor Operator Vigilance; Engine had been showing gradually lowering frequency for some time; No timely
action taken. Alert operator action could have saved severe damage.
3
Inadequate Safeties, Alarms & Trips; The Low Frequency Trip did not operate. Expansion Doors did
not operate. No evidence or record of testing of safeties, alarms and trips - before putting the engine online postoverhaul, nor on a regular basis. Crucial protections missing; Engine not adequately protected.
Expansion Tank Refilled on ‘Auto’; The Expansion Tank gets refilled automatically. No alarm,
Crucial indication missed / Emergency steps not taken. Timely action against Jacket Water leakage
indication could have prevented excessive damage
Manufacturer Protocols for GenSet Early Operation Ignored; Engine put on full load,
sharing-mode within a very short time (1 hr) without allowing for ‘running-in’.
Root Cause of Failure … 3
Poor Management
‘Partial Overhaul’ with Inadequate Technical Expertise, Supervision; Overhaul
undertaken suddenly, without adequate preparation, arrangements. Inadequate Technical expertise. GenSet Integrity
Compromised.
Decisions not Traceable; ‘Benchmark’ information and maintenance records not kept; Decisions taken are
4
not clear. The Expansion Tank gets refilled automatically. No alarm, Crucial indication missed / Emergency steps not
taken. GenSet Integrity Compromised
Inadequacy of Tools; Requisite Tools were not available with maintenance persons; Records of Calibration of
Tools not kept. Some tools not in working order. GenSet Integrity Compromised.
Spare Parts Installation Traceability; Alternate-source Spare Parts installed without (i) waiting for
Supplier response to clarifications / documentation, (ii) mix & match of existing parts and from new Suppliers - No
Trace-ability records kept.
Other Failures
Poor Workmanship - Maintenance
Poor Injector Calibration; No evidence or record of Injector Calibration for each Position (sound profile,
spray profile, pressures, flows ??). GenSet Efficiency Compromised.
Inadequate Valve - Valve Seat Fitting Checks; No record kept of valve and valve seat polishing
and mutual ‘mating’. Lack of Care for GenSet Efficiency.
1
Inadequate Rocker Arm Checks; No record kept of ‘Cam wear’, ‘Yoke wear’, ‘Push Rodd Assembly Play’.
Lack of Care for GenSet Efficiency.
Connecting Rods Swapped; ConRods for Positions 3 & 4 found installed on Position 5 & 6. Compromised
Engine balance and Wear parts Performance.
Incorrect Big End Bearing Assembly Fitting; No record kept of bearing clearances, ‘shims’,
torque-tightening of hold-down bolts. GenSet Reliability Compromised
Other Failures … 2
Poor Workmanship – Maintenance … (cont)
Head Pressure Test Checks; No record kept of Head Pressure Tests (Pressure Setting, Time maintained,
observations, etc). GenSet Integrity Compromised.
Non-compliant Maintenance Logbook; Record of overhaul measurements, settings, clearances,
1
etc. not recorded, kept per Manufacturer protocols. GenSet Integrity Compromised.
No Crankshaft Verification; No record kept of Crankshaft deflection, damage after Accident on 15 Sep
2022; No trace-able records, data available. GenSet Reliability Compromised.
Other Failures … 3
Poor Operations
Inadequate Fire-fighting Response; Firefighting response (equipment, training for opertaors and
workmen, etc. is not in place). Asset Integrity Compromised
‘Robot’ Data Logging by Operators; Records and physical evidence permitted to be ‘adjusted’ Some
2
records are clearly unmistakably created ‘post-event’. Some data continued to be logged as ‘normal’ by ‘rote’ after the
event - e.g. bearing Temps. Compromised Vigilance.
Low Fuel Oil Temp Readings Unresponded; Readings went unreported, unaddressed for an
extended period of time. This is an important parameterneeds to be connected to and alarm, tracking.
Summary
1. Jacket Water ingress into combustion chamber Position 5 caused
‘Hydraulic Lock’
Defective maintenance, lack of confirmatory checks, tests.
2. Position 6 - next on the ‘exhaust stroke’ and being a neighbour - took
maximum remaining brunt.
3. Hydraulic Lock caused of failure / breakage of (i) Connecting Rod, (ii) Big
End Bearing & Mountings.
4. Blow Out from Expansion Doors, Breakage in Lower Block - Due to (i)
Broken Parts flying out, (ii) Fire.
Ignition - Friction of rotating broken parts; Fuel - Very Hot Lube Oil
Vapours.
5. Power Plant Protections Compromised - Trips, Alarms.
Jacket Water Pressure, Under Current Safeties should have operated
in time. Expansion Tank Level Safety should be in place and responded to,
operated. Expansion Doors in body did not operate.
Summary … 2
6. Overspeed ‘Theory’ - No evidence supports this.
Wartsila certificate available;
Governor link open triggers mechanical ‘fail-safe open’; a fuel cut-off
would have saved GenSet from damage.
7. HMI - Functional and Accessible
Passwords are available with personnel the Power Plant; No problem
faced in powering up.
Reported False data post-event due to short-circuited wiring.
Current Alarm List showing for 15 Nov 2022 does not tie up with Trending
Records
8. ‘Valves Dropping’ Theory – No evidence supports this.
Valves mountings on Position 5 and Position 6 found intact;
No ‘strike marks’ or embedment found on the Piston Crown; Presence of
water under very high pressure doesn’t allow ‘valve drop’.
Summary … 3
9. Spare Parts - Use of Chinese source
No situation arose where spare parts quality could play a role in
the Accident or their contribution questioned.
Maintenance records unbale to show which parts were used
where in the overhaul.
10.HMI & the Plant Safeties Inadequate
HMI programming, settings of Alarms, Trips need to cater for the
sensitivity of MAN GenSet 9L21/31S type.
GenSet design requires specific system protections to prevent Hydraulic
Lock, Combustion Chamber pressurisation.
11. Possible Unauthorised handling of Evidence
HMI was been accessed on 16 Nov 2022 by someone, and many alarms
were \ acknowledged. The residual record point to ‘over-speed’ situation.
Summary … 4
11. Possible Unauthorised Handling of Evidence (cont)
Many of the operating and maintenance records seem to have been done
from ‘memory’ post-event. Documents have been amended.
Position 5 Injector Port seems to have been used to access the
Combustion Chamber post-accident.
12. Cooling Tower Operations
Cooling Tower Operations are not in good shape; require serious
attention.
12
Lake City Power Plant operations, maintenance; ‘loss of control’
Advisory
|
Excellence
|
Assurance
2. Way Forward
Way Forward
1.
2.
3.
Transition from ‘Vendor Operated Unit’ to ‘Self Managed Unit’ – Continue with
Controls
This is a valid shift and has been done successfully by many companies. This, however,
is done with due diligence and risk assessment.
Owing to unique circumstances, this was rushed at Lake City Power Plant. Thorough
diligence must be done of resources, capacities, capabilities to make it successful.
Control of ‘Change’ should be in place.
Overhaul of Lake City Power Plant Management - Immediate Steps
Multiple failures have been observed in all aspects of power plant management. The
other GenSets at the Power Plant are at a risk with existing systems, methods.
It is recommended that a complete overhaul of the Management, Systems be done.
The current system (and the previous one with IMS) does not fit in with Lake City
standards / promise to its Customers.
Recover & Rehabilitate Generating Set 1 - Undertake Feasibility
The Generating Set is not a ‘total loss’ / ‘condemned’. Value can be derived by lake
City.
Mishandling in removal, relocation can further damage, increase cost. Also, it needs
‘preservation’ in storage.
Way Forward
4.
5.
6.
Controls and Protections - Immediate Assessment, Improvement
HMI and other protections do not cater for the Power Plant Assets, and leave it
vulnerable. Poor design.
A hazard / Risk based assessment, and certain improvements are needed.
Emergency Fire Response – Immediate Assessment, Improvement
The hard and soft systems are wholly inadequate at the Lake City Power Plant.
It is recommended that a complete assessment are done, and reasonable
hardware and systems are put in place.
Alternate Supply Chain & Spare Parts Sourcing – Continue with Controls
Operations of plants with spare parts from Alternate Supply Chain / OEMs is a genuine
aspiration of Power Plant.
This needs to be undertaken systematically and with ‘control of change’ mechanisms in
place.
Advisory
|
Excellence
|
Assurance
3. Way Forward
Advisory
www.eexcellence.com
Value
o
Quality
o
Excellence
|
Excellence
|
Assurance
info@eexcellence.com
o
Relationship
+92 42 35787980
o
Integrity
o
Dependability
Advisory
|
Excellence
|
Assurance
Back Up Slides
OEM Spare Parts – Sourced from China
Connecting Rod Bearings / ‘Big End’ Bearings
Connecting rod Bearing/ Big End Bearing: In the Accident bearing shells blown out with big end bearing caps of
unit no.5 and unit no. 6. Bearing shell found among broken parts are mixed up. There are no evidence of
bearings melting or heating up. They are deformed identical to deformed bearing caps which suggest they were
deformed under stress applied freely moving and striking connecting rod.
Normally There is no identity mark on bearing shell, we can not differentiate which shell belongs to which unit.
Valve Guides
These long cylindrical shaped part are in which valve travels, This part is fitted in Cylinder Head. This part cannot
have any involvement in this accident
OEM Spare Parts - Sourced from China
Valve Seats
Inlet valve Seats: Inlet valve reside on this seat in close mode. This part is fitted in cylinder head. This part cannot
have any involvement in this accident.
Exhaust valve seats: Exhaust valve reside on this seat in close mode. This part is fitted in cylinder head. These
seats are water cooled, if there are any chance of water leakage it can only happen during cold condition of
engine as water flow stream is upward and exhaust valve seat faced downward. This part cannot have any
involvement in this accident.
Valve Cones
During inspection we found all valves of unit no. 5 & unit no. 6 firmly held by conical pieces. There was no sign of
valve dropping or guttering.
Valve of unit no. 5 seems bend inside guide due to piston crown stuck at TDC while crank shaft was moving as
well as cam shaft so valve pressed against opposite forces and got bend. There was one groove near top of stem,
which normally have rubber gasket to resist excessive flow of lubricant into exhaust side.
OEM Spare Parts – Sourced from China
Rotocaps
This parts resides on top of valve & spring in Rocker Arm area. Its purpose is to rotate valves for even facing of
seat and valve. This part cannot have any involvement in this accident.
Kit for Air Cooler
This part cannot have any involvement in this accident
Piston Rings
There are no abrasion or seizure marks on liner wall. Piston ring for unit no. 5 stuck with piston crown while for
unit no.6, they broke in small piece and scattered in engine unable to retrieve. There is no evidence of piston
ring involvement in accident. Lake City team confirmed they use piston ring after taking measurements and with
satisfaction. This part cannot have any involvement in this accident.
Kits for Water Pumps
This part cannot have any involvement in this accident.
OEM Spare Parts – Sourced from China
Cylinder Liners
There is no crack, scratch or scuffing mark in liner. Lake City team confirmed, prior to inserting they checked
ovality and take measurements and insert them after satisfaction. There is no evidence that accident happened
due to any problem in liner. This part cannot have any involvement in this accident.
Cylinder Heads - bare
Old Cylinder Heads were used on unit no. 5 and unit no. 6. HNH supplied Cylinder Head installed on unit no. 7 &
8. TriTech supplied Cylinder Heads were used on other positions. This part cannot have any involvement in this
accident.
OEM Spare Parts
Typical Cylinder and Piston
Assembly
Trips & Alarms Comparison with Trending … 2
Description
OEM alarm set
point
OEM tripping Values from CCP
set points
(trending)
Values from GCP 1
(trending)
Alarms on HMI Trippings on
of GCP 1
HMI
Lubricating oil system
1
2
3
4
5
6
7
temp. after cooler
press. after filter
differential pressure
pre lube pressure
Sump level
pressure before filter
cranckase protection
80 deg c
3.5 bar
1.5 bar
0.12bar
LAL /LAH
8
Main bearing temp.
level High
100 deg
4k
100 deg c
9
10
11
pressure after filter
leaking oil
tempreture
4 bar min.
level high
N/A
12
13
14
15
LT water pressure
HT water pressure
HT water tempareture
LT water temp.
0.4 bar
0.4 bar
90 deg c
3 bar
72
not available
not available
not available
not available
not available
level High
not available
105 deg
5k
105 deg c
not available
Fuel oil system
not available
not available
67 deg
Cooling water system
not available
not available
95 deg C
52 deg c
38 deg c
not available
not available
not available
not available
not available
not available
no
no
no
no
no
no
no
no
no
no
no
no
not available
no
no
no
no
not available
not available
not available
no
no
no
no
no
no
not available
not available
not available
not available
no
no
no
N/A
no
no
no
N/A
not available
Trips & Alarms Comparison with Trending
Description
16
17
18
19
20
21
OEM alarm set
point
OEM tripping Values from CCP
set points
(trending)
Exhaust & charge air system
600 deg c
not available
480 deg c
not available
50 deg c
not available
450
not available
not available
not available
22
exh. Gas temp. before TC
exh. Gas temp. out let cyl
temp. difference between each cyl
exh gas temp. after TC
Charge air press.. after cooler
Charge air temp. after cooler
Compressed air system
press. inlet Engine
23
over speed
1130
24
25
frequency
turbo speed
47.5 hz
26
winding temp.
6.5 bar
100
not available
Speed control system
1150
not available
45 hz
130
46 hz @ 17:46 hrs
not available
Alternator
85
Values from GCP 1
(trending)
Alarms on
Trippings on
HMI of GCP 1 HMI
not available
not available
not available
not available
not available
not available
not available
not available
no
no
no
no
no
no
no
no
no
no
no
no
no
no
not available
not available
not available
no
alarm @
17:53**
no
no
alarm @
17:54
no
not available
no
no
** Note: @ 17:53 safety chain tripped event occur at first as per HMI Data of GCP 1 , all alarms later can be considerd false