IF IT AIN’T BROKE, FIX IT
Factors Effecting the Life Cycle of Mission Critical Systems
By
Douglas H. Sandberg
Director of Operations ASCO Services Inc.
I’m quite certain that anyone reading this
recognizes the title play on an old axiom. With
the tremendous financial impact and the risk of
personal liability, many of those responsible for
mission critical operations are rethinking their
approach to systems. This paper focuses on the
factors, which effect the useable life of these
systems. For illustration, we’ll focus on the
engine generator / turbine and switching
elements however, the rational applies to all
electrical systems.
What is the life cycle of a given piece of gear? It
could be simply expressed as the period of time
beginning at installation and ending when it is no
longer usable. The expected usable life of
equipment may be dependent on number of
operations, characteristics, model etc. Figure 1
illustrates the factors, which make up the
continuing cycle of maintenance and determine
product life.
The equipment / system life cycle depends on all
of these factors beginning with the engineering
concept.
Proper Design & Application
The length of the usable life of any piece of gear
begins long before installation.
All too often we find some aspect of system
design or application lacking. It may be physical
restrictions, which prevent proper maintenance.
It may also be improper application of a product
type. Most manufacturers provide equipment
models that satisfy various requirements of the
facility. Automatic Transfer Switch
manufacturers, for example offer specific
technologies for specific applications. The major
types supplied by most manufactures are as
follow.
• Standard break before make with no service
bypass.
•
•
•
•
Standard break before make with service
bypass. This is basically 2 switches (one
automatic, one manual) in parallel. The
manual switch is used to bypass the
automatic for maintenance or replacement of
the automatic unit without impact to the
critical load.
Make before break or closed transition with
or without service bypass.
Delayed transition is designed to pause in
mid position thereby allowing electrical
fields to collapse, which in turn limits
current inrush.
Softload is basically a closed transition ATS
with a synchronizer and power controller.
This type of unit is employed to “walk” the
load from one source to the other. This
design is especially useful where the standby
source size is marginal or soft like natural
gas or methane plants.
These basic types of automatic transfer switches
are employed today in mission critical systems.
While individual components are important, how
they are configured in a system is equally
important.
For example, a closed transition switch is ideal
for feeding a UPS. While the initial power
outage transition to the emergency source of
power operates the same as if a basic break
before make ATS was installed, that’s where the
similarity ends.
• After the event, a seamless transfer back to
the normal source prevents the UPS from
loading the battery again.
• Should the owner decide to place the facility
on standby power in the event of impending
bad weather or for other reasons, a seamless
transfer can be made in both directions and
battery depletion is minimized.
• Should the UPS module or system be in the
bypass mode, a no break transition prevents
loss of load.
1
•
If the ATS is equipped with an isolation /
bypass feature, the ATS mechanism and
controls may be serviced without effect to
critical loads. Maintenance access means
this essential element will be in top
condition when called upon to operate.
Having decided what type is best suited for the
system demands; one must consider how the
individual ATS will interface with other system
components.
• How will loads be managed? It’s paramount
to protect the highest priority loads while
preventing the standby engine system from
being subjected to an overload. Are the
ATSs associated with the system equipped
with the proper controls and features?
• Are the governors, voltage regulators and
controls of the proper type to interact with
the UPS?
• What about communications and
monitoring? It is essential in today’s critical
environment that operators know instantly
the status of or changes in the system they
depend upon.
• Are time delays and voltage sensing
setpoints properly coordinated between all
critical members of the system?
Careful Handling & Installation
Here again events occur which can severely
effect the useful life of a piece of equipment
even before it’s installed.
• Did it ship well? Only incoming inspection
can tell. Inspection should take place
obviously before you accept the gear. The
presence of the manufacturer’s
representative is recommended. Waiting
until the gear is installed to discover a
problem creates many problems and may
impact commissioning schedules.
• Store the gear in a dry, safe environment.
Moisture or condensation can and will
degrade insulators or stand-offs. Moisture
also attracts dust and dirt, which may
become conductive.
• Those charged with installing the gear
should be thoroughly familiar with the
manufacturer's recommendations. Extreme
caution must be observed not to let metal
shavings or other debris contaminate the unit
or controls.
• Personnel should read the installation and
operations manual. Do not force controls or
•
the mechanisms. If you’re not sure what to
do, call the manufacturer for guidance.
Remove all shipping blocks, bolts etc.
Benchmark Testing
This aspect of commissioning cannot be
emphasized enough. The manufacturer should be
contacted to participate in the planning of the
benchmark or initial commissioning test. This
may well be part of the purchase package,
however if it is not, a few dollars invested at this
point may save a few thousand later.
• A team should include the owner, design
engineer, consultant and all manufacturers’
representatives. Usually, these benchmark /
acceptance tests include functional testing of
each component, functional testing of the
system and specific performance criteria.
For example, measure and record cold start
time from initiation of the start signal until
each generator closes to the bus.
• You must insure that all programmable set
points and time delays are understood,
properly set and indeed function. For
example, are the low voltage set points for
the ATS, UPS and engine control system
properly adjusted? Failure to coordinate
these values may leave the UPS on battery
while the ATS & engine system are unaware
of a problem or it may be that voltage is
sufficient and the UPS set point is simply set
incorrectly. Either way, the critical load
could be in jeopardy.
• Unrealistic tests invalidate results and
diminish the value of this important
requirement.
• Are the time delays adjusted properly?
Overriding short outages is very important.
Again, this feature should match the UPS
settings to prevent unnecessary hits to the
battery.
• Insure that load priority blocks are properly
assigned. For example, the ATSs feeding the
most important loads should be of the
highest priority and admitted to the
generator bus first. Conversely, these should
be the last to go in the event of generator
failure.
• Document and retain all setpoints and
adjustments on each piece of gear. Test
records should become a clearly defined
package in the system master file.
2
Document Control
Each piece of gear has documentation. The
system also has documentation associated with
it. This body of information includes operator’s
manual, bills of material, one line schematic
diagrams, detailed control and power schematics,
mechanical details, maintenance records,
software and test results.
•
•
•
•
Maintaining this information is paramount.
It is your only reference in an emergency.
Benchmark control settings provide a
default reference.
A master one-line diagram on the wall under
glass or plastic and adjacent to switchgear is
a quick reference point in an emergency or
when operating the system.
Personnel should drill on system operational
knowledge so that it becomes second nature
in an emergency situation.
Contact information for all key service
providers and manufacturer’s
representatives is critical and must be
maintained and updated as changes occur.
Routine Preventive (Scheduled) Maintenance
The point to be made here is while routine
maintenance according to the manufacturer’s
recommendations is essential; it does not totally
preclude a problem. Mission critical systems are
complex. Many actions occur when they are
called upon to change state. Regular scheduled
maintenance does however increase the odds in
your favor.
This is one of the most neglected aspects of the
critical power system. In this context, neglected
can mean that maintenance is simply not done on
a routine basis, may not be able to be done
properly due to improper physical or electrical
design, may be performed by a company, which
is not qualified or simply because the owner does
not understand the process.
Recently I attended a conference and heard a
presentation, which questioned the value of
scheduled maintenance. The “expert” suggested
that there is inherent risk associated with the
performance of maintenance and tasks required
to change the state of a critical system from “on
line” to the maintenance mode, performance of
the actual maintenance tasks and then returning
the system or piece of equipment to it’s “on line”
state. I cannot dispute the risk involved.
However, the presenter inferred that a good
portion of the risk was due to the probability that
the maintenance provider would make some sort
of error. The human factor is always of concern
however, the owner or owner’s representative
have a responsibility here as well. Successful
maintenance requires a team effort. The team is
the maintenance provider, the owner and the
user. This joint approach provides for a wellplanned and coordinated event. In this
environment, specific actions are planned and
examined for risk. A master plan or method of
procedure is essential. Contingencies are planned
as well. The most important aspect is the
verification of proper configuration and system
performance following a maintenance routine. If
you do feel that the risk of performing
maintenance is too great, I would recommend
you update your resume.
The maintenance provider, the owner, operator
and design engineer or consultant are part of an
essential team. Failure of any team member may
result in system malfunction. It is essential that
all parties understand their unique
responsibilities to the other team members. A
manufacturer may hold the contract and
responsibility to maintain a specific piece of
system equipment however; they cannot be
effective without the cooperation and
understanding of the owner or operator.
Too often the decision on the maintenance
provider or type of program is made strictly on
price. The problem is that all programs are not
created equal.
In order to understand exactly what you’ll be
receiving for your maintenance dollar you have
to ask some probing questions.
Ask any prospective maintenance provider what
kind of a job they do and they’ll tell you they are
great. Before you run down to purchasing to cut
the check, perhaps a little deeper analysis will
reveal the true nature of a service company. If
there’s a $2,500 dollar difference between
service providers, they may be a reason. The
following 10 questions will help you determine
the value of the program, by comparing all
perspective service providers on a level playing
field.
3
•
Q. How are your service technicians
selected, trained and qualified to work on
my critical equipment?
This is important. Some maintenance
companies have only a few specialists.
While they may have many people, the
person arriving at your site in an emergency
may not be skilled or trained to effect the
repair or worse, may cause a system level
failure. Ensure that the folks who may
respond are qualified on the equipment
installed and understand how each piece
relates to the system.
•
Q. How many folks do have on staff?
Are there enough people to properly support
your site in the event of a disaster or heavy
schedule?
•
•
•
Q. Do you provide 24-hr. service? How?
(Have them explain call handling,
escalation and response procedures).
The owner must understand how people will
be contacted and directed when you need
them. A good escalation plan provides for
uniform steps and involvement of escalating
levels of management. The critical system
maintenance company should be ready,
willing and able to commit all the necessary
resources to solve a situation regardless of
the date or time.
Q. Is your company authorized and or
trained by the manufacturer? Prove it.
No big deal right? Wrong. Your emergency
power system is the lifeblood of the facility.
The people working on it must have the
proper skill set and training. Your life may
literally depend on it. And let’s not forget
the personal liability you may have as an
owner or operator. Civil actions and
penalties are becoming the law of the land,
protect your facility and yourself.
Maintenance providers must have an official
relationship with the manufacturer they
claim to represent. Failure to do so may
mean the maintainer is not aware of current
procedures or service bulletins.
Q. Can your company provide training
for my staff? This is also important. Why?
I’m not suggesting that your staff be trained
to do the maintenance or repair on critical
gear. I’m also not suggesting that they are
incapable of learning how. I am suggesting
that without constant exposure, they will not
remain sharp. They should be taught how
the equipment and system is expected to
operate under normal and abnormal
conditions. They should understand how to
operate the system and what steps to take
should the equipment not operate as
expected. In the case of a transfer switch for
example, one must understand how to
operate the bypass if so equipped or
manually operate the ATS safely. The
engine control system is equally important.
The ability to manually start and parallel
engines may mean a lot to your data center
or hospital in the middle of the night.
•
Q. How do you support the technicians
who will be on my site? (Technical
support, parts etc.)
What happens when the technician can’t
figure out the problem? Does the
prospective service provider have help
available 7 x 24?
•
Q. How does your company become
aware of product changes, technical
bulletins etc. from the manufacturer?
This is normally limited to the
manufacturer’s own Service Company or
authorized agents. You will want your
provider to be aware of anything effecting
your system. How well they are connected
makes a huge difference.
•
Q. What type of maintenance agreements
do you offer?
Take the time to understand what types of
agreements are available and which one best
fits the requirements of the system and
facility. Some equipment is relatively simple
and very dependable. So it is possible to
little or nothing, call it maintenance and get
by. This “dust off and test” philosophy is
dangerous. You may feel comfortable in that
your gear has been maintained when if fact,
the system may be on the verge of failure.
Trust but verify.
•
Q. What is the average experience of your
field force on the equipment I have?
There are many companies who have
general knowledge or are not qualified.
Make sure you understand these
qualifications of your prospective
maintenance company. For example, one
major manufacturer advertises thousands of
“feet on the street” to provide service to
4
critical gear. The truth is that only a few are
experts while the rest are generalist. I guess
the idea is the sooner someone shows up, the
better regardless of weather they can fix the
problem. This makes for great marketing but
is shortsighted.
•
Q. What spare parts do you carry for my
equipment?
You will find that many companies do not
invest in spares. They may bank on the
manufacturer having spares available in an
emergency. The bottom line is that if you’re
in the maintenance business, spare parts are
a cost of doing business. Will they have the
parts you need in an emergency? If they are
taking your money, they damn well better.
Make sure they do.
Maintenance is more important than ever. The
dollars associated with critical operations are
substantial, the maintenance window is shrinking
or non-existent, in certain applications, life may
literally depend on a properly operating system
(should you find yourself in a hospital on life
support you may want to think about that). In
any case, the emphasis seems to be on the lowest
possible dollar and very little is done to qualify
the provider. Personal liability is becoming of
concern. Some levels of staff are subject to civil
or legal penalties should the system fail.
Manufacturers strive to produce a better product
for the market while reducing or containing cost.
When a re-design takes place, the manufacturer
must decide how long the prior model will be
supported with parts and maintenance. Some
manufacturers are better than others are and there
is usually some quantity of new old or obsolete
parts. The point is you need to understand the
degree of support available for the systems your
business depends upon.
When it becomes clear that older equipment may
lack support, it’s time to consider replacement or
upgrade. Upgrading or modernization provides
cost effective alternative to replacement.
Testing
Periodic testing is very important to the health
and life expectancy of your mission critical
system. It allows you to determine the system is
fully operational following maintenance, track
performance degradation and monitor
performance set points. The benchmarks
established during initial installation are the
standard used for comparison.
Results often tip you off to problems, which if
left un-addressed will lead to unplanned failure.
Normally as we said, testing is part and parcel of
a well-designed maintenance program. If not, it
should be.
Modifications & Upgrades
Summary
Replace or upgrade? Upgrading systems is a
viable alternative to replacement. Depending on
the equipment or system, the manufacturer or
service provider may be able to provide a range
of services from a simple control replacement to
a complete rebuild and modernization. The cost
involved may be considerably lower than the
total cost of replacement. This is especially true
when you consider the collateral costs associated
with de-installation, rigging and re-installation.
Still not convinced? Take a long hard look at
your equipment / systems. Ask the manufacturer
if all components are readily available.
Buildings are becoming increasingly complex, as
are the systems they depend on. The evolution of
technology coupled with the convergence of the
truly connected environment indicates this trend
will only continue. Mission critical systems must
remain reliable. The concepts presented in this
paper are really common sense but as with many
other common sense issues, they must be dusted
off and consciously examined.
Weather it’s a data center, healthcare facility or a
factory; electrical systems are the lifeblood of the
facility. During the blackout of 04, many
facilities were shocked to learn that systems they
depended on failed.
5
EQUIPMENT / SYSTEMS LIFE CYCLE
Design &
Application
Quality
Testing
Transport
&
Protection
Modify &
Upgrade
Maintenance
Proper
Installation
Training
Benchmark
Testing
Documentation
Figure 1
6