Network Management of Critical Wireless Systems
Brian Cunningham
Applications Engineer
Eaton/Cooper Bussmann, Wireless Business Unit*
Port Coquitlam, BC V3C 6G5
1-800-663-8806
www.cooperbussmann.com/wireless
Keywords
radio, mesh, wireless, network, management, systems, monitoring, spread spectrum,
telemetry, frequency hopping, transmitter, receiver, transceiver
Abstract
With increasing reliance on wireless networks for process control, the question has
arisen of how can these be monitored and maintenance proactively scheduled. The
benefits of wireless communications have been proven through hundreds of thousands
of applications in water/wastewater, oil and gas, and manufacturing, to name a few.
Traditionally, once the network was installed, it was forgotten until a problem occurred.
Wireless manufacturers’ recommendations of regularly scheduled maintenance
traditionally involved significant labor and hours of driving only to discover no faults, or
trends too small to be indicative. As IT and other technical professionals have been
using software tools to monitor copper networks for years, now software is available to
do this automatically and conveniently for wireless networks. This paper will discuss the
importance of network management on critical wireless systems that are relied on by
large industrial facilities to supply drinking water to families, electricity to households and
fuels for heating and transportation.
0813
Reorder 10144
Page 1
1.0
Wireless Systems for Industrial Applications
Before a discussion of network management can begin, the reasons why the network is
there in the first place need to be covered. As in the title, this paper will focus on
wireless systems – for specific reasons. Wireless systems are now in use by industrial
users for process control and monitoring. Ranging from Ethernet links for video
surveillance and PLC connections, to WirelessHART for sending 4-20mA signals, to
controlling pumps and monitoring flows, wireless connections offer significant savings in
many applications. The savings over wired connections have outweighed the reliability
and security concerns, in the many business cases that are presented to management
by engineering departments. Particularly for long distance applications, such as
controlling water pumping stations, the only hardwired alternatives are leased phone
lines. Industrial facilities using 4-20mA signals often use buried conduit to transport
signals – digging a trench when soil remediation is required such as in older refineries
has become so expensive that every alternative is explored first. If the distance exceeds
½ mile, wireless is often the only cost effective approach. Rotating or moving machinery
is a natural for wireless applications.
Wireless connectivity can be deployed on a variety of frequency spectrums. Fixed
frequency 5 watt radios can transmit 80+km and are common in the oil patch for
monitoring and controlling pump jacks. Spread spectrum radios allow shorter links, but
without the regulatory hassles, making them ideal for in-plant applications. Cellular
modems offer the unique capability of access to a site that has cellular coverage, from
anywhere in the world with an internet connection. Wireless connectivity options will vary
depending on the application, the terrain, the distance and the regulatory requirements of
the jurisdiction.
2.0
Traditional Wireless Network Maintenance/Monitoring
A short discussion of how things were done in the past will explain the pain and expense
that motivated the quest for a better way of doing things. Routine maintenance of
wireless systems involved cumbersome site visits. With fixed frequency radios up to
80km away, this involves long distance driving and many unproductive hours. Some
networks are as large as 2000 radios using dozens of repeaters. Often the roads
connecting sites do not take the most direct path – meaning to drive to a site with an
80km radio link can involve 100+km of mileage on the vehicle. Multiply this by 2000 sites
and you have a very motivated organization looking for efficiencies.
Once on site, the most common parameter checked is the Received Signal Strength
Indicator (RSSI). By measuring this on a regular basis, a trend can be obtained which
will indicate a point in time of failure. Foliage growth, antenna and co-axial cable
corrosion, insufficiently waterproofed connectors, electronic component degradation and
0813
Reorder 10144
Page 2
drift can all lead to reduced RF performance. If this continues, intermittent operation will
be followed by longer and longer outages.
Figure 1 Technician at Remote SCADA Site
One issue with snapshot measurements of the RSSI taken once or twice a year, is that
signals can bounce around a fair bit making a trend hard to spot. This is particularly true
with frequency hopping spread spectrum radios; as the frequency changes, multipath
can occur reducing the RSSI, which may recover as the radio transmits on a new
frequency. Moving machinery can also cause this to occur, as reflective paths come and
go. The author has also witnessed this in high wind conditions blowing a tree branch in
and out of the radio path.
Background noise level is another important parameter that can change over time due to
increased proliferation of wireless devices. A rule of thumb is the RSSI should be 10dB
stronger than the background noise level. Therefore if the background noise level rises
too high, the receiver will be unable to lock onto the lead-in tone of the transmitting radio.
Again, this type of signal degradation can occur over a long period of time.
Other maintenance functions on a wireless system include simple visual inspection of the
hardware. Any mechanical damage caused by storms or infiltration of water into
electrical cabinets or antenna connectors should also be corrected. This type of
maintenance can be done by non-technical personnel and does not require a laptop with
specialized software.
Another parameter that should be monitored, that would enable a technician to bring the
right tools to the right site, is traffic out the serial or Ethernet port of the radio. In order for
the radio to be functioning correctly, the attached devices need to be sending polling
requests/replies into the radio. If communications port activity can be monitored
independent of the RF port activity, the nature of a failure can be determined with far less
troubleshooting.
0813
Reorder 10144
Page 3
3.0
Mesh Network Complications
Mesh networks with their capabilities to change RF links at any time depending on the
network conditions add additional complications. Are a large number of remote radios
relying on one or two repeaters? How much traffic is going through those repeaters? Is
the traffic reaching the maximum throughput of those repeaters? A site visit to monitor
this only gives a snapshot in time – how did it perform last week or last month?
As these questions illustrate, network traffic is an important parameter to monitor.
Consider the scenario where 2 repeaters are available, but one has a stronger signal,
and therefore is the “chosen one” with the other as a backup. As network traffic
increases, it may exceed the bandwidth of the “chosen repeater”. You may wish to force
some remote sites to go through the weaker radio path to even out the traffic or even
consider a 3rd dedicated repeater. This is a common scenario when deploying a network
of 100+ radios. When initially installed, typically a few remote sites are commissioned
every week. Every addition can impact the performance of the network and therefore
continuous network performance monitoring of the entire system is an essential task to
keep the network stable.
Figure 2 Network Topology Showing Repeater Paths
If 3 or 4 repeater paths are available, which one is being used right now? Critical
information for troubleshooting involves finding out which radio has a problem to narrow
down the source. Otherwise driving to remote sites is required to log into each radio. If
presented with a network topology map, showing which nodes are up and which are
down, the network can be easily visualized and performance understood.
0813
Reorder 10144
Page 4
4.0
Network Management Using Software – the Requirements
A dedicated computer with access to the internet would be a basic requirement. The
computer must be running 24 hours in order to monitor the network and sent out
email/text notifications if alarms occur. Remote access needs to be available so an
operator can log in from anywhere and decide if immediate action needs to be taken.
Ideally, multiple operators should be able to log in simultaneously, with different levels of
authority – some for monitoring only, others with full control access, able to change
wireless settings on remote radios. This of course requires that an effective security
system be in place to prevent intruders from accessing the system.
A Network Management System (NMS) will periodically request status updates from the
wireless network. This information flow can have some impact on the bandwidth
available for regular SCADA traffic. Therefore the monitoring data needs to be
minimized and prioritized so not to interfere with the payload data. Allowing time based
reporting, as well as on demand – to spot check a potential problem, would be required
to make the NMS compatible and complimentary to the wireless network. Additionally, a
large network may have a mix of wireless and wired equipment installed and may be
from different manufacturers. An agnostic NMS will take all of this into account
Visualization is a critical element of an NMS to provide immediate and accurate
information to the user at any point in time so that device or link issues can be located
and action can be taken. Detailed information like RSSI, SNR, Tx/Rx throughput, channel
utilization etc. are critical parameters for analyzing problems without traveling to a remote
site. Especially early warning thresholds can inform you of potential issues ahead of time
and issues can be resolved before any serious degradation of the network can occur.
The number of faults or alarms - if recorded can sometimes be traced to external
environmental factors such as extreme weather, or passing rail cars – both can obstruct
wireless signals. Statistics of uptime and com status can allow contractors to sign off on
a new installation.
0813
Reorder 10144
Page 5
Figure 3 Statistics of Wireless Hosts
Figure 4 Graphical Analysis of Host Performance
0813
Reorder 10144
Page 6
Can you remotely change wireless settings to modify performance? Can you view the
wireless nodes on a map to get an idea of distances and clusters? Perhaps you need to
turn down the transmit power of one radio to minimize interference to a neighboring
system, or add a fixed routing path to a mesh network? If you can see the network on a
real world map like Google Earth, visualization of the issues becomes much simpler. If
you can change a parameter from the control room, then observe the results, hours/days
of labor and mileage can be saved.
Figure 5 NMS Showing Wireless Locations on a Google Map
The performance of the network or of individual devices can easily be documented using
an integrated graph generating and reporting tool that allows the user to print or export
such data for external use.
0813
Reorder 10144
Page 7
Figure 6 NMS showing all State Transitions
Figure 7 Availability Summary of a Remote Site
0813
Reorder 10144
Page 8
5.0
Network Management Using Software – the Business Case
If all industrial wireless users were to employ network management software, the author
estimates technical support inquiries would drop by 20%. By trending the supply voltage
of a solar powered remote site, a technician could be dispatched long before a remote
site failed. In the solar world, these failures commonly occur in the middle of the night,
typically in the dead of winter. Remote monitoring could ensure a technician is only sent
when needed, for common minor maintenance such as cleaning the solar panels or
replacing the battery.
Consider the following case study: A large biotech company that requires monitoring of
fridge, freezer and incubator temperatures. These chambers contain research that leads
to cutting edge new drugs helping cure cancer, improve recovery times and treat a host
of other illnesses. Often the samples in these chambers are worth hundreds of
thousands of dollars. Now consider this – the chambers are on castor wheels and
frequently moved from one room to another, and sometimes from one building to
another. The company has over 70 buildings on this particular campus. There are over
2000 chambers on campus. It is an ideal application for a mesh network, since the
repeater paths will need to change. It is also an ideal application for NMS, since the
network layout, data traffic levels, RSSI, etc. may all vary with the chambers movements.
In this installation, remote monitoring is essential to keep track of the physical location of
the chambers.
After commissioning, the NMS determined that traffic levels at some key points were too
high, and on rare occasion, an entire building would lose communications. This led to
the installation of fixed repeaters. These repeaters, left with the choice of many dozens
of paths, sometimes chose paths with higher traffic levels on chambers that were subject
to frequent movement. This led to fixed routing paths being created to alleviate
congestion.
By trending the RSSI, facilities staff could see if a chamber was at risk of losing
communications. An abrupt drop in signal strength often meant the magnetic mount
antenna had been knocked over onto its side. A slow decline would be indicative of an
antenna connector or cable issue. Overall, alarm levels were set to alert facilities staff
well in advance of failure. When a failure or temperature alarm occurred, protocol
dictated the researcher was notified and if necessary, required to drive on site to transfer
the samples to another chamber. By providing early warning notifications, the user was
able to make an informed decision about the severity of the problem and often avoided
an unnecessary trip to the facility. It was essential to provide immediate status updates
of network link failures, which if not repaired immediately, could result in the loss of R&D
samples and the value that research represented.
0813
Reorder 10144
Page 9
6.0
Conclusion
Wireless communications can offer significant cost savings over hardwiring, and in some
cases such as moving machinery, are the only choice. Historically, maintenance of these
systems involved site visits to gather a snapshot of today’s performance. Mileage and
labor costs add up to make the decision to do preventative maintenance a business
decision weighed against the odds of failure and the loss of productivity or material.
Mesh networks further complicate manual network monitoring as the repeater paths may
change at any moment. A network management system needs to be capable of
alarming, trending, visualization as well as the ability to remotely change wireless
parameters upon discovery of an anomaly. The rewards of utilizing an NMS system
include higher reliability, greater product quality, increased throughput, workforce
mobility, as well as more measurements at a lower cost.
www.cooperbussmann.com/wireless
www.eaton.com/wireless
Technical Support:
United States: +1 866 713 4409
Australia:
+61 7 3352 8624
Email: ELPRO-Support@cooperindustries.com
ELPRO-US-Support@cooperindustries.com
0813
Reorder 10144
Page 10
Note: Eaton acquired Cooper Industries plc in November 2012
0813
Reorder 10144
Page 11