!
TELECOMMUNICATION INFRASTRUCTURE IN UTILITIES
For the last decade utilities used to have serial low-bandwidth communication to monitor
the entire grid, however the increasing regulatory requirements of power quality,
cybersecurity compliance, video surveillance and smart-grid technologies such as
syncrophasors, distributed generation, distribution automation, and AMI, are requiring
more bandwidth and flexibility to assign resources in their telecommunication
infrastructure according with traffic characteristics, reason why almost all utilities had
began to upgrade their communication infrastructure.
!
!
1. Moving from circuit switch to packet switch networks
Digital communication networks can be generally classified as either circuit switched or
packet switched. Historically, utilities implemented circuit-switching technology
establishing a fixed path with a constant latency and dedicated bandwidth. Knowing
constant latency, path delay can be factored into various time sensitive applications. For
simultaneous transmission of multiple services sharing the same medium (i.e. data, voice,
and video), these signals are combined using Time Division Multiplexing (TDM), where
each independent signal is allocated a time slot and gap in a cyclical fashion. The
disadvantage of this technology is that the network resources are inefficiently used by
bursty traffic like video or voice. For example, a channel assigned for video will be
permanently occupied, even during the times that you are not transmitting any monitoring
or video conferencing data.!
This inefficiency is eliminated in packet switched network technology, which inserts the
data into a digital packet. The packet enters into a queue at the ingress of the network,
and the network devices make the necessary decisions to choose the best path for sending
the packet through the network. Although this process eliminates the wasted bandwidth
that results from circuit switching, the unknown queuing time and packet route would
result in a highly variable path delay. This unknown and variable delay makes it difficult
for a time sensitive applications to know with certainty how much time a given packet
spent in transit.!
2. Problems that are facing the utilities when upgrade their infrastructure
Utility are now expanding their networks beyond the realm of this deterministic network
by connecting to communication nodes that rely on asynchronous, non-deterministic,
packet switched networks.
The asymmetric delay and latency!
There are several protection schemes that power engineers can implement using
communication network to maintain the grid stable and avoid outages expand across
several areas. The most common schemes for power system protection, specifically for
protecting high voltage transmission lines, use distance (impedance) protection, current
differential protection, or a combination of both. !
Asymmetry delay is a concern basically when there are implementations such as modern
differential protection operates over a digital communication channel, where analog
values of the Alternating Current (AC) waveform are digitally sampled as vectors. These
vectors consist of an array that includes magnitude, angle and time stamp information
related to each sample of the measurement. Inaccurate synchronization between clocks in
two different nodes can result in measurements shifted from their true time reference. The
consequence of phase and magnitude shift can result in a false positive, where isolation
happens unnecessarily, or a true negative, where a dangerous fault condition is permitted
to propagate further into the power distribution system. The reason is that packet
networks can be configured dynamically according with the network necessities, which
not happens on TDM networks. Packets network can optimize the routes between two
points according to different objectives such as bandwidth, latency, cost, etc. For more
applications this is the best option, however for differential application a change of the
route or chance on latency could generate a complete blackout on the system. So when IT
engineers configure the network to implement power applications have to be aware to
setup those circuit with special characteristic in order to avoid those changes!
The mainly downside for the most common packet based technology is that utilities they
don’t have the right Operational Technology (OT) or Information Technology (IT)
engineers to support the new infrastructure. This will imply utilities high amount of
investment and a long learning curve that in the time is money.
From our research among several utilities around the globe, we found the followings
facts.
!
•
•
•
•
•
•
•
Utilities prefer flexible network products that can meet changing business needs
and growth requirements. To support growth over the long-term, networks must
adapt to changing business priorities, enabling new services in a timely fashion.
Utilities need to quickly and reliably add services to their networks, without
introducing unnecessary risk to the production operations environment.
Major network architecture decisions are based on the requirement to support
business functions.
For utilities the network have become a critical part of their infrastructure.
Utilities focus on support uptime, security and equipment costs.
Some utilities typically use products from multiple vendors for wide area
networks optimization, which drives additional ongoing operating expenses.
When utilities introduce multiple vendor products to satisfy functional
requirements; they typically do not evaluate the tradeoff of additional
functionality with the impact on long-term costs and overall business operations
risks.
!
3. Components of a packet based architecture
The basic building blocks are identified and defined below. It defines the key segments
and maps each segment onto each associated power and energy system layer.
Wide Area Networks (WAN): The WAN interconnects to the public Internet
network, transmission substations, utility control centers, and enterprise/IT/OT
networks using secure communications. See the description for “Backhaul”
below to identify special cases for Wide Area Networks.
It is comprised of the core network/backbone that connects to major service
providers backbones, datacenters, inter-utility backbones, and the regional and/or
Metropolitan Area Network (MAN) (metro fiber rings or wireless networks).
The WAN could be either a utility owned or a public service provider network.
Utility Local Area Network (LAN): It is comprised of utility operations and
enterprise LANs to manage operations, control and enterprise processes and
services (billing and automation, meter reading, outage management, demand
response, load control, etc). It interconnects to the WAN through secure wireline
or wireless communications. Utilities consider two types of LANThere are two
types on LAN It also interconnects all the IEDs and sensors in the substations, to
the public Internet to exchange customer data to third party providers.
!
Backhaul: It is the spur that connects the WAN (major POPs – point-ofpresence) to the last mile network. The backhaul can be owned by the utilities or
provided by third party service providers. It aggregates and transport customers
smart grid telemetry data, substations automation critical parameters data,
distribution plant intelligent devices data field information, mobile workforce
information to/from the utility head end to/from the last mile network. (It should
be noted that in certain network architectures a backhaul segment may exist with
no actual WAN. Legacy connections consisting of point-to-point or multi-point
data circuits are just such examples.)
EDGE: The last mile is a two-way wireless or wireline communications network
overlaid on top of the power distribution system. It is usually named as
Neighborhood Area Network (NAN), Field Area Networks (FAN), or Advanced
Metering Infrastructure (AMI) depending on the utility network system
characteristics, services offered, network topology and demographics and the
vendor technology utilized. The last mile could be an integrated and multipurpose network technology alternative for AMI (smart meters, Demand
Response, etc) services, Distribution Automation (IEDs in the field) and
!
substation automation. Alternatively, the last mile could be comprised of
individual networks technologies with different purpose and network
characteristics (performance, security, management, etc) for each particular
application. In one end it interfaces with the smart meters - at the customer
premise edges, the field IED devices and sometimes the distribution substation
hotspots. In another end, its network Access Point (AP) interfaces with the
backhaul network, where the data is collected/aggregated to be transported to/
from the backhaul to the WAN. The last mile may also provide communications
to the Distributed Resources (DR) - renewable and non-renewable energy sources
- connected to the distribution grid.
4. Options to build the telecommunication infrastructure
Utilities are a very special case from telecommunication perspective. Utilities are at the
same time data carriers as LEVEL 3, access provider as AT&T, datacenters as GOOGLE,
all in one. Utilities have very difficult decision to take: stay with a single vendor to meet
their telecommunication needs, or build a multi-vendor environment that focuses on the
best features and functionality in the market. Some utilities don’t have their own
telecommunication infrastructure; they subcontract some portion of the infrastructure
(Backbone, backhaul, EDGE) with a third party, but still they need to integrate the
service in some point.
a. The multi-vendor option
The utilities telecommunication infrastructure is usually separated in different domains,
backbone, backhaul, LAN, and EDGE. For each domain there is a responsible for the
data network. There is debate between the IT and the OT department about the support
the telecommunication infrastructure. The discussion basically IT says OT don’t know
anything about security, or data network infrastructure so we have to provide those
services. In the other, hand OT says that IT doesn’t know anything about the power grid.
There are several arguments to support each side, but Meanwhile utilities have to deal
with different monitors tools like HPOpenview, Trivoli, one for each domain (left boxes),
and different configuration tools, one per each manufacturer and one for each service
(bottom boxes).
Figure 1 Multivendor Architecture
!
Multi-vendor environment allows the enterprise to pick and choose technologies that best
fit the needs of its users and its different business. Products from multiple vendors may
provide some additional functionality, but also introduces new functional and operational
complexities and support challenges.
!
Multiple vendors create network interoperability challenges: the support is more complex
and difficult, and involves additional indirect costs and performance/security risks.
Vendors often are unable or unwilling to develop and maintain a working knowledge of
the hardware, software features and functions, and platform-specific limitations that
affect compatibility with other vendors’ equipment in the network. This impacts
customers’ ability to operate their networks, contributing to longer issue resolution times,
more complex change management, and more serious service impacts during network
outages.
When multiple vendors are contacted to diagnose and correct a network issue, it is not
unusual for “finger pointing” and vendors’ inability to work together to delay resolution
of issues. The increased operational impact increases the burden on the customer’s
networking team. This results in substantial negative impact to business operations, and
increased cost due to time and effort required to resolve problems.
Some utilities may be tempting to choose one vendor to mitigate interoperability and
integration issues. There is no a right answer all depends on number of devices and the
geographical area that utilities have to cover and the actual technology utilities have on
place.
!
Networking equipment from different vendors is not designed to interoperate under all
production conditions; when these products are implemented on a common network,
greater customer effort is required around design, testing, and management
!
b. The single-vendor option
Some companies choose to stay with one vendor for ease of use and guaranteed
interoperability across all their communications platforms. However the downside of this
approach is the telecommunication industry is highly vertical integrated tying single
vendors for specific solutions, which allow manufacturers execute market power
increasing the price, limit the applications, and constrain the innovation.
Telecommunication manufactures are not quite there, where they're all really good at
everything. Some vendors are really strong in datacenters, others in backbone networks,
and others in the edge or distribution networks,
Some providers have developed intermediate solutions to mediate temporally the
problem, but the problems remain under the hood. Cisco, Juniper, Huawei, Alcatel
Lucent, or any of big telecommunication providers have tools that allow IT/OT engineers
create templates in their graphical user interface and build standards configuration for
each type of service including wide area protection schemes.
!
Figure 2 Single Vendor Architecture
!
!
Even customers who prefer a single vendor network approach may occasionally be
required to incorporate multiple vendors’ products into their networks for certain periods
of time, due to circumstances such as mergers and acquisitions, or where a nonincumbent vendor is the only supplier of a critical functional capability.
!
From the security stand point multivendor are has an embedded risk for utilities.
Communications networks grow through expansion of the power grid or nowadays
because upgrading new substations, this usually requires network equipment from
different vendors to be integrated. Integrating these networks into a single multivendor
network provides special security challenges for these customers, which contribute to
higher cost, complexity, and operational challenges
!
Clear Creek Networks (CCN) is developing software and hardware platform based on
open architecture that will helps utilities get all the benefits from single vendor plus the
benefits of market competition allowing a soft transition from their legacy system to the
latest telecommunication technology of the 21st century.
!
5. How CCN helps utilities move to packet base technologies.
As was discussed above, today’s legacy communication network is vertically integrated:
the interface, applications, network operating system, and switching/routing hardware are
integrated into a single device. Different switch vendors allow different access and
functionality through proprietary interfaces. This is a problem and frustration of today’s
network engineers – they need to be network specialists to solve their network problems
and even then, they are constrained in what they are able to configure.
!
Clear Creek Networks’ solution is software that runs on a commodity server. Our network
control software interfaces with Software-Defined Networking (SDN) switches that have
become essential in today’s data centers and used extensively by Google, Microsoft and
Verizon. We have built a solution using this technology specifically for the power grid.
!
a. How SDN works
Using SDN as a foundation, Clear Creek Networks has built a self-operating network that
is upgraded through software. You no longer rip out your old hardware and replace it with
new, expensive, proprietary hardware that requires new training. Your infrastructure
investment is safe and any new unforeseen functionality is adapted by simply installing a
new application into our open controller such as the Self-Conf, Link Monitor and Cyber
Security apps shown below.
!
This state of affairs has been compared to the mainframe era of computing. In the era of
the mainframe, applications, the operating system, and the hardware were vertically
integrated and provided by a single vendor. All of these ingredients were proprietary and
closed, leading to slow innovation. Today, most computer platforms use the x86
instruction set, and a variety of operating systems (Windows, Linux, or Mac OS) run on
top of the hardware. The OS provides APIs that enable outside providers to develop
applications, leading to rapid innovation and deployment. In a similar fashion,
commercial networking devices have proprietary features and specialized control planes
and hardware, all vertically integrated on the switch. As will be seen, the SDN
architecture and the OpenFlow standard provide an open architecture in which control
functions are separated from the network device and placed in accessible control servers.
This setup enables the underlying infrastructure to be abstracted for applications and
network services, enabling the network to be treated as a logical entity.
!
Using this technology, CCN platform is able to self-configure and operate a computer
network that supports the grid.
!
!
Figure 3 CCN Architecture
!
CCN software creates a middle layer between SDN architecture and Energy applications,
removing all the complexity from different configuration and management tools. One
single platform allows utilities monitor all the behaviors on the grid plus configure the
devices. In Addition CCN has created links between energy applications so the network
and the power grid necessities are consistence.
!
!
!
6. The Clear Creek Networks Advantage
CCN platform provides utilities with a value from the distribution side to the datacenters:
!
!
!
1.
2.
3.
4.
Reduce the Total Cost of Ownership
Future proof implementation
Vendor agnostic,
Less total cost of acquisition
a. Total Cost of Ownership (TCO)
The total cost of acquisition is not only the capital expenses designated to buy the
infrastructure but also the operational expenses to support the infrastructure. CCN will
reduce dramatically the both cost.
!
One of the things that companies often face when deploying new technologies are the
costs involved. Many times they have to decide which is more important to them; the
cost of acquisition or the total cost of ownership. When considering scale any services
above 100s of units either cloud computing or wide area networks, the cost of ownership
can get considerably higher.
!
With such a wide range of choices out there on the market, you have to make a series of
choices that will impact the way utilities operate and how much it will cost them over
time. The temptation that many companies have is to go with the cheapest solution they
can, but in the end, they find out how much that cheap solution costs them much more.
Using SDN technology CCN can simplify the infrastructure reducing 1/3 of total cost of
ownership compared with legacy technology.
!
CCN reduce the initial cost of investment by simplifying the architecture, then less is a
single type of device that can create the routes, ACLs, flows and define the security, there
is no more need of specialized routers of firewall or independent devices for the WAN
and LAN networks. SDN centralize the knowledge in the controller given each
communication devices the right instruction how to deliver each packet.
!
CCN can also reduce operational. The lack of experienced engineers with knowledge not
only of the power grid but also of the communications infrastructure is holding utilities
back from realizing the potential of their grids. Clear Creek Networks has created a selfoperating technology that helps experienced power engineers move into the digital age by
giving them the tools to create their own telecommunications architecture.
!
Figure 4 Total Cost of Ownership
!
!
!
b. Future proof implementation
The giants on the Internet and mobile communications are supporting the SDN
development; only Google upgrading their datacenter telecommunication infrastructure
improved the performance of the network from 30% to 100%. This is motivating to the
entire industry move from legacy network technology to SDN, in few years of
development SDN have become the mainstream topic for research, development, and
standardization. More and more innovations will come to be part of the new ecosystem
and the utilities that had take this decisions will get all the benefits.
!
Since no one knows what smart grid technologies are coming next, you need a crystal
ball to design your network. Clear Creek Networks future-proofs your grid by providing a
highly programmable communication network – a solution that truly evolves by simply
installing new software.
!
!
c. Vendor agnostic
CCN allow utilities not be tie for the single provider for the next decade. With CCN
platform utilities are able to buy SDN hardware for any vendor while maintain the same
software or keep the hardware and upgrade only the software piece. More over utilities
can add different types of technology such as wireless radios, mobile terminals, to the
same management infrastructure. This will open the market for more competition and
flexibility to add new functionalities or devices capabilities.
!
Network components provide greater value if they interoperate without affecting
performance, security, manageability, or operational stability. Maintaining
interoperability is crucial for an organization, which operates a multivendor network, and
this carries additional associated costs.
!
d. Enhance Security protection
Ensuring a properly secured computer network is a daunting task, and the unfortunate
state of network management is that a computer network’s security is often ignored or
improperly setup. With Clear Creek Networks, a security wrapper is created to protect
your computer network. Clear Creek Networks ensures that only devices that are
supposed to talk to each other actually do.
!
When you add new devices in the field, these devices are automatically discovered and,
with a push of a button, the network is updated. There is no need for router configuration
to allow it into the network; a simple interface enables easy approval of the new device.
!
Network security: Security policies are established and enforced to protect the
availability, access, operational security, data privacy, and integrity of the network.
Security controls are typically managed by specialized groups responsible for
maintaining security across the network, regardless of network vendor.
!
!
!
1. Our communications devices
The CCN hardware is a rugged SDN switch built under the standard IEC 61850-3:2013.
The IEC requirements, mainly regarding construction, design and environmental
conditions for utility communication and systems in power plant and substation
environments. All the interfaces of the CCN switch support Small Form-factor Pluggable
(SFP) connector, that provides the utilities the flexibility to choose the right combination
of RJ45 or optical ports and can be easy interchangeable.
!