Intelligent Power Routers: Robust Computational Support for
Self-Healing Electric Energy Networks
White Paper (DRAFT)
1. Introduction
Virtually every crucial social, military, and economical function of our society depends on the secure and
reliable operation of electric power networks, the most significant engineering achievement of the 20th
century, according to the National Academy of Engineering. As demand for digital goods and services
has increased, so has the demand for reliable and disturbance-free electricity. Failures in power delivery
systems can have significant economical and societal effects due to the interaction of the power networks
with critical infrastructure systems such as defense, health, finance and communications. For instance,
power outages and disturbances on the National Grid cost the U.S. economy more than $100 billion
annually. Certainly, protecting the National Grid must be a priority for the defense of the citizens of the
U.S. and allies. At the same time, the development of new advanced electrical capabilities can play a key
role in the defense of the U.S. by enabling the deployment of a next generation of ships and vehicles that
are far more efficient and effective, and have superior survivability to ensure that missions are
accomplished and casualties are minimized. The Office of Naval Research (ORN) is currently seeking
technologies to achieve these goals with “The Electric Warships and Combat Vehicles Future Naval
Capability” (FNC)1.
Modern power delivery systems are complex networks comprised of highly interactive, nonlinear,
dynamic components that have different spatial and temporal requirements for operation. Power delivery
systems consist of two major infrastructures: the electric power grid and the communications system.
These two infrastructures are highly interdependent, since reliable operation and control of the electric
power grid relies heavily on its communications system. A breakdown in one of these infrastructures can
often lead to failures in the other. The operating state of power delivery systems can be adversely affected
at any time by a variety of entities or factors, including: system operators, power producers, energy
consumers, natural phenomena, and purposeful attacks. Another factor contributing to catastrophic
failures is the stressed state caused by operating power system components close to their operational
limits. New defense strategies and technologies must be developed in order to reduce the vulnerability of
power delivery infrastructures to catastrophic failures.
There are many key technological challenges that must be overcome to deploy highly reliable, efficient
and survivable electric systems. State-of-the-art power delivery systems for ships, vehicles, civil power
grids are designed with redundant power generators and delivery lines to make the system as a whole
tolerant to some of the most common types of failures. However, control and coordination of the
generation and distribution of power still occurs in a rather centralized manner, with only a few sites
managing mission-critical tasks for power generation and delivery. Moreover, recovery from failures
often involves decisions that are unnecessarily left for human operators to make, often based on human
analysis of stale and incomplete system status information. Excessive centralization, lack of automation,
and inadequate system information dissemination and analysis are three fundamental barriers that may be
overcome with a more aggressive application of computing and information technologies. The enabling
technologies that are lacking are the capabilities for electric networks to automatically detect dangerous
situations, and then activate countermeasures that stop or contain the damage, restore service to
undamaged portions of the system, and maintain the largest possible segments of the system within an
acceptable operational range. Thus, electrical networks should have embedded intelligence and other
support infrastructure to dynamically reconfigure and self-heal to guarantee survivability.
Borrowing ideas from the realm of computer networks, we propose a multi-layer approach to achieve
self-healing power delivery systems based on the concept of the Intelligent Power Router (IPR) as the
1
http://www.onr.navy.mil/fncs/
1
fundamental building block [1,2,3,4]. IPRs are strategically deployed over the electric network, and by
controlling power flow control devices (e.g. switches, FACTS), IPRs can “route” power to various areas
in a similar fashion as routers forward packets in a computer network. Groups of IPRs are responsible for
executing distributed algorithms for disseminating system status information among fellow IPRs, storing
data into local database, and using this information for making local decisions in the event of system
failures. During a contingency, groups of neighboring IPRs work together to contain the damage, bring
back critical lines, activate emergency generators, deliver power to critical loads, and continuously
monitor the system to maintain an acceptable level of operation.
In this project, we propose to work the protocols, distributed algorithms, mathematical models, and
decision making schemes that can make IPRs become enablers of self-healing electrical networks. The
core of the IPR consists of its distributed algorithms, so in essence, the work here proposed is
computational in nature. Our multi-disciplinary research group combines two Computer Scientists, having
expertise in dependable distributed algorithms and data management, with two Electric Power Engineers
having expertise in power system operations during emergencies and power system risk assessment.
During the first phase of our research we will develop the distributed information dissemination, data
mining and decision control algorithms necessary for the IPRs. We shall work to demonstrate the
advantages of IPR-based systems by creating simulations, models, and integrating software tools
developed separately for simulating networks (e.g. NS2) and power systems (e.g. Virtual Testbed). Later
in the project we will work to develop prototypes demonstrating the feasibility of IPRs as a new type of
building block technology for self-healing power systems.
The idea of an IPR-based self-healing power delivery system is of critical importance for the much
needed re-engineering of the National Power Grid, as well as for the survivability to both attacks and
failures of complex machinery containing major electrical subsystems such as a large aircraft carrier
ships, and land vehicles. Our research will explore the applicability of IPRs to the design and
development of fault tolerant electrical systems for complex machinery such as these as the demonstration
testbed. This project can have direct impact on the Electric Warship and Combat Vehicles FNC initiative
from ONR, particularly dealing with Priority Area 1: Ship tactical endurance, and Priority Area 3: Ship
enhanced survivability.
2. Intelligent Power Routers
In our framework for self-healing electrical networks, the intelligence that can be used for control and
coordination operations is embedded into a series of computing devices called the Intelligent Power
Routers (IPRs). As shown in Figure 1, IPRs are strategically deployed over the electric network at buses,
power lines, and generators. By controlling power flow control devices (e.g. switches, FACTS), IPRs can
“route” power to various areas in a similar fashion as routers forward packets in a computer network. For
example, when power is lost on a given region due to a generator failure, several IPRs in charge of that
region might request another generator to increase its power output, and then coordinate to open alternate
lines to route power into the affected region. Likewise, IPRs can oversee load shedding operations to
remove low-priority loads from the system in favor of other loads with higher priorities. Groups of IPRs
are responsible for executing distributed algorithms for disseminating system status information among
fellow IPRs and for using this information for making local decisions in the event of system failures.
During a contingency, groups of neighboring IPRs work together to contain the damage, bring back
critical lines, activate emergency generators, deliver power to critical loads, and continuously monitor the
system to maintain an acceptable level of operation. Our goal is to show that by distributing network
intelligence and control functions using the IPR, we will be capable of achieving improved survivability,
security, reliability, and re-configurability of the electrical network. An IPR-based power delivery system
scales much in the same way as a computer network scales. Groups of local IPRs form “local area power
networks” and are interconnected by border IPRs, which enable the formation of larger networks of
networks. Border IPRs are responsible form attempting to contain recovery actions inside local area
2
power networks to avoid failures from cascading across large regions of the system. IPRs view currently
existing relay-based load shedding schemes as a lower layer in a multi-layer power delivery network.
Border IPRs
~
~
~
~
~
~
~
~
~
Zone A
~
Zone B
Interior IPRs
IPR-based Electric Grid
Figure 1. Electrical Network Featuring IPRs
Power System
Energy Flow
Control Devices
Sensor
Input
Switching
Commands
Interfacing
Circuits
ICCU
Intelligent Power Router
Figure 2. Organization of an Intelligent Power Router
The architecture for an Intelligent Power Router is shown in Figure 2. The router consists of Interfacing
Circuits (ICKT) that operates existing Energy Flow Control and Sensing Devices (EFCD) and Intelligent
Control and Communication Unit (ICCU). Examples of EFCDs are: circuit breakers, regulating
transformers, series capacitors, static var compensators, or their combination as Flexible AC
Transmission System (FACTS) devices. The ICKT is the hardware component that interacts with the
energy flow control devices of the electric power system. These already existing devices will control the
power flow, opening and closing lines as needed, or regulating the amount of power that flows through a
given corridor. The Interfacing Circuits sends commands to the EFCD to dynamically change the
behavior of the power system. Also, the ICKT receives information already being collected by
conventional sensors (e.g., CTs and PTs) and other sensing devices, in order to assess the current status of
the system. The ICKT will operate under the direct control of the ICCU, which will have the necessary
logic to determine how to re-route power, change load set point in generators, shed load or take any other
3
corrective or preventive action to enhance system security. The ICCU can be implemented as an
embedded computer located inside the IPR. The ICCU could feature a RISC-type CPU, high speed RAM,
non-volatile data storage, and a network interface. The ICCU should be made out of commodity
components to keep its cost low, make it easy to fix or replace, and to leverage on the latest advances in
the computing technology.
3. A Multi-Layer Architecture for Self-Healing Energy Networks
Computer networks are typically designed following a multi-layer approach that partitions the
functionality of the network infrastructure in well-defined abstractions that lie one atop another. Each
layer abides by a well-defined contract to provide a specific functionality that can be extended by building
further layers above it. For instance, the Internet Protocol (IP) implements the abstraction of a global
network of computers each one having a unique identifying number (a.k.a. an IP number). Using IP
numbers upper layers can send messages among computers without having to worry about how these
messages get routed around the network to reach their destination. Absent from IP is a facility for
applications to communicate among themselves when more than one application runs in a given
computer. This functionality is most often provided by the Transport Control Protocol (TCP) by
assigning each application a distinct port number which combined with the IP number serves as a globally
unique address for applications. This simple example illustrates how each layer in the architecture
delegates responsibilities to lower layers in order to reduce the complexity of the contract that it must
fulfill.
Intelligent Control n
Intelligent Control 0
Data Management
Data Dissemination
Device Drivers
Physical
breakers, switches, flow control
devices
Figure 3. Layered Architecture
for Self-Healing Networks
In essence our proposal encompasses the application of a similar divideand-conquer approach to the design and implementation of self-healing
energy networks. As a starting point we propose the architecture
depicted in Figure 3. As in data networks, each component in the energy
distribution network will implement enough of the “protocol stack” to
support the level of abstraction supported by that device. For instance,
simple devices such as software controllable switches may only need to
implement up to the device driver layer, while more complex devices
such as IPRs will need to implement most or all of the protocol stack.
At the lowest level the physical layer is comprised by the collection of
energy flow control and communication hardware devices that form the
energy and data distribution infrastructure. The physical layer can be
responsible for guaranteeing, for instance, that no energy transmission
lines remain above a threshold load. The device driver layer serves as
the hardware/software interface and its responsibility can be limited to
providing a programmatic interface for controlling the physical layer
devices.
The data dissemination layer supports effective and reliable information dissemination across the
network. This layer provides an interface that allows upper layers to exchange information among
components in the network in a location transparent way, that is, without having to be concerned with low
level details such as routing algorithms and message reliability and security. This layer can be used to
propagate both system status information and control decisions.
The data management layer archives the information disseminated by the lower layers and provides a
distributed database application programming interface (API) that can be used by the upper layers to
query the operational state of the system. One or more intelligent control layers are responsible for
implementing the actual decision making algorithms that respond to sudden changes in system status
triggered by a myriad of possible external and internal factors such as attacks, device failures, human
errors, sudden unexpected power demand shifts, among others. Our research will focus on developing
decentralized controls algorithms that make local decisions aimed at maintaining the system at its
4
maximum functional level according to a configurable set of functional priorities. For instance, in the
event of a power generation capacity loss due to the destruction of a generator in a navy air craft carrier,
sustaining the capacity of the ship to maneuver and communicate could be configured to have higher
priority than sustaining other functions such as reconnaissance and long range weapons systems.
The main concept behind the proposed IPRs scheme is to provide self-healing and adaptive
reconfiguration capabilities to power delivery systems based on contingency assessment. IPRs will
continuously assess the vulnerability of the power delivery system with respect to cascading events. For
instance, whenever the IPRs determine that the power grid is approaching a vulnerable condition, a
controlled islanding scheme could be triggered in order to contain or limit the extent of the damage to the
power delivery system.
This time scale coordination is intrinsically linked to the physical constraints associated to the design and
operation of an energy processing network. For instance, wave phenomena on a transmission network
manifest itself in a time frame between tenths of micro seconds (10 -7 s) and milliseconds (10-3 s) [5],
probably too fast for an IPR network to react to the phenomenon as it unfolds. In this instance the
proposed IPR network will have to react to mitigate the effects of a disturbance on the transmission
network and avoid further deterioration. On the other hand, the proposed IPR scheme could react to
events of electromechanical nature that occur between tenths of milliseconds and minutes.
4. Distributed Algorithms for Fault-Tolerant Information Dissemination
Informally, the information dissemination problem consists of distributing a particular value or data item
(such as the power flowing through a line) to a set of interconnected components distributed across a
network in a way that maximizes the fraction of the components that receive the value and minimizes the
amount of time required for the dissemination to take place. In an ideal environment where failures are
inexistent and components have global knowledge about other components, data dissemination is rather
simple. Under more realistic assumptions the problem becomes significantly more difficult since
components are not absolutely certain about which other components belong to the system at any given
time.
Xie, et.al. [6] identified that problems in real-time monitoring, operating control systems,
communications and information systems contribute to the highest percentage of disturbances in a power
system. They also identify inadequate exchange of real-time operating information and real-time
coordination among control centers, lack of automatic communication systems to receive rapid and
automatic information and lack of timely communication and failure to use available communication
systems as key bottlenecks in the communication and information systems that led to the operation and
restoration problems found in many disturbances.
Self-healing energy networks require effective means for exchanging information among the components
of the system even in the presence failures. Algorithms should be capable of re-routing power and
information even when some channels become destroyed or temporarily unavailable. We propose to
build on recent advances in resilient network overlays [7] and gossip-based routing [8] in order to support
protocols that allow IPRs to stay aware of the operational state of the system even after some components
or communication channels become unavailable. In a gossip-based algorithm all the components of the
system collaborate in the information dissemination by optimistically forwarding information to other
components believed to belong to the system. This distributed and redundant forwarding of messages
reduces the chances that messages main not reach their destination due to destroyed components or links.
Due to the high level of parallelism, gossip-based algorithms are able to propagate information in time
that is logarithmic in the number of components.
To test our algorithms in a realistic environment we propose to integrate the popular NS2 network
simulator2 with the Virtual Testbed3, a multi-technical simulation system developed under the auspices
2
http://www.isi.edu/nsnam/ns/index.html
5
of the Office of Naval Research (ONR) at the University of Southern Carolina. The resulting integrated
environment would be used to test the interaction between the communications and energy distribution
infrastructures under realistic configurations. NS2 will permit the simulation of systems encompassing
cross-discipline subsystems such as electrical, mechanical, and thermal, among others.
5. Algorithms for Reliable Data Management
IPRs need to gather information about system parameters (e.g., frequency, voltage, current and power) in
order to assess current system conditions, detect hazardous events, and make decisions about necessary
actions to keep the system operational. It is critical that neighboring IPRs share their local data with each
other, since this will enable each IPR to be aware of its own status, and the status of neighboring IPRs.
This will enable IPRs to make local decisions, and coordinate damage controls actions that must be taken
as a group. Hence, each IPR needs to maintain a local database to store the various system parameters that
must be analyzed to effectively maintain the electrical system in an acceptable operational state. The
collection of databases at the IPRs become a distribute database system, whose components can be
accessed to extract various views of the data and make appropriate decisions based on these data.
Unfortunately, the vast of amount of information that will be generated by sensors located at various
components in the electrical system makes it impractical to store all these values each of the databases
owned by each IPRs. Hence, it is necessary to sample, aggregate, and compress values based on an
appropriate time intervals, and store these new derived abstractions into each database. Clearly, the
granularity of the time intervals needs to chosen according to the operational nature of the ship, vehicle,
or facility being serviced by the electrical system to make sure that accurate and up-to-date system
behavior is being captured in each database. Another complication arises from the distributed nature of
the data. When, an IPR needs to query data from other neighboring IPRs, this query must be answered
quickly and accurately since the IPR will use this information to possibly make decisions to affect the
system. Hence, the speed and quality of the data must follow within certain quality value range that gives
IPRs a confidence interval to make their decisions. There is also a need to constantly perform data mining
operations on the data to detect patterns that help diagnose problems, and perhaps forecast the occurrence
of a disturbance based on the data gathered from previous disturbances.
We propose to work on algorithms to efficiently aggregate, query and perform data mining operations on
the data in the distributed databases maintained by the IPRs. We shall employ Data Streams [9] and
Continuous queries [10], adapted to the IPRs domain to incorporate time, quality and reliability
constrains. Data Streams refers to data collections that are continuously updated and disseminated by a
data source. Continuous queries are queries that constantly run, inspecting the data from a Data Stream
and generating new query results continuously. Our first trust will be geared towards developing
algorithms to sample the data from the sensors, filter and aggregate these data, and store them at multiple
redundant databases. The idea is for an IPRs to have critical portions of its database replicated at
databases belonging other peer IPRs. In the event of a catastrophic failure that destroys and IPRs, some of
its data can still be recovered from its peers. In addition, we shall develop protocols that enable two or
more IPRs to perform the same computation, thus enabling the given computation to be completed in the
event of a failure that disables or destroys several IPRS.
In order to execute queries efficiently, we propose to develop Peer-to-Peer (P2P) protocols and algorithms
to enable collaboration between IPRs to run a query. We shall investigate algorithms in which IPRs
compete trying to finish the same query or portion of the query, and the winner will receive a reward,
such as extra space to store data, more power for its loads, etc. This type of competitive paradigm is
inspired by systems based on market economies, in which sites try to solve queries by bidding on
operations to be run, and then get paid for their services. With this infrastructure, we shall explore
algorithms for mining the data in the databases looking for trends that can be used to forecast
3
http://vtb.engr.sc.edu
6
contingencies, or for the system to learn how to react to certain event. For example, an IPRs might
observe that a drop in frequency at a nearby generator usually causes its loads to get disconnected. The
IPRs might then monitor the frequency in this generator, and when frequency drops, it might ask other
IPRs or generators to bring in more power to prevent the loads from being disconnected. Likewise, an
IPRs might ask other IPRs to open lines, shed loads, or bring auxiliary generators online. Such IPRs
might use data in the distributed system to justify his request, and the other IPRs might use the same data
to validate the situation, and concur on the solution. This can be seen as a protection mechanism for the
IPRs, to prevent malicious or malfunctioning IPRs from starting actions that do more harm that good to
the electric system. We propose to build simulations and prototypes to validate all these architectures,
protocols, and algorithms.
6. Algorithms for Distributed Decision & Control
Central to the IPR architecture here proposed is the concept of distributed decision and control; the
process of making autonomous decisions based on potentially incomplete or imprecise local information.
Centralized power distribution optimization algorithms have the advantage they can make stronger
assumptions about the system as a whole and as a result have a better chance of finding globally optimal
solutions. Their main drawback is that they are highly susceptible to component failures. Decentralized
algorithms have received considerably less attention from researchers due in fact to the difficulty of
formulating such problems in closed mathematical form in the presence of a variety of failure modes.
Our research is focused on maximizing the survivability of complex power delivery network in the
presence of partial failures as opposed to finding, for instance, the most economical way of supplying
energy to satisfy a given load.
As a starting point in our research the operational behavior of IPRs will be based on two configurable
factors associated with the links interconnecting the IPRs: a priority factor and a reliability factor. The
priority factor is associated with lines that are acting as loads and will establish an ordering in which loads
should be serviced. The reliability factor will be associated with links that are acting as sources and
establishes an ordering in which sources should be used according to the reliability that they provide.
IPRs respond to messages from neighbor components requesting that power flow be increased, such as in
a reconnection or addition of a load, or decreased, such as in the removal or disconnection of a load.
Central to the decision process will be the goal of controlling the propagation of such requests in order to
avoid cascading failures. Therefore the logic inside the IPR will attempt to resolve the request with local
resources before propagating the request to upper IPRs. Strategies that will be attempted will include
shutting down loads with lower priority or increasing the power supplied by nearby generators.
7
7. References
Carlos M. Torres-Ortolaza and Agustín A. Irizarry-Rivera, “Failure Modes and Failure Probability of
Intelligent Power Routers”, Submitted to Eighth Probabilistic Methods Applied to Power Systems (PMAPS)
International Conference, Ames, Iowa, September 13-16, 2004.
2. Agustín A. Irizarry-Rivera, Manuel Rodríguez, Miguel Vélez-Reyes, José R. Cedeño, Bienvenido Vélez,
Efraín O’Neill-Carrillo and Alberto Ramírez, “Intelligent Power Routers for Distributed Coordination in
Electric Energy Processing Networks”, In Proceedings of the 2003 EPNES Workshop, Orlando, Florida,
October 23-24, 2003.
3. Juan J. Jiménez and José R. Cedeño. “A Particle Swarm Optimization Approach for Electric Power System
Restoration”, In Proceedings of the 35th North American Power Symposium, Rolla, MO, October 20-21,
2003.
4. Idalides J. Vergara and Manuel Rodriguez-Martinez, “IPRS: A Decentralized Framework for Controlling
Electrical Energy Distribution Networks with Intelligent Power Routers”, In Proceedings of the 2003
IASTED PowerCon International Conference, New York City, NY, December 2003.
5. Machowski J., Bialek J.W. & Bumby J.R., Power System Dynamics and Stability, Wiley, 1997.
6. Zhaoxia Xie, G. Manimaran, Vijay Vittal, A. G. Phadke, and Virgilio Centeno. “An Information
Architecture for Future Power Systems and Its Reliability Analysis”, In IEEE Transactions on Power
Systems, Vol. 17, No. 3, August 2002.
7. David Andersen, Hari Balakrishnan, M. Frans Kaashoek, and Robert Morris. "Resilient Overlay
Networks". In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP '01),
Chateau Lake Louise, Banff, Canada, October 2001.
8. Kenneth P. Birman, Jie Chen, Ken Hopkinson, Bob Thomas, Jim Thorp, Robbert van Renesse, Werner
Vogels. "Overcoming Communications Challenges in Software for Monitoring and Controlling Power
Systems". Submitted to special issue of the Proceedings of the IEEE, "Energy Infrastructure Defense
Systems", October 2003
9. Alin Dobra, Minos N. Garofalakis, Johannes Gehrke, Rajeev Rastogi, “Processing complex aggregate
queries over data streams”. In Proceedings of 2002 ACM SIGMOD International Conference, Wisconsin,
WI, May 2002.
10. Samuel Madden, Mehul A. Shah, Joseph M. Hellerstein, Vijayshankar Raman, “Continuously adaptive
continuous queries over streams”, In Proceedings of 2002 ACM SIGMOD International Conference,
Wisconsin, WI, May 2002.
1.
8