RINSE Design Document
for SCADA DNP3 Project
(draft)
Revision History:
Revision #
00.01
Date
May-10-2010
Author
David Bergman
Description
Initial draft
Introduction
Purpose
The purpose of this document is to document the software design for extending the
RINSE simulator with the capability of simulating a Supervisory Control and Data
Acquisition (SCADA) network and interaction with both real control equipment and
PowerWorld, a power flow simulator. The current efforts have focused on providing
virtual models for SCADA relays and data aggregators.
These devices communicate
using a virtual DNP3 protocol, which also enables these devices to communicate with
a real control station. Another virtual device, the “State Server” allows virtual
relays to pull values in real-time from the PowerWorld simulator. This document
will cover the necessary design changes (delta changes) to the existing RINSE
Simulator applications and backend to support the simulation and emulation of this
network.
Target Audience
This document will be used by software developers who are involved in this project
to complete the implementation of the design.
It may also be used by anyone
looking for more technical information about this project and its proposed
implementation.
Abbreviations, Acronyms and Definitions
DNP
Distributed Network Protocol (v3.0)
SCADA
Supervisory Control and Data Acquisition
RTU
Remote Terminal Unit
CRC
Cyclic Redundancy Check
IED
Intelligent Electronic Device
Emu
Emulation
App
Application
IP
Internet Protocol
DML
Domain Modeling Language
RINSE
Real-time Immersive Network Simulation Environment
SSF
Scalable Simulation Framework
Module Affected By This Change/Design
The following applications have been changed/added:

DNP Message (new)

BlockingDNPRelay Application (new)

BlockingDNPOutstation Application (new)

BlockingDNPStateServer Application (new)
Constraints and Limitations
External traffic must be properly set up to be directed to the correct
locations. For instance, in order for RINSE to gather information
from PowerWorld, simpleproxy and openvpn must be set up to
direct traffic to/from the PowerWorld server.High-Level Design
Decomposition: DML files
Ground-Up Designs
The DML description of the Relay, Data Aggregator and the State Server (a
virtual host with the capability of acquiring data from an external source),
are shown as follows. Also, there are two purely emulated nodes in the
design as well – corresponding to the Control Station and PowerWorld.
Note: The netvis key is for use with the network visualizer. It tells the
GUI where to position the icon and what icon to use. RINSE just ignores it.
Each host must use tcp and socket ProtocolSessions as well as an appropriate
application layer ProtocolSession.
Relay
Host [ id 1
name "DNPRelay_generator_31_1"
netvis [
position "600.0, 300.0"
icon /home/ssf/demo/images/sel_421_small.jpg
]
graph [
ProtocolSession [ name app use "SSF.OS.DNP.blockingDNPRelay"
pw_id generaor_31_1 _extends .dict.dnprel_setup ]
ProtocolSession [ name socket use "SSF.OS.Socket.blockingSocketMaster"
]
ProtocolSession [ name tcp use "SSF.OS.TCP.tcpSessionMaster" _find
.dict.tcpinit ]
ProtocolSession [ name icmp use "SSF.OS.ICMP" ]
ProtocolSession [ name ip
use "SSF.OS.IP" ]
]
interface [ id 0 _extends .dict.iface1]
]
Outstation
Host [ id 1
name "DNPOutstation_31"
graph [
ProtocolSession [ name app use "SSF.OS.DNP.blockingDNPOutstation"
master_addr 100 _extends .dict.dnpout_setup ]
ProtocolSession [ name socket use "SSF.OS.Socket.blockingSocketMaster"
]
ProtocolSession [ name tcp use "SSF.OS.TCP.tcpSessionMaster" _find
.dict.tcpinit ]
ProtocolSession [ name icmp use "SSF.OS.ICMP" ]
ProtocolSession [ name ip
use "SSF.OS.IP" ]
]
interface [ id 0 _extends .dict.iface1 ]
interface [ id 1 _extends .dict.iface1 ]
]
StateServer
host [ id 1
name "StateServer"
graph [
ProtocolSession [ name app use "SSF.OS.DNP.blockingDNPStateServer"
_extends .dict.dnpss_setup ]
ProtocolSession [ name socket use
"SSF.OS.Socket.blockingSocketMaster" ]
ProtocolSession [ name tcp use "SSF.OS.TCP.tcpSessionMaster" _find
.dict.tcpinit ]
ProtocolSession [ name icmp use "SSF.OS.ICMP" ]
ProtocolSession [ name ip
use "SSF.OS.IP" ]
]
interface [ id 0 _extends .dict.iface1 ]
]
Control Station proxy
host [ id 1
name "Control_Station"
graph [
ProtocolSession [ name ip use "SSF.OS.IP" ]
ProtocolSession [ name icmp use "SSF.OS.ICMP" ]
ProtocolSession [ name emu use "SSF.OS.Emulation" ip_forwarding true]
]
interface [ id 0
bitrate 1e8
latency 0
]
interface [ id 1
bitrate 1e8
latency 0
]
]
PowerWorld proxy
host [ id 1
name "PowerWorld"
graph [
ProtocolSession [ name emu use "SSF.OS.Emulation" ip_forwarding true
host_mirrored true]
ProtocolSession [ name icmp use "SSF.OS.ICMP" ]
ProtocolSession [ name ip
use "SSF.OS.IP" ]
]
interface [ id 0 _extends .dict.iface1]
]
Delta Design Changes
None
Decomposition: DNP Message
Ground-Up Design
The DNP Message class is defined to contain all the DNP message fields and all
associated primitives within the scope of this project. The class can assist the
DNP host session classes by parsing and creating bytestreams, and by calculating
CRCs.
Every DNP Message is composed of three layers with various fields in each
layer:
Data-link layer:
Length
Control bits
Source
Destination
CRC
Transport Layer:
Control bits
Sequence Number
Application Layer:
Control bits
Sequence Number
Function code
CRC
Indicator bits
Array of application requests (Exists in virtual representation, but gets
converted to bytes when creating a bytestream)
Data pointer
Data length
Delta Design Changes
Creation
Decomposition: Outstation App
Ground-Up Design
The Oustation app is a SSF procedure and a protocol session. It represents an
outstation data aggregator, with the intelligence to handle DNP message received
from the socket session below it.
The app has two threads running at any given time: a listener thread and a request
thread.
The listener thread waits for a request from the control station and
responds with an appropriate response. The request thread sends requests to each
of the relays in its relay list and waits for their response in a round-robin
manner. Once it receives a response, there is currently a stub method to process
the data.
TODO List:

Aggregate the data that comes from the relays and place it into the response
to the control station accordingly.
Delta Design Changes
Creation
Decomposition: Relay App
Ground-Up Designs
The Relay app is a SSF procedure and a protocol session. It represents a relay in
the SCADA control network, with the intelligence to handle DNP message received
from the socket session below it.
The app has a response thread that is created once and persists for the run of the
simulation. This thread waits for a request from the data aggregator and responds
with an appropriate response.
Depending on the type of request, the relay will
respond accordingly. When queried for data, the relay will access memory which is
shared with the state server. In this manner, the relay retrieves the most current
electrical data, as reported by PowerWorld’s API.
Delta Design Changes
Creation
Decomposition: State Server App
Ground-Up Designs
The State Server app is a SSF procedure and a protocol session. It has no realworld analog in a SCADA network. It sits above the socket app session layer and
its main purpose is to communicate with the PowerWorld server to retrieve real-time
data.
PowerWorld is an external electrical grid simulator.
In our test, our copy of
PowerWorld has been modified to include a server which can be queried for values of
the buses, etc. in the simulation. It provides an API, and the State Server issues
a series of commands that sets up a session with the Server. From there, the State
Server issues commands to request values, and parses these values into the
appropriate shared memory upon their arrival.
Delta Design Changes
Creation
Detailed (Low-Level) Design
Classes
New Classes
1.
DNPMessage Class
The constructor
prime::ssfnet::DNPMessage::DNPMessage()
The destructor
prime::ssfnet::DNPMessage::~DNPMessage()
Member Functions
/** Return the protocol number that the message is representing. */
int type() { return SSFNET_PROTOCOL_TYPE_DNP;
/** Return the length. */
int getLength() { return length; }
/** Takes in a bytestream and and sets corresponding fields in the DNPMessage */
void fromByteStream(byte* msg, int msg_length)
/** Converts data structure into a byte stream and returns a pointer to the stream.
* Stores size of message into length*/
byte* toByteStream(uint32& length);
/** Pack this DNP message to a real DNP header starting from the given offset,
although some information is missing. */
virtual void toRealBytes(byte* buf, int& offset);
/** Unpack the DNP message from the buffer that stores the real DNP header at the
given offset. */
virtual void fromRealBytes(byte* buf, int& offset);
/* Set of functions that add CRCs to the DNP Message.
should be called */
DNPAddCRCs is the one that
byte* DNPAddCRCs(byte* msg, uint16&length, byte* app_data, uint16
&app_data_length);
void DNPComputeCRC(byte, uint16*);
uint16 calcCRCRoom(uint16 length);
2.
BlockingDNPOutstationSession Class
constructor
The default constructor
prime::ssfnet::BlockingDNPOutstationSession(ProtocolGraph* graph);
destructor
virtual prime::ssfnet::~BlockingDNPOutstationSession();
Member Functions
virtual void config(prime::dml::Configuration *cfg);
/**
* Return the protocol number. This specifies the relationship
* between this protocol and the other protocols defined in the same
* protocol stack.
*/
virtual int getProtocolNumber() { return SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_OU
TSTATION; }
/** Initialize this protocol session. */
virtual void init();
/** The main process of the dnp outstation. */
void main_proc(prime::ssf::Process*); /
/** The process for handling each incoming client connection. */
void session_proc(prime::ssf::Process*);
// called by blockingDNPOutstationSessionTimer upon timer expiration
void timer_callback();
/** Sends a request to a specific relay */
void sendRequest(IPADDR relay_ip, uint16 relay_port, uint32 ridx);
/** This method is used by the outstation to send requests to the relays in
* the relay_list
*/
void sendRequests();
/** Processes a response from a relay */
void processData(int index, DNPMessage* msg);
/** This method should not be called; it is provided here to prompt
error message if it's called accidentally. */
virtual int push(ProtocolMessage* msg, ProtocolSession* hi_sess,
void* extinfo = 0, size_t extinfo_size = 0);
an
/** This method should not be called; it is provided here to prompt
an
error message if it's called accidentally. */
virtual int pop(ProtocolMessage* msg, Packet* packet, ProtocolSession* lo_ses
s, void* extinfo = 0, size_t extinfo_size = 0);
/** This method is used by the outstation to generate a response to a
* request from the control station,
*/
byte* generateResponse(DNPMessage* dnpRequest, uint32& response_size);
/** This method is used by the outstation to generate a request to send
to a relay
*/
byte* generateRequest(uint32& slave_address, uint32& response_size);
/** This method is used by the outstation to generate a confirmation
message * to send to a relay
*/
byte* generateConfirm(uint32& slave_address, uint32& response_size);
*
/** called by DNP hosts that need to print messages */
void displayDnpMsg(DNPMessage* Dnpmsg);
/** This method alters Dnpmsg's app_data and app_data_length
* such that it
will now carry a message. Used to send responses
* to control stations.
*/
void insertData(DNPMessage* Dnpmsg);
/** This method alters Dnpmsg's app_data and app_data length
* such that it
now carries a request for data. Used to send
* requests to relays.
*/
void insertRequestData(DNPMessage* Dnpmsg, uint32& options);
//** Populates the vector 'relays' with relay info */137
getRelayList();
bool
/**Returns the ip and port for a given relay with id ridx */
bool getRelayInfo(IPADDR& relay_ip, uint16& relay_port, uint32 ridx);
Member Data
// the following stores the configuration from read from DML
SSFNET_STRING poll_message;
SSFNET_STRING poll_peer_nhi;
SSFNET_STRING poll_peer_ip;
// keep an account on how many polling messages sent and received
int nsent;
int nrcvd;
// state information of this protocol session
IPADDR poll_peer; // the IP of the peer we send periodic hello messages
ProtocolSession* sock_session; // the IP layer below this protocol
private:
// configurable parameters
byte* reqbuf_sp; //! SSF STATE
//byte* reqbuf_srs; //! SSF STATE
byte* reqbuf_sr; //! SSF STATE
byte* respbuf_sr; //! SSF STATE
byte* confbuf_sr; //! SSF STATE
VirtualTime start_time;
///< Time before a session starts.
VirtualTime start_window; ///< Size of random window before session starts.
VirtualTime poll_interval; ///<Time between sending out requests
VirtualTime user_timeout;
///< Timeout before aborting a session.
VirtualTime proc_time; ///<Time it takes to process a request
uint16 outstation_port; ///< Port number to receive incoming request.
uint32 outstation_master_address; ///<Master address for the outstation
uint32 outstation_slave_address; ///<Slave address for the outstation
uint32 request_size; ///< Size of the request from/to control station /
relay.
uint32 response_size; ///<Size of the response to/from control station /
relay
.
uint32 confirm_size; ///<Size of the confirmation message to relay.
uint32 client_limit; ///< Number of client sessions that can be handled
simult
aneously.
bool show_report; ///< Whether we print out the result or not.
SSFNET_STRING relay_list; ///< name used to filter traffic patterns
// state variables
BlockingSocketMaster* sm; ///< Point to the socket master.
prime::ssf::ssf_semaphore*
sessions.
bound_sem;
//
Bound
the
number
of
client
prime::ssf::ssf_semaphore* next_sem; // Used for continuation.
prime::ssf::ssf_semaphore* bound_sem_relay; // Bound the number of client
sess
ions.
prime::ssf::ssf_semaphore* next_sem_relay; // Used for continuation.
IPADDR outstation_ip; // IP address of this server (interface 0).
int payloadCounter; //Added as last byte of dnp message
//byte dataPayload[DNP_MESSAGE_SIZE];
byte* dataPayload_p;
byte* dataPayload_id;
byte* dataPayload_ird;
DNPMessage* dnpInRequest;
//Request from Control Station
DNPMessage* dnpInResponse; //Response from Relay
DNPMessage* dnpInConfirm;
//Confirm from Control Station
DNPMessage* dnpOutResponse; //Response to Control Station
DNPMessage* dnpOutRequest;
//Request to Relay
DNPMessage* dnpOutConfirm;
//Confirm to Relay
TrafficServerData*
address
currentRelayData;
//List
of
Relay's
nhi's
to
ip
SSFNET_VECTOR(uint16) tran_seqs;
numbers for each relay
//Holds the transmission layer sequence
SSFNET_VECTOR(uint16) app_seqs;
numbe rs for each relay
//Holds the application layer sequence
SSFNET_VECTOR(int) relay_socks;
with each specific relay
//Holds the sockets used to communicate
//List of Relays
SSFNET_VECTOR(TrafficServerData*) relays;
3.
BlockingDNPRelaySession Class
constructor
prime::ssfnet: BlockingDNPRelaySession(ProtocolGraph* graph);
destructor
prime::ssfnet::~BlockingDNPRelaySession();
Member Functions
/**
* Return the protocol number. This specifies the relationship
* between this protocol and the other protocols defined in the same
* protocol stack.
*/
virtual
int
getProtocolNumber()
{
return
SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_RELAY; }
/** Initialize this protocol session. */
virtual void init();
/** The main process of the dnp relay. */
void main_proc(prime::ssf::Process*); //! SSF PROCEDURE
/** The process for handling each incoming client connection. */
void session_proc(prime::ssf::Process*); //! SSF PROCEDURE
/** Sends a request to a specific relay */
//void sendUpdateRequest(); //! SSF PROCEDURE
// called by blockingDNPRelaySessionTimer upon timer expiration
void timer_callback(); //! SSF PROCEDURE
protected:
/** This method should not be called; it is provided here to prompt
an error message if it's called accidentally. */
virtual int push(ProtocolMessage* msg, ProtocolSession* hi_sess,
void* extinfo = 0, size_t extinfo_size = 0);
/** This method should not be called; it is provided here to prompt
an error message if it's called accidentally. */
virtual int pop(ProtocolMessage* msg, Packet* packet,
ProtocolSession* lo_sess,
void* extinfo = 0, size_t extinfo_size = 0);
/** This method is used by the relay to generate a response to a
* request from the control station,
*/
byte* generateResponse(DNPMessage* dnpRequest, uint32& response_size);
/** called by DNP hosts that need to print messages */
void displayDnpMsg(DNPMessage* Dnpmsg);
/** This method alters Dnpmsg's app_data and app_data_length
* such that it will now carry a message
*/
void insertData(DNPMessage* Dnpmsg);
/** Grabs data from PowerWorld and puts it into the payload
*/
void insertPWData(DNPMessage* Dnpmsg);
Member Data
VirtualTime update_interval; ///<Time between sending out requests for
line value
VirtualTime proc_time; ///<Time it takes to process a request
uint16 relay_port; ///< Port number to receive incoming request.
char* pw_id; //<Corresponds to the bus ID counterpart in Powerworld
char* dummy;
uint32 request_size; ///< Size of the request from client (must be
consistent).
uint32 client_limit; ///< Number of client sessions that can be handled
simultaneously.
bool show_report; ///< Whether we print out the result or not.
// state variables
byte* reqbuf; //! SSF STATE
BlockingSocketMaster* sm; ///< Point to the socket master.
prime::ssf::ssf_semaphore* bound_sem; // Bound the number
of
client
sessions.
prime::ssf::ssf_semaphore* next_sem; // Used for continuation.
IPADDR relay_ip; // IP address of this server (interface 0).
//byte dataPayload[DNP_MESSAGE_SIZE];
byte* dataPayload_p;
byte* dataPayload_id;
uint32 response_size;
DNPMessage* dnpRequest;
DNPMessage* dnpResponse;
short type;
// Is this a line/generator/shunt/load?
4.
BlockingDNPStateServerSession Class
constructor
prime::ssfnet::BlockingDNPStateServerSession(ProtocolGraph* graph);
destructor
prime::ssfnet::~BlockingDNPStateServerSession();
Member Functions
/** Configure the blocking DNP state server test protocol session. */
virtual void config(prime::dml::Configuration *cfg);
/**
* Return the protocol number. This specifies the relationship
* between this protocol and the other protocols defined in the same
* protocol stack.
*/
virtual
int
getProtocolNumber()
{
return
SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_RELAY; }
/** Initialize this protocol session. */
virtual void init();
/** The main process of the dnp state server. */
void main_proc(prime::ssf::Process*); //! SSF PROCEDURE
/** Sends a request to a specific state server
void sendUpdateRequest(int); //! SSF PROCEDURE
*/
// called by blockingDNPStateServerSessionTimer upon timer expiration
void timer_callback(prime::ssf::Process* p); //! SSF PROCEDURE
protected:
/** This method should not be called; it is provided here to prompt
an error message if it's called accidentally. */
virtual int push(ProtocolMessage* msg, ProtocolSession* hi_sess,
void* extinfo = 0, size_t extinfo_size = 0);
/** This method should not be called; it is provided here to prompt
an error message if it's called accidentally. */
virtual int pop(ProtocolMessage* msg, Packet* packet,
ProtocolSession* lo_sess,
void* extinfo = 0, size_t extinfo_size = 0);
/** This method is used by the state server to generate an update request
to send
* to PowerWorld. Returns the size of the request in bytes
*/
short generateUpdateRequest(int);
Member Data
VirtualTime update_interval; ///<Time between sending out requests for line
value
uint16 state_server_port; ///< Port number to receive incoming request.
uint32 request_size; ///< Size of the request from client (must be
consistent)
.
uint32 client_limit; ///< Number of client sessions that can be handled
simultaneously.
bool show_report; ///< Whether we print out the result or not.
// state variables
byte* pw_data_size; //! SSF STATE //<Used to set the size of powerworld
data in bytes
byte* pw_data; //! SSF STATE
//<pointer to the data itself
short* request_size_short; //! SSF STATE
long* padding; //! SSF STATE
short* clientVersion; //! SSF STATE
byte* respbuf; //! SSF STATE
char* getSCADAMsg; //! SSF STATE
bool first;
BlockingSocketMaster* sm; ///< Point to the socket master.
prime::ssf::ssf_semaphore* bound_sem; // Bound the number of client
sessions.
prime::ssf::ssf_semaphore* next_sem; // Used for continuation.
IPADDR state_server_ip; // IP address of this server (interface 0).
int payloadCounter; //Added as last byte of dnp message
byte dataPayload[DNP_MESSAGE_SIZE];
byte* dataPayload_p;
Nhi* sn_nhi;
//Holds the nhi of the super node
int sn_port;
//Holds the port used to communicate with the super node
Nhi* pw_nhi;
//Holds the nhi of the super node
int pw_sock;
//Holds the socket used to communicate with the super node
int pw_port;
//Holds the port used to communicate with the super node
IPADDR sn_ip; // IP address of the super node (interface 0).
IPADDR pw_ip; // IP address of the super node (interface 0).
typedef
typedef
typedef
typedef
typedef
std::map<std::string,
std::map<std::string,
std::map<std::string,
std::map<std::string,
std::map<std::string,
typedef
typedef
typedef
typedef
typedef
std::pair<std::string,
std::pair<std::string,
std::pair<std::string,
std::pair<std::string,
std::pair<std::string,
pwtype_generator> genMap;
pwtype_load> loadMap;
pwtype_shunt> shuntMap;
pwtype_line> lineMap;
std::string> stringMap;
pwtype_generator> genStringPair;
pwtype_load> loadStringPair;
pwtype_shunt> shuntStringPair;
pwtype_line> lineStringPair;
std::string> stringPair;
//Deprecated?
stringMap real_to_nhi_map; //Contains mapping from pw IDs to virtual IDs
stringMap real_to_nhi_bus_map; //Contains mapping from pw IDs to virtual
IDs for busses
loadMap real_to_nhi_load_map; //Contains mapping from pw IDs to virtual IDs
for loads
genMap real_to_nhi_gen_map; //Contains mapping from pw IDs to virtual IDs
for generators
lineMap real_to_nhi_line_map; //Contains mapping from pw IDs to virtual IDs
for lines
shuntMap real_to_nhi_shunt_map; //Contains mapping from pw IDs to virtual
IDs for shunts
Member Classes:
class pwtype_generator{
public:
int bus;
byte ID;
};
class pwtype_load{
public:
int bus;
byte ID;
};
class pwtype_shunt{
public:
int bus;
byte ID;
};
class pwtype_line{
public:
int bus_src;
int bus_dst;
int dst;
byte ID;
};
Changes To Existing Classes
NONE
Functions
NONE
Files
Changes To Existing Files
Added to ssfnet.cc:
SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_OUTSTATION= 239
SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_RELAY= 240
SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_STATE_SERVER = 241
Added to ssfnet/Makefile.common.in:
## os/dnp: ##
DNP_HDRFILES =
$(SRCDIR)/os/dnp/blocking_dnp_outstation.h
$(SRCDIR)/os/dnp/blocking_dnp_relay.h
$(SRCDIR)/os/dnp/blocking_dnp_state_server.h
$(SRCDIR)/os/dnp/dnp_session.h
$(SRCDIR)/os/dnp/dnp_message.h
DNP_SRCFILES =
$(SRCDIR)/os/dnp/blocking_dnp_outstation.cxx
$(SRCDIR)/os/dnp/blocking_dnp_relay.cxx
$(SRCDIR)/os/dnp/blocking_dnp_state_server.cxx
$(SRCDIR)/os/dnp/dnp_session.cc
$(SRCDIR)/os/dnp/dnp_message.cc
References
1. http://www.scribd.com/doc/2542661/DNP-v3-0-Guide-A-Protocol-Primer
2. DNP Users Group DNP Primer (www.dnp.org/About/DNP3%20Primer%20Rev%20A.pdf)
3. A
Taxonomy
of
Attacks
on
the
DNP3
Protocol
(http://www.springerlink.com/content/k48k4733v0367120/)
4. The DNP Users group at dnp.org. Contact Himanshu or others for login info
5. Wikipedia for a high-level view... but nothing technical
TODO List:
A. Data Aggregation in Outstation host
B. Large network size – in the PW2R.py file, just have to add multiple layers of
networks.
C. Connect EMS to multiple Data Aggregators