RINSE Design Document for SCADA DNP3 Project (draft) Revision History: Revision # 00.01 Date May-10-2010 Author David Bergman Description Initial draft Introduction Purpose The purpose of this document is to document the software design for extending the RINSE simulator with the capability of simulating a Supervisory Control and Data Acquisition (SCADA) network and interaction with both real control equipment and PowerWorld, a power flow simulator. The current efforts have focused on providing virtual models for SCADA relays and data aggregators. These devices communicate using a virtual DNP3 protocol, which also enables these devices to communicate with a real control station. Another virtual device, the “State Server” allows virtual relays to pull values in real-time from the PowerWorld simulator. This document will cover the necessary design changes (delta changes) to the existing RINSE Simulator applications and backend to support the simulation and emulation of this network. Target Audience This document will be used by software developers who are involved in this project to complete the implementation of the design. It may also be used by anyone looking for more technical information about this project and its proposed implementation. Abbreviations, Acronyms and Definitions DNP Distributed Network Protocol (v3.0) SCADA Supervisory Control and Data Acquisition RTU Remote Terminal Unit CRC Cyclic Redundancy Check IED Intelligent Electronic Device Emu Emulation App Application IP Internet Protocol DML Domain Modeling Language RINSE Real-time Immersive Network Simulation Environment SSF Scalable Simulation Framework Module Affected By This Change/Design The following applications have been changed/added: DNP Message (new) BlockingDNPRelay Application (new) BlockingDNPOutstation Application (new) BlockingDNPStateServer Application (new) Constraints and Limitations External traffic must be properly set up to be directed to the correct locations. For instance, in order for RINSE to gather information from PowerWorld, simpleproxy and openvpn must be set up to direct traffic to/from the PowerWorld server.High-Level Design Decomposition: DML files Ground-Up Designs The DML description of the Relay, Data Aggregator and the State Server (a virtual host with the capability of acquiring data from an external source), are shown as follows. Also, there are two purely emulated nodes in the design as well – corresponding to the Control Station and PowerWorld. Note: The netvis key is for use with the network visualizer. It tells the GUI where to position the icon and what icon to use. RINSE just ignores it. Each host must use tcp and socket ProtocolSessions as well as an appropriate application layer ProtocolSession. Relay Host [ id 1 name "DNPRelay_generator_31_1" netvis [ position "600.0, 300.0" icon /home/ssf/demo/images/sel_421_small.jpg ] graph [ ProtocolSession [ name app use "SSF.OS.DNP.blockingDNPRelay" pw_id generaor_31_1 _extends .dict.dnprel_setup ] ProtocolSession [ name socket use "SSF.OS.Socket.blockingSocketMaster" ] ProtocolSession [ name tcp use "SSF.OS.TCP.tcpSessionMaster" _find .dict.tcpinit ] ProtocolSession [ name icmp use "SSF.OS.ICMP" ] ProtocolSession [ name ip use "SSF.OS.IP" ] ] interface [ id 0 _extends .dict.iface1] ] Outstation Host [ id 1 name "DNPOutstation_31" graph [ ProtocolSession [ name app use "SSF.OS.DNP.blockingDNPOutstation" master_addr 100 _extends .dict.dnpout_setup ] ProtocolSession [ name socket use "SSF.OS.Socket.blockingSocketMaster" ] ProtocolSession [ name tcp use "SSF.OS.TCP.tcpSessionMaster" _find .dict.tcpinit ] ProtocolSession [ name icmp use "SSF.OS.ICMP" ] ProtocolSession [ name ip use "SSF.OS.IP" ] ] interface [ id 0 _extends .dict.iface1 ] interface [ id 1 _extends .dict.iface1 ] ] StateServer host [ id 1 name "StateServer" graph [ ProtocolSession [ name app use "SSF.OS.DNP.blockingDNPStateServer" _extends .dict.dnpss_setup ] ProtocolSession [ name socket use "SSF.OS.Socket.blockingSocketMaster" ] ProtocolSession [ name tcp use "SSF.OS.TCP.tcpSessionMaster" _find .dict.tcpinit ] ProtocolSession [ name icmp use "SSF.OS.ICMP" ] ProtocolSession [ name ip use "SSF.OS.IP" ] ] interface [ id 0 _extends .dict.iface1 ] ] Control Station proxy host [ id 1 name "Control_Station" graph [ ProtocolSession [ name ip use "SSF.OS.IP" ] ProtocolSession [ name icmp use "SSF.OS.ICMP" ] ProtocolSession [ name emu use "SSF.OS.Emulation" ip_forwarding true] ] interface [ id 0 bitrate 1e8 latency 0 ] interface [ id 1 bitrate 1e8 latency 0 ] ] PowerWorld proxy host [ id 1 name "PowerWorld" graph [ ProtocolSession [ name emu use "SSF.OS.Emulation" ip_forwarding true host_mirrored true] ProtocolSession [ name icmp use "SSF.OS.ICMP" ] ProtocolSession [ name ip use "SSF.OS.IP" ] ] interface [ id 0 _extends .dict.iface1] ] Delta Design Changes None Decomposition: DNP Message Ground-Up Design The DNP Message class is defined to contain all the DNP message fields and all associated primitives within the scope of this project. The class can assist the DNP host session classes by parsing and creating bytestreams, and by calculating CRCs. Every DNP Message is composed of three layers with various fields in each layer: Data-link layer: Length Control bits Source Destination CRC Transport Layer: Control bits Sequence Number Application Layer: Control bits Sequence Number Function code CRC Indicator bits Array of application requests (Exists in virtual representation, but gets converted to bytes when creating a bytestream) Data pointer Data length Delta Design Changes Creation Decomposition: Outstation App Ground-Up Design The Oustation app is a SSF procedure and a protocol session. It represents an outstation data aggregator, with the intelligence to handle DNP message received from the socket session below it. The app has two threads running at any given time: a listener thread and a request thread. The listener thread waits for a request from the control station and responds with an appropriate response. The request thread sends requests to each of the relays in its relay list and waits for their response in a round-robin manner. Once it receives a response, there is currently a stub method to process the data. TODO List: Aggregate the data that comes from the relays and place it into the response to the control station accordingly. Delta Design Changes Creation Decomposition: Relay App Ground-Up Designs The Relay app is a SSF procedure and a protocol session. It represents a relay in the SCADA control network, with the intelligence to handle DNP message received from the socket session below it. The app has a response thread that is created once and persists for the run of the simulation. This thread waits for a request from the data aggregator and responds with an appropriate response. Depending on the type of request, the relay will respond accordingly. When queried for data, the relay will access memory which is shared with the state server. In this manner, the relay retrieves the most current electrical data, as reported by PowerWorld’s API. Delta Design Changes Creation Decomposition: State Server App Ground-Up Designs The State Server app is a SSF procedure and a protocol session. It has no realworld analog in a SCADA network. It sits above the socket app session layer and its main purpose is to communicate with the PowerWorld server to retrieve real-time data. PowerWorld is an external electrical grid simulator. In our test, our copy of PowerWorld has been modified to include a server which can be queried for values of the buses, etc. in the simulation. It provides an API, and the State Server issues a series of commands that sets up a session with the Server. From there, the State Server issues commands to request values, and parses these values into the appropriate shared memory upon their arrival. Delta Design Changes Creation Detailed (Low-Level) Design Classes New Classes 1. DNPMessage Class The constructor prime::ssfnet::DNPMessage::DNPMessage() The destructor prime::ssfnet::DNPMessage::~DNPMessage() Member Functions /** Return the protocol number that the message is representing. */ int type() { return SSFNET_PROTOCOL_TYPE_DNP; /** Return the length. */ int getLength() { return length; } /** Takes in a bytestream and and sets corresponding fields in the DNPMessage */ void fromByteStream(byte* msg, int msg_length) /** Converts data structure into a byte stream and returns a pointer to the stream. * Stores size of message into length*/ byte* toByteStream(uint32& length); /** Pack this DNP message to a real DNP header starting from the given offset, although some information is missing. */ virtual void toRealBytes(byte* buf, int& offset); /** Unpack the DNP message from the buffer that stores the real DNP header at the given offset. */ virtual void fromRealBytes(byte* buf, int& offset); /* Set of functions that add CRCs to the DNP Message. should be called */ DNPAddCRCs is the one that byte* DNPAddCRCs(byte* msg, uint16&length, byte* app_data, uint16 &app_data_length); void DNPComputeCRC(byte, uint16*); uint16 calcCRCRoom(uint16 length); 2. BlockingDNPOutstationSession Class constructor The default constructor prime::ssfnet::BlockingDNPOutstationSession(ProtocolGraph* graph); destructor virtual prime::ssfnet::~BlockingDNPOutstationSession(); Member Functions virtual void config(prime::dml::Configuration *cfg); /** * Return the protocol number. This specifies the relationship * between this protocol and the other protocols defined in the same * protocol stack. */ virtual int getProtocolNumber() { return SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_OU TSTATION; } /** Initialize this protocol session. */ virtual void init(); /** The main process of the dnp outstation. */ void main_proc(prime::ssf::Process*); / /** The process for handling each incoming client connection. */ void session_proc(prime::ssf::Process*); // called by blockingDNPOutstationSessionTimer upon timer expiration void timer_callback(); /** Sends a request to a specific relay */ void sendRequest(IPADDR relay_ip, uint16 relay_port, uint32 ridx); /** This method is used by the outstation to send requests to the relays in * the relay_list */ void sendRequests(); /** Processes a response from a relay */ void processData(int index, DNPMessage* msg); /** This method should not be called; it is provided here to prompt error message if it's called accidentally. */ virtual int push(ProtocolMessage* msg, ProtocolSession* hi_sess, void* extinfo = 0, size_t extinfo_size = 0); an /** This method should not be called; it is provided here to prompt an error message if it's called accidentally. */ virtual int pop(ProtocolMessage* msg, Packet* packet, ProtocolSession* lo_ses s, void* extinfo = 0, size_t extinfo_size = 0); /** This method is used by the outstation to generate a response to a * request from the control station, */ byte* generateResponse(DNPMessage* dnpRequest, uint32& response_size); /** This method is used by the outstation to generate a request to send to a relay */ byte* generateRequest(uint32& slave_address, uint32& response_size); /** This method is used by the outstation to generate a confirmation message * to send to a relay */ byte* generateConfirm(uint32& slave_address, uint32& response_size); * /** called by DNP hosts that need to print messages */ void displayDnpMsg(DNPMessage* Dnpmsg); /** This method alters Dnpmsg's app_data and app_data_length * such that it will now carry a message. Used to send responses * to control stations. */ void insertData(DNPMessage* Dnpmsg); /** This method alters Dnpmsg's app_data and app_data length * such that it now carries a request for data. Used to send * requests to relays. */ void insertRequestData(DNPMessage* Dnpmsg, uint32& options); //** Populates the vector 'relays' with relay info */137 getRelayList(); bool /**Returns the ip and port for a given relay with id ridx */ bool getRelayInfo(IPADDR& relay_ip, uint16& relay_port, uint32 ridx); Member Data // the following stores the configuration from read from DML SSFNET_STRING poll_message; SSFNET_STRING poll_peer_nhi; SSFNET_STRING poll_peer_ip; // keep an account on how many polling messages sent and received int nsent; int nrcvd; // state information of this protocol session IPADDR poll_peer; // the IP of the peer we send periodic hello messages ProtocolSession* sock_session; // the IP layer below this protocol private: // configurable parameters byte* reqbuf_sp; //! SSF STATE //byte* reqbuf_srs; //! SSF STATE byte* reqbuf_sr; //! SSF STATE byte* respbuf_sr; //! SSF STATE byte* confbuf_sr; //! SSF STATE VirtualTime start_time; ///< Time before a session starts. VirtualTime start_window; ///< Size of random window before session starts. VirtualTime poll_interval; ///<Time between sending out requests VirtualTime user_timeout; ///< Timeout before aborting a session. VirtualTime proc_time; ///<Time it takes to process a request uint16 outstation_port; ///< Port number to receive incoming request. uint32 outstation_master_address; ///<Master address for the outstation uint32 outstation_slave_address; ///<Slave address for the outstation uint32 request_size; ///< Size of the request from/to control station / relay. uint32 response_size; ///<Size of the response to/from control station / relay . uint32 confirm_size; ///<Size of the confirmation message to relay. uint32 client_limit; ///< Number of client sessions that can be handled simult aneously. bool show_report; ///< Whether we print out the result or not. SSFNET_STRING relay_list; ///< name used to filter traffic patterns // state variables BlockingSocketMaster* sm; ///< Point to the socket master. prime::ssf::ssf_semaphore* sessions. bound_sem; // Bound the number of client prime::ssf::ssf_semaphore* next_sem; // Used for continuation. prime::ssf::ssf_semaphore* bound_sem_relay; // Bound the number of client sess ions. prime::ssf::ssf_semaphore* next_sem_relay; // Used for continuation. IPADDR outstation_ip; // IP address of this server (interface 0). int payloadCounter; //Added as last byte of dnp message //byte dataPayload[DNP_MESSAGE_SIZE]; byte* dataPayload_p; byte* dataPayload_id; byte* dataPayload_ird; DNPMessage* dnpInRequest; //Request from Control Station DNPMessage* dnpInResponse; //Response from Relay DNPMessage* dnpInConfirm; //Confirm from Control Station DNPMessage* dnpOutResponse; //Response to Control Station DNPMessage* dnpOutRequest; //Request to Relay DNPMessage* dnpOutConfirm; //Confirm to Relay TrafficServerData* address currentRelayData; //List of Relay's nhi's to ip SSFNET_VECTOR(uint16) tran_seqs; numbers for each relay //Holds the transmission layer sequence SSFNET_VECTOR(uint16) app_seqs; numbe rs for each relay //Holds the application layer sequence SSFNET_VECTOR(int) relay_socks; with each specific relay //Holds the sockets used to communicate //List of Relays SSFNET_VECTOR(TrafficServerData*) relays; 3. BlockingDNPRelaySession Class constructor prime::ssfnet: BlockingDNPRelaySession(ProtocolGraph* graph); destructor prime::ssfnet::~BlockingDNPRelaySession(); Member Functions /** * Return the protocol number. This specifies the relationship * between this protocol and the other protocols defined in the same * protocol stack. */ virtual int getProtocolNumber() { return SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_RELAY; } /** Initialize this protocol session. */ virtual void init(); /** The main process of the dnp relay. */ void main_proc(prime::ssf::Process*); //! SSF PROCEDURE /** The process for handling each incoming client connection. */ void session_proc(prime::ssf::Process*); //! SSF PROCEDURE /** Sends a request to a specific relay */ //void sendUpdateRequest(); //! SSF PROCEDURE // called by blockingDNPRelaySessionTimer upon timer expiration void timer_callback(); //! SSF PROCEDURE protected: /** This method should not be called; it is provided here to prompt an error message if it's called accidentally. */ virtual int push(ProtocolMessage* msg, ProtocolSession* hi_sess, void* extinfo = 0, size_t extinfo_size = 0); /** This method should not be called; it is provided here to prompt an error message if it's called accidentally. */ virtual int pop(ProtocolMessage* msg, Packet* packet, ProtocolSession* lo_sess, void* extinfo = 0, size_t extinfo_size = 0); /** This method is used by the relay to generate a response to a * request from the control station, */ byte* generateResponse(DNPMessage* dnpRequest, uint32& response_size); /** called by DNP hosts that need to print messages */ void displayDnpMsg(DNPMessage* Dnpmsg); /** This method alters Dnpmsg's app_data and app_data_length * such that it will now carry a message */ void insertData(DNPMessage* Dnpmsg); /** Grabs data from PowerWorld and puts it into the payload */ void insertPWData(DNPMessage* Dnpmsg); Member Data VirtualTime update_interval; ///<Time between sending out requests for line value VirtualTime proc_time; ///<Time it takes to process a request uint16 relay_port; ///< Port number to receive incoming request. char* pw_id; //<Corresponds to the bus ID counterpart in Powerworld char* dummy; uint32 request_size; ///< Size of the request from client (must be consistent). uint32 client_limit; ///< Number of client sessions that can be handled simultaneously. bool show_report; ///< Whether we print out the result or not. // state variables byte* reqbuf; //! SSF STATE BlockingSocketMaster* sm; ///< Point to the socket master. prime::ssf::ssf_semaphore* bound_sem; // Bound the number of client sessions. prime::ssf::ssf_semaphore* next_sem; // Used for continuation. IPADDR relay_ip; // IP address of this server (interface 0). //byte dataPayload[DNP_MESSAGE_SIZE]; byte* dataPayload_p; byte* dataPayload_id; uint32 response_size; DNPMessage* dnpRequest; DNPMessage* dnpResponse; short type; // Is this a line/generator/shunt/load? 4. BlockingDNPStateServerSession Class constructor prime::ssfnet::BlockingDNPStateServerSession(ProtocolGraph* graph); destructor prime::ssfnet::~BlockingDNPStateServerSession(); Member Functions /** Configure the blocking DNP state server test protocol session. */ virtual void config(prime::dml::Configuration *cfg); /** * Return the protocol number. This specifies the relationship * between this protocol and the other protocols defined in the same * protocol stack. */ virtual int getProtocolNumber() { return SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_RELAY; } /** Initialize this protocol session. */ virtual void init(); /** The main process of the dnp state server. */ void main_proc(prime::ssf::Process*); //! SSF PROCEDURE /** Sends a request to a specific state server void sendUpdateRequest(int); //! SSF PROCEDURE */ // called by blockingDNPStateServerSessionTimer upon timer expiration void timer_callback(prime::ssf::Process* p); //! SSF PROCEDURE protected: /** This method should not be called; it is provided here to prompt an error message if it's called accidentally. */ virtual int push(ProtocolMessage* msg, ProtocolSession* hi_sess, void* extinfo = 0, size_t extinfo_size = 0); /** This method should not be called; it is provided here to prompt an error message if it's called accidentally. */ virtual int pop(ProtocolMessage* msg, Packet* packet, ProtocolSession* lo_sess, void* extinfo = 0, size_t extinfo_size = 0); /** This method is used by the state server to generate an update request to send * to PowerWorld. Returns the size of the request in bytes */ short generateUpdateRequest(int); Member Data VirtualTime update_interval; ///<Time between sending out requests for line value uint16 state_server_port; ///< Port number to receive incoming request. uint32 request_size; ///< Size of the request from client (must be consistent) . uint32 client_limit; ///< Number of client sessions that can be handled simultaneously. bool show_report; ///< Whether we print out the result or not. // state variables byte* pw_data_size; //! SSF STATE //<Used to set the size of powerworld data in bytes byte* pw_data; //! SSF STATE //<pointer to the data itself short* request_size_short; //! SSF STATE long* padding; //! SSF STATE short* clientVersion; //! SSF STATE byte* respbuf; //! SSF STATE char* getSCADAMsg; //! SSF STATE bool first; BlockingSocketMaster* sm; ///< Point to the socket master. prime::ssf::ssf_semaphore* bound_sem; // Bound the number of client sessions. prime::ssf::ssf_semaphore* next_sem; // Used for continuation. IPADDR state_server_ip; // IP address of this server (interface 0). int payloadCounter; //Added as last byte of dnp message byte dataPayload[DNP_MESSAGE_SIZE]; byte* dataPayload_p; Nhi* sn_nhi; //Holds the nhi of the super node int sn_port; //Holds the port used to communicate with the super node Nhi* pw_nhi; //Holds the nhi of the super node int pw_sock; //Holds the socket used to communicate with the super node int pw_port; //Holds the port used to communicate with the super node IPADDR sn_ip; // IP address of the super node (interface 0). IPADDR pw_ip; // IP address of the super node (interface 0). typedef typedef typedef typedef typedef std::map<std::string, std::map<std::string, std::map<std::string, std::map<std::string, std::map<std::string, typedef typedef typedef typedef typedef std::pair<std::string, std::pair<std::string, std::pair<std::string, std::pair<std::string, std::pair<std::string, pwtype_generator> genMap; pwtype_load> loadMap; pwtype_shunt> shuntMap; pwtype_line> lineMap; std::string> stringMap; pwtype_generator> genStringPair; pwtype_load> loadStringPair; pwtype_shunt> shuntStringPair; pwtype_line> lineStringPair; std::string> stringPair; //Deprecated? stringMap real_to_nhi_map; //Contains mapping from pw IDs to virtual IDs stringMap real_to_nhi_bus_map; //Contains mapping from pw IDs to virtual IDs for busses loadMap real_to_nhi_load_map; //Contains mapping from pw IDs to virtual IDs for loads genMap real_to_nhi_gen_map; //Contains mapping from pw IDs to virtual IDs for generators lineMap real_to_nhi_line_map; //Contains mapping from pw IDs to virtual IDs for lines shuntMap real_to_nhi_shunt_map; //Contains mapping from pw IDs to virtual IDs for shunts Member Classes: class pwtype_generator{ public: int bus; byte ID; }; class pwtype_load{ public: int bus; byte ID; }; class pwtype_shunt{ public: int bus; byte ID; }; class pwtype_line{ public: int bus_src; int bus_dst; int dst; byte ID; }; Changes To Existing Classes NONE Functions NONE Files Changes To Existing Files Added to ssfnet.cc: SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_OUTSTATION= 239 SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_RELAY= 240 SSFNET_PROTOCOL_TYPE_DNP_BLOCKING_STATE_SERVER = 241 Added to ssfnet/Makefile.common.in: ## os/dnp: ## DNP_HDRFILES = $(SRCDIR)/os/dnp/blocking_dnp_outstation.h $(SRCDIR)/os/dnp/blocking_dnp_relay.h $(SRCDIR)/os/dnp/blocking_dnp_state_server.h $(SRCDIR)/os/dnp/dnp_session.h $(SRCDIR)/os/dnp/dnp_message.h DNP_SRCFILES = $(SRCDIR)/os/dnp/blocking_dnp_outstation.cxx $(SRCDIR)/os/dnp/blocking_dnp_relay.cxx $(SRCDIR)/os/dnp/blocking_dnp_state_server.cxx $(SRCDIR)/os/dnp/dnp_session.cc $(SRCDIR)/os/dnp/dnp_message.cc References 1. http://www.scribd.com/doc/2542661/DNP-v3-0-Guide-A-Protocol-Primer 2. DNP Users Group DNP Primer (www.dnp.org/About/DNP3%20Primer%20Rev%20A.pdf) 3. A Taxonomy of Attacks on the DNP3 Protocol (http://www.springerlink.com/content/k48k4733v0367120/) 4. The DNP Users group at dnp.org. Contact Himanshu or others for login info 5. Wikipedia for a high-level view... but nothing technical TODO List: A. Data Aggregation in Outstation host B. Large network size – in the PW2R.py file, just have to add multiple layers of networks. C. Connect EMS to multiple Data Aggregators