COORDINATED SCIENCE LABORATORY
ENGINEERING AT ILLINOIS
Virtual Time Consistency in Smart Grid Test-­‐beds David M. Nicol Franklin W. Woeltge Professor of ECE Director, InformaFon Trust InsFtute ITI.ILLINOIS.EDU
2 What purpose for a Test-­‐bed? Create a model of a physical system that •  Captures salient features of interest •  Can be observed through controlled presentaFon of inputs or boundary condiFons 3 Test-­‐bed Behavior We want the test-­‐bed to “act like” the system it models •  Present some input Test-­‐bed input component component component component component component 4 Test-­‐bed Behavior We want the test-­‐bed to “act like” the system it models •  Input presented to some test-­‐bed component Test-­‐bed input component component component component component component 5 Test-­‐bed Behavior We want the test-­‐bed to “act like” the system it models •  IniFates some sequence of acFons and inputs/
outputs Test-­‐bed component component component component component component 6 Test-­‐bed Behavior We want the test-­‐bed to “act like” the system it models •  IniFates some sequence of acFons and inputs/
outputs Test-­‐bed component component component component component component 7 Test-­‐bed Behavior We want the test-­‐bed to “act like” the system it models •  IniFates some sequence of acFons and inputs/
outputs Test-­‐bed component component component component component component 8 Test-­‐bed Behavior We want the test-­‐bed to “act like” the system it models •  IniFates some sequence of acFons and inputs/
outputs Test-­‐bed component component component component component component 9 Test-­‐bed Behavior We want the test-­‐bed to “act like” the system it models •  IniFates some sequence of acFons and inputs/
outputs Test-­‐bed component component component component component component 10 Test-­‐bed Behavior We want the test-­‐bed to “act like” the system it models •  Output observed Test-­‐bed component component component component component component Output One wants the components to interact in the same way, with the same input/outputs, as in the field, to get the same output 11 Test-­‐bed Behavior We want the test-­‐bed to “act like” the system it models •  Output observed Test-­‐bed component 1 2 5 3 4 2 component 3 component 6 5 component component 4 7 Output 3 component As in the field, sequencing in the test-­‐bed is governed by real-­‐
Fme delays 12 Hold That Thought 13 Our View of a Smart Grid Test-­‐bed Devices •  Meters, relays, PMUs, data aggregators, adapFve mulF-­‐
channel source, etc. So*ware •  Control staFon, data historian, authenFcaFon servers, etc. Power System Simulators •  Hardware assisted (e.g., RTDS, Opal-­‐RT) •  Soaware only, PowerWorld, GridLab-­‐D, OpenDSS, PSAT, etc. Device Emula5on •  Xen, QEMU, LXC Network Emula5on •  Emulab, CORE, Deter Device/Network Simula5on •  ns-­‐3, S3F, OmNet++, etc. 14 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  The VMM sends them when it gets around to it… VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Best Effort Network Simulator/emulator 15 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  The VMM sends them when it gets around to it… VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Best Effort Network Simulator/emulator 16 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  The VMM sends them when it gets around to it… VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Best Effort Network Simulator/emulator 17 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  The VMM sends them when it gets around to it… VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Best Effort Network Simulator/emulator 18 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  The VMM sends them when it gets around to it… VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Best Effort Network Simulator/emulator 19 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  The VMM sends them when it gets around to it… VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Best Effort Network Simulator/emulator 20 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  The VMM sends them when it gets around to it… VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Best Effort Network Simulator/emulator 21 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  The VMM sends them when it gets around to it… VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Best Effort Network Simulator/emulator 22 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  The VMM sends them when it gets around to it… VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Best Effort Network Simulator/emulator 23 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  But if synchronized in virtual 5me VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Virtual Time Network Simulator/emulator 24 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  But if synchronized in virtual 5me VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Virtual Time Network Simulator/emulator 25 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  But if synchronized in virtual 5me VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Virtual Time Network Simulator/emulator 26 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  But if synchronized in virtual 5me VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Virtual Time Network Simulator/emulator 27 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  But if synchronized in virtual 5me VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Virtual Time Network Simulator/emulator 28 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  But if synchronized in virtual 5me VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Virtual Time Network Simulator/emulator 29 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  But if synchronized in virtual 5me VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Virtual Time Network Simulator/emulator 30 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  But if synchronized in virtual 5me VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Virtual Time Network Simulator/emulator 31 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  But if synchronized in virtual 5me VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Virtual Time Network Simulator/emulator 32 Keeping it all coordinated… Time mahers a lot Normal emulaFon execuFon is best effort Suppose in the modeled system 3 devices all send messages to the same router at the same .me •  But if synchronized in virtual 5me, dispatch is concurrent VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VMM Virtual Time Network Simulator/emulator 33 Release That Thought 34 Test-­‐bed Behavior We want the test-­‐bed to “act like” the system it models •  Output observed Test-­‐bed component 1 2 5 3 4 2 component 3 component 6 5 component component 4 7 Output 3 component To sequence as in the field, we have to sequence with respect to Virtual Time 35 Imagine the PossibiliFes…. With a test-­‐bed embedded in virtual Fme, you can •  model larger systems on smaller test-­‐beds –  More simulaFon, fewer devices •  Mask latencies in test-­‐bed federaFon –  Run xN slower, turns 50ms real delay into a 50/N ms virtual delay –  SynchronizaFon 36 Virtual Time Sequencing Every component acFon needs a 5me-­‐stamp Every component acFon needs a 5me delay Virtual Fme management framework •  Schedules component acFons in virtual Fme •  Manages inter-­‐component input/output 37 Virtual Time Sequencing Every component acFon needs a 5me-­‐stamp Every component acFon needs a 5me delay Virtual Fme management framework •  Schedules component acFons in virtual Fme •  Manages inter-­‐component input/output SomeFmes known as Discrete-­‐Event SimulaFon QuesFons •
•
•
•
How to embed emulaFon in VT? How to embed power system flow simulaFon in VT? How to embed device execuFon in VT? How to coordinate it all? 38 Embedding EmulaFon in Virtual Time Requires •  calls to clock return virtual Fme –  Based on measured execuFon and Fme dilaFon factor •  Scheduling –  Advance virtual machines concurrent w.r.t. virtual Fme Examples •  Versions of Xen just shia off clock bits –  TDF of 2,4,8, etc. –  Ordinary VM scheduling •  Timekeeper scales Fme by TDF –  Advances all LXCs paced by slowest container Network SimulaFon/EmulaFon CoordinaFon Requires synchronizaFon •  Virtual machines and simulators need to advance at the same rate –  Best effort interacFons within that •  Fine grained synchronizaFon –  Ensure that no simulator or VM receives a Fme-­‐stamped communicaFon “in its past” –  Achieved using synchronizaFon protocols from parallel discrete-­‐event simulaFon •  S3F and Timekeeper Power System Flow Simulators and Virtual Time Requirements •  Export state, with virtual Fme-­‐stamps •  Pause/Restart, or scaled release execuFon •  Buffer state Time-­‐stamped state Time-­‐stamped state Device ExecuFon and Virtual Time No “always works” methodologies •  But some important special cases –  PMU GPS Grid state Time-­‐stamped data Device ExecuFon and Virtual Time Possible approaches •  Tinker with the input GPS translate Grid state Time-­‐stamped data Device ExecuFon and Virtual Time Possible approaches •  Tinker with the input •  Tinker with the output GPS Grid state Time-­‐stamped data translate Assembling Pieces of the Puzzle (TimeKeeper) Combined Physical / Virtual Network S3F Examples 46 Example Use of a Smart Grid Testbed DDoS Ahack Using C12.22 Trace Service in AMI C12.22 Trace Service •
•
AmplificaFon –  Increased volume of traffic ReflecFon –  Spoofed source address Components used •  Meter emulaFon •  Meter simulaFon •  Access point simulaFon •  Zigbee wireless simulaFon Might have included power simulaFon but wasn’t needed 47 Experiment •
4x4 blocks, 448 meters •
5 ahackers •
VicFm: the single egress point (meter gateway) •
ZigBee wireless network, 1 Mb/s bandwidth •
Normal traffic: 100-­‐byte packet per 10 second •
Ahacking traffic: 200 Fmes faster, 15-­‐30 hops 48 Experimental Results 49 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter View of power system state seen at
Test-bed Control station
- its view derived from network simulation
Power simulaFon 50 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Tes-­‐tbed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter Control Station polls system devices
for status and state
Power simulaFon 51 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter Control Station polls system devices
for status and state
Power simulaFon 52 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter Control Station polls system devices
for status and state
Power simulaFon 53 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter Control Station polls system devices
for status and state
Power simulaFon 54 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter Control Station polls system devices
for status and state
Power simulaFon 55 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter Control Station polls system devices
for status and state
Power simulaFon 56 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter 1. Holes in FW permit outbound http
connection to site with hidden
malware
Power simulaFon 57 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter 2. HMI monitor infected, maps PCS,
IDs devices
Power simulaFon 58 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter 3. Network map returned to malware
master
Power simulaFon 59 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter 3. Malware master returns infection
program to compromised host
Power simulaFon 60 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter 3. Compromised host infects others,
creates botnet
Power simulaFon 61 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test-­‐bed Power System Control StaFon INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter 4. Attack on relay causes line to open
Flow topology changes
Power simulaFon 62 Example : Ahack on SituaFonal Awareness simulaFon/emulaFon Test=bed Power System Control StaFon X X INTERNET OSI Control StaFon SubstaFon Data Aggregator RTU relay SubstaFon Test-­‐bed SubstaFon Interface SubstaFon AMS SubstaFon Data Converter 5. Simultaneous launch of bandwidth
consumption attack by botnet blinds
control station
Power simulaFon 63 Conclusion Virtual Fme consistency in Smart Grid test-­‐beds •  Allows greater flexibility in what can be studied IntegraFon of device/communicaFon simulaFon + emulaFon well underway IntegraFon of power flow simulaFon with device/
communicaFon simulaFon has been accomplished •  Flow simulator needs special hooks IntegraFon of devices with virtual Fme is a Work in Progress 64