ACM CCS Conference Tutorial
Nov. 2009
Cyber Security
for the Power Grid:
Cyber Security Issues &
Securing Control Systems
Andrew Wright
Cyber Security Solutions
For <Client Name>
CTO, N-Dimension
andrew.wright@n-dimension.com
borrowed from NIST Smart Grid Twiki
Power Grid Communications & Control Systems
Internet
Control Systems
Agenda
• High-Level
– Industrial Control Systems and Cyber Security Issues
– Securing Control Systems
• Detailed
– Security Issues in Industrial Control Systems
– Today’s Threats
– Securing Control Systems
A Control System
Sensor(s) +
Actuator(s) +
Controller(s)
Types of Industrial Control Systems (ICS)
Supervisory Control And Data
Acquisition (SCADA)
Distributed Control
Systems (DCS)
Process Control
Systems (PCS)
Automation
Historical ICS
•
•
•
•
Proprietary
Complete vertical solutions
Customized
Specialized communications
– Wired, fiber, microwave, dialup, serial, etc.
– 100s of different protocols
– Slow; e.g. 1200 baud
• Long service lifetimes: 15–20 years
• Not designed with security in mind
Modern ICS Trends
Internet
Enterprise Network
Workplaces
Firewall
Enterprise
Optimization
Suite
Enterprise
Network
IP
Third Party
Application
Server
Mobile
Operator
Services
Network
Connectivity
Server
Historian
Server
Application
Server
Control
Network
Serial, OPC
or Fieldbus
Redundant
Device Network
Third Party
Controllers,
Servers, etc.
Serial
RS485
Engineering
Workplace
Technology Trends in ICS
• COTS (Commercial-Off-The-Shelf) technologies
–
–
–
–
Operating systems—Windows, WinCE, embedded RTOSes
Applications—Databases, web servers, web browsers, etc.
IT protocols—HTTP, SMTP, FTP, DCOM, XML, SNMP, etc.
Networking equipment—switches, routers, firewalls, etc.
• Connectivity of ICS to enterprise LAN
– Improved business visibility, business process efficiency
– Remote access to control center and field devices
• IP Networking
–
–
–
–
Common in higher level networks, gaining in lower levels
Many legacy protocols wrapped in TCP or UDP
Most new industrial devices have Ethernet ports
Most new ICS architectures are IP-based
New IP-Based Industrial Control Systems
•
•
•
•
•
ODVA (Rockwell)
Profinet
Foundation Fieldbus HSE
Telvent
ABB 800xA
•
•
•
•
•
Honeywell Experion
Emerson DeltaV
Yokogawa VNET/IP
Invensys Infusion
Survalent
• IP to the Control Network or even Device Network
• Not all are fully compatible with “ordinary IP”
Security Risks to Modern ICS
• COTS + IP + connectivity = many security risks
• All of those of Enterprise networks and more
Worms and Viruses
DOS and DDOS impairing availability
Unauthorized access
Unknown access
Unpatched systems
Little or no use of anti-virus
Limited use of host-based firewalls
Improper use of ICS workstations
Unauthorized applications
Unnecessary applications
Open FTP, Telnet, SNMP, HTML ports
Fragile control devices
Network scans by IT staff
Legacy OSes and applications
Inability to limit access
Inability to revoke access
Unexamined system logs
Accidental misconfiguration
Improperly secured devices
Improperly secured wireless
Unencrypted links to remote sites
Passwords sent in clear text
Default passwords
Password management problems
Default OS security configurations
Unpatched routers / switches
When ICS Security Fails
• Loss of production
• Penalties
•
•
•
•
•
Lawsuits
Loss of public trust
Loss of market value
Physical damage
Environmental damage
• Injury
• Loss of life
• USSR pipeline explosion, 1982
• Bellingham pipeline rupture, 1999
• Queensland sewage release, 2000
$$$.$$
• Davis Besse nuclear plant infection, 2003
• Northeast USA blackout, 2003
• Browns Ferry nuclear plant scram, 2006
So How Do We Secure
Industrial Control Systems?
ACM CCS Tutorial
Nov. 2009
There is No Silver Bullet!
No Silver Bullet!
Defense in Depth
• Perimeter Protection
– Firewall, IPS, VPN, AV
– Host IDS, Host AV
– DMZ
• Interior Security
–
–
–
–
–
Firewall, IDS, VPN, AV
Host IDS, Host AV
IEEE P1711 (AGA 12)
NAC
IDS
Scanning
IPS
• Monitoring
• Management
DMZ
VPN
AV
NAC
Intrusion Detection System
Intrusion Prevention System
DeMilitarized Zone
Virtual Private Network (cryptographic)
Anti-Virus (anti-malware)
Network Admission Control
50000 Foot View
Internet
IT Stuff
Enterprise Network
VPN
FW
Proxy AV
IPS
Host
AV
Log Mgmt
Host IPS
Scan
IDS
Control Network
NAC
IT Stuff
Event Mgmt
Host IDS Host AV
Reporting
IPS
FW
Partner
62351 Site
VPN
FW
VPN
IDS
FW
AV
Scan Field Site NAC
P1711
Field Site
Field Site
Security Issues in
Industrial Control Systems
ACM CCS Tutorial
Nov. 2009
Availability, Integrity and Confidentiality
• Enterprise networks require C-I-A
– Confidentiality of intellectual property matters most
• ICS requires A-I-C
– Availability and integrity of control matters most
– control data has low entropy—little need for confidentiality
– Many ICS vendors provide six 9’s of availability
• Ensuring availability is hard
– Cryptography does not help (directly)
– DOS protection, rate limiting, resource management, QoS,
redundancy, robust hardware with high MTBF
• Security must not reduce availability!
DoS and DDoS Attacks
• Denial of Service (DoS) attack overwhelms a system
with too many packets/requests
– Exhausts TCP stack or application resources
– Defenses include connection limits in firewall
• Distributed Denial of Service (DDoS) attack
coordinates a botnet to overwhelm a target system
– No single point of attack
– Requires sophisticated, coordinated defenses
– Weapon of choice for hackers, hacktivists, cyber-extortionists
• DoS, DDoS particularly effective when Availability is
critical, i.e. against ICS
Fragile ICS Devices
• Many IP stack implementations are fragile
– Some devices lockup on ping sweep or NMAP scan
– Numerous incidents of ICS shut down by uninformed IT staff
running a well-intentioned vulnerability scan
• Modern ICS devices are much more complex
– Some IEDs include web server for configuration and status
– More lines of code leads to more bugs
– Modern IEDs require patching just like servers
Unpatched Systems
• Many ICS systems are not patched current
– Particularly Windows servers
– No patches available for older versions of windows
• OS and application patches can break ICS
– OS patches are tested for enterprise apps
• Uncertified patches can invalidate warranty
• Patching often requires system reboot
• Before installation of a patch:
–
–
–
–
Vendor certification—typically one week
Lab testing by operator
Staged deployment on less critical systems first
Avoid interrupting any critical process phases
Limited use of Host Anti-Virus
• AV operations can cause significant system
disruption at inopportune times
– 3am is no better than any other time for a full disk scan on a
system that operates 24x7x365
• ICS vendors only beginning to support anti-virus
– Anti-virus is only as good as the signature set
– Signatures may require testing just like patches
• AV may be losing ground in enterprise deployments
– impact on hosts, endpoint security not getting better
– virus writers have learned to test against dominant AV
• application whitelisting can be a good alternative
– enumerate goodness rather than badness
Poor Authentication and Authorization
• Machine-to-machine comms involve no “user”
• Many ICS have poor authentication mechanisms
and very limited authorization mechanisms
• Many protocols use cleartext passwords
• Many ICS devices lack crypto support
• Sometimes passwords left at vendor default
• Device passwords are hard to manage appropriately
– Often one password is shared amongst all devices
and all users and seldom if ever changed
– This is happening AGAIN in Smart Meter deployments!
Poor Audit and Logging
• Many ICS have poor or non-existent support for
logging security-related actions
– Attempted or successful intrusions may go unnoticed
• Where IDS logs are kept, they are often not reviewed
• Various regulatory requirements are driving some
change in this area
– NERC—North American Electric Reliability Corporation
– FERC—Federal Energy Regulatory Commission
– Sarbanes Oxley and PCAOB (Public Company Accounting
Oversight Board)
– FISMA—Federal Information Security Management Act
Unmanned Field Sites
•
•
•
•
Many unmanned field sites
Many with dialup access
Some with high-speed connectivity to control center
Most with poor authentication and authorization
backdoor to the
control center!
Legacy Equipment
• Much legacy equipment
• Usually impossible to update to add security features
• Difficult to protect legacy communications
– but see IEEE P1711 for serial encryption
• Password protection is weak
• Little or no audit and logging
Unauthorized Applications
• Unauthorized apps installed on ICS systems can
interfere with ICS operation
• Many types of unauthorized apps have been found
during security audits
–
–
–
–
–
Instant messaging
P2P file sharing
DVD and MPEG video players
Games, including Internet-based
Web browsers
Inappropriate Use of ICS Desktops
• Web browsing from HMI can infect ICS
–
–
–
–
Browser vulnerabilities
Downloads
Cross-site scripting
Spyware
• Email to/from control servers can infect ICS
– Sendmail and outlook vulnerabilities
• Disk storage exhaustion can crash OS
– Storage of music, videos
Little or No Cyber Security Monitoring
• internal monitoring is essential to detect low profile
compromises
–
–
–
–
IDS
port scanning
vulnerability scanning
system audit
• without internal monitoring don’t know whether
systems have been compromised
Requirement for 3rd Party Access
• Firmware updates and PLC, IED programming are
sometimes done by vendor
– Many ICS have open maintenance ports
– Infected vendor laptops can bring down ICS
• Partners may require continuous status information
– Partner access is often poorly secured
– Partner channels can serve as backdoors
• 3rd parties may include:
– ISO, transmission provider or grid neighbor,
equipment vendor, emissions monitoring service or agency,
water level monitoring agency, vibration monitoring service,
etc.
People Issues
• ICS network often managed by “Control Systems
Department”, distinct from “IT Department” running
enterprise network
– ICS personnel are not IT or networking experts
– IT personnel are not ICS experts
• Majority of control systems workforce is
older and nearing retirement
– Few young people entering this field
– Few academic programs
Harsh Environments
• Temperature
• Vibration
• Dust
• Humidity
• Electrical
Transients
Attack Vectors into Control Systems
Includes Infected
Laptops and Is Growing
Source: 2003–2006 data from Eric Byres, BCIT
Security Assessments on ICS
• Various groups perform security assessments and
penetration tests on ICS (generally under NDA)
–
–
–
–
Idaho National Labs
Sandia National Labs
N-Dimension Solutions
Other private organizations
• Vulnerability assessments always uncover problems
• For penetration tests, we always get in
– Not a question of “if”, but “how long”
Other Issues
•
•
•
•
•
Unusual physical topologies
Many special purpose, limited function devices
Static network configurations
Multicast
Long service lifetimes
For More Information ...
• See Smart Grid Cyber Security Strategy and
Requirements, NISTIR 7628, www.nist.gov/smartgrid
– particularly Appendices C and D
Today’s Threats
ACM CCS Tutorial
Nov. 2009
Intense Media Visibility on the Cyber Security Issue
Smart Grid Security Frenzy: Cyber War Games,
Worms and Spies in Smart Grid (June 09)
President Obama: securing the electric
infrastructure is a national security priority (June 09)
Hiroshima, 2.0 – Cyberspying of the
US Electric Grid (April 09)
Cyberspies penetrate electrical grid (April 09)
'Smart Grid' vulnerable to hackers (March 09)
CIA: Hackers Have Attacked Foreign Utilities
(Jan 2008)
earth2tech.com
Limited Information About Incidents
• Little information sharing about actual attacks
– BCIT incident database has about 30 incidents per year vs.
100s of thousands of incidents per year in CERT database
– Few cyber attacks on ICS for which details are public
• Little information sharing about actual vulnerabilities
– some are not easily or rapidly fixed
– assessments are done under NDA
• Difficult to estimate risk
– Difficult to demonstrate ROI for security spending
• But… lots of data about significant financial losses in
enterprise and e-commerce
– Why would control systems be immune?
Accidents Happen ...
Attacks Can Cause Similar Results
INL National Lab Aurora Demonstration, March 2007
Cyber Security Regulatory Requirements
FERC releases Smart Grid Policy - cyber security
mandatory for Utility rate recovery (July 09)
Regulators provide Smart Grid Stimulus Funding
criteria - cyber security is mandatory (June 09)
Strengthened Cyber Security Standards Approved for
North American Utilities (May 09)
Ontario Green Energy Act Drives Smart Grid With Security
(May 09)
NIST developing interoperability and security standards
for Smart Grid
AMI-SEC working group developed security
requirements for AMI
AMI-SEC Task Force
Securing
Control Systems
ACM CCS Tutorial
Nov. 2009
Adversaries
•
•
•
•
•
•
•
•
•
Script kiddies
Hackers
Organized crime
Disgruntled insiders
Competitors
Terrorists
Hactivists
Eco-terrorists
Nation states
How an Attack Proceeds—Step #1
Engineering
Workstation Management
Console HMI
IED
IED
Modem Pool
Data
Historian
RTU
FEP
Email
Server
Control
System
Network
Web
Server
enterprise
Firewall
Enterprise
Network
ICS
Firewall
Internet
Web
Server
Business
Workstation
Domain Name
Server (DNS)
Database Server
Attacker
How an Attack Proceeds—Step #2
Engineering
Workstation Management
Console HMI
IED
IED
Modem Pool
Data
Historian
RTU
FEP
Email
Server
Control
System
Network
Web
Server
enterprise
Firewall
Enterprise
Network
ICS
Firewall
Internet
Web
Server
Business
Workstation
Domain Name
Server (DNS)
Database Server
Attacker
How an Attack Proceeds—Step #3
Engineering
Workstation Management
Console HMI
IED
IED
Modem Pool
Data
Historian
RTU
FEP
Email
Server
Control
System
Network
Web
Server
enterprise
Firewall
Enterprise
Network
ICS
Firewall
Internet
Web
Server
Business
Workstation
Domain Name
Server (DNS)
Database Server
Attacker
How an Attack Proceeds—Step #4
Engineering
Workstation Management
Console HMI
IED
IED
Modem Pool
Data
Historian
RTU
FEP
Email
Server
Control
System
Network
Vendor Web
Server
Web
Server
enterprise
Firewall
Enterprise
Network
ICS
Firewall
Internet
Web
Server
Business
Workstation
Domain Name
Server (DNS)
Database Server
Attacker
How an Attack Proceeds—Step #5
Engineering
Workstation Management
Console HMI
IED
IED
Modem Pool
Data
Historian
RTU
FEP
Email
Server
Control
System
Network
Vendor Web
Server
Web
Server
enterprise
Firewall
Enterprise
Network
ICS
Firewall
Internet
Web
Server
Business
Workstation
Domain Name
Server (DNS)
Database Server
Attacker
How an Attack Proceeds—Step #6
Engineering
Workstation Management
Console HMI
IED
IED
Modem Pool
Data
Historian
RTU
FEP
Email
Server
Control
System
Network
Web
Server
enterprise
Firewall
Enterprise
Network
ICS
Firewall
Internet
Web
Server
Business
Workstation
Domain Name
Server (DNS)
Database Server
Attacker
How an Attack Proceeds—Step #7
Engineering
Workstation Management
Console HMI
IED
IED
Modem Pool
Data
Historian
RTU
FEP
Email
Server
Control
System
Network
Web
Server
enterprise
Firewall
Enterprise
Network
ICS
Firewall
Internet
Web
Server
Business
Workstation
Domain Name
Server (DNS)
Database Server
Attacker
Defending ICS
• Separate control network from enterprise network
– Harden connection to enterprise network
– Protect all points of entry with strong authentication
– Make reconnaissance difficult from outside
• Harden interior of control network
– Make reconnaissance difficult from inside
– Avoid single points of vulnerability
– Frustrate opportunities to expand a compromise
• Harden field sites and partner connections
– mutual distrust
• Monitor both perimeter and inside events
• Periodically scan for changes in security posture
50000 Foot View
Internet
IT Stuff
Enterprise Network
VPN
FW
Proxy AV
IPS
Host
AV
Log Mgmt
Host IPS
Scan
IDS
Control Network
NAC
IT Stuff
Event Mgmt
Host IDS Host AV
Reporting
IPS
FW
Partner
62351 Site
VPN
FW
VPN
IDS
FW
AV
Scan Field Site NAC
P1711
Field Site
Field Site
Logical Overlay on SP99 / Purdue Model of Control
Enterprise Network
Level 5
Level 4
Email, Intranet, etc.
Site Business Planning and Logistics Network
Terminal
Services
Patch
Mgmt
Enterprise
Zone
AV
Server
DMZ
Historian
(Mirror)
Web Services
Operations
Level 3
Production
Control
Optimizing
Control
Level 2
Supervisory
Control
HMI
Level 1
Level 0
Batch
Control
Discrete
Control
Historian
Application
Server
Site Operations
and Control
Engineering
Station
Supervisory
Control
Continuous
Control
HMI
Hybrid
Control
Area
Supervisory
Control
Basic
Control
Process
Control
Zone
Logical Architecture
• Enterprise Zone contains typical business systems
– Email, web, office apps, etc.
• DMZ provides business connectivity
– Contains only non-critical systems that need access to both
Control and Enterprise Zones
– Enforces separation between Enterprise and Control Zones
– Consists of multiple functional sub-zones
• Separated by Firewall, IPS, Anti-Virus, etc.
• Control Zone demarcates critical control systems
– Consists of multiple functional sub-zones
• Internally protected by Firewall, IDS, Anti-Virus, etc.
How NOT to connect Control / Enterprise
•
•
•
•
•
Dual-homed server
Dual-homed server with Host IPS / AV
Router with packet filter ACLs
Two-port Firewall
Router + Firewall combination
• See NISCC Good Practice Guide on Firewall Deployment for
SCADA and Process Control Networks, NISCC and BCIT, Feb
2005
DMZ—Logical View
Emergency
Disconnect
AV
Terminal
Services
Patch
Mgmt
Proxy
AV
Proxy
VPN
IDS
FW
IPS
Multiple
Functional
Sub-Zones
Host AV
Host IPS
Historian
Mirror
DMZ
Scan
IDS
Web
Services
Operations
Application
Server
No Direct
Traffic
Emergency
Disconnect
DMZ Design Principles
•
•
•
•
•
•
•
•
•
•
•
•
•
DMZ contains non-critical systems
Multiple functional security sub-zones
Traffic between sub-zones undergoes firewall (& IPS or IDS)
DMZ is only path in/out of Control Zone
Default deny for all firewall interfaces
No direct traffic across DMZ
No control traffic to outside
Limited outbound traffic from Control Zone
Very limited inbound traffic to Control Zone
No common ports between outside & inside
Emergency disconnect at inside or outside
No network management from outside
Cryptographic VPN and Firewall to all 3rd party connections
DMZ Implementation (1)
Enterprise
LAN
NAT
DMZ LAN 2
DMZ LAN 3
Security
Appliance
With
Multiple
Ports
Routing
FW
IPS
DMZ LAN 4
Anti-Virus
Proxy
Host IPS / Anti-virus
DMZ/Control
Interconnect
WAN/LAN
DMZ Implementation (2)
Enterprise
LAN
VLAN-capable
L2 switch
NAT
DMZ VLAN 2
dot1q
trunk
DMZ VLAN 3
DMZ VLAN 4
NOT L3!
Security
Appliance
Routing
FW
IPS
VLAN
Anti-Virus
Proxy
Host IPS / Anti-virus
DMZ/Control
Interconnect
WAN/LAN
DMZ Implementation
• Sub-zones implemented by physical LANs or VLANs
– Physical LANs require multi-port Security Appliance
– VLANs require:
• VLAN-capable Security Appliance and Switch
• anti-VLAN hopping protections on switch and FW
• NO L3 (routing) on switch
• FW implements policy between
– DMZ LANs, Enterprise Zone, Control Zone
• Anti-virus proxy controls outbound HTTP and/or FTP
access to enterprise or Internet resources
• Host IPS and/or Host Anti-virus protects DMZ servers
Remote Access
Enterprise
LAN
Terminal
Services
AAA
Server
Remote
Access
VPN
Certificate
Authority
DMZ
Remote Access Pool
DMZ/Control
Interconnect
WAN/LAN
Remote Access
• Security Appliance terminates Host-to-site VPN into
remote access pool
– IPSEC VPN, SSL VPN, PPTP VPN
• Authenticates user via:
– AAA server, LDAP, Active Directory, etc.
– Can enforce use of multi-factor hardware token
• Time-varying password tokens for vendor access
• Clients use VNC, Citrix, or Remote Desktop (RDP) to
connect to Terminal Server
• Then VNC, Citrix, RDP, or Control System Apps to
Control System Servers
Control Zone—Logical View
DMZ
Level 3
Level 2
Level 1
Level 0
Production
Control
Optimizing
Control
Supervisory
Control
HMI
Batch
Control
Discrete
Control
Historian
Engineering
Station
Supervisory
Control
Continuous
Control
Site Operations
and Control
HMI
Hybrid
Control
Area
Supervisory
Control
Basic
Control
Process
Control
Zone
Control Zone Design Principles
•
•
•
•
Multiple functional security sub-zones
Firewall and IDS between sub-zones
Minimal number of connections to DMZ
Control Zone independent of DMZ, Enterprise
–
–
–
–
Separate Security Appliance from DMZ
Separate Time Server
Separate AAA
Allows emergency disconnect from DMZ
• Cryptographic VPN and Firewall to all offsite IP connections
(Field Site or Partner)
• IEEE P1711 for all offsite serial ICS connections
• Host IDS, Host AV, or app whitelisting where feasible
• Management only from management zone
Control Zone Implementation—Hierarchical
• Fast routing between
VLANs via L3 switch
• ACLs between VLANs
but no Stateful Firewall
DMZ/Control Interconnect WAN/LAN
FW
Level 3
IDS
FW
L3
L3
SPAN
Scan
Level 2
L2
Gigabit
dot1q Trunks
Control
Zone
L2
QoS, Shaping, Policing
Port Security
10/100
Level 1
Host IDS
Host AV
Control Zone Implementation—Ring
• Ring reduces wiring for linear
sites like power dams
• but spanning tree can have
problems with large rings
DMZ/Control Interconnect WAN/LAN
FW
Level 3
IDS
FW
L3
L3
SPAN
Scan
Level 2
L2
Gigabit
dot1q Trunks
Control
Zone
L2
QoS, Shaping, Policing
Port Security
10/100
Level 1
Host IDS
Host AV
Perimeter Protection in Utilities
Firewall
IDS/IPS
Client VPN
DMZ
Proxy
Network AV
Host IDS/IPS
NAC
Site-to-site VPN
Interior Protection in Utilities
IDS
Port Scan
Vuln Scan
Firewall
NAC
Firewall
SCADA VPN
Port Scan
IDS
SCADA VPN
Monitor, Log, Analyze, Report
Log
Analyze
Report
Compliance
Managed
Security
Beyond Network Security
•
Planning, processes, procedures, physical security, etc. are also
important
•
NERC CIP Regulatory Requirements provide reasonably good
guidance in this area:
•
•
•
•
•
•
•
•
•
CIP-001: Sabotage Reporting
CIP-002: Critical Cyber Asset Identification
CIP-003: Security Management Controls
CIP-004: Personnel & Training
CIP-005: Electronic Security Perimeters
CIP-006: Physical Security
CIP-007: Systems Security Management
CIP-008: Incident Reporting & Response Planning
CIP-009: Recovery Plans for Critical Cyber Assets
See www.nerc.com -> Standards -> Reliability Standards -> CIP
Summary
• Today’s ICS are mix of
modern and legacy
– vulnerabilities due to both
lack of security design in
legacy and security issues
in newer equipment
• Defense in depth is essential
– both perimeter (DMZ) and
interior security are crucial
• Regulation and government
action is driving change
• Smart Grid must be
designed with strong security
Thanks!
andrew.wright@n-dimension.com
ACM CCS Tutorial
Nov. 2009
Standards Efforts
•
•
•
•
•
•
•
•
NERC CIPs
NIST Smart Grid Interoperability Standards Project
NIST SP800-82
NIST SP800-53
NIST PCSRF Protection Profiles
AMI-SEC
ISA SP99
ODVA
• IEEE P1711 (AGA 12) -- serial SCADA encryption
A Few References
• www.nist.gov/smartgrid
• Securing Your SCADA and Industrial Control
Systems, Version 1.0, DHS, ISBN 0-16-075115-8
• Guide to SCADA and Industrial Control System
Security, NIST SP800-82
• ISA99 Industrial Automation and Control Systems
Security,
www.isa.org/MSTemplate.cfm?MicrositeID=988&Co
mmitteeID=6821
• AGA 12/IEEE P1689 SCADA Encryption Standard,
scadasafe.sf.net