Narrowband & Distribution
ABB Ethernet Solutions
AFS650 / AFS670 / AFR677 Family
© ABB Group
October 2, 2014 | Slide 1
AFS6xx Family – ABB Ethernet Solutions
Table of contents
Time
Day 1
09.00 –
12.45 am
-Introduction
-Networking Basics
01.30 –
04.00 pm
Lunch
-Equipment Introduction
L2 Overview
-User-SW “AFS Finder” PC- Installation
-Local Management Access
-Basic Programming
Day 2
-Layer 2 Network
VLAN Foundation
Spanning Tree Protocol
Lunch
-Layer 2 Programming
VLAN Programming
Spanning Tree Programming
Day 1
Networking Basics
OSI Stack & TCP/IP Architecture
© ABB Group
October 2, 2014 | Slide 3
Principles of the Internet


Edge vs. core (end-systems vs. routers)

Dumb network

Intelligence at the end-systems
Different communication paradigms

Connection oriented vs. connection less

Packet vs. circuit switching

Layered System

Network of collaborating networks
The network edge
end
systems (hosts):

run application programs

e.g., WWW, email

at “edge of network”
client/server
model:

client host requests, receives
service from server

e.g., WWW client (browser)/
server; email client/server
peer-peer

model:
host interaction symmetric
e.g.: teleconferencing
Network edge: connection-oriented service
Goal:
data transfer between end
sys.
setup (prepare
for) data transfer ahead of time
handshaking:
Hello,
hello back human protocol
set
up “state” in two
communicating hosts
TCP
- Transmission Control
Protocol
Internet’s
service
connection-oriented
TCP service
[RFC 793]
in-order byte-stream
data transfer
reliable,
loss:
acknowledgements and
retransmissions
flow
control:
sender
won’t overwhelm receiver
congestion
senders
control:
“slow down sending rate”
when network congested
Network edge: connectionless service
Goal: data transfer between end systems

UDP - User Datagram Protocol [RFC 768]: Internet’s connectionless service

unreliable data transfer

no flow control

no congestion control
The IP Model
Application layer
SMTP HTTP
FTP
Telnet
TCP
UDP
DNS
RTP
PPP
ATM
Optics
Video
Transport layer
Network layer
IP
Ethernet
Audio
ADSL
Satellite
3G
Physical and Data link layer
The OSI Model
7
Application
6
Presentation
5
Session
4
Transport
3
Network
2
Data Link
1
Physical
Upper Layers
Application oriented
“End-to-End”-Layers
Lower Layers
Network oriented
“Hop-by-hop” layers
OSI Model and the Internet

Internet protocols are not directly based on the OSI model

However, we do often use the OSI numbering system. You should at least
remember these:

Layer 7: Application

Layer 4: Transport (e.g. TCP, UDP)

Layer 3: Network (IP)

Layer 2: Data link

Layer 1: Physical
Layer Interaction:
TCP/IP Model
End
to
end
Hop
by
hop
Application
Application
TCP or UDP
TCP or UDP
IP
IP
Link
Physical
Host
Link
IP
Link
Link
IP
Link
Physical
Router
Link
Physical
Router
Host
End-to-end layers

Upper layers are “end-to-end”

Applications at the two ends behave as if they can talk
directly to each other

They do not concern themselves with the details of what
happens in between
Hop-by-hop layers

At the lower layers, devices share access to the same
physical medium

Devices communicate directly with each other

The network layer (IP) has some knowledge of how many
small networks are interconnected to make a large
internet

Information moves one hop at a time, getting closer to the
destination at each hop
Layer Interaction:
TCP/IP Model
Application
Application
TCP or UDP
TCP or UDP
IP
IP
Link
Physical
Host
Link
IP
Link
Link
IP
Link
Physical
Router
Link
Physical
Router
Host
Layer Interaction:
The Application Layer
Applications behave as if they can talk to each other,
but in reality the application at each side talks to the
TCP or UDP service below it.
Application
Application
TCP application
or UDP
or UDP
The
layer doesn't care about whatTCP
happens
atIPthe lower layers,
IP provided the
IP transport layer
IP
carries the application's data safely from end to end.
Link
Physical
Host
Link
Link
Link
Link
Physical
Router
Link
Physical
Router
Host
Layer Interaction:
The Transport Layer
The transport layer instances at the two ends act as if
they are talking to each other, but in reality they are
each talking to the IP layer below it. The transport
layer doesn't care about what the application layer is
Application
Application
doing above it.
TCP or UDP
TCP or UDP
IP
IP the
The IP
transport layerIPdoesn't care what
happens in
IPLink
layer or below,
long asLink
the IPLink
layer can move
Link as
Link
Link
datagrams from one side to the other.
Physical
Host
Physical
Router
Physical
Router
Host
Layer Interaction:
The Network Layer (IP)
The IP layer has to know a lot about the topology of
the network (which host is connected to which router,
which routers are connected to each other),
but it
Application
Application
doesn't care about what happens at the upper layers.
TCP or UDP
IP
TCP or UDP
IP
IP
IP
LinkIP layer works
Link Link
Link hop byLink
The
forwardsLink
messages
hop
from one side
to the other side.
Physical
Physical
Physical
Host
Router
Router
Host
Layer Interaction:
Link and Physical Layers
The link layer doesn't care what happens above it, but
it is very closely tied to the physical layer below
it.
Application
Application
All or
links
have
no
TCP
UDPare independent of each other, andTCP
or UDP
way of communicating with each other.
IP
IP
Link
Physical
Host
Link
IP
Link
Link
Link
Physical
Router
IP
Link
Physical
Router
Host
Layering: physical communication
data
application
transport
network
link
physical
application
transport
network
link
physical
network
link
physical
application
transport
network
link
physical
data
application
transport
network
link
physical
Frame, Datagram, Segment, Packet


Different names for packets at different layers

Ethernet (link layer) frame

IP (network layer) datagram

TCP (transport layer) segment
Terminology is not strictly followed

we often just use the term “packet” at any layer
Encapsulation & Decapsulation

Lower layers add headers (and sometimes trailers) to data from higher
layers
Application
Transport
Network
Network
Data Link
Data Link
Data
Header Transport Layer Data
Header
Network Layer Data
Header Header
Data
Header
Link Layer Data
Header Header Header
Data
Trailer
Trailer
Layer 2 - Ethernet frame
Preamble
Dest


Type
Data
2 bytes
46 to 1500
bytes
Destination and source are 48-bit MAC addresses (e.g.,
00:26:4a:18:f6:aa)
6 bytes

Source
6 bytes
CRC
4 bytes
Type 0x0800 means that the “data” portion of the Ethernet frame contains an
IPv4 datagram. Type 0x0806 for ARP. Type 0x86DD for IPv6.
“Data” part of layer 2 frame contains a layer 3 datagram.
Layer 3 - IPv4 datagram
Version
IHL
Differentiated Services
Identification
Time to Live
Total Length
Flags
Protocol
Fragment Offset
Header Checksum
Source Address (32-bit IPv4 address)
Destination Address (32-bit IPv4 address)
Options
Padding
Data (contains layer 4 segment)

Version = 4
If no options, IHL = 5
Source and Destination
are 32-bit IPv4
addresses

Protocol = 6 means data
portion contains a TCP
segment. Protocol = 17
means UDP.
Layer 4 - TCP segment
Source Port
Destination Port
Sequence Number
Acknowledgement Number
Data
Offset
Reserved
UAE R S F
RCO S Y I
GKL TNN
Checksum
Window
Urgent Pointer
Options
Padding
Data (contains application data)

Source and Destination are 16-bit TCP port numbers (IP addresses are
implied by the IP header)

If no options, Data Offset = 5 (which means 20 octets)
Day 1
Networking Basics
IP Addressing
Purpose of an IPv4 address

Unique Identification of:


Source

So the recipient knows where the message is from

Sometimes used for security or policy-based
filtering of data
Destination


So the networks know where to send the data
Network Independent Format

IP over anything
Purpose of an IP Address

Identifies a machine’s connection to a network

Physically moving a machine from one network to
another requires changing the IP address

Unique; assigned in a hierarchical fashion

IANA (Internet Assigned Number Authority)

IANA to RIRs (AfriNIC, ARIN, RIPE, APNIC, LACNIC)

RIR to ISPs and large organisations

ISP or company IT department to end users

IPv4 uses unique 32-bit addresses

IPv6 used similar concepts but 128-bit addresses
Basic Structure of an IPv4 Address


32 bit number (4 octet number):
(e.g. 133.27.162.125)
Decimal Representation:
133

27
162
125
Binary Representation:
10000101 00011011 10100010 01111101

Hexadecimal Representation:
85
1B
A2
7D
Addressing in Internetworks

The problem we have
 More than one physical network
 Different Locations
 Larger number of hosts
 Need a way of numbering them all

We use a structured numbering system

Hosts that are connected to the same physical network
have “similar” IP addresses

Often more then one level of structure; e.g. physical
networks in the same organisation use “similar” IP
addresses
Network part and Host part

Remember IPv4 address is 32 bits

Divide it into a “network part” and “host part”


“network part” of the address identifies which network in
the internetwork (e.g. the Internet)
“host part” identifies host on that network

Hosts or routers connected to the same link-layer
network will have IP addresses with the same network
part, but different host part.

Host part contains enough bits to address all hosts on
the subnet; e.g. 8 bits allows 256 addresses
Dividing an address

Hierarchical Division in IP Address:

Network Part (or Prefix) – high order bits (left)


describes which physical network
Host Part – low order bits (right)

describes which host on that network
Network Part

Host Part
Boundary can be anywhere

choose the boundary according to number of hosts

very often NOT a multiple of 8 bits
Network Masks


“Network Masks” help define which bits are used to describe the
Network Part and which for the Host Part
Different Representations:

decimal dot notation: 255.255.224.0

binary: 11111111 11111111 11100000 00000000

hexadecimal: 0xFFFFE000

number of network bits: /19


count the 1's in the binary representation
Above examples all mean the same: 19 bits for the Network Part
and 13 bits for the Host Part
Example Prefixes

137.158.128.0/17
1111 1111
1000 1001

198.134.0.0/16
1111 1111
1100 0110

(netmask 255.255.128.0)
1111 1111 1 000 0000 0000 0000
1001 1110 1 000 0000 0000 0000
(netmask 255.255.0.0)
1111 1111
1000 0110
0000 0000
0000 0000
0000 0000
0000 0000
205.37.193.128/26
(netmask 255.255.255.192)
1111 1111 1111 1111 1111 1111 11 00 0000
1100 1101 0010 0101 1100 0001 10 00 0000
Special Addresses


All 0’s in host part: Represents Network

e.g. 193.0.0.0/24

e.g. 138.37.64.0/18

e.g. 196.200.223.96/28
All 1’s in host part:
Broadcast

e.g. 193.0.0.255
(prefix 193.0.0.0/24)

e.g. 138.37.127.255 (prefix 138.37.64.0/18)

e.g. 196.200.223.111 (prefix 196.200.223.96/28)

127.0.0.0/8: Loopback address (127.0.0.1)

0.0.0.0: Various special purposes
Exercise


Verify that the previous examples are all broadcast
addresses:

193.0.0.255
(prefix 193.0.0.0/24)

138.37.127.255 (prefix 138.37.64.0/18)

196.200.223.111 (prefix 196.200.223.96/28)
Do this by finding the boundary between network part and
host part, and checking that the host part (if written in
binary) contains all 1's.
Day 1
Networking Basics
ARP
Encapsulation Reminder

Lower layers add headers (and sometimes trailers) to data
from higher layers
Application
Transport
Network
Network
Data Link
Data Link
Data
Header Transport Layer Data
Header
Network Layer Data
Header Header
Data
Header
Link Layer Data
Header Header Header
Data
Trailer
Trailer
Ethernet Reminder

Ethernet is a broadcast medium

Structure of Ethernet frame:
Preamble
Dest
Source
Type
Data

Entire IP packet makes data part of Ethernet frame

Delivery mechanism (CSMA/CD)

back off and try again when collision is detected
CRC
Ethernet/IP Address Resolution



Internet Address

Unique worldwide (excepting private nets)

Independent of Physical Network technology
Ethernet Address

Unique worldwide (excepting errors)

Ethernet Only
Need to map from higher layer to lower
(i.e. IP to Ethernet, using ARP)
Address Resolution Protocol

ARP is only used in IPv4

ND replaces ARP in IPv6

Check ARP cache for matching IP address

If not found, broadcast packet with IP address to every host
on Ethernet

“Owner” of the IP address responds

Response cached in ARP table for future use

Old cache entries removed by timeout
Types of ARP Messages

ARP request


Who is IP addr X.X.X.X tell IP addr Y.Y.Y.Y
ARP reply

IP addr X.X.X.X is Ethernet Address hh:hh:hh:hh:hh:hh
ARP Procedure
1. ARP Cache is checked
5. ARP Entry is added
2. ARP Request is Sent using broadcast
4. ARP Reply is sent unicast
3. ARP Entry is added
ARP Table
IP Address
Hardware Address
Age (Sec)
192.168.0.2
08-00-20-08-70-54
3
192.168.0.65
05-02-20-08-88-33
120
192.168.0.34
07-01-20-08-73-22
43
Day 1
AFS650, AFS670 Family
Drivers & Typical Applications
Packet Switched Networks Drivers
Additional applications trend towards Ethernet

Today's SCADA communication uses increasingly the IEC 608705-104 (IEC 104)

Utility Internal Voice communication for operational purpose is
moving towards Voice over IP service (VoIP)

Video over IP: IT infrastructure can be used for Video surveillance
(CCTV)

In-Plant applications

Intrusion Detection System

Fire Detection System

Access Control System

ICCP Links between Control Centres

Substation Automation IEC61850
Packet Switched Networks Applications
Distribution Management System
Requires:

FE/GbE backbone for

SCADA communication
Voice over IP
CCTV (video)

Electrical ports required

Depending on design Switches or
Routers required

Fast redundancy required

Management System for supervision
required
Packet Switched Networks Applications
In-plant communication
Requires:

FE/GbE backbone & GbE for server
connections

Optical & electrical ports

Depending on design Switches or
Routers required

Power over Ethernet (PoE)

Port Authentication (802.1x) for security
on outdoor cameras

SNTP (Simple Network Time Protocol)
for time distribution
Packet Switched Networks Applications
IEC61850 Substation Automation
Requires:
IED = Intelligent Electronic Devices

Utility hardened switches supporting
IEC61850 (Station & Process bus)

High number of optical ports

Pure Layer 2 Ethernet switches

Fast redundancy protocol needed

QoS and VLAN used to provide
required performance

IEEE 1613 compatibility for substation
EMI/EMC
Day 1
AFS650, AFS670 Family
Feature Overview
AFS Family
Overview
AFS650/655
ABB FOX Switch Family
Utility Grade Switches
AFS670/675
AFS677/AFR677

IEC 61850 approved

IEEE 1613 compliant
EMI substations requirement

Extended temperature range
0°C up to 60°C
option -40°C to 85°C (70°C)
option conformal coating (high humidity)

Fanless design

Redundant power supply

AC power supply (90 .. 265 VAC)
and/or
DC power supply (18 .. 60, 77 .. 300 VDC)
AFS670/ AFS675

19” managed Switch of utility grade

Metal housing, Ports on front or on rear

High port density, modular concept

Up to 28 FE ports (electrical, optical or SFP cages)

Huge variety on optical ports: SFP cages, SC, ST,.., MM and
SM

SFP’s in various versions available (MM, SM, single fibre)

Up to 4 GbE ports (optical, electrical or combo ports)

Up to 4 Ports PoE

24/36/48 VDC or 60/120/250 VDC, 110/230VAC Power supply,
Redundant Power supply possible

CRA (Configuration Recovery Adapter) port for easy exchange of
product
AFS650/ AFS655
Technical Features

DIN Rail mounted managed Ethernet Switch of utility grade
(wall mounting possible)

Metal housing

Up to 10 FE ports (electrical, optical or SFP cages)

Up to 3 GbE ports (electrical, SFP or Combo ports)

Multimode and Single Mode versions available

Huge variety on optical ports: SFP cages, SC, ST, …

SFP’s in various versions available (MM, SM, single
fibre)

24/36/48 VDC or 60/120/250 VDC, 110/230VAC Power
supply, Redundant Power supply possible

CRA (Configuration Recovery Adapter) port for easy
exchange of product
General Features AFS650, AFS670 Family

Full managed Ethernet Switch supporting

Protection features such as
RSTP Rapid Spanning Tree Protocol (IEEE802.1w),
MSTP Multiple Spanning Tree Protocol on AFS677 (IEEE802.1s)
MRP Media Redundancy Protocol (IEC62439)
E-MRP Enhanced Redundancy Protocol (Proprietary)
 Protection Scheme in case of link failure

VLAN’s (IEEE802.1Q)
 Separation different applications

Traffic prioritization (IEEE802.1D/p), 4 queues
 Priorization for mission critical applications

Port Rate Limiting (Ingress & Egress) in steps of kbps
 Avoids blocking of mission critical data's

Link Aggregation (IEEE802.3ad)
 Bundling of several parallel links
General Features AFS650, AFS670 Family

Full managed Ethernet Switch supporting

PTP Timing (IEEE1588): Transparent Mode on AFS650/670,
full hardware support on AFS677

Authentication (IEEE802.1x) / Port Security
 Security of end user

Port mirroring
 Packet sniffing for troubleshooting

Traffic Monitoring
 Packet counters for troubleshooting

IGMP (Internet Group Management Protocol) snooping
 Multicast traffic handling

SNMP V1, V2, V3, SSH (Secure Shell)
 Network management
General Features AFS650, AFS670 Family


Configuration

Command Line Interface

Telnet

WEB Interface

Management System AFS-View
Configuration Recovery Adapter (CRA)
 Configuration stored on a USB stick for easy product replacement

Watchdog & Rollback feature
 Restores old configuration in case of cutting of the equipment

Diagnostic LEDs

2 x Alarm contacts

SFP Diagnostics

Local Logfile & Syslog Function

LLDP (Link Layer Discovery Protocol)
 Automatic topology discovery of network
Day 1
AFS677 & AFR677
Hardware Overview
AFS677 & AFR677
Look and Feel
Power Supplies
Alarm contacts
USB Port
for CRA
Gigabit Ethernet
Combo-Port
Configuration
Port (serial)
GbE SFP ports
(optical)
GbE
Electrical Ports
AFS677 & AFR677
Hardware Features

19” managed Full Gigabit Switch of utility grade

Metal housing, Ports on front or on rear (typical substation config)

Equipped with 16 GbE Combo-Port

Combo-Port offers either electrical or SFP-based interface


Electrical 100Mbit/s or 1000Mbit/s
SFP’s in various versions available (MM, SM, short and long
haul)

SFPs for 100Mbit/s or 1000Mbit/s

Up to 4 Ports PoE

Low/ High Voltage as well as AC/ DC power supply,
redundancy possible

CRA (Configuration Recovery Adapter) port for easy exchange of
product
Day 1
AFS677 & AFR677
Specific L2 Feature Overview
Event Time Stamping
Reference Time required for event recording
Event A
Event B
Event A
Event B
Event A
Event B
IRIG-B Interface
IRIG-B Interface
IRIG-B Interface
GPS Receiver
GPS Receiver
GPS Receiver
Event A
Event B
Event A
Event B
IRIG-B Interface
GPS Receiver
Event A
Event B
IEEE1588: Precision Time Protocol (PTP)
SNTP
PTP
SNTP
PTP
SNTP
PTP
SNTP
PTP

Time synchronisation is needed at many levels of utility
networks, typically to time-stamp different types of events.

Simple Network Time Protocol (SNTP) offers accuracy in
the milisecond range, which is not always sufficient

For higher accuracy (microsecond range) PTP can be used
=> accuracy in the microsecond range

PTP includes correction of the run-time between switches

PTP is supported in hardware in AFS677 & AFR677
Day 2
Layer 2 Network
Network Design
Layer-2 Network Design

A good network design is modular and hierarchical, with a
clear separation of functions:

Core: Resilient, few changes, few features, high
bandwidth, CPU power

Distribution: Aggregation, redundancy

Access: Port density, affordability, security features,
many adds, moves and changes
Layer-2 Network Design - Simple
ISP1
Network Border
Core
Distribution
Access
Layer-2 Network Design - Redundant
ISP1
ISP2
Network Border
Core
Distribution
Access
In-Building and Layer 2


There is usually a correspondence between building
separation and subnet separation

Switching inside a building

Routing between buildings
This will depend on the size of the network

Very small networks can get by with doing switching
between buildings

Very large networks might need to do routing inside
buildings
Layer 2 Concepts

Layer 2 protocols basically control access to a shared
medium (copper, fiber, electro-magnetic waves)

Ethernet is the de-facto wired-standard today


Reasons:

Simple

Cheap

Manufacturers keep making it faster
Wireless (802.11a,b,g,n) is also Layer-2 technology.
Ethernet Functions

Source and Destination identification


MAC addresses
Detect and avoid frame collisions

Listen and wait for channel to be available

If collision occurs, wait a random period before retrying

This is called CASMA-CD: Carrier Sense Multiple
Access with Collision Detection
Ethernet Frame

SFD = Start of Frame Delimiter

DA = Destination Address

SA = Source Address

CRC = Cyclick Redundancy Check
Evolution of Ethernet Topologies

Bus


Everybody on the same coaxial cable
Star


One central device connects every other node

First with hubs (repeated traffic)

Later with switches (bridged traffic)
Structured cabling for star topologies standardized
Switched Star Topology Benefits

It’s modular:

Independent wires for each end node

Independent traffic in each wire

A second layer of switches can be added to build a
hierarchical network that extends the same two benefits
above

ALWAYS DESIGN WITH MODULARITY IN MIND
Hub

Receives a frame on one port and sends it out every other
port, always.

Collision domain is not reduced

Traffic ends up in places where it’s not needed
Hub
Hub
A frame sent by one node is always sent to
every other node. Hubs are also called
“repeaters” because they just “repeat” what
they hear.
Switch

Learns the location of each node by looking at the source
address of each incoming frame, and builds a forwarding
table

Forwards each incoming frame to the port where the
destination node is

Reduces the collision domain

Makes more efficient use of the wire

Nodes don’t waste time checking frames not
destined to them
Switch
Forwarding Table
Address
Port
AAAAAAAAAAAA
1
BBBBBBBBBBBB
5
Switch
B
A
Switches and Broadcast


A switch broadcasts some frames:

When the destination address is not found in the table

When the frame is destined to the broadcast address
(FF:FF:FF:FF:FF:FF)

When the frame is destined to a multicast ethernet
address
So, switches do not reduce the broadcast domain!
Switch vs. Router

Routers more or less do with IP packets what switches do
with Ethernet frames


A router looks at the IP packet destination and checks
its routing table to decide where to forward the packet
Some differences:

IP packets travel inside ethernet frames

IP networks can be logically segmented into subnets

Switches do not usually know about IP, they only deal
with Ethernet frames
Switch vs. Router


Routers do not forward Ethernet broadcasts.

Switches reduce the collision domain

Routers reduce the broadcast domain
This becomes really important when trying to design
hierarchical, scalable networks that can grow sustainably
S
R
S
Traffic Domains
Router
Switch
Hub
Switch
Hub
Broadcast Domain
Hub
Hub
Collision Domain
Traffic Domains


Try to eliminate collision domains

Get rid of hubs!

Actually hubs are very rare today.
Try to keep your broadcast domain limited to no more than
250 simultaneously connected hosts

Segment your network using routers
Layer 2 Network Design Guidelines

Always connect hierarchically

If there are multiple switches in a building, use an
aggregation switch

Locate the aggregation switch close to the building
entry point (e.g. fiber panel)

Locate edge switches close to users (e.g. one per floor)

Max length for Cat 5 is 100 meters
Minimize Path Between Elements
✗
✔
Virtual LANs (VLANs)

Allow us to split switches into separate (virtual) switches

Only members of a VLAN can see that VLAN’s traffic

Inter-vlan traffic must go through a router
VLAN introduction

VLANs provide segmentation based on broadcast domains.

VLANs logically segment switched networks based on the functions, project teams, or
applications of the organization regardless of the physical location or connections to
the network.

All workstations and servers used by a particular workgroup share the same VLAN,
regardless of the physical connection or location.
Local VLANs

2 VLANs or more within a single switch

VLANs address scalability, security, and network
management. Routers in VLAN topologies provide
broadcast filtering, security, and traffic flow
management.

Edge ports, where end nodes are connected, are
configured as members of a VLAN

The switch behaves as several virtual switches,
sending traffic only within VLAN members.

Switches may not bridge any traffic between VLANs,
as this would violate the integrity of the VLAN
broadcast domain.

Traffic should only be routed between VLANs.
Local VLANs
Switch
VLAN X
VLAN Y
Edge ports
VLAN X nodes
VLAN Y nodes
Broadcast domains with VLANs and routers
10.1.0.0/16
10.2.0.0/16
Without VLANs:
10.3.0.0/16



Without VLANs, each group is on a different IP
network and on a different switch.
One link per VLAN or a single VLAN
Trunk (later)
Using VLANs. Switch is configured with the ports
on the appropriate VLAN. Still, each group on a With
different IP network; however, they are all on the VLANs
same switch.
10.1.0.0/16
10.2.0.0/16
What are the broadcast domains in each?
10.3.0.0/16
VLANs
Switch 1
172.30.1.21
255.255.255.0
VLAN 1
o
r
t
123456. P
L
A
N
121221. V
172.30.2.12
255.255.255.0
VLAN 2
172.30.2.10
255.255.255.0
VLAN 2
172.30.1.23
255.255.255.0
VLAN 1
TwoTwo
VLANs
= Two subnets
VLANs
Ÿ



Two Subnets
Important notes on VLANs:
VLANs are assigned to switch ports. There is no “VLAN” assignment
done on the host (usually).
In order for a host to be a part of that VLAN, it must be assigned an IP
address that belongs to the proper subnet. Remember: VLAN =
Subnet
VLANs
ARP Request
Switch 1
172.30.1.21
255.255.255.0
VLAN 1
o
r
t
123456. P
L
A
N
121221. V
172.30.2.12
255.255.255.0
VLAN 2
172.30.2.10
255.255.255.0
VLAN 2
172.30.1.23
255.255.255.0
VLAN 1
TwoTwo
VLANs
= Two subnets
VLANs
Ÿ


Two Subnets
VLANs separate broadcast domains!
e.g. without VLAN the ARP would be seen on all subnets.
Assigning a host to the correct VLAN is a 2-step process:
 Connect the host to the correct port on the switch.
 Assign to the host the correct IP address depending on the VLAN
membership
VLAN operation

As a device enters the network, it automatically assumes
the VLAN membership
of the port to which it is attached.

The default VLAN for every port in the switch is VLAN 1
and cannot be deleted.
(This statement does not give the whole story. More
in the lab later for interested groups…)

All other ports on the switch may be reassigned to alternate
VLANs.
VLANs across switches

Two switches can exchange traffic from one or more
VLANs

Inter-switch links are configured as trunks, carrying frames
from all or a subset of a switch’s VLANs

Each frame carries a tag that identifies which VLAN it
belongs to
VLANs across switches
No VLAN Tagging
VLAN Tagging

VLAN tagging is used when a single link needs to carry traffic for more than
one VLAN.
VLANs across switches
Tagged Frames
802.1Q Trunk
Trunk Port
VLAN X
VLAN Y
VLAN X
Edge Ports
This is called “VLAN Trunking”
VLAN Y
802.1Q

The IEEE standard that defines how ethernet frames
should be tagged when moving across switch trunks

This means that switches from different vendors are able to
exchange VLAN traffic.
802.1Q tagged frame
Tagged vs. Untagged

Edge ports are not tagged, they are just “members” of a
VLAN

You only need to tag frames in switch-to-switch links
(trunks), when transporting multiple VLANs

A trunk can transport both tagged and untagged VLANs

As long as the two switches agree on how to handle
those
VLANS increase complexity


You can no longer “just replace” a switch

Now you have VLAN configuration to maintain

Field technicians need more skills
You have to make sure that all the switch-to-switch trunks
are carrying all the necessary VLANs

Need to keep in mind when adding/removing VLANs
Good reasons to use VLANs

You want to segment your network into multiple subnets,
but can’t buy enough switches


Hide sensitive infrastructure like IP phones, building
controls, etc.
Separate control traffic from user traffic

Restrict who can access your switch management
address
Day 2
Layer 2 Network
Switching Loops
Switching Loop
Switch A

When there is more
than one path
between two
switches

What are the
potential problems?
Switch B
Swtich C
Switching Loop

If there is more than one path between two switches:

Forwarding tables become unstable


Source MAC addresses are repeatedly seen coming from different ports
Switches will broadcast each other’s broadcasts

All available bandwidth is utilized

Switch processors cannot handle the load
Switching Loop
Switch A
Switch B
Swtich C
Node 1
Node1 sends a
broadcast frame
(e.g. an ARP request)
Switching Loop
Switch A
Switch B
Swtich C
Node 1
Switches A, B and
C broadcast node
1’s frame out
every port
Switching Loop
Switch A
Switch B
Swtich C
Node 1
But they receive
each other’s
broadcasts, which
they need to
forward again out
every port!
The broadcasts are
amplified, creating
a broadcast
storm…
Good Switching Loops???

But you can take advantage of loops!


Redundant paths improve resilience when:

A switch fails

Wiring breaks
How to achieve redundancy without creating dangerous
traffic loops?
What is a Spanning Tree
“Given
a connected,
undirected graph, a
spanning tree of that
graph is a subgraph
which is a tree and
connects all the vertices
together”.
A
single graph can
have many different
spanning trees.
Spanning Tree Protocol

The purpose of the protocol is to have bridges dynamically
discover a subset of the topology that is loop-free (a tree)
and yet has just enough connectivity so that where
physically possible, there is a path between every switch
Spanning Tree Protocol

Several flavors:

Traditional Spanning Tree (802.1d)

Rapid Spanning Tree or RSTP (802.1w)

Multiple Spanning Tree or MSTP (802.1s)
Traditional Spanning Tree (802.1d)

Switches exchange messages that allow them to compute
the Spanning Tree

These messages are called BPDUs (Bridge Protocol
Data Units)

Two types of BPDUs:

Configuration

Topology Change Notification (TCN)
Traditional Spanning Tree (802.1d)

First Step:

Decide on a point of reference: the Root Bridge

The election process is based on the Bridge ID, which
is composed of:

The Bridge Priority: A two-byte value that is
configurable

The MAC address: A unique, hardcoded address
that cannot be changed.
Root Bridge Selection (802.1d)

Each switch starts by sending out BPDUs with a Root
Bridge ID equal to its own Bridge ID


Received BPDUs are analyzed to see if a lower Root
Bridge ID is being announced


I am the root!
If so, each switch replaces the value of the advertised
Root Bridge ID with this new lower ID
Eventually, they all agree on who the Root Bridge is
Root Bridge Selection (802.1d)
32678.0000000000AA
Switch A
Switch B
32678.0000000000BB
 All switches have the same priority.
 Who is the elected root bridge?
Switch C
32678.0000000000CC
Root Port Selection (802.1d)

Now each switch needs to figure out where it is in relation
to the Root Bridge

Each switch needs to determine its Root Port

The key is to find the port with the lowest Root Path
Cost

The cumulative cost of all the links leading to the
Root Bridge
Root Port Selection (802.1d)

Each link on a switch has a Path Cost

Inversely proportional to the link speed
e.g. the faster the link, the lower the cost
Link Speed
STP Cost
10 Mbps
100
100 Mbps
19
1 Gbps
4
10 Gbps
2
Root Port Selection (802.1d)

Root Path Cost is the accumulation of a link’s Path Cost
and the Path Costs learned from neighboring Switches.

It answers the question: How much does it cost to
reach the Root Bridge through this port?
Root Port Selection (802.1d)
1.
2.
3.
4.
Root Bridge sends out BPDUs with a Root Path Cost
value of 0
Neighbor receives BPDU and adds port’s Path Cost to
Root Path Cost received
Neighbor sends out BPDUs with new cumulative value
as Root Path Cost
Other neighbor’s down the line keep adding in the same
fashion
Root Port Selection (802.1d)

On each switch, the port where the lowest Root Path Cost was received
becomes the Root Port

This is the port with the best path to the Root Bridge
Root Port Selection (802.1d)
32678.0000000000AA
1
Switch A 2
Cost=19
1
Switch B
Cost=19
2 Cost=19
32678.0000000000BB
2
1
Switch C
32678.0000000000CC
 What is the Path Cost on each Port?
 What is the Root Port on each switch?
Root Port Selection (802.1d)
32678.0000000000AA
1
Switch A 2
Cost=19
Root Port
1
Switch B
32678.0000000000BB
Cost=19
2 Cost=19
2
1
Switch C
32678.0000000000CC
Root Port
Electing Designated Ports (802.1d)

OK, we now have selected root ports but we haven’t solved the loop problem
yet, have we?The links are still active!

Each network segment needs to have only one switch forwarding traffic to and
from that segment

Switches then need to identify one Designated Port per link

The one with the lowest cumulative Root Path Cost to the Root Bridge
Root Port Selection (802.1d)
32678.0000000000AA
1
Switch A 2
Cost=19
1
Switch B
32678.0000000000BB
Cost=19
2 Cost=19
2
1
Switch C
32678.0000000000CC
Which port should be the Designated Port on
each segment?
Electing Designated Ports (802.1d)

Two or more ports in a segment having identical Root Path Costs is possible,
which results in a tie condition

All STP decisions are based on the following sequence of conditions:

Lowest Root Bridge ID

Lowest Root Path Cost to Root Bridge

Lowest Sender Bridge ID

Lowest Sender Port ID
Root Port Selection (802.1d)
Designated
Port
32678.0000000000AA
1
Switch A 2
Cost=19
1
Switch B
Designated
Port
Cost=19
2 Cost=19
32678.0000000000BB
Designated
Port
2
1
Switch C
32678.0000000000CC
In the B-C link, Switch B
has the lowest Bridge
ID, so port 2 in Switch B
is the Designated Port
Blocking a port

Any port that is not elected as either a Root Port, nor a
Designated Port is put into the Blocking State.

This step effectively breaks the loop and completes the
Spanning Tree.
Root Port Selection (802.1d)
32678.0000000000AA
1
Switch A 2
Cost=19
1
Switch B
32678.0000000000BB
2 Cost=19
✗
2
Cost=19
1
Switch C
32678.0000000000CC
Port 2 in Switch C is put into the Blocking
State, because it is neither a Root Port nor
a Designated Port
Spanning Tree Protocol States

Disabled



Port is shut down
Blocking

Not forwarding frames

Receiving BPDUs
Listening

Not forwarding frames

Sending and receiving BPDUs
Spanning Tree Protocol States


Learning

Not forwarding frames

Sending and receiving BPDUs

Learning new MAC addresses
Forwarding

Forwarding frames

Sending and receiving BPDUs

Learning new MAC addresses
STP Topology Changes

Switches will recalculate if:

A new switch is introduced

It could be the new Root Bridge!

A switch fails

A link fails
Root Bridge Placement


Using default STP parameters might result in an undesired
situation

Traffic will flow in non-optimal ways

An unstable or slow switch might become the root
You need to plan your assignment of bridge priorities
carefully
Bad Root Bridge Placement
Out to router
32678.0000000000DD
Swtich D
Switch B
32678.0000000000BB
Root
Bridge
32678.0000000000CC
Switch C
Switch A
32678.0000000000AA
Good Root Bridge Placement
Alternative
Root
Bridge
1.0000000000DD
Out to standby
router
Swtich D
32678.0000000000CC
Switch C
Out to active
router
Switch B
Switch A
Root
Bridge
0.0000000000BB
32678.0000000000AA
STP Design Guidelines



Enable spanning tree even if you don’t have redundant
paths
Always plan and set bridge priorities

Make the root choice deterministic

Include an alternative root bridge
If possible, do not accept BPDUs on end user ports
802.1d Convergence Speeds

Moving from the Blocking state to the
Forwarding State takes at least 2 x Forward
Delay time units
(~ 30 secs.)


Some vendors have added enhancements such
as PortFast, which will reduce this time to a
minimum for edge ports


This can be annoying when connecting end user
stations
Never use PortFast or similar in switch-to-switch links
Topology changes typically take 30 seconds too

This can be unacceptable in a production network
Rapid Spanning Tree (802.1w)

Convergence is much faster


Communication between switches is more interactive
Edge ports don’t participate

Edge ports transition to forwarding state immediately

If BPDUs are received on an edge port, it becomes a
non-edge port to prevent loops
Rapid Spanning Tree (802.1w)

Defines these port roles:

Root Port (same as with 802.1d)

Alternate Port

A port with an alternate path to the root

Designated Port (same as with 802.1d)

Backup Port

A backup/redundant path to a segment where
another bridge port already connects.
Rapid Spanning Tree (802.1w)

Synchronization process uses a handshake method

After a root is elected, the topology is built in cascade,
where each switch proposes to be the designated
bridge for each point-to-point link

While this happens, all the downstream switch links are
blocking
Rapid Spanning Tree (802.1w)
DP
Root
Proposal
RP
Agreement
Switch
Switch
Switch
Switch
Rapid Spanning Tree (802.1w)
DP
Root
DP
Proposal
RP
RP
Agreement
Switch
Switch
Switch
Switch
Rapid Spanning Tree (802.1w)
DP
Root
DP
RP
RP
Switch
Switch
DP
Proposal
Agreement
RP
Switch
Switch
Rapid Spanning Tree (802.1w)
DP
Root
DP
RP
RP
Switch
Switch
DP
DP
Proposal
Agreement
RP
RP
Switch
Switch
Rapid Spanning Tree (802.1w)

Prefer RSTP over STP if you want faster convergence

Always define which ports are edge ports
Multiple Spanning Tree (802.1s)

Allows separate spanning trees per VLAN group



Different topologies allow for load balancing between
links
Each group of VLANs are assigned to an “instance” of
MST
Compatible with STP and RSTP
Multiple Spanning Tree (802.1s)
Root VLAN A
Vlan A
Root VLAN B
Vlan B
✕
✕
Multiple Spanning Tree (802.1s)

MST Region

Switches are members of a region if they have the
same set of attributes:

MST configuration name

MST configuration revision

Instance-to-VLAN mapping

A digest of these attributes is sent inside the BPDUs for
fast comparison by the switches

One region is usually sufficient
Multiple Spanning Tree (802.1s)

CST = Common Spanning Tree


In order to interoperate with other versions of Spanning
Tree, MST needs a common tree that contains all the
other islands, including other MST regions
IST = Internal Spanning Tree

Internal to the Region, that is

Presents the entire region as a single virtual bridge to
the CST outside
Multiple Spanning Tree (802.1s)

MST Instances

Groups of VLANs are mapped to particular Spanning
Tree instances

These instances will represent the alternative
topologies, or forwarding paths

You specify a root and alternate root for each instance
Multiple Spanning Tree (802.1s)
CST
MST Region
MST Region
IST
IST
802.1D switch
Multiple Spanning Tree (802.1s)

Design Guidelines

Determine relevant forwarding paths, and distribute
your VLANs equally into instances matching these
topologies

Assign different root and alternate root switches to each
instance

Make sure all switches match region attributes

Do not assign VLANs to instance 0, as this is used by
the IST
Additional Information
AFS677 & AFR677
Specific L3 Feature Overview
Data transfer: SWITCH (Layer 2)
Client B
Client A
Client A
One IP network
Client C
10.10.1.0/16
Application
Application
Presentation
Presentation
Session
Session
Transport
Switch 2
Switch 1
Client B
Client D
Switch
Network
Transport
Network
Data Link
Data Link
Data Link
Physical
Physical
Physical

All NE are in the same IP network and form one broadcasting domain

Packet forwarding is done based on MAC-address

Fast convergence in case of link failure (RSTP, E-MRP..)

No logical separation of the connected Networks

Unmanaged and Managed switches available
All AFS switches are fully managed
Data transfer: ROUTER (Layer 3)
Client A
Client C
10.10.1.0/16
Client B
Client A
2 different IP Networks
192.168.1.0/24
Application
Application
Presentation
Presentation
Session
Router
Transport
Client B
Router
Client D
Session
Transport
Network
Network
Network
Data Link
Data Link
Data Link
Physical
Physical
Physical

Router separates different IP networks = logical separation

Router creates several broadcast domains
(1 IP network = 1 broadcast domain)

Packet forwarding is done based on IP-address

Slower convergence in case of link failure (routing protocol)

Routing can be implemented in software (slower) or hardware (faster)
AFR677 supports routing in hardware

Routers are managed devices
Switching & Routing



Switching is required for:

Mission Critical services

Substation automation applications

Distribution communication applications
Routing is required for:

Big and complex communication networks

Connection of different LAN/ WAN networks

At the borders of substations or control centres
Most applications can run over a Switched or a routed network

But IEC61850 GOOSE & SV messages can not be routed today!
(update to IEC standard under development)
 Network structure needs to be planned carefully
IP Routing Table

Any routing table requires at least three fields:
DESTINATION
NEXT HOP
DESTINATION NEXT HOP
Network A
Network B
Network C
local
local
B.2
OUTBOUND
INTERFACE
A.1
B.1
B.1
OUTBOUND INTERFACE
DESTINATION NEXT HOP
Network A
Network B
Network C
B.1
local
local
OUTBOUND
INTERFACE
B.2
B.2
C.1
Static vs. Dynamic Routing

Routing tables can be populated in two ways:


Static Routing:

The routes are manually entered by the network administrator.

Used in small network with only little changes.

No traffic redundancy is supported
Dynamic Routing:

Used in bigger, more dynamic networks

The routes are automatically 'learned' by the router, based on a
routing protocol (OSPF, RIP, BGP, …)

In case of a path failure, the routing protocol can calculate
alternate paths thus providing redundancy.
L3 Functionality on AFR677


Routing Features

Static Routing

Dynamic Routing: OSPF or RIP

Port-based and VLAN-based routing
Network Protection

OSPF/RIP recalculation in case of network failure

Routing functionality can be combined with L2
protection
(e.g. E-MRP)

Virtual Router Redundancy VRRP / E-VRRP & Tracking
AFR677
Port-based and VLAN-based routing
AFR677


There are two different possibilities to configure routing:

Port-Based Routing: Individual physical ports are considered to be router interfaces.

VLAN based Routing: A VLAN is assigned a virtual router interface.

Physical interfaces and VLAN virtual interface are assigned an IP address
The configurations can also be mixed.
AFR677
Port based and VLAN base router configuration
R
R
Switch
Switch
Layer 2 Switch
R
VLAN
Switch
Router & Switch
Definition of VLANs
VLAN
Router
R
R
VLAN
R
VLAN
Switch
Routing between VLANs Free Configuration
AFR677
Virtual Router Redundancy Protocol (VRRP)
Router A
(active)
Router A
(standby)
X
Router B
(standby)
Router B
(active)
Additional Information
ABB FOX Firewall
Introduction to AFF650
AFF650: ABB FOX Firewall
Hardware


2 ports 100BASE-X

TX/TX

FX/FX

FX/TX including single mode fibre
1 RS232 port with RJ11 socket for

configuration of AFF

secure modem access to network for
remote diagnosis

1 USB socket for CRA support

Temperature range:



Standard: 0°C to +60°C

On request: -40°C to +60°C
Voltage range:

9.6 to 60 VDC

18 to 30 VAC
Power consumption: 9.6 VA max. at 24 VDC
Why is Cyber Security security an issue?

Cyber security has become an issue by introducing Ethernet
(TCP/IP) based communication protocols to industrial
automation and control systems.
 e.g. IEC60870-5-104 or IEC61850

Connections to and from external networks (e.g. office
intranet) to industrial automation and control systems have
opened systems and can be misused for cyber attacks.

Cyber attacks on industrial automation and control systems
are real and increasing, leading to large financial losses
What is Cyber Security?

Main goals of Cyber Security are:

Availability
– avoid denial of service

Integrity
– avoid unauthorized modification

Confidentiality
– avoid disclosure

Authentication
– avoid spoofing / forgery

Authorization
– avoid unauthorized usage
ABB’s Security Landscape
System Overview
User
Corporate Network
NCC or Regional Centre
Internet
WAN
Plant / Substation
Plant / Substation
Stateful Inspection
Insecure
Secure
Request
Response
Request
X
Response
Packet Filtering
Insecure
Secure
Examples:
MAC Filter
IP Filter
Port Filter
Only allow traffic from 00:02:A3:02:60:00
Only allow traffic to 192.168.1.100
Block all TELNET traffic (port 23)
What is a VPN?
VPN is a secure L3 connection

Tunneling private communication through an
insecure network (e.g. the Internet)

Function of VPNs:


Authentication

Encryption
Applications: Connecting geographically
distributed participants

Connection of RTU in substation to control
center (through public network)

Connection of remote offices to headoffice.
AFF650 - Main Functions

Operates in Routing or Bridging Mode

Router Functionality

Firewall Functionality (in routing or bridging mode)

Stateful Inspection

Packet filtering (IP address or protocol)

Packet filtering (MAC address)

Protection against DoS attacks

Network Address Translation - NAT (1:1, 1:n)

VPN Functionality (IPSec)

Router Redundancy

Dynamic DNS
Example: Customer projects with firewalls
© ABB Group
October 2, 2014 | Slide 169