Introduction To Storage
Overview of the storage subsystem
Objectives for this Unit



Understand Storage Basics
Introduce Direct Attached Storage
Differentiate between types of storage






DAS
NAS
ISCSI
SAN
Introduce Network Attached Storage
Introduce Fibre Channel SAN
A Few Storage Basics….



Where will data finally end up?
How will it get there?
What will it pass through?
Direct Attached Storage
(Internal)
Computer System
CPU
Memory
Bus
I/O - RAID
Controller
Disk Drives
Direct Attached Storage
(Internal)
12345
Data
Computer System
CPU
Memory
Bus
I/O - RAID
Controller
Disk Drives
Direct Attached Storage
(Internal)
Computer System
12345
CPU
Memory
Bus
I/O - RAID
Controller
Disk Drives
DAS w/ internal controller and
external storage
Computer System
12345
CPU
Disk Enclosure
Memory
Bus
I/O - RAID
Controller
Disk Drives
Disk Drives
Disk Drives
Comparing Internal and External
Storage
Server
Storage
Disk Drives
RAID Controller
Internal Storage
RAID controllers
and disk drives
are internal to the
server
SCSI, ATA, or
SATA protocol
between controller
and disks
RAID controller is
internal
SCSI or SATA
protocol between
controller and
disks
Disk drives are
external
Server
RAID Controller
Storage
Disk Drives
SCSI Bus w/ external storage
DAS w/ external controller and
external storage
Storage System
Computer System
12345
CPU
Memory
Bus
Disk Enclosure
Disk Drives
Disk Drives
Disk Drives
HBA
RAID
Controller
DAS over Fibre Channel
Server
HBA
HBA is internal
Fibre Channel
protocol
between HBAs
and external
RAID controller
Disk drives and
RAID controller
are external
Storage
Disk Drives
RAID Controller
External SAN Array
I/O Transfer

RAID Controller



Contains the “smarts”
Determines how the data will be written (striping,
mirroring, RAID 10, RAID 5, etc.)
Host Bus Adapter (HBA)




Simply transfers the data to the RAID controller.
Doesn’t do any RAID or striping calculations.
“Dumb” for speed.
Required for external storage.
Storage types









Single Disk Drive
JBOD
Volume
Storage Array
SCSI device
DAS
NAS
SAN
iSCSI
NAS: What is it?




Network Attached Storage
Utilizes a TCP/IP network to “share” data
Uses file sharing protocols like Unix NFS
and Windows CIFS
Storage “Appliances” utilize a strippeddown OS that optimizes file protocol
performance
Networked Attached Storage
Public or Private Ethernet
network
NIC
Server
NIC
All data converted
to file protocol for
transmission (may
slow down
database
transactions)
Server has a
Network Interface
Card
No RAID
Controller or HBA
in the server
NAS Server
Storage
Disk Drives
RAID Controller
iSCSI: What is it?





An alternate form of networked storage
Like NAS, also utilizes a TCP/IP network
Encapsulates native SCSI commands in TCP/IP
packets
Supported in Windows 2003 Server and Linux
TCP/IP Offload Engines (TOEs) on NICs speed
up packet encapsulation
iSCSI Storage
Public or Private Ethernet
network
NIC or iSCSI HBA
Server
NIC or iSCSI HBA
SCSI commands
are encapsulated
in TCP/IP packets
Server has a
Network Interface
Card or iSCSI
HBA
iSCSI HBAs use
TCP/IP Offload
Engine (TOE)
iSCSI Storage
Disk Drives
RAID Controller
Fibre Channel: What is it?


Fibre Channel is a network protocol
implemented specifically for dedicated
storage networks
Fibre Channel utilizes specialized




Switches
Host Bus Adapters
RAID controllers
Cables
Fibre Channel Components
Server A
HBA
Server B
HBA
HBA
Server C
HBA
HBA

Servers

switch
switch

RAID Controller

RAID Controller

FC Storage Array
Two switches for redundancy
Fibre Channel Storage Array

Disk Drives
Fiber optic or copper
Fibre Channel Switches


Host Bus Adapters
Cables


HBA
Two RAID Controllers for
redundancy
4–100+ disk drives per array
A true storage network



Multiple servers
Multiple switches
Multiple Storage Arrays
SAN: What is it?




Storage Area Network
A network whose primary purpose is the transfer
of data between storage systems and computer
systems
Fibre Channel is the primary technology utilized
for SANs
Recently, SANs have been implemented with
dedicated iSCSI networks
Benefits of SAN/Consolidated
Storage






Reduce cost of external storage
Increase performance
Centralized and improved tape backup
LAN-less backup
High-speed, no single-point-of-failure
clustering solutions
Consolidation with > 70TB of storage
Fibre Channel Technology








Provides concurrent communications between servers,
storage devices, and other peripherals
A gigabit interconnect technology
FC1: Over 1,000,000,000 bits per second
FC2: Over 2,000,000,000 bits per second
A highly reliable interconnect
Up to 127 devices (SCSI: 15)
Up to 10 km of cabling (3-15 ft. for SCSI)
Physical interconnect can be copper or fiber optic
Fibre Channel – (continued)




Hot-pluggable - Devices can be removed
or added at will with no ill effects to data
communications
Provides a data link layer above the
physical interconnect, analogous to
Ethernet
Sophisticated error detection at the frame
level
Data is checked and resent if necessary
Fibre Channel – Frame
Dissection


Up to 2048 byte
payload
4 byte checksum
for each frame
Fibre Channel

What’s with the funny name?




Some background history required
Originally developed to only support fiber optic
cabling
When copper cabling support was added, ISO
decided not to rename the technology
ISO changed to the French spelling to reduce
association with fiber optics only medium
Fibre Channel

How does it work?



Serial interface
Data is transferred across a single piece of
medium at the fastest speed supported
No complex signaling required
Fibre Channel Interface
Layers
Device Driver
SCSI Protocol
Fibre Channel
Fiber Optic or Copper Cabling
SCSI vs. Fibre Channel
Protocol

SCSI





SCSI protocol vs. SCSI device
SCSI is an established, tried and true protocol
Provides services analogous to TCP/IP
Supported in every major OS on market
Fibre Channel



Fibre Channel runs on top of SCSI
No re-inventing the wheel
Immediate OS support
SCSI vs. FC Transmission
RAID Controller
Disk Drive
Fibre Channel
RAID Controller
Disk Drive
SCSI
SCSI







vs.
Interface for internal storage to
external disks
Potential down time w/ SCSI
Single bus
RAID controller is SCSI hardware
Standards:

Ultra2 (80 MB/sec)

Ultra 160 (160 MB/sec)

Ultra 320 (320 MB/sec)
Media specific (copper only)
SCSI Limitations:

Cables can’t be any longer
than 3 feet for single ended;
15 feet for LVD (low voltage
differential)

No more than 15 devices on
a SCSI bus

# of disk drives
Fibre Channel







Used with SAN
Lots of built-in redundancy with connections
Redundant network
HBA is fibre channel hardware
Standards:

FC1: 100 MB/sec

FC2: 200 MB/sec
Provides a data link layer above the physical
interconnect

Analogous to Ethernet

FC is a network of devices

It can be media independent- copper or
fibre optic
Fibre Channel limitations:

Cable length: Up to 10 kilometers (more
a limitation of cable than FC itself)

Up to 127 devices

# of disk drives
Fibre Channel vs. iSCSI

Fibre Channel






The current market leader for shared storage technologies
Provides the highest performance levels
Designed for mission-critical applications
Cost of components is relatively high, particularly per server
HBA costs
Relatively difficult to implement and manage
iSCSI






Relatively new, but usage is increasing rapidly
Performance can approach Fibre Channel speeds
A better fit for databases than NAS
A good fit for Small to Medium Size Businesses
Relatively inexpensive, compared to Fibre Channel
Relatively easy to implement and manage
Microsoft Simple SAN Initiative


Make operating system aware of SAN and SAN
capabilities
Shift integration burden from IT staff or services back
to VENDORS products:




Microsoft
Storage hardware and software
Application developers
Key storage technologies:




Volume Shadow Copy Service (VSS)
Virtual Disk Service (VDS)
Microsoft Multipath Input/Output (MPIO)
Microsoft iSCSI driver


Software Initiator (client)
Software Target (attached to disk subsystem)
Review






What is the difference between a RAID
Controller and an HBA?
How many protocols may be used for DAS?
Name two types of storage that rely on ethernet
cables
Name two benefits of SANs
Describe the four interface layers of the Fibre
Channel protocol
Describe a scenario where an iSCSI SAN may
be preferred over a Fibre Channel SAN
Summary


How data is routed through a server to I/O
Types of storage





Benefits of SAN technology




Storage consolidation
Reduced costs
Centralized, LAN-free backup and restore
The Fibre Channel protocol



DAS
NAS
iSCSI
SAN
How it works
Fibre Channel protocol vs. SCSI protocol
Comparing Fibre Channel SANs and iSCSI SANs


Fibre Channel SANs offer mission-critical performance, with relatively high
costs and high complexity
iSCSI SANs offer moderate to high performance at an attractive
price/performance ration and are relatively easy to administer
Fibre Channel
Objectives

Define what is Fibre Channel

Standards

Fibre Channel Architecture

Simple example of a Network Connection

Fiber Channel Layers
 Summary FC Layers (Picture)
Fibre Channel Topologies



Technology Comparison
Conclusion
What is Fibre Channel?

A high-speed transmission technology used as
a peripheral channel or network backbone.

It is a 100MB/sec, full-duplex, serial, data
communication technology.

It supports several common transport protocols
like Internet Protocol (IP) and SCSI.

It operates over copper and fiber optic cables at
distances of up to 10 Kilometers.

It is supported by many suppliers like Compaq,
Hewlett-Packard, IBM, Seagate, and Sun
Microsystems.
Standards for Fibre Channel

The T11 Committee of NCIT, a U.S. standardsdevelopment organization under the ANSI (American
National Standards Institute) meets 6 times a year to
develop Fibre Channel standards.
Current standards:

Some examples of current standards.

Performance from 266 megabits/second to
over 4 gigabits/second.

Support for distances of up to 10Km.

High-bandwidth utilization with distance
insensitivity.

Support for multiple cost/performance levels,
from small systems to super computers.

Ability to carry multiple existing interface
command sets, including Internet Protocol (IP),
SCSI, IPI, and audio and video.
Fibre Channel
Architecture

Fibre Channel transfers digital data between
sources and users of information.

This digital data represents different types of
information like programs, files, graphics,
videos and sound.

Each having its own structure, protocol,
connectivity, measures of performance and
reliability requirements.
Network Connection
Example
Node Ports
Printer
Node Ports
Computer/
Workstation
Node Ports
Server
Storage
Device
Node Ports
Computer/
Workstation
Fabric Ports
Network connection (continued):

Network Connections are established
between the node ports (N_Ports), that are in
computers, servers, storage devices, and
printers, and the fabric ports (F_Ports), that
are on the periphery of the Fibre Channel
Fabric.

The Fibre Channel Architecture specifies in
detail the link Characteristics and protocol
used between the node ports and the fabric
ports.

The Fibre Channel can interconnect more
than 16 million node ports in a single address.
Fibre Channel Layers

Five layers FC-1, FC-2, FC-3, and FC-4.

Define the physical media, transmission rates,
encoding scheme, framing protocol, flow
control, common services, and the upper level
interfaces.

FC-0, FC-1, and FC-2 - define how Fibre
Channel ports interact with other ports.

They are refereed to as the Fibre Channel
Physical levels (PC-PH Levels).

FC-3, and FC-4 - define how Fibre Channel
ports interact with applications in host systems.
FC-0 : Media and Interfaces

Covers the physical characteristics of the
interface and media, including cables,
connectors, drivers, transmitters, and
receivers.

Examples of media :





twisted pair
coaxial
multi mode/single mode fiber
fiber light sources
long wave lasers
FC-1, and FC-2:

FC-1: Transmission Protocol


Defines how FC-0 signals are patterned to carry data
and how port-to-port links are initialized.
FC-2: Framing and Signaling Protocol


Defines the rules for signaling and the transfer of data.
Defines various classes of services, some examples:
 Class 1: Is a full-duplex dedicated link between 2
ports. (Highest quality of service because it is the
most effective in transferring large amounts of data
at very high speed.
 Class 2: Multiplexed connection , where 1 port can
carry different exchanges with many other ports.
 Class 3: Multicast and broadcast where there’s no
confirmation of receipt.
FC-3 and FC-4:

FC-3: Common Services


Defines commons services provided by two or
more node ports in a host system. (Ex. Two or
more node ports, sharing a common port
address, which increases the bandwidth available
from node port to fabric ports).
FC-4: Protocol Mappings


Formed by series of profiles defining how to map
legacy protocols to Fibre Channel.
Profiles for protocols like IP, SCSI, for disk drives,
and several others are already defined here.
Fibre Channel Layers
FC4
FC-3
Source: www.fibrechannel.com/layers/
Fibre Channel Layers

The previous picture illustrates the relationship
between the media type and the operating range
for each Fibre Channel, which is defined by the FC0 layer.



For example we see that a Multimode Fiber medium
has a transfer rate of 133Mbps-266Mbps.
And Singlemode Fiber medium has a range of
531Mbps-1.06Gbps.
Whereas a copper medium has a transfer rate
2.12Gbps-4.25Gbps.
Fibre Channel Layers
(continued)

We also observe the relationship between FC-0 and FC1, where FC-1 defines how the signals are carried by the
FC-0 layer.



We also observe that the FC-2 layer defines the framing
protocol, and flow control.
We also observe that FC-3 defines the common services.
And that FC-4 is the layer defining the protocols like IP.
Fabric Topologies

There are three topologies for Fibre Channel
Fabrics:

Point-to-point


Switched


Where two node ports have the same signaling
rate and class of service.
Where 16 million node ports can be
interconnected.
Loop (Ring)

Organizes up to 127 Fibre Channel ports on a
ring, and distributes the routing functions among
them.
It is used more than the switched topology.

It also costs less than switched topology.

Technology
Comparison
Fibre Channel
Technology
Storage, network,
applicatios
video, clusters
Point-to-point loop
Topologies
hub, swithced
Scability to
2.12 Gbps,
higher data rates
4.24Gbps
Guaranteed
Yes
Delivery
Congestion data
None
loss
Frame size
Variable, 0-2KB
Flow Control
Credit Based
Physical media
Protocols
Supported
Copper and Fiber
Gigabit Ethernet
Network
ATM
Network
video
Point-to-point hub,
switched
switched
Not defined
1.24Gbps
No
No
Yes
Yes
Variable, 0-1.5KB
Rate Based
Fixed, 53B
Rate Based
Copper and
Fiber
Network
video
Copper and Fiber
Network, SCSI,
Network
Video
Table 3.3 Technology comparison
Source: www.fibrechannel.com
Technology
Comparison

Clearly from this table we can observe that Fibre
Channel is the best technology:

Because it provides a higher data rate than ATM.

Because it can be employed in more topologies,
when compared to the Ethernet, and ATM.

It is more reliable since the delivery of data is
guaranteed and there’s no loss of data.
It has a bigger frame size of up to 2KB when
comparing it to Ethernet’s 1.5KB, and ATM’s 53B
And also because it supports Network, SCSI, and
video protocols, whereas Ethernet only supports
Network, and ATM only supports Network and Video
Protocols.


Conclusion





Once again Fibre Channel is a high speed
peripheral transmission technology used in
networks.
With a transmission rate of 100MB/sec and with a
full-duplex flow of transmission.
It is defined by five layers which are FC-0, FC-1,
FC-2, FC-3, and FC-4, and they define the media,
transmission rates, coding/encoding, framing, flow,
and protocols supported.
And when compared to the Ethernet and ATM
technologies it is the best.
Experts agree that Fibre Channel is the first
technology with the potential to move the data
communications industry into a low-cost-ofownership, commodity phase.
iSCSI Introduction
Contents
•
•
•
•
Definition of iSCSI
iSCSI vs Fibre Channel
iSCSI Session Types
Key Negotiation Overview
Definition of iSCSI
•
•
iSCSI is a Storage Area Network (SAN)
protocol that allows for SCSI command
transmission over a TCP/IP network
iSCSI allows for the sharing of I/O
devices over a long distance, especially
storage devices
–
Typically high speed disk arrays
Definition of iSCSI Continued
•
•
iSCSI maintains the SCSI notion of an
Initiator and Target device
iSCSI removes the traditional bus
structure of SCSI and maps SCSI over
a network
Bus vs Network Architecture

Bus Architecture

Network Architecture
Problems Mapping SCSI to
a Network
•
•
In order not to violate Bus model of SCSI,
there can only ever be one (1) concurrent
Session between an iSCSI Initiator and iSCSI
Target
–
Commonly called “The ISID Rule”
Multiple Connections may exist within that
Session, but all Connections must contain the
same Initiator Session Identifier (ISID)
iSCSI vs Fibre Channel
•
Both protocols have advantages and
disadvantages
–
–
iSCSI is typically less expensive
Fibre Channel is typically faster
iSCSI
•
iSCSI is traditionally less expensive
than Fibre Channel to implement
–
•
•
Can be done with only software
iSCSI can run over preexisting copper
infrastructure and use preexisting
switches and routers
iSCSI does not require specially trained
network administrators to manage the
network
Fibre Channel
•
•
•
In most cases, Fibre Channel will outperform
iSCSI in data transfer speeds
Fibre Channel requires a special fiber
network to be created in order to be
implemented
Total Cost of Ownership (TCO) is much
higher than iSCSI
–
Special network, administrators, fiber
maintenance, etc.
Comparison Conclusion
•
•
•
Fiber Channel is much faster than traditional
iSCSI
– iSCSI Extensions for RDMA (iSER) aims to
lessen this difference
iSCSI can run over preexisting copper
infrastructure
– iSCSI devices can be accessed over the
Internet
In almost all cases, iSCSI is less expensive to
implement and maintain
iSCSI Session Types
•
iSCSI has two (2) Session types
–
–
•
Discovery
Normal
Each Session has two (2) Phases
–
Login
–
–
Contains Security Negotiation and Operational
Parameter Negotiation
Full Feature Phase
Discovery Session
•
•
A Discovery Session is used to allow
the iSCSI Initiator to find iSCSI Targets
that it can connect to
The Initiator must explicitly tell the
Target that the Session to be created be
a Discovery Session
–
SessionType=Discovery
Login within Discovery
•
Allows the
Initiator and
Target to
establish 'who' it
is
Full Feature Phase within
Discovery
•
•
Once the Login Phase has completed, the
Session can transfer into Full Feature Phase
(FFP)
Within a Discovery Session, FFP allows the
sending of Text Requests (Initiator) and Text
Responses (Target)
–
•
Used to let the Target inform the Initiator what
devices it can access
No other types of data units may be transmitted
Discovery Session
•
•
Initiator transmits
the SendTargets
key to the Target
Target replies with
a list of devices
accessible to the
Initiator and their
addresses
Discovery Session
•
•
Once a Discovery Session has
completed, the Session must be
terminated
If the Initiator then desires to perform
data I/O, a Normal Session must be
created
Normal Session
•
•
•
An iSCSI Initiator will explicitly tell the
iSCSI Target that the Session to be
created is a Normal Session
A Normal Session is defined in RFC
3720 to be an unrestricted session
Access to SCSI devices can only be
achieved while in a Normal Session
Login within Normal Session
•
Allows the Initiator and Target to
declare/negotiate data transmission
rules and other connection settings
–
Ex. Size of individual data units, number of
simultaneous data transfers
Full Feature Phase in Normal
Session
•
Once Full
Feature Phase
(FFP) within a
Normal Session
has been
reached, data I/O
can take place
Key Negotiation Overview
•
All iSCSI parameters are contained
within key=value pairs
–
•
Ex. “TargetAlias=UNH-IOL”
Some keys are declared and others are
negotiated
–
–
Declared keys do not need a response
from the receiver
Negotiated keys must always receive a
response from the receiver
iSCSI Key Negotiation
•
•
Key negotiations can
allow the enabling or
disabling of features
Also allows for value
negotiation or
declaration
•
Ex. MaxBurstLength=512
More on Key Negotiation will be
covered in the iSCSI Login
Presentation
Summary
•
•
•
•
iSCSI is a low cost, low maintenance
alternative to Fibre Channel
iSCSI networks can use existing copper
infrastructures
Unlike Fibre Channel, iSCSI networks can
span the Internet
iSCSI Key Negotiation allows for
enabling/disabling of features as well as
value negotiations
IP Storage
The Purpose of this Tutorial

IP Storage as “block” vs. “file” storage






NAS will be discussed peripherally
To provide details about IP Storage
To provide factual information
To clarify issues
To facilitate understanding
Key point

This is will be pragmatic education not
cheerleading
76
IP Networked Storage
iSCSI – New Possibilities
Overview





Introduction
Benefits of IP Storage
IP Storage technologies
iSCSI
Conclusions
78
Introduction
“Ethernet wins. Again. In time… Ethernet will eventually triumph
over all other storage networking technologies, including Fibre
Channel”
Source: March 2001 Forrester Research
“If we were starting with a clean piece of paper … we would
probably use gigabit Ethernet and IP”
Source: Bill Miller CTO StorageNetworks, Industry Standard
“... 76% of senior IT executives believe IP will make it easier to
implement large-scale storage networks”
Source: Enterprise Storage Group 9/11/2000
“75% perceive iSCSI as the IP storage standard”
Source: Marc Staimer , Dragon Slayer Consulting – May 2001
79
Network Storage Models
Direct Attached Storage
•High Cost of Ownership
•In-flexible
Network Attached Storage
Storage Area Network
•Transmission optimized for file
•Transmission optimized for database
transactions
transactions
•Storage traffic travels across the LAN •Separate LAN and SAN
•Increases Data availability
•Flexible and scalable
80
Moving from Dedicated to Networked Storage
81
Benefits of IP Storage






Brings the SAN concept to Ethernet
networks
Lower total cost of ownership
Creates a single integrated network
Makes remote data replication possible
Improves enterprise networks
management
Provides higher degree of interoperability
82
Advantages of IP Storage



Storage access over distance
Transparent to Applications
Leverage Benefits of IP





IT Skills
Ethernet & SCSI Infrastructure
Network Management
R&D Investment
iSCSI
IP Network
Storage Router
GE
FC or SCSI
Universal Access to Storage
Storage appears
local to servers
83
Key Business Trends Favor IP Storage
Network Performance
Overall System Cost
100Gbps
IP Storage
Switches
FC Switches
40Gbps
10Gbps
10Gbps
1Gbps
IP Storage
Switches
1.7Gbps
FC Switches
0.85Gbps
2000
2001
2002
2003
Trained Staff Available
2000
2001
2002
2003
Total Cost of Ownership
IP Storage
Switches
FC Switches
IP Storage
Switches
FC Switches
2000
2001
2002
2003
2000
2001
2002
2003
84
IP Storage Standards
Storage Networking Industry
Association

IETF IP Storage (IPS) Working Group





iSCSI
FCIP
iFCP
iSNS
Storage Networking Industry Association (SNIA)

SNIA IP Storage Forum
85
IP Storage Technologies
What are the technologies? (iSCSI, iFCP, FCIP

iSCSI


iSCSI is a TCP/IP-based protocol for establishing and managing
connections between IP-based storage devices, hosts and
clients
FCIP


)
FCIP is a TCP/IP-based tunneling protocol for connecting
geographically distributed Fibre Channel SANs transparently to
both FC and IP
iFCP

iFCP is a TCP/IP-based protocol for interconnecting Fibre
Channel storage devices or Fibre Channel SANs using an IP
infrastructure in place of Fibre Channel switching and routing
elements
87
IP Storage: iSCSI, FCIP, iFCP
End
Devices
Fabric
Services*
iSCSI
iSCSI/IP
Internet
Protocol
FCIP
Fibre
Channel
Fibre
Channel
iFCP
Fibre
Channel
Internet
Protocol
* Fabric Services include routing, device discovery,
management, authentication, inter-switch communication
88
iSCSI, iFCP and FCIP Protocol Stacks
Applications
Operating System
Standard SCSI Command Set
FCP
New Serial SCSI
FCP
FC-4
FC-4
FC Lower Layers
TCP
TCP
TCP
IP
IP
IP
iSCSI
iFCP
FCIP
89
iFCP
iFCP
iFCP is a gateway-to-gateway protocol for implementing a
fibre channel fabric over a TCP/IP transport
 Traffic between fibre channel devices is routed and
switched by TCP/IP network
 The iFCP layer maps Fibre Channel frames
to a predetermined TCP connection for transport
 FC messaging and routing services are terminated at the
gateways so the fabrics are not merged to one another
 Dynamically creates IP tunnels for FC frames

Ethernet
Header
IP
TCP iFCP
FCP
//
SCSI Data …
CRC
Checksum
91
iFCP Approach
FC
Server
iFCP provides F port to
F port connectivity only
FC Tape
Library
iFCP
Gateway
iSNS
Server
iFCP
Gateway
iFCP
Gateway
IP Network
iFCP
Gateway
FC
Server
FC
JBOD
Device-to-Device
Session
FC Tape
Library
FC
Server
iFCP
Gateway iFCP
Gateway
iFCP
iFCP Gateway
Gateway
iSNS
Server
Device-to-Device
Session
IP Services at individual device level
IETF Standards for Routing, Naming,
Security, QoS, CoS, Discovery (iSNS)
FC
Server
FC
JBOD
92
FCIP
FCIP






FCIP encapsulates FC frames within TCP/IP, allowing islands of FC
SANs to be interconnected over an IP-based network
TCP/IP is used as the underlying transport to provide congestion
control and in-order delivery FC Frames
All classes of FC frames are treated the same as datagrams
End-station addressing, address resolution, message routing, and
other elements of the FC network architecture remain unchanged
IP introduced exclusively as a transport protocol for an inter-network
bridging function
IP is unaware of the Fibre Channel Payload and the FC fabric is
unaware of IP
//
Ethernet
Header
IP
TCP FCIP
FCP
SCSI Data …
CRC
Checksum
94
FCIP Approach—IP Tunneling
FC Tape
Library
FC Server
FC Switch
FC
Switch
Fibre
Channel
SAN
FCIP
Tunnel
FC
Server
FC Tape
Library
FC Switch
IP Network
FCIP
Tunnel
Tunnel Session
FC Switch
Fibre
Channel
SAN
FC Switch
FC Switch
FC
Server
FC Switch
IP Services
Available at Aggregated
FC SAN Level
FC
JBOD
FC
Server
FC
JBOD
FCIP provides E port to E port connectivity
95
iSCSI
iSCSI


iSCSI is a SCSI transport protocol for
mapping of block-oriented storage data
over TCP/IP networks
The iSCSI protocol enables universal
access to storage devices and Storage
Area Networks (SANs) over standard
TCP/IP networks
//
Ethernet
Header
IP
TCP
iSCSI
SCSI Data…
CRC
Checksum
97
iSCSI, iFCP, FCiP
//
Ethernet
Header
IP
TCP FCIP
FCP
SCSI Data …
CRC
SCSI Data …
CRC
Checksum
Ethernet
Header
IP
TCP iFCP
FCP
Checksum
//
Ethernet
Header
IP
TCP
iSCSI
SCSI Data…
CRC
Checksum
98
iSCSI – Cont.

iSCSI (Internet SCSI) specifies a way to
“encapsulate” SCSI commands in a
TCP/IP network connection:
IP
Header
TCP
Header
iSCSI
Header
SCSI commands and data
Explains how to extract
SCSI commands and data
Provides information necessary to
guarantee delivery
Contain “routing” information
So that the message can find its
Way through the network
99
iSCSI Deployment
100
iSCSI Implementations
iSCSI
Client
Native iSCSI Device
IP
Network
iSCSI
Server
iSCSI
Gateway
FC
Switch
Disk
101
Storage Consolidation
NT
Servers
NT
Servers
Tape
Library
RAID
RAID
(Email)
Tape Drive
Switch
Switch
Switch
Switch
RAID
LAN
Tape Drive
Mission-Critical RAID
(Oracle, ERP DB)
SAN
RAID
Tape Drive



Server and LAN bottlenecks

Single points of failure

Poor scalability (management overhead, resource
inefficiencies)
Tape Drives => Tape Library
Departmental => Application-centric disc arrays
102
iSCSI Architecture

Overview



Architectural Model
Features Beyond // SCSI
Issues Beyond // SCSI
103
iSCSI - Layered Model
Initiator I/O System
SCSI
Application
Layer
SCSI Application
Target I/O System
SCSI Application
Protocol
SCSI Device
Server
SCSI CDB
Protocol Service
Interface
iSCSI Protocol
Layer
iSCSI Protocol
Services
iSCSI Protocol
iSCSI Protocol
Services
iSCSI PDU
iSCSI Transport
Interface
TCP/IP
TCP/IP
TCP/IP
TCP/IP
TCP/IP Protocol
TCP/IP
TCP/IP
TCP/IP
TCP segments
in IP
datagrams
iSCSI session
Ethernet
Data link +
Physical
Data link +
Physical
Ethernet
Frame
Ethernet



Replaces shared bus with switched fabric
Transparently encapsulates SCSI CDBs
Unlimited target and initiator connectivity
104
iSCSI Sessions
iSCSI Host
iSCSI Device
iSCSI Session
iSCSI Initiator
iSCSI Target
TCP Connection
TCP Connection
TCP Connection
iSCSI Target
iSCSI Session

Session between initiator and target
One
or more TCP connections per session
Login phase begins each connection


Deliver SCSI commands in order
Recover from lost connections
105
iSCSI Encapsulation
Data Servers
IP
Network
SCSI Initiator
iSCSI Initiator
Ethernet
Header
iSCSI Target
FC
SCSI
Header
DATA
C
R
C
Ethernet
Header
T
I
C
P
P
C
R
C
DATA
T
I
C iSCSI SCSI DATA
P
P
External
Network
End Users
C
R
C
SCSI Target
Fibre Channel SAN
LUNs
106
iSCSI Packet Order
Data Servers
1
2
3
IP
Network
SCSI Initiator
iSCSI Initiator
1
iSCSI Target
1 Target
2
SCSI
3
2
3
Fibre Channel SAN
LUNs
107
iSCSI Packet
//
Ethernet
Header
IP
TCP
iSCSI
SCSI Data…
CRC
Checksum
108
iSCSI Packet
46–1500 bytes
Preamble
Destination Source
Type
Address Address
8
6
6
Well-known
Ports:
21 FTP
23 Telnet
25 SMTP
80 iSCSI
http
5003
IP
TCP
Data
FCS
2
4 Octet
iSCSI
Encapsulated
Opcode
Opcode Specific Fields
Length of Data (after 40Byte header)
Sourced Port
Destination Port
LUN or Opcode-specific fields
Sequence Number
Acknowledgment Number
OffsetReserved U A P R S F
Window
Checksum
Urgent Pointer
Options and Padding
Initiator Task Tag
Opcode Specific Fields
Data Field …
TCP Header
109
iSCSI Commands

SCSI Commands



Command phase
Optional data phase
Response phase

iSCSI Commands

Binds command phase with
associated data into iSCSI
Protocol Data Unit (PDU)
110
iSCSI Architecture Features
Beyond // SCSI

Sessions


Device sharing


Comprises one or more TCP connections
used for fail over and/or link aggregation
Any host on the network can potentially use
the same iSCSI device
Device scalability

Hosts can connect to an effectively limitless
number of iSCSI devices
111
iSCSI Architecture Issues
Beyond // SCSI





Naming, addressing and discovering
Security & Data Integrity
Ordering and numbering
Error handling/recovery
Networking Overhead
112
iSCSI Architecture Issues
Naming, Addressing & Discovery

// SCSI uses a simple NAD scheme:



Devices discovered by polling the bus
Devices given unique id between 0 and 15
iSCSI requires:


Internet addressing
Location independent naming





operation beyond firewalls
multiple addresses to one target
multiple targets behind one address
3rd party commands
Scalable discovery (poll the Internet??)
113
iSCSI Storage Device Discovery Process

1) Host driver requests available iSCSI
targets
from the SCSI router

2) SCSI router sends available iSCSI target
names to host

3) Host logs into iSCSI targets that were
received

4) SCSI router accepts the login and sends
target identifiers to Host (numbers)

5) Host queries targets for device information

6) Targets respond with device information

7) Host creates table of internal devices
114
iSCSI Sequence
Initiator
TCP
Single TCP Session
Establish normal TCP Session
Target
TCP port
5003
0X03 Command—Login
iSCSI Driver
Send Targets
0X43 Login Response—Reject Login Status 1
In text area, list of assessable target names.
Keeps TCP session up.
0X03 Command—Login
List of Target names sent
This
device
has
already
initialized
onto the
Fibre
Channel
0X43 Login Response
Response with target drive mapping
115
iSCSI Architecture Issues:
Security Levels


0: None – ok in controlled environments
1: Initiator and target authentication


2: Digests for header and data integrity


Prevents unauthorized access
Prevents against man-in-middle, insertion,
modification and deletion
3: Encryption (IPSEC)

Prevents against eavesdropping
116
iSCSI Architecture Issues Ordering & Numbering

Unlike // SCSI, iSCSI PDUs may



Arrive out of order (by taking different routes)
Not arrive at all
iSCSI requires

Command numbering


Status numbering


Ordered delivery over multiple connections
Detection of a failed connections
Data sequencing

Detection of missing data PDUs
117
iSCSI Architecture Issues Error Handling & Recovery

// SCSI errors incur costly recovery:



Aborted commands; target, bus and host
resets
OK, because bus errors are infrequent
iSCSI errors will be more frequent




Link failures
TCP failures
Bad “middle box” (firewall, router)
Does the Internet have a “reset” option??
118
iSCSI Architecture Issues Networking Overhead


Software iSCSI can achieve near GbE
wire speed – but at 100% CPU
Traditional TCP stacks are expensive




multiple memory copies
too many interrupts
checksums calculations
We needs TCP offload engines (TOE)
119
iSCSI - TCP Offload
Ethernet
Header





IP
TCP
iSCSI
SCSI Data
CRC
Ethernet frame requires additional CPU processing
Headers must be stripped
Packets ordered
Data copied into memory buffers
CRC checked
120
iSCSI Architecture  Issues
 Networking

TOE

The challenge rests on the TOE vendor



Interrupt host on command boundaries
Offer zero-copy from NIC to app
Eliminate TCP reassembly buffer



Provides true zero-copy
Requires RDMA or synchronization
Proposed IETF solutions for framing


WARP - an RDMA mechanism
Markers – a synchronization mechanism
121
What’s Next for iSCSI




CRC
SLP (Service Location Protocol)
Authentication
Encryption
122
Conclusions
Conclusions
 IP-based
storage will proliferate
 Benefits are strong
 Significant players
 Clear need
 Standards will be established
 Work with industry leaders
124
Backup
iSNS
 iSNS
(Internet Storage Name Server)
 Provides registration and discovery of
SCSI devices and Fibre Channel-based
 In IP-based storage like iSCSI end devices
registered with iSNS
 In iFCP, Fibre Channel-based storage end
devices register with iSNS by a iFCP
gateway
126
iSNS Operation
iSNS
server
FC network 1
FC network 2
Local
iFCP Portal
Server_1
N_port ID
#24
IP
Network
IP address
10.1.2.3
IP address
10.1.2.4
Remote
iFCP portal
Server_2
N_port ID
#24
Problem: Two identical N_port IDs
Solution: Create new ID (based on IP address + N_port ID) = 2422
127
Tracing an iSCSI Block I/O
Server
Database
Application
1
iSCSI Appliance
Application
File I/O requests
2
Operating System
Database System
Raw Partition
Manager
iSCSI Appliance Storage
Storage I/O Bus
File System
Volume Manager
SCSI Device Driver
iSCSI Device Driver Layer
TCP/IPP stack
Network Interface Card
RAID Host Bus Adapter
SCSI Device Driver
iSCSI Device Driver Layer
TCP/IPP stack
Network Interface Card
Device specific requests to TCP/IP network
Block I/O / data / storage location
128
Challenge 1 - TCP Overhead
Consider a SCSI WRITE command. How many times do you think
the data is copied before eventually reaching the target HBA?
Linux Host System
Application
File System
1
Buffer Cache
Linux Target System
SCSI Subsystem
2
iSCSI Host Driver
TCP/IP
Ethernet Driver
Ether
Bridging Software
iSCSI Target Driver
3
TCP/IP
Ethernet Driver
Ether
4
Block Device
Driver
HBA
Application –copy-> Buffer Cache –copy-> TCP/IP –DMA-> Ether (2 copies
1 DMA) Ether –DMA-> Ring Buffer –copy-> TCP/IP –copy-> Bridge –DMA->
HBA (2 copies 2 DMA)
129
TCP Overhead (2)

TCP Processing

Every TCP connection that is part of an iSCSI session has
processing overhead potential


Connection setup / teardown
TCP state machine:







Acknowledge, Timeout, Retransmission
Window management
Congestion Control
TCP segmentation
IP fragmentation
Checksum calculations
Partial or Complete TCP Offload mechanisms are
assumed to be required to make iSCSI performance
comparable to FC
130
Challenge #2 – Framing

Message Boundaries (The Framing - HW-Issue)


iSCSI messages have no alignment relationship with
TCP segments
And TCP does not have a “built in mechanism” for
signaling message boundaries.


IETF considered leverage the urgent pointer for some time
So how can an iSCSI adapter determine where a
message begins and ends??

By reading the length field in the iSCSI header




Determines where in byte stream current message ends and
next begins
NIC must stay “in sync” with beginning of byte stream
Works well in a perfect world (Maybe a SAN or LAN ????)
In a MAN/WAN we have issues


IP Frags leading to out-of-order packet delivery and/or packet
loss
Any “middle box” may fragment an IP packet until, sending each
along potentially different routes
131
Framing (2)

Message Boundaries Continued

THE SCENARIO:


THE ISSUE:


An iSCSI header is not received when expected because the TCP
segment that it was part of was delivered out of order
The receiver does not know where to put the trailing data packets
until the packet with the header arrives
The different options?

Drop all packets until the header arrives


They will be retransmitted
Buffer packets until the header arrives. Then “re-assemble.”


On a 1Gbit WAN link,16MB of buffer memory is required per TCP
connection
On a 10 Gbit WAN link, 125MB of buffer memory required per TCP
connection
132
Framing (3)

Message Boundaries Continued

THE BAD NEWS:


Dropping packets greatly impacts performance and
significantly increases network congestion
Local buffering is expensive and NIC logic is
complex
133
Into – SAN View
Storage Management & Apps
Hosts
Infrastructure
Targets
134
SAN Components

Server Platforms:




Storage Platforms:




Fibre Channel Host Bus Adapters
IP Storage NICs (SNICs)
SAN Software
RAID subsystems
JBOD
Tape subsystems
SAN Interconnect:




Fibre Channel hubs and switches
IP Storage switches
SAN-to-SCSI bridges
MAN and WAN gateways
135
SAN, NAS, iSCSI Comparison
DAS
SAN
iSCSI
iSCSI
Appliance Gateway
NAS
Computer System
Application
Application
Application
Application
Application
File System
File System
File System
File System
File System
Volume Manager
Volume Manager
Volume Manager
Volume Manager
SCSI Device Driver
iSCSI Driver
SCSI Device Driver
iSCSI Driver
I/O Redirector
NFS/CIFS
TCP/IP stack
NIC
SCSI Device Driver
SCSI Device Driver
SCSI Bus Adapter
Fibre Channel HBA
TCP/IP stack
TCP/IP stack
NIC
NIC
File I/O
Block I/O
SCSI
SAN
IP
IP
IP
FC
NIC
TCP/IP stack
iSCSI layer
Bus Adapter
NIC
TCP/IP stack
iSCSI layer
Bus Adapter
NIC
TCP/IP stack
File System
Device driver
FC switch
Block I/O
136
137
Potential Outcomes and
Success Probability
138
I/O Adapters “Data Movers”
Intel and other vendors will have
ONE Ethernet Wire
for
ALL Storage & LAN Traffic
I/O Block
Data
GbE
R
010101
Port
LAN Data
010101
139
Storage
Functions/Applications

Current Functions/Applications






Storage Consolidation
Tape Backup
Clustering
Replication
Disaster Recovery
New Capabilities with IP Storage



SAN Extension
QoS
Security
140
LAN-free Tape Backup
Users
Servers
RAID

SAN Switch
SAN Bridge Tape Subsystem
SAN Advantages for LAN-free Tape Backup:




Removes backup traffic from the LAN
Tape becomes SAN shared resource
High performance SAN infrastructure
SCSI attached via SAN bridge
141
Remote Backup Application
NT
Server
Backup Server :
• Veritas Shared Storage Option
• Tivoli Storage Manager
Tape
Library
NT
Server
RAID
(Email)
HBAs
LAN
Mission-Critical RAID
(Oracle, ERP DB)
RAID
GE, 10GE ( iSCSI, iFCP )
Fibre Channel
SCSI
iSCSI
Servers

Tape
Library
Allows customers to move archiving off-site for higher
disaster protection
142
Server Clustering
Users
Heartbeat
Servers
RAID

SAN Switch
RAID
SAN Advantages for server clustering:




Server access to common storage resources
Failure of a single server still provides data access
Scalable to > 30 servers in a cluster
Simplified storage resource management
143
SAN Extension: Replication over WAN
NT
Server
Tape
Library
NT
Server
RAID
(Email)
HBAs
LAN
IP WAN
RAID
RAID


iSCSI
Servers


Tape
Library
Unified Management of Data Center and WAN storage routers
Not vulnerable to disruption at a local SAN
IP WAN Link
Leverage current infrastructure
GE, 10GE ( iSCSI, iFCP )
Fibre
Channel
Expandable to iSCSI devices
SCSI
(OC-3, T1, etc)
144
TCP/IP Layers
TCP/IP Protocols
OSI Model
TCP/IP layers
7
FTP
Telnet HTTP SNMP
TFTP
Process layer
6
5
4
TCP/IP
UDP
Connection oriented
Connectionless oriented
Host to host
layer
3
IP
Internet layer
2
LAN/WAN
Network
access layer
1
Ethernet, token ring, ATM, Frame Relay, FDDI
145
Internet Protocol
Storage Area Networks (IP
SAN)
IP Storage Area Networks
Upon completion of this module, you will be able to:
 Describe the benefits of IP SAN.
 Describe IP convergence in the SAN and its
implications.
 Describe and discuss the basic architecture of
 FCIP
 iFCP
 iSCSI
 Explain potential applications of IP SAN
technology.
- 147
In this module …
This module contains the following lessons:
 IP SAN Overview.
 IP SAN Protocols.
 Applications of IP SAN.
- 148
Lesson: IP SAN Overview
Upon completion of this lesson, you will be
able to:
 Describe the benefits of IP SAN.
 Describe the IP convergence in the SAN
and its implications.
 List the three common IP SAN approaches.
 List the three deployment models
(topologies) for IP SAN.
- 149
Introduction

Traditional SAN technology is built around
Fibre Channel.
Servers
Servers
Switch
Switch
Storage
Storage
Servers

= IP
= FC
Servers
IP technology is emerging as an alternative
or supplemental transport for storage traffic.
Block Storage over IP –
Protocol options

iSCSI

SCSI over IP



IP
IP
FCIP

Fibre Channel-to-IP
bridge / tunnel (point
to point)


IP FC
IP encapsulation done on
host / HBA(host bus
adapter)
Hardware-based gateway
to Fibre Channel storage
IP as the inter-switch
fabric

IP FC
IP
Fibre Channel end points
iFCP

FC IP
Fibre Channel end points
FC/
IP
IP/
FC
IP
FC/
IP
- 151
IP/
FC
IP Storage Approaches
FCIP
iFCP
iSCSI
Servers
FC
FC
FCIP Router
IP
Network
iFCP Switch
IP
Network
FCIP Router
FC
IP
Network
iFCPSwitch
FC
iSCSI/FC
Gateway
FC
Storage
- 152
IP
Network
IP Storage Deployment
Models
= IP
= FC
Native
Ÿ
Ÿ
Ÿ
All Ethernet (No Fibre Channel)
iSCSI Protocol
Ethernet Switches & Routers
Bridging
Ÿ
Ÿ
Ÿ
Servers - Ethernet Attached
Storage - FC Attached (SAN or DAS)
iSCSI Protocol
= IP
= FC
Extension
Ÿ
Ÿ
Ÿ
Servers & Storage - SAN Attached
FCIP or iFCP Protocol
SRDF
- 153
= IP
= FC
Market Drivers for SAN
Internetworking



Fibre Channel SAN challenges.
IP SAN enablers.
Easy to leverage IP equipment and
expertise to help manage data in
conjunction with Fibre Channel SANs.
- 154
Benefits of IP SAN


Cost Effective
Extend the reach of a SAN
- 155
IP is Cost Effective


Most organizations already have IP
networks and familiarity with traditional
network management.
Leverages existing Fibre Channel
applications.
- 156
Extend the Reach of Your
SAN



Standard Fibre Channel Distances.
IP Extends Fibre Channel applications
over regional/global distances.
At higher link speeds, IP can handle
synchronous applications.
- 157
Lesson Summary
Topics in this lesson included:
 Describe the benefits of IP SAN.
 Describe the IP convergence in the SAN
and its implications.
 List the three common IP SAN
approaches.
 List the three deployment models
(topologies) for IP SAN.
- 158
Lesson: IP SAN Protocols
Upon completion of this lesson, you will be
able to:
 Describe and discuss the basic
architecture of



FCIP
iFCP
iSCSI
- 159
Fibre Channel over IP - FCIP



Encapsulates FC frames in IP packets.
Creates virtual FC links that connect
devices and fabric elements.
Includes security, data integrity, congestion
and performance specifications.
SCSI Data
FCIP Encapsulation
IP
TCP
FCIP
Header Header Header
IP Payload
IP Datagram
- 160
EOF
FC
Header
CRC
SOF
Fibre Channel Frame
FCIP Benefits

FCIP

Best of both technologies
IP
Fibre Channel
• Widely available
• Low latency
• High reliability
• Off-the-shelf solutions
• Mature standards



•
•
•
•
•
Widely available
Accepted technology
Trained user base
Affordable
Mature standards
Support for existing applications
Cost effective
-Multi-point networking
161
Internet Fibre Channel
Protocol
iFCP
 Gateway-to-gateway protocol
IP switches & routers replace FC switches
 Transparent to FC drivers

 FC

transport uses TCP connections
Point-to-multipoint networking possible
SCSI Data
iFCP Address Translation &
Encapsulation
IP
TCP
iFCP
Header Header Header
- 162
IP Payload
EOF
FC
Header
CRC
SOF
Fibre Channel Frame
iFCP Benefits



Works with wide range of devices.
Flexible.
Less potential bottlenecking vs. FCIP.
- 163
iFCP Maps FCP to an IP
Fabric
FC Tape Library
FC Loop Disks
FC Server
iFCP Gateway
iFCP Gateway
IP Network
iFCP Gateway
iFCP Gateway
FC Switch
- 164
Device-to-device
session
Addressing and Routing



Local Mode.
Address Transparent Mode.
Routing between gateway regions is IPonly.
- 165
iFCP
Servers
Gateway
Servers
Gateway
Switch
Switch
SAN A
SAN B
The two fabrics
remain separate
Storage
- 166
Storage
iSCSI



A method to transfer blocks of data using
the TCP/IP network.
Serialized service delivery subsystem.
SCSI protocol over IP.
- 167
iSCSI Model Layers
Storage
SCSI Data
SCSI encapsulation
IP
TCP
iSCSI
Header Header Header
IP Payload
IP Datagram
- 168
iSCSI Storage Models


Native
Bridging
Switch
Storage
Gateway
Servers
= IP
= FC
- 169
Controlling IP SANS
A
iSNS
Target Z
B
Target Z - Device A – Initiator A
Target Z - Device B – Initiator B
Target Z - Device C – Initiator C
IP Network
Initiator A
Initiator C
Initiator B
- 170
C
Internet Storage Name Server
Overview




iSNS is a client/server model.
The iSNS server is passive.
iSNS clients register & manipulate the
objects in the iSNS server.
An iSNS server can be hosted on a target,
initiator, or stand-alone server with a
specified IP address.
- 171
iSNS Example
Management Platform
Storage
Security
Keys
Device B
iSNS
Discovery
Domains
Host A
Device A
Storage
Host C
Host B
- 172
iSCSI Nodes



A single Initiator or Target
Names are assigned to all Nodes
Independent of address
- 173
Architectural Model
Network
Portal
10.1.1.1
iSCSI
Node
Initiator
IP Network
Network
Portal
10.1.1.2
Network Entity
iSCSI Client
- 174
Network
Portal
10.1.2.1
TCP port
3260
Network
Portal
10.1.2.2
TCP port
3260
iSCSI
Node
Target
Network Entity
iSCSI Server
FCIP, iFCP and iSCSI Comparison
FC
FCIP
iFCP
iSCSI
Simple Name
Service
Simple Name
Service
Manual/iSNS/SLP
Manual/iSNS/SLP
IP Routable
-
Per Virtual ISL
Per Connection
Per Connection
TCP Session Basis
-
Manual definition
of virtual ISL path
Dynamic by nodeto-node login
Dynamic by nodeto-node login
Intermediate
Long
Unlimited
Unlimited
FC Content
-
Full Frames
FCP
None
Transmitted
Overhead
Low
High
Medium
Medium
Conversion
Overhead
-
Low
Medium
High
Resiliency
Medium
Low
High
High
Fabric Contiguity
YES
YES
NO
NO
Port Aggregation
YES
YES
YES
YES
Medium
Low
High
High
Node Discovery
Relative Distance
Scalability
Lesson Summary
Topics in this lesson included:
 The basic architecture of FCIP.
 The basic architecture of iFCP.
 The basic architecture of iSCSI.
- 176
Lesson: IP SAN Applications
Upon completion of this lesson, you will be
able to:
 Describe common applications of IP SAN
technology such as:



Remote Backup and Restore
Remote Data Replication
Storage Consolidation
- 177
Remote Backup and Restore
Disks
Server
Servers
FC
FC
IP Network
Storage
FC
Disks
Server
Disks
Remote Data Replication
Disks
Server
Servers
FC
FC
IP Network
Storage
FC
Disks
Server
- 179
Disks
Storage Consolidation/Dual
Purpose IP Network
Servers
Disks
Servers
Switch
Switch
Servers
IP Network
Disks
Disks
Switch
Switch
Disks
Disks
- 180
Disks
Lesson Summary
Topics in this lesson included:
 Remote backup and restore
 Remote data replication
 Storage consolidation
- 181
Module Summary
Topics in this module included: :
 The benefits of IP SAN.
 The IP convergence in the SAN and its
implications
 The basic architecture of FCIP
 The basic architecture of iFCP
 The basic architecture of iSCSI
 Application of IP SAN technology
- 182
 Check Your Knowledge



What are the benefits of IP SAN?
What are the three common protocols for
IP SAN?
What data storage application could
benefit from an IP SAN solution?
- 183
The TCP/IP Protocols
Objectives


Describe how the TCP/IP protocol stack is
organized compared to the ISO/OSI
protocol stack
Discuss the different protocols that make
up the TCP/IP suite


Includes IP, TCP, UDP, ARP, and RARP, as
well as support protocols DNS, BOOTP, TFTP,
DHCP, ICMP, SMTP, HTTP, HTTPS, and NTP
Show how TCP/IP data are encapsulated
inside a hardware frame for transmission
185
Objectives (cont’d.)




Describe the relationship between IP
addresses and MAC addresses
Discuss the role of the PING, TRACERT,
NBTSTAT, and NETSTAT applications
Explain the new features provided by IPv6
Describe the purpose and use of a
protocol analyzer
186
Introduction

TCP/IP network model contains five
layers
Figure 8-1 ISO/OSI and TCP/IP protocol stacks
187
Introduction (cont’d.)

Data packet transmitted from source
computer

Contains an informational header for each
layer
Figure 8-2 TCP/IP stack layering
188
Introduction (cont’d.)

TCP/IP supports many protocols
Figure 8-4 TCP/IP protocol suite
189
RFCs


Describe TCP/IP protocols in detail
Official Internet standards

Published as electronic documents
Table 8-2 Several important TCP/IP RFCs
190
IP

TCP/IP protocol suite base layer


Used at the TCP/IP stack Internet layer
Datagram: unit of packaged TCP/IP data

Encapsulates all TCP/IP protocols


Exception: address resolution protocols
Eventually encapsulated inside a hardware
frame
Figure 8-5 An IP datagram encapsulated into an Ethernet frame
191
IP (cont’d.)

Basic IP datagram transmission

Source computer sends packets via local LAN



IP datagram pieces treated as independent
entities



Forwarded to destination
Router forwards packet to another router or local
LAN computer
May take different routes
Destination computer may receive packets out of
order
IP is unreliable

Offers best effort delivery
 No
192
guarantees data delivery
IP (cont’d.)

Congested network



Datagrams discarded
ICMP contains error message: may/may not be
returned
IP datagram encapsulated in hardware
frame


Uses local network maximum transmission unit
(MTU): typically 1500 bytes
Fragmentation occurs when different MTUs
encountered
Figure 8-7 ICMP encapsulated in an IP datagram
193
IP Addresses

Routes IP datagram across network


Uses dotted decimal notation
Assigned by software (statically or
dynamically)


Must be unique
Differs from MAC addresses

Fixed 48-bit addresses encoded into Ethernet
controller hardware
Figure 8-11 Class C network IP address
194
IP Addresses (cont’d.)

Five Internet address classifications

Each contains a network ID, host ID
Figure 8-12 IP address classes
195
IP Addresses (cont’d.)


Some IP addresses reserved
IP version 4 addressing scheme
shortcoming


Unused hosts in Class A, Class B
Shortage of all Internet addresses
Table 8-4 Special IP addresses
196
TCP

Allows two computers to communicate
together over interconnected networks

Connections established through predefined
ports, sockets


Reliable, error checking, packet received
acknowledgment, packet sequencing
Provides communication link between
application program and IP

Uses a set of function calls

Provides application process different options
197
TCP (cont’d.)

TCP header and associated data

Encapsulated into IP datagram and hardware
frame
Figure 8-13 TCP datagram format
198
TCP (cont’d.)

Primary purpose of TCP


Provide guaranteed delivery on top of IP
Areas requiring attention to guarantee
delivery:


Data transfer, reliability, flow control, multiplexing,
connections, message precedence and security
Port numbers


Data communicated between two TCP
applications utilizes specific ports
Port


Associated with network socket, used by the
application
Pair of sockets uniquely identify network
connection
UDP

Connectionless and unreliable

Many applications do not require TCP
handshaking overhead and complexity

Domain Name Service (DNS), Dynamic Host
Configuration Protocol (DHCP), network games
Figure 8-16 UDP datagram format
200
ARP and RARP

Address Resolution Protocol (ARP)

Uses a directed broadcast message


ARP messages



Obtains MAC address for a given IP address
Not encapsulated within an IP datagram
Encapsulated directly into an Ethernet frame
Not routable


Locates machines on local network
Internet traffic forwarded to the default gateway
201
ARP and RARP (cont’d.)

Reverse Address Resolution Protocol
(RARP)


Provides IP address for a specific MAC
address
RARP message format


Same as ARP message
ARP application program runs from DOS
prompt

Displays/manages IP/MAC address table
202
ARP and RARP (cont’d.)
Figure 8-18 ARP message in an
Ethernet frame
203
Figure 8-19 Using ARP and RARP
TCP/IP Support Protocols



Domain Name Service (DNS)
 Converts host name to IP address (vice versa)
 Makes it easier to remember an address
Bootstrap Protocol (BOOTP)
 Routable UDP protocol
 Allows diskless workstation to:
 Discover its own IP address
 Discover IP address of a BOOTP server
 Specify file to be downloaded (via TFTP) and
executed
Trivial FTP: simple file transfer protocol
 Used when user authentication not necessary
TCP/IP Support Protocols
(cont’d.)

DHCP

Dynamically assigns IP addresses to network
devices during bootstrap process


ICMP


Simplifies network administration
Contains error message for IP datagram error
SMTP

Routes electronic mail on the Internet using
TCP, IP
205
TCP/IP Support Protocols
(cont’d.)




SNMP
 Defines format and meaning of messages exchanged
by manager and agents
HTTP
 Transfers multimedia information over the Internet
HTTPS
 Exchanges encrypted Web pages between a
client/server connection
NTP
 Synchronizes the time of multiple network computers
206
TCP/IP Applications
Table 8-12 Comparison of different network utilities
207
TCP/IP Applications (cont’d.)

FTP

Allows users to:



Log in to a remote computer
Transfer files back and forth through simple
commands
Telnet

Allows user to:

Establish remote computer terminal emulation
connection
208
IP Version 6 (IPv6)

Next generation Internet


Resolves many problems in IPv4
Changes from IPv4

Address: 128 bits



Accommodates foreseeable network growth
Different header formats, new extension
headers, support for audio and video
All possible protocol features not specified


Allows for adding new features
No need to update the protocol
209
IP Version 6 (cont’d.)

IPv6 addresses


128-bits long, written using colons (:),
Uses 4-digit groups of hexadecimal numbers



Hexadecimal group represents 16 bits of binary
addressing information
Shorthand technique eliminate groups of 0000
patterns: double-colon (::)
Composed of network, host addressing bits


Subnet mask determines network associated with a
specific IP address
Can attach subnet mask IP address
210
Protocol Analyzers

Hardware or software devices



Listen to network traffic
Capture packets for examination
Double as cable testers
211
Troubleshooting Techniques

Windows ipconfig utility and Linux ifconfig
utility


Displays various IP address information
Network utilities



Handy for many reasons
On the spot troubleshooting
Can be combined to resolve issues
212
Summary








TCP/IP protocol model contains five layers
Internet Protocol (IP): TCP/IP base layer
 Provides best effort delivery
 Connectionless communication
TCP/IP data packaged in units called IP datagrams
TCP: reliable and connection oriented
UDP: unreliable and connectionless
ARP and RARP obtain MAC and IP addresses
IPv6 adds addressing improvements to IPv4
Many protocols and tools available
213
Network Management and Security
Objectives






Discuss the various elements of network
management
Describe the elements of network security
Identify the main components of Disaster
Recovery
Explain the characteristics of Storage
Management
Differentiate between various encryption
and authentication techniques
Describe the purpose of fault tolerance
215 tunneling
and
Network Management

Important task in a networked computer
environment

Function performed by:


Maintain current network view (status)
through:




System administrator, network manager, or
engineer
Planning essential network elements and
categories
Record keeping
Review and select new network technology
Perform research and invest time for success

Read trade magazines, attend conferences, and
make friends with other IT professionals
Disaster Recovery

Computer lab or network operations center


Many potential disasters
Disaster recovery

Planning and methodologies





Reduce or eliminate threats; recover from disaster
Identify critical systems first
Implement safeguards for quick, painless
recovery
Get business running again after disaster
Includes Service Level Agreements (SLAs)
217
 Specific
service level guarantees in the event of
Disaster Recovery (cont’d.)

Business continuity plan


Looks at long-term business operations
Specifies alternate site use in the event of
disaster





Cold site: electricity, space for equipment,
bathrooms
Warm site: minimal networking hardware, no data
Hot site: all equipment required, current data
Downtime affects accessibility over time
Operating systems include backup tools
 218
Full
backup
Disaster Recovery (cont’d.)
Table 11-1 Uptime* percentage
and its effect on downtime
Figure 11-3(a) Windows XP Backup Utility
219
Disaster Recovery (cont’d.)



Fault tolerance
 System ability to withstand hardware or software fault
 Keep functioning
Increase fault tolerance by:
 Having backup archives
 Using RAID drives
 Using redundant hardware
 Utilizing multiple, different Internet connections
Money spent on fault tolerance
 Worth the expense on networks requiring high
reliability
220
Figure 11-4 Fault-tolerant network topology
221
Disaster Recovery (cont’d.)

Protocol analyzers




Report computer network traffic type and
usage
Quickly identify network problems
Proactively monitor and plan future network
growth
Stand-alone hardware unit
Connects to network or software program running
on a network host
 Collects statistical network performance
information
 Statistics converted into graphical real-time views
222

Figure 11-5 WireShark window showing protocol statistics
223
Disaster Recovery (cont’d.)

Simple Network Management Protocol (SNMP)
 Gathers statistics on a device-by-device basis
 Network devices categorized as managed
 Include hubs, switches, routers, and other network
devices
 Agents gather network statistics
 Management stations report on data
 Popular due to its power, ease of use
 Four operations (GET, SET, GET-NEXT, TRAP)
 Three SNMP information categories define scope
 SNMP protocol, Structure of Management
Information rules, Management Information Base
224
Network Security

Two categories



Methods to secure data
Methods to regulate data transmission
Threats

Many types





Interruptions
Interceptions
Modifications
Fabrications
Controlled by security measures
225
Table 11-3 Different types of security
measures
Figure 11-6 Typical information exchange scenarios between A and B
226
Network Security (cont’d.)

Viruses
 Software
 Enter computer system; corrupt the hard drive files
 Exhibit various behaviors
 Important to have system, user file backup archives
 Stored appropriately
 Four main category types
 Boot sector, file or program, macro, multipartite
viruses
 Latest virus types: stealth, polymorphic, armored
 Also have worms, Trojan horses, and rootkits
227
Network Security (cont’d.)

Network sniffers


Similar to protocol analyzer
Capture and decode network traffic


Operates in passive mode



Passwords, trade secrets, and other proprietary
information
Device attached to the network
Silently collects information
One of the worst security breach types

No one knows network security compromised
228
Network Security (cont’d.)

Plain-text encryption

First line of defense in protecting network data



Prevent plain-text password exchange on the
network
Prevents sensitive information disclosure
NetBIOS plain-text encryption


Not difficult to crack
Provides easy, small measure of security to
networked resources
229
Table 11-4 Common terminology used in computer and
network security
230
Network Security (cont’d.)

Kerberos


Uses secret-key ciphers for encryption,
authentication
Authenticates requests for network resources


Does not authenticate document ownership
SSL


Facilitates secure Internet communication
Uses public key encryption



Encrypts data before transmission
Supported by most browsers
Enabled when URL specifies https protocol
231
Network Security (cont’d.)

Public-key encryption

Public key


Private key



Encrypts transmitted
Decrypts data
Eliminates problem when same key used to
encrypt and decrypt data
More secure data encryption and decryption
method


Public key posted for public access
Private key guarded very carefully
232
Network Security (cont’d.)

Certificates of authority



Guarantee e-mail sender’s identity
Available for enterprises, small businesses,
and home use
Provides opportunities:




Manage Web site domain names, server
certificates
Safeguard network resources using public-key
encryption
Secure network applications (e-mail, messaging)
Enable e-commerce applications supporting online
payments, purchases
233
Network Security (cont’d.)

Pretty Good Privacy (PGP)

Provides confidentiality and authentication
services


Important elements:




Used with electronic message, file storage
applications
Uses best cryptographic algorithms
Unlimited source code, documentation distribution
Not controlled by government, standards
organization
Support mechanisms
 Digital signatures,
234compression
message encryption,
Network Security (cont’d.)

Masquerading



Spoofing attacks
IP masquerading (IP spoofing)
One computer forwards information from one
or more other computers


Using IP addresses not officially assigned
Firewalls

Hardware device or software program


Examines information packets
Determines if communication exchange occurs
 235
Packet-filtering,
proxy-service, stateful-
Network Security (cont’d.)

Virtual LANs (VLANs)

Partitioning VLAN-capable switch ports into
LAN groups


Isolates traffic
Intrusion Detection Systems


Combines traffic sniffing with analysis
techniques
Network-based Intrusion Detection (NIDS)


Centralized approach
Host-based Intrusion Detection (HIDS)
 Decentralized
236
approach
Network Security (cont’d.)


Proxy Server
 Server-type application program
 Sits between a user workstation and the Internet
 Monitors, logs network activity
 Used with a gateway to isolate LAN from the Internet
 Used with a firewall to protect network from intruders
IP Security (IPSec)
 Secures communications across a LAN, between
public, private networks, across the Internet
 Incorporated into TCP/IP protocol stack
 Below UDP, TCP transport protocols
237
Network Security (cont’d.)

Tunneling

Uses public network infrastructure


Data encapsulated


As part of a Virtual Private Network
Encrypt original source address, destination
address, payload data
Layer 2 Tunneling Protocol (L2TP)

Combination of protocols



Point-to-Point Protocol (Microsoft)
Layer 2 Forwarding Protocol (Cisco)
Standard tunneling protocol for VPNs
238
Network Security (cont’d.)

Denial-of-Service Attacks (DOS attack)

Prevents legitimate users from using a
service


Three basic attack types:




Essentially disables a computer or network
Consumption of limited resources
Destruction or alteration of configuration files
Physical destruction or modification of network
components
Hardening

System: process vulnerabilities reduced or
239
eliminated
Storage Management

Network Attached Storage (NAS)


High-capacity file storage directly connected
to the network
NAS devices characteristics:




May use hard drives or writable CD-DVDs to store
data
Fault tolerance possible using RAID technology
Communication via TCP/IP and IPX/SPX
Storage Area Network (SAN)
Uses a special switch to connect servers, storage
devices on their own physical network
 One server acts as a gateway to the SAN
240

Storage Management
(cont’d.)

User management

Areas to consider








Acceptable Use Policy
Logon Policy
Password Policy
Access Control
Security Awareness Training
Auditing Policy
User Termination Policy
All areas require enforcement
241
Troubleshooting Techniques

Network management


Requires patience and determination
Several different possible troubleshooting
choices



Each choices may provide a workable solution
No one choice better than the others
Network security
May involve significant research
 Requires a good normal system activity
baseline
 242
Most important tool

Summary

Network management


Managing network technologies to provide a
cost-efficient, reliable, and secure computer
network
Disaster recovery

Planning and methodologies



Reduce or eliminate threats
Recover when disaster occurs
Network security
Methods used to secure data
 243
Methods used to regulate transmitted data

IP SAN
Module Objectives
Upon completion of this module, you will be
able to:
 Identify components of IP SAN
 Describe benefits of IP SAN
 Describe IP convergence in the SAN and
its implications
 Describe basic architecture of



iSCSI
FCIP
FCoE
- 245
Lesson: IP SAN Protocols
Upon completion of this lesson, you will be able to:
 Discuss market drivers for IP SAN technology
 List IP SAN technologies
 List components and connectivity options of
iSCSI
 Describe iSCSI architecture and topologies
 Explain iSNS operation
 Describe architecture of FCIP
- 246
Driver for IP SAN Internetworking



In FC SAN transfer of block level data takes
place over Fibre Channel
Emerging technologies provide for the
transfer of block-level data over an existing
IP network infrastructure
IP is being positioned as a storage transport
because:



Easier management
Existing network infrastructure can be leveraged
Reduced cost compared to new SAN hardware
and software
- 247

Block Storage Over IP –
Protocol
Options
iSCSI
Storage Array
iSCSI Gateway
SCSI over IP
 IP encapsulation
 Ethernet NIC card
 iSCSI HBA
 Hardware-based gateway
to Fibre Channel storage
 Used to connect servers
FCIP
 Fibre Channel-to-IP bridge /
tunnel (point
to point)
 Fibre Channel end points
 Used in DR implementations


IP
FC Port
Server
iSCSI HBA
iSCSI Port
(a) iSCSI Implementation
IP
Server
FC Port
FC HBA
(b) FCIP Implementation
- 248
Storage Array
Storage Array
What is iSCSI ?



IP based protocol used to connect host
and storage
Carries block-level data over IP-based
network
Encapsulate SCSI commands and
transport as TCP/IP packet
- 249
Components of iSCSI



iSCSI host initiators
 Host computer using a NIC or iSCSI HBA to connect
to storage
 iSCSI initiator software may need to be installed
iSCSI targets
 Storage array with embedded iSCSI capable network
port
 FC-iSCSI bridge
LAN for IP storage network
 Interconnected Ethernet switches and/or routers
- 250
iSCSI - Host Connectivity
Options

Three options for iSCSI configuration:

Software Initiators


TCP Offload Engine (TOE)


Code that can be loaded onto a host to provide the
translation between the storage I/O calls and the
network interface
Moves the TCP processing load off the host CPU
onto the NIC card, to free up processing cycles for
application execution
iSCSI HBA

A network interface adapter with an integrated
SCSI ASIC (application-specific integrated circuit)
Simplest option for boot from SAN
- 251
iSCSI Protocol Stack
OSI Model
iSCSI Initiator
iSCSI Target
Layer 7 Application
SCSI
Commands and Data
SCSI
Layer 5 Session
iSCSI
Login and Discovery
iSCSI
Layer 4 Transport
TCP
Windows and Segments
TCP
Layer 3 Network
IP
Packets
IP
Layer 2 Data Link
Ethernet
Frames
Ethernet
Interconnect
Ethernet
- 252
IP
TCP
iSCSI
SCSI
Data
iSCSI Topologies



Native iSCSI Connectivity
Bridged iSCSI Connectivity
Combining FCP and Native iSCSI
Connectivity
- 253
Native iSCSI Connectivity
Native iSCSI Port
IP
Servers
iSCSI HBA


Storage Array
No FC components
Each iSCSI port on the array is configured
with an IP address and port number

iSCSI Initiators Connect directly to the Array
- 254
Bridged iSCSI Connectivity
iSCSI Gateway
Storage Array
IP
Servers
iSCSI HBA
FC SAN
FC Port
FC HBA



Bridge device translates iSCSI/IP to FCP
 Standalone device
 Integrated into FC switch (multi-protocol router)
iSCSI initiator/host configured with bridge as target
Bridge generates virtual FC initiator
- 255
Combining FCP and Native
iSCSI Connectivity
Native iSCSI Port
IP
Servers
iSCSI HBA
FC SAN
Storage Array
FC Port
FC HBA


Array provides FC and iSCSI connectivity
natively
No bridge devices needed
- 256
Internet Storage Name Server


For iSCSI communication, initiator must
discover location of the target on a
network
iSCSI discovery take place in two ways:

SendTargets discovery


Internet Storage Name Service (iSNS)



initiator is manually configured with the target
Initiators and targets automatically register
themselves with iSNS server
iSNS is a client/server model
An iSNS server can be hosted on a target,
initiator,
or stand-alone server with a
257
iSNS Operation
A
iSNS
Target Z
Target Z - Device A – Initiator A
B
Target Z - Device B – Initiator B
Target Z - Device C – Initiator C
C
IP
Initiator A
Initiator C
Initiator B
- 258
iSCSI Names


All initiators and targets require a unique
iSCSI identifier
Two types of iSCSI names

iqn.: iSCSI Qualified Name

iqn.2008-02.com.example:optional_string


Example: iqn.1992-05.com.emc:apm000339013630000-10
eui.: Extended Unique Identifier

eui.0300732A32598D26
- 259
iSCSI Error Handling and
Security

The error detection and recovery in iSCSI can be classified into
three levels:
 Level 0
 If SCSI session is damaged, all TCP connection needs to be
closed
 Session restart via repeat login
 Level 1
 Each node selectively recover damaged packet
 Identification of error and data recovery at SCSI task level is
performed
 Attempt to repeat the transfer of lost or damaged packet
 Level 2
 New TCP connections are opened to replace a failed
connection
 New connection picks up where old one failed
- 260
What is FCIP (Fibre Channel
over
IP)
 FCIP is an IP-based storage networking



technology
Combines advantages of Fibre Channel and
IP
Creates virtual FC links that connect
devices in a different fabric
FCIP is a distance extension solution

Used for data sharing over geographically
dispersed
SAN IP
FC SAN
FC SAN
Host
Storage Array
- 261
FCIP Frame


Encapsulates FC frames in IP packets
 FCIP Router is used for encapsulation
 FC Router at other end removes IP wrapper and
sends FC data to other fabric
Includes security, data integrity, congestion and
performance specifications
FC Frame
SOF
FC
Header
SCSI Data
CRC
FCIP Encapsulation
FCIP Frame
- 262
IP
Header
TCP
Header
FCIP
Header
IP Payload
EOF
Lesson Summary
Key points covered in this lesson:
 iSCSI components
 iSCSI frame structure and topologies
 iSNS operation
 iSCSI error handling and security
 Architecture of FCIP
- 263
Lesson: Fibre Channel over
Ethernet (FCoE)
Upon completion of this lesson, you will be
able to:

Discuss the FCoE and its benefits

Describe how FCoE works

Describe FCoE physical and logical
elements

Compare different protocol stack
- 264
Fibre Channel over Ethernet (



FCoE)
A new protocol that maps Fibre Channel protocol
natively over Ethernet
Based on two new standards that are currently in active
development:
 FCoE standard, being developed by T11 Fibre
Channel Interfaces Technical Committee
 Enhanced Ethernet standard, being developed by the
Ethernet IEEE Data Center Bridging Task Group
 Both standards are expected to be ratified in 2009
Enables the consolidation of SAN traffic and Ethernet
traffic onto a common 10 Gigabit network infrastructure
- 265
FCoE: Benefits

Lower capital expenditure



Reduced power and cooling requirement
Enabler for consolidated network
infrastructure



Dramatic reduction in the number of adapters,
switch ports and cables required
Potentially lower administration cost, with
convergence of LAN and SAN
Effective sharing of high-bandwidth links
Lower Total Cost of Ownership (TCO)
- 266
I/O Consolidation with FCoE
CNA
LAN Ethernet
FC HBA
SAN (FCoE)
SAN (FC)
LAN
(Ethernet)
NIC
CNA
Typical Server Connectivity in data
center environments
- 267
Server with CNA
supporting both SAN and LAN
FCoE - Physical Elements

Host Interface: CNA
(Converged Network Adapter)



PCIe card on host
consolidates NICs and
HBAs
Provides a 10 Gigabit
Ethernet link that
carries consolidated
traffic
FC
10Gbps
Ethernet
/FCoE
Menlo
ASIC
10 GE
CNA
10 Gbps connectivity options:
Host to FCoE switch

Option1: Copper-based
Twinax

- 268
Cost effective option
Twinax Cable
PCIe
Bus
Infrastructure - Before FCoE
………
LAN
Storage
Arrays
IP Edge
Switches
FC
Switches
…………
Hosts
- 269
Infrastructure – With FCoE
…………
LAN
Storage Arrays
FCoE Switch
FCoE links
…………
Hosts
- 270
FCoE – Frame Structure
Normal Ethernet Frame: Ether Type = FCoE
Control Information: Version and Ordered Sets (SOF and EOF)
24 Bytes
16 Bytes
12 Bytes (MAC Addresses) + 4 Bytes
(802.10 TAG)
- 271
FCS
EOF
Fibre Channel Payload
CRC
FC
Header
FCoE
Header
Ethernet
Header
Same as a Physical Fibre Channel Frame
4 Bytes
Up to 2112 Bytes
1 Byte(EOF) + 3 Bytes (Padding)
Lossless Ethernet

No frame drop due to congestion or buffer
overflow



Fibre Channel manages congestion through link
level, credit based flow control
Ethernet uses drop flow control method which is
not lossless
PAUSE capability of Ethernet is used to
achieve the lossless fabric
Busy receive portEthernet
can traffic
send the control
frame
to
Receiver
–
Port
B
Sender – Port A
the transmit port forFRAMES
pause in transmission

PAUSE
- 272
QUEUE
FULL
Network Stack Comparison
SCSI
SCSI
SCSI
SCSI
SCSI
iSCSI
FCP
FCP
FCP
FC
FC
FC
Less Overhead
than FCIP, iSCSI
FCIP
TCP
TCP
IP
IP
FCoE
Ethernet
Ethernet
Ethernet
PHYSICAL WIRE
SCSI
- 273
iSCSI
FCIP
FCoE
FC
Lesson Summary
Key points covered in this lesson:
 Basics and benefits of FCoE
 Storage infrastructure with FCoE
 FCoE Architecture and stack comparison
- 274
Module Summary
Key topics covered in this module:
 Benefits of IP SAN
 IP convergence in the SAN and its
implications
 Architecture and implementation of iSCSI
 Architecture and implementation of FCIP
 Convergence of new protocol FCoE
- 275
 Check Your Knowledge






What is the difference between a native
and bridged iSCSI implementation?
Explain the benefits and drawbacks of
using: NIC, TOE and iSCSI HBA
Name two iSCSI discovery mechanisms
What are two types of iSCSI names, and
which one is similar to a Fibre Channel
name?
What are the physical elements of FCoE?
Explain the benefit of using CNA card
- 276
Storage Area Network, an
Introduction of basic concepts
Antonella Corno – CCIE Storage CM
Summary






What is a SAN
Basic Building Blocks of a SAN
A zoom into the Storage
Architectures
SAN elements and architecture
Basic Protocols and Mechanisms
Who is who: Standardization
Bodies and Industry
Organizations
Summary






What is a SAN
Basic Building Blocks of a SAN
A zoom into the Storage
Architectures
SAN elements and architecture
Basic Protocols and Mechanisms
Who is who: Standardization
Bodies and Industry
Organizations
Definition


A SAN (Storage Area Network) is a
network designed to transfer data from
servers to targets, and it is alternative to a
directly attached target architecture, or to
a DAS architecture, where the storage is
connected to the servers on general
purpose networks
Additional definitions of a SAN imply that
the SAN should also be highly performing,
and should be such to enable storage
devices to communicate with one another
and with computer systems
Different technologies


Multiple technology can be used when
building a SAN; traditionally the dominant
technology is Fiber Channel, but IP based
solutions are also quite popular for specific
applications
The concept of SAN is also independent
from the devices that are attached to it.
Can be disks, tapes, RAIDs, file servers,
or other
SAN and NAS: network and
node


SAN vs NAS: while a SAN is a network
connecting storage subsystems, the NAS
is a storage subsystem, making use of a
general purpose network.
The SAN is an extension of the disk
channel outside the server, while the NAS
is a disk subsystem connected to the
servers, in most cases via an IP network.
Summary






What is a SAN
Basic Building Blocks of SAN
A zoom into the Storage
Architectures
SAN elements and architecture
Basic Protocols and Mechanisms
Who is who: Standardization
Bodies and Industry
Organizations
SAN interconnections




As said, different technologies can be used to interconnect the
network nodes, extending the Disk interface outside the server
Fiber Channel is a dedicated channel based high performance and
highly available network based on Fiber Channel Protocols
iSCSI is SCSI protocol carried over an IP network. In this case the
network infrastructure can be shared with other applications
SCSI network is an extension of the internal SCSI bus, used for
short distances due to its parallel architecture
Initiator and Target in FC
SANs



Fiber Channel Node: can be the source or
the destination of information
If the node is an Initiator (source), it is
usually connected to the network via an
HBA (Host Bus Adaptor), which is the
physical connection interface, and can be
based either on electrical or (more often)
optical technology
If the node is a target (destination), it can
be a JBOD (Just a Bunch of Disks), a
RAID (Redundant Array of Independent
Disks), or a Storage array
Hard Drives




The basic storage element is an Hard
Drive. They are made into complex
devices composed of platters, heads,
cylinders and tracks
The Logical Block Addressing (LBA)
addresses the sector within the disk.
Modern drives have 512 byte sectors
File systems arrange files into sectors so
that they can be stored and retrieved
The File system usually deals with clusters
of blocks and uses a FAT (File Allocation
FAT
Cluster
Free / Next / Final
LBA
0
1
0
1
5
4
…
10
14
40
11
free
44
12
free
48
13
free
52
14
final
56
…
62
free
248
63
free
252
File Allocation Table
JBODs and RAIDs


While a Jbod is a group of disks
packaged in an enclosure and
connected via a FC loop, a RAID
is a more sophisticated device,
that may improve performance
and/or reliability of the storage
device
RAID is improving performances
reading/writing information from a
set of disks at the same time, and
reliability adding parity and/or
mirroring information on multiple
RAID 0 or striping



Data are split onto different disks for
performance increase: performances depend
on information unit size vs stripe size
No redundancy added
Cost is limited (no additional hardware)
RAID 1 - mirroring




Data are replicated on
multiple disks for
redundancy
Performance may be
impacted if copy is
done serially
Increase of cost
proportional to the
amount of
redundancy
More complex
algorithm to manage
multiple copies
RAID 3




Data protection via ECC
(Error Correction Control
code)
Good redundancy
Performance not
changed for reads but
lower for writes since the
ECC need to be
calculated
Cost is only 1 extra disk
for the entire logical
array
RAID 5




Data protection
with ECC, but
parity is spread on
the array
Good redundancy
Same speed reads,
slower writes
One disk per array
of added cost
Summary






What is a SAN
Basic Building Blocks of a SAN
A zoom into the Storage
Architectures
SAN elements and architecture
Basic Protocols and Mechanisms
Who is who: Standardization
Bodies and Industry
Organizations
RAID array vs DAS vs NAS vs
SAN (I)


A RAID array is an enclosure containing a
set of disks and a RAID controller
providing in hardware the features of a
RAID 0 – 5 and usually some caching
engine
A DAS (Direct Attached Storage) is an
architecture for which the storage is
“privately” attached to the servers: cannot
be shared, it is hard to scale, expensive
and complex to manage. 80% of the
market it is still DAS
RAID array vs DAS vs NAS vs
SAN (II)



NAS (Netwrok Attached Storage) is an
architecture for which the storage is
attached to the servers via a multi-purpose
network, and it is accessed at a file level
via protocols like CIFS or NFS
The network is usually an IP network
TCP can be tuned to optimize storage
transport
DAS vs NAS architecture
Clients
Application
Servers
LAN
Win2k Linux
LAN
Win2k Linux Unix
Unix Win2k Linux Unix
Generic
SCSI
FC
FC
Application
Servers
Tape
Direct Attached
Storage
Generic
NAS Appliances
or
NAS Head Ends
SAN architecture




Storage is
accessed at
block level
not at file level
Very high
performances
Storage is
shared
Good
management
tools
Clients
LAN
Database
Servers
Fibre
Channel
SAN
Block
Storage
Devices
Storage Area Network (SAN)
Summary






What is a SAN
Basic Building Blocks of a SAN
A zoom into the Storage
Architectures
SAN elements and architecture
Basic Protocols and Mechanisms
Who is who: Standardization
Bodies and Industry
Organizations
SAN



SAN is the convergence of an almosterror-free, very reliable interface and the
advantage of a networking infrastructure
It takes advantage of a controlled and
reliable data transfer interface and adds
the capability of switching and sharing
proper of the networks
Relies on at least 10 to -12 error
probability infrastructure
SAN basic requirements



The concept of SAN is to bring a block
level interface on a very reliable support,
and add the advantages of a networking
infrastructure
Need to be scalable in terms of speed, fast
and efficient getting rid of all the low level
error recovery mechanisms: will treat the
errors as exceptions at application or
firmward level
Allow extensions like long distance, multi
protocol support (SCSI, ESCON,
TCP/IP..), rely on great simple connectivity
SAN supported topologies:
point to point



Point to point is the
simplest topology for
very limited connectivity
needs
It guarantees in order
delivery and full
bandwidth access
The application can
handle any multipath
connectivity to a set of
disks in case this is
provided, since no other
elements are present in
this topology
TX
RX
TX
RX
SAN supported topologies:
arbitrated loop




Designed to scale to a limited number of nodes (up to 127)
Low cost (no interconnecting devices needed)
Arbitration protocol is designed to manage media sharing across
nodes; may be disruptive when a node gets added/removed from
loop and loop initialization protocol kicks in
A arbitrating hub can be used instead of a distributed protocol
Each node can be
a server or a storage device
SAN supported topologies:
switched fabric



In a switched fabric topology,
switching element get added
to the nodes to allow
interconnections via point-topoint links
Extended number of devices
(potentially thousands) and
greater distances can be
achieved
Scalable, robust and reliable
architecture, but the cost of
the interconnection devices
adds on
Fibre
Channel
SAN
Block
Storage
Devices
Switched Topology
Summary






What is a SAN
Basic Building Blocks of a SAN
A zoom into the Storage
Architectures
SAN elements and architecture
Basic Protocols and Mechanisms
Who is who: Standardization
Bodies and Industry
Organizations
Fiber Channel and the others


Fiber Channel and SCSI: they are not
alternatives: FC will act as a transport
layer for SCSI, amongst other protocols
Fiber Channel and IP: sometimes that are
alternatives, sometimes not. While FC is
used instead of a TCP/IP network to
connect reliable the servers to the storage,
IP can still be used to extend FC networks
over long geographical distance; in turn in
some applications, FC can be used to
carry IP traffic (mostly for in-band
management)
FC protocols and services







FC offers a layer 2 transport service, and uses layer 3
services to provide it, and to map upper layer protocols
like SCSI, HIPPI, IP or others
FC0 defines the physical interface (data rates,
connectors, media, distances, power and so on
FC1 deals with encoding/decoding of bits on the
physical interface and low level signalling
FC2 defines framing and classes of services
FC3 provides common services like multicasting or
possibly encryption
FC4 maps upper layer protocols onto Fiber Channel
FC-0 to 2 are the core of the FC link protocol, and will
be examined in more detail in the next set of slides
FC0: physical interface




FC supports many different variations of physical interfaces:
Electrical and Optical both Multi Mode and Single Mode. They vary
in terms of cost and performances (distance and speed)
The basic physical access to a FC port is so called GBIC (Gigabit
Interface Connector) which is a hot swappable connector supporting
all types of media, and comes in different form factors. The MIA
(Media Interface Adaptor) is an electrical DB-9 to optical interface
converter, that can be applied externally to the enclosure (disk or
host)
Fiber channel connections are always bi-directional, point to point,
serial. No drops or taps are considered in FC
All links should be capable of BER (Bit Error Rate) of 10 to -12 or
less
FC-1: Line encoding



FC-1 is in charge of bit transmission on the line. Since FC
uses a serial transmission, an encoding in needed in order
to transmit the clock signal along with the data. This is
achieved mapping any pattern of 8 bits into a 10 bit pattern
that will guarantee enough signal transitions, and so enough
clock signal power on the line. The clock signal is filtered
and recovered at the receiving point, and the bits are
decoded
Not all the possible 10 bits sequences are used: only the
ones that guarantee enough transitions are allowed. The
others are invalid chars. Each 8 bits char has 2 coding on 2
10 bit patterns with opposite disparity (numbers of 0s and
1s), or 1 coding if the 10 bit pattern has no disparity (Same
number of 0s and 1s)
To avoid accumulation of DC at the receiver, the encoder
remembers every time it has transmitted a char with
disparity (different numbers of ones and zeros) and will send
the next char coded with opposite disparity. This mechanism
is called running disparity
FC-1 characters



FC is a word oriented architecture: 4
characters form a word
All possible 256 chars are mapped into 10
bit strings, and further 10 bits strings are
available outside the coding space. Those
are called special characters.
The encoder can encode data char as well
as data char, and an input signal to the
encoder will signal the encoder itself if the
coming pattern is data or special char
FC-1: Transmission words
and Ordered sets



The transmitted words are
composed by a sequence of 4
characters.
The first character of the
sequence will tell if the remaining
three need to be interpreted as
control signals or data. If it is a
K28.5 then the next 3 char are to
be considered signals, and the
word is called ordered set
The ordered set are divided into
FC-2: the frames




Before any data is echanged, a session need to
be established between the end ports, FC-2 is in
charge of login session
Once the link is established, FC-2 will handle
frames by using sequences and exchanges
FC-2 is also defining the frame format for FC
It also handles flow control and class of services
FC-2 Session





Before any data can be exchanged between end points,
a session need to be established. During login the end
nodes enchange information like identification, credits
(number of outstanding frames supported)
When the session is not needed, it gets cleared by a
logout operation
Within the session each operation happens within an
exchange, that is a set of frames logically related to each
other (example all the frames implementing a SCSI
operation)
A set of frames that constitutes a high-level protocol
information unit, is grouped into a sequence. An
exchange can host multiple sequences
An FC frame is 2112 bytes long, and contains addressing
fields as well as control fields like exchange ID,
sequence ID
FC-2: flow control




Flow control is performed in FC via use of Credits.
At the beginning of a transmission session, the two end point will
negotiate how many frames the sender can send before they get
acknowledged by the receiver (end to end credits – end to end flow
control)
If there are connection elements in between the two end point the
credits will also be negotiated at each link (buffer – to – buffer
credits)
Each end point of a link will acknowledge receipt of the frames
sending back credits (link by link or end to end) to the sender, which
is now allowed to send more frames
FC switching fabric
architecture services



FC switching fabric relies upon a number
of software applications to deliver the
transport services to the mapped upper
level protocols
Amongst the many the most relevant ones
are Zoning, Alias services, Domain
Controller, Name Server, FSPF
Those services allow the fabric to function
with no disruption, and maintain a
consistent set of distributed information
across the nodes of the fabric, and the end
nodes (hosts and storage)
Summary






What is a SAN
Basic Building Blocks of a SAN
A zoom into the Storage
Architectures
SAN elements and architecture
Basic Protocols and Mechanisms
Who is who: Standardization
Bodies and Industry
Organizations
Standardization Bodies and
Industry Organizations






FC is standardized by ANSI, in T11 committee (www.T11.org)
SCSI is standardized by ANSI, in T10 committee (www.T10.org)
IETF is dealing with iSCSI, FCIP, IPFC (IP/FC protocols) requests
for comments (RFC) (www.ietf.org)
SNIA is the Storage Networking Industry Association and it is not in
charge of standardization (www.snia.org)
FCIA is the Fiber Channel Industry Association and it is not in
charge of standardization (www.fibrechannel.org)
UNH is University of New Hampshire, and it is traditionally involved
with interoperability third party testing. It is not a standard body
(http://www.unh.edu)