




11/3/2011

tSAS/sreddy 2

SCSI (Small Computer System Interface)
• The SCSI protocol is an application layer storage protocol.
• It's a standard for connecting peripherals to your computer via a standard hardware interface, which uses standard
SCSI commands.
SAS (Serial Attached SCSI)
• SAS is the successor of SCSI technology
• Initially introduced at 3Gb/s in 2004
• Currently, supports 6Gb/s
• 12Gbps by 2012
• Significantly increased the available bandwidth offered by legacy SCSI storage systems.
• Use of expanders increases the scalability over legacy SCSI
11/3/2011 tSAS/sreddy 3

Topology
SCSI
Parallel Bus
Speed 3.2 Gbps
Distance 1 to 12 meters
Number of Targets 14 devices
Devices
Connectivity
SCSI only
Single Port
11/3/2011
Drive Form Factor 3.5”
Cost Low
SAS
Serial Bus
Reduces Noise
(Better signal integrity)
3 Gbps, 6Gbps,
12Gbps
8 meters
128 expanders.
> 16,000 with cascaded expanders
SAS & SATA
Used widely in
Data Centers today
Dual Port
Fail-over/fail-back
2.5”
Medium
4 tSAS/sreddy

11/3/2011 iSCSI protocol describes a means of transporting of the SCSI packets over
TCP/IP
 The iSCSI specification is revised and updated by the
Internet Engineering task
Force (IEFT).
 Work by S. Aiken, D.
Grunwald, A. Pleszkun and
J. Willeke shows the performance of a commercial iSCSI software implementation compared quite favorably with fibrechannel [7]
5 iSCSI protocol layering model tSAS/sreddy

Driver
~8m
DISK DRIVES
EXPANDERS
11/3/2011 tSAS/sreddy 6

 Overcome the distance and scalability limitations of traditional Serial Attached
SCSI (SAS) by using tSAS.
 Like iSCSI, tSAS takes advantage of existing internet infrastructure, internet management facilities as well as addresses distance limitations
 Provide research results for future industry specification for tSAS and iSCSI.
11/3/2011 tSAS/sreddy 7

11/3/2011
 Michael Ko’s patent at IBM Research-Almaden on
Serial Attached SCSI over Ethernet proposes a very similar solution to the tSAS solution provided in this project.
 iSCSI specification (SCSI over TCP) itself is similar to a tSAS solution (SAS over TCP). The iSCSI solution can be heavily leveraged for a tSAS solution.
 The Fibre Channel over TCP/IP specification also can be leveraged to design and implement a tSAS solution.
tSAS/sreddy 8

tSAS Topology
SCSI
Driver
SCSI
Driver
- DISK DRIVES
- EXPANDERS
11/3/2011 tSAS/sreddy
-
TCP link
-
SAS link
9

tSAS Topology
SCSI
Driver
SCSI
Driver
- DISK DRIVES
- EXPANDERS
11/3/2011 tSAS/sreddy
-
TCP link
-
SAS link
10

 Investigate how tSAS can be implemented
 Design and develop a mock tSAS application
− Write a client program using C/C++ that sends a set of tSAS commands to a server.
− Write a server program using C/C++ that receives a set of tSAS commands, processes them and responds to the client with tSAS responses.
 Evaluate the tSAS solution and compare with a mock iSCSI client/server application.
11/3/2011 tSAS/sreddy 11

 Software based tSAS solution
− Cheaper than a hardware based solution
− All tSAS processing is done by the processor and TCP/IP operations are also executed by the CPU
− NIC is merely an interface to the network
 TCP Offload Engine (TOE) solution
− NICs with integrated TOEs have hardware built into the card that allows the TCP/IP processing to be done at the interface
11/3/2011
 Hardware based tSAS solution
− In a hardware-based tSAS environment, the initiator and target machines contain a host bus adapter (HBA) that is responsible for both TCP/IP and tSAS processing tSAS/sreddy 12

 The tSAS Request is initially sent by the tSAS Initiator to the tSAS Target over TCP.
 The tSAS Target strips off the TCP header and sends the SAS frames using the SAS
Initiator block on the tSAS Target to the
SAS expander/drive.
 The SAS expander/drive sends SAS frames to the tSAS Target.
 Finally, the tSAS Target embeds the SAS frames received from the expander/drive over TCP and sends it to the tSAS Initiator.
11/3/2011 tSAS/sreddy 13

Ethernet
Header
IP
Header
TCP
Header tSAS
Header tSAS
Data
Ethernet
Trailer
TCP Segment
IP Datagram
Ethernet Frame
 The above shows how a legacy SAS header and data are embedded in an Ethernet frame.
 iSCSI uses the same approach where the iSCSI header and data are encapsulated in an Ethernet frame.
11/3/2011 tSAS/sreddy
14

tSAS
Initiator tSAS
Target tSAS
Request Open Address
Frame
SAS
Expander
Open Accept tSAS Xfer
Rdy Frame
Data Frame
Data Frame
SAS/SSP
Request Frame
Xfer Rdy
Frame
.
.
Data Frame
Data Frame
SSP tSAS
Response
Response
Frame
Frame
 The above diagram shows a typical tSAS (SSP Write )
Request & Response sequence diagram.
 iSCSI uses a similar approach where the back-end between the iSCSI target and SCSI drives uses the legacy
SCSI protocol.
15
11/3/2011 tSAS/sreddy

 Primitives
− Primitives are special 8b/10b encoded characters that are used as frame delimiters, for out of band signaling, control sequencing in SAS
− Most primitives can be conveniently ignored on the tSAS side except a few such as
Broadcast primitives used to initiate discovery
− SAS primitive can be encapsulated in an
Ethernet frame
 Discovery
− Discovery in tSAS will be similar to SAS and will be accomplished by sending Serial management protocol (SMP) commands over
TCP to the initiators and expanders downstream to learn the topology.
11/3/2011 tSAS/sreddy 16

11/3/2011
SAS Primitives tSAS/sreddy 17

tSAS
Client
SMP
Initiator on
Expander
SMP
Request
Open
Address
Frame
Open
Accept
SMP
Target on
Expander
SMP
Request
SMP
Response
Close
SMP
Response
SMP Request & Response Sequence Diagram
11/3/2011 tSAS/sreddy 18

11/3/2011
Test benches used for performance calculations
 tSAS and iSCSI Mock Application
− The client application and server application run on two different Windows machines connected using a NetGear ProSafe Gigabit
Switch
 Intel Core 2 Duo CPU
 2.53 GHz, 3.45 GB RAM
 Intel 82567LM Gigabit Network Adapter
− Client makes Read/Write requests and the server processes and responds to requests made by the Client
Windows
Machine A
(Running
Client App)
NetGear
Switch
Windows
Machine B
(Running
Server App) tSAS/sreddy 19

Test benches used for performance calculations
 Legacy SAS
− A 6 Gbps SAS Host Bus Adapter in a PCIe slot of a Super Micro server
− A 6 Gbps SAS target (Seagate SAS drive) connected to the Host Bus Adapter
− IOMeter in Windows and VDBench in RHEL used to generate Read/Write IOs and measure performance
 IOMeter and VDBench are IO generator tools used to measure IO performance of a storage system
− A LeCroy SAS Protocol Analyzer placed between the HBA and the SAS Target
11/3/2011 tSAS/sreddy 20

Test benches used for performance calculations
Driver
SAS Protocol Analyzer
SAS Protocol Analyzer
11/3/2011
SAS DISK DRIVE
EXPANDER tSAS/sreddy 21

11/3/2011
Test benches used for performance calculations
 Legacy iSCSI
− Two windows machines/servers were used connected using a NetGear ProSafe Gigabit
Switch
− On one machine the StarWind iSCSi Initiator was running
− On another machine, the KernSafe iSCSi target software was used to create a iSCSI target
− IOMeter was used to send Read/Write requests from the iSCSI Initiator to the iSCSI
Target and measure performance
− Wireshark used to capture Network Traffic iSCSI
Initiator on
Windows
Machine A
NetGear
Switch tSAS/sreddy iSCSI
Target on
Windows
Machine B
22

tSAS vs iSCSI Mock Application at 10 Mbps iSCSI vs tSAS READ Completion Time at 10
Mbps
2500
2000
Time (Milliseconds)
1500
1000
500
0
0 200 400 600 800 1000 1200
Transfer Size (Kilobytes) tSAS READ Completion time iSCSI READ Completion Time tSAS vs iSCSI Write 10Mbps
8000
6000
Time (milliseconds) 4000
2000
0
0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200
Transfer Size (Kilobytes) tSAS Write 10MBps iSCSI Write 10MBps
11/3/2011 tSAS/sreddy 23

tSAS vs iSCSI Mock Application at 100 Mbps tSAS vs iSCSI Read 100 Mbps
Time
(Milliseconds)
700
600
500
400
300
200
100
0
0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200
Transfer Length (Kilobytes) tSAS Read 100 Mbps iSCSI Read 100 Mbps tSAS vs iSCSi Write 100 Mbps
Time
(Milliseconds)
900
800
700
600
500
400
300
200
100
0
0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200
Transfer Length (KB) tSAS Write 100 Mbps iSCSI Write 100 Mbps
11/3/2011 tSAS/sreddy 24

tSAS vs iSCSI Mock Application at 1 Gbps tSAS vs iSCSI Read 1 Gbps
50
40
Time
(Milliseconds)
30
20
10
0
0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200
Transfer Length (KB) tSAS Read 1 Gbps iSCSi Read 1 Gbps tSAS vs iSCSI Write 1 Gbps
Time
(Milliseconds)
80
70
60
50
40
30
20
10
0
0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200
Transfer Size (KB) tSAS Write 1 Gbps iSCSI Write 1 Gbps
11/3/2011 tSAS/sreddy 25

Time for READ Completion with transfer size of 512K at 10 Mbps,
100Mbps and 1Gbps
1200
1000
800
Time
(Milliseconds)
600
400
200
0
0 200 400 600 800
Connection Rate (Mbps)
1000 1200
Time for READ Compeltion
11/3/2011 tSAS/sreddy 26

 Using several iSCSI Target Softwares that were not licensed/purchased made it challenging since several features were either not supported or did not work on the unlicensed version.
 Even using KernSafe iSCSI target, I was not able to create a iSCSI RamDisk Target or a iSCSI Hard disk target. I used a USB Flash drive as the iSCSI target
− Subtracted the delay at the Flash Drive from the iSCSI IOMeter performance results
− Delay of Flash Drive calculated by running IOMeter to the SCSI device on the target machine
11/3/2011 27 tSAS/sreddy

 Jumbo frames
 The NetGear ProSafe switch is a Gigabit switch that supports Jumbo Frames.
However, I could not get jumbo frames to work since the Network Adapter on the
Windows machines did not support Jumbo frames
 One needs cabling and adapters as well that support Jumbo frames to use a switch that supports Jumbo frames
 Understanding the SAS protocol and tools is very involved
− Took substantial time to understand and learn the workings of SAS as well as the tools
11/3/2011 tSAS/sreddy 28

 More data points
− The tSAS mock application can be run using a faster switch with connection rates greater than 1
Gbps to get more data points (40G/100G Ethernet)
 Piggybacking
− Response frames are piggybacked with the last
DATA frame or a DATA frame is piggy backed with a request frame. This would increase performance
 Jumbo frames
− Can be used to increase the amount of DATA that is passed from the initiator and target per Ethernet packet improving the performance results.
 The Storage Associations can be motivated with similar work to fuel the conception of a tSAS specification
 tSAS code can be implemented in a SAS HBA and performance can be measured using this direct implementation
11/3/2011 tSAS/sreddy 29

 tSAS is a viable solution
 tSAS will be faster than a similar iSCSI implementation
− Overhead in tSAS smaller than in iSCSI
 Request, Response and Transfer Ready frames in tSAS smaller than Request,
Response and Ready to Transfer frames in iSCSI
− Back-end of tSAS will be legacy SAS
− tSAS more visibly faster at larger transfer sizes than iSCSI
− tSAS more reliable at smaller transfer sizes at
10 Mbps, 100 Mbps and 1 Gbps due to higher
TCP retransmission rate at larger sizes
− tSAS hardware solution would be the fastest vs a software solution
− tSAS would perform better at smaller transfer sizes
11/3/2011
 tSAS will overcome the distance limitation of legacy SAS tSAS/sreddy
30

11/3/2011
References
[1] T10/1760-D Information Technology – Serial Attached SCSI – 2 (SAS-2),
T10, 18 April 2009,
Available from http://www.t10.org/drafts.htm#SCSI3_SAS
[2] Harry Mason,
Serial attached SCSI Establishes its Position in the Enterprise,
LSI Corporation, available from http://www.scsita.org/aboutscsi/sas/6GbpsSAS.pdf
[3] J Satran, K Meth, C. Sapuntzakis, M. Chadalapka, E. Zeidner,
RFC 3720 Internet Small Computer Systems Interface (iSCSI) Specification
IETF, April 2004, available from http://www.ietf.org/rfc/rfc3720.txt
[4] Cai, Y.; Fang, L.; Ratemo, R.; Liu, J.; Gross, K.; Kozma, M.;
A test case for 3Gbps serial attached SCSI (SAS)
Test Conference, 2005. Proceedings. ITC 2005. IEEE International, February
2006, available from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1584027 tSAS/sreddy 31

11/3/2011
References
[6] Kalmath Meth, Julian Satran,
Design of the iSCSI Protocol,
Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings. 20th
IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies,
April 2003, available from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1194848&tag=1
[7] Stephen Aiken, Dirk Grunwald, Andrew R. Pleszkun, Jesse Willeke,
A Performance Analysis of the iSCSI Protocol
20 th IEEE/11 th NASA Goddard Conference on Mass Storage Systems and
Technologies, 2003, available from http://www.storageconference.org/2003/papers/20-Aikens-Performance.pdf
[8] M. Rajagopal, E. Rodriguez, R. Weber,
RFC 3821 Fibre Channel over TCP/IP (FCIP) standard,
IETF, July 2004, available from http://tools.ietf.org/html/rfc3821
[9] BLi Bigang, Shu Jiwu, Zheng Weimin,
SCSI Target Simulator Based on FC and IP Protocols in TH-MSNS *
Department of Computer Science and Technology, Tsinghua University, Beijing
China, 2005 tSAS/sreddy 32

11/3/2011
References
[10] S. Chaitanya, K. Butler, A. Sivasubramaniam, P. McDaniel, M. Vilayannur,
Design, Implementation and Evaluation of Security in iSCSI-based Network
Storage Systems,
StorageSS '06 Proceedings of the second ACM workshop on Storage security and survivability,
October 2006, available from http://portal.acm.org/citation.cfm?id=1179564
[11] 1Gbps and 10Gbps Ethernet
Server Scalability, NetApp, available from http://partners.netapp.com/go/techontap/matl/downloads/redhatneterion_10g.pdf
[12] Michael A. Ko, LAYERING SERIAL ATTACHED SMALL COMPUTER SYSTEM
INTERFACE (SAS) vOVER ETHERNET, United States Patent Application
20080228897, 09/18/2008 available from http://www.faqs.org/patents/app/20080228897
[13] SCSI Primary Commands - 4 (SPC-4), Revision 31, T10, 2011/06/13, available from http://www.t10.org/members/w_spc4.htm
[14] Broadcom iSCSI HBA FAQ available from http://www.broadcom.com/collateral/faq/iSCSI-FAQ100-R.pdf
tSAS/sreddy 33