The University has decided to allocate some of its Science Research Infrastructure Funds
(SRIF) towards establishing a Storage Area Network (SAN) Facility to support the storage requirements of future e-science related research projects. The purpose of this paper is to discuss to what extent the large-scale storage requirements of other services could be satisfied by a SAN based storage consolidation strategy and if possible include these requirements within the overall procurement process. The SRIF funding becomes available from April this year and it is hoped to have the procurement completed by late summer. The Computing Service and the University’s purchasing department are handling the procurement process and a notice, inviting expressions of interest from potential suppliers, has been placed in the European Journal. To ensure that the requirements of the research community are being addressed the computing service will be liasing with a sub group of the research committee and other researchers in IBLS, Gla-
NESC, DCS, Eng and HATTI. It is expected that the requirements identified during this consultation phase will be included in the Operational Requirement (OR) document that will be sent to short listed suppliers as part of the Invitation To Tender (ITT)
Documentation. The Operational Requirement is intended to elicit fully costed options and features from suppliers rather than committing the University to a particular spend or course of action.
Traditional solutions to file store provision are based on Direct Attach Storage (DAS) whereby each File or Application server maintains direct connectivity with its own local file store.

Figure 1 – Direct Attach Storage.
Some refinements to this model exist whereby client systems mount file store from other servers over a data network, however these fileservers have themselves Direct Attach
Storage. The main problems associated with this model are

File store growth is exponential leading to o A proliferation of hardware and systems administration load o Less than optimal provision with spare capacity is some areas and under provision in others o Inefficient file store delivery with respect to the number of clients a server can support and the need to tune servers for general-purpose use rather than predominately I/O operations.

Close coupling of storage with servers present single points of failure and potential loss of service to large communities

Backup regimes become stressed, costly and labour intensive

Disaster recovery procedures become expensive or impractical
‘It is proposed that a storage consolidation strategy based on Storage Area Network technology would provide a flexible, manageable and cost effective solution for the majority of the University’s file store requirements.’
In order to explore this strategy further it would be beneficial to define what is meant by
‘storage consolidation’, identify the potential benefits that may be derived from it as well as the likely implementation scenarios including support effort and costs.

Storage consolidation is about centralising and sharing storage resources amongst a variety of file and application servers. In effect the management of storage resources is separated from the management of servers by establishing a network that removes the physical mapping of storage devices to physical servers. This network allows isolated islands of storage to be pooled together and allocated to servers in a more flexible fashion. The major benefits of storage consolidation include:

Heterogeneous environment - cross platform support

Protection against single points of failure

Storage aggregation into flexible virtual storage pools

Less complex storage management

Lower administrative costs

Disaster Recovery (DR) procedures would be easier to implement through remote mirroring/replication and clustered file/application servers

Potential for server consolidation reducing costs and systems administrator overheads

Addresses legislative requirements for data retention

Improved backup and restore capabilities

Improves data security, accessibility and availability
Storage consolidation could be achieved by implementing any of the following architectures:

Enterprise RAID

Storage Area Network (SAN)

Network Attached Storage (NAS)
Enterprise RAID and SAN architectures provide block level access between the storage subsystem and servers. The NAS architecture provides file level access between the storage subsystem and clients using standard file sharing protocols (e.g., CIFS, NFS, DFS etc.) over a traditional data network
Redundant Array of Independent Disks (RAID) architectures can support a heterogeneous environment with as many as 64 direct-attached server interfaces

Figure 2. – Enterprise RAID
Enterprise RAIDs typically provide a boxed solution with a RAID header, incorporating the RAID controllers, Sever interfaces and disk enclosure interfaces.
SAN architectures extend the functionality of RAID sub-systems by connecting them to a
SAN fabric that also provides interconnects for servers, tape libraries and fabric gateways. The SAN fabric may consist of a number of technologies including

Fibre Channel (FC) supporting switches, ALs, HBAs operating at 1Gbs or 2Gbs interconnect speeds

iSCSI. An IETF standard for communication SCSI commands over an IP network

FC/IP. A method of tunnelling FC connections over an IP network
 iFCP. Encapsulates FC over IP whilst maintaining the visibility of individual FC devices
Connecting RAID storage subsystems to a SAN increases the number of servers and other devices that can access the available storage capacity well beyond what could be supported via DAS solutions.

Figure 3 – SAN fabric incorporating iSCSI client connections
Network Attached Storage is implemented via dedicated NAS devices that share files, over a data network, with other servers and clients using standard protocols e.g., CIFS,
NFS, and DFS etc. NAS devices can be configured to provide many of the advantages of
Enterprise RAID and SAN solutions.
Figure 4 – Network Attach Storage

As storage consolidation gains acceptance within the enterprise storage market place there is a drive towards solutions that integrate the benefits of the discrete architectures into a common storage consolidation model. In this model the Storage Area Network is enhanced with NAS head functionality and fabric extension via iSCSI, FC/IP and iFCIP capabilities.
NAS heads provide for efficient file serving via:

SAN attached dedicated file servers tuned for sustained i/o operation

Resilience via tight or stretched NAS head clustering

High speed FC SAN interfaces and Gigabit Ethernet LAN interfaces with TOE
Fabric extension via iSCSI, FC/IP and iFCIP provide:

Remote SAN island integration over existing high speed campus backbones

Lower cost server to SAN interconnects

Ability to deliver SAN connectivity to distributed servers
Figure 5 – Bringing it together

Clearly implementing a SAN solution would provide many benefits for the University in terms of:

More efficient use of storage resources

Less systems administration effort

Resilience via fault tolerance

Disaster recovery plans

Backup and restore operations

Server consolidation and resilience

Core service provision – Directories, E-Mail, CSCE, SSD, Samba etc

Research projects

Faculty/Department Block and File level requirements
However we need to identify the SAN architecture that would be most suited to the
University environment and how the various University application and file servers could be linked to it.
It is proposed that the University implement a flexible Storage Area Network model based on the following:

A core fabric spanning 2 sites. NB the core fabric would consist of 2 or more director class Fibre Channel switches

A number of FC edge switches linked to the FC core

A number of disk arrays providing the storage requirements for: o Research projects o Mirroring o Core services o Other applications

A number of servers connected via FC Host Bus Adaptors (HBAs) to either the core or edge FC switches

An enterprise class backup facility based on a mixture of disk and tape technology

NAS heads for file sharing functionality
 iSCSI support for fabric extension to non FC attached clients

FC/IP support for native fabric extension

Figure 6 – Possible SAN Model Core to Edge Fabric with extension capabilities
This model would provide a reasonable mix of high performance SAN fabric facilities that could be extended to non-fabric aware clients as and when required.
Although SAN deployments are increasing and overall costs decreasing there are still many factors that need to be addressed when considering a SAN solution e.g.

Fabric costs – FC switches and HBAs are significantly more expensive than other
Fabric options e.g., iSCSI and Gigabit Ethernet switching. However FC fabrics currently deliver superior performance and security

Fabric design – The Fabric design will impact the extent to which SAN clients may connect directly to the SAN fabric. The common design of core to edge seems applicable but there would be several limiting factors affecting its implementations e.g., o Fabric switch costs – Circa 60K per Director switch and 25k per FC switch o HBA costs – FC HBAs cost around 1.5k per dual channel adapter o Fibre costs – Implementing acceptable levels of resilience whilst maintaining low contention ratios between the Fabric, disk arrays and clients would require many Fibre circuits

o Manufacturers limits – Many SAN manufacturers impose their own limits on SAN design that must be adhered to. These limits typically affect the number of Director and no-director switches in the Fabric as well as the number of switch hops, ISL trunks and end-to-end latency

Vendor and OS support – It will be necessary to define the Operating Systems and
File Systems that the SAN must support initially and over time. Vendor compatibility will be critical for the successful operation of the SAN.

SAN Fabric extension via: o NAS heads – For many applications NAS heads may prove to be the preferred method of access to the SAN as there is no need for full hardware interface compatibility at the HBA/firmware level o ISCSI and FC/IP – Again these extension capabilities may provide lower cost client interconnect options for servers that require block level access to the storage sub-systems

SAN management – SAN management will be a critical factor as this will provide the main storage mappings, operational and security functions e.g., o The partitioning and grouping of disks into flexible file-systems o Implementing security policies i.e., which servers can access which filesystems via LUN masking, Virtual SANs etc o SAN performance and fault monitoring

Disk Virtualisation – This technique presents a logical representation of the storage sub-system (a storage pool) to a server. This technique can be applied at the block level or file level providing a virtualisation of storage services.

Virtualising Filesystems – This capability provides concurrent access to the same file-system between a number of different servers and possibly different operating systems

Data Storage Lifecycle Management – Provides the ability to transparently move files that have not been accessed for a pre-determined period to a lower cost media

Backup regimes need to be specified with respect to: o Type - Snapshotting, full and incremental overheads o Extent – Compromises may need to be made between Backup regimes, backup capacity and the total available SAN storage o Media - LTO tape or staging disk then Tape o Costs for media and hardware

Fabric connection guidelines – It is envisaged that an open SAN architecture would be impractical due to cost and potential compatibility problems with
HBAs, OS’s, firmware revisions etc. For these reasons it is envisaged that the number of direct server connections to the fabric would be kept to a manageable set of proven hardware compatible systems

Funding – Whilst some funding is available to help establish an enterprise SAN facility, significant funds will be required over time to establish a facility capable of meeting the University’s file storage requirements

This paper has concentrated on enterprise class SAN solutions that would require significant funding to implement. It is perhaps worth pointing out that lower cost IP/SAN solutions based on stock hardware and open source software may provide satisfactory operation for some applications. However when considering the University’s overall requirements it is felt impracticable to consider these for core service provisions.
Some examples of alternative provision are:

Open source NAS with SAN possibility - http://www.techworld.com/news/index.cfm?fuseaction=displaynews&NewsID=6
15

Open Source Based NAS appliances - http://www.nwfusion.com/reviews/2003/0428nasrev2.html

SANmelody - http://www.datacore.com/products/prod_SANmelody.asp