Redpaper
Using an IBM System Storage N
series with VMware to Facilitate
Storage and Server Consolidation
© Copyright IBM Corp. 2006. All rights reserved.
ibm.com/redbooks
1
Introduction
Many companies are experiencing a rapid proliferation of commodity Intel®
servers. These systems are typically equipped with a small amount of internal or
direct-attached disk storage. As server numbers grow into the hundreds, the
management complexity for both servers and storage increases exponentially,
putting a strain on limited administrator resources and IT budgets.
Server and storage consolidation using VMware ESX Server software and IBM®
N series storage systems significantly reduces these problems. VMware allows
multiple operating system instances to run on a single physical machine and
does not require major changes to existing application software.
Previous VMware implementations have demonstrated that it is possible to
reduce the number of deployed Intel-based servers by 50% or more while
achieving:
򐂰
򐂰
򐂰
򐂰
2
60% to 80% utilization rates (up from typical rates of <15%)
Application provisioning time measured in seconds, not days
Response times for change requests measured in minutes
Zero-downtime hardware maintenance
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
The benefits of VMware are uniquely complemented by IBM N series storage
solutions. Using IBM N series technology, storage can be consolidated onto one
or a few storage systems accessed across Fibre Channel SANs or regular IP
networks see (Figure 1). This dramatically simplifies storage provisioning,
increases asset utilization, and allows a company to take advantage of IBM N
series data protection and disaster recovery functionality. This paper describes
the real-world experiences of a customer that reduced its server count 50% and
implemented a tapeless storage infrastructure using VMware and technology
that is used today with the IBM N series.
SAN
Enterprise
NAS
Departmental
Enterprise
Departmental
iSCSI
Fibre
Channel
Dedicated
Ethernet
SAN
(Block)
16
Corporate
LAN
NAS
(File)
IBM
N series
© 2005 IBM Corporation
Figure 1 IP and SAN networks
Customer challenges
Customer challenges are:
򐂰 Server proliferation results in rising complexity and costs.
򐂰 Low asset utilization.
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
3
VMware and IBM N series value proposition
This offers:
򐂰
򐂰
򐂰
򐂰
Simple server/storage virtualization
Reduced operating costs
Increased business flexibility
Advanced data protection and disaster recovery
Deployment results
These are:
򐂰
򐂰
򐂰
򐂰
򐂰
򐂰
4
Servers reduced >50%
Server management cut 20%
Tapeless backup for rapid recovery
Bare metal recovery in 30 minutes
Disaster recovery for all critical business applications
Dramatically increased IT flexibility
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
Background
Using many of the features and capabilities available today in the IBM N series
the customer hosts Oracle via CIFS while providing Fibre Channel SAN storage
used by VMware and Microsoft® Exchange servers. If you were using an IBM N
series it would look similar to Figure 2.
SAN
Exchange
NAS
Oracle
Fibre
Channel
CIFS
Ethernet
SAN
(Block)
IBM
N series
16
© 2005 IBM Corporation
Figure 2 Microsoft Exchange and Oracle
Like many organizations, the company's Intel-based server environment has
grown exponentially over the years to support additional business requirements
and applications. At the time, it relied on over 100 servers. After internal
benchmark studies against similar companies in the same industry showed that
the number of servers was significantly above average for the number of users
supported, the company launched an initiative to lower operating costs by
reducing servers.
In 2004 the company selected VMware as an enabling technology to replace
physical servers with virtual hardware instances running on powerful
multiprocessor systems.
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
5
VMware requires either direct-attached or SAN-based disk storage. The
company chose a Fibre Channel SAN to allow storage to be consolidated onto
the existing storage systems utilizing the functions made available by Data
ONTAP. While building a new Fibre Channel infrastructure can be expensive, the
VMware server consolidation project server count was cut dramatically and
overall expenditures were significantly reduced. Costs were also offset by the
advantages of consolidated storage and the advanced data management
capabilities offered by technology also used in the IBM N series today.
6
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
Infrastructure
The company's SAN infrastructure utilizes two storage systems using the same
software suite available in the IBM N series today—one in its main data center
and another in a secondary data center. Both storage systems are configured to
support SAN-attached VMware servers (see Figure 3) using the multiprotocol
capabilities of Data ONTAP. An Exchange server in the main data center also
utilizes SAN storage with SnapManager for Exchange software. Gigabit Ethernet
is deployed throughout each data center. Redundant DS3 circuits link the
primary data center to the secondary data center.
Primary
Secondary
VMware Servers
SAN
26
SAN
VMware
Servers
© 2005 IBM Corporation
Figure 3 SAN attached VMware servers
Both storage systems have FCP Target cards installed, while all VMware servers
are configured with QLogic 2342 adapters. Microsoft Exchange 2003 hosts have
Emulex initiator cards installed. Servers and storage systems are connected to a
Brocade model 3800 fabric switch using factory default settings and no zoning.
Three four-processor Xeon® servers have been configured with VMware ESX
server software. These VMware ESX servers are configured to access LUNs on
the storage system in the primary data center. Data ONTAP provides the
capability to create these LUNs using traditional or FlexVol volumes. An
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
7
additional four-processor machine is connected to the SAN in the secondary
data center and configured in standby mode. VMware SAN volumes and other
volumes on the primary storage system are mirrored to the secondary storage
system in the secondary data center using SnapMirror software. In the event of a
disaster affecting the primary data center, critical applications running under
VMware can be quickly restarted in the secondary location.
LUNs are provisioned in their own qtrees. Following best practices, each
virtualized server is allocated a qtree in a SAN_vmware volume on the storage
system. In addition, each VMware LUN is formatted as a VMFS-2 volume. This
allows the VMware Virtual Center to be used to migrate a virtual machine from
one VMware ESX server host to another while the server is live. This facilitates
maintenance or other administrative needs while minimizing application
disruptions.
Configuring an IBM N series Fibre Channel SAN with
VMware
Once the appropriate software is installed, configuring the VMware ESX server
to access IBM N series storage via Fibre Channel protocol (FCP) is simple.
VMware allows the administrator to map LUNs to the virtual SCSI disks used by
each virtual machine instance. Provisioning LUNs on the IBM N series storage
systems requires the following steps:
1. Decide the size of the LUN, the LUN ID number, and where it will reside on
the storage system (use Linux® as the Host OS type).
2. If not already present, create an initiator group for the host.
3. Run LUN setup on the IBM N series storage system via the command line, or
use the LUN Wizard tool in IBM N series FilerView (see Example 1).
itsotuc1> lun setup
This setup will take you through the steps needed to create LUNs
and to make them accessible by initiators. You can type ^C (Control-C)
at any time to abort the setup and no unconfirmed changes will be made
to the system.
Do you want to create a LUN? [y]:
Example 1 lun setup
Once the LUN is created, use the VMware administrative interface to map the
LUN from the storage system to the specific virtual server.
8
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
In sizing the LUN for VMware, increase the size slightly to account for VMware
administrative space. VMware uses this space to accommodate information
about the instance and to store the memory contents of the virtual server when it
is suspended.
VMware migration
The company created LUNs that were 2 GB larger than the storage requirement
to accommodate VMware. After the company created its LUNs and virtual
servers, the LUNs were mapped to individual virtual servers and the instances
started. A migration tool from VMware called P2V was used to move data from
existing physical servers to VMware virtual servers. After a period of testing and
a brief outage for each, the physical servers were retired.
Figure 4 Storage system Fibre Channel SAN configuration with VMware software
Backup and disaster recovery
In any data center, backup and recovery is critical. In this deployment, Snapshot
and SnapMirror technology help the company protect virtual server data and
speed recovery in the event of a disaster.
The VMware Perl scripting API makes it simple to automate the capture of a
consistent Snapshot copy of active data. Every night at 8 p.m. the virtual
machines (VMs) are suspended. Then the VMware instances and the Host OS
buffers are flushed to disk, creating a consistent OS image on the storage
system LUNs. Once the instances are suspended, the Perl scripts direct the
primary storage system to take a Snapshot copy of the VMware volume. Once
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
9
the copy has been created, the VMs are resumed. This process is similar to
using a database's hot backup mode — suspending I/O to the disks while the
backup process creates a consistent copy of the database.
The entire process, from begin suspend to resume VM, usually completes within
30 to 120 seconds on normally loaded servers in the lab. The Snapshot portion
takes less than 5 seconds.
At 9 p.m. the Snapshot copy is mirrored to the secondary data center using
SnapMirror software. As described above, the secondary site is running an
identical storage system plus a single standby VMware ESX server.
In the event of a disaster or inadvertent loss of data at the main site requiring an
extended outage, the remote site can use the SnapMirror copy of the data to
restart critical virtual servers with minimal disruption to ongoing operations.
Once the situation is resolved, the SnapMirror technology allows a reverse
mirror to be initiated. This is transparent to the applications, and when the mirror
has been synchronized, a brief outage allows operations to resume at the main
site.
Figure 5 Process to automate the capture of a consistent Snapshot copy of active data
10
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
In addition to allowing recovery in the event of a disaster, the Snapshot copies at
the main site or from the mirror at the remote site provide consistent copies of the
LUN images for backup purposes, allowing tape copies to be made for off-site
archival without any additional virtual server downtime.
Results
When the project began, the company relied on 105 servers across 9 locations,
supporting a wide variety of applications including Oracle, Exchange, Web
serving, Microsoft SMS, print, and file serving. By simplifying its IT infrastructure
with VMware and technology that is used in the IBM N series today, the company
was able to reduce the number of deployed physical servers to fewer than 50 —
a reduction of more than 50% — without eliminating any services.
To further decrease complexity, the company reduced the number of locations
where business servers were deployed by 66%, with the remaining three main
sites interconnected with high-speed networking connections. This effort
significantly simplified the company's IT infrastructure while making it easier to
provide appropriate protection and disaster recovery capabilities for critical
applications. As a result, the head count directly responsible for server
management functions fell 20% from 2.5 people to only two.
Consolidating server storage on the storage systems has given the company
much greater control over its data. In the past, backups were performed across
the company's networks and had to be throttled to prevent network bottlenecks.
With the high bandwidth between the three major sites, this is no longer an issue.
For individual file restores, the company has completely eliminated a previous
service level agreement of 4 hours. The metric is no longer needed because
restores occur in minutes.
In addition, the previous tape environment had a much higher risk of losing data,
and bare metal recovery of a failed server could take from 8 to 12 hours. Today,
the company uses Acronis True Image software to back up virtual server images
to a FlexVol or traditional volume. The volume is also replicated to the secondary
site for DR. In testing, bare metal recovery of critical servers can now be
accomplished in 30 minutes.
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
11
Improved disaster recovery is another major advantage. Snapshot technology
and SnapMirror software have replaced tape backups with a completely tapeless
solution. In the past, the company provided disaster recovery for only a small
part of its overall environment. With SnapMirror, the company provides DR for all
its critical applications and can rapidly add DR capability for any application that
requires it (see Figure 6).
Multi-protocol, Heterogeneous Environment
iSCSI
FCP
CIFS
NFS
Disaster
Recovery
Disk-based
Backup
18
© 2005 IBM Corporation
Figure 6 Disaster recovery and disk-based backup
The company has also found that resource usage is very efficient in its VMware
environment. VMware efficiently shares processor and memory resources
between instances of the same OS, so fewer processors and less memory has
been needed than originally anticipated, while end users report that application
performance has actually increased noticeably.
VMware allows the company to quickly implement new servers for proofs of
concept or new projects. The company no longer has to budget immediately for
new hardware and software; it can just configure a new virtual server. Their
storage systems and SAN technology complement this capability. Today, the
company makes changes to its storage environment four or five times a week
with no disruption to users.
12
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
Conclusion
The combination of VMware software and IBM N series storage solutions offers
significant advantages for companies consolidating servers and storage to
reduce costs and increase operational efficiency. VMware allows companies to
reduce server counts by 50% or more without major application changes, while
making it possible to adapt much more rapidly to support new projects and new
applications.
IBM N series solutions simplify a company's overall storage infrastructure,
creating a storage system that adapts rapidly to changing needs while improving
data protection and disaster recovery — often by reducing or eliminating the
need for tape-based solutions.
The team that wrote this IBM Redpaper
Much of the material in this IBM Redpaper was provided by Network Appliance,
Inc. and reproduced for use by IBM by Alex Osuna of the International Technical
Support Organization, Tucson.
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
13
14
Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation
Notices
This information was developed for products and services offered in the U.S.A.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area.
Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM
product, program, or service may be used. Any functionally equivalent product, program, or service that
does not infringe any IBM intellectual property right may be used instead. However, it is the user's
responsibility to evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document.
The furnishing of this document does not give you any license to these patents. You can send license
inquiries, in writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.
The following paragraph does not apply to the United Kingdom or any other country where such
provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION
PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer
of express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may
make improvements and/or changes in the product(s) and/or the program(s) described in this publication at
any time without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any
manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the
materials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without
incurring any obligation to you.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm
the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on
the capabilities of non-IBM products should be addressed to the suppliers of those products.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrates programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the
sample programs are written. These examples have not been thoroughly tested under all conditions. IBM,
therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy,
modify, and distribute these sample programs in any form without payment to IBM for the purposes of
developing, using, marketing, or distributing application programs conforming to IBM's application
programming interfaces.
© Copyright International Business Machines Corporation 2006. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by
GSA ADP Schedule Contract with IBM Corp.
15
This document created or updated on January 9, 2007.
Send us your comments in one of the following ways:
򐂰 Use the online Contact us review redbook form found at:
ibm.com/redbooks
򐂰 Send your comments in an email to:
redbook@us.ibm.com
򐂰 Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099, 2455 South Road
Poughkeepsie, NY 12601-5400 U.S.A.
®
Redpaper
Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
Redbooks (logo)
™
IBM®
System Storage™
The following terms are trademarks of other companies:
VMware, the VMware "boxes" logo, GSX Server, ESX Server, Virtual SMP, VMotion and VMware ACE are
trademarks (the "Marks") of VMware, Inc.
Snapshot, Data ONTAP, SnapMirror, SnapManager, FilerView, Network Appliance, The Network Appliance
logo, the bolt design,Camera-to-Viewer, Center-to-Edge, ContentDirector, ContentFabric, NetApp Availability
Assurance, NetApp ProTech Expert, NOW, NOW NetApp on the Web, RoboCache, RoboFiler,
SecureAdmin, Serving Data by Design, Smart SAN,The evolution of storage, Virtual File Manager, and Web
Filer are trademarks of Network Appliance, Inc. in the U.S. and other countries. All other brands or products
are trademarks or registered trademarks of their respective holders and should be treated as such.
Microsoft, and the Windows logo are trademarks of Microsoft Corporation in the United States, other
countries, or both.
Intel, Xeon, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of
Intel Corporation or its subsidiaries in the United States, other countries, or both.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
16
Using a IBM N series with VMware to Facilitate Storage and Server Consolidation