Fusion-io
directCache
Non-Virtualized Server Caching
Solution
NetApp Confidential – Limited Use
1
Why Deploy Flash Technology?
 Need low latency (<1ms)
– Lower application run time
– Provide better user experience
 Need more IOPS
– Offload IOPS from spinning disk to flash
– Increased workload on same infrastructure
 Green computing
– Reduce spindle/rack requirements
– Reduce power requirements
2
Server Flash as Cache
Flash emerging in server as IOPS tier
• Significant performance benefits, but relatively costly
However, flash alone is just fast DAS (Direct Attached Storage)
• No sharing across servers
• No high availability, data protection, disaster recovery
• Management complexity
Ideal: Server flash as a cache for network storage
• Flash to improve performance, network storage to protect and
manage data
• Optimal $/GB and $/IOPS
Target use cases
• Application-specific IOPS acceleration, OLTP, virtualized OLTP, VM
density
3
Virtual Storage Tiering
Server
Flash
Virtual Storage Tier
Data-driven  Real-time  Self-managing
Array Flash
Intelligently place “hot” data
on the highest performing
media in real time
 Efficient use of flash
 Simple to install
 Self-managing
 Nondisruptive operations
 Caching vs. data migration
 Minimal HDD I/Os
 Highly granular
HDD Storage
 Real-time responsiveness
4
Server Side Flash as Cache
 Utilizing flash without compromising enterprise
array capabilities
– Application integrated Snapshot®/SnapRestore®
technologies
 SnapDrive® for Windows®/SnapManager® for SQL Server ®
– Disaster recovery capabilities
 SnapMirror®/MetroCluster ™
– High availability
 CFO/MetroCluster
– Shared storage
 Virtualization and host clustering
5
Server Cache Partner Alliance – Software
Data Coherency Implications with Caching
What is data coherency?
Consistency between data that is
being presented in cache and data
that actually resides on disk
Why is data coherency
Important?
Storage Array
Physical Server
Application
Cache
0100100100101
0101010101101
0101010010100
1000100101001
Data
Coherency
0100100100101
0101010101101
0101010010100
1000100101001
Caches that don’t recognize
changes to the primary data copy
(e.g. via SnapRestore®) will cause
data corruption issues
Data coherency issues can vary in severity based on the cache implementation.
Key criteria include:
• When does the cache invalidate data?
• Is the cache a write-through or a write-back cache?
• Does the cache retain data across a reboot, etc?
6
Server Cache Partner Alliance
Creating
Customer Value
• Provide industry's broadest coverage for use
cases, operating systems and hypervisors - today
• Provide partners with integration points, to
expose NetApp value at the server
Enabling
“Best of Breed”
solutions
• Provides customers with best of breed solutions
• Provide qualification criteria (SW & HW) to
ensure interoperability, performance, and data
management
Simplifying
Deployment
• “Best Integrated” into solutions such as FlexPod®
for rapid customer deployment
7
Server Cache Partner Alliance
 Program for joint testing/validation of server
side caching solutions
– Ensures existing data coherency issues are
known and understood
– Collaboration between NetApp® and partner to
design resolution for data coherency issues
8
Fusion-io Bare Metal
Cache Solution
9
Fusion-io Solution Components
 ioMemory Virtual Storage Layer (VSL)
– Block interface for ioMemory (driver)
 ioMemory device – ioDrive2
– Hardware PCIe device providing flash capacity
 directCache software
– Software that transforms ioMemory into a block
based cache device
 ioSphere
– GUI for ioMemory management, alerting, and
performance monitoring
10
Fusion-io ioMemory VSL & ioDrive2
 Enterprise reliability from MLC flash
 Simplified data path from CPU to flash =
lower latency and increased data integrity
 Lower CPU and RAM utilization
 Flash endurance-extending capabilities
11
Fusion-io directCache
 Integrated with ioMemory
VSL
 Caches frequently read
data closer to the
application
 Caches block storage
(not NAS)
 Simple software
configuration
 Maintain existing SAN
configuration
Physical Server
Application
OS
Disk
Controller
VSL
NetApp®
Storage
ioMemory
12
Fusion-io directCache
 directCache operates as a transparent filter
driver in the host
 Primary storage device identity is maintained:
no changes in existing storage identity
– Support for NetApp® SnapDrive® for Windows®
and SnapManager® for SQL Server®
13
Fusion-io directCache
Application
NetApp®
Storage
 For bare-metal Windows and
Linux
 Reads are cached to
ioMemory
 Writes are passed through to
primary storage and then
cached for later reads
 All following reads of same
data are from ioMemory
 Maintains existing SAN
configuration
14
Multi-Volume Caching Support
 One ioMemory device caches multiple LUNs
 64 backing stores per ioMemory device
 Independent settings for each volume
ioMemory
Cached data blocks
15
SnapDrive for Windows Data Coherency
 Maintaining data coherency during SDW
SnapRestore® operation requires the NetApp®
SnapRestore Support feature of directCache
be installed
 When a SDW SnapRestore operation is
detected, cache is automatically invalidated
for the affected device
 Cache must be manually invalidated if a
SnapRestore is performed manually at the
array
16
directCache Product Features
Product Type
Add-on software
Cache type
Fully associative cache
Cache mode
Write-through
Minimum cacheable block size
512 bytes
Max cache instances per host
64
Additional CPU&RAM overhead
No noticeable increase over the ioMemory
VSL driver
Product Architecture
Independent Linux® kernel module
Windows® volume/disk filter driver
Management
CLI and ioSphere-based GUI
17
Fusion-io ioSphere (for Bare Metal)
 Fusion-io offers two different ioSphere
products:
– ioSphere – for bare metal environments using
ioMemory and directCache software
– ioSphere VM Edition – management system for
ioTurbine software in virtual environments
– These are different products
 Customers with valid Fusion-io support
contracts are entitled to download ioSphere
18
Fusion-io ioSphere (for Bare Metal)
 GUI management and
monitoring software for
ioMemory devices
 Real-time visibility into all
deployed ioMemory
 Historical performance
monitoring
 Warranty forecasting
 Intuitive Web user
interface
ioSphere - VM or
Physical Host
ioDrive2
ioDrive2
ioDrive2
19
Fusion-io ioSphere (for Bare Metal)
 ioSphere capabilities
Interface
Web GUI for remote access
Authentication
Local users and/or LDAP
accounts
Monitoring
Multihost ioMemory device
monitoring, auto discovery
Alerting
SMS/Email, customizable and
searchable history
ioMemory Performance
metrics
Live display and historical
reports
Configuration
Configure ioMemory devices
Usage warranty forecasting
20
Supported Systems
and Requirements
21
NetApp ioTurbine Validation
 NetApp® validation includes verification of certain
product combinations and features
– Data coherency for snapshot restore events
– SAN interoperability – see note in IMT Policies &
Guidelines http://support.netapp.com/matrix/
 Other operating systems, product combinations, and
protocols are fully supported by Fusion-io but have
not been validated by NetApp
22
NetApp directCache Validation
 NetApp directCache data coherency
validation
SnapDrive® for
Windows®
SnapDrive® for UNIX®
SnapProtect®
directCache 1.1.6
Validated – Requires directCache
NetApp® SnapRestore® Support
feature
Not supported*
To be validated
Current as of August, 2012
* SDU is inoperable when directCache caching is enabled
23
NetApp directCache Validation
 Configuration used for NetApp validation
directCache
1.1.6 with directCache NetApp®
SnapRestore® Support feature installed
ioMemory VSL
3.1.5
SnapDrive® for Windows®
6.4.1
SnapManager® for SQL®
5.2
Server Operating System
Windows Server® 2008 R2 SP1
Storage Connectivity
FCoE
24
Fusion-io directCache System Support
 Other operating systems and features
supported by Fusion-io for directCache
Fusion-io directCache
1.1.6
Compatible cache devices
ioDrive1 and ioDrive2 family devices
ioDrive, ioDrive Duo, and ioDrive Octal
products, and mezzanine form factors
Operating Systems - Windows®
64-bit Microsoft® Server 2008/2008 R2
Operating Systems - Linux®
RHEL 5.x, RHEL 6.x, SLES 11.x, Debian
Squeeze, Cent OS 6.2
Host clustering
MSCS, VCS 5.x and above on RHEL,
Linux native cluster
Supported primary storage connectivity
FC, iSCSI (does not cache NAS)
25
ioDrive2 Requirements
 Fusion-io ioDrive2 device requirements
Form Factor
Low-profile PCI Express
Connectivity
PCIe 2.0 x4 electrical x8 physical
26
Fusion-io NetApp
Resell
NetApp Confidential – Limited Use
27
Fusion-io ioMemory – ioDrive2
 ioMemory devices available from NetApp
– ioDrive2 365GB MLC
– ioDrive2 785GB MLC
ioDrive2 Capacity
365GB MLC 785GB MLC
Read Bandwidth (1 MB)
Write Bandwidth (1 MB)
910 MB/s
590 MB/s
1.5 GB/s
1.1 GB/s
Random Read IOPS (512B)
137,000
141,000
Random Write IOPS (512B)
535,000
535,000
Device raw throughput numbers
NetApp Confidential – Limited Use
28
Hardware & Software Resell
 NetApp resell of Fusion-io hardware and
software
– Hardware: ioDrive2 365GB & 785GB MLC
– Software: ioTurbine, directCache & ioSphere
Note: No SKU for ioSphere needed. Hardware support contract
provides ioSphere access
SKU
Description
SW-FS0106
directCache Standard (per host)
SW-FS0120
ioTurbine (per host)
X-F00-001-365G-CS-0001-R6
ioDrive2 365GB MLC
X-F00-001-785G-CS-0001-R6
ioDrive2 785GB MLC
NetApp Confidential – Limited Use
29
Product Support
 Hardware and software support provided by Fusion-io
directly to customer:
– NetApp® support staff trained to help triage
– NGS has Fusion-io hardware/software
– Joint customer issues managed through TSANet
 License and account registration performed by
customer/Fusion-io
 All software provided by Fusion-io from fusionio.com
– No Fusion-io software hosted on NetApp site
NetApp Confidential – Limited Use
30
Product Support SKUs
 Fusion-io support contracts resell by NetApp
Software Support
SW-SSPVN-FS0110
FusionIO SSP, directCache Gold 1Yr
SW-SSPVN-FS0111
FusionIO SSP, directCache Gold 3Yr
SW-SSPVN-FS0124
FusionIO SSP, ioTurbine Gold 1Yr
SW-SSPVN-FS0125
FusionIO SSP, ioTurbine Gold 3Yr
Hardware Support
SVCVN-FS1-GLD-320-HE-1YR
HW Svc, FusionIO ioDrive1 320GB SLC Gold 1Yr
SVCVN-FS1-GLD-320-HE-3YR
HW Svc, FusionIO ioDrive1 320GB SLC Gold 3Yr
SVCVN-F00-GLD-365G-CS-1YR
HW Svc, FusionIO ioDrive2 365GB MLC Gold 1Yr
SVCVN-F00-GLD-365G-CS-3YR
HW Svc, FusionIO ioDrive2 365GB MLC Gold 3Yr
SVCVN-F00-GLD-785G-CS-1YR
HW Svc, FusionIO ioDrive2 785GB MLC Gold 1Yr
SVCVN-F00-GLD-785G-CS-3YR
HW Svc, FusionIO ioDrive2 785GB MLC Gold 3Yr
NetApp Confidential – Limited Use
31
directCache
Administration and
Management
32
directCache Terms and Definitions
 Important directCache terms and definitions
Cache device
The ioMemory ioDrive2 device used for flash
memory capacity
ioMemory control device
The device name for the ioMemory ioDrive2
device. Example: \\.\fct5
Backing store
The primary storage device to be cached. A
device name, drive letter, or mount point.
Example: PhysicalDrive3, F:\, or
C:\MSSQL\tempdb
Bind
To configure a backing store to be cached
Unbind
To remove a backing store from a cache
device and drop cache contents
33
directCache Installation Overview
 Installation high-level steps
1.
2.
3.
4.
Install ioDrive2 card in host
Install ioMemory VSL
Install directCache software
If using SDW install the directCache NetApp ®
SnapRestore® Support feature
5. Install ioSphere software (optional) on
different windows host
6. Configure storage for caching
34
Configuring directCache
1. Identify control device name for ioMemory
C:\Users\Administrator> fio-status
Found 1 ioMemory device in this system
Fusion-io driver version: 3.1.3 build 27
Adapter: Single Controller Adapter
Fusion-io ioDrive2 785GB, Product Number:F00-001-785G...
Connected ioMemory modules:
fct5: Product Number:F00-001-785G-CS-0001, SN:1210D0132
fct5
Attached as 'fct5' (block device)
ioDrive2 Adapter Controller, Product Number:F00-001-785G...
PCI:05:00.0
Firmware v6.0.0, rev 107004 Public
785.00 GBytes block device size
Device name
\\.\fct5
35
Configuring directCache
2. Nondisruptively bind cache device to backing
store
C:\Users\Administrator>dc-bind C:\MSSQL\tpce_tempdb \\.\fct5
About to create binding using
backing store 'C:\MSSQL\tpce_tempdb’
caching device '\\.\fct5’
write-through mode
WARNING: Binding will destroy any existing data on the caching
device \\.\fct5!
Do you wish to continue [y/n]? y
Binding: [====================] (100%) /
Successfully bound
backing store 'C:\MSSQL\tpce_tempdb’
with cache device '\\.\fct5’.
Warning occurs only
for first backing store
to bind to this device
Note: binding a system/boot drive is not supported by directCache.
36
Removing Cache Configuration
 Nondisruptively unbind cache device from
backing store
C:\Users\Administrator>dc-unbind C:\MSSQL\tpce_tempdb
About to tear down binding:
backing store 'C:\MSSQL\tpce_tempdb’
cached by '/dev/fct5’
cache UUID: 52c1edca-0914-40f7-9275-b23aded2a2e8
Do you wish to continue [y/n]? y
Unbinding: [====================] (100%) \
Successfully unbound:
backing store 'C:\MSSQL\tpce_tempdb’
cached by '/dev/fct5'
Note: Unbinding the backing store invalidates cache contents for that backing store.
37
Viewing Cache Status & Statistics
 Cache status and statistics can be displayed
with dc-status command
C:\Users\Administrator>dc-status –s
fct5: Fusion-io ioDrive2 785GB, SN:1210D0132
Device fct5 cache bindings:
Cache device:
Backing store:
Cache instance:
VSU 8:
Status: enabled, Mode: Write-through
C:\MSSQL\tpce_tempdb
dab3fedd-7338-4ea0-b181-9b1b352fb182
dda8d294-7d71-49bf-9b55-1099ed3aa59a
In-use: 13.98GBytes
Write stats:
Hits: requests 0%, sector 0%, Misses: requests 100%
Latency: hits 0.0s, misses 2.37247ms, partials 0.0s, total 2.37247ms
Read stats:
Hits: requests 72%, sectors 93%, Misses: requests 27%
Latency: hits 327.666us, misses 50.3810ms, total 14.1122ms
38
Cache Statistics in Windows Perfmon
 Standard performance counters in Windows®
Perfmon—reads/sec, sec/read—will not show
correct stats
– Cache hits bypass the part of the Windows
stack where statistics are gathered
– Standard Perfmon counters show stats only for
IOPS that were writes or read cache misses
 directCache provides WMI Perfmon counters
for monitoring various cache statistics
39
Cache Statistics in Windows Perfmon
 directCache provides Perfmon counters for
monitoring various cache statistics
40
directCache Cache Invalidation
 Data in directCache cache is invalidated in
the following cases:
– System reboot
– Binding/unbinding the backing store
(dc-bind/dc-unbind)
– Disabling/enabling caching
(dc-disable/dc-enable)
– During SDW SnapRestore® operation (only when
directCache data coherency add-on is installed)
41
Collecting Data For
Support/Troubleshooting
 fio-bugreport generates a detailed report
about the state of the device and directCache
– fio-bugreport runs several ioMemory VSL and
directCache commands which must be located in
your PATH environment variable for commands to
complete successfully
C:\Users\Administrator> fio-bugreport
Report output: fio-bugreport-20120723_153906.cab
Generating bug report. Please wait, this may take a
while...
--------------------------------------------------Gathering all Windows Event Logs...DONE
Gathering Fusion-io Windows Event Logs...DONE
Gathering System Information...
42
Recommendations &
Best Practices
43
Recommendations for Server Caching
 If using SnapDrive® for Windows®, you must
install the directCache NetApp® SnapRestore®
Support feature
 Manual revert of storage at the array requires
manual cache invalidation
When performing manual storage reverts:
1.
2.
Stop applications using storage
Invalidate cache data (dc-unbind or dc-disable)
3.
Disconnect storage devices from host or shut down host
4.
Perform SnapRestore®
5.
Reconnect storage devices to host or boot host
6.
Re-enable cache
44
Recommendations for Server Caching
 Do not cache LUNs that are heavily written to
– Example: Database logs, Snapinfo LUNs
 Do not use cache reservation feature
– Let cache software manage cache utilization
 When using multiple cache devices, spread
busiest backing stores across cache devices
45
Resources
NetApp Confidential - Internal Use Only
46
References
 Server Caching Partner Alliance binder on
field portal
 Overview of Server Caching Program
 Server Caching Partner Alliance FAQ
 Server Caching Program Customer
Presentation
NetApp Confidential - Internal Use Only
47
© 2012 NetApp, Inc. All rights reserved. No portions of this document may be reproduced without
prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp,
the NetApp logo, Go further, faster, MetroCluster, SnapDrive, SnapInfo, SnapManager,
SnapMirror, SnapProtect, SnapRestore, and Snapshot are trademarks or registered trademarks
of NetApp, Inc. in the United States and/or other countries. Microsoft and Windows are registered
trademarks of Microsoft Corporation. Linux is a registered trademark of Linus Torvalds. All other
brands or products are trademarks or registered trademarks of their respective holders and
should be treated as such.
48