SPO3292
Veeam
Backup &
Replication:
Tips and Tricks
Anton Gostev
Veeam Software
@Gostev
Doug Hazelman
Veeam Software
@VMDoug
#vmworldsponsor
Quick Overview of v6 Architecture
 Backup servers
Backup proxy servers
Backup repositories
 “Automated everything”
 Intelligent load balancing
 Centralized management via Enterprise Manager
Scaling your backups
3 simple rules
Scaling your backups
1. Keep the management server happy
● Disable default proxy (requires 6.1 or later)
● Allocate enough RAM for job manager processes
● Keep concurrent (running) jobs under 100 per management server
2. Backup proxy servers: the more, the better?
● No! Too much load on storage and network
● Use max concurrent tasks limit on proxies or repositories
● Watch for “job timed out waiting for resources” messages
3. Backup repository considerations
● Be careful with the reversed incremental backup mode!
● Limit concurrent jobs on backup repositories to a reasonable amount
● Use ingest rate throttling for cross-SAN backups
A word on backup repositories
 Don’t underestimate the importance of performance!
● By far, the most commonly reported bottleneck
 What makes the best backup repositories?
● Windows or Linux server (can be same as backup proxy server)
● Local storage, DAS or SAN mounted for physical server
● pRDM disk (vSphere 5+), or iSCSI LUN connected via in-guest iSCSI
for virtual server
 Sub-optimal backup storage
● NAS or network share
● VMDK on VMFS (size and recoverability considerations)
 RAID level
● If you can afford it, use RAID10 (again, performance)
Processing Modes
All you need to know
Direct SAN Access: The good
 Fastest processing mode
 Least impact on production
● Backup processing is fully offloaded to dedicated backup proxy servers
● Backup traffic is isolated to the storage network (aka LAN-free)
 Does not impact consolidation ratio, so cheapest too
Direct SAN Access: The bad
 Supports block storage only
● FC (fibre channel): physical backup proxy server only!
● iSCSI: physical and virtual backup proxy servers both supported
 Physical backup proxy server requirement for FC SAN
● Might not go along well with your virtualization project
● Consider repurposing older servers
 Might be hard for beginners to setup
● See Veeam Forums FAQ for step-by-step guide
 Manual datastore mapping might be required
● For certain SANs, B&R may not able to detect proxy connectivity
Direct SAN Access: The ugly
 What’s the worst that could happen?
 Windows re-signaturing your VMFS LUNs!
● vSphere will no longer recognize datastores
● Don’t panic, VMware Support should be able to fix
 Three easy ways to get into trouble
1. Windows Explorer automounting new volumes (but not with Veeam)
2. Clicking Disk Management snap-in popup without reading
3. Giving Local Administrator rights to random people
Direct SAN Access: The safe way
 Present VMFS LUNs to backup proxy server as read-only
● Most SANs support it these days—chase your vendor if yours does not
 Disable automount on your backup proxy servers
● Do it the right way: use SANPolicy Windows setting!
● Veeam backup proxy server setup does this automatically for you
 Disable Disk Management snap-in with Group Policy
User Configuration > Administrative Templates
Window Components > Microsoft Management Console >
Restricted/Permitted snap-ins > Disk Management
 Keep Local Administrator rights on backup proxy servers
to yourself
● Cannot really do this for default proxy due to FLR requirement
● Another reason to use dedicated backup proxy server!
Direct SAN Access: Tips & tricks
 Got a fast SAN? Get a modern backup proxy server!
● Multi-core CPU (compression) and fast RAM (inline deduplication)
 Update firmware and drivers across the board
 Disabling MPIO may increase performance
 iSCSI SAN? Tweak TCP/IP on backup proxy
netsh interface tcp set global autotuning level = disable
 Increase read-ahead buffer
● Default is 4MB (optimal setting for most SANs)
● To change, create the new value in bytes:
VddkPreReadBufferSize (DWORD)
Hot Add: The good
 Easy to setup—very little planning involved
● Any Windows VM can be made a Hot Add backup proxy
 Fast data transfers with any storage
● Direct storage access (albeit through ESXi storage stack)
 Supports all types of storage (including NFS)
● Shared storage: at least 1 backup proxy server per vSphere cluster
● Local storage or DAS: at least 1 backup proxy server per host
 Use your existing Windows VMs (save on licensing)
● Data processing engine process runs with lower priority (6.1)
● Further CPU usage reduction in 6.5
 Allows for 100% virtual deployment
Hot Add: The bad
 Not as mature as other modes
 Affects your consolidation ratio
● Backup proxy servers take host resources
● Ultimately means more ESXi hosts, and more VMware licenses
 Hot Add process itself is slooow
● Can take up to 1–2 minutes to complete for each VM—adds up quickly!
 Hot Add as a vSphere feature has a number of limitations
● Good news—many are being removed as VDDK matures
● See FAQ on Veeam forums FAQ for the complete list
Hot Add: The ugly
 Snapshot removal problems due to locks
● Veeam B&R: multiple hooks in place to work around
 CBT must be disabled on backup proxy VM
● Prevents stun on Hot Add due to CBT initialization
 NFS-specific issue
● Extended VM stun on hot remove in some scenarios
Hot Add: Tips & tricks
 Add extra virtual SCSI controller to backup proxy server
● A single SCSI controller can have a maximum of 16 disks attached
● Concurrent jobs on the same backup proxy server can result in more!
 Keep vSphere and Veeam up to date
● Single block size in VMFS5 removes the most common hot add issue
● Latest Veeam Backup & Replication will have latest and greatest
VDDK version
 Try increasing read-ahead buffer
● Seems to really help with certain NFS storage
 Avoid cloning backup proxy VM
● For example, to provision additional backup proxies
Network (NBD): The good
 Easy to setup—in fact, no setup is required
● Any existing server (physical or virtual) would do
 Supports all types of storage, including NFS
● Server placement does not matter (unlike with Hot Add)
 Very quick to initialize data transfer
 Can be quite fast—with 10Gb Ethernet
Network (NBD): The ugly
 Painfully slow performance on 1Gb Ethernet
● Average speed reported is 10-20 MB/s
 Leverages ESXi management interface
Network (NBD): Tips & tricks
1 Gb Ethernet
 Use for sites with low change rate
● Works faster than other processing modes in such conditions
 Keep at least one Hot Add backup proxy server around
● Full VM and virtual disk restores take forever over NBD
 Keep in mind intelligent load balancing algorithms
● Network backup proxy servers have lowest priority!
 Upgrade to 6.1 or later
● Improved network proxy location awareness
One last thing
 This hack significantly reduces supportability!
 Cut up to 5 minutes of processing time per VM by
disabling VDDK logging
 Apply in stable environments only!
● Create the new value and set to 1:
DisableVDDKNetworkOutput (DWORD)
Deduplicating Storage
Yes, you can afford it!
Deduplicating storage: The good
 What gives? Global dedupe!
● Deduplication across backup files from different jobs
● Perfect for long-term backup archival
 Top hardware appliances among Veeam users
● EMC DataDomain
● ExaGrid
● HP StoreOnce
 Top software appliances among Veeam users
● ZFS-based appliances
● StarWind
 Windows Server 2012 dedupe is awesome
Deduplicating storage: The bad
 Hardware appliances are expensive
● Although they do provide excellent dedupe ratio
 Software appliances are resource hogs
● Both performance and dedupe ratio are sub-par, too
 Windows Server 2012 dedupe is awesome
● Included free of charge—start using it today!
● Provides very decent dedupe ratio
Deduplicating storage: The ugly
 Random access performance is lacking
● A problem for all solutions featuring inline deduplication
● Typically insufficient out-of-the-box for large-scale vPower usage
 Exception: post-process deduplication
 ExaGrid
● Raw disk landing zone (full-speed vPower from recent backups)
● Veeam-specific logic further optimizes performance
 Windows Server 2012
● Backups “land” on raw storage at full speed
● Only old backup files are deduplicated—great for vPower
● Decent speed even off already deduped backups
Deduplicating storage: Tips & tricks
 Already own storage with inline deduplication?
● Inline data “rehydration” process is what makes vPower slow
 Reduce the block size in Advanced job settings
● WAN (256KB) and LAN (512KB)
● Reduced block size might impact backup performance
 Use Linux-based backup repository
● Large client cache, or even caching client file system (FS-cache),
can significantly improve vPower performance
Deduplicating storage: Tips & tricks (continued)
 Keep Veeam dedupe on
 Use incremental backup mode
● Choice of synthetic or active fulls depending on actual storage
 For best dedupe ratio on device side…
● Disable compression (significantly increases amount of data transferred
from backup proxy server to backup repository over network)
 For best backup performance and smallest window
● Keep compression at default
 If you like to avoid extremes…
● Set compression to Low (dedupe-friendly)
Deduplicating storage: Tips & tricks (continued)
Got more than one deduplicating storage device?
 Use internal replication to sync backups offsite!
● Extremely traffic-efficient approach
● Many customers use and report great success!
 Keep the backups imported for easy DR
● To automate repository refresh in DR site, use:
Get-VBRBackupRepository -Name "DR_Repository" |
Sync-VBRBackupRepository
WAN Accelerators
Your WAN on steroids
WAN accelerators: The good
 Two types of WAN accelerators
● Caching WAN accelerators provide significant bandwidth savings with
Veeam replication, but are typically quite expensive
● Transport layer WAN optimizers are unlikely to offer significant
bandwidth savings with Veeam replication, and are usually cheap
 Both improve reliability of TCP
● Long distance wireless or satellite links
● IPsec rekey operations on a VPN tunnel
● WAN links with high jitter, packet loss or occasional drops
 Both allow long-running jobs to finish more consistently
● For example, initial replication over network
 Both improve WAN utilization for most workloads
WAN accelerators: What’s hot?
 Top caching WAN accelerators among Veeam users
● Cisco WAAS
● Riverbed
● SilverPeak
 Top WAN optimizers among Veeam users
● Hyper-IP
WAN optimizers: Tips & tricks
 Veeam Backup & Replication leaves little room for
bandwidth reduction by WAN optimizers
 Built-in WAN optimizations in v6:
● Multiple TCP streams to maximize throughput
● Network traffic compression
 Consider using on unreliable networks, but don’t
expect them to add you extra bandwidth
● Update to Backup & Replication 6.1 Patch 1 before evaluating
 Veeam Backup & Replication not using all available
bandwidth?
● Increase the amount of TCP streams (default is 5)
DownloadStreamsNumber (DWORD)
Caching WAN accelerators: Tips & tricks
 Multiple TCP streams can cause issues!
● Disable multiple streams in B&R traffic throttling settings
 Disable network traffic compression in Veeam
● Low (dedupe-friendly) compression level might be a better option
 If required, have network admins configure bypass on
Veeam backup proxy servers to avoid polluting caches
Veeam Backup & Replication 6.5
We never stand still
What’s coming in 6.5
 Veeam Explorer for Microsoft Exchange
 Veeam Explorer for SAN Snapshots
 VMware vSphere 5.1 support
 Windows Server 2012 support
Veeam Explorer for Microsoft Exchange
 Visibility into Exchange VM backups
● Immediate: No need to provision storage, restore the VM or
restore the mailbox store
● Agentless
● Requires no special backups or metadata collection—even works with
existing Veeam backups (and SAN snapshots)
 Free!
● Included in all versions of Veeam Backup & Replication 6.5,
including Free Edition
● Eliminates need for expensive standalone tools licensed per-mailbox
 Currently available in “exclusive beta”
Just restored a 145 GB #MSExchange Public
Folder database in 2 min. using @veeam. Then
restored a single item all under 10 min. SWEET!
The new #veeam explorer for exchange
looks veeamy. That's right, I just made up a
new word (it means awesomesauce)
Even the *beta* of @veeam Exchange
Explorer works a treat. Saved literally, hours of
work.. and saved my bacon. #recommend
Really excited with the new @veeam Explorer
beta for #Microsoft Exchange VM backups - we
have lots of interested customers ready for this!
Veeam Explorer for Microsoft Exchange
 Capabilities
● Browse: familiar Explorer-type interface
● Search: familiar Outlook-like Find, including Advanced Find
● Export: export to PST file, MSG file or attachment
 Uses cases
● E-discovery
● Item-level restore: export and send to affected user
● Mailbox archive
 Supports Exchange Server 2010
(continued)
Veeam Explorer for SAN Snapshots
 Veeam restores from SAN snapshots
 Supports tiered data protection strategy
 Perform all restores through familiar, easy-to-use
Veeam interface
 Supports HP StoreVirtual VSA and HP LeftHand
SAN snapshots + Veeam restore =
Best RPOs and RTOs for operational recovery
Veeam Explorer for SAN Snapshots (continued)
 Fast: recover entire VM or individual items in < 2 minutes
● Fully automated: clone & promote snapshot, present to vSphere, clean up
● Restores directly from VM files on the SAN snapshot: no staging or
intermediate restores required
 Flexible
● Specific VM
● Individual guest files: Windows, Linux, et al
● Individual Microsoft Exchange items
 Free
● Worry-free: automated process eliminates human errors and protects
integrity of SAN snapshots and production LUNS
● Agent-free: no agents to deploy on hosts or VMs
● Literally free: included in all editions of Veeam Backup & Replication 6.5,
including Free Edition
Questions? Comments?
Thanks for attending!
FILL OUT
A SURVEY
AT WWW.VMWORLD.COM/MOBILE
COMPLETE THE SURVEY
WITHIN ONE HOUR AFTER
EACH SESSION AND YOU WILL
BE ENTERED INTO A DRAW
FOR A GIFT FROM THE
VMWARE COMPANY STORE
SPO3292
Veeam
Backup &
Replication:
Tips and Tricks
Anton Gostev
Veeam Software
@Gostev
Doug Hazelman
Veeam Software
@VMDoug
#vmworldsponsor