Protecting Your Crown Jewels in
a Virtual World: Strategic
Systems Recovery
January 29, 2013
3:45 - 5 PM
New York Hilton
LegalTech® New York
January 29 – 31, 2013
Speakers:
• Marcus Bluestein, Kraft Kennedy, Moderator
• Steven Shock, Irell & Manella LLP
• Jim McCue, Rodey Law Firm
• Steve Skidmore, Martin, Clearwater & Bell
LLP
LegalTech® New York 2013
2
Industry Trends
Marcus Bluestein
Chief Technology Officer
Kraft Kennedy
bluestein@kraftkennedy.com
LegalTech® New York 2013
3
Industry Trends
Use of server virtualization increase again…
Servers are highly virtualized
70
60
50
40
30
20
10
0
2012
LegalTech® New York 2013
2011
2010
2009
4
Industry Trends
…as does use of SANs.
Firms using SANs
90
80
70
60
50
40
30
20
10
0
2012
LegalTech® New York 2013
2011
2010
2009
5
Industry Trends
Traditional backup systems are declining
LegalTech® New York 2013
6
Industry Trends
Firms increase use of disk-based backup
LegalTech® New York 2013
7
Industry Trends
…increase use of replication technology,
LegalTech® New York 2013
8
Industry Trends
…and slowly begin to explore online backup.
LegalTech® New York 2013
9
RPO and RTO
RECOVERY POINT OBJECTIVE (RPO)
The amount of data loss associated with a service failure.
The Recovery Point Objective (RPO) is the point in time to which you must recover data as
defined by your organization. This is generally a definition of what an organization
determines is an "acceptable loss" in a disaster situation. If the RPO of a company is 2
hours and the time it takes to get the data back into production is 5 hours, the RPO is still 2
hours. Based on this RPO the data must be restored to within 2 hours of the disaster.
RECOVERY TIME OBJECTIVE (RTO)
The amount of downtime associated with a service failure.
This is meant to describe the amount of time between when a service becomes
unavailable to users and when service is restored. This includes time to try to fix the
problem without resorting to the recovery procedures, the time to enact the recovery
process, testing of the recovered system and communication to the users.
LegalTech® New York 2013
10
RPO and RTO
Applications
Application
Perceived
RTO
Perceived
RPO
Achievable
RTO
Achievable
RPO
Desired
RTO
Desired
RPO
Messaging
DMS
Accounting
Lit Support
Remote Access
CRM
Docketing
HR
Records
Management
Knowledge
Management
Other practice
area specific
systems
LegalTech® New York 2013
11
Strategic Disaster Recovery
with Exchange 2010
Steven Shock
Chief Technology Officer
Irell & Manella LLP
SShock@irell.com
(310) 203-7511
LegalTech® New York 2013
12
Irell & Manella LLP
• 450 users
• 2 Offices
• IP Litigation
LegalTech® New York 2013
• Concierge Service
• Retention Policy
13
Irell Environment
•
•
•
•
•
•
•
•
•
Los Angeles Primary Data Center
Newport Beach Data Center
Arizona Data Center
HP Server – DL 380: 20 Hosts
HP Storage – HP P4000 Lefthand: 1.5 Petabyte raw storage
VMWare 5.1 - SRM
Cisco Nexus Core Switching
MPLS – 100mbps each location
Watchguard Branch Office VPN – LA & AZ
LegalTech® New York 2013
14
BASIC CONCEPTS
LegalTech® New York 2013
15
Client Connections
MAPI Connects
through the CAS
Role
What “Mailbox”
Server is listed in
Outlook Profile?
LegalTech® New York 2013
16
Database Availability Group
• Replication Boundary
• High Availability vs. Site Resiliency
– HA is Site-based
– Site Resiliency Across Sites
• Consider Recovery Objectives
• Quorum Isolation
LegalTech® New York 2013
17
Redundancies
• Servers / VMs
• Load Balancing
LegalTech® New York 2013
• Multiple Database Copies
• Shadow Redundancy
18
Design Philosophy
• New Design Philosophy
– Recovery Objectives
– Database Design
• Dependent on Environment
– Simple
– Complex
– Moderately Complex
• Staggering Number of Options!
LegalTech® New York 2013
19
Design Options
Too Many Options!
• High Availability
•
•
•
•
Exchange Native
VMware
Third Party On-premise
Third Party Email Continuity
• Client Connectivity
•
•
•
•
•
Outlook AnyWhere
Outlook WebApp
SSLVPN / UAG
Mobile Devices
MDM
LegalTech® New York 2013
• Replication
•
•
•
•
Exchange Native
Third Party Software
SAN-based
Future Integration
• Disaster Recovery
• Exchange Native
• Third Party Email Continuity
• Third Party On-premise
• Software Based
• SAN Solutions
20
Recovery Objectives
• Reasonable Objectives are Usually Exceeded
– Don’t Increase Firm Expectations!
• What Failures are you Designing for?
– Failure Domain(s)
• Time Objective (RTO)
• Point Objective (RPO)
LegalTech® New York 2013
21
Database
• Design Philosophies: Old and New
– Policy
– Backups
– Recovery Objectives
• Large Databases
• Archive Database
• Lagged Copies
LegalTech® New York 2013
22
Database
• Single Item Recovery
• Recovery Database
• Override Mount Dial
• Offline Database Copy Seeding
–
http://blogs.technet.com/b/timmcmic/archive/2011/07/12/exchange-2010-using-vss-toperform-an-online-offline-database-seed.aspx
LegalTech® New York 2013
23
CAS / Hub Roles
• Multiple Servers Hosting Roles
• Production and DR
• NLB for CAS
• Hub – Native HA
LegalTech® New York 2013
24
Network / WAN
• WAN Redundancy
– Links
– Carriers
– Equipment
• Replication Network
• DAG Replication Maximum Latency
– Adjust Cluster Settings?
LegalTech® New York 2013
25
Backup / Recovery
Portable
Database
Recovery
Database
SAN
Replication
LegalTech® New York 2013
Monolithic
Point in
Time
No
Backups
26
Other Options
• HA at Virtualization Level
• SAN Replication
– Database or Full Environment
• Managed & Cloud Services
LegalTech® New York 2013
27
DR Testing
• Worse Than a Disaster?
• Full Failover vs. Unit Testing
• VMware / Site Recovery Manager
– Restore Environment in a “Bubble”
– Restoring DAG Can be Difficult
– Consider New Server and Database Portability
LegalTech® New York 2013
28
Site Failover
• Database Failover
• Activation Blocking
• CAS Proxying
LegalTech® New York 2013
• Site Failover
– Start/Restore DAG
– Update DNS
• Internet Mail
29
Site Failback
• Restore / Resume Servers
• Resume / Update Database Copies
• Verify Health
• Failover Databases
• Update DNS
LegalTech® New York 2013
30
Lessons Learned - 1
• NLB / Kerberos Authentication
– Consider DR site
– http://technet.microsoft.com/en-us/library/ff808312.aspx
• DAG in a Single Site?
– Put DAG in DAC Mode Anyway
– DatacenterActivationMode [DAGName] –DatacenterActivationMode DAGOnly
• Only One CAS Server?
– Still Consider using CASArray Setting
– Future Growth
LegalTech® New York 2013
31
Lessons Learned - 2
• Failback URL
– http://blogs.technet.com/b/mbaher/archive/2010/11/27/robert-s-rules-ofexchange-namespace-planning.aspx
• Namespace Considerations
– Internet Name(s)
– MAPI
– Don’t Mix!
LegalTech® New York 2013
32
Steven Shock
Chief Technology Officer
Irell & Manella LLP
SShock@irell.com
(310) 203-7511
LegalTech® New York 2013
33
Q & A: Steven Shock
LegalTech® New York 2013
34
Back Up/Recovery with VMWare
Jim McCue
Information Systems Manager
Rodey Law Firm
jmccue@rodey.com
LegalTech® New York 2013
35
Rodey Environment
• Overview
– 75 attorneys/150 people/2 offices
– Litigation/Business
– 30 mb QMOE WAN Albuquerque – Santa Fe
LegalTech® New York 2013
36
Rodey Environment
• Virtualization
– 80% virtual
– 2 datacenters (Albuquerque, Santa Fe)
– 4 VMWare 4.1 esxi hosts w H/A
– 30 Guests
• Exchange 2010
• Aderant
• Autonomy
• BES, File Servers, DCs, MailMarshal
LegalTech® New York 2013
37
Rodey Environment
• EqualLogic SANS
– Albuquerque 2 x PS6000XV
• 13 TB usable
• 3.5 TB data + replication
– Santa Fe PS6000E
• 11 TB usable
• 3 TB data + replication
• Local disk backup storage
– Albuquerque, Santa Fe
• Drobo B1200i: 12TB
• Disk to tape: Dell Powervault lto5 tape autoloader
LegalTech® New York 2013
38
Rodey Goals
• RPO (Recovery Point Objective)=4 hour
• RTO (Recovery Time Objective)=8 hours, 16 hours
– Tier one applications=8 hours
•Network Authentication
•E-Mail
•Documents
•Time Entry & Billing
– Tier two applications=16 hours
•E-discovery & Litigation DBs
•BES
•Citrix
LegalTech® New York 2013
39
Rodey Goals
• D/R
– Move operations to other office
– Replication
• Restores
– VM Level Recovery
– Application Level Recovery
•Files, SQL, Exchange, A/D
• Archive to tape->move to disk offsite in the future
LegalTech® New York 2013
40
EqualLogic
• Thin provisioning
• Snapshots
• Replication
– 6.0 firmware: Synchronous
• EqualLogic Snapshot Manager/VMware Edition (ASM/VE)
– VMWare Tools quiescence
– VMWare consistent image
• EqualLogic Snapshot Manager/Microsoft Edition (ASM/ME)
–
–
–
–
Volume Shadow Copy: Exchange, SQL
Requires guest attached storage
Transactional consistent image
No log truncation
LegalTech® New York 2013
41
Veeam Backup & Replication
• Bare metal hypervisor backup
– Vmware/Hyper-V
• Deduplication
• Compression
• Changed Block Tracking (VMware CBT)
LegalTech® New York 2013
42
Veeam Backup & Replication
• Application aware processing
– Exchange, SQL, AD, Sharepoint
– Veeam agent -> MS VSS agents
– No permanent guest agent
– Transactional consistent image
– Log truncation
LegalTech® New York 2013
43
Veeam 6 – distributed processing
LegalTech® New York 2013
44
Veeam Backups
• Full,
• Incremental
• Reverse Incremental
LegalTech® New York 2013
45
Veeam Replication
• Replication
– Full + VMWare snapshots
– Network mapping and re-IP
LegalTech® New York 2013
46
Veeam Replication
• Replication Recovery
– Failover
– Failback
LegalTech® New York 2013
47
Veeam Verification
• SureBackup
–
–
–
–
Live Verification
Virtual Lab
Application Group
Scheduled
LegalTech® New York 2013
48
Veeam Verification
• Application Group - Role
LegalTech® New York 2013
49
Veeam Restores
LegalTech® New York 2013
50
Veeam Restores
• Instant Recovery
– Mount backup file as NFS
– Track VM changes to store
– Vmotion backup file +changes
• VM
– Overwrite, new location
• VM Hard Disks
• Guest Files
• Application Items: Application Item Recover
– Exchange, SQL, AD
LegalTech® New York 2013
51
What does Rodey do?
• Backups
– Veeam to disk
– Backup Exec disk to tape
• Replication
– Veeam to remote SAN
• Snapshots
– Equallogic to same SAN
LegalTech® New York 2013
52
EqualLogic & Veeam
• Application specific tips
– Active Directory
•avoid restore/failover
– SQL
•Use native SQL backups to disk
•Don’t truncate logs with Veeam
– Exchange
•Tie to A/D backup
•DAGs - watch for failover
LegalTech® New York 2013
53
EqualLogic & Veeam
• Things to watch out for
– Stagger scheduling
•Backups
•Replication
•Snapshots
•Backup of backups
– Open snapshots: alarms
– Virtual or physical
– Don’t backup to VMWare disks!
•Guest attached storage
LegalTech® New York 2013
54
Jim McCue
Information Systems Manager
Rodey Law Firm
jmccue@rodey.com
(505) 766-7504
LegalTech® New York 2013
55
Q & A: Jim McCue
LegalTech® New York 2013
56
Backing Up with Evault
Steve Skidmore
IT Director
Martin Clearwater & Bell LLP
skidms@mcblaw.com
LegalTech® New York 2013
57
Backing Up with Evault
•Martin Clearwater & Bell LLP
– Litigation Firm
– IT Department
LegalTech® New York 2013
58
Backing Up with Evault
•Risk Assessment
– Security Risk Tolerance
•Interruption to business can be costly
•Responsibility to clients
– Geographical Location
•United Nations, Israeli Consulate, Grand
Central Station
– Hardware / Software Infrastructure
– Regulatory Compliance Requirement
LegalTech® New York 2013
59
Backing Up with Evault
•Risk Assessment
– Disaster
– Outages
– Planned Maintenance
LegalTech® New York 2013
60
Backing Up with Evault
•Risk Assessment
– Essential Applications
– Recommended Applications
– Non-Essential Applications
LegalTech® New York 2013
61
Backing Up with Evault
•Risk Assessment
– Who is going to access the recovery
system?
– Where are they going to access from?
– How will they access the system?
LegalTech® New York 2013
62
Backing up with Evault
•Things to consider
– Traditional backups
•Significant capital
•Costly to scale
•Multiple systems
– Cloud-based DR services
•Turnkey solution
•Virtualized environments
•Overburdened
•Budget-friendly
LegalTech® New York 2013
63
Backing up with Evault
•Cloud Disaster Recovery Service
– Is data securely transferred?
– Is data securely stored?
– How does the authentication process work?
– Meet regulatory requirements?
– Bandwidth capacity?
LegalTech® New York 2013
64
Backing up with Evault
•Cloud Disaster Recovery Service
– Secure,Virtual hot cloud environment
– SSAE 16-compliant data center
– ISO Certified
– HIPAA Compliant
– Offsite Solution
– 4hr SLA
LegalTech® New York 2013
65
Backing up with Evault
•Hybrid Approach
– On Prem
– Cloud
LegalTech® New York 2013
66
Backing up with Evault
•Setup
– Application Plug-ins
– Backup Agents
– Collector S/W
– Additional NIC
– Subnet
– VPN
LegalTech® New York 2013
67
Backing up with Evault
•Setup
– Replication
•Vaults
•Servers
– Utilities
•Migrate
•Optimization
•Test
LegalTech® New York 2013
68
Q & A: Steve Skidmore
LegalTech® New York 2013
69
Q & A: General
LegalTech® New York 2013
70