Crisis Management – Process &
Review
Robert DiLossi – Director Crisis Management
Robert.DiLossi@sungard.com
215-446-2205 Office
www.sungardas.com
Steps to declare a disaster…
1.
(866) 722-1313 or 215-351-1313 (Service Desk) or
2.
Call Logged, Appropriate Staff Notified
3.
Call Back Process
4.
Disaster Declaration Authorization List (DDA)
5.
Password Authorized, if applicable
6.
Identify and Review Declaration
7.
Needs/Alternatives
8.
Site Preparation
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
2
‘Alerts’ – what are the advantages…
1.
No cost to You, our Customer
2.
Heightened awareness
3.
Review last test, DDA, etc.
4.
Logistics
5.
Review contract requirements/alternatives
6.
Document configuration requirements
7.
Follow up on every alert until issue
is resolved
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
3
# of Alerts
Alert Totals – 2010
2010
Alerts via Phone
Alerts via RSPortal
(Source: SunGard Availability Services US data) – Chart 1
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
4
Proactive Calling Map
Mt. St. Helens
Eruption
2003 Northeast
Power Outage
N. CA
Earthquake
Hurricanes
California Fires
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
5
Proactive Calling, what we review…
1.
Communication Tools and Protocols
2.
Review Contract, DDA, etc.
3.
Recovery Plans and Last Test
4.
Logistics / Travel / Evacuations
5.
Roles & Responsibilities
6.
Tape and Vendor Availability
7.
Damage Assessment Teams
8.
Timeline
9.
Corporate and External Agency Activation
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
6
Crisis Management Center
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
7
SunGard Availability Services
Key Lessons Learned over the years…
 Key Players Unavailable or Missing
 Communication Issues
 Lack of Transportation/Lodging/Logistics
 No Plan for Extended Recoveries
 Outdated Documentation/Scripts
(i.e., DDA, Technical Scripts, Employee Contact Lists, etc.)
 Difficulty Accessing Tapes/Data/
Documentation
 Incomplete Contracts – Any Vendor
(i.e., Media Storage, Insurance, Fuel, etc.)
 Lack of Content “Business Continuity Plans”
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
8
SunGard Total Resource Utilization
Declarations
# of Disasters per Event
900
783
800
Alerts
700
600
481
500
400
300
200
241
121
155
105
66
100
81
87
38
0
September 11,
2001 Terrorist
Attack
2003
2004
2005
Northeast Power Hurricane Season Hurricane Season
Outage
2008 Hurricane
Season
Total Resource Utilization
39%
35%
40%
43%
32%
Note: SunGard total utilization has never exceeded 43%
(Source: SunGard Availability Services US data) – Chart 2
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
9
SunGard Disaster experience per year
Disaster Experience
200
176
180
162
160
# of Disasters
141
140
124
116
120
100
127
91
80
60
121
78
76
71
90
77
69
66
54
52
46
53
32
40
33
13
20
0
90
91
92
93
94
95
96
97
98
99
00
01
02
03
04
05
06
07
08
09
10
11
Year
(Source: SunGard Availability Services US data) – Chart 3
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
10
Types of Disasters or Major Business Disruptions:
Declaration Events
 SunGard: 100% success rate for customer recoveries
(2,406 to date)
700
573
# of Disasters
600
500
400
349
275
300
176
200
100
38
30
63
105
98
27
19
0
Bomb
Building Earthquake
Fire/
Threat/
Damage,
Electrical,
Evacuation
Gas/
Explosion
Water
Break
Flood
Hardware Hurricane/ Network
Failure
Weather
Outage
Power
Outage
Software Terrorism
Event Type
(Source: SunGard Availability Services US data) – Chart 4 – Since 1999
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
11
Disasters by Platform
900
814
800
# of Disasters
700
600
476
500
379
400
300
252
185
200
80
100
74
212
100
80
86
71
23
9
0
SGN, TS,
Internet,
Sorter
AS/400
DEC
Dedicated/ HP
HA
9000
Managed
Services
Mainframe
Megavoice
MMC,
PCLAN RS6000
Gensets, (Intel)
VSAT
SUN
Micro
Trading Workforce
Floor
Continuity
Platform Type
(Source: SunGard Availability Services US data) – Chart 5 – Since 1999
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
12
Disaster Experience – Recovery Options
500
450
435
# of Disasters
400
350
300
250
185
200
150
100
29
50
27
0
Hotsite
(Center Based
or Mobile in
Hotsite)
Mobile
Generators
Shipped
Mobile
(Shipped
Equipment)
Mobile
Datacenter or
Metrocenter
Deployed
Recovery Options
(Source: SunGard Availability Services US data) – Chart 6 – Since 2004
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
13
Disasters Declared by Business Type
30
26
25
20
18
15
14
14
2010
2009
2008
10
10
9
9
5
5
5
5
4
3 3
3
2
3
3
2
1
0 0
4
3
0 0
1
2
1
0
0
1
0
0
1
0
1
0 0
2
1
0
0
(Source: SunGard Availability Services US data) – Chart 7
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
14
Workforce Continuity seats used - by Facility - 2010
1200
980
1000
800
600
575
# of seats used
# of disasters
400
262
200
100
30
3
1
10
1
3
1
22
40
3
10
1
5
2
0
Carlstadt
Copley
Cypress
Long Island
City/Queens
Marlborough
Mexcio City
San Ramon
Toronto
Wood Dale
(Source: SunGard Availability Services US data) – Chart 8
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
15
Workforce Continuity disaster event types - 2010
12
10
10
8
6
5
4
2
2
2
1
0
0
Hurricane
Flood
0
0
Power Outage
Strike/No Access
Network Outage
Water Main Break
Gas Main Break
Fire/Explosion
(Source: SunGard Availability Services US data) – Chart 9
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
16
SunGard Disaster Experience
Average # of Days in Disaster Mode
19.2
19
19.0
18.8
18.6
18.4
18.2
18
18.0
17.8
17.6
17.4
Hotsite
Mobile
(Source: SunGard Availability Services US data) – Chart 10
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
17
SunGard Disasters allocated by Facility
2010 # of Disasters by Facility
10
9
9
8
7
7
6
5
5
4
4
2010 # of Disasters
3
2
2
2
2
1
1
1
1
1
1
0
(Source: SunGard Availability Services US data) – Chart 11
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
18
What can you do at – www.mysungard.com
DDA Online – Your Disaster Declaration Authorization (DDA) Form is a list
of individuals who are authorized to declare a disaster for your company.
Logon to www.mysungard.com
• Access and Update DDA Information 24/7 by person or by schedule


Delegate access to DDA Information by company, person and by schedule
Data is updated within the hour for final changes
• Alerts Online



Access and submit online Alerts 24/7 via the Recovery Services Portal
Customer will receive call back on all alerts
No costs to Customer
• Crisis Management Information and Tips




Sample disaster timeline
Alert, Disaster and Mock Disaster Flow Chart
Disaster Workbook
Crisis Management Statistics
• Document Repository (new in 2010)

Add critical business recovery documents 24x7x365
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
19
Thank You! – Questions?
© 2010 SunGard Availability Services LP. - All Rights Reserved - www.sungardas.com
20