Exercising Moving From Good to Great
A member discussion facilitated by
the MCPF Board
March 17, 2011
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
1
Overview
The purpose of this 90 minute facilitated discussion
is to look at why and how we conduct business
continuity and emergency management exercises
and to share “best practices” that can in turn be
used to improve our own exercises.
Active Participation = Greater Success
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
2
Agenda
•
•
•
•
•
Review September 2010 meeting
Provide updates
Facilitated Group Activity
Wrap Up
Close
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
3
Review - Types
Exercise Format
•
•
•
•
Talk-Through / Table Top
Simulation / Connectivity
Integrated
Live
Testing
• Building / Data Center
Infrastructure
• Business Continuity
• Crisis Management
• Emergency Responder
• Executive
• Human Capital
• Supply Chain
• Technology
• Third Party Vendor
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
4
Review Types - Updated
Exercise Format
•
•
•
•
Talk-Through / Table Top
Simulation / Connectivity
Integrated
Live
•
•
•
•
•
Orientation
Discussion based
Functional
Drill
Full-scale
Of Testing
• Building / Data Center
Infrastructure
• Business Continuity
• Crisis Management
• Emergency Responder
• Executive
• Human Capital
• Supply Chain
• Technology
• Third Party Vendor
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
5
Review Exercise Phases
• Planning
• Preparing
• Executing / Conducting
• Follow-up / Resolution
• Closure / Next Exercise Date
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
6
Review Exercise Phases –
Updated
• Planning
• Preparing
• Executing / Conducting
• Follow-up / Resolution
• Closure / Next Exercise Date
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
7
Review - Testing Partners
• Business Leaders
• Customers
• Internal Auditors
• Third Party Vendors
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
8
Review - Testing Partners
Updated
• Business Leaders
• Customers
• Internal Auditors
• Third Party Vendors
Don’t forget about:
• Local, state, federal
government
• Regulators
• First responders
• Utility providers
• Suppliers
• Media (cautiously)
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
9
Review - Discussion Questions
for the Types of Testing
• Assumptions
• Challenges
• Actions/Tasks
• Budget
• Resource Commitments
• Requirements – Goal
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
10
Types of Testing
Disaster Recovery
Power Down
Building/Data Center Infrastructure:
Assumptions:
•
Do you have a back-up building?
•
Valid security policies and monitoring of data center entry
•
When does timing of RTO start – point of disaster or disaster
declaration?
Challenges:
•
Measure against RTO
•
Tenant of a building - POP of telecommunication
•
Identifying test time
Actions/Tasks:
•
Ensure diesel fuel supply in case of emergency
•
Map on infrastructure from generator – ask data center
facilities team to ensure this exists (what is failing over and
what is not).
•
Communication, communication, communication…
Budget:
•
Part of IT budget
Resource Commitments:
•
Timing – weekend or off hours, peak versus non-peak season
•
Customer communication
Requirements – Goals
Considerations:
•
Telecommunications
•
Underload
•
Third-party vendors
•
Fire control/suppression systems
•
Generator – failover to UPS or generator
•
Ensure emergency responders understand location of data
center, critical infrastructure – NO WATER in data center!
•
Announced versus un-announced for building
•
Reasonable test if thinking of unannounced – ramifications
•
Tabletop Exercise – un-announced – after fire drill, then
notify team going to conduct a tabletop exercise – do not
allow them access to work area to get plan.
Question & Answers:
•
Does generator need to cycle gas? Can use stabilizer, run
once a month or week?
Lessons Learned:
•
Need to determine method to get folks back to work after an
evacuation like a bomb scare.
•
Ensure the right folks can get access to facility after an
incident.
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
11
Types of Testing
Business Continuity
Crisis Management
Types of Test:
•
Talk-Through / Tabletop
•
Simulation / Connectivity
•
Integrated
•
Live
Options:
•
Alternate Work Area – Internal, 3rd party
•
Work from Home – ensure capability is set up prior to event,
test capacity of VPN, training employees on process, security
guidelines
Assumptions
Challenges:
•
Maintaining a living document
•
Business to own their plan
•
Actions/Tasks
Budget
Resource Commitments
Requirements – Goals:
•
Meet RTO
•
Ensure business understands RTO
Lessons Learned:
•
Transportation to alternate site
Reference:
•
Red Cross Ready Rating Program
Assumptions:
Define crisis for your organization
Challenges
Actions/Tasks
•
Training employees on how to react to an emergency
•
Quick reference cards/wallet cards
•
Communication - practice
Budget
Resource Commitments
Requirements – Goal
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
12
Types of Testing
Executive
Third-Party Vendor
Tabletop exercise
•
Scenario, walk-through guides and wallet cards
Engage Executives
Assumptions:
•
If plan is not in place, executives will take charge,
take over.
Action Items:
•
Ensure executives know and understand their roles
- training
•
Executives can talk publicly in front of a camera
•
Separate command center for
executive/management team and technical teams
•
Train assistants that manage logistics for the
executives
Requirements/Goals:
•
Test support personnel
•
Engage them in the process
•
Test executive response
Lessons Learned:
•
Do not have executives in command center
Assumptions:
•
If vendor does not have plan, they don’t test.
Actions:
•
Send questionnaires/risk assessment to top #
vendors
•
Test with a couple of vendors
•
Ensure contractual agreements with vendors
include disaster recovery testing
•
Ability to audit vendor
Challenges:
•
Coordinating test dates, time, resources with
vendor
•
Include vendor in your own disaster planning and
testing
•
Identify secondary and tertiary individuals to fill key
roles
•
“Rehearsals” – have to rehearse enough that
employees can seamlessly move into crisis response
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
13
Things That Contribute
to a Good Exercise
•
•
•
•
•
•
•
•
•
•
•
An understanding by all that there are no wrong answers or actions. The purpose
of the exercise is to identify what works and doesn't work.
Realistic exercise scenario with no "cutesy gotchas" or trick injects.
It should be Realistic, Relevant and Revealing
No worst-case scenarios (because I already know how to pray, and that is the only
solution to the worst case scenario).
An invested planning group
Have a good planning committee that is committed to the overall outcome of the
exercise
As a planner be flexible and look for workable solutions to benefit all
Challenging for the participants/responders
SMART Objectives - additionally, make sure that your objectives identify the
Audience, the Behavior that you what the audience to perform, under what
Condition it (the behavior) is to be performed under, and to what Degree of
accuracy it (the behavior) is to be performed.
Make it fun
Putting into practice learned lessons.
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
14
SMART Objectives
• Specific – Is the wording precise and
•
•
•
•
unambiguous?
Measurable – How will achievements be
measured?
Action Oriented – Is an action verb used to
describe expected accomplishments?
Realistic – Is the outcome achievable with given
available resources?
Time Sensitive – What is the timeframe (if
applicable)?
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
15
Things That Can Hurt an Exercise
• Trying to cram too much into the exercise or making it too complex
- "KISS“
• Time frame too short for adequate planning will kill an
exercise...adequate time for planning is a must
• Inadequate preparation by all participants prior to the exercise
• Less than 100% by all participants in terms of physical and mental
involvement.
• Lack of Commitment, Communication, and Coordination
• Unrealistic scenario for the players or jurisdiction
• Too many people on the planning committee
• Trying to make the exercise "'everything to everybody," and too
much artificiality.
• Any exercise not tied into a cycle of training and exercises is a waste
of time.
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
16
Which of the following are you most worried about being not up-to-speed in
your organization should an incident occur?
Security Executive Council Poll
03/15/3011
www.securityexecutivecouncil.com
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
17
Group Activity
Congratulations – you are a member of the MCPF Exercise
Team!
1. Break up into five groups – Planning, Preparing (Design &
Development), Executing & Conducting, Follow-up &
Resolution (Evaluation), Closure & Next Exercise
(Improvement Planning).
2. Select a Spokesperson and a Documenter for your group
3. Your goal is to brainstorm (WHAT?) related to your team
assignment
4. Use the easel paper and markers to capture your ideas
5. You will have 30 minutes to complete this task and then
participate in a group debriefing
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
18
Today’s Scenario
•
•
•
•
•
•
•
Earlier this morning at 9:00 AM, a 7.7 magnitude earthquake occurred along the New Madrid
Seismic Zone (NMSZ). Fifteen county regions along the Mississippi River, with a population of
approximately 1.9 million people are seriously affected. The counties include Dunklin,
Pemiscot, New Madrid, Stoddard, Butler, Mississippi, Scott, Bollinger, Cape Girardeau, Perry,
Ste. Genevieve, Jefferson, St. Louis, St. Louis City, and Wayne.
Our region is considered to be in the affected area.
Police and MODOT authorities are asking motorists to stay in place and not drive unless it’s
an emergency, as inspectors are assessing damage to roadways and bridges.
Utility outages are widespread throughout the metro St. Louis area.
The area within 50-75 miles of the epicenter was subjected to shaking on the Modified
Mercalli scale at an intensity of VII or greater, strong enough to destroy well-built structures,
damage dams and reservoirs, cause landslides, and severely damage or destroy
transportation structures such as roads, highways, bridges, and railroad tracks.
Soil-liquefaction occurred in some areas, thereby increasing the level of destruction as
quicksand-like conditions contributed to the destabilization and collapse of numerous
buildings, transportation, and utility structures.
Assume current day, time, and weather conditions.
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
19
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
20
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
21
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
22
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
23
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
24
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
25
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
26
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
27
Debriefing
•
•
•
•
What did we set out to do?
What actually happened?
Why did it happen?
What are we going to do differently next
time?
• Are there lessons learned that should be
shared?
• What follow-up is needed?
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
28
Additional Resources
•
•
•
•
•
•
•
•
•
Disaster Recovery Journal: www.drj.com
Disaster Resource Guide: www.disaster-resource.com
Continuity insights: www.continuityinsights.com/
FEMA HSEEP: https://hseep.dhs.gov/pages/1001_HSEEP7.aspx
MCPF Group in Linked In: www.linkedin.com
CUSEC: www.cusec.org
USGS: www.usgs.gov
Missouri SEMA: www.sema.dps.mo.gov
Natural Hazards Observer: http://www.colorado.edu/hazards
• Each other!
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
29
NLE 2011
• In May 2011, the Federal Emergency Management Agency (FEMA)
will conduct the National Level Exercise 2011 (NLE 2011;
www.ready.gov/nle2011/index.html ). The purpose of the exercise
is to prepare and coordinate a multiple-jurisdictional integrated
response to a national catastrophic event.
• NLE 2011 is a White House directed Congressionally-mandated
exercise that will focus on regional catastrophic response and
recovery activities between federal, regional, state, tribal, local and
private sector participants.
• The focus of the exercise will simulate the catastrophic nature of a
major earthquake in the central United States region of the New
Madrid Seismic Zone (NMSZ). The year 2011 is the bicentennial
anniversary of the 1811 New Madrid earthquake, for which the
NMSZ is named. NLE 2011 will be the first NLE to simulate a natural
hazard.
MidAmerica Contingency Planning Forum - Not to be used or
reproduced without written authorization.
30