INTRODUCTION TO EXERCISE DESIGN & EVALUATION INTRODUCTIONS Name Agency Exercise Experience Course Goals OBJECTIVES Familiarize participants with and/or review the exercise design and evaluation process Introduce MTDES evaluation documentation Examine and analyze HSEEP Toolkit documents and systems REASONS TO EXERCISE Establish a comprehensive exercise program Reveal planning weaknesses Reveal gaps in resources Satisfy regulatory requirements Clarify roles and responsibilities Test and evaluate plans, policies, and procedures BENEFITS OF EXERCISING Individual Training System Improvement REASONS TO USE HSEEP Common exercise policy and program guidance Consistent terminology Provides tools for exercise design, conduct, and evaluation PREPAREDNESS CYCLE CAPABILITIES-BASED PLANNING Reviewing current plan Conducting needs assessment Assessing jurisdiction’s capability to conduct an exercise EXERCISE PROGRAM MANAGEMENT Project Management Multi-Year Planning Budgeting Grant Management Staff Hiring Funding Allocation Expenditure Tracking MULTI-YEAR TRAINING & EXERCISE PLAN Goals Components Capabilities-Based PROGRESSIVE EXERCISES Component of a comprehensive exercise program Goal: Large-scale functional or full-scale exercise Small-scale discussion-based and operations-based exercises in preparation for large-scale exercise EXERCISE TYPES DISCUSSION-BASED EXERCISES SEMINAR Overview/Introduction Purpose Leadership Participants Applications WORKSHOP Purpose Characteristics Participants Applications TABLETOP Purpose Characteristics Participants Applications GAME Purpose Characteristics Applications OPERATIONS-BASED EXERCISES DRILL Purpose Characteristics Applications FUNCTIONAL EXERCISE Purpose Characteristics Applications FULL-SCALE Purpose Characteristics Participants Applications EXERCISE PROJECT MANAGEMENT Foundation Design and Development Conduct Evaluation Improvement Planning FOUNDATION FOUNDATION Base of Support Exercise Planning Timelines Exercise Planning Team Exercise Planning Conferences BASE OF SUPPORT Senior Officials Buy-In Purpose EXERCISE PLANNING TIMELINES Target timeframes Multi-Year Training and Exercise Plan Origin & Composition EXERCISE PLANNING TEAM Exercise Planning Team Leader Purpose Responsibilities Safety (Ops only) Composition ICS Operations Site Liaison Resources Planning Exercise Documentation Evaluation Logistics Admin Finance Props Reporting Actors Budgeting EXERCISE PLANNING CONFERENCES Concept and Objectives Meeting Initial Planning Conference Mid-Term Planning Conference Master Scenario Events List Conference Final Planning Conference DESIGN & DEVELOPMENT EXERCISE DESIGN STEPS Injects Assess Needs Expected Actions Define Scope Major & Detailed Events Write a Statement of Purpose Define Objectives Narrative Compose a Narrative Write Major and Detailed Events List Expected Actions Prepare Injects Define Objectives Statement of Purpose Define Scope Assess Needs ASSESS NEEDS Purpose Needs Assessment Process Evaluations from Past Exercises Needs Assessment Product TARGET CAPABILITIES Capabilities-based planning Capabilities-based exercises Tasks & Objectives DEFINE THE SCOPE Purpose Determining Functions Key Elements WRITE A STATEMENT OF PURPOSE Purpose Goal Content DEFINE OBJECTIVES Purpose Origin SMART System Simple Measurable Number Achievable “Good” Objectives Realistic SMART Objectives Task Oriented SCENARIO Purpose Characteristics Narrative Outline WRITE MAJOR AND DETAILED EVENTS Major Events Sample 1. Fuselage breaks apart and hits buildings below 2. Jet fuel ignites several homes in the area 3. About 60 survivors are thought to be trapped in the front section of the aircraft 4. Estimates of passenger casualties rise between 200 & 220 deaths and at least 70 severe burn victims Purpose Goal Strategy Origin Sample WRITE MAJOR AND DETAILED EVENTS (CONT.) Detailed Events Sample 1. The mortuary is unable to accept the large numbers of remains resulting from the crash. 2. Local hospitals lack specialized facilities and personnel to treat large numbers of severe burn victims. 3. The American Red Cross has agreed to set up a family information center to link victims and their families Purpose Goal Strategy Origin Sample LIST EXPECTED ACTIONS Purpose Goals Types of Actions Guidelines Relationship to Objectives Example DOCUMENTATION Situation Manual Exercise Plan Controller & Evaluator Handbook Master Scenario Events List Player Handout Exercise Evaluation Guides Exercise Policies LOGISTICS Discussion-based Exercises Operations-based Exercises EXERCISE CONDUCT EXERCISE CONDUCT Setup Briefings Management of Facilitators, Controllers, Evaluators, Players, and Actors Wrap-Up Activities SETUP Site Visits Pre-Exercise Check Discussion-Based Exercises Operations-Based Exercises PRESENTATIONS/BRIEFINGS Discussion-Based Exercises Operations-Based Exercises PERSONNEL Facilitators Controllers Evaluators Players Actors Simulators EXERCISE EVALUATION EVALUATION GOALS Identify successes and shortfalls of: Response organizations Planning documents Facilities Emergency Management System Training/Staffing deficiencies Equipment Improve Emergency Management Capabilities EVALUATION STEPS Plan and Organize the Evaluation Observe the Exercise and Collect Data Analyze Data Develop the Draft After Action Report & Improvement Plan PLAN & ORGANIZE THE EVALUATION EVALUATION TEAM LEADER Qualifications Responsibilities Designation EVALUATION REQUIREMENTS & METHODOLOGY Purpose Defining the Team Structure Criteria Strategies Evaluation Plan EVALUATION TEAM Team Structure Responsibilities Strategies RECRUIT, ASSIGN, AND TRAIN EVALUATORS Recruiting Experience & Expertise Time Commitment Establishing Base of Evaluators Assigning Strategy When to Assign Training Purpose Materials Instructions FINALIZE EVALUATION PLAN Exercise-Specific Information Evaluator Team Organization, Assignments, and Location Evaluator Instructions Evaluation Tools CONTROLLER AND EVALUATOR BRIEFING Purpose Composition OBSERVE THE EXERCISE AND COLLECT DATA EXERCISE EVALUATION GUIDES Purpose Use Composition DISCUSSION-BASED EXERCISES Higher Level Capability Issues Types of Information Collected Debrief OPERATIONS BASED EXERCISES Task & Activity Level Observations Types of Information Collected Use Events to Gauge Success Evaluator Guidelines HOT WASH Purpose Roles and Responsibilities Guidelines COLLECTING SUPPLEMENTAL DATA Delegation Purpose of Data Examples ANALYZE DATA IDENTIFYING ROOT CAUSE AND DEVELOPING RECOMMENDATIONS What Happened Why It Happened Root Cause Analysis DATA ANALYSIS Discussion-Based Exercises Operations-Based Exercises Controller & Evaluator Debrief Controller & Evaluator Debrief Draft AAR Content Draft AAR Content Preliminary Analysis Content Exercise Event Timeline Preliminary Analysis Content DEVELOP THE DRAFT AFTER ACTION REPORT & IMPROVEMENT PLAN AFTER ACTION REPORT Purpose Who is responsible for it? Form Format IMPROVEMENT PLANNING CONDUCT AFTER ACTION CONFERENCE Purpose Content Format IDENTIFY CORRECTIVE ACTION TO BE IMPLEMENTED Corrective Actions Purpose Content Realistic Goals IMPROVEMENT PLAN Purpose Components Capability Observation Recommendations Corrective Action Description Responsible Party/Agency Agency POC Start Date End Date CAPABILITY ELEMENTS Planning Organization and Leadership Personnel Equipment and Systems Training Exercises, Evaluations, and Corrective Actions FINALIZE AAR/IP Purpose Composition Goals TRACK IMPLEMENTATION Event Points of Contact Participating Entity Points of Contact Action Officers Continual Improvement QUESTIONS?