Business Continuity Planning vs. Disaster Recovery Planning Marilyn A. Blake, AU, CRM Joyce A. Hermann, AU, CISR There’s an old saying… No one plans to fail, they just fail to plan. What’s the Difference? Getting beyond just information systems recovery requires a more comprehensive type of plan than just a disaster recovery plan. Telecommunications companies cannot underestimate the importance of business continuity planning. What’s the Difference? Disruptions in service can be caused by power outages, floods, snowstorms, earthquakes or something as severe as a chemical or physical attack. It doesn’t have to be terrorism, hackers, or computer viruses—but it could be. Downtime from the disruption - whether it's hours, days or longer - can be costly. Parts of Continuity Planning Emergency/Disaster Planning Business Continuation Planning Crisis Management What is an Emergency? Any unplanned event that can cause deaths or significant injuries to employees, customers, or the public; Or, that can shut down your business, disrupt operations, cause physical/environmental damage, or threaten the company’s financial standing or public image. Every Year Emergencies Take Their Toll on Business in Lives and Dollars Goal of the Plan: Limiting injuries and damages and returning more quickly to normal operations Who’s Job is it? Preparedness is EVERYONE’s job; during the first few hours/days following an emergency, essential services may not be available. So, EVERYONE must be ready to act. (according to their assigned roles) Think About This... How long will your business last without computers or operating switches/equipment? What would happen if you were denied access to your facilities, server, or customer records? How long could you work without telephone service, electricity, water (utilities) or run only on generators? Even if these situations only kept your operations closed for a few days, it would be more than an inconvenience — especially if you had not planned how to handle it. Think about this…. If your building survived, without an business continuity plan, you have no guarantee that your business would. What if your customers didn’t all return? Even if emergency events only shut you down for a short period of time, your business would be interrupted and cause you discomfort. What is Business Continuity Planning? It is the process of preparing for (through a business impact analysis), mitigating, responding to, and recovering from an “emergency” to your operations/employees/ customers/property The process is dynamic Planning is critical, but training, drills, testing equipment, and community coordination are also essential components Reasons to Develop a Plan 1. 2. 3. 4. 5. 6. It is likely an emergency of some kind will effect you Safeguarding life and property (physical and financial) Employee morale Liability as utility provider (public utilities commission) Public image OSHA requirement (1910)--must be written if you have more than 10 employees What are Some Examples of an Emergency? Fire Computer shutdown Flood Tower damage Hurricane Power surges/failure Tornado Explosion Winter storm Civil disturbance (snow/ice/hail) Earthquake Lightning Wind Storm Unexpected loss of key supplier Labor Strife Pandemic flu 5 Steps in the Planning Process 1. Establish a planning team 2. Analyze capabilities and hazards 3. Develop the plan 4. Implement the plan 5. Re-evaluate annually or after it’s been used for updates/corrections/nuances #1-Establish the Team Size of the team will depend on the facility, but a group is best Functional areas to include are: – – – – – – – – Upper management Safety coordinator Line management Human Resources Engineering/maintenance PR/Community relations (links to community organizations) Accounting/purchasing Legal #2-Analyze Many documents are already in place (evacuation plan, employee manuals, insurance/risk management policies, purchasing procedures, etc.) List potential emergencies (historical examples, technological possibilities, human error factor), their probability, and the best way to minimize it Local organizations can help (Fire department, Red Cross, National Weather Service, Police department, construction companies, etc.) Vulnerability Analysis Chart/Example Rank on a scale 1-5 (low impact-high impact) The lower the score the better Type of Emergency Hurricane Power Outage Servers Down Probability Human Impact Property Impact Business Impact Internal Resources External Resources Total #2-Analyze (Con’t) Review your insurance & risk management policies – Are property values up-to-date? – Do you have coverage for floods, earthquakes, winter storms, tornadoes, etc.? Is NFIP available? – Do you have redundant systems to minimize your business interruption exposure in case of emergency? – What are your deductibles? – What about replacement for lost toll or data records? – Do you know how to call/fax/email in a claim? – What if you can’t get into the building? – Agent’s phone number in your cell phone #3-Develop the Plan Executive summary/mission statement Procedures (for reporting, escape, evacuation, resumption of operations) Support documents (call lists, site maps) Write the document (review and distribute) Establish a training schedule for employees Obtain upper management approval Distribute to employees Telcom has prepared a sample fill-in-the-blank telco-specific document as a starting point for Step #3 3.The Plan…at the Beginning Mission Statement—Sample In order to responsibly serve our customers, our communities, and your employees, ABC Telecom must be able to respond efficiently and effectively in all emergency situations and restore lost communications as rapidly as possible. The overall objective shall be returning customers communications service and the Cooperative’s operations to normal working conditions, while observing all safety precautions, as soon as possible. Table of Contents Areas to Consider Organizational Structure Plan—notification plans Employee Information Contractors Generators—locations/rental options Safety/Security/First Aid Vehicles/Equipment Utility Companies Table of Contents Areas to Consider Insurance Important Vendors Public Relations-releases/messages Central Office/Tower sites CATV Directories: NTCA, VTIA, other local associations Maps Organizational Structure Plan—notification plans Key functional areas/responsibilities – – – – – – – – Crisis Manager/Site Coordinator Engineering/Maintenance Officer Finance/Accounting Officer Human Resources Officer Security Officer Communications Officer Public Relations Officer Outside Members—Police/Fire/Rescue Communication Plan: first & second point of contact; employees; public: TV/radio/newspaper notification; twoway/cell phones/text messages Employee Information Departmental Organizational Charts Employee pager/cell/home phone numbers Employee Information List—of Crisis Team including connection to the internet or your network capabilities Contractors It may be necessary to bring in contractors either in preparation or during an emergency or to help clean-up afterwards – – – – – Splicing Construction CATV Engineering Computer/Network specialist Generators In many situations, generators may be necessary to continue your business operations. Don’t forget, refueling plans – – – – Portable trailer generators Portable generators COW Rental options Safety/Security/First Aid Security company contact information for your building (who has access) Security—who’s allowed where First-aid—list of responders/kits location (someone to inspect them on a monthly basis) Evacuation plans from all buildings (posted) Shelter/safe areas—identified and supplied (in each building with regular employees) Identify local hospitals/medical treatment options Vehicles/Equipment Vehicles: assigned to whom/VIN Trailers: haul fuel to generators, equipment to repair, sandbag before a storm, etc. Extra equipment in your warehouse to replace damaged equipment (inventory) Utility Companies Local emergency numbers – Emergency Management – City/County officials (for all of your locations) Local utility companies – Electric – Water & Sewer – Public Works Insurance Property-Casualty Agent/Claims reporting information Group Health Insurance Contact/claims reporting information Life insurance or AD&D contact/claims reporting information Important Vendors Banks/financial institutions Computer/data back-up company emergency contact numbers Building contractors NTCA and VTIA and other associations (others who can help you) Fuel companies Tower maintenance Towing services Central Offices/Tower Sites All 911 addresses identified with specifics on what equipment is at that location Is it Fiber or Copper? Circuit IDs and any passwords necessary Towers—owned and where you have leased equipment or shared tower space CATV CATV distribution layout from the headend Nodes Channel line-up Dish layout 2-way CATV areas Directories NTCA VTIA Other associations Local associations--community Maps I&R areas Generator locations Tower/CO/Switch sites City/County Buildings you own/have people or equipment Life Safety Plan-NFPA 101 Sample Areas Automatic Sprinkler Alarm system Emergency signs and lights 2 means of egress Exit doors unlocked Handicapped occupants/helpers Basement and upper levels to consider Emergency Pre-Storm Checklist Sample 72 hours Prior: make sure all generators are serviced, vehicles are fueled, security for the buildings, contact information for insurance/FEMA updated, equipment/ supplies tied down/inside (as much as possible) 48 hours Prior: backhoes/chainsaws checked; generators to appropriate places, educate employees on work orders/timesheets, maps of assigned areas 24 hours Prior: food preparation, secure buildings—caulking, sand bags, lock down building Emergency Pre-Storm Checklist Sample 12 hours Prior: check latest weather, distribute information/communications equipment to local emergency responders Don’t forget to have employees change their voice mails and emails to say you’re closed or have different hours; make sure there is a main line for customers to call Samples of Plan Contents Policies/Procedures Pandemic Flu—different from a traditional emergency because it’s not that you are shut-down from a disaster, your employees are sick and can’t come to work and/or your customers potentially are sick. Computer/Server Shut-down Bomb Threat Inclement Weather Storm—Pre-event planning Pandemic Flu Influence in the Plan Avian influenza (H5N1) is a virus capable of mutating from birds to humans of which there is no vaccine available Pandemics usually last 12-24 months Last 3 pandemics 1968 (3m deaths), 1957 (2m deaths), and 1918 (50m deaths) Medical community would not be inadequate Could effect 50% of our world populations World Bank estimates $800B in economic impact Pandemic—Business Effects 40% fewer staff (either sick or caring for loved ones who are sick) Huge demand for telecommuters—can your network handle it and can you install high speed connections for your customers Customers—coming in to pay their bills Local governments may quarantine Pandemic—Employees and Customers Identify essential employees and functions/ operations (procedures manuals/cross training) Modify frequency of face-to-face contact (hand-shaking, meetings, shared workspace How will sick leave and FMLA react Epidemics usually last 6-8 weeks and spreads randomly (not just the young and the old) and go in waves Identify how techs will enter homes/businesses or not during a wave in the community Keep up with www.pandemicflu.gov for updates on what the government is doing Emergency Example: Computer Server is Down Whether it’s a hacker, service interruption, or mechanical problem in your office: Identify essential or key employees Can employees work from home on a temporary basis? What computer equipment/connection do they have? How can you continue to serve your customers? Do you have off-site replication? How long does it take to “switch over”? What happens when you switch back to the data on the off-site server? Key providers’ contact information available? Emergency Example: Bomb Threat In the event you receive a bomb threat, the following info should be obtained and provided to your supervisor. It is paramount in case the threat is carried out and will assist the authorities: – – – – What the person said Male or Female Bomb locations and time of activation Anything additional Emergency Example: Inclement Weather Procedure Do you have a plan for bad weather (hurricane to blizzard)? Do hourly, salaried-non supervisor, and supervisors know what they are to do? How will they know updates? What if there is mandatory evacuation? Do you pay people still? Emergency Incidents While most emergency situations are handled locally, when there’s a major incident help may be needed from other jurisdictions, the state and the federal government. National Incident Management System (NIMS) was developed so responders from different jurisdictions and disciplines can work together better to respond to natural disasters and emergencies, including acts of terrorism. NIMS benefits include a unified approach to incident management; standard command and management structures; and emphasis on preparedness, mutual aid and resource management. ICS Features Designed to coordinate responders so they use the same terminology/equipment and apply the same principles Plain language with specific titles and terminology are key Titles of personnel are based on their function at the incident, not their rank/regular job title Incident Action Plan Spells out the strategy for managing the incident Provides supervisory personnel with directions Addresses 4 main elements: – What do we have to do here? – Who is responsible for doing it? – How do we communicate with each other? – What is the process if someone is hurt? Can be written or oral as the site safety plan It’s a chain of command system (fashioned similarly to the military system) Incident Command Posts Command post is positioned outside established and potential hazard zone, but close enough to maintain command Marked with a diagonally divided green/white square Staging Area Temporary locations where personnel and resources are kept between assignment and deployment. May be more than one staging area Equipment and personnel are considered “available” if they have checked in. Designated by a circle with a “S” inside Communications Poor communication can disrupt, slow down, hamper any incident response When an incident occurs, all responders must observe strict communication rules: – Use only equipment you’ve been trained on – Follow radio/phone procedures, like check-in and out; permitted frequencies, and radio silence – Use plain English—avoid jargons or codes that not everyone understands – Limit communication to essential information – Use secure communications when appropriate – Use full names/locations so everyone is on the same page (i.e. could be more than one Jim) #4-Implement the Plan Must become part of the corporate culture Should have walk-through and functional drills—at least annually and document them Evaluate and modify the plan as new operations begin or as situations dictate Make sure employees have read the plan and understand it and their roles Restoration Activities Check all buildings/equipment for damage, generators as appropriate (especially to 911 stations, police, hospitals, etc) Restore services to customers using employees, vendors, contractors, etc Report to insurance, FEMA, RUS/mortgage company Access back-up for billing, payables, disbursements, and payroll All completed: Thank you letters to all involved Conduct a post emergency review and recommend changes 5-Re-evaluate Any major changes in your “core” business continuity staff? Any new operations? Discontinue any operations? Any new rules/laws in your industry or state Has to be a work in progress…as you keep progressing! “An Ounce of Prevention…” No business continuity plan can guarantee that your telecommunications company won’t suffer any losses--but it can minimize the damage and help use all of your resources to protect your employees and your business. Resources Telcom Insurance Group (sample Emergency Preparedness Plan) www.FEMA.gov www.OSHA.gov www.EPA.gov www.RedCross.org www.ntca.org www.cvtma.org