SRT610 Business Continuity Plan Liquid Networks Page 1 of 47 Business Continuity Plan for Liquid Networks Prepared for SRT610 by: Jamaal Joseph Chris Lai Shawn Williams SRT610 Business Continuity Plan Liquid Networks Page 2 of 47 Table of Contents INTRODUCTION .................................................................................................................................3 Copies of This Plan .................................................................................................................................... 3 Calling a Disaster ....................................................................................................................................... 3 Scope and Objectives ................................................................................................................................ 4 FUNCTION 1: Authentication Services .................................................................................................5 Overview ................................................................................................................................................... 5 Scenarios covered ..................................................................................................................................... 5 Prepared by ............................................................................................................................................... 5 Team ......................................................................................................................................................... 5 Authentication Service, Scenario1: Power outage at the office ............................................................... 6 Recovery Steps Summary...................................................................................................................... 7 Recovery Step Detail: ............................................................................................................................ 8 Function 2: Remote Customer Service .............................................................................................. 24 Overview ................................................................................................................................................. 24 Scenario Covered .................................................................................................................................... 24 Prepared by ............................................................................................................................................. 24 Team ....................................................................................................................................................... 24 Remote Customer Service, Scenario 1: No internet connection from ISP.............................................. 26 Recovery Steps Summary.................................................................................................................... 26 Recovery Step Details ......................................................................................................................... 27 FUNCTION 3: Inventory Management System Overview.................................................................... 34 Scenarios covered ................................................................................................................................... 34 Prepared by ............................................................................................................................................. 34 Team ....................................................................................................................................................... 34 Inventory Management Service, Scenario1: Destruction of warehouse stock ...................................... 35 Recovery Steps Summary.................................................................................................................... 36 Recovery Step Detail: .......................................................................................................................... 37 Appendix A: Update Information ...................................................................................................... 46 Appendix B: Testing Information ...................................................................................................... 47 SRT610 Business Continuity Plan Liquid Networks Page 3 of 47 INTRODUCTION Copies of This Plan There are 4 copies of the emergency plan in three filing cabinets in the main office. The warehouse has 2 copies and each branch offices have one copy of the plan each. Office One copy is kept in the red Emergency Plan folder in the sales department manager’s filing cabinet in the lowest drawer (Floor 1 room 103). This cabinet is locket and the Sales manager has a key. The second plan is located in the HR filing cabinet in administrative room (Floor 2 room 231) in HR. This filing cabinet is protected by a key pad and the manager for the HR department has the key code. The third is located in the marketing departments’ filing cabinet in their storage room (Floor 3 room 331) under the red Emergency Plan folder. This cabinet is locked and the manager of the marketing department has a key. The fourth is kept in the IT department managers filing cabinet (Floor 4 room 410) filed in the lowest drawer in the red Emergency Plan folder the manager of the helpdesk has the key to this filing cabinet. Warehouse This plan is located in the supervisor’s office (Floor 1 room 30) in the records filing cabinet in the bottom drawer in the red emergency plan folder. The floor supervisor should have a key for this filing cabinet as well as the Warehouse manager. The other emergency plan folder is located in the Basement of the warehouse in the server room filing cabinet (Floor 0 room 3) in the bottom drawer in the red Emergency Plan folder. Branch Offices The branch offices have one copy of the Emergency plan in the manager’s office filing cabinet. The manager and supervisor on duty have keys to this cabinet. The emergency recovery plan is located in the red folder in the bottom drawer of that cabinet. Who knows about the locations of the Emergency Recovery Plans? Main Office Managers of the HR Department, All of the IT Department, Sales Department, and Marketing Department are aware of the locations of the plan. Warehouse Inventory mangers and Floor Supervisors are aware of the locations for the recovery plan. Branches Branch Managers and supervisors are aware of the location of the plan at the branches. Calling a Disaster Any employee that observes a potential disaster must alert their manager, any onsite manager or their managers supporting manager based on the contact list cell phone contact list via txt msg and phone call to all available numbers. Contact lists are deployed to every employee and should be in their desk at all times. The managers when alerted to an emergency situation must receive the approval of the time of a SRT610 Business Continuity Plan Liquid Networks Page 4 of 47 disaster must contact the CIO, CEO or Director of their department to get approval to begin the BCP for the function that is at risk. Once the available managers have declared a disaster one manager must retrieve the BCP and contact the responsible team and their backups to begin the BCP processes. Scope and Objectives What disaster scenarios and functions (function=service=system) does this plan cover? It might be useful to have this section in the following format or something similar. Note: the data in the table above is just a sample; you would use your own data. Disaster Ice Storm Scenario Power outage at head office due to downed power lines. Communications Failure at ISP Communications failure with main office ISP due to human error. Fire at warehouse destroys available inventory and warehouse physical structure. Fire Function Authentication Services are at risk and running on 6hr backup battery. SRT610 Business Continuity Plan Liquid Networks Page 5 of 47 FUNCTION 1: Authentication Services Overview This service is a domain authenticating service that allows users to remotely sign in to the domain and allows users in the office to use the company’s internal network to access domain services. All services for employees are accessible after authentication and authorization by the domain authentication service. Scenarios covered Ice Storm knocks down power lines to the main office causing power outage. Prepared by This scenario was prepared by Jamaal Joseph Team Team Leader Sr. Network Administrator, Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com First Alternate Helpdesk Manager, Duncan Terrell Work Number: Ext 10 Cell Number: 416.789.2345 Home Number: 416.628.1567 Email: d.terrell@liquid.com Second Alternate Network Administrator Supervisor, William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Team Member Helpdesk Supervisor, Kanye Fazooli Address: 312 Thatch Blvd., Thornhill, ON Contacts: Work Number: Ext 15 Cell Number: 416.223.4521 Home Number: 905.692.3949 Email: k.fazooli@liquid.com First Alternate Sr. Helpdesk Technician, Claire Silver Work Number: Ext 14 Cell Number: 416.678.9876 Home Number: 905.435.4523 Email: c.silver@liquid.com Second Alternate SRT610 Business Continuity Plan Liquid Networks Page 6 of 47 Helpdesk Supervisor, Kanye Fazooli Address: 312 Thatch Blvd., Thornhill, ON Contacts: Work Number: Ext 15 Cell Number: 416.223.4521 Home Number: 905.692.3949 Email: k.fazooli@liquid.com Team Member First Alternate Helpdek Technician, Barney Baggins Address: 9202 Shire Rd., Barrie, ON Contacts: Work Number: Ext 39 Cell Number: 416.734.1461 Home Number: 293.234.6578 Email: b.baggins@liquid.com Kanye Fazooli Second Alternate Claire Silver BCP For Authentication Services Function Authentication Service, Scenario1: Power outage at the office Overview of Scenario Overview of Recovery Procedure Recovery Time Objective (RTO) Recovery Location and directions After a significant amount of ice rain in an ice storm the power lines to the office snap and power is cut off to the building. Backup generators are activated immediately and they last 6hrs. We have 6hrs to restore Authentication services and the domain services that depend on authentication services at a new site. Data will be migrated and restored to the backup site. The estimated recovery time to for these services are 6 hrs Cold Site 1203 Wellington Road South, London, ON From main office in Waterloo (1282 Ottawa St N). Go East on Ottawa St N from the South Facing Parking lot of the main office Take hwy 7 South for to Hwy 8 South Hwy 8 South to 401 West 401 West to Wellington St. N in London, 1203 Wellington St. North, London, ON SRT610 Business Continuity Plan Liquid Networks Dependencies Other Considerations Page 7 of 47 This is dependent on ISP communications being functional at the office and the physical computers and devices at the office being functional and accessible NA Recovery Steps Summary Step 1 – Initiate the BCP Step 2 – Lockout users so that technicians can before data backups Step 3 – Obtain keys for the cold site and have a technician go to the cold site Step 4 – Perform a local backup of the services at the office Step 5 – Enter the backup site and begin the remote synchronization process. Step 6 – Notify all employees when services are restored Step 7 – Remove backup of local systems and lock the server room at the head office. Take the backup to the third party backup site for storage. Step 8 – Contact the network administrators and have them contact the hydro helpdesk to restore the services. Step 9 – When hydro utilities are restored at the head office sync the head office servers with the cold site servers. Step 10 - Notify all employees that services are fully recovered at head office SRT610 Business Continuity Plan Liquid Networks Page 8 of 47 Recovery Step Detail: Step 1. Once power outage has been declared the Network Administrator or their backup should appoint a technician to initiate the BCP run the script located as listed below \\server1\911scripts\poweroutage\notifactionprocess.bat. Log in to a computer on the network with a user account that has liquid network administrative privilege. This can be accomplished by logging in to a computer on the network in the office or by logging into the computer with admin privilege and then logging into the VPN service. Click on the start menu and then click on the run option in the menu. In that window type the absolute filename as seen below and press the enter key. o This script will prompt the technician for their contact number and name and then will begin. o It will then send out an email, voice mail, and text message to all network administrators using their contacts located in the HR database advising the technicians of two things. The time the power outage began Asking them to contact the team member whose details were inputted into the program. o It will then send an email, text message and voice mail to all of the liquid network employees using the HR records advising them that the systems are running on backup power and that the network administrators are doing unplanned maintenance. Lastly, that they will be notified when the power is restored. Responsibility: Paul Smith Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Workstation Where Any workstation located in the office on the company network Script Server 1 available through remote desktop on any workstation Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com Any alternate for Paul Smith \\server1\911scripts\poweroutage\notifactionprocess.bat SRT610 Business Continuity Plan Liquid Networks Page 9 of 47 Step 2. Paul Smith or his alternate will activate the following script located at: \\server1\911scripts\poweroutage\db_lockdown.bat. To do this log in to a computer on the network with a user account that has liquid network administrative privilege. This can be accomplished by logging in to any computer on the network with administrator privileges locally or via remote desktop. Click on the start menu and then click on the run option in the menu. Type cmd at the run prompt and press ok. In that window type the absolute filename as seen here in the document and press the enter key. This script will text message, email and leave a voice mail message for all Liquid Network Employees. The statement will say for all employees that… o The authentication services will be off line while network administrators investigate the issue so databases will all be inaccessible. o Users are to log out of the databases as soon as they can conclude their current data inputs. They will receive a warning to log out again at 10minutes. At 15 minutes they will be logged out. Responsibility: Paul Smith or Alternate: _________________ Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Contact Info Jr Network Administrator Contact Info Where Contact Contact List located Paul Smith in bottom drawer Address: 4030 Waybill Rd., Oakville, ON of all employee Work Number: Ext 12 desks. Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com Contact List located William Marx in bottom drawer Address: 506 Isling Ave., Toronto, ON of all employee Work Number: Ext 23 desks. Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Helpdesk Manager Contact Info Contact List located Duncan Terrell in bottom drawer Work Number: Ext 10 of all employee Cell Number: 416.789.2345 desks. Home Number: 416.628.1567 Email: d.terrell@liquid.com Helpdesk Supervisor Contact Info Contact List located Helpdesk Supervisor, Kanye Fazooli in bottom drawer of all employee Work Number: Ext 15 desks. Cell Number: 416.223.4521 Home Number: 905.692.3949 SRT610 Business Continuity Plan Liquid Networks Page 10 of 47 Email: k.fazooli@liquid.com Workstation Any workstation located in the office on the company network Script Server 1 available through remote desktop on any workstation Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com Any alternate for Paul Smith \\server1\911scripts\poweroutage\db_lockdown.bat SRT610 Business Continuity Plan Liquid Networks Page 11 of 47 Step 3. Step 3 – The response team must obtain the cold site keys and ID card. All managers must have cold site building keys and a swipe card on them at all times. If needed the response team may contact another member of the response team to obtain the keys and swipe card. The senior more member of the response team must go to the cold site using directions provided in the BCP. Responsibility: Paul Smith or Alternate: _________________ Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Contact Info Where Contact List located in bottom drawer of all employee desks. Jr Network Administrator Contact Info Contact List located in bottom drawer of all employee desks. Helpdesk Manager Contact Info Contact List located in bottom drawer of all employee desks. Helpdesk Supervisor Contact Info Contact List located in bottom drawer of all employee desks. Cold Site Keys In the possession of the network administrators, helpdesk manager and helpdesk supervisor Col d Site Location of Cold Site Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Duncan Terrell Work Number: Ext 10 Cell Number: 416.789.2345 Home Number: 416.628.1567 Email: d.terrell@liquid.com Helpdesk Supervisor, Kanye Fazooli Work Number: Ext 15 Cell Number: 416.223.4521 Home Number: 905.692.3949 Email: k.fazooli@liquid.com Paul Smith William Marx Duncan Terrell Kanye Fazooli Col d Site SRT610 Business Continuity Plan Liquid Networks 1203 Wellington Road South, London, ON Page 12 of 47 1203 Wellington Road South, London, ON From main office in Waterloo (1282 Ottawa St N). Go East on Ottawa St N from the South Facing Parking lot of the main office Take hwy 7 South for to Hwy 8 South Hwy 8 South to 401 West 401 West to Wellington St. N in London, 1203 Wellington St. North, London, ON SRT610 Business Continuity Plan Liquid Networks Page 13 of 47 Step 4. Second most senior technician in response team should be appointed to as the onsite recovery leader. They should begin the physical backup of the head office servers. They must log in to a computer at the office with administrative privileges and run the automated script the script located at \\server1\911scripts\poweroutage\TOTAL_local_backup.bat. Responsibility: William Marx or Alternate: _________________ Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Where Contact List located in bottom drawer of all employee desks. Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Jr. Network Administrator Contact List located in bottom drawer of all employee desks. Helpdesk Manager Contact List located in bottom drawer of all employee desks. Duncan Terrell Work Number: Ext 10 Cell Number: 416.789.2345 Home Number: 416.628.1567 Email: d.terrell@liquid.com Helpdesk Supervisor Contact List located in bottom drawer of all employee desks. William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Servers Room 304 at Main Office “Emergency Backup Media” Server Room 10Tb drive located in server room in Paul Smith William Marx Duncan Terrell Kanye Fazooli Power on device and click on the sync button located on the top left of drive. When sync is complete drive will flash complete on LCD on front panel. SRT610 Business Continuity Plan Liquid Networks Script back left shelf. Device is labeled “Emergency Backup Media”. Server 1 available through remote desktop on any workstation Page 14 of 47 \\server1\911scripts\poweroutage\TOTAL_local_backup.bat SRT610 Business Continuity Plan Liquid Networks Page 15 of 47 Step 5. Step 5. When the technician appointed to the cold site recovery arrives at the backup site they should enter the building via the main entrance and travel to the 5th floor room 503. Once at the cold site use the swipe card to enter room, turn on servers labeled 1 through 6 using the power button, and run the following commands on each server from the command line. When prompted the user must enter their name, and phone number. The scripts are as listed below where X is the server number on the front of the server. Username for each server is ADMIN password is ADM1Nr3c0veR%. All servers are labeled with their names on the front panel and are accessible through ssh on the console in the server room. o \\coldserverX\databasessync\remotesync.bat These scripts will sync the existing files on server 1-3 with servers 4-6 and then begin a remote synchronization of the data available on the Authentication Server as well as the other servers at the office. The authentication servers new address will be tested and then made available to network users. Once script is complete an email and voice mail will be sent to the technician that started the scripts to login that this step is complete. Responsibility: Paul Smith or Alternate: _________________ Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Where Contact List located in bottom drawer of all employee desks. Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Jr Network Administrator Contact List located in bottom drawer of all employee desks. Helpdesk Manager Contact List located in bottom drawer of all employee desks. Duncan Terrell Work Number: Ext 10 Cell Number: 416.789.2345 Home Number: 416.628.1567 Email: d.terrell@liquid.com Helpdesk Supervisor Contact List located in bottom drawer of all William Marx Address: 506 Isling Ave., Toronto, ON SRT610 Business Continuity Plan Liquid Networks employee desks. Cold Site Keys Location of Cold Site Servers Script Page 16 of 47 Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com In the possession of the network administrators, helpdesk manager and helpdesk supervisor Located and direction in BCP Paul Smith William Marx Duncan Terrell Kanye Fazooli Room 203. Located at Backup Site in racks on the backmost wall. Server 1 -6 Username: ADMIN Password: ADM1Nr3c0veR% Col d Site 1203 Wellington Road South, London, ON From main office in Waterloo (1282 Ottawa St N). Go East on Ottawa St N from the South Facing Parking lot of the main office Take hwy 7 South for to Hwy 8 South Hwy 8 South to 401 West 401 West to Wellington St. N in London, 1203 Wellington St. North, London, ON \\coldserverX\databasessync\remotesync.bat SRT610 Business Continuity Plan Liquid Networks Page 17 of 47 Step 6. The technician appointed to the cold site recovery should then run the following script located on cold server 1. o \\coldserver1\databasessync\srvsrestored_notifications.bat This script will notify all employees of liquid networks that the services are restored and that they should resume their tasks from home until further notice. Responsibility: Paul Smith or Alternate: _________________ Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Where Contact List located in bottom drawer of all employee desks. Jr Network Administrator Contact List located in bottom drawer of all employee desks. Helpdesk Manager Contact List located in bottom drawer of all employee desks. Duncan Terrell Work Number: Ext 10 Cell Number: 416.789.2345 Home Number: 416.628.1567 Email: d.terrell@liquid.com Helpdesk Supervisor Contact List located in bottom drawer of all employee desks. William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Server1 Located at Backup Site in racks mounted on the backmost wall. Coldserver 1 Username: ADMIN Password: ADM1Nr3c0veR% Script Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com \\coldserver1\databasessync\srvsrestored_notifications.bat SRT610 Business Continuity Plan Liquid Networks Page 18 of 47 Step 7. Once new authentication server is up and running and the physical backup of authentication server and all other servers at the office is complete technicians at the office must remove the physical backup device ‘Emergency Backup Media’. The server room must be locked using the keys to the server room from Paul Smith or his alternate. That technician must take the backup to the third party data backup service site when it is safe to do so and then go to the cold site to work from that office. Responsibility: William Marx or Alternate: _________________ Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Where Contact List located in bottom drawer of all employee desks. Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Jr Network Administrator Contact List located in bottom drawer of all employee desks. Helpdesk Manager Contact List located in bottom drawer of all employee desks. Duncan Terrell Work Number: Ext 10 Cell Number: 416.789.2345 Home Number: 416.628.1567 Email: d.terrell@liquid.com Helpdesk Supervisor Contact List located in bottom drawer of all employee desks. William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Servers Room 304 at Main Office Paul Smith William Marx Duncan Terrell Kanye Fazooli SRT610 Business Continuity Plan Liquid Networks “Emergency Backup Media” Third Part Site Server Room 10Tb drive located in server room in back left shelf. Device is labeled “Emergency Backup Media”. 3Party Backup Services Page 19 of 47 Power on device and click on the sync button located on the top left of drive. When sync is complete drive will flash complete on LCD on front panel. Business Services Manager,Ken Finler Cell number: 415.627.2341 Helpdesk Number – 1800.536.7383 Account - #4524525325230 Password – L1Qn3t 122 Yonge Street Ave, Barrie, ON, L6L7L4 SRT610 Business Continuity Plan Liquid Networks Page 20 of 47 Step 8. Contact network administrator team lead and have them contact the utilities company to coordinate service restoration of hydro to the office. Responsibility: Paul Smith Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Where Contact List located in bottom drawer of all employee desks. Jr Network Administrator Contact List located in bottom drawer of all employee desks. City Hydro Services Property Management Services Hydro One Summit Property Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Business Services - 1800.345.6789 Business Services - 1800.304.6975 32 Lopter Ave, Barrie, ON, M3N5L6 Business Services Manager -Sam Tekger Office Number: 416.423.6574 Cell Number: 796.234.3456 SRT610 Business Continuity Plan Liquid Networks Page 21 of 47 Step 9. When hydro has been restored to the office, the servers at the office should be turned on and synced using script listed below on Server1 o \\server1\databasessync\complete_remoterecovery.bat This script will do the following. Once the script is begun the technician will enter their name and phone number and press ok. Advise all users via email, voice mail, and text message that in 1 hour access to all network resources will be locked out for network maintenance. All network data from the cold site will be synced with the head office servers and tested. The network administrator that began the script will be notified via voice mail, email and text message that the synchronization is completed. Responsibility: Paul Smith Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Where Contact List located in bottom drawer of all employee desks. Jr Network Administrator Contact List located in bottom drawer of all employee desks. Server Room Keys In the possession of the network administrators, helpdesk manager and helpdesk supervisor Room 304 at Main Office Any workstation located in the office Servers Workstation Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Paul Smith William Marx Duncan Terrell Kanye Fazooli Paul Smith William Marx Paul Smith William Marx SRT610 Business Continuity Plan Liquid Networks Script on the company network Server 1 Page 22 of 47 \\server1\databasessync\complete_remoterecovery.bat SRT610 Business Continuity Plan Liquid Networks Page 23 of 47 Step 10. Step 10. Once the services are restored the technician should run the script listed below to notify employees that they may return to the working out of the office on the following business day. o \\server1\databasessync\srvsrestored_notifications.bat Responsibility: Paul Smith Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Where Contact List located in bottom drawer of all employee desks. Jr Network Administrator Contact List located in bottom drawer of all employee desks. Server Room Keys In the possession of the network administrators, helpdesk manager and helpdesk supervisor Room 304 at Main Office Any workstation located in the office on the company network Server 1 Servers Workstation Script Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Paul Smith William Marx Duncan Terrell Kanye Fazooli Paul Smith William Marx Paul Smith William Marx \\server1\databasessync\srvsrestored_notifications.bat SRT610 Business Continuity Plan Liquid Networks Page 24 of 47 Function 2: Remote Customer Service Overview Remote Customer Service (RCS) a form of customer support remotely over the internet. When Liquid Networks receives a call requesting service, The RCS individual would remotely access the customer’s computer and diagnose and fix the problem. In the event of hardware failure the request would be forwarded to onsite customer support. Scenario Covered - There are problems at the ISP causing the main office to lose all forms of internet access. Requests from customers who require support are piling up but without internet we cannot provide any support. Prepared by Chris Lai Team Team Leader Mark Twain Address: 125 Oldtown St., Markham, ON Contacts: Office Tel: Ext: 13 Home Tel: 728-2162 Cell Phone: (416) 555-4399 Personal E-Mail: mtwain@liquid.com First Alternate David Copperfield Second Alternate Jack Bauer Team Member First Alternate David Copperfield Address: 1255 Jeorge St., Markham, ON Contacts: Office Tel: Ext: 14 Home Tel: 364-2562 Cell Phone: (416) 346-2639 Harry Potter Address: 55 David St., Mississauga, ON Contacts: Office Tel: Ext: 15 Home Tel: 364-6263 Cell Phone: (416) 324-3446 Personal E-Mail: hpotter@liquid.com Personal E-Mail: dcopperfield@liquid.com Second Alternate Adam Hotchner Address: 1526 Macrill St., Missauga, ON Contacts: Office Tel: Ext: 16 Home Tel: 364-2353 SRT610 Business Continuity Plan Liquid Networks Page 25 of 47 Cell Phone: (416) 354-3147 Personal E-Mail: ahotchner@liquid.com Team Member First Alternate Jack Bauer Address: 1672 Rose St., Markham, ON Contacts: Office Tel: Ext: 17 Home Tel: 163-1622 Cell Phone: (416) 316-6126 Dan Brown Address: 62 Jane St., Vaughan, ON Contacts: Office Tel: Ext: 18 Home Tel: 319-6623 Cell Phone: (416) 384-3666 Personal E-Mail: dbrown@liquid.com Personal E-Mail: jbauer@liquid.com Second Alternate Barney Stinson Address: 6126 Mclarens St., Markham, ON Contacts: Office Tel: Ext: 19 Home Tel: 867-3474 Cell Phone: (416) 256-2846 Personal E-Mail: bstinson@liquid.com SRT610 Business Continuity Plan Liquid Networks Page 26 of 47 Remote Customer Service, Scenario 1: No internet connection from ISP Overview of Scenario There is a problem with the ISP causing our office to not have internet connection. We are unable to respond to customer support requests remotely without internet. Overview of Recovery The recovery procedure consists of informing managers and customer Procedure service representatives of the situation. Must contact the RCS team so that they can be prepared and move to the cold site office which has a different ISP. Recovery Time Objective For this scenario the RTO is within 24 hours when the ISP is confirmed that (RTO) there will be a long period of downtime. Within this time, team and staff will have to move to the cold site and continue operations. Because the cold site would have enough workstations for all employees, availability of this service will not be interrupted Recovery Location and Liquid Networks Cold Site directions 8257 Nestea Street Aurora, Ontario L8C 5U1 Phone: (416)275-1729 From Highway 407 ETR: 407 East to 404 North and get off at Burlington east. Turn left at the second traffic light and on the right hand side. Dependencies This requires that the employees have a means of travel in order to get to the cold site. Other Considerations N/A Recovery Steps Summary Step 1: Contact ISP requesting information on the cause of connection issue and the length of downtime. Step 2: RCS staff and customer service representatives are informed that the internet connection is down. Customers are informed through representatives that remote connection support will be delayed. Step 3: Alert RCS team members and representatives of disaster and prepare for transport Step 4: RCS representatives move to cold site and settle in as well as perform checks on software Step 5: Inform customer service that RCS can perform customer queries again Step 6: Depending on estimated downtime by ISP, perform routine checks on ISP progress Step 7: When ISP is up and running again, informs RCS representatives and have them move back to head office in groups of 4. SRT610 Business Continuity Plan Liquid Networks Page 27 of 47 Recovery Step Details RCS Scenario 1- Internet connection is down Step 1: Contact ISP requesting information on the cause of connection issue and the length of downtime. Manager contacts the ISP (Rogers) and requests information regarding the downtime they are facing. The cause of the connection issues and the length of downtime must be made clear. In event of long downtime (greater than 2 hours) proceed to step 2. Responsibility: Billy Walsh Date Completed: _______________ Initials: ________________ People/Services Name Rogers Representative Rogers Customer Support Description Person in charge of Markham ISP office Contact Andrew Davis Office Number: (416)572-5183 ext 173 E-Mail: adavis@rogers.ca Markham ISP office (if Andrew is unreachable) Office Number: (416)823-1782 SRT610 Business Continuity Plan Liquid Networks Page 28 of 47 Recovery Step Details RCS Scenario 1- Internet connection is down Step 2: RCS staff and customer service representatives are informed that the internet connection is down. Customers are informed through representatives that remote connection support will be delayed. Manager using his/her mobile phone to contact RCS manager and customer service manager and informs them of the ISP downtime information. RCS and CS manager would forward information and instructions to proceed. Customer who request remote support would be informed of technical difficulties and assistance would be delayed. Responsibility: Billy Walsh Date Completed: _______________ Initials: ________________ People/Services Name Customer Service Manager Remote Customer Service Manager Where - Manager’s mobile phone address book - Contact information document that all employees have a copy of - Manager’s mobile phone address book - Contact information document that all employees have a copy of Contact Janice Splosh Office Number: (416)555-5555 ext 25 E-Mail: jsplosh@liquid.com Mark Twain Office Number: (416)555-5555 ext 13 E-Mail: mtwain@liquid.com SRT610 Business Continuity Plan Liquid Networks Page 29 of 47 Recovery Step Details RCS Scenario 1- Internet connection is down Step 3: Alert RCS team members and representatives of disaster and prepare for transport E-mail notifications sent are sent out to all RCS staff members by the RCS manager. The e-mail will inform them of the current situation and to prepare for transport to cold site. Responsibility: Mark Twain Date Completed: _______________ Initials: ________________ Location / Directions Liquid Networks Cold Site 8257 Nestea Street Aurora, Ontario L8C 5U1 Phone: (416)275-1729 From Highway 407 ETR: 407 East to 404 North and get off at Burlington east. Turn left at the second traffic light and on the right hand side. SRT610 Business Continuity Plan Liquid Networks Page 30 of 47 Recovery Step Details RCS Scenario 1- Internet connection is down Step 4: RCS representatives move to cold site and settle in as well as perform checks on software Staff members are transported to cold site through most efficient means of transport. Once there they report in and proceed to their assigned terminals. They ensure all required software is functioning properly for them to perform their normal tasks. Responsibility: Mark Twain/staff members Date Completed: _______________ Initials: ________________ Service Checklist: Internet SSH Telephone communication Telnet VPN SRT610 Business Continuity Plan Liquid Networks Page 31 of 47 Recovery Step Details RCS Scenario 1- Internet connection is down Step 5: Inform customer service that RCS can perform customer queries again RCS manager contacts and informs CS manager that remote customer service is operational again and ready to process customer queries. Responsibility: Mark Twain Date Completed: _______________ Initials: ________________ People/Service Name Customer Service Manager Where - Manager’s mobile phone address book - Contact information document that all employees have a copy of Contact Janice Splosh Office Number: (416)555-5555 ext 25 E-Mail: jsplosh@liquid.com SRT610 Business Continuity Plan Liquid Networks Page 32 of 47 Recovery Step Details RCS Scenario 1- Internet connection is down Step 6: Depending on estimated downtime by ISP, perform routine checks on ISP progress Office manager proceeds to contact and check ISP status on their downtime every hour. After each check, Manager would contact CS and RCS manager on the ISP’s progress Responsibility: Mark Twain Date Completed: _______________ Initials: ________________ People/Services Name Rogers Representative Description Person in charge of Markham ISP office Contact Andrew Davis Office Number: (416)572-5183 ext 173 E-Mail: adavis@rogers.ca Rogers Customer Support Markham ISP office (if Andrew is unreachable) Office Number: (416)823-1782 Name Customer Service Manager Where - Manager’s mobile phone address book - Contact information document that all employees have a copy of - Manager’s mobile phone address book - Contact information document that all employees have a copy of Contact Janice Splosh Office Number: (416)555-5555 ext 25 E-Mail: jsplosh@liquid.com Remote Customer Service Manager Mark Twain Office Number: (416)555-5555 ext 13 E-Mail: mtwain@liquid.com SRT610 Business Continuity Plan Liquid Networks Page 33 of 47 Recovery Step Details RCS Scenario 1- Internet connection is down Step 7: When ISP is up and running again, informs RCS representatives and have them move back to head office in groups of 4. When manager confirms that ISP is operational again at main office, he/she will contact RCS and CS manager of this information. RCS staff will proceed to slowly move back to main office in groups of 4 to prevent the service going down again during transport. When the each group will move only after the group ahead of them has confirmed their arrival. Responsibility: Mark Twain Date Completed: _______________ Initials: ________________ People/Services Name Description Contact Rogers Representative Person in charge of Andrew Davis Markham ISP office Office Number: (416)572-5183 ext 173 E-Mail: adavis@rogers.ca Rogers Customer Support Markham ISP office (if Andrew is unreachable) Office Number: (416)823-1782 Name Customer Service Manager Where - Manager’s mobile phone address book - Contact information document that all employees have a copy of - Manager’s mobile phone address book - Contact information document that all employees have a copy of Contact Janice Splosh Office Number: (416)555-5555 ext 25 E-Mail: jsplosh@liquid.com Remote Customer Service Manager Mark Twain Office Number: (416)555-5555 ext 13 E-Mail: mtwain@liquid.com SRT610 Business Continuity Plan Liquid Networks Page 34 of 47 FUNCTION 3: Inventory Management System Overview The inventory management system is responsible for maintaining accurate inventory counts of the various products we have available within all stores and the warehouses. This database is mandatory in the sense that it is frequently used by our staff to pull up what items are in stock upon customer request. It is also directly linked to wholesale purchasing system. Scenarios covered Fire in the warehouse destroy mostly all inventory, customers as a result are calling in inquiring about why orders are late. Prepared by This scenario was prepared by Shawn Williams Team Team Leader Sr. Network Administrator, Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com First Alternate Helpdesk Manager, Duncan Terrell Work Number: Ext 10 Cell Number: 416.789.2345 Home Number: 416.628.1567 Email: d.terrell@liquid.com Second Alternate Network Administrator Supervisor, William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Third Alternate Lead Database Designer, Billy Bob Address: 555 Rathburn Rd, Etobicoke, ON Work Number: Ext 2 Cell Number: 416.234.2347 Home Number: 905.234.2347 Email: Bbob@liquid.com Team Member Helpdesk Supervisor, Kanye Fazooli First Alternate Sr. Helpdesk Technician, Claire Silver Work Number: Ext 14 SRT610 Business Continuity Plan Liquid Networks Page 35 of 47 Address: 312 Thatch Blvd., Thornhill, ON Contacts: Work Number: Ext 15 Cell Number: 416.223.4521 Home Number: 905.692.3949 Email: k.fazooli@liquid.com Cell Number: 416.678.9876 Home Number: 905.435.4523 Email: c.silver@liquid.com Second Alternate Helpdesk Supervisor, Kanye Fazooli Address: 312 Thatch Blvd., Thornhill, ON Contacts: Work Number: Ext 15 Cell Number: 416.223.4521 Home Number: 905.692.3949 Email: k.fazooli@liquid.com Team Member First Alternate Helpdek Technician, Barney Baggins Address: 9202 Shire Rd., Barrie, ON Contacts: Work Number: Ext 39 Cell Number: 416.734.1461 Home Number: 293.234.6578 Email: b.baggins@liquid.com Kanye Fazooli Second Alternate Claire Silver BCP for Inventory Management Service Function Inventory Management Service, Scenario1: Destruction of warehouse stock Overview of Scenario Overview of Recovery Procedure Due to one of our generators exploding, a fire began in the west side of the warehouse. Before it was contained it destroyed an unknown amount of inventory in the ‘to be shipped’ area of the warehouse as well as some surrounding inventory. The resulting incident has severely disrupted our ability to determine what product we have available. In addition, several customer products that were supposed to be shipped today were destroyed. In order to restore service back to normal, the following steps have been devised; We need to; 1. 2. 3. 4. Evaluated the impact of the damage. Notify insurance Run automated scripts Clean up. SRT610 Business Continuity Plan Liquid Networks Recovery Time Objective (RTO) Recovery Location and directions Page 36 of 47 The estimated recovery time to for these services are 12 hrs Drop off point Until, the mess in the west end of the warehouse has been cleared up, all arriving shipments are to be stored in the east end of the building. Dependencies Other Considerations This is dependent on ISP communications being functional at the office and the physical computers and devices at the office being functional and accessible Service is linked to the wholesale purchasing system Recovery Steps Summary 1. 2. 3. 4. 5. 6. 7. 8. Get an estimate on what and how much stock was lost. Contact insurance company. Find out how many customers orders were lost. Contact all customers who lost stock, explain that their order will be late and offer a 5% discount on their next purchase. Re-synchronize what is left over with the inventory management system. Update the webs status of all products. Products that we don’t have, change status to “not in stock” Re-order any products in high demand. Clean up and salvage what can be salvaged. SRT610 Business Continuity Plan Liquid Networks Page 37 of 47 Recovery Step Detail: Step 1. Fill out form B-57. This form can be retrieved found at \\server1\damages\B-57.doc. These forms are also readily available in hardcopy within all offices. Just ask for the B-57 incident report. This file can also be retrieved from a remote system via ftp. If you are accessing the form from a PC not a part of our network or from our wireless network, use your regular network login when credentials are needed. Responsibility: Paul Smith Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Workstation Where Any workstation located in the office on the company network or outside network via ftp. Damages form Server 1 available through remote desktop on any workstation or ftp. A hardcopy is also available in all offices, in the front desk. Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com Any alternate for Paul Smith \\server1\damages\B-57.doc remote desktop (LAN) http://www.liquidnetworks/file10/911/damages/B57.doc SRT610 Business Continuity Plan Liquid Networks Page 38 of 47 Step 2. After form B-57 (contains information on what was lost, and value) is fill out either by Paul Smith, Duncan Terrell or Williams Marx, contact insurance. They can be reached at; Phone: 1-800-HELPME Tell them about the fire and give them and estimate on what was lost along with picture of damages. Also, fax the a copy of form B-57 @ 1-888-HELP-ME Responsibility: Paul Smith or Alternate: _________________ Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Contact Info Jr Network Administrator Contact Info Where Contact Contact List located Paul Smith in bottom drawer Address: 4030 Waybill Rd., Oakville, ON of all employee Work Number: Ext 12 desks. Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com Contact List located William Marx in bottom drawer Address: 506 Isling Ave., Toronto, ON of all employee Work Number: Ext 23 desks. Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Helpdesk Manager Contact Info Contact List located Duncan Terrell in bottom drawer Work Number: Ext 10 of all employee Cell Number: 416.789.2345 desks. Home Number: 416.628.1567 Email: d.terrell@liquid.com Helpdesk Supervisor Contact Info Contact List located Helpdesk Supervisor, Kanye Fazooli in bottom drawer of all employee Work Number: Ext 15 desks. Cell Number: 416.223.4521 Home Number: 905.692.3949 Email: k.fazooli@liquid.com Workstation Any workstation located in the office on the company network Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 SRT610 Business Continuity Plan Liquid Networks Insurance Phone or fax Page 39 of 47 Email: p.smith@liquid.com Any alternate for Paul Smith Phone number: 1-800-HELPME Fax: 1-888-HELP-ME SRT610 Business Continuity Plan Liquid Networks Page 40 of 47 Step 3. Step 3 – Find out how many customers orders were lost. o This can be found out by accessing the orders database and querying ordered products with product reported damaged or lost. Or by executing script EM-1.bat which will not only do a product query, but also notify customers. o If we have a few products left over, but a much higher demand, ship only the left over products to the customers who ordered first. o Script can be located here via remote desktop \\server1\911scripts\damages\EM-1.bat. Responsibility: Paul Smith or Alternate: _________________ Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Contact Info Where Contact List located in bottom drawer of all employee desks. Jr Network Administrator Contact Info Contact List located in bottom drawer of all employee desks. Helpdesk Manager Contact Info Contact List located in bottom drawer of all employee desks. Helpdesk Supervisor Contact Info Contact List located in bottom drawer of all employee desks. Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Duncan Terrell Work Number: Ext 10 Cell Number: 416.789.2345 Home Number: 416.628.1567 Email: d.terrell@liquid.com Helpdesk Supervisor, Kanye Fazooli Work Number: Ext 15 Cell Number: 416.223.4521 Home Number: 905.692.3949 Email: k.fazooli@liquid.com SRT610 Business Continuity Plan Liquid Networks Page 41 of 47 Step 4. Contact all customers who lost stock, explain that their order will be late and offer a 5% discount on their next purchase. The automated system should handle this once EM-1.bat is executed. Responsibility: William Marx or Duncan or Automated message: _________________ Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Helpdesk Manager Where Contact List located in bottom drawer of all employee desks. Contact Duncan Terrell Work Number: Ext 10 Cell Number: 416.789.2345 Home Number: 416.628.1567 Email: d.terrell@liquid.com Helpdesk Supervisor Contact List located in bottom drawer of all employee desks. William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Script Server 1 available through remote desktop on any workstation \\server1\911scripts\damages\EM-1.bat Step 5. Re-synchronize what is left over with the inventory management system. After manually finding out how many products that were un-effected by the fire, update the inventory management system with the correct figures. Billy Bob and his team of database programmers are in charge of the updating of the database tables. Script EM2.bat is not to be executed until approved by Billy Bob. Execute script EM-2.bat after all entries are registered into the system. This script can be executed by anyone once Billy Bob has fixed the database entries. Script EM-2.bat can be found here; \\server1\911scripts\damages\EM-2.bat This script should also automate steps 6 and 7 of the recovery process. Responsibility: Billy Bob or Alternative: _________________ SRT610 Business Continuity Plan Liquid Networks Page 42 of 47 Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Where Contact List located in bottom drawer of all employee desks. Jr Network Administrator Contact List located in bottom drawer of all employee desks. Helpdesk Manager Contact List located in bottom drawer of all employee desks. Duncan Terrell Work Number: Ext 10 Cell Number: 416.789.2345 Home Number: 416.628.1567 Email: d.terrell@liquid.com Database Designer Contact List located in bottom drawer of all employee desks. Billy Bob Server 1 available through remote desktop on any workstation \\server1\911scripts\damages\EM-2.bat Script Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com William Marx Address: 506 Isling Ave., Toronto, ON Work Number: Ext 23 Cell Number: 416.629.1231 Home Number: 905.234.2345 Email: w.marx@liquid.com Work Number: Ext 2 Cell Number: 555.555.555 Home Number: 905.234.2347 Email: Bbob@liquid.com SRT610 Business Continuity Plan Liquid Networks Page 43 of 47 Step 6. Update the webs status of all products. Products that we don’t have, change status to “not in stock” Automatic process, EM-2.bat handles this. Responsibility: Automated: System Date Completed: ______________ SRT610 Business Continuity Plan Liquid Networks Step 7. Re-order any products in high demand. Automatic process, EM-2.bat handles this. Responsibility Automated: System Date Completed: ______________ Page 44 of 47 SRT610 Business Continuity Plan Liquid Networks Page 45 of 47 Step 8. Clean up and salvage what can be salvaged. This job is to be done by operators of the warehouse, janitors and should be overseen by Paul Smith. Also, all products that were not damaged but were in the disaster zone, need to be retested before shipping them. Responsibility: Paul Smith Date Completed: ______________ Initials: ____ Equipment/Office Supplies Description Network Administrator Where Contact List located in bottom drawer of all employee desks. Contact Paul Smith Address: 4030 Waybill Rd., Oakville, ON Work Number: Ext 12 Cell Number: 4161231234 Home Number: 905.123.1234 Email: p.smith@liquid.com SRT610 Business Continuity Plan Liquid Networks Page 46 of 47 Appendix A: Update Information This plan will be updated when it is last used or new changes have been made into effect (IE: employee changes). Any changes will be discussed by BCP Team Leader, Manager and is documented below. Date April 21st 2009 Updated By Collie O Brian Cause New contact information for office manager SRT610 Business Continuity Plan Liquid Networks Page 47 of 47 Appendix B: Testing Information Testing of this plan is done annually. BCP team leader is responsible for this and ensure tests are done on time. IT team would have to perform tests and updates on terminals located at the cold site. All tests are recorded below Date Done By Comments January 15th 2007 Chris Lai Testing and updates were done th January 18 2008 Jamaal Joseph Testing and updates were done January 14th 2009 Shawn Williams Testing and updates were done