Business Continuity & Technical Risk For Auditors A Going Concern, 2015 Agenda BCP / DR Overview Auditors Perspective Current Trends in BCP / DR Practical Considerations BCP / DR Demo in RPX A Going Concern, 2015 About Us: A Going Concern Our company works with an association of highly skilled independent consultants that are brought together to provide our clients the specialized skill sets needed. This enables us to control costs and ensure our clients the best value for their consulting dollar. A Going Concern, 2015 Why do we care about BCP? Depending on where you work – it may be required Changes in organizational make-up demand it, consolidation, globalization You will need to recover your programs following a disaster (really??????) Technology advances may drive it People, Process, Technology, and Third parties all matter to us! A Going Concern, 2015 Continuity Planning Business Continuity ERP: Emergency Response Plan Event Driven Response (Site Impact) Contamination, Bomb-threat, Fire, Earthquake, Wind, Etc. ERP IT-DRP Integrations Integrations Depending on Event, The integration of all Plans is Possible. Integrations CMP: Crisis Management Plan Event Escalation Response Non-physical or physical impacts, Examples: Toyota, Recall A Going Concern, 2015 IT-DRP: IT Disaster Recovery Plan (Technology - Voice & Data Impact) Network Failure, Sabotage, Virus, Physical Loss of Systems Etc. BCP Integrations CMP BCP: Business Continuity Time Driven Response (Site and Business and Image Impact) Infrastructure Disruptions, Healthcare Unit Disruptions, Department Disruptions (Failure to deliver product or service) Business Continuity Program The Important Components Recovery Detection Emergency Response Crisis Management Business Continuity/IT Plans Minutes A Going Concern, 2015 Hours Weeks 7 Planning Concepts and Issues Scenario based approach creates problems and roadblocks – We think in terms of events – We plan in terms of impacts to build flexible and responsive plans • For example, in Healthcare, Patient Safety is key (immediate recovery need), whereas operations and administration are vital and some of them can wait a long time to recover. When building plans, the timeline to accomplish all the parts is difficult to schedule and other priorities will continue to compete for time from participants Some processes may need to be changed to make them recoverable A Going Concern, 2015 How do all these different elements work together? Incident Occurs Normal Operations Capability Recovery Time Objective Recovery Risk Acceptance Recovery Return to Normal Operations Restoration Activities Emergency Response Restoration Transfer & Finance [Plan activation, strategy] [Insurance] Prevention [ERM, Crisis Mgmt., DRP, BCP] Crisis Management Proactive Risk Activities Prevention and Preparedness Time Reactive Risk Activities Proactive Risk Activities Prevention and Preparedness Response, Recovery & Restoration A Going Concern, 2015 Minimum Acceptable Level of Capability Process criticality and recovery sequence are established with senior leadership and key stakeholders. Process Process Criticality Classification Determined by senior management Criticality is a function of tolerance for downtime and data loss at time of disaster Sub Process RPO Determined by line management 96hrs Tolerance for Data Loss Recovery Point Objective 72hrs Resources 48hrs •People •Data •Work area •Vital records •Computing •Vendors •Applications 24hrs 6hrs 6hrs 24hrs 48hrs 72hrs RTO Event Tolerance for Downtime Recovery Time Objective A Going Concern, 2015 96hrs Applications Internal and external applications Why do auditors care about BCP? Depending on where you work – it may be required Audit programs are an integral part of the mitigation/prevention - just like you help in the Infosec, Safety, Security, etc. You will need to recover your programs following a disaster (really??????) Driver for needed changes in the organization’s culture. A Going Concern, 2015 Some Audit Observations IT DR Testing – Use of “virtual” environments which do not completely replicate the actual production environment IT DR Testing – Lack of use of opportunistic testing by way of required maintenance. IT DR, Detailed Recovery Procedures – Lack of documentation to allow for appropriate hand-off between internal IT dependencies during recovery IT DR, Shared Drives – Use of network shares for critical transactional data with no means in place for failover. IT DR, Sign-Off – Appropriate level of leadership not accountable for the contents of the DR SOP. IT DR, Documentation – Lack of integration between IT DR Plan and Business Continuity Program. A Going Concern, 2015 Current Trends in BCP / DR A Going Concern, 2015 Areas to Watch Trends for 2015 Supply Chain focus (less manufacturers and suppliers) Technology – virtualization & cloud (public and private) services (continues from the previous 3 years) Outsourcing of functions (changes the dynamics and risks) Broader communications More single points of failure Doing more with less Crisis Management Issues A Going Concern, 2015 Supply Chain Focus Customers pushing BCP planning down to suppliers – Automotive industry has been doing this for some time – Food industry has begun this as well – HealthCare is poised Unreasonable demands – Partner with competitors – Suppliers maintain all inventory – Tier one suppliers bear the burden without the reward A Going Concern, 2015 Technology - Virtualization Most companies now looking at how to virtualize data center and recovery – See lower operating cost – Do not realize potential increase in risk • Fewer machines, not clustered • One breaks, many affected Applications may not handle it well Complex existing infrastructure may make it hard to achieve Vendor dependence A Going Concern, 2015 Outsourcing of Functions IT, HR, Data Centers They are not employees – their contract specifies actions and responses Critical functions may be outsourced You may not be their only client, nor their highest priority A Going Concern, 2015 Broader Communications To All employees; not just response teams 30 minute or less messaging External and internal recipients More forms – Email – Letters – Printed materials – Texts – Media releases A Going Concern, 2015 More Single Points of Failure Loss of personnel and shrinking headcount – More gaps from senior to junior personnel – Less staff = less cross-training – Retirement disaster larger than ever Less spend on technology and redundant systems Outsourced functions A Going Concern, 2015 Doing More with Less Less staff Less budget Less testing Less time with business More capability More responsibility A Going Concern, 2015 Practical Considerations A Going Concern, 2015 Practical Considerations for Auditors How often should a plan be updated? – How often do you see them updated? – The answer is: How much stuff needs to be in a plan? – How long do you think a plan will survive and event? – Does it show how to lead and make decisions? – Does it provide for how we communicate? How do you audit a plan without always being the bad guys? – Just don’t do them? – Help explain why and how the planning works? – Staff assistance! (the other guy can do the work!) A Going Concern, 2015 Tools and toolkits We commonly find plans built in MS Word or Excel, which can be housed in Sharepoint, network shares, or third party cloud solutions. There are outsourced options for you – we like RPX – Recovery Planner There are very complex and comprehensive programs with web based or locally hosted option – the old Strohl Systems LDRPS (now part of SunGard) Many are trying to use Archer to house plans. In BCP / DR you need a tool that fits your organizational need and budget! A Going Concern, 2015 Disaster Recovery BCP & IT DR Not exclusive of each other: Must have both for the system to function Realistic requirements based upon expected impacts Team effort Must be consistent in “manual” processes and procedures Must be able to update systems when they are restored to maintain accurate data and care provided record Tested in small teams, integrated into total package Training is essential – all team members must understand and be able to follow the process Leadership and supervisor decisions to the recovery are essential A Going Concern, 2015 Disaster Recovery Multi-layered approach required (Over-Arching DR Plan – DR Teams – DR SOP’s) Simple backup to tape will not suffice (understanding tomorrow's technology) Immediate availability is difficult and costly (and may still fail) If possible, design the recovery strategy into the data center(s) or Colocation / Managed Solution Minimize single points of failure Automate where possible Build resistance to virus/trojan/malicious code into the backup and recovery processes. Train, practice and demonstrate A Going Concern, 2015 Business Recovery After the event, the data from before must be restored, then the data during must be input to ensure an accurate patient record and business record Cross functional teams are best at designing and implementing these procedures. IT, Business Units, Public & Client areas, Administration are all needed in these teams This is usually the last area implemented since the other processes need to be in place prior to a restoration. The decisions in the previous steps will affect the ability and process of restoration, so often it becomes and iterative process. Keep the restoration in mind during the design phase(s) A Going Concern, 2015 A practical example A Going Concern, 2015 RPX – Recovery Planner A Going Concern, 2015 Closing thoughts A Going Concern, 2015 What about Ebola? A Going Concern, 2015 Keys to success Keep the frustration level very low Make it easy (BJ Fogg) Give it enough time Iterative processes It isn’t real until you practice http://www.behaviormodel.org/ A Going Concern, 2015 Contact Information Fred Klapetzky: 618.581.1047 fred@agoingconcern.com Keith Gregorio: 949.456.6074 keithg@agoingconcern.com www.agoingconcern.com A Going Concern, 2015