Cancer Trials Support Unit
___________________________________________________________________________________________________________________________
CTSU – A Service of the National Cancer Institute
Business Continuity
and
Disaster Recovery Plan
Revision 04
01 Sep 2008
Document Information
Revision Information: Cancer Trials Support Unit Business Continuity and Disaster Recovery Plan
Document No. CTSU/SYS0002 Rev. 04
Revision History
#
Date
Editor
Description
01
08/04/2005
Cutler
Initial version for 1st CTSU Contract
02
09/28/2006
Cutler/Wernimont
First release for 2nd Edition of CTSU Contract
03
08/21/2007
Cutler/Wernimont
Semi-Annual review
04
03/14/2008
Cutler/Patgiri
Semi-Annual review
05
07/01/2008
Patgiri
Updated and added OPEN project
06
09/11/2008
Patgiri
Updated
Last Saved By Patgiri_A on 3/7/2016 7:03:00 PM
File Location: \\rk27\vol2703\CTSU8339\Tasks\8339_15_IT\04_Security\CTSU_BCDRPlan_200809.doc
This document was prepared for:
CTSU Project / National Cancer Institute
Approvals:
Project Security Officer / Alan Cutler
Date
CTSU Asst. Project Director for IT / Jerry Wernimont
Date
CTSU Project Director / Steve Riordan
Date
This document was prepared by:
WESTAT
1650 Research Boulevard
Rockville, Maryland 20850
Phone: (301) 251-1500
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page ii
Contents
1.
INTRODUCTION ................................................................................................................................... 1
1.1
1.2
1.3
1.4
1.5
2.
OVERVIEW ........................................................................................................................................ 1
POLICY STATEMENT .......................................................................................................................... 1
REFERENCES.................................................................................................................................... 2
ROLES AND RESPONSIBILITIES ........................................................................................................... 3
SCOPE ............................................................................................................................................. 3
BUSINESS IMPACT ANALYSIS .......................................................................................................... 4
2.1
OVERVIEW ........................................................................................................................................ 4
2.2
IDENTIFICATION OF CRITICAL RESOURCES .......................................................................................... 4
2.2.1
CTSU Enterprise Database ................................................................................................................... 4
2.2.2
Enterprise Application Servers .............................................................................................................. 4
2.2.3
Regulatory Support System .................................................................................................................... 5
2.2.4
Oncology Patient Enrollment Network .................................................................................................. 5
2.2.5
Financial Management System .............................................................................................................. 5
2.2.6
Clinical Trial Management System ........................................................................................................ 5
2.2.7
Real-Time Data Transfer System (RDTS) .............................................................................................. 6
2.2.8
CTSU Web Sites ..................................................................................................................................... 6
2.2.9
Image Management Systems .................................................................................................................. 6
2.2.9.1
2.2.9.2
Regulatory Image Management System (RIMS) ............................................................................................... 6
Clinical Data Image Management System (CDIMS) ......................................................................................... 7
2.2.10
CTEP Database Access ......................................................................................................................... 7
2.3
DISRUPTION IMPACTS AND RECOVERY PRIORITIES ............................................................................. 7
3.
IDENTIFICATION OF PREVENTATIVE CONTROLS .......................................................................... 9
3.1
3.2
3.3
3.4
3.5
4.
OVERVIEW ........................................................................................................................................ 9
SERVER RELATED ............................................................................................................................. 9
NETWORK RELATED .......................................................................................................................... 9
INFRASTRUCTURE RELATED .............................................................................................................. 9
SECURITY RELATED ........................................................................................................................ 10
RECOVERY STRATEGIES ................................................................................................................. 11
4.1
OVERVIEW ...................................................................................................................................... 11
4.2
RECOVERY METHODS ..................................................................................................................... 11
4.2.1
CTSU Enterprise Applications ............................................................................................................. 11
4.2.2
CTSU Oracle Clinical Systems ............................................................................................................ 11
4.2.3
Other Systems ...................................................................................................................................... 11
5.
PLAN TESTING, TRAINING, AND EXERCISES ............................................................................... 12
6.
PLAN MAINTENANCE........................................................................................................................ 13
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page iii
Overview
1. Introduction
1.1
Overview
The CTSU is a support services organization sponsored by the National Cancer Institute, National
Institutes of Health, to provide support services for cancer treatment trials. The systems used to support
this organization have been developed, implemented, and operated on the Westat campus. The CTSU
computer systems are linked to NCI computer systems as well as other systems at the Cancer
Cooperative Groups; however, the scope of this document only applies to Westat’s systems.
The CTSU Enterprise is a collection of databases, systems, and applications that support the operations
of the CTSU. The systems are located in Westat’s headquarters in Rockville, Maryland, near
Washington, DC, in Westat’s two computer facilities. The enterprise systems are supported with an
extensive computer system infrastructure including power backup, firewall protection and communication
and Internet connections.
The CTSU Business Continuity and Disaster Recovery Plan addresses the overall plan and relates to
business continuity and disaster recovery. It is supported by more detailed documentation that includes a
CTSU Security Plan, multiple detailed Disaster Recovery Plans, and corporate security and disaster
recovery plans and procedures.
1.2
Policy Statement
The CTSU Area develops and maintains business continuity and disaster recovery plans for the major
systems/applications of the CTSU Enterprise to maintain a high availably environment capable of
sustaining major failures with minimal disruption of services. The goal is to restore operations within a
timeframe based on Client requirements and/or expectations.
The plan is consistent with and built upon the Corporate and Platform Contingency and Disaster
Recovery Plans. These plans also are shaped by the following Clinical Areas Standard Operating
Procedures:

SOP IT-109
System Security Procedures for the Clinical Trials Area

SOP IT-110
Systems Business Continuity and Disaster Recovery (BC/DR)

SOP IT-105
Computer Backup and Restoration
The plans shall be reviewed annually by senior managers. Resources necessary to meet continuity and
disaster recovery goals will be acquired and maintained. The plans shall be tested (or executed in a live
disruption) annually. The plans will also be updated whenever any major change occurs in the computing
infrastructure.
Daily Backup operations begin on weekdays after the regular workday is complete, and on weekends,
backup operations begin in the early afternoon. Backup operations are monitored by trained operators.
The details of the Backup and Recovery procedures and offsite storage are described in detail in the
CTSU Security Plan.
Specific responsibilities are assigned to the CTSU staff that support contingency and disaster recovery
operations, and are documented in the following section. Staff are provided training prior to be being
involved in the support.
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 1
References
1.3
References
The following are CTSU, Corporate, and NIH related references that support the Business Continuity and
Disaster Recovery Plan.
#
Name
Location
1
CTSU Enterprise Security Plan
Project Network Directory
2
CTSU Enterprise Disaster Recovery Plan
\\rk27\vol2703\CTSU8339\Tasks\8339_15_IT\04_S
ecurity\Docs\Current\CTSU_DR_Enterprise.doc
3
CTSU Oracle Clinical Failover Plan
\\rk27\vol2703\CTSU8339\Tasks\8339_15_IT\04_S
ecurity\Docs\Current\CTSU_DR_Clinical.doc
4
Corporate Network Systems Continuity and
Disaster Recovery Plan
Westat Corporate Internal Web Site
5
Westat Information Technology and
Systems (ITS) Security Policy and Best
Practices
Westat Corporate Internal Web Site
6
Corporate Oracle Clinical (OC), Remote
Data Capture (RDC), and AERS Platforms
Systems and Data Continuity Plan
Westat Corporate Internal Web Site
7
Westat Clinical Trials Area SOP IT-109
System Security Procedures for the Clinical
Trials Area
Clinical Trial Network Directory
8
Westat Clinical Trials Area SOP IT-110
Systems Business Continuity and Disaster
Recovery (BC/DR)
Clinical Trial Network Directory
9
Westat Clinical Trials Area SOP IT-105
Computer Backup and Restoration
Clinical Trial Network Directory
10
NIST Special Publication 800-34
Contingency Planning Guide for Information
Technology Plans
NIH Web Site
NIST Special Publication 800-30.
11
Risk Management Guide for. Information
Technology Systems
12
CTSU Enterprise Database Failover Plan
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
NIH Web Site
\\rk27\vol2703\CTSU8339\Tasks\8339_15_IT\04_S
ecurity\Docs\Current\CTSU_DR_Enterprise.doc
Page 2
Roles and Responsibilities
1.4
Roles and Responsibilities
Project Manager – Responsible for driving the business continuity and disaster recovery requirements,
and making disaster/recovery decisions based on feedback and advice from the technical
disaster/recovery team.
Disaster/Recovery Coordinator – Responsible for coordinating the development of advance
arrangements and procedures to ensure and organized systematic response to a disaster so critical
business functions can be resumed within a defined time frame and the amount of loss can be minimized.
This responsibility includes coordinating the design, development, maintenance, and exercising (testing)
of the overall disaster recovery plan. The Disaster Recovery Coordinator participates in any actual
disaster recovery effort and in that role coordinates and communicates with all members of the
disaster/recovery team, and senior Project Staff.
Disaster/Recovery Team – Members of this team are coordinated by the Disaster/Recovery
Coordinator in the event of a disaster or disaster test. They include:
Project Oracle Group – CTSU project DBAs support the non Oracle Clinical Servers for applications like
RSS/OPEN/CDIMS etc and are responsible for disaster recovery operations of both Database and
Application Servers.
Corporate Oracle Clinical Group - DBAs from this group support CTSU Oracle Clinical Servers for the
CTSU Enterprise and are called upon during disaster/recovery operations.
Corporate Systems and Network Team –- During failover, this Group provides Windows and Network
support. The System’s and Network Group are on call 24*7, and can be reached by pager.
Other Corporate Platform Teams

Corporate Email

Novell Network
1.5
Scope
This plan applies to CTSU Enterprise computing environments managed by the CTSU Project at Westat.
This plan addresses business continuity and disaster recovery operational, procedures and policies not
already covered in Westat’s Corporate Plan (Westat’s Business Continuity and Disaster Recovery Plan)
or Westat’s Platform Plan.
The Corporate Plans describes the approach for the continuous operations of Westat’s computer network
and data communication facilities in response to potential disruptive events. A Westat Vice President of
Computer Systems is responsible for developing and updating this plan with inputs and reviews from
other technical and corporate managers. This Vice President also heads the Systems Disaster
Assessment and Recovery Team (SDART). This plan builds upon the Corporate Plan to meet the specific
needs of the Clinical Trials Area. The CTCDDRP also builds upon the Westat Clinical Trials Security Plan
which describes its activities for the protection if the data and computer systems.
Westat Platform Group Plans describes the Business Continuity and Disaster Recovery Plans for major
applications/services used by Corporate Staff and Projects throughout Westat. They include:

Oracle Clinical, and Remote Data Capture Platforms

Corporate Email

Novell Network
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 3
Overview
2. Business Impact Analysis
2.1
Overview
A number of servers/applications make up the CTSU Enterprise and are deemed critical to business
processes. It is these servers/applications along with their associated interfaces, interconnections,
components and processes that have been identified as critical resources, and are included in this plan.
The primary applications and systems that comprise the CTSU Enterprise are:

Regulatory Support System

Oncology Patient Enrollment Network

Financial Management System

Clinical Trials Management System, including
-
Data Management Systems (Oracle Clinical)
-
Remote Data Capture System (Oracle RDC)
-
Patient Enrollment Credentialing

Image Management Systems

Public Web Site

Members’ Web Site
To support these systems and their integration with the Cooperative Group systems and the NCI CTEP
databases, several data provisioning and data distribution mechanisms are used, including:

Real-Time Data Transfer System (RDTS)

Clinical Data Transfer System (CDTS)
These systems work together to support the processes required by the CTSU from protocol management
and regulatory data collection to patient enrollment and data collection.
2.2
Identification of Critical Resources
2.2.1
CTSU Enterprise Database
The CTSU Enterprise Database supports most of the critical applications in some capacity, with the
exception of Oracle Clinical and RDC. This database is a critical resource required for these systems to
operate properly. The dependent systems include:

Regulatory Support System

Oncology Patient Enrollment Network

Financial Management System

Patient Enrollment Credentialing

Image Management Systems

Real-Time Data Transfer System (RDTS)

Clinical Data Transfer System (CDTS)

Members’ Web Site
2.2.2
Enterprise Application Servers
These servers host many of the user interface tiers that the CTSU Enterprise supports, and are as such
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 4
Identification of Critical Resources
as critical as the enterprise database servers.
2.2.3
Regulatory Support System
The Regulatory Support System includes a comprehensive store of IRB approvals, site registration and
institutional and person credentials. Its primary goal is to reduce the redundant collection off regulatory,
person and institutional data, and provide a mechanism to share pertinent information across the
Cooperative Groups, CTSU and CTEP. Cooperative Groups also use RSS to enter and maintain relevant
protocol and roster data. The system supports the patient enrollment process by managing the
availability of protocols to sites based on the regulatory data collected, group roster information, and the
current NCI maintained status of investigators and sites.
The CTSU Regulatory Office, operated by the Coalition of Cancer Cooperative Groups, is the primary
user of the RSS for collection and documentation of regulatory data. The supporting system architecture
includes web-based Oracle Forms-based data entry screens as a front-end to the CTSU Enterprise
database, which is integrated with data from the CTEP Enterprise system.
2.2.4
Oncology Patient Enrollment Network
The Oncology Patient Enrollment Network (OPEN) is a web-based registration system for the enrollment
of patients onto NCI-sponsored Cooperative Group clinical trials. This system has been developed to
leverage the existing CTSU Enterprise System of regulatory information and the existing integration of the
CTEP Enterprise and Cooperative Group management systems. The OPEN Portal will collect the
information needed to enroll patients, including data to verify institutional and investigator qualifications,
as well as data needed for verification of trial eligibility. This data will be sent electronically to the
Cooperative Group systems that can then assign treatments (both randomized and non-randomized
assignments), with the results presented to the site registrar in real-time
2.2.5
Financial Management System
The Financial Management System creates and processes invoices in support of CTSU operations and
patient accruals. This system does not perform any actual accounting functions, but interfaces with the
Westat CostPoint system for generation of checks and reconciliation.
The primary functions of the FMS include:

Maintaining information regarding contractual relationship between the CTSU and various
participating individuals and entities,

Maintaining the receipt, approval, and processing of payment requests from various parties
for CTSU supported activities such as travel, and technical consulting, and

Computing and generating invoices based on patient accruals and other events generated
by the collection of clinical data.
The system generates various reports in support of operation, oversight, and management of the
process. Selected reports are provided to NCI and the Cooperative Groups on a regular basis.
2.2.6
Clinical Trial Management System
The Clinical Trial Management System is comprised of the processes, procedures, and applications
needed to support patient registration and randomization, data capture, data quality management, and
data transfer to the Cooperative Groups.
The system can be organized into the following major functions and systems:
Data Collection and Management – Oracle Clinical (OC) is a commercially available clinical trials data
management system developed by Oracle Corporation. OC allows for data entry at a central location (e.g.
Westat) into a central database, and it also allows for web-based data entry via RDC from many locations
(e.g. study sites) into a central database. The core functions of this system support the entry of patient
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 5
Identification of Critical Resources
registration data, randomization of patients, and the entry of data from the clinical trial process such as
CRFs and other applicable sources.
This system manages the most sensitive data in the CTSU project - patient information. Issues of user
authentication and data confidentiality are of particular importance in the current implementation of the
Oracle Clinical Remote Data Capture system. The RDC system is configured in such a way that the site
users can access and manage data on their own patients.
Patient Enrollment- The system captures enrollment information, including enrolling site and physician,
to determine whether the site/investigator is eligible to enroll the patient through CTSU. This system is
heavily dependent on the Regulatory Support System.
Clinical Data Transfer System (CDTS) – The Clinical Data Transfer System (CDTS) is an application for
electronic transfer of data in XML format to the Cooperative Groups from the CTSU Oracle Clinical
database. The files generated are made available through a secure web site for download by the
appropriate Cooperative Group.
Quality Management – These functions support the staff in analyzing and summarizing data for use by
CTSU and NCI staff, for the following purposes:

Monitor the compliance of CTSU clinical trial activities with established standards and
procedures

Monitor the progress of individual protocols against performance goals

Monitor the quality and completeness of data provided by clinical sites or Cooperative
Groups
The CTMS applications consist primarily of Oracle Forms and Reports based on information extracted
from the Oracle Clinical database. These systems are all integrated with the CTSU Enterprise database.
There may be some data editing and limited data entry of information not collected elsewhere.
2.2.7
Real-Time Data Transfer System (RDTS)
The Real-Time Data Transfer System sends transaction details from the RSS to CTSU business partners.
These transactions are triggered by RSS screen updates made by CTSU business partners. A dedicated
leased line between Westat and NCI is used to provide access to related NCI databases and systems.
2.2.8
CTSU Web Sites
The CTSU public web site provides general information about the CTSU to physicians, patients and the
general public. The site includes a search engine to display publicly accessible information about
protocols in the CTSU menu such as: Protocol Title, Lead Group, Abstract & Trial Sites, Eligibility Criteria,
Schema, and Physician Fact Sheet.
The CTSU members’ web site provides support for members who wish to enroll patients on CTSU menu
trials. These include but are not limited to all the necessary documentation and forms for protocols,
educational material on CTSU processes, drug safety notifications, and protocol updates. Documents are
obtained from various sources, such as the Cooperative Groups, NCI, and developed by the CTSU, and
posted in the members’ web site. Members also have access to a subset of data from the Regulatory
Support System based on their membership credentials maintained by the Cooperative Groups.
2.2.9
2.2.9.1
Image Management Systems
Regulatory Image Management System (RIMS)
The Regulatory Image Management System (RIMS) is used to capture fax images electronically and
manage them for data entry into RSS. This system has components in our Regulatory Office in
Philadelphia as well as the Westat computer centers. The main repository is in the main Westat
computer center, a failover repository is in the secondary Westat computer center , with a redundant
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 6
Disruption Impacts and Recovery Priorities
image repository in Philadelphia.
2.2.9.2
Clinical Data Image Management System (CDIMS)
The Clinical Data Image Management System (CDIMS) is used to capture images of clinical forms
electronically and manage them for data entry into Oracle Clinical.
2.2.10 CTEP Database Access
Access to the CTEP Enterprise database is done through a T1 connection to the NIH network. This
access is the source of transactions that update investigator and protocol information in the RSS and is
distributed to the Cooperative groups through the RDTS.
2.3
Disruption Impacts and Recovery Priorities
The following table describes the impact on CTSU-related operations due to disruption of certain
resources. The table categorizes them and prioritizes their recovery, as well as provides a high-level
indication of the recovery method. Details about recovery strategies are described in a later section, and
details of recovery procedures are included in detailed disaster recovery plans.
Resource
Impact
Priority
Remarks / General Recovery Strategy
CTSU Enterprise Database
Critical
1
Affects operation of most CTSU applications and
web site.
Recovery: Failover to hot standby.
CTSU Application Servers
Critical
1
Access to many (but not all) applications is
disrupted.
Recovery: Use alternate server(s)
Regulatory Support System
Critical
1
Affects patient enrollment, regulatory data entry,
and many other processes
Interim Operation: Paper based for time of need
enrollments
Recovery: Use alternate server(s)
Oncology Patient Enrollment
Network
Critical
1
Affects patient enrollment.
Interim Operation: Paper/Fax-based for time of
need enrollments assuming Group Systems are
available. Interim operations available only during
business hours of Groups.
Recovery: Use alternate server(s)
Oracle Clinical
High
2
Data entry and data delivery are impacted; loss of
productivity.
Recovery: Failover to hot standby
External network access
High
2
External access to Regulatory Support System and
web sites. Data entry delays by groups into the
system.
Patient enrollments are not affected.
Recovery: Dual network connections operational; if
both fail we are dependent on one of the ISPs.
CTSU Web Sites
High
2
Access by sites is disrupted; Information is
available through the CTSU help desk if needed.
Recovery: Switch to alternate server
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 7
Disruption Impacts and Recovery Priorities
Resource
Impact
Priority
Remarks / General Recovery Strategy
Regulatory Data Transfer
System
Medium
3
Affects the automatic distribution of data updates
to the groups. For time of need enrollments, this
data is needed by the groups.
Interim Operation: Access to images of IRB and
other regulatory documentation is available to the
groups for direct keying into their system.
Recovery: Use backup server for RDTS data feed..
Image Management System
Medium
3
Affects the workflow processing
Interim Operation: Paper based facilitates timecritical operations.
Recovery: Depending on component, alternate
servers or backup restoration.
CTEP Database
OPEN credentialing is affected.
Critical
1
Investigator registrations and status updates are
delayed. Access to detailed protocol information is
unavailable. RSS can operate without the
connection.
Interim Recovery: Brownout mode for CTSU
provides access to all data through the previous
day.
Recovery: Dependent on CTEP enterprise system
managers.
NIH Network Connection
Is the connection to CTEP
database affected?
If so, OPEN credentialing is
affected.
Critical
1
Investigator registrations and status updates are
delayed. Access to detailed protocol information is
unavailable. RSS can operate without the
connection.
Interim Recovery: Brownout mode for CTSU
provides access to all data through the previous
day.
Recovery: Failover to Internet connection
CDTS
Low
4
Delivery of clinical data delayed.
Interim Recovery: This application can run on
many different servers and can be run manually.
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 8
Overview
3. Identification of Preventative Controls
3.1
Overview
There are substantial preventative controls in place for all CTSU production servers that are detailed in
Westat’s Network Systems Continuity and Disaster Recovery Plan. These preventative controls are listed
below:
3.2
Server Related
The following are server related controls that are currently in place.

RAID1/RAID6 disk redundancy

Diesel generator support (~3 days, and after that refueling of generator likely to extend
generator support)

Daily system backups to tape media

Offsite storage of backup media

Daily database exports to tape media and online storage

Duplicate copies of the Oracle online redo log files and control files are stored on separate
drives on the production database servers. In addition, a third copy of the active log files is
stored on the failover servers located in the alternate data center.

Continuous 24*7 monitoring of critical IT resources (including application servers)
3.3
Network Related
The following are network related controls that are currently in place.

Dual independently routed connections to the Internet

Redundant firewalls

Patch Management - CTSU maintains a very aggressive security patch management
program. Each alert, advisory, patch notification or vulnerability notification is analyzed the
week it has been released and a patch implementation is scheduled unless the patch is not
relevant (wrong O/S version, product not installed, etc.),
3.4
Infrastructure Related
The following are infrastructure related controls that are currently in place.

Air conditioning systems with adequate capacity and redundancy

Temperature controls

Humidity controls

Power surge protection

UPS

Water/flood sensors

Smoke detectors

HALON fire extinguishing system

Glass breakage sensors

Break-in sensors

Master power shutdown switch in the main Computer Facility
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 9
Security Related

3.5
Diesel power generator for the Main Computer Facility and the building it is located in
Security Related
The following are security related controls and procedures that are currently in place.

Anti-Virus - Trend Micro’s server protect software provides centralized managed virus
protection for all CTSU Enterprise Servers. The signature files are updated every night
automatically. Virus scanning is done in real time.

SNORT is used for network intrusion detection monitoring.

TACACS and RADIUS are used for Network user authentication.

Network Penetration Testing - Westat contracts with a qualified network security firm to
conduct network security penetration testing to identify possible vulnerabilities to Westat
systems from the public Internet. This test is performed at least twice annually. Separate
penetration tests of resources located in the WesNet and Data Zones are also performed
twice each year. All results of the tests are received by the Corporate Officer for Systems
Security (COSS) and formal reports of any identified server or system vulnerabilities are
made to the appropriate systems technical administrators and managers who are required
to respond with information on any corrective actions taken.
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 10
Overview
4. Recovery Strategies
4.1
Overview
As previously mentioned, daily backups are performed for all Critical IT Systems, and the specific details
of backup and recovery operations and offsite storage are detailed in the CTSU Security Plan.
Westat has two data centers and several satellite facilities throughout the WAN infrastructure to support
computer operations. The corporate data centers (primary and failover) operate in parallel with each other
and are supported by systems programmers, system administrators, network engineers, managers, and
operators located on our main corporate campus. The data centers operate 24 hours a day, 7 days a
week, with operators on site every day, including evening shifts during the week.
In the event of a failure of the primary data center, or computer systems in the primary data center, or the
primary network infrastructure, the redundant data center is able to provide continuing operations. The
network capabilities in the redundant data center have been tested and proven to be reliable.
4.2
Recovery Methods
4.2.1
CTSU Enterprise Applications
Due to the critical nature of many of applications within the CTSU Enterprise (e.g., the Regulatory
Support System), the most robust recovery method was implemented that consists of a Warm Mirrored
Site (Westat’s Failover Facility) to support failover operations for extended outages. This capability has
been operational for several years and its redundant and mirrored capabilities have been tested on a
regular basis since its inception.
CTSU maintains a detailed step by step Disaster Recovery Plan for the CTSU Enterprise. This plan
follows NIST SP 800-34. This plan has been followed for each Disaster Recovery exercise and actual
disruption of service.
4.2.2
CTSU Oracle Clinical Systems
Due to the critical nature of the Oracle Clinical Systems and applications hosted on these servers, the
Warm Mirrored Site (Westat’s Failover Facility) approach is also used.
CTSU maintains a detailed step by step Disaster Recovery Plan for the CTSU Oracle Clinical System.
This plan follows NIST SP 800-34. This plan has been followed for each Disaster Recovery exercise and
actual disruption of service.
4.2.3
Other Systems
For less critical systems, alternate servers are available to use to restore systems from backups if
required. It is the plan for CTSU to have all systems available with the most robust recovery method to
improve the overall availability of all system components.
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 11
Recovery Methods
5. Plan Testing, Training, and Exercises
Corporate Business Continuity and Contingency testing exercises began in 2005 and have continued
since. They are run semi-annually and each test is set up to test different failure situations, the most
severe of which simulate the entire loss of power and networking to the main computer facility. CTSU
Enterprise system disaster recovery procedures are tested during these exercises.
In addition to Corporate Business Continuity and Contingency testing exercises, the CTSU Regulatory
Support System and the Remote Data Collection System Disaster Recovery plans are exercised at least
annually each year. These plans have also performed very well against all major system, network and
power-related outages.
After a major disruption of services occurs and a recovery operation is completed, lessons learned are
reflected in updates to the plans. During any recovery operation, any new staff that are identified to
support recovery operations are brought along to observe the entire process one time, and the next time
they will do the recovery operations themselves with a veteran staff monitoring their actions and providing
guidance. On the third recovery, they can perform the recovery by themselves but in practice will have a
veteran recovery specialist available for assistance.
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 12
Recovery Methods
6. Plan Maintenance
This plan and the individual Disaster Recovery Plans will be updated semi-annually and additionally,
whenever a major change occurs in the CTSU Enterprise. The plans are updated by the lead CTSU
Disaster Recovery and Security Staff. These plans are also updated after a Risk Analysis is completed.
Copies of this plan and the disaster recovery plans are located on the CTSU Project Network directory.
Access to these plans is controlled. There is a record of changes section in each plan to identity the
changes in each version, the date the changes were made, and who the changes were made by.
Business Continuity and Disaster Recovery Plan
CTSU/SYS0002 Rev. 03-2016-03-07
Page 13