BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 BACKUP AND RECORD RETENTION POLICY POLICY The purpose of this policy is to define the need for performing periodic computer system backups to ensure that mission critical administrative applications, data and archives and applications, users' data and archives are adequately preserved and protected against data loss and destruction. Each ETS unit responsible for providing and operating a mission critical application must document and perform System Specific Data Backup or at least Minimal Data Backup on a periodic basis. Computer systems that create or update mission critical State data on a daily basis need to be backed up on a daily basis to minimize the exposure to loss of mission critical data. The unit responsible for providing and operating such systems must conduct a systematic and detailed investigation of all the influencing factors leading to the compilation of a comprehensive System Specific Data Backup Policy. System specific backup policies policy must at least fulfill the requirements of the Minimal Data Backup Policy. APPLICABILITY This policy applies to all operating units of ETS. This backup policy is defined to protect against the following situations: • Destruction of data media by force majeure, e.g. fire or water • Deliberate and/or accidental deletion of files with computer-viruses etc • Inadvertent deletion or overwriting of files • Technical failure of storage device (head crash) • Faulty data media • Demagnetization of magnetic data media due to ageing or unsuitable environmental conditions • (temperature, air moisture) • Interference of magnetic data media by extraneous magnetic fields • Uncontrolled changes in stored data (loss of integrity) BACKUP VERSUS ARCHIVE A backup process takes periodic or real-time images of active data in order to provide a method of recovering records that have been deleted or destroyed. Most backups are retained only for a few days or weeks as later backup images supersede previous versions. A backup is designed as a short-term insurance policy to facilitate disaster recovery, while an archive is designed to provide ongoing access to decades of business information. Archived (historical) records are placed outside the traditional backup cycle for a long period of time, while backup operations protect active data that's changing on a frequent basis. There are now over 10,000 regulations in place throughout the world that require records to be held for certain periods of time. Companies that do not comply face hefty financial penalties, bad PR and even imprisonment for key board members. ARCHIVING IMPLICATIONS SARBANES-OXLEY Page 1 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 A record is essentially any material that contains information about the state’s plans, results, policies or performance. In other words, anything about state business that can be represented with words or numbers can be considered a business record – and ETS is now expected to retain and manage every one of those records, for several years or even permanently depending on the nature of the information. SOX – SECTION 802 Section 802 makes it a crime for anyone to intentionally destroy, alter, mutilate, conceal cover up or falsify any records documents or tangible objects that are involved in or could be involved in, a US government investigation or prosecution of any matter, or in a Chapter 11 bankruptcy filing. Section 802 underscores the importance of record retention and destruction policies that affect all of ETS provided Email, Email attachments, and documents retained on computers – edata – as well as hard copies of all company records. The rules states that if you know ETS is under investigation, or even suspect that it might be, all document destruction and alteration must stop immediately. And, you must create a statement showing that you’ve ordered a halt to all automatic edata destruction practices. ETS also needs to consider all other regulatory rules governing records retention with the industry. For example, FFIEC, SEC, IRS, etc…most documents must be retained for 7 years. RECORD RETENTION REQUIREMENTS The federal government views just about any type of company information as a business record. This includes business documents, in hard copy and electronic form, as well as many other type of electronic files you may not think of as a business record – but the government does. E-data is also subject to disclosure in lawsuits with non-government opponents in federal and state courts, just like traditional paper documents. This Backup and Backup Retention policy does not address mandated requirements for record archiving, such as Email and business records, however this policy works is concert with the Record Management Policy. Archiving requirements are defined in the “Record Management, Retention, and Disposition Policy”. TYPES OF BACKUPS Backups are created to avoid situations of losing precious data. Backups can be created on daily basis, weekly basis, or monthly basis. Backups prove useful at the time of data loss, data inaccessibility, software malfunctions, drive corruptions etc. Before a backup strategy is developed, the types of backups that be performed need to be understood. Defined below are five (5) types. Type Of Backup Full Backup Incremental Backup Description Appropriate Use A full backup creates a copy of every file on a storage device. It is also the most costly in terms of effort, time and dollar output. The media for this can be static (tape, optical disk) or dynamic (disk to disk). These backups are often are used as mandated archive copies. Annual (verified) Backup An incremental backup creates copies of only those files or records on a storage device that have changed since the last backup. It is also more complex to restore when a complete files needs to be restored but it takes less effort to create. Weekly Backup Monthly Backup Weekly Backup Daily Backup Daily Backup When incremental backups are taken planning for full backups needs to be at a frequent enough time period so that recovery is Page 2 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 not hindered. Data Replication Replication is the process of sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility. The same data is stored on multiple storage devices – either in the same physical location or in a remote location via network connectivity Real Time Data Deduplication Data deduplication (often called "intelligent compression" or "single-instance storage") is a method of reducing storage needs by eliminating redundant data. Only one unique instance of the data is actually retained on storage media, such as disk or tape. Redundant data is replaced with a pointer to the unique data copy. Annual (verified) Backup Monthly Backup Weekly Backup Daily Backup This is often used for email where the same email can be stored for several user accounts or for attachments that are duplicated. Transaction Log Backup A transaction log backup creates copies of only those records (in some cases before and after images of records) on a storage device that are changed since the last backup. Daily Backup It requires a version of the application program to run the all of the transactions since the last full backup. STORAGE MANAGEMENT Storage Management is a data storage process which moves data between high-cost and low-cost storage media. Storage Management is needed because high-speed storage devices, such as hard disk drive arrays, are more expensive (per byte stored) than slower devices, such as optical discs and magnetic tape drives. While it would be ideal to have all data available on high-speed devices all the time, this is prohibitively expensive. Instead, Storage Management policies are set so that the bulk of the backup data is on slower devices, and then backup data is transferred to faster disk drives when needed. MINIMAL BACKUP POLICY Type Of Data Minimal Backup Policy Backup Retention Policy System Software Latest Version plus patches At Least Weekly Application Software Latest Version plus patches At Least Annual (verified) Backup Monthly Generations Weekly Generations Annual (verified) Backup Page 3 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 System Data Weekly Monthly Generations Weekly Generations Daily Annual (verified) Backup Monthly Generations Weekly Generations Daily Generations Daily Data Deduplication Application Data Daily with real time transaction files Annual (verified) Backup Monthly Generations Weekly Generations Daily Generations Annual (verified) Backup Monthly Generations Weekly Generations Daily Generations Software licenses, encryption keys & Protocol data weekly Annual (verified) Backup Monthly Generations Weekly Generations Mobile Device Data On connect or at least weekly Monthly Generations Weekly Generations REQUIREMENTS The minimal backup policy mandates the following: • System and application software - All software, whether purchased or developed for the state, is to be protected by at least one full backup which includes all updates. • Application data - All application data are to be protected by means of weekly full back-up using the multiplegeneration retention principle. • System data - System data are to be backed up with at least one generation per month. • Protocol data - All protocol data are to be protected by means of a full weekly backup using the three-generation principle. • Storage - All backup media must be stored in a safe and secure location extraneous to the location of the backed up systems. All weekly backup media must be stored in a fireproof safe. All software full backup and monthly backup media must be stored in an off-site backup archive storage location. • Software licenses and encryption keys necessary to activate both system and application software are to be backed up with at least one generation per week or daily it they change frequently. BACKUP AND RETENTION Backup cycles are defined for daily, weekly, monthly and annual periods. A daily-generation full daily backup cycle involves retaining seven sets of backups (one week, SSMTWTF). Then the seventh daily backup is retained for one month, as part of a weekly backup cycle and stored in a local safe. The fourth weekly backup is retained for one year as part of a monthly Page 4 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 backup cycle and stored in the off-site backup archive storage location. End of fiscal year and yearly archive data backup should be generated in multiple copies and each copy stored in a distinct archive storage location. In this way, the risk of catastrophic loss is minimized at a reasonable media cost. DOCUMENTATION AND BACKUP MEDIA LABELING The backup process and media should fully document the following items for each generated backup: • Date of data backup • Data backup hardware and software (with version number) • Type of data backup (incremental, full) – Monthly and annual backups are full back-up as incremental are too difficult to deal with when recovery from backups is necessary • Number of generations to be retained – destruction date and destruction processes • Responsibility for backup execution and storage • Extent of data backup (files/directories) • Media on which the operational files are recorded • Media on which the backup is recorded • Backup parameters (type of backup media – qualitative and quantitative) • Storage location of backup copies The backup documentation process needs to include the process and procedures that need to be followed to restore the media to the necessary state with the appropriate set of internal controls that comply with the security policies and procedures of ETS and meet all documented and mandated requirements such as Sarbanes-Oxley and audit requirements. STORAGE LOCATION OF BACKUP COPIES STORAGE Backup media, documentation on its use, and necessary hardware and software should be stored in a fireproof and protected location. In the case of magnetic media they should be in a case or vault that is shielded from electro-magnetic radiation. For maximum safety the archive media should be stored at a site that is removed from where the backup media is to be used if necessary CLOUD BACKUP Cloud backup, also known as online backup, is a strategy for backing up data that involves sending a copy of the data over a proprietary or public network to an off-site server. The server is usually hosted by a third-party service provider, who charges the backup customer a fee based on capacity, bandwidth or number of users. In the ETS, the off-site server might be proprietary, but the chargeback method would be similar. Online backup systems are typically built around a client software application that runs on a schedule determined by the level of service the customer has purchased. If the customer has contracted for daily backups, for instance, then the application collects, compresses, encrypts and transfers data to the service provider's servers every 24 hours. To reduce the amount of bandwidth consumed and the time it takes to transfer files, the service provider might only provide incremental backups after the initial full backup. Capital expenditures for additional hardware are not required and backups can be run dark, which means they can be run automatically without manual intervention. Page 5 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 In many states, cloud backup services are primarily being used for archiving non-critical data only. Traditional backup is a better solution for critical data that requires a short recovery time objective (RTO) because there are physical limits for how much data can be moved in a given amount of time over a network. When a large amount of data needs to be recovered, it may need to be shipped on tape or some other portable storage media. Cloud Storage versus Traditional Storage Factor Cloud Storage Traditional Storage Amount of Data Best when the total amount to protect is less than 100 GB per 1 Mb of network bandwidth. For example, 100 GB can be supported by a 1 Mb WAN connection. For large amounts of data, or for environments with limited network connectivity, traditional backup techniques are more appropriate. Rate of Data Change Best when the rate of change is less than 10% of the total data per month. For data that changes frequently, traditional back-up methods that use local disk and tape, with tape transport off-site are more appropriate RESPONSIBILITIES Each backup process should have at least one individual in a defined role in charge and one substitute. In the case of employee termination or removal from that role the Chief Information Officer (CIO) and/or Chief Security Officer (CSO) should immediately see that the substitute assumes those responsibilities and a new substitute is assigned. These responsibilities and this process should be documented in the Disaster Recovery/Business Continuity Plan. TESTING AND TRAINING On at least at irregular (unannounced intervals) and at least annual basis all backup and restoration policies and procedures are tested by individuals who are responsible for those processes. The test is to be monitored by an independent third party either internal audit, external auditors, or consultants uniquely qualified to complete these processes. Testing should verify: • The backup processes fit within the necessary operational window (i.e. a daily backup should not take 25 hours) • The restoration processes fit within the necessary operational window (i.e. master file restoration should not take 25 hours) • The restoration is effective, efficient, and accurate • The documentation is adequate to communicate to someone unfamiliar with the particular process to be able to conduct the backup, store the media, recover the media, and restore the data in an emergency situation. Page 6 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 • This testing should be used as training for other staff members in the backup and restoration policies and procedures. SYSTEM SPECIFIC BACKUP POLICY Type Of Data System Specific Policy Backup Retention Policy System Software Latest Version plus patches At Least Weekly Annual (verified) Backup Monthly Generations Weekly Generations Application Support Software Latest Version plus patches At Least Weekly Annual (verified) Backup Monthly Generations Weekly Generations Application Software Latest Version plus patches At Least Weekly Annual (verified) Backup Monthly Generations Weekly Generations System Data Daily Annual (verified) Backup Monthly Generations Weekly Generations Application Data Daily with real time transaction files Daily Generations Annual (verified) Backup Monthly Generations Weekly Generations Daily Generations Software keys & Protocol Data weekly Annual (verified) Backup Monthly Generations Weekly Generations System specific data backup policy and procedures are driven by various factors, including: • System hardware • OS • Application support systems • Application software • Volume of data (both master files and transactions) • Velocity of data updates • Criticality of the application for states’ continued viability The system specific backup policy mandates the following for each of those systems deemed as unique and necessary for the continued operation of ETS which may have to be restored independently of other applications of functions: • Software - All software, whether purchased or developed for ETS, is to be protected by at least one full backup which includes all updates. Page 7 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 • System data - System data are to be backed up with at least one generation per month. • Application support software - All application support data are to be protected by means of a weekly full back-up using the multiple-generation retention principle. • Application data - All application data are to be protected by means of a weekly full back-up using the multiplegeneration retention principle. • Protocol data - All protocol data are to be protected by means of a full weekly backup using the three-generation principle. • Storage - All backup media must be stored in a safe and secure location extraneous to the location of the backed up systems. All weekly backup media must be stored in a fireproof safe. • All software full backup and monthly backup media must be stored in an off-site backup archive storage location. • Software licenses and encryption keys necessary to activate both system and application software are to be backed up with at least one generation per week or daily it they change frequently BACKUP RETENTION Backup cycles are defined for daily, weekly, monthly and annual periods. A daily-generation full daily backup cycle involves retaining seven sets of backups (one week, SSMTWTF). Then the seventh daily backup is retained for one month, as part of a weekly backup cycle and stored in a local safe. The fourth weekly backup is retained for one year as part of a monthly backup cycle and stored in the off-site backup archive storage location. End of fiscal year and yearly archive data backup should be generated in multiple copies and each copy stored in a distinct archive storage location. In this way, the risk of catastrophic loss is minimized at a reasonable media cost. DOCUMENTATION AND BACKUP MEDIA LABELING The backup process and media should fully document the following items for each generated backup: • Date of data backup • Data backup hardware and software (with version number) • Type of data backup (incremental, full) – Monthly and annual backups are full back-up as incremental are too difficult to deal with when recovery from backups is necessary • Number of generations to be retained – destruction date and destruction processes • Responsibility for backup execution and storage • Extent of data backup (files/directories) • Media on which the operational files are recorded • Media on which the backup is recorded • Backup parameters (type of backup media – qualitative and quantitative) • Storage location of backup copies The backup documentation process needs to include the process and procedures that need to be followed to restore the media to the necessary state with the appropriate set of internal controls that comply with the security policies and procedures of ETS and meet all documented and mandated requirements such as Sarbanes-Oxley and audit requirements. Page 8 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 STORAGE Backup media, documentation on its use, and necessary hardware and software should be stored in a fireproof and protected location. In the case of magnetic media they should be in a case or vault that is shielded from electro-magnetic radiation. For maximum safety the archive media should be stored at a site that is removed from where the backup media is to be used if necessary. RESPONSIBILITIES Each backup process should have at least one individual in a defined role in charge and one substitute. In the case of employee termination or removal from that role the Chief Information Officer (CIO) and/or Chief Security Officer (CSO) should immediately see that the substitute assumes those responsibilities and an new substitute is assigned. These responsibilities and this process should be documented in the Disaster Recovery/Business Continuity Plan. TESTING AND TRAINING On at least at irregular (unannounced intervals) and at least annual basis all backup and restoration policies and procedures are tested by individuals who are responsible for those processes. The test is to be monitored by an independent third party either internal audit, external auditors, or consultants uniquely qualified to complete these processes. Testing should verify: • The backup processes fit within the necessary operational window (i.e. a daily backup should not take 25 hours) • The restoration processes fit within the necessary operational window (i.e. master file restoration should not take 25 hours) • The restoration is effective, efficient, and accurate • The documentation is adequate to communicate to someone unfamiliar with the particular process to be able to conduct the backup, store the media, recover the media, and restore the data in an emergency situation. Page 9 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 BACKUP AND RECORD RETENTION POLICY - APPENDIX BACKUP - BEST PRACTICES STORE DATA PRUDENTLY UNDERSTAND WHEN TO STORE AND WHEN TO DESTROY Consider the value of different types of data that must be stored, and how that value changes over time. While keeping all data close at hand on high speed disks might seem ideal for access purposes, in reality to do so could be prohibitively expensive in terms of both hardware purchases and the cost of power, cooling and physical space, especially when compared with tape storage. In a study, the University of California at Santa Cruz showed that 90% of data stored to NAS was never accessed again, and another 6.5% of the data was only accessed once more. It has been estimated that more than 95 percent of data stored is rarely accessed beyond 90 days after it was created. SEPARATE YOUR DATA Separate your data from your operating systems. Ideally, you should save data files on a separate drive or partition. This will make protection easier in many ways, and it could be the difference between success and failure. For example, you can restore your system to a previous state without reversing your data to that point in time. MANAGE YOUR BACKUP PROCESSES, PROCEDURES, EQUIPMENT, SOFTWARE, AND MEDIA A best practice is to have a set of defined policies and procedures that manage and control it. The policies and procedures should include: • Craft the processes and procedures you need to ensure backups are completed properly, including assigning responsibility for getting backups accomplished and monitoring the effort to spot problems, while also ensuring that those responsible are sufficiently trained. • Ensure that backup copies are valid and can be successfully restored, which requires that you rank the importance of your data and establish ways that the most important data is backed up first and restored first. Be sure that you have adequate time to back-up all the data that is important to your business, and be sure to understand the time required to restore that data in case of loss or corruption. This includes regularly checking and testing your equipment, media, and processes. • Ensure that backup copies are safe. This means storing your backups in a logically and physically secured offsite location. It also means ensuring that you haven’t backed up viruses and other malware, spam, and data that is not important or that is harmful to your business. • Maintain backup logs so you — and your auditors — can track backup activities. • Regularly revisit your backup/restore risks, procedures, and technologies to make sure they are adequate as business needs and conditions evolve. • Dispose of backup media carefully, making sure that they are physically destroyed so that their contents cannot be read by the unauthorized. Page 10 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 IMPLEMENT A REASONED STORAGE ARCHITECTURE Storage architectures provide a way of matching the value of the data to the most cost-effective form of storage. You should place the highest value, time-critical information on storage media that can be easily accessed with minimal time to access data, and to archive little-used information onto low-cost storage media with a proven shelf-life yet acceptable access time. Factors to consider are: • Recovery Time Objective (RTO) - how quickly you need to get this type of data back • Recovery Point Objective (RPO) - how recent the data must be in order to minimize impact to your business minutes, hours or a few days The requirements that need to be addressed include: • Archiving - email and business records that are static can clog storage devices; removing them and saving them to a lower tier (cost) of storage can both free up valuable “productive” storage space and reduce the costs of the overall storage environment. • Data retention for compliance and e-discovery (deep archiving) - separate from archival of more unstructured, infrequently used data is the need to retain information for compliance and business governance reasons. • Data backup and restore - ensuring the timely restoration of data following a user error, system failure or other occurrence. Critical decisions to determining which storage technology to choose include: • Business continuity and disaster recovery - in the event of a significant system failure due to malicious act or natural disaster, what provision needs to be in place to get the business back up and running? MINIMIZE RISK It is a best practice to hold at least 3 copies of data in different locations, including one of these stored in a remote region for disaster recovery purposes in the case of fire, flood, earthquake or business interruption event. Data encryption is a best practice that can and does protect data that is at rest or in transit and is mandated by a number of federal, state, and institutional regulatory bodies It's not just about the reliability of the technology you choose or the security of your location, but about the overall strategy for holding multiple copies on different media, online and offline, secured and protected. MANAGE TOTAL COST OF OWNERSHIP (TCO) CIOs need to consider all aspects of the value of a solution, not only with regard to backup window and recovery times, but also the total ongoing cost of delivering the service. In a data archiving TCO study, the total cost of ownership over a five year period for the longterm storage of data in a tiered storage archiving environment was examined. The analysis compared a disk-to disk solution to a solution consisting of a mixture of disk and tape. After factoring in acquisition costs of equipment, media, electricity costs and data center floor space, the study found that the total cost of archiving solution based on disk was about 23 times more expensive than a tape library archiving solution. VALIDATE THAT DATA CAN BE RESTORED A best practice it to have a plan and process in place to validate that data can be restored. It is therefore important to consider the following factors: Page 11 of 12 BACKUP AND RETENTION POLICY NUMBER: 107-08-nnn EFFECTIVE DATE: mm-dd-2015 • Regular testing of process and media - with all backup data, regardless of technology used for storage, frequent testing of restore the capability essential. • Shelf life - you need to ensure that the storage medium selected has sufficient expected shelflife. In general, tape offers between 4 and 6 times the life expectancy of disk, with media manufacturers specifying up to 15 years for DAT and up to 30 years for LTO tape media. • Efficient restores – the amount of time it takes to restore data needs to fall within the operational requirements of the enterprise. CLOUD BACKUP – BEST PRACTICES • Define specific business requirements for cloud data backup. Don’t forget to also address customer needs. • Conduct a Total Cost of Ownership (TCO) analysis. Use a provider that can integrate archives, so you can move data sets from a backup plan to an archive plan and provides online search and retrieval functionality. • Encrypt the backup. To ensure security, encrypt backup data. Store the encryption key in a place that is secure and will be available if you lose your facility. • Utilize Data De-Duplication. Data de-duplication reduces overall storage and data transmission requirements. This in turn lowers storage and transmission costs. • Follow governance and compliance requirements. For example, regulatory compliance related to where data may move or be stored when different countries or regions are involved, or compliance related to retention periods of data. Be aware of tax, liability, and insurance implications. • Train staff in the cloud connectivity and recovery rocess. Staff should be familiar with procedures related to bulk data import where data is shipped on removable media storage to your recovery site. This option can be critical when faster data recovery is needed for large data recovery efforts. • Do not depend 100% on your cloud. Backup locally and remotely — to both on-premise and cloud storage. • Have a local copy of all publicly accessible cloud data. Backup the data locally before storing in cloud. • Have multiple cloud vendors. Multiple vendors to mitigate risks and provide options when a recovery process is place. • Test entire process before you depend on it. Validate that the backup and recovery process will work in you environment when there is a major outage. Ensure that backed-up data can be recovered on-premise or to another cloud vendor. Page 12 of 12