2013 Edgewood College Prepared by the: Information Technology Services Office [BACKUP POLICY & PROCESS] Contents Purpose ..........................................................................................................................................................................3 Definitions ..................................................................................................................................................................3 Scope..............................................................................................................................................................................6 Resources .......................................................................................................................................................................6 Commvault Backup Agents ............................................................................................................................................7 File System - iDataAgent ............................................................................................................................................7 Exchange Database Agent .........................................................................................................................................8 SQL iDataAgent ..........................................................................................................................................................8 Active Directory iDataAgent ......................................................................................................................................9 Sharepoint Agent .......................................................................................................................................................9 Beta/Test/Development Environments .......................................................................................................................10 Decommissioning Systems ...........................................................................................................................................10 Overview ..................................................................................................................................................................10 Procedure .................................................................................................................................................................10 Test and Restore of Backups ........................................................................................................................................11 Overview ..................................................................................................................................................................11 Procedure .................................................................................................................................................................11 Backup Monitoring Process .........................................................................................................................................11 Overview ..................................................................................................................................................................11 Procedure .................................................................................................................................................................11 SQL Server Maintenance Plans ....................................................................................................................................11 Overview ..................................................................................................................................................................11 Procedure .................................................................................................................................................................11 Exceptions ................................................................................................................................................................13 Core and Network Infrastructure Backups ...................................................................................................................13 Overview ..................................................................................................................................................................13 Procedure .................................................................................................................................................................13 Definity and Audix backups (Phone/Voicemail) ...........................................................................................................13 Overview ..................................................................................................................................................................13 Procedure .................................................................................................................................................................13 Revision Table ..............................................................................................................................................................15 Purpose This document outlines the processes and procedures used by Edgewood ITSO to backup server databases and file systems. Included herein are definitions of terms, schedule of backups, retention policies, and other relevant backup information. Definitions Subclient o A subclient is a portion of a client, and it can either contain all of the client's data or a designated subset thereof. Storage Policies o Storage policies act as the primary channels through which data is included in data protection and data recovery operations. A storage policy forms the primary logical entity through which a subclient or instance is backed up. Its chief function is to map data from its original location to a physical media. Vault Tracker Policies o The VaultTracker® feature provides the capability to track media movement between multiple locations Retention Policy o Time constraints that data is stored on disk or tape before it is deleted. Retention has two parts days and cycles. Both conditions must be met before data is allowed to be overwritten. For example: (F = Full, I=Incremental, B=Bad backup) F I I I I F I I I I B = 8 days 1 cycle. If the B was an F then it would equal 9 days 2 cycles. If the retention policy was set at 14 days 2 cycles then the retention would not be met. Cycles o Point of time between successful full backups. 1 cycle = 2 Successful full backups in a period of time. For example: (F = Full, I=Incremental, B=Bad backup) F I I I I F I I I I B = 8 days 1 cycle. If the B was an F then it would equal 9 days 2 cycles. If the retention policy was set at 14 days 2 cycles then the retention would not be met. Scheduled Policies o When multiple clients or tasks require similar operations to be scheduled you can create a schedule policy as a template and attach it to each of these clients or tasks in a CommCell. Magnetic Library (disk library) o A disk library is a virtual library associated with one or more mount paths. The disk library does not represent a specific hardware entity; it is a software entity that contains a list of mount paths through which data can be sent to a disk media. A disk library can be configured by assigning one or more mount paths to it. The storage capacity of a disk library is determined by the total storage space in its mount paths. Disk media, whether individual hard disks or RAID arrays, is logically divided into partitions. A partition can include some or all of the total disk storage space available. Each partition is associated with a file system path called a mount path, through which data is written to and read from. Tape Library o Piece of hardware that uses LTO3 or LTO5 tapes to record backup data. FireSafe o Fireproof safe located in backup datacenter that stores used and available tapes that are not in the tape library. Archive o Full backup and/or specific directories of subclient that will be written to tape with an infinite retention. Tape is then stored in the FireSafe. Media Agent o The MediaAgent manages the transmission of data between clients and backup media. There can be more than one MediaAgent within a CommCell. Commcell o The CommCell Console is the graphical user interface that allows you to control and manage a CommCell. The CommCell Console can be run in two ways: as a stand-alone application, which is installed directly on to a CommServe, MediaAgent, Client, or any other computer that can communicate with the CommServe. For comprehensive information on supported platforms, see System Requirements CommCell Console as a Stand-Alone Application. as a remote web-based application, which allows you to access the CommCell Console remotely via any computer running a supported platform and Java-enabled web browser. For comprehensive information on supported platforms, see System Requirements - CommCell Console as a Remote Web-Based Application. DeDuplication o Deduplication provides a smart and efficient method to store data by identifying and eliminating the duplicate items in backups. All data types from Windows, Linux and Unix operating systems can be deduplicated before moving the data to a secondary storage. Synthetic Full Backup o Synthetic full backup is a synthesized backup. It is created from the most recent full backup (i.e., standard or synthetic) and subsequent incremental and/or differential backups. It, therefore, cannot be the first backup job of a client computer or subclient. The resulting synthetic full backup is identical to what would have been created had the last backup for the subclient been a full backup. Unlike full, incremental, and differential backups, a synthetic full backup does not actually transfer data from a client computer to the backup media. In this sense it is not really a backup at all, rather a backup consolidator. Therefore, since synthetic full backups do not back up data from the client computer, they impose no load on the client computer. Because synthetic full backups consolidate backup data into one archive file, they provide a means of enhancing restore performance, particularly when a given full backup cycle contains many incremental backups. Synthetic full backups are treated by the system as standard full backups, but cannot be the next backup job after a standard full backup. As a result, they delineate full backup cycles in the same manner as standard backups. This is an important point to remember, since retention periods are defined in part by the number of full backup cycles. Using synthetic full backups in an unguided manner may cause the unintentional expiration of data. Synthetic full backups are media-based; they read backup data from one media and simultaneously write (restore) the result to the new active media within the same media group. As a result, synthetic full backups require that at least two media drives for the same storage policy be available at the time the job is started. Synthetic full backups cannot be performed on subclients where the storage policy is associated with a stand-alone drive. Incremental Backup o An incremental backup contains only data that is new or has changed since the last backup, regardless of the type. On average, incremental backups consume far less media and place less of a burden on resources than full backups. For the DB2, DB2 DPF, Informix, Oracle, Oracle RAC and SAP iDataAgents, Incremental backup works as described in differential backups. Incremental backups behave differently on Notes Database and Notes Document clients. Differential Backups o A differential backup contains only the data that is new or has changed since the last full backup. Like incrementals, differential backups, on average, consume less media and place less of a burden on resources than full backups. For most agents, differential backups are cumulative. This means that each differential backup contains all changes accumulated since the last full backup. Each successive differential backup contains all the changes from the previous differential backup. The Oracle iDataAgent, however, provides a backup option that allows you to specify whether or not you want the backup to be cumulative. The following illustration demonstrates the nature of differential backups. For simplicity, assume there is a file system that contains six files as represented in the figure. This file system and its modifications are the same as those described for Incremental Backups. Full Backup o Backups for any client start with a full backup. For Agents that support non-full backups, the full backup becomes a baseline to which subsequent backup types are applied. For example, if an agent supports incremental backups, a full backup must be performed before an incremental backup can be initiated. A full backup contains all the data that comprises a subclient's contents. If a client computer has multiple agents installed, then the subclients of each agent require a full backup in order to secure all of the data on that client. Backups can also be performed at the backup set or instance level, and will apply to all of the subclients within the selected backup set or instance. Backup Agents o File System iDataAgent o Virtual Server Agent (VSA VADP) o SQL Server Agent o Active Directory Server Agent o Sharepoint Server Agent o Exchange Agent o Linux Core Infrastructure o Edge networking devices which include all routers, WAN links, fiber links, Firewalls, edge switches. Does not include client switches. Network Infrastructure o Client/Server switches. WAN o Wide Area Network Definity o Brand name of our main phone switch system. Audix o Auxiliary Copies (secondary copy) o Secondary copies of data associated with data protection operations, independent of the original copy Tier 1 systems o Mission critical or high traffic servers – example: JICS, Blackboard, Feith, Exchange, Sharepoint o Also includes all Unix, Macintosh Systems Tier 2 systems o Non-mission critical production system - example Tier 3 systems o Beta and test servers. o System archives DAG - Database Availability Group o Duplicate mailbox databases on multiple exchange mailbox servers that work in an Active/Passive relationship. Voicemail system attached to the Definity phone system. Scope Backup process includes but not limited to: Systems & Servers o Virtual machines o Physical Servers o Network switches o Core infrastructure devices Databases & Speciality Applications o SQL Databases o Exchange Databases o SharePoint Databases o Active Directory Definity Phone System including Audix voicemail system. Resources Software o Commvault Simpana o SQL Management Studio o VMWare Disk o o o Tape Local Storage Magnetic Library Equallogic Storage via iSCSI o o o Dell TL4000 Tape Library with four LTO-5 Tape Drives Dell PowerVault 132 with two LTO-3 Tape Drives Audix Tape Drive Flash Media o PCMCIA Flash media o Compact Flash media Physical Storage o Fire-Proof Safe Commvault Backup Agents Determine type of Agent is to be used for new system based on system tier and type. Install agent(s) on system Commvault automatically creates default subclient based on agent type o Sys/Net team member modifies default to match naming conventions o Assign storage policy Configure agent-specific settings Assign backup type and schedule policy File System - iDataAgent Overview The File System iDataAgent is installed on all tier 1 systems that back up the entire file system excluding SQL database files and drives dedicated to Exchange. Procedure After installation, Sys/Net team member modifies default to match naming conventions File system subclient is split in to System State & file system level backups. Subclients are assigned to a storage policy under “Storage Device” > “Data Storage Policy” o DeRicci virtual machines or DeRicci physical servers are assigned to the BKUP820-Dedupe-IDA storage Policy o Sonderreger virtual machines or Sonderreger physical servers are assigned to the BKUP821Dedupe-IDA storage Policy Deduplication is to be enabled in the Subclient properties under “Storage Device” > “Deduplication” o Deduplication is to be set to “On Media Agent” Subclients are assigned to scheduled policy based on traffic and number of DataAgents installed. Schedules Both FileSystem & System State sub clients are assigned to a schedule policy o Systems with front and backend database servers Front end system is assigned to “Incremental Forever 10pm” schedule policy Database backend system is assigned to “Critical 3am” schedule policy o o Single System servers, including Macintosh servers, are assigned to “Incremental Forever 10pm” schedule policy Unix file systems should be set to do a full backup daily at midnight. Exchange Database Agent Overview Exchange database agent is used to backup Exchange mailbox databases. The Exchange database agent process is different on the 2007 & 2010 Exchange environments. On the Exchange 2007 environment the agent is installed on the server that houses the mailbox databases. Whereas on the Exchange 2010 environment has the agent installed on each mailbox server, however the sub clients are attached to a DAG. Procedure Based on the duration/performance of the backup, multiple sub clients will be created to make best effort to keep exchange backups inside a 22 hour backup window. Sub clients are assigned to a storage policy under “Storage Device” > “Data Storage Policy” o DeRicci virtual machines or DeRicci physical servers are assigned to the BKUP820-Dedupe-IDA storage Policy o Sonderreger virtual machines or Sonderreger physical servers are assigned to the BKUP821Dedupe-IDA storage Policy Deduplication is to be enabled in the Sub client properties under “Storage Device” > “Deduplication” o Deduplication is to be set to “On Media Agent” Schedules Sub clients per server are split in to two halves and scheduled in 30 minute increments. o First set of sub clients are scheduled between 6pm – 8pm o Second set of sub clients are scheduled between 8pm – 10pm. SQL iDataAgent Overview Installed on Tier 1 database servers where block-level SQL backups are needed for data retention/restoration. Procedure Under the sub client properties on the Content section, discover and specify which databases should get backed up. Generally all databases should get backed up. Sub clients are assigned to a storage policy under “Storage Device” > “Data Storage Policy” and storage policy settings are duplicated under “Storage Device” > “Log Storage Policy” o DeRicci virtual machines or DeRicci physical servers are assigned to the BKUP820-Dedupe-IDA storage Policy o Sonderreger virtual machines or Sonderreger physical servers are assigned to the BKUP821Dedupe-IDA storage Policy Deduplication is to be enabled in the Sub client properties under “Storage Device” > “Deduplication” o Deduplication is to be set to “On Media Agent” Schedules SQL Database sub clients get assigned to the 10pm Database Backup schedule. Active Directory iDataAgent Overview Active Directory iDataAgent is installed on the primary domain controller. Procedure Sub clients are assigned to a storage policy under “Storage Device” > “Data Storage Policy” o DeRicci virtual machines or DeRicci physical servers are assigned to the BKUP820-Dedupe-IDA storage Policy o Sonderreger virtual machines or Sonderreger physical servers are assigned to the BKUP821Dedupe-IDA storage Policy Deduplication is to be enabled in the Sub client properties under “Storage Device” > “Deduplication” o Deduplication is to be set to “On Media Agent” Schedules Should be assigned to the “Incrementals forever 1AM” schedule policy. Sharepoint Agent Overview Microsoft SharePoint Server and Windows SharePoint Services include components that are backed up by the SharePoint Server iDataAgent, as well as data which must be backed up using the File System iDataAgent. SharePoint database files can also reside on separate SQL servers; to secure this data, you must back up these files using the appropriate SQL Server iDataAgent. This agent requires that the database server has the SQL iDataAgent installed. Procedure Documents o Under the sub client properties on the Content section, discover and specify which databases should get backed up. Generally all databases should get backed up. o Sub clients are assigned to a storage policy under “Storage Device” > “Data Storage Policy” DeRicci virtual machines or DeRicci physical servers are assigned to the BKUP820Dedupe-IDA storage Policy Sonderreger virtual machines or Sonderreger physical servers are assigned to the BKUP821-Dedupe-IDA storage Policy o Deduplication is to be enabled in the Sub client properties under “Storage Device” > “Deduplication” Deduplication is to be set to “On Media Agent” Databases o In the sub client properties check use direct database access o o Sub clients are assigned to a storage policy under “Storage Device” > “Data Storage Policy” DeRicci virtual machines or DeRicci physical servers are assigned to the BKUP820Dedupe-IDA storage Policy Sonderreger virtual machines or Sonderreger physical servers are assigned to the BKUP821-Dedupe-IDA storage Policy Deduplication is to be enabled in the Sub client properties under “Storage Device” > “Deduplication” Deduplication is to be set to “On Media Agent” Schedules Sharepoint Document subclient is to be assigned to “Incrementals forever 8pm” Sharepoint Database subclient is to be assigned to the “10pm Database Backup” schedule Beta/Test/Development Environments Overview Development/Test/Beta environments of production servers are created upon request. They are created using VMWare through the vSphere client cloning process. Procedure Request is made to the Systems and Network team to create a test environment. A Systems and Network team member initiates clone of the Virtual Machine(s). o Tier 1 systems must be cloned during non-standard work hours. Clones are stored in the Sonderegger Beta Pool. Unless specifically requested, additional backups are not done for Test/Dev/Beta systems. Decommissioning Systems Overview This section will describe the backup process when a system is to be decommissioned. Procedure System is evaluated to see if archive backup is needed. If archive is needed, then through Commvault a subclient is created for archive backups which would be assigned to the Archive Storage Policy. (see FileSystem iDataAgent) Manual full backup of the systems archive sub client is initiated. When backup is complete, tape is exported from the tape library and stored in the Firesafe. Other sub clients on decommissioned server are removed from all scheduled policies. After 35 days have passed since the archive backup, then regular backup tapes may be manually cleared as long as that sub client is the only active data on the tape media. Test and Restore of Backups Overview This section will describe the testing and restore process for systems requiring disaster recovery testing. Procedure Jenzabar o The Jenzabar system will best test restored quarterly or more often as testing is required. The restore will be tested by using a production backup and restore on the play database. The schedule with the quarterly review will be Jan, Apr, Jul, Oct unless a module manager requests access to the play database. The quarterly review schedule will be continued at the next interval following the testing in the play database. Backup Monitoring Process Overview This section will describe the monitoring process for Commvault backups. Procedure Systems and Network team member checks the Simpana Commcell regularly on a daily basis to ensure that backups are meeting the backup window. Backup reports are stored on the Operations website on a weekly basis. Appropriate actions are taken to remedy when the backup report indicates issues or backups are not being completed in the backup window. SQL Server Maintenance Plans Overview This section will describe the standard process for SQL database backup on the database servers. Procedure SystemDB Backup 1. 2. Check Database Integrity a. Connection: Local Server Connection b. Database(s): All System Databases c. Include Indexes: Yes History Cleanup a. Connection: Local Server Connection b. Backup and Restore History: Yes c. SQL Server Agent Job History: Yes 3. 4. 5. d. Maintenance Plan History: Yes e. Remove Historical Data Older Than: 1 Weeks Maintenance Cleanup a. Connection: Local Server Connection b. Delete Files of the Following Type: Backup Files c. File Location: Search Folder and Delete Files Base on Extension (create folder if doesn’t exist) i. Folder: Drive:\DBBackups\SystemDBs ii. File Extension: bak iii. Include First-Level Subfolders: Yes d. File Age i. Delete Files based on the Age of the File at Task Run Time ii. Delete Files Older than the Following: 1 Week Backup Database a. Connection: Local Server Connection b. Backup Type: Full c. Database(s): All System Databases d. Create Backup File for Every Database: Yes e. Backup to: Disk f. Create Sub Directory for Each Database: Yes g. Folder: Drive:\DBBackups\SystemDBs h. Compression: Default Rebuild Index a. Connection: Local Server Connection b. Database(s): All System Databases c. Reorganize Pages With Default Amount of Free Space d. Sort Results in Temp DB: No e. Keep Index Online While Reindexing: No UserDB Backup 1. 2. Maintenance Cleanup a. Connection: Local Server Connection b. Delete Files of the Following Type: Backup Files c. File Location: Search Folder and Delete Files Base on Extension (create folder if doesn’t exist) i. Folder: Drive:\DBBackups\UserDBs ii. File Extension: bak iii. Include First-Level Subfolders: Yes d. File Age i. Delete Files based on the Age of the File at Task Run Time ii. Delete Files Older than the Following: 2 Days Backup Database a. Connection: Local Server Connection b. Backup Type: Full c. Database(s): All User Databases d. Create Backup File for Every Database: Yes e. Backup to: Disk f. Create Sub Directory for Each Database: Yes g. h. Folder: Drive:\DBBackups\UserDBs Compression: Default Notify Operator All Numbered Subtasks Above should have a Notify Operator Task “on Error” to send email alerts to ITSOServerAdmin@Edgewood.edu Maintenance Schedule All Daily Backups are scheduled between 12:00am-5:00am. Weekly Backups are scheduled for Sunday between 12:00am-5:00am. Exceptions Blackboard & JICS are setup using vendor specified best practices. o One difference on these is that they are setup to do transactional log backups regularly. Core and Network Infrastructure Backups Overview Configuration backups of all edge networking devices which include all routers, WAN links, fiber links, Firewalls, edge switches including client switches. Procedure Core Infrastructure o Configurations are saved to flash media or tftp site before and after any configuration modifications. o Configuration files are saved to flash media on a weekly basis for Cisco 6509s. Media is then storage in the fire safe. Client Networking Infrastructure o Configurations are saved to our tftp site before and after any configuration modifications. Definity and Audix backups (Phone/Voicemail) Overview This section describes the backup procedures for the Definity phone system and Audix voicemail backups. Procedure Definity phone System o Configurations are automatically saved to flash media on a nightly basis. o Flash media is rotated on a weekly basis and stored in the firesafe. Data is retained for two weeks. Audix Voicemail o Configurations are automatically saved to tape media on a nightly basis. o Tape media is rotated on a weekly basis and stored in the fire safe. Data is retained for one month. Revision Table Date 05/16/2013 05/17/2013 05/17/2013 Revised By SysNet Team SysNet Team Patrick Gumieny