Kimberry _______ Associates www.kimberry.co.uk SVR329 Active Directory Disaster Recovery Part 1 of 2 John Craddock Principal Systems Consultant v-jcradd@microsoft.com johncra@kimberry.co.uk Sally Storey Senior Consultant sallysto@kimberry.co.uk Get it Right by Design Kimberry _______ Associates www.kimberry.co.uk Planned response to failure prevents an event turning into a DISASTER 3 Be Prepared Kimberry _______ Associates www.kimberry.co.uk How many of you proactively monitor your AD infrastructure? How many of you have a disaster recovery plan? 4 _______ Part 1 and Part 2 Session TopicsKimberry Associates www.kimberry.co.uk Infrastructure Components File Replication and SYSVOL Backing up the Directory Restoring the Directory Authoritative Restores Recovering a Forest 5 Legal Stuff Kimberry _______ Associates www.kimberry.co.uk Every effort has been made to make this seminar as complete and as accurate as possible but no warranty or fitness is implied. The presenters, authors, publisher and distributor assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. Names identifying the directory and associated objects are fictitious and are not intended to represent any organizations or people All trademarks are acknowledged and are the property of their respective owners © All materials are copyright Kimberry Associates 6 What Is an AD Disaster? Kimberry _______ Associates www.kimberry.co.uk Loss of business functionality Loss of services Loss of system(s) Delays in deploying the system Causes Hardware failure Software failure Operator error / malicious attack / virus 7 It’s the Database and More… Kimberry _______ Associates www.kimberry.co.uk Your AD infrastructure is pivotal Redundancy provides fault tolerance Designed and implemented procedures Minimise risk Minimise downtime Maximizing availability and reliability 8 Design, Plan and Test Kimberry _______ Associates www.kimberry.co.uk Don’t just assume that you will be able to recover a system when it fails Always design, plan and test your disaster recovery procedures Before they are needed Regularly test the integrity of backups 9 Kimberry _______ Associates www.kimberry.co.uk We’ll Start by Looking at the Infrastructure Components From Acorn to Forest Kimberry _______ Associates www.kimberry.co.uk Domains Operator errors Hardware failures Trusts Multi master operations Single master operations SYSVOL Backup and Restore DFS Replication topology DNS Global catalog File Replication Service Group policy AD Services Database Administration 11 Malicious users Virus attacks Authentication Time synchronization Multiple Domains Kimberry _______ Associates www.kimberry.co.uk Added complexity for all existing services, plus GC placement Trusts Cross-domain references 12 Know Your Forest is Healthy Kimberry _______ Associates www.kimberry.co.uk Systems must be monitored For large systems consider using Microsoft Operations Manager (MOM) or third Party equivalents If you are not going to deploy a monitoring solution, use the available tools Regularly check the event logs Dcdiag, netdiag, nltest, nslookup, repadmin, replmon and more… 13 Replication Model Kimberry _______ Associates www.kimberry.co.uk The replication model is described as multimaster, loose consistency with convergence Multimaster Changes can be made at any DC Loose consistency There is a latency between changes being made and their availability throughout the enterprise Convergence Eventually the changes will propagate to all DCs and conflicts will have to be detected and resolved 14 Single Replication Masters Kimberry _______ Associates www.kimberry.co.uk Certain operations are critical and must be handled by a single master There are 5 types of FSMO roles Forest-wide Schema Domain Naming For each domain PDC Emulator RID Master Infrastructure Master 15 Secondary FSMO Role Holders Kimberry _______ Associates www.kimberry.co.uk As part of your DR plan select potential candidate DCs that can take the role of the FSMO roles If both servers are online, the roles can be transferred through the UI If a FSMO role holder is offline, the role can be seized using ntdsutil 16 Special Considerations for the RID Master Kimberry _______ Associates www.kimberry.co.uk When seizing the RID master or restoring a RID master take special care The next RID pool to be issued by the master must be higher than any previously allocated pools If necessary elevate the pool 17 RID Master Kimberry _______ Associates www.kimberry.co.uk Distributes pools of RIDS Controls cross-domain moves If running in mixed-mode, place the RID master and the PDC emulator on the same DC PDC could be a high consumer of RIDs The RidAvailablePool attribute of the cn=RID Manager$,cn=system,dc=… object holds the start value of the next RID pool 64-bit number, next RID held in lower 32-bits 18 Seizing Roles Kimberry _______ Associates www.kimberry.co.uk Before seizing any roles, establish the reason for the loss of the role holder Assess the short term impact of FSMO loss before seizing a role The schema, RID and domain naming master FSMOs must never be brought back online once their roles have been seized 19 _______ So What Happened to the RIDs?Kimberry Associates www.kimberry.co.uk 2603 2602 2103 500 groups Request new pool 2102 1602 500 groups 500 groups 1103 Further group creation fails Network Request new pool down 1603 RID Master Next pool starts 2103 2603 3103 Next pool starts 2103 Replicates 20 Seize the Role (Oh No!) Kimberry _______ Associates www.kimberry.co.uk 2603 2602 2103 Duplicate SIDs 2103 More groups 2102 1602 500 groups Network down 1603 RID Master RID Master Next pool starts 3103 21 Next pool starts 2103 2603 Forced Demotion Kimberry _______ Associates www.kimberry.co.uk Retired FSMO role holders that cannot be reconnected to the network can be forcibly demoted Use dcpromo /forceremoval They can then be reconnected to the network Use dcpromo to re-establish them as domain controllers 22 Kimberry _______ Associates www.kimberry.co.uk Domain Controllers Depend on the File Replication Service and SYSVOL SYSVOL Kimberry _______ Associates www.kimberry.co.uk SYSVOL holds important domain-wide information Group Policy Scripts SYSVOL must be available and consistent The contents is replicated by the File Replication Services (FRS) If SYSVOL fails the domain controller fails Monitor the health of the FRS 24 Ultrasound Kimberry _______ Associates www.kimberry.co.uk Installs WMI providers on replica members Information gathered by Ultrasound controller Stored in database for analysis Requires SQL server or MSDE 25 Don’t Help NTFRS!! Kimberry _______ Associates www.kimberry.co.uk If files or folders are not replicating between instances of SYSVOL Don’t manually copy them across A file or folder copied into SYSVOL will be seen as new and will be stamped with a unique GUID 26 Folder Name Collisions Kimberry _______ Associates www.kimberry.co.uk If a folder name collision occurs Deletion could potentially result in the loss of a significant number of files and subfolders The situation is resolved by giving the last writer a non-conflicting name The name is said to be morphed: FolderName_NTFRS_207480c0folde The loser keeps the original name The administrator must intervene to resolve the names 27 The Morphed Results Kimberry _______ Associates www.kimberry.co.uk G1, G2, G3 – represent GUIDs Scripts…xxx – represent morphed names G1 Scripts G2 Scripts…bbb G3 Scripts…zzz 28 G2 Scripts G1 Scripts…xxx G3 Scripts…ccc G3 Scripts G2 Scripts…aaa G1 Scripts…yyy Recovering from Morphing Kimberry _______ Associates Rename folders on one server www.kimberry.co.uk G1 Scripts1 G2 Scripts2 G3 Scripts3 G2 G3 Scripts2 Scripts3 G2 G1 Scripts1 Scripts2 G1 G3 Scripts3 Scripts1 After rename has fully replicated, rename appropriately! 29 Non-Authoritative Restore of SYSVOL Kimberry _______ Associates www.kimberry.co.uk If a SYSVOL replica gets corrupted it can be restored using a non-authoritative restore Setting the BurFlags in the registry to D2 (hex) triggers a non-authoritative restore when the FRS service starts HKEY_LOCAL_MACHINE\SYSTEM\ CurrentControlSet\Services\NtFrs\Parameters\ Backup/Restore\Process at Startup 30 Non-Authoritative Restore Kimberry _______ Associates www.kimberry.co.uk SYSVOL File Structure 1 Re-initialize FRS database DO_NOT_REMOVE_NfFrs_Preinstall_Directory Policies Replica Set Scripts 2 31 files and folders Creates: NtFrs_PreExisting___See_EventLog Non-Authoritative Restore (cont’d) 1 Kimberry _______ Associates www.kimberry.co.uk Re-initialize FRS database DO_NOT_REMOVE_NfFrs_Preinstall_Directory 3 Replica Set 2 NtFrs_PreExisting___See_EventLog Policies Scripts 32 files and folders Non-Authoritative Restore (cont’d) Upstream Partner Associates www.kimberry.co.uk Non-authoritative Restore VVJoin Full sync join to upstream partner File GUIDs Creates change orders from IDTable MD5 Checksums Request non matching files Moves local file from PreExisting to replica tree 33 Kimberry _______ If files found in PreExisting calculate MD5s If MD5s don’t match If MD5s match Authoritative Restore Kimberry _______ Associates www.kimberry.co.uk If a replica set becomes corrupted it may be necessary to perform an authoritative restore Shutdown FRS on ALL servers Restore the files to one server Set the BurFlags to D4 on the server Restart the NTFRS service on this server only This will reinitialise the NTFRS database and apply new GUIDS to all the files and folders 34 Authoritative Restore (cont’d) Kimberry _______ Associates www.kimberry.co.uk Once the D4 server has successfully reinitialised Check for an 13516 event log message On a downstream server Set the BurFlags to D2 and start the service When this server has reinitialised, continue with D2 on the next and so on As you restore take replication topology into account A D2 join should only be performed with an initialised partner 35 Kimberry _______ Associates www.kimberry.co.uk Backing Up the Directory System State Kimberry _______ Associates www.kimberry.co.uk It’s not just the database that needs to be backed-up The boot files The COM+ class registration database The registry Also copied to %SystemRoot%\Repair\RegBack The system volume (SYSVOL) Certificate Server (if installed) 37 Backup Solutions Kimberry _______ Associates www.kimberry.co.uk Don’t use image backups Will result in USN rollback Pre SP1 this is very difficult to troubleshoot You must use Ntbackup or an equivalent thirdparty product AD backup/restore programs mark the restored database as a new replication source 38 What Should you Backup? Kimberry _______ Associates www.kimberry.co.uk A backup to recover the AD should include System State System Disk SYSVOL folders and files Separate backup simplifies the reinstall of group policies and scripts Consider backing up GPOs with the Group Policy Management Console (GPMC) 39 Windows Server 2003 Application Directory Partitions Kimberry _______ Associates www.kimberry.co.uk For each application directory partition, you must backup at least one of the servers that hosts the partition Failure to do this will result in the loss of the partition data if all the host systems fail 40 Backup Considerations Kimberry _______ Associates www.kimberry.co.uk Backups older than the tombstone lifetime setting cannot be restored Restored DCs could hold deleted objects Resulting in lingering objects Backups can only be restored to the DC on which they were made 41 Data Corruption Kimberry _______ Associates www.kimberry.co.uk React quickly if an event occurs that deletes or corrupts directory objects Can you stop replication of the changes propagating from the site? This may save you a lot of work particularly if the user, computer and group objects are involved 42 Stopping Replication Kimberry _______ Associates www.kimberry.co.uk Unplug the servers from the network! Stop the KCC and delete connection objects Make sure they are documented if they are manually configured The KCC is disabled via the options attribute of the NTDS Site settings object Intrasite KCC: Bit 0 = 0 ON, Bit 0=1 Off Intersite KCC: Bit 4 = 0 ON, Bit 4=1 Off 43 Special Site Kimberry _______ Associates www.kimberry.co.uk Could set up a DC to only receive inbound replication for AD and DFS replica sets Schedule at night or trigger manually Provides a window for recovery 44 DNS Requirements Kimberry _______ Associates www.kimberry.co.uk Only allow the DC to register its DSA CNAME record which is required for replication Windows 2003, block registrations via group policy Windows 2000, block registrations via the registry Do not allow WINS records to be registered 45 Please Come Back for Part 2 Kimberry _______ Associates www.kimberry.co.uk Restoring the Directory Authoritative Restores Recovering a Forest 46 And There is More… Kimberry _______ Associates www.kimberry.co.uk Order on the web www.kimberry.co.uk Discount code KB1764 (15% discount) 47 Kimberry _______ Associates www.kimberry.co.uk Thanks for coming to the seminar Hope to see you in Part 2 48 Resources Kimberry _______ Associates www.kimberry.co.uk Technical Chats and Webcasts http://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asp Microsoft Learning and Certification http://www.microsoft.com/learning/default.mspx MSDN & TechNet http://microsoft.com/msdn http://microsoft.com/technet Virtual Labs http://www.microsoft.com/technet/traincert/virtuallab/rms.mspx Newsgroups http://communities2.microsoft.com/ communities/newsgroups/en-us/default.aspx Technical Community Sites http://www.microsoft.com/communities/default.mspx User Groups 49 http://www.microsoft.com/communities/usergroups/default.mspx Live from Tech·Ed Webcast Series has Been Brought to You by: www.microsoft.com/hpc Fill out a session evaluation on CommNet and Win an XBOX 360! © 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.