Kimberry
_______
Associates
www.kimberry.co.uk
SVR329
Active Directory
Disaster
Recovery Part 1 of 2
John Craddock
Principal Systems Consultant
v-jcradd@microsoft.com
johncra@kimberry.co.uk
Sally Storey
Senior Consultant
sallysto@kimberry.co.uk
Get it Right by Design
Kimberry
_______
Associates
www.kimberry.co.uk
Planned response to failure
prevents an event turning into a
DISASTER
3
Be Prepared
Kimberry
_______
Associates
www.kimberry.co.uk
How many of you proactively monitor your AD
infrastructure?
How many of you have a disaster recovery plan?
4
_______
Part 1 and Part 2 Session TopicsKimberry
Associates
www.kimberry.co.uk
Infrastructure Components
File Replication and SYSVOL
Backing up the Directory
Restoring the Directory
Authoritative Restores
Recovering a Forest
5
Legal Stuff
Kimberry
_______
Associates
www.kimberry.co.uk
Every effort has been made to make this seminar as complete
and as accurate as possible but no warranty or fitness is implied.
The presenters, authors, publisher and distributor assume no
responsibility for errors or omissions, or for damages resulting
from the use of the information contained herein.
Names identifying the directory and associated objects are fictitious
and are not intended to represent any organizations or people
All trademarks are acknowledged and are the property of their
respective owners
© All materials are copyright Kimberry Associates
6
What Is an AD Disaster?
Kimberry
_______
Associates
www.kimberry.co.uk
Loss of business functionality
Loss of services
Loss of system(s)
Delays in deploying the system
Causes
Hardware failure
Software failure
Operator error / malicious attack / virus
7
It’s the Database and More…
Kimberry
_______
Associates
www.kimberry.co.uk
Your AD infrastructure is pivotal
Redundancy provides fault tolerance
Designed and implemented procedures
Minimise risk
Minimise downtime
Maximizing availability and reliability
8
Design, Plan and Test
Kimberry
_______
Associates
www.kimberry.co.uk
Don’t just assume that you will be able to recover
a system when it fails
Always design, plan and test your disaster
recovery procedures
Before they are needed
Regularly test the integrity of backups
9
Kimberry
_______
Associates
www.kimberry.co.uk
We’ll Start by Looking at
the Infrastructure
Components
From Acorn to Forest
Kimberry
_______
Associates
www.kimberry.co.uk
Domains
Operator errors
Hardware
failures
Trusts
Multi master
operations
Single master
operations
SYSVOL
Backup and Restore
DFS
Replication
topology
DNS
Global catalog
File Replication Service
Group policy
AD Services
Database
Administration
11
Malicious users
Virus attacks
Authentication
Time synchronization
Multiple Domains
Kimberry
_______
Associates
www.kimberry.co.uk
Added complexity for all existing services, plus
GC placement
Trusts
Cross-domain references
12
Know Your Forest is Healthy
Kimberry
_______
Associates
www.kimberry.co.uk
Systems must be monitored
For large systems consider using Microsoft Operations
Manager (MOM) or third Party equivalents
If you are not going to deploy a monitoring
solution, use the available tools
Regularly check the event logs
Dcdiag, netdiag, nltest, nslookup, repadmin, replmon
and more…
13
Replication Model
Kimberry
_______
Associates
www.kimberry.co.uk
The replication model is described as
multimaster, loose consistency with convergence
Multimaster
Changes can be made at any DC
Loose consistency
There is a latency between changes being made and their
availability throughout the enterprise
Convergence
Eventually the changes will propagate to all DCs and
conflicts will have to be detected and resolved
14
Single Replication Masters
Kimberry
_______
Associates
www.kimberry.co.uk
Certain operations are critical and must be
handled by a single master
There are 5 types of FSMO roles
Forest-wide
Schema
Domain Naming
For each domain
PDC Emulator
RID Master
Infrastructure Master
15
Secondary FSMO Role Holders
Kimberry
_______
Associates
www.kimberry.co.uk
As part of your DR plan select potential
candidate DCs that can take the role of the
FSMO roles
If both servers are online, the roles can be
transferred through the UI
If a FSMO role holder is offline, the role can be
seized using ntdsutil
16
Special Considerations for the
RID Master
Kimberry
_______
Associates
www.kimberry.co.uk
When seizing the RID master or restoring a RID
master take special care
The next RID pool to be issued by the master
must be higher than any previously allocated
pools
If necessary elevate the pool
17
RID Master
Kimberry
_______
Associates
www.kimberry.co.uk
Distributes pools of RIDS
Controls cross-domain moves
If running in mixed-mode, place the RID master
and the PDC emulator on the same DC
PDC could be a high consumer of RIDs
The RidAvailablePool attribute of the
cn=RID Manager$,cn=system,dc=… object holds
the start value of the next RID pool
64-bit number, next RID held in lower 32-bits
18
Seizing Roles
Kimberry
_______
Associates
www.kimberry.co.uk
Before seizing any roles, establish the reason for
the loss of the role holder
Assess the short term impact of FSMO loss
before seizing a role
The schema, RID and domain naming master FSMOs
must never be brought back online once their roles
have been seized
19
_______
So What Happened to the RIDs?Kimberry
Associates
www.kimberry.co.uk
2603
2602
2103
500
groups
Request new pool
2102
1602
500
groups
500
groups
1103
Further
group
creation
fails
Network
Request new pool
down
1603
RID Master
Next pool starts 2103 2603 3103
Next pool starts 2103
Replicates
20
Seize the Role (Oh No!)
Kimberry
_______
Associates
www.kimberry.co.uk
2603
2602
2103
Duplicate SIDs
2103
More
groups
2102
1602
500
groups
Network
down
1603
RID Master
RID Master
Next pool starts 3103
21
Next pool starts 2103 2603
Forced Demotion
Kimberry
_______
Associates
www.kimberry.co.uk
Retired FSMO role holders that cannot be
reconnected to the network can be forcibly
demoted
Use dcpromo /forceremoval
They can then be reconnected to the network
Use dcpromo to re-establish them as domain
controllers
22
Kimberry
_______
Associates
www.kimberry.co.uk
Domain Controllers Depend on
the File Replication Service
and SYSVOL
SYSVOL
Kimberry
_______
Associates
www.kimberry.co.uk
SYSVOL holds important domain-wide
information
Group Policy
Scripts
SYSVOL must be available and consistent
The contents is replicated by the File Replication
Services (FRS)
If SYSVOL fails the domain controller fails
Monitor the health of the FRS
24
Ultrasound
Kimberry
_______
Associates
www.kimberry.co.uk
Installs WMI providers on replica members
Information gathered by Ultrasound controller
Stored in database for analysis
Requires SQL server or MSDE
25
Don’t Help NTFRS!!
Kimberry
_______
Associates
www.kimberry.co.uk
If files or folders are not replicating between
instances of SYSVOL
Don’t manually copy them across
A file or folder copied into SYSVOL will be seen
as new and will be stamped with a unique GUID
26
Folder Name Collisions
Kimberry
_______
Associates
www.kimberry.co.uk
If a folder name collision occurs
Deletion could potentially result in the loss of a
significant number of files and subfolders
The situation is resolved by giving the last writer
a non-conflicting name
The name is said to be morphed:
FolderName_NTFRS_207480c0folde
The loser keeps the original name
The administrator must intervene to resolve the names
27
The Morphed Results
Kimberry
_______
Associates
www.kimberry.co.uk
G1, G2, G3 – represent GUIDs
Scripts…xxx – represent morphed names
G1
Scripts
G2
Scripts…bbb
G3
Scripts…zzz
28
G2
Scripts
G1
Scripts…xxx
G3
Scripts…ccc
G3
Scripts
G2
Scripts…aaa
G1
Scripts…yyy
Recovering from Morphing
Kimberry
_______
Associates
Rename folders on one server
www.kimberry.co.uk
G1
Scripts1
G2
Scripts2
G3
Scripts3
G2
G3
Scripts2
Scripts3
G2
G1
Scripts1
Scripts2
G1
G3
Scripts3
Scripts1
After rename has fully replicated, rename appropriately!
29
Non-Authoritative Restore
of SYSVOL
Kimberry
_______
Associates
www.kimberry.co.uk
If a SYSVOL replica gets corrupted it can be
restored using a non-authoritative restore
Setting the BurFlags in the registry to D2 (hex) triggers
a non-authoritative restore when the FRS service
starts
HKEY_LOCAL_MACHINE\SYSTEM\
CurrentControlSet\Services\NtFrs\Parameters\
Backup/Restore\Process at Startup
30
Non-Authoritative Restore
Kimberry
_______
Associates
www.kimberry.co.uk
SYSVOL File Structure
1
Re-initialize FRS database
DO_NOT_REMOVE_NfFrs_Preinstall_Directory
Policies
Replica Set
Scripts
2
31
files and folders
Creates: NtFrs_PreExisting___See_EventLog
Non-Authoritative Restore
(cont’d)
1
Kimberry
_______
Associates
www.kimberry.co.uk
Re-initialize FRS database
DO_NOT_REMOVE_NfFrs_Preinstall_Directory
3
Replica Set
2
NtFrs_PreExisting___See_EventLog
Policies
Scripts
32
files and folders
Non-Authoritative Restore
(cont’d)
Upstream Partner
Associates
www.kimberry.co.uk
Non-authoritative
Restore
VVJoin
Full sync join to
upstream partner
File GUIDs
Creates change
orders from
IDTable
MD5 Checksums
Request non
matching files
Moves local file
from PreExisting
to replica tree
33
Kimberry
_______
If files found in
PreExisting
calculate MD5s
If MD5s don’t match
If MD5s match
Authoritative Restore
Kimberry
_______
Associates
www.kimberry.co.uk
If a replica set becomes corrupted it may be
necessary to perform an authoritative restore
Shutdown FRS on ALL servers
Restore the files to one server
Set the BurFlags to D4 on the server
Restart the NTFRS service on this server only
This will reinitialise the NTFRS database and apply new
GUIDS to all the files and folders
34
Authoritative Restore
(cont’d)
Kimberry
_______
Associates
www.kimberry.co.uk
Once the D4 server has successfully reinitialised
Check for an 13516 event log message
On a downstream server
Set the BurFlags to D2 and start the service
When this server has reinitialised, continue with
D2 on the next and so on
As you restore take replication topology into account
A D2 join should only be performed with an initialised partner
35
Kimberry
_______
Associates
www.kimberry.co.uk
Backing Up the Directory
System State
Kimberry
_______
Associates
www.kimberry.co.uk
It’s not just the database that needs to be
backed-up
The boot files
The COM+ class registration database
The registry
Also copied to %SystemRoot%\Repair\RegBack
The system volume (SYSVOL)
Certificate Server (if installed)
37
Backup Solutions
Kimberry
_______
Associates
www.kimberry.co.uk
Don’t use image backups
Will result in USN rollback
Pre SP1 this is very difficult to troubleshoot
You must use Ntbackup or an equivalent thirdparty product
AD backup/restore programs mark the restored
database as a new replication source
38
What Should you Backup?
Kimberry
_______
Associates
www.kimberry.co.uk
A backup to recover the AD should include
System State
System Disk
SYSVOL folders and files
Separate backup simplifies the reinstall of group policies
and scripts
Consider backing up GPOs with the Group Policy
Management Console (GPMC)
39
Windows Server 2003
Application Directory Partitions
Kimberry
_______
Associates
www.kimberry.co.uk
For each application directory partition, you must
backup at least one of the servers that hosts the
partition
Failure to do this will result in the loss of the partition
data if all the host systems fail
40
Backup Considerations
Kimberry
_______
Associates
www.kimberry.co.uk
Backups older than the tombstone lifetime
setting cannot be restored
Restored DCs could hold deleted objects
Resulting in lingering objects
Backups can only be restored to the DC on
which they were made
41
Data Corruption
Kimberry
_______
Associates
www.kimberry.co.uk
React quickly if an event occurs that deletes or
corrupts directory objects
Can you stop replication of the changes
propagating from the site?
This may save you a lot of work particularly if the user,
computer and group objects are involved
42
Stopping Replication
Kimberry
_______
Associates
www.kimberry.co.uk
Unplug the servers from the network!
Stop the KCC and delete connection objects
Make sure they are documented if they are manually
configured
The KCC is disabled via the options attribute of
the NTDS Site settings object
Intrasite KCC: Bit 0 = 0 ON, Bit 0=1 Off
Intersite KCC: Bit 4 = 0 ON, Bit 4=1 Off
43
Special Site
Kimberry
_______
Associates
www.kimberry.co.uk
Could set up a DC to only receive inbound
replication for AD and DFS replica sets
Schedule at night or trigger manually
Provides a window for recovery
44
DNS Requirements
Kimberry
_______
Associates
www.kimberry.co.uk
Only allow the DC to register its DSA CNAME
record which is required for replication
Windows 2003, block registrations via group policy
Windows 2000, block registrations via the registry
Do not allow WINS records to be registered
45
Please Come Back for Part 2
Kimberry
_______
Associates
www.kimberry.co.uk
Restoring the Directory
Authoritative Restores
Recovering a Forest
46
And There is More…
Kimberry
_______
Associates
www.kimberry.co.uk
Order on the web www.kimberry.co.uk
Discount code KB1764 (15% discount)
47
Kimberry
_______
Associates
www.kimberry.co.uk
Thanks for coming to the seminar
Hope to see you in Part 2
48
Resources
Kimberry
_______
Associates
www.kimberry.co.uk
Technical Chats and Webcasts
http://www.microsoft.com/communities/chats/default.mspx
http://www.microsoft.com/usa/webcasts/default.asp
Microsoft Learning and Certification
http://www.microsoft.com/learning/default.mspx
MSDN & TechNet
http://microsoft.com/msdn
http://microsoft.com/technet
Virtual Labs
http://www.microsoft.com/technet/traincert/virtuallab/rms.mspx
Newsgroups
http://communities2.microsoft.com/
communities/newsgroups/en-us/default.aspx
Technical Community Sites
http://www.microsoft.com/communities/default.mspx
User Groups
49
http://www.microsoft.com/communities/usergroups/default.mspx
Live from Tech·Ed Webcast
Series has Been
Brought to You by:
www.microsoft.com/hpc
Fill out a session
evaluation on
CommNet and
Win an XBOX 360!
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not
be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.