Maybe that’s funny but I would strongly recommend using basic Windows tools first, to see if there is no issue with networking on a server itself.
to see network settings on a server. Just simply review IP address, network mask, default gateway and DNS servers list to be sure that nothing has changed there. This is really useful in case that you are using DHCP with reservation for a server. When you see in IP address section 169.254.x.y with network mask 255.255.0.0 (APIPA address) you may be sure that there is no connection to DHCP server from your machine. In other case when settings are ok, you should check another command
another step checks if network card is not broken and if network communication is working on a server. For the first check, you should ping loopback interface (127.0.0.1) address to see if it replies. If so, then you can be sure that NIC is not broken, in other case that means you have a network card issue
When you have done the step above you need to check if server works properly on layer 3 and try to ping its real IP address or another machine in the same subnet
after that check if there is communication with default gateway to be sure that another subnets are reachable
the last check should be performed to remote host in another subnet. You will see if there is no problem with routing between current network and if communication goes out from the subnet.
at this stage that’s all you can check using ping command.
this tool allows you to see how network communication is transmitted to another subnet. You can see how many routers (hops) are in the path and you can evaluate if transmission goes over proper path. Tracert shows you also time delay on each hop which allows for detecting lags for connection. When specified point is not reachable, you will see at which step it is not working or where the communication has the longest delay.
tracert or tracert -d <DestinationIPAddress>
as you can see there are 2 variants of this command. I would suggest to use the second option with -d switch. When your run tracert with -d then reverse DNS resolution is skipped and command is executed a little bit faster.
this command is a summary of ping and tracert. Its output shows you response time of each hop in the routing path.
after verification if there is network communication, you should also check if all required ports are opened and your server is listening on them
netstat -an -p tcp
and you will get all ports on which server is listening.
The whole list with required ports to be opened for Active Directory communication over firewall, you can find on Microsoft Technet site.
this tool is similar to netstat but the main difference is that portqry checks specified port or ports range if they are opened. Netstat shows only ports which are already opened and server listens on them.To use that, you need to download it from Microsoft: Portqry Download
OK, now you have an overview about networking part. If there are no errors, you should investigate somewhere else. The next place to check if something is going wrong is Microsoft Event Viewer
- Event Viewer
please review logs regularly to avoid any server issue. I know that is a lot of work but it would prevent your server to crash suddenly, if you detect error earlier. The most important logs are:
- System log
- Application log
- Security log (if you are investigating user account lockout or unauthorized access to domain machine)
This is the only one place where you can get information about lingering objects in your Active Directory environment. That’s why this is important to review event logs on regular basis.
To replace standard Event Viewer you can download Event Log Admin which is 3rd party tool but it’s free and much more convenient in logs management
this is one of the most important commands to troubleshoot Active Directory: Directory Services. Thanks to it, you can get quick overview of your domain/forest condition and start resolving issue, if any exists.
dcdiag /e /c /v /f:c:\dcdiag.log
will scan all Domain Controllers in entire forest to check for potential issues with them.
Note! In large environments where is big amount of Domain Controllers, I would suggest to skip using /e switch to avoid generating huge output file and shorten reporting time
does some fixes with DNS in the environment (this command superseded netdiag from Windows Server 2003)
another very important command for enterprise/domain administrator. This allows you for:
- Checking replication status
- Forcing replication
- Removing lingering objects
- Checking status for system state backup of Domain Controllers
- Identifying issue with USN rollback
Checking replication status
to display information about replication status, you should use below syntax
repadmin /showrepl /intersite /all /verbose >c:\repadmin.log
and check if there are any error statuses for replication. If you want to get summarized report about replication and its state, please use
that’s all about replication status possibilities.
One option of repadmin is forcing replication with other DCs. If you wish to push standard Active Directory update in your domain only, try this way
repadmin /syncall /APd
to force replication to all Domain Controllers in entire forest, you need to use additionally /e switch
repadmin /syncall /APed
but to be able to run above command, you need to be a member of Enterprise Administrators group in forest root domain
Removing lingering objects
This happens not very often in environment but if it appears, you need to clean up AD database. Repadmin is really helpful for this activity. Below syntax is only an overview of its usage, if you’re interested how to really clean up database from lingering objects, please read Microsoft article “Use Repadmin to remove lingering objects“
repadmin /removelingeringobjects DomainControllerName DomainControllerGUID DirectoryPartition
if you wish to only enumerate lingering objects on particular directory partition, use at the end /advisory_mode switch. It does not remove these objects, just only reports them.
repadmin /removelingeringobjects DomainControllerName DomainControllerGUID DirectoryPartition /advisory_mode
Checking status for system state backup of Domain Controllers
As you know, really important part of regular Domain Controller management is to have up-to-date System State backup. Mostly this is done by 3rd party backup tool or using Windows default command used as scheduled task. How often do you verify if this kind of backup has been finished successfully?
Believe me, you should do that to be sure that you have available the most actual System State backup for at least one Domain Controller. In case of DC failure, you would be able to restore it. However, you should take care about backing up:
- Domain Controller with forest-wide FSMO roles
- Domain Controller with domain-wide FSMO roles in each domain
In case of any domain failure you are able to restore the most important DC in a short time.
To verify if System State backup is performed properly by your Domain Controllers run below query
repadmin /showbackup *
and this will give you an output with last successful System State backup. One more important command to execute is vssadmin which allows to see if all necessary writers are running properly. When their status is stable you can be sure that System State backup would be performed without any issue. Vssadmin command needs to be run on each Domain Controller separately.
vssadmin list writers
and check status of these writers:
- System writer
- FRS writer
- Registry writer
- COM+ REGDB writer
if all of them have stable in status it’s good
Identifying issue with USN rollback
Normally, you should not worry about that but virtualization is used almost in every company now. Administrators try to use virtual machines snapshots to revert them to the previous state if they found any error. This may be a good option for domain member servers, however it is not the proper way to restore Domain Controllers. You cannot use virtual Domain Controller snapshot to revert it back as a DC restoration. This is not supported by Microsoft and may lead to USN Rollback
Information! Please do not use Domain Controller snapshots to restore it. Always use the latest available System State backup to avoid any issues
To start identifying USN rollback issue run below syntax
repadmin /showutdvec "dc=domain,dc=local"
where “dc=domain,dc=local” is distinguished name of your domain
That was only short introduction to repadmin command. If you’re more interested in its full possibilities, I would recommend reading Microsoft whitepaper.
- ADRepl Status
that separately downloaded tool from Microsoft is really helpful when you want to see Active Directory replication status. It is available at this location.
Another helpful command for domain administrator. Thanks to this command, you can see if all required DNS records exist for specified Domain Controller
dnslint /ad /s "DomainController-IP-Address"
if some records are missing, you can try to re-add them using below steps. On that DC in command-line run
ipconfig /flushdns dcdiag /fix nltest /dsregdns ipconfig /registerdns
and check once again if they appeared.
This command has many useful switches. It allows to see:
- All Domain Controller for specified domain
- Check and verify secure channel for domain
- Reset secure channel for domain
- Site to which DC belongs to
- Sites covered by particular Domain Controller
- Query domain/forest trust information
and many others. If you want to see all of its possibilities run command with /? switch in command-line to get full help.
To get all Domain Controllers and compare it to your list (mostly administrators are sure that this machine is no longer DC but it shows up in query results)
To query and verify secure channel use
nltest /server:DomainController /sc_query:DNSDomainName nltest /server:DomainController /sc_verify:DNSDomainName
if you want to troubleshoot issue related with Sites, you may list all of them firts
and to see which Sites are covered by particular Domain Controller
If you organization has established forest trust with another company, you can simply check information about that
and the same can be done to get domain trust information
command which is forgotten by admins but it is really helpful. You can use it for:
- Active Directory authoritative restoration of any object
- Change DSRM password (how many of you changes this password due to company password policy? )
- Transfer/Seize FSMO roles
- Do metadata cleanup
- Create Install from Media (IFM)
- Create application partitions
- Compress AD database
- Duplicate SID cleanup
and much more I did not mention here.
You can use this command to modify schema and AD objects attributes. In Windows Server 2008R2 you can also use this tool for deleted object restoration. Similar tool for almost the same tasks (except object restoration) is ADSIEdit which allows operating in GUI.
- dsacls, dsrevoke and LIZA
these 2 commands and one GUI tool are really needed when you troubleshoot Active Directory Delegated rights. You can verify any Organizational Unit to see if user/group of user are allowed to do required tasks on that OU. For these commands you need to know distinguished name on an OU. To get that in short way, you need run dsquery command
dsquery ou -name "OUName"
and copy its output to dsacls command
dsacls "OUDistinguishedName" >c:\dsacls.log
dsrevoke works a little bit different and needs to be downloaded first. It reports/removes specified user or group permissions from all objects where user/group is premitted
dsrevoke /report "DomainName\userNameorGroup" >c:\dsrevoke.log dsrevoke /remove "DomainName\userNameorGroup"
Note! Be aware of using /remove switch as this will remove user or group from object ACL
and finally LIZA. This is really good and free tool to see delegated permissions and it works as GUI tool. To be able to run it, you need to have .NET 2.0 installed on machine. Download LIZA
- Account Lockout and Management Tools
this is really helpful when you troubleshoot user/computer account lockout issue. Just download it from Microsoft: AL Tools
- LockoutStatus.exe to see on which Domain Controller an account is locked out
- eventCombMT.exe to review security log on all Domain Controllers or only on specified
I think, that’s all regarding direct tools for AD database troubleshooting. I probably did not mentioned all of available tools as I could not remember them. If you found that something important is missing here, please let me know, I will update this article.
SYSVOL replication tools (GPO, logon scripts)
Another part which needs to be checked separately is SYSVOL replication as it is not checked in details by mentioned tools above. First tool which allows you to see SYSVOL FRS replication is
- FRS Diag
this is old tool and Microsoft does not recommend using it anymore, however it may be still helpful in the old environments. This is not a part of operating system and needs to be downloaded separately from Micrsoft: FRSDiag download
newer and more appropriate tool for FRS diagnosis is Utrasound which is also free and needs to be downloaded from Microsoft: Ultrasound download
- GPO Tool
this may be helpful to you when you are troubleshooting Group Policies. It is a part of Microsoft Windows 2003 Resource Kit but still can be used on Windows Server 2008/2008R2. To be able to use GPO Tool, you need to download it from Microsoft.
- GPO Log View
this tool allows you in much more convenient way to review GPO logs. It is free and can be downloaded from Microsoft: GPO Log View
I think that’s all. If I forgot something, please let me know and I will update this article as it might be helpful to us during Active Directory troubleshooting process. Thank you in advance