Guide To
tm
Active Directory
Troubleshooting
Don Jones
Volume 6
Note to Reader: This book presents tips and tricks for Active Directory troubleshooting topics.
For ease of use and for cross referencing, the questions are numbered.
Q.33: How does Active Directory communicate? ...........................................................................1
Basic Communications ........................................................................................................1
RPC Endpoint Mapping.......................................................................................................2
RPC over HTTP...................................................................................................................2
Choosing Your Battles.........................................................................................................2
Q.34: How do I troubleshoot Active Directory communications issues?........................................3
Know the Sequence..............................................................................................................3
What to Expect.....................................................................................................................5
No Fixes, Just Symptoms.....................................................................................................7
Q.35: DNS works sometimes, but sometimes it doesnt. What can we do?....................................8
Low-Hanging Fruit ..............................................................................................................8
Replication Issues ................................................................................................................8
Protocol Problems................................................................................................................9
Q.36: How can we troubleshoot Group Policy application?............................................................9
Start at the Bottom ...............................................................................................................9
Centralized Troubleshooting..............................................................................................10
Remember the Pecking Order............................................................................................11
Replication Problems Equals Inconsistency ......................................................................12
Q.37: Are WAN links are being over-utilized by Active Directory replication traffic. If yes,
why?...............................................................................................................................................12
Secret Bridges ....................................................................................................................13
Being Smarter than AD......................................................................................................15
Making the Change ............................................................................................................16
Volume 6
Copyright Statement
2003 Realtimepublishers.com, Inc. All rights reserved. This site contains materials that have
been created, developed, or commissioned by, and published with the permission of,
Realtimepublishers.com, Inc. (the Materials) and this site and any such Materials are protected
by international copyright and trademark laws.
THE MATERIALS ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NONINFRINGEMENT. The Materials are subject to change without notice and do not represent a
commitment on the part of Realtimepublishers.com, Inc or its web site sponsors. In no event
shall Realtimepublishers.com, Inc. or its web site sponsors be held liable for technical or editorial
errors or omissions contained in the Materials, including without limitation, for any direct, indirect,
incidental, special, exemplary or consequential damages whatsoever resulting from the use of
any information contained in the Materials.
The Materials (including but not limited to the text, images, audio, and/or video) may not be
copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, in
whole or in part, except that one copy may be downloaded for your personal, non-commercial use
on a single computer. In connection with such use, you may not modify or obscure any copyright
or other proprietary notice.
The Materials may contain trademarks, services marks and logos that are the property of third
parties. You are not permitted to use these trademarks, services marks or logos without prior
written consent of such third parties.
Realtimepublishers.com and the Realtimepublishers logo are registered in the US Patent
& Trademark Office. All other product or service names are the property of their
respective owners.
If you have any questions about these terms, or if you would like information about licensing
materials from Realtimepublishers.com, please contact us via e-mail at
info@realtimepublishers.com
ii
Volume 6
User Datagram Protocol (UDP) port 88 is used for Kerberos authentication. Transmission
Control Protocol (TCP) port 88 can also be used, although its less common.
TCP and UDP ports 135 are needed for remote procedure call (RPC) endpoint mapping.
RPCs are used for a number of domain controller-to-domain controller and client-todomain controller operations. Unfortunately, not all communications take place over port
135, as Ill discuss later.
TCP port 139 and UDP port 138 are needed for file replication between domain
controllers. This port combination is the standard NetBIOS session service port set.
UDP port 389 handles LDAP queries and is used for normal domain controller
operations.
TCP and UDP ports 445 are used for file replication and are the standard Windows file
sharing ports.
TCP and UDP ports 464 are the Kerberos password change protocol ports.
TCP port 593 is used by the RPC over HTTP transport. Although you dont technically
need this port for normal operations, Ill discuss later how this feature can make working
with domain controllers through firewalls a bit easier.
TCP port 636 is for LDAP over Secure Sockets Layer (SSL), which is the default LDAP
methodology for Windows Server 2003 and later.
TCP port 3268 and 3269 handle Global Catalog (GC) queries. Port 3269 handles secure
queries. Any domain controller that needs access to a GC or that is acting as a GC server
will use these ports.
TCP and UDP ports 53 are used to communicate with Domain Name System (DNS),
which is a vital part of AD communications.
Generally, opening these ports between clients and domain controllers, or between domain
controllers, will enable AD to function normally. One exception is RPC traffic.
Volume 6
RPC Endpoint Mapping
Most RPC communications first start on TCP port 135. However, thats merely the RPC
endpoint mapper service. Its function is to select a new destination port for further
communications in that RPC session. Exchange Server is a major user of RPC endpoint
mapping, and its very difficult to get Exchange traffic through a firewall as a result. The range
of potential endpoint addresses used by RPC communications is huge, essentially requiring the
entire firewall to be opened to allow all the possibilities. The ports selected by the endpoint
mapper can range from TCP 1024 to TCP 65535.
Fortunately, you can force AD to always map endpoints to specific ports. The Internet Assigned
Numbers Authority (IANA) has set aside ports 49152 to 65535 for private port assignments, so
choose ports from this range and force AD to always use them. Youll then be able to open a
much smaller range of ports in your firewalls.
To force port selection, modify the registry key
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters. Youll
need to create or modify a DWORD value named TCP/IP Port, and set it to whichever port
youre going to use. However, there are some downsides to this modification:
Youll need to modify every domain controller on your network; otherwise, they wont
be able to communicate properly. Youre effectively disabling endpoint mapping, so all
domain controllers will have to be manually told which port to use.
Your domain controllers will have to work a bit harder to handle the same number of
connections. The servers communicate less efficiently when forced to use a single port
for all communications because they cant rely on the port number to identify individual
conversations.
Volume 6
If you absolutely must have a firewall between domain controllers, youll need to restrict the
ports they use. The File Replication Service (FRS) will need to be restricted, as will general
communications. I explained earlier how to force an RPC port for general communications;
Microsoft can help you with other types of traffic.
See the Microsoft articles Restricting Active Directory Replication Traffic to a Specific Port for
information about restricting AD replication traffic and How to Configure a Firewall for Domains and
Trusts for information about configuring firewalls to support domain and trust communications. In
addition, see How to Restrict FRS Replication Traffic to a Specific Static Port for information about
restricting FRS traffic.
For more information about AD communications, see Question 10, Question 19, and Question 33.
Additional traffic can occur when clients attempt to look up cross-domain references by
contacting a Global Catalog (GC) server. Clients might contact GCs for other reasons, such as
during the logon process, to resolve Exchange 200x Address Book lookups, and more.
Troubleshooting this traffic can be difficult. Kerberos traffic, for example, is always encrypted
using temporary encryption keys established between the Key Distribution Center (KDC
domain controller) and clients. Even simple Lightweight Directory Access Protocol (LDAP)
queries to GC servers or domain controllers are generally encrypted by Secure Sockets Layer
(SSL) encryption, particularly in Windows Server 2003, in which LDAP over SSL is the default.
Know the Sequence
The only way to effectively troubleshoot AD communications is to use Network Monitor.
Although other utilities are very good at troubleshooting ADs operations, they dont get into
low-level communications. Youll need to know what to expect, then you can see what traffic is
actually being transmitted.
Volume 6
The one caveat youll need to be aware of with Network Monitor is that it doesnt have built-in
parsers for most of the traffic AD uses, including Kerberos. Thats OK; you dont need a parser
to troubleshoot traffic, you just need to know what youre doing. Network Monitor might not, for
example, recognize Kerberos traffic when it sees it (because the tool lacks a parser), but Ill show
you what to look for so youll recognize the traffic.
Figure 34.1 shows a typical Network Monitor capture. In the top pane are all the capture frames
of traffic; the bottom pane shows a breakdown of the frame selected in the top pane. Notice that
the protocol in this case is listed as TCP. Network Monitor will show the Transmission Control
Protocol (TCP) or User Datagram Protocol (UDP) for most AD traffic. To determine the actual
traffic type, youll need to look at the packets details.
Volume 6
In this example, the packet has a source (src) port of 80 and a destination (dst) port of 4254. I can
see that the source address was a server and the destination was a client, so I can presume that
this is a reply. That means the servers source port is the same port that the client originally used
to contact the server: port 80. Port 80 is HTTP traffic, although Network Monitor obviously
didnt recognize this frame as a traditional HTTP packet. The destination port was randomly
chosen by the client when the client initiated the communication session.
Most AD communications will appear as TCP, and youll need to look at the source port used by
the server or the destination port used by the client to determine the traffic. Expect to see the
following TCP and/or UDP ports:
88Kerberos
389LDAP query
3268GC LDAP
What to Expect
The exact sequence of packets will differ from environment to environment, especially because
AD traffic is usually interleaved with many other, unrelated forms of communication. If youre
experiencing any form of problem, check the following items (these dont require you to
memorize complex sequences of communications):
If traffic is passing through a firewall or router, check the traffic on both sides of the
device to make sure its all getting through.
Make sure DNS queries are working. Network Monitor will parse these and display them
as DNS queries; ensure that queries are receiving appropriate responses and that clients
are querying the correct servers.
You cant actually read most exchanges between domain controllers or clients, but you
can ensure that theres an actual conversation. In other words, make sure youre seeing
the client make a request, then see the domain controller send something back.
Watch for patterns. For example, if some operation seems to be taking too long or
repeatedly times out, youll likely see an identical sequence of packets repeat over and
over. This repetition generally means something internal to the traffic has gone wrong,
such as a user password being wrong or an incorrect configuration parameter.
Volume 6
One way that Network Monitor can help draw out the AD traffic from everything else on your
network is with filters. When viewing a capture, click Filter. Youll see a dialog box similar to
the one that Figure 34.2 shows.
Click Expression to add a filter expression. As Figure 34.3 shows, create an expression that
filters for packets having a specific destination port. In this example, the filter will eliminate all
traffic except Kerberos traffic, which uses UDP and TCP 88.
Volume 6
Be sure to create filters that allow for a source or destination port so that youll capture both the
client and server sides of the exchange. You can use the Or button on the filter dialog box to
ensure that any packets meeting any one of your criteria will be included in the display. Figure
34.4 shows an example of the configured filters.
Filtering for specific types of traffic will help you focus on different areas to be sure theyre
working properly without having to manually wade through all of the packets that Network
Monitor may have captured. You can load and save filters for later use, making it easier to
quickly look at Kerberos traffic, DNS traffic, LDAP queries, and so forth.
No Fixes, Just Symptoms
Network Monitor wont allow you to directly fix any problems. However, it can point the way to
whats broken. For example, if capturing packets from each side of a firewall reveals that
Kerberos (port 88) packets arent making it through, youll know that you need to reconfigure
the firewall. If DNS queries contain incorrect IP addresses, youll need to fix the DNS zone
database. If traffic seems to be repeating itself, theres something wrong or misconfigured within
the traffic itself (such as a username or password contained within the traffic), and youll need to
closely examine the configuration of the computers involved.
Volume 6
If replication latency is a problem for your DNS zones, consider upgrading to Windows Server
2003. In Windows Server 2003, the DNS zone is stored in an AD partition, and you can control
which domain controllers contain a copy of the partition. By limiting the partition to just those
domain controllers that are acting as DNS servers, youll force a new replication topology to be
generated for that partition. The result will be fewer servers replicating the information. Thus,
replication will be able to occur more quickly, causing the different copies of the partition to
converge more quickly and reducing problems caused by replication latency.
Volume 6
Protocol Problems
Another problem can occur if your network is assuming that DNS uses User Datagram Protocol
(UDP) port 53 and blocks access to Transmission Control Protocol (TCP) port 53. The DNS
specification requires DNS to use the connectionless UDP transport protocol, but only for small
queries. Larger queriesor, more accurately, larger query responsesthat wont fit into a single
UDP packet may be broken into multiple TCP packets instead. This switch to TCP can cause
bewildering problems on your network because some DNS queries will work fine and others will
simply time out.
DNS queries will nearly always go out via UDP (the notable exception being the Simple Mail Transfer
ProtocolSMTPservice in IIS, which seems to always use TCP); replies will come in on UDP or
TCP depending upon the number of hosts and IP addresses contained within the replies.
If youre not sure whether this circumstance relates to your problems, try using Network Monitor
to capture DNS traffic on both sides of your firewall. If youre not seeing identical traffic on both
sides of the firewall, the firewall is obviously blocking some DNS traffic, most likely large
replies. To play it safe, I recommend opening your network to incoming DNS traffic on both
UDP and TCP ports 53.
Volume 6
By default, Gpresult doesnt show you which individual policies are applied or what they are set
to; because GPOs successively overwrite one another as they are applied, you can still be left
with a troubleshooting task to figure out which of the GPOs listed is responsible for the settings
youre seeing. Fortunately, Gpresult has a superverbose mode, enabled by running
Gpresult /z
This mode not only displays which GPOs have been applied, but lists every single policy thats
enabled in each GPO, allowing you to see which GPO modified which setting, and which GPO
finally won out in the end. Figure 36.1 shows a portion of Gpresults superverbose output. In this
example, the GPO being applied is Local Group Policy, and you can see exactly which registry
keys each setting is modifying.
Superverbose mode also breaks down the user and computer policies, allowing you to see every
setting that is affecting the current users or their machines.
Centralized Troubleshooting
For Windows Server 2003, Microsoft introduced a sort of server-side Gpresult. Its called RSOP,
and its built into the Active Directory Users and Computers console. With this tool, you can
actually view applied policies in a graphical user interface, which can be a bit easier to work with
than the text-only Gpresult output.
The Windows Server 2003 version of the Active Directory Users and Computers console will work
fine against Windows Server 2000 domains running Service Pack 3 (SP3) and later; however, you
might need to purchase a copy of Windows Server 2003 to be able to legally use the new the new
Active Directory Users and Computers console.
10
Volume 6
To launch the new tool, open the Active Directory Users and Computers console and select a
user or computer. Right-click the object, and select Resultant Set of Policy (Planning) from the
All Tasks menu. The RSOP wizard will step through a number of screens that allow you to
specify the user or computer account location, determine whether you want to simulate the effect
of a slow network link or loopback processing, choose a specific site, or modify either the users
or computers security group memberships. The final result, which Figure 36.2 shows, is a
console that looks a lot like the Group Policy Object Editor, displaying each of the policies that
have been applied to the user. For each policy, youll see which GPO was responsible for the
final application of the policy, making it easy to see where you need to go to make changes.
Figure 36.2: Output from the new Active Directory Users and Computers RSOP tool.
11
Volume 6
Replication Problems Equals Inconsistency
If users are experiencing inconsistent GPO application, the problem is most likely a failure in
Active Directorys (ADs) GPO replication process. Although AD defines GPO links in the AD
database, the GPOs themselves are contained in normal files, which are replicated from domain
controller to domain controller by the File Replication Service (FRS). A failure in the FRS can
result in inconsistent GPOs on domain controllers, which results in users having inconsistent
GPO application.
For more information about the FRS, see Question 31.
Perhaps the simplest way to verify that the GPOs have replicated consistently is to check the files
themselves, located in each domain controllers SYSVOL share. Provided each domain
controller has the same files, with the same date and time stamps (which are replicated by FRS,
not recreated on each domain controller), then everything should be consistently applied to all
users.
12
Volume 6
If you set up one site link per T1 link, youll wind up with a design somewhat like the one in
Figure 37.2. This configuration is straightforward and reflects the way that most companies build
their sites and networks. If you have additional backup links between sites, you might even
configure separate site links for those, configuring a higher link cost to reflect the links nature as
a backup.
Secret Bridges
What you might not realize is that AD, by default, creates site link bridges for every site link.
This setup isnt necessarily a bad idea. For example, consider what happens if an administrator
locks out a user account in the Bath office. Obeying only the site links, AD will have to replicate
that change from a bridgehead in the Bath office to a bridgehead in the Boston office, then to
New York, LA, Las Vegas, and finally Reno. Depending upon your replication schedules, it
could be quite some time before a Reno domain controller actually locked out the users account,
even though account lockout is a high-priority change for replication. In the meantime, a user
could be logging on to a Reno or Las Vegas domain controller, relying on the fact that those
domain controllers havent yet heard about the lockout.
13
Volume 6
ADs automatic site link bridges create a topology similar to the one in Figure 37.3, in which
each site is logically linked to the others.
When a change is made at the Bath office, its bridgehead domain controllers replicate directly to
bridgehead domain controllers in each of the other offices. Effectively, AD is ignoring the
physical layout of your network a bit in order to speed replication. The cost is that your WAN
links are going to carry more traffic than you might expect. For example, the link connecting Las
Vegas and LA will carry Baths change twiceonce as Bath replicates with Las Vegas, and once
as Bath replicates with Reno. The link between Bath and Boston will carry the same replication
traffic five timesonce for each of the other offices.
For more information about how site link bridges are created and how the replication topology is
generated, see Question 19.
14
Volume 6
In this case, a change made at Bath would replicate to Boston and LA, because LA is virtually
connected to Bath. LA would then replicate to New York and Las Vegas. Reno would receive
the information last, either from New York or Las Vegas. You might reconfigure this setup a bit
to have the Reno site link bridge connecting to LA rather than New York; doing so would place
more burdens on LA-based domain controllers, but would disseminate the information in fewer
steps. The site link bridges effectively shortcut the physical topology, wasting a small amount of
WAN bandwidth but providing minimal replication latency.
15
Volume 6
If you disable this behavior, you should definitely review your manual site link bridges and
create new ones as necessary to provide the desired amount of replication latency. I recommend
testing your new topology by making changes in your furthest-out office (Bath, in our example)
and measuring the amount of time it takes the change to replicate to the opposite corner of your
network (Reno, in our example). Adding site link bridges that bridge from the edges of your
WAN to the middle of your WAN is the most effective strategy, as it provides the most efficient
shortcut for replication traffic.
16