One of the top problems IT administrators face is poor client performance, which is typically related to network performance. In a domain environment, users need access to many resources -- documents, printers, applications and so forth -- on local and wide-area networks. Many organizations employ large Distributed File System environments that require good network performance for a satisfactory user experience in locating, reading and editing documents. Desktop users have become very sensitive to poor performance because cloud and other Internet applications dominate their working hours.
Of course, a number of factors can affect the performance of a workstation, including viruses, malware, spyware, bad software updates, and saturation of resources such as memory and disk space. It's fairly easy to isolate network problems simply by observing when they occur such as during opening files in certain shares. The best way to find a problem between the client and the server is with a network trace. To properly interpret a network trace, administrators need to understand protocols and how they send and receive data. Typically, the admin will place a support call to an engineer who has those skills, but it's fairly easy to learn how to gather a trace. Getting the trace at the right place at the right time is critical to providing the correct data to a network analyst.
A trace is simply a way of hooking a machine to the network and watching what data is going over the wire. Trace tools can convert and open each packet and identify the protocol used, the IP address of the source and destination, and other payload information like the path of the share. Figure 1 shows a simple trace taken when I performed some Lightweight Directory Access Protocol (LDAP) searches and mapped a drive. Here you can see the destination and source address.
Note the path of the search request in the lower part of the trace window. This information tells us if it is searching for what we expect, which servers are responding and how long it takes. So if a user complains about slow performance opening a file, we can find the section in the network trace that has a long wait period between the request and the response. You can break it down from there. You might find that the endpoint is querying a server in a remote site in a Distributed File System (DFS) configuration rather than the one in the local site. Or, it may be a problem with client site awareness configuration, but at least you can find where the problem is.
The two most popular network trace packages are WireShark) and Microsoft's Network Monitor (NetMon). Each tool can read a capture taken by the other, but NetMon has can reveal information for Windows processes. This is especially important in diagnosing DFS problems. So rather than just seeing that an operation on a DFS share is taking a long time, we can see deeper, such as to a firewall configuration that may affect share access. Windows Server 2003 contains an old copy of NetMon, but it's very old and not very useful. To get the good stuff, download Network Monitor v3.4 from Microsoft.
Figure 2 shows a simple network with clients and servers connected through the WAN.
It's usually a good idea to take the trace on both sides -- the server and client. In the case of authentication or domain name system (DNS) name resolution, you want to trace between the client and its authenticating domain controller. On the client, open a command window, and enter the following command:
C:\>set log LOGONSERVER=\\ALF-DC1
This shows that the authenticating controller is ALF-DC1, so you would set the trace on it to get both sides of the request.
Figure 2 also shows a "promiscuous" station. A promiscuous trace station is like a third party to the network transaction. This is particularly useful if you want to get a network trace of a slow desktop boot. Obviously, you can't turn on the tracing application until the machine is booted, and by then, the problem may be gone. If we set up a different client, connecting the target client with the tracing client in a simple hub, it will reduce the traffic on the wire. One of the challenges in network tracing is limiting the noise that you are not interested in, so just temporarily configuring a dumb hub between the stations will help a lot.
Setting up the trace
Some engineers prefer to put filters on the trace when capturing data, but I like to just deal with the extra size and not put any filters on it. When I look at a trace, I can specify filters to limit the data to just what I want to see. Note that in Figure 1, there is a trace filter set at the top of the screen for LDAP, limiting the data to that protocol -- very quick and easy. While WireShark and NetMon differ in how they implement certain features, the principles are the same. Here are some basic tips for configuring the trace.
- Select the interface. If the machine is multi-homed -- has two network interface cards or NICs --0 then you need to specify in WireShark or NetMon which interface you want to trace.
- Locate the start and stop trace buttons in the toolbar.
- Start the trace, reproduce the problem as quickly as possible, and then stop the trace. This limits the data in the capture and makes it easier to find the important stuff.
- Specify the location of the trace files to not be on the system disk.
- If the problem happens at random, you have some other options:
1. Let it run continuously, and configure the trace files to be limited to a certain size.
2. Manually monitor the trace files, and make sure they don't fill up the disk. I usually leave the two or three most recent files and delete the others until the problem occurs.
- If the problem happens at a certain time every day, then use the command-line option for the tool (both WireShark and NetMon have such options). Configure a simple .bat file, enter the command for the trace, and enter it as a scheduled task in the Windows Task Scheduler. Start the trace 15 to 30 minutes before the incident is to occur, and stop it immediately after.
- Many companies don't like third-party freeware on their systems, so NetMon is usually a more acceptable option. But if you prefer to use WireShark to do your analysis, you can read NetMon traces with it. Likewise, NetMon can read WireShark traces.
- If you suspect a problem with a Windows process such as DFS, then NetMon is better to use because it can terrace the process data to a lower level than WireShark can.
Once you start capturing traces, you will probably want to start reading them. There are some excellent resources available for both WireShark and NetMon.
The best WireShark book is Laura Chappell's WireShark Network Analysis. It's available on Amazon.com but is often backordered, so try a more direct link to the book. Chappell is an expert network analyst and teaches online classes, and her website has excellent resources. She frequently speaks at Tech Ed and other conferences, so attend her sessions if you have a chance. You can download WireShark for free.
Make sure you get the latest version of Microsoft's Network Monitor, currently in Version 3.4. As I noted, some versions of Windows Server have an ancient version of this tool, so make sure you get the new stuff. For help and information, check out the NetMon TechNet blog.
ABOUT THE AUTHOR:
Gary Olsen is a systems software engineer in Global Solutions Engineering at Hewlett-Packard. He authored Windows 2000: Active Directory Design and Deployment and co-authored Windows Server 2003 on HP ProLiant Servers. Olsen is a Microsoft MVP for Directory Services and formerly for Windows File Systems.
This was first published in November 2010