Fingerprinting, or identifying the operating system that's running on a remote computer, has traditionally been used as a hacker technique. The idea is that a hacker often gathers as much information about the target system as possible prior to performing the actual hack. Of course one of the most critical pieces of information that a hacker can gain prior to the hack is what operating system the target computer is running. Although fingerprinting has been traditionally considered to be "a bad thing" there are ways that you can use fingerprinting to your advantage.
There are two ways you can use fingerprinting to find a hacker. One way is to fingerprint the hacker's computer in an effort to gain evidence against the hacker. Another way is to reconfigure your servers so that they basically lie to a hacker.
Whether you want to fingerprint hackers or obscure your server's own fingerprints, you need to understand some of the basics of how fingerprinting works. Fingerprinting works because each operating system performs common functions just a little bit differently. For example, every operating system configures the TCP/IP protocol stack differently. An example of this is the TTL (Time to Live) stamp on outbound packets. As innocent as this stamp seems, it reveals a lot about the server. For one thing, not every operating system uses TTL stamps. Those operating systems that do use TTL stamps tend to use different values. For example, Linux operating systems tend to set the TTL at 64 hops.
Fingerprinting the Hacker
Suppose that you wanted to fingerprint a hacker's activity. You could use this general concept to perform what's known as passive fingerprinting. The idea behind passive fingerprinting is that you can fingerprint the would-be intruder without them knowing that they are being fingerprinted, just by looking at the inbound packets. To see how this works, let's go back to my TTL example. We already know that Linux machines tend to set the TTL at 64. Imagine however that an inbound packet had a TTL of 49. What operating system sets the TTL at 49?
Actually, this is kind of a trick question. The TTL wasn't originally set at 49. The TTL value is decremented by one every time the packet hops from one router to another. Therefore, to determine the true TTL value, you need to add the number of hops that the packet took to reach you to the current TTL value. For example, if the packet took 15 hops to reach you and the current TTL value is 49, then the original TTL was 64. We have already established that Linux uses a TTL of 64.
Things aren't quite that simple, but these things never are. This method of fingerprinting only works if the intruder machine's TCP/IP values haven't been modified. In fact, there are no TCP/IP packet components that can positively identify an intruder's machine beyond a shadow of a doubt. However, if you look at enough parts of the TCP/IP packet, you can usually determine the intruder's operating system with a high degree of certainty. Some of the packet's attributes that you might want to look at include the Don't Fragment bit, the Type of Service, and the Window size.
Fooling Hackers with Fingerprinting
If a hacker is trying to fingerprint your Windows based Web server, you could have the server configured to make the hacker think that the server is running Linux instead of Windows. This is advantageous because many of the hacking exploits that are used against Windows will not work against a Linux machine and visa versa. Disguising your server's fingerprints can cause a hacker to attempt to use ineffective measures to try to hack the system. Using Linux related hacking commands against a Windows Server will really stand out from your server's normal traffic patterns. It should be very easy for your intrusion detection system to pick up on these anomalies and alert you to the attempt.
There are a couple of ways to alter your servers fingerprint. One way is to edit the registry and modify the TCP/IP protocol stack to emulate that of a different operating system. There are also a number of third party utilities that can somewhat obscure your server's operating system. One of my personal favorites is a utility called ServerMask. It's both cheap and effective. Regardless of what you use, you must keep in mind that there is no way to completely obscure a server's operating system. A skilled hacker with enough determination will realize pretty quickly that the server is posing as a different type of server. The trick is to have your intrusion detection system catch the attack before the hacker realizes what is going on.
About the author: Brien M. Posey, MCSE, is a Microsoft Most Valuable Professional for his work with Windows 2000 Server and IIS. He has served as CIO for a nationwide chain of hospitals and was once in charge of IT security for Fort Knox. As a freelance technical writer, he has written for Microsoft, TechTarget, CNET, ZDNet, MSD2D, Relevant Technologies and other technology companies. You can visit his personal Web site at www.brienposey.com.
More information from SearchWindowsSecurity.com
This was first published in September 2005