Check IT List: Five steps for rootkit detection

Rootkit technologies are rapidly cropping up in a variety of places, including commercial security products and seemingly benign, third-party application extensions. Whatis.com defines rootkit technology as "a collection of tools or programs that enable administrator-level access to a computer or computer network. A hacker installs a rootkit on a computer after first obtaining user-level access, either by exploiting a known vulnerability or cracking a password. Once the rootkit is installed, it allows the attacker to mask intrusion and gain root or privileged access to the computer and, possibly, other machines on the network."

Sony and Symantec have recently been reported to be engaged in questionable practices involving rootkits. Both companies have reportedly had rootkit-based functionality in certain products. This technique uses a simple approach to disguise activity and filesystem information from the Windows API, thus preventing direct observation of the application's behavior. It also causes the operating system to misreport systemwide activity.

Finding and eradicating potential rootkit installations is not an exact science. Rootkits can be installed on a computer in many ways. No single tool (and no combination of tools) can correctly identify all rootkits and rootkit-like behavior. The following Check IT list provides pointers and references to purpose-built tools that are useful for examining application behavior and determining if further investigation is necessary.

1. Search your system memory. Monitor all ingress points for a process as it is invoked, keeping track of imported library calls (from DLLs) that may be hooked or redirected to other functions, loading device drivers, etc. The drawback to this approach is that it is tedious, time-consuming and cannot account for all possible avenues in which a rootkit can be introduced into the system.
2. Seek the truth -- expose API dishonesty. One good rootkit detection application for Windows is the RootkitRevealer by Windows security analysts Bryce Cogswell and Mark Russinovich. This tiny (190 KB) binary scouts out file system locations and registry hives, looking for information kept hidden from the Windows API, the Master File Table, and directory index. In addition, Jamie Butler, author of the highly recommended trade book Subverting the Windows Kernel: Rootkits, has created a tool called VICE, which systematically hunts down hooks in APIs, call tables and function pointers.

   RootkitRevealer may take a while to complete because it performs an exhaustive search. First it dumps the registry hives, then it examines the C:\ directory tree for known rootkit sources and signatures, and finally performs a cursory analysis of the entire C: volume.
   

3. Keep abreast of the latest antivirus and malware protection software from leading antivirus and security vendors. Sysinternals and F-Secure offer standalone rootkit detection tools. Even Microsoft has implemented rootkit detection features in its own Malicious software removal tool.
   Sysinternals RootkitRevealer
   F-Secure Blacklight
   Microsoft Malicious Software Removal Tool

4. Update your firewall protection. Remember, for the concealment process to be effective to a potential attacker, it is vital that the hacker can get back into a machine once it's been compromised. Although firewalls do nothing to mitigate application-level risks, they can pose a significant challenge to attackers when they prohibit re-entry into a victim machine.

5. If possible, harden your workstation or server against attack.This proactive step prevents a attacker from installing a rootkit in the first place. The National Security Agency publishes a guideline for hardening Windows environments, which is a great jump-off point for educating yourself on preventive actions against system intrusion.
   NSA Guideline to Securing Windows XP

Note: If you do discover a rootkit (or rootkit-like software) on your machine, unless there's a targeted removal tool (as is the case for the Sony DRM software), the only way to remove a rootkit from an infected machine is to wipe the drive and reinstall everything. If that proves necessary, be sure to back up all files and settings or try to roll back to an earlier backup, drive image or restore point.

Ed Tittel is a full-time freelance writer and trainer based in Austin, Texas, who specializes in markup languages, information security, and IT certifications. Justin Korelc is a longtime Linux hacker who concentrates on hardware and software security topics. Ed and Justin contributed to a recent book on Home Theater PCs and "Tom's Hardware 2005 Holiday Buyer's Guide."