A few years ago, I worked on a project that investigated more than 10,000 computers that had been made into a botnet because of a targeted malware attack. Weak security practices, such as no vulnerability testing and an overreliance on traditional antivirus software, were part of the problem. We also discovered a communications breakdown among the security team, the help desk, IT administrators and other involved parties. It was ugly.
"Bots" and their command-and-control (C&C) servers are categorized as advanced malware. As we learn more about the intricacies of advanced malware and just how complicated and widespread a problem it can be, it's clear that botnet cleanup is not a simple matter. Unfortunately, at the enterprise level, you don't have the luxury of simply shutting everything off and re-imaging systems.
A bot infection such as the one I encountered could be one of the nastiest things you ever deal with as an IT professional, but it doesn't have to ruin your career.
Here are five key steps for enterprise IT admins to focus on when responding to an advanced malware infection and conducting bot removal.
If you're going to effectively manage IT risks, you need well-developed incident-response procedures. The mere lack of a flight plan is arguably the greatest impediment to effective security responses. Rather than be reactive, start developing proactive measures today to minimize the potential effects of an advanced malware attack. How would your organization deal with systems hijacked by a botnet? A good plan addresses varied endpoints, network access, data management and unknowing users.
At the saying goes, diagnosis is half the cure. So, just where is the infection? Of course, this is the $64,000 question with advanced malware.
Using encryption, quick Domain Name System changes called "fast flux" and covert channels, much of the typical botnet and C&C code and communication flies under the radar of traditional security controls. That's why it can be so difficult to detect botnets without the right tools. But if you can find the hosts from which advanced malware is originating, you'll be able to tighten investigation and containment. Hint: Windows workstations are likely the larger percentage, but it could be your Windows servers.
Microsoft's Sysinternals tools are a good place to start. Just be careful what passwords you enter, what other systems you access, etc. on the suspect machines. A network analyzer such as Wireshark or, better yet, OmniPeek can provide additional insight into what's happening at the network level, which can be even more beneficial because of the higher-level perspective it provides.
In addition, you may end up needing more advanced technologies from vendors such as Damballa and FireEye to effectively track down infections and begin bot removal.
If you know enough about an infection, you can implement and perhaps even apply some stopgap network access control lists or firewall rules to block the malware's inbound or outbound network traffic until you get things cleaned up.
You can also take a whitelisting approach to fight the malware infection with something as rudimentary as local policies or Group Policy Objects or something as advanced as Bit9's "positive security" controls.
The thing with bot removal is that you likely won't be able to run a simple antivirus scan and be done with it. Odds are great that you won't even be able to detect anomalous behavior from advanced malware. Even if detection is possible, the malicious code is often so intertwined with the operating system/registry that mainstream antivirus software won't have a clue about how to handle it.
One of the best things you can do is to run multiple antimalware tools -- especially those that know about more advanced threats such as Webroot and Malwarebytes. You might not have a choice but to reimage your systems.
Also, be aware of what's at risk of being lost when you reimage systems. There's hardly any internal security assessment I've performed where I didn't see the only copy of sensitive information located on a workstation that has never been backed up.
As far as malware infections go, one of your worst enemies is all of the Java, Adobe and related third-party software updates your users haven't yet applied. A close second will be Windows XP as Microsoft support for it ends.
More on malware detection and bot removal
Web-based malware detection can reinvigorate defenses
Fileless and Java malware pose new threats to desktops
Targeted attacks prompt new approaches to malware defense
Better defenses needed as malware becomes ubiquitous
Malware requires defense in depth
The thing is, simply patching enterprise systems could eliminate the threat or at least stop it from spreading. So start thinking about third-party patch management now and you'll have another tool in your toolbox when the time comes to respond.
When all else fails, you may have to call in the experts. Botnets in particular can be quite difficult to handle. As I found out in my project and in hearing from other incidents, botnets are not all that different from cancer in the body. If one single bit of the botnet remains on the network, you'll likely experience another infection. Incident response and forensics professionals who deal with significant breaches on a weekly basis can help get an organization out of an IT security bind.
Desktop malware and bot removal is only one aspect of minimizing risks. Threat intelligence -- knowing what to look for and having good information to base decisions on -- is key. This goes back to a fundamental admin principle: Know your network. As boring as it sounds, you really do need to know what "normal" means so you can detect abnormal activity.
Even if you have no tools or processes to know what's what, start today. Acquire the endpoint controls, a good network analyzer and event monitoring tools you need to fight the botnet threat. As my favorite security saying goes, you cannot secure what you don't acknowledge.
I suspect the problem with advanced malware is not going to get any better. Now is the time for desktop and network admins to enhance their skill sets and become threat analysts, data scientists and incident responders. Even if these areas of IT don't currently affect your job, they will in time.
This was first published in December 2013