|Windows patch maintenance and post-patch security|
Here are some basic tenets to keep in mind after applying patches -- especially after Microsoft's Patch Tuesday -- culled from many different people's experiences with administering both remote servers and local workstations.
Be mindful of changes in general system behavior. If something is clearly wrong, it'll tend to announce itself. That said, this is a little easier to do when you are solely responsible for the system in question -- for instance, a server. If you deal with multiple workstations, stay close to one of the patched machines, if you can, and if anyone reports bizarre behaviors, you can investigate them and then try and duplicate them on your own "pet" machine. Check this tip out for some advise on handling patch emergencies.
Inspect the system logs. Compare error logs both before and after a patch, and see if anything new or unusual jumps out at you. Sometimes it might not be anything, or it might be coincidental, but error logs are one of the best places to go to for concrete information about something not working correctly. This is especially important if what is going wrong has no other outward symptoms yet, except for a logged error. (Also keep in mind that some errors may simply be false alarms and have no real connection to anything; sometimes it can be hard to tell the difference.)
Check compatibility on any potentially affected applications. If there's a chance a given patch might affect the way a program works, test it before and after the fact. For instance, test a patch to a middleware component by making sure no database connections are suddenly throwing errors. In addition, make sure other non-trivial database operations that take place in your environment -- and that require middleware (like retrieving large amounts of data or opening many connections at once -- also get tested whenever possible.
Rolling back Windows patches
- Roll back by hand
Microsoft's hotfixes and service packs for Windows come pre-equipped with their own Windows patch rollback mechanisms that can be activated manually if the need arises. If you want to uninstall a given hotfix, here's the procedure for doing so.
- Set Explorer to show hidden and system files if you haven't already done so.
- Open the %SystemRoot% directory and look for a series of directories with the name $NTUninstallKBXXXXXX$, where XXXXXX is the Knowledge Base article number for the hotfix in question.
- Within that directory is another directory named spuninst.
- Inside spuninst is an executable named spuninst.exe. Run it, and the hotfix in question will be rolled back through a Wizard interface.
- If spuninst.exe doesn't work or is unavailable, type batch spuninst.txt. This executes a batch-file version of the same recovery options.
- System restore
Windows also has a global mechanism for restoring settings and components to an earlier state. It's one most of us should be familiar with: System Restore. This method is something of a brute force way to move back to before a hotfix was installed. And it's slow -- it can take many minutes for a System Restore to complete -- but it covers absolutely everything that might have been touched by a hotfix.
Be mindful, though. You cannot run System Restore from the Recovery Console, at least not without a good deal of manual hacking. However, you can run an individual patch rollback as described before from the Recovery Console.
- Third-party software
The most complete way of dealing with patch roll back is probably through a third-party package. Plenty of third-party software products exist for rolling a system back to an earlier state, with undoing changes made by patches as part of that.
Fixing post-patch problems: Auditing revision levelsJust because you patch something once doesn't mean that you won't have to patch it (or something else) later. The list below details how you can audit revision levels to fix problems after deploying Windows patches.
- In Explorer: The most obvious way to determine the revision of a component is just to right-click on it in Explorer and select Properties | Version. Or, you could switch to the Details view in Explorer and show the File Version and Product Version as columns. But, with this view, you can't easily export the results. Note that .DLLs will have a Version tab, but .EXE files will not, so this limits its usefulness a bit.
- Through Process Explorer: The endlessly useful Process Explorer utility from Sysinternals lists the revision levels of all loaded components. If you click on the name of a process and select View | Lower Panel View | Show DLLs, you can see all of the loaded DLLs in use by that process as well as their revision levels. This is only useful for running processes, but the program does support exporting the information shown to a delimited text file. Note that it may take several seconds for the program to poll all the used .DLLs for a given process.
- Through an external resource: This method is best if you want to find out what other revisions there might be for a process or component. For Microsoft components, Microsoft itself has a site called DLL Help. There you can look up any component from a Microsoft or Microsoft-supported product, see all of the tracked revisions for the component and learn more about each of them. However, DLL Help is only useful for Microsoft components, not third-party apps.
- Through a script: This option is the most effective way to report back on a whole slew of components at once. For instance, use a script if you want to audit all of the items in a directory that represent what a patch will put into place, and you want to see a quick side-by-side comparison of component revision information. One such script is available online at JSWare and, with a little work, it can be used to obtain the revision information for all files that match a wildcard or are in a directory.
Most of the time Windows Server Update Services (WSUS) does a pretty good job of deploying patches to computers throughout an organization. If you want to make sure that WSUS deploys each patch in as timely a manner as possible, then it's worth spending a little time to optimize the patch management process. WSUS is a Web application, so it doesn't have any settings of its own that are directly related to performance, but there are several WSUS settings that you can use to improve the efficiency of your patching operation though.
Download files when available
One of the first things that I recommend in Windows patch management is configuring WSUS to download patches as soon as they are available, not when they are approved. Normally, patches are not downloaded until you approve them. The problem with that is that as soon as patches are approved, computers try to install them. If the patch has not yet been downloaded though, the update process has to stop and wait for the patch to be downloaded. This whole process can be made more efficient by downloading files as soon as they become available.
To change the download option:
- Open the WSUS Admin console and click the Options button in the upper left corner of the screen.
- When the Options screen appears, click the Synchronization Options link
- Scroll all the way to the bottom of the screen and click the Advanced button.
- The Advanced Synchronization Options dialog box will appear.
- Make sure that the Store Update Files Locally option is enabled and that the Download Update Files to This Server Only When Updates Are Approved option is not selected.
Download express installation files
By default, WSUS downloads patches and pushes those patches to the clients. As you can imagine though, if a patch is large or if you have a large number of clients, this method of installation can consume a considerable amount of network bandwidth and it may take a long time to update all of the clients.
Express installation files tend to be larger than the normal patches that WSUS downloads, which means that the download may take a little bit longer to complete. This extended download time is usually more than made up for by the reduced time and bandwidth requirements when updating clients.
Check out other helfpful hints on optimizing WSUS performance in security expert Brien Posey's article, including why to use a dedicated WSUS server and why you should be downloading patches in specific languages only.
This was first published in September 2007