A poorly written application or service can choke system resources, and while each succeeding version of Windows has included that many more provisions to ameliorate this problem, it can still leave a system -- server or desktop -- unresponsive.
I've looked into several programs to help deal with this problem -- not just for myself, but for other folks I've known who are forced to deal with poorly written programs that affect their systems periodically. One of the best is Process Lasso, which superficially resembles a replacement for Windows Task Manager but is something else entirely. For instance, it determines a program's working set size behavior and can do much more.
Process Lasso is a process governor. It watches over all existing processes in the system, and if any given process uses more than a certain percentage of CPU for a certain length of time, it's throttled back (set to idle priority).
Process Lasso can also insure that when a given process (specified by name) starts up, it will always start at a given priority -- such as Vista's TrustedInstaller.exe module, a source of grief to more than a few people. Process Lasso can automatically terminate a process (again, specified by name) that attempts to run, something that you can use to control malware and block unwanted update checkers or other unessential applications from running.
In addition, individual processes can be excluded from restraint, and the program's activity can be logged to a file. The program can run in an entirely GUI-less mode, with only the process governor portion of the program loaded and running silently in the background.
Another included feature, auxiliary to the program's real use but worth mentioning, is its ability to force all programs to trim their working set size. It's essentially an implementation of those "memory compacting" utilities that were briefly all the rage but which are essentially snake oil. This feature is mainly of interest to program developers as a way to determine a program's working set size behavior and is not intended to enhance system performance per se.
About the author: Serdar Yegulalp is editor of Windows Insight (formerly the Windows Power Users Newsletter), a blog site devoted to hints, tips, tricks and news for users and administrators of Windows NT, Windows 2000, Windows XP, Windows Server 2003 and Vista. He has more than 12 years of experience working with Windows and contributes regularly to SearchWinComputing.com and other TechTarget sites.