Creating a testing environment for hotfixes and patches

It is definitely a wise idea to thoroughly test patches before deployment. First, I would suggest creating a test environment. Ideally, you'd be able to duplicate all of the various system types and implementations you have in your environment (all applications, operating systems, and combinations thereof). However, in the real world, this is unrealistic for most organizations.

Instead, for a test environment, you can create systems that represent most of the technologies (if not in all their incarnations) you use in production. For example, if you use the Apache web server on Debian Linux, you would have a representative server using these technologies (not necessarily mirroring all content and applications). You might have representative desktop systems loaded with common business applications. To make this even simpler, you can use a virtualization tool like VMWare to create multiple operating system images that can be run simultaneously on the same machine.

Once your environment is reasonably composed (the more critical your systems, applications, and data are, the more rigorous and representative your test environment should be), you can work out test plans for evaluating the effects and results of patches and hotfixes. First, you will want to ensure the actual installation and deployment of the hotfix works on your representative systems. Of course, the patch management and distribution tools you use should be loaded in the test environment.

Once you have tested the installation of the patch, you can proceed to user and application testing. This consists of normal operating tests to ensure that the system and applications continue to work as expected post-patch. Depending on how sophisticated your needs are, you may use an automated suite of testing tools for this purpose.

After you feel comfortable with the patch in your test environment, most organizations opt for a staged deployment. In a staged deployment, the early phases of patch distribution are used as a sort of 'live test'. The patch is deployed to a subset of systems that are monitored closely. Based on the results of these initial deployments, further decisions are made regarding a more comprehensive installation across the environment.