Ask the Expert - Mr. Database
The 10 Tenets of Patching
by Eric Gross, DBA

We all install and upgrade databases far less frequently than we patch them. Patches are the lifeblood of your existing installations; they bring the latest features or fix the latest issues. Due to the increased frequency of patching, it is particularly important for the process to run smoothly—your databases depend on it. Here are my 10 tenets of patching:

1. Come up with a unified timetable for changing your systems (installing, upgrading, patching, and application changes).

• It may be true that the frequency of patching far exceeds installations or software upgrades, but it still needs to fit into an overarching plan that specifies how far back in the patching pocket your various databases will sit. Some databases need the most current security patches because of exposure, but other databases may sit far behind security layers and provide the revenue for the business. These better-protected databases may not need to be touched unless there is a problem that threatens to cause downtime regardless of the security implications. Each organization has a different tolerance for the risks associated with changing systems that are working. These requirements must be documented.

2. Determine if the patch provides its intended result.

• Patches are released in order to solve problems that are either discovered or resolved after the release of the product. Vendors test patches far less than other releases and upgrades; regression testing is usually performed in only a limited few environments. It is vital that you determine if a patch deemed as a candidate for your environment actually fixes the problem to which it is ostensibly being applied. Granted, this can be difficult to investigate for some kinds of fixes. In many cases of security situations, for example, the specific attack vectors are not available for independent verification. Downtime caused by a valid patch is one thing, but loss of service for a patch that doesn't even do what it should is egregious.

3. Patch objects in the proper order.

• As a general rule, you should patch dependent objects before the objects which depend on them. Failure to do this can cause issues that are difficult to diagnose because these cases are rarely tested and the resulting issues are seldom documented.

4. Always apply a patch the same way.

• Some platforms have a patching process that is exactly the same for most patches. Other platforms’ processes can vary wildly depending on the patch and the environment being patched. You should come up with a standardized set of steps that you follow for each patch. Add and remove specific requirements for each patch as necessary, instead of creating a plan from scratch for each patch. It is too easy to forget an important step (such as creating a backup, or requiring the proper testing before applying a patch to a production system) if you are making a new plan each time.

5. Maintain a list of approved patch configurations, test patches, and rollbacks.

• It is in your best interest to have confidence in any patch you are applying to an important system. In order to achieve this level of confidence, test the exact process you intend to follow in production. Any testing needs to be conducted in an environment that is as similar as possible to the target environments (this is as challenging as it sounds!). You'll need environments that can be created, returned to a particular starting configuration, and patched at will - and you'll need to perform each of these tasks many times. If this is done manually, it can become quite a job for each patch, let alone for the increasing stream of patches that is flowing from our vendors.

6. Monitor unpatched candidate nodes and keep track of your deployments.

• As new systems are brought online, there must be some proactive process (manual or otherwise) that recognizes the loose unpatched systems. These systems should be patched as soon as possible -- in fact, they should have been patched before going live -- in order to keep the range of deployed configurations at a minimum. Keeping track of patches in the wild is also key because you'll need to tie issues caused by (or affected by) patches you have applied in order to get them resolved in a timely fashion.

7. Save the logs (and make them verbose).

• As with other provisioning activities, patch applications and rollbacks can produce voluminous log files, especially when the tracing is set too high. These log files are the key to investigating patching issues, but oftentimes they are not collected or they are collected with no logging beyond the default. Turn up the tracing when patching if you can and save the logs; they are useful for many things, including comparing working vs. non-working procedures to see what first went wrong.

8. Perform backups before you do anything.

• No matter how likely a patch is to succeed, you should – if possible – back up everything you are patching first. Patch rollback procedures do not always work properly, and your backup will allow you to return to a functional system. If you are patching a set of binaries that runs a database, not only backup the database software but the actual data if you can - patches can require that the database contents change.

9. Go Local - avoid SEE (Shared Everything Else) errors.

• In clustered databases where the database binaries exist on each node (not shared between nodes), you should be wary of the patch application routines performing exactly the same operations to every applicable node in the cluster. The only way you can be sure that the proper actions are taken on every node is to individually do each one. In Oracle, you can pass the "-local" flag to instruct opatch to act exclusively on the local node. Keep in mind that you'll need to take care of patching all the required nodes yourself. Make sure that the system you use to patch all the nodes is more reliable than the vendor’s solution. Check out Kevin Closson's great post on this topic.

10. Use an automated service.

• Here’s the kicker: Life can be simpler! If you use the only available cross-database patching solution, all of the above is handled automatically. More importantly, the workload is drastically reduced. Your service levels will never be better.

<< back