« PRKO-2016 Errors in CRS Clusters | Main | Back on the Train »
Time to Automate
By Eric Gross | August 22, 2007
At this point we’re all on the same page that automation is essential for properly managing the database layer. Many factors play into this, including the number of steps involved with performing the activities at hand, the number of inputs involved, and the difficulty of verifying that any particular activity is successful. Being able to determine which activities are successful is quite a step up from what is generally done in the “wild”.
Typically, database management steps are performed one after another until a failure becomes obvious to the operator, at which point an investigation is required to determine which activity did not perform correctly. Given that there can be thousands of clicks and key presses between a bare-metal server and a completely configured database layer, it is impossible for a person to be able to reliably complete these activities in a consistent fashion. This is especially true when other business units are responsible for fundamental aspects, such as server provisioning, OS and update provisioning, and storage. It is critical that all of these requirements are validated before any database activities are undertaken. Checking all of these inputs manually is impractical, based on the number of verifications, especially when the input is correct 99% of the time. This high success rate is an invitation for verifications to be ignored, and in turn creates an unreliable system.
Creating an automation framework is the first step toward implementing a complete automation solution, but it is only a small subset of the overall work required. An automation framework can be created to handle certain situations, such as the deployment of an entity to a server, a script that can be executed in the context of an existing object, and the removal of the object. However, it is quite challenging to create a framework that can handle any number of the same entities on a particular server, and most solutions don’t measure up. Beyond a sufficient framework that is able to handle the particulars of the managed database entities, code must be written to actually interact with the applications to be managed. This is a huge challenge for all of the same reasons it is a challenge to manually manage the database layer. Interacting with a complex application, such as Oracle DBMS and Microsoft SQL Server, requires running a large number of commands, each with unique arguments and each in a unique way to determine if the task was successful or not – frequently requiring the execution of additional commands just to judge the results of a prior act.
Each organization also has their own customizations beyond the scope of managing the actual database entities, such as clusters and database instances. These requirements can match or exceed the complexity of the core automation required to harness the database applications. In many cases, preexisting work can be leveraged to automate the work around the core database elements. Many times, these steps need to be adapted so as not to require user interactions. User interactions were acceptable in the past, though now that the entire process can be automated, these ancillary tasks need to be brought under the management of the automation software.
An example that calls out for automation in the database layer from the Microsoft SQL Server side is the creation and maintenance of clustered SQL Server instances. Before installing the instance there are a multitude of checks to be performed beyond those included in the installation utility. These checks must be run on all the nodes that will be added to the cluster, so that any missing requirements can be found before starting the work. Next the base install is performed to all the nodes. There are a number of steps that must precede this, such as the stopping of certain services and moving the resource group to hold the new instance to the local node. After the installation is complete, there is still the need to install a service pack and potentially one or more hotfixes. Each of these steps requires activities to take place on each of the nodes participating in the cluster to confirm that the updates have been sent properly.
It is not a wise use of your time to design/implement your own automation framework and scripts to harness the desired database applications. It’s unlikely that you are the only candidate for the automation, thus if you are considering automation, it’s best to buy rather than build this aspect of your infrastructure. There is a constant stream of work required to update an automation framework based on new requirements. Beyond that, the actual code used to call out to the managed applications is even more fluid due to updates from the software which are managed and therefore less attractive for internal development.
Automation is an essential part of running an effective database environment. If you have asked yourself when your organization should automate, the answer is now—you can only benefit from customized database automation.
Topics: Uncategorized
December 4th, 2007 at 12:32 pm
[...] Automation is easy to start but hard to complete, at least without our software. [...]