Database Automation is Demanding

By Eric Gross | December 9, 2008

There are a number of Data Center Automation (DCA) players: HP (Opsware) and BMC (BladeLogic) being the most notable. These tools provide significant operating system related content which allow them to create living servers out of bare metal boxes. This is the bread and butter of these offerings and companies can expect to generally satisfy their OS provisioning requirements with these tools. General DCA products are extensible allowing users to run arbitrary scripts that can presumably do ‘anything’.

While there is quite a lot you can do with scripts they tend to be insufficient for handling the deployment and ongoing administration of complex applications. It is one thing to demonstrate the ability to run a complicated task for a specific use case and a totally different thing to handle the thousands of unforeseen use cases that will inevitably surface. This is a key point. GridApp Systems core software product does only one thing - it knows about databases. You can’t manage databases with a single custom script (hack) that is completely static. Database requirements vary dramatically based on version, OS, and architecture.  Ultimately the complexity and associated cost of databases make them special. They are the center of your organization and they require special treatment. They are fickle and require constant TLC - the computer equivalent of a meticulously manicured garden. We know exactly what databases require, what they emit, what they inspect, what they step on, and when they are unhappy. GridApp typically knows all of these things even before anything is changed.

Ignore the marketing. Ask for a demonstration. Let your vendor attempt to demonstrate the utopian scenario: An inexperienced person should be able to sit down at a computer with a single piece of paper and deploy your enterprise standard database architecture in the time it takes for system processing plus, let’s be generous here, an hour of human time. Let them show you how they do it, and then let us show you how we do it.

Topics: Data Center Automation | No Comments »

A Day in the Life of an Oracle RAC DBA

By Eric Gross | December 4, 2008

This is where the rubber hits the road – the day to day activities of database professionals around the world who are responsible for creating and managing Oracle RAC databases. How do these peoples’ lives differ? What factors indicate how much they’ll get done and to what degree they will enjoy their work? How likely are they to meet expectations?

Ann and Bob work at a company that has so far failed to embrace database automation. Paul and Raul work at a more forward-thinking company which has invested in database automation. Ann and Paul are both extremely experienced DBAs who have been working with Oracle RAC since it was released while Bob and Raul are rather green. Ann and Paul can both create RAC databases because they have the required skills. In order for Ann to offload the RAC database work to Bob she would need to document the entire procedure that could be followed in her absence. Raul, just like Bob, does not know how to create a RAC database but since automation has been put into use he will be able to do so without comprehensive documentation – he just has to use the automated solution.

Name Company Skill Level
Ann Company F Experienced
Bob Company F Novice
Paul Company A Experienced
Raul Company A Novice

This is the story of a typical day in the life of Bob and Raul.

Bob gets into work and finds a beautifully bound two-volume book containing every step that needs to be run sequentially, along with each step’s prerequisites and required output. There are approximately one thousand commands to be run. In the event of any misconfiguration or procedural error the process will need to stop until Ann can be involved to determine the issue and provide Bob information how he can proceed. Since these things are complicated, Bob gets three hundred pages into the document, experiences an unexpected result, gives up, emails Ann, and goes home. Tomorrow’s another day – maybe the database will work soon.

Raul’s day is rather different. He gets in to a note on his keyboard asking him to create another RAC database. Unfortunately he did not receive a beautiful book but he’s not afraid. Raul accesses the database automation software and, using the details provided by his in-house expert, Paul, he creates the database without accessing the database server for even a moment. Raul has accomplished a great task before lunch so he is able to spend the rest of his day working on tuning SQL and running reports requested from the business unit.

Which DBA do you think adds more value, Bob or Raul?

Topics: Database Automation | No Comments »

Can I Apply an Oracle Security Patch to a RAC Database Without Taking Downtime?

By Eric Gross | September 5, 2008

In short, it depends. The first step involved with applying an Oracle CPU is using the OPatch tool to apply the patch to the required Oracle Homes. This requires that no processes are running out of the home at the time of patching. This requirement does mean that the instances running on one instance on one node have to become unavailable for at least as long as it takes to patch the binaries. Since a RAC database can be alive on multiple instances simultaneously, other nodes can service the database during this part of the process. So the binaries can be patched without taking any downtime.

Similarly, the data dictionary can be updated while the database is online, on multiple instances without any downtime; this part of the procedure also supports not taking downtime across the database. Following that, the suggestion is to recompile all invalid objects online.

Finally, the view recompilation phase; this only has to be run in certain circumstances based on the history of the database, and it only has to be run once during the life of the database. This process does require that the database be altered so that CLUSTER_DATABASE is false and opened in a way such that no connections are allowed. This clearly meets the definition of downtime. It depends if your downtime is required for application of a CPU based on your environment.

Topics: Oracle, Patches, Security, Uncategorized | No Comments »

Network World: Database administrators in high demand

By Eric Gross | August 28, 2008

Matt Zito was quoted in this week’s IT Careers and Training Alert by Jon Brodkin.

Click to read the full article.

Topics: Data Center Automation, Data Management, Database Automation, People, Uncategorized | No Comments »

eWEEK.com Podcast: Automate Mundane Database Tasks to Increase Value

By Eric Gross | August 26, 2008

Check out the following eWEEK.com Podcast: Mike Vizard talks with Robert Gardos, GridApp CEO.

http://www.eweek.com/c/a/Storage/Automate-Mundane-Database-Tasks-to-Increase-Value/

Length: 00:19:21

Database administrators spend too much time performing mundane database tasks, such as deploying, patching and troubleshooting, and managing a diverse range of databases, rather than using the data to create value for the company, says Robert Gardos, CEO of GridApp Systems. To get more out of your DBAs, companies need to drive more automation, efficiencies and standards into database management.

Topics: Clarity, Data Management, Database Automation, Efficiency | No Comments »

How do I know when to patch or to leave well enough alone?

By Eric Gross | July 23, 2008

Every business wants the same thing: a database that services clients with the utmost performance and is always available. In the war against downtime, patches can be a rogue force. When something goes wrong, patches may let instability—the enemy—in through the front door. Patches always promise to fix a problem. If not for that promise, there would be no reason to let such an untested piece of code into your environment. Before you let any patch into your camp, you need to make sure it won’t have a malicious impact or any other negative effects. It makes sense to initially let it only into a bastion where you can analyze its every impact. Patches should always be held to the standard of providing a positive result—not creating any negative effects which will outweigh the intended result. If possible, businesses should always wait for the subsequent patchset, which normally contains all recommended patches made public since the last release. If it’s not feasible to wait that long, a patch bundle may be an alternative since bundles can be more widely deployed and tested.

Whenever a patch presents itself, you should vet it just as you would challenge any other change to a working system. Sometimes patches turn into candidates for installation based on a clear-cut symptom a specific patch has been issued to address. Other times the confidence that a particular patch will fix a symptom is lower, and sometimes, it can be a shot in the dark. It is always prudent to search the web for patch-related issues specifically related to that patch. Beyond that, take advantage of the limitless number of service requests available to you to confirm all implications of using this patch. You may find during your interaction with product support that you are missing a prerequisite for patch installation. This approach makes even more sense, because Critical Patch Updates clutter the landscape in different ways based on the releases you are running. No matter how much confidence you have that a patch will remedy an issue, you should always test it in an environment that is as similar as possible to the production case. A point patch can break seemingly unrelated things, so testing needs to include a full regression suite (in addition to the obvious and critical confirmation that the patch fixes its intended symptom).

Patching is one of the endless reasons that you should minimize the number of configurations in play at any one time. Since you’ll need to test each candidate environment for each patch, you should—if possible—reduce the number of unique configurations. Once you have decided to apply a patch, the next step is creating a procedure that you’ll use to deploy and rollback this patch. Ideally this script will be completely automated so that one command takes care of the entire apply/rollback use case: stop dependent processes, backup, apply patch, and restart processes. All of this should be performed in addition to your custom requirements—those related to change control or regulatory requirements. Each part of the procedure that is manually executed increases the probability of failure and inconsistent results. Equally important to the application process is the rollback process. This can be required without notice and in critical situations where unattended work is far preferable for its speed and reliability. Backups are a fundamental part of most rollback procedures, so you should be confident in your ability to restore to a working state.

Remember: Once a patch is let loose, it is important that it be monitored. This crucial final step ensures that the intended results are realized and any negative effects are tracked and potentially resolved.

Topics: Database Automation, Patches | No Comments »

Deploying SQL Server 2000 onto Windows 2003 clusters with more than 4 nodes

By Devin Heitmueller | June 11, 2008

So you’re deploying a clustered instance of SQL Server 2000 onto two nodes in a Windows 2003 cluster. The installer runs to completion, but all of the resources in your newly created cluster group are in the offline state, and any attempt to bring the group online results in an unknown failure.

In many cases, this is the result of some unexpected behavior in Microsoft’s license enforcement. SQL Server 2000 documentation states that it supports instances of up to four nodes. However, if you are running Windows 2003, you can have Microsoft clusters up to eight nodes.

Where the documentation becomes unclear is that the limit is not on the number of nodes the instance is deployed on, but rather on the total number of nodes in the Microsoft cluster. So if you have an eight node Microsoft cluster, with the intention of putting four 2-node instances onto it, then you will hit the above failure condition.

It would have been nice if the SQL Server 2000 installer would just tell you this is the problem when you try the above scenario. Unfortunately, it just proceeds with the install and then the instance doesn’t startup. Here at GridApp, we didn’t realize what was causing the problem until we dug through the error logs and put the following string into Google.

[clushelp.cpp:150] : 259 (0×103): No more data is available

Only then do we arrive at the Microsoft Knowledge base article describing the issue (KB811054).


— The Clarity Advantage —
GridApp Clarity has a precheck built in to automatically check how many nodes are in your cluster if you are deploying SQL Server 2000, and to show you an error message informing you of the problem before the install starts. This allows you to correct the problem (such as picking a different cluster to use or making the cluster smaller) before you deploy a broken instance and spend the time trying to figure out why it won’t startup.

Topics: SQL Server 2000 | No Comments »

Boring Jobs Dull the Mind (and Insight Failures)

By Eric Gross | June 9, 2008

Here is an interesting article:

BBC News: Dull jobs really do numb the mind

New research shows that people who are asked to perform repetitive tasks eventually go into “autopilot” mode in order to economize the work required to complete a task. I know this is true for me - ask me to create 20 databases and after the third one, I’m in drone-mode. Unfortunately, after having completed it a few times, I know I’ll eventually miss a step because I’ve pretty much stopped thinking about the procedure. Drone-mode means my mind wanders elsewhere - such as my next post to the blogosphere.

Brain scans can predict when someone has gone into this minimal thinking state. So if you want to improve the quality of your database management team you could fashion a hockey helmet with EEG electrodes, pop them on your DBAs, and start monitoring when they go into drone-mode.

Better yet, either enrich their jobs or - *gasp* - try automation!

Topics: Data Center Automation, Efficiency, People | No Comments »

Unleash Your “cognitive surplus”

By Eric Gross | June 5, 2008

A recent meme - cognitive surplus - has risen that describes the resources that have been brought to bare to accomplish a useful task once people get tired of wasting their time. The discussion started with a talk that shows how people have so much free time to work on improving the Wikipedia (the estimate is that about 100 million human-hours has gone into the project so far). Imagine if your car suddenly learned how to drive itself, leaving you to be able to do what you wish with your commute - you would suddenly have a few free hours per day for accomplishing something new - perhaps a college course, reading the paper… even putting on your make-up or chatting on the phone without fear of a ticket or accident.

Imagine then if your databases could “drive themselves”, while database vendors have been pumping out databases for decades, only recently has extensive automation been available to drastically reduce peoples’ needs to manually perform common tasks. GridApp’s automation technology reduces the human time required to do everything from provision to patch.

What could you do now with the cognitive surplus released by automating your database stack? There are plenty of tasks that DBAs can still do which are not yet fully automated such as performance tuning and planning for future growth/migrations - coincidentally, tasks which make money for the company. Or perhaps it leaves more time for applying the often-ignored security patches or testing backups and restorations.

GridApp has automated many of the common database tasks that suck up your team’s valuable time and energies; this not only improves database management efficiency, but also yields increased value from each resource because of the intrinsic standardization that comes with automation.

Never leave your potential surplus on the table!

Topics: Agile, Clarity, Consistency, Database Automation, Efficiency, People | No Comments »

Only 10% of Oracle Databases are Secure!

By Eric Gross | April 29, 2008

A recent article in eWEEK revealed alarming statistics surrounding patch and CPU application. The survey was conducted by a company that appears to benefit from the patch application process being as painful as possible (they sell a solution that attempts to mitigate security risks surrounding non-patched databases). Back to the survey, here are a few of the more disturbing stats:

What the survey data actually reveals is a deficiency that screams to be solved with patch automation. Automation reduces time-to-patch dramatically by separating the definition/packaging of a patch (including any pre & post scripts that may be a part of an organization’s standard for patch activity), from the actual process of installing the patch.

The benefits of this type of automation are immediate and quite clear:

Don’t get me wrong, the fault is not with the DBA. This security-catastrophe-waiting-to-happen is being caused by the delayed adoption of automated database administration practices, and this falls squarely in the lap of an organization’s decision makers.

Topics: Data Center Automation, Database Automation, Efficiency, OPatch, Patches, Security | 2 Comments »


« Previous Entries Next Entries »