THE DATA ADMINISTRATION NEWSLETTER – TDAN.com
ROBERT S. SEINER – PUBLISHER

Published in TDAN.com April 2002

The recent attacks in New York and Washington has hopefully placed the importance of “Business Continuity” plans and processes in the forefront of everyone’s mind. Of course, Business Continuity is a new enough term that you may not know what that is. In short, it is your disaster prevention, planning, and recovery system. Many companies have a backup and restoration process, a virus policy, and even a systems security policy for hacker prevention. However, most company’s policies in all three of these areas are woefully incomplete. To make matters worse, these plans are scattered about the company AND they leave out one of the most important pieces, actual recovery after a major disaster has occurred. First off, lets tackle each item one at a time.

Backup and Restoration

The obvious reason for doing this one is in the case of some form of database corruption, disk failure, or even user error (someone deletes a file then realizes they shouldn’t have). If (when really) one of these happens you can recover from it no worse for the wear. However, what happens if the building that houses your data center burns to the ground? Do you have a policy to take care of that? In this case a simple backup and restoration policy is not sufficient. You need to start looking at things such as off-site storage of backup tapes. In this case, off-site storage does not refer to keeping tapes at somebody’s house or simply in another building on your campus. What happens in case of massive flood, hurricane, tornado, earthquake, or bomb? You need a secure place that is designed to house sensitive, mission critical, data. These types of places will walk you through their physical security (man-traps, metal detectors, etc) as well as natural disaster security. They should have things such as high capacity sump pumps in case of flood, heavily buttressed walls in case of tornado or hurricane, and shelving units designed to collapse together in case of massive earthquake. You should have a FULL backup going to a place like this once a week.

Only one other thing … that still isn’t enough. Think about it this way. If there has been some major natural disaster, don’t you think almost every company in your area is going to be looking to purchase new computer equipment? Don’t you think that will put a significant strain on the supply chain for computers and computer supplies? Is your building even still standing? That means you need more. What you need is a reciprocal agreement with another company in a geographically and geologically separated area (meaning one that is at least 900 miles away to either the east or the west). The agreement you make is that if some natural disaster hits your area then they will provide enough computing resources and manufacturing space to allow you to do business (basically this means take orders, process AR, process AP, do payroll, and squeak out enough orders to keep you in business and that’s it) until the computers you order can get to you and you can find other manufacturing space. As the name “reciprocal agreement” implies, you agree to do the same for them. Now, if a disaster does hit, you can get your tapes out of storage, head over to your partner company and get your company back online while your competitors are still fighting to get computer orders filled and figuring out a way they can manufacture anything in the first place.

Virus Protection

While most companies have virus scanners in place, few truly stay on top of keeping their virus signature files updated. Few really stay on top of the CERT advisories (CERT stands for Computer Emergency Response Team).

A truly robust virus policy includes not only making sure that a scanner is on every computer that attaches to your network (no matter how that attachment happens) but also includes checking for new virus signature files from your virus protection vendor daily. It also includes a daily scan of the newsgroups on the subject of viruses and keeping up to date with the CERT advisories.

Hacker Protection

This is a tricky one. Most companies have some sort of firewall to protect themselves from hackers that are coming from outside the company. The only problem is that there are now so many different types of attacks that merely putting a firewall in place is no longer enough. Additionally, most unauthorized accesses to you systems will not come from outside your company. Because entire books have been written on this subject I’m not going to go into it here. Let me say this, even supposing you are keeping up on the latest security measures, you still need to ensure that you are keeping up with the CERT advisories (www.cert.org), monitoring alt.2600 (and it’s sub newsgroups) and you are a part of the user groups for all of your hardware mission critical software. This will ensure that you stay on top of any new security holes being exploited.

Having Enough Protection

There is a final problem with most companies. The above three policies are virtually always created independently of one another. Well, think of it this way. You have a backup and restoration policy to prevent the loss of your company’s data. You have a virus policy to prevent the loss of your company’s data. You have Hacker policies to prevent the loss (and unauthorized use) of your company’s data. Anyone see a pattern here? You see, all three of them are created to prevent the loss of your data, so they should be created together. In short, they should be part of a unified “Business Continuity” document. Essentially, this document explains, in minute detail all of the processes in the even of a disaster. In this case, disaster means anything that prevents you from getting at your data when you need to. It means a virus hits even one computer on your network, an accidental file deletion, a hacker breaks through your security system, a hurricane causes the collapse of every building in your campus … anything that prevents you from doing business exactly when it needs to be done.. Getting something comprehensive like this setup can be a bit time-intensive. However, once it is in place much of the maintenance can be automated. That only leaves a little necessary hand-holding to keep the reciprocal agreement in tact I had talked about earlier. If done properly, this can be an enjoyable break from the daily grind. I would like to say now that the creation of the systems and processes should be overseen by a specialist in this area. If this specialist knows what they are doing, they will have created the following document before their work is completed.

The Business Continuity Table of Contents

This document, once fully put together will be quite thick. It should contain:

• The exact procedure on how to fully startup every server on your network. (this includes exact “click by click and word by word” description of any commands that need to be typed in after the “on button” has been hit)
• The name of your virus software
• Virus software rev level
• The phone number to the technical support for your virus software.
• Virus signature update frequency
• Location of the scripts that obtain the new virus signature file from your vendor. If you do not have scripts to do this, then it should contain an exact “click by click and word by word” explanation on how to obtain a new signature file.
• A printout of the script that performs the update (if your company uses one)
• Exact “click by click and word by word” explanation on how to launch the script
• Exact procedures for performing a manual virus scan.
• Exact procedures for what to do when a virus is found that can be cleaned from the file
• Exact procedures for what to do when a virus is found that can NOT be cleaned from the file
• The name of your backup and recovery software
• The rev level of your backup software
• The phone number to the technical support for your backup software.
• The storage location of your tapes
• The rotation schedule for your tapes, where and when you obtain a new clean tape, where to put the tapes that are being swapped out, and where and when the offsite people pickup the tapes
• The exact procedure to startup your backup software if it is not started upon system launch
• The exact procedure to perform a full manual backup
• The exact procedure to restore a file and how to find the appropriate tape the file will be on
• Exact procedures (including passwords and the like) for obtaining tapes from your offsite storage company
• A detailed description of your security system (e.g. firewalls, proxy servers, security groups, access levels, etc.).
• A listing of the rev levels for each piece of software
• The phone numbers to technical support for each piece of software
• Contact phone numbers for each of your security personnel.
• Contact phone numbers for network security EXPERTS in case you suspect a hacker attack is taking place or you suspect your security has been breached.
• The name, phone number and address to your reciprocal partner company
• The reciprocal agreement
• An English language description of what the contract says
• A detailed procedure to get your reciprocal partner company online with your data
• Detailed procedures on how to “do business” with all of your data sitting on servers you are not used to working with.
• This should include how to input orders, process them, how to run payroll, how to run AR, how to run AP, and anything else you simply must have computer access to run the basics of your business.
• Detailed procedures on how to send your reciprocal partner your data, and an explanation on how your partner gets that data.
• Detailed procedures on how to get your reciprocal partner up and running with the most up to date data possible for the systems being run on their servers.
• Finally, it should contain the dates of when you tested all of this and exactly what was tested.

Putting It Together

By looking at what belongs in the document you may have trouble putting together why this all belongs together. Well, look at it this way. If you have a hacker breach your security and they start deleting files don’t you want to know how those files get restored AND how to eliminate the attack, and possibly track them? If a virus hits and it can’t be cleaned -- same thing. You’ll need to restore the file back to a point where it was clean. If a real disaster hits and your network personnel that take care of this stuff on a daily basis are killed or otherwise incapacitated don’t you want VERY detailed procedures in place so you can get your business back up and running AND don’t you want it to be all in one place so you don’t have to hunt for four or five different documents right in the middle of a national crisis?

Now, having one copy of this thing printed out and put on a shelf somewhere is useless. If the “disaster” is a fire that guts your building and ruins your computers what are you going to do? This document needs to have several copies printed. One goes in each and every server room, one goes to your offsite storage company, another goes to your reciprocal partner, and another goes to each and every corporate officer (the “C” level folks such as your Chief Executive Officer, Chief Financial Officer, Chief Operating Officer, etc). Also, you will want this file a part of EVERY backup that is performed, and it should be sent in electronic form to your reciprocal partner. Now you can be sure that not only is your network safe, but should something truly disastrous happen you can find the document that will allow you to get your business back up and running again well ahead of your competition.