What you will learn in this tip: Branch offices often don't have experienced technical staff to deal with data backup and recovery chores, making branch office backup a tricky issue for enterprises. Learn about the best remote office backup and recovery strategies.
Backing up and restoring remote office/branch office (ROBO) data is very much a different situation than doing so for the primary data center. Whereas the primary data center typically has dedicated backup and recovery administrators, procedures and even periodic testing to ensure data is recoverable, ROBOs usually do not. Someone in the ROBO, who is most likely not part of the IT organization, handles the data backup responsibilities as a secondary responsibility.
Common sense would dictate ROBO backup procedures and processes with this in mind, insuring successful data backups. Regrettably, it does not. Backup is a means to an end, with the end being able to restore or recover the data should it be lost, deleted, corrupted or infected. It's hard for many backup administrators to remember or even realize this basic truth because at a bare minimum, they perform backup tasks every single day. Restores/recoveries, on the other hand, are a lot less frequent, and that's with dedicated primary backup admins. The remote backup experience is typically an order of magnitude less.
Since recoveries are always urgent, it is logical to assume that IT organizations frequently test their ability to recover data. That's an incorrect assumption, especially for ROBOs. A backup admin at a large IT organization (that preferred not to be named) said at the June 2010 Chicago Storage Decisions conference that although they back up their ROBOs to tape every single night, they have no confidence they could recover the data in the event of a disaster, and was frightened to even test their backups. What's really scary is that this is not atypical.
The most common reasons (beyond the already discussed lack of dedicated personnel and inexperience) include: tapes not being replaced when full with backups being overwritten; tapes being mishandled or mislabeled; tapes not sent offsite or sent offsite via amateurish means (loaded in the trunk of an employee's car); unclear backup procedures; insufficient disk capacity when backing up to disk (again overwriting some or all of the backup job or previous backup jobs); backup windows not being met causing the admin to abort the job; and corrupted backups.
Fixing your branch office backup blues
What can IT organizations do about these branch office backup headaches? An IT organization could bite the bullet and hire dedicated branch office backup admins. But that only solves some of the problem. The backup and recovery technology at the ROBO should also be upgraded to disk, virtual tape library or deduplicated disk that can replicate back to the primary data center as well or backed up directly to cloud storage. This provides for both local and disaster recoveries, increasing the probability of success. Most backup and recovery software can work with these types of storage targets as well as tape. But this can be an expensive solution.
Another solution is to centralize the backup and recovery disciplines, processes, and procedures within the data center. There are two ways to do this. The first is to centralize all of the storage in the primary data center. In this manner, the requirement for remote backup and recovery is eliminated because all of the data resides in the data center and utilizes data center backup and recovery. The second is to utilize backup software capable of centralizing and controlling all backup disciplines, processes, and procedures from the primary data center. Some products that can do this are Asigra Inc. Cloud, CommVault Simpana, products from FalconStor Software, and Symantec Corp. NetBackup. The key advantage to this is that they take most of the ROBO personnel out of the equation. But there are unintended side effects as well.
Centralizing the storage is subject to increased end-user response times because of speed of light latency, TCP latency, and WAN packet loss, resulting in significantly reduced productivity. Minimizing or eliminating that side effect requires WAN optimization controllers/appliances or virtual equivalents. This once again adds cost. Depending on the number of ROBOs, this can be very expensive.
Using backup and recovery software to centralize and control the branch office backup fixes most backup problems. But this typically makes recoveries significantly longer because they are done over the WAN. The WAN is bandwidth, latency and packet loss constrained. This can be mitigated with WAN optimization (physical or virtual controllers). The cost would depend on the amount of ROBO offices you have. It can also be mitigated at a cost by providing a hybrid of local and centralized backup by keeping the most recent backups on local disk target storage (deduped or not), and moving all backup data to the primary data center backup vault.
Cloud backup for remote offices
Another answer is to outsource branch office backup to a cloud backup service. Cloud backup services often provide a mix of local physical or virtual appliances to provide local fast restore/recoveries for recent revisions of backups, as well as remote vaults for all the data in case of a disaster. And best of all, contractual service-level agreements (SLAs) help keep someone's feet to the fire to ensure the ROBO recovery point objectives (RPOs) and recovery time objectives (RTOs) are met. This answer also eliminates remote office backup personnel. The cloud backup service takes the place of data center central repository.
Going the cloud backup route requires research, customer referral verification, and for you to do your homework. Cloud backup and recovery service providers should guarantee customer data security. They should be able to wall off and isolate each client's data. They should also provide extensive encryption -- preferably FIPS 140-2-certified encryption -- to make sure even if an accidental access occurs, the data cannot be read. It's important for them to be able to provide customizable RPOs, RTOs, retention policies, and a mix of local and remote recovery capabilities. But all of this comes at a price. Cloud backup services are usually priced based on the amount of data stored in their facilities.
One way to mitigate that cost of a cloud backup service is to negotiate a fixed price per quarter, semiannually, or annually where the cost is negotiated to a fixed price at the end of each time period. Another way to mitigate costs is to utilize tiered cloud backup, taking advantage of local storage for fast local recoveries controlled by the cloud backup service provider; online cloud storage for disaster recoveries and older restores; and nearline storage for non-mission critical data. One other way to mitigate costs is to stop keeping everything forever. Having realistic data retention and destruction policies significantly reduces the amount of data stored. Then enforce them vigorously. These methods are very effective especially when combined.
Effective branch office backup today is not common, especially the recovery aspect of it, but there are ways to fix this problem. Choose the one that works best within your IT ecosystem, mores and budget.
About the author: Marc Staimer is the founder, senior analyst, and CDS of Dragon Slayer Consulting in Beaverton, OR. The consulting practice of more than 12 years has focused in the areas of strategic planning, product development, and market development. With over 30 years of marketing, sales and business experience in infrastructure, storage, server, software, and virtualization, he's considered one of the industry's leading experts. Marc can be reached at [email protected].
26 Jul 2010