Published: 10 Nov 2006
Surprise Winner: BakBone NetVault is leader of the backup pack
BakBone NetVault placed first in the enterprise category, besting better known backup apps such as Symantec Veritas NetBackup, EMC NetWorker and IBM TSM in the latest Diogenes Labs–Storage magazine Quality Awards survey.
Backup and recovery can often seem like eating your vegetables: You know you have to do it, but you don't have to like it. Among all of the product categories surveyed for the Diogenes Labs–Storage magazine Quality Awards, backup and recovery (B/R) has generated the lowest overall scores. User satisfaction apparently remains elusive, as our 2006 survey yielded even lower scores than the 2005 edition. There's also a greater separation between the top and bottom scores, and one out of three respondents wouldn't make the same purchase decision today if they could do it again.
Perhaps the relative antipathy toward B/R is the result of the attention it requires. "There's no backup solution in the world that doesn't require daily intervention," says Lynn Haueter, senior systems engineer at Alliance Coal LLC, Tulsa, OK. "It's not a ship that you can put on autopilot."
But not all of the B/R survey results are bleak, as three products significantly increased their scores compared to last year. BakBone Software Inc.'s NetVault, which came in near the bottom in 2005 with a weak 4.60 overall score, was a surprise enterprise-class winner this year with a strong 5.97 (see the "Overall rankings" section of the "Rankings of enterprise backup and recovery software" chart, below). NetVault's score surpassed last year's winner, CommVault Inc.'s Galaxy, even though Galaxy's score improved from 5.32 to 5.59 (second place). Hewlett-Packard (HP) Co.'s OpenView Storage Data Protector fell from second place to third with a 5.13. IBM Corp.'s Tivoli Storage Manager (TSM), Symantec Corp.'s Veritas NetBackup and EMC Corp.'s NetWorker--which combined control the majority of the B/R software market--all finished in the bottom half of the finalist group. TSM's 5.04, Veritas NetBackup's 4.68 and NetWorker's 4.66 were all slight declines compared to last year's scores. Syncsort Inc.'s Backup Express didn't have enough responses to be statistically valid, but it still wouldn't have been a contender based on the responses it received. CA Inc.'s BrightStor Enterprise Backup was not included in the 2006 survey because the company discontinued new sales of the product a year ago.
In the small- to medium-sized business (SMB) category, Symantec's Veritas Backup Exec placed first (5.14), narrowly edging out last year's winner, EMC Retrospect (5.12) (see the "Overall rankings" section of the "Rankings of SMB backup and recovery software" chart,). In the closest competition of any category thus far, Veritas Backup Exec won three categories while Retrospect snared two. Veritas Backup Exec's overall score in this category improved from a 4.99 in 2005, while Retrospect's fell from a 5.42. CA BrightStor ARCserve Backup fell from a slightly positive rating of 4.58 in 2005 to a negative satisfaction rating of 3.93 in 2006. Arkeia Software's Network Backup, Atempo Inc.'s Time Navigator, BridgeHead Software Inc.'s HT Backup and Yosemite Technologies Inc.'s Backup Standard (formerly TapeWare) were included in the survey, but didn't get enough responses to be statistically valid.
|Products in our survey|
The following vendors and products were included in the 2006 Diogenes Labs-Storage magazine Quality Award survey for backup and recovery software:
Enterprise backup software
BakBone Software Inc. NetVault
CommVault Inc. Galaxy
EMC Corp. NetWorker
Hewlett-Packard (HP) Co. OpenView Storage Data Protector
IBM Corp. Tivoli Storage Manager (TSM)
Symantec Corp. Veritas NetBackup
Syncsort Inc. Backup Express*
SMB backup software
Arkeia Software Network Backup*
Atempo Inc. Time Navigator*
BridgeHead Software Inc. HT Backup*
CA Inc. BrightStor ARCserve Backup
EMC Corp. Retrospect
Symantec Corp. Veritas Backup Exec
Yosemite Technologies Inc. Backup Standard
*Not enough responses to be included in the results.
How much with how many
For 2006, we also asked how much storage was backed up by respondents and how many administrators they used. Out of the 442 respondents who supplied this information, 131 (30%) use two administrators, while 357 (80.1%) use five or fewer. Of the 25 organizations that use 20 or more administrators, most were in the financial or IT services industries.
We were curious to see which product could lay claim as the "heavy lifter" of the industry. We defined heavy lifting as a workload greater than 20TB backed up. Based on this criterion, IBM's TSM hoists the most bytes, with 54.6% of respondents using it to back up more than 20TB. It's followed by Veritas NetBackup (43.3%), EMC NetWorker (31.4%), HP Data Protector (28.8%), CommVault Galaxy (28.5%) and BakBone NetVault (7.7%). However, Veritas NetBackup had the largest number of respondents (21.4%) using it to back up more than 100TB vs. TSM's 19.6%. NetVault, the enterprise category winner, had no respondents using it to back up more than 50TB. Among SMB products, more than 80% of respondents use the products to back up less than 10TB. Two Veritas Backup Exec respondents and one CA BrightStor ARCserve Backup respondent indicated they use these products to back up more than 100TB; there were no respondents using EMC Retrospect for more than 20TB.
Recent market hype regarding disk-to-disk backup prompted us to ask which B/R applications are used for this function. It came as no surprise that IBM's TSM had the largest number of such users (84.5%), as disk staging is part of its basic architecture. "It does it really well," says Alliance Coal's Haueter of TSM's disk-based backup prowess. But we were surprised that CommVault Galaxy was almost as high (81%), while products like Veritas NetBackup and EMC NetWorker were relatively low (49.7% and 34.3%, respectively). We were also surprised to see EMC Retrospect used for disk backup by 57.1% of its users, a high percentage for the SMB group.
"We were switching from backup to tape, to disk-to-disk-to-tape backup," says Keith Price, system administrator at Johnson & Wales University in Providence, RI. "Since we were doing a significant redesign, we went through and did a full investigation into which product would work best for us." That re-evaluation resulted in replacing Veritas NetBackup with CommVault Galaxy. "They [CommVault] came out better priced and seemed to be a better product," says Price.
The majority of backup apps are sold through resellers, so some of the survey results are more a reflection of a conglomeration of resellers rather than the vendor directly. However, the results highlight a vendor's ability to recruit, train, service and support quality resellers.
All winners of our Quality Awards have scored high on sales-force competence. BakBone NetVault did so in style this year, with a very high 6.16. CommVault Galaxy also scored well, but with a relatively distant 5.47. For the statement "My sales rep is flexible," NetVault gained its highest score (6.31). More importantly, it scored 6.15 for the statement "The vendor's licensing formula offers good value." EMC Retrospect had the second highest rating among all products for this statement with a 6.38.
Licensing (how the licenses are counted and what their price is based on: CPU, tape drives or volumes) is one of the key areas of contention in the sales process. "I just really don't like the idea that every time I add a few more servers, I've got to go buy some more licenses," says Cindy Yoho, a systems engineer at a Nashville, TN, publishing house, in reference to CA BrightStor ARCserve Backup. Yoho's company also uses HP Data Protector and she notes that "HP's licensing works a lot better for us. It's just licensed by tape drive." Price thinks CommVault's licensing is much easier vs. Veritas NetBackup. "The system monitors the licenses and you can only use exactly what you have," he says.
Interestingly, Veritas Backup Exec (with a 5.54) didn't win this category among SMB products, while EMC Retrospect had its best category score for sales-force competence (6.03). In fact, this was the only score in any SMB product category that exceeded 6.0. This was CA BrightStor ARCserve Backup's strongest category score (4.86), and the only one in which it posted a positive score.
You can consider product features to be the category where EMC Retrospect stumbled and lost the award among SMB products. In a significant change, Retrospect's score fell from a generally positive 4.62 in 2005 to a negative 3.97 this year. This category was also Retrospect's lowest rated section in both the 2005 and 2006 surveys. Had the product achieved even a neutral 4.5 rating, it would have handily won the award for the second year in a row. While Veritas Backup Exec's 4.81 was almost a full point higher, it was one of that product's lowest category scores.
EMC Retrospect's score for this section was particularly hurt by a 3.43 rating for the statement "This product is a complete solution." But perhaps more damaging was that the product didn't get a positive rating to any statement in this section, which queried respondents on management functionality, and file-system and database backup features.
Products in the enterprise category fared significantly better in this category. CommVault Galaxy bested BakBone NetVault for product features, the only category in which NetVault didn't achieve the highest score. CommVault respondents rewarded the product with a 6.07, higher than last year's 5.77. Respondents especially liked Galaxy's database backup, giving it a 6.24 for "This product's database backup features meet my needs." NetVault had its best rating (6.23) for "This product is interoperable with other vendors."
Sara Weylman, a SAN engineer at a New York-based payroll processing company, is an IBM TSM advocate in a Veritas NetBackup shop. She says that "there's nothing elegant about it [NetBackup]." Yoho notes that the Nashville publishing house where she works is only using CA BrightStor ARCserve Backup because of a single shortcoming of HP Data Protector. "There is no Mac client," she says. "That's the one thing that Data Protector doesn't do."
Alliance Coal's Haueter, another IBM TSM devotee, particularly likes the product's disaster recovery (DR) features. "It prepares a whole DR plan that tells you exactly, step for step, what process you have to go through to rebuild your TSM environment," he says.
Initial product quality
In the initial product quality section, we asked respondents to tell us about their experiences when first using a product. According to respondents, BakBone NetVault is the easiest product to deploy among enterprise and SMB products. NetVault rated a 6.38 for "This product is easy to install," while CommVault Galaxy placed second with a 5.55. The highest rated SMB product for this statement was EMC Retrospect, which placed third overall with a 5.52.
Ease of use is another important consideration, especially in a product category not noted for its easy operation. "I think the HP product is a little easier to use," says systems engineer Yoho, comparing CA BrightStor ARCserve Backup and HP Data Protector. "I wouldn't call it as easy to use as Word," Johnson & Wales University's Price says of CommVault Galaxy, "but I don't consider it any more difficult than any other backup product I've worked with."
We also wanted to know how users perceive the financial aspects of B/R products. Often, the cost of media exceeds the cost of the backup software itself. Therefore, we asked respondents to tell us how efficiently the products use tape. In this regard, respondents indicated EMC Retrospect as best overall (6.10). Among enterprise products, CommVault Galaxy was rated the best with a 5.90.
We're always interested to see the responses to the statement "This product offers good value for the money." Here again, BakBone NetVault and Galaxy led the enterprise group with ratings of 5.85 and 5.79, respectively. These are relatively high satisfaction scores for an often contentious issue that involves money. However, the category ranges down to a "Dissatisfied" rating of 4.25 for EMC NetWorker. Among SMB products, Veritas Backup Exec was rated higher than Retrospect--5.01 to 4.62, respectively. Given that CA scored a 3.63 on this statement, it's fair to say that users aren't overwhelmed by the value delivered by any of these SMB products.
"The main thing you want in your backup system is reliability," says Yoho. "They [CA and HP] don't go down, and they don't have to have a lot of patches."
Our product reliability section gauges how well each product meets service-level requirements, how often unplanned patches are required, how easy patches are to apply and so on. For this category, enterprise products had the widest spread among scores of all of the categories. BakBone NetVault achieved its second highest category score with a 6.14. The bellwether score we've detected in all of our surveys is for the statement "Patches/updates can be applied nondisruptively." In this regard, NetVault garnered an exceptionally high 6.77. Veritas NetBackup's 4.43 was the only negative reliability category rating of any enterprise product, and the product was also given a 4.01 for nondisruptive patch management. We've determined that there's a more direct connection between patch management and overall product satisfaction than any other single element.
But given the experiences of Johnson & Wales University's Price, CommVault Galaxy's 5.45 score for upgrade proficiency may be something of an understatement. "To download all the updates, it's a two-click thing," says Price. "To install the updates on every machine, it's another three clicks." He says that he was able to do a service pack upgrade on 150 servers across four campuses in approximately an hour and a half.
The overall reliability ranking for SMB products wasn't particularly inspiring either. Veritas Backup Exec won the category, but with a relatively low 4.81. EMC Retrospect wasn't far behind with a 4.76, while CA BrightStor ARCserve Backup suffered its worst category score with a very negative 3.45.
We also found that the statement "This product's error handling is easy and intuitive" revealed a low level of user satisfaction. Although not all products scored poorly, the responses to this statement were among the lower scores in the category for every vendor.
About the survey
The Diogenes Labs-Storage magazine Quality Awards are designed to identify and recognize products that have proven their quality and reliability in actual use. The results are derived from a survey of qualified Storage readers who assessed products in five main categories: sales-force competence, product features, initial product quality, product reliability and technical support. Our methodology incorporates statistically valid polling that eliminates market share as a factor. Our objective is to identify the most reliable products on the market regardless of vendor name, reputation or size. Products are rated on a 1.0-8.0 scale, where 8.0 is the most favorable score.
For this year's survey, 500 respondents provided 667 valid system evaluations. This yields a margin of error of +/- 4% with a 95% confidence factor. This pool of respondents was larger than in 2005 (465), but provided fewer system evaluations than last year's 775. Apparently, IT organizations are consolidating backup and recovery (B/R) application operations. In 2005, only 35% of respondents indicated they had a single B/R application. In 2006, that number doubled to 70.8%. In fact, 90.6% of respondents indicate that they have two or fewer B/R products this year. The highest number of products in a single organization was nine, the same as in 2005.
The range of scores narrowed for technical support. BakBone's NetVault was again the leader among enterprise products, but its 5.79 rating in this category and for product features were its lowest category scores. EMC's NetWorker trailed the group with a somewhat positive 4.69. Among the SMB products, EMC Retrospect and Veritas Backup Exec had solid technical support scores of 5.70 and 5.35, respectively.
"We don't have to call them that often," says Alliance Coal's Haueter of IBM's support for TSM. "But when we do call them, they pretty much get us what we need."
Of course, everyone would like their tech support issues resolved on the first call, but expeditious problem escalation is equally desirable. "HP has stellar support," says systems engineer Yoho. "They're very willing to escalate the problem quickly--if they can't find a solution, they immediately bump it up to someone else."
While Price hasn't had any serious problems aside from the usual implementation and learning curve issues, he senses that CommVault is trying to strengthen its tech support. "They've been hiring more people, and you can tell that calls are much shorter and resolutions are much faster," he says.
We realize that first-level technical support is often provided by a reseller or company other than the manufacturer. Therefore, we asked respondents to react to the statement "Vendor's third-party partners are knowledgeable." BakBone NetVault received a 6.62 for this statement, well above its category average. EMC Retrospect was also highly rated with a 6.38. No other product was rated above a 6.0 in this regard.
Would you buy again?
We've never found a direct correlation between user satisfaction and a willingness to repurchase the product. Not all organizations use a "best in class" purchase criterion and all of them factor numerous other considerations into their decision. Moreover, organizations can't retool their B/R environment annually and generally need a compelling reason to change.
However, for organizations consolidating operations or considering a change, and for those companies that are still developing their B/R strategy, knowing which products customers would buy all over again is instructive. In the 2006 survey, CommVault Galaxy was highest rated in this regard with 92.9% of its respondents indicating a willingness to repurchase. IBM TSM received such consideration from 82.5% of its respondents followed by HP Data Protector (82.2%), BakBone NetVault (76.9%) and Veritas NetBackup (76.1%). EMC NetWorker was last in the enterprise group with 59.7%. Among SMB products, Veritas Backup Exec had 69.7% positive responses, followed by EMC Retrospect (52.4%) and CA BrightStor ARCserve Backup (39.3%).
Although products must meet a threshold of functionality, products that delivered trouble-free operation were generally more highly regarded than those with the longest feature lists.