Each year, Storage magazine publishes its Quality Awards, which is an awards program that seeks to identify and recognize the best products that have proven their quality and reliability in actual use.
The results are derived from a survey of qualified readers who assess products in five main categories: sales-force competence, initial product quality, product features, product reliability and technical support. Our methodology incorporates statistically valid polling that eliminates market share as a factor. Indeed, our objective is to identify the most reliable products on the market regardless of vendor name, reputation or size.
Products were rated on a scale of 1.00 to 8.00, where 8.00 is the best score. A total of 358 respondents provided 567 backup and recovery software evaluations.
And the winner is...
In the past, EMC Corp.'s Retrospect has won top honors in the midrange backup and recovery category, but this time, Acronis Inc.'s Acronis Backup & Recovery topped the field in four of five categories.
Backup and recovery software is the only software product group in the Quality Awards program, and over the many surveys Storage magazine has conducted, software seems to be judged more critically than hardware products, with ratings that are typically (and sometimes significantly) lower. But if there's any solace for the software vendors, it's that the scores have been steadily, if modestly, climbing since our first survey. In fact, Hewlett-Packard (HP) Co.'s second-place score of 6.09 would have been good enough to win three of the four previous enterprise surveys.
The story's the same for midrange backup programs. Symantec's second-place overall score of 5.47 for Backup Exec would have bettered the winners of the first three Quality Awards surveys, including its own win in the second survey.
Other promising signs for data backup and recovery software vendors are the improvements in the average scores for product features. Over the first three midrange surveys, those numbers languished in the 4.18 to 4.74 range, which, on our 1.00 to 8.00 rating scale, indicated the functionality of the rated apps was middling at best. The enterprise-class apps fared slightly better. Satisfaction with product feature sets, however, seems to be on the rise; for enterprise apps, the average feature scores on the last two surveys have been impressive, just barely exceeding 6.00 both times. Midrange products have also shown improvement, although less impressively so, with feature ratings finally climbing over the 5.00 mark.
Buying data backup and recovery software
Any IT purchase can be a gut-wrenching experience, not just because the buy might involve some big bucks, but because each purchase is a commitment. For data backup and recovery software, the commitment goes beyond time, effort and other resources as you're also committing your company's data to the product. So the sales process is particularly critical for a data storage manager who's about to lock into a particular backup technology.
CommVault scored highest among enterprise backup applications in the sales-force competence judging category with a 6.20; HP, which proved to be a game competitor for CommVault, was second with a solid 6.10, well ahead of third-place finisher EMC NetWorker (5.85).
CommVault scored highest on six of the seven statements in this category, with its best ratings for "The vendor's sales support team is knowledgeable" (6.64) and "My sales rep is knowledgeable about my industry" (6.33). HP also did well on those statements (6.06 and 6.13, respectively), but topped CommVault -- 6.63 to 5.91 -- for when the sales process gets closest to signing on the dotted line ("The vendor's licensing formula offers good value"). All of the vendors fared well for having a knowledgeable sales support team, with EMC (6.37), IBM (5.77) and Symantec (5.65) getting their highest scores in this category for that statement.
The midrange products didn't fare quite as well in the sales-force competence category. Acronis picked up a couple of 6.00-plus ratings on its way to an category winning score of 5.65, but neither Symantec (5.36) nor CA (4.77) could muster one. Smaller businesses may be tougher customers for backup software vendors, but Acronis seems to be following the lead of its enterprise-class brethren, with its highest scores coming for those statements related to having a knowledgeable sales support team (6.11) and offering favorable licensing terms (6.11).
Is your backup software easy to install?
In four of the five evaluation categories for enterprise applications, HP nipped at CommVault's heels but ended up in second place. But in the initial product quality category, HP turned the tables to outdistance CommVault, 6.13 to 5.93, coming out on top for four of the seven statements in the process.
With software products, the key to long-term satisfaction is getting out of the gate fast with a quick and relatively pain-free installation. HP is apparently doing a good job of setting the tone early with its users by having a product that installs easily without customers requiring a lot of help from HP's professional services team. HP received its highest score in this category for the statement "This product was easy to install," and was the only vendor in the group to rise above a score of 5.00 for "This product did not require professional services."
Although it finished second to HP, CommVault was no slouch in the initial product quality category, with especially high ratings for "This product uses tape efficiently" (6.55) and "This product delivers good value for the money" (6.48). Its overall score was dragged down by a 4.52 for the statement about requiring professional services.
Arguably, midrange backup products should be easy to install and get up and running simply because the users of these products are likely to have fewer resources available to get through the initial stages of implementation. All three products showed their mettle in the initial product quality category, with Acronis (5.99) coming out on top, followed by Symantec (5.82) and CA (5.76); in addition, all three had 6.00-plus scores for the critical "easy to install" statement. Acronis scored very well for all seven statements in the category with one glaring exception -- a 4.00 for "This product uses tape efficiently," which, ironically, was where Symantec racked up one of its highest marks on the way to a very respectable performance.
A no-sweat implementation may indeed set the tone for a user's overall experience, but a backup application still has to deliver the goods with the required features and functionality. With the highest score recorded for the product features category (6.70), CommVault proves that there's plenty of substance behind the style of its backup suite. Its victory in this category is a tour de force, as it scored the highest for every statement, highlighted by the only over-7.00 score in the survey, a 7.02 for "This product's file system backup features meet my needs." CommVault also scored impressively for database backups (6.88), for being a "complete solution" (6.81) and for its disk backup functionality (6.80).
Second place HP also did well for features, with its overall 6.14 score bolstered by strong showings for database backup (6.45) and file system backup (6.38). IBM's Tivoli Storage Manager was hard on HP's heels with a category average rating of 6.12, highlighted by a 6.52 for its disk-based backup features. Symantec's NetBackup made a strong showing with an overall 5.89 built on three 6.00-plus statement scores.
Once again, survey respondents were a little tougher with their ratings for midrange backup apps, with the products earning scores lower than their enterprise counterparts. Symantec's Backup Exec showed why it's one of the most widely installed midtier backup applications by topping its competition on seven of the eight statements in the product features category for an average winning score of 5.68. Backup Exec did particularly well for file system backup features (6.05), backup to disk (5.99) and database backup (5.85); its lowest score -- a 5.27 for archiving features -- was just a hair shy of Acronis' score of 5.31. Acronis' Achilles' heels in this category were database backup (5.17) and scalability (5.17).
Backup software reliability
The true test of a backup application comes with the daily grind of protecting a company's data assets; first impressions and feature sets are all well and good, but long-term reliability and stability are paramount. CommVault was the only enterprise-class backup app with a 6.00-plus average for the product reliability rating category, picking up a 6.24 while leading on six of eight statements. Its strongest showings were for meeting service-level requirements (6.86) and operating system/platform support (6.84).
Second place HP (5.94 overall) also showed strength for those two reliability criteria (a 6.18 and 6.59, respectively) among its consistent marks in this category. For the two statements that CommVault didn't have the leading score, EMC's NetWorker led the pack with a 6.07 for "Requires very few unplanned patches/updates," while IBM Tivoli Storage Manager garnered a top mark of 5.87 for "Very few bugs."
Within the midrange product set, Acronis bested CA (5.38) and Symantec (5.34) for five of the eight reliability statements on its way to winning the category with a 5.68 average. This was CA's sole second-place finish; its ARCserve Backup product came out on top for meeting service-level requirements (5.94) and for requiring few unplanned patches or updates (5.83). Backup Exec finished a close third with a winning score of 5.17 for "Vendor provides comprehensive upgrade guidance" -- an area where all three vendors' ratings were relatively low, indicating that some improvement in the upgrade process would be welcomed by users.
Does your backup software vendor have good support?
Backup software vendors have labored diligently to make their products easier to use with improved GUIs, wizards and other interface aids. But with an application as complex as backup, users will invariably need some help from time to time, so how well a vendor supports its product will figure significantly in a user's perception of ease of use. After backup vendors as a group registered a disappointing overall average of 4.88 in the first Quality Awards survey for backup software, they have shown steady progress in the quality of support they provide, with the best numbers yet recorded this time around (a very healthy 5.89).
CommVault's 6.20 score for the technical support category was enough to nudge out HP's 6.12. IBM (5.82), EMC (5.73) and Symantec (5.58) turned up pretty good ratings as well. CommVault is apparently meeting users' support expectations, as its highest score in this category was a 6.86 for "Vendor supplies support as contractually specified." HP topped CommVault on the statements dealing with documentation (a 6.15 vs. CommVault's 6.00) and "Support issues rarely require escalation" (6.00 and 5.69, respectively). EMC was rewarded with a statement-high score of 6.27 for training its third-party partners well.
Again, we detect some grumbling among midrange users as they aren't nearly as satisfied with their vendors' support efforts. Although the overall support score average is higher than it's ever been, it's still a rather unimpressive 5.23.
Acronis topped the category with a modest 5.39 overall support score, but did manage to get the only 6.00-point rating in the category with an even 6.00 for providing support as contracted. All three vendors failed to reach the 5.00 plateau for providing adequate training, with Acronis earning a 4.80, followed by Symantec (4.74) and CA (4.59). And the three didn't fare much better for providing adequate documentation, with scores ranging from Acronis' 5.05 to Symantec's 5.08, with CA sandwiched in between at 5.07. The message to midrange backup software vendors seems clear: Provide more support and you're likely to see better all-around scores on future surveys.
As on all Quality Awards surveys, we ask respondents if, given what they now know, they would make the same purchase again. Given its overall performance on this survey, it's not too surprising that CommVault had the highest percentage of users who said they'd do it all over again (83.7%). Only 62.9% of HP's users said they would buy Data Protector again, a surprisingly low number considering how well the product was rated overall.
Although not necessarily related to user satisfaction, it's interesting to see how heavily users are taxing these backup products. Among enterprise programs, IBM's Tivoli Storage Manager is the workhorse, backing up an average of 67.1 TB weekly. For the midrange apps, Symantec's venerable Backup Exec is the heaviest lifter with an average of 13.3 TB of data backed up each week.
Products included in the survey
The following backup and recovery software products were included in the survey. The number of responses for finalist products is shown in parentheses.
Asigra Inc. Televaulting*
BakBone Software Inc. NetVault:Backup*
CommVault Systems Inc. Galaxy (or Simpana) (48)
EMC Corp. NetWorker (36)
Hewlett-Packard (HP) Co. Data Protector (43)
IBM Corp. Tivoli Storage Manager (81)
Symantec Corp. NetBackup (81) Symantec NetBackup PureDisk* Syncsort Inc. Backup Express*
Zmanda Inc. Amanda Enterprise*
Acronis Inc. Backup & Recovery (33)
Arkeia Software Arkeia Network Backup*
Atempo Inc. Time Navigator*
BarracudaWare Yosemite Server Backup*
BridgeHead Software Ltd. HT Backup*
CA ARCserve Backup (43)
Double-Take Software Inc. Double-Take Backup*
FalconStor Software Inc. Continuous Data Protector*
Microsoft Corp. Data Protection Manager (DPM)*
PHD Virtual Technologies esXpress*
Symantec Backup Exec (108)
Veeam Software Veeam Backup & Replication*
Vizioncore Inc. vRanger Pro*
*Did not receive enough survey responses to be included as a finalist.
About this author: Rich Castagna ([email protected]) is editorial director of the Storage Media Group.
This article was originally published in Storage magazine.