Published: 12 Sep 2008
Although users gave favorable marks to most enterprise and midrange backup software products, they still have concerns about product complexity and usability.
If the results were any tighter, they'd squeak. The 2008 edition of the Diogenes Labs-Storage magazine Quality Awards for backup and recovery software recorded lower high scores, but substantially improved low scores. In past surveys, midrange backup products scored significantly lower than enterprise-class products. This year, the enterprise products still rated higher, but the discrepancy was much smaller.
CommVault's Galaxy solidified its position as the top enterprise backup product, repeating last year's win and finishing first for the third time in the four years we've conducted the survey.
Galaxy's overall winning score of 6.01 came with the slimmest possible margin--.01--to barely nudge out Hewlett-Packard (HP) Co.'s Data Protector. HP has nipped at the winner's heels in past surveys, and its final tally of 6.00 left it just shy of an upset victory. CommVault and HP each won two of the five survey categories.
IBM Corp.'s Tivoli Storage Manager held onto third place again and improved its score to 5.85 from last year's 5.72. EMC Corp. NetWorker showed the greatest improvement in the enterprise group, climbing from a 4.88 last year to 5.42 in this survey--a high enough score to slip by Symantec Corp.'s Veritas NetBackup (5.40) and take fourth place. Syncsort Inc.'s Backup Express, which was a finalist last year, had too few survey responses to be considered for this year's award. BakBone Software Inc.'s NetVault:Backup, the 2006 winner, also failed to garner enough responses to be in the running.
Among the midrange backup products, EMC Re-trospect retained its crown as the group champion. Its 5.56 score outdistanced second place Symantec's Backup Exec's 5.32. EMC took top honors by winning every section of the survey. Retrospect, Backup Exec and CA ARCserve Backup finished 1-2-3 in all sections of the survey.
While CA ARCserve finished third in the midrange group once again, it deserves some sort of honorable mention. Two years ago, ARCserve finished with a very negative 3.93. Since then, its scores have steadily risen and it finished with a positive overall score of 4.60 in 2008. In our 1.0-8.0 scoring system, a score of 4.50 represents the demarcation between positive and negative ratings. CA's scores were above 4.50 in three of the five categories, and only slightly negative in the other two. Clearly, CA has its product moving in the right direction.
Of all the product categories we survey for the Quality Awards, backup and recovery software is apparently the least loved. Last year, there were a number of negative comments from respondents. This year, negative comments outnumbered positive comments by approximately 4:1. Moreover, one-third of the respondents say they wouldn't make the same product purchase decision again.
In cases where respondents provide evaluations for two or more products, we compare these head-to-head matches to determine which vendor is preferred. This year, the results were quite surprising. Among enterprise products, NetWorker was involved in 29 direct comparisons and won 62.1% of them, the highest percentage among enterprise products. ARCserve was in 26 such match-ups and won 57.5% of them. Note that head-to-head comparisons could be against any product in our survey, not just the finalists.
Sometimes sales competence isn't measured only by the reps that show up in your shop. "Our new sales team is still kind of learning what we do," says Chris Wible, backup and recovery lead at Richmond, VA-based Performance Food Group, of the Symantec reps supporting his Veritas NetBackup and Backup Exec installations. But Wible was pleasantly surprised when an inside sales rep contacted him to follow up on an inquiry he made about adding a software option. "She called up the people that I needed to talk with for Veritas NetBackup PureDisk, scheduled the meeting and got it going," he says.
Licensing schemes are often a source of user ire. We asked respondents to tell us about their vendor's licensing methodology. HP had the highest score for "This vendor's licensing formula offers good value" (5.97). One Data Protector user notes that HP's "licensing structure is very generous, very cost-effective and it affords us a tremendous amount of flexibility." CommVault was the subject of several unhappy comments regarding its licensing formula ("the licensing model makes the product expensive to own"), but received a decent 5.46 for this statement. Veritas NetBackup had the worst score among enterprise products for this statement with a 4.76.
EMC Retrospect led the midrange group with an overall sales score of 5.24, while Backup Exec and ARCserve scored a 5.20 and 4.77, respectively. Retrospect had an interesting dichotomy with its highest section score (a 5.50) for the statement "My sales rep is flexible" and its lowest section score (a 4.86) for "My sales rep is easy to negotiate with." Backup Exec had the highest midrange score in this section for "My vendor's sales support team is knowledgeable" (5.77).
Backup Exec users have struggled at times with the product's licensing process, both before and after Symantec acquired Veritas. "The notification of releases, the license renewal process was an absolute nightmare," says Esmond Kane, senior manager of information systems at the Harvard Art Museum in Cambridge, MA. "The license activation process has gotten better with [Version] 12d, but certainly with the 11d release it was a complete and utter disaster," notes Kane.
"Very robust" is how Performance Food's Wible describes NetBackup's management interface. He also likes the product's ability to scale. "It's infinite," he says. "If you need to, you can always throw another media server [at it] to handle load and stuff like that."
Because backup to disk has become such an important part of many data protection strategies, respondents were asked how well their product met their backup to disk needs. Perhaps not surprisingly, Tivoli Storage Manager scored very well with a 6.62 for this issue. Even so, Galaxy bested the field with a 6.75. Data Protector was third with a still-strong 6.45. But these results didn't correlate to the degree that respondents say they use these products for backup to disk. Tivoli has the highest percentage of users backing up to disk (88.4%), while 83.3% of Galaxy users do so. HP had the lowest percentage of users backing up to disk (46.7%). Interestingly, 81% of Retrospect users use the product to back up to disk.
Retrospect received a 5.68 for its product features vs. Backup Exec's 5.50 and ARCserve's 4.58. Retrospect topped the section with a 6.15 for the statement "This product's features meet my needs." However, it had a relatively disappointing 4.67 for "This product is interoperable with other vendor's products." Kane says Retrospect's "interface to configure multiple, simultaneous jobs is wonderful and really intuitive. Once you have the system up and running, it's real easy to tweak and tune."
Backup Exec didn't have any scores over 6.0 in this section, but came very close with a 5.98 for the statement "This product's file-system backup meets my needs." All of its other section scores were above 5.0. ARCserve had its highest section score (5.21) for its file-system backup features.
"[Backup Exec] is centrally manageable and user friendly, so even if you're someone who's not real familiar with backups, it's pretty explanatory," says Louis Lawson, server operations supervisor at Truckee Meadows Community College (TMCC) in Reno, NV. "When we hire our new server guys, they can catch on pretty quick. It's pretty intuitive."
Initial product quality
"Highly scalable, highly flexible, but also highly complex," was how one Tivoli Storage Manager user describes the product. That sentiment is shared by a CommVault user who says, "CommVault works well, but not without trade-offs. The product is complex, [and] difficult to configure and maintain."
HP bucked the negative trend among enterprise products by posting a 6.19 score in this section. This was HP's highest score of any section and well above CommVault's second-place rating of 5.80. HP received a 6.31 for "This product is easy to use," while CommVault had a 6.11 for the same statement. HP had the highest score of any product for "This product delivers good value for the money" (a 6.45), which was also its highest score in the section. CommVault's Achilles' heel in this section was a 4.83 for the statement "I'm satisfied with the level of professional services required for this product." This was CommVault's only rating below 5.40 throughout the survey.
In the midrange, EMC Retrospect's 5.74 section score compares favorably to Backup Exec's 5.63 and ARCserve's 4.86. Retrospect had a solid rating of 6.12 vs. Backup Exec's 5.60 for "This product is easy to use." Retrospect also had the highest score for our value statement (a 5.72).
"They now have built-in update notification capability, and you can roll out the clients right from within the interface," says Harvard Art Museum's Kane about Retrospect. Kane also uses Backup Exec in his shop and finds its interface "much more intuitive," while Retrospect's requires more of a learning curve.
"It's 100% reliable," says Performance Food Group's Wible of NetBackup, which notched a 5.83 for the service-level statement. Also a Backup Exec user, Wible says of NetBackup's sibling app, "overall, our reliability with that is definitely in the mid- to upper-90s."
Among midrange products, Retrospect scored a 5.62 for reliability, followed by Backup Exec's 5.18 and ARCserve's 4.36. This score was ARCserve's lowest of the survey. Retrospect's highest score in the section was for "This vendor provides comprehensive upgrade guidance," (5.94), but it also rated highly for "This product's OS platform support is adequate" (5.89). Backup Exec also had its highest rating for this statement (5.93). Apparently, however, users aren't enamored with the way any of the vendors handle errors. "This product's error handling is easy and intuitive" generated the lowest section score for all three midrange products: ARCserve (3.60), Backup Exec (4.56) and Retrospect (4.89).
"It reports a lot of errors," says Performance Food Group's Wible about Backup Exec, minor stuff he doesn't think affects the actual backup, "errors that I don't want to see."
"IBM has clearly the best and knowledgeable people in support roles," noted one survey respondent, adding that IBM "furnishes fixes in appropriate timeframes." Another satisfied Tivoli user also had praise for IBM: "Their support is top notch."
Backup and recovery software, more than most other storage product categories, is sold primarily through value-added channels. We therefore ask respondents to evaluate the vendor's partners. Based on the response to the statement "Vendor's third-party partners are knowledgeable," IBM's are apparently head and shoulders above the others with a 6.31 rating; Symantec was second with a 5.82, followed by EMC (5.43), CommVault (5.42) and HP (5.35).
The Web has become an increasingly important support tool for users, often used in lieu of placing service calls to vendors or resellers. "Documentation is really good," says TMCC's Lawson of Symantec's Backup Exec support. "The product is widely used, and you don't have to have a logon ID to go to their support center." Lawson relies on online support because the college didn't re-up its tech support contract. "We weren't really using it to justify the price," says Lawson, who notes that their backup environment is stable enough to do without the security blanket of a support contract.
Retrospect was also highly rated for the statement "Vendor provides support as contractually specified" (6.16). Its overall section score of 5.50 topped Backup Exec (5.12) and ARCserve (4.43).
Looking backwards--and forward
Because backup is the foundation of data protection, users demand that it operate properly. Based on this year's survey, users are quite happy with the feature sets available. However, it's clear they would dearly like improved quality and reliability, at least at the enterprise level.
About the survey: The Diogenes Labs-Storage magazine Quality Awards are designed to identify and recognize products that have proven their quality and reliability in actual use. The results are derived from a survey of qualified Storage readers who assess products in five main categories: sales-force competence, product features, initial product quality, product reliability and technical support. Our methodology incorporates statistically valid polling that eliminates market share as a factor. Indeed, our objective is to identify the most reliable products on the market regardless of vendor name, reputation or size. Products were rated on a 1.0-8.0 scale, where 8.0 is the most favorable possible score.
Midrange: Arkeia Software Arkeia Network Backup*, Atempo Inc. Time Navigator*, BridgeHead Software Ltd. HT Backup*, CA ARCserve Backup, EMC Retrospect, NovaStor Corp. NovaBackup (Server)*, Symantec Backup Exec and Yosemite Technologies Inc. Yosemite Backup (TapeWare)*