Managing and protecting all enterprise data


Quality Awards IV: On top again: CommVault Galaxy, EMC Retrospect

CommVault's Galaxy solidified its position as the top enterprise backup product by finishing first for the third time in the four years we've conducted the Quality Awards. Among the midrange products, EMC Retrospect retained its crown as group champion.

Although users gave favorable marks to most enterprise and midrange backup software products, they still have concerns about product complexity and usability.

If the results were any tighter, they'd squeak. The 2008 edition of the Diogenes Labs-Storage magazine Quality Awards for backup and recovery software recorded lower high scores, but substantially improved low scores. In past surveys, midrange backup products scored significantly lower than enterprise-class products. This year, the enterprise products still rated higher, but the discrepancy was much smaller.

CommVault's Galaxy solidified its position as the top enterprise backup product, repeating last year's win and finishing first for the third time in the four years we've conducted the survey.

Galaxy's overall winning score of 6.01 came with the slimmest possible margin--.01--to barely nudge out Hewlett-Packard (HP) Co.'s Data Protector. HP has nipped at the winner's heels in past surveys, and its final tally of 6.00 left it just shy of an upset victory. CommVault and HP each won two of the five survey categories.

IBM Corp.'s Tivoli Storage Manager held onto third place again and improved its score to 5.85 from last year's 5.72. EMC Corp. NetWorker showed the greatest improvement in the enterprise group, climbing from a 4.88 last year to 5.42 in this survey--a high enough score to slip by Symantec Corp.'s Veritas NetBackup (5.40) and take fourth place. Syncsort Inc.'s Backup Express, which was a finalist last year, had too few survey responses to be considered for this year's award. BakBone Software Inc.'s NetVault:Backup, the 2006 winner, also failed to garner enough responses to be in the running.

Among the midrange backup products, EMC Re-trospect retained its crown as the group champion. Its 5.56 score outdistanced second place Symantec's Backup Exec's 5.32. EMC took top honors by winning every section of the survey. Retrospect, Backup Exec and CA ARCserve Backup finished 1-2-3 in all sections of the survey.

While CA ARCserve finished third in the midrange group once again, it deserves some sort of honorable mention. Two years ago, ARCserve finished with a very negative 3.93. Since then, its scores have steadily risen and it finished with a positive overall score of 4.60 in 2008. In our 1.0-8.0 scoring system, a score of 4.50 represents the demarcation between positive and negative ratings. CA's scores were above 4.50 in three of the five categories, and only slightly negative in the other two. Clearly, CA has its product moving in the right direction.

Of all the product categories we survey for the Quality Awards, backup and recovery software is apparently the least loved. Last year, there were a number of negative comments from respondents. This year, negative comments outnumbered positive comments by approximately 4:1. Moreover, one-third of the respondents say they wouldn't make the same product purchase decision again.

In cases where respondents provide evaluations for two or more products, we compare these head-to-head matches to determine which vendor is preferred. This year, the results were quite surprising. Among enterprise products, NetWorker was involved in 29 direct comparisons and won 62.1% of them, the highest percentage among enterprise products. ARCserve was in 26 such match-ups and won 57.5% of them. Note that head-to-head comparisons could be against any product in our survey, not just the finalists.

Sales-force competence
HP's sales force has been highly ranked in recent Quality Award surveys for other products, and won this survey's sales-force competence section. Its 6.06 handily topped CommVault's 5.83. HP's best rating in this section (a 6.35) was for the statement "My sales rep is flexible"; however, its sales support group was also highly rated for "My sales support team is knowledgeable" (6.27). CommVault had one category score higher than 6.0, which was a 6.22 for the same sales support statement as HP.

Sometimes sales competence isn't measured only by the reps that show up in your shop. "Our new sales team is still kind of learning what we do," says Chris Wible, backup and recovery lead at Richmond, VA-based Performance Food Group, of the Symantec reps supporting his Veritas NetBackup and Backup Exec installations. But Wible was pleasantly surprised when an inside sales rep contacted him to follow up on an inquiry he made about adding a software option. "She called up the people that I needed to talk with for Veritas NetBackup PureDisk, scheduled the meeting and got it going," he says.

Licensing schemes are often a source of user ire. We asked respondents to tell us about their vendor's licensing methodology. HP had the highest score for "This vendor's licensing formula offers good value" (5.97). One Data Protector user notes that HP's "licensing structure is very generous, very cost-effective and it affords us a tremendous amount of flexibility." CommVault was the subject of several unhappy comments regarding its licensing formula ("the licensing model makes the product expensive to own"), but received a decent 5.46 for this statement. Veritas NetBackup had the worst score among enterprise products for this statement with a 4.76.

EMC Retrospect led the midrange group with an overall sales score of 5.24, while Backup Exec and ARCserve scored a 5.20 and 4.77, respectively. Retrospect had an interesting dichotomy with its highest section score (a 5.50) for the statement "My sales rep is flexible" and its lowest section score (a 4.86) for "My sales rep is easy to negotiate with." Backup Exec had the highest midrange score in this section for "My vendor's sales support team is knowledgeable" (5.77).

Backup Exec users have struggled at times with the product's licensing process, both before and after Symantec acquired Veritas. "The notification of releases, the license renewal process was an absolute nightmare," says Esmond Kane, senior manager of information systems at the Harvard Art Museum in Cambridge, MA. "The license activation process has gotten better with [Version] 12d, but certainly with the 11d release it was a complete and utter disaster," notes Kane.

Product features
Given the maturity of the backup and recovery market, most products are packed with features. The most common complaint we've observed is product complexity. Perhaps because Galaxy is one of the newest enterprise products in our survey is why it received the highest score for product features (6.50), which is also the highest section score. ("CommVault rocks!" noted one particularly enthusiastic respondent). Even so, we did note several user comments that took issue with Galaxy's complexity. Galaxy received its highest section score for the statement "This product's file-system backup features meet my needs." Its database backup is almost equally well regarded, garnering a 6.62 for "This product's database backup meets my needs." Coincidentally, those were also the highest rated statements for Data Protector.

"Very robust" is how Performance Food's Wible describes NetBackup's management interface. He also likes the product's ability to scale. "It's infinite," he says. "If you need to, you can always throw another media server [at it] to handle load and stuff like that."

Because backup to disk has become such an important part of many data protection strategies, respondents were asked how well their product met their backup to disk needs. Perhaps not surprisingly, Tivoli Storage Manager scored very well with a 6.62 for this issue. Even so, Galaxy bested the field with a 6.75. Data Protector was third with a still-strong 6.45. But these results didn't correlate to the degree that respondents say they use these products for backup to disk. Tivoli has the highest percentage of users backing up to disk (88.4%), while 83.3% of Galaxy users do so. HP had the lowest percentage of users backing up to disk (46.7%). Interestingly, 81% of Retrospect users use the product to back up to disk.

Retrospect received a 5.68 for its product features vs. Backup Exec's 5.50 and ARCserve's 4.58. Retrospect topped the section with a 6.15 for the statement "This product's features meet my needs." However, it had a relatively disappointing 4.67 for "This product is interoperable with other vendor's products." Kane says Retrospect's "interface to configure multiple, simultaneous jobs is wonderful and really intuitive. Once you have the system up and running, it's real easy to tweak and tune."

Backup Exec didn't have any scores over 6.0 in this section, but came very close with a 5.98 for the statement "This product's file-system backup meets my needs." All of its other section scores were above 5.0. ARCserve had its highest section score (5.21) for its file-system backup features.

"[Backup Exec] is centrally manageable and user friendly, so even if you're someone who's not real familiar with backups, it's pretty explanatory," says Louis Lawson, server operations supervisor at Truckee Meadows Community College (TMCC) in Reno, NV. "When we hire our new server guys, they can catch on pretty quick. It's pretty intuitive."

Initial product quality
The initial product quality section asked backup software users to respond to issues regarding usability and manageability of the products. Users of the enterprise products seemed to take out some of their frustrations about complexity in the product quality section of the survey. The difference between user reactions to the enterprise products compared to the midrange group was rather stark. For enterprise products, the quality section elicited some of the lowest scores. For midrange products, this section yielded the highest scores for each vendor.

"Highly scalable, highly flexible, but also highly complex," was how one Tivoli Storage Manager user describes the product. That sentiment is shared by a CommVault user who says, "CommVault works well, but not without trade-offs. The product is complex, [and] difficult to configure and maintain."

HP bucked the negative trend among enterprise products by posting a 6.19 score in this section. This was HP's highest score of any section and well above CommVault's second-place rating of 5.80. HP received a 6.31 for "This product is easy to use," while CommVault had a 6.11 for the same statement. HP had the highest score of any product for "This product delivers good value for the money" (a 6.45), which was also its highest score in the section. CommVault's Achilles' heel in this section was a 4.83 for the statement "I'm satisfied with the level of professional services required for this product." This was CommVault's only rating below 5.40 throughout the survey.

In the midrange, EMC Retrospect's 5.74 section score compares favorably to Backup Exec's 5.63 and ARCserve's 4.86. Retrospect had a solid rating of 6.12 vs. Backup Exec's 5.60 for "This product is easy to use." Retrospect also had the highest score for our value statement (a 5.72).

"They now have built-in update notification capability, and you can roll out the clients right from within the interface," says Harvard Art Museum's Kane about Retrospect. Kane also uses Backup Exec in his shop and finds its interface "much more intuitive," while Retrospect's requires more of a learning curve.

Product reliability
Because backup and recovery is the cornerstone of any data protection scheme, reliability is a critical factor. CommVault returned to the top of the group in the product reliability section with a score of 6.11. No other product rated above 6.0, although HP came close with a very respectable 5.92. CommVault had a very high 6.83 rating for the statement "This product's OS platform support is adequate." More importantly, it had a very strong 6.46 score for "This product meets my service-level needs," which gets to the core of reliability. Tivoli and HP had ratings of 6.35 and 6.31, respectively, for this same statement. Responses to the statement "This product operates in a lights-out mode" are also designed to illustrate key reliability. CommVault again led the group with a 5.67, followed by HP (5.54) and Tivoli (5.28).

"It's 100% reliable," says Performance Food Group's Wible of NetBackup, which notched a 5.83 for the service-level statement. Also a Backup Exec user, Wible says of NetBackup's sibling app, "overall, our reliability with that is definitely in the mid- to upper-90s."

Among midrange products, Retrospect scored a 5.62 for reliability, followed by Backup Exec's 5.18 and ARCserve's 4.36. This score was ARCserve's lowest of the survey. Retrospect's highest score in the section was for "This vendor provides comprehensive upgrade guidance," (5.94), but it also rated highly for "This product's OS platform support is adequate" (5.89). Backup Exec also had its highest rating for this statement (5.93). Apparently, however, users aren't enamored with the way any of the vendors handle errors. "This product's error handling is easy and intuitive" generated the lowest section score for all three midrange products: ARCserve (3.60), Backup Exec (4.56) and Retrospect (4.89).

"It reports a lot of errors," says Performance Food Group's Wible about Backup Exec, minor stuff he doesn't think affects the actual backup, "errors that I don't want to see."

Technical support
Tivoli Storage Manager broke up the back-and-forth 1-2 finishes of CommVault and HP by winning our technical support section. Its 6.16 section score was quite respectable and well above Galaxy (5.79) and Data Protector (5.71). Lately, IBM support has done very well in our surveys. Continuing this trend, its support scored high marks for the statement "Vendor provides support as contractually specified" (6.61).

"IBM has clearly the best and knowledgeable people in support roles," noted one survey respondent, adding that IBM "furnishes fixes in appropriate timeframes." Another satisfied Tivoli user also had praise for IBM: "Their support is top notch."

Backup and recovery software, more than most other storage product categories, is sold primarily through value-added channels. We therefore ask respondents to evaluate the vendor's partners. Based on the response to the statement "Vendor's third-party partners are knowledgeable," IBM's are apparently head and shoulders above the others with a 6.31 rating; Symantec was second with a 5.82, followed by EMC (5.43), CommVault (5.42) and HP (5.35).

The Web has become an increasingly important support tool for users, often used in lieu of placing service calls to vendors or resellers. "Documentation is really good," says TMCC's Lawson of Symantec's Backup Exec support. "The product is widely used, and you don't have to have a logon ID to go to their support center." Lawson relies on online support because the college didn't re-up its tech support contract. "We weren't really using it to justify the price," says Lawson, who notes that their backup environment is stable enough to do without the security blanket of a support contract.

Retrospect was also highly rated for the statement "Vendor provides support as contractually specified" (6.16). Its overall section score of 5.50 topped Backup Exec (5.12) and ARCserve (4.43).

Looking backwards--and forward
As noted earlier, nearly one-third of our respondents say they would make a different purchase decision based on what they know now. Typically, this "buyer regret" number is approximately 25% on other surveys. Among users willing to repurchase, EMC Retrospect led all products with an 85.7%. Among enterprise products, HP users seem to be the happiest overall, as 84.4% would make the same decision. CommVault users are also happy, as 81.3% would make the same decision. CA had a 63.3% positive response, significantly higher than in prior years, while NetBackup had the lowest rate of all products with 63% saying they would buy it again.

Because backup is the foundation of data protection, users demand that it operate properly. Based on this year's survey, users are quite happy with the feature sets available. However, it's clear they would dearly like improved quality and reliability, at least at the enterprise level.

About the survey: The Diogenes Labs-Storage magazine Quality Awards are designed to identify and recognize products that have proven their quality and reliability in actual use. The results are derived from a survey of qualified Storage readers who assess products in five main categories: sales-force competence, product features, initial product quality, product reliability and technical support. Our methodology incorporates statistically valid polling that eliminates market share as a factor. Indeed, our objective is to identify the most reliable products on the market regardless of vendor name, reputation or size. Products were rated on a 1.0-8.0 scale, where 8.0 is the most favorable possible score.

To ensure market coverage, our survey seeks to include every product from Arkeia to Zmanda. However, we restrict the list to general-purpose backup and recovery software products. Any product in the survey must receive at least 13 responses to be statistically valid and therefore considered as a finalist for the award. In 2008, 460 respondents provided 637 product evaluations, a response rate that yields a 4.5% margin of error with a 95% confidence factor. The largest response group belonged to Backup Exec with 151 individual ratings, followed by NetBackup with 127. Reflecting the consolidation that has gone on in the industry, 92.4% of respondents have just one or two backup and recovery products in their organization.

The industry most represented in this survey was financial services followed by government and non-profits. As a group, the respondents were skewed toward small- and medium-sized enterprises; 64% were from organizations with revenue of less than $1 billion.

Products: The following backup and recovery software products were included in the survey.

    Enterprise: Asigra Inc. Televaulting*, BakBone Software Inc. NetVault:Backup*, CommVault Galaxy (or Simpana), EMC Corp. NetWorker, Hewlett-Packard Co. Data Protector, IBM Corp. Tivoli Storage Manager, Symantec Corp. Veritas NetBackup, Syncsort Inc. Backup Express* and Zmanda Inc. Amanda Enterprise*

    Midrange: Arkeia Software Arkeia Network Backup*, Atempo Inc. Time Navigator*, BridgeHead Software Ltd. HT Backup*, CA ARCserve Backup, EMC Retrospect, NovaStor Corp. NovaBackup (Server)*, Symantec Backup Exec and Yosemite Technologies Inc. Yosemite Backup (TapeWare)*
*Did not receive enough survey responses to be included as a finalist.

Article 28 of 37

Dig Deeper on Data storage backup tools

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.

Get More Storage

Access to all of our back issues View All