Published: 09 Sep 2005
The Diogenes Labs-Storage magazine Quality Awards debuted in our August issue with the results of our service and reliability survey on enterprise-class arrays. This month, we turn to backup and recovery (B/R) software, and evaluate leading products in two categories--products for small- to medium-sized businesses (SMBs) and products for enterprise backup applications. Rather than a line-item comparison of product features or a measure of market share, the Quality Awards process quantifies the factors that contribute to users' perceptions of a product's overall quality. Toward that end, we surveyed hundreds of storage professionals who have practical experience with specific B/R products to gauge their levels of satisfaction. As a follow-up to the survey, a number of respondents were then contacted and interviewed.
|About the survey|
Respondents were asked to rate their level of agreement or disagreement with a series of evaluative statements in each of five categories using a 1.0-8.0 scale. A response of 1.0 indicated strong disagreement, while 8.0 indicated strong agreement. Because the 1.0-8.0 scale doesn't accommodate a neutral response, all responses indicated some degree of either favorable or unfavorable experiences. Responses were required for all questions, although "Don't know" and "Not applicable" responses were permitted.
With 4.5 as the mean on this scale, scores below 4.5 are generally negative while those above are generally positive. The survey included 14 backup and recovery software products from leading vendors in the small- to medium-sized business (SMB) and enterprise sectors. Vendors whose products were represented in the survey had no prior knowledge of the survey, and didn't influence the selection of specific products included in the survey.
Respondents were asked to rate the products in the following categories: sales competence, product features, product quality, product reliability and product support. Each category had up to eight questions. The overall scores for the backup and recovery (B/R) survey were lower than the scores for our previous survey on enterprise arrays. We believe this is because IT organizations find B/R to be a problem area, and are therefore more generally dissatisfied with it. "It touches everything—servers, networks, database applications-everything," says Roman Van Liempt, storage administrator for xwave Solutions in St John's, Newfoundland, an IT services company that provides support for New Brunswick Power. "Backup software will bring to light any problems that you have in your network."
The survey used to determine the winners of the Quality Awards for B/R software included five series of service and reliability questions in the following key categories:
- Sales competence
- Product features
- Product quality
- Product reliability
- Technical support
Respondents were asked to rate their level of agreement or disagreement with a series of evaluative statements in each of these categories using a 1.0-8.0 scale, where a response of 1.0 indicated strong disagreement and an 8.0 indicated strong agreement. With 4.5 as the mean on this scale, scores below 4.5 are generally negative and those above are generally positive.
|SMB backup software|
The winners are ...
We divided our field of 14 products into SMB and enterprise groups, and awarded Quality Awards to each. We recognize that many SMB products offer an "enterprise edition" and enterprise products offer a "workgroup edition," but we placed products into each group based on their most common deployments.
Based on the survey results, we're pleased to award CommVault Systems Inc.'s Galaxy with the Diogenes Labs-Storage magazine Quality Award in the enterprise backup product category and EMC/Dantz's Retrospect with the Quality Award in the SMB backup category. Retrospect ran away with the award in its group, outpolling the competition in every category. The tally in the enterprise group was tighter, with Galaxy winning the product features, quality and reliability categories, and Hewlett-Packard (HP) Co.'s OpenView Storage Data Protector earning top honors for sales force and product support (see the "Overall ranking" categories in the SMB backup software and Enterprise backup software charts). Underscoring CommVault's solid showing in the survey, one respondent notes, "The company I work for is currently replacing all instances of backup software installed with CommVault Galaxy." Another survey participant switched to CommVault because "the Tivoli support people were not able to make TSM's [IBM's Tivoli Storage Manager] Exchange agent work in our environment."
We also asked respondents if they'd buy the product they were using again. In the case of Retrospect, 81.3% of those surveyed said "Yes." CommVault Galaxy, our overall winner, actually placed third among enterprise products for this question with a respectable 78.6% indicating "Yes." Placing first and second were IBM's TSM and Veritas' (now owned by Symantec Corp.) NetBackup, with 84.8% and 81.8% of current users, respectively, stating they'd buy those products again. It appears that TSM and NetBackup have a high user loyalty, even though their overall ratings weren't as high as either Galaxy's or Data Protector's (see SMB backup software: Would you buy this product again?, and Enterprise backup software: Would you buy this product again?).
Arkeia Corp.'s Server Backup and StoneFly Networks Inc.'s Backup Advantage products were included in our survey, but we didn't rate them. Server Backup received only one response, while Backup Advantage didn't receive any.
|SMB backup software: Would you buy this product again?|
The success of any application implementation begins with the competence of the person selling the product. The product must be sold into the appropriate environment based on realistic expectations. Most SMB backup products are sold through VARs, while enterprise products are sold through VARs or directly to large accounts. For sales made through VARs, vendors have less control over sales quality.
You might expect a vendor's own sales team would deliver superior satisfaction. However, EMC/Dantz--which is sold almost exclusively through VARs--received the highest sales force rating with a 6.38 (see the "Sales force," category in the SMB backup software chart). HP Data Protector, also sold primarily through resellers, topped this category among enterprise products with a score of 5.56 (see the "Sales force," category in the Enterprise backup software chart). Although Computer Associates (CA) International Inc.'s products didn't fare particularly well in this survey, the company achieved its highest marks in the sales force section. The vast majority of CA's B/R product sales are through VARs, so the high ratings in this category speak well of the company's ability to train, manage and motivate resellers.
In the sales force category, Veritas' score for NetBackup was particularly hurt by the survey question, "The vendor's licensing model offers good value," where it received a 4.19 rating--the lowest of any product.
Karl Lewis, storage administrator at the University of Michigan in Ann Arbor, says NetBackup has been a blessing to his shop, but adds, "[Veritas] licensing is my No. 1 complaint. Of my top five things, it's the top three." According to our survey, HP ranked the highest related to licensing with a score of 5.62. "The HP licensing model is excellent," says Roman Van Liempt, storage administrator for xwave Solutions in St John's, Newfoundland, an IT services company that provides support for New Brunswick Power. "It's the best I've seen," he adds.
|Enterprise backup software|
B/R software products have been around for a long time, tend to be comparable and new functionality is incremental. Surprisingly, all of the major SMB products hovered near a neutral 4.5 rating in the product feature category--Retrospect had a modest 4.62 (the highest score in the category), while the lowest rating was a 3.25 for Yosemite Technologies Inc.'s TapeWare (see the "Product features" category in the SMB backup software chart).
In the enterprise product group, five of the eight products rated above 5.0. CommVault's Galaxy led all products with a very high 5.77, the highest category score among the enterprise products (see the "Product features" category in the "Enterprise backup software" chart, right). Veritas' NetBackup received its highest category rating for features (5.28). Six of the eight products had generally favorable ratings above 4.5 (only Syncsort Inc.'s Backup Express and CA's BrightStor Enterprise Backup rated below a 4.5). This suggests that the majority of enterprise products are suitable for most applications and environments. As opposed to features, product differentiators center on support, price, quality and reliability.
Lewis appreciates NetBackup's operational prowess after running into some problems with HP's Data Protector. "A lot of the features that you admittedly pay for in NetBackup just aren't there in [Data Protector]," he comments. "NetBackup is definitely much more robust," he says.
|Enterprise backup software: Would you buy this product again?|
Product quality and reliability were the lowest rated categories as measured by the average of all vendors. Given IT's frustration regarding B/R operations, it's not surprising these were the lowest rated areas. To assess product quality, we asked respondents to rate the products in terms of ease of use, configuration and the need for manual intervention. Only two SMB products--Retrospect and Backup Exec--received scores above 5.0 for quality, while Galaxy was the only enterprise-class product to score above 5.0 (see the "Product quality" section of the SMB backup software chart, and the "Product quality" section of the Enterprise backup software chart).
Media represents the largest single, non-personnel B/R cost. As another indicator of quality, we asked which product utilized media most efficiently. CommVault Galaxy's 5.68 led the ratings in this area, followed closely by IBM'S TSM with a 5.55. Among SMB products, Retrospect rated the highest with a 5.50.
We also asked respondents how well they felt the products delivered value for the money. They rated Galaxy as the best value (5.50); Data Protector and TSM were nearly tied for second with a 5.22 and 5.21, respectively.
Gary Wood, manager of server engineering for the County of Westchester, NY, explains why his firm switched to EMC Legato's NetWorker after using Syncsort's Backup Express for five years. "As our server base increased, and consequently the load on the product [Syncsort], we began to see some cracks appear," he says. He concedes that while implementing NetWorker "we did have a bumpy road along the way, as it wasn't configured properly by Legato and their consultants." It took about a month to iron out the problems and, as far as Wood is concerned, the configuration snafu is no longer an issue. "All in all, we're very pleased with the Legato experience."
The University of Michigan's Lewis also experienced some configuration problems when installing Veritas' NetBackup. While he's pleased with NetBackup's performance, he notes that "it's extremely cumbersome to install and configure, possibly because of the flexibility of the product."
Determining product reliability for B/R software is different than from most other storage components because a discussion of five-nines uptime isn't relevant. B/R rarely needs to operate on a 24/7 basis, but it must run when needed. Therefore, we examined a variety of operational characteristics to derive reliability. Many of the reasons for a B/R job failure--including tape drive failures/errors, media errors and scheduling errors--are beyond the control of B/R software. However, B/R software does control the ability to operate unattended (i.e., "lights out") without interruption due to bugs and can handle errors effectively.
The statement, "I can operate this product in a lights-out mode" generated low marks for most products. Only Galaxy (5.71) and Data Protector (5.30) stood out with scores above 5.0. The other enterprise products with favorable ratings were NetBackup (4.81) and TSM (4.79). Among SMB products, Retrospect's 4.71 narrowly nosed out Backup Exec's 4.68.
Also generating low scores across the board was, "This product experiences very few bugs." With no product breaking the 5.0 barrier for this statement, Galaxy was the highest rated with a 4.96, followed by Retrospect (4.88). Interestingly, we noted an almost direct correlation between the ratings for this statement and the overall rankings of the products. The message to vendors is consistent: Spend more time solving existing problems to create stable products rather than implementing the latest "whiz-bang" feature.
Given the low user satisfaction concerning product reliability, and especially the number of product bugs, product support gains importance as a purchase criteria (see the "Product support" category in the "SMB backup software" chart, p. 38, and the "Product support" category in the "Enterprise backup software," chart, p. 39). Because most B/R software is sold through resellers, Level-1 support is usually handled by the reselling organization. When asked to respond to the statement, "The vendor's third-party partners are knowledgeable," 11 of the 14 products in our survey rated above 5.0. Retrospect led the SMB group with a whopping 6.56; among enterprise products, IBM's TSM earned the highest mark on this question with a 5.83, followed by a 5.75 for EMC Legato NetWorker.
But support isn't the only criterion for sticking with a product. "We uncovered some shortcomings in the [Syncsort] product along the way," says Westchester County's Wood. "The product was faulty in some areas, but the support was just great."
The timeliness of problem resolution is also an important issue. Responses to the statement, "Problems are resolved in a timely manner," were consistent with the overall curve of the study. Retrospect received the highest score (5.94) among SMB products, and HP led the enterprise group with a 5.20. We also found a correlation between this statement and "Support issues rarely require escalation." When problems escalate from Level-1 support (reseller) to Level-2 support (vendor), there's a necessary hand-off between the two. Individual vendor scores for these two questions were similar in every case. Clearly, knowledgeable Level-1 support that avoids problem escalation is an important factor in overall customer satisfaction.
|The Diogenes Labs-Storage magazine Quality Awards|
|The Diogenes Labs-Storage magazine Quality Awards is a joint project of Storage magazine and Diogenes Analytical Laboratories Inc. It's an ongoing process that surveys users to gather their perceptions related to service, reliability and quality of key storage infrastructure products. Storage reported the results for enterprise-class arrays in its August 2005 issue; other product categories will be addressed in the coming months.|
Diogenes Labs and Storage magazine congratulate CommVault Galaxy and EMC/Dantz Retrospect on taking top honors among a large field of B/R products. And thanks, too, to the nearly 500 users who took the time to complete the survey and provide such insightful responses.