animind - Fotolia
Backup managers seem to be a little grumpier these days, no doubt due to the spiraling data capacities they're expected to back up and protect. That crankiness was evidenced by the results of last year's Quality Awards for backup and recovery software, and continues with the tenth edition of the service and reliability survey.
While overall scores just barely topped last year's marks, users of Asigra and Veeam backup products reaffirmed their satisfaction by returning the two vendors to the winner's circle in a reprise of last year's finish. Along the way, the two upstarts once again nudged aside some of the most venerable backup applications in the industry.
Typically, storage pros are tough judges of the software they're using and don't seem to cut the apps any slack. Reflecting those higher standards, overall scores for midrange backup applications were down again; while the enterprise gang upped its overall ratings by a small margin. However, that score was also low compared to past years. With 774 product evaluations from 346 respondents, we had a field of six enterprise and seven midrange products.
Enterprise. As it did last year, Asigra flexed its muscles across all five rating categories, leading the field in each one and rolling up an overall score of 6.66 to easily outdistance number two finisher Commvault (6.09). Symantec's NetBackup was third with a 5.75 rating, ahead of the closely packed cluster of HP, IBM and EMC -- all separated by only .07 points. Asigra swept the categories last year, too, but actually improved its scores for every category this time around. While Asigra didn't yield on too many of the ratings statements, when it did, Commvault stepped in, finishing second in four categories and third in the final one. While Asigra's performance is a new-kid-on-the-block-makes-good story, Commvault has established a strong record over the years, with five victories and two second-place finishes.
Midrange. Veeam, too, was a repeat winner, although not in as convincing a manner as last year. But Veeam's overall score of 6.08 was enough to outdistance EMC Avamar (5.82), which also finished second last year. The win was built by besting the field in four of the five rating categories; Veeam's top mark came for initial product quality, a key factor in getting an application up and running and productive quickly. EMC Avamar prevailed in the fifth category -- sales competence -- while notching very consistent scores in the other categories. VMware, with its vSphere Data Protection Advanced, cracked the top three only a year after it earned a spot among the finalist products. The rest of the midrange gang represented some of the more veteran product lines, including ARCserve, Symantec Backup Exec, Acronis and Microsoft.
Enterprise. Asigra demonstrated that it has a very adept sales team and partners, outscoring the competition for all seven sales competence rating statements with a very strong 6.61 category tally. Commvault (5.81) and Symantec (5.80) finished in a virtual tie for second with EMC just barely off that pace at 5.73. Asigra's dominant performance was highlighted by a couple of near 7.0s -- 6.95 for having a knowledgeable sales support team and 6.91 for "My sales rep understands my business." It was hardly a slouch on the other five statements, gaining a 6.82, for example, for sales reps that are knowledgeable about users' industries. That was Commvault's strongest statement as well, with a 6.20. Symantec performed consistently in the category, highlighted by a 6.02 on the knowledgeable sales support team statement.
Midrange. EMC Avamar cruised to a comfortable category win with a 6.13 score built on scoring highest on six of the seven category statements. EMC's highpoints included a 6.35 for "My sales rep is knowledgeable about my industry," and a 6.32 earned for its knowledgeable sales support team. EMC was also had a strong showing for reps who understand users' businesses (6.23) and for keeping customers interests foremost (6.21). Second-place VMware came out on top for the remaining statement, with an even 6.00 for "The vendor's licensing formula offers good value." Veeam finished third in the category at 5.88, tallying consistent scores that ranged from 5.20 for "My sales rep is easy to negotiate with" to 5.96 for the knowledgeable sales support team statement.
Initial product quality
Enterprise. Getting an app installed, configured and up and running will often set the tone for the overall user experience. Asigra proved it's up to the task by copping top marks for six of the seven statements in the initial quality category. HP rode four 6.00-plus scores to 5.94, good enough for second place, and Commvault snared the final statement (6.35 for efficient use of tape) on its way to finishing third (5.85). Asigra flirted with a 7.00 score, earning a 6.96 for the key statement, "This product is easy to use." Asigra also did very well on another bellwether statement, posting a 6.88 for "This product delivers good value for the money." HP's best score was 6.18 for a product that requires little daily intervention; Commvault also rated well for that statement (6.12); HP also scored well for ease of use (6.15) and using tape efficiently (6.14).
Midrange. Veeam swept away its midrange competition with ease -- gaining top marks for all seven statements and posting a category score of 6.50 that was nearly a full point higher than the 5.54 earned by both ARCserve and EMC Avamar as they tied for second place. Veeam's best showing was a lofty 6.93 for "This product was easy to install," but it also scored very well for easy configuration (6.72), ease of use (6.69) and delivering good value (6.56). ARCserve's top grades were for ease of installation (5.82) and delivering good value (5.73); EMC posted a solid 6.28 for ease of use and a 6.00 for the good value statement. The results did yield good news for software developers: as a group, the two top averages were for easy installation (5.90) and ease of use (5.74).
Enterprise. Ultimately, even after a good initial experience, it comes down to what a product is capable of, and its features must match users' needs to be relevant. Asigra resoundingly resonated with its users, posting 7.00-plus scores for half of the category's statements and leading the field on all eight for a total of 6.84. Its top tallies were for backup-to-disk functionality (7.31), database backup (7.19), file system backup (7.15) and a 7.00 for "This product is a complete solution." Commvault also impressed, with all 6.00-plus scores for a total of 6.49. Commvault netted a pair of 6.70s for being a complete solution and for database backup. Symantec scored well enough for file system backup (6.14) and management features (6.09) to finish third in the category, just ahead of IBM.
Midrange. Veeam rolled up a 6.18 for features among the midrange apps, based on the best marks in the group for six of the eight category statements. Its highest score was a 6.78 for backup to disk functionality, bolstered by a solid 6.51 for management features and a 6.35 for scaling well to meet new requirements. EMC Avamar's 5.82 put it in second place, with the group's highest grades for file system backup (6.46) and for being a "complete solution" (6.03). EMC also scored very well for management features (6.26) and for database backup (6.06). Symantec's Backup Exec finished third with a category score of 5.40, nudging out ARCserve (5.38). As a group, this statement, "This product is interoperable with other vendor's products" was a stumbling block, producing an average of only 5.09.
Enterprise. After features, the reliability rating category was the enterprise group's strongest, with Asigra again outrunning Commvault, 6.70 to 6.17, as both products scored very well. Asigra led for seven of the category's eight statements, and Commvault snared the eighth with the highest statement score in the group. Asigra posted excellent scores for all statements, highlighted by a 6.93 for meeting service level requirements, 6.85 for having very few bugs and a 6.69 for requiring few unplanned patches. Commvault's group-leading mark was an imposing 6.95 for OS/platform support, and it also tallied a 6.91 for the service-level statement. The third-, fourth- and fifth-place finishers -- IBM, Symantec and HP -- finished just .02 points apart for an effective three-way tie for third.
Midrange. Veeam's score for reliability wasn't quite as high as Asigra's, but it managed to snag top marks for seven statements and just fell shy by .04 points on the eighth. Veeam cobbled together its winning 6.19 mark with impressive grades of 6.51 for meeting service-level requirements, and a pair of 6.41s for requiring few unplanned patches and for providing comprehensive upgrade guidance. Second-place EMC Avamar (5.80) notched some solid scores, notably a top-of-the-group 6.53 for OS/platform support and 6.42 for the service-level statement. ARCserve was a fairly distant third, totaling a 5.38 average for the category, just ahead of VMware's 5.36. As a group, the weakest showing -- an across-the-board average of only 5.05 -- was for the statement, "This product's error handling is easy and intuitive."
Enterprise. We often say that good tech support can turn a so-so product experience into a good one, but for Asigra, its performance in the technical support category is more like icing on the cake. Posting the highest scores on six of seven statements, Asigra rolled up an average of 6.68 -- the second best we've seen to date for enterprise backup apps. The whole category was a highlight reel for Asigra, but its best marks were a 6.96 for supplying support as contractually specified, 6.92 for knowledgeable support personnel and 6.91 for knowledgeable third-party partners. Commvault edged out Asigra, 6.41 to 6.40, for the final statement regarding supplying adequate documentation and supporting materials. Commvault's best score was a 6.56 for supplying support as promised. Symantec finished third with a solid 5.80 category score.
Midrange. Veeam (5.94) prevailed in the tech support category, although in less dominating fashion, netting top marks for four statements and yielding the spotlight to third-place VMware (5.79) for two statements and second-place finisher EMC Avamar (5.82) for the final one. Veeam's top grades were for "Vendor's support personnel are knowledgeable" (6.38) and for delivering support as contractually specified (6.22). Veeam also earned a couple of 6.00s for support issues that rarely require escalation and for resolving problems in a timely manner. EMC led the group with a 5.66 for providing adequate training, and added a couple of 6.14s for delivering support as promised and having knowledgeable support personnel. VMware's two winning scores were for documentation/supporting materials (5.97) and a 5.90 for having knowledgeable third-party partners.
Would you buy this product again?
The final question in every Quality Award survey is a simple, subjective query: Given what you now know about the product you've rated, would you buy it again today? Often, we find the "buy again" results run counter to the category rating scores, a situation that could be described as simply sticking with what you know.
Enterprise. After dominating in the rating categories, it's no surprise that Asigra's 82% buy-again tally led the enterprise group. It was, however, a somewhat less-resounding coda to excellent survey results than Asigra's mark of 93% that capped last year's win. Still, a win is a win and if 82% of your customers are ready to buy your product again, you must be doing something right. The big "buy-again" surprise in the enterprise group was EMC NetWorker's result -- a healthy 78% said they would buy the product again, a stronger endorsement perhaps than the marks it received in the ratings categories.
Midrange. The midrange "buy-again" results were more closely aligned with the products category, with Veeam, EMC Avamar and VMware repeating their overall ranking order. Veeam's 92% tally was just a couple of points shy of last year's mark; EMC's 89% was six percentage points higher than its tally from last year, and VMware added five percentage points this time around for 79%.
A closer look at today's backup and recovery apps
The inside scoop on data backup applications