News Stay informed about the latest enterprise technology news and product updates.

Hifn offloads data deduplication to hardware

Hifn is adding data deduplication processing and encryption to its boards, which should appear in OEM products later this year.

Hifn Inc. has developed a new set of chip boards that will perform data deduplication, compression and encryption processing with the goal of eliminating some of the performance issues associated with software-based approaches in storage devices.

"All of these are processor intensive features," said John Matze, Hifn vice president of business development. "Up until now, all data deduplication products have done hashing calculations in software, and they haven't been able to offer encryption and compression with it in one pass."

Hifn will use a chip that provides network-based encryption and compression with two chip boards. The Express DR 250 board is an update to a product that has already been on the market. The DR 255 offers a faster four-lane PCI-e interface.

More on data deduplication
HDS ships VTL bundles with Diligent deduplication

FalconStor clusters VTL data deduplication

U.S. Army drafts deduplication, sends tape on leave

Copan adds data deduplication to MAID
"Right now, the fastest these products will go is 250 MBps," Matze said. That limits their use to secondary storage, but a third card based on new silicon that could perform at 1.6 GBps is planned for later in the year. The faster card could be used for primary storage data reduction.

The two cards Hifn released today also are capable of deduplication, encryption and compression with one board, but not in one pass. It does MD5 hashing for data deduplication in one process, and compression and encryption are done in a second pass. The card due out later this year will complete all three processes in one pass, Matze said. "Right now, we're recommending as a best practice that two cards be used, one for hashing, and one for compression and encryption, since they're already separate processes," he said. Today's cards will also leave data deduplication lookup -- the matching of identification keys produced by the hashing algorithm to existing data -- to software.

Hifn sells compression cards to OEM partners including Sepaton Inc., which uses it to add LZS-standard two times compression to its virtual tape libraries (VTL) with data deduplication. That increases its data reduction ratio from 25:1 to 50:1, in some cases.

The new boards and software APIs for unlocking the hashing, compression and encryption features will ship to OEMs this week, and Matze estimated products could be available to end users by the middle of the summer. He was mum on whether Hifn will offer its own product based on the boards, or which OEMs have signed up to incorporate the hardware into products.

Data deduplication vendors have debated about whether inline or post-process deduplication is better. Inline deduplication vendors argue that their products finish the hashing process before data is written to disk, which more effectively reduces disk space requirements and allows data to be replicated immediately. Post-process deduplication vendors argue that inline deduplication creates a performance bottleneck. This new hardware could short-circuit that argument entirely, according to Arun Taneja, founder of the Taneja Group.

"Inline data deduplication vendors might be able to put this card in and equalize that performance situation," Taneja said. "Even if throughput isn't exactly the same as post-process products, it could at least come in close, and it could take one huge knock on inline deduplication off the table."

Taneja also pointed out that chip and component makers are porting more storage services into hardware, citing the storage controller chip with embedded encryption that PMC-Sierra launched today. The chip can also be placed into host bus adapters (HBA).

"We're going to have to start deciding, as an industry, where we want these services, especially encryption," Taneja said. "Right now you can do encryption at the disk-drive level, the controller level, the SAN-switch level, the HBA level … the industry is in a bit of an overkill situation, and it's going to need to settle on the best approach."

Dig Deeper on Data reduction and deduplication

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.