Problem solve Get help with specific problems with your technologies, process and projects.

What are the differences between file-level vs. block-level deduplication?

Dave Ellis, Principal Technologist at Instrumental Inc., discusses the differences between file-level and block-level data deduplication in this ask the expert response.

What are the differences between file-level vs. block-level deduplication?

Many different data backup vendors provide data deduplication services, so expect to find subtle differences in how both file-level deduplication and block-level deduplication are implemented. In general, file-level deduplication watches for multiple copies of the same file, stores the first copy, and then just links the other references to the first file. Only one copy gets stored on the disk/tape archive. Ultimately, the space you save on disk relates to how many copies of the file there were in the file system.

Block-level, sometimes called variable block-level deduplication, looks at the data block itself to see if another copy of this block already exists. If so, the second (and subsequent) copies are not stored on the disk/tape, but a link/pointer is created to point to the original copy. For example, John's copy of a file may in fact just be a pointer to Mary's file -- if Mary's file was the first to be archived.

There are pros and cons associated with both file-level and block-level deduplication. For instance, if 1,000 identical attachments are sent out by a benefits coordinator, file-level dedupe will find those 1,000 attachments that are exactly the same, but it won't find the exact duplicate copies you saved (i.e., Benefits_file.Aug, Benefits_file.Sep, Benefits_file.Oct, etc.). Block-level dedupe will find all of the duplicates, even if named differently, and will store the name variations with pointers to the original blocks. Variable block-level will also account for misaligned data sets on disk and would also detect the duplicate files and exact copies with different names.

File-level deduplication will save a relatively small amount of space on your disk/tape archive. Block-level deduplication will save more space on your disk/tape archive, and variable block-level deduplication will save even more space on your disk/tape archive. However, as with any other data storage technology or technique, be advised that your mileage will vary depending on the amount of replicated data you have in your file systems.

For more articles, tips and expert advice on data deduplication, visit the data reduction and deduplication section of our site.

Dig Deeper on Data reduction and deduplication