What are the differences between file-level vs. block-level deduplication?
Many different data backup vendors provide data deduplication services, so expect to find subtle differences in how both file-level deduplication and block-level deduplication are implemented. In general, file-level deduplication watches for multiple copies of the same file, stores the first copy, and then just links the other references to the first file. Only one copy gets stored on the disk/tape archive. Ultimately, the space you save on disk relates to how many copies of the file there were in the file system.
Block-level, sometimes called variable block-level deduplication, looks at the data block itself to see if another copy of this block already exists. If so, the second (and subsequent) copies are not stored on the disk/tape, but a link/pointer is created to point to the original copy. For example, John's copy of a file may in fact just be a pointer to Mary's file -- if Mary's file was the first to be archived.
There are pros and cons associated with both file-level and block-level deduplication. For instance, if 1,000 identical attachments are sent out by a benefits coordinator, file-level dedupe will find those 1,000 attachments that are exactly the same, but it won't find the exact duplicate copies you saved (i.e., Benefits_file.Aug, Benefits_file.Sep, Benefits_file.Oct, etc.). Block-level dedupe will find all of the duplicates, even if named differently, and will store the name variations with pointers to the original blocks. Variable block-level will also account for misaligned data sets on disk and would also detect the duplicate files and exact copies with different names.
File-level deduplication will save a relatively small amount of space on your disk/tape archive. Block-level deduplication will save more space on your disk/tape archive, and variable block-level deduplication will save even more space on your disk/tape archive. However, as with any other data storage technology or technique, be advised that your mileage will vary depending on the amount of replicated data you have in your file systems.
For more articles, tips and expert advice on data deduplication, visit the data reduction and deduplication section of our site.
Dig Deeper on Data reduction and deduplication
Related Q&A from Dave Ellis
Sometimes tape failure can be hard to predict, but certain monitoring tools can help detect problematic tapes. Continue Reading
Dave Ellis discusses how to get started with a backup procedure in your company in this expert response. Continue Reading
Dave Ellis explains what LTO-4 is and its compatibility with other versions of LTO in this expert response. Continue Reading