This article is part of an Essential Guide, our editor-selected collection of our best articles, videos and other content on this topic. Explore more in this guide:
3. - How deduplication is used today: Read more in this section
- Data backup process angst persists, but dedupe and cloud offer relief
- What types of data yield a high deduplication ratio?
- Software-based deduplication and cloud backup
- Windows Server 2012 deduplication vs. backup software dedupe
- Using deduplication when backing up VMs
- Education group goes all-dedupe for data center, remote data backups
- How centralized backup can ease remote site data protection challenges
- Using data dedupe technology as part of your disaster recovery strategy
- Finding data deduplication solutions for DR
- Deduplication and data lifecycle management
Explore other sections in this guide:
- 1. - Backup deduplication technology today
- 2. - How deduplication is performed
- 4. - Data deduplication challenges
This is the second part of a nine-part series on deduplication. For easy access to all nine parts, check out our quick overview of Deduplication 2013.
Today, practically every backup product vendor includes a deduplication feature in its software. Over the last few years, backup deduplication has become the norm for major backup software vendors, such as Veeam, Symantec and Acronis. In spite of the fact that deduplication has become little more than a feature checklist item for backup software providers, not all deduplication features are created equal. Deduplication has become an essential feature for cloud-based backup providers as well. All of the major Backup as a Service (BaaS) providers employ some form of data deduplication. Part 2 of our series on deduplication in 2013 takes a look at software dedupe and cloud backups.
Some backup applications perform source-based deduplication. This form of deduplication compacts the data on the source server prior to performing a backup. Although the deduplication process is completely transparent, deduplication, by its very nature, is CPU-intensive; it also tends to generate a considerable amount of disk I/O.
Check out the entire series on data deduplication
Get the inside scoop on hardware-based deduplication
Part 3 of our series details software and hardware dedupe
Learn more about target deduplication
Explore more about the deduplication tax and why it's important
Not all backup software performs source deduplication. Some products perform target deduplication. Target deduplication performs the deduplication process on the backup target rather than on the source server. In the case of a software-based solution, this usually means deduplicating data on the backup server.
Target deduplication can occur in line, as data is being written to the backup server, but often the deduplication is post process, which means that deduplication is scheduled rather than occurring in real time.
Backup vendors are not the only ones who are performing software-based deduplication. Microsoft has included a native file-system deduplication feature in Windows Server 2012. This native deduplication occurs at the data source, but it is post-process. Because source-based deduplication can be resource-intensive, Microsoft has taken measures to make the deduplication process less disruptive to the server.
One of the things that Microsoft has done is to delay the deduplication of new data. When a new file is created, that file is not scheduled for deduplication for several days. This prevents Windows from wasting resources deduplicating temporary files. Another thing that Microsoft has done is to allow exclusions. Compressed data, such as ZIP files or compressed media files (MP3, MPEG 2, JPEG, etc.), tend not to deduplicate very well, because they are already compressed. System resources can be conserved by configuring Windows not to deduplicate these types of files.
Deduplication is essential to making cloud-based backups practical. The deduplication process reduces the amount of data that must be uploaded to the cloud, which improves the speed of the backup and reduces storage and Internet bandwidth costs for the BaaS provider.
Some BaaS providers, such as Mozy and LiveDrive, use file-level deduplication (single-instance storage). The way in which this type of deduplication is implemented varies considerably from one provider to the next. Some providers perform file-level deduplication on a per-computer basis. In other words, if multiple copies of a file exist on a computer, then only a single copy of the file is backed up.
Mozy uses a slightly more sophisticated form of single-instance storage that is based on private key encryption. If multiple computers share the same private key, then single-instance, storage-based deduplication can occur across all of those computers.
Some BaaS providers perform deduplication behind the scenes without really exposing the process to the administrator. However, there are some BaaS providers, such as CrashPlan, that let administrators configure the deduplication process. CrashPlan lets administrators choose the degree to which data is deduplicated. Full deduplication is CPU-intensive, but very effective in reducing the size of the data that needs to be uploaded. CrashPlan also offers less CPU-intensive, but less efficient deduplication models. They also have an automated setting, in which the software automatically chooses the most appropriate form of deduplication.
Part 3 of this series explores compares hardware and software dedupe and discusses using them together.
Brien M. Posey, MCSE, has received Microsoft's MVP award for Exchange Server, Windows Server and Internet Information Server (IIS). Brien has served as CIO for a nationwide chain of hospitals and has been responsible for the department of information management at Fort Knox. You can visit Brien's personal website at www.brienposey.com.