Tip

How to estimate your data deduplication ratio

What you will learn: This tip details the factors that influence the data deduplication ratio (the ratio of data before deduplication to the amount of data after deduplication), so you can estimate

Requires Free Membership to View

the deduplication ratio you can reasonably expect to achieve.


Every data deduplication vendor claims that their product offers a certain ratio of data reduction. However, the actual data deduplication ratio can vary according to many factors, some of which are within a user's control. Below are a few variables.

Redundant data

The more redundant data you have on your servers, the higher the data deduplication ratios you can expect to achieve. If you have primarily Windows servers with similar files and/or databases, you can reasonably expect to achieve higher ratios of data deduplication. If your servers run multiple operating systems and different files and databases, expect lower data deduplication ratios.

Rate of data change

Data deduplication ratios are related to the number of changes occurring to the data. Each percentage increase in data change drops the ratio; the commonly cited 20:1 ratio is based on average data change rates of approximately 5%.

Precompressed data

Data compression is a key component in every vendor's data-reduction algorithm. Vendors base their advertised data-reduction ratios on the premise that compression will reduce already deduplicated data by a factor of 2:1. In a case where data deduplication achieves 15 times, compression could take that ratio up as high as 30:1. However, users with large amounts of data stored in compressed formats such as jpeg, mpeg or zip, aren't likely to realize the extra bump compression provides.

Data retention period

The length of time data is retained affects the data-reduction rate. For example, to achieve a data-reduction ratio of 10 times to 30 times, you may need to retain and deduplicate a single data set over a period of 20 weeks. If you don't have the capacity to store data for that long, the data-reduction rate will be lower.

Frequency of full backups

Full backups give data deduplication software a more comprehensive and granular view into the backup. The more frequently full backups occur, the higher the level of data deduplication you'll achieve. Deduplicating backup software products have a slight edge over disk libraries because they run a full server scan every time they execute a server backup, even though they only back up changes to existing files or new files. In between full backups, disk libraries usually only receive the changes sent as part of the backup software's daily incrementals or differentials.

Check out the complete text of the Storage magazine article, Catching up with deduplication.

Jerome M. Wendt is a storage analyst specializing in open-systems storage and SANs.


This was first published in June 2007

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.