Analysts claim that the volume of data requiring safe haven in business data centers will continue an upward trajectory that could decimate your storage infrastructure by 2020. According to industry watchers, 2009 saw the creation of about one zettabyte (ZB) of new data. This volume climbed to 2.75 zettabytes in 2012, then on to about 8 ZBs in 2015. Now, IDC and others are suggesting that between 20 and 60 zettabytes of new data will be created by humans and machines by 2020. That will be a nightmare for both private and public data center operators who have not prepared for a "Z-pocalypse."
Some folks have been inclined to spin these projections to underscore the promise of, and to project a bright future for, cloud storage services. In conferences and trade shows, clouds are treated like a magical fifth storage medium -- flash, disk, optical, tape and cloud storage -- though this view is clearly in error. Clouds are a service delivery model; they are not a storage technology in and of themselves.
What is really interesting is listening to cloudies talk about data burgeon. They are actually more concerned about it than are many private businesses I visit, probably because they expect to catch more of the growth of zettabytes of new data on their chips and spindles -- and maybe cartridges -- than traditional business data centers.
A recent presentation by Microsoft provided a back-of-envelope calculation on the state of storage and the capability of the industry to handle the data deluge. With a production capacity of only about 500 exabytes per year, the speaker noted, flash memory would have neither the capacity nor the cost metrics to store all the bits. Even if disk manufacturers made good on the promise of 24 TB HAMR (heat-assisted magnetic recording) drives by 2020 or sooner, again, we would be looking at a capacity shortfall and extraordinary cost to store the zettabytes of data being created. The optical industry, despite Facebook's interest, might get us to a 1 TB BluRay disk by 2020 -- and that is a stretch, as well as being very insufficient to meet the storage challenge. Ultimately, Microsoft concludes, it will be up to tape technology to shoulder the lion's share of the storage of all the new data.
Really? We heard tape was a dead tech. Not so, says Microsoft
Of the three "industrial farmers" of the cloud storage space, only Google has admitted to using tape in its infrastructure. Microsoft talks very favorably about the technology in its discussions of cloud storage at conferences, though they do not disclose the details of their Azure storage infrastructure. Amazon, on the other hand, seems to have the same allergy to tape that many businesses developed a decade or so ago. In 2015, they began offering a strange disk-based appliance called Snowball to collect terabytes (TB) of data for "sneakernet" delivery to their storage cloud. Finally, a tacit acknowledgement that networks provide neither the speed nor the capacity to handle the transfer of that much data to or especially from the cloud!
Listening to cloud storage vendors discuss how they will handle that many zettabytes of data (note: a zettabyte is a billion terabytes) always requires what Hollywood calls "the suspension of disbelief." Truth be told, there are serious questions that need to be asked by anyone considering cloud storage. Some have to do with security and cost, of course. But one also needs to wonder whether the cloudies will simply give up on building the capacity that will be required to store all of the data produced from 2020 forward.
Yes, there is money to be made from storing as much data as possible, even if it is only for pennies per GB per month. However, if capacity is limited, there will probably be more money to be made by keeping supply limited and selling it at a premium to companies that can afford it. Because of this, I seriously doubt that the cloud service providers are going to offer a magical answer to the issue of the growth of zettabytes of data we are facing.
You will need a strategy of your own, home-grown and probably inclusive of heavy doses of tape archiving. The time to get going on an archive strategy is right now and the good news is that there are numerous products for file-system archiving to tape -- via gateway appliances that leverage Linear Tape File System and object storage technologies -- that make it relatively easy to get your act together and win the war against zettabytes.
Rapid data growth and the search for capacity
Meet zettabyte's cousin, the petabyte
Cisco predicts spike in data center traffic