Tech Talk: Understanding LTFS and its place in the storage marketDate: Sep 13, 2013
Can you give a brief overview as to what LTFS is and what it's designed to do?
Jon Toigo: LTFS (Linear Tape File System) is a technology from IBM. It's free; you can download the core software that's required to make it work. You load it on a server; the server connects up to the tape library, and as long as you're using something called partitioned media in the library (which is LTO-5, LTO 6, Oracle tape in its current format, and IBM tape in its current format), you can start using your tape library like a file system, like a filer, like a [network-attached storage] device, like NAS on steroids.
Basically, you write standard files there; the files are organized by the bar code on the end of the tape; so, what you see if a file folder that has a bar code number on it, you click on that file folder and it shows you all the files that have been written to that folder. OK? That's in its raw format with no addition. That can be accessed across the network like any other file server or any other NAS device by end users. So, it becomes a way to store a ton of data; the most dense kind of storage you can get for files. We're talking petabytes of storage on a couple of raised floor tiles, consuming less than three or four lightbulbs of electricity.
Twenty-seven years ago when I first got into IT, tape was not just [a] backup medium; tape was a production storage medium. We kept files that we didn't access very frequently on tape, but whenever we needed it, we just went to the tape library and pulled the file out. And the response time was roughly that of what we would regard as the 'World Wide Wait' today. You know, you go to a TechTarget website and you want to download a white paper; and you have to find it, and then when you find it, you download it to your desktop -- same difference, except the files are all organized in something that looks like a regular file system.
Who is using LTFS today?
Toigo: The original uptake was in the broadcast and media space, because they were already used to using tape to store their assets anyway. However, it's recently blossomed into other areas. There's hardly a vertical segment of the market now that isn't finding a use for it, but oftentimes early adoption is among companies that have what are known as long block files, files that benefit from a consistent rate of streaming. Once you find the beginning of the file, you're streaming a long amount of data off after it.
Video is a good example [of a long block file]: not only broadcast video, but surveillance video. Telemetry from satellites is another; oil and natural gas wellhead data is another, and human genome sequencing. There's a huge uptake of LTFS in medical now for genome sequencing and medical imaging, because those MRI scans and the CT scans and stuff are huge long block files. And this is [a] very inexpensive way to store very infrequently accessed data.
Can you offer any specific examples of LTFS being used in the media or entertainment industry today?
Toigo: Major League Baseball's kind of remarkable, because they've got what, 11 cameras on every angle of every play of every game, and they're recording that video; and they have wedded LTFS to a front-end content management system that basically allows a bunch of interns to quickly type in 'This is Lou Gehrig in the ninth inning of game No. 7 with the New York Yankees' and this is where he hits the home run. And they've got 11 different views of the home run from all the cameras that they're pointing at once. They've got a content management system that contains all of those details in a database format, because they want to be able to use, you know, somebody wants to do a documentary someday and they want to get this angle and that angle and that angle, OK. That's fine; they can just use those portions of Major League Baseball's historical video record.
They've wedded that content management system to LTFS so, instead of having tape cartridge bar code numbers, the file folders actually have names on them, like this is game No. 4 of the World Series of 1973 or whatever. And then underneath that are different innings and different plays and different people, and different angles.
Do you see LTFS being embraced as general-purpose storage in the future?
Toigo: Well, you know, it's really hard to gauge that, only because nobody's tracking how many downloads there are of LTFS software. Here at Storage Decisions today, I've had a whole bunch of people come up and they say, 'We're throwing together an LTFS right now just for test purposes,' and they are from all different kinds of companies. There was an insurance company, there was a government institution, there was a local government thing, there were hospitals. In fact, there was a person who asked me a question today about LTFS and how it could be used effectively to store medical data for long-term file-keeping for HIPAA.
So, I would say, there's an adage [about] the difference between discovery and innovation, OK: It took 6,000 years after the discovery of the wheel to innovate and put them on the outside of luggage so you could pull your luggage through O'Hare Airport. OK, how long does it take for the discovery to materialize in terms of an innovative platform?
Perpendicular magnetic recording, which we use to store a lot of bits onto a disk drive or onto a tape -- that technology was actually discovered in the early 1980s. It's taken 35 years to put it on a disk drive. Now it's dedupe, where all disk drives have perpendicular magnetic recording on them, all the SATA [Serial Advanced Technology Attachment] drives and the terabytes-sized drives. How long will it take for LTFS to become a dominant file storage platform? I think it's being adopted a lot faster than some of those other technologies were. I'm already seeing it in a lot of shops right now. In a big way, not so much -- not yet. But I think more and more companies are going to get used to the idea.
The problem of backup [has] always been complicated with tape, and they think that because backup sucks -- and it does, you know; it's slow and it's lumbering and you know, it has a lot of things that can go wrong and the software generally isn't fully baked.
The fact is, tape always worked just fine. The software wasn't the greatest thing in the world for writing data to the tape, but the tape library itself was [a] pretty interesting and well-defined piece of hardware with a lot of pedigree behind it, and it's pretty effective stuff. So, what we did was we were throwing the baby out with the bath water -- didn't want to do the backup software, so let's throw the tape out, too. And you also had [a] $50 million marketing campaign from disk vendors who wanted to get rid of tape and have disk become the platform for archive, which makes no economic sense. In five years, it's 450 times the expense of tape to do archiving on disk.
For disk to disk backup, a 100-terabyte (TB) system is going to cost you somewhere in the realm of $2.5 million. With tape, I can get by with $500,000. So, there are all kinds of economic advantages to retaining tape, but there's been a lot of noise made that's said tape is no good anymore, or tape is outmoded, tape is yesterday's news. And in fact, there was a lot of paid analytical work that was done in the late '90s where claims were made that were patently false about tape -- things like one in 10 tapes fails on restore, you know. That 73% of tapes can't be read after you write them -- this is all hogwash.
But it didn't stop marketing folks who were working against tape, including the cloud guys today. You're hearing a lot of cloud vendors resuscitate all these old, bogus claims, because they think they're real. But you also see analysts who would normally not be friends of tape, a Gartner for example, backing off and saying, 'We never said that. Show us the document where we said, "One in 10 tapes fail on restart." We did not say that.'
So, we're kind of seeing tape having to fight a many-front war to come back in the mainstream news. And in smart organizations where they're confronting huge rates of storage growth and no budgetary increase to go along with it to help them buy the kind of storage they need, tape offers a very dense, very capable, very capacious storage modality for not a lot of money. And that may win at the end of the day.