Two things became apparent as the creators of the often-controversial show South Park closed out a ninth season this past year: it was time to overhaul their storage hardware, and it was time to get new backup and recovery software to replace Legato Networker, which had been giving the production studio fits for years.
"It wasn't just that it would crash, said Keith Nesson, officially credited as an e-mail, Internet and electronic information officer for the show. "It's that we wouldn't even know it had crashed except the robot in the tape library wouldn't be running."
Often, getting Legato back online meant that IT at South Park Studios would have to reboot its production servers -- and in the process have to tell the dozens of artists working feverishly to get the show produced for its weekly airing on Comedy Central to stop what they were doing.
In fact, part of South Park's contract with Comedy Central requires it to sign up for 24/7 support with all IT vendors in order to ensure new weekly episodes are delivered on time.
"Still, they were never able to tell me exactly what the problem was," Nesson said. The show had been using Legato for several years before it was acquired by EMC Corp. in 2003. Nesson said the acquisition not only didn't solve his issues, but it probably spelled the beginning of the end for Legato's presence in his shop.
"Maybe it was the transition during their acquisition [by EMC]," Nesson said. "Sometimes I think that's what it was. It wasn't that I was using an old version of it -- I always kept up to date with the newest version." The last version of Legato the studio used, according to Nesson, was 7.1 -- the latest version to date is 7.3.
"They just didn't seem to have a lot of energy for a small installation like ours," he said. "We're not some mammoth IT client like Bank of America or a huge studio, but we demanded some of the same availability."
Another problem, Nesson acknowledged, was that the show was clunking along with seriously outdated storage dating to around 1996 -- a Ciprico Inc. 7000 attached via 1 Gbps Fibre Channel through a similarly antique Silicon Graphics Inc. Origin 3200 server to an Exabyte Corp. 221L two drive, 21-slot tape library.
Suffice to say that between the backup software and the backup system, the IT infrastructure wasn't adequate for a weekly show on a tight schedule, and with no room for error.
As part of the upgrade shopping spree, the IT crew for the studio replaced outdated Apple Computer Inc. Macintosh G3s with PowerMac G5 workstations, ordering three Apple XServe RAID arrays to go along with them, and building them into a small SAN by adding a QLogic Corp. 5200 Fibre Channel switch. They stuck with Exabyte for the tape library, but upgraded aging DLT drives and cartridges to LTO-2.
After updating the storage, the studio turned to the backup application. A new contract with Comedy Central gave them a handsome capital budget to work with, according to Nesson. "We figured, 'while we're at it, we should replace as much as we can.' "
Saving time with snapshotsIt was clear they wouldn't be signing on with Legato again. As Nesson's boss, technical supervisor J. J. Franzen, would state later in a press release, "We were looking for something with more functionality -- and less cantankerousness."
The studio looked at software from BakBone Software Inc. and Atempo, as well as open source options, before going with Atempo's Time Navigator.
In the end, Nesson said, the startup won its business with people skills, sending an engineer and a marketing person down to look at the studio's environment instead of just e-mailing a price quote.
Technologically speaking, Time Navigator's ability to quickly restore files from tape using chronologically organized snapshots has come in handiest for the show, Nesson said.
"Now when our artists want to reuse a background or prop, they can say, 'Cartman had a skateboard in the first season but I don't remember which episode,' and find it easily," Nesson said. "It saves them time."
And time is of the essence -- Nesson said together, the upgraded disk and new backup application have allowed the show to do four incremental daily backups instead of just one, and dropped the time for a full system backup from a week to half a day.
Finally, Nesson said the studio also appreciated Atempo's pricing structure, which charges for licenses according to number of drives in a library rather than by the number of cartridges, which was the method many competitors used. The pricing structure would allow the studio to add more tape cartridges without a software licensing bump, a godsend since the show expects data to explode as more episodes are created and archived, and animated sequences become more complex.
"We bought 19 terabytes (TB) of the XServe storage even though we only use a little over 2.5 TB today," Nesson said. "We're obviously looking ahead."