Data Management – The Modern Day Wild West

 The concept of data management holds no mystery.  Digital information has to be stored and filed in such a way that it can be retrieved and used when necessary.  However, in the film and television production community, data management offers a vast, under-explored territory. 

"We grew up in a world of physical objects: paper, file folders, reels of film, and we have developed certain conventions based on how we handle those physical objects." said Andy Maltz, Executive Director, Science and Technology Council, Academy of Motion Picture Arts and Sciences. "Imagine a filing system for all the tens of millions of objects that are created for a digitally-produced motion picture.   We're in the early days on the digital side.  It's not just how you recover the ones ands zeros from the digital storage media, but how you interpret them once you get them back." 

Data management difficulties begin at the earliest stages of production.  Regardless of the type of camera used, it’s the handling of the captured data and transferring the files into a storage system that initiates the management worries.

"What you start with is a raw file," said Steven Poster, A.S.C., National President, International Cinematographers Guild.  "That has to be put into a pipeline.  You apply a look up table to it in order to be consistent with what you were looking for once it comes out of the pipeline.  Film, inherently, is a logarithm space.  Computer files are inherently linear.  That look-up table has to be done a certain ways to make it display properly on your display device."  

The transit of data is the next major management hurdle.  Outside vendors such as Sohonet, used on recent James Bond and Harry Potter movies, and  Aspera, the software used to move data between global facilities involved in the effects works on "Avatar," are relied on to quickly and safely transport data from one pipeline to another.

"Aspera is a software that packetizes the data and puts it over the IP network.  The data is transferring over internet connections; it's literally going through the wires and the routers of the internet.  It's moving along as a bit stream, “said Michelle Munson, President and Co-Founder of Aspera.

To move the size of the data in a movie like "Avatar," a tremendous amount of speed is required.  The fiber optic networks that have to be set in place sometimes require physical modifications to buildings and designated drives to store the information.   These modifications can be very costly for a facility to establish and maintain.  

"The network capacities have not caught up with the growth, the storage, and the data management requirements," said Maltz.  "There are a lot of fire wire and USB drives that are being messengered because, in most cases, it's faster and less expensive to move data around that way than to encore the expense of a fiber optic or photonics network."

 Wherever files travel along a pipeline and network, security breaches become a major concern.  Both Aspera and Sohonet use a form of encryption to main privacy of the data as it is transferred, and both products monitor the transmission of data to ensure hackers are not compromising the data.  Despite these precautions, security is still a concern for anyone moving data. 

Once data is received, or a film is completed, the next management issue is storage.

"Many directors and cinematographers are experiencing liberation of not exposing film.  They feel that bits are free," said Christopher Carey, Executive Vice President, Worldwide Technical Operations, Paramount Pictures.  "There are data management build up worries.  What should be thrown out, what is necessary to keep."

"Content can range from a few terabytes to limitless," said Paul Kocher, President, Cryptography Research.  "The choice of how much storage is needed is usually under-estimated.  There are no patterns"

Some companies are looking into ways to assist with a production's data storage decisions. Sohonet has been working on "Vessel," a software package that files the data that has been both viewed and used.  The goal of such a project will help directors and producers make selective decisions as to what data should be kept depending on what was used during the production process.

Adding to the complexities of dealing with data management is the future of data.  With such rapid changes in technology, there is no certainty that the data stored today will be accessible a few years down the line.

“The problem that remains, is the ultra long term storage problem,” said Munson.  “What do we do to ensure that the data we have kept 100 years from now can actually be access, because the computers, the operating systems, and the software that can read back those bits may be obsolete?  That is a very significant technical unsolved problem right now."

To assist the production industry in overcoming data management hurdles, ICG has developed a training course for digital loaders, ensuring a consistency in encoding data on sets.

"We've developed a series of systems to safely, effectively and efficiently download images from any form of digital camera," said Poster.  "Before this, there were no standards and procedures.  It's very dangerous.  We're developed a two day program.  When a loader takes the program, he can walk onto the set the next day and do the job effectively."

AMPAS is also working on developing a set of standards for the industry to following in managing data.  Their goal is to not only focus on how productions currently manage data, but enable fluid transitions of data management systems in the future.

”What we're trying to do here at the Academy is develop some of this infrastructure on a path towards standards, so everybody agrees on metadata structures, everybody agrees on the basic file formats and the way you represent the colors digitally, and the best practices for the creation of the data so you can find it, and the preservation of it over the long term, " said Maltz.