A 35mm film projector in operation.

As part of preservation-level digital standards, myself and colleagues have worked since 2004 to develop a best practice specification for digitizing moving images.  Our initial standard document was developed for the NJVid Portal, and was very basic in its specification.

Since then, some minor tweaks have periodically been added to the document.  But recently, some major developments have occurred with our campus infrastructure that have resulted in our need to consider slightly more substantial changes to our spec:

• RUcore is in the process of implementing Wowza as it’s new streaming server platform, and it is already in operation for the libraries reservation streaming media service.  In conversations with the reserves and media teams handling video for those efforts, there has been some real-world testing of video quality improvements for MP4 streaming, and tweaks have been suggested for improving the quality of our streamed videos.
• RUwireless Wifi campus connections have been upgraded to support higher bitrates (up to 3Mbps per connection).  Previously they were capped at 1Mbps.  This means that we can now push video and other content at nearly triple the data rate we were accustomed to over campus Wifi, given proper conditions.

In light of this, coupled with user demand for improvements in video streaming quality, and in preparation for Wowza streaming support on RUcore, we’re proposing changes to the digitization specs for moving images. A draft for comments is available.  Changes are noted in the document red, but to summarize:

1. MPEG-4 streaming bitrates have been increased to a minimum of 860kbps, recommendation of 2.1Mbps for high quality.
2. HD resolution is now supported at a minimum of 720p resolution.

Language has also been added to address digitization of motion picture film, and calls for a minimum of DCI 4K resolution, with support for MXF wrappers and Motion JPEG2000 where appropriate. Motion Picture Film scanning is still a moving target however, and mention is made that film digitization projects should start with a Digital Curation consult.

A Peirce 55-B dictation wire recorder from 1945. Courtesy of Stanford University Libraries. Source: Wikipedia.

A good portion of our nation’s heritage has been immortalized in sound recordings.  From the late 19th century to the present, sound recordings have been used to capture music, speeches and historic events, the oral histories of people who have lived through important events in our nation’s history.

As with many electronic and mechanical recordings, however, this vast heritage is in danger.  In an effort to save what we can of these timeless recordings, the Library of Congress has put together a blueprint in the form of a National Preservation Plan.  This plan is the result of nearly a decade of work that was mandated by Congress as part of the National Recording Preservation Act of 2000.

As the Library of Congress puts in in their press release:

Experts estimate that more than half of the titles recorded on cylinder records—the dominant format used by the U.S. recording industry during its first 23 years—have not survived. The archive of one of radio’s leading networks is lost. A fire at the storage facility of a principal record company ruined an unknown number of master recordings of both owned and leased materials. The whereabouts of a wire recording made by the crew members of the Enola Gay from inside the plane as the atom bomb was dropped on Hiroshima are unknown. Many key recordings made by George Gershwin no longer survive. Recordings by Frank Sinatra, Judy Garland, and other top recording artists have been lost. Personal collections belonging to recording artists were destroyed in Hurricanes Katrina and Sandy.

The National Preservation Plan for sound recordings is available as a PDF file. In it, multiple recommendations are made, including:

• Create a publicly accessible national directory of institutional, corporate and private recorded-sound collections and an authoritative national discography that details the production of recordings and the location of preservation copies in public institutions;
• Develop a coordinated national collections policy for sound recordings, including a strategy to collect, catalog and preserve locally produced recordings, radio broadcast content and neglected and emerging audio formats and genres;
• Establish university-based degree programs in audio archiving and preservation and continuing education programs for practicing audio engineers, archivists, curators and librarians;
• Construct environmentally controlled storage facilities to provide optimal conditions for long-term preservation;
• Establish an Audio-Preservation Resource Directory website to house a basic audio-preservation handbook, collections appraisal guidelines, metadata standards and other resources and best practices;
• Establish best practices for creating and preserving born-digital audio files;
• Apply federal copyright law to sound recordings created before February 15, 1972;
• Develop a basic licensing agreement to enable on-demand secure streaming by libraries and archives of out-of-print recordings;
• Organize an advisory committee of industry executives and heads of archives to address recorded sound preservation and access issues that require public-private cooperation for resolution.

A Con Ed worker makes efforts to restore power in the wake of Hurricane Sandy. The East Coast’s Infrastructure was heavily damaged by this storm, testing not just our survival skills, but how well we preserve our data and memories. Photo by Robert Francis on Flickr.

No greater a test of our resolve and our preparedness exists than a true trial by fire, and the past two weeks have been living proof of this. There isn’t any way to overstate or exaggerate it.  Hurricane Sandy devastated New Jersey, New York and other sections of the East Coast, taking away lives, homes, power, and safety.  It could take years before a sense of normalcy is restored for the lives of those most impacted, and as has been stated by many officials and news outlets: it will never be the same.

A stark reality coming from this is the notion that landmarks and attractions washed away by the storm will now exist only in memories, and in people’s photo and video archives.  But what hasn’t quite been acknowledged fully just yet, is that Sandy has also taken a toll on the archival and digital front as well.

For some, Sandy was just a rather annoying inconvenience.  Power and heat were out for a while.  Cell phones didn’t work as well as they used to. Internet access was scarce, and websites were taken offline for a few days until power was restored. This very blog, for instance, had an emergency plan that kicked in when its usual home base at Rutgers sat safe but idle, without electricity. It was digitally “evacuated” to a backup cloud datacenter in Los Angeles for a few days, until all was clear and the power was back on.

These were the lucky ones. Others fared so much worse.

With homes and businesses being washed away, so too were all of the things inside.  We’re starting to hear about this in the media: trillions of dollars in on-paper riches, potentially wiped out. Computers with important family documents and personal data, gone.  Photos, and keepsakes, destroyed as in this article chronicling the situation in Breezy Point:

[Shamus] Barnes, 43, has spent every summer he can remember here at the sandy tip of the Rockaway Peninsula. Those years were lovingly documented in photos of what his family calls “the pyramid” — the intergenerational group photo op that seemed to grow larger each year.

Those photos were lost when, in the midst of Sandy’s assault, fire destroyed more than 100 houses on Monday, including Barnes’ and his parents’ homes.

“We’ll never be able to replace those things,” he says. He is standing in the mud, holding the lighthouse-shaped sign for No. 16 Fulton Walk, all that was left of his bungalow. “It’s just pictures, but they show the legacy of what’s gone on here. That’s the backbone of everything out here — memories.”

Sadly, the events of these past two weeks have meted out a cruel lesson: use the technology you have to save your important memories, before it’s too late to save them.  Storms can take away our physical possessions, but our photos, videos, recordings and documents can always be saved, if we work to keep them safe. We can’t always get back what is lost, but we can take steps to prepare for what may come.

Always Have a Plan B

How do we do this?  A year ago, I wrote a couple of articles on keeping your stuff safe.  The information in those articles is still relevant today, now more than ever:

1.  Keep a local backup of all your important stuff.
2. Supplement that local backup with a remote, or cloud, backup.  And I’ll add: do some research to make sure that this cloud provider is in a different geographical area from where you normally keep your digital stuff. This way if your local backup is just as waterlogged as your computer, you can still get your stuff back.
3. I haven’t written about this part yet, but you should even back up your mobile devices. It’s easy to do, whether you have an iPhone or an Android device.

“If you cannot access your data from your computer or storage device, no matter what its been through, the data may still be recoverable,” said Chris Bross, Strategic Technical Alliance Engineer at DriveSavers Data Recovery. “We have repeatedly been successful in recovering data from storage devices that have been exposed to sustained water and fire damage, corruption, corrosion and erosion. We have the most advanced technology and methods available to help Hurricane Sandy victims get their data back safely.”

Here’s to hoping that no one will ever have to experience such great losses again. But in case we must, let’s take the steps we need to better prepare for next time.

Recently, members of the Rutgers University Libraries at both integrated Information Systems and the Scholarly Communication Center began an auspicious collaboration with the Asia Society in New York City, in our first project to digitally preserve, to standards, their digital video archive for The China Boom Project.  It is the first time that RUcore has ingested a fully born-digital video archive, using the original source content and project files, and creating presentation video from those source files.

The China Boom Project’s goal is to seek an answer to the question, “Why did China Boom?” The site comprises taped interviews with individuals and experts with insights into China’s rapid economic expansion in recent decades. It offers to site visitors packaged video content from these interviews arranged by subject matter and relevant time periods in China’s history, in a very effective and attractive format that is described as a “mosaic explanation.”

But while the China Boom site itself provides snippets and prepackaged commentary, an ancillary goal of the project has been to partner with educational institutions to make the full-length content available to researchers, and to have the video archived and preserved.  This is where Rutgers University Libraries, and RUcore, come into the picture.

Read the rest of this entry »

MobileMe was shut down on July 1 by Apple, as part of its efforts to transition users to other services. Not making the transition, however, were users’ public web sites, shareable Photo Galleries and iDisk, a cloud storage service similar to Dropbox.

Note: It’s important that readers of this article understand that the purpose of this post is to document a growing, grass-roots movement to archive the web, in spite of some rather controversial methods practiced by this movement. While I sympathize with the philosophy, I am not affiliated with, nor do I condone all of their actions, nor is this something we at Rutgers would do without first clearing permissions and rights to archive any content.

One of the big problems with the web is its inherent lack of permanence. There is no formal archiving structure, and like anything digital, it’s very easy for something deemed important by someone to just disappear overnight, with little or no notice.   Sometimes these deletions happen on a mass scale, affecting millions of websites of varying quality, and sometimes arguably of significant cultural value.

Now it appears that, for better or for worse, a group of individuals are working to do something about it… with or without our permission.

Read the rest of this entry »