Salman Rushdie. Source: Wikipedia. Click on image for link to source.

The New York Times today published an article that reflects some of the challenges of preserving born digital content – that is, documents, data and other content that has been created digitally, on a computer or electronic device, and for which there is no physical original (such as on paper).

In particular, they highlight the efforts of Emory University, in preserving Salman Rushdie’s archival materials.

Among the archival material from Salman Rushdie currently on display at Emory University in Atlanta are inked book covers, handwritten journals and four Apple computers (one ruined by a spilled Coke). The 18 gigabytes of data they contain seemed to promise future biographers and literary scholars a digital wonderland: comprehensive, organized and searchable files, quickly accessible with a few clicks.

But like most Rushdian paradises, this digital idyll has its own set of problems. As research libraries and archives are discovering, “born-digital” materials — those initially created in electronic form — are much more complicated and costly to preserve than anticipated.

Electronically produced drafts, correspondence and editorial comments, sweated over by contemporary poets, novelists and nonfiction authors, are ultimately just a series of digits — 0’s and 1’s — written on floppy disks, CDs and hard drives, all of which degrade much faster than old-fashioned acid-free paper. Even if those storage media do survive, the relentless march of technology can mean that the older equipment and software that can make sense of all those 0’s and 1’s simply don’t exist anymore.

Imagine having a record but no record player.

An interesting aspect of this collection and its exhibition is that it emulates the experience Rushdie had in creating the content.  Rather than just viewing the finished documents, you get to see the computer desktop as he saw it, open up the same applications he used, all in the 1980s and 1990s technological contexts… and not using the modern, Web 2.0, Windows 7 or Mac OS X trappings we’re accustomed to in today’s computers.

I think this article is an excellent read, irrespective of what one’s views may be on the subject matter.  Material of all kinds, in increasing amounts, faces the same perils as this collection every day, and archivists everywhere, including this one, wrestle with how best to retain it all.  So far, the only tried and true method for such types of preservation is to obsessively manage and migrate the content, and that requires making tough decisions as to how to proceed, what formats to migrate to, and hoping the decisions made are the right ones to keep the content viable, at least until the next generation of technology requires that the hard decisions be made again.

As the person responsible for handling the various file formats in RUcore, the digital library repository for Rutgers University Libraries, I’ve been looking with trepidation at the increasing sizes of the digital assets people are starting to create.  In 2004 when the architecture for this was first envisioned, very few digital items grew past the hundred-megabyte point.

How things have changed!  Video and even audio files are routinely pushing into the gigabytes, now that technology has progressed to the point where high-definiteion video and audio can be originated for ubiquitous mobile devices.  And as RUcore and other large repositories seek to preserve this content, we are finding ourselves running into a hurdle we did not anticipate: the ability for our architectures to handle these very large digital files.  In particular, files larger than 2 Gigabytes has posed some exceptions forFEDORA, our infrastructure of choice, and this is a very big deal for video content in particular.  Consider that 2 Gigabytes can comprise less than 5 minutes of HD content, and you can see our dilemna.

Added mechanisms to support these large items has been slow in coming, and have presented some difficulties of their own in implementing.  For this reason, I’ve drafted a document which explains our position on why we need uniform large file support in digital repositories.  Feel free to have a look and provide feedback.

With any luck, developers will heed the call presented here and in other institutions,a nd work to make better support for big files a reality.

• White paper: “The Case for Large File Support in FEDORA/Duraspace Object Management Architectures” (101KB, PDF File)

If you’re interested in grabbing some of the last of Polaroid’s instant film, or even interested in picking up a Polaroidcamera, today may be your last remaining chance to do so, at least for a while.  Starting at 10:00 a.m. today, a number of Urban Outfitters stores in New York, Los Angeles, Cambridge, MA and Vancouver BC will begin selling remaining Polaroid Supplies.  On August 28, anything that’s left will be sold at these store locations.

This sale is in partnership with the Impossible Project, which purchased up Polaroid’s last stocks of instant film (theyceased production in 2008) as well as the last factory to produce it, in the Netherlands.

The Impossible Project’s eventual aim, according their site, is “NOT to re-build Polaroid Integral film but (with the help of strategic partners) to develop a new product with new characteristics, consisting of new optimised components, produced with a streamlined modern setup. An innovative and fresh analog material, sold under a new brand name that perfectly will match the global re-positioning of Integral Films.”

A documentary on Polaroid film’s final year is also in the works.

For two years, owners of Apple’s iPhone have complained bitterly about the lack of video recording capability, something that has become more and more common on mobile devices, if not yet matured.  Three weeks ago, they finally got their wish, and the results have been quite dramatic.

The weekend after Apple’s latest reveal – the iPhone 3GS – was released to the public, YouTube reported a massive 400% surge in the number of mobile video uploads, attesting to the mass appeal of Apple’s product and opening the floodgates for new video content.

Admittedly, seeing the sudden unleashing of newly-minted cellphone videographers made me cringe a bit, at first.  For a while now, I’ve been kind of annoyed with the popularity of cell phone video.  Let’s face it, although lots of people want something small and compact to make quick videos with, the image and sound quality coming from these devices has been utterly deplorable.  Blocky video, muddy sound… in general, not something you really want to store cherished memories with.  But what really stuck in my craw was when news sources started relying on cellphone videography as poor substitutes for actual newsgathering.  I began to ask myself: When did yanking video off a YouTube account start to pass as acceptable broadcast-quality content?

Within the past couple of months, however, things have changed quite a bit.  It actually started when camera makers like Canon began including video capabilities into their Digital SLR cameras, such as the Canon T1i.  Although it looks nothing like a camcorder, this model actually records some incredibly good HD-quality video, and can store it on a postage-stamp sized SD card.  It’s still rather bulky and expensive though; not the kind of thing an average person on the street would carry with them at all times.

Apple, however, changed that.  They finally decided to include video capabilities on their latest device.  And, they did the job so well that some aspiring independent film makers are now shooting films solely with the iPhone.  While the critical acclaim of the subject matter is best left open to judgement, one can’t deny the video quality is astoundingly good, compared to what everyone is used to from a cellphone.

Additionally, a professionally produced music video has been recorded and edited using the same gadget. But you wouldn’t know this came from an iPhone unless someone told you in advance:

Of course, the raw video underwent a lot of post-processing before yielding the finished product we see above.  But the unprocessed, full-color raw footage direct from the iPhone camera shows that it’s certainly no slouch on its own:

The ramifications are clearly huge.  There’s been lots of talk about how accessible good video recording technology has become, but now the technology to have a high quality video recorder literally in your pocket is available to the masses.

The technical specs of the videos created aren’t lightweight by any means, either.  Some users have reported that an hour of video from an iPhone 3GS can take up as much as 8GB of storage.  Still not on par with the roughly 20GB per hour that broadcast-level DV video consumes, but not anything to sneeze at, either.  The wide availability of this level of video production is going to require lots of supporting memory and storage to back it up.  And the metadata and details surrounding each and every video produced could be enough to turn the average cell phone user into amateur video catalogers, as well.

For professional curators, this could be both good and bad news.  From my perspective, it’s good that there’s a better chance history-making content will actually be recorded on a camera worthy of capturing it, as opposed to poor specimens of history like this. On the other hand, this means that good quality video will be coming from multitudes of sources, more than we’ve ever been accustomed to, as more and more individuals have the technology within their grasp.

One of the more frequent debates that I see cropping up often in preservation circles is how best to preserve “born digital” photographs: those photos that never began as physical film, but originated on a digital camera.

This isn’t an easy topic. There is no industry standard for born digital image preservation. Digital cameras of different vintages and configurations will output in one of a handful of differing file formats, and their metadata will often differ as well. And so, preservationists have been largely left to their own devices, fabricating their own methods, preferred formats and storage procedures for handling this type of material.

One controversial method that has been suggested is to forget about digital altogether, and to use a pigment-based inkjet or die-sub printer to print physical copies of digital photographs and rely on the hard copies as the long-term archive. This is a tempting method for lots of curators who have been trained to trust the physical, and without delving too deep into the specifics this seems at first blush like sound reasoning.

Unfortunately, it can be a very bad idea, and here’s why.

Loss of image fidelity

This is by far the most important reason, and yet not really the most obvious to some. For laypeople, and for the less-experienced in digital formats, creating a print from a digital files is a lot like doing the same from analog film. However, inkjet and photo printers are not going to give you the same level of quality as a true analog photographic print. And the print, while fine to the naked eye, will suffer a significant degradation compared to the original.

The best way to prove this is to take a digital image, make a print, and then rescan it. Here, for instance, is a born digital image taken from a Canon EOS 30D, shot and preserved in Camera RAW format, and presented here as a 24-bit PNG file:

(Note: clicking on the above image will take you to the full-resolution photograph, a 16MB file.)

I printed this image on a Kodak Photo Printer, using pigment inks, on 4×6 Kodak photo paper. Then, I rescanned the image at 1200dpi, using the scanner attached to the same photo printer. Here’s the resulting re-scan:

(Note: clicking on the above image will take you to the full-resolution re-scanned photograph, also a 16MB file.

At these reduced resolutions, there doesn’t seem to be much difference. The color appears slightly off, but it isn’t so bad… right? Well, let’s look a little closer at the re-scan:

Yikes! Clearly, there’s a significant compromise in image quality here, and this is because photo printers, regardless of how good they are, rely on printing methods that are unlike the traditional photograph, and through which the same level of quality doesn’t translate if you’re doing a bit-per-bit scan. This becomes even more evident when you compare the re-scan with the digital master, at the same scale.

If this argument isn’t compelling enough, there are other reasons for not relying on a hard copy as your preservation master.

Loss of technical metadata

Most modern digital cameras embed technical metadata into their image files, either by using EXIF, or as built in fields into their own Camera Raw format. This information can contain information about the camera which too the photo, what settings were used, what lenses, time and date, and even the GPS location of the camera, i properly equipped. It goes without saying that all of this potentially valuable metadata is lost if a hard copy is used as a preservation master, in lieu of the digital.

Limited ability to adjust or enhance the image.

Having and preserving the original file created by a digital camera affords a curator, editor or researcher a great deal of leeway and making adjustments to derivative presentation copies. Things like localized color adjustments are very easy to do with the digital master present, particularly if the master is a Camera Raw. On the other hand, your options are very limited if all you have is a print.

The best practice: preserve the digital

The best option for preserving born-digital photos remains keeping them digital. This does have implications for curators wanting to do right by their collections, and it can make the uninitiated very anxious. Capital purchases for technology, backups, and whole new workflows and best practices must be established. Fortunately, the world of digital curation is starting to come into its own, and others have already begun to tread these waters. In future articles, I will outline some best practices and case studies I’ve undertaken and encountered, to help guide those seeking answers to the digital dilemma.