1/2 inch, consumer, open reel video format used in the 1960s-70s. This format suffers from Sticky Shed Syndrome, making playback difficult. Working machines for this long-obsolete format are scarce. Source: IUB Media Preservation Initiative, used with permission. Note: IUB has asked me to stress that the above photo is not representative of all media collections at the university.

The Media Preservation Site at Indiana University – Bloomington (IUB) makes its message loud and clear the moment you first set your eyes on it: “Our History is At Risk.”

Home to at least 3 million media objects, including sound and moving image recordings, photos, documents, and artifacts, IUB has come to grips with the issue at hand: a great deal of their heritage is locked within obsolete electronic and analog playback formats for sound and moving images.  As an old format becomes obsolete, it gets harder by the day to find working equipment to play back these objects.  And that assumes that the objects can be played back, not having succumbed to age, wear and physical decay. Lacquer from old aluminum audio discs can delaminate, making them unplayable. Video tapes from the 1960s, 70s and 80s can suffer from a condition where the binding agent that holds the recording material to the plastic base sheds, allowing audio and video recordings to literally flake into nothingness. Film, too, has its own serious decay problems.

With its vast collection, faculty and staff at IUB knew the situation could become serious if nothing was done.  Their first step was to take stock of the situation, and consult outside experts (myself included) to get input how how best to address the problem.

Their efforts began nearly 18 months ago when a group of IUB faculty and staff, concerned about the potential fate of important special collections on campus, approached their Office of the Vice Provost for Research about the critical issues of media, and to impress upon them that time was of the essence to address these issues.

“Even though [IU Bloomington's] needs are now documented, and it is far better equipped than most universities in the country to meet them, there is no guarantee that IU can adequately preserve its collections in the near future.”

— The State of Recorded Sound Preservation in the United States, Council on Library and Information Resources for The Library of Congress, Washington 

The culmination of their efforts to date have been documented on the IUB Media Preservation Website, where they document their comprehensive effort to preserve IUB’s vast audio, video, and film holdings.  Some important documents from their study and efforts including IUB’s Director of Media Preservation Services Mike Casey‘s  Media Preservation Survey (PDF), outlining the collection holders, preservation stakeholders, the risks involved, and potential preservation strategies.  A follow-on public report (12MB PDF) also lays out the situation and what steps are being taken to save their special collections and historic content.  Continued engagement, updates, and discussion on decisions made and procedures undertaken are regularly made available on their Media Preservation Blog.

The IUB Media Preservation Blog

IUB has many years of work ahead of it, not only to transfer older content into more modern digital formats, but also to continue to maintain those archives, preserve new content, and keep pace with new technologies and formats to ensure that their collections are accessible.  It’s encouraging to see them in action, and their efforts stand as a potential framework for other organizations in a similar bind to model their initiatives after.

A cloud storage datacenter, housing multiple storage bays. Each red box holds 135 Terabytes of storage. Source: backblaze.com

Previously in this series, I focused on local storage; primarily, hard drives and similar media were discussed.  But a lot has changed since that last article was posted!  In particular, flooding in Thailand threatened to severely restrict the supply of hard drives, and retail prices had doubled and nearly tripled for the most common capacities and models.  Pricing for hard drives has begun to stabilize, but supplies are still constrained, and the situation is not expected to go back to anything resembling normal until March 2012 at the earliest.  For you and I, this means that the price of a hard drive, or even a new computer, might bit higher for the next few months and harder to come by, until the region can recover and production of hard drive components can resume.

And so, it makes sense to look at other solutions for backup strategies, with cloud storage being a lead contender.  Cloud (or online) backup services have become quite popular over the last few years.  They offer an attractive option for keeping your stuff safe: for a fee, you get the ability to send your files to a remote datacenter, where maintaining the storage and hardware required for backing up all of your data becomes the responsibility of the backup service you subscribe to.  They upgrade the hardware when it needs to be upgraded.  They fix and replace hard drives that go bad. The idea is to further simplify the backup process so that even buying hard drives and hooking them up to your computer aren’t part of the equation.

To be sure, cloud computing providers will also be impacted by the hard drive crunch.  After all, cloud computing is nothing more than large scale computing and storage platforms housed in data centers and they, too, must buy hard drives and equipment from many of the same vendors that businesses and consumers would get them from. However, such large scale outfits still have an advantage in that they buy in bulk, and can sometimes ride out price spikes for hardware with the excess capacity they’ve already built up.  So, if there is a price hike in cloud backup services, it might not be as huge as the cost of retail hard drives.

In any case, cloud backup solutions have proliferated quite a bit, and users have many options.  But, how does a cloud backup work?  And what should you look for in choosing a cloud storage vendor?

What the cloud can do for you

The primary promise of cloud backup services is that they seek to offer a hassle-free experience for their users.  There are no hard drives to buy or hook up, and like modern local backup software, the backup operation can be a set-and-forget affair.  What you need is a computer of fairly recent vintage, a broadband internet connection,  and some software that runs on your computer to monitor and schedule the backup processes.

In most cases, you can tell the cloud backup service what files or folders  you want backed up and which you’d prefer they not touch, but by default, most will try without any specific instructions to backup your computer’s entire primary hard drive.  From there, you can expect an initially large backup session to start with, as the software sends a full snapshot of all of your files to the cloud.  Form that point, backups are incremental: only new or changed files are then uploaded to the cloud backup service going forward.  And if it turns out that you need to rely upon that backup later, you can either download the whole archive to rebuild everything, or just select individual files that you might need.

Online backup services generally come with a fixed monthly or annual fee.  Some people might find this more attractive than the upfront cost of buying hard drives for backup purposes, as the cost is typically pretty low.  For this, you might get a set mount of storage that you ca use to back up whatever you like, or you might get an “unlimited” amount of storage offered by certain services.. though they sometimes carry limitations on how much you can upload in a given month.

The Advantages

Set and Forget. For many, the allure provided by online backup services is that it usually provides a truly stress-free option for backing up your data.  As long as you pay into the service, you won’t have to worry about buying and maintaining hard drives. All of that is taken care of by the backup provider, who can enjoy  economies of scale by buying all of their storage and needed hardware in bulk.  As a result, multiple customers sharing the resources of a massive storage system somewhere means that the online backup service can usually buy their hard drives for a lower price than most individual consumers can.  Which brings us to our next advantage…

Price. Most online backup services aren’t expensive.  Some offer the service for as low as $5 a month for a single computer, or $50 for an entire year, regardless of how large that computer’s drive may be.  Such low price points make it easier for most users to want to try such a service.

Off-site. online backup services are, by definition, remote operations. This means that you backup is usually pretty safe, either ion a secure server room, or spread out over multiple locations.  This is ideal in a case where your home or office might suffer the effects of a disaster, and an on-site backup might be destroyed along with the backed-up computers.  With the cloud, your work site might be gone, but your data isn’t.

Expandability.

To allay most users’ fears about privacy, most online backup services offer encryption capabilities, so that only you (or someone with your password) can gain access to what’s in your backup.

The Drawbacks

Although the advantages make cloud backup service compelling, there are some caveats that must be considered as well.

High Bandwidth Usage. Online backup services, by virtue of storing your content in remote locations that are accessible on the internet, will use your internet connection to make those backups happen. It’s important to carefully consider the capabilities and limitations of your internet connection.  High-speed broadband is absolutely required to make cloud services work. Consider that most computer hard drives can store hundreds of megabytes – in some cases even terabytes – of information.  If you choose to backup everything, your very first complete backup could take days to complete, although subsequent backup sessions will only record changes to your files, and should take a much shorter time.

Another important consideration is whether your internet provider enforces a usage cap.  Some ISPs, like Comcast and AT&T in the US, and Rogers in Canada, limit how much data you can transfer in a monthly period, ranging from as little as 50GB to 250GB.  Other ISPs, like Verizon, do not impose such limits.

These problems can be mitigated through some careful timing of your initial backup session.  Your cloud provider might offer a feature that lets you schedule when and how much data gets backed up in a set amount of time, helping you “budget” the usage of any limited broadband services.  Some services even allow you, for a fee, to send in a hard drive with your initial backup image, to save time and bandwidth.

Longevity of the backup service. Another concern is whether the online backup service you’ve chosen has been in business for a one time, as stands of good chance of staying in business.  Startup services like Backify have entered the fray and then abruptly shut down, leaving users who thought their files were being safely backed up suddenly in the lurch. Even big names like Vodafone and HP are not immune from instances where they tried to offer backup services to the masses, but found they couldn’t make a profit and chose to shut down.

You can help mitigate this by having a good relationship with the provider you choose.  See if they have a blog, Twitter feed or Facebook page that’s regularly updated, and keep checking it to ensure they are active and still dedicated to serving their customers’ backup and storage needs.

Some other things to consider

Privacy and encryption.  It’s very likely that you’ll want to backup sensitive information, such as personal documents, financial data, saved passwords, and other information you might not want others to see and have unrestricted access to. This is why, if you’re considering the cloud, that you make sure the cloud backup service you’re considering offers a private key encryption feature.  This will allow you to encrypt your data locally, on your computer, using a password that you set. Effectively, this prevents even the backup service from gaining access to your files unless you provide the password to unlock your backup. And, remember to use a truly strong but easy to remember password to secure your backup.  With a private key system, your data is so secure that if you lose or forget your password, you’re out of luck!

Also be sure to carefully read the backup service’s privacy policy, and make sure you are comfortable with any limitations they spell out.

Getting your files back. A good backup service will not only clearly explain how their service works, but how you’re going to get your files back when you need them. Ideally, you should be able to browse the contents of your backup image online (after you provide you encryption password, of course!), and select either the individual files you need, or pull down your whole backup image to recover from something catastrophic.

The control panel for BlackBlaze's cloud backup service. Here, the status screen tells us the computer is mostly backed up, save for about 75MB in newly created files that will be uploaded at 2:00 am the following morning, presumably when no one is using the internet connection.

Some common cloud backup providers

No one backup service is perfect for everyone.  Each cloud provider offers different features that work best for different users, so it’s a good idea to research a few before making a decision.  Below, I’ve listed a few major backup services for desktops that have been around for a while, have been highly recommended by colleagues, and appear relatively stable:

Carbonite: Probably the most heavily advertised backup service, Carbonite offers “unlimited” backups for as low as $59 per year.  There is a catch though: “unlimited” on the basic plan refers to files in the “User” folders of your desktop, and user generated files.  Software and operating system files aren’t included, unless you play for premium services.  Carbonite will also slow down the speeds of backups that exceed 200GB in a month.

Backblaze: Backblaze provides native Mac and Windows backup support, and offer unlimited backups for $50 per year without any catches. They also allow you to budget your internet usage, allowing backups to occur at night, and allowing you to set limits on how much data is sent up within a set period of time.  Their blog also details the insights of the company, keeping everyone up to date on their efforts, and even describing the custom hardware they use to provide their service.

Incidentally, it is their ease of use, reasonable pricing and consistent openness that makes Backblaze the service that I use for my cloud backup strategy.

Mozy: This service is a bit more up front on their limits, with their home service being offered as either a 50GB or 120GB plan. With that limitation though, comes perks; namely, the ability to access your backed up files on iOS and Android mobile devices.

Crashplan: This service has a bit of a unique crowdsourcing-type model for backups.  In addition to the traditional cloud backup service, Crashplan also allows you to share your backup resources with friends and family, for free. It’s worth having a look if you have a network of friends you can rely on with fast internet connections that are willing to pool storage resources with you.

These and many other services are available for you to try out Cloud backup services.  They definitely have a place in many users’ personal data security plans. I suggest checking some of these services out and see if they fit your needs!

An External Hard Drive: the easiest method for making a quick copy of your important stuff. Photo taken by flickr user Miss Karen

All, right, so you’ve heard it over and over and again, and you know it’s true: you need to make regular backups of your stuff. But how?  What options do typical computer users have?

In the past few years, the options for preservation and backups have expanded quite a bit, giving users an enormous array of solutions to choose from.  Of course, the diverse options can be confusing: what’s the best choice for you?

The backup options out there can be grouped into two major categories: local storage and cloud storage.  Each has their strengths and weaknesses, and will appeal to different users based on where and how they use their computers.  Some of the best and most secure backup strategies make use of both solutions… a backup-of-the-backup, so to speak.  I’ll discuss that further in a later write-up.

In this article, I’ll talk about local storage.  In the next article, we’ll go into cloud-based solutions.

Physical Media

Local storage is pretty straightforward stuff.  It involves using what we commonly call physical media; something you can hold in your hand, physically insert, remove, connect or disconnect with your computer.  In the past this was pretty much the only option available to most people on their home computers, and so it’s the one people are most familiar with.

Until recently, backing up with local storage often meant using removable disk media like CDs and DVDs to make copies of small groups of files at a time (or if you were really old school, you might remember 100/250MBMB Zip Disks, 1/2GB Jaz Drives,  or even floppy disks and old, expensive backup tape drives).  And for some people, DVDs CDs and Blu-ray disks are still the medium of choice for making copies of their data and their work.  It requires a lot of juggling of multiple discs if you have a lot of stuff to back up, and is often labor intensive too, but can work reasonably well if your needs require it.

But more modern solutions are faster and easier to work with: External drives using USB, FireWire, Thunderbolt and LightPeak interfaces make quick work of adding lots of storage and making thorough copies.  And making it even easier are the built-in backup services now available in Windows (starting with Vista) and Mac OS X (starting with Leopard) that automate the backup process, without your having to buy any extra software to make it all work.

Costs

Most of the investment required for local backups is up-front: you will need to buy your storage media.  Often, that’s an external hard drive, and as of this writing, a relatively roomy (for now anyway) 3 TB external USB hard drive can be had for about $130, with lower capacities going for half as much.  Software might also cost you some additional money if you choose to go with third party vendors, though most modern operating systems now offer a built-in or easily-installable backup solution.  Most of this article will be spent talking about those built-in solutions, because they work quite well for most people.

Pros and Cons

No doubt, the physicality of the backup – the fact that it’s right there, on separate box or separate set of media – is the prime advantage for local backups. You know where your data is, and you know how safe ti is at all times.  It’s also relatively inexpensive. Once you buy the media you need to handle the backups (and any software), there tend to be no ongoing costs to keep things running smoothly.  Of course, once your backup storage fills up, you will need to decide whether you want to buy more, or are simply happy with making room on the existing storage by dropping old copies in favor of newer ones.

The disadvantage to this physicality is that care must be taken in the event a catastrophe of some kind takes out both your computer and  the backup.  Having your backup hard drive and your computer in the same room won’t help you very much in the event of theft, fire, flood or other physical disaster that wipes out the hardware. For this reason, if local storage is all you plan on using for your backup strategy, it might make sense to buy more than one set of hard drives for example, make backups on both, and keep one in an off-site location, just in case.

Windows: Backup and Restore

Windows Backup and Restore is useful for taking a total snapshot of your computer, or if you’re the tinkering type, you tell it exactly which folders and files you want to keep backup copies of.  Paired with an external drive, you can tell it how often you want you want to refresh your backups, and how many different versions you’d like to keep.

Backup & Restore has two basic modes: disk image and incremental backup.  In the former, a total snapshot of your computer’s hard drive and all of its files are kept in an archive.  In the latter, the backup service adds to the archive as it goes, making note of files that are added, removed or changed as time goes by.

Backup & Restore was made available starting with Windows Vista and 7, and is a nice, ubiquitous, easy to use utility that really made backups simple for Windows users.  For previous versions of Windows (including XP), there is a built-in service called NTBackup, but it’s far less user-friendly, not quite as automated and not commonly used.

If you’re still using Windows XP, you might want to consider some third party options for local backups.  Vendors such as Acronis and Symantec offer solutions that are just as automated, but of course will cost some extra money.

Mac: Time Machine

For the Mac, Time Machine is an equally-ubiquitous solution that was released in late 2007.  The instructions for using Time Machine are incredibly simple: plug in an external drive, and your Mac will ask you if you want to use it for backups.  Click “Use as Backup Disk,” and that’s it!

The features Time Machine offers are also leagues above most other backup software out there.  The files that changed aren’t just updated in a flat incremental backup, but actual versions are kept.  You can virtually “go back in time” with your computer’s hard drive and see what changed when, hence “Time Machine.”  And it does all this with a minimum of storage overhead: aside from your first full backup (which takes up the most space), each additional snapshot of your Mac will only take up whatever space is absolutely necessary to note what’s changed since the last time the software had a look.

This level of richness does come at a price though: the settings and intervals that time Machine uses for backups aren’t customizable.  So, whether you like it or not, you’re getting snapshots of your computer for every hour over the past 24 hours, every day for the past 30 days, and every week past that, until your backup hard drive gets full (at which point, the oldest backups are discarded to make room for newer ones).  As a general rule, Time machine will try to backup everything on your hard drive, but you can give it a list of which individual folders you’d rather it not keep copies of, should you be so inclined.

If full, detailed customization is absolutely what you need, then fortunately there are still a few pieces of third party software out there that can do the job.  The most commonly recommended (outside of Time machine itself) is Carbon Copy Cloner, which will let you schedule and specify the times and levels of thoroughness your backups should have.

What About Linux?

The Linux platform has  a lot of great virtues going for it, primary of those being that there is a rich, diverse mix of developers working hard to contribute to it, and that diversity means there’s so many different flavors of the operating system that people can chose from.  The same is true for the type and number of backup utilities that exist out there.

However, the double-edged sword to this diversity is that there really isn’t a single go-to utility that is already bolted on and ready to go.  Though, if you’re an avid linux user, there’s a good chance that you’re well aware of the diversity out there, and probably chose your platform on that very basis.

That said, I would highly recommend looking at this article: 10 outstanding Linux backup utilities.  It pretty much runs the gamut of the more commonly used solutions available, and they vary widely in their ease of use, customization and features.

I’ll also take this opportunity to point out that I haven’t even come close to an exhaustive list of local backup software for Windows or Mac platforms, and searching around online will no doubt yield plenty of other options.  That said, the fact that both of these Operating Systems have recently incorporated their own easy-to-use (and free) backup solutions, the field of vendors has shrunk quite a bit in recent years.

Plain old copying

Last but not least, there’s the tried-and-true method of making a backup that works on every operating system out there: the good old manual copy.  By far this is the most customizable option: nothing gets backed up unless you make it happen.  The bad news?  Well… nothing gets backed up unless you make it happen, either through dogged determination and remembering to run the backup procedure, or fashioning your own script or bit of code and scheduling it to run at a certain frequency.

If this is the way you’d like to back up your stuff, then more power to you.  However, based on the experience I’ve had (both my own stuff and with how colleagues and friends behave), I’ve found this to be the least reliable way of making sure your stuff is kept safe.

What do I use?

Since 2007, I’ve been a Mac user, so my laptop and desktop systems are faithfully and easily backed up on local drives using Time Machine.  I’ve used it on multiple occasions to restore accidentally-deleted or changed files, and have a couple of times used it to do full restorations (including once when my laptop was stolen, and all that was left of the stuff on it was the Time Machine backup image on a hard drive kept at home).

I also use Linux desktops and servers for a number of projects (including this blog), and maintain local backups of those systems using rsnapshot, a nice utility that runs in the background and makes periodic incremental backups of everything I tell it to.  In many respects it has the functional utility of Time Machine for the Mac, though it lacks the pretty graphical interface.  I highly recommend it if you run linux servers and want a complete but basic, no-nonsense local backup strategy.

A hard disk drive with damaged platters, caused by a head crash. The data on this drive is not recoverable.

Whether we like it or not, those of us who rely on electronics to get our work done are guaranteed one thing: a data loss event.  This means that at least once in our  lifetimes (and sometimes more than that), every one of us who uses a computer, laptop, tablet, smartphone or similar device is going to one day stare at our screens and realize that the piece of information we expected to be there, just isn’t.

It can happen any number of ways.  Sometimes, we users make a mistake and accidentally erase something we shouldn’t have… or someone else might’ve accidentally deleted something of ours that they shouldn’t have. Other times, it’s the computer’s fault: buggy software might’ve claimed to save something but didn’t, or a 10-year-old hard drive finally decided to give up the ghost.  And sometimes, acts of nature (power outage, natural disaster, or other events beyond our control) will intervene and cause vital work to be lost.

Of course, we’ve all heard it time and time again: to protect your documents, photos, drawings, artwork, and other important data, you need to have backups.  Unfortunately, while we all have heard this before and know it to be true, we don’t always follow through.  In the past it’s been tedious to do regular backups; a chore we all dread.  And so, it always falls but he wayside, and often, we get back into a backup regimen only after something bad has happened, and it’s already too late.

But take heart.  A lot has changed recently.  There ARE personal backup solutions out there that are surprisingly easy… and even automatic!  keeping your stuff safe doesn’t have to be a tedious chore anymore… as long as you’re willing to invest a little time and effort at the beginning, and in some cases a small amount of cash on an ongoing basis.

Not only are there different options, but different methods for backing up your online stuff.  There’s hard disks, removable media, cloud storage, duplication… even tape if your needs are hardcore.

Obviously, the uninitiated (and maybe even some veterans using older methods) will have questions.  Like, what IS cloud storage exactly? Is just dragging-and-dropping my files to another hard drive good enough?  Do I really have to buy extra software?

Over the next few days, I’ll be posting a series of articles that lays out all of the options, why they work best for, and describe whether and how they might fit how you work and play online.  Hopefully most of your questions will be answered, or least these articles will help arm you with enough information to get you started.  And with luck, the information provided will help make sure that you don’t have to face an expected data loss disaster without a well-rehearsed Plan B.

Michael Stern Hart and Gregory Newby, founders of Project Gutenberg, an effort to digitize and make available public domain books in electronic forms.

My colleagues and I sadly mourn the passing of a pioneer in digital curation.  Michael Stern Hart, the founder of Project Gutenberg, has passed away at age 64.

While not a household name, Hart has an important place in modern technology, given he is credited as the inventor of the eBook.  His contribution can be felt every time someone reads a book from a Kindle, Nook or iPad.

Some might say that fans of traditional paper books, and those bemoaning the demise of bookstores such as Borders, might take issue with his work.  But everyone should agree that his legacy will be the irreversible revolutionizing of how we read, going forward.

From Project Gutenberg’s obituary for Mr. Hart:

Hart was an ardent technologist and futurist. A lifetime tinkerer, he acquired hands-on expertise with the technologies of the day: radio, hi-fi stereo, video equipment, and of course computers. He constantly looked into the future, to anticipate technological advances. One of his favorite speculations was that someday, everyone would be able to have their own copy of the Project Gutenberg collection or whatever subset desired. This vision came true, thanks to the advent of large inexpensive computer disk drives, and to the ubiquity of portable mobile devices, such as cell phones.

Project Gutenberg is an effort to freely make available a collection of 36,000 out-of-copyright books that were painstakingly proofed and digitized into various open digital formats.

I spend a lot of time on this blog talking about how to keep “big data” safe: research datasets, archival documents, that sort of thing.  But, the lessons we learn here are also pretty useful for smaller setups, artists, businesses… and people at home.  Everyone can benefit from digital preservation.  Everyone needs to know how they can keep their digital stuff safe.

And so, I’m partnering with Intervention ’11 to get the message out about how to keep your digital items of value accessible for the long run.  I’ll be holding a panel titled “Keeping Your Stuff Safe: Strategies for Preserving Your Digital Life.”  In that panel, we’ll be talking about ways to easily, affordably, and safely back up your digital media, for now and the future.

 What is Intervention?

Our goal is to bring together fans and different independent creators to party, educate, and appreciate the opportunities the Internet gives to all of us. We are similar to the Sci-Fi/Anime/Other fandom cons that we love so much, but we are focused on people who use the internet as the primary way to distribute their work and talk to their friends.

It’s a great event for geeks and non-geeks alike who do just about anything related to internet culture to get together for a weekend.  Feel free to give ‘em a visit!

Cranberry Harvest in New Jersey. Source: USDA

A few months back, I wrote about our efforts to leverage RUcore for the benefit of the academic research community at Rutgers. The result is RUresearch, a place for Rutgers researchers to share their data with the global scholarly community.  This data sharing is particularly important in light of a National Science Foundation mandate to openly share research data that has been funded through them.

Over the summer, the RUcore team has been working with a few researchers to better understand their needs, and to work on preserving and sharing our first samples of actual research data.  In collaboration with the Philip E. Marucci Center for Blueberry and Cranberry Research and Extension, our efforts – if you’ll pardon the pun – have begun to bear fruit.

As part of funded by the U.S. Department of Agriculture Specialty Crop Research Initiative, Marucci Center researchers have extracted a genome for a cultivar of the cranberry; a fruit for which New Jersey is the third-largest producer in the US, devoting some 3,600 acres to its cultivation.

The genome research is part of a study in genetics of fruit rot-rresistance, and the data generated (using Applied Bioscience’s SOLiD 3 Plus System) takes up over 60GB of storage when compressed.  Sharing of this data to researchers who would find it useful obviously requires a system that can not only spare the storage, but be robust enough to permit open access.  Enter RUcore.

Although further refinements are in progress, the result of our collaboration is one of our first research data records in RUcore, located at this link.  The PDF attached to that record describes the link to the download point for the data sets.

While the data itself isn’t something the general public will easily recognize and interpret, the ability to share this information with other researchers can benefit all of us, through continued study into which genetic factors can make certain fruits resistant to rotting.  And it’s also a learning experience for us, in how to make that sharing among researchers a little bit easier.

On July 7, Sony announced that production of MiniDisc playback equipment would cease in September of 2011. According to Sony, the format’s creator, the blank MiniDisc recording media will continue to be manufactured for up a year beyond the players’ discontinuation.

MiniDisc never made as big a splash as Sony had hoped, at least in markets outside of Asia.  Introduced in 1992, Sony had envisioned that the format would be just as ubiquitous in the 1990s as the audio cassette – and another Sony invention, the Walkman – was in the 1980s.  Unlike Audio CDs, MiniDiscs offered a more compact design to increase portability, greater durability and anti-skip capabilities, and all MiniDisc playback equipment was capable of writing to recordable and re-writeable media from the outset.  By contrast, the first sub-$10,000 CD writers wouldn’t become available until 13 years after Compact Disc’s 1982 introduction to the market, and almost 3 years after MiniDisc was widely available.

Unfortunately, MiniDisc had barriers to adoption from the outset, most of which were placed – deliberately or otherwise – by the company who introduced the format in the first place.

The ’90s marked an era where audio vendors knew there was a growing appetite for improved, portable digital playback, but couldn’t agree on a single platform.  Consequently, the market was flush with differing standards, and MiniDisc had to find a place in an environment already crowded with other options vying to be the successor to analog cassettes.  Sony never really made an effort to rationalize this, and ignoring past lessons, such as the Beta-vs-VHS wars of the video world, instead seemed to think that this was a perfectly sustainable situation.  From Sony’s initial press release announcing MiniDisc:

 ”The announcement of this important new technology for personal listening supports our view that no single audio format can meet every consumer’s needs,” noted Ron Sommer, president and COO, Sony Corporation of America. “We do not see MD displacing any current formats. Instead, we expect it to co-exist with CD, DAT and other cassette formats, each of which meets specific consumer requirements.”

There were also limitations imposed which dampened consumer interest.  MiniDisc used ATRAC audio compression; a proprietary, lossy format which degraded the quality of the audio enough to cause most audiophiles to criticize the medium.  And even less-discerning consumers found that while MiniDisc seemed to be a competent digital format, digital duplication was hobbled by a Digital Rights Management scheme known as SCMS, severely limiting one of the format’s most attractive features.  Even worse, although the format was licensed to other audio equipment makers, MiniDisc was largely a Sony venture, with little interoperability.  But the largest inhibitor to its adoption was price: in North America, 1992-era MiniDisc models sold for between $800 and $1,000; out of reach for many casual listeners, and mostly only affordable by those same affluent, dedicated audiophiles who found its compression methods unattractive.

While it never really caught on in mass markets outside of Asia, MiniDisc did find a niche in some broadcast and studio operations. By the mid to late ’90s, the price of the recorders and discs dropped enough that smaller, independent artists and musicians found it to be a reasonably good digital recording medium.  The small size of portable recorders also made it a useful tool for journalists in the field, being both more durable and providing recordings of better quality than most tape recorders. And smaller radio and broadcast outfits – particularly college radio stations – used MiniDisc as a slightly cheaper alternative to Digital Audio Tape-based systems, and far more reliable than analog cart systems.

A Sony Vaio Desktop with a built-in MiniDisc drive. Source: minidisc.org.

Sony did try to further expand MiniDisc’s uses to improve adoption.  Limited data storage capabilities were added, and Sony manufactured a line of music-centric desktop PCs which allowed authoring of audio recordings to MiniDiscs.  An improved-quality, higher-density version of the format called Hi-MD made its debut in 2004.

Even so, MiniDisc’s days were numbered.  Having enjoyed success in these niche markets for a few years, MiniDisc slowly fell to the wayside as by the early 2000s, digital recording evolved into more fluid stateless, file-based systems.  The appearance of Flash Memory paved the way for solid state recording of audio and video, and hard-drive based playback devices allowed both consumers and producers to embrace audio file formats such as WAV, MP3 and AAC, playable on multiple devices regardless of the physical storage medium being used.

No doubt, the merging of media recording and playback devices with nearly-ubiquitous smartphones have only hastened MiniDisc’s demise. In an era where music is downloaded, synced, and played back wirelessly via multiple storage devices, or even using intangible forms of storage, the writing is on the wall for most physical containers of digital media, and MiniDisc could be one of many types of this physical media to go obsolete in the near future.

Some recent, high-profile security-related events are adding another wrinkle of complexity for those who are trusting the cloud for their data storage and content delivery: who your neighbors are, and what they might be doing.

On June 21, the FBI raided a Reston, Virginia based server farm for Swiss hosting provider Digital One.  While the agency isn’t commenting, the speculation is that they were looking for data related to a single hacker group - LulzSec – responsible for recent numberous high-profile security breaches waged against Sony Corporation and several law enforcement agencies.

Unfortunately, that raid entailed the physical removal of multiple pieces of server hardware that, among other things, served as the virtual, cloud-based home for dozens of other websites.  Most of these affected parties are presumed to be legitimate customers that were storing data or serving web content… conducting real business that wasn’t running afoul of any laws.

As a result, several high profile corporate content developers, including Instapaper, Curbed Network, and Digital One’s own website and support system, were either suffering degraded service or were taken completely offline for more than a day.  Without a backup, the data could have been lost indefinitely while the FBI conducts whatever investigation on whatever client captured their interest.

The ramifications of this event are clear: Cloud services are shared services.  One of the big advantages of the Cloud is the notion that multiple entities can share the same large datacenter and resources without necessarily having to buy it all themselves.  Unfortunately, it’s rare in a public Cloud setting that you are allowed to choose who you’re sharing your resources with.  Often, this isn’t a big deal, but if your “neighbor” happens to be attracting a lot of attention (from hackers or law enforcement agencies), then your data and operations may also be affected as a result.

This is yet another reason to consider having a backup plan, and not totally entrusting all of your data to a single Cloud vendor.

A Sony HVR-Z1U camera. This device is a digital video workhorse at the SCC, and relies heavily on digital video tape... something which could be rather hard to come by in the near future.

My heart, thoughts, and a donation goes to those affected by the Earthquake, Tsunami, and now radiological crisis that Japan must grapple with.  It’s not exaggeration to say this turn of events is truly unprecedented.  Sitting thousands of miles away, and only observing the events through websites and television screens, I’m aware that I cannot possibly grasp the ordeal that survivors now face.

With that preface, it’s difficult to even think at this point of how the disaster will inconvenience those of us far removed.  However, there will be a rather significant impact for quite some time, given our technological dependencies in a digital world, the number of electronic components and supplies that are produced in Japan, and how we use those components to capture our current history and cultural heritage.

Our first hints of trouble began with an advisory issued to consumers of magnetic tape media. Sony, a major manufacturer of various varieties of tape media as well as semiconductors, optical discs such as DVD and Blu-ray, and electronic components, has been hit hard.  Sony was forced to shut down a number of factories in the region while recovery efforts continue. The earthquake has forced a halt to production in various manufacturing facilities in Japan, including those of magnetic media manufacturers, and suppliers are now warning of an impending shortage and possible price spikes:

“Our industry has already been affected by a halt in media manufacturing operations – professional media supply shortages are evident, namely HDCam SR,” explained a post on the Comtel Pro Media web site. “Worldwide stock shortages present a realistic threat to our industry and the immediate needs of the television and motion picture production.”

Of particular note is a shutdown of the Sony Corporation Sendai Technology Center, currently the only facility in the world producing HDCAM-SR tapes.

HDCAM-SR Tape is widely used as a Master Tape recording medium in the broadcasting industry, and one of the formats at potential risk of a shortage as a result of the disaster in Japan.

In a landscape increasingly colored by the use of disc media, USB flash drives, hard drives and cloud-based storage, not many connected users think that tape is such a big deal.  But in reality, tape is still very widely used and plays an important part in what we see, hear and access online.  Quite a few existing studio, professional, and prosumer HD camcorders use tape to record video.  At our own facility, our standing policy is to use a combination of media when recording video for events: a digital Direct-To-Edit hard drive for speed in editing, and a MiniDV tape for archival purposes and to act as a backup in case the hard drive fails.

Tape is important in other media-rich fields as well.  Google’s recent e-mail outage would have been much harder to recover from had the company not made extensive of use of LTO data tape to back up users’ stored messages.  Tape is cheap and reliable, and although it’s painstakingly slow to play back and retrieve information, it serves as a very effective safety net when you absolutely need to keep your video, audio or digital data safe.

The most chilling thing about all of this are anecdotal reports of suppliers advising their customers to start re-using old tape stock to preserve existing new supplies.  History has taught us important lessons about reusing tape.  Countless television programs not deemed worthy of replay were lost when studio staff and executives chose to wipe their media in the name of cutting costs.  Even recordings of the first moonwalk from the historic Apollo 11 mission were all but lost thanks to the practice of tape re-use.

Given the possibility of having no choice but to re-use existing media and possibly lose things whose value is underestimated until after it’s gone, getting Japan on the road to recovery takes on a new sense of urgency in the cultural heritage  and preservation community.

The immediate human and financial losses are monumental, indeed.  But the historical impact could radiate far beyond the region.  It could ultimately affect what you watch, read, interact with, and how future generations recall – or lose – culture and history for many decades to come.  The inability to easily access historically important documents, photo, audio and video due to an inability to read obsolete formats is seen as a contributing factor to a coming Digital Dark Age.  The worst-case scenario fro this disaster could mean something just as bad: a loss of those records because a lack of anything reliable to store them on, at all.

Investigators are expected to share with other researchers, at no more than incremental cost and within a reasonable time, the primary data, samples, physical collections and other supporting materials created or gathered in the course of work under NSF grants. Grantees are expected to encourage and facilitate such sharing. See Award & Administration Guide (AAG) Chapter VI.D.4.

To that end, researchers are now required to submit a Data Management Plan with their grant requests, detailing how the project will comply with research sharing guidelines set forth by the NSF.

These requirements leave researchers with a choice: either come up with a plan on their own, or seek help from their institutions on a comprehensive data sharing and preservation model.  Fortunately, the resources and tools exist at Rutgers for its researchers to easily take the latter route.

In anticipation of these data sharing requirements, the university has setup a site to guide researchers through the ins and outs of data sharing.  The Rutgers University Research Data Archive site clearly explains the importance of sharing and preserving research data, and details some of the current offerings for researchers who need a platform to share their research data to comply with NSF guidelines.

It goes without saying that one such option listed on the site (and the platform I recommend) is the Rutgers University Community Repository.  In anticipation of this need, the RUcore team has developed the RUResearch Data Portal, a section of our digital repository meant specifically for serving research data needs.

Already trusted by faculty members to store their academic publications, and the mandatory platform for Theses and Dissertations in the Graduate School of New Brunswick, RUResearch is a natural extension of RUcore’s mission to preserve and make accessible the university’s academic output from a centralized resource that adheres to established digital preservation standards.  With RUResearch, you can not only be assured of meeting NSF’s requirements on paper, but you will also have the security of knowing your research data is truly safe and preserved.

More information on data preservation services can be found on the Rutgers Libraries Website, including dates for in-person presentations on the services we offer the academic research community.  And, if you are a researcher interested in how RUcore and the RUResearch platform can help you, contact our Data Services Librarian, Ryan Womack, and he will be able to give you the information you need to get started.

GMail kept users notified through a status page of their ongoing recovery efforts.

This past week offered up a little dose of panic to an estimated tens of thousands of users to Google’s free Gmail service, when they logged in to discover that all of their e-mail was missing.  According to Google:

We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version.

0.02% may not sound like much, but with an estimated 193 million users, that means that close to 39,000 of them had to briefly cope with the unpleasant possibility of losing thousands of e-mails, that could span several years for some of GMail’s earliest adopters.  In fact, some sources put the estimate of affected users somewhat higher.

To their credit, Google is working on the problem and insists that all e-mail for everyone affected will be restored by the end of the day.  How? Google uses tape backups to archive the contents of your Gmail account, which is their assurance against data loss.  Still, Google’s web-based applications, including e-mail, are the primary experience most users have with a type of service offering known as Cloud Computing.  And a threat of users losing important data stored on the Cloud is understandably casting doubt on the confidence of users in this type of offering.

From a data curation perspective, this is an important event.  Although in the end no harm was done, this incident illustrates how willing most users are to fully trust an online service with data they feel is important, and even critical to their personal histories, without giving a thought to what’s being done to keep that information safe.  Most Gmail users – or users of Facebook, Yahoo, Microsoft Live, or Flickr (which had its own high-profile data loss recently) – aren’t aware of what kinds of backup plans, if any, are put in place by these service providers.  And it’s likely that few of them have made backups of their own.

So, what can we learn from this experience?  A few things…

1. The cloud isn’t perfect.
   Contrary to  what Microsoft would like us to believe (illustrated in the video above), the Cloud isn’t a magical place that solves every media problem.  In fact, it’s not a whole lot more complex than the computer equipment that sits on your desk.  The Cloud is little more than a bank of servers with lots of storage located in datacenters in (sometimes) remote locations, connected by high speed internet. And much like what sits on your desk, that equipment will eventually suffer errors and failures.  In most cases, there are plenty of backup and redundant systems in place to keep these failures from being noticed.  But once a while – much like what we saw this week – a perfect storm can brew that causes one of those incidents to get very, very public.
2. Making backups is still a very good idea
   Although Google (thankfully) maintains their own backups, people should still keep in mind that services such as Gmail are offered to the public for free.  Although Google would have earned some very bad publicity had they not restored the lost data, there really isn’t any agreement or guarantee between Google and most of its users to protect any content that they store for you.  As a result, the responsibility for keeping your data intact still falls on you, the user. 

   Fortunately, Google does offer basic instructions on how to use common desktop programs to make local backup copies of your e-mail.  Other tech sites are also joining the fray with more detailed descriptions and alternatives.

   Definitely consider this week’s events a warning… if it’s important to you, then it’s time to back up your mail.

3. GMail does make backups of their data… which could be good news and bad news.
   The good news: most people didn’t realize it, but Google does have a backup plan for the media you store on their servers.
   The bad news:  most people didn’t realize that Google does back up the media you store on their servers. 

   How can this be both good and bad?

   For those solely concerned about the longevity of their e-mails, this is a good thing.  Google is making an effort to preserve the content that users generate and store, for free, on their platform.

   For those who are concerned about their privacy, on the other hand, this could be a very bad thing.  We now know that Google uses tapes as a backup medium.  What we don’t know: How long are those tapes kept, and where?  Are those tapes transported off-site?And if so, are they encrypted to protect against the risk of theft in transit? And if someone deliberately deletes e-mails from their GMail account, are those deleted e-mails still kept on tape somewhere?  If so, for how long?

   For most people though, the prospect of Google backing up their e-mails is a mixed blessing: nice to know they take steps to keep the data guarded against loss, but the unanswered questions might lead that same user to be concerned about potentially sensitive personal data (e.g. links to bank statements) being stored in an unknown medium, for an unknown period of time.

Conclusions

This event should be seen as an opportunity for users of Cloud services – Gmail, Google Apps, Mobile Me, Live, Yahoo, Flickr, Facebook, and even more complex business offerings – to reflect on how important that stored data is to them, and what impact it would have on their lives if that content were lost.  This isn’t to say the Cloud shouldn’t be used or trusted – quite the contrary; it seems that Cloud providers are taking the importance of that data and its integrity seriously.  But, that’s not always enough.  Users of the Cloud need to be sure that they are well-informed about how safe their content is… and if they have doubts, some simple backup procedures to non-cloud media should do the trick.

This is also a time to extra-mindful of the implications of storing your content on the Cloud, and what that means to risks to your privacy.  Perhaps it’s time to call upon these service providers to be more transparent about what actions they take to secure not only the integrity of the data they store, but the privacy of their backups as well.

The drumbeat continues to sound for the preservation of obsolete and endangered moving image formats, particularly videotape.  As older tape formats become unplayable, either through decay or lack of equipment to play them back, the urgency grows to find ways to preserve their content using modern digital formats.

The problem has been considered by multiple organizations acting separately over the past decade, and all of them have wrangled over the same question: What digital format should be used to preserve this content in the digital space, and help ensure that we aren’t finding ourselves in the same obsolescence predicament right away?  Interestingly, those analyzing this problem and making decisions for their respective organizations have often come up with different answers.

Those differing opinions, and the rationale behind them, were the subject of a talk held earlier this month in Philadelphia, at the Association of Moving Image Archivists / International Association of Sound and Audiovisual Archives (AMIA/IASA) 2010 Conference.  Representatives from the respective digital preservation projects underway at the Library of Congress, Rutgers University Libraries, and Stanford University were each on hand to offer their perspectives and the paths their organizations took for digitally preserving their video.

The abstract of the talk, as well each presenter’s slides and notes, can be found here:

• AMIA/IASA 2010 • Wrappers and Codecs: A Survey of Selection Strategies

I feel an important conclusion to take away from this talk is that there isn’t always a single right answer to the digital preservation conundrum.  There is a common desire among preservationists to have and use a widely accepted standard format for keeping our digital objects safe in the long term.  However, while formats and standards can be recommended and can work very well for a wide variety of use cases, there are always those local requirements and special needs that need to be considered, and adjustments made accordingly.

Fortunately, a great deal of progress has been made in the last several years, as those who were once wading into this problem alone have experimented and learned from past mistakes.  It’s venues like this which permit that knowledge and experience to be shared, so that those preservationists just starting to consider the problem can use that wisdom, and have multiple case studies to consider in making decisions of their own.

Some of our best digital preservation projects have been the direct result of collaboration; working with dozens of separate entities that all have valuable materials that they want to share with the online world.  That collaboration brings some challenges though, and one of biggest problems we’ve run into has been how people name files after they’ve created or digitized them.

For experienced computer users who store lots of valuable informartion digitally, it goes without saying that clearly naming files is extremely important.  Often, the filename is the first thing a user sees that identifies what’s in a file; the information it contains. Without any other cataloging system in place, file names become the way to figure out what’s inside the hundreds of thousands of individual files that can sit on the average persons’ desktop computer, and having countless “untitled” or ambiguously-named files can make finding the information you want nearly impossible.

Fortunately, modern computer operating systems give people a wide latitude in how they can name files.  Most people have a file naming method that works best for them, and for the most part, individual systems can work well, so long as they stay consistent and aren’t too hard for most people to easily comprehend.  However, things can get a bit more tricky when such files are destined for a digital library, online repository, or other type of internet-based storage and delivery medium.  When these types of architectures come into play, some of that wide latitude that modern computers give us in naming files can cause some complications.  Web-based content management systems aren’t always as flexible or forgiving with filenames, and can sometimes reject or even mangle files that are more liberally-named.

For this reason, it’s helpful to establish and follow a few simple ground rules when working on a digital preservation project that requires file handling.

In this article, I’ll list some of the basic recommendations that ensure file processing on a digital library project will go smoothly.  I’ll also point out some tips and tricks that have worked well for us in projects handled here at the Digital Curation Research Center.

Online File Naming: The Ground Rules

1. Keep It Simple
   Some of the clearest and best file names avoid complication, and keep things clean and neat.  This means sticking to standard alphanumeric characters as much as possible (Upper and Lowercase A-Z as well as numbers 0-9).  Filenames such as “EndOfYearReport-2010.doc” and “EdSmithBiography.pdf” can tell you quite a lot about what to expect in the file’s contents, without having to be overly complex or lengthy. 

    

2. Avoid Special Characters
   Although most desktops are fine with the following characters in file names, most web-based systems are not.  For this reason, you should avoid the following characters in your file names:
   ! # $ % & ‘ ( ) + , .
   ; = @ [ ] ^ ` { } ~
   
   Additionally, these characters will often cause problems even on some desktop filesystems, and should never be used when naming a file:
   ” * / : < > ? \ | 

    

3. Avoid Using Spaces
   Some of the earliest file systems for computers had a strict limitation that prevented the use of spaces.  Although current operating systems for desktop computers eliminate this restriction and make it very easy to use them, the use of a space in a file name can still be very tricky for online, web-based repositories.  Some may replace the spaces in a file name with special character codes (see illustration to the right and above) that can come up as gibberish when retrieved later.  For this reason, spaces should be avoided if you know that the file is going to be uploaded or retrieved from an online system. 

   Instead of using spaces, consider using hyphens (-) or underscores (_) as separators instead. The use of CamelCase in file names might also be a good way to help you make out individual words.

Tips and Tricks that have worked for us

Having dealt with creating, sorting, transferring and storing literally hundreds of thousands of digital files over the past several years, staff members at the DCRC have come up with a few tricks that help making the handling of multiple files very straightforward.  A few of them include:

• Handling Dates
  For date-specific items, it’s a common habit for curators and catalogers to put the date in the file name.  Doing this is fine, but unless you format your date conventions a certain way, a chronological sorting of such filenames is not guaranteed. 

  First, seriously consider whether putting the date in the file name is necessary at all.  All current computer systems automatically add a date stamp to files as they are created.  If the item in question is a born digital document (such a digital photo or MS Office Document),  then chances are, this date stamp will correlate pretty accurately to when the file was created or last modified.

  If it turns out the contents of a file correlate to a different date that’s worth recording, you’ll have the best luck using a numeric date format with the year first, followed by the month and date  (YYYY-MM-DD).  This will allow you to sort these files in a way that fits naturally with the numeric sorting that computers will perform on filenames.

Changing Trends: The legacy of the three-character file extension

It used to be that a common tenet of file naming best practices was to mind the number of characters you used in the file extension (the last few characters at the end of a file name, separated by a “.” ).  The extension is used by older operating systems to identify what type of content a file might have, and which software package was best suited to open it.  These extensions have become a quick way for humans to figure out what’s in a file as well: experienced computer users are familiar with the .doc extension referring to a document created in MS Word; a .jpg file is usually an image.

Very old operating systems were limited in how they handled these identifiers, and restricted these extensions to just three alphanumeric characters.  And so for the longest time, a common recommendation was to make sure all files had no more than three characters in the extension.

For the most part, this is still a good practice, but isn’t always possible.  In particular, new file formats, such as documents made with newer versions of Microsoft Office, have mandatory four-letter file extensions by default (.docx, .xlsx, .pptx, and so on).

Additionally, as computers have grown more sophisticated, they’ve relied less on file extensions to identify file types, and more on embedded metadata in the file itself to figure out how best to treat it.  As a result, it’s now possible to make files that carry no file extension at all.

For this reason, as more software packages start taking advantage of modern file systems, the three-character file extension will start to become less important in the near future.

A last resort: Batch renaming to remove bad filenames

Not everyone will consult a file naming document before they start organizing their content, and so you may find (as we have many times) that you end up stuck with hundreds of files created by someone else, with all kinds of special characters that your digital library platform is rejecting.  You can’t really blame the people creating these files; the computer certainly allowed the files to be named this way, and in an ideal world, naming systems would be universally easy.  Nonetheless, having to go back and manually rename each individual file is tedious, time-consuming work that no one is really eager to tackle.

Fortunately, there are software packages that tackle this very problem, and can make short work out of renaming multiple files to conform to a standard.  Using these tools, it’s possible to go through a directory of files, locate spaces and special characters in their filenames, and either remove the characters or replace them with something less exotic.

For Windows, the most commonly available (and free!) program that tackles this issue is the Bulk Rename Utility.  For Macs, there’s the free Name Changer application, along with some commercial tools such as NameMangler (Free Trial, then $10), and Renamer4Mac ($25).  And for linux desktops, there’s KRename, and the built-in command-line tool, rename (which also exists in command-line form on the Mac).

Over time, it’s hoped that file naming systems on different computer platforms will standardize and converge as they continue to evolve, and perhaps online web-based systems might become more lenient in how files are named.  Until that day comes, a lot of time and aggravation can be saved on the part of catalogers and digital curators by heeding a few rules about how files get their names, and keeping things simple and straightforward.

If you have any Kodachrome Film stashed away, the last days to get it developed are at hand.  According to Dwayne’s Photo, the last commercial developer of the film format, they will be developing their last received rolls of Kodachrome film on December 31, 2010.  After this date, their remaining equipment to handle this type of film will be shut down forever, and discarded.  Per the statement on their website:

The last day of processing for all types of Kodachrome film will be December 30th, 2010.  The last day Kodak will accept prepaid Kodachrome film in Europe is November 30th, 2010.  Film that is not in our lab by noon on December 30th will not be processed.

To be sure, this deadline has been known for a while now, since the announcement by Kodak that it had manufactured its last run of the film format.  Still, people have continued to use Kodachrome up to the present.

What is purported to be the last roll of Kodachrome film ever made by Kodak was shot and developed in July of this year, by Freelance photjournalist Steve McCurry:

There are definite advantages to digital photography by comparison to film, McCurry said.

“You have the ability to view, edit and monitor what we are doing as we go. We can evaluate the light and composition and the design instantly. And we can shoot in extremely low light, which was impossible with film.”

Regardless, digital photography is simply not the same.

“I like having something to hold in my hand,” McCurry said. “With digital photography, it’s just a hard drive. With Kodachrome, the film is real. You can touch it, put it in a drawer, and come back to it later. It’s tangible. It’s an object. With digital, the pictures only exist in a hard drive, in a memory chip.”

It’s that last statement which still brings hope to film’s future, as it’s the psychological connection with film among some photographers that keeps them coming back to it.  Although sales of digital cameras have been rising dramatically in recent years, including the increasing prevalence of consumer-grade digital still and video cameras in cell phones, Kodak claims that sales of analog film have been steady, including a potential mini-resurgence of film as a strong niche format.

Although Rutgers University Libraries has had digitization standards in place since 2006  for the RUcore object types we currently handle, the documents were often hidden deep in places where they weren’t easily found.  This made it hard for members of the public, and other people interested in finding a resource for how best to digitize to find out what we’re doing.

Additionally, the documentation was getting a bit long in the tooth; some of the proposals hadn’t been looked over in years, some still had “Draft” markings even though committees have reviewed them and we’ve already been carrying these procedures out, and in a couple of cases the documentation has been superseded by technology advances, and doesn’t match current practice at all.

For this reason, we’ve been engaged in a review of these standards and are revising where needed to make them reflect current best practices within RUcore and the Digital Curation Research Center. Additionally, I’ve created a “home” for the complete set of documents here:

• Digital Curation: Current Digitization Standards

The link to these standards are also available on the upper-left corner of this blog, in the navigation bar.

We hope that keeping these standards in one place will greatly benefit other curators and those who need a place to get started digitizing and preserving works.

Some major policy shifts came out of the Library of Congress today that fundamentally changes how the Digital Millennium Copyright Act (DMCA) is applied and enforced.  This decision making is part of a three-year cycle in which the Librarian of Congress and the Register of Copyrights hear from the public and review policies regarding enforcement of the DMCA.  According to the Librarian of Congress’ statement:

Section 1201(a)(1) of the copyright law requires that every three years I am to determine whether there are any classes of works that will be subject to exemptions from the statute’s prohibition against circumvention of technology that effectively controls access to a copyrighted work.  I make that determination at the conclusion of a rulemaking proceeding conducted by the Register of Copyrights, who makes a recommendation to me.  Based on that proceeding and the Register’s recommendation, I am to determine whether the prohibition on circumvention of technological measures that control access to copyrighted works is causing or is likely to cause adverse effects on the ability of users of any particular classes of copyrighted works to make noninfringing uses of those works.

For this cycle, some rather significant rulings were made that are decidedly consumer-friendly and archivist-friendly.  In particular, the Register touched on:

• Decryption of DVDs for fair use. Commercial and other video DVDs that are protected by the Content Scramble System (CSS) may now be lawfully decrypted, and the copy protection circumvented, for Fair Use purposes.  This includes extraction of short pieces for comment or criticism, educational uses in college and university settings, documentary filmmaking, and noncommercial videos.
• Video games and computer programs. Recognizing that preservation of old, obsolete software packages like applications and video games can require some circumvention of anti-pirating schemes, it now appears that the LoC is giving some leeway here.  It is now legal to crack DRM on legally-obtained  games and software “when circumvention is accomplished solely for the purpose of good faith testing for, investigating, or correcting security flaws or vulnerabilities.”  It is also legal now to bypass protection measures where security dongles are required if the security measures “prevent access due to malfunction or damage and which are obsolete.”
• eBooks. With some formats heavily protected by DRM measures, potential eBook buyers are often frustrated by an inability to transfer their legally-purchased content across platforms, and the blind are often thwarted in their attempts to use software that will allow this content to be read to them.  Today’s decision partially relieves this angst.  In cases where no other alternative exists, the LoC has deemed it legal to bypass Digital Rights management for eBooks for the purpose of enabling text-to-speech.
• Mobile Devices and Wireless Phones. This part of today’s decision deals specifically with a smartphone user’s right to load “unauthorized” or modified operating systems on their mobile devices, in particular, the practice of jailbreaking on Apple iPhones.  The LoC has ruled that this activity does fall under Fair Use.

This decision has been over a year in the making, and the next review cycle is less than two years away, at which point these decision may be revisited, or possibly even more DMCA exemptions will be laid out.

The official announcement and accompanying documentation can be found on the US Copyright Office Website here.

It looks like the Library of Congress sees the social impact and significance of the medium, and even believes there is a potential academic treasure trove waiting to be unearthed within this mass of single-sentence missives.  And so, the LoC has announced – via Twitter, of course – that it has acquired the entire Twitter archive.

That’s right. Every public tweet, ever, since Twitter’s inception in March 2006, will be archived digitally at the Library of Congress.

…

So if you think the Library of Congress is “just books,” think of this: The Library has been collecting materials from the web since it began harvesting congressional and presidential campaign websites in 2000. Today we hold more than 167 terabytes of web-based information, including legal blogs, websites of candidates for national office, and websites of Members of Congress.

Twitter also made its own announcement via its blog:

It is our pleasure to donate access to the entire archive of public Tweets to the Library of Congress for preservation and research. It’s very exciting that tweets are becoming part of history. It should be noted that there are some specifics regarding this arrangement. Only after a six-month delay can the Tweets will be used for internal library use, for non-commercial research, public display by the library itself, and preservation.

The specific details of the arrangement are still a bit sketchy, and I do have some questions about how this will play out.  For instance, there’s not much direct mention of whether this archive will include the numerous photos and videos that are frequently linked to users’ tweets, but are often hosted via third party add-on sites such as TwitPic and Posterous.  A lot of Twitter users tend to use the platform as a springboard towards linking to websites and other external content, the permancnce of which can be pretty dubious.

This is still a very promising start though, and hopefully the archived twittersphere will in fact prove useful to researchers in the future.

Some may question the importance or singificane of this decision.  But Twitter isn’t just mindless banter. The LoC lists a few socially significant tweets in the archive.  Among them, the first “Victory tweet” by a president-elect.  There’s also quite a bit of historical influence that was set in motion by Twitter: political prisoners in the Middle East have used it to get their message across to followers; sometimes it was the very medium that got them into trouble, and other times it spread the word that helped set them free. Politicians in the West from all ends of the political spectrum have and continue to use Twitter to marshall their troops, as it were.  And the media have documented cases where Twitter became the source of social change in countries ruled with an iron hand, so much so that the potential outage of the service due to maintenance was once considered a serious threat to activism.  There’s PLENTY of social significance there.

If there is one thing that every organization, institution, and individual curator learns as they delve into digitally preserving their collections, it’s that digital preservation isn’t cheap.  While there are very compelling reasons for digitizing, sometimes including it being cost-effective, there are still significant startup costs and an ongoing financial commitment required to sustain and keep your digital preservation projects viable.  Planning out the initial capital outlay and budgeting the ongoing maintenance costs requires a very different funding model from traditional, physical and analog collections.

In light of this, An NSF and Mellon Foundation-sponsored Blue Ribbon Task Force on Sustainable Digital Preservation and Access was convened in 2007, to explore the problem of economic sustainability of digital preservation platforms.  Their goal is to issue “specific recommendations that are economically viable and of use to a broad audience, from individuals to institutions and corporations to cultural heritage centers.”

Their final report has been issued. and is publicly available on their site.  I highly recommend reading through the report for any curator, business, library, or educational or heritage institution that is considering a long term preservation project and needs to get a grasp on the economic realities of such an endeavor:

• Sustainable Economics for a Digital Planet: Ensuring Long-Term Access to Digital Information
  Final Report of the  Blue Ribbon Task Force on Sustainable Digital Preservation and Access
  February 2010

They also have a complete listing of their publications, including preliminary and interim reports.  And, on April 1, a Symposium to celebrate the report’s release and open discussion is being held in Washington, DC.

This volume is devoted to the broad topic of distributed digital preservation, a still-emerging field of practice for the cultural memory arena. Replication and distribution hold out the promise of indefinite preservation of materials without degradation, but establishing effective organizational and technical processes to enable this form of digital preservation is daunting. Institutions need practical examples of how this task can be accomplished in manageable, low-cost ways.

Definitely something that I think every digital archivist and technophile should have in their virtual library.