Saving our data from digital decay

Digital objects -- documents, images, databases -- require specific software to open and read them, which in turn requires specific operating systems, device drivers, and hardware to run them depending on the format in which they're stored, whether magnetic, optical or some other system. The pace of change in the world of technology is so rapid that applications as well as media technology have only short life spans and archived data has to be migrated at frequent intervals on to new data carriers and into new file formats to maintain its integrity. For instance, data that once might have been held on magnetic tape or floppy disks is unreadable on today's equipment and CD-ROMs and other media will go the same route in the future.

If the 0s and 1s of digitised information can succumb to the vagaries of technological change, one thing is certain, analogue archives will always be readable to future generations provided they retain language skills. After all, we can read the works of modern authors almost as readily as the words of Shakespeare and Chaucer and ancient hieroglyphics bear close scrutiny once we cracked the code. With this point in mind, Steffen Schilke of the Gemeinsame IT-Stelle der hessischen Justiz in Frankfurt am Main in Germany and Andreas Rauber of the Department of Software Technology and Interactive Systems, Vienna University of Technology, in Austria discuss how e-government archives might be safely stored using an alternative to digital media -- the microfilm format beloved of spy fiction.

Writing in the International Journal of Electronic Governance, the team explains how e-government applications have to archive data or documents for long retention periods of 100 years or more for legal reasons and also such materials are often worthy of storage for future historians. They and many others have recognised the problems of storing such materials in digital media and suggest that in terms of cost, stability and technology independence, microfilm offers a promising solution for "off-line" storage.

The team has carried out a feasibility study that analysed encoding techniques to allow digital data to be saved on to microfilm and then to test data recovery as well as cost issues. Aside from precluding the need for frequent technology updates, storage of documents and data on microfilm will give future generations access to the information by scanning the microfilm into whatever system they are currently using and applying optical character recognition to re-digitize und subsequently decode the data.

The team further suggests that in order to reduce the amount of microfilm used for any given repository and so cut conversion and re-digitization times it would be possible to convert a stream of text into a bar-code type system that would still be entirely analogue but would rely on knowledge of the conversion key to return the data to digital form from microfilm. Using such a system could render a tested 170 kilobyte file that requires 191 pages of microfilm space as just 12 or so "printed as a two-dimensional barcode. Such a barcode would incorporate redundancy and be self-checking unlike a straight digital to analogue image scan of the text. Further compression is possible, if colour microfilm and barcodes were used for storage. This may provide a valuable, low-maintenance additional back-up for the original digital objects in addition to preservation activities needed for the on-line access copies.

For microfilm data storage, no tapes, no spinning discs, no electronic devices are needed and the only precautions necessary for the storage of the microfilm is a dry, temperature-controlled and locked closet or even just a hermetically sealed box. "The advantage of such a method is clearly the 'don't care' factor for the media migration," the team asserts, "With a lifetime of more than 100 years a lot of media migration projects (which are usually necessary every 3 to 7 years) could be avoided, saving money, effort, and precluding the risk of data loss as technology diverges away from today's standards."