Master’s Project Overview
The public library faces an interesting challenge over the next few years: as it progresses to the digital age at a feverish pace it must meet an insatiable demand for data storage. It is estimated that the at-risk audio/video material in the library’s collections – items that exist only in NYPL and for which there is only one copy – will require 10 petabytes of disk space to store in digital format. The amount is staggering, but not unrealistic. This number can grow exponentially if other materials are considered. Given the recent emergence of “big data” storage solutions, such as that provided by Isilon Systems, the technology to grow such large systems for unstructured data is readily available. However, the cost is considerable and great challenges exist in the concepts of vendor lock-in, data validity, bit-rot, and ongoing migrations. This challenge exists not only for libraries and similar institutions vying to keep digital representations online for prosperities sake, it also bleeds into other industries such as health-care, financials, and media.
Implementing a commercially available storage system does not guarantee such availability. Having this data in one place for example exposes it to total loss in case of a disaster. Replicating the content, while necessary, can greatly increase capital expenditures. Tape backup of multi-petabyte environments are impractical, costly, and unreliable. Hierarchical storage systems offer varying levels of technology to yield cost reductions but are complex, include less reliable tape, and can be slow. Write-Once-Read-Many (WORM) technology is attractive but can significantly increase the storage requirement projects. The challenge is fundamental, how do companies looking to archive vast amounts of data and ensure its integrity over a long period of time do it in a simple and cost effective manner.
Oddly enough, the components to do this exist in the market, but they are disparate. Piecing a system together for long-term preservation is an oxymoron of sorts – software and hardware do not last forever, but the files may be expected to! I propose the creation of a technology company that will build the software stack to provide a low-cost, easy to manage, all-in-one archival storage system. The technical philosophy will be to provide a system leveraging open source software and commodity servers or cloud storage. The management layer will give data owners multiple preservation options through a simple web based GUI. The technology will be offered in multiple flavors, allowing firms to deploy it on their own selected hardware, on certified hardware, as a virtual machine, or entirely in the cloud. Firms can leverage a cloud offering as either their primary storage or to copy on-premise data to. Furthermore, a mix of different cloud storage providers can be leveraged to diversify the underlying technology and reduce risk.
The end result is that the concept of creating a large-scale data repositories/archives is no longer a combined development and architecture project or pieced together using a host of independent products. The packaged solution offers time and money saving opportunities, resource efficiency, risk mitigation, and a reduction in overall costs compared to the alternatives. In my current position at the public library, I have evaluated multiple storage offerings in the past few years. I will explore this subject in relation to the library’s goals and extend it to other industries. If such a product existed, would the library buy it, at what cost, and will such a product be attractive in other industries? I believe the answer the answer is yes!