HathiTrust

“Hathi” is the Hindi word for elephant, and this project uses the elephant’s association with wisdom and memory in its name, HathiTrust.  HathiTrust is a shared digital repository whose content is composed largely of digitized books from the Google Books Library Project.  The idea to create the shared digital repository comes from Committee on Institutional Cooperation and the University of California system.  The associated universities are all partners in HathiTrust, and partnership is open to other research libraries.

As far as preserving the content within the repository, HathiTrust touts that it “provides a no-worry, pain-free solution to archiving vast amounts of digital content. You can rely on the expertise of other librarians and information technologists who understand your needs and who will address the issues of servers, storage, migration, and long-term preservation.”

HathiTrust envisions itself as becoming a very credible and large digital library as well as being able to provide a viable preservation service for its partners.

The Content and Google

In the HathiTrust shared repository, outside users will mostly find content that is in the public domain.  Due to copyright and licensing regulations, most the content that is currently in the repository cannot be viewed by non-owning parties.  In fact, only 16% of the total volume stored in HathiTrust are in the public domain as of August 8, 2009.  The FAQ also state that, “as it becomes possible to expand access to the materials through permissions or other agreements, other materials will be made available. HathiTrust has already been contacted by some rights holders wishing to provide broader access to their content.”
HathiTrust has made all of the public domain content available via search within the repository and through Google.  You can also browse a very cool visualization of the public domain content by the LC classification.  The search functionality of the full catalog is still under development in partnership with OCLC, but they offer this at the current time.

Most of the public domain content of this shared digital repository comes from books that have been digitized through the Google Books Library Project. Google partnered with institutional libraries to digitize books that are hard to find and may otherwise be unavailable.  HathiTrust stemmed from some of the libraries in this partnership banding together to create a way to preserve and share their copies of the digitized books…independently of Google.

Having two sources of the same digitized content might sound a little redundant.  The Google Books Library Project corresponds directly with Google’s mission to make the world’s information accessible.  However, when we look a little more closely at the Library Project’s goal, it seems that Google is really trying to create some sort of visual card catalog, and is less concerned with the actual content.  Google also does not have the historical role that research libraries do in creating consistent access to materials.  Additionally, HathiTrust is focused on the long-term retention and storage of the digital content.  We can’t be sure of what Google’s plan is.  So perhaps it’s good that there is another project attempting to serve researchers and the public good that has corporate or commercial ties.

HathiTrust aims to include digitized content from printed materials (including journals) that extend beyond the scope of the Google digitization project.  They also hope to include born-digital materials and, eventually, items from institutional repositories.

Accessibility and Preservation Benefits

In my mind, one benefit of HathiTrust’s efforts that concerns a large number of potential researchers is that fact that it will provide an opportunity to remotely access items from libraries’ collections that would have otherwise required travel…or would have made side by side comparison an impossibility given the fragile and protected nature of many rare library items.
But why limit the benefit of accessibility to scholars?  One thing about digitization is that it can create a user audience where there once was no audience…so it is impossible to say how the general public can utilize newly available resources.

As for libraries, becoming a partner in the program will enable a library to store vast amounts of digital content, be it a special collection that has been digitized, endangered books (rare, brittle, etc.), purchased ebooks, and other digital items.  This will be a huge benefit to libraries who cannot start or handle a digital collection and storage project on their own.  The partner library will provide all the bibliographic data records for items to be ingested, and HathiTrust takes on all the hardware, staffing, and migration concerns.

Even non-partner libraries and their patrons will find an enormous benefit to utilizing HathiTrust: “HathiTrust is making bibliographic records for the public domain HathiTrust materials available so that institutions around the world can load them into their online catalogs, alerting users to the availability of these digitized volumes.”

Funding

HathiTrust is a non-profit, with no intention of becoming for-profit.  The original partners from the CIC and the University of California have apparently provided a large sum of the project’s existing funding.  Further partnerships from joining libraries will continue to provide funding.  The library partners will be charged once upon joining depending on the number of volumes they are adding to the repository using a per GB calculation system.

Advertisements

OAIS Reference Model Part II: The Model

Welcome to Part II of my OAIS Reference Model crash course!  By now you probably have noticed that I have refrained from including in this post any of the many graphed images that are in the OAIS reference model document.  This is because before I had a basic understanding of the model, these images seemed supremely complicated and confusing…kind of like Power Point slides with too many words.  I hope that what I provide here is a substantial enough understanding of the OAIS model to make the images less frightening when you do eventually encounter them.

Model Roles:

To start, it is important to recognize the three types of people that will be affiliated with a repository within the OAIS framework: the Producers of the repository’s content, the Managers of the content and repository, and the Consumers who use the content stored in the repository.  Each phase of the preservation process effects these three roles.  The ingest, the processing and storage, and the accessing of digital objects

The Model in Brief:

The document for the OAIS reference model has several key areas of content:

  • Terminology: An awesome vocabulary and glossary for the operations and information structures of repositories is located in Section 1.
  • Mandatory responsibilities: A list of the things that a repository must do in order be considered an OAIS-type repository comprises Section 3.  One particular action that this section calls for is identifying a designated producer/consumer community and ensuring that the information within the repository (metadata, etc), should be independently understandable (and accessible) by this community.  This means that “the community should be able to understand the information without needing the assistance of the experts who produced the information.”  Read this for more detail about the other mandatory responsibilities.
  • A model for ingesting, storing, and providing access to stored items, including a very smart model for capturing each item’s metadata (Content Information) and preservation metadata (Preservation Description Information).  Together, this data is discussed as an item’s “packaging information.”  It is intended to include information about an item’s context in order to fulfill one of an OAIS-type repository’s mandatory responsibilities.  This is all discussed in Section 2.
  • An outline for administrative management of the repository and the OAIS functions is presented in Section 4.  This discusses working with the creators of the digital objects and the objectives behind the day-to-day mangement of the repository. The administrative role also oversees the general planning and governance of the repositories, and include policy and preservation decisions.
  • Actual preservation methods: Preservation processes such as digital migration and emulation are examined in Section 5.  Preservation Planning is obviously a central part of any repository’s role.
  • Archive and repository interoperability: concepts behind repository interoperability and federation are discussed and explained in Section 6.  Heavy cooperation between repositories to develop common local standards in order to make this a possibility.

By following the OAIS model and the mandatory responsibilities which it entails, a repository will gain recognition as an OAIS-type archive or repository.  It is beneficial for a repository to be recognized as such because it means that the well-documented archival standards of the OAIS model will have been applied to help ensure the effective long-term storage, retrieval, and preservation of digital documents.  Another benefit is that communication with similarly-purposed OAIS repositories will be easy and fluid.

OAIS in Action:

DSpace and Fedora are two repository software platforms that have included OAIS-compliance capabilities in their product.  This helps pave the road for any repository that is built using either of these open source systems to follow procedures from the OAIS model.

What I would love to find or collect is a list of actual digital archives and repositories that are following the OAIS model either by the book or in some variation.  If anyone has a suggestion, please post a comment!

OAIS Reference Model Part I: Background and Influence

The OAIS model is an international standard that has been adopted for guiding the long term preservation of digital data and documents.  In fact, the OAIS model is an ISO standard (ISO 14721:2003): it was developed by the Consultative Committee for Space Data Systems (CCSDS) in 2002, and was adopted as an ISO standard in 2003.  The document is freely available, despite the fact that most ISO documentation is usually sold as a service.  It’s a hefty 148-pages, available in PDF form here.

oais
Photo by OliBac licensed under Creative Commons

The OAIS model is a standardized model describing a way that digital repositories intended for preservation purposes can be run.  Within this model, you will not find a standard for metadata.  It also does not endorse any particular repository platform, software, protocols or implementation procedure.  The OAIS model is simply a set of standardized guidelines intended to aid the people and systems behind a repository that has been designated with the responsibility of maintaining documents for archival purposes over a long period of time.

OAIS stands for Open Archival Information System, the word open referring to the open and public process under which this model was developed.  Participation in its initial development was encouraged by the CCSDS, and as an ISO standard, it will go under review every five years.

Because the OAIS model is a recognized standard, its users have formed a default sub-community within the digital preservation community.  But it has also been very beneficial to the digital preservation community at large and has helped promote progressive thinking and discussion.  Here are some key reasons why the OAIS model is so helpful to the digital preservation process and community:

  • It has standardized the terminology associated with digital preservation
  • It has outlined the duties and services of a preservation repository
  • It has outlined a way that information should be attributed and managed within a repository
  • It has mobilized community discussions about repository standards and certification
  • It has included preservation metadata as an important part of the preservation process
  • It focuses on long-term preservation, but lets “long-term” be defined by the repository managers
  • OAIS-type archives are committed to a set of defined responsibilities

As a final note, is important to make it clear that the OAIS model is by no means a requirement for a digital repository; while it is a recognized way of running a repository, it is not the only way.  It may not fit for some repositories, depending on their intended size, resources, and designated communities.  But admittedly, when a repository chooses not to follow the OAIS recommendations, it cannot fall under the umbrella of the most widely-used and understood digital archive standard.

————

Here are some resources that were incredibly useful for me while writing this post and the one to follow:

  • I really benefited from reading this post by John Mark Ockerbloom, the editor of the blog Everybody’s Libraries.  I almost considered forgoing my own entry and just directing readers directly to his!
  • And then I found this post and was blown away by how thorough it is.  It’s really well done and I’d encourage you to check it out.
  • This page is a brief run-down of OAIS from the JISC Standards Catalogue.

Continue on to Part II

Cloud Computing

Let’s talk about cloud computing.

At its simplest, things that are in the “cloud” are things that float around in a sort of digital airspace and don’t exist on your computer. They exist on remote servers which can be accessed from many computers.

cloud1
Photo by mansikka under a Creative Commons license

For this reason, the cloud is a good metaphor for the Internet. For most of us, keeping things in a cloud results in a convenient and logical way to make life simple.

You can access things in the cloud from anywhere that is connected to the Internet…depending on the service and its security (private cloud or public cloud). It’s kind of like your email or Facebook account. You have lots of stuff stored in these accounts that is specific to you, but you can log in from anywhere. And it will always look the same and have all your stuff in it. Your stuff is always just…there.

cloud
Photo by AJC1 under a Creative Commons license

Getting a bit more technical, your stuff is actually physically stored somewhere as bits on servers that are run by whoever is providing the service. For example, some institutions have servers dedicated to an institutionally-based digital repository. These servers might live on the campus and will store everything that is added to the repository. But the whole repository will not exist on the specific machine that you might use to access documents stored there. Your computer will connect to the remote server to access the repository.

What makes this fun for digital preservationists is that cloud computing can really increase the scale and sharing of preservation duties. Maureen Pennock, the Web archive preservation Project Manager at the British Library, recognizes this in her blog: “This minimises costs for all concerned, addresses the skills shortage, and produces a more efficient, sustainable and reliable preservation infrastructure.”

In the future – and as we are seeing with DuraCloud – all the tech work behind producing ways to store and retrieve data may be provided as part of a single repository product. (This type of service, by the way, is referred to as IaaS – Infrastructure as Service.) This would be excellent news for a great deal of institutions that don’t have the means or skills to set up a repository themselves.

Cloud computing offers a huge potential for an off-site alternative to the out-of-the-box repository products that most institutions currently must use. Instead, external organizations will be able to do the tech work while the institutions will be able to focus on non-technical repository maintenance.

Images by Flickr users mansikka and AJC1

Original publication date: 7/15/09

DuraSpace and DuraCloud

Here’s some news: the projects behind the digital repository platforms DSpace and Fedora have joined efforts and will now live under a bigger umbrella called DuraSpace. This new organization will still enable full and independent functioning of Fedora and DSpace, but there will be new joint projects aimed at advancing digital preservation technology and addressing a larger group of stakeholders.duraspace

The May 9, 2009 press release that announced the DuraSpace partnership emphasizes the first new project of its portfolio: DuraCloud.  DuraCloud is currently in a year-long pilot phase, and has the advantage of being backed as an NDIIPP project.  What makes it special is that it seems to be the first repository project to use cloud technology to store data.  Institutions will be relieved of a huge economic and technological burden if they no longer have to store the data themselves.  The Library of Congress announcement states that, “Duracloud will let an institution provide data storage and access without having to maintain its own dedicated technical infrastructure.”  Which means the servers (and knowledgeable techies) are provided with the DuraCloud product.

This means that the duties of the institutions with repositories that are supported by cloud storage technology will be refined to making the repository data standardized and accessible, which is probably a better way to spend time and funding for them.  DuraSpace and DuraCloud will maintain the open source and non-profit legacies of DSpace and Fedora, which makes this new organization and its first project even more appealing to institutions on tightened budgets.

Original publication date: 7/14/09