Workshop paper for Very Large Digital Libraries Workshop, ECDL, Aarhus, 2008.

On September 19th 2008 Digital Library Consulting will be presenting a paper at the First Workshop on Very Large Digital Libraries in Aarhus, Denmark (in conjunction with the 12th European Conference on Research and Advanced Technologies on Digital Libraries).

http://www.delos.info/vldl2008

Greenstone has developed—rather unfairly, we feel—a reputation as a ‘toy’ document system not capable of handling large-scale, enterprise level collections.  While our latest ‘million page’ newspaper collections should help change this preconception, there are indeed some scalability issues encountered in large collections.  Similar problems have been encountered in large-scale databases and have been answered by the use of distributed computing, where the processing and storage workload is shared and balanced between several computers instead of just one.  However, Greenstone didn’t provide this functionality… until now.

While still in its early development stage, Greenstone has been integrated with the recently released IBM DB2 Express-C database.  This database meets Greenstone’s requirements for metadata storage and—using the Net Search Extensions add-on—full text search, while its license allows users to download and install for free.  Most importantly, Greenstone is then able to leverage the power of ‘Federation’, DB2’s implementation of distributed computing.  The ‘front-end’ DB2 server transparently manages interaction with an arbitrary number of ‘back-end’ DB2 data servers.  This provides the potential to dramatically increase Greenstone’s large-scale performance just by adding further ‘back-end’ servers without having to drastically change Greenstone itself.

Here at DL Consulting we’re continuing to make improvements to Greenstone’s support for importing and displaying METS/ALTO data. METS/ALTO is an XML schema published by the Library of Congress, and being used by the US National Digital Newspaper Project (NDNP), as well as many other newspaper digitization projects (as well as some collections of books, journals, and other textual resources). In addition to extracting machine-readable text from the page a process resulting in METS/ALTO also records information about individual articles within a page. This allows a user interface to be built where newspaper articles can be displayed on their own, as well as within the pages on which they were printed.The Papers Past site we built last year with the National Library of New Zealand (and which uses METS/ALTO) continues to grow. There are now over 600,000 searchable pages (that’s about 6.5 million newspaper articles!) in the system. We’re happy with how well the system is scaling, but continue to work on further improvements, with the eventual goal being infinite scalability with large collections distributed across multiple computers. We’re making good progress towards that goal thanks to a research grant from the Foundation for Research, Science, and Technology.In addition to the Papers Past collection we’ve built two further METS/ALTO based newspaper collections over recent months. Neither of these sites are accessible to the public yet unfortunately, but we’ll post links on www.dlconsulting.com once they are.

• Cornell University – The Cornell Daily Sun Digitization project. This project has been using a basic Greenstone system for some time (and which is still online now) but we implemented a major upgrade so the system can now import METS/ALTO data (which Cornell have switched to for the digitization of all remaining newspaper issues) as well as the older (proprietary) data format that was used in the earlier digitization work. METS/ALTO is more flexible than the older format but the system was implemented so that all the data (both old and new formats) are displayed very similarly. The Cornell Daily Sun project also switched to generating web-accessible images on demand with image server software, similar to the way Papers Past does.
• National Library Board of Singapore. We’ve also been working for many months on a large newspaper collection for the National Library of Singapore, building upon the software written for Papers Past. The Singapore collection will be released later this year, initially with around 600,000 pages of digitized content. That will grow to around 2 million pages over time. The Singapore project has some added complexity, including integration with a digital rights management system (because some of the digitized newspapers are still in copyright) and integration with automated concept (i.e. subject heading) extraction software. In addition, the Singapore project uses large grayscale JPEG2000 source images, as opposed to the black-and-white TIFF images used by Papers Past. We had to redevelop our image server software quite significantly to get good performance when processing these JPEG2000 images.

We’ve been asked several times if the code written to import and display METS/ALTO data is open source, and if it has been committed back to Greenstone. The answer is yes, of course it’s open source, but no it hasn’t yet been committed back to the Greenstone code base. The reasons for not committing it back are as follows.

• It’s a lot of highly specialized code, and is only useful to those with METS/ALTO data. My personal belief is that at times we have too much highly specialized functionality added into Greenstone, and that Greenstone2 isn’t currently modular enough to make it easy to add these sorts of major changes.
• We’ve worked with a number of METS/ALTO based projects and the data itself is always subtly different. That is, the code always needs to be modified to suit the METS/ALTO schema used, so is only useful as a starting point.

Having said all of the above, we are of course happy to make the code available to those who want it. Please contact us at contact@dlconsulting.com if you’re planning on building a METS/ALTO based Greenstone collection.

DLConsulting recently received a grant from the Foundation for Research Science & Technology to further improve Greenstone’s performance when scaled up to very large collections. Of particular interest are scalability issues caused by large collections of uncorrected OCRed text (e.g. digitized newspaper collections). As part of this research we are testing and benchmarking the performance of a number of different search engine and metadata database options, as well as improving Greenstone’s ability to distribute a collection across multiple servers. The research grant runs until April 2008.

I finally remembered to add my paper on The Adoption of Open Source technologies in New Zealand to the main DLC site.  This was the final report for the first module of the Post Graduate Certificate in Professional Development (Electronics and ICT) that I’m currently undertaking.

While the report could probably do with some more polish, it does provide an interesting background to open source issues (such as intellectual property and copyright/left issues) and provides statistics and references which could be used as evidence in future contracts. Earned me an A too!