Open Source “Digital Library” Systems: Streetprint & Greenstone An introduction and comparison Prepared by Martha Chantiny, Head, Desktop Network Services 114th Annual ASEE Conference Workshop 2 University of Hawaii at Manoa, Hamilton Library June 24, 2007 Systems to make digital collections available, searchable & harvestable •NOT a complex Content Management System •Intended for the one-person project/organization •Does not require technological or programming wizard level skills •Mostly platform agnostic 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 History http://wiki.greenstone.org/wiki/gsdoc/others/Greenstone_history.htm Greenstone – created 1997-98 Greenstone software grew out of the New Zealand Digital Library Project at the University of Waikato Developed and distributed in cooperation with UNESCO Licensed under the Gnu Public License (GPL) First CD-ROM the Humanity Development Library 1.3 issued in April 1998 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 History http://www.crcstudio.arts.ualberta.ca/streetprintorg/streetprintMLA.pdf Streetprint – created 2003 Grew out of Canada Research Chair Humanities Computing Studio project to create “computerized and integrated multimedia humanities research centre to unite humanities research and digital technology “ Funded by Canada Research Chairs Infrastructure Fund $54,860 in June 2001 Licensed under the Gnu Public License (GPL) First collection was digitized British street literature Streetprint: Revolution and Romanticism 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Pros & Cons FEATURES Greenstone CD runtime output Any metadata scheme Cross-collection searching Large user base, active listserv Multi-lingual Highly scaleable Streetprint CSS & templates provided PHP (easy to change functionality) Comment feature Easy to use 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Pros & Cons Greenstone Complex mix of specialized code – more difficult to understand - mix of GUI and coding Requires more SysAdmin level knowledge & skills DRAWBACKS Streetprint No batch load Code not bug free or finished Only Dublin Core No cross-collection search Small user base, inactive lists How scaleable? 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Installation - Streetprint 1. A server (Linux or Mac with OSX or a Windows server) 2. Web server software (e.g. Apache) 3. MySQL version 3.23 or higher 4. PHP version 4.1 or higher 5. Know how to download & expand tar.gz or zip files 6. Knowledge of (or access to someone who can) change permissions to allow user running apache to write to the Streetprint directory, create databases, & assign rights in MySQL, configure web server software for PHP 7. Desktop computer with any web browser 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Installation - Greenstone 1. A server (Linux or Mac with OSX or a Windows server) 2. Web server software (e.g. Apache) 3. PERL/CGI 4. GCC Gnu C++ compiler and GDBM Gnu database manager compiler (if the precompiled binaries don’t work for you) 5. Knowledge of how to download and correctly expand tar.gz or zip files 6. Ability to (or access to someone who can) change web server config & cgi-bin config files; add users 7. Desktop computer with Java Runtime Environment software, installed GLI java applet and web browser 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection –Streetprint Database creation Geek Alert! This chapter of the Streetprint Engine manual is by far the geekiest of the lot. If you need any help with this section, we recommend you consult your nearest internet wizard or technical support staff member. 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection –Streetprint Name the DB in MySQL 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection –Streetprint 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection –Streetprint User profiles 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection –Streetprint User profiles 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection - Greenstone 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection - Greenstone User Accounts Info only Editable Info only 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection - Greenstone User Accounts 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection –Greenstone Database creation The installation begins by asking which directory to install in. By default, all Greenstone software is installed in the directory /usr/local/gsdl if it is the root user who is doing the installation. Next you choose to install binaries or source code. You are then asked some questions about your webserver setup. A valid cgi executable directory (normally called “cgi-bin” on Unix systems) is required and you need to enter the web address of the cgi directory. A directory called “collect” will be created during the install process. 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection –Greenstone Database creation “Greenstone Librarian Interface” or GLI applet should be installed at this point --New Collection-- 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection – Greenstone Description, Contact, About 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection – Streetprint Description, Contact, About 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Setting up a collection – Streetprint Description, Contact, About 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Adding Data - Streetprint Important note: By default, most PHP servers limit uploaded file sizes to 2MB or smaller. If you plan to add media files which are larger than 2MB, you may need to contact your server administrator and have them change this setting first. 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Adding Data - Streetprint Document type This can be used to distinguish items in the collection, e.g. different report series, or different sizes 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Adding Data - Streetprint Defining types 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Adding Data - Streetprint Defining types Category = Subject Heading By default this creates a pulldown list in the Add/Edit mode 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Adding Data - Streetprint Metadata 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Adding Data – Greenstone Plug-ins (*.pm) 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Adding Data – Greenstone more about plug-ins From the documentation: Two types, metadata and document plugins to process files, some •group several files into one document, •some split one file into several documents, and •some have a one to one mapping. ImagePlug •uses the imagemagic utilities to •create derivatives(thumbnail images) and, •extract image metadata (width height format) ImagePlug can easily be extended to extract more extensive image metadata if required 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone customization Plug-ins 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Adding Data – Greenstone configuring plug-ins I am building a collection that is mainly JPEG files, and 1 PDF file of a pamphlet. Is it possible to type a description page in MS Word for each JPEG in the collection, and associate the word document with the JPEG so that when the JPEG image is retrieved, the user also has the option of clicking an icon and reading MS Word document description of the image. Do you want the text in the Word document to be indexed? What you could try is naming your documents like photo1.jpg, photo1.doc i.e. same filename for word doc and image doc. Then have WordPlug and ImagePlug in the collection. On one of the plugins use the associate-ext option - if you add it to WordPlug, set the value of the option to jpg. If you use the option on IMagePlug, set the value to doc. This will mean that one version is processed as a document, the other is associated with it. Which way round you do this affects the processing of the files. (whether you get the word document indexed or not, whether you get the image processed into thumbnails etc) You'll also need to play around with format statements. 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Diverging Paths At this point using the 2 systems involves very different workflows Greenstone = quasi-batch Streetprint = record by record 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Input - Greenstone 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Input – Greenstone drag & drop all file types 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Input – Streetprint default form for metadata entry 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Input – Streetprint error checking & publish 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Input – Streetprint adding images 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Input – Streetprint adding images A minimum of two files per record need to be pre-processed and uploaded 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Input – Streetprint adding full text Cut and paste (or type in!?) only plain ASCII 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Input – Streetprint adding media Unlike images – media files must be uploaded from the PC 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Adding Data – Greenstone build After adding files … build collection 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Adding Data – Greenstone adding & editing metadata If you haven’t carefully manipulated and coded your files, the extracted metadata isn’t always very useful 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Break? 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Searching - Greenstone setup 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Searching - Greenstone setup 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Searching - Greenstone user interface 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Searching - Greenstone Preferences What the end user sees: 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Searching - Greenstone Preferences Functionality and look of page controlled in prefs.dm macro file: # page content ############################################################## _pagetitle_ {_collectionname_: _textpreferences_} _imagethispage_ {_textpreferences_} _htmloptions_ {} _htmloptionson_ { <p><tr>_fcoption_</tr> <p><tr>_hloption_</tr> <p><tr>_extlinkoption_</tr> _If_(_PreferenceDocsFromWeb_,<p><tr>_intlinkoption_</tr>) } _fcoption_ { <td valign=baseline>Document page layout</td> <td><input type=radio name=fc value=1 onClick="updatefc(1);" _If_(_cgiargfc_, checked)>navigation bar at top<br> <input type=radio name=fc value=0 onClick="updatefc(0);" _If_(_cgiargfc_,, checked)>no navigation bar</td> } 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Searching - Streetprint setup 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Searching - Streetprint user interface Default is ultra-simple keyword search 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Searching - Streetprint user interface Advanced search with instructions or multiple pull down menus 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Streetprint - user interface Comments 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Customizing Which code looks like more fun? 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Customizing format VList <td valign=top>[link][icon][/link]</td> <td valign=top>[highlight]{Or}{[Title],Untitled}[/highlight] <i><small>{If}{[Date],<br>_textdate_[Date]}{If}{[NumPages], <br>_textnumpages_[NumPages]}{If}{[Source], <br>_texts\source_[Source]}</small></i></td> $tmpl->addVars( "viewtext", array ( "TEXT_NAV" => $textnav->getHtml(), "IMAGE1" => $image1, "MAIN_IMAGE_WIDTH" => $mainImgWidth, "TEXT_TITLE" => $text->title, "TEXT_TITLE_UC" => strtoupper($text->title), "RECORD" => $text->getRecord($gfx), "RIGHT_COL_WIDTH" => $rightColWidth,)); 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Customizing $tmpl->addVars ( "searchresults", array("THUMBS" => $thumbnails, "PAGINATION" => $pag, "VIEW_SWITCHER" => $switcher));if(!empty($legend)) {$tmpl->addVar("commentLegend", "COMMENT_LEGEND", $legend); } else { $tmpl->setAttribute("commentLegend", "visibility","hidden"); } <format><td valign=top>[link][icon][/link]</td><td valign=top>[srclink][srcicon][/srclink]</td> <td valign=top>[highlight]{Or}{[ dls.Title],Untitled}[/highlight]<i><small>{If}{[Date ],<br>{If}{[NumPages],<br>_textnumpages_[NumPages]} {If}{[Source]</small></i></td> </format> 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Streetprint customization Templates, Pages, Includes /streetprint/DB/Pages •about_contact.php •about.php •advancedsearch.php •basicsearch.php •browseby.php •browselist.php •browse.php •bytitle.php •search.php •searchresults.php •viewfulltext.php •viewimage.php •viewmedia.php •viewtext.php /streetprint/DBs/templates/Pages/ •about_tmpl.html •advancedsearch_tmpl.html /streetprint/DB/include/ •basicsearch_tmpl.html •about_navigation.inc •browseby_tmpl.html •allObjects.inc •browselist_tmpl.html •navigation.inc •browse_tmpl.html •sp_fieldnames.inc •index_tmpl.html •sp_footer.inc •searchresults_tmpl.html •sp_graphicfiles.inc •viewfulltext_tmpl.html •sp_header.inc •viewimage_tmpl.html •sp_initialization.inc •viewmedia_tmpl.html •sp_subdir_header.inc •viewtext_tmpl.html 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone customization Macros,Config files /gsdl/macros •about.dm •browse.dm •collect.dm •dateqry.dm •extra.dm •gli.dm •help.dm •home.dm •html.dm •nav_css.dm •pref.dm •query.dm •style.dm •yourhome.dm There are 103 macro files, most of them are for language definitions: arabic to urdu 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone customization Format statements & string items [link][/link] [srclink][/srclink] [icon] [srcicon] [num] [numleafdocs] [Text] [RelatedDocuments] [highlight][/highlight] [Summary] [DocOID] [DocRank] [DocImage] [collection:meta-name] VList HList DateList SearchVList DocumentVList CL1VList CL1HList CL1DateList 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone customization Format statements & string items 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone customization CGI cgi arguments used by library encode the parameters of each page, e.g., what kind of page to display, what query to carry out. a is the main cgi argument and determines which action is used to display the page. 1 a=p: Page action 2 a=q: Query action 3 a=d: Document action 4 a=extlink: External Link action Queries then have many options: t search type - if b=0, 0=all, 1=some. if b=1, 0=boolean, 1=ranked m max number of documents to return in search results o how many documents to display per page of search results r which result number to start from for the current page of results 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 OK, have we had enough tech? Time for a tour … 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Sites using Greenstone http://nzdl.sadl.uleth.ca/cgi-bin/library http://www.booksfromthepast.org/ 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Sites using Greenstone http://image2.nybg.org/cgi-bin/nybg.exe http://bridges.lib.lehigh.edu/cgi-bin/library?a=p&p=home 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Sites using Greenstone Hawaiian Language Newspapers http://nupepa.org/ 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Sites using Greenstone Hawaiian Language Newspapers part of The Hawaiian Electronic Library http://www.ulukau.org/ 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 http://www.infoapolice.sk/streetprint/ Sites that use Streetprint http://cushing.tamu.edu/collections/images/ http://www.opmagazine.com/streetprint/ 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Sites that use Streetprint University of Victoria Maltwood Art Museum and Gallery http://maltwood.uvic.ca/physics/ http://www.maltwood.uvic.ca/lorenzen 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Sites that use Streetprint http://www.spleencoffin.com/streetprin http://www.edmontonsmallpress.org/catalogue/ 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone at UH Manoa: not as far along as we thought we’d be http://128.171.57.25/gsdl/cgi-bin/library 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone at UH Manoa: HSPA 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone at UH Manoa: HSPA records - default 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone at UH Manoa: HSPA - labelled 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone at UH Manoa: HSPA search types Content type Sugar Company Keywords 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone at UH Manoa: Building a Rainbow 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone at UH Manoa: Trust Territory of the Pacific Archive Photos http://libweb.hawaii.edu/digicoll/ttp/ttpi.html 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Greenstone at UH Manoa: Trust Territory of the Pacific Archive Photos 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Streetprint at UH Manoa http://digicoll.manoa.hawaii.edu/ 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Streetprint at UH Manoa Social Movements http://digicoll.manoa.hawaii.edu/socmovements/ Converted from a home-made MySQL implementation 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Streetprint at UH Manoa Krauss Index http://digicoll.manoa.hawaii.edu/krauss/ Being input one by one from 3x5 index cards. Eventual goal is to link each card to an image of the article to which it refers. 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Streetprint at UH Manoa Save Our Surf http://digicoll.manoa.hawaii.edu/sos/ Scanned, OCR’d & completed over two semesters by one student assistant 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Streetprint at UH Manoa Demonstration site - TRAIL http://digicoll.manoa.hawaii.edu/techreports/ Created as a “proof of concept” to present to GWLA Directors with a proposal to fund a multi-year digitization effort 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Streetprint at UH Manoa Demo site - Hawaiian Music http://digicoll.manoa.hawaii.edu/music/ 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Streetprint at UH Manoa Steve Thomas Traditional Navigation http://digicoll.manoa.hawaii.edu/satawal/ Scanned & proofed by two students in just a little over a year. Using comment feature to obtain expanded description info 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Links •New Zealand Digital Library http://nzdl.sadl.uleth.ca/cgi-bin/library •Greenstone http://www.greenstone.org/ •Greenstone Wiki http://wiki.greenstone.org/wiki/index.php/GreenstoneWiki •Streetprint http://www.crcstudio.arts.ualberta.ca/streetprintorg/ (http://www.streetprint.org) •Streetprint Sourceforge download https://sourceforge.net/projects/streetprint/ 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007 Open Source “Digital Library” Systems: Streetprint & Greenstone Questions? Comments? Presentation available online: http://www2.hawaii.edu/~chantiny/ASEE.ppt 114th Annual ASEE Conference Workshop 2 – University of Hawaii at Manoa, Hamilton Library, June 24, 2007