Digital Treasures of
the Treasure State
An Evaluation of Montana’s Statewide Digital Library Projects
Rachel Rawn
Rutgers University | 17:610:553
DIGITAL TREASURES
I.
2
Introduction
As our digital world expands and more users are introduced, it is extremely important that we
keep reviewing and evaluating the digital services we provide. This paper will look at two statewide
digital libraries in Montana, evaluate them as they are, and provide suggestions for improvement where
needed.
The Montana State Library has several digital library initiatives developed for consumption by a
statewide library consortium. This includes the MontanaLibrary2Go collection of ebooks and digital
audiobooks, and the Montana Memory Project collection of local historical artifacts, such as oral
histories and photographs. MontanaLibrary2Go is a digital library of titles purchased from Overdrive.
Libraries may opt in to this service, and then pay an annual fee according to a cost share formula. All
library users who have a card at a participating Montana library have access to this digital library. The
Montana Memory Project is a user-driven collection of Montana historical objects. Montana libraries,
museums, genealogical societies, etc., submit digitized collections to the Montana State Library for
inclusion in the Montana Memory Project. This paper will evaluate MontanaLibrary2Go and the
Montana Memory Project for the benefit of the users and the consortia.
II.
Literature Review: Criteria for Evaluation
Usability
The following criteria for evaluation fall under the umbrella term of usability. Usability is a very
general term that covers a wide range of evaluation criteria. Usability is widely covered in the literature.
“ISO defines usability ‘as the extent to which a product can be used by specific users to achieve specified
goals with effectiveness, efficiency and satisfaction in a specified context of use’” (Saracevic, 2004).
DIGITAL TREASURES
A.
3
User Interface
In a 2008 study on user evaluation of digital libraries, Xie found that fifty-four percent of
participants emphasized the importance of the interface. One subject summed up the importance of the
interface by saying “if the users can’t access the DL (digital library) effectively or finds it cumbersome,
the information is not really accessible to them” (Xie, 2008). Jeng suggests that “interface is one of the
most important aspects of usability as it is the medium that users communicate and interact with the
system” (2005). Hariri and Norouzi point out that “when a system is hard to use, the users may fail to
find important results, may misinterpret what they do find, or may give up in disgust believing that the
system is unable to help them. A DL is only as good as the interface it provides to its users” (2010). The
evaluation of user interface encompasses issues such as technical and visual design, such as color, font,
images and icons, navigation, and information architecture. The user interface has a large impact on
other aspects of usability, such as learnability.
B.
Search & Browse Functions
Related to the general user interface, the ease and usability of the search function is extremely
important for digital libraries. Xie (2008) identifies searchability and browsability as essential criteria for
user evaluation of digital libraries. Both of these options must be available in an online environment, as
they would be in a physical environment. One subject in Xie’s study points that both searchability and
browsability “must exist, to satisfy both users with search skills and specific needs as well as those with
lesser skills or a less defined need. Having both is a small way to ease the digital divide between those
with search skills and those without. Additionally, the browse function allows the user to see what the
collection holds” (Xie, 2008). Elements for evaluating searchability and browsability include any sort of
quick search feature, advanced search options, limiters, and the results display. Availability and usability
of a search results manager is also an element that can be evaluated here.
DIGITAL TREASURES
C.
4
Learnability
Chen, Germain, and Rorissa (2011) identify learnability as one of the characteristics of usability,
consistently seen in the literature and corroborated in their study of library professionals. Learnability
refers to ease of use. “The system should be easy to learn, easy to use, and intuitive, so that its user can
rapidly start accomplishing work” (Chen, Germain, and Rorissa, 2011). Learnability is especially
important for novice users (Jeng, 2005). Learnability can be measured from the amount of time required
for users to perform tasks accurately.
D.
Content
The primary objective for most, if not all, digital libraries, it so give users access to content. Xie
points to the importance of the contents of the digital collection, calling usability and collection quality
“two sides of the coin” (2008). One subject in Xie’s study said “If the content of the collection is of no
use to the user, the user will have little or no reason to even browse the collection.” Scope, format
availability, accuracy, authority, and completeness of the collection are important elements for
evaluating content. Evaluating content is not always about the size of the collection, in many cases it is
about relevancy to the purpose of the collection.
III.
Evaluations
MontanaLibrary2Go
MontanaLibrary2Go is a consortium service that offers circulating downloadable digital e-books
and audiobooks to patrons at participating libraries. The service is provided by OverDrive, a digital
content distributer, and facilitated through the Montana State Library. Any public, academic, or special
library may join the MontanaLibrary2Go consortium. Membership costs are calculated using a cost share
formula based on patron count. There are currently 92 participating libraries. Montana Library 2 Go is
DIGITAL TREASURES
5
governed by two committees, an elected Executive Committee and a volunteer Selection Committee.
The Executive Committee is responsible for developing and reviewing collection development policy and
procedures, developing the annual budget and cost-share formula for approval by the membership, and
making other recommendations to the membership as needed. New content is added monthly, based
on patron requests and selections made on a rotating basis by members of the Selection Committee.
The Selection Committee follows the collection development policy and procedures as approved by the
Executive Committee and the membership (libraries.msl.mt.gov).
A.
User Interface
While not strikingly elegant or beautiful, MontanaLibrary2Go’s user interface is certainly
functional. A bar across the top allows users to sign in, or see a list of the participating libraries. Users
can refer to the list to see if their local library gives them access to the service. Below that bar is the logo
(a cowboy reading a book, which does seem like a visual that would appeal to many Montana library
users), menu button, account button, help button, and a search bar with an option to redirect to the
advanced search interface. The benefit provided by the menu button isn’t immediately apparent, but
clicking on it reveals a drop down element with quick links to some of the most frequently accessed
MontanaLibrary2Go collections. This includes the top ebooks, top audiobooks, recommendations,
recently added, and several links under each of the major collections: ebook fiction, ebook nonfiction,
audiobook fiction, audiobook nonfiction, and children and teen. The account button and help button
both redirect to their respective sections of the website. These elements remain at the top of the page
throughout the website, which lends an element of stability and constituency throughout the website.
Clicking on the reading cowboy returns the user to the front page of the website. Design elements
reflect the priorities of the user community: efficiency is prized.
DIGITAL TREASURES
6
From montanalibrary2go.org
The account page features a virtual bookshelf, where users can manage the items they have
checked out (out of a total possible five), see the number of titles they have on hold (out of a total
possible ten), and whether any of the books they have on their wish list are available. The wish list
function seems mildly superfluous, as only items already purchased through OverDrive can be added to
it, and the hold function actually puts the user on a list to check out the book. The wish list can be used
to keep a personal list of books, and it doesn’t have the title limit of ten that the hold function does.
From the account page users can also adjust several settings such as their default lending period (seven
or fourteen days), range of maturity level (juvenile, young adult, general adult, mature adult) displayed
when browsing, whether they want mature adult covers displayed (if “no” is selected the cover art will
be replaced with a basic cover), and high contrast and dyslexic font options for accessibility. The account
page also has a recommendation feature, based on the user’s current checkouts, a list of titles that the
DIGITAL TREASURES
7
user has rated, and a list of digital titles the user has recommended for purchase by the
MontanaLibrary2Go selection committee.
B.
Search & Browse Functions
As discussed previously, different users will have different preferences for finding content,
making browsability and searchability indispensable. MontanaLibrary2Go has browsing and searching
options on display with equal prominence: the search bar is at the top of every page, with a link to
advanced search immediately below it, and links to various collections allow the user to jump into
browsing. When a search term is entered limiters allow results to be narrowed further, including
displaying only titles currently available, format (Kindle, ePub, MP3), subject, publisher, rating,
compatible devices, interest level, and tools to measure reading level (ATOS level, Lexile measure, and
text difficult). There is also another search bar that can be used to search within the results. Several
practice searches were run to measure result accuracy. Results proved quite accurate, but further
testing could be done.
The advanced search option allows the user to search by title, author/creator, and/or ISBN.
Limiters available when performing an advanced search, and addition to the limiters listed above, allow
the user to narrow their search by year titles were added to MontanaLibrary2Go. There is also an option
called “additional titles to recommend” that, when selected, makes additional recommendations based
on the search terms. These are indicated with a symbol in the corner of the title cover. When limiters
are not selected for format, ebook titles in the results are indicated by a black book symbol in the corner
of the cover, and audiobooks are indicated by a back headphone symbol. Additionally, if the “available
now” limiter is not selected, titles that are not available because all licensed copies are checked out are
indicated by the appropriate book or headphone symbol being gray, rather than black.
DIGITAL TREASURES
8
Browsing on MontanaLibrary2Go can be accessed several different ways. On the front page
several collections are displayed, including “Top eBooks,” “Top Audiobooks,” “May We Recommend,”
and “Recently Returned.” The user can begin at one of these collections by clicking the title of the
collection, and then browse using the same limiters provided in the search option. Additionally, across
the top of every page on MontanaLibrary2Go run the two bars mentioned in the User Interface
evaluation. One option there is the “menu,” which when clicked displays links to a large selection of
MontanaLibrary2Go’s collections. One of these can be clicked on, and again, limiters can then be used to
browse. A third option for browsing is to click on advanced search and apply limiters without entering
anything in search fields, and then clicking the search button. Any of these methods can be used to
browse MontanaLibrary2Go, depending on the user’s preference.
From montanalibrary2go.org
DIGITAL TREASURES
C.
9
Learnability
Learnability is difficult to assess accurately without quantifiable data from a user group. In Jeng’s
2005 study data was collected to examine the effort needed for new users to learn how to use a website
and how long it takes new users to orient themselves on a new website. Learnability was examined by
looking at how long it took subjects using a new website to answer questions and how many they
answered correctly. They found that all of their subjects were able to begin searches almost
immediately, but this was a rather narrow pool of subjects who were all students at either Rutgers
University or Queens College. This study most likely reflected much more accurate results for academic
library websites than it would for public library websites, as the user group at academic libraries is
already familiar with the kind of website they were using. The website itself was new, but the concept
and layout was probably not new. If such a study were to be undertaken for MontanaLibrary2Go it’s
quite likely that public library users unfamiliar with the OverDrive model would display more varied
results in learning how to use such a website. For the purpose of this paper, evaluation of learnability
will be based on ease of use as demonstrated in the evaluation of the User Interface and Search and
Browse Functions. Further research could be conducted based on Jeng’s structure of evaluation using
quantifiable data.
The users who will be accessing MontanaLibrary2Go’s content are potentially an incredibly wide
range. Anybody who has a library card at one of the participating libraries is eligible for this service.
Many Montana libraries allow anyone to get a card at their library, regardless of residency in Montana.
However, it is reasonable to assume that the majority of users will be Montana residents of varying
technology skill sets, including some users who will be completely new to the technology used to access
MontanaLibrary2Go content. Ease of use is an extremely important issue for MontanaLibrary2Go. The
user interface is straightforward and simple to use, and there are several different ways to browse or
search. Additionally, all MontanaLibrary2Go users will be connected to a Montana library by nature of
DIGITAL TREASURES
10
the service. This means help is available from librarians for those who are not familiar with the
technology. However, it is possible that MontanaLibrary2Go as a digital library is too reliant on this
access that the users have to libraries and librarians, stunting the website’s improvements to
learnability. Further testing for quantifiable data is recommended.
D.
Content
MontanaLibrary2Go contains 13,160 ebooks and 11,366 audiobooks, including fiction,
nonfiction, and titles for children and young adults. The scope of material provided is important because
MontanaLibrary2Go has a wide variety of users, as discussed in the evaluation of Learnability.
Audiobook titles are available in the widely used MP3 format, ebooks are available in the ePub format
and for Kindle. Support is provided by OverDrive for a wide range of devices, from ereaders to MP3
players to phones. However, there is an issue with availability. With such a large user population and
limited licenses purchased for each title, many titles are unavailable at any given time. Some more
popular titles have hundreds of holds on them. Users who are accustomed to instant access may find
the limited availability and long waiting periods frustrating.
E.
Analysis
Overall, MontanaLibrary2Go’s site is simple, even pleasant, to use. The interface is intuitive,
many features are offered for searching and browsing content on the website, and a wide range of
content is available. Further research might be undertaken to evaluate learnability of the website;
caution should be taken that learning how to use the website is not dependent on the availability of
physical libraries and librarians, as this may be discouraging, or even a barrier to accessibility for some
users. The lack of availability of content at any given time may be frustrating for some users.
Unfortunately, there is no cheap way to fix this. The Executive Committee may consider a slight increase
DIGITAL TREASURES
11
to each library’s annual fee, or seeking additional funding through grants. Funds should be allocated to
purchasing additional licenses, especially for high interest titles.
Montana Memory Project
The Montana Memory Project is an online source for digital collections related to Montana’s
cultural heritage. In part, these collections document the Montana experience. Access is free and open
through the Internet. Many of these items are digitized copies of historical material, some items are
contemporary. All serve as a resource for education, business, pleasure, and lifelong learning. Forty-one
Montana libraries, museums, archives, and cultural institutions have contributed materials to this
collection. Possible contents include digital newspapers, maps, copies of photographs, rare books,
historic documents, diaries, oral histories, audio and video clips, paintings, illustrations and art. The
Montana State Library employs a Montana Memory Project Director (Jennifer Birnell), who works with
contributors and institutions who wish to contribute, including assistance in defining the collection,
submitting an application, finding funding, and digitizing the collection according to the MMP standards.
Some funding for digitization of collections can be provided by the Montana Memory Project, up to
$1500 per project. For 2015 the Montana State Library has allocated $6000 in LSTA funds for adding
special collections to the Montana Memory Project. The Montana Memory Project hosts 85 collections,
with two more collections currently in development. An Advisory Council was formed in 2013 to help
develop a strategic plan for the Montana Memory Project, and to help facilitate outreach and promotion
of the project (libraries.msl.mt.gov).
DIGITAL TREASURES
12
From montanamemory.org
A. User Interface
Interestingly, the front page of the Montana Memory Project website features a video called
“How to Use the Advanced Search on the MMP.” A bright blue hyperlink announces “Instructional
Videos,” and beneath that “Short instructional videos designed to help viewers use the Montana
Memory Project pages. Take a tour of the web pages and learn how to view and manipulate
photographs, books, documents and more. Learn how to search our pages and how to improve search
results by using the Advanced Search options.” MMP Director Jennifer Birnell says they used to rotate
the instructional video featured on the front page about once a month, but they have left it on the
Advanced Search video since April 2015 because that seems to be the one needed the most. This will be
discussed further in the evaluation of the website’s learnability.
Across the top of the page reads “Montana Memory Project, Where Montana’s Rich Past Meets
Its People.” The logo is an old fashioned pocket watch. Underneath is a bar with links to the MMP
DIGITAL TREASURES
13
collections (Artwork, Audio, Photographs, and Print Materials), a link that takes the user to a list of
contributors, a link to the instructional videos, a link to educational resources, and a link to a collection
map. The educational resources include PowerPoint presentations for classroom use that give lessons
on seventeen subject areas in their collection. They also provide ideas for classroom activities that tie in
to their collections, and an option to share lesson plans for teachers who are using MMP in the
classroom. The educational resources page also contains links to resources on the Montana Historical
Society website, Library of Congress and National Archives resources for teaching, and other relevant
digital libraries such as Chronicling America and the Digital Public Library of America. The map of
collections (pictured below) is a virtual map of Montana that with pins that indicate that there is a
collection with information about that area. Clicking on a pin brings up a note about the collection, and a
link to the objects in that collection.
From http://montanamemory.org/Map
DIGITAL TREASURES
14
The front page also features a search bar, with links next to it for Advanced Search and Browse.
Overall, the user interface is easy to use, and all features on the site are easily accessible from anywhere
on the site thanks to the bar of links at the top. The videos can help new users navigate the site, but
most users who are familiar with digital libraries should be able to start using the website without any
issues.
B. Search & Browse Functions
Clicking on the browse button on the front page brings up all the MMP collections. While this is
interesting to look at, it might be overwhelming at first glance. The user can use the bar at the top to
browse different collections by type. The artwork and audio links lead straight to the collections in those
respective formats, while the photograph and print collections are more numerous and can be
narrowed further. Photographs are narrowed by subject area while print materials are narrowed by
format (books, documents, maps, newspapers & periodicals, and yearbooks) and then further by subject
area.
Searching on Montana Memory Project is an interesting endeavor. The search results consist of
objects relevant to the search terms, while on the left the limiters allow the user to narrow search
results by collection, creator area covered, date, and subject. The user can sort the search results by
various criteria such as name and date. Search results themselves display a thumbnail image, title, date,
subject areas (tags), and a brief description of the object (image below). The user can also keep track of
results by saving them to favorites. These can be accessed through favorites option on the link bar at the
top of the website, and exported to PowerPoint or a zip archive. There is no way to create an account,
however, so items added to favorites will not persist past the current session.
DIGITAL TREASURES
15
From http://mtmemory.org/cdm/search/searchterm/havre/order/title
The advanced search interface is simple but efficient. It gives the user the option to perform
different kinds of searches based on Boolean logic, with a dropdown menu providing the options “the
exact phrase,” “all of the words,” “any of the words,” and “none of the words” in any given field (such as
author, title, etc.) or all fields. Multiple fields can be added, allowing the user to perform quite precise
searches. The user can also choose which collections to search in. For those with some practice in
performing searches with this sort of system, the interface will be easy to use. For others who are
unfamiliar with performing searches like this, advanced search may not be as intuitive. For those who
are new to this kind of search, the “How to Use the Advanced Search on the MMP” video is easy to find,
and gives clear, easy-to-follow instructions.
C. Learnability
The instructional videos for using the Montana Memory Project are a very interesting aspect to
evaluating the site’s learnability. Some might argue that the videos should not be necessary because the
DIGITAL TREASURES
16
website should be intuitive to use, while others would probably argue that there is no such things as
“intuitive” for the complete novice, making the videos quite useful for those newly initiated to this type
of research. With or without taking the videos into account, the interface of the Montana Memory
Project is fairly straightforward, if not perfectly so. For example, pictures do not fully load immediately,
and a small bar must be tugged down to display the full view. Search results can seem overwhelming at
first glance, but the information provided is quite useful at second glance. It is part of the Montana
Memory Project’s mission to be a growing project, meaning the interface and search options may need
to evolve as the collections become more numerous. Ease of use should be assessed consistently as the
Montana Memory Project grows and expands. Again, learnability is extremely difficult to measure
accurately without a formal study. To gather data on the learnability of the Montana Memory Project
website, a study designed after Jeng’s 2005 study discussed in the evaluation of MontanaLibrary2Go
could be invaluable.
D. Content
Due to the nature of the project, the content of the Montana Memory Project should be
growing continuously. As mentioned earlier, the Montana Memory Project has an advisory council to
help the Montana Memory Project director develop a strategic plan. The advisory council should make
writing a collection development policy a priority after, or in conjunction with, drafting the strategic
plan. The content of the Montana Memory Project should be very specific, and serves the purpose of
digitally preserving Montana’s historical and cultural heritage. For users who are interested in the
history of Montana, the Montana Memory Project is a wonderful, growing source. A strong collection
development policy will allow MMP content to remain a wonderful treasure for the Montana
community for decades to come. Outreach and promotion of this project is vital; knowledge institutions
should be aware that this program offers a solution for preserving our important historical and cultural
DIGITAL TREASURES
17
artifacts. Individuals should be aware that the Montana Memory Project is a resource they can use to
learn about Montana’s history.
E. Analysis
Overall, the Montana Memory Project features an easy-to-use interface with some kinks, such
as pictures not fully loading at the first try, and search results that can seem a little overwhelming. Some
minor adjustments to the interface and design may be beneficial. Further research should be done to
assess learnability with new users, to see whether any improvements can be made to make the website
more intuitive. The instructional videos and a discussion with the MMP director, Jennifer Birnell, suggest
that learnability has in fact been an issue. This should be addressed, especially as the Montana Memory
Project is expected to expand. The content of the Montana Memory Project is truly a treasure for
Montana residents, and all who are interested in Montana’s history and culture. The drafting of a
collection development policy should be a priority, as should continued outreach and promotion.
IV.
Conclusion
As we expand our digital world, the digital treasure of the Treasure State will become even more
valuable. Evaluation is an important part of the growing process of the digital library. As
MontanaLibrary2Go and the Montana Memory Project continue to grow, they should be evaluated
consistently for any possible improvements that can be made. The purpose of this paper was to evaluate
MontanaLibrary2Go and the Montana Memory Project, two statewide digital libraries developed by the
Montana State Library for consumption by consortia and users at consortia libraries. With continued
evaluation, improvement, and growth, these digital libraries will remain treasures for Montana and
Montana residents for the foreseeable future.
DIGITAL TREASURES
18
References
Chen, Y.-H., Germain, C.A., Rorissa, A. (2011). Defining usability: How library practice differs from
published research. portal: Libraries and the Academy, 11(2), 599–628.
Hariri, N. & Norouzi, Y. (2010). Determining evaluation criteria for digital libraries’ user interface: A
review. The Electronic Library, 29(5), 698-711. doi:10.1108/02640471111177116.
Jeng, J. (2005). What is usability in the context of the digital library and how can it be measured?
Information Technology & Libraries, 24(2), 47-56.
Saracevic, T. (2000). Digital library evaluation: Toward evolution of concepts. Library Trends, 49 (2), 350369.
Xie, H.I. (2008). Users’ evaluation of digital libraries (DLs): Their uses, their criteria, and their assessment.
Information Processing and Management, 44, 1346-1373. Doi:10.1016/j.ipm.10.003.