MOAC II User Evaluation: Making Museum Content Useful, Part 2
Anne Gilliland-Swetland
UCLA Department of Information Studies, GSE&IS Building, Box 951520, Los Angeles, CA 900951520, swetland@ucla.edu
Carina MacLeod
UCLA Department of Information Studies, GSE&IS Building, Box 951520, Los Angeles, CA 900951520, cmacleod@appleisp.net
Kathleen Svetlik
UCLA Department of Information Studies, GSE&IS Building, Box 951520, Los Angeles, CA 900951520, mkomara@earthlink.net
Layna White
San Francisco Museum of Modern Art, 151 Third Street, San Francisco, CA 94103,
lwhite@sfmoma.org
The Museums and the Online Archive of California
II User Evaluation (MOAC II) is a partnership
between museums, a library, a digital library, and
researchers to study the needs and behaviors of
four user constituencies in their use of select
digitized collections available on the Web. Until
recently museum information professionals have
been less accustomed than their counterparts in
libraries and archives to assessing and
documenting the usefulness of their online
resources. One expectation is that the MOAC II
project will make recommendations for best
practices regarding how museums might evaluate
the use, usage, and usability of Web-accessible
content over time. This interim report focuses on
lessons learned in the project’s second year.
MOAC II builds on the Museums and the Online
Archive of California (MOAC),1 a concurrent project that
tests the appropriateness of the Encoded Archival
Description (EAD) as a data structure standard for museum
objects and collections.2 MOAC partners produce EAD
finding aids for contribution to the Online Archive of
California (OAC), a Web-based collaborative cultural
information resource managed by the California Digital
Library. In MOAC II we are attempting to discover if
current efforts to provide and deliver resources – such as
EAD finding aids, as well as digital objects delivered
independently of finding aids – via the Web are useful and
meaningful for education and research. Questions guiding
the MOAC II study address issues of content provision,
access, and use:

The MOAC II Study
Museums and the Online Archive of California II User
Evaluation (MOAC II) is a two-year collaborative project,
supported by the Institute of Museum and Library Services,
with the objective of studying the needs and behaviors of
four user constituencies in their use of select digitized
collections available on the Web. MOAC II is a
partnership between several California museums, a library,
a digital library, and researchers. We are examining and
comparing the potential crossover or complementary user
contexts of K-12 teachers, university students, academics in
the humanities and social sciences, museum professionals,
librarians, and archivists. One outcome of the evaluation
will be recommended strategies for improving the usability
and value of digitized collections as well as for
encouraging their use in research and learning contexts.



Why and how do teachers, academics, students,
and information professionals use digitized
museum objects?
What do these users need to understand about
museum objects as sources of information and
evidence?
Does the use of a schema like EAD benefit use by
one group over another?
How might digitized museum objects be used to
enhance classroom teaching at K-12,
undergraduate, and graduate levels?
Because there are scope and temporal dynamics to be
addressed, as well as complex potential interactions
between some of the variables under examination, our
formal evaluation of MOAC is multifaceted. We are
gathering and triangulating both quantitative and qualitative
data from several different sources. Our goal is to obtain a
better overall idea of the complexity of issues associated
with providing online access to museum objects in general,
as well as to be able to tease apart aspects that relate to the
design of MOAC in particular. Our data sources comprise
transaction logs, pre-existing use data, feedback forms,
high-level questionnaires, and in-depth interviews with
participants. The project’s first year and research
methodology are described elsewhere,3 including “part 1”
of our report to ASIST as a poster session in 2004.
Subsequent papers will report findings from the MOAC II
study and the resulting recommendations. This interim
report focuses on lessons learned in the project’s second
year, with particular emphasis on our work with museum
information professionals.
Interim Report
Museums are increasingly accustomed to putting
digitized content to use for short and long-term purposes,
such as sharing and exchanging in galleries, websites, and
in collaboration with libraries and archives. Museum
information professionals have been less accustomed than
their counterparts in libraries and archives to assessing and
documenting the usefulness of their online resources, in
particular. Our experience in MOAC II suggests there are
differences between museums and libraries in their
readiness to conduct formal evaluation. For example,
MOAC II museum partners did not have, at the ready,
statistical and qualitative data about the use of their online
resources. Such data could help museums learn what, if
any, impact the availability of the online resources in which
they are investing (like MOAC) is having upon collections
and services. One expectation is that the MOAC II project
will make recommendations for best practices regarding
how museums might evaluate the use, usage, and usability
of online information systems like MOAC over time.
While museum partners had little pre-existing use data
available for the research team, partners offered anecdotal
information about what people might want to know about
museum objects and how people might want to use objects
in their work or learning. This anecdotal information is
based on partner interactions with school groups,
researchers, and other patrons during visits to museum
galleries or other display areas; that is, interactions with
patrons when the museum object is present and a museum
professional is on hand to offer contextual information
about objects on display or answer questions.
By extension, museum professionals may assume
patrons will be interested in viewing digitized
representations of objects. For the MOAC online resource,
then, the question becomes, what do we know about user
needs for information about museum objects when users
are not situated in front of the physical object and are not
interacting with a museum professional, other than the
rather limited mediation provided by standard museum
descriptive cataloguing and surrogate images? The MOAC
II study will add to this area of investigation by providing
valuable experience, information, and suggestions for
museum (and library) digital content producers and
providers. We anticipate sharing our preliminary findings
in Fall 2004.
As content producers and providers, MOAC II partners
have offered considerable feedback about the MOAC II
research questions and research instruments and their
involvement has revealed the importance of collaboration
in this evaluation. Partners have brought an understanding
of audiences and content to bear when validating the
research methodology – making the methodology
applicable for this particular project and community. In
particular, museum educators and curators are bringing
considerable insights to the development of MOAC II –
these professionals interact regularly with patrons.
Together with museum partners, we are exploring
issues related to providing, accessing, and using online
cultural content. For instance, how do users use visual
information? Are teachers ready to teach with museum
objects? If teachers do not have the mechanisms or time to
widen contextualization beyond that found in familiar
textbooks, what tool sets might be developed to enhance
teaching with museum objects? We expect the MOAC II
study to elicit detailed information about the relevance of
MOAC, especially in terms of how users might want to use
descriptions and images in their work, and the effectiveness
of the online display and navigability aspects of MOAC
and the OAC.
Partners have continued to develop content and design
elements for MOAC even as the MOAC II evaluation
progresses. In fact, the evolving nature of MOAC is typical
of Web-accessible resources and has added a “real world”
dynamic to the evaluation. That is, MOAC and the OAC
are not artificial test databases; rather, they are live
resources, with changes being made to the interface and
new or revised content added regularly. In addition,
MOAC content can be accessed from two locations on the
Web and each point of access has a distinct style or look.
Feedback from the MOAC II study may suggest that
more or different changes to MOAC are necessary,
especially if MOAC is to accommodate diverse user needs
for picking, mixing, and using digitized content. MOAC
partners will assess the practicality of integrating what
users want into their routine production and distribution
activities. Already, partners have begun to consider what
can and cannot change about the content, packaging, and
presentation of their MOAC contributions, as well as
discuss with whom responsibility lies for implementing
what uses want (or deciding to maintain a status quo).
Notes
1MOAC is a partnership between the California Digital
Library (CDL), the UC Berkeley Bancroft Library, UC Berkeley
Art Museum and Pacific Film Archive, Japanese American
National Museum, Oakland Museum of California, UC Berkeley
Phoebe A. Hearst Museum of Anthropology, UCLA Grunwald
Center for the Graphic Arts, UCLA Fowler Museum of Cultural
History, UCR/California Museum of Photography, and San
Francisco Museum of Modern Art. The CDL leads a subset of
these partners in MOAC II. The UCLA Department of
Information Studies leads the MOAC II research team.
2Rinehart,
R. Museums and the Online Archive of California:
Museum Consortia, Digital Library Projects, Standards for
Complex Multimedia Objects, and More Fun than a Barrel of
Monkeys. SPECTRA 28, no. 1, 20-27, 2001.
3See
for example:
Gilliland-Swetland, A., R.L. Chandler & L. White. MOAC II User
Evaluation: Making Museum Content Useful. Proceedings of the
66th ASIST Annual Meeting. Medford, NJ: Information Today,
2003.
Gilliland-Swetland, A., C.M. MacLeod, M.K. Svetlik & L. White.
Evaluating EAD as an Appropriate Metadata Structure for
Describing and Delivering Museum Content: MOAC II
Evaluation Study. Proceedings of the International Conference on
Digital Libraries, 2004 (forthcoming).
Gilliland-Swetland, A., R.L. Chandler & L. White. We’re
Building It, Will They Use It? The MOAC II Evaluation Project.
Proceedings of the Museums and the Web conference, 2004.
Consulted June 14, 2004
http://www.archimuse.com/mw2004/papers/g-swetland/gswetland.html.