Usability Testing: Art Museum Web Sites
Woo, Jeongwon
A research project for the course
EDC 385G: Designs and Strategies for New Media
Spring 2006
Table of Contents
Introduction………………………………………………………….…….3
The Websites………………………………………………………………3
Methodology………………………………………………………………6
Data collection…………………………………………………………….8
Results…………………………………………………………………....10
Discussion………………………………………………………….…….33
Recommendations……………………………………………………….35
Conclusion……………………………………………………………….37
Summary…………………………………………………………………37
References…………………………………………………………….….38
Related Links……………………………………………….………….…38
Appendices (from A to G)………………………………………………..39
2
Introduction
Generally, museums exist to display extensive collections such as works of art, scientific
artifacts and historic objects. People can appreciate and experience valuable exhibits by
visiting various museums. Especially, thanks to the advancement of World Wide Web,
we also can appreciate exhibits in a virtual museum at any time without visiting the
museum actually. Thus, most museums nowadays employ both actual museums and
virtual museums. So we can take full advantage of museum as vast source for education
and a means to enrich the culture.
Art museum is the exemplar of classical museums and the most familiar type of museums.
This research paper is to test the usability of art museum web sites. Most popular and
famous three art museum web sites were selected and tested by two users. Tasks and
procedures for usability testing were developed refer to Nielson (2000)’s Methodology
report and Rubin (1994)’s usability handbook. Participants were required to clear both
specific and open-ended tasks after answering pre-test questionnaire. Also, there was a
short interview and survey about the websites. The recommendations for each site were
made based on results from usability testing. This study will provide useful information
to improve virtual museum or museum web sites. In addition, it will give some ideas for
the people who plan to develop online museums.
The Websites
The three websites studied in this paper were selected based on the following criteria:
- Popularity and reputation of actual museums
- Abundance of art collections
- Art museums in the U.S.
- Famous to the general public
- Have virtual museums(able to see the collections through the website)
1) The Metropolitan Museum of Art
www.metmuseum.org
The Metropolitan Museum of Art is the biggest art museum in the U.S. and located at
New York City. It has almost over 3,000,000 collections including Egyptian art, Greek
art, Medieval and European paintings, American paintings, ancient artifacts and so on. It
holds over 30 exhibits a year and 5,500,000 visitors from all over the world. It surely
works as the center of art museum in the U.S. The purpose of museum is to cultivate the
quality of life through arts.
3
2) The Museum of Modern Art
www.moma.org
The Museum of Modern Art (so-called MOMA) is also located at New York City. As the
name explains, it was established in 1929 to exhibit paintings, sculptures and prints as
well as photographs, architecture and commercial designs which were made after 1880’s.
MOMA has over 20,000 collections and be famous for collection of Picasso, Rousseau
and Matisse. MOMA intends to enlighten visual art through modern art.
4
3) Museum of Fine Arts, Boston
www.mfa.org
Museum of Fine Arts at Boston is the next comprehensive art museum after Metropolitan
Museum of Art. MFA has exhibits divided by 7 themes: Egyptian Art, Ancient Art, Asian
Paintings, European Paintings, Prints, Dyeing and Weaving and American Art. Especially,
it is famous for Asian collections like quality arts of Indian, China, Korea and Japan. It
also has subsidiary art school (School of the Museum of Fine Arts) to educate artist and
provide various certificate and continuing education programs.
5
Methodology
The Participants: Two participants were selected to perform the usability test of the
three art museum web sites. Their interests and background information was collected
through pre-test questionnaire (see the Appendix B). They were selected because they
ordinarily showed interest about works of art and artists. The information is as follows:
• Lee, S:
- 38 years old, single, male
- Background: Ph. D. student in Curriculum & Instruction Department of University of
Texas at Austin
- Computer experience: He usually uses the Internet at home and spends approximately 4
hours a day. He uses Internet for checking e-mail, researching, shopping, entertaining,
information and blogging. He has managed his blog for almost 2 years and regularly
updated it.
- Interest about art: His level of interest about fine art is very high. Even though he had
no chance to visit museums in U.S., he enjoys visiting museums. When he was in Korea,
he often visited Korean National Museum of Contemporary Art. He also visits online art
museums frequently like Rijksmuseum in Amsterdam, Museum of Modern Art, and The
6
National Gallery in London. He likes Netherlands painters and their paintings of 17C. He
utilizes these pictures with his essays in his blog. He knows well about his area of interest.
• Kim, S:
- 28 years old, single, female
- Background: Ph. D. student in Neuroscience of University of Texas at Austin
- Computer experience: She uses the Internet both at work and home. She spends almost
3 hours a day to check e-mail, research, entertaining and get information. She hardly
shops by online.
- Interest about art: She has medium level of interest about fine art. But she enjoys
visiting museums and has some knowledge about art and artists. The most impressive
museum she ever visited was the Guggenheim Museum in New York City because of the
beautiful structure of it. Also, she actually visited Museum of Modern Art (MoMA) twice.
She hardly visits art museum web sites. She likes modern artists such as Monet, Gogh,
Kandinsky and Klimt.
The Tasks: The tasks were developed based on Nielsen’s Methodology (Nielsen et al.,
2000). And also, after considering the general use of art museum web sites, specific tasks
were designed. For example, one of the common tasks for three sites was to find a
specific painting among their collections. The tasks are divided into three parts:
TaskⅠ(general impression), TaskⅡ(specific tasks), and Task Ⅲ(open-ended tasks).
The tasks given to each user were the following (To see more detailed tasks, see the
Appendices).
TaskⅠ- General impression: This task was to get the general first impression of the web
sites. The questions are described in a similar way as in Nielsen’s methodology. Minor
adjustments were made to the questions to reflect information that users would typically
want to find out from an art museum website.
Task I
a) What do you think this website is about?
b) Who do you think is its target audience?
c) If it is the first visit of this web site, what would you say about that?
d) At first glance, do you think this site could provide you with any information of
your interest?
Task Ⅱ- Specific tasks: Participants are required to perform specific tasks at each web
sites. The tasks are the most common functions for the art museum web sites. Some
changes are made according to each websites but the basic requirements for the tasks are
similar.
7
Metropolitan Museum of Art
Task II
Museum of Modern art
• Find the painting ‘Boating’
of Édouard Manet from its
collections and get the
information about when the
painting was drawn. Zoom the
image.
• Find the painting ‘Les
Demoiselles d'Avignon’
of Pablo Picasso from its
collections and get the
information about when
the painting was drawn.
• Appreciate the exhibition
‘Samuel Palmer: Vision and
Landscape’ paintings with
audio guide.
• Find the educational
material: ‘How Van Gogh
made his mark’
• Send an E-card to a
friend with the painting
you’ve found in previous
task.
• Appreciate the
exhibition ‘Edvard
Munch: The Modern Life
of the Soul’ paintings
with audio tour.
• Find the information about
• Find the information
Metropolitan museum’s
about MOMA’s
admission fee, directions to go admission fee, directions
there, open hours written in
to go there, open hours.
Korean.
Boston Museum of Fine Art
• Find the painting
‘Postman Joseph Roulin’ of
Vincent Van Gogh from its
collections and get the
information about when the
painting was drawn. Then,
listen to the audio file
explaining the painting.
Zoom the image.
• Send an E-card to a friend
with the painting you’ve
found in previous task.
• Find out how many
exhibitions are held now.
• Find the information about
Boston Museum’s
admission fee, directions to
go there, open hours.
Task Ⅲ- Open-ended tasks: This type of task is to find out what is going on at the
websites and get the insights we hardly see (Neilson et al. 2000). In this task, users are
allowed to explore the web sites freely based on their own interests.
Task III
a) Buy some fine art product(e.g. posters, panels, books, jewelry…) you want at
the Museum’s online store
b) Explore and find more about your favorite artist or art
Data Collection
Participant Background Questionnaire: After getting signature of consent form from
the participants, each participant was given with a pre-test questionnaire. The
questionnaire is divided into three parts: demographics, computer experience and interest
about art. The first part of the questionnaire is about age, occupation, and the filed of
study now. Second part is about usage of the Internet such as Internet access place, time
of usage per day, and the purpose of using the Internet. The last part is to find out the
8
participant’s usual interest about fine art. It has questions about interest level about fine
art, museum visit experience, and online art museum experience (See the Appendix B).
Task Construction: The three tasks are made by researcher. The first task is to know the
first impression about the art museum website. In this task, participants had 2-3 minutes
to explore the sites and then answered the questions: what do you think this website is
about? , who do you think its target audience?
Task Ⅱ is composed by specific tasks and open-ended tasks for each site. Then, there
was a simple survey about the easiness during performing the tasks. Specific tasks are a)
find a specific painting from the museum’s collection using its search function and zoom
the image if possible, b) send an E-card to a friend with the painting found in previous
task (Not for the Metropolitan Museum), c) Appreciate special exhibition held in the
Museum now with audio guide (Not for the Boston Museum of Art), and d) Find the
information about admission fee, directions to go there and open hours. The paintings are
different for each museum and some tasks are replaced by similar one because one
museum doesn’t provide some functions which other two museums provide.
Task Ⅲ is open-ended tasks. It is to buy fine art products such as posters, panels, books,
jewelry and so on. Then, participants can explore freely according to their favorite artists
or areas of interest.
Short survey is about the easiness of use which is related to perform the tasks. For
example, the data about the easiness of searching a specific painting and quality of the
images seen on the screen were gathered. The data was collected by 7 point Likert scale
(See Appendix C, D, E, and F).
Art Museum Web site Usability Questionnaire: When all the tasks are performed,
participants are required to complete the overall usability questionnaire. The
questionnaire has 27 questions divided by 4 sections: overall impression, information
design, interaction design, and interface design. The data was also estimated by 7 point
Likert scale (See Appendix G).
Usability testing execution: During the process of task execution, researcher sat beside
the participant and took notes of their behavior and actions when participants attempted
to complete tasks. Their reactions during performing the tasks were observed and
recorded by the researcher. If the participants had problems about the task, researcher
gave some hint and recorded the problem. Moreover, their comments and
recommendations about websites were documented. The average usability testing hour
was about one and half hour per a participant.
Testing Facilities: The usability tests were conducted at the researcher’s home with
personal desktop computer. Thus, the testing environment was not influenced by
distracter factors like other people or noises. The web browser used for the test was
Internet Explorer 6.0 operated by Windows xp. The setting of computer was Pentium 4
processor and the Internet speed was 100M bps. The monitor was 19’’ flat LCD monitor
and resolution was 1280x1024.
9
Results
Findings from the task execution of two participants are discussed below. The results are
organized by each art museum web site and the three tasks.
1) The Metropolitan Museum of Art (www.metmuseum.org)
Task Ⅰ- The first impression of the Met museum website was that it seems very formal
and traditional. It seems to have many information and lots of collections of art. However,
one participant said that it is too informative or educative and thus boring. Both
participants reported that the target audience of this site is the adult and general public.
The index page is too overwhelming by small texts. The collection and exhibition are not
clearly divided. And it may confuse the users. The web site seems to have the
information we need but it is not neatly organized.
Task Ⅱ- The first task is to find ‘boating’ of Édourad Manet from the museum’s
collection and get information about that painting. Both of the participants had no
problem finding that picture using the search function. If the user input the title of
painting or the artist name, the search engine shows all the items which has that words
and organizes the result according to the categories: works of art, special exhibition,
membership, miscellaneous, and met store.
10
Zooming the image was satisfying for both users. Because Met Museum provides
advanced zooming function, it is easy to look closely the whole painting. Met Museum’s
image quality got the highest score in the questionnaire.
11
The second task is to appreciate special exhibition ‘Samuel Palmer: Visions and
Landscape’ with audio guide. Both participants succeeded doing this task but they were
12
confused by the audio guide function between ‘download the audio file’ and ‘stream the
full audio program’. Because the menus for exhibition are basically just text only menus
and links, users are easy to be confused. Moreover, the website does not provide the
paintings of special exhibition and audio explanation in combined way. Compared to
MoMA’s online exhibition, it looks like the graphic and audio do not match.
The third task is to find the special material made by the Met Museum: How Van Gogh
made his mark. As we can see form its first page, the Met Museum concerns about
educational programs for children and students. Thus, the museum prepared various flash
movie clips about some artist or certain kind of art. The materials are under the ‘Explore
13
and Learn’ menu. But one of the participants didn’t find that information and needed help.
However, both participants showed satisfaction about making that kind of educational
material.
The final task is to find the visitor’s information such as admission fee, directions, and
open hours. It was easy for both users because that ‘visitor information’ was one of the
main menus. Especially, the Met museum provides 10 different languages on visitor’s
information page. The participants showed satisfaction with its consideration of foreign
users. And the information was very detailed for actual visitors.
14
Task Ⅲ- In the open-ended tasks, shopping was the main task. The male participant
bought a book and the female participant bought a magnetic after looking around the
online store. The female participant showed interests about shopping function of museum
web site but she said that the ‘Met Store’ is not attractive enough compared to other two
museum online stores. They provide shopping status bar for the customers.
15
When exploring the website, the male participant said that the timeline and collection
matching was very impressive. He told that it’s a good idea to provide a separate
submenu ‘Timeline of Art history’. Also, he reported that it was very informative and
useful both for general public and students.
Although the Met museum expressed that it is famous for kid-friendly art museum. It was
pointed out that it actually not that friendly. Because it only has the list of materials
which already exist in the museum web site, the kid-museum does not have its own
purpose and mission. Although one of the participants said the Met Museum is too
educative, it’s not clear that the special features were made for kids or for adults.
16
The Met Museum’s permanent collection is organized by its own categories. But the
search function is too simple considering its vast amount of collections.
17
And in the survey about the easiness of usage, the average point for easiness of search
function was 4.5 which was same with MOMA but lower than Boston Museum. About
the quality of images, the point was 5.5 which was top with the Boston Museum. The
overall easiness to appreciate museum collection/exhibitions was 5.5. And the easiness of
shopping was 6.5 and easiness of navigation was 5. Compared to other two museums, the
results say that Met museum has advantage in quality of images, easiness of appreciation,
and the easiness of shopping.
2) Modern Museum of Art (www.moma.org)
Task Ⅰ- The first impression of MoMA was that it was neat and simple. However, one of
the participants said that the first page of MoMA has too much information which is
basically texts with links. Perceived target audience was general public and adults. One
participant said that the menus are not intuitive and obtrusive. The menus were designed
like text although they are actually buttons which changes color when mouse rolls over.
Task Ⅱ- The first task of finding the painting ‘Les Demoiselles d’Avignon’ of Pablo
Picasso was successful for both participants. However, the participants had problems
with searching. If one enters ‘Avignon’ in the search box at the top, it shows many results
related to the word ‘Avignon’ and it also organized search result into categories similar to
Met Museum: The collection, exhibitions, press, online store and other. The main
problem is that the first searching result was not the actual painting but the conservation
of the painting ‘Les Demoiselles d’Avignon’. Only by using the searching function in the
‘collection’ menu, participants were able to find that painting. One participant was trying
to find that painting from ‘paintings and sculpture’ menu, but the web site shows all the
thumbnails of pictures, it was impossible to find specific painting in that. Even though it
provides collection highlight at each areas, users said that it was a little difficult.
18
19
There are two versions of search function, but users do not feel that both of them are easy
to use.
20
As you can see above, the wrong search result is the link to the conservation of ‘Les
Demoiselles d’Avignon’ not the painting itself. Users can get the right result only from
searching ‘browse and search the online collection’ menu.
In addition, MoMA does not provide enlarge or zoom function, so it was impossible to
look the painting closely.
The second task was to send an E-card to a friend with the painting ‘Les Demoiselles
d’Avignon’ of Pablo Picasso. Both participants said that it was a good function but the
buttons were confusing. Because combining the word ‘preview & send’ into one button,
participants hesitated click that button. One participant pointed out that generally the
‘preview’ and ‘send’ buttons are separated.
21
The third task was to appreciate exhibition ‘Edvard Munch: The Modern Life of the Soul’
with the audio tour. The reaction about online exhibition with voice was positive for both
participants. If one clicks ‘full program’, it shows a pop-up window which has the
exhibition paintings and related audio guide. However, they expressed confusion among
‘full program’, ‘Flash 7 or higher’, and ‘download audio clips’. The reason is that they
are too closely located in the same page, and they are same in color, font, and size. And
one participant said that it was inappropriate to have main menus and supplementary
menu such as downloading specific programs at the same location.
22
The final task was to find the information about admission fee, directions and open hours
of MoMA. It provides this visitor information in 5 different languages.
23
Task Ⅲ- At the shopping task, one participant confused between online store and MoMA
stores. After finding the right page, both participants had no special problems with
shopping. However, both users complained that they can not get out of the store directly
and go back to MoMA main page. Instead of making link to MoMA homepage at the left
top, they provide just small button ‘MoMa.org’ at the left bottom. And it opens a new
window.
24
Exploring the website, one participant noticed that MoMA provides various educational
programs. Unlike Met Kid Museum, it has some materials made by Flash which children
can learn about modern art/artists with fun. Also, it has many adult programs,
international programs, and teacher programs. Participants expressed affirmative reaction
about these menus but they reported that overall navigation is too complex.
25
MoMA’s collection page is not very different from other museum’s collection page. But
it provides viewing options such as single object, browse, list and text only. But users
doubt that function is necessary.
The average point for easiness of search function was 4.5 which was same with Met
Museum but lower than Boston Museum. About the quality of images, the point was the
lowest 2.5 because it does not provide zooming function. The overall easiness of
appreciating museum collection/exhibitions was 4.5 and it was the lowest point. They
responded the search function is uncomfortable. And the easiness of shopping was 3.5
and easiness of navigation was also 3.5. It was hard to move between museum website
and online store site. And the buttons and menus are not clear. Compared to other two
museums, MoMA showed the lowest scores in all aspects in the easiness of using.
3) Museum of Fine Art at Boston (www.mfa.org)
Task Ⅰ- The first impression about Boston Fine Art Museum was that it was attractive
and neat. One participant said that the main page looks simple and menus are clearly
categorized. He said the accessibility to the information what he needs was the highest
among the three museum sites (only 6 menus). The other participant also expressed
positive reactions about the overall impression. Although the index page does not seem to
have too much information, the menus are very natural and intuitive.
26
Task Ⅱ- The first task was to find ‘Postman Joseph Roulin’ of Vincent Van Gogh from
its collections and then listen to the audio file explaining that painting. One participant
failed once because she confused between ‘site search’ and ‘collection search’, but she
completed the task. One participant said that he was very satisfied with their categories of
collections and menus.
27
About the audio guide, one participant didn’t find it easily because the sound icon was at
the bottom of the whole page. Also, the ‘enlarge function’ of the whole painting is
restricted to 800x800 pixels. Instead, it provides zoomed version of some partials of the
painting.
28
Same as the MoMA website, Boston Museum also provides E-card function. Both
participants had no problem performing this task. Yet, they said that send button was not
explicit and too small. Also, they can not preview the final version of E-card before
sending it.
29
Third task was to find out how many exhibitions are held now in Boston Museum. This
task was different from the other two museum sites because Boston Museum does not
provide online exhibition with audio guide. It just presents the introduction about
exhibition. The participants had no problem to find that information.
The final task was to find Boston Museum’s admission fee, directions, and open hours.
Like other museum web site, Boston Museum provides this information by main menu.
30
So participants had no problem completing this task. It also provides this information in 7
different languages.
Task Ⅲ- In this open-ended task, participants bought book and magnetic each. Both of
them had no problem in shopping these items. However, it found out that once the
customer checks out, it is hard to come back to museum website because there are no
buttons or links.
31
And during exploring the Boston Museum website, it was found that they also provide
special features about fine art history or specific themes. However, unlike other two
32
museum site, Boston Museum provides rather static information basically made by html
files. They has ‘online tour’ menu, but it was just the collection of pictures.
The average point for easiness of search function was 7 point which was the highest and
perfect score. About the quality of images, the point was 5.5 and it was the same with
Met Museum. The overall easiness of appreciating museum collection/exhibitions was 5
point. And the easiness of shopping was 5.5 and easiness of navigation was also 7 point.
Compared to other two museums, Boston Museum showed the most satisfactory results
in all aspects of the easiness of using.
Discussion
In this part, the usability of each site will be discussed under the perspective of
information, interface, and interface design. Data was collected from the “Art Museum
Web site Usability Questionnaire” (See Appendix G) which was implemented after
performing the tasks.
The Questionnaire collected data according to overall impression, information design,
interaction design, and interface design. And 7 point Likert scale was used. The results
are as follows:
33
Met Museum
MoMA
Boston Museum
Overall Impression
24/35
24/35
29/35
Information Design
41/63
34/63
49.5/63
Interaction Design
32/49
24.5/49
34/49
19.5/42
25.5/42
30.5/42
116.5/189
108/189
143/189
Interface Design
Total
As one can see, the Boston Museum website recorded the highest point in all the areas.
Met Museum is the second and MoMA is the third place based on the total score.
Overall Impression:
The overall impression is whether the website has users feel to explore more and revisit
the website. Also, it is related to affective aspects when using the website. Boston
Museum had the highest score in this section, but the other two sites had slightly lower
score. Due to the general characteristics of Art Museum, all three museum websites in
this study had their own attractive features. And also, art museum sites used various
images and cool movie clips from its first page. This affects the user’s experience of the
art museum website in positive way.
Information Design:
From information design perspective, the most important function of art museum site is
the organization/categorization of its vast online collections. It is related to find some
specific painting or artist among its collection. The title of menu and content should be
matched correctly. Moreover, as famous art museum, these websites should provide
meaningful and valuable information about fine art and present it in appropriate way.
They also should consider the amount of information shown for each of their exhibits.
The reason Boston Museum got the highest score in information design was because it
had simple but clear menus. Although it has only 6 main menus (Home, Visit, Calendar,
Exhibition, Collections, and About the MFA), they are the most frequently used functions
in online art museum. One participant of this study called this as ‘intuitive’. Considering
the Norman (2002)’s concept of natural mapping, these might be the most natural
functions as art museum. MoMA has the lowest score in information design. The reason
is that it mainly concerned about the modern appearance of the website. It had 11 main
menus at the first page but they are sometimes overlapped. Thus, people got confused
during using the site. Besides, the way they organize their collections seem to be
unnatural for finding specific pictures and artists. To manage the vast resources in
organized way, museums need to be focus on the strategies of information design.
Interaction Design:
In interaction design perspective, museum website should provide enjoyable and
memorable experience as virtual museum. It should consider special functions possible
for online website such as E-card and audio tour guide. The art museum website needs to
34
have good search engine for visitors to find a specific painting. Also, it has to be easy to
manipulate graphic images of the paintings (e.g. zoom, enlarge…). Moreover, the website
should be developed considering various visitors for the museum. For example, all the
museums provide multiple language version of visitor’s information. Boston Museum
website meets all these conditions. Also, Met Museum got high score in this part as the
biggest museum among the three. Users feel satisfaction about Boston Museum website
based on their easiness of appreciating works of art and appropriate search function. In
Met Museum website, users can manipulate the images easily and see them as closely as
possible. And users can save their preferred image in their ‘My Met Gallery’ menu if they
registered. Boston Museum also provides this service. And it is helpful for each museum
to have its own special materials. For example, Met Museum provides a very well
designed ‘timeline of art history’. MoMA has funny and interactive materials for kids.
These characteristics may be the main reason for revisiting the website.
Interface Design:
As famous art museums, all museum websites used different colors in different functions
smartly. Like other website, museum website should use consistent and attractive visual
style. Also, it should promote and advertise the museum’s image in positive way by using
charming graphics or multimedia. Still, the overall interface needs to be neat and simple.
The texts and graphics should be well-balanced in the web pages. Boston Museum has
the most simple and neat presentation of its content. Although MoMA gives us the first
impression of modernism and refinement, their menus and buttons are just text-based
links. Thus, users lost their interest easily and feel bored. Met Museum site has the most
complicated interface. Met Museum seems to follow the traditional general website
interface. Therefore, people feel that the website has too much information from its first
page. And it also has ‘educative’ environment not the enjoyment of appreciating works of
art.
Recommendations
The recommendations were made based on the results from usability tasks and
questionnaires. Each websites has its own features for improvement.
1) Metropolitan Museum of Art
Met Museum needs redesign from its first page. It should lower the density of texts and
change into neat interface. The main page gives the impression of formal and boring
museum. Thus, they should smartly adopt active and multimedia factors. About the
collection menu, they need to organize materials more clearly. Also, Met’s Kid museum
seems more like parents-friendly museum. It is just the list of educational materials in the
museum resources. The purpose of the materials is ambiguous because same material is
used for children and for adults. The materials need to be redesigned to focus on its target
audience. In other words, resources should be divided into adult materials and children
materials. It is good to appreciate full exhibition program in computer. But the menus are
complex under a special exhibition page. For example, ‘listen to full program’ and
‘downloading audio clips’ make users to think what the difference is. As Krug (2006)
35
said, well-designed web pages do not make users think deeply. And the online exhibition
service is not possible for all the exhibitions. Met Museum should consider adopting
‘fun’ factors and ‘customer-service’ spirit. For example, E-card service was very popular
among participants in other websites.
Met Museum has many attractive factors such as virtual reality tour and image
manipulate functions. Especially, they provide many options like enlarge, zoom, and
alternative view. Users in this study were impressed by this function. It is important to
provide rich resources but they have to meet various visitors’ need at their best. Met
Museum has long history and the biggest collection in the U.S. If it utilizes their capital
more actively for online environment, Met Museum can achieve two goals at the same
time: informative and enjoyable virtual museum.
2) Museum of Modern Art
MoMA is the center of Modern Art in the U.S. Its online museum seems very modern
and fancy. However, it showed many problems in usability. First of all, it should consider
more about user-friendly interface not the looking-good interface. In addition, it needs to
redesign the menu buttons and reduce the number of menus. Visitors are confused by
similar title of menus. For example, they put the ‘MoMA stores’ and ‘online store’ in the
same space. More importantly, they need to improve their search functions. It was found
that the search box on top and advanced search function below had different results when
entering same word. And the first item of the result list should be actual works of art not
from the ‘conservation’ menu. If MoMA intended to separate ‘site search’ and ‘collection
search’, that intention needs to be presented explicitly like the case of Boston Museum.
And the search result needs to utilize images with the texts. Because result from the top
search box only shows texts, users should read the text first and decide that was the
information he needed. If they provide thumbnails with the text, users can quickly and
easily process that information. And also, MoMA basically uses text links so it has to be
changed into icons or image links. MoMA does not have image zooming function.
Compared to other museums, it is a critical flaw because zooming and looking the image
closely is one of the merits of online museums.
But MoMA also provides useful functions such as E-card and online exhibition.
Although the buttons ‘send’ and ‘preview’ should be separated, it has the most advanced
version of e-card function. Especially in the exhibition menu, matching the paintings and
the audio guide was very beneficial for online museum visitors. Users can easily
appreciate online exhibitions with the audio guidance. Besides, MoMA has special
materials developed only for children. For example, ‘destination: modern art’ Flash
movie clip has an alien character who wants to know about earth’s modern art. It has
interactive learning factors which attracts children. MoMA maintains the image of active
and young museum. With the redesign of the site interface, menu structure and search
function, MoMA will be a pioneer virtual museum as the offline museum is so.
3) Museum of Fine Art at Boston
Boston Museum was the best museum site in all design perspectives in this study. It
might be hard to generalize this result, but it has good features as a museum website.
36
Boston museum simplified its menus according to the most frequently used functions.
Thus, visitors are not overwhelmed by vast information and resources. Although it has
good search function, it has some things to be improved. The ‘site search’ and ‘collection
search’ may confuse visitors. Moreover, the ‘image enlarge’ function falls behind
compared to Met Museum. And Boston Museum lacks interactive special materials.
Online museum visitors cannot see all the pictures of special exhibition. There is no way
to appreciate special exhibition by online. Boston museum should consider developing
online exhibition programs. Like MoMA, Boston Museum also provides E-card function
for all their collections. But the interface is a little weird to use comfortably. They should
provide ‘preview’ function before sending the E-card and make buttons more evident.
Moreover, Boston Museum website is mainly for adults. It does not provide any special
online materials for children. Although it provides many offline field trip programs for
educators, children actually don’t have chance to appreciate its collections by themselves.
Boston Museum needs to make special section for the young visitors.
Boston Museum succeeds in attracting online users by the simple menus and good search
function. It is thought to invite ‘natural mapping’ concept by Norman (2002) in its overall
design. The Boston museum website offers system which matches the user’s need and
online museum services. If it incorporates more interactive parts in the website, Boston
Museum will become an excellent museum website.
Conclusion
Due to the small number of participants of usability test, the result of this research is hard
to apply to all users. However, it is possible to find out important ideas about improving
online virtual museums. Based on the finding of this research, good museum website
should maintain the following characteristics.
• Information design
- moderate amount of information at the first level.
- simple menus which are used frequently in online museums
- clear organization of museum’s collections
- natural categorization of its exhibits
• Interaction design
- appropriate search function
- unrestricted manipulation of images (e.g. zoom, enlarge)
- consideration about various visitors (e.g. children, international visitors)
- specially designed materials which have clear purpose.
- characteristics for each museum website
- enjoyable virtual tour experience using multimedia technologies.
- matching of images and audios.
- supplementary functions only possible in online environment such as E-cards, online
shopping
• Interface design
37
- neat and attractive main page as an art museum
- smart use of color coding in different menus
- well-balanced text and graphics
- contains consistent image with its offline museum
Summary
This study was conducted to find useful implications when designing art museum web
sites. Three famous museums websites (Metropolitan Museum of Art, Museum of
Modern Art, and Boston Museum of Fine Art) were selected and tested by two users who
were interested in fine art for a long time. The data were collected from the tasks and
questionnaires. The results say that the best museum website was Boston Museum and
the worst website was MoMA. Results are discussed in the perspective of information,
interaction and interface design. Then, recommendations for each website are made based
on the usability test results. In conclusion, critical considerations for designing museum
website were proposed.
References
Krug, S.(2006). Don’t Make Me Think: A Common Sense Approach to Web Usability (2nd
ed.). New Riders Press.
Nielsen, J., Snyder, C., Molich, R., and Farrell, S. (2000). E-Commerce User Experience:
Methodology. Nielsen Norman Group. [Online] Available
http://www.nngroup.com/reports/ecommerce
Norman, D.A. (2002). The Design of Everyday Things. NY: Basic Books.
Rubin, J. (1994). Handbook of Usability Testing: How to Plan, Design, and Conduct
Effective Tests. John and Wiley & Sons.
Related Links
Metropolitan Museum of Art
Museum of Modern Art
Museum of Fine Art, Boston
www.metmuseum.org
www.moma.org
www.mfa.org
38
Appendix A
Consent Form
Study Administrator is:
Participant is:
Understanding Your Participation
You have been asked to participate in a usability study about how websites meet your
needs. Our purpose in conducting this study is to understand what makes some websites
easier to use than others. The results of this study will NOT be published in professional
reports. In the session, we’ll ask you to visit some websites and look for various things.
Please keep in mind that this is a test of websites; we are not testing you. We may
audiotape all or some of the test for evaluation purposes but we will not release the
recording to anybody. All information we collect concerning your participation in the
session is confidential.
Participation is voluntary. To the best of our knowledge, there are no physical or
psychological risks associated with participating in this study.
During the session, the study administrator will assist you and answer any questions. You
may take short breaks as needed and may withdraw from this evaluation at any time.
If you have any questions, you may ask now or at any time during the test.
Statement of Informed Consent
I have read the description of the study and I am aware of my rights as a participant. The
conductor of the research study has assured me that my identity will remain anonymous
and confidential. I agree to participate in the study.
Signature _______________________________
Date ___________________________________
39
Appendix B
Participant Background Questionnaire
Thank you very much for participating in this research study. Before you start your
usability testing, please answer the following questions. This information will be kept
strictly confidential. Please circle the appropriate answer.
User ID:
Date:
Ⅰ. Participant Information
1. Your age: under 18
2. Gender:
Male
19-24
25-30
31-35
36 or older
Female
3. Please state your current occupation:
4. If you are a student, state your field of study/research:
Ⅱ. Computer Experience
5. Where do you access the Internet?
Home
Work
Both
6. How many hours a day do you use the Internet?
7. What do you use the Internet for?(Choose all that apply)
E-mail
Research
Shopping
Entertaining
Information
Other, Please specify
Ⅲ. Interest
8. How’s your interest level about fine art?
Low
Medium
High
9. Do you enjoy visiting museums whenever you have chance to go? Yes
10. What is your most impressive museum you’ve visited?
11. Have you visited art museum web site before? If yes, please list them.
40
No
Appendix C
Usability Tasks
User ID:
Date:
Circle the website being tested.
Metropolitan Museum of Art
Museum of Modern Art
Museum of Fine Art
This is an exploratory exercise. Please take 2-3 minutes to just browse this web site.
Please jot down your first impressions of this web site in the space provided below.
Please be honest in your responses, your objective opinion will only support the purpose
of this study.
◈ Task Ⅰ
1. What do you think this website is about?
2. Who do you think is its target audience?
3. If this is the first visit of this web site, what would you say about this site?
4. At a first glance, do you think this web site could provide you with any
information of your interest?
41
Appendix D
Usability Tasks
User ID:
Date:
You will have four tasks for each site. Please follow the instructions and finish your
tasks. Then, complete the short survey below.
◈ Task Ⅱ (for Metropolitan Museum of Art web site)




Find the painting ‘Boating’ of Édouard Manet from its collections and get
the information about when the painting was drawn. Zoom the image.
Appreciate the exhibition ‘Samuel Palmer: Vision and Landscape’ paintings
with audio guide.
Find the educational material: ‘How Van Gogh made his mark’
Find the information about Metropolitan museum’s admission fee, directions
to go there and open hours written in Korean.
◈ Task Ⅲ

Buy some fine art product(e.g. posters, panels, books, jewelry….) you want at
the Museum’s online store
 Explore and find more about your favorite artist or art.
1- How easy is it to find the information about specific painting?
1
2
3
4
5
6
7
 
 
Not at all
Very easy
2- How satisfying the quality of images seen on screen?
1
2
3
4
5
6
 
7

Not at all

Very good
3- How easy is it to appreciate museum’s collections/exhibitions on line?
1
2
3
4
5
6
7
 
 
Not at all
Very easy
4- How easy is it to buy the fine art products?
1
2
3
4
5
 
6
7

Not at all

Very easy
5- How easy is it to navigate the web site?
1
2
3
4


Not at all
5
6
7


Very easy
42
Appendix E
Usability Tasks
User ID:
Date:
You will have four tasks for each site. Please follow the instructions and finish your
tasks. Then, complete the short survey below.
◈ Task Ⅱ (for Museum of Modern Art web site)




Find the painting ‘Les Demoiselles d'Avignon’ of Pablo Picasso from its
collections and get the information about when the painting was drawn.
Send an E-card to a friend with the painting you’ve found in previous task.
Appreciate the exhibition ‘Edvard Munch: The Modern Life of the Soul’
paintings with audio tour.
Find the information about MOMA’s admission fee, directions to go there,
and open hours.
◈ Task Ⅲ


Buy some fine art product(e.g. posters, panels, books, jewelry….) you want at
the Museum’s online store
Explore and find more about your favorite artist or art.
1- How easy is it to find the information about specific painting?
1
2
3
4
5
6
7
 

Not at all

Very easy
2- How satisfying the quality of images seen on screen?
1
2
3
4
5
6
 
7

Not at all

Very good
3- How easy is it to appreciate museum’s collections/exhibitions on line?
1
2
3
4
5
6
7
 
 
Not at all
Very easy
4- How easy is it to buy the fine art products?
1
2
3
4
5
 
6
7

Not at all

Very easy
5- How easy is it to navigate the web site?
1
2
3
4


Not at all
5
6
7


very easy
43
Appendix F
Usability Tasks
User ID:
Date:
You will have four tasks for each site. Please follow the instructions and finish your
tasks. Then, complete the short survey below.
◈ Task Ⅱ (for Boston Museum of Fine Art web site)




Find the painting ‘Postman Joseph Roulin’ of Vincent Van Gogh from its
collections and get the information about when the painting was drawn.
Then, listen to the audio file explaining the painting. Zoom the image.
Send an E-card to a friend with the painting you’ve found in previous task.
Find out how many exhibitions are held now.
Find the information about Boston Museum’s admission fee, directions to go
there, and open hours.
◈ Task Ⅲ


Buy some fine art product(e.g. posters, panels, books, jewelry….) you want at
the Museum’s online store
Explore and find more about your favorite artist or art.
1- How easy is it to find the information about specific painting?
1
2
3
4
5
6
7
 

Not at all

Very easy
2- How satisfying the quality of images seen on screen?
1
2
3
4
5
6
 
7

Not at all

Very good
3- How easy is it to appreciate museum’s collections/exhibitions on line?
1
2
3
4
5
6
7
 
 
Not at all
Very easy
4- How easy is it to buy the fine art products?
1
2
3
4
5
 
6
7

Not at all

Very easy
5- How easy is it to navigate the website?
1
2
3
4


Not at all
5
6
7


very easy
44
Appendix G
Art Museum Web Site Usability Questionnaire
User ID:
Date:
Circle the website being tested.
Metropolitan Museum of Art
Museum of Modern Art
Museum of Fine Art
Strongly
Overall impression
Strongly
Disagree
Agree
The website makes me explore more of it.
1
2
3
4
5
6
7
It is pleasant to go through the website.
1
2
3
4
5
6
7
It is easy to go thorough the website.
1
2
3
4
5
6
7
I would like to revisit this website.
1
2
3
4
5
6
7
I had no problem when using this site.
1
2
3
4
5
6
7
Information Design
Strongly
Strongly
Disagree
Agree
The menus and sections are
appropriately categorized as museum
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
website.
The information is well organized and
structured.
It is easy to find the information I need.
The information of this website is
meaningful and valuable.
The website provides rich database of
works of art and artists.
The title and the content of the
information match exactly.
They use accurate and clear language.
The way they present works of art with
information is appropriate.
The amount of information shown for one
45
page is moderate.
Strongly
Interaction Design
Strongly
Disagree
I can fully appreciate the works of art in
this website.
The website keeps letting me know where
I am and what I do.
They provide useful functions as a virtual
museum (e.g. audio tour, e-card…).
I can manipulate the images to look as
closely as I want like zooming.
The quality and size of images are
satisfying as a virtual museum.
They provide different functions based on
different visitors of museum.
Agree
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
They provide appropriate search
functions to manage their online
collections.
Interface Design
Strongly
Strongly
Disagree
They use consistent visual style (color,
font, graphic etc.).
I am attracted by the visual elements of
the site.
Agree
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
They use media effectively to promote
specific information (e.g. special
exhibition).
The design of the screen is simple and
neat.
They use space wisely to keep the
interface tidy.
Texts and graphics are well-balanced.
46
Author info:
This paper is written by Jeongwon Woo for the course EDC 385G Designs & Strategies
for New Media at the University of Texas – Austin.
47