Art Images Online:
Leveraging Social Tagging and
Language for Browsing
Judith Klavans1, Jen Golbeck1, Susan Chun2, Rob Stein3,
Ed Bachta3, Irene Eleta1, Raul Guerra1, Rebecca LaPlante1
University of Maryland1 Independent Museum Consultant2
Indianapolis Museum of Art3
http://umiacs.umd.edu/research/t3/
High Level Goals
• Images in Museums and Libraries
• Words…words… words
– Traditional cataloging
– Handbook and other descriptive text
– Social tagging
2
Record from American Institute for College Teaching
Minimal
metadata for
image, no
descriptive
terms.
3
Nefertiti
Gardner (v. 11, pl. 3-33)
The famous painted limestone bust of
Akhenaton’s queen, Nefertiti (fig. 3-33),
exhibits a similar expression of entranced
musing and an almost mannered sensitivity
and delicacy of curving contour. The piece was
found in the workshop of the queen’s official
sculptor, Thutmose, and is a deliberately
unfinished model very likely by the master’s
own hand. The left eye socket still lacks the
inlaid eyeball, making the portrait a kind of
before-and-after demonstration piece. With this
elegant bust, Thutmose may have been
alluding to a heavy flower on its slender stalk
by exaggerating the weight of the crowned
head and the length of the almost serpentine
neck…
4
Excerpt of descriptive text from Gardner (v. 11, pl. 333), suggested CLiMB terms highlighted in yellow
5
The famous painted limestone bust of
Akhenaton’s queen, Nefertiti (fig. 3-33),
exhibits a similar expression of entranced
musing and an almost mannered sensitivity
and delicacy of curving contour. The piece
was found in the workshop of the queen’s
official sculptor, Thutmose, and is a
deliberately unfinished model very likely by
the master’s own hand. The left eye socket
still lacks the inlaid eyeball, making the
portrait a kind of before-and-after
demonstration piece. With this elegant bust,
Thutmose may have been alluding to a heavy
flower on its slender stalk by exaggerating the
weight of the crowned head and the length of
the almost serpentine neck…
Woman
User Tags
Hat
One blind eye
Beautiful features
Long neck
Beautiful woman
Blue
Yellow
Blue hat
Graceful eyes
Ceramic statue of an elegant woman
Half an ear
6
I’d like this in my living room
Many kinds of Words…..
• Terms informed by art historical
criteria:
– deliberately unfinished model
• Ability to find related images
– elongated neck
– bust
• Potential for using thesaural
resources
– painted limestone bust
7
CLiMB:
Computational
Linguistics for
Metadata Building
Columbia University
University of Maryland – UMIACS
and
Computational Linguistics and Information Processing (CLIP) Lab
9
STEVE.MUSEUM
• 18 museum partners
• Over 90,000 tags
• Nearly 1800 images tagged
• But tags are “unruly” and “chaotic”
10
T3: Tags, Terms, and Trust
• What computational linguistic techniques can
be used to bring all these words to use?
• What is the nature of the tags?
• What tools can be helpful to the museums
and library user communities?
• How does tagging in different languages
compare?
11
T3 Research Group
Judith Klavans
Museum Partners
Rob Stein
Jen Golbeck
Ed Bachta
Irene Eleta
Susan Chun
Raul Guerra
… and
Rebecca LaPlante
12
18 museums
Funding
• Mellon Foundation
– CLiMB-1 (Columbia Univ)
– CLiMB-2 (Univ of Maryland, UMIACS-CLIP
Lab)
• IMLS –
– Steve.museum (Indianapolis Museum of Art)
– T3 – IMA and University of Maryland
• National Science Foundation
13
Major Contribution – Creating
Order over Chaos
• Developed techniques for processing social
tags
– What tags are related to other tags?
– How are tags related (or not)?
– What is the impact of cultural and language
differences in tagging?
• Completed user analysis of tagging behavior
• Explored the value of tags compared with
descriptive text
14
Highlight Two Specific Areas
The value and use of
–Computational linguistic analysis for tags
–Multilingual social tags
15
Computational Processing
Pipeline
Woman
Hat
One blind eye
Beautiful features
Graceful - [Adjective] {graceful}
Long neck
Eyes – [Noun-Plural] {eye}
Beautiful woman
Blue
One - [Number]
Blind – [Adjective] {blind}
Eye –[Noun-Singular] {eye}
Blue hat
Graceful eyes
Word Count
Ceramic statue of
an elegant woman
Woman – 3
Half an ear
Eye - 2
16
Yellow
What is a tag ?
Important to treat tags related to the same topic as
related, e.g.
Line, line, lines, lining
How many “tags”? Four, Three, or one?
Leaves
Puppies
Babies
17
Leaves
Stemmer
puppi
Puppies
Babies
Lemmatizer
puppy
What is a tag ?
Comparing Stemmers and Lemmatizers
• Stemmers: reduce words to roots.
–
Advantage: fast with big data
– Disadvantage: stems are not necessarily words
• Lemmatizers: reduce words to ‘dictionary’
form.
– Advantage: lemmas are more useful
– Disadvantage: slower because they rely on external
resources like Wordnet
18
How do we do?
• Checking Lemmatizer performance against
Lemma Gold Standard of 850 user-tags:
• Default configuration of the pipeline
– 64% accuracy on all tags
– 68% accuracy on all the correctly spelled tags
• Fine grained configuration of the pipeline
– 76.47% accuracy on all tags
– 81.35% accuracy on all the correctly spelled tags
19
Part of Speech Labeling
“graceful eyes” - [[Adjective] [Noun]]
“blue” – [Adjective]
“face” – [Noun] , [Verb]
20
21
Part-of-Speech Labeling
• Harder than annotating text because of lack of context.
22
The famous painted limestone bust of
Akhenaton’s queen, Nefertiti (fig. 3-33),
exhibits a similar expression of entranced
musing and an almost mannered sensitivity
and delicacy of curving contour. The piece
was found in the workshop of the queen’s
official sculptor, Thutmose, and is a
deliberately unfinished model very likely by
the master’s own hand. The left eye socket
still lacks the inlaid eyeball, making the
portrait a kind of before-and-after
demonstration piece. With this elegant bust,
Thutmose may have been alluding to a heavy
flower on its slender stalk by exaggerating the
weight of the crowned head and the length of
the almost serpentine neck…
23
Handbook Text Chunking
• As a result of the way users tag images,
– FEW images have LOTS of tags
– LOTS of images have FEW tags
• Can we add tag-like phrases from text
– Two types of phrases
• Names of things (Named Entities)
• Nouns and all the words that go with it ( Noun Phrases )
24
Results
Precision for Named Entities is 46.39% for Full
Match and 69.24% for partial matches.
– Columbia University in the City of New York
– Columbia University
Precision for Noun Phrases is 79.95% for Full
Match and 93.03% for Partial matches.
– Famous painted limestone bust
– Limestone bust
25
Conclusions
• Stemmers/Lematizers are the first step for
removing variation from the tags.
• Part of Speech Tagging of social tags useful for
getting a better understanding of the tags
• Handbook Text Analysis helps to add tags for
poorly annotated images.
• The pipeline enables to add information to the
user tags that is useful for applications using
them.
26
Multilingual Social Tagging of Art Images
Cultural Bridges and Diversity
Polynesian women
jungle
baby
Polinesia
Virgen roja
Jesús
La Orana Maria, by Gauguin
(Metropolitan Museum of Art)
Questions
• How similar are the tags provided by different
language communities when tagging art
images?
• Does it depend on the type of painting?
• Do tags reflect cultural differences in the
interpretations of the paintings?
28
Contributions of this Study
• Expands T3 to include Spanish
• Identifies tags that could be used
for multilingual search
• Identifies tags that could be used
for cross-cultural discovery and
understanding
• Proposes the separation of tagging
environments by language
29
Data Collection
American participants
Spanish participants
IN NUMBERS…
5 questions
33 paintings
773 English tags
566 Spanish tags
10 images
30Genesis
First Version By Lorser Feitelson (San Francisco MOMA)
Processing and Comparing Tags
• Tag tokenization and normalization
• Creation of lexical correspondences
The Cotton Pickers by Winslow
Homer (Los Angeles Contemporary
Museum of Art)
31
Semantic Analysis
LaPlante,
R., Klavans, J., Golbeck, J (under review). Subject Matter
32
Categorization of Tags Applied to Digital Images from Art Museums.
Results
• Many exact translations
The Cotton Pickers by Winslow Homer
(LACMA)
33
Translation Pairs by Type of Painting
34
Differences
The Cotton Pickers by Winslow Homer (LACMA)
35
Translation pairs > multilingual search
– “general person or thing” in realistic paintings
– “visual elements” in abstract paintings
Different perspectives > design for discovery
– “emotions and abstract ideas”
36
Publications
Computational techniques to examine tags and phrases
of importance for browsing on art image collections.
• Klavans, Judith, Guerra, Raul, LaPlante, Rebecca, Stein, Rob & Bachta, Ed
(2011). Beyond Flickr: Not all image tagging is created equal. In AAAI 2011
Workshop: Language-Action Tools for Cognitive Artificial Agents, San
Francisco, CA.
• Klavans, Judith, Stein, Rob, Chun, Susan, & Guerra, Raul (2011).
Computational linguistics in museums: Applications for cultural datasets.
Museums and the Web 2011, Philadelphia, PA.
37
Publications
Comparing social tagging patterns in two languages to
inform design for multilingual access to art images.
• Eleta, Irene and Jennifer Golbeck (2012, to appear) A Study of Multilingual
Social Tagging of Art Images: Cultural Bridges and Diversity. 2012 ACM
Conference on Computer Supported Cooperative Work (CSCW 2012),
Seattle, Washington.
38
Thank you!
Publications:
http://umiacs.umd.edu/research/t3/
Tools available through Steve in Action:
http://www.steve.museum/
39