T. Lombardi, S. Cha, and C. Tappert
January 19th, 2005

Students of art history learn three primary skills:



Formal analysis
Comparison
Classification
How can computer science contribute to the development of these skills?
Figure 1: Girl with a Pearl Earring,
Jan Vermeer, 1665
Electronic Imaging 2005


An Interactive Indexing and Image Retrieval
System (IIR) for fine-art paintings can aid students in these endeavors by providing:
 a mathematical summarization of an image
 a measurable basis for comparing two images
 an elementary way to classify an image relative to those in a database
Electronic Imaging 2005

We synthesize the goals of two research areas:

Classification of paintings:

R. Sablatnig, P. Kammerer, and E. Zolda, “Hierarchical Classification of Paintings
Using Face- and Brush Stroke Models”, in Proc. of the 14th International Conference on
Pattern Recognition (1998).

D. Keren, “Painter Identification Using Local Features and Naïve Bayes”, in Proc.
of the 16th International Conference on Pattern Recognition (2002).

Image retrieval which aims to bridge the semantic gap:

J. Corridoni, A. Del Bimbo, and P. Pala, “Retrieval of Paintings using Effects
Induced by Color Features”, in Proc. of the International Workshop on Content-Based
Access of Image and Video Databases (1998).

Can we construct a feature set that satisfies the objectives of both areas while providing analytically relevant data to students?
Electronic Imaging 2005

The system consists of two major components:


Image Database stores images, thumbnail images, and extracted features for later retrieval and analysis.


Graphical User Interface provides interactive query capabilities to the end user
Electronic Imaging 2005


An XML index file stores extracted features and control information.

A file system stores images and thumbnail images.

The open design of the database contributes to the goals of ease of use and exchange of information.
Electronic Imaging 2005

Figure 2: XML Index File
Electronic Imaging 2005
Figure 3: File System

Two different kinds of features are extracted:

Palette features

 concern the set of colors in an image (color map) examples: palette scope


Canvas features concern the spatial and frequency distribution of colors in an image (image index)
 examples: max, min, median, mean (for each color channel)
Electronic Imaging 2005

Table 1: Sample Features used for Web Museum Interactive Test
Feature Name Description and Notes
Max Max value of H, S, and V channels
Min
Mean
Median
Standard Dev.
Color Entropy
Line Count
Intensity Mean
Min value of H, S, and V channels
Mean of H, S, and V channels
Median of H, S, and V channels
Std of H, S, and V channels
Measures the frequency distribution of color
Normalized number of detected edges – Sobel edge detector
Arithmetic mean of values in a grayscale image
Electronic Imaging 2005

Figure 4: Hallucinogenic
Toreador
Salvador Dali, 1970
Figure 5: Composition with Large Blue Plane,
Red, Black, Yellow, and Gray
Piet Mondrian, 1921
Palette Scope -- the total number of unique colors used in an image.
We expect Dali’s piece to have a higher palette depth than Mondrian’s work.
Electronic Imaging 2005

Formal definition of Palette Scope (U):
U = C/P
Where
C=Total # of unique colors measured in RGB or
HSV triples.
P= Total # of pixels in an image.
Electronic Imaging 2005

Table 2: Palette Scope statistics.
Artist
Mondrian
Dali
Total Pixels (P) Total Colors (C) Palette Depth (U)
359700 2242 0.00623
165775 3899 0.02351
We see that Dali uses more of the color spectrum than Mondrian.
Palette depth is an important feature for artist and period style identification because many styles are defined by color, i.e. Picasso’s
Blue Period and fauvism.
Electronic Imaging 2005


The GUI consists of three primary windows for:

Analysis

Comparison

Classification
Electronic Imaging 2005

Figure 6: The Analysis Window
Electronic Imaging 2005

Figure 7: The Comparison Window
Electronic Imaging 2005

Figure 8: The Classification Window
Electronic Imaging 2005

Two types of tests were conducted:



Feature tests
Feature tests focus on the accuracy of specific collections of features.

Interactive tests
Interactive tests assess the accuracy of the system as a whole.
Electronic Imaging 2005

Figure 9: Les Demoiselles d’Avignon,
Pablo Picasso, 1907.
Figure 10: Road with Cypress and Star,
Vincent Van Gogh, 1890.
Table 3: Feature test to distinguish the work of Picasso and Van Gogh.
Training Set
36
200
200
Test Set
36
200
200
Percent Correct
94
88
83
Electronic Imaging 2005

Database of 10 works of each of the following ten artists:
Braque, Cezanne, De Chirico, El Greco, Gauguin,
Modigliani, Mondrian, Picasso, Rembrandt, and Van
Gogh.
Training Set
100
Table 4: Initial Interactive Test
Testing Set
90
Percent Correct
81
Electronic Imaging 2005

Table 5: Results from Web Museum Interactive Test
Artist
Aertsen
El Greco
Hopper
Malevich
Monet
Morisot
Rembrandt
Renoir
Turner
Velazquez
Overall
Training Set
9
10
10
10
10
10
10
10
10
10
500
Queries
9
7
7
11
10
11
33
38
10
8
293
Electronic Imaging 2005
Success
5
4
3
8
6
7
25
14
4
8
165
Percent
55.6
57.1
42.9
72.7
60.0
63.6
75.8
36.8
40.0
100.0
56.3


Overall result: 56.3% accuracy

36.3% better than blind guessing (10 guesses/50 artists = 20%)

Dissecting the classification mistakes reveals some intelligent mistakes

Rembrandt is most often confused with Caravaggio,
Ast, and Vermeer
Electronic Imaging 2005





Simple palette and canvas features are sufficient for an interactive classification system
A single feature set can serve for classification and image retrieval applications
A general feature set can adequately serve for educational applications
Although showing promise, we currently have a low confidence system
Electronic Imaging 2005


Add texture features

Improved color features: hue histograms

Improved distance metrics: modulo comparison of hue histograms

Test against larger datasets
Electronic Imaging 2005