Hybrid Intelligent Automatic
Image Annotation Using
Machine Learning
By
Mohamed Sami
Nashwa El-Bendary
Cairo University, Faculty of Computers and Arab Academy for Science,Technology, and
Information; ABO Research
Maritime Transport; ABO Research
Laboratory; Cairo, Egypt
Laboratory; Cairo, Egypt
e-mail: codemiles@gmail.com
e-mail: nashwa m@aast.edu
Aboul Ella Hassanien
Gerald Schaefer
Cairo University, Faculty of Computers and
Information; ABO Research
Laboratory; Cairo, Egypt
Department of Computer Science,
Loughborough University; U.K.
e-mail: aboitcairo@gmail.com
e-mail: gerald.schaefer@ieee.org
AGENDA
Introduction
 Hybrid Intelligent Automatic Image
Annotation Schema

◦
◦
◦
◦
Segmentation phase
Feature extraction phase
Feature optimization phase
Classification and annotation phase
Experimental Results
 Conclusions and Future Work

2
INTRODUCTION

Images retrieval process can be done using image
analysis techniques like content-based image
retrieval (CBIR) that includes low level image
features such as color, shape, and texture to find the
similarity for the purpose of image matches.

Image retrieving can be done using keyword-based
queries to target the images annotated by semantic
words.
3
INTRODUCTION (cont’d)

Retrieving images based on visual features leave a
semantic gap between the actual results and the
user’s expected ones.

Automatic image annotation can bridge the
semantic gap between low level features and high
level semantic based features that are recommended
by users.
4
Hybrid intelligent automatic
image annotation system
5
Hybrid intelligent automatic
image annotation system
1. Segmentation phase

The normalized cuts segmentation algorithm
formulates image segmentation as a graph
partitioning problem and uses the normalized
cuts value between different graph groups.
6
Hybrid intelligent automatic
image annotation system
2. Feature extraction phase
7
Hybrid intelligent automatic
image annotation system
3. Feature optimization phase
GAs coupled with SVMs were used to optimise
the feature set for image annotation.
 For this, we have used real, binary, and bi-coded
chromosomes in our experiments.
 Binary representation is used for feature
selection, real representation for weighting of the
features, while bi-coding is employed using both
representations in each chromosome.

8
Hybrid intelligent automatic
image annotation system
4. Classification/annotation phase

Four different types of SVM kernel functions have
been experimented; namely linear, quadratic,
Gaussian RBF (with s 1 and 16), and polynomial
kernel functions.

The proposed model was tested using GAs on
single SVMs, and using multiclass SVMs.
9
EXPERIMENTAL RESULTS
Results for different SVM kernel functions
10
EXPERIMENTAL RESULTS

Images retrieval process The proposed approach
was evaluated using six classes of fruits images
which are apple, banana, grapes, mango, orange
and watermelon.

Each class has 10 training images and 5 testing
images.

Each image is segmented into five segments using
normalized cuts algorithm.
11
EXPERIMENTAL RESULTS
Classes segments distribution
12
EXPERIMENTAL RESULTS
Normalized cuts segmentation results
13
EXPERIMENTAL RESULTS
Classification and annotation results (1)
E: Expected label output
A: Actual label output
BG: Background label
14
EXPERIMENTAL RESULTS
Classification and annotation results (2)
E: Expected label output
A: Actual label output
BG: Background label
15
EXPERIMENTAL RESULTS
Classification and annotation results (3)
E: Expected label output
A: Actual label output
BG: Background label
16
CONCLUSIONS

The presented automatic image annotation
approach for region labeling is based on multi
classifiers SVM and genetic-based feature
selection, in conjunction with normalized cutsbased segmentation.

It takes advantage of both context and
semantics present in segmented images.
17
CONCLUSIONS (cont’d)

This hybrid approach refines the output of
multi-class classification for automatically
labelling images regions from difference classes.

Experimental results demonstrated that the
proposed approach provides a good
improvement in terms of classification
performance.
18
FUTURE WORK

Increasing the number of data sets
and also to apply different machine
learning approaches such as neural
networks and rough sets that can be
applied for image retrieval and
annotation.
19