67
ONTOLOGY DRIVEN APPROACH TO IMAGE UNDERSTANDING1
P. Astrelli2, S. Colantonio2, I. Gurevich3, M. Martinelli2, O. Salvetti2 and Yu. Trusova3
2
Institute of Information Science and Technologies-CNR, Via G. Moruzzi 1, 56124 Pisa, Italy
{Patrizia.Asirelli, Sara.Colantonio, Massimo.Martinelli, Ovidio.Salvetti}@isti.cnr.it
3 Dorodnicyn Computing Center of the Russian Academy of Sciences,
Vavilov str. 40, 119333 Moscow, Russian Federation, {igourevi, ytrusova}@ccas.ru
In this paper, we describe an ontology-driven approach to image understanding. The
main necessary kinds of ontologies are introduced and an algorithm ontology for image
understanding is presented. To illustrate the validity of the approach, a case study on
cell image analysis is discussed, introducing the corresponding domain ontologies.
1. Introduction1
Many applications require the processes
applied to media to be concisely recorded for
re-use, re-evaluation or integration with other
analysis data. Quantifying and integrating
knowledge about algorithms for media is a
challenging problem; this is even more crucial
for the visual outcomes of algorithms.
Algorithms for image analysis are difficult to
manage, understand and apply, particularly,
for non-expert users. For instance, a user, who
is unfamiliar with the specific algorithms but
needs to reduce the noise and improve the
contrast in a radiologic image, would greatly
benefit from a tool able to guide him in the
choice of relevant algorithms to apply to his
particular image example.
Image understanding is a very difficult task
involving
integrating
different
image
processing techniques, pattern recognition
algorithms, and artificial intelligence tools.
Lots of image understanding systems have
been developed for a wide range of tasks [e.g.,
7, 9, 11]. Some attempts to build a general
1
The work was partly supported by the cooperative
grant within agreement between Italian National
Research Council and Russian Academy of Sciences, by
European Project Network of Excellence MUSCLE –
FP6-507752, by the Russian Foundation for Basic
Research (projects nos. 06-01-81009, 06-07-89203, 0707-13545) and by the project no. 2.14 of the Program of
the Presidium of the Russian Academy of Sciences
“Fundamental Problems of Computer Science and
Information Technologies”.
framework IU have also been presented,
involving different methods [6, 12]. Many
ontologies are being developed in various
subject areas of science for great variety of
application tasks, including visual data
interpretation [3, 10, 13].
The solution proposed consists of a
sufficiently detailed and well-constructed
algorithm ontology that describes the
knowledge about available image processing
and analysis tasks and techniques (methods,
algorithms, operators, etc.) and two domainrelated ontologies: a domain ontology and a
domain algorithm ontology. Algorithm
ontologies can be used during the image
understanding process in order to plan the
image processing operations to be done and to
assist in adjusting algorithms parameters
interactively with the user. Furthermore, the
ontological
representations
of
image
processing operations can be mapped onto the
functions from a particular image processing
library for solving application tasks. It will
also help to design user-friendly interfaces
providing a terminology for goal formulation
as well as informal description of algorithms
in natural language. The domain ontology will
instead define the user application domain.
The benefits of this approach include: (i)
modularity through the use of independent
ontologies to ensure usability, flexibility and
extensibility (e.g. including semantics aspects
and/or annotations); (ii) a general framework
on which different users can interchange their
68
applications and set up common knowledge;
(iii) a general framework where reasoning
about algorithms for image understanding can
be performed.
2. Description of the approach
Image Understanding (IU) can be defined as
the process of automatically discovering,
identifying and interpreting the relevant
structures contained in the image, in order to
perform an image-based task, i.e., an activity
that relies on vision.
The primary goal of IU is to endow a
computerised application with the ability of
approximating, in some way, the similar
capability of human beings. To this end, a
complex process is usually needed involving
sophisticated image interpretation, knowledge
representation and reasoning ability. Advanced
IU systems require (i) powerful and rich
representations of image content, in order to
derive relevant and invariant information from
their input; (ii) symbolic and/or sub-symbolic
reasoning techniques, for interpreting the
represented visual content; (iii) some “metalevel” capabilities, for modelling and
reasoning about their goals and the success of
their approaches.
In this paper, we discuss a standard approach
to IU based on the use of different kinds of
ontologies. We consider the process of image
understanding as a complex multi-stage
procedure that requires solving the following
tasks:




Image pre-processing;
Image segmentation;
Feature selection and extraction;
Object recognition.
Solving each of these tasks needs the selection
of the most relevant algorithm and its best
parameters.
Formally, in our problem-solving vision, a task
T represents an IU problem which can be
decomposed into a sequence of sub-tasks.
Hence, by an algorithm A, solving a task T, we
mean one of the possible sub-task
decompositions: A(T) = {T 1, T 2,…, T N}. Each
task (or sub-task) T i is solved by a method Mi,
which can be performed by applying a
sequence of operators {Oi}, i=1,…,k. An
operator can be described in terms of the
attributes it uses, the data it operates on, and
any additional knowledge about how to use it
for satisfying the goal.
In this framework, the following main kinds
of knowledge, required for automated
problem solving in the domain of image
analysis and understanding, can be defined:
(i)
image
understanding
background
knowledge and (ii) user application
knowledge. The latter becomes essential for
handling the specific tasks at hand. The
former, on the other hand, is fundamental for
defining new solutions using the several IU
techniques.
Representing knowledge by means of
ontologies is nowadays considered to be most
promising. There are many reasons for this,
such as: ontology languages allow for
expressing rich semantics and provide
reasoning capabilities; free and Semantic Web
based languages and ontology tools can be
easily found; ontologies make it easier to share
the results and the re-use of knowledge
provided by others.
In the IU problem-solving process, the
following main kinds of ontologies are
involved:
(a) algorithm ontology (AO), which is
functionally divided into two parts:
1. main concepts for describing general
tasks: the general problems in the
domain of IU and information about
their decomposition into sub-tasks;
2. main methods (techniques, approaches,
etc.) for image processing, analysis,
recognition and understanding;
(b) domain ontology (DO), which describes
the domain of application.
The AO represents the fundamental domainindependent knowledge on IU. For solving any
specific task, it should be combined with the
DO, thus obtaining a domain algorithm
ontology (DAO) which describes domain tasks
and algorithms (see Fig. 1).
69
DO1
Mammography
DO2
Cytology
AO
Tasks
PO
DAO1
DAO2
Methods
DON
Cardiography
DAON
IAT contains the most important terms and
their definitions for describing and classifying
image processing techniques in the given
domain and allows us to avoid building the
ontology starting from scratch. To automate
the process of converting the IAT into the
OWL ontology a special tool was developed at
the Institute of Information Science and
Technologies of the Italian National Research
Council (ISTI-CNR) [5].
Figure 1. The involved ontologies
4. Ontology-driven cell image analysis
(Domain ontologies)
Moreover, as an extension, a problem ontology
(PO) could be defined for storing all the
DAOs, obtained for different domains, so that
the developed solutions can be re-used for
solving new domain tasks.
The framework presented has been applied for
supporting the analysis of microscopic cell
images. Image-based cytometry mainly
consists in computing a set of measurements
from images obtained by magnification of
opportunely stained cell specimen. Depending
on the problem at hand, different types of
measurements, i.e. features, can be extracted
from the images in order to reproduce the
diagnostic evaluations done by physicians.
In our framework, the considered tasks
encompassed the analysis of different types of
blood cells for different diagnostic problems
(e.g., anaemia or chronic lymphatic
leukaemia). So far, only the solution of the
analysis tasks was accomplished, leaving to
further developments the solution of the
corresponding diagnosis tasks.
We defined the following ontologies for
microscopic cell image analysis:
 DO for cell biology;
 DO for microscopy domain;
 DO for microscopic cell images.
By combining the DO and AO, we obtained
the DAO for cytological image analysis. In the
DAO, the definitions of two segmentation
methods were added: (i) an algorithm based on
a neuro-fuzzy classification approach for
automatically extracting all the cells [6] and
(ii) an algorithm based on active contours [8].
Moreover, we decided to extend the class
feature, defined in AO, including new
instances [4]. Fig.2 shows a graphical
representation of the main classes of the three
DO and the class feature.
3. Image understanding ontologies:
Algorithm ontology
The formal representation of the semantics of
image processing techniques (algorithms,
methods, approaches) enables recording of
provenance, provides reasoning capabilities,
facilitates
application
and
supports
interoperability of data.
We defined the following five top-level classes
of the algorithm ontology: “Data”, “Task”,
“Method”, “Operator” and “Attribute”.
The description of a task contains the
following information: goal description; input
data; output data; information about the
decomposition of the task into subtasks; a list
of keywords.
The description of a method contains the
following information: the name of a method;
goal (task); input data; output data; the
name(s) of operator(s) to be applied;
information about the usage of the method;
informal natural language description of the
method; a list of keywords.
The description of an operator contains the
following information: the name of an
operator; formal definition; input data; output
data; attributes (parameters); information
about the use of the operator; informal natural
language definition; list of keywords.
As the basis for building the ontology we used
the Image Analysis Thesaurus (IAT) [2]. The
70
enumerated
{positive;
negative}
cell
io
e
string
fu
nc
t
typ
string
highlights
n
applic
ation
isSampledIn
Staining_procedure
consistsOf
isS
date
string
acquisition_procedure
tain
edB
y
date
consistsOf
contains
acquiredBy
specimen
string
ion
ens
dim
tion
examina
string
n
lutio
reso
image
string
application
locus
string
rdf:subClassOf
describes
hasInput
hasOutput
rdf:subClassOf
transformation
type
feature
Image _region
func
tion
appl
icat
ion
enumerated
{positive;
negative}
string
string
rdf:subClassOf
morphological_
feature
rdf:subClassOf
densitometric_
feature
rdf:subClassOf
textural_
feature
rdf:subClassOf
structural_
feature
Figure 2. Graph-based representation of the main ontology
classes
A showcase tool (see Fig. 3) was developed
for testing the ontologies, by integrating a
library of algorithms for segmenting the
microscopic cell images and for extracting the
different types of features. The main
functionalities of the tool include (i) the
formulation of the task, which can regard the
analysis of different types of images; (ii) the
selection of the segmentation algorithm; (iii)
the suggestion of the set of features to be
computed according to the task.
5. Conclusions
An ontology-driven approach to image
understanding has been discussed. The
solution proposed consists of using an
algorithm ontology that describes the
knowledge about available image processing
and analysis tasks and techniques and a
domain ontology, the union of the two
resulting in a domain algorithm ontology. A
case study on ontology-driven cell image
analysis has been presented for showing the
use of domain ontologies.
Figure 3. Screen shot of the testing software
We are planning to revise and refine the
algorithm ontology to extend its applicability
by mean of introducing more precise
information on algorithms. The cell image
analysis case study will be extended to include
classification besides analysis. The resulting
ontologies will be integrated within the
architecture 4M [1] that has already been
developed within the MUSCLE NoE and that
will also provide capabilities for multimedia
knowledge acquisition, feature extraction and
annotation.
References
1. Asirelli, P., Little, S., Martinelli, M., and Salvetti, O.
MultiMedia metadata management: a proposal for
an infrastructure. In Semantic Web Applications and
Perspectives. SWAP 2006. Proceedings online. G.
Tummarello, P. Bouquet, O. Signore (eds.) (Pisa, 1820 December 2006). CEUR, 2006.
2. Beloozerov, V.N., Gurevich, I.B., Gurevich, N.G.,
Murashov, D.M., Trusova, Yu.O. Thesaurus for
Image Analysis: Basic Version. Pattern Recognition
and Image Analysis: Advances in Mathematical
Theory and Applications, 13 (4) (2003), 556-569.
3. Bittner, T. and Winter, S. On Ontology in Image
Analysis In P. Agouris, A. Stefanidis (Eds.): ISD'99,
LNCS 1737, Springer-Verlag, (1999), 168-191.
4. Colantonio, S., Gurevich, I., Martinelli, M., Salvetti,
O., and Trusova, Y. Ontology driven Approach to
Cell Image Analysis. In Proc. of the 7th Open
German-Russian Workshop on Pattern Recognition
and Image Understanding (Ettlingen, Germany,
August 20-23, 2007) (in press).
5. Colantonio, S., Gurevich., I., Martinelli, M., Salvetti,
O., and Trusova, Yu. Thesaurus-based Ontology on
Image Analysis. In Proc. of the Int. Conf. on
Semantics and Digital Media Technologies (Genova,
Italy, December 5-7, 2007) (accepted).
6. Colantonio, S., Gurevich, I., Salvetti, O. Automatic
Fuzzy-Neural based Segmentation of Microscopic
Cell Images. Int. J. of Signal and Imaging Syst. Eng.
Inderscience. 1(2007) (in press).
7. Crevier, D., and Lepage, R. Knowledge-Based
Image Understanding Systems: A Survey. Computer
Vision and Image Understanding, 67, 2 (1997), 161–
185.
8. Gurevich, I., Kharazishvili, D., Murashov, D.,
Salvetti, O., and Vorobjev, I. Technology for
Automated Morphologic Analysis of Cytological
Slides. Methods and Results. In Proc. of the 18th Int.
Conf. on Pattern Recognition (Hong Kong, China,
August 20-24, 2006), The IEEE Inc., 2006, 711-714.
9. Hanson, A.R., and Riseman, E.M. VISIONS: a
computer system for interpreting scenes. Computer
vision systems. Academic, 1978, 303–333.
71
10. Maillot, N., Thonnat, M., Boucher, A. Towards
ontology-based cognitive vision. Mach. Vision and
Appl. Springer-Verlag, 2004. 16 (2004), 33–40.
11. Matsuyama, T., Hwang, V. SIGMA – a knowledgebased aerial image understanding system. Plenum,
New York, 1990.
12. Rincon, M., Bachiller, M., and Mira, J. Knowledge
modeling for the image understanding task as a
design task. Expert Systems with Applications.
Elsevier Ltd. 29 (2005), 207–217.
13. Town, C. Ontological inference for image and video
analysis. Machine Vision and Applications.
Springer-Verlag, 2006. 17, 2 (2006), 94–115.