Chris Catton, Simon Sparks and David M. Shotton (e-mail firstname.lastname@zoo.ox.ac.uk)
Image Bioinformatics Laboratory, Department of Zoology, University of Oxford, Oxford OX1 3PS, UK
Queries of image databases rarely result in complete and accurate data retrieval. The reasons for this are well documented and include inadequate metadata, ambiguity of search terms and poorly designed interfaces. These shortcomings in turn result from difficulties in annotating multimedia objects and in designing efficient data storage and retrieval systems for large sparsely populated data sets. Additionally, many image databases fail to distinguish adequately between the content and the meaning of an image, or take into account different interpretations of the same data.
A complete image description requires a large number of entities in a wide variety of relations. While traditional image databases might record that an image has a panda as its subject, it is difficult to record that an image depicts a ‘panda eating bamboo’ or ‘panda exhibiting use of pseudothumb in feeding’ other than as free text. Storing descriptive data as free text forces image databases to rely on natural language processing in order, subsequently, to extract meaning from the descriptions. This results in inaccurate retrieval as a result of the inevitable confusion between images of members of the species Ailuropoda melanoleuca and images of Fiat Panda automobiles.
These difficulties are further compounded by a lack of tools for managing image content and metadata coherently during image production. Without such tools, valuable information is often lost, and once lost it is difficult or impossible to retrieve. While text documents can be analysed by natural language processing or statistical tools and their meaning extracted with some degree of success, tools intended to automatically extract meaning from images have been disappointing.
Ontologies promise to help address many of the difficulties currently experienced in managing large image databases. In addition to domain-specific ontologies, image databases require a generic image ontology describing the structural content and media representations of the images themselves, in order to record and distinguish between several semantic ‘worlds’: the Real World, in which exist the objects and events that have been partially captured by various forms of media (micrographs, videos, etc.), the Media World, in which exist such representations of aspects of the real world, and the Cognitive World, in which exist man’s ideas, interpretations and conceptualisations both about objects and events in the Real World and about the portrayal of these objects and events in the Media World. The formalizations (i.e. ontologies) of the conceptualisations and ideas of all three worlds and of their interconnections exist in a fourth Logical World.
For scientific image databases, the distinction between these worlds is a vital one that the image database ontology and data model must seek to capture.
Specifically, the ontology for a scientific image database must specify the metadata requirements for a wide range of information, namely image acquisition (including details of who took the original micrograph, where, when, under what conditions, for what purpose, etc.), the media object itself (source and derivation, image type, dynamic range, resolution, format, codec, etc.), the denotation of the referent: (e.g. the name, age and condition of the subject), and the connotation of the referent (the image’s interpretation, meaning, purpose or significance, its relevance to its creator and others, and its semantic relationship to other images).
Provision must additionally be made for describing field aspects of the real world that cannot conveniently be attached to any particular object (e.g. variations of illumination intensity or chemoattractant concentration across the field of view of a light microscope image). Furthermore, in imaging sequences of change, for example the development of a live Drospohila embryo, there is a need to preserve the concept of object identity in the face of radical spatio-temporal changes in appearance, and to devise mechanisms for modelling such changes and the appearance of new structures.
There is also a need to be able to include estimates of the confidence with which particular assertions of image interpretation are made, and to build into the metadata structures curation records of the provenance of the data and any revisions to which they have been subjected. All this information, or at least a good part of it, should be structured in such a way that it is searchable.
In addition to these ancillary metadata about the image, none of which are directly derived from analysis of the content of the image itself, there is yet a further area for which an image database must provide. This is a framework in which to record semantic metadata related directly to analysis of the information content of the images or videos themselves. These semantic metadata carry very high information value, since they relate

Abstract for Seventh Annual Bio-Ontologies Meeting, Glasgow, Friday 30th July 2004 2 directly to spatio-temporal features that are of most immediate relevance to human understanding of media content, namely “Where, when and why is what happening to whom?” (Shotton et al ., 2002).
Depending upon the complexity of the images, such semantic content metadata can be derived either by fully automated or by interactive image analysis procedures (see, for example, Machtynger and Shotton, 2001;
Shotton et al
., 2000; Rodríguez et al., 2004). Subsequent queries based upon semantic content metadata extends the query domain from the conventional one of text searching or simple image matching to include direct interrogation of the spatio-temporal attributes of the objects of interest, and of their associated events.
We will describe our ontology-based work to optimise submission and retrieval procedures for the Bioimage
Database (Shotton, 2003). The BioImage Database ( http://www.bioimage.org
), part of the EC-funded
ORIEL Project (Online Research Information Environment for the Life Sciences; http://www.oriel.org
), is designed to provide the scientific community with a freely accessible database of high quality multidimensional digital research images of biological specimens. Over the last two years, we have completely redesigned the prototype BioImage Database (Carazo et al ., 1999) using Semantic Web technologies, with the semantic reasoning layer between the users and the stored metadata. In addition, the metadata storage structure (the data layer) is a novel hybrid of conventional relational tables in which commonly searched metadata are stored and an RDF triple store used for hierarchically structured Entity-
Attribute-Value metadata that would otherwise populate conventional relation tables only sparsely. This hybrid approach has allowed us to retain the speed and power of traditional SQL query engines and relational databases while providing the flexibility of RDF to store Entity-Attribute-Value (EAV) triples. Entity-
Attribute-Value databases have recently found favour among healthcare professionals as a way of recording patient data (Nadkarni, 2002). Like image descriptive data, patient data is sparse – an image represents a very small subset of the objects in the real world, just as a patient will have only a very small subset of all possible diseases and treatments. Whereas in conventional relational database models each description is stored in a specific column, the EAV approach uses row modelling: each description generates a row consisting of an entity (in our case a resource in the ImageStore Ontology), an attribute (a property that the
ImageStore resource is allowed to have, as specified in the ontology) and corresponding value (a reference to resource in the ImageStore ontology or an external ontology, or a literal, depending upon the property used).
The new BioImage Database is standards-based, using open source components throughout and employing the latest Semantic Web technologies. It is based on an ImageStore software stack that has an MVC2
(Model-View-Controller) architecture ( http://java.sun.com/blueprints/ patterns/MVC-detailed.html
) in which the data is organised separately from the presentation layer (Fig. 1).
Figure 1 The structure of the BioImage Database
ImageStore software stack
Apache Web server
VideoWorks Web server
Tomcat
Data
Local image filestore
Browser interfaces
SOAP clients
SOAP interfaces
Controller Struts
Model
Logic layer (servlets)
Query servlet
Administration servlet
Submission servlet
OWLBase
Session servlets
(Java beans)
OBO server
Ontologies
NCBI server
Taxonomies
BioImage metadata -
PostgreSQL
Key HTTP SOAP protocols Internal processes

Abstract for Seventh Annual Bio-Ontologies Meeting, Glasgow, Friday 30th July 2004 3
The View is provided by XSL stylesheets, JSP pages and SiteMesh, while the Struts 1.1 framework
( http://jakarta.apache.org/struts ) provides the Controller. The Model has been encapsulated in a Logic Layer containing the Submission, Query, and Administration Services, and is instantiated by OWLBase, which acts as a mapping layer between the ImageStore Ontology and the BioImage Database itself. OWLBase is our
Java middleware library that provides a layer of abstraction, permitting users and applications to communicate with the underlying data layer by accessing the ImageStore Ontology.
The ImageStore Ontology is central to the functioning of the database, and all interactions with the database occur through it. Written in DAML+OIL, it was constructed using the Jena toolkit
( www.hpl.hp.com/semweb/index.html
) and our own open source Ontology Organiser, a Jena-based Javacoded ontology constraint propagator and datatype manager ( http://www.bioimage.org/software.do
). It has been developed in accordance with the SUMO ontology (Suggested Upper Merged Ontology; http://protege.stanford.edu/ontologies/sumoOntology/sumo_ontology.html
, http://ontology.teknowledge.com
), a development of the IEEE Standard Upper Ontology (SUO) Working
Group ( http://suo.ieee.org
). That well documented and well structured upper level ontology has helped us to structure our own, specialist image database ontology in a coherent way. In particular, the ImageStore
Ontology describes three knowledge domains. It uses a subset of the class model of the Advanced Authoring
Format ( http://sourceforge.net/projects/aaf/ , http://www.aafassociation.org
) to describe media objects, it expands upon the original BioImage data model to describe scientific experiments and their associated metadata, and it uses a subset of MPEG-7 to describe multimedia content. Biological content is described by plugging in external ontologies - currently the GO ontology ( http://www.geneontology.org
) to describe genes and gene products, and the NCBI taxonomy ( http://www.ncbi.nlm.nih.gov/Taxonomy/ ) to identify species.
We have written code to allow clickable NCBI taxonomy search trees to be displayed within the BioImage client browse interface, which will allow BioImage users to find a particular group or species by navigating up or down the taxonomic tree. Since the database structure is entirely determined by the ImageStore
Ontology, we have developed procedures that permit submission interfaces to be dynamically adapted to users’ requirements, and others that permit us to dump the BioImage Database as XML, modify the
ImageStore Ontology as necessary, and then recreate the entire database from our revised ImageStore
Ontology.
References
Carazo, J. M., Stelzer, E., A., E., Fita, I., Henn, C., Machtynger, J., McNeil, P., Shotton, D. M., Chagoyen,
M., Alarcon, P. A., Lindek, S., Fritsch, R., Heymann, B., Kalko, S., Pittet, J._J., Rodriguez_Tome, P. and Boudier, T. (1999). Organizing multidimensional biological microscopic image information: the
BioImage Database. Nucleic Acid Research 27 : 280_283.
Machtynger, J. and Shotton, D. M. (2002). VANQUIS, a system for the interactive semantic content analysis and spatiotemporal query by content of videos. J. Microscopy 205 : 43-52.
Rodriguez, A., Guil, N., Shotton, D. M. and Trelles, O. (2004). Automatic analysis of the content of cell biological videos, and database organization of their metadata descriptors. IEEE Multimedia 6 : 119-
128.
Shotton, D.M. (2003) BioImage: From image database to Grid service. Proc. 2nd e-BioSci / ORIEL
Summer Conference . 2-5 Sept 2003, Villa Monastero, Varenna, Italy. Available online at http://www.bioimage.org/publications.do.
Shotton, D. M., Rodriguez, A., Guil, N. and Trelles, O. (2000). Object tracking and event recognition in biological microscopy videos. Proc. 15th International Conference on Pattern Recognition: ICPR
2000 , Barcelona, Spain. (3-8 September 2000) ) 4 : 226-229.
Shotton, D. M., Rodriguez, A., Guil, N. and Trelles, O. (2002).
A metadata classification schema for semantic content analysis of videos. J. Microscopy 205 : 33-42.
Nadkarni, P. (2002) An introduction to entity-attribute-value design for generic clinical study data management systems. On-line at http://ycmi.med.yale.edu/nadkarni/Introduction%20to%20EAV%20systems.htm.