PROGRESS IN KNOWLEDGE ENGINEERING FOR IMAGE INTERPRETATION AND
CLASS I F I CPI''/'' I Ot···.I"
N.J. Mulder, H.Middelkoop, J.Miltenburgu
I ..
·r . c.
PuO.Box 6, Enschede.
Thf:::O t···~E~t:.her I i:'-l.nds
OJ
U II It
(.~BSTRACT
The common grounds between pattern recognition, geo information
s·ys;·tems and (~;< per-t systf?mS
.. The e;.~ pert systE·~m is
Ii
II
de-mystified~
useful techniques in knowledge engineering are
identified. The effects of the degeneration of the method of
statistical pattern rscognition into a rulebase are studied.
ThE~ dev(~l opment of an IIf2~·{pE~rt tutor
fol"- the i nterpretati on of
II
soil classes from SPOT images is reported.
Implementations are
based on a home developed PROLOG based shell with binary logic
and a commercial shell (ENVISAGE) providing limited reasoning
with uncertainties.
INTRODUCTION
Expert systems and artificial
intelligence are popular /
fashionable subjects probably because of mans fascination with
building things after the likeness of his own image. From this
driving force in the computing community a number of
software
packages have resulted, mostly in the field of man
machine
interfacing. If we first remove the mystic layers covering the
lines of code then we discover techniques for knowledge
engineering which are based on finding and recording traces in
a space (matri:;) which relates the instantiation of a problem
to the instantiation of a solution.
At ITC we are mostly interestEd in the engineering of
knowledge used in classification and interpretation. When
expert systems are refered to in the rest of the paper those
will be assumed to be of the diagnosis type.
Work on the use of knowledge engineering in the fields of
image interpretation and mapping at I.T.C. was initiated in
1980, resulting in publications in 1982, (1984) on context
dependent classification using terrain knowledge in a selective
111 ... 395
f.,,,c:t.y ..
A first meta rule seemed to be
rules
~
use
data
selectively
defi~e
a hierachy of decision
•
In 1985 v.d.Pluym and Mulder (not published) investigated
the relation between rules in expert systems and rules in
pattern recognition. This investigation concentrated on the
role of certainty factors. It was soon apparent that the
certainty factors used in medical expert systems were not
suitable because fuzzy logic does not optimise anything. The
Bayes decision rule on the other hand, when combined with a
cost! benefit matrix maximises the <economical, social,
political) benefit of the application of the classification
r-ule.
So the second meta rule appears to be :
base -c€:rtainty factors
the
on
like.lyhood estimation.1f
bound to
bene·fit 01· proper classification and cost of NroTlg classifica.tion
•
The objection against statistical certainty factors in
med i cal 8iq:iert ~;;ystems i is
there are not enough samp I es
available to define the multi variate statistical
d i f::;tr- i but ions
In remote sensing and mapping this is not a problem and
moreover an intuitive estimation of statistical relations is
more apt to provide a good parameter than the ill defined
measures of fuzzy logic.h
II
if
In order to make some of the knowledge engineering techniques
clear to staoff and students se··veral versions of an lIe}·~pert"
system have been developed at ITe by the authors based on micro
PROLOG and ENVISAGE.
DE-MYSTIFICATION OF THE EXPERT or CAN KNOWLEDGE BE COMPILED ?
As the decision rules map the observation vectors in a unique
way into class likelyhood vectors, and as the maximum
likelyhood class,
Pm~M(class:X)
is choosen ; the relation
between., all possible inputs (X) and outputs (class), is fixed
and could be compiled in one large decision (truth) table.
An important part of the de-mystification of expert systems
and artificial intelligence is the realisation that any expert
system could be replaced by a large enough library with a
sophisticated indexing system. This is demonstrated in one of
111 ... 396
the PROLOG based versions. (In pattern recognition use is made
of classification look-up -tables ).
CASE STUDY
SOIL MAPPING FROM SATELLITE IMAGERY.
Chemical and textural characteristics of a soil profile,
which are used in soil classification systems, such as the US
Soil Taxonomy (SOIL SURVEY STAFF, 1975) , cannot be observed from
space. Therefore it is nescessary to define the relation
between soil class and landscape features, e.g. types of
vegetation, altitude, shape of natural landsegments.
After a first inventarisation of the world soil types and
possible relevant landscape features it turned out that this
relation strongly depends on local (agri) cultural
interferences in the past and present. This is contrary to the
more definition based, rules for the ordinary classification of
soil profiles. Therefore ,a large amount of local contextual
knowledg~ has to be stored in the knowledge base.
Knowledge acquisition by interviewing soil experts proved to
be difficult. The present generation experts have not yet been
accustomed to express there knowledge in a way compatible with
rule based knowledge engineering. This factor and the
requirement for local knowledge led the authors (physical
geographers) to select an area in Holland familiar to them. As
the extend of the area was defined by the available SPOT scene
(60km x 60km) a knowledge base could be set up which was not
too large and still of practical use for the case study"
After generalizing the information from soil maps of the
Netherlands (scales 1:50,000 and 1:250,000 (STIBOKA 1979;
1985»
to a scale compatible with the major soil classes of the
region, a relation matrix (Table 1) was constructed relating
ten soil classes to nine observable features separable into 39
attributes. The names of the soil, classes follow Soil Taxonomy
(SOIL SURVEY STAFF, 1975; De Bakker, 1979).
Input into the system is defined by a segmented SPOT image. For
the present the extraction of attributes of image segments
natural "landunits") is performed by visual interpretation.
Attributes can be spectral (image tone) or spatial (parcel
size, parcel shape, .aw).
III
THE I t···jTEh:FP,CE
The user is provided with a SPOT image scale 1~100,000 with
a segment overlay. The segmentation has been prepared by
trained image interpretors and is based on landscape features.
The student is asked to input observed features /
attributes for each landunit. The system guides the student
through a hierachy of observations according to a backward
chaining search strategy. On request the system will show the
log of input and trace the decision making process, and it will
provide explanation text , in reaction to WHY (do you ask this
question) and EXPLAIN (how could I answer the question)
HANDLING UNCERTAINTY
- WHAT CAUSES UNCERTAINTY IN CLASSIFICATION RESULTS ?
1- Uncertainty associated with inexact evidence e.g. caused by
noisy measure ments or the inaccuracy of the human decision
making process. The last one could be quantified by
accumulating the hit/miss ration <confusion matrix) for a
certain interpretor working in his field of expertise.
2- Uncertainty associated with rules. This means that the
assertion
xn> THEN <class Cl>
IF <attributes xl
is not true in all cases.
DISCRIPTION OF THE PROTOTYPES
In this case study different approaches to rule based expert
systems have been tried. Thse have been implemented in models
written in (micro-)PROLOG and ENVISAGE.
BOOLEAN DECISION RULES
According to tabel l~ each soil class has a list of
attributes. Applying logical AND an~ OR functions to match the
given attributes with the list for each class would produce a
classification. The evaluation of a Boolean expression with
111 ... 398
AND/OR terms can be represented in an AND/OR tree and is
equivalent to a box classifier in Pattern Recognition. The
classification schema of figure 2 is equivalent to a set of
production rules like
IF
is
1I1 an d--use is few"est OF~ heath"
no-super"+ i c i al-drai nage
THEN lIela5s i~:; Haplortod
Pd'"~D
"drainage type
II
ll
These Boolean decision rules are eqeivalent to PROLOG HornClauses (DE SARAM, 1985)
«soil Haplorthod)
(land-use forest OR heath)
(drainage no-superficial-drainage»
Boolean rules suggest certainty, because the evaluation of
the logic expressions produce only FALSE or TRUE.
MAXIMUM LIKELYHOOD
Instead of working with TRUE and FALSE in real life one has
to do with likelyhoods" Furthermore decision rules have to
include weight factors for the costs of false decisions and the
benefit of good decisions. The maximum likelyhood (cost
weighted) decision rule fullfils these requirements. It also
allows the accumulation of evidence by using previous
classification results to provide the prior probabilities in
the BAVES rule
P(Cilx) * P(x) ::::: P(x:Ci) * P(Ci)
V'Jr',er'e
X ::::: (xi •• xn) observation tupple, attribute
class i
Ci
list ..
For a given constant P(xICi) and area dependant P(Ci) the
maximum likelyhood rule selects the class Cm with the maximum
I i kel yhood .. Stor' i ng the Ii kel yhood \/€ctor P (Ci
:::::p (Ci : ~.{) in
the database allows the accumulation of evidence when new
components of the observation tupple (new attributes) become
available ..
I
LIKELYHOODS IN A BOOLEAN ENVIRONMENT
III
)
In GIS environments it is usual to preclassify ordinal data
into nominal data by mapping the ordinal data into intervals.
Mapping ordinal data into intervals followed by an AND function
is equivalent to a box classifier decision rule. Overlapping
boxes in feature space are equivalent to OR expressions
connecting AND boxes. In figure 3 a three dimensional feature
space partitioned into boxes is shown. The value of P(x!Ci) is
stored for each soil class Ci. For a given P(Ci) the
likelyhoods PC Ci :x) are also known for each box. Under the
maximum likelyhood decision rule the classification result for
each box is also defined. Consultation of the expert system is
now equivalent to determine the box address from the
observations (xl .. xn) and read the corresponding class.
As the box classifier contains Boolean AND functions it can
also be represented by PROLOG clauses. The difference with the
Boolean approach is that in the learning fase frequences of
co-occurance are used for the definition of the terms in the
Bayes rule and the maximum likelyhood is explicitly stored as
an attribute of each classified segment.
In this study PROLOG was used for the storage of the
likelyhoods in a six dimensional attribute (feature) space
«
(;.~:l"
6 ) , so i I -- c I ass
P (C i : (;.( 1 .. .. H 6»
»
H
;.{
I
where (xl .. x6) is interpreted as a library label pointing to
the content of a cell in six dimensional feature space.
Redundancy in the featurespace is apparent from the fact that
of the 6480 boxes in featurespace only 300 contained
information. PROLOG has the nice property of compressing the
sparsely populated likelyhood lookup-table into a list with 300
ter'm~:; ..
For a different area with different P(Ci) the classification
lookup-table has to be recalculated from the generic relation :
«
L<l ... ~·~6) 'sc)il"-cla.ss' P( (;<1 .... H6) lei) »
and from the new P(Ci) according the the Bayes rule.
FUZZY L..DGIC: '?
For an N-dimensional featurespace vector the rule for
obtaining the certainty of fuzzy-AND is to take the minimum
P(class:xi) over i. For fuzzy-Or the maximum over i is taken.
As the minimum over all projections is less than the average
over a box and the maximum over all projections is more fuzzy
111 ... 400
logic is not supported by statistical reasonining.
Figure 4 illustrates the difference between statistical correct
likelyhoods and fuzzy AND/OR rules for certainty estimation.
LIKELYHOODS IN ENVISAGE
The ENVISAGE shell offers AND/OR trees, Fuzzy logic and a
form of likelyhood calculatioh. The likelyhood calculations are
implicitly based on the assumtion that all observations are
statistically independant. This assumption is rather
unrealistic and in practice the method is statistically weak.
The statistical parameters are not derived from a co-occurance
matrix but the user has to express a measure of belief (odds)
in the answer he provides. After sevral sessions playing with
the intuitive likelyhoods may produce acceptable results but
unexpected results will still occur. Our measure of belief ~n
the systems results remained relative low.
MODE OF INFERENCE
Both the Boolean model and part of the ENVISAGE model use a
mode of inference (reasoning) called backward chaining. This
means that·the inference process is started with the most
likely hypothesis based on prior class probabilities. Th~ most
likely hypothesis will define the first querry to the user or
to a database. The querry process will continue until either
there is sufficient evidence for the hypothesis under
consideration or there is insufficient evidence. In the last
case the reasoning system will redirect its attention to a more
likely hypothesis given the accumulated list of observations so
far. The main advantage of backward chaining is that not all
possible observations (attributes, features) have to be
generated and examined for all classes.
CONCLUSIONS
.I.. •• _
- The stimulus provided by applying knowledge engineering LU
the art of image interpretation was remarkable. It motivates a
better contact between the staff specialised in photo (image)
interpretation and the staff responsible for digital techniques
111 . . 401
in remote sensing. For future knowledge engineering projects it
is desirable that application domain experts learn to program
at the level of expert systems shells.
- Replacing basic tutorial sessions by computer assisted
instruction is cost effective once the building of the
knowledqe bases has become a routine.
- These kind of exercises demonstrate a direct path from
remotely sensed data to a GIS environment enhanced with
certainty factors and explanation facilities. Users of GIS's
have to be trained in thinking and do-ing in terms of
likelyhoods and cost/benefit analysis.
- The knowledge concerning the relations between landscape
elements and soils was less general than had been expected. An
expert system can only be implemenmted for a limited domain. It
is better to combine a number of low level expert systems in a
hierarchical way than try to put a large and too complicated
knowledge base in one large model.
- In any future operational system of knowledge engineering
historical economical and social effects on observable features
have to be taken into account. This means that any rule base
for Earth resource applications will contains a limited body of
general knowledge and a large body of local knowledge in the
form of context maps and context rules"
- Models for e.iN soil genesis have to be included explicitly
in the procedural part of the knowledge base, using e.g. the
digital elevation data as a parameter file.
FUTUHE RESE{-iHCH
- In the near future the critical step of segmenting the image
into landunits will be more automated.
- spatial features which now must be observed by the student
interpretor will gradualy be extracted by pattern recognition
routines. As these routines may require large computing
resources, the knowledge engineering approach allows selective
calling of subroutines based on the current state of the
backward chaining inference process"
- next generation GIS must be developed supporting operational
use of certainty factors (likelyhood vectors) and cost/benefit
analysis applied to the decision making process.
- more features will be stored in a GIS, this will also reduce
111 .. 402
the required input from the user.
-
the interface between knowledge engineering systems KEG and
image processing and pattern recognition must be greatly
:i.
i'fI F' r- C"j \/ F!! cl
F' 1'"- E' S f2 n t
n
El ,::i_i-:: 1-:: E~ r- , H .. cI E-? '!
Netherlands.
t
C) C)
1i
in i
t
E' d
t o ' l tal kin 9 II
11 <:!1_ j Drs 0 i 1 san d
r:5 0 i 1
a
r- e 9 i on s i n
the
Soil Survey Institute, Wageningen.
198;
F'hotogr-ammetr-ia
H ..
a 1"- (-;2
1 979 ,
Mulder,N.J. ,
Sar-l.~_m'l
~::: E:~ ~:::;
l
~o
de,
Classification
and
decision
making,
95-,,6, Elsevier,Arnsterdam.
1985;
Progra_mming
in
micro-PF<OLOG,
Ellis
Not-- t h lI-.,Iood 1 t d "
Soil
Survey
Service~
U.S.
8taff,1975;
80i 1
-ra~{onomy,
80il
Conservation
Dept of Agriculture.
STIBOKA,
1985;
Bodem kaart van Nederland 1:250,000.
STIBOKA, Wageningen.
SYSTEM DESIGNERS,
1985; ENVISAGE Reference Manual,
111 ... 403
Penbr-ooke ..
17017"'111
...
il
~
n
;}
.g ~
...
ttl
I-'
III
()
,Q
I
C
~< ~....
C
.0
C
ttl
,S
(0
'0
rt
8.
>'.!
§ ~
....
()
I
III
()
t
It
::l
..,
g'
I-'
r1"
1"1
0
C
<
It
::l
()
::r
r1"
....
1"1
,
~
rt
0
rt
ttl
ttl
~
t'"
III
r1"
III
,Q
t'"
C
111
()
I
'0
rt
~
+
I
I
I
I
+
+
I
,
+
'0
rt
.Q
It
,
t'"
0
I
0
1"1
I
rt
::r
III
8.
::l
rt
I
~
I-'
0
(II
(II
I-'
III
HI
It
III
t'"
r1"
C
I
It
III
-
tr
c
rt
III
[/J
I»
'1
111
rt
rt
(II
'1
t'"
I
I
+
+
I
+
+
heath
I
,
I
grass
g
arable
III
I
I
I-'
(II
(0
forest
I-'
0.
+
I
I
+
+
+
,
+
+
I
+
+
I
I
I
I
,
+
I
+
I
I
I
I
+
I
I
I
I
I
I
I
I
+
+
+
+
+
I
I
I
+
I
,
+
,
=B
{
+
+
I
I
I
I
I
I
mixed-grass-arable
,
+
I
mixed-forest-arable
I
I
I
mixed-forest-grass
I
+
more-than-5-meters
I
less-than-S-meters
I
I
river
+
I
~
It
III
....0
rt
+
I
I
+
+
+
+
+
larger-than-average
I
I
I
+
average
+
I
I
I
I
I
I
I
+
+
I
I
I
,
canals
+
+
no-superficial-drainag
<
11/
I-'
rt
'1
I»
'!'
+
+
I
I
I
I
+
I
I
I
I
I
I
I
,
I
+
I
I
+
+
I
+
+
+
+
c
+
i
+
0.
~
t'"
::l
III
lit
.Q
III
I
'0
....
I'l
PI
III
e.t
JfI
'"0
1"1
n
smaller-than-average
not-uniform
square
----
- - - I----
.g-
rectangular
Il/
'1
()
III
I
+
I
+
I
I
I
+
+
I
+
I
I
,
elongated
I
polygonal
,
I
I
not-uniform
random
regular-blocks
+
+
+
+
+
+
I
+
I
I
I
,
+
I
+
I
+
+
+
+
+
I
I
.;-
.:-
I
I
+
1
n
irregular-blocks
~0
'0
[/J
1"1
0
t'"
t-'
....
PI
III
t
t'"
I
I
+
I
+
+
+
+
I
I
I
,
,
parallel
I
I
+
circular
I
+
I
+
I
:
I
I
+
I
I
I
+
+
I
I
0
;:!
following-meanders
I
I
I
+
I
I
+
+
+
,
+
I
+
I
+
+
normal
I
+
I
I
I
,
+
+
light
I
I
I
+
+
+
I
I
linear-settlement
+
I
+
i
I
I
+
+
+
other
I
+
I
+
+
+
+
+
no-village
,
+
I
I
+
black
dark
[/J
0
0
.....
t'"
I-'
,
II)
I-'
I
,
"
I
[/J
_.
'1
III
I-'
It
brook
+
I
I
+
,
+
o
+
+
I
A)
+
I
I
"
.......,;
'0
rt
0
ttl
.g
0.
t'"
::l
~
i"
'0
I-'
I
;'
~
"l:
+
I
~
.Q
'tI
I-'
[/J
I
+
c::
.fl::l .fl
....
tr
'1
....
I
+
'tI
....
()
::r
HI
I
I
'0
I
I
&1
~
0
....
tr
....'1
HI
+
'3
j-
::l
til
>'.!
~
;.Q
(II
C
"' "'
til
'"0
1
rt
0
;:!
co
I
....
:aco
<:
t'"
t-'
I-'
III
\Q
ro
I
I
+
+
I
I
I
I
I
+
+
I
I
I
part of-anelliptic-structure
1
JfI
til
t'"
ro
situated-along-a-river
~
I
bordersa-forest-or-villaqe
....
I
borders-a-brook
. 1 ass f'1.l a .\.() .lOJ X1.l1BUI sa1 n q1.l11 B - SSB10 1108
0
rt
I
III
It
HI
ro
;:!
c::
t'.
0
::l
1
31
1
J .,
(J
3ack·....ard :haining :lede of r-easen1.ng for 3 S01.!.-classes ·"ith 3 features
1501.1 :Jdipsamment:!
,\,
"
I
no~a teh
", ... ancuse --------"T--~) r.
Imatch
A ~levatl.onl
~
----------r--
G-anduse!
Imatcn
r./A lelevat1.onl __n_o_-_m_a_t_c_h'---_-!
no-cnatch
1match
A era1.!)age
Isod Plaggeptl
no-match
r;-:-::-::;-:-:
L !landusel __n_o_-_cna
__t_c~h____~
1
A
match
Itone I __....:n~o:..-..:m:.:a:..:t:..:c;:.:h:._._--l
1match
I
no-cnatch
!cnatch
no-match
no-match
L/A ~rainagel~:':::-::'=':"::":'~---l
L/A !dra1.nagel--------....
1
1match
ma tcn
,...--""::"---.
cOnChlS1.0nj
Haplortl',od
lcOnC1USl.On
Udipsammen t
I
lmatch
I
conclusionl
fPlaggept I
I:
can It
help you
A: ask
L: look up
L/A: look up else ask
Fig.
/
/
/ /
/ / / / / /
/ / / / / / / / / /
/7/ / / / / / / / V
p
"a
r
c
e
1
~f
V
V
po
V V
el
V
re
l/
V
(R)
1/ Vh
Vm
sq
Vi
ht
f
a
ub
fh
ga
fa
g
fg
..
ianduse (L)
figure
3
The three-dimensional feature space.
landuse (L)
fa
ub
unk
fg
p
unk
po
all:
re
p
sq
(5)
po
81
(S)
-
ht
fh
ub
Ii
ga
fa
fg
unk
l anduse (Ll
figure
It
Feature spaces in situations where one or more features
n.
are unknown.
The ~.ti34;a&.. h\ocAc.l fa.ke!> peel r It', •• ~,,) 1~~ c.ell D
,
L
( I L..
LL._
...........
D ,.... (Gi J 'l.J')
t'hL 1"...,:~ty
rIA e 't"cl.~~ "tTUii n,..; t'\,.,.CA......
'"l"i'\ ..
0
~ bo-'lte sA, 8 JUt&. C .
0t
1ic..
111 ... 405
I