From: AAAI Technical Report WS-93-05. Compilation copyright © 1993, AAAI (www.aaai.org). All rights reserved.
The Meta-Knowledge Level: A Methodology for Validation
Robert T. Plant
Department of Computer Information
University of Miami
Coral Gables
Florida 33124
Systems
ABSTRACT
The aim of tiffs paper is to showthat whena developmentlife cycle of representation
refinementis utilized, that follows the principles of Newell’sKnowledge-Level,
then the system
will becomeself validating. This is illustrated througha rigorous development
methodologythat
utilizes formal techniquesin the specification of the domainknowledge,the cognitive aspects
and the representation. The paper introduces the concept of the Meta Knowledge-Level,a
variant of Newell’sKnowledge-Level
that facilitates the construction of a meta knowledge
model. This provides the knowledgeengineer with a dynamicperspective of the system which
can be used in conjunctionwiththe static aspects foundin the intermediaterepresentation, an
implementationindependentrepresentation that is created through the use of a knowledge
filter.
1. Introduction
Thefocus of research and developmentin the area of methodologiesfor knowledge-based
systems has
primarily beenuponthe functionality, formalaspects and refinementof tiffs formof softwaresystem[Plant.93]
[Miller.90]. This has beensubsequentlyreflected in the researcharea of validationand verification for knowledgebasedsystems,wherethe primaryfocus has beenuponthe static structures of the systems. In tiffs paper wewill
attempt to moveawayfrom tiffs static perspective towardsan alternative philosophyof systemdesign, one that
relies not only uponthe use of formality and refinementbut one that utilizes a variant of Newell’sKnowledgeLevel [Newell.82]to influence the design, tiffs being the Meta-Knowledge
Level. The paper will present an
overviewof this approachand indicate howtiffs metaknowledge
can be obtainedandused to assist in the design
and validation processes.
2. Background
Thecreation of knowledge-based
systemsand their subsequentverification can be considered from two
aspects: the static and the dynamicaspects of the system. Thedevelopmentmethodsare focused uponthe use
of increasinglyformalaspects of systemdevelopment,
such that the proof obligation for systemscan be ultimately
determined and met. These developmentmechanismsare at or approaching the TRILLIUM
k Level 2 capability
with certain aspects movingtowardsLevel 3 capability [Preece.93]. This movetowardsformal functionality in
systemdesign has, as Bellmannotes, movedthe validation research to movein a parallel direction:
"The bulk of rule-based methodsconcernthe systematic analysis of the static structure and
dynamicbehavior of a rule-base, using analysis based on certain correctness criteria"
[Bellman.91]
Examplesof such static structure validation mechanisms
are those utilized in the EVAtool set [Chang.90]:
87
¯
¯
¯
¯
¯
Structure Checker
Logic Checker
Omission Checker
Control Checker
Rule Refiner
Theprimaryintent of such checkers is to identify inconsistencies and incompletenessstates, whichMorellhas
defined in the followingway:
"A system is inconsistent if it asserts somethingthat is not true of the modeleddomain"
[MoreU.89]
"Asystemis incompleteif it lacks deductivecapability" [Morell.89]
However,the question thus arises as to what are weto verify our knowledge-based
systemsagainst, given the
difficulty of obtaining a complete/total specification for the domain.Oneapproacharoundthis is to create a
partial or compositespecification of the desired systemfunctions that are known.Morellnotes:
"In general there are manydesirable properties that need to be specified that can be used as
incompleteor semi-specifications. This knowledgeabout the knowledgebase is called metaknowledge
and is a vital necessityfor verification. [Morell.89]
and this is reflected by Bellmanwhosuggests that the static and dynamicanalysis of systemsis:
"Based upon ad-hoc models of rule-based inference, or more generally, on ad-hoc metaknowledgeabout the rule-base or the rule-inference process" [Bellman.91]
Anexampleset of meta-knowledge
categories that a knowledgeengineer mayutilize are of the form:
¯
¯
¯
¯
¯
¯
¯
¯
¯
¯
¯
Accuracy
Applicability
Assessment
Consistency
Completeness
Disambiguation
Justification
Life Span
Purpose
Source
Reliability [Morrell.89]
Thesecategories of metaknowledge
can then be utilized in the verification and validation of all aspects of system
development
- specification, elicitation, representation, implementation,and maintenance.Wewill later in the
paper consider examplesof these meta knowledgecategories in different aspects of developmentverification.
Afurther aspect to the utilization of metaknowledge
is the knowledge
engineersability to obtain this
meta knowledge,as Bellmannotes:
"This knowledge
is evenharder to elicit froma domainexpert than ordinary knowledge..,it is
often omitted fromoriginal rule base descriptions and providedlater by independentevaluators"
[Bellman.91]
88
Thus, we can see that the meta knowledgeis often collected (if at all) from manyad-hoc sources in
unorganized mannerand subsequently needs validation and verification techniques of its own, to ensure
correctness. Theremainderof this paper will consider howmeta knowledge
could be elicited and organizedinto
explicit models, a need originally suggested by Bellmanand Walter at the AAAI’88workshop
on validation and
verification [Bellman.88].
3. A Meta-Knowledge
Level KBS Development Methodology
3.1 Introduction
Alan NeweU
showedthe relationship betweenthe notion of levels in terms of ComputerScience. He
describeda level to consist of:
"A mediumthat is to be processed, componentsthat provide primitive processing, laws of
compositionthat permit componentsto be assembledinto systems, and laws of behavior that
determinehowthe system behavior dependson the componentbehavior and the structure of
the system."[Neweil.82]
A level is defined in two ways: i) "Autonomously
without reference to any other level" [NeweU.82]
& ii) "Each level can be reduced to the level below. Eachaspect of a level - Medium,Components,Lawsof
Composition,and Behavior,can be defined in terms of systemsat the next level. [Newell.82]
This thus providesus with a framework
throughwhich
wecan showhowa series of specifications can fit together
in a sequencethat then act as a series of Knowledge-Levels.
3.2.
The Specification
of Knowledge-based Systems
The natural point from whichto developany software system is the creation of a specification. The
specification shouldideally detail everyaspect of the systemin unambiguous
termsthat all interested parties can
consider. Thecreation of such a specification for knowledge-based
systemsis howevera far from easy task for
any but the most trivial of systems. In light of this problem,knowledgeengineers have often been forced to
proceedwith only a minimalspecificationor no specificationat all. This is a less than ideal situation anda source
from which manysubsequent developmental problems emanate. In order to overcomethe problem of weak
specifications in knowledge-basedsystem developmentweadvocate the use of two techniques: Prototyping &
Composite-Specificationsthrough formal methods.
Thefirst of these techniques:Prototyping,is utilized to achieve the creation of a base-line document:
the Initial Specification. FollowingMiller [MiUer,90]this phase utilizes prototyping to create an initial
specification and this phasedoes not end until all parties {customer,developer,user} agree that they finally
understandwhat the systemis intended to do, and in particular howit is supposedto do it, what Miller terms
"The Operational Concept"[Miller,90].
Theprototypeprocessis primarilyintendedto establish the boundariesof the solution space. It is very
important that the prototyping is used only to this end, as it is extremelydetrimental to consider the more
complexdevelopmentissues at this stage e.g., representation, interface etc. As these decisions wouldbe made
on incomplete knowledgeof the domainand environment.
Embedded
within the initial specification developmentprocess is an aspect of Cognitive Engineering
knownas CognitiveTask Analysis [Roth,89],where:"CognitiveTaskAnalysis is used to derive a description of
the cognitive demandsimposedby a task and the sources of goodand poor task performance"[Woods,87].
89
Theaim of cognitive task analysis can then be seen as an attempt by the knowledgeengineer to:
"Definewhatmakesthe domainproblemhard, what errors domainpractitioners typically make
and howan intelligent machinecan be used to reduce or mitigate those errors or performance
bottlenecks" [Roth,89]
Theuse of cognitive engineeringtechniques,is not howeverlimited to the creation of the initial specification.
Wielinga, Breuker and others, have created a developmentmethodologyKADS
[Breuker,87] [Hesketh,90] that
also attempts to modelexpertise, such that it can be utilized in a knowledge-based
softwaredevelopment
project.
Weshall utilize other cognitive engineeringpractices later in the methodologyto
assist in the assessment
of validation and verification, quality assurance, in the selection of a representation as well as developingthe
systeminterfaces.
Thecreation of an initial specification providesthe knowledge
engineerwith the first specification in
the creation of the composite-specification
of the system.Acomposite-specificationbeing
a set of specifications,
each of which focuses upon an aspect of the developmentprocess: domainspecification, representation
specification etc. Thecompositeof whichenablesan approximationof a total specification for the systemto be
made.Figure I illustrates the six areas wherespecifications can be derived in a knowledge-based
system, to
varyingdegrees of formality.
I
Co~ositeSpecification
I
I
I
Problem
Description
Specification
Specification
I
Cognitive
Engineering
Specification
I
Hera
Know[ edge
Model
[
I
I
Control
Architecture
Specification
Figure I: A Composite-Specification
FromFigure I wecan see that there are two distinct types of specification present: The dynamic
specifications and the static specifications. Dynamic
specifications refer to aspects of the systemthat are under
constant changeor for whichthe interaction of the componentsare undetermineddue to their combinatorial
complexity.Static specifications refer to those aspects that do not change,but rather remainconsistent over a
period of time. Thus,there are four static specifications:
The Specification of the DomainKnowledge
Wecan easily modelthis aspect as the knowledgeelicited from the domainexpert(s) and knowledge
source(s) is finite and that though the use of transformational processes this can be specified formally in
languagesuch as "Z"[Spivey,90].Froma specification in a languagesuch as this, weobtain several advantages.
Firstly, the Z notation in whichit is written is clear, concise, unambiguous
and allowsfor both a technical and
non technical readership. Secondly,the use a formal notation has significant maintenancebenefits, such as
allowing knowledgeengineers to keep a correct documentof the domaininformation in an implementation
independentform, allowing the implementationlanguageto vary if necessary.
9O
The Specification of the Representation
Theaim of this specification is to allow the knowledge
engineeran opportunityto consider and identify
those aspects of the knowledgerepresentation languageto be used and specify themin a formal manner.The
selection of a representationis a difficult consideration,that weshall discussfurther in the next section, however
oncea representational formhas beenselected then it is imperativethat this be specified fully in terms of its
denotational semantics and its syntax. For without these, it is extremelydifficult to reason about a domain
description/representationwith any certainty. Includedin this specification is the Specification of the Control
Architectureto be used in the system.
Specification of the CognitiveEngineetingAspects
Thecognitive engineeringaspects of the systemdefinition are those that involve:
¯ Specification of the man-machine
interface
¯ Cognitive Task Analysis
¯ Knowledge-Encoding
¯ Competencemodeling
¯ Performance modeling
The man-machine
interface can be subjected to formal specification techniques, as demonstratedby Sufrin et
al [Sufrin,90],whouse the Z notation to specify an interface, and Jacob whoformally specifies a man-machine
interface [Jacob,83]. The Z notation is again a superior formof specification to the pseudo-code,or natural
languagedescriptions that are usually used.
As mentioned, Cognitive Engineering has an impact upon manyaspects of system development, from
the initial specification,throughacquisition,elicitation, quality assuranceto validationandverification. Weshall
considereach of these aspects later.
In addition to the static specifications, there are those aspects that are dynamicin nature.
Specification of the ProblemDescription.
Aprincipal aspect of the systemsdynamicaspect is the specification of the problemdescription. The
nature of the techniquesfor formally specifying systemsare howeverlimited to static aspects of a systemand
do not facilitate full specifications of the dynamicaspects and thus weare unableto fully define the systemin
its entirety hencethis necessitatedthe utilization of prototypingin the creation of the initial specification in
addition to the utilization of the composite-specification
techniqueandthe designtenets identified earlier.
Meta KnowledgeModel(Specification)
In the developmentof a knowledge-based
systemthe knowledgeengineeris obligated to not only elicit
the static domainspecific knowledge,but to incur the overhead of eliciting the dynamicmeta knowledge
associated with the domainknowledge.This knowledgeis inclined to changeover time and thus there is a need
to utilize a separate specification of this knowledge,this is the role of the metaknowledge
model.This model
should utilize as strong a degree of formality as is possible, again the use of a notation such as Z or VDM
is
advocated[Jones.80] as this will enable the relationships betweenthe static domainknowledgeand the meta
knowledgebe identified and assist in showingtheir correctness, completenessand consistency.
91
I
I nitial
Specification
~-~Prototypes
I
t
I BaseLine Requirements Spec. I
I
KnowLedge
eticitation
]
KnowLedge
FiLter
l E t ici ted KnowLedge
Representation]
r
1
Meta
KnoN
t edge
Model
/
KnowLedge
Acquisition X
I
°o,n
I
Specification
Engineering
Specification
Representation
~-~ Specification
I
I
1
ConcreteRepresentation
I
Code Creation
l
l
I
Static AnaLysis
I
t
Figure II: Design of Knowledge-BasedComponent
Thus,wehaveidentified the needfor specifications in the creation of knowledge-based
systems. Wenow
discuss howthese specifications can be broughttogether throughthe use of a rigorous development
methodology
and howthese specifications can be equated to Knowledge-Levels.The methodologyas a whole can be
introduced by considering Figure II above. These stages will nowbe examinedin greater detail.
As wehave already seen, the initial specification is a documentthat can act as a baseline for the
remainder of the systems development.Each of the resultant phases can be comparedto the objectives and
systemspecifications laid downin this document.
92
3.3 The Knowledge Elicitation
Process
Theuniquenature of knowledge-based
systemsis that they utilize domainspecific informationthat is
"expert"in nature. This has several implications. Theinformationmayin itself be unique, scarce or uncommon,
howeverit is the waythat the expert employeesthat informationthat makesthe informationvaluable. Thus, one
of the most importanttasks befalling the knowledgeengineer is to ensure that he elicits as muchstructural,
control and relational knowledge
fromthe expert source as possible. This thus formsthe basis of the knowledge
elicitation task and the knowledge-based
systemdevelopment
process itself. As later in the processthe knowledge
engineerwill haveto consider specifyingthe static domainknowledge
{facts, rules, heuristics etc.,} and select
a representationin whichto manipulatethis knowledge,whichentails considerationof suchfactors as structural,
control and hierarchical knowledge
types. Thusthe knowledge
engineerstask in knowledge
elicitation can be seen
as falling into twocategories:¯ Elicitation of static knowledge
¯ Elicitation of dynamicknowledge
The knowledgeengineer has several different approaches to the knowledgeelicitation
[Welbank,83],for example:
process, [Roth,89]
¯ Verbaltransfer of knowledgee.g., Interviewing- structured, focused and unstructured.
¯ Reportingtechniques: e.g., On-line, Off-line and Hybrid.
¯ Psychologicaltechniques:e.g., Repertorygrid, critical incident, Inference structure, Goal
decomposition& Distinguishing evidence.
¯ Knowledge
engineer investigates literature.
The choice of elicitation technique will dependheavily upon the domainunder consideration, the type of
knowledgeto be extracted and the point the elicitation has reached. For example,the elicitation maycommence
with the knowledgeengineer performinga series of unstructured interviews to extract high level conceptual
knowledge.This maythen be followedby structured interviewswherethe relationship of the domain,its structure
and moredetailed informationare obtained. This maythen be followedby a series of focusedinterviews to fill
in the low level informationof a fine grain size. Several frameworks
for the analysis of these techniqueshave
been proposed[Dhalival,90] [Burton,87], including Cognitive Mappingand KnowledgeEncoding,two aspects of
Woods’scognitive engineering paradigm[Woods,88][Roth,89].
Theresultant of the elicitation process, dependinguponthe techniqueemployed,will be, whatwehave
termedthe elicited representation.This will, for examplebe a transcript in the case of an interviewor an on-line
report. Theaim of this stage in the life cycle is to providea permanentrecord of the knowledge,
in the formin
whichit wasextracted. This will enablethe knowledge
engineerto followa knowledgetrail later in the process
if necessary(e.g., maintenance
phase).
Theprocess of eliciting the different knowledgetypes, perhapsfromdifferent sources, with differing
Knowledge-Levels,
usingdifferent techniquesat different periods of time meansthat there will be a set of elicited
representations whichtogether form a historical database of elicited knowledge.
3.3.1 The Knowledge Filter
In this section wewill attemptto illustrate howthe conceptof the specificationsas levels is realized in
practice. In order to do this weintroduce the softwaredevelopment
process weterm the Knowledge
Filter, which
is itself a series of subprocesses,eachof whichtake the elicited representationandprocessit in order to distil
a resultant output that reflects the sub-processfunction. For example,weutilize the sub-processof conversational
coherenceto obtain an understandingof the alignmentwithin the elicited representation [Ragan.83].
93
Theaim of these sub-processesis to act as a series of knowledge
filters, eachof whichenablea different
perspective of the elicited representation to be obtained. Thesecan then be utilized in the subsequentsystem
development.
Theknowledge
filter processcan be used to illustrate the notion of specifications as levels. Aswehave
noted, Newellidentified four componentsthat define a Knowledge-Level,
whichcan be summarizedas follows:
¯
¯
¯
¯
A medium
Components
Lawsof composition
Lawsof behavior
to be processed
that provide primitive processing
that permit componentsto be assembledinto systems
that determine howthe system behavior dependsupon a componentbehavior
and the structure of the system
Plus the two constraints:
¯ The system can be defined autonomouslywithoutreference to any other level
¯ Each level can be reduced to the level below. Each aspect of a level - Medium,Components,Laws
of Composition,and Behavior,can be defined in terms of systemsat the next level.
Thus,in relation to the knowledge
filter describedabove,wecan see that the elicited representationacts as the
mediumto he processed. The sub processes of the knowledgefilter act as the componentsthat provide the
primitive processing.Theanalysis of the informationresulting fromthe primitive processingresults in the meta
knowledgemodeland the intermediate representation, both of whichcan be defined formally, these act as the
laws of compositionwhichpermit the componentsto be assembledinto systems. The meta knowledgemodeland
the intermediaterepresentation are also boundthroughformaldescriptions of their behaviore.g., what can and
can not be addedto them, and thus these form Neweli’slaws of behavior. Further each of the representations
can be considered in isolation or rigorously transformedinto the next representation, thus obeyingNewell’s
constraints.
Wecan see therefore, that these form one tier of a methodology
whichwhenaddedto the other tiers
provides a description of a complete Knowledge-Level
developmentenvironment.
3.3.2
The Meta Knowledge Model
Aswehavediscussedin the previoussection, the knowledge
filter acts to isolate the different aspects
of informationcontainedwithin the elicited representation. A resultant of this is the MetaKnowledge
Model.
Theconceptof whichis illustrated in FigureIII:
I
I
Knowledge Elicitation
]
I Elicited
Representation
I
Knowledge
Filter
Anatysia
I Pr°t°c°t
I
Coherenc~
I Con,eraational
I
I
l
L°gic I
IEr°tet’c
ConceptualAoo,,~i~
I
Heta
Knowt~ge
Quest | On/Answer TheOry
l TaskAnaly’i"
I
l
Figure III. Meta KnowledgeFilter and FeedbackLoop
This showshowthe meta knowledgeidentified during the knowledgefiltrationprocess is representedin the Meta
Knowledge
Modelin a formal manner.This knowledgeis then fed back into the elicitation process to enable
the knowledgeengineer to obtain the granularity and scope of knowledgerequired in conjunctionwith further
meta knowledge.Oncethe elicitation process has reached a steady state the meta knowledgemodelis used in
conjunction with the intermediate representation to feed into the next level, wherebya moreformal domain
specification, cognitive engineeringspecification and representationspecification are constructed. Theadvantage
of using the meta knowledgemodelis that the knowledgeengineer is no longer only creating a system based
uponstatic domaininformation but information from the Meta-Knowledge
Level.
3.3.3. TheIntermediateRepresentation
The second output of the knowledgefiltration process is the production of the intermediate
representation. Theprimaryfunction of whichis to provide a mechanism
that is rigorous enoughto allow several
demanding
analyses to take place uponit. Oneof these ultimately producesa formal specification of the domain
knowledge
and another acts as the basis for the selection of the high level "classical"representation such as a
production system, which ultimately will be used to represent the domainknowledgeheld in the formal
specification.
As stated the aim of this phase is to producefromthe elicited knowledgea morerigorous, intermediate
representation [Scott,91]. This representation will be structured in form, syntax and semantics, such that the
knowledgeacquisition necessary to transform the elicited representation will identify the inconsistencies,
incompletenessand any incorrectness in the elicited representation. The knowledgeengineer will use this
intermediaterepresentationto drawtogether the knowledge
fromthe varyingelicited forms:transcripts, repertory
grids, questionnaires etc. Intermediate representations are of the form: decision tables, AND/OR
graphs,
decision trees ; each of whichencouragecompleteness,correctness & consistency; allow for refinement and
reduction while havingclean yet concise structures.
3.4. DomainSpecifications
Thecreation of the dynamicmeta knowledgemodeland the static intermediate representation allows
the knowledgeengineer to have a dual perspective in the remainderof the systemdevelopment.The aim of the
intermediate representation is to provide a morerigorous formthan the elicited representation with whichto
reason about the domain, while the meta knowledgerepresentation contributes by allowing the knowledge
engineer to understandand be moresensitive to the dynamicaspects of the systemsdevelopmente.g., the use
of recta knowledgeallows us to enhanceour understandingof the systemsexplanationcapability, as defined in
our cognitiveengineeringspecification.
The meta knowledgemodel and the intermediate representation are used in the creation of three
specifications at the next Knowledge-Level:
the domainspecification, the cognitive engineeringspecification and
the representationspecification. Thefirst of these specifications, the domainspecification is intendedto provide
a specification that focuses exclusively uponthe domainknowledge,the static knowledgeof rules, facts, and
heuristics. Theaim of this specification is to allow a knowledge-based
systemto have a repository fromwhich
the domaincan be consideredin isolation. This has several advantages,for examplein the course of maintenance
or subsequentsystem updates the domainspecification will be the unique location for the domainknowledge
to be added, deleted or modified.Theknowledgeengineerwill be able to maintainthe correctness, completeness
and consistency of the system as far as possible. These changescan then be traced throughoutthe remainder
of the developmentprocess. The formalized procedures for updating the domainspecification can also be
specified for addedrigor.
The domainspecification therefore will have to have mathematicsas its basis and this lead to the
adoption of the "Z"notation, a formal specification language. Specifications in "Z"consist of formal text and
natural languagetext. Theformerprovidesa precise specification whilethe latter is usedto introduceandexplain
the formal parts. Specificationsare developedvia small pieces of mathematics
that are built up using the schema
languageto allow specifications to be structured. This leads to formalspecifications that are morereadablethan
a specification presented in mathematicsalone.
It is also advantageoustohavea domainspecification from the perspective of knowledgeengineer-userdomainexpert communication,
as the specification can act as a medium
for communication.Thus,
it can be seen
that the use of a formal languagein the developmentof a knowledgebase is very advantageous.
3.5. The Cognitive Engineering Specification
The cognitive engineering specification, as we have already noted, is composedof manyaspects,
including: cognitive task analysis, knowledge-encoding,
competenceand performancemodelling. The combined
effect of utilizing these cognitive componentsis very powerful, and can be considered as a chief factor in
maintainingthe semanticcorrectness of the systemas a whole, filling the gaps in the decision makingprocess.
This can be seen as aspects of differing phases feed into each other. For example,the knowledgecontainedin
the domainspecification can be consideredin light of the knowledge-encodingtechniques
and this has an impact
uponthe choice of representation in the representation specification, whichin turn will determinethe systems
ability to manipulatedomainknowledge.
The cognitive engineeringspecification also providestwo models:
¯ The CompetenceModel:that provides a modelof the required competenceexpected from the model
in the domain.[Roth,89]
¯ ThePerformanceModel:that describes the knowledgeand strategies that characterize goodand poor
performancein the domain.[Roth,89]
Theadoptionof the cognitiveengineeringspecification in these tworoles can then act as the basis of a quality
assurance mechanism,for competenceand performance.
96
3.6. The Representation Specification
The next step in the developmentmethodologyis to identify which(if any) classical or hybrid
representation is the most suitable form, around whichto base the representation specification, wherethe
~, "productionsystems’, "semanticnetworks
~ etc. In order to find the most
classical representations are "frames
suitable form, several influencingfactors haveto be taken into account:
¯ Information obtained from performingknowledgeacquisition uponthe intermediate representation.
¯ Informationpertaining to representation selection that can be obtained fromconsidering the
compositionof the domainspecification.
¯ Informationresulting from the cognitive engineeringprocesses.
Eachof these informationsources provide valuable insights on whichrepresentationwouldprovide the best basis
for the domainunderconsideration. Theanalysis of the intermediate representationwiUallow a coarse analysis
of the underlyingdomainstructure to he obtained. This is refined by consideringthe compositionof the domain
specification in termsof its knowledge
anddata types, their inter-relationships, and structures. This is then
enhancedby the cognitive mappingdrawnfrom the cognitive engineering processes.
In order for a suitable match, the characteristics lookedfor in the intermediate representation, the
domainspecification, and the cognitive modelshaveto be re-engineeredin the examinationof the representation
schemes themselves (rules, frames, etc). Oncethis is achieved the match can then be made. The chosen
representation is formally specified in terms of its semantics and syntax, this forms the representation
specification[Craig,91].
3.7. The Concrete Specification
Havingcreated the domain,cognitive and representation specifications, weare nowat a point at which
these specifications can be combinedinto a form that will allow us to movetowardsimplementation.This stage
is knownas the concretespecification.
Thecreation of the concretespecification is in stages, firstly the domainknowledge
is transformedfrom
its "Z" specification into the form advocated by the representation specification, and secondly a formal
specification of the control architecture that is associatedwith the representationis created.
It shouldbe noted that this is not the implementation
as the representation is a hybridbetweena high
level version of whatis to be implemented
and a formal specification in the style suggestedby the syntax and
semanticsof the representation specification (e.g., pseudocode). Theaim being to producean implementation
independentrepresentation of the system. This will allow the knowledgeengineer to have a simplified version
(minusthe complexsyntax) with whichto reason about the implementationlater in the systemslife cycle e.g.,
maintenance.
3.8. Coding
Havingcreated the concrete specification this is then used as the basis of the systemimplementation.
The interface issues being resolved by referral to the cognitive engineering and man-machineinterface
specifications.
Theimplementationof the systemshould be the moststraight forwardof all the stages, due to the high
degree of structuring and refinement that has been performeduponthe system in the previous phases.
The mechanism
throughwhichthe system is implementedis left open to the knowledgeengineer as this
is considereda trivial exercise oncethe specifications havebeendeveloped.
97
3.9. Validation and Verification
The methodologywe
have presented does not have an explicit validation and verification stage, as the
approachhas aimedat presenting a philosophyof total quality development.Wecan quali~y this by equating
verification to the Knowledge-Level.
Verification has been defined by IEEE(no. 729-1983)as:
"The process of determining whether or not the products of a given phase of software
developmentmeet all the requirementsestablished during the previous phase."
This wecan equate to Neweil’ssecondKnowledge-Level
constraint (cited earlier in the paper), whereby each
Knowledge-Level
can be shownto be reducedto the level below. Thus, by following a refinement process such
as wehaveadvocated,the verification of the systemwill naturally follow.
3.10. Testing and Integration
Havingcompletedthe developmentof the knowledge-basedcomponentthe developer can nowconsider
the integration of the systeminto any other system. Anaim of this methodologyis to minimizethe amountof
overheadinvolved in the process of embeddingthe knowledge-based
component.This is the reason for the use
of system wide developmentstandards, historical databases, metrics, formal methodsand an adherence to
integration throughoutthe system/processlife-cycle.
4. Institutionalization
Anaspect of developmentthat wehave not yet addressedis that of institutionalization.This has been
identified by Liebowitz[Liebowitz,91]and others as being the critical factor affecting systemacceptance,usage,
and ultimate success. Theprocess of institutionalization can be broken downinto three fundamentalaspects:
implementation,transitioning and maintenance,all three of whichhave historically been weakwhenconsidered
with respect to the case of expert system development. Liebowitz identifies four areas vital to the
institutionalizationprocess:i) Anawarenessof expert systemsfor managers,ii) Usertraining strategies, iii) User
support service strategies, iv) maintenance.All of these areas incorporatewhatBadiru[Badiru,88]terms Triple
C - Communication,
Co-Operation& Co-Ordination,importantmanagement
aspects to the creation of an expert
system. Thus, wesee the process of institutionalization as the connecting link betweenthe management
and
technical perspectives of systemdevelopment[Plant.93b].
The institutionalization of system developmentcan be considered as the holistic approach to
development,whereall levels of personnel, from users to managersare involved in the developmentprocess,
what Leonard-Bartonterms "integrative innovation" [Leonard-Barton,87]. The model wehave proposed here
attempts to overcomethese problems of institutionalization through the participation of management
and
instilling an awarenessof the technologyinvolved to them. Further, the modelaims to actively involve the
user/client in all aspectsof the development.
This is vital to the institutionalizationprocessin that the technology
transfer is greatly eased. This is apparent in manyways; the user becomesmoreunderstandingofthe technology
involved,the developeris relieved of the total obligation for systemcorrectnessas this is nowsharedwith other
membersof the developmentteam, and the probability of system success is increased if there is a continual
involvementand thus continual feedbackfrom all membersand levels of the organization. Liebowitzhas also
identified other influencingfactors that effect the institutionalizationprocess,includingthe following:
¯ System Migration
¯ Standards
¯ Configuration Management
¯ Testing
¯ User Support Services
¯ Maintenance
¯ User Training
98
Wewill nowbriefly touchuponsomeof these areas as they relate to our model.However,for a fuller treatment
the reader is referred to Liebowitz[Liebowitz,91].
The ability for the developers to perform system migrations is greatly eased by having a set of
specifications fromwhichto work.This facilitates the changesin platformthat mayoccur over the systemslife
cycle. Thesespecifications and the adoption of a rigorous development
strategy also allow for close adherence
to standards and subsequentchangesin those standards.
It has been our aim to maximizethe systemscorrectness, howevera by-productof the representation
refinement approachis to allow for, and support maintenance.Bypartitioning the specifications into their
functional areas, the knowledge
engineercan integrate any maintenance
needs into the existing specifications in
sucha wayas to assess the impactthis will haveuponthe existing specification - thus maintainingintegrity and
correctness. As stated by Liebowitz"maintenanceis
a key issue in institutionalizingexpert systems"[Liebowitz,91],
and hence wehave placed a heavyemphasisin designing our methodologyto support this function.
5. Summary & Conclusions
This paper has attemptedto illustrate several points. Firstly, that whena developmentlifecycle of
representation refinementis utilized, that follows the principles of Newell’sKnowledge-Level,
then the system
will becomeself validating.
The second issue addressed was the use of a meta Knowledge-Level
that allowed for the construction
of a meta knowledgemodel. This model allows for a dynamicperspective of the system to be obtained in
conjunctionwith the static aspects foundin the intermediaterepresentation. Theuse of the knowledge
filter to
produce a better meta knowledgemodeland intermediate representation was introduced along with the meta
knowledgefeedbackloop to induce the elicitation of further meta knowledge.
Thepaper therefore is intendedto showthat the ability to validate knowledge
basedsystemsis not one
that can he seen only fromthe static testing of the knowledgebase but mustemanatefrom a holistic KnowledgeLevel developmentmethod.
In conclusion the paper has showna methodology
for the creation of knowledge-based
systemsthat has
attempted to utilize the rigor of software engineeringand cognitive engineering towardsthe obtainmentof a
better understandingofthe knowledgeengineeringprocess. The use of formalized techniquesin conjunctionwith
a rigorous Knowledge-Level
development
style will enable better degrees of software quality to he obtained. The
future of knowledgebased software developmentmethodologieslies in the utilization of formal methodsin
conjunctionwith software metrics. The use of metrics is vital for us to obtain a better understandingof our
methodologies,transformationprocessesand developmenttechniques. Throughthe use of metrics wewill be able
to better establish the correct or most appropriate representation to use for a given domainsegment,or assist
the knowledgeengineer to better understandthe interface issues. Thus, the next stage in the research is to
developa modelthat utilized these techniques to better understandthe Knowledge-Level
developmentprocess.
References
Badiru, A.B., (1988), Successful Initiation of Expert SystemsProjects", IEEETransactions on Engineering
Management,Vol. 35, No. 3, IEEE, August1988.
Bellman, K., & Landauer, C. (1991), Testing and Evaluating Knowledge-basedSystems, AAAIworkshop
Validation and Verification, August1991, BostonMA.
Breuker, J. & Wielinga, B., (1987), Use of Modelsin the Interpretation of Verbal Data. In, Knowledge
99
Acquisition for Expert Systems: A Practical Handbook,Kidd, A.L (Ed), NewYork Plenum
Burton, A.M., Shadbolt, N.R., Hedgecock, A.P., & Rugg, G. (1987), A Formal Evaluation of Knowledge
Eiicitation techniquesfor ExpertSystems:domain1., In, Researchand Development
in Expert SystemsIV, Editor:
D.S. Moralee., BCSSeries, CambridgeUniversity Press. Cambridge,England.
Craig, I.D., (1991),FormalSpecification of Advanced,41Architectures, Chichester, England,Ellis Horwood.
Culbert, C, (1990), Verification and Validation of Knowledge-Based
Systems,Special Issue: Expert Systemswith
Applications, Vol. 1., No. 3. PermagonPress, NewYork.
Dhaliwal, J.S, & Benbasat, I. (1990), A Framework
for the ComparativeEvaluation of Knowledge
Acquisition
Tools and Techniques. Knowledge
Acquisition, 2, 145-166.
Hesketh, P., Barett, T. (1990). AnIntroduction to the KADS
methodology.Esprit Project P1098,Deliverable
M1,EECESPRITProject, Brussels, Belgium.
Jacob, RJ.K. (1983), Using Formal Specifications in the Design of a Human-ComputerInterface.
Communications
of the ACM,
April 1983, Vol. 26, No. 4.
Jones, C. (1980), SoftwareDevelopmenta Rigorous Approach,Prentice Hall.
Leomard-Barton, D. (1987), The Case for Integrative Innovation: An Expert System at Digital, Sloan
ManagementReview, MIT, Cambridge, MA.
Liebowitz,J. ( 1991),InstitutionalizingExpertSystems:A Handbook
for Managers,Englewood
Cliffs, Prentice Hall
Liebowitz, J. (1986), Useful Approachfor Evaluating Expert Systems,Journal of Expert Systems. 3(2):86-96.
Miller, L. (1990), A Realistic Industrial Strength Life Cycle Modelfor Knowledge-Based
SystemDevelopment
and Testing. AAAIWorkshop
Notes: Validation and Verification, August31st, Boston Mass.
Moreil, LJ., (1989), Use of Metaknowledge
in the Verification of Knowledge-Based
Systems, NASA
Contractor
Report 181821,Langleyresearch Center, Virginia.
Newell, A. (1982), TheKnowledge-Level,
Artificial Intelligence 18, pp 87-127,North Holland.
O’Leary, D.E. (1987), Validation of Expert Systems- with Applications to Auditing and AccountingExpert
Systems.DecisionSciences, Vol 18. pp 468-486.
O’Leary,D.E. (1993), Collected Papers1989-92:AAAI
Workahops
on Validation &Ve~fication. (In Preparation)
Plant, R.T. (1993), A Rigorous DevelopmentMethodologyfor Knowledge-Based
Systems. Communications
the ACM(To Appear)
Preece, A.D., Grogono, P., & Shinghal, R. (1993), Assessing the Capability of Knowledge-BasedSystem
Developers, IEEECAIAWorkshopon Validation and Verification, Orlando, FI, March2nd 1993.
Ragan, S.L. Aligment and Conversational Coherence, In, Conversational Coherence: Form, Structure, and
Strategy. Edited by Robert T. Craig & KarenTracy, Sage.
I00
Roth, E.M, & Woods,D.D., (1989), Cognitive Task Analysis: An Approachto KnowledgeAcquisition for
Intelligent SystemDesign., In: Topics in F~ert System Design, (3. Guida& C. Tasso (Editors), pp 233-264,
Elsevier SciencePublishers.
Royce, W.W.(1970), Managingthe developmentof large software systems. Proc. WESTCON,
Ca. USA.
Rushby, J. (1988), Quality Measuresand Assurancefor AI Software. NASA
Contractor Report 4187.
Scott, C. (1991), A Practical Guideto Knowledge
Acquisition. ReadingMA,AddisonWesley.
Spivey, J.M. (1990), TheZ Notation. Prentice Hall
Sufi’in, BA.&He, J. (1990), Specification, Analysisand Refinementof Interactive Processes. In, FormalMethods
in Human-Computer
Interaction. Editors, Harrison, M., & Thimbleby,H. pp 153-200, CambridgeUniversity
Press.
Welbank,M. (1983), A Reviewof KnowledgeAcquisition Techniques for Expert Systems. British Telecom
Research Labs. MartleshamConsultancyServices.
Woods,D.D., & Hollnagel, E. (1987), MappingCognitive Demandsin ComplexProblem Solving Worlds.
International Journal of Man-Machine
Studies, 26, 257-275.
Woods,D.D, & Roth, E.M., (1988), Cognitive SystemsEngineering. In M.Helander(Ed.), Handbook
of HumanComputerInteraction. NewYork: North Holland.
i01