Intelligent Approaches to
Lessons Learned Processes
Rosina O Weber
University of Wyoming
Navy Center for Applied Research in AI, Naval Research Lab
Intelligent Approaches to
Lessons Learned Processes
Collaborators:
David W. Aha
Hector Munoz Avila
Len Breslow
Nabil Sandhu
R.O.Weber Calgary 3 Aug 2001
Outline
Introduction

Context
Problems with lessons learned systems
Methodology
Monitored Distribution
Case Representation
Lesson Elicitation Tool
Next Steps
R.O.Weber Calgary 3 Aug 2001
Context
Knowledge management context
Lessons learned systems (LLS)
Organizations adopting LLS
Lessons learned definition, representation
and example
Lessons learned process
R.O.Weber Calgary 3 Aug 2001
Knowledge management context
Three types of KM initiatives*
knowledge repositories
 knowledge access and transfer
 knowledge environment

Knowledge repositories
Internet
 industry oriented (alert systems)
 organization oriented (lessons learned systems)

5
R.O.Weber Calgary 3 Aug 2001
Lessons learned systems
Lessons learned
systems are
knowledge
repositories of
knowledge artifacts
KNOWLEDGE
ARTIFACTS
Examples of knowledge artifacts are lessons,
alerts, best practices, reports, video clips, etc.
6
R.O.Weber Calgary 3 Aug 2001
Lessons learned systems
in organizations
Aha & Weber (Eds.)
Intelligent lessons learned Systems.
Papers from the AAAI 2000 Workshop
(Technical Report WS-00-03) AAAI Press
Weber, Aha & Becerra-Fernandez (survey)
Intelligent lessons learned Systems. 2001
International Journal of Expert Systems
Research & Applications, Vol. 20, No. 1., 17-34.
7
R.O.Weber Calgary 3 Aug 2001
government
non-government
Construction Industry Inst.
non-military
military
Honeywell
GM
Hewllet Packard
US
Air Force
Army
int’l
Bechtel Jacobs Company
European Space Agency
Italian (Alenia)
French (CNES)
Japanese (NASDA)
Lockheed Martin E. Sys, Inc
United Nations
DynMcDermott Petroleum Co.
Xerox
IBM
Coast Guard
BestBuy
Joint Forces
Siemens
Marine Corps
Navy
int’l
Canadian Army Lessons Learned Centre
8
US
Department of Energy: SELLS
NASA (Ames, Goddard)
Lessons learned definition…
…or organizational lessons, lessons, lessons identified
Definition:
A lesson learned is a knowledge or understanding gained
by experience. The experience may be positive, as in a
successful test or mission, or negative, as in a mishap or
failure. A lesson must be significant in that it has a real or
assumed impact on operations; valid in that is factually
and technically correct; and applicable in that it identifies a
specific design, process, or decision that reduces or
eliminates the potential for failures and mishaps, or
reinforces a positive result.” (Secchi et al., 1999)
9
R.O.Weber Calgary 3 Aug 2001
Lessons learned example
applicable task
Installing custom stereo speakers.
conditions for applicability
The car is the Porsche Boxster.
lesson suggestion
Make sure you distinguish the wires leading to the
speakers from the wires leading to the side airbag.
Rationale
Somebody has cut the wrong wire because they look
alike and the airbag went off with explosive force. This
means spending several thousand dollars to replace the
airbag in addition to be a potential hazard.
11
From article “Learning from Mistakes” about Best Buy
in Knowledge management magazine, April 2001.
R.O.Weber Calgary 3 Aug 2001
Lessons learned process
12
R.O.Weber Calgary 3 Aug 2001
Lessons distribution sub process
20
R.O.Weber Calgary 3 Aug 2001
Lesson distribution methods
Broadcasting
bulletins, doctrine
Passive
Standalone repository
Active casting
list servers,
information gathering tools
20
R.O.Weber Calgary 3 Aug 2001
Problems with
lesson distribution methods
 Distribution
is
divorced
from
targeted
organizational processes.
 Users may not know or be reminded of the
repository, as they need to access a standalone
tool to search for lessons.
 Users may not be convinced of the potential
utility of lessons.
 Users may not have the time and skills to retrieve
and interpret relevant lessons.
 Users may not be able to apply lessons
successfully.
21
R.O.Weber Calgary 3 Aug 2001
Here is a gap
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
Monitored distribution
Repository of
lessons learned
Organization’s
members
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
Monitored distribution
Lesson repository
is integrated with
targeted processes
…and lessons are
distributed when
and where they
are needed.
22
Organization’s
members
Repository of
lessons learned
Organizational
processes
R.O.Weber Calgary 3 Aug 2001
Problems with
lessons learned systems
Technological
Human
Managerial
13
R.O.Weber Calgary 3 Aug 2001
Technological issues
Standalone distribution outside context of reuse
Lessons disseminated in context of reuse
Low precision and recall in text databases
Case retrieval for lesson dissemination
 cases indexed by applicability
Convert lessons into cases/Collect cases
Requirement: machine recognizable format
Textual representation of lessons
Case strutcture for lessons
14
R.O.Weber Calgary 3 Aug 2001
Human issues
Lesson authors

Lack of training/instructions: content and format
Lesson validators

Hard to validate textual descriptions
Lesson (re)users




15
Have
Have
Have
Have
to access the repository in another context
to accept the potential benefit of lessons
the skills to search for lessons
to interpret textual lessons
R.O.Weber Calgary 3 Aug 2001
Managerial issues
Knowledge collection
 Determine, communicate and enforce standards
for lesson collection and representation
Knowledge validation
 Define structured format
Knowledge reuse
 Embed knowledge in targeted processes
 Monitor knowledge transfer
 Oversee knowledge reuse
16
R.O.Weber Calgary 3 Aug 2001
Technological, Human, Managerial issues
Standalone repository
Monitored distribution
Retrieval method
Case retrieval
Textual format of lessons
Lessons as structured cases
Collection method
embed instructions
for lesson submission
Lesson elicitation tool that
embeds instructions for
lesson submission and
converts them into the
structured format
17
R.O.Weber Calgary 3 Aug 2001
Methodology
Prescriptive KM infra structure for frameworks
human
users
LET
organizational
processes
lesson repository for
monitored distribution
18
 Monitored distribution
 Case representation
 Lesson elicitation tool that
embeds instructions for lesson
submission and converts them
into the structured format
Casecase
base
of lessons
base
R.O.Weber Calgary 3 Aug 2001
Frameworks
Monitored Distribution
Case represebtation
Lesson Elicitation Tool
19
R.O.Weber Calgary 3 Aug 2001
Monitored Distribution
Problems
&
Solutions
Distribution is divorced from
targeted organizational processes.
Lessons are integrated to targeted
organizational processes.
Users may not be reminded of the
repository, as they need to access a
standalone tool to search for
lessons.
Users don’t need to be reminded of
the repository because they don’t
need to access a standalone tool.
Users may not have the time and
skills to retrieve relevant lessons.
No additional time or skills are
required.
Text databases have low levels of
precision and recall
Case retrieval of disambiguated
knowledge increase recall and
precision
Users may not be able to apply
lessons successfully.
Whenever possible, an ‘apply’
button allows the lesson to be
automatically executable.
20
Monitored distribution
characteristics
 Distribution is tightly integrated to the targeted processes
and distribute lessons when and where they are needed.
 Represent lessons as form-like cases.
 Distribute lessons using case-retrieval/ and retrieve lessons
based on similarity.
 Additional benefits are:
 Case representation facilitates interpretation.
 Users access the lesson rationale to evaluate its potential
utility.
 Cases are retrieved in the context of similar experiences.
21
R.O.Weber Calgary 3 Aug 2001
Noncombatant Evacuation Operations:
Military operations to evacuate noncombatants
whose lives are in danger and rescue them to a safe haven
R.O.Weber Calgary 3 Aug 2001
Assembly
Point
Campaign headquarters
Intermediate Staging Base
.
safe haven
NEO site
Example in HICAP
• HICAP is a plan authoring tool suite
http://www.aic.nrl.navy.mil/hicap
• Muñoz-Avila et al., 1999
• Users interact with HICAP by refining an
HTN (hierarchical task network) through
decompositions
24
R.O.Weber Calgary 3 Aug 2001
NEO site
R.O.Weber Calgary 3 Aug 2001
safe haven
Selecting the Suggested Case…
R.O.Weber Calgary 3 Aug 2001
Expanding yields…
R.O.Weber Calgary 3 Aug 2001
And the user is notified of a lesson
RATIONALE:
TYPE: advice
Clandestine SOF should not be used alone
WHY: The enemy might be able to infer that SOF are involved, exposing them.
R.O.Weber Calgary 3 Aug 2001
After applying the lesson
R.O.Weber Calgary 3 Aug 2001
Evaluation
Hypothesis

Using lessons will improve plan quality
Methodology
Simulated HICAP users generated NEO
plans with and without lessons
 NEO executor implemented plans

Plan total duration
 Plan duration before medical assistance
 Casualties: evacuees, FF, enemies

30
R.O.Weber Calgary 3 Aug 2001
Plan evaluator
non deterministic (100 plans 10 times each)
30 variables: 12 random
length of plans 18 steps
size of planning space 3,000,000
13 lessons
Actions and their influences
31
R.O.Weber Calgary 3 Aug 2001
Plan evaluator: actions



32
Plans where evacuees were transported by land
modes have an increased chance of being
attacked by enemies.
Plans that combined weather with too strong
wings have a small chance of helicopter crash.
When an attack or crash happens it increases the
number of casualties among evacuees and FF (in
proportion to # of evacuees).
R.O.Weber Calgary 3 Aug 2001
Plan evaluator: lessons
Conditions for applicability:
There are representatives of different branches assigned to
participate.
Lesson suggestion:
Assign representatives of all forces to plan.
Rationale:
Lack
of
representatives
prevent
good
communication causing delays and miscommunication.
Conditions for applicability:
There are hundreds or more evacuees as to justify a security
effort.
Lesson suggestion:
Assign EOD personnel.
Rationale: An evacuee once asked what to do with their
weapons.
33
R.O.Weber Calgary 3 Aug 2001
Results
no lessons
with lessons
reduction
39h50
32h48
18 %
duration until
medical assistance*
29h37
24h13
18 %
casualties
among evacuees
11.48
8.69
24 %
casualties among
friendly forces
9.41
6.57
30 %
casualties
among enemies
3.08
3.14
-2 %
NEO plan
total duration*
34
*The resulting values are averages
R.O.Weber Calgary 3 Aug 2001
Case representation
CBR Cycle and Knowledge Processes
New
knowledge
Case Representation
case problem 
case solution 
indexing elements
reuse elements
Distributing process knowledge in the
context of the targeted process.
36
R.O.Weber Calgary 3 Aug 2001
Lessons learned representation
indexing elements:


applicable task
conditions for applicability
reuse elements:
lesson suggestion
 rationale

37
R.O.Weber Calgary 3 Aug 2001
Problems
&
Solutions
Users may not be able to correctly
interpret retrieved lessons because
they may be long and not well written
If more than one lesson happens to be
retrieved, the representation allows
the user to assess its relevance
immediately
Free text collection allows unlimited
content and have the potential to
cause misuse of lessons
Pre-defined content and format limit
the content to what is applicable,
facilitating correct use
Users may not be able to interpret
long and ambiguous texts
The conversion into cases forces
disambiguation facilitating
interpretation
Textual representations complicate
validation, sometimes validation
results in more text added
Structured representation can
potentially allow automatic
verifications, without adding
unnecessary content
38
Lesson Elicitation Tool
Elicitation Tool




39
LET is supported by generic and domain-specific lexicons
of expressions and verbs (do not store unless relevant)
Confirmations search for clues indicating user’s need for
help
A domain lexicon supports disambiguation at run-time
Uses a subset of NL based on the case representation by
using a template-based elicitation with pre-defined
grammar structures thus avoiding NL parsing
LET supports conversation to acquire new concepts for
the ontology
R.O.Weber Calgary 3 Aug 2001
1.2 Fields Elicitation
n=0
Is
n>4
?
No
Yes
Display
view text
Elicit field n
Word comparison
Do
words entered
match existing
ones?
Yes
n++
n: elicitation field
40
No
start
vocabulary
elicitation
1.2.1 APPLICABLE ACTION Elicitation
1.2.2 SUGGESTION Elicitation
1.2.3 CONDITIONS Elicitation
1.2.4 ORIGINATING EVENT Elicitation
1.2.5 View text (edit)
1.2.a Word comparison
1.2.b Vocabulary Elicitation
41
R.O.Weber Calgary 3 Aug 2001
42
R.O.Weber Calgary 3 Aug 2001
43
R.O.Weber Calgary 3 Aug 2001
44
R.O.Weber Calgary 3 Aug 2001
45
R.O.Weber Calgary 3 Aug 2001
46
R.O.Weber Calgary 3 Aug 2001
Para 5 Narr5. (U) OBSERVATION: An ODA on a low-visibility
special recon mission was used to implement a NEO involving
notional U.S. citizens. SOF MH-53 helicopters were used to
evacuate personnel from Tinian. The original plan called
for a SOC qualified MARFOR to accomplish this operation
with the clandestine assistance of the SOF personnel.
Para 6 Narr6. (U) DISCUSSION: Using covert SOF personnel
and helicopters to implement a NEO comprises these assets
if high-visibility conventional forces are not also
utilized in theoperation. An alert OPFOR commander would
question exactly how the forces assisting with the NEO got
on the island, if no conventional forces were inserted. He
ought to be able to infer that SOF were involved, which
compromises them.
Para 7 Narr7. (U) LESSON LEARNED: The CJTF must be made
aware that, if he implements a plan that uses only visible
SOF on a NEO without conventional forces being on the
scene, this increases the risk to clandestine SOF personnel
performing missions supporting his campaign plan.
Para 8 Narr8. (U) RECOMMENDED ACTION: Clandestine SOF
assets on low-visibility missions generally should not be
used alone to perform a NEO where they can be observed by
the OPFOR.
Para 9 Narr9. (U) COMMENTS: The CJTF should use lowvisibility SOF assets alone on a NEO only when the cost of
leaving U.S. citizens in harm's way (possibly as hostages)
exceeds the risk that compromised SOF personnel may not be
able to accomplish their missions.
47
R.O.Weber Calgary 3 Aug 2001
SOF and
conventional
forces
Problems
Current Collection:
&
Solutions (iii)
Lesson Elicitation Tool:
Users do not know what specific LET educates users indicating
content to communicate when
what contents to communicate
submitting lessons
Self-explanatory elicitation tool
Users have to compose textual guides users in using a predescriptions of their experiences defined format by filling out
blank fields and selecting from
drop-down lists
Lack of training and instructions Embedded training with
examples and confirmations*
for lesson submission
48
Next steps

Requirement for monitored distribution is that lessons are
represented as cases.
• Collection tool;
• Verify methods;
• Reasoning (?)

Evaluation with human subjects (simulated users in HICAP)
and let human subjects decide on applying lessons.
Extend Monitored Distribution to other knowledge artifacts.
Extend Monitored Distribution to other DDS.
Integrate experiential knowledge with training knowledge.
Human issues (e.g., disclosing identity of lesson authors)




49
R.O.Weber Calgary 3 Aug 2001