David W. Aha
Rosina Weber
Héctor Muñoz-Avila
Leonard A. Breslow
Kalyan Moy Gupta
Navy Center for Applied Research in Artificial Intelligence, Naval Research Laboratory
Booth # 214

Introduction


Contributions
Context:
 lessons learned systems, process, organizations
Lesson distribution gap
How to bridge this gap? Monitored Distribution
Example
Evaluation, Results
Next Steps
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Describe lessons learned process
Identify gap in lesson distribution
Propose Monitored Distribution
Test hypothesis in evaluation


Monitored Distribution can improve plan quality
Plan evaluator
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Knowledge management context
Three types of KM initiatives


 knowledge repositories knowledge access and transfer knowledge environment
Working Knowledge


 industry oriented (alert systems, best practices) organization oriented (lessons learned systems) for example, ..
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

government non-government military non-military
US int’l
European Space Agency
Italian (Alenia)
Air Force
French (CNES)
Army
Japanese (NASDA)
United Nations
Coast Guard
Joint Forces
Marine Corps
Construction Industry Inst.
Honeywell
GM
Hewllet Packard
Bechtel Jacobs Company
Lockheed Martin E. Sys, Inc
DynMcDermott Petroleum Co.
Xerox
IBM
BestBuy
Siemens
Navy int’l
Canadian Army Lessons Learned Centre
US
Department of Energy: SELLS
NASA (Ames, Goddard)

Lessons learned systems
KNOWLEDGE
ARTIFACTS
Lessons learned systems are repositories of a knowledge artifact called lessons learned
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

…or organizational lessons, lessons, lessons identified
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

REUSE
RETRIEVE
REVISE
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
RETAIN

indexing elements (case problem)

 applicable task preconditions reuse elements (case solution)

 lesson suggestion rationale
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

applicable task
Installing custom stereo speakers.
preconditions
The car is the Porsche Boxster.
lesson suggestion
Make sure you distinguish the wires leading to the speakers from the wires leading to the side airbag.
rationale
Somebody has cut the wrong wire because they look alike and the airbag went off with explosive force. This means spending several thousand dollars to replace the airbag in addition to be a potential hazard.
From article “Learning from Mistakes” about Best Buy in knowledge management magazine, April 2001.

Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Lesson distribution methods
Broadcasting bulletins, doctrine
Passive standalone repository
Active casting list servers, information gathering tools
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA






Distribution is divorced from targeted organizational processes.
Users may not know or be reminded of the repository, as they need to access a standalone tool to search for lessons.
Users may not be convinced of the potential utility of lessons.
Users may not have the time and skills to retrieve and interpret textual lessons.
Users may not be able to apply lessons successfully.
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Here is the gap
Repository of lessons learned
Organization’s members
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

How to bridge this gap?
Organization’s members
Repository of lessons learned
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

How to bridge this gap?
Organization’s members
Repository of lessons learned
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

How to bridge this gap?
Organization’s members
Repository of lessons learned
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

How to bridge this gap?
Organization’s members
Repository of lessons learned
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

How to bridge this gap?
Organization’s members
Repository of lessons learned
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

How to bridge this gap?
Organization’s members
Repository of lessons learned
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

How to bridge this gap?
Repository of lessons learned
Organization’s members
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

How to bridge this gap?
Repository of lessons learned
Organization’s members
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

How to bridge this gap?
Repository of lessons learned
Organization’s members
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Repository of lessons learned
Organization’s members
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Lesson repository is in the same context as targeted processes
Repository of lessons learned
Organization’s members
Organizational processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Distribution is divorced from targeted organizational processes.
Lessons are distributed to users in the context of the organizational processes.
Users need to access a standalone tool to search for lessons.
Users don’t need to access a standalone tool.
Distribute in the same context does not suffice!

Users may not have the time and skills to retrieve relevant lessons.
No significant additional time or skills are required.
Users may not be convinced of the potential utility of lessons.
Users can assess the potential utility of lessons easily.
Users may not be able to apply lessons successfully.
Whenever possible, an ‘apply’ button allows the lesson to be automatically executable.
Intrusive methods may cause more problems than solutions.
Distribution tightly integrated to the targeted processes.

Monitored Distribution Characteristics
 Distribution tightly integrated to the targeted processes so that lessons are distributed when and where they are needed.
• Represent lessons as cases (knowledge modeling).
• Lessons are indexed by their applicability.
 Additional benefits are:
• Case representation facilitates interpretation.
• Users assess potential utility with lesson rationale.
• Whenever possible, an ‘apply’ button allows the lesson to be automatically executable.

Noncombatant Evacuation Operations (NEO)
Military operations to evacuate noncombatants whose lives are in danger and rescue them to a safe haven
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Assembly Point Campaign headquarters
NEO site
Intermediate Staging Base
.
safe haven

•HICAP is a plan authoring tool suite
•Users interact with HICAP by refining an
HTN (hierarchical task network) through decompositions
• http://www.aic.nrl.navy.mil/hicap
•Muñoz-Avila et al., 1999
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

NEO site
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA safe haven

Selecting the Suggested Case…
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

After applying the lesson
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Hypothesis

Using lessons will improve plan quality
Methodology


Simulated HICAP users generated NEO plans with and without lessons
Plan evaluator implemented plans

Plan total duration

Plan duration before medical assistance

Casualties: evacuees, friendly forces, enemies
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

non-deterministic (100 plans 10 times each)
30 variables: 12 random e.g., weather, airports length of plans 18 steps e.g., transportation mode, supplies, team size of planning space 3,000,000
13 actual lessons
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA


Plans where evacuees were transported by land modes have an increased chance of being attacked by enemies.

When an attack happens it increases the number of casualties among evacuees and friendly forces
(in proportion to # of evacuees).
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

*The resulting values are averages no lessons
39h50
29h37
11.48
with lessons
32h48 NEO plan total duration* duration until medical assistance* casualties among evacuees casualties among friendly forces casualties among enemies
24h13
8.69
9.41
6.57
3.08
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
3.14
reduction
18 %
18 %
24 %
30 %
-2 %

Collection tool
Verify methods, reasoning
Integration of informal groups’ and user’s individual features
Evaluation with human subjects (simulated users in HICAP) and let human subjects decide on applying lessons
Extend MD to other decision support systems and other knowledge artifacts
Investigate distribution of experiential knowledge with training knowledge
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

David W. Aha
Navy Center for Applied Research in Artificial Intelligence,
Naval Research Laboratory, Washington, DC
Rosina Weber
Department of Computer Science, University of Wyoming
Fall 2001 at Drexel University, PA
Héctor Muñoz-Avila
Department of Computer Science, University of Maryland
Fall 2001 at Leehigh University, PA
Leonard A. Breslow
Navy Center for Applied Research in Artificial Intelligence,
Naval Research Laboratory, Washington, DC
Kalyan Moy Gupta
IIT Industries, AES Division, Alexandria, Virginia

Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Aamodt & Plaza 1994


Intrusive method requires good precision

Knowledge representation is costly and so are lives!

What’s the worth of 35,000 unused lessons?

Knowledge representation can be also support validation

Good news: collect lessons into case representation.
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

applicable task:
Assign security element.
Conditions for applicability:
There are hundreds or more evacuees as to justify a security effort.
Lesson suggestion:
Recommend that EOD* personnel is utilized in security element.
Rationale:
Success.
EOD two DET ten personnel were employed in a force protection role and assisted USS Nassau security teams in identifying and investigating suspect items brought aboard by evacuees.
*EXPLOSIVE ORDNANCE DISPOSAL
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

How is monitored distribution (MD) different from Clippie?
• In MD, case base task is applicability
• MD distributes experiential knowledge collected from users in similar roles as the potential reuser
•
•
Clippie is activated by single word
Clippie distributes general instructions/information
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA

Conditions for applicability:
There are representatives of different branches assigned to participate.
Lesson suggestion:
Assign representatives of all forces to plan.
Rationale: Lack of representatives prevent good communication causing delays and miscommunication.
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA