CHAPTER 6
OTHER KNOWLEDGE CAPTURE TECHNIQUES
TEST YOUR UNDERSTANDING
1.
Explain in your own words the distinctive features of on-site observation. What kinds of
problems does it pose?
On-site observation or action protocol involves knowledge-capture through direct
observation. By observing the expert’s actions and work, the knowledge developer (KD) is
able to witness first-hand the processes in completing the tasks and responsibilities of the
expert. During the process, the knowledge developer is able to ask questions but does not
correct or modify, in any way, what the expert is doing.
This process enables the KD to witness the expert’s work first-hand, but also poses
problems:
a. The KD’s presence may be distracting to the expert, affecting adversely the expert’s
work.
b. Possible problem with the accuracy or completeness with which the knowledge captured
is recorded, since there exists a gap from when the knowledge is observed and when it is
documented.
c. Some experts do not like to be observed. They prefer to explain rather than demonstrate
their expertise.
d. The reaction of others in the observation setting can be a distracting problem
2.
Distinguish between:
a.
brainstorming and consensus decision making
b.
protocol analysis and Delphi method
c.
repertory grid and nominal group technique
d.
Blackboarding and electronic brainstorming
a. Brainstorming is idea generation of multiple solutions to a problem, is unstructured, and
results in a wide variety of creative ideas. Consensus decision making, however,
involves a controlled exercise in which the knowledge developer focuses the group on
one or two of the solutions from the brainstorming exercise. It follows brainstorming.
The goal is to arrive at a clear agreement regarding the best solution to a problem.
b. Protocol analysis or think-aloud requires the expert to solve a problem or complete a
task (using cases, scenarios, or protocol), while verbally describing what goes on in
his/her head. Without interruption, the knowledge developer writes down the steps the
expert follows. The Delphi method is a series of questionnaires to solicit input from
multiple experts. These questions are then used to develop a standardized approach to a
problem. The results from one are used to construct the next questionnaire(s), and so on
until a consensus is achieved.
CHAPTER 6
OTHER KNOWLEDGE CAPTURE TECHNIQUES
c. The repertory grid is a bipolar construct on which elements are placed within gradations.
The knowledge developer elicits the constructs and asks the domain expert to provide
the elements, which are rated according to the constructs. In contrast, the nominal group
technique is an idea generation technique. It provides an interface between consensus
and brainstorming. A panel of experts becomes a “nominal” group, whose meetings are
structured in order to effectively pool individual judgment.
d. Blackboarding gets the experts together around a “blackboard” that puts ideas on the
board from the experts until the problem is solved. Electronic brainstorming is similar
to blackboarding, but the inputs are done by the individual experts via a network of PCs.
So, those who are not comfortable speaking in a group will be more confident and offer
more ideas or solutions. The ideas are sorted and condensed into an organized format.
3.
How is brainstorming conducted? Provide an example.
There are several steps involved in brainstorming:
a. Introduction of the session along with the problem to be addressed.
b. Prompt the experts to generate ideas. Ideas need to be given by the group on how to
solve the problem.
c. Watch for signs of convergence. Once the ideas begin to focus on one or a few
solutions, the group should come to a consensus. If a decision cannot be made, a vote
could decide the solution. Throughout the session, all ideas should be considered and
none cut short or shot down.
Students may come up with all kinds of brainstorming examples. For example, one topic
might be “How many exams should be given in this course?” “How should the text material
be covered?” “How should course projects be solved—individually or via teams?”
4.
If no interruptions are allowed or no questions asked while the expert is answering a
question or solving a problem, how does the facilitator control a protocol analysis
session?
In one respect, there isn’t much control involved. The domain expert is going through a
problem-solving protocol and attempts to produce results. All the facilitator can do is listen,
observe, and record details. The control factor becomes important when the expert goes off
on a tangent or when there is degradation in the quality of the expert’s think-aloud process
or when the facilitator finds out he or she is running out of time. Otherwise, it is not so easy
to break into a think-aloud session without adversely affecting the concentration of the
domain expert.
5.
In what way does consensus decision making follow brainstorming? Be specific.
Consensus decision-making follows brainstorming in two ways:
a. Each method allows all experts to present their views or ideas, but discussions invariably
proceed before consensus.
b. The knowledge developer tries to rally the group in each method towards one final
6-2
CHAPTER 6
OTHER KNOWLEDGE CAPTURE TECHNIQUES
decision that will hopefully lead to a consensus.
6.
Give an example of your own to illustrate the procedure followed in consensus decision
making.
Students should be asked to come up with a problem domain of their own and apply the
procedure on page 166 in the text. This short exercise is usually conducted in a team, where
a team member becomes the facilitator or knowledge developer and the other 2-4 team
members are the experts. The session should be timed not to exceed twenty minutes for two
reasons:
a. If time turns out to be too brief for the session, the students begin to realize the
importance of scoping the problem domain to the time constraint.
b. Setting a time limit up front promotes discipline in the way a session is considered.
7.
"The grid is used to acquire and evaluate the expert's model--the way the expert works
through the solution". Do you agree? What is so unique about the grid? What are some
of its drawbacks?
The repertory grid is a representation of the experts’ way of looking at a problem. Its
gradations break down the scale to allow for more accurate ratings. The rating is useful in
comparing individual rankings. It is a comparative subjective rating of elements along a
scale. It may prompt the expert who sees the grid laid out on paper to think more seriously
about the problem and how to solve it.
The main drawback of the grid is that it tends to be a difficult tool to manage effectively
when large grids are accompanied by complex details. Large grids also defeat the tool’s
goal of simplicity, clarity, and manageability. That is the reason why knowledge developers
normally use the grid in the early stages of knowledge capture.
8.
In what way is the nominal-group technique an alternative to the consensus technique?
Explain.
The nominal-group technique is an alternative to the consensus technique, because it is able
to mitigate some of the process losses associated with multiple experts. This technique also
provides an interface between consensus and brainstorming. It provides results similar to
the consensus technique but under a more structured group environment.
9.
Illustrate by an example of your own the procedure followed in NGT.
Students should be encouraged to apply NGT by team. The procedure is laid out on page
170 in the text. One possible NGT exercise is “should there be class on Saturday?”
10.
If you were asked to apply the Delphi method involving four experts, what procedure
would you follow? What limitations would you expect in this knowledge capture
technique?
6-3
CHAPTER 6
OTHER KNOWLEDGE CAPTURE TECHNIQUES
In using the Delphi method, one would first provide the experts with a brief synopsis of the
issue, followed by a questionnaire for each expert to respond to. After these opinions are in,
they would be anonymously presented to the rest of the group. Second, opinion/analysis
from the experts would be requested after each has been the previous responses. Following
this step several times, extreme opinions should be eliminated and the four opinions would
converge on a main idea.
11.
Blackboarding offers certain characteristics. Explain each briefly. What makes up the
basic model?
There are several characteristics of blackboarding:
a. Blackboarding allows for diverse approaches to problem solving, thus allowing the
various viewpoints of multiple experts to be accounted for.
b. Blackboarding must also use a common language (including charts, diagrams, etc.) so
all may communicate on the same level.
c. Flexibility exists concerning the representation of information as long as all the
experts can understand.
d. The blackboard stores all the knowledge input and provides easy access to the
information.
e. Blackboarding is organized. Since only one expert may respond at a time, the session
becomes more controlled and organized.
f. The structure of blackboarding promotes a step-by-step building process, since each
expert adds to or refines what the previous expert did.
The basic model is comprised of the knowledge sources, the blackboard, and a control
mechanism. The knowledge source is the expert, the blackboard is the memory
structure/database, and the control mechanism controls the flow and pattern of the
problem solution.
KNOWLEDGE EXERCISES
1.
Divide into small groups of five to seven persons. Select a group discussion leader and a
person to record responses. Use the brainstorming guidelines to conduct a five-minute
brainstorming session on the following topic. Your goal is to identify creative solutions to
the problem:
Employees in large companies often complain that personal worth perception is low.
They feel that the company does not overtly reward them for their contributions and set
procedures that allow them to be most productive and creative.
This exercise organizes the team into participants, a leader, and a scribe. The 5-minute limit
should demonstrate how it is not so easy to conduct brainstorming or identify creative
solutions in such a short time frame.
6-4
CHAPTER 6
OTHER KNOWLEDGE CAPTURE TECHNIQUES
2.
Based on the problem presented in #1, complete the following tasks:
a.
Brainstorm how the company can reward efforts and increase the perception of
personal worth other than issuing pay increases.
b.
Use nominal group technique to find the best solution to the employee personalworth perception problem. Consider the solutions from the brainstorming activity
and select the "best" solution from that set.
c.
Use consensus decision making with the goal of selecting a solution to the
employee personal-worth perception problem to which all members of the group
can commit.
a. There is no specific answer offered for this part. The focus, however, is on the creativity
of the participants regarding ways in which the company can reward efforts and increase
the perception of personal worth in ways other than issuing pay increases. The
challenge for the team leader is to discourage drifting away from the topic.
b. In this exercise, the participants become the nominal group. One expert (participant)
should be appointed to avoid potential contradictions between experts. This role
becomes more important the larger the nominal group. For details on the NGT
procedure, refer to the text, page 170. At the end of the exercise, the team leader should
ask the team about their experience. Has it been time-consuming? Tedious? How
much concentration did this technique require?
c. Consensus decision making can be used to follow brainstorming. Note possible
intensity of the participants. How willing were they to resolve conflict? Was there a
clear agreement regarding the best solution to the employee personnel-worth perception
problem to which all members of the group could commit?
3.
Publisher KM System
As discussed in this chapter, knowledge capture uses a variety of tools. In Publisher KM
system, several knowledge capture tools were used:
Knowledge was captured from a series of four interviews with the expert. The first two
interviews encompassed fairly broad questions about the expert's decision-making
process. The last two interviews elicited more detailed information about the expert's
decision process by discussing the outcome of specific sample case scenario decisions, to
which confidence levels were assigned.
In one interview, an attempt was made at rapid prototyping, but the format of the table
used was difficult for the expert to understand. So it was not used as productively as
expected.
An induction table (similar to a decision table) was developed, which included possible
combinations of the five variables and their outcomes--acceptance, defer, or rejection of a
book manuscript. (See Table 6-3.) Evaluate the approach taken in the knowledge
capture process to determine the rationale and reliability of the process used.
6-5
CHAPTER 6
OTHER KNOWLEDGE CAPTURE TECHNIQUES
For a problem domain the size of Publisher Advisor, four sessions were adequate to produce
a prototype. The domain expert was quite cooperative and anxious to use the resulting
prototype for training a newly hired junior editor. The induction table was quite effective
and worked well using EXSYS Corporation’s CORVID package (see www.exsys.com).
Rule
#
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
Market
Author’s
Reputation
Reviewers
Comments
Author’s
Requirements
High
High
High
High
High
High
High
High
High
High
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Low
Low
Low
Low
Low
Low
Excellent
Average
Average
Average
Average
Average
Average
Poor
Poor
Poor
Poor
Poor
Poor
Poor
Poor
Average
Average
Average
Average
Average
Average
Average
Average
Average
Poor
Poor
Poor
Excellent
Excellent
Excellent
Excellent
Excellent
Excellent
Excellent
Excellent
Excellent
Excellent
Very good
Very good
Mixed
Mixed
Below avg
Below avg
Very good
Very good
Very good
Mixed
Mixed
Below avg
Below avg
Below avg
Very good
Very good
Very good
Mixed
Mixed
Mixed
Below avg
Below avg
Below avg
Very good
Very good
Very good
Very good
Mixed
Mixed
Below avg
Very good
Very good
Very good
Mixed
Mixed
Below avg
Av. or below average
Above average
Av. Or below average
Above average
Av. Or below average
Above average
Above average
Average
Below average
Above average
Av. Or below average
Above average
Average
Below average
Above average
Average
Below average
Above average
Average
Below average
Above average
Average
Below average
Above average
Average
Below average
Av. Or below average
Above average
Above average
Average
Below average
Av. Or below average
Above average
6-6
Publisher’s Decision
Confidence
Level
Offer contract
Offer contract
Offer contract
Offer contract
Get sample chapters
Get sample chapters
Reject manuscript
Offer contract
Offer contract
Offer contract
Reject manuscript
Get sample chapters
Reject manuscript
Reject manuscript
Reject manuscript
Offer contract
Offer contract
Offer contract
Get sample chapters
Offer contract
Offer contract
Reject manuscript
Reject manuscript
Reject manuscript
Offer contract
Offer contract
Offer contract
Offer contract
Offer contract
Get sample chapters
Get sample chapters
Offer contract
Offer contract
Offer contract
Get sample chapters
Reject manuscript
Reject manuscript
0.90
0.80
0.85
0.75
0.85
0.65
0.75
0.90
0.70
0.75
0.70
0.80
0.90
0.90
0.60
0.70
0.70
CHAPTER 6
OTHER KNOWLEDGE CAPTURE TECHNIQUES
37
Low
Average
38
Low
Average
39
Low
Average
40
Low
Average
Table 6-3. Induction Table
4.
Very good
Very good
Very good
Mixed
Below average
Average
Above average
Av. Or below average
Offer contract
Offer contract
Reject manuscript
Get sample chapters
0.70
0.60
A total of six knowledge capture sessions were conducted with the expert at the expert's
location, except for the few occasions when the session required use of the system.
In the first session, the knowledge developer became familiar with the domain expert
elaborating on the admissions materials previously provided. This overview made it
possible to establish a basis of knowledge, which evolved into the concept dictionary. In
the second session, I used the interview to extract basic parameters and then had the
expert organize these parameters according to her view of the domain. We transformed
this organization into a visual concept diagram of the domain. The concept diagram was
referred to and refined throughout the entire acquisition process. This diagram provided
a clear direction for the acquisition process. It also became the foundation for the
knowledge representation structure. See Figure 6-10.
Other tools employed were process tracing and case scenarios. I used process tracing by
asking the domain expert to provide an example of a typical admissions decision. From
following her decision-making process, I began to understand how the parameters
interacted with one another. This helped me assign parameters to groupings based on the
domain concept diagram. This also alerted me to the minimum and maximum values that
could be assigned to each parameter. In addition, I was able to derive the minimum
requirements for consideration, below which the applicant would be immediately rejected.
A list of parameters is shown in Table 6-4.
Adaptability
Potential
Extracurricul
ar
Parents
Grades
Essay
Test
scores
Discipline
Previous
school
Desire
to attend
Courses
Achievement
Motivation
6-7
CHAPTER 6
OTHER KNOWLEDGE CAPTURE TECHNIQUES
Figure 6-10. Concept diagram
In order to capture the expert's heuristics, I devised a matrix of parameters and their
potential values. This would allow me to vary parameters in relation to each other. This
provied to be a formidable task. Ten multi-valued parameters, each with three or more
permissible ranges, produced more scenarios than could easily be generated via manual
means. The difficulty of this task also has implications for completeness of the rule base,
ergo, scenarios could easily be overlooked. To assist me in generating scenarios, I
discovered Logic Gem, a logic processor software that made life easier for the rest of the
capture process.
Questions
a.
b.
Based on the nature of the problem domain, evaluate the tools used in this project.
How important is listing parameters early in the knowledge capture phase?
Explain.
a. Unlike Publisher Advisor in number 3 above, the knowledge developer for ADMIT set in
advance the importance of structuring the approach to knowledge capture. The tools used
were designed to emulate structured design methodology, in which a problem is broken
down into several modules for easy programmability. In “ADMIT”, the following tools
were used:




Concept dictionary and concept sorting via the interview. The goal is to extract the
primary parameters that make up the prospective knowledge-based system.
Visual concept diagram, in cooperation with the domain expert, maps out graphically
the parameters and their interrelationships. This made it easier to understand the
variables.
Process tracing and case scenarios were used to elicit deep knowledge and validate
decision thought processes enunciated in earlier sessions. Considering the subjective
nature of admission decisions for the private secondary school, these tools were
appropriate.
Induction table, which essentially is a matrix of parameters and their respective values.
This is the closest to what EXSYS Corvid (see www.exsys.com) provides as a
beginning step to building a knowledge-based system.
6-8
CHAPTER 6
OTHER KNOWLEDGE CAPTURE TECHNIQUES
Name
ENGLISH
SCIENCE
LANGUAGE
US HIST.
WORLD
MATH
GRADES
PSAT
SAT
SSAT
AP_ESSAY
TM_ESSAY
MOTIVATION
CHALLENGE
DESIRE
P_SUPPORT
EXTRA_C
SCHOOL
DISCIPLINE
DEMEANOR
MINORITY
LEGACY
FACKID
STAYED
BOARD
SIBENROLL
NEW
Type
Number
String
Boolean
Boolean
Boolean
String
String
Number
Number
Number
String
String
String
Boolean
String
String
Boolean
String
String
String
Boolean
Boolean
Boolean
Boolean
Boolean
Boolean
Boolean
Values
0-3
Biology, Chemistry, Physics
Yes/No
Yes/No
Yes/No
Algebra I, Algebra II, Geometry
A-A, A-B, B-C, C-C, C-D, D-F
0-160
0-1600
0Excellent, Good, Fine, Poor
Excellent, Good, Fine, Poor
Over Achiever, On Target, Under
Achiever
Yes/No
Enthusiastic, Positive, Unsure,
Negative
Excellent, Average, Weak
Yes/No
Above, Equal, Below
Acceptable, Unacceptable
Excellent, Average, Difficult
Yes/No
Yes/No
Yes/No
Yes/No
Yes/No
Yes/No
Yes/No
CHANGES (from Session
#4)
1. Remove RELIGION
2. Remove FORMER
3. Add NEW
4. Remove ERB, SSAT
5. Remove ACCSIB
(same as
SIBENROLL)
Remove individual
language, add LANGUAGE
Table 6-5 ADMIT Parameters
6-9