German Workshop on
Experience Management
2003
Experience Management within
Project Management Processes
Maya Kaner and Reuven Karni
Industrial Engineering and Management
Technion – Israel Institute of Technology
CSC Proprietary 3/16/2016 7:59:33 AM 008_fmt_wht
1
Experience Management within Project Management Processes
Experience Management within
Project Management Processes
Project management is a knowledge intensive
activity. Managers use their skills and experience to
make decisions during the execution of a
development process. Better results in terms of
meeting schedule budget and functionality are
usually achieved due to the manager’s accumulated
experiences and knowledge.
(Barros, Werner and Travassos)
Experience Management within Project Management Processes
Aims of the research
• To propose an process oriented architecture for
project management experience management
• To define metrics for interpreting knowledge and
transferring experience along the process
• To study the effectiveness of a knowledge-based
architecture in terms of these metrics
Experience Management within Project Management Processes
Definitions
• Project management
• Project management is the application of
knowledge, skills, tools and techniques to project
activities in order to meet or exceed stakeholder
expectations (PMBOK)
• Project management processes
• Those processes within a project which are
concerned with planning, organizing, managing
and controlling a project in the areas of
integration, scope, time, cost, quality, human
resources, communications, risk and
procurement (PMBOK)
Experience Management within Project Management Processes
The Project Management Body of Knowledge
• The PMBOK provides a knowledge perspective of
project management
• It divides project knowledge into an hierarchy of
project knowledge areas and project management
processes
• It details nine knowledge areas and 39 project
management processes
•This stresses the wide range of knowledge and
experience required by the project manager
Experience Management within Project Management Processes
Definitions
• Experience management
• A special form of knowledge management
which deals with task-based and experiential
knowledge that is both gained and applied when
carrying out business-related tasks (Minor)
• Experience-based knowledge
• Knowledge is not so much a capacity for
specific action, but the capacity to use
information, learning and experience resulting in
an ability to interpret information and ascertain
what information is necessary in decisionmaking
(Watson)
Experience Management within Project Management Processes
Definitions
• Knowledge focus
• The compilation and correlation of knowledge
relevant to the execution of a knowledgeintensive activity
• Metric
• A measure of a particular characteristic of
performance or efficiency
• In our case this is the ability to interpret
knowledge and transfer experience
Experience Management within Project Management Processes
Scope
• Business-related tasks
• Our specific tasks are those concerned with
knowledge application in decisionmaking within
project management
• Project-related experience
• The specific experience investigated is that
obtained and employed by project managers
when making decisions (our “target”)
Experience Management within Project Management Processes
Research theory
• Based on a schema proposed by March and Smith
“Design and natural science research in
information technology”
• Research deliverables or outputs are
• Constructs – concepts or vocabulary of the
domain
• Model – propositions expressing relationships
amongst the constructs
• Method – a set of steps to perform a task
Experience Management within Project Management Processes
Research theory
•Research activities include
• Build
• Evaluate
“We build an artifact to perform a specific task. The
basic question is: does it work?
We evaluate artifacts to determine if we have made
any progress. The basic question is: how well does
it work?
Evaluation requires the development of metrics and
the measurement of artifacts according to those
metrics”
Experience Management within Project Management Processes
Research theory
•Research metrics include
• Constructs – completeness, simplicity,
understandability, ease of use
• Model – completeness, level of detail,
consistency
• Method – operability, efficiency, generality, ease
of use
Experience Management within Project Management Processes
Our research outputs – constructs
Knowledge area level (K)
Process level (P)
Issue level (I)
Case level (C)
Entity level (E)
Value level (V)
Experience Management within Project Management Processes
Our research outputs – constructs
Knowledge area level (K)
Process level (P)
Issue level (I)
Project management
body of knowledge
(PMBOK)
Case level (C)
Case
Entity level (E) base
Value level (V)
Experience Management within Project Management Processes
Our research outputs – model (I)
(processes, issues and cases)
K
Human
resources
P
Staff
acquisition
I/C
Select
Candidate
E
V
System
Experience
Budget
Beginner
Problem
Response
Experience
Delay
Duration
Cost
1.5 – 3 wks
30 – 40K
C
Risk
mgt.
Risk
response
Corrective
action
R
Time
mgt.
Activity
definition
Duration
estimating
E
Experience Management within Project Management Processes
Our research outputs – model (II)
(dyads and paths)
(+,+,+,+,-,-)
R 14
(+,+,+,+,+,+)
C 16
E 18
(+,+,+,+,+,+)
E 18
+ value recorded
- no value
(+,+,+,+,-,-)
Experience Management within Project Management Processes
Our research outputs – method
(creation of a knowledge focus)
• Five case-based and path-based activities
• Define case retrieval input from a decision scenario using
“case-by-example”
• Retrieve all corresponding cases
• Select relevant cases to retrieve paths
• Retrieve all path dyads and interpret them
• Synthesize cases and path dyads to create a knowledge
focus related to the decision scenario
• Carry out any feedback required to retrieve
other cases or paths
Experience Management within Project Management Processes
Our research – analysis of a decision scenario
(creation of a knowledge focus)
• Scenario (text)
"The manager of an information systems
implementation project is required to select the best
candidate for analyzing a human resource
management system. He can choose a beginner,
someone with experience, or an expert. The decision
should be based on knowledge from past projects,
concentrating on the trade-off between inexperience
risks versus salary costs"
Experience Management within Project Management Processes
Knowledge and experience metrics
• Mappability
• Definition – the ability to map project processes
(PMBOK) into issues
• Experience – associating project process
outputs with the decisions (usually selection)
needed to create them
• Measure – percentage of project process
outputs (PMBOK) mapped into issues
Experience Management within Project Management Processes
Knowledge and experience metrics
• Compliance
• Definition – the ability to map a textual
decisionmaking situation into cases-by-example
expressed by the issue-case-entity lexicon
• Experience – associating textual formats with
ontological formats
• Measure – number of cases-by-example created
from the decisionmaking situation description
Experience Management within Project Management Processes
Knowledge and experience metrics
• Accordance
• Definition – the ability to map a graphical
representation of a path into a textual
interpretation of the relationships expressed by
the path
• Experience – associating nodes and arcs with
specific successive actions
• Measure – graph traverse mapped into causality
Experience Management within Project Management Processes
Knowledge and experience metrics
• Convergence
• Definition – the ability to map a set of retrieved
cases and paths onto a knowledge focus
• Experience – synthesizing knowledge from a set
of retrieved sources
• Measure – cases and paths mapped into
knowledge focus
Experience Management within Project Management Processes
Knowledge and experience metrics
• Cover
• Definition – the knowledge resources compiled
to create a knowledge focus
• Experience – the ability to retrieve and compile
knowledge (completeness)
• Measure – non-redundant cases and paths
compiled towards a knowledge focus
Experience Management within Project Management Processes
Knowledge and experience metrics
• Efficiency
• Definition – the time required to create a
knowledge focus
• Experience – the ability to retrieve and compile
knowledge (speed)
• Measure – the time required to create a
knowledge focus
Experience Management within Project Management Processes
Measures (scoring)
• We score performance by counting the number of
errors in the “knowledge items” required to
successfully carry out each step in the process.
• In this way we can learn from the experiences of
the subjects in order to reduce errors and thus
improve the methodology
Experience Management within Project Management Processes
Measures – Mappability
(issues connected with process outputs)
• Errors: outputs not mapped into issues
• (out of scope of this presentation)
Experience Management within Project Management Processes
Measures – Compliance
(decomposition of a text into issues)
• Errors: faulty search for relevant cases
• Redundant search – CBE values not directed
unneeded knowledge
• Incomplete search – CBE values not relevant
unneeded knowledge
• Constrained search – too many CBE values
missing knowledge
• Null search
– case/issue not identified
missing knowledge
Experience Management within Project Management Processes
Measures – Accordance
(interpretation of a path graph)
• Errors: arcs or nodes overlooked or misinterpreted
• Null
– arc overlooked
missing causality
• Null
– case overlooked
missing knowledge
• Incomplete – arc misinterpreted
missing causality
• Incomplete – case misinterpreted
missing knowledge
Experience Management within Project Management Processes
Measures – Convergence
(creation of a knowledge focus)
• Errors: cases or paths overlooked or misinterpreted
• Null
– focus component overlooked
missing focus
• Incomplete – focus component misinterpreted
misleading focus
Experience Management within Project Management Processes
Measures – Cover
(cases/paths required for knowledge focus)
• Errors: excessive cases or paths
• Cases – too many cases retrieved
blurred knowledge focus
• Paths – too many paths retrieved
blurred knowledge focus
Experience Management within Project Management Processes
Measures – Efficiency
(total time required to derive knowledge focus)
• Errors: excessive time required
• Time – extra time required
misdirected knowledge focus
Experience Management within Project Management Processes
Experiment – Compliance
(decomposition of a text into issues)
• Scenario (text)
"The manager of an information systems
implementation project is required to select the
best candidate for analyzing a human resource
management system. He can choose a beginner,
someone with experience, or an expert. The
decision should be based on knowledge from
past projects, concentrating on the trade-off
between inexperience risks versus salary costs"
Experience Management within Project Management Processes
Experiment – Compliance
(decomposition of a text into issues)
• Solution (cases-by-example)
• C (candidates)
• Specialization = “Analysis”
• System
= “Human resource management”
• Aim
= Experience and salary of past HRM
candidates
• E (time and cost estimates)
• Specialization = “Analysis”
• System
= “Human resource management”
• Aim
= If and when original estimates updated
• R (risks)
• Category
• Type
• Aim
= “Project team member(s)”
= “Experience”
= Risks of team inexperience and responses
Experience Management within Project Management Processes
Experiment – Compliance
(decomposition of a text into issues)
• Experiment
• Groups
• Control
• Treatment
given case lexicon at start of experiment
given case lexicon before start of experiment
• Results – average errors (p < 0.01)
• Control
n = 30; average = 3.8; variance = 2.9
• Treatment
n = 30; average = 2.0; variance = 1.0
• Inference
• Treatment group makes fewer errors
• The experience of studying the lexicon contributes to better
performance
Experience Management within Project Management Processes
Experiment – Compliance
(decomposition of a text into issues)
• Experiment
• Errors – misdirected input
• Redundant search 46%
• Constrained search 24%
• Null search
20%
• Incomplete search
9%
• Errors – cases involved
• Risks
• Estimates
• Candidates
87%
9%
3%
• Interpretation of main errors
• Fear of missing knowledge leads to redundancy
• Complexity of risk case causes problem with decomposition
Experience Management within Project Management Processes
Experiment – Accordance
(interpretation of a path graph)
• Scenario (graph)
(+,+,+,+,-,-)
C 16
R 14
(+,+,+,+,+,+)
E 18
(+,+,+,+,+,+)
E 18
+ value recorded
- no value
(+,+,+,+,-,-)
Experience Management within Project Management Processes
Experiment – Accordance
(interpretation of a path graph)
• Solution (textual interpretation)
• C 16 (+,+,+,+): A beginner with salary 10-15K was selected for
coding the maintenance system
 E 18 (+,+,+,+,-,-): This led to an estimate of the corresponding
activity duration (1.5 – 2 wks) and cost (30 – 40K)
 R 14 (+,+,+,+,-,-): This also led to anticipating a problem of lack
of experience which could be triggered by a change in
requirements
 E 18 (+,+,+,+,+,+): Foreseeing this problem then led to a
revision of the estimated activity duration (2 – 3 wks) and cost (50
– 60K)
 R 14 (+,+,+,+,+,+): This in turn led to an expectation of project
delay
Experience Management within Project Management Processes
Experiment – Accordance
(interpretation of a path graph)
• Experiment
• Groups
• Control
• Treatment
none
given case details
• Results – average errors
• Treatment
n = 30; average = 1.9; variance = 1.6
• Inference
• The graph represents “experience” in linking decisions; the
subjects were require to reproduce this experience
Experience Management within Project Management Processes
Experiment – Accordance
(interpretation of a path graph)
• Experiment
• Errors – incorrect case interpretation
• Risks
• Estimates
• Candidates
16%
9%
9%
• Arc E – R
• Other arcs (3)
• Candidates
30%
12%
3%
• Errors – incorrect arc interpretation/overlook
• Interpretation of main errors
• Inability to interpret the evolution of the risk case along the path
sequence
• Inability to interpret the apparent “loop” in the path
Experience Management within Project Management Processes
Experiment – Convergence
(creation of a knowledge focus)
• Scenario (text)
"The manager of an information systems
implementation project is required to select the
best candidate for analyzing a human resource
management system. He can choose a beginner,
someone with experience, or an expert. The
decision should be based on knowledge from
past projects, concentrating on the trade-off
between inexperience risks versus salary costs"
Experience Management within Project Management Processes
Experiment – Convergence
(creation of a knowledge focus)
• Five case-based and path-based activities
• Define case retrieval input from the decision scenario
using “case-by-example”
• Retrieve all corresponding cases
• Select relevant cases to retrieve paths
• Retrieve all path dyads and interpret them
• Synthesize cases and path dyads to create a knowledge
focus related to the decision scenario
• Carry out any feedback required to retrieve
other cases or paths
Experience Management within Project Management Processes
Experiment – Convergence
(control – no guidance given)
• Four CBR and PBR actions (icons)
• Define case retrieval input from the decision scenario
using “case-by-example”
• Retrieve all corresponding cases
• Select relevant cases to retrieve paths
• Retrieve all path dyads and interpret them
• Press any icon to activate the corresponding action
Experience Management within Project Management Processes
Experiment – Convergence
(treatment – process map given)
•
Four linked CBR and PBR actions (icons)
1.
Define case retrieval input from the decision scenario
using “case-by-example”
2.
Retrieve all corresponding cases
3.
Select relevant cases to retrieve paths
4.
Retrieve all path dyads and interpret them
•
Carry out any feedback required to retrieve
other cases or paths
Experience Management within Project Management Processes
Experiment – Convergence
(creation of a knowledge focus)
• Experiment
• Groups
• Control
• Treatment
carry out any of the activities, in any order
carry out all four activities in a given order
(process)
• Results – average errors (p < 0.01)
• Control
n = 15; average = - 0.9; variance = 0.5
• Treatment
n = 15; average = - 0.5; variance = 0.3
• Inference
• The treatment group is better able to synthesize the knowledge
retrieved and to relate it to the decision to be made
• A systematized process contributes to better performance
• Experience – knowledge about the sequence in which to retrieve
and interpret knowledge – contributes to better performance
Experience Management within Project Management Processes
Experiment – Efficiency
(total time required to derive knowledge focus)
• Experiment
• Groups
• Control
• Treatment
carry out any of the activities, in any order
carry out all four activities in a given order
(process)
• Results – average errors (p < 0.05)
• Control
n = 15; average = 23 minutes; variance = 37
• Treatment
n = 15; average = 19 minutes; variance = 21
• Inference
• The treatment group is faster in generating the knowledge focus
• A systematized process contributes to faster and more uniform
performance
• Experience – knowledge about the sequence in which to retrieve
and interpret knowledge – contributes to better performance
Experience Management within Project Management Processes
Experiment – Cover (cases)
(cases required for knowledge focus)
• Experiment
• Groups
• Control
• Treatment
carry out any of the activities, in any order
carry out all four activities in a given order
(process)
• Results – average errors (lower bound = 3) (p < 0.05)
• Control
n = 15; average = + 1.47; variance = 2.6
• Treatment
n = 15; average = + 0.40; variance = 1.4
• Inference
• The treatment group requires less to find a meaningful input
• Systematization gvies more effective and uniform performance
• Systematization gives more confidence and thus less scope
• Experience – knowledge about the sequence in which to retrieve
and interpret knowledge – contributes to better performance
Experience Management within Project Management Processes
Experiment – Cover (paths)
(paths required for knowledge focus)
• Experiment
• Groups
• Control
• Treatment
carry out any of the activities, in any order
carry out all four activities in a given order
(process)
• Results – average errors (lower bound = 3)
• Control
n = 15; average = + 3.27; variance = 2.8
• Treatment
n = 15; average = + 3.47; variance = 3.3
• Inference
• Both groups tend to search the same number of paths
• However, the treatment group retreieved these pathns in an
organized fashion
• Experience – knowledge about the sequence in which to retrieve
and interpret knowledge – contributes to better performance
Experience Management within Project Management Processes
Conclusions
• Business process oriented knowledge
management (BPOKM) means that knowledge is
structured according to process requirements
• Experience oriented knowledge management
means that knowledge is created and updated
through the experience of those applying the
knowledge
•The project manager looks for suitable candidates
(process) and seeks the effects of previous
decisions (experience) on time, costs and risks as
stored in the knowledge base
Experience Management within Project Management Processes
Conclusions
• Our aim is to create an effective knowledge focus –
the right number, identification and interpretation of
the retrieved cases and paths
•At each step of the process, knowledge focus
errors may occur
• These have been measured by a uniform metric – a
count of the errors incurred
• Errors reflect redundant or missing or
misinterpreted knowledge elements (cases or paths)
as compared to an “ideal” search for the knowledge
required
Experience Management within Project Management Processes
Conclusions
• In general we find that systematization of the
decision process and the link between process and
knowledge required leads to better performance
• Our methodology allows us to locate areas and
reasons for non-performance, such that the project
management decisionmaking process can be
improved
Experience Management within Project Management Processes
To sum up …
• Knowledge bases can be structured by adopting a
process description as one specific foundation for
the schema of the knowledge base. This facilitates
access since in a process oriented application the
user knows his exact position in the process and
can therefore easily navigate the knowledge base.
• Since the process description functions as a
structuring criterion for a knowledge base, all the
relevant information and knowledge for a process
step – that goes beyond the capability of a
conventional process model – can be captured.
(Jablonski, Horn and Schlundt)
Experience Management within Project Management Processes
To sum up …
We hope that our methodology has
contributed a useful approach, based on
error metrics at each process step, to be
able to measure the effectiveness of this
knowledge capture
German Workshop on
Experience Management
2003
Experience Management within
Project Management Processes
Thank you!
CSC Proprietary 3/16/2016 7:59:36 AM 008_fmt_wht
52