German Workshop on Experience Management 2003 Experience Management within Project Management Processes Maya Kaner and Reuven Karni Industrial Engineering and Management Technion – Israel Institute of Technology CSC Proprietary 3/16/2016 7:59:33 AM 008_fmt_wht 1 Experience Management within Project Management Processes Experience Management within Project Management Processes Project management is a knowledge intensive activity. Managers use their skills and experience to make decisions during the execution of a development process. Better results in terms of meeting schedule budget and functionality are usually achieved due to the manager’s accumulated experiences and knowledge. (Barros, Werner and Travassos) Experience Management within Project Management Processes Aims of the research • To propose an process oriented architecture for project management experience management • To define metrics for interpreting knowledge and transferring experience along the process • To study the effectiveness of a knowledge-based architecture in terms of these metrics Experience Management within Project Management Processes Definitions • Project management • Project management is the application of knowledge, skills, tools and techniques to project activities in order to meet or exceed stakeholder expectations (PMBOK) • Project management processes • Those processes within a project which are concerned with planning, organizing, managing and controlling a project in the areas of integration, scope, time, cost, quality, human resources, communications, risk and procurement (PMBOK) Experience Management within Project Management Processes The Project Management Body of Knowledge • The PMBOK provides a knowledge perspective of project management • It divides project knowledge into an hierarchy of project knowledge areas and project management processes • It details nine knowledge areas and 39 project management processes •This stresses the wide range of knowledge and experience required by the project manager Experience Management within Project Management Processes Definitions • Experience management • A special form of knowledge management which deals with task-based and experiential knowledge that is both gained and applied when carrying out business-related tasks (Minor) • Experience-based knowledge • Knowledge is not so much a capacity for specific action, but the capacity to use information, learning and experience resulting in an ability to interpret information and ascertain what information is necessary in decisionmaking (Watson) Experience Management within Project Management Processes Definitions • Knowledge focus • The compilation and correlation of knowledge relevant to the execution of a knowledgeintensive activity • Metric • A measure of a particular characteristic of performance or efficiency • In our case this is the ability to interpret knowledge and transfer experience Experience Management within Project Management Processes Scope • Business-related tasks • Our specific tasks are those concerned with knowledge application in decisionmaking within project management • Project-related experience • The specific experience investigated is that obtained and employed by project managers when making decisions (our “target”) Experience Management within Project Management Processes Research theory • Based on a schema proposed by March and Smith “Design and natural science research in information technology” • Research deliverables or outputs are • Constructs – concepts or vocabulary of the domain • Model – propositions expressing relationships amongst the constructs • Method – a set of steps to perform a task Experience Management within Project Management Processes Research theory •Research activities include • Build • Evaluate “We build an artifact to perform a specific task. The basic question is: does it work? We evaluate artifacts to determine if we have made any progress. The basic question is: how well does it work? Evaluation requires the development of metrics and the measurement of artifacts according to those metrics” Experience Management within Project Management Processes Research theory •Research metrics include • Constructs – completeness, simplicity, understandability, ease of use • Model – completeness, level of detail, consistency • Method – operability, efficiency, generality, ease of use Experience Management within Project Management Processes Our research outputs – constructs Knowledge area level (K) Process level (P) Issue level (I) Case level (C) Entity level (E) Value level (V) Experience Management within Project Management Processes Our research outputs – constructs Knowledge area level (K) Process level (P) Issue level (I) Project management body of knowledge (PMBOK) Case level (C) Case Entity level (E) base Value level (V) Experience Management within Project Management Processes Our research outputs – model (I) (processes, issues and cases) K Human resources P Staff acquisition I/C Select Candidate E V System Experience Budget Beginner Problem Response Experience Delay Duration Cost 1.5 – 3 wks 30 – 40K C Risk mgt. Risk response Corrective action R Time mgt. Activity definition Duration estimating E Experience Management within Project Management Processes Our research outputs – model (II) (dyads and paths) (+,+,+,+,-,-) R 14 (+,+,+,+,+,+) C 16 E 18 (+,+,+,+,+,+) E 18 + value recorded - no value (+,+,+,+,-,-) Experience Management within Project Management Processes Our research outputs – method (creation of a knowledge focus) • Five case-based and path-based activities • Define case retrieval input from a decision scenario using “case-by-example” • Retrieve all corresponding cases • Select relevant cases to retrieve paths • Retrieve all path dyads and interpret them • Synthesize cases and path dyads to create a knowledge focus related to the decision scenario • Carry out any feedback required to retrieve other cases or paths Experience Management within Project Management Processes Our research – analysis of a decision scenario (creation of a knowledge focus) • Scenario (text) "The manager of an information systems implementation project is required to select the best candidate for analyzing a human resource management system. He can choose a beginner, someone with experience, or an expert. The decision should be based on knowledge from past projects, concentrating on the trade-off between inexperience risks versus salary costs" Experience Management within Project Management Processes Knowledge and experience metrics • Mappability • Definition – the ability to map project processes (PMBOK) into issues • Experience – associating project process outputs with the decisions (usually selection) needed to create them • Measure – percentage of project process outputs (PMBOK) mapped into issues Experience Management within Project Management Processes Knowledge and experience metrics • Compliance • Definition – the ability to map a textual decisionmaking situation into cases-by-example expressed by the issue-case-entity lexicon • Experience – associating textual formats with ontological formats • Measure – number of cases-by-example created from the decisionmaking situation description Experience Management within Project Management Processes Knowledge and experience metrics • Accordance • Definition – the ability to map a graphical representation of a path into a textual interpretation of the relationships expressed by the path • Experience – associating nodes and arcs with specific successive actions • Measure – graph traverse mapped into causality Experience Management within Project Management Processes Knowledge and experience metrics • Convergence • Definition – the ability to map a set of retrieved cases and paths onto a knowledge focus • Experience – synthesizing knowledge from a set of retrieved sources • Measure – cases and paths mapped into knowledge focus Experience Management within Project Management Processes Knowledge and experience metrics • Cover • Definition – the knowledge resources compiled to create a knowledge focus • Experience – the ability to retrieve and compile knowledge (completeness) • Measure – non-redundant cases and paths compiled towards a knowledge focus Experience Management within Project Management Processes Knowledge and experience metrics • Efficiency • Definition – the time required to create a knowledge focus • Experience – the ability to retrieve and compile knowledge (speed) • Measure – the time required to create a knowledge focus Experience Management within Project Management Processes Measures (scoring) • We score performance by counting the number of errors in the “knowledge items” required to successfully carry out each step in the process. • In this way we can learn from the experiences of the subjects in order to reduce errors and thus improve the methodology Experience Management within Project Management Processes Measures – Mappability (issues connected with process outputs) • Errors: outputs not mapped into issues • (out of scope of this presentation) Experience Management within Project Management Processes Measures – Compliance (decomposition of a text into issues) • Errors: faulty search for relevant cases • Redundant search – CBE values not directed unneeded knowledge • Incomplete search – CBE values not relevant unneeded knowledge • Constrained search – too many CBE values missing knowledge • Null search – case/issue not identified missing knowledge Experience Management within Project Management Processes Measures – Accordance (interpretation of a path graph) • Errors: arcs or nodes overlooked or misinterpreted • Null – arc overlooked missing causality • Null – case overlooked missing knowledge • Incomplete – arc misinterpreted missing causality • Incomplete – case misinterpreted missing knowledge Experience Management within Project Management Processes Measures – Convergence (creation of a knowledge focus) • Errors: cases or paths overlooked or misinterpreted • Null – focus component overlooked missing focus • Incomplete – focus component misinterpreted misleading focus Experience Management within Project Management Processes Measures – Cover (cases/paths required for knowledge focus) • Errors: excessive cases or paths • Cases – too many cases retrieved blurred knowledge focus • Paths – too many paths retrieved blurred knowledge focus Experience Management within Project Management Processes Measures – Efficiency (total time required to derive knowledge focus) • Errors: excessive time required • Time – extra time required misdirected knowledge focus Experience Management within Project Management Processes Experiment – Compliance (decomposition of a text into issues) • Scenario (text) "The manager of an information systems implementation project is required to select the best candidate for analyzing a human resource management system. He can choose a beginner, someone with experience, or an expert. The decision should be based on knowledge from past projects, concentrating on the trade-off between inexperience risks versus salary costs" Experience Management within Project Management Processes Experiment – Compliance (decomposition of a text into issues) • Solution (cases-by-example) • C (candidates) • Specialization = “Analysis” • System = “Human resource management” • Aim = Experience and salary of past HRM candidates • E (time and cost estimates) • Specialization = “Analysis” • System = “Human resource management” • Aim = If and when original estimates updated • R (risks) • Category • Type • Aim = “Project team member(s)” = “Experience” = Risks of team inexperience and responses Experience Management within Project Management Processes Experiment – Compliance (decomposition of a text into issues) • Experiment • Groups • Control • Treatment given case lexicon at start of experiment given case lexicon before start of experiment • Results – average errors (p < 0.01) • Control n = 30; average = 3.8; variance = 2.9 • Treatment n = 30; average = 2.0; variance = 1.0 • Inference • Treatment group makes fewer errors • The experience of studying the lexicon contributes to better performance Experience Management within Project Management Processes Experiment – Compliance (decomposition of a text into issues) • Experiment • Errors – misdirected input • Redundant search 46% • Constrained search 24% • Null search 20% • Incomplete search 9% • Errors – cases involved • Risks • Estimates • Candidates 87% 9% 3% • Interpretation of main errors • Fear of missing knowledge leads to redundancy • Complexity of risk case causes problem with decomposition Experience Management within Project Management Processes Experiment – Accordance (interpretation of a path graph) • Scenario (graph) (+,+,+,+,-,-) C 16 R 14 (+,+,+,+,+,+) E 18 (+,+,+,+,+,+) E 18 + value recorded - no value (+,+,+,+,-,-) Experience Management within Project Management Processes Experiment – Accordance (interpretation of a path graph) • Solution (textual interpretation) • C 16 (+,+,+,+): A beginner with salary 10-15K was selected for coding the maintenance system E 18 (+,+,+,+,-,-): This led to an estimate of the corresponding activity duration (1.5 – 2 wks) and cost (30 – 40K) R 14 (+,+,+,+,-,-): This also led to anticipating a problem of lack of experience which could be triggered by a change in requirements E 18 (+,+,+,+,+,+): Foreseeing this problem then led to a revision of the estimated activity duration (2 – 3 wks) and cost (50 – 60K) R 14 (+,+,+,+,+,+): This in turn led to an expectation of project delay Experience Management within Project Management Processes Experiment – Accordance (interpretation of a path graph) • Experiment • Groups • Control • Treatment none given case details • Results – average errors • Treatment n = 30; average = 1.9; variance = 1.6 • Inference • The graph represents “experience” in linking decisions; the subjects were require to reproduce this experience Experience Management within Project Management Processes Experiment – Accordance (interpretation of a path graph) • Experiment • Errors – incorrect case interpretation • Risks • Estimates • Candidates 16% 9% 9% • Arc E – R • Other arcs (3) • Candidates 30% 12% 3% • Errors – incorrect arc interpretation/overlook • Interpretation of main errors • Inability to interpret the evolution of the risk case along the path sequence • Inability to interpret the apparent “loop” in the path Experience Management within Project Management Processes Experiment – Convergence (creation of a knowledge focus) • Scenario (text) "The manager of an information systems implementation project is required to select the best candidate for analyzing a human resource management system. He can choose a beginner, someone with experience, or an expert. The decision should be based on knowledge from past projects, concentrating on the trade-off between inexperience risks versus salary costs" Experience Management within Project Management Processes Experiment – Convergence (creation of a knowledge focus) • Five case-based and path-based activities • Define case retrieval input from the decision scenario using “case-by-example” • Retrieve all corresponding cases • Select relevant cases to retrieve paths • Retrieve all path dyads and interpret them • Synthesize cases and path dyads to create a knowledge focus related to the decision scenario • Carry out any feedback required to retrieve other cases or paths Experience Management within Project Management Processes Experiment – Convergence (control – no guidance given) • Four CBR and PBR actions (icons) • Define case retrieval input from the decision scenario using “case-by-example” • Retrieve all corresponding cases • Select relevant cases to retrieve paths • Retrieve all path dyads and interpret them • Press any icon to activate the corresponding action Experience Management within Project Management Processes Experiment – Convergence (treatment – process map given) • Four linked CBR and PBR actions (icons) 1. Define case retrieval input from the decision scenario using “case-by-example” 2. Retrieve all corresponding cases 3. Select relevant cases to retrieve paths 4. Retrieve all path dyads and interpret them • Carry out any feedback required to retrieve other cases or paths Experience Management within Project Management Processes Experiment – Convergence (creation of a knowledge focus) • Experiment • Groups • Control • Treatment carry out any of the activities, in any order carry out all four activities in a given order (process) • Results – average errors (p < 0.01) • Control n = 15; average = - 0.9; variance = 0.5 • Treatment n = 15; average = - 0.5; variance = 0.3 • Inference • The treatment group is better able to synthesize the knowledge retrieved and to relate it to the decision to be made • A systematized process contributes to better performance • Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance Experience Management within Project Management Processes Experiment – Efficiency (total time required to derive knowledge focus) • Experiment • Groups • Control • Treatment carry out any of the activities, in any order carry out all four activities in a given order (process) • Results – average errors (p < 0.05) • Control n = 15; average = 23 minutes; variance = 37 • Treatment n = 15; average = 19 minutes; variance = 21 • Inference • The treatment group is faster in generating the knowledge focus • A systematized process contributes to faster and more uniform performance • Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance Experience Management within Project Management Processes Experiment – Cover (cases) (cases required for knowledge focus) • Experiment • Groups • Control • Treatment carry out any of the activities, in any order carry out all four activities in a given order (process) • Results – average errors (lower bound = 3) (p < 0.05) • Control n = 15; average = + 1.47; variance = 2.6 • Treatment n = 15; average = + 0.40; variance = 1.4 • Inference • The treatment group requires less to find a meaningful input • Systematization gvies more effective and uniform performance • Systematization gives more confidence and thus less scope • Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance Experience Management within Project Management Processes Experiment – Cover (paths) (paths required for knowledge focus) • Experiment • Groups • Control • Treatment carry out any of the activities, in any order carry out all four activities in a given order (process) • Results – average errors (lower bound = 3) • Control n = 15; average = + 3.27; variance = 2.8 • Treatment n = 15; average = + 3.47; variance = 3.3 • Inference • Both groups tend to search the same number of paths • However, the treatment group retreieved these pathns in an organized fashion • Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance Experience Management within Project Management Processes Conclusions • Business process oriented knowledge management (BPOKM) means that knowledge is structured according to process requirements • Experience oriented knowledge management means that knowledge is created and updated through the experience of those applying the knowledge •The project manager looks for suitable candidates (process) and seeks the effects of previous decisions (experience) on time, costs and risks as stored in the knowledge base Experience Management within Project Management Processes Conclusions • Our aim is to create an effective knowledge focus – the right number, identification and interpretation of the retrieved cases and paths •At each step of the process, knowledge focus errors may occur • These have been measured by a uniform metric – a count of the errors incurred • Errors reflect redundant or missing or misinterpreted knowledge elements (cases or paths) as compared to an “ideal” search for the knowledge required Experience Management within Project Management Processes Conclusions • In general we find that systematization of the decision process and the link between process and knowledge required leads to better performance • Our methodology allows us to locate areas and reasons for non-performance, such that the project management decisionmaking process can be improved Experience Management within Project Management Processes To sum up … • Knowledge bases can be structured by adopting a process description as one specific foundation for the schema of the knowledge base. This facilitates access since in a process oriented application the user knows his exact position in the process and can therefore easily navigate the knowledge base. • Since the process description functions as a structuring criterion for a knowledge base, all the relevant information and knowledge for a process step – that goes beyond the capability of a conventional process model – can be captured. (Jablonski, Horn and Schlundt) Experience Management within Project Management Processes To sum up … We hope that our methodology has contributed a useful approach, based on error metrics at each process step, to be able to measure the effectiveness of this knowledge capture German Workshop on Experience Management 2003 Experience Management within Project Management Processes Thank you! CSC Proprietary 3/16/2016 7:59:36 AM 008_fmt_wht 52