Eliciting Expert Judgement
advertisement
Eliciting Expert Judgement Ian James Chief Engineer Electronics & Software Technology Aero Engine Controls February 2009 A Rolls-Royce and Goodrich Corporation joint venture Eliciting Expert Judgement Contents of Lecture Introduction - Speaker & Company Background Elicitation Overview The Expert Elicitation Process Using Expert Information Elicitation Updates Summary 1 Engine Control Systems Take an Aircraft view of an Engine Fuel Control System .... Cockpit Controls Fuel Tanks and Fuel Pumps Engine and Fuel Control System Cathay Pacific Operating Airbus A330 powered by Rolls-Royce Trent 700 Engines Engine Control Systems Engine Control System Rolls-Royce Trent 700 Engine Left View 2 Engine Control Systems Electronic Engine Control - E.E.C. Expert Elicitation Overview Data Characteristics Expert knowledge is defined as a general term encompassing what qualified individuals know with respect to their: Technical Practices - What they do Training - What they have been taught Experience - What they have experienced Expert knowledge can be categorized into Expertise Expert Judgement 3 Expert Elicitation Overview Expert Judgement Expertise refers to the tacit thinking processes used in everyday decision making and is implicitly used in problem solving and decision making Expert judgment, expert estimates, or expert opinion are terms that refer to the contents of the problem— estimates that populate the structure of the problem Estimates, outcomes, predictions, uncertainties, utilities and their corresponding assumptions, heuristics, and conditions are all examples of expert judgment. Expert Elicitation Overview Expert Judgement Is affected by the process used to gather it Has uncertainty (which must be characterized and subsequently analyzed) Is conditioned on various factors (such as question phrasing, information considered, assumptions and problem solving) Can be combined with other data and information. 4 Expert Elicitation Overview History of Elicitation of Expert Judgement The formal elicitation and analysis of expert knowledge dates back to the early 1980s It is rooted in early human cognition studies and the emergence of Probabilistic Risk Assessment (PRA). PRA relied heavily on expert judgment because, for most early problems, ‘hard’ (test, experimental, observational) data were sparse or nonexistent Just as the ‘P’ in PRA implies, these early efforts were based in probability theory. As a construct of risk, probability theory offered a logical mechanism for handling uncertainties in these analyses and studies. Expert Elicitation Overview Eliciting Expert Judgement Data Extracting knowledge in as raw and unbiased a form as possible, according to the way experts think and problem solve has been shown to be most successful An analyst who excludes expert opinion as a source of data denies the subtle uses of expert tacit knowledge in everyday decision making Any analyst who aims to include expert opinion as a source of data faces the difficult task of extracting tacit knowledge from one or more members of the subject area under study The approach is interdisciplinary, there is a shared emphasis on frequent, direct interactions of the expert and interviewer/ analyst, often in face-to-face sessions. 5 Example - Design Changes Data Analysis Airline Operator Records Warranty Claims Repair Records Product Support Database Design Improvements Common Manufacturing Process Similar Component Types Reduced Interconnections Reduced Number of Assemblies Increased Circuit Integration Managing Design Decisions Balancing the Payoff Versus The Risk Where there is Change you will always find Concern How to weigh up the Benefit/Concern for each change Consequence to the end user Cost saving .. Cost of failure When will we know if its ok ? How to affect the decision making gp process Absence of hard data necessitates subjective decision making Are the views: Personal, Political, Blinkered, Ill-advised ? We need expert knowledge that can be relied upon and …. De-risk test & analysis with defined decision points A PROCESS to systematically elicit expert concerns 6 The Expert Elicitation Process The Expert Elicitation Process Initial Launch Meeting Describe elicitation objectives to interested parties Define Base product and changes from Base to new design Select Experts / Interviewees Interview Planning Review relevant Service Data Develop proformas and facilitation aids Carry Out Interviews Individual interviews up to 2 hours maximum Foster an informal, open, no blame environment Collate Concerns Classify Concerns into Root Cause Categories Distribute Processed concerns for comment The Expert Elicitation Process Initial Launch Meeting Define f Base Product Use the group to brainstorm any similarities between the new design and existing products Agree the most appropriate product (s) for comparison Break down the Base product (s) into major functional parts Define Changes from Base to New Design Are there any new functions ? Are any functions no longer required ? Which functions remain similar between designs ? Are there any major technology changes ? Any there any manufacturing process changes ? 7 The Expert Elicitation Process Example – Major Changes Brainstorm Element Changes from Product X General Mechanical Design Changes from Product Y Higher Operating temperature 3 large modules versus 12 smaller modules All Surface Mounted Component versus mixed type on Product X and Y Thermal Design via pillars versus cold wall As Product X Shape of bonding Strap Connector Types not used before Installation angle achieved via mounting bracket Change of Casting Material Aircraft Interface Similar to Product X but radically different communications Lightning Strike / HIRF F l Fl Fuel Flow Nothing like Product Y New design based on product X.. Uses SILS used in place of hybrids Si il to Similar t Product P d t X but b t a novell switching it hi system t h has b been iimplemented l t d Torque Motor Drives As Product X redesign Pressure Measurement Multi Chip Module Housing Mechanical installation change, but electrical similar Similar installation None used. Use of Simple Hybrid Technology Similar installation to Product X but Mounted directly onto gear box Linear Circuits Common Circuit blocks comprising discrete components versus of the shelf functionality The Expert Elicitation Process Initial Launch Meeting Candidates For Elicitation Engineering Consultants Project Designers System Design Engineers Quality Engineers Manufacturing Staff g Staff Purchasing Product Support Representatives Test Engineers Customer Representative Supplier Representative 8 The Expert Elicitation Process Interview Planning Familiarisation with Base Product Service Data Current Issues, Major Trends, Top 10 Reasons for failure, etc Surface Mount Wiring / Interconnect Power Lighting Strike Pressure Plated Thru Hole Confirmed Fault [140] [27] A/C Power [9] Dedicated Generator [70] Filters [3] Connector Pins Other [31] Capacitors [14] Hybrid Failure [4] Capacitors [26] Hybrids [12] Solder Joints [5] Solder Joint [2] PCB [1] Ignition Relay [3] The Expert Elicitation Process Interview Planning How to promote useful discussion during interviews Environment Comfortable Surroundings / Room Layout Availability of Flipchart / Whiteboard Access to Computer Network Tools Summary of Base Product Information Proformas to expedite capture expert’s knowledge Gauge to aid selection of Likelihood 9 The Expert Elicitation Process Example – Elicitation Concern Sheet Product Design Stage Functional Block Change Concern Description Mitigating action Likelihood of Occurrence The Expert Elicitation Process Example – Concern Likelihood Scale Likelihood Scale Concern: …………………………. Engineer: ………………………… Certain Never Very y Unlikely y 0 Unlikely Fairly y Unlikely y Fairly y Likely 0.5 Lik ly Likel Very y Likely y 1 What is the chance the concern may lead to a fault in use if no corrective action is taken? 10 The Expert Elicitation Process Carry Out Interviews Focus on difference between new design and base product Develop the list brainstormed during Launch Meeting Understand motivation for each design change For each difference discuss the impact upon reliability Performance improvement or concern Can Test or Analysis evaluate the concern ? What is the ultimate mitigation action ? Is the work planned ? What is the probability of a service incident? The Expert Elicitation Process Example – Elicitation Summary: Individual Expert Concern 1 2 3 4 5 6 7 8 Sub-system General Change Vibration environment Concern Not on Pump so potential to be better or worse depending on mounting bracket arrangement less corrosion Case Material now through hardened xyz IMPROVEMENT was 1234 tapping into harder more brittle material resulting in cracking and subsequent leaking or jamming Diodes Higher Operating Temperature Data Sheet does not Specify max. New Component Supplier No experience with new package type EIDs Reduced life so higher fail rate of EIDs SCM New Package Type Little experience with soldering parts; relies on new manufacturing process Seal Location of Fire Seal Increased chance of Seal compression set causing leakage Mitigating action Likelihood Ensure adequate bracket 0.1 design RET and Qual Test Supplier Audits 0.1 0.5 Design Evaluation Test 0.5 0.5 RET and Qual Test 0.5 Prototype runs, SPC, RET and design evaluation RET and Qual Test 0.5 0.5 11 The Expert Elicitation Process Collate Concerns Combine Identical Concerns It is likely that different experts will raise identical concerns Ensure there is no double counting Categorise each Concern into a generic fault class 5-10 Classes will ensure a reasonable root cause level See next page Review Likelihood of Occurrence Review range of likelihood for concerns from multiple experts Review the use Max, Min or Average, or combinations Distribute Collated Comments for Review Seek agreement / approval from individual experts Configuration Control: Formally issue concerns & comments The Expert Elicitation Process Manufacture and Assembly Root Cause Classification Build Fault Soldering Process Component Parts Wear-Out Supplier Quality Environment Handling Damage Extreme Operation Design Margin Tolerances Component Selection 12 The Expert Elicitation Process Example – Elicitation Summary: Collated Concern Eng. Ref 1 7.11, 3.3 3.4, 6.2 2 3.5, 6.3, 8.3 3 3.6, 1.7, 13.7 4 5 3.7, 3 7 7 7.1 1 13.6 6 3.8, 7.2 7 3.9, 13.4, 10.7 8 13.8, 7.3 Sub-system General Change Vibration environment Concern Not on Pump so potentail to be better or worse depending on mounting bracket arrangement Case Material now through hardened xyz less corrosion was 1234 IMPROVEMENT tapping into harder more brittle material resulting in cracking and subsequent leaking or jamming Diodes Higher Operating Temperature Data Sheet does not Specify max. New Component Supplier No experience with new package type EIDs Reduced life so higher fail rate of EIDs SCM New Package Type Little experience with soldering parts; relies on new manufacturing process Seal Location of Fire Seal Increased chance of Seal compression set causing leakage Mitigating action Likelihood Ensure adequate bracket 0.1, 0.2 design RET and Qual Test Fault category Environment 0.1, 0.25, 0.2 Build Fault 0.5, 0.5, 0.5 Design Margin Design Evaluation Test 0.5, 0 5 0 0.75 75 0.5 Supplier Quality Wear-Out RET and Qual Test 0.5, 0.3 Environment Prototype runs, SPC, RET and design evaluation RET and Qual Test 0.5, 0.5, 0.25 Soldering Process Supplier Audits 0.5, 0.75 Design Margin Using Expert Information Reliability Assessment Estimates reliability of a variant design at specified times in product lifecycle using appropriate mathematical models Aid engineering understanding of performance and to help inform downstream processes that will support enhancement Alternative scenarios can be investigated through ‘what if’ analysis so that the impact on reliability can be estimated Provides a tracking system to analyse / record how reliability evolves. 13 Using Expert Information New design Similar design Elicit engineering judgement about failure classes of concern Changes Event history data for relevant failures Model Flag failure classes of concern Estimate reliability of new design Using Expert Information Event History Data Group Service failure data into Root Cause Categories Generate a Survival Function, R(t), for each Fault Category The distribution then represents all concerns in this category Nonparametric Survival Plot for Wear Out Nonparametric Survival Plot for Supplier Quality Kaplan-Meier Method Kaplan-Meier Method Censoring Column in Wear Out Censor Censoring Column in Supplier Censor 1.00 MTTF Median IQR 19318 * 0.000000 1.00 MTTF 0.99 Median IQR 20068 * 0.000000 0.98 Probability Probability 0.97 0.95 0.96 0.95 0.94 0.93 0.92 0.90 0.91 0.90 0 10000 Time to Failure 20000 0 10000 20000 Time to Failure 14 Using Expert Information Reliability Estimate for New System Design Failure Class Expert Judgement Design Component NC ND Event Data Build NB RD(t) RC(t) RB(t) Using Expert Information Concern Elicitation Interviews with engineering experts generates a list of potential concerns with the new design. For each concern a likelihood of failure is recorded. Concerns are then categorised into root-cause classes. Supplier Quality Soldering Process Build Fault Environment Wear Out Design Marginality Service Data P1 Failure Distribution Review Develop Reliability distribution for each major category of concern. Design Marginality Wear Out Environment Build Fault Soldering Process Supplier Quality P2 P3 P4 P5 P6 R1(t) R2(t) RSystem(t) R3(t) R4(t) R5(t) R6(t) Statistical Model 15 Using Expert Information 1 0.9 0.8 0.7 0.6 0.5 0.4 Soldering Process 0.3 Supplier Quality Build Fault Wear Out Environment 0.2 Factored Inherent Design Marginality 0.1 System Reliability 0 0 5000 10000 15000 20000 25000 30000 Elicitation Updating Concern Elicitation Test & Analysis RSystem(t) Failure Distribution Review Statistical Model 16 Elicitation Updating Updating Expert Judgment Following a period off action which may affect ff Concerns Arrange interviews with a subset of experts Review Concerns recorded during initial interviews Has test or analysis activity successfully mitigated any issues ? To what extent has the likelihood of occurrence changed ? Are any new test or analysis activities planned ? Capture changes not recorded during initial interviews Develop action plan to minimise likelihood of occurrence Elicitation Updating Soldering Process 16 Updating Expert Judgment 14 12 10 Design Marginality Build Faults 8 Review changes in Opinion Prioritise Actions Focus Attention 6 4 2 Initial Assessm ent 0 Assessm ent #2 Environment Supplier Quality Wear Out 18 16 14 Initial Assessment Assessment #2 12 10 8 6 4 2 0 Soldering Process Build Faults Supplier Quality Root Wear Out Environment Design Marginality C a us e 17 Reliability Assessment Product X Reliability Assessment 1 0.9 0.8 P Probability 0.7 0.6 0.5 0.4 0.3 Worst Case Assessment 0.2 Target [100k] 0.1 0 0 5000 10000 15000 20000 25000 30000 Operating Hours Reliability Assessment - Update Product X Reliability Assessment 1 0.9 0.8 Prrobability 0.7 0.6 0.5 04 0.4 0.3 Worst Case Assessment 0.2 Target [100k] 0.1 0 0 5000 10000 15000 20000 25000 30000 Operating Hours 18 Summary Summary Expert Judgment is a valuable piece of information The extraction of Expert knowledge may be very difficult But …. its exclusion can undermine the decision making process Thorough Planning is Required for Expert Elicitation General Logistics:- Room, Time, Expert Selection, Facilitation Aids Scope of Elicitation:- Selection of Prior Products, Design Changes Processing the Elicitation Output Correlation of Concerns and Root Cause Classification Practical Usage Reliability Modelling Test Development Continuous Updating 19
