10
Section
WARWICK MANUFACTURING GROUP
Product Excellence using 6 Sigma (PEUSS)
Elicitation of
expert judgement
Warwick Manufacturing Group
ELICITATION OF EXPERT JUDGEMENT
Contents
1
Introduction
1
2
The elicitation and use of expert knowledge
2
3
An approach to eliciting expert knowledge for
reliability analyses - REMM
2
4
Summary and conclusions
30
5
References
30
Copyright © 2007 University of Warwick
Warwick Manufacturing Group
Elicitation of Expert Judgement
Page 1
ELICITATION OF EXPERT JUDGEMENT
1 Introduction
Research on the elicitation of expert knowledge originates in the fields of psychology,
decision analysis, probabilistic risk assessment, and knowledge acquisition. Examples of
works in these fields include [1–6], respectively, and uncertainties in these early works were
primarily characterized using probability theory. However, as the understanding of human
cognition advanced, more formal elicitation methods were developed and the need arose for
more flexible theories for uncertainties to properly capture expert knowledge [5]. Various
forms of expert knowledge from the explicit judgment estimates of unknown quantities to the
more tacit, implicit use of expertise, are present in handling uncertainties for a problem.
Expert knowledge is defined as a general term encompassing what qualified individuals know
with respect to their technical practices, training and experience. It is critical to understand
that the term knowledge is contextual, referring to knowledge gained in practice rather than
knowledge in the abstract.
Expert knowledge, in turn, can be categorized into expertise and expert judgment.
Expertise refers to the tacit thinking processes used in everyday decision making and is
implicitly used in problem solving and decision making. The definition and structure of the
problem, its representation and scope, determining relevant information, information
organization and flow through the problem are all aspects of the tacit knowledge defined as
expertise.
In contrast, expert judgment, expert estimates, or expert opinion are terms that refer to the
contents of the problem—estimates that populate the structure of the problem. Explicitly
stated estimates, outcomes, predictions, uncertainties, utilities and their corresponding
assumptions, heuristics, and conditions are all examples of expert judgment.
While expert judgment can be used as a complement to test, experimental or observational
data, it is often the sole source of information representing the current state of knowledge.
Like other sources of data, expert judgment:
•
•
•
•
Is affected by the process used to gather it,
Has uncertainty (which must be characterized and subsequently analyzed),
Is conditioned on various factors (such as question phrasing, information considered,
assumptions and problem solving), and
Can be combined with other data and information.
Warwick Manufacturing Group
Page 2
2 The elicitation and use of expert knowledge
The formal elicitation and analysis of expert knowledge dates back to the early 1980s, and is
specifically rooted in early human cognition studies and the emergence of Probabilistic Risk
Assessment (PRA). PRA relied heavily on expert judgment because, for most early problems,
‘hard’ (test, experimental, observational) data were sparse or nonexistent. Just as the ‘P’ in
PRA implies, these early efforts were based in probability theory. As a construct of risk,
probability theory offered a logical mechanism for handling uncertainties in these analyses
and studies. Moreover, the experts involved in these studies were scientists and engineers
who were used to thinking in terms of probability.
However, as more became known about human cognition, experience has indicated the
limitations of probability theory in handling the different kinds of uncertainties involved in
complex problems [5].
Not all experts (even in technical fields) think in terms of probability. Even those who do may
not consistently think according to its rules or axioms of use. No one theory is therefore
universally applicable for characterizing all uncertainties in all problems.
Extracting knowledge in as raw (or perhaps pure) and unbiased a form as possible, according
to the way experts think and problem solve has been shown to be most successful[5].
An analyst who excludes expert opinion as a source of data denies the subtle uses of expert
tacit knowledge in everyday decision making, such as narrowing the focus of a problem,
making assumptions, and deciding what data sources are relevant.
Any analyst who aims to include expert opinion as a source of data faces the difficult task of
extracting tacit knowledge from one or more members of the subject area under study.
However, the past decade has seen the development of a portfolio of formal elicitation
methods drawn from various disciplines: for example, cognitive psychology, anthropology,
statistics, decision analysis, and knowledge acquisition. Although the approach is
interdisciplinary, there is a shared emphasis on frequent, direct interactions of the expert and
interviewer/ analyst, often in face-to-face sessions.
3 An approach to eliciting expert knowledge for reliability analyses REMM
3.1
Purpose of the REMM Model
The REMM (Reliability Methodology and Modelling) model aims to estimate reliability of a
new design at specified times in the product lifecycle to aid engineering understanding of
performance and to help inform downstream processes that will support enhancement. The
model will support ‘what if’ analysis. This means that alternative scenarios can be
investigated and the impact on reliability estimated. Therefore the REMM statistical model
Warwick Manufacturing Group
Page 3
provides a tracking system to help analyse how reliability will and does evolve throughout
the lifecycle by integrating the numerical estimates with the engineering understanding of
reliability.
The REMM statistical model requires engineering concerns about potential faults in the item
of equipment to be identified and estimates when they are likely to occur as failures under
different scenarios. For example, assuming no action is taken to mitigate the faults, or
assuming a particular action is to be adopted.
The model will have most impact during design and development when it should help
inform cost-effective decisions to enhance reliability. For example, during design reliability
estimates will be provided for the current state of the item and the need for enhancement
activities can be flagged. By exploring the likely impacts of scheduled downstream actions
on reliability through ‘what if’ analysis, it will be possible to forecast future reliability and to
assess the added-value of alternative actions. Therefore the model has the capability of
providing in-service predictions. However, like all forecasts, such predictions need to be used
with extreme caution.
Therefore the REMM statistical model is essentially a growth model for use in design and
development when high-level estimates of reliability, including the uncertainties in the
estimates of reliability, are sufficient to support decision-making. The REMM statistical
model should be regarded as a complement, but not necessarily a replacement, to
conventional systems reliability models, which may be used to support more detailed analysis
of, for example, systems configuration and operational scenarios.
The following notes will describe the process for collecting the data for input to the REMM
model. The principle purpose of these notes is to give an example of collecting expert
judgement data in a structured manner. The notes do not give the mathematical formulation
of the model but concentrate on the data input and output from the model.
3.2
Types of Items that can be Modelled
The implementation of the REMM statistical model is suitable only for repaired or nonrepaired items that have an associated failure, or event, time. This failure time may be
expressed in terms of, for example, operating hours, flying hours and cycles. The model
should be applicable with any units of measurement specified and the results will be output in
the same units.
The model requires appropriate data to be input and will support analysis at different levels of
system indenture and by classification of faults. Both nonparametric and parametric
estimates of reliability can be computed. The former requires raw failure time data to be
input, while the latter requires the parameters of the appropriate failure time distribution to be
input.
As usual, the quality of the estimates will be influenced by the quality of the input data. If
good input quality data is not available, then compromises might be made to adjust data for
Warwick Manufacturing Group
Page 4
comparative purposes. However the consequences of such action will be that more uncertainty
will be introduced into the modelling process and less confidence can be attached to any
estimates calculated.
In principle, the REMM statistical model is suitable for use with designs that are classed as
variant. By variant design, the model assumes that the new design is an evolutionary change
of an existing design and hence historical event data for the similar item can be used to
characterise the base reliability function.
3.3
REMM Statistical Model Data Inputs and Outputs
Two basic types of input are required to populate the model:
•
•
Engineering concerns with the new design and an estimate of their probability of
occurrence in service assuming no actions taken
Reliability profile of the engineering concern or fault class to which it belongs
The data will come from two sources – structured engineering judgement and, in the case of
variant designs only, historical event data.
The basic output from the REMM statistical model will be the estimated reliability function,
also known as the survival function. This provides an estimate of the probability of
surviving until a specified time.
Additional output will be a list of engineering concerns that is provided at the end of an
elicitation exercise although the prior distribution computed from this data will be an input to
the model.
3.4
Levels of Equipment Indenture, Fault Classes and Correction Strategies
The REMM model can be applied to items of equipment at the level of indenture required for
reliability analysis; for example, system, LRU, sub-assembly. All modelling will be
conditional on the definition of the item boundary.
In addition, the model can be applied at different levels of fault class aggregation. A fault
class is classification of the type of concern, for example, design, build, component, supplier
etc. These will be specific to each application and are defined by investigating the data.
The fault classes should be defined to be consistent with those used within the company and
considered appropriate for analysis. The model allows the fault classes to be customised for
the each application.
Warwick Manufacturing Group
Page 5
3.5
Use of the REMM Model During the Product Lifecycle
The model can be used at any stage during the product lifecycle. The stages of interest
should be agreed with the design team.
During concept design the model will provide a rough cut analysis of potential faults and
should give insight into areas of particular risk associated with a design.
During detailed design the model will use judgement data collected from the design team and
all other relevant engineers and, if appropriate will be combined with appropriately screened
and detailed historical event data on the base item selected. It is during this stage that the
model is likely to have most impact as it should support a detailed evaluation of design
options.
During development, production and use the model can be revised in the light of data
collected about the item. The times to realisation of particular faults can be revised in the
light of the data collected and the engineering review may lead to a revision of actual and the
addition of new faults.
At each implementation of the model a record of the input and output data can be made for
use in other applications.
3.6
The Role of the Modelling Facilitator
A relevant person should take responsibility for managing the modelling process.
The facilitator should possess a good engineering understanding of the designs being
evaluated, have a basic understanding of the model and be a good communicator.
This is because the facilitator will need to:
•
•
•
•
•
•
•
interact with the engineers and managers responsible for designing the item of
equipment
manage the process of eliciting judgement from relevant engineers about the reliability
of the new design
lead the screening process to identify relevant historical event data for the base design
understand the basic ideas underpinning the model
run the modelling software and assist in the interpretation of the results
assist in the specification and interpretation of what if analysis, if required
provide evidence that will contribute to the reliability case
The facilitator is responsible for ensuring the data integrity, the model set-up and the
interpretation of the results within the context of the reliability requirements for the design
under consideration.
Warwick Manufacturing Group
Page 6
3.7
REMM Model for Variant Design
The diagram below shows a flowchart of the modeling process.
Warwick Manufacturing Group
Page 7
Variant design –
Event data
available for
base item
New
project?
yes
Input project
details
no
Update
project
details
Definition of
data structures
Preparation of
event data
Elicitation of
engineering
concerns
Modelling
Sensitivity
analysis
Exit model
Warwick Manufacturing Group
Follow-up
elicitation
Page 8
3.8
Input Project Details
If the model is to be run for the first time with a variant design then a new project should be
defined. This provides a record of which item is being modelled, any relevant description and
names the project manager to whom results are to be reported.
3.9
Definition of Data Structures
The data structures to be used in modelling are defined as:
•
•
•
•
•
Level of modelling
Fault correction strategy
Model for in-service reliability
Prior distribution for the number of engineering
3.10 Preparation of Event Data
3.10.1 Identify Base Item, Define, Select and Prepare Relevant Events
A record should be made of the similar design that is identified the base item and the data
source used.
The selection and preparation of relevant events is essential. The goal is to ensure that
those events experienced in the base design that remain representative of the events that can
occur with the new design should be identified. Those events for which there is evidence that
a corrective action has removed the chance of a re-occurrence within the new design should
be excluded from analysis. Note that events will usually correspond to confirmed failures.
However, if appropriate, different forms of events such as removals may be used.
The definitions of relevance and other assumptions made in cleansing historical event data for
the similar item should be noted and recorded. This will inform the interpretation of
subsequent analysis.
If the analysis is to be conducted at the fault class level, then fault classes must be defined.
For the nonparametric case, the raw event data as well as the population size should be
input. The population size corresponds to the number of items in the fleet and is required to
ensure that the impact of items that did not experience an event is taken into consideration in
modelling.
For each item in the population, the total operating hours (or equivalently, total flying hours,
cycles as appropriate) should be recorded. For those items in the population which have
experienced an event, the accumulated operating times to each event should be recorded.
Note that the number of events per item may vary. For most this may be 0 or 1, however as
many events as are relevant may be recorded by simply adding the next time in the
consecutive column.
For the parametric case, the failure time distribution (choices are exponential, Weibull and
lognormal) is required and the appropriate assumed or estimated parameters. For example,
Warwick Manufacturing Group
Page 9
for the exponential distribution the user may either input the hazard (instantaneous failure
rate) or the mean time between failures (MTBF) or the mean time to first failure (MTTF). For
the Weibull and the lognormal, the user should input the two parameters. For the Weibull this
is the shape and scale parameter, for the lognormal this is the mean and standard deviation of
the logged failure times.
3.10.2 Estimated Reliability Function for Base Design
The estimated reliability function for the base design given the model assumptions is made
and the event data entered can be drawn.
An example plot of the superimposed non-parametric reliability functions corresponding to
three fault classes is shown below. This display shows that fault classes 1 and 2 are the major
reliability drivers since they show the greatest reduction in the proportion surviving the range
of operational hours considered.
Warwick Manufacturing Group
Page 10
3.11
Elicitation of Engineering Concerns
3.11.1 Process flow
The following diagram illustrates the elicitation process:
Warwick Manufacturing Group
Page 11
Elicitation of
engineering
concerns
Plan
elicitation
exercise
Conduct
briefing
session
Conduct
individual
interviews
Review and
categorise
engineering
concerns
Conduct
follow-up
group session
Tabulate
verified
concerns and
probabilities
Warwick Manufacturing Group
Page 12
3.11.2 Plan Elicitation Exercise
Plan
elicitation
exercise
Define
relevant
engineering
domains
Identify
appropriate
experts to provide
domain coverage
Record details
about engineering
experts
Plan interview
schedule
Prepare
interview
packs
Confirm
project plan
and schedule
Warwick Manufacturing Group
Page 13
Define, Identify and Record Relevant Engineering Experts and Knowledge Domains
•
Specify the domains of expertise required. These will relate to, for example, the
functionality, parts, technology, manufacture and use of the design and its variant.
•
Assess the importance of each domain for the design. The default is to assume each
domain is equally important.
•
List all potential engineers and their domain of expertise.
•
Select (at least) one engineer from each domain to ensure representative coverage of
knowledge about the design.
•
The engineer is to be selected as an expert within a specified domain expert should be
the engineer who is judged by the facilitator and accepted by his peers as being the
person with the most expertise and experience relating to the design.
The number of engineers per domain may be increased in certain domains in line with its
assumed importance if required and if this is deemed to accrue benefits in relation to costs.
It is acceptable, and even preferred; to select engineers with overlapping knowledge as this
provides better coverage of interfaces between domains.
Failing to select engineers from a domain should be avoided.
A record of the facilitator’s reasoning should be maintained to justify the design of the
elicitation exercises.
For each selected engineer, complete an expert record of the reasons for selection.
Plan Interview Schedule, Prepare Interview Packs and Confirm
It is recommended that interviews be held with those engineers with high-level domains of
knowledge (e.g. systems engineer, lead engineer) first. These engineers possess over-arching
knowledge of the design and so can help the facilitator understand the broad nature of design
changes and potential concerns. This approach provides a better basis for interviewing those
experts with more specialized knowledge about the design and hence more detailed concerns
about specific aspects of design. Next the designers (e.g. electronic, mechanical, software)
should be interviewed followed finally by those with specialist knowledge (e.g. thermal,
vibration, manufacture, components).
Warwick Manufacturing Group
Page 14
The interview schedule may be arranged at the briefing workshop, if appropriate.
An interview pack should be prepared. For example, this may contain
•
•
•
•
Photos or diagrams of the item to be assessed.
A1 sheets (around 6-10 per interview), marker pens, post-its for mapping
List of fault classes, including descriptions
Probability (and severity) scales to be used, handouts for probability scale to be
used for each concern, probability scale overlay
3.11.3 Conduct Briefing Session
It is important to brief experts about the elicitation process and how the information they
provide will be used prior to the first elicitation exercise.
This briefing aims to inform and condition the experts so that honest and full accounts of
potential concerns, and their probabilities, are provided.
The preferred briefing method is to hold a pre-elicitation workshop to provide engineers with
the required information, allow discussion around the process and its implications, arrange
convenient times for interviews.
The presentation should cover the following topics:
•
•
•
•
•
*
Elicitation process within REMM model – what and why?
Individual interview requirements
Some definitions* for interview
Elicitation process example
Output of elicitation
Note - the definitions are:
•
•
•
•
Failure: the termination of the ability of an item to perform an required function
Concern: an issue of concern with any aspect of the design
Sub-concern: more details about the concern to capture underlying reasons for raising
issue
Fault class: mutually exclusive categories of root causes of failures as specified in
elicitation requirements
It is important to ensure that experts do not perceive the elicitation of concerns as a judgement
of their own performance – this is why ‘concern’ is used rather than, say, fault. It is
recommended the facilitator explains this and adapts the definitions to suit the understanding
of the audience.
3.11.4 Conduct Individual Interviews
The diagram below shows the process for conducting individual interviews.
Warwick Manufacturing Group
Page 15
Conduct
individual
interviews
Establish whether
design is to be
decomposed
by
changes
or
function
Build concern map
for
each
design
decomposition
Review all concern
maps and agree
concerns
and
probabilities
Record concern
description and
b bilit
For variant designs, the norm will be to conduct the elicitation by focussing upon the changes
between the base and the new design.
Throughout, the facilitator should be vigilant in identifying any biases exhibited by the
engineer. For example, engineers who are consistently optimistic/pessimistic, engineers who
anchor chances of failure on historical data rather than gut reaction, engineers who are
unwilling to share information. Such biases can be detected by observing, for example, body
language, behaviour and wording of responses. If bias is suspected then the facilitator has
two choices. Either explore the source of bias through discussion and try to overcome, or to
disqualify the engineer as an expert if it is thought that the judgement being provided is not
constructive.
Warwick Manufacturing Group
Page 16
Build Concern Map
The individual interview involves the facilitator getting the engineer to identify the changes
from the previous design(s), to identify their concerns and to provide some information on the
chance that each would occur as a failure if no corrective action is taken. A consistency
check of the engineer’s responses should be made and reasons for any differences should be
explored, the process partially repeated until consistency is reached.
In preparation for each interview the facilitator should post all A3 sheets onto wall, white
board etc to form a master worksheet and label areas respectively as shown below.
Chang
The
Fault
Concern
Sub(Master worksheet)
facilitator should open the interview by summarising the process and its purpose, and asking
the engineer if he has any questions about the process or any aspect of the briefing workshop.
This should relax the interviewee and interviewer before the formal process begins.
The facilitator asks the engineer to identify the relevant design changes to the expert, discuss
the high level reasons for them and implications of the changes and note each one in turn.
As a further check the facilitator should ask the engineer about other related aspects of the
design that may affect reliability. In particular, pre-existing design weaknesses from parent
design, perhaps noted from lessons learnt, and novel features that are specific to the new
design.
The facilitator must tackle each identified change, novel feature and weakness in turn to
identify all associated concerns.
For a given change, the facilitator should ask the engineer to identify any issues of concern
and to offer reasons behind each concern. This path of reasoning should be built up within
the sub-concerns section of the worksheet.
Warwick Manufacturing Group
Page 17
Each concern should be recorded in the ‘Concerns’ section of the master worksheet as
bubbles containing a description of the concern. The facilitator must ensure that the engineer
agrees with the description noted for each concern.
The facilitator must judge whether or not the concern is an accurate representation of an
aspect of engineering that could lead directly to failure. For example, an engineer may
provide some concerns that are not specific enough in detail to relate to a specific root cause
of failure. To explore concerns and clarify understanding, the facilitator should invite the
engineer to explain why and how the concern might manifest itself as a failure in use. This
reasoning behind the concerns are recorded as ‘sub-concerns’ on the worksheet and linked to
the concern to which they refer.
This process may be iterative until both facilitator and engineer are comfortable that the
concerns recorded for the given change are ‘potential faults’ within the design.
The facilitator should repeat this process for all changes. Concerns with reasons should be
recorded in the same way. Note that any reason can be mapped to any number of concerns
and concerns may relate to more than one change.
Once all changes have been exhausted, the facilitator should ask the engineer to identify any
other concerns that they have not yet identified.
The facilitator will also have knowledge of the concerns that may be transferred from the
previous designs and if they have not been raised then the facilitator should explore these
issues further with the relevant engineer.
To close this stage of the process, the facilitator should review all concerns in turn and ask the
engineer to allocate a fault class by referencing the list provided. The fault class should be
recorded above the concerns and each concern realised as such must be mapped to the
associated fault class bubble.
Warwick Manufacturing Group
Page 18
An example of a completed map is shown below.
Changes
Design Change
1
Class
Class 1
Class 2
Concern 1
Concerns
Sub -Concerns
Concern 2
Reasoning
Reasoning
Experts
initial
concern
Other Risk
Result 1
Reasoning
Result 2
Failure 1
Result 3
Failure 2
Result 4
Failure 3
Reasoning
Optional – Prompting Questions
The above lays out the generic process, however from experience it may be useful to have
questions prepared to prompt engineers and to engage them in discussion. Selections of
question that have been used include the following.
•
Design General
1. Is the design more complex and is this a concern?
2. Is there more integration and therefore fewer solder joints?
3. Has a better design review process been introduced giving greater confidence?
4. Has more use been made of lessons learnt and therefore some problems eliminated?
•
Electrical performance
1.Are there novel features to the design which could be a reliability risk?
2. Is there any circuit function marginality that is a concern?
3. Are there any derating concerns or improvements?
4. Are there any possible concerns due to external influences?
4.1 EMC
4.2 Lightning strike
Warwick Manufacturing Group
Page 19
4.3 Radiation
5. How good is the fault coverage (testability) - is there any risk of unidentified faults
getting into delivered product?
•
Reliability
1.Are there any components in the design that are a concern for long term reliability?
2. Are there any suppliers that give the same concern?
3. Are any components to be used that will have a connection/joint that gives rise to
long term reliability concerns?
•
Environmental performance
1. Are you concerned that any components may be vulnerable to the known
application environment?
1.1
temperature extremes, cycling or rate of change
1.2
vibration or shock
1.3
humidity
1.4
dormant storage
2. Could any devices be incompatible with the board proposed for long term use?
•
Manufacturing issues
1. Do you have any concerns with the assembly process to be used?
2. Are there any layout concerns - difficult components - large size - sensitive to
temperature solvents anything else?
3. Is increased automation to be used (more consistency)?
4. Is better process control to be employed?
5. Has key process capability improved or got worse?
6. Is operator skill important to any assembly and is the maintenance of this a
concern?
7. Is there any concern that the screening process could cause undetected damage?
8. Has there been any change in Manufacturing environment or provider that is a
concern for reliability?
Review Maps and Agree Probabilities
The facilitator should ask the engineer to assess the chance a concern might result in a fault in
use, assuming no corrective action is taken.
The model requires probabilities to be input in the range 0 to 1, where 0 implies the concern
will never be realised as a fault, while 1 implies it will always be realised as a fault. The
closer to 1, the greater the probability that the concern will be realised as a fault.
Warwick Manufacturing Group
Page 20
A template for noting the probabilities (likelihoods) is given below. Note that this template
can be adapted to align with company standards, so that literal descriptions of probability
ranges that may be in use and hence familiar to the expert.
11 Lk lh
d
Fairly
Very
Unlikely
Unlikely
Unlikely
Don’t
Know
Fairly
Likely
Likel
Very
Likely
Never
C is taken?
What is the chance the concern may lead
to a fault in use if no corrective 2action
50/50
The engineer should mark a point on the scale with ‘x’ that best represents his understanding
of the chance each concern will be realized as a fault in use given no change to the design.
The engineer may also mark a range on the scale to indicate his uncertainty in the judgement.
A separate likelihood scale should be used for each concern to reduce the possibility of the
engineer anchoring on the initial value.
Finally, the experts should be asked to review and verify that the probabilities recorded as a
correct representation of their belief that the concern may be realised as a fault in operation.
To close the interview, the facilitator should review data gathered, ask engineer if he has any
additional comments on data or process, explain the next stage (collate into report for review
and follow-up group session) and thank engineer for sharing views.
Record Concern Description and Probability
At the end of an interview the facilitator should review the data collected and ensure all
records are complete and legible.
Warwick Manufacturing Group
Page 21
At the same time the facilitator should note any issues that arose in the course of the
interview. For example, problems with expert bias or changes required to customise process
better for context.
3.11.5 Review and Categorise Engineering Concerns
The following flowchart illustrates the process flow for categorising engineering concerns.
Review
categorise
engineering
concerns
and
Review
all
concerns from all
experts
Identify concerns
that are unique or
related
Categorise concerns
as
independent,
correlated
or
identical
Prepare group session to
reach agreement about
concerns for model
Warwick Manufacturing Group
Page 22
Prior to the group session it is important to review all engineering concerns to assess which
are unique to an individual engineer and which are common to two or more engineers. Such
common faults might arise due to overlapping domains of knowledge due to reasons such as
job sharing, job sequencing or the compatibility and overlap of certain functions. The
common faults must be identified and managed so that they are not double counted in
analysis.
At this stage the facilitator must make a judgement about which concerns are related based on
the evidence collected during the interviews. The facilitator may want to distinguish between
concerns that are identical between engineers and those that are related but not necessarily
referring to the same fault.
The three classes that concerns may be allocated are:
Category
Rules
Independent
Engineering concern is unique to individual expert
Identical
Same engineering concern is shared by more than one expert
Correlated
Engineers share concerns over aspects of the systems, the result of an
engineers concern being realised will increase/decrease likelihood of
another engineers concerns but not preclude or necessitate other
engineers concerns being realized
It is recommend that the group session involves all engineering experts so that the
facilitator’s treatment of concerns can be verified, or corrected, as appropriate, and the
common concerns can be explored in more detail so that probabilities values, or ranges, are
agreed.
A summary report of all the data collected may be prepared and distributed to the engineers.
This will be a compilation of the summary tables of data and a supporting description.
The following are required for the group session:
• Introductory presentation and briefing example
• Copies of original data for reference
• A3 sheets (one per grouping of concerns)
• Coloured pens (one per expert and facilitator)
• A5 likelihood sheets (at least 10 per expert) as used in individual probability
elicitation
The presentation should be prepared to focus discussion at the group session. It should cover:
•
•
•
•
A summary of who interviewed about which aspect of design
Summary of overall results
Check of data correctness and ability to update results real-time
Agree on concerns to be discussed in this session
Warwick Manufacturing Group
Page 23
•
Statement of what data will be feedback to the design team and how it will be used.
3.11.6 Conduct Follow-up Group Session
The following flowchart illustrates the process of following up the results from the individual
elicitation interviews; this is primarily a consensus meeting:
Conduct
follow-up
group
session
Explain
purpose and
goals
of
session
Review all
correlated
and
identical
concerns
Agree concern
description and
(range of)
Note disagreements
between concerns and
probabilities to inform
sensitivity analysis
Note agreed
concerns and
probabilities
Revise records of
concerns
and
probabilities to be
input to model
Warwick Manufacturing Group
Page 24
If one large group session is considered appropriate then the facilitator should follow the
presentation to achieve:
•
•
•
•
A review and agreement that all individual concerns are correctly recorded
A note of all unique (or independent) concerns and their probabilities
A discussion around common concerns to negotiate wording and probabilities.
Description of how the data will be used and feedback to the design team.
Note there is no requirement to arrive at a single probability value. If there are differing
perspectives then these should be noted as they can be used to estimate the uncertainty in the
judgements about particular concerns.
The group discussions also provide a chance to explore any uncertainties in terms of the
concerns, their descriptions, and their allocation to fault classes as well as their probability
values. The facilitator should allow these uncertainties to be examined and to note the
outcomes as this can help scope different scenarios for examination in REMM modelling
‘what if ‘ analysis.
The summary table collated by the facilitator will be presented back to all engineers at the
group session for verification and further discussion of the common issues. The output of the
session will be a record of the agreed likelihoods and any comments about uncertainties.
Concern
Fault Concern Number of Likelihood Note
description Class
occurrences range
Number
e.g.
C
Textual
description
2.5, 4.2,
4
0.45, 0.33, Identical
0.60, 0.55
e.g.
B
Textual
description
3.3, 5.2
2
0.80, 0.95
Correlated?
e.g.
A
Textual
description
1.1
1
0.90
Independent
5.1, 6.2
Agreed
Comments
likelihood
Post the group session, the agreed concerns and probabilities should be updated in
preparation for modelling.
Warwick Manufacturing Group
Page 25
3.12 Modelling
The flowchart below illustrates the modelling process after all the data has been collected:
Modelling
Confirm
model
options
Run model
Select
preferred
output
displays
Save
results, if
required
Interpret
findings
Now that all the input data has been entered, the model can be run.
shown below.
Warwick Manufacturing Group
Two typical plots are
Page 26
On the left hand side we have the estimated reliability functions for the three fault classes
for the new design. These are the same data for which the base data were shown earlier. The
plot on the right shows the aggregated reliability function for the new item, which is naturally
lower than the functions for each class and the main drivers for unreliability remain fault
classes 2 and 3.
It is important that the profiles displayed should be assessed to ensure that they are credible
in relation to the engineering understanding of the item and are not just statistics taken at face
value.
3.13
Sensitivity Analysis
There are two routes through which sensitivity analysis may be conducted. The first to
explore uncertainties in any of the concerns or fault classes that have, for example, varying
probabilities attributed by different types of experts. This type of analysis provides insight
into the best and worst case scenarios for reliability under different perspectives. The second
is to examine the anticipated impact of future design changes and corrective actions to
enhance reliability. This type of analysis provides a prediction of future reliability. As
usual such forecasting should be treated with care, as there will be accumulated uncertainty in
the input data
Warwick Manufacturing Group
Page 27
Sensitivity
analysis
.
Review
options
presented
Prediction of future
reliability
given
tasks to remove
concerns
Sensitivity of
model
to
uncertainty
given inputs
Create whatif scenario
Create
prediction
scenario
Record data
changes and
reasons
Record tasks
and impact
on concerns
Re-run model
Save results
if required
Warwick Manufacturing Group
Page 28
A ‘what if’ scenario can be defined for each sensitivity case by identifying the concerns and
the probabilities to be used, noting reasons for any adjustments made to values.
The model updates and provides reliability profiles for each sensitivity scenario and output
can be customised to superimpose the predicted and the original reliability estimate for the
design. For example, the plot on the left shows the reliability estimate for fault class 2 under
a given ‘what if ‘ scenario leading to an improvement in reliability while the plot on the right
shows the revised estimate corresponding to an in-service prediction. This indicates that the
changes proposed will have most impact on reliability after 10000 hours.
3.11
3.12
3.13
3.14
3.15
3.16
3.14
Update project and Conduct Follow-up Elicitation
The major set-up costs of modelling are incurred in the first pass. When this is completed,
there will be considerable data in the model and this can be updated as required. For all
projects it will be necessary to update the engineering judgement in the light of analysis
and test conducted since the last phase of modelling.
The process displayed below outlines the key steps in a follow-up elicitation and proposes
that the current tables of data are reviewed, updated and justified (in a similar way to the
sensitivity analysis) with, as far as possible, the same group of experts. This review process
should be conducted using the information from the last elicitation as the basis of discussion
and the revised output should be noted under the project updating. When all new information
is input for the new phase of modelling, the analysis can be run in the same way as before to
generate updated estimates.
Warwick Manufacturing Group
Page 29
Follow-up
elicitation
Confirm or update
details
regarding
facilitator
and
experts
Review
and
update status of
engineering
concerns
Prepare
information for
follow-up
interviews
Review concerns and
revise
probabilities,
establish
if
new
concerns
for
each
expert
Update
and
confirm records
Warwick Manufacturing Group
Page 30
3.15
Results from the REMM model
The notes above indicate how expert judgement data can be collected and used as part of a
new design project. It illustrates the richness of such data as well as the usefulness.
The REMM model as described above provides a reliability assessment of a new variant
design of a product. It also provides information on concerns elicited from experts in the
project team. This elicited data can be used for informing and updating the risk register and
hence is an integral part of the design process.
4 Summary and conclusions
These notes provide discussion and guidance on how to collect qualitative and quantitative
data from experts. They identify the problems associated with collecting such data and give a
proposed method for obtaining expert opinion in a structured manner. Further information on
eliciting expert judgement can be found in the references below.
5 References
1. Hogarth R. Judgement and choice: the psychology of decisions. Chicago, IL: Wiley;
1980.
2. Spetzler CS, Stael von Holstein CA. Probability encoding in decision analysis. Mgmt Sci
1975;22:340–52.
3. US Nuclear Regulatory Commission (NRC), Office of Nuclear Regulatory Research PRA
procedures guide: a guide to the performance of probabilistic risk assessments for nuclear
power plants, vol. 1–2, prepared under the auspices of the American Nuclear Society and
the Institute of Electrical and Electronic Engineers under a grant from the Nuclear
Regulatory Commission, Nuclear Regulatory Commission Report NUREG/CR-2300,
Washington, DC; 1983.
4. Gaines BR, Boose J, editors. Knowledge acquisition for knowledge based systems.
London, England: Academic Press; 1988.
5. Meyer MA, Booker JM. Eliciting and analyzing expert judgment: a practical guide.
Philadelphia, PA: Society of Industrial and Applied Mathematics; 2001.
6. Hora SC, Iman RL. Expert opinion in risk analysis: the NUREG 1150 methodology. Nucl
Sci Engng 1983;102:323–31.
7. Information integration technology. In: Khattree CR, Rao R , editors. Handbook of
statistics: statistics in industry. 2003.
8. Papers by Dr Lesley Walls, University of Strathclyde
Warwick Manufacturing Group