Mathematical Methods in Reliability Research
Friday 31st August 2012
GH510 Graham Hills Building, University of Strathclyde
Expert Judgment
Programme
10:00
Refreshments
10:25
Welcome by John Quigley, University of Strathclyde
10:30
Importance of the Process for Expert Judgement Elicitation
Lesley Walls, University of Strathclyde, UK
11:00 Conditional rank correlation elicitation exercise
Oswaldo Morales Napoles & Anca Hanea, TU Delft, Netherlands
11:30 Uncertainty in Competitive Bidding – Supporting Pricing Decision with Probabilistic Models
Yee Mey Goh, University of Loughborough, UK
12:00 Mathematical Aggregation of Probabilistic expert judgements
Taposhri Ganguly, University of Strathclyde, UK
12:30-1:30
1:30
Lunch
Expert Judgment, Meta-Analysis and Participatory Risk Analysis
Simon French, University of Warwick, UK
2:00
The Moment Model Alternative to Cooke’s Classical Model
Tim Bedford, University of Strathclyde, UK
2:30
Calculating a Health Index to make decisions on asset replacement
Simon Blake, Durham University, UK
3:00
Discussion, wrap up & drinks reception.
Abstracts
Lesley Walls, University of Strathclyde, UK
Importance of the Process for Expert Judgement Elicitation
Drawing upon experience of eliciting probabilistic engineering judgement to support reliability
modelling of industrial problems, this talk will examine key challenges and suggest ways of designing
a robust elicitation process. An overview of our elicitation experience space will be given to provide
a frame of reference. Examples of how common biases manifest themselves within technical risk and
reliability applications will be described. Perceptions of engineers and managers about the role of
probabilistic expert judgement in reliability modelling will be shared. The important role that
theoretical frameworks play as the basis for an elicitation process design will be discussed.
Oswaldo Morales Napoles & Anca Hanea, TU Delft, Netherlands
Conditional rank correlation elicitation exercise
Dependence measures such as rank correlations are commonly used in different types of models.
Whenever field data is available these may be estimated directly from data. However, field data is
often not available and structured expert judgment is sought instead.
Methods for eliciting rank correlations from experts have been proposed in the past (Cooke &
Goossens 1999), (Clemen et al. 1999). One option is to ask experts directly for an estimate of the
rank correlation between pairs of variables. Another option is to acquire estimates of other related
quantities, e.g. conditional probabilities of exceedance, probabilities of concordance or discordance.
The rank correlations of interest could be then estimated using these quantities. Previous results
indicate that an accurate way to obtain a subjective measure of bivariate dependence is by simply
asking the experts to estimate the correlation between the two variables in question (Clemen and et
al 2000).
Recently non-parametric Bayesian networks have been introduced as flexible tools for high
dimensional dependence modelling (Hanea et al. 2006). The input for these models consists of
univariate marginal distributions and (conditional) rank correlations. In absence of field data
(conditional) rank correlations have been assessed from experts through the elicitation of
conditional probabilities of exceedance or ratios of rank correlations (Morales Napoles et al. 2008).
To our knowledge, there is no experimental study available that would, at least, give some indication
about the accuracy of estimating conditional rank correlations from experts. Even less is known
about the accuracy of one method with respect to the other. In this study we describe data collected
from a controlled exercise. The question of interest is whether experts can estimate more accurately
conditional rank correlations through estimates of conditional probabilities of exceedence or
through ratios of rank correlation coefficients.
Yee Mey Goh, University of Loughborough, UK
Uncertainty in Competitive Bidding – Supporting Pricing Decision with Probabilistic Models
The suppliers of long-life products such as submarines and airplanes no longer simply sell these
products but provide their capability or availability. This means that companies that traditionally
design and manufacture long-life products now compete through the provision of a service. These
companies face a high level of uncertainty due to the novelty of the process of designing and
managing the contract and the long-term nature of services. As a consequence, it is difficult for
companies to determine an appropriate price bid for the service, which will enable them to win the
contract as well as make a profit. The research aims at supporting these companies in their pricing
decision process by providing a tool that models the uncertainties at the competitive bidding stage
and depicts their influence on the probability of winning the contract and the probability of making a
profit. An industrial case study will be described to demonstrate the approach.
Taposhri Ganguly, University of Strathclyde, UK
Mathematical Aggregation of Probabilistic expert judgements
Mathematical aggregation of expert judgements is one of the widely addressed topics within
Bayesian statistics especially for situations when the true value of a parameter of interest is
unknown and the judgements are correlated. As stated by Winkler (1981), decisions in the face of
uncertainty should be based on all available information. Literature suggests that aggregating the
probabilistic expert judgements would be useful especially when there are multiple experts on the
same area of expertise, judging the same scenario. Further, based on the literature, it may be argued
that there is a significant gap between aggregation of expert judgements where judgements are
statistically dependent and there is a gap in aggregation of between and within expert judgements
when multiple variables are assessed by a set of experts. We propose a mathematical model of
aggregating probabilistic expert judgements, where the judgements are correlated. We assume the
expert judgements to follow a Normal distribution, assuming that the experts are providing
estimates of the true value of the unknown parameter. These judgements are correlated and we
construct a multivariate normal likelihood function with Normal priors to estimate the posterior
distribution. The dependence between and within expert judgements is captured in the construction
of the covariance matrix. The model developed, is further validated by deriving the unbiased
estimators of the parameters. This modelling approach demonstrates aggregation of subjective
judgements when the judgements are correlated. A realistic example on data from the Forth Road
bridge demonstrates this approach.
Simon French, University of Warwick, UK
Expert Judgment, Meta-Analysis and Participatory Risk Analysis
There are three contexts in which one might wish to combine expert judgments of uncertainty: the
expert problem, the group decision problem and the textbook problem. Much has been written on
the first two, which have the focus of a single decision context, but little on the third. The textbook
problem arises when one needs to draw together expert judgments into a decision analysis when
their judgments were made originally in a context free manner or perhaps for other decision
contexts. In many ways the textbook problem parallels that of performing a meta-analysis of
empirical studies. However, there are differences. In this paper, we discuss those difficulties and
then focus on two closely related issues: how should expert judgment studies be published so as to
facilitate subsequent meta-analyses and how should such meta-analyses be performed?
Tim Bedford, University of Strathclyde, UK
The Moment Model Alternative to Cooke’s Classical Model
Quantitative expert judgement is used in many areas of risk analysis to provide assessments of
uncertainty. A leading method is Cooke’s classical model, which provides a way of weighting experts
depending on their performance in answering so-called calibration questions. The moment method
for expert judgement combination is an alternative to Cooke’s method which provides a different
way of calibrating experts. It has better theoretical properties in that it uses a strictly proper scoring
rule, that is, an expert who wishes to maximise his score can only do so by stating what he actually
believes. The method also requires the specification of weights for the calibration process. In this
paper we consider a special case of the moment model that scores location and variability
assessments. We provide an approach to setting the calibration weights and repeat the comparison
of this moment model with Cooke’s classical model we conducted earlier for this special case.
Simon Blake, Durham University, UK
Calculating a Health Index to make decisions on asset replacement
A Distribution Network Operator in the electricity industry manages a large portfolio of ageing
assets, with an annual budget for replacing those in the worst state. This is determined by allocating
a health index (HI) to each asset, on a scale of 0 to 10, which takes into account a diverse range of
factors including asset age, location, utilisation, inspection data and laboratory analysis. Weighting
and balancing these factors is done using a complex set of formulae, derived by incorporating expert
judgment as far as possible. How this is done, and the results and consequences, constitutes an
informative insight into the industrial decision-making process.