Public Procurement Evaluation by
Evidence-based Multiple Criteria Decision Analysis
— From conventional scoring to systematic profiling
Professor Jian-Bo Yang
Director of Decision and Cognitive Sciences Research Centre
Manchester Business School (MBS)
The University of Manchester
Tel: 0161 306 3427 (Ext: 63427), 07715 175 723 (O2)
Email: jian-bo.yang@mbs.ac.uk
Web: www.mbs.ac.uk/dsrc,
www.personal.mbs.ac.uk/jbyang
Public Procurement Evaluation and MCDA by J B Yang of MBS
Outline of This Presentation
 Public procurement evaluation and Multiple Criteria
Decision Analysis (MCDA)
 Typical MCDA models
– Decision matrix – Scoring
– Pairwise comparison decision matrix - Rating
– Belief decision matrix – Profiling or grading
 Evaluation aggregation based on scores
– Linear aggregation or weighted sum
– Reference point approach using nonlinear distance measures
 Evaluation aggregation based on beliefs (IDS)
–
–
–
–
–
Evidence collection, mapping and grading
Evidential reasoning for generating bidder profiles
Expected utility as score for ranking
Sensitivity analysis for testing the robustness of ranking
Communication based on both bidder profiles and scores
Public Procurement Evaluation and MCDA by J B Yang of MBS
Public Procurement Evaluation and MCDA
Procurement evaluation criteria (weight)





Contractor’s organisation (0.1)
Financial considerations (0.3)
Management resources (0.2)
Past experience (0.2)
Past performance (0.2)
–
–
–
–
Failure of a contract (0.25)
Overruns: time (0.25)
Overruns: cost (0.25)
Actual quality achieved (0.25)
– ……
Public Procurement Evaluation and MCDA by J B Yang of MBS
Multiple Criteria Decision Analysis
Under uncertainty – Summary of main features
 A hierarchy of performance or risk criteria
 Quantitative and qualitative criteria
 Precise data and uncertain numbers
 Subjective judgements with uncertainty
 Possible absence of data
 Non-commensurability among criteria
 Conflict among criteria
 Ranking may not be precise
Public Procurement Evaluation and MCDA by J B Yang of MBS
Modelling for Procurement Evaluation
 Transparency and fairness via knowledge sharing
 Objectivity via data collection and management
 Systematic analysis via information aggregation
 Panoramic view of bidder profile
 Sensitivity analysis for uncertainty clarification
 Consistency in evaluation
 Simulation for improvement and feedback
 Communication with evidence (original / aggregated)
Public Procurement Evaluation and MCDA by J B Yang of MBS
MCDA Models for Procurement Evaluation
Scoring based model – decision matrix
MCDA problem with numbers: Decision maker is faced with
assessing and ranking several alternatives with all attributes being
considered simultaneously, with no attribute being absolutely more
important than others. The problem can be represented as follows
Decision Matrix (Table)
Bidder 1
Bidder 2
…
Bidder l
Criterion 1 Criterion 2
y11
y12
y21
y22
…
…
yl1
yl2
…
…
…
…
…
Criterion m
y1m
y1m
…
ylm
How should the assessment and ranking be made ?
Public Procurement Evaluation and MCDA by J B Yang of MBS
Scoring-based Decision Matrix
– Job evaluation
Decision Matrix for Job Evaluation
Criteria
Job offer 1
Job offer 2
Job offer 3
Salary
£32,500
£28,500
£26,000
Quality of
life
Interest of
work
Average
(50%)
Poor
(25%)
Poor
(25%)
Poor
(25%)
Good
(75%)
Good
(75%)
Good
(75%)
Average
(50%)
Good
(75%)
Location
Public Procurement Evaluation and MCDA by J B Yang of MBS
Pairwise Comparison Matrix
Compare each pair of job offers on a criterion
Pairwise Comparison Matrix for Job Evaluation
Quality of
life
Job offer 1
Job offer 2
Job offer 3
Job offer 1
1
2
0.5
Job offer 2
0.5
1
0.25
Job offer 3
2
4
1
Job 1 is judged (rated) twice as good as Job 2 in
terms of “Quality of life” (Interval comparison ?)
Public Procurement Evaluation and MCDA by J B Yang of MBS
Evidence-based Belief Decision Matrix
– Take into account judgmental information
MCDA problem with both numbers and judgements:
Belief Decision Matrix
Criterion 1 Criterion 2
…
Criterion m
Alternative 1
y11
S12
…
S1m
Alternative 2
y21
S22
…
S1m
…
…
…
…
…
Alternative l
yl1
Sl2
…
Slm
Belief distribution: Sij ={(H1, βij1), (H2, βij2), …… , (HN, βijN)}
Public Procurement Evaluation and MCDA by J B Yang of MBS
Belief Decision Matrix
Assessment based on evidence collected
House House 1 in
Criteria
Altrincham
Location
Distance
(mile)
Asking
Price (£)
Attractiveness
House 2 in
Heaton
House 3 in House 4 in
Mercy
Didsbury
{(G, 0.5),
(E, 0.5)}
{(G, 0.5)}
{(A, 0.2),
(G, 0.8)}
{(G, 0.2),
(E, 0.8)}
7
5
6
5.5
113,000
110,000
118,000
150,000
{(P, 0.05),
(G, 0.35),
(E, 0.60)}
{(A, 0.4),
(G, 0.6)}
{(G, 0.3),
(E, 0.7)}
{(G, 0.6),
(E, 0.4)}
Belief is generated from the assessment of evidence
Public Procurement Evaluation and MCDA by J B Yang of MBS
Belief Decision Matrix
Assessment based on evidence collected and mapped
Assessing the Location of House 1 in Altrincham using the collected
evidence against the agreed assessment standards (mapping)
Public Procurement Evaluation and MCDA by J B Yang of MBS
Procurement Evaluation Aggregation
– Weighted sum or Multiple Attribute Value Function
General form of an additive (linear) value function is given by:
m
v   i vi ( yi )  1v1 ( y1 )  2v2 ( y2 )    mvm ( ym )
i 1
Conditions for use of Additive MAVF:
1. Satisfaction of preferential independence among any groups of
attributes. This is only a necessary condition.
2. Satisfaction of the corresponding trade-off, or Thomsen condition.
3. Interval scale property for constructing marginal value function.
4. Weights of attributes need to be assessed as scaling constants
(trade-offs), or swing weights, not necessarily relative importance.
5. Linear & complete compensation among criteria without any limit.
Public Procurement Evaluation and MCDA by J B Yang of MBS
MCDA – Value Measurement Theory
– Preferential independence violation example
Chinese Restaurant Menu: Combination of soup and main dish
Attribute 1: Choose soup
Attribute 2: Choose main dish
Are you preferentially independent when choosing soup and main dish?
Soup
Value score
Mixed veg &
8
bean curd
Egg and
3
tomato
Main dish
Value score
Bean curd
Pork with
Spring
Onions
10
7
If you are preferentially independent in choosing soup and main dish, you
would ask for a main dish without considering what soup you have taken.
However, is this the case for you? Would you really choose both Mixed
veg & bean curd as soup and Bean curd as main dish?
Public Procurement Evaluation and MCDA by J B Yang of MBS
Limitation or Bias of Additive MAVF
Efficient frontier: A, B, D, E, F, G
Efficient convex hull: A, E, G
Additive MAVF cannot find B or F
as the most preferred solution
25
G(2, 20)
Safety (Maximising)
20
E(12, 15)
F(5, 17)
15
ωsvs+ωpvp=v
D(12, 12)
10
C(11, 9)
B(14, 7)
5
A(20, 2)
0
0
5
10
15
20
Profit (Maximising)
Public Procurement Evaluation and MCDA by J B Yang of MBS
25
Distance-based Aggregation
Ideal point models (minimax distance)
Ideal point models: Set an ideal reference point and find an
alternative closest to the ideal point in certain distance measure.
Set criterion
weights
Criterion 2 (Maximising)
25
Ideal point
G(2, 20)
20
E(12, 15)
F(5, 17)
15
Reference point
D(12, 12)
10
C(11, 9)
B(14, 7)
5
A(20, 2)
0
0
5
10
15
20
Criterion 1 (Maximising)
Public Procurement Evaluation and MCDA by J B Yang of MBS
25
Evidential Reasoning MCDA
Assessment distribution by a belief structure
ER Example 1: A qualitative assessment that the quality yq
of a bidder A is assessed to be “Good” or “Excellent” by an equal
number of assessors, with no assessment below “Average”, can
be described by the following distribution
S(yq(A)) ={(Bad, 0), (Average, 0), (Good, 0.5), (Excellent, 0.5)}
which is termed as a belief distribution of assessment, with
“Bad”, “Average”, “Good” and “Excellent” defined as “assessment
grade” and 0 (0%) and 0.5 (50%) as “belief degree” (frequency
to which “Good” or “Excellent” is ticked by the assessors).
The above distribution shows the quality profile of the bidder.
Public Procurement Evaluation and MCDA by J B Yang of MBS
Evidential Reasoning MCDA
Assessment Using ER – What’s different
Traditionally, only scores are used
ER uses both scores and belief degrees
Bidders
6.1 Give examples of STRATEGIC
Partnering, Alliances and Collaborative
Working
Bidder 1
Score 76%
({Best, 28%}, {Good, 51%}, {Average,17%},
{Poor, 4%}, {Worst, 0%}
Bidder 2
Score 76%
({Best, 46%}, {Good, 29%}, {Average,15%},
{Poor, 3%}, {Worst, 7%}
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using a Decision Support System
– Intelligent Decision System (IDS)



IDS is supported by the Evidential Reasoning (ER) approach
ER has been developed over a period of over 15 years
ER results from multi-discipline research
-




Decision Sciences
Artificial Intelligence
Statistical Analysis
Fuzzy Sets
ER addresses subjectivity and uncertainties
ER can handle heterogeneous information
ER guarantees to generate rational results
ER is gaining popularity in both academia and industry
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Advantages

Structured and natural
No modification needed in IDS for procurement
evaluation modelling

Flexible in modelling
Model can be modify, attributes changed, added
and deleted easily

Improved consistency and efficiency
Through knowledge management and using an
systematic evidence mapping process
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Advantages

No unnecessary assumption
No need to use scores for subjective judgement
No need to assume missing data

Transparent
Candidates compared on any attribute at any level
Weaknesses and strengths of each candidate identified

Rational, convincing and informative
Examine impact of changes in any factor on decisions
easily so that the decisions are made in a more
rational, convincing and informative way
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Modelling
Build an evaluation criteria hierarchy
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Modelling
Define qualitative attribute:
Number of grades
can be changed
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Modelling
Define grades to assess a qualitative attribute:
Wording of grades
can be changed
Preference value of
grades can be changed
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Modelling
Define grade standard to assess a qualitative attribute:
This can be used as guidelines to help
improve consistency in assessment
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Modelling
Define quantitative attribute:
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Modelling
Assign weights
Drag and drop to change weight
Or type weight here
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Review Document
Evidence classified and recorded
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Mange Knowledge
Evidence examined and comments provided
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – Make assessment
Evidence mapped and belief degrees assigned to grades
Grade guidelines entered earlier
More than one grades may be selected
Optional
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – View Results
Performance distribution (profile) of bidder generated
Can be any attribute in the hierarchy
Unknown element due to lack of data in
Economic Test & Interview
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – View Results
Performance scores – when there is missing data
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – View Results
Compare candidates on multiple criteria – by scores
Public Procurement Evaluation and MCDA by J B Yang of MBS
Assessment Using IDS – View Results
Compare candidates – by performance profile
Public Procurement Evaluation and MCDA by J B Yang of MBS
Other Applications - Siemens UK
Supplier pre-qualification assessment
Public Procurement Evaluation and MCDA by J B Yang of MBS
Other Applications
 Product design and evaluation
car, motorcycle, ship, aircraft, computer, …
 Safety and risk assessment
 Quality management
 Supply chain management
 Environmental management
 Financial services and investment
 Customer satisfaction survey
 Web based survey
 Data collection only – remote or onsite audit
Public Procurement Evaluation and MCDA by J B Yang of MBS
Summary and Conclusions
 Public procurement evaluation and multiple criteria
decision analysis (MCDA)
 Typical MCDA procurement evaluation models
– Decision matrix – Scoring
– Pairwise comparison decision matrix - Rating
– Belief decision matrix – Profiling or grading
 Evaluation aggregation based on scores
– Linear aggregation or weighted sum
– Reference point approach using distance measures
 Evaluation aggregation based on beliefs (IDS)
–
–
–
–
–
Evidence collection and mapping or grading
Evidential reasoning for generating bidder profile
Expected utility as score for ranking
Sensitivity analysis for testing the robustness of ranking
Communication based on both bidder profile and score
Public Procurement Evaluation and MCDA by J B Yang of MBS