Results of BBA/ISDA/RMA
IRB Validation Study
BBA/ISDA/RMA
Advanced IRB Forum
Monika Mars
London - June 23, 2003
BRITISH
BANKERS’
ASSOCIATION
Agenda
 Survey Approach & Participants
 Background – Use of Ratings
 Survey Findings
 Conclusions and Implications
pwc
2
BRITISH
BANKERS’
ASSOCIATION
Survey Approach
Survey research
and design
4th Quarter 2002
Data collection,
and analysis
Interviews
Jan – Feb 2003
Feb – Mar 2003
Report
preparation
1st Draft Mid March
2003
Final Report Draft
– early May
Report
presentation
June
19/23
pwc
3
BRITISH
BANKERS’
ASSOCIATION
Survey responses covered all asset classes
representing a diverse group of institutions
pwc
4
BRITISH
BANKERS’
ASSOCIATION
Agenda
 Survey Methodology & Participants
 Background – Use of Ratings
 Survey Findings
 Conclusions and Implications
pwc
5
BRITISH
BANKERS’
ASSOCIATION
Internal ratings are key to managing the business
at most firms
pwc
6
BRITISH
BANKERS’
ASSOCIATION
Most banks use “Master Scales” to compare
ratings information across portfolios
pwc
7
BRITISH
BANKERS’
ASSOCIATION
Default definitions, time horizons and alignment
to external sources vary among institutions
 The definition of default is not in all cases in line with the
BASEL II definition – this is particularly the case for retail
portfolios
 Time horizons of one year are most common, however
the estimate of a 1-year PD might be based on a
multiyear sample
Some banks use more than one year as a time horizon
while a few use less than a one year time horizon to
estimate PD
A small number of banks estimate PDs over the life of the
loan
 Most participants align a “majority” of their ratings in the
corporate asset class to an external source, while the
majority don’t do this in the retail asset class
pwc
8
BRITISH
BANKERS’
ASSOCIATION
Agenda
 Survey Methodology & Participants
 Background – Use of Ratings
 Survey Findings
 Conclusions and Implications
pwc
9
BRITISH
BANKERS’
ASSOCIATION
Key Findings
 Banks employ a wide range of techniques for
internal ratings validation
 Ratings validation is not an exact science
 Expert judgment is of critical importance in the
process
 Data issues are centred around quantity not quality
 Regional differences exist with respect to the
validation of internal ratings
 Defining standards for stress testing requires
additional work
pwc
10
BRITISH
BANKERS’
ASSOCIATION
Banks employ a wide range of techniques to
validate internal ratings - key differences exist
between corporate and retail ratings
 Corporate Asset Class
 Statistical models where the quantity of default data
allows for strong estimation (particularly in middle
market)
 Expert judgment models for portfolios where default data
is limited
 Hybrid and/or Vendor models to complete the picture
 Retail Asset Class
 Statistical models are heavily relied upon due to the
greater availability of internal data history
pwc
11
BRITISH
BANKERS’
ASSOCIATION
A variety of model types are employed within
each asset class
Corporate
Middle
Market
Retail
Statistical
7
4
23
Expert
Judgement
15
11
8
External
Vendor
7
2
17
Hybrid
10
7
5
Model Type
pwc
12
BRITISH
BANKERS’
ASSOCIATION
Models for bank and sovereign exposures
extensively use external information and expert
judgement
 Ratings for bank exposure are mostly derived by
benchmarking against external ratings as well as
using expert judgment or hybrid models
 Ratings for sovereign exposures are similarly
derived by benchmarking against external ratings as
well as using expert judgment
 Published default statistics are used for PD
estimation for both bank and sovereign exposures
pwc
13
BRITISH
BANKERS’
ASSOCIATION
Most banks surveyed have a rating system for
specialised lending in place but face major issues
in its validation
 A common theme is the lack of default data
 Validation issues specific to specialised lending include:
 differentiation of borrower and transaction,
 definition of default (particularly the restructuring clause),
 inconsistent data history,
 and the time horizon of the model
pwc
14
BRITISH
BANKERS’
ASSOCIATION
Rating validation is not an exact science
 Even with the use of statistical techniques to assess
model performance absolute triggers and thresholds are
not used
 There is no absolute KS statistic, GINI coefficient, COC or
ROC measure that models need to reach to be
considered adequate
 Default statistics published by the major rating agencies
are used differently from bank to bank depending on each
bank’s assessment of the most appropriate use of the
external data
 Benchmarking against external ratings raises many
issues including the “unknown” quality of external ratings,
methodology differences, and the like
pwc
15
BRITISH
BANKERS’
ASSOCIATION
The performance of statistical rating models is
achieved through a number of different
techniques
pwc
16
BRITISH
BANKERS’
ASSOCIATION
Different triggers are used to evaluate the overall
performance of expert judgement rating models
pwc
17
BRITISH
BANKERS’
ASSOCIATION
A variety of techniques are employed for
evaluating vendor models
Methods used to determine the appropriateness of the vendor
model to your internal rating system and approvals process:
The model outputs a default rate and risk rating
aligned to ECAI
The model outputs a rating that can be mapped to
our IRS
The model considers factors we feel important to the
credit process
The approach is based on sound theory of how
businesses operate
Other
0
2
4
6
8
Corporate
10
12
14
16
Retail
pwc
18
BRITISH
BANKERS’
ASSOCIATION
Expert judgement is essential in the validation
process
 Data scarcity prevents the use of statistical models
for some asset classes: corporate, bank, sovereign,
and specialised lending
 Most respondents use judgemental overlay by rating
experts (account officer, credit analyst) to confirm or
modify the risk rating output of their assessment
model (statistical, hybrid, vendor)
 Large proportions of banks’ exposures are covered
by expert-judgment type rating systems
pwc
19
BRITISH
BANKERS’
ASSOCIATION
Most data issues centre around quantity of data
available not the quality of the data
 Most banks surveyed have initiated projects to
collect the necessary data in a consistent manner
across the institution to allow for statistical modelling
in the future
 The quantity of default data around large corporate,
bank, sovereign, and specialised lending exposure
classes is a real problem for most institutions
 Institutions have begun data pooling initiatives for
PD and LGD data, however there is scepticism as to
whether these measures will solve the data quantity
problem
pwc
20
BRITISH
BANKERS’
ASSOCIATION
Clear regional differences exist with regard to
internal ratings for corporate assets and their
validation
 Expert judgment models are used for large corporate
portfolios, however the structure of the ratings differ
significantly
In North America fixed weightings are not assigned for the
factors to be assessed by the experts
In Europe specific weights for each factor are often set
 Models based on equity market information (KMV) or
balance sheet information (Moody’s RiskCalc) are used
for corporate and middle market portfolios
In North America, these models tend to be an integral part of
the rating and are used in conjunction with expert judgment
in a hybrid approach
In Europe, these models are more likely to be used as a
benchmark or a validation of the internal rating model
pwc
21
BRITISH
BANKERS’
ASSOCIATION
Similar differences can be observed for the retail
asset class
 Statistical (scorecard) techniques for retail exposures tend
to be product specific in the US and UK, while in
Continental Europe the focus is on customer
scores/ratings
 US and UK scorecards are redeveloped more often than
those on Continental Europe, where robustness of ratings
and long-term stability factors are of higher priority
 This often has direct implications for validation, as longer
term more stable models tend to show – for example lower GINIs than models using the latest available data
pwc
22
BRITISH
BANKERS’
ASSOCIATION
More work needs to be done in defining
standards for stress testing
 There is currently no uniform approach regarding the
type of stress testing undertaken, its frequency, or
actions taken in response to stress testing results
 At the moment, stress testing is performed on the
portfolio level with risk ratings being a key input in
stress testing scenarios for economic capital
requirements
 There is uncertainty around BASEL II requirements
with respect to stress testing of rating model inputs –
and also considerable debate as to its usefulness
pwc
23
BRITISH
BANKERS’
ASSOCIATION
Agenda
 Survey Methodology & Participants
 Background – Use of Ratings
 Survey Findings
 Conclusions and Implications
pwc
24
BRITISH
BANKERS’
ASSOCIATION
The industry, regulators and other stakeholders
must continue a dialogue to address Basel II
implementation issues
 Recognition of different techniques for validating
internal rating systems – no one “right” method
 Increased debate and guidance with respect to
validation of expert judgement based rating systems
 Recognition of regional / cultural differences as they
impact internal ratings and the consequences for
validation
 Guidance on requirements for the use of pooled
data
 Additional discussion and clarification with respect to
stress testing requirements
pwc
25
BRITISH
BANKERS’
ASSOCIATION
pwc
26
BRITISH
BANKERS’
ASSOCIATION