Validating Credit Rating Models
Stuart Burns
Senior Director
Standard & Poor’s Rating Services
6 May 2015
Permission to reprint or distribute any content from this presentation requires the prior written approval of
Standard & Poor’s. Copyright © 2013 by Standard & Poor’s Financial Services LLC. All rights reserved.
Agenda
Standard & Poor’s Ratings Services
Career Journey
Case Study #1: Expert Judgement Scorecard
Case Study #2: Modelling The Credit Cycle
Conclusions
Questions
2
Standard & Poor’s Ratings
Services
Standard & Poor’s Global Reach
• In business for 150+ years
• Provides global reach and
local knowledge with an office
network spanning 23 countries
• 1,400+ research analysts
• More than 1.1 million ratings outstanding
• $3.5+ trillion in new debt rated in 2011
• Provides perspective on a
company’s creditworthiness,
including its environmental,
social and governance (ESG)
performance
4
Standard & Poor’s EMEA
• EMEA operations
established in 1984
• 400+ analysts and
analytical supervisors
(excluding Criteria and Quality)
• Offices include: London, Frankfurt, Paris, Madrid, Milan,
Stockholm, Moscow,
Dubai, Johannesburg, Tel Aviv
5
Standard & Poor’s Ratings Leadership
S&P: More Than 1.1 Million Ratings Outstanding Globally
Credit Ratings Outstanding Globally1
Global Analytical Capability by NRSROs3
NRSROs
36.88%
45.67%
Moody’s
923,363
S&P
1,143,300
13.99%
3.46%
Fitch
350,370
Others2 86,571
Credit
Analysts
Credit Analyst
Supervisors
Standard & Poor’s Ratings Services
1,436
245
1,681
Moody’s Investors Service, Inc.
1,123
149
1,272
783
309
1,092
A.M. Best Company, Inc.
84
42
126
DBRS Ltd.
93
33
126
Japan Credit Rating Agency, Ltd.
27
32
59
Morningstar
26
10
36
Kroll Bond Ratings
22
6
28
2
3
5
Fitch, Inc.
Egan-Jones Rating Company
1. Nationally Recognized Statistical Rating Organization (NRSRO) filing 2013 (Exhibit 8). Note: Morningstar and Kroll — 2012 filing, 2013 N/A.
2. Others include A.M. Best Company, Inc; DBRS Ltd.; Japan Credit Rating Agency, Ltd.; Morningstar; Kroll Bond Ratings; Egan-Jones Rating Company.
3. NRSRO filing 2013 (Item 7). Note: Morningstar and Kroll — 2012 filing, 2013 N/A.
6
Total
Career Journey
Career Journey
• Built Economic Capital models for specialised areas at Abbey National
Treasury Services.
• At RBS Financial Markets headed up Basel II project. My team built the
models and validated them.
• At Standard Chartered worked in the Group Risk function, managing
stress testing and economic capital as well as building a team in
Singapore to validate wholesale and retail models.
• At HSBC plc Head of Corporate Analytics. AIRB and IAA models passed
FSA (now PRA) waiver.
• At Barclays Capital Global Head of Credit Risk Methodology. Portfolio of
industry leading models.
• At RBS worked on FiRST – Finance/Risk/Treasury transformation
project, then Rainbow – Williams & Glyn model suite development.
• Now Senior Director Model Validation at Standard & Poor’s.
8
Case Study #1: Expert
Judgement Scorecard
What are Credit Rating Models
Credit Rating Models attempt to predict the ability and willingness of an
issuer, such as a corporation or state or city government, to meet its
financial obligations in full and on time.
Credit rating models can also speak to the credit quality of an
individual debt issue, such as a corporate note, a municipal bond or a
mortgage-backed security, and the relative likelihood that the issue
may default.
Note that models are only one part of a rigorous process of credit
assessment undertaken by a rating agency. They may do this:
•
Via a Credit Rating (e.g. an agency rating such as AAA, AA+, etc.)
•
Or via a “risk parameter” which gives an estimate of credit risk
• Typically this may be via a “Probability of Default” (PD) (estimate)
• Often this will be supplemented by other parameters such as “Loss Given Default” (LGD) – this is
the expected recovery from a defaulted obligation (e.g. “cents in the dollar”). Again this is an
estimate
10
Credit Risk - IRB Rating Systems
Banks were incentivised under Basel II to quantify their Credit Risk
using an Internal Ratings Based (IRB) approach.
Various parameters needed for IRB:
Probability of Default (PD) – generated by the appropriate rating
model
Exposure at Default (EAD) – this is based on a model, typically
looking at drawn amount and a percentage (determined by
factors such as product) on the headroom (difference between
limit and drawn amount)
Loss Given Default (LGD) – again produced by a model, looking
at both product and customer characteristics. This represents the
percentage of the exposure we would lose in the event of default
11
Credit Risk associated with an exposure is a function of the characteristics of both
borrower and facility
Credit Risk of
Transaction
Borrower
Characteristics
Facility
characteristics
How much
exposure do
you have to
this facility?
Exposure at
Default (EAD - £)
12
What collateral
or other loss
mitigation is in
place for this
facility?
Loss Given
Default (LGD %)
Who are you lending to?
Probability of Default
(PD - %)
PD Modelling: Logistic Regression
13
What is validation
Validation represents the process by which credit risk models are
qualified for the following attributes:
•
Discriminatory Powers – such as Gini coefficient
•
Accuracy of Calibration – effectiveness of the model to quantify
risks, e.g. Binomial test
•
Stability and Consistency – does the model treat similar risks on a
consistent basis
•
Override rates – is the model output being used, or overridden within
acceptable levels
14
Discriminatory Power
The Gini coefficient is equal to double the area between the two
plots below
15
15
R-squared
16
16
Population Stability Index
17
Source: infoagora.com
Population Stability Index
PSI=∑((Actual% - Expected% ) ∗ ln( Actual%/Expected% )) PSI Value
Inference
Action
Less than 0.1 Insignificant change No action required
0.1 – 0.25
Some minor change
Greater than Major shift in
0.25
population
18
Source: ucanalytics.com
Check other scorecard monitoring
metrics
Need to delve deeper
Case Study: Expert Judgement Scorecard
A business area within a bank wanted to expand market share in a
particular business line.
• Historically we had not done many deals
• This meant internal historical data was insufficient for model build
• Buying in external data was ruled out
• An “Expert Judgement Scorecard” approach was hence proposed:
• An Expert Judgement Scorecard is an approach where empirical data is lacking
• Typically an expert panel will do some or all of the following
• “Blind rank” – i.e. rank order customers
• Propose factors and weights to be used in model
• Score customers for these factors
• There were problems from the outset in how this approach was
executed
19
Development Approach for Expert Judgement Scorecard #1
A model developer with a theoretical background combined with an
expert panel and adopted the following approach
• Divide existing portfolio 60%/40% into development/validation
samples
• Dependent variable cannot be “default/no default” so instead credit
grade (~ agency rating) was used
• Expert panel assembled and initially took a day out of their diaries to
meet in the same physical location as model developer
• Factors and weights were discussed
• Development Portfolio was scored for factors using “expert weights”
• Rank order from overall score was then bucketed into internal credit
grades and adjusted to match expert panel’s view of correct rank
order.
20
Development Approach for Expert Judgement Scorecard #2
The model developer then performed the following analysis and
reverted to a new meeting of the expert panel:
• Model developer then used regression analysis to estimate weights
for factors
• Some factors correlated or had no explanatory power
• On advice of expert panel these were left in with expert weight
• Weights for other factors estimated using stepwise regression
• New model then presented to expert panel
• Expert panel then scored validation sample
• Validation sample rank order was presented and validated by expert
panel
• R-squared on validation dataset was high and seen as proving model fit for purpose
• Calibration to a default rate admissible using “low default portfolio methodology”
• Population stability index acceptable between development and validation datasets
21
Problems with Development Approach
The following were all noted in the independent validation of the model
• Sample was too small to justify not using 40% of cases in
development
• Timing and psychology of approach present issues
• Groupthink in expert panel
• All had worked together before
• Everyone ultimately reported to one person (also on expert panel)
• Emphasis on reaching conclusion in time for scheduled model development
• Don’t blindly use regression when factors are correlated!
• Validation performed (R-squared) in no way independent
Population stability was also calculated on the wrong data:
• The “new business” was larger deals, which were unlike the
development or validation data
• Model was overfitted and gave unstable results on new data
22
Case Study #2: Modelling the
Credit Cycle
Probabilities of Default and the Credit Cycle
Probability of Default (PD) is a major input into a bank’s regulatory
capital calculation under Basel II
• For a retail bank a number of the inputs into PD may be volatile
• E.g. behavioural score, current account variables
• Classical wholesale PD models tend to be less volatile
• Typically these are based off published financials
• However under the Merton approach, equity price can be used to derive the PD of a firm. This
measure is potentially even more volatile than retail PDs
Volatile PDs means volatile capital requirements. Industry literature
discussed Retail/Merton PDs as “Point In Time” (PIT), and proposed
methodology to transform these into “Through The Cycle” (TTC
=stable) PDs.
• One formulation of this is that the PIT PD is conditioned on current
economic conditions, TTC PD is conditioned on “cycle average”
credit conditions.
24
Idealised Point In Time and Through The Cycle PDs
0.5
0.45
0.4
0.35
0.3
y_PIT
0.25
yTTC
0.2
0.15
0.1
0.05
0
0
25
1
2
3
4
5
6
7
The Merton Model
0.04
0.03
0.02
0.01
0
-5
-3.75
-2.5
-1.25
0
1.25
2.5
3.75
5
The horizontal axis above represents “default distance” (DD), the area to the right is
PD.
• One way to model cycle effects is to use proprietary PD data and imply the DD
• We may decide to use the assumption that a firm’s DD varies in line with a peer group by industry and region
• Form an industry-region cohort
• Current cohort average DD vs. long term average DD represents PIT vs. TTC
• A “Credit Cycle Adjustment” approach would use this difference to estimate a TTC
PD for each counterparty firm
26
Credit Cycle Adjustment and Scalars
0.5
0.45
0.4
0.35
0.3
y_PIT
0.25
yTTC
0.2
0.15
0.1
0.05
0
0
1
2
3
4
5
6
7
Adjustment
Scalar
0.2
2.5
0.15
2
0.1
0.05
1.5
0
0
1
2
3
4
5
6
7
1
-0.05
-0.1
0.5
-0.15
0
-0.2
0
1
2
3
4
5
6
7
27
Validation of Credit Cycle Adjustment Methodology
Validation of such a theoretical methodology is problematic.
However absurd behaviour should be avoided – and stress testing is
one way to check for this.
• When a counterparty is downgraded (PIT PD), the TTC PD should not
get better
• Critical assumption is that the average default distance between PIT and TTC for the group is a
proxy for that of the individual counterparty
• Industries are not always cyclical – e.g. UK Automotive
• Re-running history would the TTC flex in line with industry expansion or decline (sometimes
terminal)?
28
Inaccurate cycle amplitude and TTC estimates
0.6
0.5
0.4
y_PIT
0.3
z_PIT
y_sc_z
0.2
0.1
0
0
1
2
3
4
5
6
7
29
Inaccurate cycle phase and TTC estimates
0.6
0.5
0.4
y_PIT
0.3
z_PIT
y_sc_z
0.2
0.1
0
0
1
2
3
4
5
6
7
30
Conclusions
Expert Judgement vs. Analytics
Expert judgement
•
considered transparent
•
(can be) easy to get buy in
•
“groupthink”/ war stories
•
death by meetings
Analytics
•
(better) portfolio insight
•
real data used (not always the appropriate data)
•
model risk
Ideally we need robust debate to ensure a balance of the two approaches. A
balanced approach is needed by model developers and validators.
32
Questions
Thank You
Stuart Burns
Senior Director, Model Validation
T: +44(0)207 176 7627
Stuart.burns@standardandpoors.com
Permission to reprint or distribute any content from this presentation requires the prior written approval of
Standard & Poor’s. Copyright © 2013 by Standard & Poor’s Financial Services LLC. All rights reserved.
Copyright © 2013 by Standard & Poor’s Financial Services LLC. All rights reserved.
No content (including ratings, credit-related analyses and data, valuations, model, software or other application or output therefrom) or any part thereof (Content) may be modified,
reverse engineered, reproduced or distributed in any form by any means, or stored in a database or retrieval system, without the prior written permission of Standard & Poor’s
Financial Services LLC or its affiliates (collectively, S&P). The Content shall not be used for any unlawful or unauthorized purposes. S&P and any third-party providers, as well as their
directors, officers, shareholders, employees or agents (collectively S&P Parties) do not guarantee the accuracy, completeness, timeliness or availability of the Content. S&P Parties are
not responsible for any errors or omissions (negligent or otherwise), regardless of the cause, for the results obtained from the use of the Content, or for the security or maintenance of
any data input by the user. The Content is provided on an “as is” basis. S&P PARTIES DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, ANY WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE OR USE, FREEDOM FROM BUGS, SOFTWARE ERRORS OR
DEFECTS, THAT THE CONTENT’S FUNCTIONING WILL BE UNINTERRUPTED OR THAT THE CONTENT WILL OPERATE WITH ANY SOFTWARE OR HARDWARE
CONFIGURATION. In no event shall S&P Parties be liable to any party for any direct, indirect, incidental, exemplary, compensatory, punitive, special or consequential damages, costs,
expenses, legal fees, or losses (including, without limitation, lost income or lost profits and opportunity costs or losses caused by negligence) in connection with any use of the Content
even if advised of the possibility of such damages.
Credit-related and other analyses, including ratings, and statements in the Content are statements of opinion as of the date they are expressed and not statements of fact. S&P’s
opinions, analyses and rating acknowledgment decisions (described below) are not recommendations to purchase, hold, or sell any securities or to make any investment decisions,
and do not address the suitability of any security. S&P assumes no obligation to update the Content following publication in any form or format. The Content should not be relied on
and is not a substitute for the skill, judgment and experience of the user, its management, employees, advisors and/or clients when making investment and other business decisions.
S&P does not act as a fiduciary or an investment advisor except where registered as such. While S&P has obtained information from sources it believes to be reliable, S&P does not
perform an audit and undertakes no duty of due diligence or independent verification of any information it receives.
To the extent that regulatory authorities allow a rating agency to acknowledge in one jurisdiction a rating issued in another jurisdiction for certain regulatory purposes, S&P reserves the
right to assign, withdraw or suspend such acknowledgement at any time and in its sole discretion. S&P Parties disclaim any duty whatsoever arising out of the assignment, withdrawal
or suspension of an acknowledgment as well as any liability for any damage alleged to have been suffered on account thereof.
S&P keeps certain activities of its business units separate from each other in order to preserve the independence and objectivity of their respective activities. As a result, certain
business units of S&P may have information that is not available to other S&P business units. S&P has established policies and procedures to maintain the confidentiality of certain
non-public information received in connection with each analytical process.
S&P may receive compensation for its ratings and certain analyses, normally from issuers or underwriters of securities or from obligors. S&P reserves the right to disseminate its
opinions and analyses. S&P's public ratings and analyses are made available on its Web sites, www.standardandpoors.com (free of charge), and www.ratingsdirect.com and
www.globalcreditportal.com (subscription), and may be distributed through other means, including via S&P publications and third-party redistributors. Additional information about our
ratings fees is available at www.standardandpoors.com/usratingsfees.
STANDARD & POOR’S, S&P, GLOBAL CREDIT PORTAL and RATINGSDIRECT are registered trademarks of Standard & Poor’s Financial Services LLC.