Challenges in Validating
Catastrophe Models
Dr. Paul Rockett
Casualty Actuaries in Europe, 31 May 2013
Agenda
►
►
►
►
►
Introduction
Validation tools
Validating catastrophe models
The bigger picture
Summary
Page 1
31 May 2013
Challenges in Validating Catastrophe Models
Introduction
Page 2
31 May 2013
Challenges in Validating Catastrophe Models
What are catastrophe models?
►
Tools designed to evaluate the risk
profile of a risk or set of risks exposed
to catastrophe events
Event Set Module
Hazard Module
0.06
Exposure Module
Density
Ratio
LossProbability
0.05
Over Limit
Damage
Distribution
0.04
Vulnerability Module
0.03
0.02
Financial Analysis Module
Insurance
Cover
0.01
0
0
0.2
0.4
0.6
0.8
Damage Ratio
Intensity
Page 3
31 May 2013
Challenges in Validating Catastrophe Models
Deductible
1
What are catastrophe models?
►
They combine the frequency of catastrophe events with their severity
to produce annual exceedance probability (EP) curves
Convolution
►
EP Curves can be thought of like VAR. SII SCR is a VAR measure
corresponding to the 99.5% conf level of non-exceedance in 1-year
Page 4
31 May 2013
Challenges in Validating Catastrophe Models
What do they cover?
►
►
►
►
There are catastrophe models covering
►
Earthquakes
►
Windstorms, Tropical Cyclones
►
Storm Surge, River Floods, ...
►
Severe Convective Storm (Tornado, Hail, ...)
►
Bushfire, wild fire, ...
And man-made catastrophes
►
Terrorism, Explosion, Conflagration
►
Pandemic
Most important models for the insurance industry are US Hurricane,
US Earthquake and Europe Windstorm
But it varies by company
Page 5
31 May 2013
Challenges in Validating Catastrophe Models
What are catastrophe models used for?
Capital model
inputs
Estimate post
event loss
size
Reinsurance
decision
making
Catastrophe
Models
Accumulation
management
Page 6
31 May 2013
Pricing risks
Challenges in Validating Catastrophe Models
Who builds catastrophe models?
AIR
Insurers
Cat
Modelling
Companies
EQECAT
RMS
Brokers
Reinsurers
Others
Consultants
Marketplace
– e.g.
OASIS LMF
Page 7
31 May 2013
Challenges in Validating Catastrophe Models
Why validate catastrophe models?
Material
Uncertainty
► Catastrophe models can
drive the Solvency Capital
Requirement
► Extreme events and
subsequent losses e.g.
► Hurricane Katrina
► Hurricane Ike
► Christchurch Earthquakes
History
Page 8
31 May 2013
► Limited data
► Many assumptions
Why
Validate?
Solvency II directive:
► Article 124
(Validation Standards)
► Article 229 TSIM18
(Validation Process)
Regulatory pressure
Challenges in Validating Catastrophe Models
Validation Tools
Page 9
31 May 2013
Challenges in Validating Catastrophe Models
Solvency II Validation tools
Tools for use by all firms (Article 230 TSIM19)
Results against
experience
Purpose:
 Modelling and
estimation errors,
 Any model
weaknesses
Model robustness
Purpose:
 Are capital
requirements
produced robust?
 Back-testing
 Goodness of fit
 Dependency
assumptions
 Supplement with
qualitative analysis
Purpose:
 Impact of single or
multiple events
Applications
Applications
Applications
Stress & scenario
testing
 Sensitivity of results
to changes in the
key underlying
assumptions
 Stability of the
model
 Challenge results
and parameters
 Dependencies and
tails of distributions
 Understanding of
the risk profile
 Supports capital
allocation decisions
 Exceptional but
plausible large-loss
events
 Reverse stress
testing
 Limitations of the
model
Page 10
31 May 2013
Challenges in Validating Catastrophe Models
Profit & loss
attribution
Purpose:
 Are risks covered
by the model
complete?
 Explain Profits and
Losses
Applications
 Explains actual
results using the
model.
 Assess level of
unexplained profits
and ability to reflect
risk profile of the
business
Solvency II Validation tools:
Further (optional) tools set out by EIOPA
Hypothetical
portfolio
Benchmarking
Analysis of change
Purpose:
 Reinforce the
appropriateness of
Pillar I and II results
Purpose:
 Analyse how the
results of the model
have changed
Applications
Applications
Consider results of
peers
 Reconciles results
Applications
Use test
 Validates that the
model is working as
expected
Hypothetical portfolios
of assets and/or
liabilities
 Using the model in
BAU demonstrates
sufficient level of
comfort that the
results are
appropriate for use
 to investigate
whether alternative
modelling
approaches could
improve quality
Note differences in
company risk profiles
Purpose:
 Improve
comparisons
between different
models or versions
31 May 2013
Purpose:
 Additional
qualitative
validation tools
Applications
 chosen by the
company to
analyse outcomes
from different
versions of internal
models
Care needs to be
taken to avoid
creating systemic risk
Page 11
Qualitative review
Challenges in Validating Catastrophe Models
Qualitative review
With dialogue and
review, gain comfort:
 Theoretical basis of
the internal model
 Any part of the
internal model
Validating Catastrophe Models
Page 12
31 May 2013
Challenges in Validating Catastrophe Models
Validating catastrophe models
Input
Cat Modules
Output
Event Set
Hazard
Exposure Data
Exposure
Vulnerability
Financial Analysis
►
Each component needs validation
►
Main focus is on model output
Page 13
31 May 2013
EP Curves
Challenges in Validating Catastrophe Models
ELTs
Input
Exposure data
►
Key Aspects
Exposure Data
Geospatial
Resolution
Risk
Attributes
Valuation
Data
Heuristics
Cat Modules
Event Set
Hazard
Exposure
Vulnerability
►
Typical Approach
Qualitative review inc. data audit
►
Validation Challenge
Ensuring processes in place track all the
exposure information, and that it passes quality
control checks before modelling
Output
Financial
Page 14
EP Curves
ELTs
31 May 2013
Challenges in Validating Catastrophe Models
Input
Event set module
►
Key Aspects
Exposure Data
Event
Frequency
Clustering of
Events
Event
Characteristics
Cat Modules
Event Set
Hazard
Exposure
►
Typical Approach
Back-testing, Sensitivity Testing,
Benchmarking
►
Validation Challenges
Need an understanding of the physical
mechanism to define suitable tests
Vulnerability
Output
Financial
Page 15
EP Curves
ELTs
31 May 2013
Challenges in Validating Catastrophe Models
Input
Hazard module
►
Exposure Data
Event
Footprints
Event Set
Cat Modules
Key Aspects
Hazard
Distribution
Hazard
Exposure
►
Typical Approach
Back-testing, Qualitative Review
►
Validation Challenge
Sensitivity testing very difficult
Vulnerability
Output
Financial
Page 16
EP Curves
ELTs
31 May 2013
Challenges in Validating Catastrophe Models
Spatial
Correlation
Input
Exposure module
►
Key Aspects
Exposure data
Geocoding
Data
Recognition
Cat Modules
Event Set
Hazard
Exposure
Vulnerability
►
Typical Approach
Benchmarking, Audit
►
Validation Challenge
Very involved to properly check
Output
Financial
Page 17
EP Curves
ELTs
31 May 2013
Challenges in Validating Catastrophe Models
Input
Vulnerability module
►
Key Aspects
Exposure Data
Vulnerability
Functions
Secondary
Modifiers
Cat Modules
Event Set
Hazard
Exposure
►
Typical Approach
Qualitative Review, Sensitivity Testing
►
Validation Challenge
Limited data
Vulnerability
Output
Financial
Page 18
EP Curves
ELTs
31 May 2013
Challenges in Validating Catastrophe Models
Input
Financial module
►
Key Aspects
Exposure Data
Implementation
Uncertainty
Aggregation
Cat Modules
Event Set
Hazard
Exposure
►
Typical Approach
Qualitative Review, Audit
►
Validation Challenge
Understanding the real level of uncertainty
Vulnerability
Output
Financial
Page 19
EP Curves
ELTs
31 May 2013
Challenges in Validating Catastrophe Models
Input
Output
►
Key Aspects
Exposure Data
EP Curves
Event Loss
Tables
Historical Event
Losses
Cat Modules
Event Set
Hazard
Exposure
►
Typical Approach
Back-testing, Sensitivity Testing, Qualitative
Reviews, Benchmarking
►
Validation Challenge
Limited data
Vulnerability
Output
Financial
Page 20
EP Curves
ELTs
31 May 2013
Challenges in Validating Catastrophe Models
Bigger Picture
Page 21
31 May 2013
Challenges in Validating Catastrophe Models
Scope of validation
►
Validating catastrophe models should be seen within the
context of internal model validation
Understanding
Capital
Modelling
Accumulation
Corporate
Governance
Pricing
Cat
Model
Model Risk
Management
Nonmodelled
Risks
►
Each aspect needs validation
Page 22
31 May 2013
Challenges in Validating Catastrophe Models
Data
Quality
How to validate non-modelled perils?
►
Non-modelled
Risks
Key Aspects
Limited
Geographic
Coverage
Missing Lines
of Business
Corporate
Governance
►
Typical Approach
Qualitative Review
►
Validation Challenge
Subjective
Understanding
Model Risk
Management
Page 23
31 May 2013
Challenges in Validating Catastrophe Models
Missing SubPerils
Corporate governance
►
Non-modelled
Risks
Key Aspects
Materiality
(SCR, Pricing,
Accumulation)
Processes and
Controls
Documentation
Change
Management
Corporate
Governance
Understanding
Model Risk
Management
Page 24
31 May 2013
►
Typical Approach
Qualitative Review, Audit
►
Validation Challenge
Checking the right structures, processes and
controls are in place to ensure the IM is robust
Challenges in Validating Catastrophe Models
Understanding cat models
Knowledge of Catastrophe Models
Non-modelled
Risks
Corporate
Governance
Understanding
►
Model Risk
Management
Page 25
31 May 2013
Challenges in Validating Catastrophe Models
Challenge
Improving knowledge flow
Model Risk Management. Is the cat model
really reasonable?
►
Scenario: After a validation exercise, model C is
selected for the IM.
Loss (€)
Non-modelled
Risks
Corporate
Governance
Model A
Model B
Model C
0
Understanding
50
100
150
Return Period
31 May 2013
250
►
Model C is revised. The company doesn’t approve.
►
Validation Challenge
Robustness of the IM
Model Risk
Management
Page 26
200
Challenges in Validating Catastrophe Models
Summary
Validating catastrophe models is challenging
► Catastrophe
models are complex, and comprise a number of sub-modules
► Each
sub-module is subject to significant uncertainties
► Data
to validate them is generally limited, but there are tools to help
Validation needs to be seen in a broad context
► Internal
Model perspective includes adequate understanding by stakeholders,
corporate governance, model risk management and non-modelled risks
► It
also considers the interface between cat models and capital models, pricing
and accumulation management tools
► Effort
Page 27
needs to be proportional to materiality
31 May 2013
Challenges in Validating Catastrophe Models
Contact
Paul Rockett
Senior Manager
T: +44 (0)20 7951 1098
M: +44 (0)78 2408 4806
E: prockett@uk.ey.com
Page 28
31 May 2013
Challenges in Validating Catastrophe Models
Reliance
►This
document is provided on the basis that you agree that Ernst & Young LLP (including its partners and staff) accepts
no responsibility and shall have no liability in contract, tort or otherwise in relation to the contents of this document and
that any use you make of this document is entirely at your own risk.
►The
information in this presentation pack is confidential and contains proprietary information of Ernst & Young LLP. It
should not be provided to anyone other than the intended recipients without our written consent.
►The
information in this pack is intended to provide only a general outline of the subjects covered. It should not be
regarded as comprehensive or sufficient for making decisions, nor should it be used in place of professional advice.
►Accordingly,
Ernst & Young LLP accepts no responsibility for loss arising from any action taken or not taken by anyone
using this pack.
►The
information in this pack will have been supplemented by matters arising from any oral presentation by us, and
should be considered in the light of this additional information.
Page 29
31 May 2013
Challenges in Validating Catastrophe Models
Thank you