Reserve Variability:
From Methods to Models
Mark R. Shapland, FCAS, ASA, MAAA
Casualty Actuaries of New England
Spring Meeting – Sturbridge, MA
March 20, 2007
© Milliman, Inc. 2005-2007. All Rights Reserved.
Overview

Definitions of Terms

Ranges vs. Distributions

Methods vs. Models

Types of Methods/Models

What is “Reasonable?”

Model Evaluation
© Milliman, Inc. 2005-2007. All Rights Reserved.
Definitions of Terms




Reserve – an amount carried in the liability section
of a risk-bearing entity’s balance sheet for claims
incurred prior to a given accounting date.
Liability – the actual amount that is owed and will
ultimately be paid by a risk-bearing entity for claims
incurred prior to a given accounting date.
Loss Liability – the expected value of all estimated
future claim payments.
Risk (from the “risk-bearers” point of view) – the
uncertainty (deviations from expected) in both timing
and amount of the future claim payment stream.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Definitions of Terms

Process Risk – the randomness of future outcomes
given a known distribution of possible outcomes.

Parameter Risk – the potential error in the
estimated parameters used to describe the
distribution of possible outcomes, assuming the
process generating the outcomes is known.

Model Risk – the chance that the model (“process”)
used to estimate the distribution of possible
outcomes is incorrect or incomplete.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Definitions of Terms
Measures of Risk from Statistics:

Variance, standard deviation, skewness, average
absolute deviation, Value at Risk, Tail Value at Risk,
etc. which are measures of dispersion.

Other measures useful in determining
“reasonableness” could include: mean, mode,
median, pain function, etc.

The choice for measure of risk will also be
important when considering the “reasonableness”
and “materiality” of the reserves in relation to the
capital position.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Ranges vs. Distributions

A “Range” is not the same as a “Distribution”

A Range of Reasonable Estimates is a range
of estimates that could be produced by
appropriate actuarial methods or alternative
sets of assumptions that the actuary judges to
be reasonable.

A Distribution is a statistical function that
attempts to quantify probabilities of all
possible outcomes.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Ranges vs. Distributions
A Range, by itself, creates problems:
 A range can be misleading to the
layperson – it can give the impression
that any number in that range is equally
likely.

A range can give the impression that as
long as the carried reserve is “within the
range” anything is reasonable.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Ranges vs. Distributions
A Range, by itself, creates problems:
 There is currently no specific guidance
on how to consistently determine a
range within the actuarial community
(e.g., +/- X%, +/- $X, using various
estimates, etc.).

A range, in and of itself, needs some
other context to help define it (e.g., how
to you calculate a risk margin?)
© Milliman, Inc. 2005-2007. All Rights Reserved.
Ranges vs. Distributions
A Distribution provides:
 Information about “all” possible
outcomes.

Context for defining a variety of other
measures (e.g., risk margin, materiality,
risk based capital, etc.)
© Milliman, Inc. 2005-2007. All Rights Reserved.
Ranges vs. Distributions
A Distribution can be used for:
 Economic Capital
– Reserve Risk
– Pricing Risk


Pricing / ROE
Reinsurance Analysis
– Aggregate Excess
– Stop Loss


Dynamic Risk Modeling (DFA)
Strategic Planning / ERM
© Milliman, Inc. 2005-2007. All Rights Reserved.
Ranges vs. Distributions
Should we use the same:
 criterion for judging the quality of a
range vs. a distribution?

basis for determining materiality? risk
margins?

selection process for which numbers
are “reasonable” to chose from?
© Milliman, Inc. 2005-2007. All Rights Reserved.
Methods vs. Models

A Method is an algorithm or recipe – a
series of steps that are followed to
give an estimate of future payments.

The well known chain ladder (CL) and
Bornhuetter-Ferguson (BF) methods
are examples
© Milliman, Inc. 2005-2007. All Rights Reserved.
Methods vs. Models

A Model specifies statistical
assumptions about the loss process,
usually leaving some parameters to
be estimated.

Then estimating the parameters gives
an estimate of the ultimate losses and
some statistical properties of that
estimate.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Methods vs. Models

Many good probability models have been
built using “Collective Risk Theory”

Each of these models make assumptions
about the processes that are driving claims
and their settlement values

None of them can ever completely eliminate
“model risk”
“All models are wrong. Some models are
useful.”
© Milliman, Inc. 2005-2007. All Rights Reserved.
Types of Models
Triangle Based Models
Single Triangle Models
vs.
vs.
Individual Claim Models
Multiple Triangle Models
Conditional Models
vs.
Unconditional Models
Parametric Models
vs.
Non-Parametric Models
Diagonal Term
Fixed Parameters
vs.
No Diagonal Term
vs.
Variable Parameters
© Milliman, Inc. 2005-2007. All Rights Reserved.
Methods vs. Models

Processes used to calculate liability ranges
can be grouped into four general categories:
1) Multiple Projection Methods,
2) Statistics from Link Ratio Models,
3) Incremental Models, and
4) Simulation Models
© Milliman, Inc. 2005-2007. All Rights Reserved.
Multiple Projection Methods
Description:

Uses multiple methods, data, assumptions

Assume various estimates are a good proxy
for the variation of the expected outcomes
Primary Advantages:
 Better than no range at all
 Better than +/- X%
© Milliman, Inc. 2005-2007. All Rights Reserved.
Multiple Projection Methods
Problems:
 It does not provide a measure of the density of the
distribution for the purpose of producing a probability
function
 The “distribution” of the estimates is a distribution of
the methods and assumptions used, not a distribution
of the expected future claim payments.
 Link ratio methods only produce a single point
estimate and there is no statistical process for
determining if this point estimate is close to the
expected value of the distribution of possible
outcomes or not.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Multiple Projection Methods
Problems:
 Since there are no statistical measures for these
models, any overall distribution for all lines of
business combined will be based on the addition of
the individual ranges by line of business with
judgmental adjustments for covariance, if any.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Multiple Projection Methods
Uses:
 Data limitations may prevent the use of more
advanced models.
 A strict interpretation of the guidelines in ASOP No.
36 seems to imply the use of this “method” to
create a “reasonable” range
© Milliman, Inc. 2005-2007. All Rights Reserved.
Statistics from Link Ratio Models
Description:
 Calculate standard error for link ratios to calculate
distribution of outcomes / range
 Typically assume normality and use logs to get a
skewed distribution
 Examples: Mack, Murphy, Bootstrapping and others
Primary Advantages:
 Significant improvement over multiple projections
 Focused on a distribution of possible outcomes
© Milliman, Inc. 2005-2007. All Rights Reserved.
Statistics from Link Ratio Models
Problems:
 The expected value often based on multiple
methods
 Often assume link ratio errors are normally
distributed and constant by (accident) year – this
violates three criterion
 Provides a process for calculating an overall
probability distribution for all lines of business
combined, still requires assumptions about the
covariances between lines
© Milliman, Inc. 2005-2007. All Rights Reserved.
Statistics from Link Ratio Models
Uses:
 If data limitations prevent the use of more
sophisticated models
Caveats:
 Need to make sure statistical tests are satisfied.
 ASOP No. 36 still applies to the expected value
portion of the calculations
© Milliman, Inc. 2005-2007. All Rights Reserved.
Incremental Models
Description:
 Directly model distribution of incremental claims
 Typically assume lognormal or other skewed
distribution
 Examples: Bootstrapping, Finger, Hachmeister,
Zehnwirth, England, Verrall and others
Primary Advantages:
 Overcome the “limitations” of using cumulative values
 Modeling of calendar year inflation (along the
diagonal)
© Milliman, Inc. 2005-2007. All Rights Reserved.
Incremental Models
Problems:

Actual distribution of incremental payments may not
be lognormal, but other skewed distributions
generally add complexity to the formulations

Correlations between lines will need to be
considered when they are combined (but can
usually be directly estimated)

Main limitation to these models seems to be only
when some data issues are present
© Milliman, Inc. 2005-2007. All Rights Reserved.
Incremental Models
Uses:
 Usually, they allow the actuary to tailor the model
parameters to fit the characteristics of the data.
 An added bonus is that some of these models allow
the actuary to thoroughly test the model parameters
and assumptions to see if they are supported by the
data.
 They also allow the actuary to compare various
goodness of fit statistics to evaluate the
reasonableness of different models and/or different
model parameters.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Simulation Models
Description:
 Dynamic risk model of the complex interactions
between claims, reinsurance, surplus, etc.,
 Models from other groups can be used to create
such a risk model
Primary Advantage:
 Can generate a robust estimate of the distribution of
possible outcomes
© Milliman, Inc. 2005-2007. All Rights Reserved.
Simulation Models
Problems:
 Models based on link ratios often exhibit statistical
properties not found in the real data being modeled.
 Usually overcome with models based on incremental
values or with ground-up simulations using separate
parameters for claim frequency, severity, closure
rates, etc.
 As with any model, the key is to make sure the
model and model parameters are a close reflection
of reality.
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
$11M
$16M
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Premise:

We could define a “reasonable” range based on
probabilities of the distribution of possible
outcomes.

This can be translated into a range of liabilities
that correspond to those probabilities.
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
A probability range has several advantages:

The “risk” in the data defines the range.

Adds context to other statistical measures.

A “reserve margin” can be defined more precisely.

Can be related to risk of insolvency and materiality
issues.

Others can define what is reasonable for them.
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Comparison of “Reasonable” Reserve Ranges
by Method
Relatively Stable LOB
Method
More Volatile LOB
Low
EV
High
Low
EV
High
Expected +/- 20%
80
100
120
80
100
120
50th to 75th Percentile
97
100
115
90
100
150
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Comparison of “Normal” vs. “Skewed” Liability Distributions
“Normally” Distributed Liabilities
“Skewed” Liability Distribution
Probability
Probability
50th Percentile
Expected Value, Mode & Median
Value
50th Percentile
Mode
Median
Expected Value
Value
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Comparison of Aggregate Liability Distributions
Probability
Liability Distribution for Line A
50 th Percentile
Mode
Median
Value
Expected Value
Probability
Aggregate Liability Distribution with 100% Correlation
(Added)
50 th Percentile
Liability Distribution for Line B
Probability
Mode
Median
Value
Expected Value
50 th Percentile
Aggregate Liability Distribution With No Correlation
(Independent)
Value
Expected Value
50 th Percentile
Probability
Liability Distribution for Line C
50 th Percentile
Mode
Median
Value
Probability
Mode
Median
Mode
Median
Value
Expected Value
Expected Value
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Comparison of Required Capital
0% Correlation
100% Correlation
Expected Value
Probability
Probability
Expected Value
99th Percentile
99th Percentile
$500M
Liability Estimate
Required Capital = $1,000M
$1,500M
$500M
$1,100M
Liability Estimate
Required Capital = $600M
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
50 th Percentile
Probability
Reasonable & Prudent Margin
75th Percentile
Reasonable & Conservative Margin
Mode
Median
Expected Value
Liability Estimate
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Comparison of “Reasonable” Reserve Ranges
with Probabilities of Insolvency
“Low” Reserve Risk
Corresponding Surplus Depending on Situation
Loss Reserves
Situation A
Prob.
Amount Of Ins.
Situation B
Situation C
Amount
Prob.
Of Ins.
Amount
Prob.
Of Ins.
Amount
Prob.
100
50%
80
40%
120
15%
160
1%
110
75%
70
40%
110
15%
150
1%
120
90%
60
40%
100
15%
140
1%
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Comparison of “Reasonable” Reserve Ranges
with Probabilities of Insolvency
“Medium” Reserve Risk
Corresponding Surplus Depending on Situation
Loss Reserves
Situation A
Situation B
Prob.
Amount Of Ins.
Prob.
Amount Of Ins.
Situation C
Amount
Prob.
Of Ins.
40%
160
10%
100
40%
140
10%
80
40%
120
10%
Amount
Prob.
100
50%
80
60%
120
120
75%
60
60%
140
90%
40
60%
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Comparison of “Reasonable” Reserve Ranges
with Probabilities of Insolvency
“High” Reserve Risk
Corresponding Surplus Depending on Situation
Loss Reserves
Situation A
Prob.
Amount Of Ins.
Situation B
Situation C
Amount
Prob.
Of Ins.
Amount
Prob.
Of Ins.
Amount
Prob.
100
50%
80
80%
120
50%
160
20%
150
75%
30
80%
70
50%
110
20%
200
90%
-20
80%
20
50%
60
20%
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Satisfying Different Constituents:

Principle of Greatest Common Interest – the
“largest amount” considered “reasonable” when a
variety of constituents share a common goal or
interest, such that all common goals or interests are
met; and the

Principle of Least Common Interest – the
“smallest amount” considered “reasonable” when a
variety of constituents share a common goal or
interest, such that all common goals or interests are
met.
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
$11M
$16M
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Probability
50 th Percentile
75th Percentile
$16M
$11M
Mode
Median
Expected Value
Liability Estimate
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
$11M
Probability
50 th Percentile
75th Percentile
$16M
Mode
Median
Expected Value
Liability Estimate
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Probability
50 th Percentile
75th Percentile
$16M
$11M
Mode
Median
Expected Value
Liability Estimate
© Milliman, Inc. 2005-2007. All Rights Reserved.
What is “Reasonable”?
Probability
50 th Percentile
75th Percentile
$11M
Mode
Median
$16M
Expected Value
Liability Estimate
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Selection and Evaluation

Actuaries Have Built Many
Sophisticated Models Based on
Collective Risk Theory

All Models Make Simplifying
Assumptions

How do we Evaluate Them?
© Milliman, Inc. 2005-2007. All Rights Reserved.
Probability
How Do We “Evaluate”?
Liability Estimates
© Milliman, Inc. 2005-2007. All Rights Reserved.
Fundamental Questions

How Well Does the Model Measure
and Reflect the Uncertainty Inherent in
the Data?

Does the Model do a Good Job of
Capturing and Replicating the
Statistical Features Found in the Data?
© Milliman, Inc. 2005-2007. All Rights Reserved.
Modeling Goals

Is the Goal to Minimize the Range (or
Uncertainty) that Results from the
Model?

Goal of Modeling is NOT to Minimize
Process Uncertainty!

Goal is to Find the Best Statistical
Model, While Minimizing Parameter
and Model Uncertainty.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Selection & Evaluation Criteria

Model Selection Criteria

Model Reasonability Checks

Goodness-of-Fit & Prediction Errors
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Selection Criteria


Criterion 1: Aims of the Analysis
– Will the Procedure Achieve the Aims
of the Analysis?
Criterion 2: Data Availability
– Access to the Required Data
Elements?
– Unit Record-Level Data or
Summarized “Triangle” Data?
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Selection Criteria

Criterion 3: Non-Data Specific
Modeling Technique Evaluation
– Has Procedure been Validated
Against Historical Data?
– Verified to Perform Well Against
Dataset with Similar Features?
– Assumptions of the Model Plausible
Given What is Known About the
Process Generating this Data?
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Selection Criteria

Criterion 4: Cost/Benefit
Considerations
– Can Analysis be Performed Using
Widely Available Software?
– Analyst Time vs. Computer Time?
– How Difficult to Describe to Junior
Staff, Senior Management,
Regulators, Auditors, etc.?
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Reasonability Checks

Criterion 5: Coefficient of Variation
by Year
– Should be Largest for Oldest
(Earliest) Year

Criterion 6: Standard Error by Year
– Should be Smallest for Oldest
(Earliest) Year (on a Dollar Scale)
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Reasonability Checks

Criterion 7: Overall Coefficient of
Variation
– Should be Smaller for All Years
Combined than any Individual Year

Criterion 8: Overall Standard Error
– Should be Larger for All Years
Combined than any Individual Year
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Reasonability Checks
Accident Yr
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
Total
Mean
26,416
26,216
50,890
90,705
148,110
186,832
418,461
638,082
607,107
1,521,202
3,714,020
Standard
Coeffient of
Error
Variation
37,927
143.6%
38,774
147.9%
54,508
107.1%
74,824
82.5%
99,986
67.5%
117,230
62.7%
183,841
43.9%
268,578
42.1%
477,760
78.7%
1,017,129
66.9%
1,299,184
35.0%
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Reasonability Checks
Accident Yr
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
Total
Mean
25,913
25,708
50,043
89,071
145,388
183,864
411,367
628,347
1,113,073
1,263,550
3,936,326
Standard
Coeffient of
Error
Variation
37,956
146.5%
38,846
151.1%
54,780
109.5%
74,987
84.2%
100,373
69.0%
118,502
64.5%
185,211
45.0%
271,722
43.2%
229,923
20.7%
253,596
20.1%
599,048
15.2%
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Reasonability Checks


Criterion 9: Correlated Standard Error
& Coefficient of Variation
– Should Both be Smaller for All LOBs
Combined than the Sum of Individual
LOBs
Criterion 10: Reasonability of Model
Parameters and Development
Patterns
– Is Loss Development Pattern Implied
by Model Reasonable?
© Milliman, Inc. 2005-2007. All Rights Reserved.
Model Reasonability Checks


Criterion 11: Consistency of
Simulated Data with Actual Data
– Can you Distinguish Simulated Data
from Real Data?
Criterion 12: Model Completeness
and Consistency
– Is it Possible Other Data Elements or
Knowledge Could be Integrated for a
More Accurate Prediction?
© Milliman, Inc. 2005-2007. All Rights Reserved.
Goodness-of-Fit & Prediction Errors

Criterion 13: Validity of Link Ratios
– Link Ratios are a Form of Regression
and Can be Tested Statistically

Criterion 14: Standardization of
Residuals
– Standardized Residuals Should be
Checked for Normality, Outliers,
Heteroscedasticity, etc.
© Milliman, Inc. 2005-2007. All Rights Reserved.
Goodness-of-Fit & Prediction Errors
18,000
16,000
14,000
Loss @24
12,000
10,000
8,000
6,000
4,000
2,000
0
0
1,000
2,000
3,000
4,000
5,000
6,000
Loss @12
© Milliman, Inc. 2005-2007. All Rights Reserved.
Goodness-of-Fit & Prediction Errors
18,000
16,000
14,000
Loss @24
12,000
10,000
8,000
6,000
4,000
2,000
0
0
1,000
2,000
3,000
4,000
5,000
6,000
Loss @12
© Milliman, Inc. 2005-2007. All Rights Reserved.
Goodness-of-Fit & Prediction Errors
18,000
16,000
14,000
Loss @24
12,000
10,000
8,000
6,000
4,000
2,000
0
0
1,000
2,000
3,000
4,000
5,000
6,000
Loss @12
© Milliman, Inc. 2005-2007. All Rights Reserved.
Standardized Residuals
Plot of Residuals against Predicted
1.5000
1.0000
Residuals
0.5000
0.0000
-0.5000
-1.0000
-1.5000
-2.0000
5.0000
6.0000
7.0000
8.0000
Predicted
© Milliman, Inc. 2005-2007. All Rights Reserved.
Standardized Residuals
Plot of Residuals against Predicted
0.8000
0.6000
0.4000
Residuals
0.2000
0.0000
-0.2000
-0.4000
-0.6000
-0.8000
4.0000
5.0000
6.0000
7.0000
8.0000
Predicted
© Milliman, Inc. 2005-2007. All Rights Reserved.
Goodness-of-Fit & Prediction Errors


Criterion 15: Analysis of Residual
Patterns
– Check Against Accident, Development
and Calendar Periods
Criterion 16: Prediction Error and
Out-of-Sample Data
– Test the Accuracy of Predictions on
Data that was Not Used to Fit the
Model
© Milliman, Inc. 2005-2007. All Rights Reserved.
Standardized Residuals
Plot of Residuals against Accident Period
0.8000
0.8000
0.6000
0.6000
0.4000
0.4000
0.2000
0.2000
Residual
Residuals
Plot of Residuals against Development Period
0.0000
0.0000
0
-0.2000
-0.2000
-0.4000
-0.4000
-0.6000
-0.6000
-0.8000
2
4
6
8
10
12
-0.8000
Development Period
Accident Period
Plot of Residuals against Payment Period
Plot of Residuals against Predicted
1.0000
0.8000
0.6000
0.5000
0.4000
0.0000
2
4
6
8
10
12
0.2000
Residuals
Residual
0
-0.5000
0.0000
-1.0000
-0.2000
-1.5000
-0.4000
-0.6000
-2.0000
-0.8000
4.0000
-2.5000
Payment Period
5.0000
6.0000
7.0000
8.0000
Predicted
© Milliman, Inc. 2005-2007. All Rights Reserved.
Standardized Residuals
Plot of Residuals against Accident Period
800.00
800.00
600.00
600.00
400.00
400.00
200.00
200.00
Residual
Residuals
Plot of Residuals against Development Period
0.00
0.00
-200.00
-200.00
-400.00
-400.00
-600.00
-600.00
Development Period
Accident Period
Plot of Residuals against Predicted
800.00
800.00
600.00
600.00
400.00
400.00
Residuals
Residual
Plot of Residuals against Payment Period
200.00
0.00
200.00
0.00
-200.00
-200.00
-400.00
-400.00
-600.00
0
-600.00
Payment Period
200000
400000
600000
800000
1000000
1200000
1400000
Predicted
© Milliman, Inc. 2005-2007. All Rights Reserved.
Goodness-of-Fit & Prediction Errors

Criterion 17: Goodness-of-Fit
Measures
– Quantitative Measures that Enable
One to Find Optimal Tradeoff
Between Minimizing Model Bias and
Predictive Variance
• Adjusted Sum of Squared Errors (SSE)
• Akaike Information Criterion (AIC)
• Bayesian Information Criterion (BIC)
© Milliman, Inc. 2005-2007. All Rights Reserved.
Goodness-of-Fit & Prediction Errors


Criterion 18: Ockham’s Razor and
the Principle of Parsimony
– All Else Being Equal, the Simpler
Model is Preferable
Criterion 19: Model Validation
– Systematically Remove Last Several
Diagonals and Make Same Forecast
of Ultimate Values Without the
Excluded Data
© Milliman, Inc. 2005-2007. All Rights Reserved.
Questions?
© Milliman, Inc. 2005-2007. All Rights Reserved.