Testing Models on
Simulated Data
Presented at the
CAS Annual Meeting
November 13, 2007
Glenn Meyers, FCAS, PhD
ISO Innovative Analytics
The Application
Estimating Loss Reserves
• Given a triangle of incremental paid losses
– Ten years with 55 observations arranged by
accident year and settlement lag
• Estimate the distribution of the sum of the
remaining 45 accident years/settlement
lags
• Loss reserve models typically have many
parameters
– Examples in this presentation – 9 parameters
Danger – Possible Overfitting!
• Model describes the sample, but not the
population
• Understates the range of results
• The range is the goal!
• Is overfitting a problem with loss reserve
models?
• If so, what do we do about it?
Outline
• Illustrate overfitting with a simple example
– Example – fit a normal distribution with three
observations
– Illustrate graphically the effects overfitting
• Illustrate overfitting with a loss reserve
model
– Reasonably good loss reserve model
– Show similar graphical effects as normal
example
Normal Distribution
• MLE for parameters m and s
1 n
m
ˆ   xi and s
ˆ 
n i 1
• n = 3 in these examples
1 n
2
 xi  mˆ 

n i 1
Simulation Testing Strategy
• Select 3 observations at random
Population - m = 1000, s = 500
• Predict a normal distribution using the
maximum likelihood estimator
• Select 1,000 additional observations at
random from the same population
• Compare distribution of additional
observations with the predicted distribution
Simulated Fits
Simulation 24
7000
7000
9000
11000
Simulation 91
7000
9000
11000
Loss
Loss
Simulation 132
Simulation 4
9000
Loss
11000
7000
9000
Loss
11000
PP Plots
0.6
0.4
0.2
0.0
Predicted Percentiles
If predicted percentiles
are uniformly
distributed, the plot
should be a 45o line.
0.8
1.0
Plot
Predicted Percentiles x Uniform Percentiles
0.0
0.2
0.4
0.6
Uniform Percentiles
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.4
0.8
Simulation 91
0.0
Predicted Percentiles of Outcomes
0.4
0.8
Simulation 24
0.0
Predicted Percentiles of Outcomes
PP Plots
0.0
0.2
0.4
0.6
0.8
Uniform Percentiles
1.0
0.6
0.8
1.0
0.4
0.8
Simulation 4
0.0
Predicted Percentiles of Outcomes
0.4
0.8
Simulation 132
0.0
0.4
Uniform Percentiles
0.0
Predicted Percentiles of Outcomes
Uniform Percentiles
0.2
0.0
0.2
0.4
0.6
0.8
Uniform Percentiles
1.0
Simulated Fits
Simulation 24
7000
7000
9000
11000
Simulation 91
7000
9000
11000
Loss
Loss
Simulation 132
Simulation 4
9000
Loss
11000
7000
9000
Loss
11000
View Maximum Likelihood
As an Estimation Strategy
• If you estimate distributions by maximum
likelihood repeatedly, how well do you do
in the aggregate?
• Consider a space of possible parameters
for a model
• Select parameters at random
– Select a sample for estimation (training)
– Select a sample for post-estimation (testing)
Continuing Prior Slide
• Select parameters at random
– Select a sample for estimation (training)
– Select a sample for post-estimation (testing)
• Fit a model for each training sample
• Calculate the predicted percentiles of the
testing sample.
• Combine for all samples.
• In the aggregate, the predicted percentiles
should be uniformly distributed.
1.0
PP Plots for Normal Distribution (n = 3)
0.6
0.4
0.0
0.2
Predicted Percentiles of Outcomes
0.8
ML
Bias Corrected ML
0.0
0.2
0.4
0.6
0.8
1.0
Uniform Percentiles
S-Shaped PP Plot - Tails are too light!
Problem – How to Fit Distribution?
• Proposed solution – Bayesian Analysis
• Likelihood = Pr{Data|Model}
• We need Pr{Model|Data} for each model in
the prior
Predictive Distribution 
 Model Distribution  Pr Model|Data
Model
Bayesian Fits
• Predictive
distributions are
spread out more
than the MLE’s.
• On individual fits,
they do not always
match the testing
data.
7000
Simulation 24
Simulation 91
Bayes
ML
Bayes
ML
9000
11000
7000
11000
Loss
Loss
Simulation 132
Simulation 4
Bayes
ML
7000
9000
9000
Loss
Bayes
ML
11000
7000
9000
Loss
11000
Bayesian Fits as a Strategy
Combined PP Plot for
0.0
0.2
0.4
0.6
0.8
1.0
Bayesian Fitting Strategy
Predicted Percentiles of Outcomes
• Parameters of model
were selected at
random from the prior
distribution
• Near perfect uniform
distribution of
predicted percentiles
• At least in this
example, the
Bayesian strategy
does not overfit.
0.0
0.2
0.4
0.6
Uniform Percentiles
0.8
1.0
Analyze Overfitting in Loss
Reserve Formulas
• Many candidate formulas - Pick a good one
E Paid LossAY ,Lag   PremiumAY  ELR  Dev Lag
• Paid LossAY,Lag ~ Collective Risk Model
• Claim count distribution is negative binomial
• Claim severity distribution is Pareto
– Claim severity increases with settlement lag
• Calculate likelihood using FFT
Simulation Testing Strategy
• Select triangles of data at random
– Payment pattern at random
– ELR at random
– {Loss|Expected Loss} for each cell in the
triangle from Collective Risk Model
• Randomly select outcomes using the same
payment pattern and ELR
• Evaluate the Maximum Likelihood and
Bayesian fitting methodology with PP plots.
Background on Formula
• Same formula appeared in SessionV2
• Fit a Bayesian model to over 100 insurers
and produced an “acceptable” combined
PP plot on test data from six years later.
• This paper tests the approach to simulated
data, rather than real data.
0.25
Prior Payout Patterns
0.00
0.05
0.10
D ev Lag
0.15
0.20
Fixed Subpopulation
2
4
6
L ag
8
10
Prior Probabilities for ELR
ELR
0.600
0.625
0.650
Prior
3/24
4/24
5/24
ELR
0.675
0.700
0.725
Prior
4/24
3/24
2/24
ELR
0.750
0.775
0.800
Prior
1/24
1/24
1/24
Selected Individual Estimates
Simulation 4
Simulation 6
Bayes
ML
50000
70000
90000
110000
Bayes
ML
50000
70000
90000
Loss
Loss
Simulation 12
Simulation 17
Bayes
ML
50000
70000
90000
Loss
110000
110000
Bayes
ML
50000
70000
90000
Loss
110000
0.6
0.4
0.2
0.0
Predicted Probability
• PP plot reveals
the S-shape that
characterizes
overfitting.
• The tails are too
light
0.8
1.0
Maximum Likelihood Fitting Methodology
PP Plots for Combined Fits
0.0
0.2
0.4
0.6
Uniform Probability
0.8
1.0
0.6
0.4
0.2
0.0
Predicted Probability
Nailed the
Tails
0.8
1.0
Bayesian Fitting Methodology
PP Plots for Combined Fits
0.0
0.2
0.4
0.6
Uniform Probability
0.8
1.0
Summary
• Examples illustrate the effect of overfitting
• Bayesian approach provides a solution
• These examples are based on simulated
data, with the advantage that the “prior” is
known.
• Previous paper extracted prior distributions
from maximum likelihood estimates of
similar claims of other insurers
Conclusion
• It is not enough to know if assumptions are
correct.
• To avoid the light tails that arise from overfitting,
one has to get information that is:
Outside
The
Triangle