The COTOR Challenge
Committee on the Theory of Risk
November 2004 Annual Meeting
History of the Challenge

Last spring a COTOR member challenged
actuarial geeks to estimate 500k xs 500k
layer based on list of 250 claims

Emails flew back and forth furiously

A number of different approaches were
used

Literature about heavy tailed distributions
was recommended

Winner was Phil Heckman using mixture of
2 lognormals
History cont.


Criticism existed around the sample since
some sample statistics were too far from
the real distribution
COTOR feels that the solution of this
problem is of interest ot the actuarial
community
• Our data is almost never normal/lognormal
• Our data is typically heavy tailed
• It is likely that in many real situations, a
sample of 250 claims would not represent a
random draw from any distribution
History cont.

Another challenge was issued under
well defined conditions

Stuart Klugman picked the sample

250 claims randomly generated from
an inverse transformed gamma

Challenge was to estimate severity in
the $5M xs $5M layer (mean and
95% confidence intervals)
The Sample
250 claims randomly selected from an inverse transformed gamma
Claim Size
Count
Greater than 5,000,000
1
500,000 to 1,000,000
2
100,000 to 500,000
7
50,000 to 100,000
10
25,000 to 50,000
8
10,000 to 25,000
26
5,000 to 10,000
30
2,500 to 5,000
56
1,000 to 2,500
74
500 to 1,000
32
250 to 500
4
Under 250
0
Purpose of Session

Raise awareness of audience of how
frequently extreme values need to be
dealt with

Present relatively easy to use
approaches

Make audience aware of how difficult
this problem is to solve
Normal Distribution Assumption

The normal or lognormal assumption is
common in finance application
• Option pricing theory
• Value at risk
• CAPM

Evidence that asset return data does not
follow the normal distribution is widely
available
• 1968 Fama paper in Journal of the American
Statistical Association
Test of Normal Distribution
Assumption
Normal Q-Q Plot of Monthly Return on S&P
1.15
T
h
e
o
r
e
t
i
c
a
l
V
a
l
u
e
1.10
1.05
1.00
0.95
0.90
0.85
0.8
0.9
1.0
Observed Value
1.1
1.2
1.3
Test of Normal Distribution
Assumption
Descriptive Statistics
Monthly Return on S&P
Valid N (lis twis e)
N
Statis tic
251
251
Mean
Statis tic
.9931
Std.
Deviation
Statis tic
.04585
Skewness
Statis tic
Std. Error
1.410
.154
Kurtos is
Statis tic
Std. Error
6.081
.306
Consequences of Assuming
Normality

The frequency of extreme events is
underestimated – often by a lot

Example: Long Term Capital
• “Theoretically, the odds against a loss such as
August’s had been prohibitive, such a debacle
was, according to mathematicians, an event so
freakish as to be unlikely to occur even once
over the entire life of the universe and even
over numerous repetitions of the universe”
 When Genius Failed by Roger Lowenstein, p. 159
Criteria for Judging

New and creative way to solve the
problem

Methodology that practicing actuaries
can use

Clarity of exposition

Accuracy of known answer

Estimates of confidence interval
Table of Results
Respond
er
Mean
A
9,500.0
0
B
Lower
CL
Upper
CL
Method
450.00
17,500.
00
Inverse Logistic Smoother
6,000.0
0
0.00
26,000.
00
Kernel Smoothing/Bootstrapping
C
12,533.
00
2,976.00
53,049.
00
Log Regression of Density Function on
large claims
D
2,400.0
0
?
?
E
6,430.0
0
1,760.00
14,710.
00
Fit distributions to triple logged data.
Used Bayesian approach for mean and
CI
F1
10,282.
00
2,089.00
24,877.
00
Scaled Pareto
F2
30,601.
00
6,217.00
74,038.
00
Pareto
G
4,332.6
5
297.34
7,645.8
6
Empirical Semi Smoothing
Generalized Pareto
Observations Regarding
Results





These estimations are not easy
Nearly 13 to 1 spread between lowest and
highest mean
Only 10% of answers came within 10% of
right result
All responders recognized tremendous
uncertainty in results (range from upper
to lower CL went from 8 to infinity)
Our statistical expert could not understand
the description of the method of 30% of
the respondents
Observations



All but 2 of the methods relied on approaches commonly
found in the literature on heavy tailed distributions and
extreme values
It is clear that it is very difficult to get accurate estimates
from a small sample
The real world is even more challenging than this
• 250 claims probably don’t follow any known distribution
• Trend
• Development
• Unforeseen changes in environment
• Consulting with claims adjusters and underwriters should
provide valuable additional insights
Observations





The closest answer was 5% below the true
mean
Half of the responses below the true mean,
Half were above
Average response was 40% higher than the
mean
Average response (ex outlyer) was within 2%
of the mean
Read:
“The Wisdom of Crowds: Why the Many are Smarter
than the
Few and How Collective Wisdom Shapes
Business, Economics,
Societies and Nations”
by: James Surowiecki

Implications for Insurance Companies?
Speakers

Meyers

Evans

Flynn

Woolstenhulme

Venter

Heckman
Announcement of Winners

Louise Francis – COTOR Chair
Possible Next Steps
Make the results of the challenge
available to the membership
 COTOR subcommittee to evaluate
how to make techniques readily
available
 Another round making the challenge
more real world
 Include trend and development
 Give multiple random samples
