Research on the Effectiveness of a
Trend Test in the
Property/Casualty RBC Formula
Chris Nyce
KPMG Senior Manager
kpmg
Disclaimer

These results are based on research conducted by
a subgroup of the American Academy P/C RBC
Committee

Views expressed today are based on the research,
but do not necessarily reflect the views of the
Academy, KPMG, or the NAIC who of course makes
all decisions about changes to the RBC formula

Examples used are illustrative, and not a reference
to any specific company

Anyone who says otherwise is not only wrong, but
is itching for a fight
Our Mission

Began the research with a charge
- “Given the use of a trend test in the life RBC formula, is the
application of a trend test in the Property/Casualty RBC formula a
good idea?”

Our interpretation- Not a “Yes/No” question

Instead-”What is the most effective way of differentiating
between companies above the Company Action Level that
are likely to fall below it, and those that are likely to remain
above it.”

We approached this with a one year time horizon, i.e. based
on observable data this year, what will happen next year
Status of the Work

Ideas to be discussed here have cleared the
Academy RBC committee

Formal report has been written, and is being
modified based on comments for review by the
AAA counsel
- Note this is the normal procedure for AAA committee
work product

Will be submitted to NAIC for consideration at the
June meeting
Background

Life test currently uses a trend test

Applies to companies with RBC between 250% and
the company action level (“CAL”) of 200%
- RBC ratio is the ratio of capital to RBC required capital

Life test looks at past changes in RBC ratio
- Max of last year and the three year average RBC decline
for each company
- Subtract result from current RBC ratio
- If below 190%, company is deemed to be at the CAL

Note that even before our work, the feeling of
committee members was that the life trend test did
not work well for P/C companies
- We quickly confirmed this to be true
Our Approach

Basic question-”What is the most effective predictor of
decline in capital adequacy?”

In general terms, used “Hypothesis Testing”

Examined specific cases of past company failures

Formulated hypotheses on the causes of RBC decline

Tested the hypotheses using statistical tests on annual
statement data

Conducted additional tests by examining the
effectiveness retrospectively

Measured the results using a specific set of metrics

Selected one approach that produced the best metrics
Boundaries of our Study

Did not constrain ourselves to examining the life
formula

Based on publicly filed data from the NAIC blank

Outcome has to be intuitively correct, and simple

All research also from public data sources

For NAIC data, company names remained
confidential

Outcome had to be based on empirical data, not on
our preconceived opinions
Data Considerations

For “micro” analysis we used public data sources
such as AM BEST and press reports

NAIC provided 5 year history of all requested data
elements
- Confidential as to company identifier
- About 2400 companies
- Used data through 2002 for statistical tests, updated
through 2003 for retrospective test

We scrubbed the data, in general separately for
each test to maximize data points utilized
- Screened out invalid entries and extreme values
Micro Results-Initial Hypotheses

Companies we examined could be characterized as
experiencing trouble due to various causes, such
as:
- High levels of reinsurance recoverables, causing high
leverage in estimating reserves, and exposure to
disputed balances
- High leverage of premiums and reserves to surplus
- Reserve inadequacies coming to roost
- Poor operating results
- Fraud and misrepresentation
- Ill-liquid or incorrectly valued assets
- Under-funded pensions: (usually a contributor, not a
cause)
What is the Best Early Indicator of
Future Capital Declines?
Lack of Liquidity
Bad Assets
Poor
Profitability
Reserve
Inadequacies
Fraud
Past
Capital
Declines
Leverage
Overall “Macro” Approach

Performed statistical tests on the NAIC database
- Explored the basic relationships behind each hypothesis

Performed retrospective tests on characteristics of
companies just prior to falling to the CAL

Set up metrics to evaluate the outcome of the
retrospective test

Determined recommendations based on all of the
above
Statistical Tests

Explored relationships between hypothesized variables

Performed tests on the NAIC database of 2400
companies for 5 years ending 2002

Looked for statistical tendencies

Generally used correlation and regression analysis
- Examined the percentage of variation explained
- Calculated the measures of significance

Used to corroborate and explain retrospective result

Note that a poor result in our tests does not necessarily
mean that the measure is not good for IRIS or other
financial evaluations
- And high correlations don’t necessarily mean the hypothesis
would form a good trend test
Statistical Test of Life Type Trend Test

Does a simple life type of trend test work?
- Correlation between year to year changes in RBC ratio
for all companies= -23% (wrong sign)
- For only companies near the CAL = 1%
- In 2001 and 2002, the direction of the change in
subsequent years was only the same 41% of the time

Changes in market asset valuations dominated any
characteristics of companies themselves

Implication: Life type of trend test is worse than
random guessing for P/C Companies
What About Underwriting Results and
Reserve Runoff?

Underwriting Results
- Correlation between subsequent year combined ratios=
25% to 34% between 2000-2002
- For only companies near CAL correlation is 33% to 75%
(highly significant)

Reserve Runoff
- Correlation between subsequent year runoff ratios=33%
to 37% between 2000 and 2002
- For only companies near CAL correlation is 29% to 35%

This is good and bad news
- Statistical relationship is strong
- But still only predicts a portion of the subsequent year
outcome
What is the Predictive Power of
Leverage?

Gross Leverage
- Correlation between gross leverage and subsequent year RBC
ratio change= -1% to 1% between 2000-2002
- For only companies near CAL correlation is –5% to –3% (not
significant)

Net Leverage
- Correlation between net leverage and subsequent year RBC ratio
change= 3% to 4% between 2000-2002
- For only companies near CAL correlation is 1% to 16% (wrong
sign)

This is not a good outcome
- Statistical relationship is weak and sign is sometimes wrong
Well then it must be Liquidity?

Correlation between liquid assets to surplus and
subsequent RBC change is –4% to 1% over 2001 to
2002

Depending on sample, relationship is not
significant, or sign is wrong
In 2002 and 2003, Portion of
Companies Falling to CAL
RBC Ratio in
Prior Year
Total
Companies in
Sample
Number of
Percentage
Companies
Falling to CAL
Falling to CAL
200% to 300%
314
30
9.6%
300% to 350%
166
9
5.4%
350% to 400%
205
4
2.0%
400% to 450%
176
3
1.7%
Greater than
200%
3582
55
1.5%
Retrospective Tests

Performed on NAIC database of 2400 companies ending
2003

Generally “Yes/No”
- Measured whether the hypothesis accurately predicted the
subsequent year outcome, or not
- Therefore, scrubs were oriented toward invalids, but not toward
extremes

Measured on three metrics
- Effectiveness-Percentage of overall correct predictions
- False alarms-Percentage of companies flagged that did not
deteriorate to CAL
- Failing Companies Flagged-Percentage of companies that
subsequently declined to the CAL that were correctly flagged
Retrospective Approach

Started by setting a threshold such as leverage
above industry average, or combined ratio above
110%, etc.
- Based on the threshold, companies were “flagged” or “not
flagged”

Allowed for mixed approaches;
- Leverage, reserve runoff, and combined ratio
- Reserve runoff and combined ratio
- Three year tests of reserve runoff and combined ratio

Adjusted the threshold to optimize the metrics
- Based on trial and error

Understand, this test tells us not what causes RBC
decline, but what best predicts it
- Although the implication for the cause is pretty clear
Retrospective Metrics
Life Trend Test
Test Year UW Ratio
Three Year Average UW Ratio
Test Year Runoff Ratio
Three Year Average Runoff Ratio
Gross Leverage, End of Test Year
Net Leverage, End of Test Year
Composite UW and Runoff
Composite uw/runoff/leverage
Two Tiered Underwriting Test
Threshold
trend
-20%
-17%
7%
5%
650%
350%
Composite
Composite
-20%/34%
Ratio of failing False Effectiveness Total # of
cos flagged alarms/total Ratio Companies
62%
47%
50%
480
67%
30%
67%
480
64%
36%
61%
480
46%
43%
53%
480
46%
37%
59%
480
56%
48%
48%
480
54%
46%
51%
480
64%
34%
63%
480
69%
42%
56%
480
67%
26%
71%
480
Why not 100% Effective

Formula approach doesn’t account for capital
changes (contributed, dividend)

Financial statements can always be subject to
restatement

RBC ratio decline could involve fraud, or an other
wise solid looking asset losing value
- Or pension funding

The statistical relationship is strong, but is not
100% predictive of direction and magnitude

Need to keep the test simple
An Effective Approach Based on Tests
RBC
Ratio
Current Year
Combined Ratio
Company Status
200%300%
Greater than 120%
CAL
Less than 120%
No Regulatory Action
Greater than 134%
CAL
Less than 134%
No Regulatory Action
All
No Regulatory Action
300%350%
Above
350%