A View Inside the “Black Box”:
A Review and Analysis of Personal Lines
Insurance Credit Scoring Models Filed in the
State of Virginia
By
Cheng-sheng Peter Wu, FCAS, ASA, MAAA
John Lucker, CISA
Deloitte Consulting, LLP
CAS 2004 Ratemaking Seminar, Call-3
Philadelphia
March 11, 2004
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
-2-
Our Efforts on this Topic to Date
• Does Credit Work?
“Does Credit Score Really Explain Insurance Losses?
Multivariate Analysis from a Data Mining Point of View”
• How Can You Go Beyond Credit?
“Mining the Most from Credit and Non-Credit Data”
• How Do Credit Score Models Work?
“A View Inside the “Black Box”: A Review and Analysis of
Personal Lines Insurance Credit Scoring Models Filed in
the State of Virginia”
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
-3-
The Motivation for Our Paper
What we did want to do?
What we didn’t want to do?
• Contribute substantive
material and insights to the
ongoing debate over credit
scoring
• Attribute our findings directly to
the filing companies and their
business practices
• Assist companies, regulators
and the public in
understanding how the Credit
Scoring “Black Box” works
• Expose proprietary information
beyond what is publicly available
• Render opinions on the
superiority of one model over
another
• Show the similarities and
differences in credit scoring
models
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
-4-
The History of Personal Lines
Pricing and Class Plans
• Few class plan factors before World War II
• Proliferation of class plan factors after the war
• Class plans for Personal Auto – territory, driver, vehicle,
coverage, loss and violation, others, tiers/company, etc.
• Class plans for Homeowners – territory, construction
class, protection class, insurance amount, coverage, prior
loss, others, tiers/company, etc.
• Insurance credit scoring started in late 80’s and early 90’s
as research and a developing concept – became
widespread from the mid-1990’s onward
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
-5-
The History of Personal Lines
Credit Scoring
• Credit Score was the first important rating factor identified in 20 years
• Credit Score is a composite multivariate score vs. raw credit info
• Until recently, it was viewed as a “secret weapon” worthy of secrecy
• Today 90+% of Personal Lines insurers use credit scoring for some
form of new biz acquisition, risk selection, pricing, and renewal
• Credit Score has been easy and relatively inexpensive to get, “quiet”
to use, confidential, and straight forward in its implementability
• Today, it is the hottest, most widely contested and debated topic in the
Personal Lines insurance industry
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
-6-
The Current Environment
• Continues to be a hot topic for debate
• Many entities have conducted studies on the true
correlation with loss ratio and the Disparate Impact issue
• Virginia, Washington, Maryland, Texas, Missouri
• NAIC, CAS, Tillinghast Towers-Perrin, EPIC
• Many states have restricted (or are considering restricting)
the usage of the score or certain credit information
• More states want the “black box” filed and opened
• More companies are considering proprietary credit models
for greater transparency and non-credit scoring models
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
-7-
Study of VA Credit Score Filings
• Insurers filed over 40 credit scoring models in
Virginia in 2002
• Deloitte obtained copies of 11 of these filings,
covering:
• 9 filings for Personal Auto and 2 filings for Homeowners
• 8 insurance groups
• $45 billion in personal lines premiums
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
-8-
Types of Models
• Industry Model – Fair Isaac (FICO)
• 4 different FICO scores used by 3 insurance groups
• Uses credit information from TransUnion
• Multiple models by line, by market segment, and by version
• Industry Model – ChoicePoint
• 3 insurance groups for Auto
• Uses credit information from Experian
• Open model
• Insurance Company Proprietary / Custom Models
• 2 insurance groups
• Uses credit information from TransUnion
• Home and Auto are the same models
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
-9-
Scoring Functions
• Rule-based
•
•
•
•
Table driven format
If factor x is equal to y, then get z points, etc...
Sum all the points to generate a raw score
All FICO models and one of the two proprietary models use this
technique
• Formula
• Can be linear or non-linear
• Need to determine the parameters/weights
• One of the two proprietary models uses this technique
• The ChoicePoint model is a mix of the two, but is more of
a formula function
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 10 -
Scoring Functions
• Rule-based
• Advantages: simplicity, easier to explain, easier
integration with a company’s class plan
• Disadvantages: must predetermine the groupings,
potential limitations in the number of variables used in
the model
• Formula
• Advantages: easier to include more variables, formula
is a direct result of the modeling process and doesn’t
require transformation
• Disadvantages: more difficult to explain and interpret
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 11 -
Scoring Functions
• One way to compare a rule-based function and a
formula function: review the “delta”
• A formula function:
• Z = 2 X + 3 Y,
• An increase of 1 in X – an increase of 2 in Z
• An increase of 1 in Y – an increase of 3 in Z
• A rule-based function:
• If X = 1 then 20 points; if X=2 then 40 points, if X=3 then 60
points.
• If Y=1 then 10 points; if Y=2 then 40 points; if Y=3 then 70
points.
• These two functions are essentially the same!
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 12 -
Scoring Process
• Step 1 – calculate the raw score
• Step 2 – scale the raw score to the final score,
(Score Scaling)
• Transform a raw score to a final score (e.g. 0.34778
becomes 570)
• Monotonic functions are used
• Simple Scaling Functions vs Complex Scaling Functions
• Simple scaling functions: linear shift and expand
(a*score+b)
• Complex scaling: non-linear formula / transformation
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 13 -
Scoring Scaling Function:
Simple vs Complex
Simple Scaling Function
Complex Scaling Function
100
900
90
800
80
70
Raw Score
Raw Score
700
600
500
400
60
50
40
30
20
10
0
300
200
300
400
500
600
700
0.0
Final Score
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
5.0
10.0
Final Score
- 14 -
15.0
20.0
Score Ranges
357 - 818 Auto
Fair Isaac (higher score is better)
326 - 845 Auto
389 - 806 Auto
200 - 884 Home
ChoicePoint (higher score is better)
220 – 998
Proprietary (higher score is worse)
#1 100-1000
#2 1-100
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 15 -
Model Variables
• Fair Isaac – 10 to 13 variables, depending on the
models
• ChoicePoint – 29 variables for “thin file” scores,
and 37 variables for “thick file” scores
• Proprietary #1 – 10 variables
• Proprietary #2 – 36 variables
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 16 -
Model Variables
More
= worse
Recent
Late Payment / Past Due / Delinquency
Public Derogatory
Leverage Ratio, Unsatisfactory, Default, Bad Debt
Info (in all but one proprietary model)
Collection (in all but one proprietary model)
Inquiry (in all but one FICO model)
# of Accounts, Account History, Account History,
Recent Account Activity
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 17 -
Varies
More Comparisons Possible
• More model comparisons that could be
performed:
• Variable strength comparison between models
• Score changes from one model to another
• Model lift and stability from one model to another
• To find out the answers to these questions:
“Normalization of the Score Ranking and
then Testing with Real Data”
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 18 -
Normalization of the Score
Ranking & Testing with Real Data
• Score a group of risks with different models
• Sort the scores for the risks from the best to the worst
• Group the sorted risks into deciles (or quintiles, quartiles,
etc)
• Use the deciles (or quintiles, quartiles, etc) as the “score”
for comparison between models for
• Predictive Power / Lift
• Variable Strength
• Score Changes / Migration
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 19 -
Score Comparison Between 2
FICO Models - Original Score
Exhibit 8
Score Distribution Comparison between a Auto Model and a Home Model by Fair Isaac
3000
Number of Data Points
2500
2000
1500
1000
500
0
<=300
325
375
425
475
525
575
625
Mid Point of the Score Range
675
725
775
Fair Isaac - Assist 2.0, Preferred Auto
Fair Isaac - Assist 2.1, HO3
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 20 -
>800
Score Comparison Between 2 FICO Models –
Normalization with Decile Score Ranking
Change in Decile
Ranking
% of Data
0
27.0%
+-1
32.2%
+-2
20.9%
+-3
11.6%
+-4
5.8%
+-5
2.1%
+-6
0.4%
Total
100.0%
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 21 -
Considerations for Building or
Selecting a Model
• How does your competitive advantage impact your choices?
• Degree of predictive power desired relative to other factors?
• How stable is the score from one period to the next?
• How flexible do you want your company’s models to be?
• What is your resource availability for development, “care & feeding”?
• What are your expectations with regards to the regulatory climate?
• What is the impact of the regulatory environment on your company?
• What is your potential cost savings for credit scores & credit data
purchases?
• How can model performance be measured and monitored?
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 22 -
Conclusions
• All models are similar in the type, form and structure of the
variables and the data sources they come from
• The models use different scoring functions and
implementation approaches
• The models produce scores with different score ranges
• To perform a real comparison we must rank test the
various models with real data
• This will continue to be a hot topic in the industry – stay
tuned…!
Copyright © 2004 Deloitte Development LLC. All Rights Reserved.
- 23 -