Using AFT and FSI Tools to Validate Components of
Interest Rate Risk Models
Page 1
ALCO Partners, LLC
Agenda
 Validation Tests of Prepayment Models
• Models applicable to securities
• Models applicable to portfolio loans
 Validation Tests of Stochastic Rate Generators using FSI
• Lattice
• Monte Carlo
Page 2
ALCO Partners, LLC
Regressor and Back-testing
 Regressor is applicable to AFT prepayment models:
• Securities
o MBS
o CMOs
 MBS data at the CUSIP level require extraction routines from
Bloomberg or Intex.
 CMO data at the CUSIP level require license with Intex, Chasen or
other cash flow model vendor.
 Agency CMO data can be extracted from Bloomberg.
Page 3
ALCO Partners, LLC
Regressor and Back-testing
 Batch “Back-test” in the Regressor Model
• Output created:
o
Actual CPR
o
AFT forecast of CPR using actual rates
• Output is written to text files with number of files equal
to the number of collateral types.
o Example: 300 CUSIPs of 15 and 30 FNMA, FHMLC, and
Whole loans might reside in six text files
• All CUSIPs can be run at once in Regressor with the
text files created automatically.
• Text files can be opened in EXCEL and standard
“goodness of fit” or other statistics applied.
Page 4
ALCO Partners, LLC
Regressor and Back-testing
 Once a macro is written, process takes less than an hour for the
entire list of CUSIPs (We did this for 300 CUSIPs.)
 Procedure can be learned in about 15 minutes via phone support, if
all required extract files are available.
 Regressor allows the aggregation across CUSIPs drawn from same
collateral. Performance on sub-portfolios can be assessed.
Page 5
ALCO Partners, LLC
Why can AFT Claim Regressor Provides a Back-test”?
 AFT Prepayment models are estimated using collateral data
• Securities are small samples of the collateral.
• Output in Regressor at CUSIP level is based on both
overlapping historical periods and projected periods, depending
on the length of history and when collateral model was
estimated.
o
Over the historical data model period this is an “in-sample” backtest.
o
When CPRs are projected beyond historical data period used for
building models, the test is an “out-of-sample” back-test.
 If the connectivity in the ALM system and AFT models has been
validated, then the ALM model is not needed for back-testing AFT
prepayment models.
Page 6
ALCO Partners, LLC
Back testing Sample Output from Regressor
55265KTS8
90
80
70
CPR (%)
60
50
40
30
20
10
R2 = 73%
Actual
Predicted
Sep-04
Jul-04
May-04
Mar-04
Jan-04
Nov-03
Sep-03
Jul-03
May-03
Mar-03
Jan-03
Nov-02
0
80
One of 180 graphs created for an ALCO Partners client
Page 7
ALCO Partners, LLC
Dynamic Aggregator and Back testing
 Dynamic Aggregator is applicable to portfolio loans
• Similar output created as in Regressor
o AFT prepayment models provide the basis for modeling.
Discrete prepayment factors can be scaled to improve
model fit.
o Historical data used to fit model and model statistics can be
created as part of the estimation process.
o In-sample and out-of-sample model testing can be applied
with similar statistics.
o If AFT-ALM model link has been validated, then the process
can substitute for back test using ALM model.
Page 8
ALCO Partners, LLC
Findings from Dynamic Aggregator Projects
 Dynamic Aggregator has coincidentally “solved” a classic data
bottleneck in many banking institutions
• IT departments now regularly archive historical data relevant
for testing prepayment models
• They are very efficient at providing raw data.
• They are inefficient at “mapping” data
o Internal resources for “research and analysis” are frequently low
on a priority list
o External projects where only raw data is required appear not to
face the same internal hurdles because they demand very few IT
resources
Page 9
ALCO Partners, LLC
Findings from Dynamic Aggregator Projects
 A part of prepayment model validation projects applicable to
portfolio mortgages is obtaining and mapping data. But this
process is no longer a large cost.
o Mapping costs have not been large. Key variable is number of
systems
o Mapping rules typically remain constant over time, so annual
updates become very cost effective
 Once the data has been mapped into Dynamic Aggregator, the
process for testing and is very cost-effective, particularly when the
process is done annually as required by banking regulations
Page 10
ALCO Partners, LLC
Testing Stochastic Rate Generators
 In many banks stochastic rate generators are being utilized to
measure value based IRR
• Base case is calibrated to market instruments
• There are known nuances and problems associated with this
• How does a bank “validate” a stochastic rate model?
Page 11
ALCO Partners, LLC
Testing Stochastic Rate Generators
 Approach I
• Obtain model documentation
• Build a separate model independently
• Perform double-blind tests and compare inputs and outputs
 Issues with Approach I
• Vendors may be unwilling to provide proprietary code
documentation
• Building a separate model can be very expensive, including its
validation
• Specialized coding/math expertise required to “build” a stochastic
rate generator
Page 12
ALCO Partners, LLC
Testing Stochastic Rate Generators
 Approach II
• Obtain a qualified stochastic rate generation system
• Calibrate to the same set of market instruments
• Compare inputs and outputs for tested samples
 FSI: A qualified stochastic rate generation system
Page 13
ALCO Partners, LLC
Testing Stochastic Rate Generators
 FSI is a fully specified two factor lattice & Monte-Carlo system
designed as a portfolio management product with validation
capabilities
• Provides a basis for comfort or discomfort depending on the model
results and how the model results are integrated with portfolio
management decisions
• Since it contains both single factor (BK) and two-factor processes,
it provides a basis for understanding the sensitivity of the risk
measure to the underlying process utilized
• Helps improve calibration procedures and allows a look into the
“black box” of stochastic rate generation capability
Page 14
ALCO Partners, LLC
Testing Stochastic Rate Generators
 Test Procedures
• Compared stochastic output from a widely used ALM model with a
stochastic generator
o Compared three portfolios in two periods
o
MBS & CMO
o
Portfolio FRMs
o
Callable Bonds
o Greeks are computed at the instrument level
o Market values and market values under rate shocks can be computed
o Market values changes compared over time at instrument level
o Results used to refine and understand calibration process and
reasonableness of risk results
Page 15
ALCO Partners, LLC
Callables: MV Sensitivity
12/31
3/30
Callables: MV Sensitivity Compared
Callables: MV Sensitivity Compared
8%
8%
FSI1F
% D MV
4%
6%
FSI2F
4%
2%
0%
0%
-2%
-4%
-4%
-6%
ALM
2%
-2%
-150
FSI1F
FSI2F
ALM
% D MV
6%
-6%
-100
0
100
150
200
-150
-100
Shock (bp)
0
100
150
Shock (bp)
Market value sensitivity measures for
callable bond portfolio where very similar
in both time periods
Page 16
ALCO Partners, LLC
200
Callables: Duration & Convexity
Duration
Convexity
Callables: FSI 2F vs. ALM Model
8
Callables: FSI 2F vs. ALM Model
8
Convexity (ALM)
Duration (ALM)
7
6
5
4
3
2
6
4
2
0
1
0
-2
0
1
2
3
4
5
6
Duration (FSI 2Factor)
7
8
-2
0
2
4
6
Convexity (FSI 2Factor)
Red line represents parity
Results indicated some non-random differences on
individual securities. Further investigation revealed pattern
associated with “at the money” calls. Cross checks with
market values showed sensitivity to calibration mechanics
Page 17
ALCO Partners, LLC
8
Callables: FSI 2 Factor vs. ALM Model: 12/31 to 3/31
How does Market Value Change over Time?
%D MV (ALM)
4%
Callables: 3/31 vs. 12/31, FSI vs. ALM Model
3%
2%
1%
0%
-1%
-1%
0%
1%
2%
3%
%D MV (FSI 2Factor)
Red line represents parity
Page 18
ALCO Partners, LLC
FRMs: MV Sensitivity
12/31
3/31
FRMs: MV Sensitivity
FRMs: MV Sensitivity
6%
4%
4%
2%
2%
0%
-2%
-4%
%D MV
%D MV
0%
FSI1F
FSI2F
ALM
-2%
-4%
-6%
FSI1F
FSI2F
ALM
-6%
-8%
-8%
-10%
-150
-10%
-100
0
100
Shock (bp)
150
200
-12%
-150
-100
0
100
150
Shock (bp)
Prepayment models were identical
Page 19
ALCO Partners, LLC
200
FRMs:
Duration & Convexity
Duration
Convexity
FRMs
6
Convexity (ALM)
5
Duration (ALM)
FRMs
0
4
3
2
-1
-2
-3
-4
1
0
-5
0
1
2
3
4
Duration (FSI 2Factor)
5
6
-5
-4
-3
-2
-1
0
Convexity (FSI 2Factor)
Red line represents parity
Page 20
ALCO Partners, LLC
FRMs: FSI (2 Factor) vs. ALM: 12/31 to 3/31
FRMs
% D MV (ALM)
0%
-1%
-2%
-3%
-3%
-2%
-1%
0%
% D MV (FSI 2Factor)
Red line represents parity
FRMs where there are good market value indicators
combined with the OAS calculations lead to similar results
Page 21
ALCO Partners, LLC
CMOs & MBS: MV Sensitivity
12/31
CMOs &MBS: MV Sensitivity
Compared (12/31/2004)
4%
2%
2%
0%
0%
  MV
  MV
4%
3/31
-2%
-4%
FSI1F
-2%
-4%
-8%
-150
ALM
-100
-6%
0
100
Shock (bp)
Page 22
FSI1F
FSI2F
FSI2F
-6%
CMOs &MBS: MV Sensitivity
Compared (12/31/2004)
150
200
-8%
-150
ALM
-100
0
100
150
200
Shock (bp)
ALCO Partners, LLC
CMOs & MBS: Duration & Convexity
Duration
Convexity
CMOs & MBS: Duration
8
CMOs & MBS: Convexity
1
Convexity (ALM)
Duration (ALM)
0
6
4
2
-1
-2
-3
-4
-5
0
0
2
4
6
8
-6
-6
-5
-4
-3
-2
-1
0
Convexity (FSI 2Factor)
Duration (FSI 2Factor)
Red line represents parity
Circled outliers were later identified as input errors in the
ALM model
Page 23
ALCO Partners, LLC
1
CMOs & MBS: FSI (2 Factor) vs. ALM: 12/31 to 3/31
CMOs& MBS: 3/31 vs. 12/31
1%
%D MV (ALM)
0%
-1%
-2%
-3%
-4%
-5%
-5%
-4%
-3%
-2%
-1%
0%
1%
%D MV (FSI 2Factor)
Red line represents parity
Four outliers are the same CUSIPs as in the prior graph
Page 24
ALCO Partners, LLC
Yield Curve Distributions: Single Factor vs. Two Factor
Stochastic Processes
30%
25%
Max
Max
MC Long Rate
MC Short Rate
20%
20%
5Yr LIBOR Rate (%)
1M LIBOR Rate (%)
Percentile
Percentile
25%
97.5
15%
90th
10%
15%
97.5
10%
90th
75th
Mean
75th
Median
Mean
5%
Median
5%
25th
10th
2.5
25th
10th
2.5
Min
0%
0
24
48
72
96
120
144
168
192
216
Min
0%
0
240
24
48
72
96
120
144
168
192
216
240
Month
Month
Single Factor 3-31
Single Factor 3-31
3%
MC Yield Curve
2%
Percentil
e
1%
97.5
Max
90th
75th
Median
Single Factor
Process:
Yield Curve
Distribution
Yield Curve (%)
0%
Mean
25th
10th
-1%
-2%
2.5
-3%
-4%
-5%
-6%
0
24
48
72
96
120
Month
Page 25
Single Factor 3-31
144
168
192
216
240
Min
ALCO Partners, LLC
Yield Curve Distributions: Single Factor vs. Two Factor
Stochastic Processes
25%
30%
MC Long Rate
MC Short Rate
Percentil
eMax
20%
15%
Percentil
eMax
20%
5Yr LIBOR Rate (%)
1M LIBOR Rate (%)
25%
97.5
90th
10%
15%
97.5
90th
10%
75th
75th
Mean
Median
Mean
Median
5%
5%
25th
10th
2.5
25th
10th
2.5
Min
0%
0
24
Two Factor 3-31
48
72
96
120
144
168
192
216
Min
0%
240
0
24
48
72
96
Month
120
144
168
192
216
240
Month
Two Factor 3-31-2005
3%
MC Yield Curve
2%
Percentil
e
Max
1%
97.5
90th
75th
Median
Two Factor
Process:
Yield Curve
Distribution
Yield Curve (%)
0%
Mean
25th
-1%
10th
-2%
2.5
-3%
-4%
Min
-5%
-6%
0
Page 26
24
48
72
96
120
Month
Two Factor 3-31
144
168
192
216
240
ALCO Partners, LLC
Yield Curve Distributions: Single Factor vs. Two Factor
Stochastic Processes
Single Factor
Process:
Two Factor
Process:
Yield Curve
Distribution
Yield Curve
Distribution
3%
3%
MC Yield Curve
MC Yield Curve
2%
2%
Percentil
e
97.5
Max
90th
75th
Median
10th
-1%
-2%
2.5
Mean
25th
-1%
10th
-2%
2.5
-3%
-3%
-4%
-4%
-5%
-5%
Min
-6%
-6%
0
24
48
72
96
120
Month
Single Factor 3-31
Page 27
97.5
90th
75th
Median
0%
Yield Curve (%)
Yield Curve (%)
Mean
25th
Max
1%
1%
0%
Percentil
e
144
168
192
216
0
240
24
48
72
96
120
144
168
192
216
240
Month
Min
Two Factor 3-31
ALCO Partners, LLC
FRMs: MV Sensitivity: Single vs. 2 Factor
12/31
3/31
FRMs: MV Sensitivity
FRMs: MV Sensitivity
6%
4%
4%
2%
2%
0%
-2%
-4%
%D MV
%D MV
0%
FSI1F
FSI2F
ALM
-2%
-4%
-6%
FSI1F
FSI2F
ALM
-6%
-8%
-8%
-10%
-150
-10%
-100
0
100
Shock (bp)
150
200
-12%
-150
-100
0
100
150
Shock (bp)
Impact of process is limited in this
case even though distribution of yield
curves is noticeable
Page 28
ALCO Partners, LLC
200
Observations from Test
 Stochastic models will frequently produce different
measures of risk at the instrument level for instruments
with embedded options
Page 29
•
We believe – but it is difficult to confirm without access to
source code – that this is largely due to differences in
calibration procedures as well as differences in modeling
methods.
•
With larger portfolios that include instruments with a wide
variety of option structures, the differences in results will
tend to cancel each other out if the overall levels of implied
volatility used in the calibrations are similar.
ALCO Partners, LLC
Observations from Test
 Validation tests of stochastic models using a qualified
stochastic model, such as FSI, can:
• Identify modeling errors, because large differences in
results invite further investigation
• Enhance understanding of the “black box” of ALM model
calibration procedures, when calibrations aren’t “all that
close”
•
Provide guidance for the product characteristics that are
stochastic modeling technique sensitive
 Validation tests of stochastic models lead to the
following warning:
Beware of relying on a single stochastic model when
measuring risk at the instrument level for instruments
with embedded complex options
Page 30
ALCO Partners, LLC
Observations from Test
Tests of stochastic models can identify modeling errors,
because large differences invite further investigation
Word to the warning about using stochastic models for risk
management involving individual transactions
Page 31
ALCO Partners, LLC