Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Value-at-Risk Models
Christophe Hurlin
University of Orléans
Séminaire Validation des Modèles Financiers. 29 Avril 2013
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Introduction
The Value-at-Risk (VaR) and more generally the Distortion Risk
Measures (Expected Shortfall, etc.) are standard risk measures
used in the current regulations introduced in Finance (Basel 2), or
Insurance (Solvency 2) to …x the required capital (Pillar 1), or to
monitor the risk by means of internal risk models (Pillar 2).
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Introduction
De…nition
Let frt gTt=1 be a given P&L series. The daily (conditional) VaR for
a nominal coverage rate α is de…ned as
Pr[ rt <
where Ft
1
VaR t jt
1 (α)
Ft
1]
=α
denotes the set of information available at time t
Christophe Hurlin
Backtesting
1.
Introduction
Backtesting Principles
Testing strategies
Recommandations
Introduction
Who does use VaR?
What for?
Bank risk manager
Measure …rm-level market, credit, op. risk
Bank executives
Set limits (management)
Banking regulators
Determine capital requirements
Exchanges
Compute margins
Regulators
Forecast systemic risk (CoVaR)
Industry
Ex: EDF, spot prices of electricity
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
"Disclosure of quantitative measures of market risk, such
as value-at-risk, is enligthening only when accompanied
by a thorough discussion of how the risk measures were
calculated and how they related to actual performance",
Alan Greenspan (1996)
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Introduction
De…nition
Backtesting is a set of statistical procedures designed to check if
the real losses are in line with VaR forecasts (Jorion, 2007).
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Introduction
Whatever the type of use of VaR, the VaR forecasts are
generated by an internal risk model.
This model is used to produced a sequence of pseudo out-of
sample VaR forecasts for a past period (typically one year)
The backtesting is based on the comparison of the observed
P&L to these VaR forecasts.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Outlines
1
How to test the validity of a VaR model?
2
What are the backtesting strategies?
3
What are the good practices?
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Principles
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Principles
Remark 1: Ex-post VaR is not observable, so it is impossible to
compute traditional statistics or criteria such as MSFE.
Remark 2: There is no proxy for the VaR contrary to the volatility
(realized volatility, Andersen and Bollerslev 1998)
Patton, A.J. (2011) Volatility forecast comparison using imperfect volatility
proxies, Journal of Econometrics, 260, 246-256.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Principles
Backtesting procedures are based on VaR exceptions
De…nition
We denote It (α) the hit variable associated to the ex-post
observation of an α% VaR exception at time t :
(
1 if rt < VaR t jt 1 (α)
It (α) =
0 else
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Principles
Christo¤ersen (1998) : VaR forecasts are valid if and only if the
violation process It (α) satis…es the following two assumptions:
1
The unconditional coverage (UC) hypothesis.
2
The independence (IND) hypothesis.
Christo¤ersen P.F. (1998), Evaluating interval forecasts, International Economic
Review, 39, pp. 841-862.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Principles
De…nition (unconditional coverage hypothesis)
The unconditionnal probability of a violation must be equal to the
α coverage rate
Pr [It (α) = 1] = E [It (α)] = α.
If Pr [It (α) = 1] > α, the risk is under-estimated
If Pr [It (α) = 1] < α, the risk is over-estimated
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Principles
De…nition (independence hypothesis)
VaR violations observed at two di¤erent dates must be
independently distributed.
It (α) and Is (α) are independently distributed for t 6= s
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Principles
Figure: Illustration: violations’cluster
8
VaR(95%)
P&L
6
4
2
0
-2
-4
-6
0
50
100
Christophe Hurlin
150
Backtesting
200
250
Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Principles
Figure: Illustration: violations’cluster
8
VaR(95%)
P&L
6
4
2
0
-2
-4
-6
0
50
100
Christophe Hurlin
150
Backtesting
200
250
Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Principles
De…nition (conditional coverage hypothesis)
The violation process satis…es a di¤erence martingale assumption.
E [ It (α) j Ft
Christophe Hurlin
1]
=α
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Principles
Remark: These assumptions can be expressed as distributional
assumptions.
Under the UC assumption, each variable It (α) has a Bernouilli
distribution with a probability α.
Itt (α)
Bernouilli (α)
Under the IND assumption, these variables are independent, and
the number of violations has a Binomial distribution.
T
∑ It (α)
B (T , α )
t =1
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies
What are the backtesting strategies?
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies
Let us consider a sequence of daily VaR out-of-sample forecasts
T
VaR t jt 1 (α) t =1 and the corresponding observed P&L.
How to test the validity of the internal risk model?
Hurlin C. and Pérignon C. (2012), Margin Backtesting,
Review of Futures Market, 20, pp. 179-194
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies
Testing strategies:
1
Frequency-based tests
2
Magnitude-based tests
3
Multivariate tests
4
Independence tests
5
Duration-based tests
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: frequency-based tests (1/5)
Figure: BIS "Tra¢ c Light" System
Note: VaR(1%, 1 day), 250 daily observations
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: frequency-based tests (1/5)
De…nition
Christo¤ersen (1998) proposes a Likelihood Ratio statistic for UC
de…ned as:
i
h
LRUC =
2 ln (1 α)T H αH
h
i
d
! χ2 (1)
+2 ln (1 H/T )T H (H/T )H
T !∞
where H = ∑Tt=1 It (α) denotes the total number of exceedances.
For a nominal risk of 5%, the null of UC can not be rejected if
and only if H < 7 for T = 250 and α = 1%.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: frequency-based tests (1/5)
Example
Berkowitz and O-Brien (2002) consider the VaR forecasts of six US
commercial banks
Berkowitz, J., and O-Brien J. (2002), How Accurate are the
Value-at-Risk Models at Commercial Banks, Journal of
Finance.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: frequency-based tests (1/5)
Figure: Bank Daily VaR Models
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: frequency-based tests (1/5)
Figure: Violations of Banks’99% VaR
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: frequency-based tests (1/5)
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: frequency-based tests (1/5)
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies
Testing strategies:
1
Frequency-based tests
2
Magnitude-based tests
3
Multivariate tests
4
Independence tests
5
Duration-based tests
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
All these tests do not take into account the magnitude of the
losses beyond the VaR
Example
Consider two banks that both have a one-day 1%-VaR of $100
million. Assume each bank reports three VaR exceptions, but the
average VaR exceedance is $1 million for bank A and $500 million
for bank B.
In this case, standard backtesting methodologies would indicate
that the performance of both models is equal and acceptable.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
Figure: Daily VaR and P/L for SocGen 2008
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
Figure: Daily VaR and P/L for SocGen 2008
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
The Risk Map
Colletaz G., Hurlin C. and Perignon C. (2013), The Risk
Map: a new tool for Risk Management, forthcoming in
Journal of Banking and Finance
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
We propose a VaR backtesting methodology based on the number
and the severity of VaR exceptions: this approach exploits the
concept of "super exception".
De…nition
We de…ne a super exception using a VaR with a much smaller
coverage probability α0 , with α0 < α. In this case, a super
exception is de…ned as a loss greater than VaRt (α0 ).
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
Figure: VaR Exception vs. VaR Super Exception
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
Solution
Given VaR exceptions It (α) and VaR super exception It (α0 ), we
de…ne a Risk Map that jointly accounts for the number and the
magnitude of the VaR exceptions
Let us consider a given UC test with a statistic Z (α) based on the
violations sequence fIt (α)gTt=1 .
H0 : E [It (α)] = α
H1 : E [It (α)] 6= α.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Number of VaR Exceptions (N)
Testing strategies: magnitude-based tests (2/5)
Non-rejection area for test
on VaR exceptions
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
Based on the same UC test, it is possible to test for the
magnitude of VaR exceptions, via the VaR super exceptions
fIt (α0 )gTt=1
H0 : E It α0 = α0
H1 : E It α0
Christophe Hurlin
6= α0
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
Non-rejection
area for test
on VaR super
exceptions
Number of VaR Super Exceptions (N’)
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
We can also test jointly for both magnitude and frequency of VaR
exceptions:
H0 : E [It (α)] = α and E It α0
= α0
Multivariate approach
Perignon C. and Smith, D. (2008), A New Approach to
Comparing VaR Estimation Methods, Journal of
Derivatives
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
15
14
13
Number of VaR Exceptions (N)
12
11
10
9
8
7
6
5
4
3
Nominal risk 5%
2
Nominal risk 1%
1
0
0
1
2
3
4
5
6
Number of VaR Super Exceptions (N')
Christophe Hurlin
Backtesting
7
8
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
Figure: Backtesting Bank VaR: La Caixa (2007-2008)
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: magnitude-based tests (2/5)
15
14
13
Number of VaR Exceptions (N)
12
11
10
9
8
7
6
5
4
3
Nominal risk 5%
2
Nominal risk 1%
1
0
0
1
2
3
4
5
6
Number of VaR Super Exceptions (N')
Christophe Hurlin
Backtesting
7
8
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies
Testing strategies:
1
Frequency-based tests
2
Magnitude-based tests
3
Multivariate tests
4
Independence tests
5
Duration-based tests
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: multivariate tests (3/5)
Intuition: Testing the validity of the VaR model for M coverage
rates, with M > 1.
Perignon C. and Smith, D. (2008), A New Approach to
Comparing VaR Estimation Methods, Journal of
Derivatives
Hurlin C. and Tokpavi, S. (2006), ”Backtesting
Value-at-Risk Accuracy: A Simple New Test”, Journal of
Risk
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: multivariate tests (3/5)
Perignon and Smith (2008) consider the null:
H0,MUC : E [It (α)] = α and E It α0
= α0 .
Let us denote:
J0,t
= 1
J1,t
=
J2,t
=
J1,t J2,t
1 if
VaR t jt
0 otherwise
1 (α
1 if rt < VaR t jt
0 otherwise
Christophe Hurlin
0)
< rt <
1 (α
0)
Backtesting
.
VaR t jt
1 (α)
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: multivariate tests (3/5)
De…nition (Perignon and Smith, 2008)
The multivariate unconditional coverage test
h
H
LRMUC =
2 ln (1 α)H 0 α α0 1
"
H 0 H 0 H0
+2 ln 1
T
T
is a LR test given by:
i
H
α0 2
#
H1 H 1 H 2 H 2
.
T
T
where Hi = ∑Tt=1 Ji ,t , for i = 0, 1, 2, denote the count variable
associated with each of the Bernoulli variables.
Christophe Hurlin
Backtesting
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Introduction
Backtesting Principles
Testing strategies
Recommandations
Testing strategies: multivariate tests (3/5)
Hurlin and Tokpavi (2006):
A natural test of the CC is the univariate Ljung-Box test of
H0,CC : r1 = ... = rK = 0
where rk denotes the k th autocorrelation:
K
LB (K ) = T (T + 2)
Christophe Hurlin
∑
T
k =1
b
rk2
d
k
Backtesting
! χ2 (K )
T !∞
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Introduction
Backtesting Principles
Testing strategies
Recommandations
Testing strategies: multivariate tests (3/5)
De…nition (Hurlin and Tokpavi, 2006)
Let Θ = fθ 1 , .., θ m g be a discrete set of m di¤erent coverage rates
0
and Hitt = [Hitt (θ 1 ) : Hitt (θ 2 ) : ... : Hitt (θ m )]
(
1 θ i if rt < VaR t jt 1 (θ i )
Hitt (θ i ) =
θi
else
Under the null of CC (martingale di¤erence):
0
H0,CC : E [Hitt Hitt
Christophe Hurlin
k
= 0m
Backtesting
8k = 1, ..., K
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies
Testing strategies:
1
Frequency-based tests
2
Magnitude-based tests
3
Multivariate tests
4
Independence tests
5
Duration-based tests
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: independence tests (4/5)
LR tests
Christo¤ersen (1998) assumes that the violation process It (α) can
be represented as a Markov chaine with two states:
Π=
1
1
π 01 π 01
π 11 π 11
π ij = Pr [ It (α) = j j It
1
(α) = i ]
De…nition
The null of CC can be de…ned as follows:
H0,CC : Π = Πα =
Christophe Hurlin
1
1
Backtesting
α α
α α
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: independence tests (4/5)
LR tests
Christo¤ersen (1998) assumes that the violation process It (α) can
be represented as a Markov chaine with two states:
Π=
1
1
π 01 π 01
π 11 π 11
π ij = Pr [ It (α) = j j It
1
(α) = i ]
De…nition
The null of IND can be de…ned as follows:
H0,IND : Π = Π β =
Christophe Hurlin
1
1
Backtesting
β β
β β
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: frequency-based tests (4/5)
The corresponding LR statistics are de…ned by:
h
i
LRIND =
2 ln (1 H/T )T H (H/T )H
+2 ln [(1
LRCC
=
h
2 ln (1
+2 ln [(1
By de…nition:
b 01 )n00 π
b n0101 (1
π
α)
T
H
(α)
H
i
b 01 )n00 π
b n0101 (1
π
T !∞
b 11 )n10 π
b n1111 ]
π
T !∞
LRCC = LRUC + LRIND
Christophe Hurlin
d
b 11 )n10 π
b n1111 ]
π
Backtesting
! χ2 (1)
d
! χ2 (2)
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: independence tests (4/5)
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: independence tests (4/5)
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: frequency-based tests (1/5)
Figure: Violations of Banks’99% VaR
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: independence tests (4/5)
Regression based tests
Engle and Manganelli (2004) suggest another approach based
on a linear regression model. This model links current margin
exceedances to past exceedances and/or past information.
Let Hit (α) = It (α)
with It (α):
Hitt (α) =
α be the demeaned process associated
1
α if rt < VaR t jt
α otherwise
Christophe Hurlin
Backtesting
1 (α)
.
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: independence tests (4/5)
Regression based tests
Consider the following linear regression model:
K
Hitt (α) = δ +
∑
K
βk Hitt
k (α) +
k =1
∑ γk zt
k
+ εt
k =1
where the zt k variables belong to the information set Ωt
(lagged P&L, squared past P&L, past margins, etc.)
Christophe Hurlin
Backtesting
1
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: independence tests (4/5)
Regression based tests
The null hypothesis test of CC corresponds to testing the joint
nullity of all the regression coe¢ cients:
H0,CC : δ = βk = γk = 0,
8k = 1, ..., K .
since under the null :
E [Hitt (α)] = E [It (α)
α] = 0 () Pr [It (α) = 1] = α
Christophe Hurlin
Backtesting
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Introduction
Backtesting Principles
Testing strategies
Recommandations
Testing strategies: independence tests (4/5)
De…nition (Engle and Manganelli, 2004)
Denote Ψ = (δ β1 ...βK γ1 ...γK )0 the vector of the 2K + 1
parameters in this model and Z the matrix of explanatory variables
of model, the Wald statistic, denoted DQCC , then veri…es:
DQCC =
b 0Z 0Z Ψ
b
Ψ
α (1 α )
d
! χ2 (2K + 1)
T !∞
b is the OLS estimate of Ψ.
where Ψ
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: independence tests (4/5)
Regression based tests
Extension: A natural extension of the test of Engle and Manganelli
(2004) consists in considering a (probit or logit) binary model
linking current violations to past ones
Dumitrescu E., Hurlin C. and Pham V. (2012),
Backtesting Value-at-Risk: From Dynamic Quantile to
Dynamic Binary Tests, Finance
Christophe Hurlin
Backtesting
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Introduction
Backtesting Principles
Testing strategies
Recommandations
Testing strategies: independence tests (4/5)
De…nition (Dumitrescu et al., 2012)
We consider a dichotomic model:
Pr [ It (α) = 1 j Ft
1]
= F (π t ) .
where F (.) denotes a c.d.f. and the index π t satis…es the
following autoregressive representation:
q1
πt = c +
∑
j =1
q2
βj π t
j
+ ∑ δj It
j =1
q3
j
(α) + ∑ γj xt j ,
j =1
where l (.) is a function of a …nite number of lagged values of
observables, and xt is a vector of explicative variables.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: independence tests (4/5)
Regression based tests
H0 : β = 0, δ = 0, γ = 0 and c = F
1
(α) .
since under the null of CC:
Pr(It = 1 j Ft
1)
= F (F
1
(α)) = α.
The Dynamic Binary (DB) LR test statistic is:
DBLR CC =
2 ln L(0, F
1
(α); It (α), Zt )
d
! χ2 (dim(Zt ))
T !∞
Christophe Hurlin
Backtesting
ln L(θ̂, ĉ; It (α), Zt )
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies
Testing strategies:
1
Frequency-based tests
2
Magnitude-based tests
3
Multivariate tests
4
Independence tests
5
Duration-based tests
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: duration-based tests (5/5)
The UC, IND, and CC hypotheses also have some implications
on the time between two consecutive VaR margin exceedances.
Let us denote by dv the duration between two consecutive
VaR margin violations:
dv = tv
tv
1
where tv denotes the date of the v th exceedance.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: duration-based tests (5/5)
Under CC hypothesis, the duration process dv has a geometric
distribution:
Pr [dv = k ] = α (1
α )k
1
k2N .
This distribution characterizes the memory-free property of
the violation process It (α) with E (dv ) = 1/α
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: duration-based tests (5/5)
De…nition
Christo¤ersen and Pelletier (2004) use under the null hypothesis
the exponential distribution:
g (dv ; α) = α exp ( αdv ) .
Under the alternative hypothesis, they postulate a Weibull
distribution for the duration variable:
h
i
h (dv ; a, b ) = ab bdvb 1 exp
(adv )b .
H0,IND : b = 1
Christophe Hurlin
H0,CC : b = 1, a = α
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: duration-based tests (5/5)
Drawback: we have to postulate a distribution for the duration
under the alternative (misspeci…cation of the VaR model).
Solution: Candelon et al. (2001) propose a J-test based on
orthonormal polynomials associated to the geometric distribution.
Candelon B., Colletaz G., Hurlin C. et Tokpavi S. (2011),
"Backtesting Value-at-Risk: a GMM duration-based
test", Journal of Financial Econometrics,
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
In the case of continuous distributions, the Pearson distributions
(Normal, Student, etc.) are associated to some particular
orthonormal polynomials whose expectation is equal to zero.
These polynomials can be used as special moments to test for
a distributional assumption (see. Bontemps and Meddahi,
Journal of Econometrics, 2005).
In the discrete case, orthonormal polynomials are de…ned for
distributions belonging to the Ord’s family (Poisson, Pascal,
hypergeometric, etc.).
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
De…nition
The orthonormal polynomials associated to a geometric
distribution with a success probability β are de…ned by the
following recursive relationship, 8d 2 N :
Mj +1 (d; β) =
(1
β) (2j + 1) + β (j d + 1)
p
Mj (d; β)
(j + 1) 1 β
j
Mj 1 (d; β) ,
j +1
for any order j 2 N , with M
1
E [Mj (d; β)] = 0
Christophe Hurlin
(d; β) = 0 and M0 (d; β) = 1 and:
8j 2 N , 8d 2 N .
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
Example
We can show that if d follows a geometric distribution of
parameter β, then:
p
M1 (d; β) = (1 βd ) / 1 β
with
E [M1 (d; β)] = 0
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
Our duration-based backtest procedure exploits these moment
conditions.
More precisely, let us de…ne fd1 ; ...; dN g a sequence of N
durations between VaR violations, computed from the
sequence of the hit variables fIt (α)gTt=1 .
Under the CC assumption, the durations di , i = 1, .., N, are
i.i.d. geometric(α). Hence, the null of CC can be expressed
as follows:
H0,CC : E [Mj (di ; α)] = 0,
j = f1, .., p g ,
where p denotes the number of moment conditions.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
De…nition
The null hypothesis of CC can be expressed as
H0,CC : E [M (di ; α)] = 0,
where M (di ; α) denotes a (p, 1) vector whose components are the
orthonormal polynomials Mj (di ; α) , for j = 1, .., p. Under some
regularity conditions:
!|
!
1 N
1 N
d
p ∑ M (di ; α)
! χ2 (p )
JCC (p ) = p ∑ M (di ; α)
N !∞
N i =1
N i =1
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Frequency-based tests
Magnitude-based tests
Multivariate tests
Independence tests
Duration-based tests
Testing strategies: duration-based tests (5/5)
Candelon et al. (2001)
De…nition
Under UC, the mean of durations between two violations is equal
to 1/α, and the null hypothesis is
H0,UC : E [M1 (di ; α)] = 0.
with a test statistic equal to
JUC =
1
p
N
N
∑ M1 (di ; α)
i =1
Christophe Hurlin
!2
Backtesting
d
! χ2 (1)
N !∞
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Recommandations
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Recommandation 1: Test, test and test
Recommandation 2: Check the P&L data
Recommandation 3: The power of your tests may be low..
Recommandation 4: Take into account the estimation risk
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Recommandation 1: Test, test and test
Each type of test (frequency, severity, independence,
conditional coverage, multivariate test etc..) captures one
type of potential misspeci…cation of the VaR model.
It is important to use a variety of tests
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Recommandation 2: Check the P&L data
Frésard, L., C. Perignon, and W., Anders (2011), The
Pernicious E¤ects of Contaminated Data in Risk Management,
Journal of Banking and Finance.
1
A large fraction of US and international banks validate their
market risk model using P&L data that include fees and
commissions and intraday trading revenues.
2
Distinction between dirty P/L and hypothetical P/L (JP.
Morgan, Romain Berry 2011).
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Recommandation 3: The power of your tests may be low..
De…nition
The power of a backtesting test corresponds to its capacity to
detect misspeci…ed VaR model.
Pr [ Rejection H0 j H1 ]
Example
Berkowitz, J., Christo¤ersen, P. F., and Pelletier, D., 2013,
Evaluating Value-at-Risk Models with Desk-Level Data.
Management Science.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Hurlin C. et Tokpavi S. (2008), ”Une Evaluation des
Procédures de Backtesting : Tout va pour le Mieux dans le
Meilleur des Mondes", Finance
Idea: we use 6 di¤erent methods (GARCH, RiskMetrics, HS,
CaviaR, Hybride, Delta Normale) to forecast a VaR(5%) on the
same asset (GM, Nasdaq), and we apply the backtests (LR, DQ,
Duration based tests) on a set of 500 samples (rolling window) of
T = 250 forecasts.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Example
LRCC tests: for 47% of the samples, we don’t reject (at 5%) the
null for any of the six VaR forecats. In 71% of the samples, we
reject at the most one VaR.
Example
DQCC tests: for 20% of the samples, we don’t reject (at 5%) the
null for any of the six VaR forecats. In 51% of the samples, we
reject at the most one VaR.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
The power of a consistent test tends to 1 when the sample
size tends to ini…nity.
Recommandation: increase at the maximum the sample size
of your backtest.. (T = 500, 750 or more.)
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Recommandation 4: Take into account the estimation risk
The risk dynamic is usually represented by a parametric or
semi-parametric model, which has to be estimated in a
preliminary step. However, the estimated counterparts of risk
measures are subject to estimation uncertainty.
Replacing, in the theoretical formulas, the true parameter
value by an estimator induces a bias in the coverage
probabilities.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Escanciano and Olmo (2010, 2011) studied the e¤ects of
estimation risk on backtesting procedures. They showed how
to correct the critical values in standard tests used to assess
VaR models.
Escanciano, J.C. and J. Olmo (2010) Backtesting Parametric
Value-at-Risk with Estimation Risk, Journal of Business and
Economics Statistics.
Escanciano, J.C. and J. Olmo (2011) Robust Backtesting Tests
for Value-at-Risk Models. Journal of Financial Econometrics.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Backtesting
Estimation Adjusted VaR
Gouriéroux and Zakoian (2013) a method to directly adjust the
VaR to estimation risk ensuring the right conditional coverage
probability at order 1/T :
Pr rt <
EVaR t jt
1
(α) = α + oP (1/T )
Gouriéroux C. and Zakoian J.M. (2013), Estimation Adjusted
VaR, forthcoming in Econometric Theory.
Christophe Hurlin
Backtesting
Introduction
Backtesting Principles
Testing strategies
Recommandations
Test, test and test
Check the P&L data
The power of your tests may be low...
Estimation risk
Thank you for your attention
Christophe Hurlin
Backtesting