Coherent functional forecasting
1
Coherent functional forecasts
of mortality rates
and life expectancy
Rob J Hyndman
Business & Economic Forecasting Unit
Joint work with
Farah Yasmeen (Monash) and Heather Booth (ANU)
Coherent functional forecasting
2
Mortality rates
−4
−6
−8
Log death rate
−2
Australia male mortality: 1950
0
20
40
60
Age
80
100
Coherent functional forecasting
3
Mortality rates
−4
−6
−8
Log death rate
−2
Australia female mortality: 1950
0
20
40
60
Age
80
100
Coherent functional forecasting
The problem
Let ft,j (x) be the smoothed mortality rate for
age x in group j in year t.
4
Coherent functional forecasting
The problem
Let ft,j (x) be the smoothed mortality rate for
age x in group j in year t.
Groups may be males and females.
4
Coherent functional forecasting
The problem
Let ft,j (x) be the smoothed mortality rate for
age x in group j in year t.
Groups may be males and females.
Groups may be states within a country.
4
Coherent functional forecasting
The problem
Let ft,j (x) be the smoothed mortality rate for
age x in group j in year t.
Groups may be males and females.
Groups may be states within a country.
Expected that groups will behave similarly.
4
Coherent functional forecasting
The problem
Let ft,j (x) be the smoothed mortality rate for
age x in group j in year t.
Groups may be males and females.
Groups may be states within a country.
Expected that groups will behave similarly.
We want to forecast whole curve ft,j (x) for
future years.
4
Coherent functional forecasting
The problem
Let ft,j (x) be the smoothed mortality rate for
age x in group j in year t.
Groups may be males and females.
Groups may be states within a country.
Expected that groups will behave similarly.
We want to forecast whole curve ft,j (x) for
future years.
Coherent forecasts do not diverge over time.
4
Coherent functional forecasting
The problem
Let ft,j (x) be the smoothed mortality rate for
age x in group j in year t.
Groups may be males and females.
Groups may be states within a country.
Expected that groups will behave similarly.
We want to forecast whole curve ft,j (x) for
future years.
Coherent forecasts do not diverge over time.
Existing functional models do not impose
coherence.
4
Coherent functional forecasting
5
Functional time series model
(Hyndman and Ullah, CSDA, 2007)
log[ft,j (x)] = µj (x) +
K
X
k =1
iid
where et,j (x) ∼ N(0, v(x)).
βt,j ,k φk ,j (x) + et,j (x)
Coherent functional forecasting
5
Functional time series model
(Hyndman and Ullah, CSDA, 2007)
log[ft,j (x)] = µj (x) +
K
X
βt,j ,k φk ,j (x) + et,j (x)
k =1
iid
where et,j (x) ∼ N(0, v(x)).
1
Estimate smooth functions ft,j (x) using
nonparametric regression.
Coherent functional forecasting
5
Functional time series model
(Hyndman and Ullah, CSDA, 2007)
log[ft,j (x)] = µj (x) +
K
X
βt,j ,k φk ,j (x) + et,j (x)
k =1
iid
where et,j (x) ∼ N(0, v(x)).
1
2
Estimate smooth functions ft,j (x) using
nonparametric regression.
Estimate µj (x) as mean log[ft,j (x)] across years.
Coherent functional forecasting
5
Functional time series model
(Hyndman and Ullah, CSDA, 2007)
log[ft,j (x)] = µj (x) +
K
X
βt,j ,k φk ,j (x) + et,j (x)
k =1
iid
where et,j (x) ∼ N(0, v(x)).
1
2
3
Estimate smooth functions ft,j (x) using
nonparametric regression.
Estimate µj (x) as mean log[ft,j (x)] across years.
Estimate βt,j ,k and φk ,j (x) using functional
principal components.
Coherent functional forecasting
5
Functional time series model
(Hyndman and Ullah, CSDA, 2007)
log[ft,j (x)] = µj (x) +
K
X
βt,j ,k φk ,j (x) + et,j (x)
k =1
iid
where et,j (x) ∼ N(0, v(x)).
1
2
3
4
Estimate smooth functions ft,j (x) using
nonparametric regression.
Estimate µj (x) as mean log[ft,j (x)] across years.
Estimate βt,j ,k and φk ,j (x) using functional
principal components.
Forecast βt,j ,k using time series models.
Coherent functional forecasting
5
Functional time series model
(Hyndman and Ullah, CSDA, 2007)
log[ft,j (x)] = µj (x) +
K
X
βt,j ,k φk ,j (x) + et,j (x)
k =1
iid
where et,j (x) ∼ N(0, v(x)).
1
2
3
4
5
Estimate smooth functions ft,j (x) using
nonparametric regression.
Estimate µj (x) as mean log[ft,j (x)] across years.
Estimate βt,j ,k and φk ,j (x) using functional
principal components.
Forecast βt,j ,k using time series models.
Put it all together to get forecasts of ft,j (x).
Coherent functional forecasting
6
0.4
0.2
φ3(x)
−0.2
0
20 40 60 80
0
Age
20 40 60 80
0
Age
20 40 60 80
Age
1970
1990
Year
1950
0.2
0.0
βt3
−0.2
−0.4
βt2
1950
−0.6 −0.4 −0.2
−2
−1
βt1
0
0.0
1
0.2
2
Age
−0.4
−0.6
0.1
20 40 60 80
−3
0
0.0
φ2(x)
−0.2 0.0
φ1(x)
0.2
−4
−6
−8
µM(x)
0.3
0.2
−2
0.4
0.4
Male fts model
1970
1990
Year
1950
1970
1990
Year
Coherent functional forecasting
7
φ3(x)
−0.4
0
20 40 60 80
0
Age
βt2
−0.1 0.0
0.1
2
1
0
1950
1970
1990
Year
1950
1970
1990
Year
−0.4
−0.3
−2
−1
20 40 60 80
0.0 0.1 0.2
Age
0.2
Age
βt1
−0.2
0.0
20 40 60 80
βt3
0
Age
−0.2
20 40 60 80
−3
0
−0.4
0.1
−0.2
0.0
φ2(x)
φ1(x)
0.2
−4
−6
−8
µF(x)
0.3
0.2
−2
0.2
0.4
0.4
Female fts model
1950
1970
1990
Year
Coherent functional forecasting
8
Forecasting the coefficients
log[ft,j (x)] = µj (x) +
K
X
βt,j ,k φk ,j (x) + et,j (x)
k =1
We use ARIMA models for each coefficient
{β1,j ,k , . . . , βn,j ,k }.
Coherent functional forecasting
8
Forecasting the coefficients
log[ft,j (x)] = µj (x) +
K
X
βt,j ,k φk ,j (x) + et,j (x)
k =1
We use ARIMA models for each coefficient
{β1,j ,k , . . . , βn,j ,k }.
The ARIMA models are non-stationary for
the first few coefficients (k = 1, 2)
Coherent functional forecasting
8
Forecasting the coefficients
log[ft,j (x)] = µj (x) +
K
X
βt,j ,k φk ,j (x) + et,j (x)
k =1
We use ARIMA models for each coefficient
{β1,j ,k , . . . , βn,j ,k }.
The ARIMA models are non-stationary for
the first few coefficients (k = 1, 2)
Non-stationary ARIMA forecasts will
diverge. Hence the mortality forecasts are
not coherent.
Coherent functional forecasting
9
0.4
0.2
φ3(x)
−0.2
20 40 60 80
0
20 40 60 80
0.2
1
βt3
−0.2
βt2
−0.4
−1
−2
2000
Year
20 40 60 80
Age
0
−5
βt1
−10
1960
0
Age
0
Age
2040
0.0
0
Age
−0.4
−0.6
0.1
20 40 60 80
−15
0
0.0
φ2(x)
−0.2 0.0
φ1(x)
0.2
−4
−6
−8
µM(x)
0.3
0.2
−2
0.4
0.4
Male fts model
1960
2000
Year
2040
1960
2000
Year
2040
Coherent functional forecasting
10
20 40 60 80
φ3(x)
−0.4
0
20 40 60 80
Age
0.0
βt3
−0.2
βt2
−0.2
−0.4
−0.6
2000
Year
20 40 60 80
0.0 0.1 0.2
2
0
−2
−4
−6
βt1
0
Age
0.2
Age
1960
−0.2
0.0
0
Age
2040
1960
2000
Year
2040
−0.4
20 40 60 80
−8
0
−0.4
0.1
−0.2
0.0
φ2(x)
φ1(x)
0.2
−4
−6
−8
µF(x)
0.3
0.2
−2
0.2
0.4
0.4
Female fts model
1960
2000
Year
2040
Coherent functional forecasting
11
Australian mortality forecasts
−6
−4
0
20
40
60
Age
80
100
−12
−10
−8
Log death rate
−6
−8
−10
−12
Log death rate
−4
−2
(b) Females
−2
(a) Males
0
20
40
60
Age
80
100
Coherent functional forecasting
Mortality product and ratios
Key idea
Model the geometric mean and the mortality
ratio instead of the individual rates for each sex
separately.
12
Coherent functional forecasting
Mortality product and ratios
Key idea
Model the geometric mean and the mortality
ratio instead of the individual rates for each sex
separately.
q
q
.
pt (x) = ft,M (x)ft,F (x) and rt (x) = ft,M (x) ft,F (x).
12
Coherent functional forecasting
Mortality product and ratios
Key idea
Model the geometric mean and the mortality
ratio instead of the individual rates for each sex
separately.
q
q
.
pt (x) = ft,M (x)ft,F (x) and rt (x) = ft,M (x) ft,F (x).
Product and ratio are approximately
independent
12
Coherent functional forecasting
Mortality product and ratios
Key idea
Model the geometric mean and the mortality
ratio instead of the individual rates for each sex
separately.
q
q
.
pt (x) = ft,M (x)ft,F (x) and rt (x) = ft,M (x) ft,F (x).
Product and ratio are approximately
independent
Ratio should be stationary (for coherence)
but product can be non-stationary.
12
Coherent functional forecasting
13
Mortality rates
−4
−6
−8
Log death rate
−2
Australia gmean mortality: 1950
0
20
40
60
Age
80
100
Coherent functional forecasting
14
Mortality rates
3.0
2.5
2.0
1.5
1.0
sex ratio: M/F
3.5
4.0
Australia mortality sex ratio 1950
0
20
40
60
Age
80
100
Coherent functional forecasting
15
Model product and ratios
q
q
.
pt (x) = ft,M (x)ft,F (x) and rt (x) = ft,M (x) ft,F (x).
log[pt (x)] = µp (x) +
log[rt (x)] = µr (x) +
K
X
k =1
L
X
`=1
βt,k φk (x) + et (x)
γt,` ψ` (x) + wt (x).
Coherent functional forecasting
15
Model product and ratios
q
q
.
pt (x) = ft,M (x)ft,F (x) and rt (x) = ft,M (x) ft,F (x).
log[pt (x)] = µp (x) +
log[rt (x)] = µr (x) +
K
X
k =1
L
X
βt,k φk (x) + et (x)
γt,` ψ` (x) + wt (x).
`=1
{γt,` } restricted to be stationary processes:
either ARMA(p, q) or ARFIMA(p, d , q).
Coherent functional forecasting
15
Model product and ratios
q
q
.
pt (x) = ft,M (x)ft,F (x) and rt (x) = ft,M (x) ft,F (x).
log[pt (x)] = µp (x) +
log[rt (x)] = µr (x) +
K
X
k =1
L
X
βt,k φk (x) + et (x)
γt,` ψ` (x) + wt (x).
`=1
{γt,` } restricted to be stationary processes:
either ARMA(p, q) or ARFIMA(p, d , q).
No restrictions for βt,1 , . . . , βt,K .
Coherent functional forecasting
15
Model product and ratios
q
q
.
pt (x) = ft,M (x)ft,F (x) and rt (x) = ft,M (x) ft,F (x).
log[pt (x)] = µp (x) +
log[rt (x)] = µr (x) +
K
X
k =1
L
X
βt,k φk (x) + et (x)
γt,` ψ` (x) + wt (x).
`=1
{γt,` } restricted to be stationary processes:
either ARMA(p, q) or ARFIMA(p, d , q).
No restrictions for βt,1 , . . . , βt,K .
Forecasts: fn+h |n,M (x) = pn+h |n (x)r.n+h |n (x)
fn+h |n,F (x) = pn+h |n (x) rn+h |n (x).
Coherent functional forecasting
16
0
20 40 60 80
0.2
20 40 60 80
0
20 40 60 80
Age
Age
1
0.2
2
βt3
0
βt2
−2
−0.2
−1
0.0
0.1
0
−2
−4
−6
βt1
0.0
φ3(x)
−0.4
0
Age
0.3
Age
−0.2
0.0
φ2(x)
0.1
20 40 60 80
−8
0
−0.4 −0.2
φ1(x)
0.2
−4
−6
−8
µP(x)
0.3
0.2
−2
0.4
0.4
Product model
1960
2000
Year
2040
1960
2000
Year
2040
1960
2000
Year
2040
Coherent functional forecasting
17
0.6
φ3(x)
0.0
0.0
φ2(x)
0.2
0.2
φ1(x)
0.4
0.4
0.4
0.5
0.4
0.3
−0.2
−0.4
0.0
0.2
20 40 60 80
0
20 40 60 80
0
20 40 60 80
Age
0
20 40 60 80
Age
Age
0.10
βt3
0.0
βt2
1960
2000
Year
2040
−0.10
−0.2
−0.2
−0.1
βt1
0.0
0.1
0.2
0.2
Age
−0.4
0
0.00
0.1
µR(x)
0.8
Ratio model
1960
2000
Year
2040
1960
2000
Year
2040
Coherent functional forecasting
18
−4
−6
−8
−10
Log of geometric mean death rate
−2
Product forecasts
0
20
40
60
Age
80
100
Coherent functional forecasting
19
3.0
2.5
2.0
1.5
1.0
Sex ratio: M/F
3.5
4.0
Ratio forecasts
0
20
40
60
Age
80
100
Coherent functional forecasting
20
Coherent forecasts
−4
−10
−8
−6
Log death rate
−6
−8
−10
Log death rate
−4
−2
(b) Females
−2
(a) Males
0
20
40
60
Age
80
100
0
20
40
60
Age
80
100
Coherent functional forecasting
21
Ratio forecasts
4
3
1
2
Sex ratio of rates: M/F
3
2
1
Sex ratio of rates: M/F
4
5
Coherent forecasts
5
Independent forecasts
0
20
40
60
Age
80
100
0
20
40
60
Age
80
100
Coherent functional forecasting
22
Life expectancy difference: F−M
8
Life expectancy forecasts
4
Number of years
80
2
75
70
Age
85
6
90
95
Life expectancy forecasts
1960
2000
Year
2040
1960
2000
Year
2040
Coherent functional forecasting
23
Coherent forecasts for J groups
and
pt (x) = [ft,1 (x)ft,2 (x) · · · ft,J (x)]1/J
.
rt,j (x) = ft,j (x) pt (x),
Coherent functional forecasting
23
Coherent forecasts for J groups
and
pt (x) = [ft,1 (x)ft,2 (x) · · · ft,J (x)]1/J
.
rt,j (x) = ft,j (x) pt (x),
log[pt (x)] = µp (x) +
K
X
βt,k φk (x) + et (x)
k =1
L
X
log[rt,j (x)] = µr,j (x) +
l =1
γt,l ,j ψl ,j (x) + wt,j (x).
Coherent functional forecasting
23
Coherent forecasts for J groups
and
pt (x) = [ft,1 (x)ft,2 (x) · · · ft,J (x)]1/J
.
rt,j (x) = ft,j (x) pt (x),
log[pt (x)] = µp (x) +
K
X
βt,k φk (x) + et (x)
k =1
L
X
log[rt,j (x)] = µr,j (x) +
l =1
pt (x) and all rt,j (x)
are approximately
independent.
γt,l ,j ψl ,j (x) + wt,j (x).
Coherent functional forecasting
23
Coherent forecasts for J groups
and
pt (x) = [ft,1 (x)ft,2 (x) · · · ft,J (x)]1/J
.
rt,j (x) = ft,j (x) pt (x),
log[pt (x)] = µp (x) +
K
X
βt,k φk (x) + et (x)
k =1
L
X
log[rt,j (x)] = µr,j (x) +
γt,l ,j ψl ,j (x) + wt,j (x).
l =1
pt (x) and all rt,j (x)
are approximately
independent.
Ratios satisfy constraint
rt,1 (x)rt,2 (x) · · · rt,J (x) = 1.
Coherent functional forecasting
23
Coherent forecasts for J groups
and
pt (x) = [ft,1 (x)ft,2 (x) · · · ft,J (x)]1/J
.
rt,j (x) = ft,j (x) pt (x),
log[pt (x)] = µp (x) +
K
X
βt,k φk (x) + et (x)
k =1
L
X
log[rt,j (x)] = µr,j (x) +
γt,l ,j ψl ,j (x) + wt,j (x).
l =1
pt (x) and all rt,j (x)
are approximately
independent.
Ratios satisfy constraint
rt,1 (x)rt,2 (x) · · · rt,J (x) = 1.
log[ft,j (x)] = log[pt (x)rt,j (x)]
Coherent functional forecasting
24
Coherent forecasts for J groups
log[ft,j (x)] = log[pt (x)rt,j (x)] = log[pt (x)] + log[rt,j ]
K
L
X
X
= µj (x) +
βt,k φk (x) +
γt,`,j ψ`,j (x) + zt,j (x)
k =1
`=1
Coherent functional forecasting
24
Coherent forecasts for J groups
log[ft,j (x)] = log[pt (x)rt,j (x)] = log[pt (x)] + log[rt,j ]
K
L
X
X
= µj (x) +
βt,k φk (x) +
γt,`,j ψ`,j (x) + zt,j (x)
k =1
`=1
µj (x) = µp (x) + µr,j (x) is group mean
Coherent functional forecasting
24
Coherent forecasts for J groups
log[ft,j (x)] = log[pt (x)rt,j (x)] = log[pt (x)] + log[rt,j ]
K
L
X
X
= µj (x) +
βt,k φk (x) +
γt,`,j ψ`,j (x) + zt,j (x)
k =1
`=1
µj (x) = µp (x) + µr,j (x) is group mean
zt,j (x) = et (x) + wt,j (x) is error term.
Coherent functional forecasting
24
Coherent forecasts for J groups
log[ft,j (x)] = log[pt (x)rt,j (x)] = log[pt (x)] + log[rt,j ]
K
L
X
X
= µj (x) +
βt,k φk (x) +
γt,`,j ψ`,j (x) + zt,j (x)
k =1
`=1
µj (x) = µp (x) + µr,j (x) is group mean
zt,j (x) = et (x) + wt,j (x) is error term.
{γt,` } restricted to be stationary processes:
either ARMA(p, q) or ARFIMA(p, d , q).
Coherent functional forecasting
24
Coherent forecasts for J groups
log[ft,j (x)] = log[pt (x)rt,j (x)] = log[pt (x)] + log[rt,j ]
K
L
X
X
= µj (x) +
βt,k φk (x) +
γt,`,j ψ`,j (x) + zt,j (x)
k =1
`=1
µj (x) = µp (x) + µr,j (x) is group mean
zt,j (x) = et (x) + wt,j (x) is error term.
{γt,` } restricted to be stationary processes:
either ARMA(p, q) or ARFIMA(p, d , q).
No restrictions for βt,1 , . . . , βt,K .
Coherent functional forecasting
Li-Lee method
Li & Lee (Demography, 2005) method is a
special case of our approach.
ft,j (x) = µj (x) + βt φ(x) + γt,j ψj (x) + et,j (x)
where f is unsmoothed log mortality rate, βt is a
random walk with drift and γt,j is AR(1) process.
No smoothing.
25
Coherent functional forecasting
Li-Lee method
Li & Lee (Demography, 2005) method is a
special case of our approach.
ft,j (x) = µj (x) + βt φ(x) + γt,j ψj (x) + et,j (x)
where f is unsmoothed log mortality rate, βt is a
random walk with drift and γt,j is AR(1) process.
No smoothing.
Only one basis function for each part,
25
Coherent functional forecasting
Li-Lee method
Li & Lee (Demography, 2005) method is a
special case of our approach.
ft,j (x) = µj (x) + βt φ(x) + γt,j ψj (x) + et,j (x)
where f is unsmoothed log mortality rate, βt is a
random walk with drift and γt,j is AR(1) process.
No smoothing.
Only one basis function for each part,
Random walk with drift very limiting.
25
Coherent functional forecasting
Li-Lee method
Li & Lee (Demography, 2005) method is a
special case of our approach.
ft,j (x) = µj (x) + βt φ(x) + γt,j ψj (x) + et,j (x)
where f is unsmoothed log mortality rate, βt is a
random walk with drift and γt,j is AR(1) process.
No smoothing.
Only one basis function for each part,
Random walk with drift very limiting.
AR(1) very limiting.
25
Coherent functional forecasting
Li-Lee method
Li & Lee (Demography, 2005) method is a
special case of our approach.
ft,j (x) = µj (x) + βt φ(x) + γt,j ψj (x) + et,j (x)
where f is unsmoothed log mortality rate, βt is a
random walk with drift and γt,j is AR(1) process.
No smoothing.
Only one basis function for each part,
Random walk with drift very limiting.
AR(1) very limiting.
The γt,j coefficients will be highly
correlated with each other, and so
independent models are not appropriate.
25
Coherent functional forecasting
Some conclusions
New, automatic, flexible method for
coherent forecasting of groups of
functional time series.
26
Coherent functional forecasting
Some conclusions
New, automatic, flexible method for
coherent forecasting of groups of
functional time series.
Based on geometric means and
ratios, so interpretable results.
26
Coherent functional forecasting
Some conclusions
New, automatic, flexible method for
coherent forecasting of groups of
functional time series.
Based on geometric means and
ratios, so interpretable results.
More general and flexible than
existing methods.
26
Coherent functional forecasting
Some conclusions
New, automatic, flexible method for
coherent forecasting of groups of
functional time series.
Based on geometric means and
ratios, so interpretable results.
More general and flexible than
existing methods.
Easy to compute prediction
intervals.
26