How reliable are household surveys?
Phil Bunn, Gareth Anderson, Alice Pugh, Arzu Uluc
Bank of England
Household Wealth Data and Public Policy Conference, 9-10 March 2015
1
Motivation
• Need to understand how reliable surveys are if using them to
inform policy decisions
• Evaluate the Bank’s own household survey, run by NMG
Consulting
2
Outline
• Comparing household surveys to each other and aggregate
data: income and mortgage debt
• What have we learnt about methodology from running the Bank
of England/NMG survey: moving online
3
Previous literature
• IFS work shows consumption has been increasingly under-recorded in
the Living Costs and Food Survey, (Banks and Johnson (1998) and
Barrett et al (2013))
• Previous Bank work looked at reliability of balance sheet data in the
British Household Panel Survey, (Redwood and Tudela (2004))
• Our focus is on balance sheet data and financial distress
4
Surveys considered
Survey
Annual sample
Main focus
Family Resources Survey
25,000
Income
Living Costs and Food Survey
5,000
Consumption
Understanding Society/
BHPS
25-30,000/
5000
Social & economic
change
Wealth and Assets Survey
20-30,000 per 2
year wave
Wealth
Bank of England/NMG Survey
2-6,000
Balance sheets/
Attitudes to spending
5
Bank of England/NMG Survey
• Annual survey since 2004, run during September
• Data on Bank website and annual Quarterly Bulletin article
• Run online since 2012
• Now covers 6,000 households a year, previously 2,000
• Focus on balance sheets and attitudes to spending
• Main benefits for us are timeliness and ability to add bespoke questions
6
A measure of gross cash income
+
+
+
+
=
Employee wages and salaries
Self-employment income
Government benefits
Private pension payments
Investment income
Total gross cash income
7
Average household income
Annual income per household, £
50000
45000
40000
35000
30000
ONS aggregate
FRS
LCFS
USoc/BHPS
NMG
WAS (2010-12)
25000
20000
15000
10000
5000
0
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
8
Distribution of individual incomes (2011)
Percentage of income
30%
HMRC 2011-2012
USOC 2011
WAS 2010-12
25%
20%
15%
10%
5%
0%
Lower value of income band
9
Income coverage by component (2011)
Percentage coverage relative to aggregate data
250
FRS
LCFS
200
Usoc
WAS
150
100
50
0
Wages and salaries
Self employment
Benefits
Investment income
10
Reliability of income data: summary
• Average income in surveys about 20% lower than implied by
aggregate data, except WAS, which is in line
• Degree of under-recording fairly constant over time
• WAS better captures high debt to income households, although also
has higher investment income
• NMG survey 5-10% below other household surveys
11
Average mortgage debt
Outstanding mortgage debt per household, £
160000
ONS aggregate
FRS
LCFS
USoc/BHPS
NMG
WAS
140000
120000
100000
80000
60000
40000
20000
0
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
12
Average mortgage debt
Outstanding mortgage debt per household, £
160000
ONS aggregate
FRS
LCFS
USoc/BHPS
NMG
WAS
ONS less BTL estimate
140000
120000
100000
80000
60000
40000
20000
0
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
13
Mortgage debt to income distribution (2012)
Percentages of mortgagors
35
FRS
LCFS
NMG
WAS (2010-12)
30
25
20
15
10
5
0
<1
1-2
2-3
3-4
Mortgage debt to income ratio
4-5
>5
14
Reliability of mortgage data: summary
• All surveys under-record growth of mortgage debt since early 2000s
• Some, but not all, of that reflects buy-to-let lending
• Differences in share of high debt households quite small between
surveys, although proportion is a bit higher in NMG survey
15
What have we learnt moving the NMG
survey online
16
Moving the NMG survey online
• What did we do?
• Measurement of financial distress
• Measurement of other financial variables
• Response rates
• Who has internet access?
• Bespoke questions
17
Moving the NMG survey online
• 2 online pilots in 2010 (500 households) and 2011 (1000 households)
alongside full face to face surveys (2000 households)
• Full survey online from 2012 (4000 households) and face-to-face
parallel run of small number of questions (2000 households)
• Increased sample size from 2,000 to 6,000 by 2013
18
Mortgage payment problems
Percentages of mortgagors having problems paying for accommodation
25
BHPS/USOC
NMG face to face
NMG online
20
15
10
5
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
0
2014
19
Mortgage arrears
Percentage of mortgages in arrears for at least 3 months
6
CML aggregate data
BHPS
NMG
WAS
4
2
0
1987
1992
1997
2002
2007
2012
20
Average income and mortgage debt
Online
2012
Face-to-face
Mean gross annual income
£31,524
£33,228
Mean mortgage debt (for mortgagors)
£85,189
£84,004
21
Mortgage debt distribution
Income distribution
Percentages of mortgagors
Percentages of households
12
20
NMG online
NMG face-to-face
FRS
18
NMG online
10
16
14
12
NMG face-to-face
8
FRS
6
10
8
4
6
4
2
2
Annual income (£, thousands)
0
<10
10
20
30
40
50
60
70
80
90
100
110
130
150
170
190
210
230
250
275
300
400
500
750
1000+
0
Mortgage debt (£, thousands)
22
Response rates
Proportion of refusals/don’t knows
Online
2012
Face-to-face
Income question
9%
35%
Mortgage debt question
5%
29%
Mortgage payment problems question
3%
7%
23
Household with internet access
By income
By age
Percentages of households
100
20-30
30-40
40-50
50-60
60-70
70+
Percentages of households
100
80
80
60
60
40
40
20
20
0
0
5-10 10-2020-3030-4040-5050-6060-7070-80 80+
Annual income (£, thousands)
24
Impact of a 1pp rise in interest rates
Borrowers vs savers
Impact on post-tax income
Impact on consumption via cashflow effect
By age
Per cent
4
Impact on post-tax income
Impact on consumption via cashflow effect
Per cent
4
3
3
2
1
2
0
1
-1
0
-2
-1
-3
-4
Mortgagors
Unsecured
borrowers
Savers
All
-2
18-24
25-34
35-44
45-54
55-64
65+
All
25
Benefits of moving the NMG survey online
• Cheaper/easier to administer
• More accurate reporting of financial distress, not much effect on other
variables
• Higher response rates
• Easier to include more innovative questions
• Downside is that not all households have internet access and no
opportunity to clarify questions
26
Conclusion
• Household surveys appear reasonably, but not perfectly, reliable.
• Surveys under-represent high income households, except WAS
• All surveys under-record the growth of mortgage debt
• NMG survey broadly follows other surveys. Does less well at margin
but is better at measuring financial distress
• There are advantages to conducting household surveys online –
experience with the NMG survey has been positive
27