Evaluation and Impact
Estimation at the OFT and
its applicability to CREW
Natalie Timan
Deputy Director, Office of Fair Trading
13 March 2013
1
Introduction
● Contents:
- Evaluation objectives: why do we measure impact?
- Types of evaluation within the OFT
- Essential features of an Impact Assessment
- Methodology
- Two worked examples
- Application to CREW
- Ex-post evaluation
2
Evaluation objectives
● The aim of our evaluation work is two-fold:
- External accountability: to evaluate whether the OFT delivers its
objectives and does so cost effectively and to demonstrate this externally;
- Internal management and prioritisation: to inform future OFT work
regarding how we prioritise, conduct and follow-up our work to ensure we
maximise our impact. This is not an internal evaluation of our procedures
or efficiency, but an assessment of the accuracy of our predictions about
markets and the effectiveness of our interventions.
3
Types of evaluation
● Impact estimation: ex-ante estimation of anticipated impact usually
undertaken shortly after the OFT’s intervention.
- Direct financial savings for consumers only – no wider benefits.
● Ex-Post evaluations: ex-post estimation of actual impact usually
undertaken some years after the OFT’s intervention.
● Application to CREW: ex-ante estimation may be the more useful type of
evaluation given:
- Cost effectiveness;
- Data limitations;
- Time constraints.
4
Welfare standard
● Both types of evaluation look at benefits to consumers as the OFT has a
Consumer Welfare standard.
- Typically estimate effect on prices and output as a proxy for consumer
welfare
● OFT does not estimate potential impact of its decisions on business as
part of positive impact estimation.
● Impact estimation looks at quantifiable direct financial benefits to
consumers:
- Decrease in price;
- Monetized improvements in quality, range or service;
- Monetized time savings;
- Benefits that consumers gain from making better informed choices about
what goods to purchase.
5
Essential features of
OFT’s Impact Estimation I*
● Assessed on regular (annual) basis during next year.
● Relatively undemanding in cost and time, utilising data from the original
interventions and/or simple default assumptions.
● Estimates are performed using ex-ante data.
● Assumed that no intervention can have a negative impact.
● Estimates are deliberately ‘conservative’.
* These features were presented by Professor Steve Davies to the OECD on 25 February 2013
6
Essential features of
OFT’s Impact Estimation II
● Estimates in terms of static consumer benefits.
● Annual moving averages employed.
● ‘Point’ estimates, rather than a range of plausible values.
● Covers mergers, cartels, abuse, market studies and consumer
enforcement. Does not cover advocacy of competition to other
government departments.
● Deterrence impact not included in estimate.
● Possible beneficial effect of competition policy of productivity and
innovation (and growth) not estimated.
7
Methodology: assumptions
● To estimate the impact of each individual case intervention, information
is required on:
- The size of affected turnover;
- The price increase removed or avoided;
- The length of time the increased price would have prevailed absent the
intervention.
● An adjustment could be made for the deadweight loss (i.e. Surplus lost
by consumers who are deterred from making any consumption).
● Ideally use any case-specific estimates from the case team; if not default
assumptions required:
8
Methodology:
default assumptions
Affected turnover
Price effect
Duration (yrs)
Mergers
Cartels
Abuse
Turnover of affected
goods
Turnover of affected
goods
Turnover of affected
goods
10-15%
10-15%
6
6
GUPPI (or 3-5%) plus
deadweight loss
estimate
2
9
Two recent examples:
An Abuse of dominance case
Summary of total consumer detriment two year period
Product A
£m
Firm 1
50
Firm 2
20
Firm 3
30
Firm 4
40
Total harm over 2 year period
Total harm over 2nd year alone
Assumed duration (years) into future absent OFT intervention ie we think it would have continued for further
2 years
Total consumer detriment averted in following 2 years (deflated and discounted to prices at decision date)
140
93
2
291.96
●
The figures represent the extra turnover derived from the implementation of various initiatives over a two
year period, borne directly by the UK consumer.
●
This is calculated by multiplying the volumes sold by the amount of increment, bearing in mind that the
price increases were maintained and carried over into the following years. So, for example, if the price of
one product was increased by £5 in year one and £5 in year two, then the total benefit would be £15.
10
Two recent examples:
A cartel case
Turnover
and Profit
(£m)
Net Fees
Net fees
converted
from year of
cartel to
prices at
year of
decision
Firm 1
Firm 2
Firm 3
Firm 4
Firm 5
Firm 6
Total £m
£5
£2
£3
£10
£20
£10
£50
£5.1
Saving in
net fees
from OFT
intervention
15%
£0.8
Savings in
net fees
from OFT
intervention
over 3 years
3
£2.2
•
Attribution of impact is justified by:
£2.0
£3.1
£10.0
£20.4
£10.2
£51
£0.3
£0.5
£1.5
£3.1
£1.5
£8
£0.9
£1.3
£4.4
£8.9
£4.4
£22
•
•
•
•
Evidence that cartel met after application for leniency
Assume 15% reduction in net fees
Assume 3 years duration. Duration is scaled down from rule of thumb of 6 years on account of:
•
•
•
Leniency programme was used
Instability of cartel
Some participants were not adhering to rules of cartel
Total impact: £22m
11
Impact estimation
2009-10 Benefits
2011-12 Benefits
Competition
enforcement
£84m
£83m
£151m
Merger control
£125m
£90m
£8m
Market studies and
reviews
£107m
£117m
£207m
Consumer protection
enforcement
£42m
£36m
£35m
£359m
£326m
£402m
7
7
8
Total estimated
benefits
Benefit/Cost ratio
●
2010-11 Benefits
The figures presented above are 3-year rolling averages.
12
Application to CREW I
● Rules of thumb
- Affected turnover
- Discount rate
- Cartel overcharges in developing countries
- Price rise in abuse cases
- Alternatives to merger simulation
- Duration
13
Application to CREW II
● Deterrence
● Producer surplus: barriers to entry
● Competition, innovation and growth
- Output = f [natural resources, capital, labour]
- Prioritise interventions that lift restrictions on e.g.:
technology, management
• Land (natural resource);
• Finance (capital)
• Markets associated with labour participation and higher education
(labour)
• Innovation intensive sectors e.g. Pharmaceuticals
• Tackling public restrictions on competition -> managers not sheltered
from impact of competition
14
Ex-Post Evaluation
● What is ex-post evaluation?
- An ex-post assessment of the actual impact of our interventions in
enforcement cases.
- Primary purpose: measure the net benefits to consumers of any
intervention. Secondary purpose: examine wider benefits (usually
unquantifiable), such as the effect on consumer confidence and
deterrence.
- Typically compares the quantified net benefit to the cost incurred by the
OFT in the initial intervention.
● How do we evaluate?
- Can involve: (i) desk-based research on prices and other market data, (ii)
surveys of consumers, businesses and other stakeholders.
- Usually conduct two to three evaluations a year. Can be conducted
immediately after a case is completed, but typically there is a delay,
potentially up to several years, whilst recommendations are implemented.
15
Methodology:
Ex-post evaluation
● Formal techniques:
-
Simulations and structural models
-
Event studies
-
Difference in Difference techniques
● Reality:
-
Identify changes in market characteristics (price, quality, choice etc)
-
Identify changes in consumer behaviours (shopping habits, awareness
and understanding, product selection)
-
Link those changes to actions or recommendations – are the changes
attributable to us?
-
If possible monetise the changes that are attributable to authority
16