• Fieldwork Methodology
• Interviewer Feedback
• Stated Preference Methodology
• Stated Preference Findings
• Summary of Recommended Changes

• Objectives of pilot were to test:
– the recruitment process
– the survey hit rate
– the clarity and flow of the questionnaire
– the appropriateness of the language used
– the accuracy of all routings
– ease of use of the show material
– design and understanding of the stated preference exercises
– the interview duration
• Approach:
– Businesses: 40 CATI phone-post/email-phone (41 achieved)
– Residential: 100 CAPI in-home interviews (99 achieved)
• Quotas:
– Business:
• 10 EPN (11 achieved), 10 SPN (10 achieved), 20 LPN (20 achieved)
• Mix of size (20 small; 15 medium; 6 large) & sector
• Minimum 10% with outage experience (39% achieved)
– Residential:
• 25 EPN (25 achieved), 25 SPN (25 achieved), 50 LPN (49 achieved)
• Mix of age & SEG

30
40
50
50
DOMESTIC
Mean = 32 minutes
BUSINESS
Mean = 43 minutes
43.9
40
35.3
31.3
29.3
30
21.9
20
18.2
20
15.2
10 10
4.9
0
25 minutes or less
>25-30 minutes
>30-45 minutes
45 mins+
0
25 minutes or less
>25-30 minutes
>30-45 minutes
45 mins+

• Problems with clarity & flow:
– Question 34: ‘did you find each of the levels of service described easy to follow?’… if you said
‘no’, Question 35 then said ‘which levels weren’t clear to you?’; seem like separate issues
– Need showcard for income question
• General comments on what would make it easier
– Concern that people will think we are trying to sell them services for an electricity company.
Also, many people less aware of Ofgem than of
Ofwat or the FSA, so they were very cautious.
Maybe a more elaborate explanation of the survey might help.
• Problems with routing:
– Q22b; if they say don’t know should skip subsequent question • Interview duration:
– Longer than budgeted for in most instances; cannot be done within current fee (due both to time required for each and impact of length on hit rates) and current incentive (due to length)
• Other:
– Some issues with defining rural versus urban – we will use a postcode dbase to do so in the mainstage
– Add all screening questions (eg age, SEG) to main questionnaire
– Add question about number of children 5+

• Problems with clarity & flow:
– Some people assumed it was a sales call despite the current intro. Need to further emphasize that Ofgem are a regulatory body and that the research is looking at improvements that could be made to the services provided to their company
– RQ13 – need a “don’t know code
– Q23: some said “both”; add this code?
– Q26: needs a don’t know or can’t remember
– Q40: needs rephrasing so it doesn’t suggest anything will be sent; needs to separate leaflet and email; needs to be multicode
• Interview duration:
– Longer than budgeted for in most instances; cannot be done within current fee (due both to time required for each and impact of length on hit rates)
• Other:
– Some issues with defining rural versus urban – again, we will use a postcode dbase to do so in the mainstage
• Problems with routing:
– none

• Objective: to obtain estimates of customers’ willingness-to-pay for improved service levels (or willingness-to-accept for reduced service levels)
• No revealed preference evidence, so need to rely on stated preferences
– Customers are not offered different service levels and prices in real world
• Two possible stated preference approaches:
– Contingent Valuation questions (CV) – ask respondents directly questions about valuations
– Discrete choice experiments (SP DCE) – elicit valuations through tradeoffs
• Based on hypothetical choices
• Essential that price is one of the attributes

Choice 3
Which electricity distribution service would you choose?
Frequency of Power Cuts (more than 3 minutes)
Average number of power cuts longer than 3 mins in normal weather conditions
Duration of Loss of Power
Average duration of power cut
Frequency of Short Interruption (less than 3 minutes)
Average number of power cuts shorter than 3 mins in normal weather conditions
Provision of Information
Information provided during power cuts
Cost
Annual Electricity Bill
As Now
4 in 5 years
100 mins on average
5 in 5 years
Telephone information line
£200
(no change)
Alternative 1
7 in 5 years
(more than now)
115 mins on average
(more than now)
6 in 5 years
(more than now)
Telephone information line plus call backs to provide information updates
£198
(£2 decrease)
Choice (mark "X" in preferred option)
Alternative 2
5 in 5 years
(more than now)
80 mins on average
(less than now)
5 in 5 years
Telephone information line
£205
(£5 increase)

Contingent Valuation (CV)
Discrete Choice Experiments
(DCE)
May present cognitive problems
DCE does not explicitly ask about money values, so may be easier for people to understand
Typically collect one response
More efficient - typically collect multiple responses from from each individual individuals
Argued that CV should be chosen when WTP for the good or service in total is required
Consumer surplus methods can be used to estimate the value of a total good from DCE data

Experiment 1
All Distributors except LPN LPN
Frequency of power cuts over 3 mins Frequency of power cuts over 3 mins
Average duration of power cuts over 3 mins
Average duration of power cuts over 3 mins
Number of short power interruptions Number of short power interruptions
Provision of Information
Restoration of supply (time) Restoration of supply (time)
Compensation for restoration of supply Compensation for restoration of supply
Experiment 2
Experiment 3
Compensation for multiple interruptions Compensation for multiple interruptions
Planned interruptions - notice
Undergrounding
Network resilience to major storms
Network resilience to flooding
Reduction in carbon emissions
Planned interruptions - notice
Provision of Information
Network resilience to flooding
Reduction in carbon emissions
Cost also included in all experiments

Experiment 1
All Distributors except LPN LPN
Frequency of power cuts over 3 mins Frequency of power cuts over 3 mins
Average duration of power cuts over 3 mins
Average duration of power cuts over 3 mins
Number of short power interruptions Number of short power interruptions
Provision of Information
Restoration of supply (time) Restoration of supply (time)
Experiment 2
Compensation for restoration of supply Compensation for restoration of supply
Experiment 3
Making and keeping appointments
Planned interruptions - notice
Network resilience to major storms
Network resilience to flooding
Reduction in carbon emissions
Making and keeping appointments
Planned interruptions - notice
Network resilience to flooding
Reduction in carbon emissions
Energy efficiency advice
Energy efficiency advice Provision of Information
Cost also included in all experiments

• Potential problem with separation of attributes across experiments is that results may overestimate total wtp
• Therefore we also include a higher-level experiment, including all attributes to test total wtp
– We also include a series of CV questions to elicit total wtp for best service
• The higher-level experiment is simplified so that
‘group’ of attributes are varied simultaneously
• Two levels tested
– Lowest service level
– Highest service level
• Still complicated

Choice 1
Frequency of Power Cuts over 3 mins
Average number of power cuts longer than 3 mins in normal weather conditions
Duration of Loss of Power
Average duration of power cut
Frequency of Power Cuts less than 3 mins
Average number of power cuts shorter than 3 mins in normal weather conditions
Provision of Information
Information provided during power cuts
Restoration of Supply (Normal Weather)
Restoration of electricity supply as a result of problems not related to weather
Compensation for Failure to Restore Supply (Normal Weather)
Compensation if your electricity is not restored in this time
Compensation for Multiple Interruptions
Number of interruptions lasting 3 or more hours in a year which will entitles you to compensation payment
Planned Interruption to Electricity Supply
Amount of notice that customers must be given before planned interruptions to electricity supply
Undergrounding
An ongoing commitment to underground overhead lines for amenity reasons
Network resilience to major storms
Number of customers affected by major storms
Network resilience to flooding
Expenditure on flood defences
Reduction of carbon emissions
Replacement of mobile generation equipment and vehicles with those using less polluting vehicles
Cost
Annual Electricity Bill
Which electricity distribution service would you choose?
Alternative 1 Alternative 2
7 in 5 years
(more than now)
120 mins on average
(more than now)
7 in 5 years
(more than now)
Telephone information line
1 in 5 years
(less than now)
80 mins on average
(less than now)
3 in 5 years
(less than now)
Telephone information line plus call backs to provide information updates
Guaranteed within 18 hours
£50 plus £25 for every additional 12 hour period
Compensation after
5 interruptions
2 days notice
5% of overhead lines per annum
144000 customers on average in a year
(20% less than now)
Increase by 15%
Replace 10% per year with those using less polluting fuels
£207
(£7 increase)
Guaranteed within 18 hours
£50 plus £25 for every additional 12 hour period
Compensation after
5 interruptions
2 days notice
None
216000 customers on average in a year
(20% more than now)
Current expenditure
Continue usage of current equipment and vehicles
£196
(£4 decrease)
Choice (mark "X" in preferred option)
Neither

• Analysis based on principles of utility maximisation
• ‘
Utility
’ assigned to each alternative, made up of:
– Attributes of the alternative
– Characteristics of the respondent, e.g. income, etc.
• Assumption that error distribution is extreme
value (iid) leads to logit
model
• Maximum likelihood techniques used to obtain values of unknown parameters
– Find the β s that optimise explanation of the model
• Model outputs
– Coefficient estimates (and t-statistics)
– Model fit statistics

Parameters to be estimated
U i
 

 
 X
1
 
2
X
2

...
  n
X n
  i
Characteristics of alternative/individual
Random error reflecting influence of parameters not included, differences between individuals/responses

Assumption that random terms (

) are independent and identically distributed across alternatives (IID) and that they vary with an Extreme-Value distribution
P i

 j

J e
Ui e
Uj

• Coefficients used to:
– Determine the relative importance of attributes
– Determine the monetary value for attributes
– Specify utility functions for prediction models
• Monetary values for attributes identified in lowerlevel games
• May be adjusted by values for groups of attributes identified by higher-level game or by CV question
– This is a judgement call

• Trading behaviour
• Background Questions on respondent understanding
• Model Results

• Domestic – London
Trading
Always Current Option
Always Alternative 1
Always Alternative 2
Neither/No Choice
Trading among Alternatives
Total Observation
Exp 1 Exp 2 Exp 3 Exp 4
3 2 2 n/a
0 0 1 0
0
0
0
0
0
0
0
2
46 47 46 47
49 49 49 49
Trading
Always Current Option
Always Alternative 1
Always Alternative 2
Neither/No Choice
Trading among Alternatives
Exp 1 Exp 2 Exp 3 Exp 4
4 3 3 n/a
1
0
1
0
0
0
1
0
0
45
0
46
0
47
1
48
50 Total Observation 50 50 50
• Domestic – Other DNOs

• Business – London
Trading
Always Current Option
Always Alternative 1
Always Alternative 2
Neither/No Choice
Trading among Alternatives
Total Observation
Exp 1
4
0
0
0
16
20
Exp 2
5
0
0
0
15
20
Exp 3
4
1
0
0
15
20
Exp 4 n/a
2
0
2
16
20
Trading Exp 1 Exp 2 Exp 3
Always Current Option 0 0 0
Always Alternative 1
Always Alternative 2
Neither/No Choice
0
0
0
0
0
0
0 0 0
Trading among Alternatives 21 21 21
Total Observation 21 21
• Business – Other DNOs
21
Exp 4 n/a
0
0
1
20
21

Examining choices at different cost levels implies that many domestic customers were willing to pay 15% increases in DUOS for service improvements
Experiment 1
Other DNOs
Experiment 2 Experiment 3
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0%
Cost Level of Alt A
(% adjustment to DUOS)
5% 10% 15%
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0%
Cost Level of Alt A
(% adjustment to DUOS)
5% 10% 15%
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0%
Cost Level of Alt A
(% adjustment to DUOS)
5% 10% 15%
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0%
Cost Level of Alt B
(% adjustment to DUOS)
5% 10% 15%
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0% 5%
Cost Level of Alt B
(% adjustment to DUOS)
10% 15%
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0%
Cost Level of Alt B
(% adjustment to DUOS)
5% 10% 15%

A wider range of costs was investigated in the higher-level experiment with large numbers of domestic users choosing alternatives with large cost increases
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt A
(% adjustment to DUOS)
20% 25% 30%
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt B
(% adjustment to DUOS)
20% 25% 30%

Examining choices at different cost levels implies that many business customers were willing to pay 15% increases in DUOS for service improvements
Experiment 1
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0%
Cost Level of Alt A
(as % of Annual Bill)
5% 10% 15%
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0%
Cost Level of Alt B
(as % of Annual Bill)
5% 10% 15%
Other DNOs
100%
80%
60%
40%
20%
0%
-15% -10%
Experiment 2
-5% 0%
Cost Level of Alt A
(as % of Annual Bill)
5% 10% 15%
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0%
Cost Level of Alt B
(as % of Annual Bill)
5% 10% 15%
Experiment 3
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0%
Cost Level of Alt A
(as % of Annual Bill)
5% 10% 15%
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0%
Cost Level of Alt B
(as % of Annual Bill)
5% 10% 15%

A wider range of costs was investigated in the higher-level experiment with large numbers of business users choosing alternatives with large cost increases
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0% 5% 10%
Cost Level of Alt A
(as % of Annual Bill)
15% 20% 25% 30%
100%
80%
60%
40%
20%
0%
-15% -10% -5% 0% 5% 10%
Cost Level of Alt B
(as % of Annual Bill)
15% 20% 25% 30%

• ‘Were you able to make the comparisons in the choices we presented to you’
– Domestic
• 48 of 49 respondent in London reported they were able to make the comparisons
• 47 out of 50 respondents in other DNOs reported they were able to make the comparisons
– Business
• 19 of 20 respondents in London reported they were able to make the comparisons
• 20 of 21 respondents in other DNOs reported they were able to make the comparisons

A sizeable proportion of domestic customers felt that the levels of service in the choices were unrealistic, and the experiments were difficult
• ‘Were the levels of service we have been asking about in the choices realistic?’
– London, 11 of 49 domestic respondents felt the service levels were not realistic or too difficult
• A 1 in 10 year electricity cut is not realistic because we are having electricity cut 6 times in a year
• I don’t think all the changes will be possible within the costs stated
• It is seriously insane to raise the bills again and again
• No electricity company would cut emissions, as they are not efficient enough to replace the current system
• There were too many options and everything was too similar
• I don’t believe the whole concept, it’s not clear enough
• They are all the same
• Too much complicated
– Other, 8 of 50 domestic respondents felt the service levels were not realistic or too difficult
• Because they are not achievable, ie the cost of putting all these cables underground isn’t going to be economically viable
• The last thing I want is for my bill to go up
• They are talking about increasing expenditure by certain percentages, but the original expenditure is not listed… so 15% increase of what?
• The comparisons are not obvious, got to really read in order to give an answer
• The different choices or packages didn’t always seem logical.
• Too much to absorb all at one for the average consumer who is having enough problems paying their bills

• ‘Were the levels of service we have been asking about in the choices realistic?
– London, 3 of 20 respondents felt the service levels were not realistic
•
7 power cuts in 10 years is outrageous
• Because we had 3 power cuts in the last 2 years of long duration
• They do not employ enough people, so realistically it is not going to happen
– Other, 4 of 21 respondents felt the service levels were not realistic
• Because some reductions are not able to be guaranteed. There are elements outside your control, i.e. the power cut we experienced was due to accidental cutting through power cables.
• Some options offered more for less….doesn’t make sense.
• Typical of today’s climate, people want all these things, but are not prepared to pay for them. Realistically, these improvements would probably cost too much.
• Some of the improvements would be very difficult to achieve.

• Domestic Customer Models
– Results are very encouraging
– Cost coefficient is significantly negative
– Most other coefficients have intuitive signs, many are significant
–
Resulting wtp values look reasonable
• Business Customer Models
–
Many statistically insignificant terms, probably a result of small sample sizes and heterogeneity of sample, e.g. in terms of size of business
– Expect model to be improved with larger sample size in main survey
– Most important variables:
• Power cuts
•
Restoration of supply
•
Reduction of carbon emissions

• “Very complex”
• “Too much to consider”
• “Consider reducing and rotating over 2 interviews to make each survey shorter”
• “Maybe cut down on the sets of choices; if possible, 6 instead of 8. Choice sets are quite long”
• “Most respondents found the exercise difficult or complex at first but then adapted to it. It is a bit difficult to keep them interested in the survey till the end”
• “The interview should be shortened if possible”
• “It’s a lot of reading”

• Subject area is abstract and the choices are complex, but – despite some concerns on the part of interviewers – respondents seem to have been able to undertake the SP experiments
• Some, mostly minor, revisions required for main survey
– Main area for review is cost levels
• Mainstage interviews:
– 2,100 domestic face-to-face interviews (150 interviews conducted in each of the
14 DNO territories)
– 1050 CATI business interviews (75 per DNO territory)

• Review cost levels
– Pilot cost levels:
Levels
Exp 1 - 3
Higher Level
1
-15%
-10%
-12%
-15%
2
-10%
-4%
-6%
-8%
3
-5%
0%
0%
0%
4
-2%
2%
3%
4%
5
0%
6%
7%
8%
6
2%
10%
12%
14%
7
5%
16%
18%
20%
8
10%
22%
24%
25%
9
15%
26%
28%
30%
– Fewer reductions and more increases for Experiments 2 and 3, e.g. 2 reductions, 6 increases and base
– Recommend that we incorporate larger cost increases
• Review wording of attributes and levels to simplify as much as is possible (may not be much scope for this)
– Include ‘better than now’ and ‘worse than now’ descriptors, where appropriate
• Review ‘Network resilience to flooding’ definition
– Complaint that it was difficult to make comparisons when base level is not defined
• Reduce number of choices in lower-level experiments from 8 per experiment to 6 per experiment

• Minor Text Changes
– Compensation for Multiple Interruptions: m inor error in wording ‘which will entitles you…’, plus ( £x) in text to be removed
– Network Resilience to Major Storms: include a ‘,’ separator in the numbers, e.g. 180,000 rather than 180000
• Minor Changes to Questions/Routing
– Slightly reword Q34 and/or Q35 (domestic) / Q46 and/or Q47 (business)
– Amend routing for Q22(domestic) / Q33 (business) “don’t knows”
– Use postcode dbase to define rural & urban
– Add age & SEG data from screener to main questionnaire (domestic only)
– Add question about number of children 5+ (domestic only)
– RQ13 (business) – need a “don’t know code
– Q18 (domestic)/Q23 (business): some said “both”; add this code?
– Q20 (domestic)/Q26 (business): needs a don’t know or can’t remember
– Q40 (business only): needs re-phrasing so it doesn’t suggest anything will be sent; needs to separate leaflet and email; needs to be multicode
• Expand explanation of survey & of Ofgem
• Reduce the length of the questionnaire or increase fees & incentives