Savings Estimation Methods
for Energy Efficiency Programs:
A Half-Hour Guide
Kansas Corporation Commission
Workshop on Energy Efficiency
March 25, 2008
Marian Brown
Manager of Measurement & Evaluation
Energy Efficiency Division
Southern California Edison
marian.brown@sce.com
(626) 812-7662
Why Measure Energy Efficiency Program
Savings?

Metering the un-metered resource




Measured savings for resource planning
Measured benefits from reduced pollution/GHG
Accountability for ratepayer dollars and
shareholder reward
Improve programs

More accurate savings estimates for measures

Increased energy savings from better design

Basis for future EE program funding
Southern California Edison
2
Types of Programs and Impact
Evaluations

Resource Acquisition Programs –


Education/Information Programs


Create direct energy savings
Create indirect energy savings
Market Transformation Programs

Create long-term changes in product availability,
cost and features, plus customer awareness,
understanding and purchase behavior
Southern California Edison
3
Direct Energy Savings

Program participation is defined as purchasing or
adopting the energy efficiency measures.



That’s how rebate and direct installation programs work.
So the first job is to estimate the energy savings
achieved from those adoptions (called gross energy
savings).
Then the follow-up question is in which cases the
program caused the purchase/installation/adoption
(net energy savings; attribution; net-to-gross ratio).
Southern California Edison
4
Indirect Energy Savings

The program activity is intended to influence a
decision to adopt efficiency measures,
rather than require it as a condition
for participation.



Examples: Energy audits, energy efficiency training,
education, information, market transformation.
So the evaluation must first determine whether the
program did indeed lead to adoption of energyefficient measures (attribution/net-to-gross ratio).
Then, it must estimate what the resulting energy
savings were.
Southern California Edison
5
Estimating Gross
Energy
and Demand
Savings
For Both Direct and Indirect
Savings Programs
Southern California Edison
6
The Two Basic Approaches
Statistical Billing (Energy Use)
Analysis



Simple billing analysis
normalized for weather
Regression analysis
Multi-stage regressions
(can include net savings
analysis)
Engineering Measurement and
Analysis

Pre-post measurements and
engineering algorithms

Pre-post building energy
simulation models
Sometimes, the two approaches are combined.
We try to include economic/behavioral effects wherever possible.
Southern California Edison
7
Which Measurement & Analysis Method
to Use for the Energy Savings?



Methods range from simple and direct to complex
and indirect and sometimes combined.
More complex methods generally require more
detailed data and higher cost.
Guidelines for good measurement/analysis:


The Model Energy Efficiency Program Impact Evaluation
Guide covers both billing analysis/regression methods and
engineering methods, plus net savings methods.
IPMVP (International Performance Measurement &
Verification Protocol) is a widely recognized standard for
engineering measurement and analysis approaches.
Southern California Edison
8
Billing Analysis
Tends To Be Preferred When:

Both pre- and post-retrofit billing data are available.

Expected energy savings are large enough to be
statistically observable in a billing analysis (at least
10% of total usage; depends on several factors).

Analysis can include large numbers of participants
that are reasonably homogeneous.

Inclusion of a comparison or control group allows for
behavioral change corrections and/or simultaneous
estimation of net savings
Southern California Edison
9
Engineering Measurement & Analysis
Tends To Be Preferred When:

No pre-installation billing data are available
(e.g., new construction).

Expected energy savings are a small fraction of
total energy usage.

Program has a small number of participants or
unique measures.

Program itself includes substantial engineering
analysis that can be built on.
Southern California Edison
10
Using a Sample to Estimate Program Savings


Choosing when to use a sample:

Can you get good enough data for the whole population of participants?
(And what is the whole population?)

Or should you gather more detailed and accurate data for a good sample?
Questions to ask to assure reliable results from a sample:

Is it unbiased (or properly weighted to be unbiased)?
•
•
Non-response bias (do some groups respond at lower rates?)
Survey selection bias: Did it exclude relevant groups from the sample?
•

Can you think of some examples?
Does it give the needed level of precision?
•
Depends on sample size and variance in the population.
•
How far off can an estimate be and still be useful to you?
Southern California Edison
11
A Key Question: What Would Energy Use
Be in the Absence of the Program?

We can measure energy use before the
program.

We can measure energy use after the program.


But how much of the change was due to the program?
Conventionally, we look at this question in two
dimensions:

One related to gross energy savings;

One related to attribution and net energy savings.
Southern California Edison
12
Gross Energy Savings Baseline

What technology/behavior choice was being
made?

Pure retrofit: Early replacement or add-on


Then savings is prior energy use minus new energy
use
Replacement on burnout or new construction (lost
opportunity market)

Then savings is efficient equipment energy use
minus usage of a standard efficiency new measure
Southern California Edison
13
Determining Attribution, the Net-toGross Ratio, and Net Energy Savings
Attribution:
What energy savings
are the result of the
program, rather than
other influences?
Southern California Edison
14
Indirect Savings Programs

Attribution is the first question for audit, training,
education, marketing, and market transformation
programs.

Step 1: Who was actually exposed to the program?

Step 2: How many of those exposed to the program
activities changed their behavior in energy-saving ways?

Step 3: Of those who changed, how much of the
changes were due to the program rather than other
influences?

Step 4: Estimate the energy savings of those changes
identified as due to the program.
Southern California Edison
15
Direct Savings Programs

Attribution is usually the second question asked
for rebate and direct install programs.

Step 1: How much energy did participants
save?

Step 2: What fraction of participants’ changes
were due to the program, rather than other
causes?
Southern California Edison
16
The Traditional Direct Savings Approach:
The Net-to-Gross Ratio


= The fraction of the apparent program energy savings
that are truly the result of the program
Example:
Program pays rebates to 100 customers who install high
efficiency windows and claims the energy savings of these 100

Sources of difference for apparent vs. actual:
 Free riders
20 of these customers were already planning to do this without
the rebate offer.

Spillover
10 neighbors also install these windows because of their
installing neighbors’ recommendations, but don’t get the rebate.
Southern California Edison
17
Measurement Methods for the Net-toGross Ratio (and their Key Difficulties)

Self-Report Survey


Regression Analysis Comparison of Participants
and Non-Participants


Minimize bias towards self-credit for decisions
Identify a comparable group of non-participants
Market Baseline

Find data for proportion of efficient sales before
and after program, for the program region and a
comparison region
Southern California Edison
18
When the Traditional Net-to-Gross
Approach Doesn’t Work



When one or more related programs have long-term
effects on a market, estimating free riders and even
some short-term forms of spillover misses the major,
long-term sources of savings due to the program(s).
Then, we need to monitor how the market itself
changes over time and try to determine how much of
the change is due to the programs.
The next presentation will discuss the measurement
of programs with these types of market
transformation effects.
Southern California Edison
19
Attribution Estimation is Difficult and
Imprecise. Can’t We Just Skip It? No!

We can’t just pretend a substantial effect doesn’t
exist, because it’s hard to measure.

We must pay attention to how much difference
the program makes, both for:


Accountability to ratepayers; and
Picking the right measures and the right programs
to maximize cost-effectiveness.
Southern California Edison
20
References and Website Locations (1)
Basic National Sources for Program Evaluation Methods
The Model Energy Efficiency Program Impact Evaluation Guide, developed by US EPA
and the National Action Plan for Energy Efficiency,
http://www.epa.gov/cleanenergy/documents/evaluation_guide.pdf
The International Performance Measurement & Verification Protocol, developed by the
Efficiency Valuation Organization, http://www.evo-world.org/
California’s database of deemed energy savings estimates for energy efficiency
measures, the Database for Energy Efficiency Resources (DEER):
http://eega.cpuc.ca.gov/deer/
Southern California Edison
21
References and Website Locations (2)
Basic California Energy Efficiency Analysis Sources
The California Standard Practice Manual (definitions and rules for cost-effectiveness tests)
http://www.calmac.org/events/SPM_9_20_02.pdf
California’s database of deemed energy savings estimates for energy efficiency measures, the
Database for Energy Efficiency Resources (DEER): http://eega.cpuc.ca.gov/deer/
The California Energy Efficiency Evaluation Framework, a comprehensive and in-depth guide to all
aspects of EE program evaluation:
http://www.calmac.org/events/California_Evaluation_Framework_June_2004.pdf
California Energy Efficiency Evaluation Protocols , the California Public Utilities Commission CPUC)
rules on how to conduct EE program evaluations:
http://www.calmac.org/events/EvaluatorsProtocols_Final_AdoptedviaRuling_06-19-2006.pdf
CALMAC Website: http://www.calmac.org/ . Toolkit sections has several CPUC and California EE
evaluation guidance documents. Searchable Database section has over 600 California energy
efficiency studies, including impact evaluations, process evaluations, market analysis studies. All
of the California utilities’ EE evaluation studies are posted here.
Southern California Edison
22