Quantitative On-Site Evaluation
of Energy Efficiency Programs
Jackie Berger
August 16, 2010
Motivation
Quotes from summary of on-site visit:
“The core finding was that the auditor’s and
crew’s engagement with the client and with
one another increased as a function of their
familiarity with their tools and their tasks.
To some degree there was more
engagement within the crew when the
logistical challenges were greater, causing
crew members to turn to one another to
work out solutions.”
2
Motivation
Quotes from summary of on-site visit:
“The agency routinely uses good-quality silicone
caulk for air sealing interior and exterior cracks,
particularly around windows. While this is
probably satisfactory, an easier-to-apply caulk
with a long lifetime and substantially lower cost is
in widespread use in weatherization and other
retrofit work. It’s an acrylic latex with silicone.
After dispensing (using the pull or push method),
it can be easily worked with a wet sponge, rag, or
finger and has a lifetime rated at 35 to 50 years…
3
Motivation
Quotes from summary of on-site visit:
…It is available in a number of standard colors
including white and clear. (The clear comes out of
the tube white and gradually turns clear during
curing, a process that takes at most a day unless
humidity levels are particularly high.) It cleans
easily with soap and water and costs about $2.25
per tube in case quantities at Home Depot and
other home supply stores...
4
Motivation
Quotes from summary of on-site visit:
…Finally, since installing air sealing measure on the
inside of the conditioned envelope is usually more
effective in limiting convective losses than is
installation on the exterior, use of an acrylic latex
product with silicone usually results in a more
aesthetically appealing result than does pure
silicone. When fully cured, it also takes paint
well (Figures 22 and 23).”
5
Motivation
Figures 22 and 23. All
caulks have their place
and matching the right
one to the right job is
part of the
weatherization craft.
6
Motivation
• Goals for on-site observation and inspections
– Collect the targeted information.
– Summarize findings over all visits.
– Understand how findings relate to the program
overall.
– Make recommendations for program improvement.
7
Session Outline
•
•
•
•
•
•
•
Introduction
Process Evaluation Research
On-Site Evaluation
Sample Selection
Data Collection Instruments
Findings from On-Site Evaluation
Summary
8
INTRODUCTION
9
INTRODUCTION
• Impact evaluation
– How much energy is saved?
– Is the program cost-effective?
– Which measures are cost-effective?
• Process evaluation
– Why succeeding or not meeting goals?
– How can the program be improved?
• Documenting service delivery
– Anecdotal – cannot be generalized.
– Quantitative – how pervasive are the issues.
10
PROCESS EVALUATION
RESEARCH
11
Research Tasks
• Background research purpose
– Understand program policies and procedures.
– Identify potential improvements.
• Background research activities
– Review program documentation.
– Conduct in-depth interviews with program
designers and managers.
– Review program performance statistics.
12
Research Tasks
• Contractor survey purpose
– Understanding of program
– Program implementation issues
– Barriers with program procedures
• Contractor survey activities
–
–
–
–
Review delivery statistics.
Review contractor characteristics.
Select sample.
Conduct survey and analyze data.
13
Research Tasks
• Client survey purpose
–
–
–
–
–
Program experiences
Energy education delivered
Energy education impact
Safety and comfort impact
Satisfaction
14
Research Tasks
• Client survey activities
– Identify key segments.
– Select sample.
– Conduct survey and analyze data.
15
ON-SITE EVALUATION
16
On-Site Importance
• Were protocols followed?
• How well did they work?
• Were important savings opportunities
missed?
17
On-Site Importance
• Protocols – provider compliance
– Program information, diagnostic tests, measure
installation, client education.
• Protocols – applicability
– How well protocols work, consistent
application.
• Use of equipment
– Tools available, adequately maintained and
used correctly.
18
On-Site Importance
• Provider adaptability
– Protocols properly adapted to individual
circumstances.
• Comprehensiveness
– Extent to which cost-effective procedures are
addressed, potential modification of procedures.
• Quality of Work
– Conformance to program standards, respect for
client home, safety.
19
On-Site Importance
• Client Education
– Focus on high potential areas, ability to
motivate clients.
• Client Interaction
– Explanation of program and work, use of client
feedback as information input.
20
SAMPLE SELECTION
21
Sample Selection
Challenges
• Limited sample size.
• Many contractor/job/home/client
characteristics to consider.
• Difficult to draw inferences about program
implementation.
22
Sample Selection
Methods
• Probability sampling
–
–
–
–
Random selection techniques are used.
Each job has a known probability of selection.
Results can be weighted to represent the program.
Confidence intervals can be developed.
• Purposive sampling
– Random selection techniques are not used.
– But can allow for assessment of program
performance.
23
Sample Selection
Utility Program Evaluation
• 5 service delivery areas, as specified by utility.
• Baseload jobs – customers without electric
heat or hot water.
• Full cost jobs – customers with installed
electric heat and >3,600 kwh seasonal heating
or cooling usage.
• 16 contractors provide services.
24
Sample Selection
Utility Program Evaluation
• One dominant contractor in each service area
for each job type.
• Budget only allowed for
observation/inspection of one contractor in
each service area for each job type.
• Dominant contractor for each area was
chosen.
25
Sample Selection
Utility Program Evaluation
Previous Year Service Delivery
Region Contractor
1
1
1
2
2
2
2
2
A
B
C
D
E
F
G
H
# Base
load
# Full
Cost
Region Contractor
6
0
1
81
36
44
53
28
272
23
107
115
58
102
0
46
3
3/4
3/4
4
4
4
5
5
I
J
K
L
M
N
O
P
# Base
load
# Full
Cost
135
46
153
38
2
63
4
57
102
128
1
27
106
0
412
0
26
Sample Selection
Utility Program Evaluation
• Selected contractors represent 60 percent of
Baseload jobs and 68 percent of Full Cost
jobs.
• Contractors use only 1-3 auditors.
• Observed auditors do 30 to 40 percent of the
audit work.
• Findings represent significant percentage of
job experiences.
27
Sample Selection
National WAP Evaluation
• Select 20 local agencies in 20 different states
Climate
Region
Agency
Size
Baseload
Measures
QA
Stratification Factors
Crews or
Urban/Rural/Suburban
Contractors
Client
Community Action
Education
Agency /Other Type
DOE Funding/ Substantial
Training
Other Resources
Dominant
Computerized
Fuel
Audit/Priority List
28
PROCEDURES AND FORMS
29
Procedures and Forms
Development
• Understand
– Program goals
– Program design and implementation
– Roles/responsibilities of providers
• Source of problems
– Protocols
– Division of responsibilities
– Implementation
30
Procedures
Utility Program Evaluation
• Baseload Observations, Full Cost
Observations, Full Cost Inspections
– Procedures: how to conduct visit, forms to
complete, visit write-up, forms to collect from
contractor.
– Data Collection Form: home, electric usage,
services provided.
– Potential Big Users: lights and appliances that
could lead to high baseload usage.
31
Baseload Procedures
Utility Program Evaluation
1. During visit
– Record auditor work and customer interactions.
– Every 10 minutes record time and auditor actions.
– Record if auditor addresses the Potential Big Users.
2. During/after visit
– Complete Baseload Observation Form.
– Complete Potential Big Users Form.
3. Immediately following visit
– Debrief customer and auditor.
32
Baseload Procedures
Utility Program Evaluation
4. Evening or next day
– Complete customer debriefing.
5. Narrative
– Chronologically document audit.
– Estimate length of key parts of audit.
– Assess interaction between auditor and client.
33
Baseload Procedures
Utility Program Evaluation
6. Forms – obtain copies of all forms used
during the visit.
–
–
–
–
–
–
–
Customer usage history
Program application
Core assessment form
Refrigerator data form
Water heater checklist
Customer profile
“Your Electric Bill” form
34
Forms
Utility Program Evaluation
• Selected parts of baseload observation form
VISIT INTRODUCTION
1. Did the customer expect the visit?
YES NO
2. Did the auditor describe the program to the customer?
YES NO
3. Did the auditor review and explain the electric bill?
YES NO
4. Did the auditor discuss whether there were any health
and safety issues?
YES NO
5. Did the auditor discuss whether there were any
comfort issues?
6. Did the auditor discuss whether there were any
problems with energy usage?
YES NO
YES NO
35
Forms
Utility Program Evaluation
• Selected parts of baseload observation form
PRIORITY LIST – AIR CONDITIONERS
1. Did the auditor inspect all window air conditioners?
YES NO
2. Did the auditor do sealing around the air conditioner?
YES NO
3. Did the auditor talk about seasonal storage?
YES NO
4. Did the auditor determine that the air conditioner had
EER of <6 or was in poor condition?
YES NO
5. If yes, did the auditor offer to replace air conditioner?
YES NO
6. If yes, did customer accept AC replacement?
YES NO
7. If no, why did the customer refuse
36
Forms
Utility Program Evaluation
• Selected parts of full cost inspection form
THERMOSTATS (FILLED OUT FOR EACH ONE)
Location
Program Temp
System controlled
Day Heat
Programmable
Night Heat
Condition
Accuracy
Day Cool
Night Cool
Exceptional (Ex): No improvements identified
Good (Gd): Minor improvements possible
Satisfactory (Sat): Some improvements recommended
Fair (Fr): Extensive improvements necessary
Poor (Pr): Substandard in all respects
Same as
recorded
Effort_________
Quality _______ Appropriateness ___________37
Procedures
WAP Evaluation
• Observe audit, measure installation, and final
inspection.
• Observe job from start to finish in one home.
• Conceptual home – observe all aspects.
38
Challenges
WAP Evaluation
• Observe up to 480 homes from 20 providers
in 20 different states.
• WAP implemented differently in every state.
– Audit procedures
– Eligible measures
– Type of education provided
• Develop general forms and procedures that
collect detailed quantitative data.
• Assess work based on whose standards?
39
Forms
WAP Evaluation
•
•
•
•
•
•
Audit observation form
Audit write-up assessment form
Measure installation observation form
Final inspection observation form
Client debriefing
Weatherization staff/contractor debriefing
40
Forms - WAP Evaluation
• Selected parts of audit observation form
Combustion
Safety Tests
Test done?
In agency
Should have Performed correctly to
standards? been done? obtain needed info?
Ambient CO
in CAZ
Y
N
Y
N
Y
N
Y
N
Ambient CO
outside CAZ
Y
N
Y
N
Y
N
Y
N
Gas/propane/
fuel oil leaks
Y
N
Y
N
Y
N
Y
N
Spillage
testing
Y
N
Y
N
Y
N
Y
N
Heating
system CO
Y
N
Y
N
Y
N
Y
N
Water heating
CO
Y
N
Y
N
Y
N
Y
N
41
Forms - WAP Evaluation
• Selected parts of audit observation form
Combustion
Safety Tests
Test done?
In agency
Should have Performed correctly to
standards? been done? obtain needed info?
Heating
system draft
Y
N
Y
N
Y
N
Y
N
Water heating
draft
Y
N
Y
N
Y
N
Y
N
CAZ –
combustion
air
Y
N
Y
N
Y
N
Y
N
Range CO
Y
N
Y
N
Y
N
Y
N
Stove CO
Y
N
Y
N
Y
N
Y
N
Addressed
immediate
safety issues
Y
N
NA
42
Forms - WAP Evaluation
• Selected parts of measure installation observation form
Air Sealing
Mark NA if not in work scope and skip section
Mark NA if not done this visit and skip section
Blower door used to guide air sealing
Sealing at top and bottom of envelope prioritized
In Work
Scope
Y
N
NA
Y
N
NA
Done This
Visit
Rate Quality
Attic floor penetration sealed
Knee walls sealed
Y
N
Y
N
1
2
3
4
Y
N
Y
N
1
2
3
4 5
Second floor rim joist sealed
Y
N
Y
N
1
2
3
4
5
5
43
Forms - WAP Evaluation
• Selected parts of measure installation observation form
Air Sealing
In Work Done This
Scope
Visit
Basement penetrations sealed
To outside
To conditioned space
Crawlspace penetrations sealed
To outside
To conditioned space
Garage penetrations to conditioned
space sealed
Rate
Quality
Y
N
Y
N 12345
Y
N
Y
N 12345
Y
Y
N Y
N Y
N 12345
N 12345
Y
N Y
N 12345
44
Forms - WAP Evaluation
• Selected parts of measure installation observation form
Air Sealing
All major opportunities sealed
Y
N
Lowest
Overall quality of air sealing work
Overall cleanliness of air sealing work
1
1
Highest
2
2
3
3
4
4
5
5
Comments
5 – Exceptional: No possible improvements identified
4 – Good: Minor improvements possible
3 – Satisfactory: Some improvements recommended
2 – Fair: Extensive improvements necessary
1 – Poor: Substandard in all respects
45
FINDINGS FROM
ON-SITE EVALUATION
46
Utility Program
Visit Introduction
# of Observations
Yes
No
Customer expected visit
Explained Program
Reviewed electric usage
Discussed health and safety issues
Discussed comfort issues
Discussed any problems with
energy usage
10
6
4
4
3
0
4
6
6
7
3
7
47
Utility Program
Baseload Visit Summary
Length of Visit
Minimum
Maximum
Visit length
(minutes)
70
180
Average
119
Rating
Excellent Good Fair Not Acceptable
Overall rating
(#of jobs)
4
2
2
2
48
Utility Program
Baseload Recommendations
• Review requirements and expectations with
contractors.
• Reinforce importance of the walkthrough
for baseload jobs with all contractors.
• Provide additional training to contractors on
the importance of 2-for-1 swaps in
refrigerator replacement, and train
contractors to work with customers to
obtain their acceptance of this measure.
49
Utility Program
Baseload Recommendations
• Review CFL replacement procedures with
contractors.
• Review customer education requirements
with contractors.
• Observe baseload service delivery to ensure
that contractors meet program standards for
service delivery.
50
Utility Program
Inspections Overview
# of Observations
Very Good Good Fair
1
1
2
1
3
2
2
Data collection accuracy
Measure selection
Measure appropriateness
Exceptional Good Satisfactory
Effort
Quality
Appropriateness
Overall Rating
3
1
3
1
1
2
1
3
1
51
Utility Program
Full Cost Recommendations
• Develop one set of required forms.
• Provide instructions on the back of each
form.
• Require all applicable diagnostic tests at the
audit visit.
52
SUMMARY
53
Recommendations
• On-site work is an important component of
technical process evaluations.
• Select sample that accurately represents
program.
• Design tools to capture what is observed in
the field.
• Train staff to systematically record data and
information.
• Analyze findings in conjunction with other
research.
54