ENERGY
HVAC Impact Evaluation – WO32
Summary of the Report Draft For Comment
WO32 Evaluation Team
September 8, 2014
1
DNV GL © 2014
August 2014
SAFER, SMARTER, GREENER
Webinar Agenda and Schedule
Webinar Agenda for WO32
 Commercial Quality Maintenance Impacts
 Residential Quality Installation Impacts
 Upstream Incentive Program Net to Gross
 Preliminary Laboratory Results
 Schedule
 8/27/2014
Energy Division Released WO 32 report to WHPA
 8/29/2014
IOU / CEC Comments due; Public Release (not modified to reflect
CEC/IOU comments)
 9/8/2014
Public Meeting and Webinar
 9/12/2014
Public comments due on the Public Documents Area (PDA)
 9/16/2014
Revised Report
2
DNV GL © 2014
August 2014
Project Timeline
3
Milestone
Timeframe
WO32 Research Plan
April 2012
Weekly CQM calls with IOUs and implementers
to coordinate pre-post monitoring and site
observations (Varies by IOU)
July 2012 –
December 2013
QI Field Data Collection
August – Dec. 2013
Memo and presentations on CQM interim
findings
August 2013
Presentations on QI interim findings
March 2014
WO32 Draft Report
August 2014
Comments and Webinar
September 2014
2013-14 Research Plans
September - October
2014
DNV GL © 2014
August 2014
Commercial Quality Maintenance
4
DNV GL © 2014
August 2014
Evaluation of Statewide CQM Measures
 The statewide programs were the focus of the evaluation research plan.
 The statewide programs generally assumed the same average energy
savings per ton of cooling independent of which measures were
installed.
 Evaluators coordinated with the statewide programs to sample and
monitor units pre- and post-maintenance. This effort presented several
logistical and technical challenges.
– Recruitment to get pre-conditions
– Maintenance observation(s)
– Sensor placement and reliability
– Unit replacement
 New - The sample size and representativeness of the sample have been
questioned by the IOUs
– Additional direct observations were made of top contractors Appendix information needs to be pulled into main report
5
DNV GL © 2014
August 2014
– Additional information on sampling and discussion being added
Pre-Post Metering
6
DNV GL © 2014
August 2014
Results
 The evaluation team found that the realization rate was zero for the
package of measures approach in the statewide program based on prepost monitoring.
 To confirm the results, the evaluation team performed bin analysis on
the power consumption data for the same units included in the primary
analysis and also found energy savings were negative.
 New - Since release of the draft report to IOUs and WHPA, additional
analysis is being conducted on both unit performance estimation and
power consumption analysis
 New – SDG&E ex ante savings for CQM need revision as records in
tracking should total zero. Meaning SDG&E did not claim savings and we
did not measure savings for that sample (Savings zero, RR – 100%)
7
DNV GL © 2014
August 2014
Site observations
 Conducted by Master Technicians under contract to DNV-GL
 Combination of pre/post and post-only
 Conducted across all pre/post metering sites in SW program
 Also conducted across third party programs
 Focus on contractors representing 75% of the savings claims
 Maintenance activities observed and cataloged
 Duplicate measurements taken using research grade instrumentation
 Maintenance activities and fault detection and diagnostic (FDD) protocols
observed compared to program requirements
 Maintenance activities FDD protocols also compared manufacturer’s
recommended service procedures specific to the unit observed
8
DNV GL © 2014
August 2014
Verification of Individual Measures – Economizer Repair
 Field observation found 25% of economizers worked after repair
 For economizers, the majority with reported repairs that we inspected
were not found to be performing their primary function of changing
damper position in response to temperature. In some cases, the
technicians opened the position of the dampers, but did not complete
working changes to sensors and linkages
 New – Additional information by IOU program based on Appendix being
compiled
9
DNV GL © 2014
August 2014
Verification of Individual Measures – Refrigerant Charge
 Field observation found 60% of units met program refrigerant charge
criteria.
 The evaluation found problems with the recommended refrigerant charge
and airflow (RCA) fault detection procedures and with incorrect
implementation of the procedure; this led to a large number of
undiagnosed faults.
 New – Additional information by IOU program based on Appendix being
compiled
 The RCA approach could not be guaranteed to determine correct
refrigerant charge faults under ideal conditions (Braun and Yuill) and was
more problematic when applied to commercial field conditions.
Braun, Jim, David Yuill, and Howard Cheung. A Method for Evaluating Diagnostic Protocols for Packaged Air Conditioning
Equipment. Purdue University and the California Energy Commission, 2012.
Braun, Jim, and David Yuill. Evaluation of the Effectiveness of Currently Utilized Diagnostic Protocols. Purdue University. 2014.
 Similar results obtained in lab
10
DNV GL © 2014
August 2014
Conclusions
 The current program, and commercial unitary HVAC maintenance
measures in general, face various inherent challenges, not addressed by
current workpapers.
 The less-than-expected savings and installation rates from both the
2006-08 evaluation and this evaluation stem from trying to address
individual faults or subsets of all faults without accounting for underlying
system issues.
 FDD protocols in commercial units especially difficult to implement due
to OA intakes and subsequent difficulty in getting an accurate mixed air
temperature measurement
11
DNV GL © 2014
August 2014
Conclusions
 Multiple faults continue to be a source of incorrect or misleading results
from program FDD protocols.
 Recovery and weigh-out of refrigerant charge implemented late this
cycle to determine true state of charge; more research planned.
 Some issues, such as cabinet and economizer damper leakage were
likely unknown until recent laboratory research.
 A comprehensive approach includes some components of maintenance,
but also requires that retro-commissioning be completed prior to
ongoing maintenance.
 Program changes since 2010-12 have taken place and additional
changes are being considered
12
DNV GL © 2014
August 2014
Recommendations
 To address these findings, we recommend that IOU programs coordinate to
clearly define the meaning of maintenance and retro-commissioning as related to
measures and programs. For example:
– Sealing economizer and unit leakage may be within the scope of maintenance,
but is not part of current programs
– The repair for basic functionality requires new parts beyond the scope of
maintenance
– Retro-commissioning involves setting integrated controls based on measuring
unit operation
 We recommend piloting approaches that have demonstrated measurable savings.
Measure recommendations relative to maintenance and repair include:
– Seal Unit Leaks.
– Reconsider Diagnostic-Based Refrigerant Charge Measures.
– Use Manufacturer Maintenance and Diagnostic Protocols.
– Develop Criteria to Replace Rather than Repair Economizers.
13
DNV GL © 2014
August 2014
Residential Quality Installation
14
DNV GL © 2014
August 2014
Residential Quality Installation
 The team conducted site visits at 50 program participants and 50 nonparticipant sites. Site staff followed identical data-collection protocols at
participant and non-participant sites.
 Data collection for all tested units included spot measurement of unit
airflow and duct leakage tests. Building envelope characteristics
informed load-calculation models for each site.
 The evaluation team calculated energy savings for participants relative
to non-participants based on system sizing, correct airflow, and reduced
duct leakage.
 The team modified the DEER single-family residential prototype
workpaper assumptions with measured values for the non-participants
as the baseline and program participants as the efficient case.
15
DNV GL © 2014
August 2014
Residential Quality Installation
 The evaluation centers on the 2010-12 workpaper assumptions
– The appropriate DEER calculation method was not used in the ex ante
or ex post
– Fan watt draw data was collected but was not incorporated into the
savings estimates
– Charge diagnostics collected, but only when weather permitted since it
was not used in the analysis
 Clarification – Sizing ratio in the report is installed size relative to
calculated size, we do not report sensible heat ratio (SHR)
 The sizing calculations use inputs based on data collected, not Manual J
defaults – report revisions are needed to stipulate these are not
“assumptions” when they are “averages”
 Window shading and overhangs observed during onsites were included in
the sizing calculations
16
DNV GL © 2014
August 2014
Quality Installation Findings
Statistic
Total System
Duct Leakage
System Airflow
(CFM per ton)
System
Oversizing
17
DNV GL © 2014
August 2014
NonParticipant
Field
Observations
(mean)
Baseline
Assumptions
Participant
Field
Observations
(mean)
QI Efficient
Case
Assumptions
17%
24%
12%
12%
300
350
338
400
13%
20%
10%
0%
Duct Leakage
18
DNV GL © 2014
August 2014
Airflow
19
DNV GL © 2014
August 2014
Sizing
20
DNV GL © 2014
August 2014
Conclusions
 A note on our realization rate calculations: The team could not separate
out free-ridership from presumed non-compliance in the gross
(workpaper) baseline. Since the workpaper assumed non-compliance, it
essentially presumes a standard practice baseline, not a code baseline.
 The evaluation team learned that installations in the programs exceeded
Title 24 code on average and that installations outside the program do
not perform as poorly as assumed in program workpapers.
 The evaluation findings showed that there was a continuous spectrum of
installation efficiencies from good to bad outside the QI program. The
team also discovered that the program workpaper does not fully capture
all aspects of program participants.
21
DNV GL © 2014
August 2014
Recommendations
 Use a difference of differences approach for workpapers and evaluation to allow
separate estimates of gross and net savings. This requires re-defining program
workpaper assumptions where the baseline is code.
 Revise workpapers to consider that the baseline or net savings as defined above
vary by code jurisdiction correlated to CZ and unit cost correlated to SEER level.
 Expand the non-participant sample to support QI programs and “to code” pilots.
 Ultimately the programs can influence savings for actions that exceed Title 24
requirements if they exist. A few areas where Title 24 can be exceeded:
– Explore downsizing to reduce peak demand.
– Explore duct sealing to reach a lower target leakage, such as the 6% threshold.
– Explore air handlers/furnaces, filters, and duct modifications that reduce
pressure drop and improve fan system efficiency.
– Explore whether all ANSI/ACCA QI elements in programs impact energy use
and align with workpapers.
22
DNV GL © 2014
August 2014
Commercial Upstream Equipment
23
DNV GL © 2014
August 2014
Commercial Upstream Net to Gross
 The evaluation efforts for the upstream component of the study focused
on the level of free-ridership in the Upstream HVAC program. Gross ex
ante savings claims are based on DEER estimates and were not
evaluated.
 A participant self-report method was used for the free-ridership/net-togross (NTG) work, and the analysis was based on in-depth telephone
interviews with 19 out of 22 participating HVAC distributors that were
conducted by DNV GL in 2013.
 The approach took into account the program’s effect on both the
stocking practices and sales practices of the distributors.
24
DNV GL © 2014
August 2014
Methods
2. Program Influence on Sales
Q3. Did rebate influence Distributor's equipment sales?
Yes
Program credit
5% for cited as a
influences on
sales.
No
Program credit
reduce to 2.5%
from 5% when
not cited initially
as a influence on
sales.
Probe: Confirm rebate was not a
factor in equipment sales.
)No program
Yes
influential
credit for
equipment
sales (-70%)
Not
influential
Q4. Rated influence of rebates'
influence on equipment sales on (0-10)
Program credit for influence on
sales [70%* score (0-10)]
Survey Question
Quetio
Consistency Check: Rebate was rated
highly influential >8 in equipment sales?
Sum of sales score=
Q3 + Q4 = (+/-) 70%
Customer Decision
Scoring Methods
Probe/Consistency Checks
_ _ _ Dash Line = Score Flow Logic
__ Solid Line = Question Flow Logic
3. Sum of Program Influence on Stocking and Sales
Sum of score= [Q1 + Q2 (30%)] + [Q3 + Q4 (70%)] =100%
25
DNV GL © 2014
August 2014
Results
 Overall, the program is achieving a savings-weighted NTG score of 0.80, versus
an un-weighted NTG score of 0.61.
 The final net to gross ratio was applied to gross energy and demand savings for
2010-12 programs.
Stocking
Score
Sales
Score
Overall
Score
Proportion
of
Program
Savings
16.2%
37.5%
53.7%
56.1%
Manufacturer-owned or
franchise distributor
4.1%
21.3%
25.4%
39.2%
Independent manufacturers’
representative
0.0%
0.6%
0.6%
4.7%
20.3%
(30%)
59.4%
(70%)
79.7%
(100%)
100%
Type of Distributor
Independent HVAC equipment
distributor
Total (Maximum Potential
Total)
26
DNV GL © 2014
August 2014
Commercial Upstream Net to Gross
 For upstream distributor incentives, the evaluation team learned that the
provided incentives influence stocking and sales for the highest volume
distributors, but have less influence on lower volume distributors.
 Some distributors supply units directly from manufacturers, and stocking
incentives have no influence there.
 The team identified some opportunities beyond the efficiency incentives:
– The program does not focus on permitting and compliance
– The program does not provide additional incentives for climateappropriate systems.
27
DNV GL © 2014
August 2014
Recommendations
 The program should consider increasing rebate levels for higher
efficiency equipment to encourage more sales in the highest efficiency
tiers.
 Program-provided support on how to market high efficiency equipment
systems may be helpful to distributors, particularly those who are less
successful at selling the concept of high efficiency.
 Providing a reservation system or rebate guarantee would encourage
more participation, and increased high-efficiency sales from distributors
who have long sales cycles or custom build equipment.
28
DNV GL © 2014
August 2014
WO 32 Laboratory Results
29
DNV GL © 2014
August 2014
Laboratory Results
 Testing conducted at AHRI-certified laboratory
 Master technicians oversaw laboratory technicians who tested packaged
HVAC units
 Additional laboratory tests are continuing and include packaged units
from a variety manufacturers.
 The laboratory tests with actual cabinet leakage and functioning
economizers provided critical new findings on the diagnosis of system
faults and the efficiency impacts of repairs.
 Critical findings from laboratory testing with an economizer include:
– All tests show that even optimally adjusted systems with an
economizer perform significantly lower than their rated efficiency.
– The diagnosis and adjustment of charge is difficult, if not impossible,
to achieve in the field since the airflow and economizer affect unit
performance.
30
DNV GL © 2014
August 2014
Affect of Outside Air and Charge
31
DNV GL © 2014
August 2014
Affect of Outside Air and Airflow
32
DNV GL © 2014
August 2014
Laboratory Results
 The following laboratory results are available based on tests of the 7.5ton dual-compressor packaged RTU:
– The test results demonstrate that reducing minimum damper position
can be a reliable measure to improve cooling efficiency
– Understanding the efficiency impacts of faults presents several
challenges. We found that the efficiency effects of outdoor conditions,
indoor conditions and faults exhibit non-linear trends and become
more complex when combined, requiring empirical testing to derive
efficiency impacts.
– Tests with the economizer open from 10% to 30% indicate that loads
and energy consumption increase 5% to 62% compared to closed
dampers that deliver 15% outdoor air through leakage.
33
DNV GL © 2014
August 2014
Laboratory Results
– Outdoor airflow is 15% with closed dampers, 20% with 1-finger open,
23% with 2-fingers open, 30% with 3-fingers open, and 62% with
dampers fully open. Designers, technicians, and program
implementers assume 2% outdoor airflow with closed dampers and
100% outdoor airflow with fully open dampers.
– Program and manufacturer’s FDD protocols could not accurately
diagnose faults induced in the laboratory
– Manufacturer EER ratings were replicated when using AHRI-specified
test conditions
– Imposition of test conditions better representing field conditions
produced much lower unit efficiency
34
DNV GL © 2014
August 2014
Thank you. Questions?
Jarred Metoyer
Jarred.metoyer@dnvgl.com
510-891-0446
www.dnvgl.com
SAFER, SMARTER, GREENER
35
DNV GL © 2014
August 2014