HVAC Road Map Summary
1 Summary of EM&V Activity for HVAC
1.1 HVAC Team Organization
The HVAC Research Roadmap lead is Jarred Metoyer. He is currently DNV KEMA’s lead on all
HVAC-related Work Orders under the CPUC 2010-2012 EM&V prime contract. Mr. Metoyer is
highly knowledgeable regarding Energy Division and IOU policies, programs and EM&V
activities, given his recent experience planning 2010-2012 HVAC impact and market research
and managing the HVAC PCG. He also has previous experiences managing the 2006-2008
HVAC HIM studies, advising on DEER 2009 HVAC updates.
Each project has a DNV KEMA manager assigned to avoid administrative issues, even if the
study is likely to include extensive use of subcontractors. The project managers and key
subcontractors are identified in the organization chart below. There are also cross roadmap
advisors which are listed in particular for HVAC. All of the ED-led Research Roadmaps will
include sampling, analysis, and data management. Additional firms on contract will also be
added to the studies as needed including Cadmus Group and UC Davis.
1.2 Overview of Original Studies from EM&V Road Map Inventory
Impact Evaluation of Commercial and Residential Upstream HVAC Programs
Commercial Upstream HVAC is a major component of the HVAC portfolio and the IOUs were
directed to include upstream incentive pilots for residential sector in their 2013-2014 portfolio,
per Decision D.12.11.014.
Impact evaluation of the Commercial Upstream HVAC incentive was included in the 2010-12
HVAC evaluations, but critical questions still need to be addressed. New commercial HVAC
equipment efficiency standards will make it more difficult to claim savings from Upstream
HVAC programs. The installed efficiency, distribution system efficiency, and loads imposed on
program incentivized HVAC equipment is not well known and is an important area of
uncertainty. Installations after July 2014 will be subject to the new Title 24 code, including
changes for fault diagnostic detection requirements.
Impact Evaluation of Commercial and Residential Quality Installation Programs and “to
Code” HVAC installations Pilot
Both residential and commercial QI programs were to be part of the 2010-12 evaluation, but the
programs were slow to start. Commercial QI pilots have been mentioned but there have been no
savings claims. The CPUC has also directed the IOUs to create a pilot program for “to code”
installations. These programs have the following known uncertainties and research risks:




Savings are highly uncertain, which causes low projected budgets and savings goals for
13-14 program cycle.
Benefits of requirements beyond Title 24 are not well known
There is a need to understand how well HVAC contractors utilize “whole building”
design aspects when sizing and installing units (such as duct sealing, insulation, and
building envelope integrity). The building envelope is not changed by QI but duct sealing
and system sizing are primary aspects; so there is a need to understand coordination
between QI and Energy Upgrade CA.
Need to determine the gross and net first year and lifecycle impacts for kWh, kW and
Therms.
Impact Evaluation of Commercial and Residential Quality Maintenance Programs
The program has the following known uncertainties and research needs:

Uncertainties
o Program participation has been slow to ramp up. The IOUs, however, are
projecting increased participation for both of these programs as they move
forward.
2

o The program design for the 10-12 cycle introduced multiple service actions for
which data is sparse.
o Program design may continue to evolve as 10-12 impact data becomes available.
Impact evaluations for 13-14 will need to take these changes into account.
o Uncertainties in impacts of service actions on equipment efficiency, in-situ v.
manufacturers’ name plate efficiency, distribution system efficiency and building
loads are important cross-cutting issues.
Research Needs
o There is a need to determine the gross and net first year and lifecycle impacts for
kWh, kW and Therms.
o Need to determine the baseline market conditions for small and medium
commercial HVAC units.
 What are the current maintenance procedures for non-participating
contractors
 How do customers view ACCA 180 maintenance?
 What motivates customers and contractors to participate in the CQM
program?
o What are the costs to perform each maintenance measure to the customer and the
contractor? This research should include new measures now incorporated in the
programs, e.g., demand controlled ventilation, notched belts, etc.
Impact Evaluation of Deemed HVAC Measures
A gap exists for measures that are not addressed in QI, QM, and upstream programs. Examples
include ductless mini splits, HVAC controls, VFDs for HVAC, and boilers and furnaces. Key
research questions include:



Within this group of measures, which measures and engineering parameters have the
highest priority?
What field data can inform DEER updates and workpaper revisions?
What field data are required to improve ex-post impact estimates?
Potential EM&V methods should include:


Scope will depend on the magnitude of the savings as well as the uncertainty in the
DEER or workpaper estimates.
Field data collection should focus on obtaining parameter level data to reduce savings
uncertainty.
3
Laboratory Testing
Lab testing is an ongoing activity spanning multiple program cycles. Lab testing is an integral
part of the overall HVAC Program impact evaluation as it provides insights into the impact of
installation techniques and service actions that are not possible in a field only study. Potential
EM&V methods should include:



Any testing identified in the 10-12 plans or during the 10-12 cycle that was not
completed will be addressed during the 13-14 cycle.
Additional tests will be identified based on the needs of the DEER updates, non-DEER
workpaper reviews and ex-post impact evaluation.
Tests will be conducted in qualified laboratories using AHRI, ASHRAE and/or ANSI
standards.
Market Assessment: Identifying Baselines and Barriers for Existing HVAC Conditions,
Building Permit and Title 24 Compliance
Objectives for these assessments are as follows:







Investigate magnitude of the failure of contractors to obtain required building permits and
to comply with building energy efficiency standards (Title 24, Part 6).and determine why
contractors are not pulling permits
Identify baselines for CA industry standard practice in obtaining building permits and
achieving 2013 Title 24 compliance for residential and commercial HVAC equipment
installations. Anecdotal information suggests that both compliance baselines are low.
Investigate permitting and compliance separately including whether non-permitted
installs are compliant and also if permitted installs are compliant.
Estimate a baseline for “existing conditions” found in the population of HVAC units
across CA using site assessments performed during the 2010-12 cycle and additional nonparticipants in 2013-14.
Coordinate with and leverage the results of similar Codes & Standards EM&V (past
cycles and 2013-14) research effort to identify market barriers to obtaining HVAC
installation permits and Title 24 compliance. Determine the variance of permit cost and
the scope across jobs.
Examine the awareness of contractors on the code requirements and inspection processes.
Examine the awareness of building code officials and HERS rater on the code
requirements and inspection process. Identify knowledge gaps where training could help
building code officials and HERS raters to better enforce HVAC requirements.
Improve the market intervention strategies for HVAC market actors identified in the
“Road to ZNE” study to pursue the state’s Big Bold Initiatives and ZNE goals for HVAC.
4
1.3 Proposed Changes to the Roadmap not redlined above
The current studies are not being changed. Changes to update the roadmap that do not entail
major reorganization include creating multiple projects within study areas and providing some
updates to the ED studies since the August 2013 HVAC roadmap.
Multiple projects within study areas
Multiple projects have been created in the large study areas to accommodate the high priority
ESPI studies and better manage the large study areas in the HVAC roadmap.

Mini-split systems, which include variable refrigerant flow systems, are on the ESPI list
and will have a high priority evaluability assessment prior to conducting the planned
upstream program impact evaluation.

Quality maintenance is on the ESPI list and will have a high priority field and laboratory
component. The 2010-12 evaluation has shown synergies in conducting field
observations and laboratory testing.
Specific clarifications to August 2013 HVAC roadmap

Work Order 32: 2010-12 Residential and Small Commercial HVAC Impact Evaluation
o Completion Date Q1 2014 changed from Q4 2013

Both residential and commercial QI were to be components of the 2010-2012 evaluation
o Only residential QI included in WO32 due to slow start for commercial and no
commercial QI workpaper in 2010-12. Not clear that there is an approved WP for
CQI in 2013-14

Both Residential and Commercial QM programs are a component of the 2010-12
evaluation
o Only commercial QM included due to slow start for residential

Note uncertainty in in-situ efficiency of equipment and systems and uncertainty in loads
is common across HVAC measures. HVAC end-use calibration data lacking for
commercial buildings.
2 Need and Application of Results
According to D.13-09-023 which established the Energy Savings Performance Index (ESPI); the
ED Staff is instructed to create a list of measures that would require an ex post evaluation of
energy savings to support an earnings claim should the IOUs include the measure in their
5
programs/portfolio. These measures are selected for ex post measurement due to several factors,
including the lack of recent program evaluation to inform/update ex ante estimates and for
measures included in relatively new programs where performance is uncertain. In addition to the
ESPI list, the ED Staff also conducted an analysis (P4) to identify measures that carry probability
factors that may pose a risk to programs and/or the portfolio reaching their goals; thereby placing
higher priority towards validating any outdated input variables for these measures that are
needed for calculating energy savings.
Commission staff issued a Final List of ESPI measures memo on December 6, 2013. Mini-splits
and quality maintenance are on the ESPI list while other upstream HVAC measures are on the P4
list.
We do not have sufficient data about HVAC market behavior to make well informed decisions
regarding long term market transformation efforts. The PCG is interested to know how current
and future HVAC research can best guide programs to target and fix the worst performing
HVAC units. The savings estimates for the HVAC sector remain an area of considerable
uncertainty. Questions exist about the true baseline of HVAC efficiency in CA:







What is the true as-found condition of HVAC systems regarding their design and
installation, as well as changes since installation (charge level, non-condensables, etc.)
How do HVAC units perform in-situ relative to the manufacturers’ specifications?
What are the building heating and cooling requirements; and distribution system
efficiencies that determine the loads imposed on the HVAC systems?
What are the energy improvements associated with bringing HVAC systems to baseline?
What are the baseline maintenance, installation, sales and distribution practices?
What is the baseline level of technician skill and diligence?
What is the role of building codes since not all HVAC work triggers code compliance?
6
3 General Approach, Issues, and List of Studies and Activities
Across studies there are a few primary types of data collection that can be used to conduct
impact analysis. Field observations, field and laboratory measurements of HVAC unit
performance and surveys are the primary methods. The cost of collecting field and laboratory
measurements is a key constraint on the studies in terms of sample design. For the market
assessment and net-to-gross components, market actor interviews and market data are also
primary data that can be collected.
For each study the sources of uncertainty contributing to the ESPI and P4 assessments was
reviewed and the priority types of data collection were defined. A large table was created to
show the timing of work orders alongside the uncertain parameters. A second table was also
created to show the specific issues or challenges with the approach in the original roadmap and a
proposed approach to overcome the issue or challenge. These tables also form the basis for the
work order proposals for the high priority and first year activities.
In all cases the Year 2 activities are roughly planned and the research activities may be
completely changed based on the WO32 report and Year 1 progress.
Studies
Upstream HVAC
Program Impact
Evaluation
Measure
Mini-splits
ESPI - P4
ESPI
Uncertainty
UES
IRate
Baseline
and
Measure
performance
NTG
Medium High
Proposed Work Orders and Timing
High Priority: Evaluability Assessment
Year 1: Surveys to determine if Mini-splits replace traditional
systems
Year 2: Measure performance of installed mini-splits
Year 1: Phone and onsite surveys to determine install rate and
applications of installed units;
Upstream HVAC
Program Impact
Evaluation
Package and
split systems
Impact Evaluation of
QI and To Code
Program
Quality
Installation,
To Code
Installation
Impact Evaluation of
QM Program
Impact Evaluation of
Deemed HVAC
Measures
P4
Uncertainty
in
applications
N/A
Baseline
and
Measure
performance
Quality
ESPI
Maintenance
Multiple
deemed
HVAC
measures
N/A
Major
Measure
performance
issues
Varies
High
Low
High
Medium
Year 1: Separate WO: Pilot remote metering and correlation to
AMI Coordinate with WB interval data study conducted by
IOUs, where meters are installed on individual HVAC units to
validate WB disaggregation and energy savings algorithms.
Year 2: Either more verifications or begin monitoring starting
with highest savings climate zone (1) and building type (1) per
IOU
Year 1: Participant and Non participant sample design with
project review followed by monitoring and testing,
Year 2: Small WO or add-on for direct observations if needed
High
High Priority: Continue Lab work and site visits with interim
reports on findings
Year 1: WO for ex-post sample followed by case studies of
Medium achieved and remaining energy savings potential.
Year 2: WO for pre-post monitoring with direct observation…
coordination challenges
Varies
Varies
Year 1: Uncertainty analysis and focused study. Perhaps on gas
measures not studied in 2010-12 and not mentioned in 2013-14
(e.g. residential furnaces)
HVAC Laboratory
Testing
Permit/Compliance
Market Assessment
Laboratory
testing
Obtain
permit data
for
participants
and nonparticipants
installations
N/A
N/A
Informs QM
and QI
Informs QI
Baseline, To
Code
Baseline
Year 1: WO to test 2 large and 2 small units already at Intertek
N/A
N/A
Year 2: Set aside contingency to plan and conduct additional lab
tests based on new field findings
Year 1: Permit rate study, market actor interviews; Await IOU
customer surveys before fielding non-participants site visits
Permit
Rate
9
NA
Year 2: Compliance study, same onsite procedure as QI and To
Code
Studies
Upstream HVAC
Program Impact
Evaluation
Upstream HVAC
Program Impact
Evaluation
Impact Evaluation
of QI and To Code
Program
Measure
Mini-splits
Package and
split systems
Quality
Installation,
To Code
Installation
Original Roadmap Approach
Issues and Considerations
Proposed Approach
Same as upstream, one of 3-4 types
Unknown if these systems are added
as new cooling capacity in
previously un-cooled space, installed
in lieu of standard efficiency minisplits or are replacing traditional
systems; Many cooling performance
issues documented by Taylor
Engineering, most data from field is
for these replacing resistance heaters
in the Northwest
Evaluability assessment: Include data
collection to first determine baseline and
applications through surveys. Several
metering challenges can be outlined
through literature review, with metering
protocols established through a
pilot/case study.
Field monitoring will be done for
both participant and non-participant
units in the residential and
commercial sectors across a variety
of climate zones and building types
Consider participant non-participant,
3 HVAC types, 6 key climate zones
and at least 5 building types means
180 strata for sampling. Therefore
even minimal samples will create
costs far exceeding the budget for
even simple metering of unit power.
Sampling detailed performance
would further limit the sample by
increasing cost per point.
Either conduct a follow-up to 10-12
study or a new study depending on
participation levels and program
design changes.
2010-12 Study sample doesn’t offer
for much follow-up beyond the
PG&E Compliance follow-up; Field
measurements without monitoring
may be required for the allocated
budget or combine this with the
Upstream effort
10
Conduct a metering pilot to determine
whether AMI data can be leveraged to
fill all the strata needed based on
comparison to direct end use meters
with remote communications.
A phone survey and followup onsite
effort to verify installations and
characterize installed application may be
able to achieve large samples.
Focus on participants in the two 201314 programs (QI and To Code) since the
budget is $500k. The larger market
assessment should conduct the nonparticipant assessments since the budget
is $1.25M. Non-participants under 2008
Title 24 would include installs from
2009 to June 2014.
Conduct field monitoring of
participants (both QI and “To
Code”) as well as sites where a
permit was not pulled for a variety
of climate zones and building types.
Ex-post observations followed by
debrief to program and technician. Subsample may include case study which
may be presented to the technicians as a
training opportunity and to estimate
achieved and remaining energy savings
potential.
Impact Evaluation
of QM Program
Evaluation should focus on
measures/treatments that have the
highest uncertainty in projected
savings for both the residential and
commercial sectors. Pre/post
Quality
monitoring and/or billing analysis
Maintenance of participating units.
Several issues with the pre-post
design as documented by WO32, but
pre-post is still the preferred ex post
method
Without pre-measurement, installation
rate is primary parameter measured,
some UES assumptions in workpapers
can be modified with measurements
after measures are installed, but does not
inform measure baseline without premeasurement.
Work on pre-post coordination, but not
on the annual reporting timeline. This
means “Pre” in 2014 and “Post” in
2015. Otherwise the program measures
may need to be all installed at the same
time which is not part of program
design, but is needed to streamline ex
post M&V.
Calibrated simulation modeling
based on monitored data. Estimate
savings by individual service
action.
11
Billing analysis will capture “noise”
based on numerous observed issues
and program allows technician
discretion increasing variability
Impact Evaluation
of Deemed HVAC
Measures
Multiple
deemed
HVAC
measures
Scope will depend on the
magnitude of the savings as well as
the uncertainty in the DEER or
Budget will not allow much field
workpaper estimates. Field data
work unless focused to a single
collection should focus on obtaining measure
parameter level data to reduce
savings uncertainty.
Any testing identified in the 10-12
plans or during the 10-12 cycle that
is not included in the current scope
will be addressed during the 13-14
cycle.
HVAC Laboratory
Testing
Laboratory
testing
Tests will be conducting in
qualified laboratories using AHRI,
ASHRAE and/or ANSI standards
Obtain total permit counts and
permit/compliance costs.
Permit/Compliance
Market Assessment
Obtain
permit data
for
participants
and nonparticipants
installations
• Conduct site surveys at a sample
of QI/QM program participants and
non-participants
(permitted/unpermitted, if
available) to obtain T24 compliance
baseline estimate and collect data
related to HVAC Market
Transformation Indicators.
12
Perhaps focus on gas measures not
studied in 2010-12 and not mentioned
in 2013-14 (e.g. residential furnaces)
given all other efforts focusing on
cooling
No issues with scope to do tests not
completed in 10-12.
Two Phases: Conduct lab work not
completed in 10-12 and reserve funds to
Do need to conduct tests at off AHRI
test based on new field findings
conditions which includes
economizers and faults. Follow
AHRAE/ANSI instrumentation
protocols and applicable. Test
conditions per test plan.
Using QM sites for permitting has
issues since they are existing
systems.
Permitting rate only requires counts
of non-participant units with permits,
while compliance requires more
detailed file review and onsite
inspections. The performance of
systems that are close to compliance
is also different than those with duct
leakage, sizing, and airflow far from
Site surveys for non-participants only as
they will serve as a baseline for both QI
and To Code. QM participants should
be excluded.
Market Actor interviews should include:
HERS Raters, Code Officials (part of
permit data collection), CSLB,
Contractors, Distributors, CEC Code
setting staff.
• Study how effectively HVAC
replacements, including emergency
replacements, are complying with
permitting obligations under SB
454 and directed in the Guidance
Decision 09-11-014.
• Conduct phone interviews with
market actors.
• Coordinate sample design and
data collection with “Market
Assessment of Residential and
Commercial Customer Decisionmaking on High Efficiency HVAC
Systems, Quality Maintenance and
Quality Installation Purchases.”
13
the standard
The IOU Market Assessment may be a
good gateway survey of customers to
identify non-participants and then offer
a follow-up onsite.
4 Research Plan and Work Order Development
See attached WO Proposals.
5 High Level Budget
5.1 Original Budgets from EM&V Road Map Inventory
2013-2014 Study Area/Title
Type
Budget
Impact Evaluation of Commercial and proposed
Residential Upstream HVAC Programs
Impact
$1,950,000
Impact Evaluation of Residential Quality Installation
Programs and proposed incentives for “to code”
installations in the Residential sector (May not include
Commercial Quality Installation)
Impact
$500,000
Impact Evaluation of Residential and Commercial
Quality Maintenance Programs
Impact
$1,500,000
Impact Evaluation of Deemed HVAC Measures
Impact
$300,000
HVAC Laboratory Testing
Impact
$1,000,000
Market Assessment: Identifying Baselines and Barriers
for Existing HVAC Conditions, Building Permit and Title
24 Compliance
Market $1,250,000
TOTAL
$6,500,000
5.2 Proposed Budgets
Studies
Impact Evaluation of Commercial
and proposed Residential
Upstream HVAC Programs
Impact Evaluation of Commercial
and proposed Residential
Upstream HVAC Programs
Impact Evaluation of Residential
Quality Installation Programs and
proposed incentives for “to code”
installations in the Residential
sector (May not include
Commercial Quality Installation)
Impact Evaluation of Residential
and Commercial Quality
Maintenance Programs
Impact Evaluation of Deemed
HVAC Measures
HVAC Laboratory Testing
Market Assessment: Identifying
Baselines and Barriers for Existing
HVAC Conditions, Building Permit
and Title 24 Compliance
Measures
Mini-splits
Package systems,
split systems
Quality installation
and “To Code”
programs
Quality Maintenance
Multiple deemed
HVAC measures
Laboratory testing
Budget
$50,000 for evaluability
assessment;
$200,000 for Year1 and 2
$700,000 for Year 1 inspections
and metering pilot
$1,000,000 for Year 2 additional
inspections and/or metering
$250,000 Year 1
$250,000 Year 2
$750,000 Year 1
$750,000 Year 2
$150,000 Year 1
$150,000 Year 2
$500,000 Year 1
$500,000 Year 2
Obtain permit data $500,000 Year 1
for participants and $750,000 Year 2
non-participants
installations
6 High Level Timeline
See above table in Section 3
15