STATE OF CALIFORNIA
Edmund G. Brown, Governor
PUBLIC UTILITIES COMMISSION
505 VAN NESS AVENUE
SAN FRANCISCO, CA94102-3298
California Public Utilities Commission
Energy Efficiency Evaluation Measurement and Verification
Work Order Cover Page
CPUC Contract Number:
09PS5863B
Prime Contractor:
KEMA, Inc.
ED Work Order Number:
KEMA013
Work Order Project Title:
Lighting Programs Process Evaluation and Market Characterization
ED Work Order Project Manager:
Mikhail Haramati
ED Contract Manager:
Jean Lamming
DMQC Contractor:
TBD
Contractor Project Manager:
Jenna Canseco
Other Project Contractors:
Itron, Inc.
Work Order Budget:
$240,000
Term:
Effective date through June 2012
Planned Budget:
$1,500,000
Purpose: This work order governs the scope of work for EM&V Work Plan Study xref #22 named “Lighting
Programs Process Evaluation and Market Characterization.”
This Work Order does not authorize work that is beyond the scope of the work authorized by the underlying prime contract, may not
expand the budget of the underlying prime contract, nor can this Work Order change, amend or modify any of the substantive terms
and/or conditions of the underlying prime contract nor add any new substantive terms and/or conditions to the underlying prime
contract.
The actual costs to complete an approved Work Order shall not exceed the amount authorized in this Work Order. If, in the
performance of the work, the Contractor determines that the actual costs are likely to exceed the estimated costs, the Contractor
shall immediately notify the Energy Division Contract Manager and Project Manager. Upon such notification, the Energy Division
Contract Manager will:
(1) Amend the Work Order scope to accomplish the work within estimated costs; or
(2) Amend the Work Order budget to accomplish the existing scope; or
(3) Terminate the Work Order.
Any expenses incurred by the Contractor that have not been authorized shall be borne by the Contractor. No amendments to this
Work Order shall be made for work undertaken without the specific approval of the Energy Division Contract Manager.
Approvals:
_________________________________________________
ED Contract Manager
Date
___________________________________________________
Contractor Project Manager
Date
1
This work order assigns staff from Prime Contractor and its subcontractor(s) to work with the Energy Division to complete the scope
outlined below. This will be accomplished through ongoing discussions and meetings with California IOU and ED staff. This work will
be performed under the overall supervision of Energy Division. In performing these tasks, the work order team will follow an
administrative and coordination process developed by ED. All work performed under this work order will be subject to pre-approval
by the ED Project Manager and shall be done at the ED Project Manager’s discretion.
Section 1. Work Order Scope
This work order assigns staff from KEMA/Itron to tasks associated with the process and market
characterization studies for IOU lighting programs (including all residential programs and non-residential
upstream programs)1. These tasks may address any program that offers lighting measures, education, or
information to IOU customers in the residential or nonresidential sectors including statewide, local,
institutional, government partnership and third party programs. Given that lighting is a major component of
the IOU portfolios, the study will be closely coordinated with relevant impact evaluations, measure cost
studies, and other lighting market monitoring and evaluation efforts (such as saturation and market share
tracking studies).
This work order includes nine tasks totaling $250,000:









Task 0 – Develop Work Order
$20,000
Task 1 – Work Order Management
$25,000
Task 2 – Program Tracking Data Review and Analysis
$10,000
Task 3 – Documentation Review and Program Staff Interviews
$35,000
Task 4 – Scoping for Fast-Track Data Collection Activities
$10,000
Task 5 – Develop Process Evaluation and Market Characterization Research Recommendations
$35,000
Task 6 – Determine Research Assignments
$ 5,000
Task 7 – Scoping and Oversight for IOU-Managed Studies
$50,000
Task 8 – Scoping and Implementation of ED-Managed Studies
$50,000
Task 0 – Develop Work Order
This task consists of developing the contents of this work order.
Deliverables:
 Final work order
Budget: Not to exceed $20,000
Task 1 – Work Order Management
1
It is anticipated that non-residential non-upstream lighting research needs will be addressed as part of Work Order 12
(Nonresidential Overarching Process Evaluation and Market Characterization).
2
KEMA’s Project Manager will provide monthly status reporting and invoicing for the duration of this work
order as well as ongoing communication with ED staff regarding scope, schedule, and budget. This task
includes ongoing and ad-hoc communications with ED staff regarding work order progress (as needed), and
also covers the process of obtaining public comment on the work order. KEMA’s Project Manager will deliver
the monthly status reports by the tenth day of each month during which the project is active.
Deliverables:
 Monthly project status reports and invoices
Budget: Approximately $25,000
Task 2 – Program Tracking Data Review and Analysis
The purpose of this task is to assess the content and quality of IOU program tracking data for programs
promoting lighting measures. This task is particularly important given that a key criticism of the 2006-2008
program impact evaluations was the lack of comprehensive and consistent tracking data in formats useful to
the evaluators.2 As such, this task will include:
(1) Assessment of data quality, consistency, and completeness; and
(2) Analysis of tracking data to produce updated information on program participation rates, measure
composition, allocations by delivery channel, and other program metrics.
This task will provide an early opportunity to address concerns regarding tracking data quality and
completeness as well as concerns regarding program direction, participation rates, and so on. Additionally,
comparing results from this assessment with those from previous lighting studies (such as 2006-2008
Upstream Lighting Program [ULP] evaluations) will enable an assessment of the programs’ contribution to
market progress since the previous data were collected. This will involve comparing 2006-2008 program
results with 2010 program results in terms of measure mix, incentive levels and delivery channels for the ULP
and other programs. As such, this task is expected to contribute to the research recommendations discussed
under task 4 (Develop Process Evaluation and Market Characterization Research Recommendations) below.
This task will be very closely coordinated with lighting impact evaluation efforts (such as those addressed in
Work order 28, Residential/Advanced/Upstream Lighting Impact Evaluation) and the majority of the effort for
this task will be captured in the the relevant impact work orders as needed (again with close coordination and
communication with the impact teams). The team will also coordinate this task with the more general data
quality control efforts being pursued through other work orders.
This task will also inform task 6 (Scoping and Oversight for IOU-managed Studies), task 7 (Scoping and
Implementation of ED-managed Studies), and task 8 (Scoping for Fast-Track Data Collection Activities), and—if
the data are available in time—the program staff interviews included in task 3. The team anticipates that this
task will commence in April and be completed in May.
Deliverables:
 Memorandum summarizing key findings and recommendations for additional research
Budget: $10,000
2
In particular, see Section 6.2.1 beginning on page 83 in the Final Evaluation Report: Upstream Lighting Program, Volume 1.
3
Task 3 – Documentation Review and Program Staff Interviews
The objective of this task is to strengthen the current understanding of the 2010-2012 portfolio of
nonresidential upstream and all residential lighting programs and to work collaboratively with IOU
representatives to identify needs for lighting process evaluation and market characterization research. The
task involves three elements:
(1) Reviewing the revised Program Implementation Plans (PIP) filed with the CPUC ED in February 2011 to
obtain updated perspectives on the programs;
(2) Reviewing the information regarding lighting programs obtained by Energy Division in response to data
requests; and
(3) Conducting interviews with IOU program managers and other staff to obtain more nuanced
information on program strategy and their perspectives on process evaluation/market characterization
research needs.
In addition, the documentation review efforts will involve collaboration with the team working on work orders
11 and 12 which have been established to identify and conduct overarching process evaluation and market
characterization research activities for all residential and nonresidential programs, respectively. Key staff
members working on task 3 of this work order are also supporting work orders 11 and 12, which should
facilitate effective communication and coordination with this activity. It is anticipated that, at least initially,
the activity completed under this task will provide the detail needed to profile the 2010-2012 programs
addressing all residential lighting measures and nonresidential lighting measures, which will be used to help
populate a program characterization database (part of task 1 in work orders 11 and 12).
The KEMA/Itron team will develop discussion guides that will include bulleted lists of topics to be addressed in
the interviews. Topics will probe interviewees regarding the status of program design and implementation
strategies outlined in the revised PIPs, challenges faced by the programs and whether/how the IOUs have
addressed those challenges, whether the programs are currently meeting expectations (in terms of customer
response, projected energy savings, alignment with Commission direction and Strategic Plan goals, etc.), and
other topics deemed appropriate. The interviews will also provide an opportunity for the KEMA/Itron team to
identify the research topics addressed in the previous round of evaluation studies, what the IOU
representatives feel they learned from that research, and how the results of those efforts feed into their
perspectives on important research topics to be addressed during the 2010-2012 evaluation cycle.
For example, based on the findings and recommendations from the evaluations of 2006-2008 lighting
programs, improvements in the quality and adequacy of program tracking data will be included as an
important topic to be discussed during the interviews. In addition, the KEMA/Itron team is reviewing
information used to estimate program savings (including as appropriate DEER estimates, work paper
assumptions, etc.) will provide a list of topics to be explored during the interviews. Finally, preliminary results
from the analysis of the 2010-2012 program tracking data (see task 2 above) may further inform the list of
interview topics.
Once the list of interview topics has been developed, the KEMA/Itron team will circulate the discussion guide
among IOU EM&V staff to identify the appropriate individuals to be included in the interviews. Based on our
experience conducting interviews about these types of programs in the past, we anticipate that the following
staff will need to be interviewed to cover the full range of interview topics:
 Program managers and other relevant implementation staff
 Program marketing and outreach staff
4


Program design and planning staff
Program tracking data staff
For the purpose of budgeting this task, we have assumed approximately 12 to 16 IOU staff will be interviewed
across the three electric IOUs and each interview will last approximately 2-4 hours to allow for sufficient time
to address specific topics identified in the interview guide as well as for more detailed discussion of issues
pertinent to specific interviewees, programs, measure groupings, or program delivery strategies.
Senior members of the KEMA/Itron team will conduct the interviews with IOU staff in person (to the extent
that scheduling and logistics allow) or by telephone. All interviewers will have been well-trained in in-depth
interviewing techniques, and all have detailed knowledge of the California IOUs’ lighting programs over the
past few cycles (many with more than a decade of knowledge and EM&V experience). We will conduct
interviews in small groups if possible and appropriate. To reduce respondent burden and increase efficiency,
to the extent possible, the team will coordinate these interviews with those to be conducted under the related
impact evaluation work orders.
This task – along with task 2 (Program Tracking Data Review and Analysis) – will help the team identify fasttrack data collection activities (task 4) as well as the broader process evaluation and market characterization
research activities necessary for lighting (task 5). While the IOUs’ PIPs offer the IOUs’ perspectives on program
plans, tasks 2 and 3 are necessary to identify ways in which the programs (as implemented) differ from the
PIPs and/or areas in which the reality for a particular program (as it’s being implemented) differs from the
IOUs’ expectations for the program at its outset. The staff interviews also serve the critical function of helping
to identify the IOUs’ perspectives on lighting process evaluation and market characterization research
priorities, including the immediate needs (to be addressed in task 4 – Scoping for Fast-Track Data Collection
Activities) and the less time-sensitive needs (to be addressed in task 5 – Develop Process Evaluation and
Market Characterization Research Recommendations).
Note that subsequent tasks in this work order may require follow-up discussions with the same IOU program
staff or additional interviews. This task includes budget for a minimal amount of follow-up discussion and
clarification, but interviews to be conducted as part of stand-alone process or market studies will be covered
by the budgets assigned to those specific studies.
The budget assigned to this task assumes that the work will be completed by mid-April, 2011. However, it may
be necessary to repeat specific elements of this task (e.g., the program staff interviews) in late 2011 and in
2012 to obtain updated program information. If such additional research is deemed necessary, additional
budget may be added to this task in the future.
Deliverables:
 Draft and final interview guides
 Memorandum summarizing key findings from the IOU staff interviews and documentation review
Budget: $35,000
Task 4 – Scoping for Fast-Track Data Collection Activities
The purpose of this task is to identify lighting process and market data needs that need to be “fast-tracked”—
undertaken quickly in the near-term—and to scope out the necessary data collection objectives, timing,
5
methods, and expected outputs (e.g., CFL installation rates) associated with this research. As described above,
tasks 2 (Program Tracking Data Review and Analysis) and 3 (Documentation Review and Program Staff
Interviews) will provide critical inputs into this task.
For example, the 2006-2008 Upstream Lighting Program evaluation collected quarterly customer telephone
survey data on CFL purchase rates and other topics. Collecting periodic and frequent data on similar issues in
the short-term would enable continued tracking and monitoring of these conditions. In this instance, we
envision streamlining the telephone survey instrument used in support of the 2006-2008 evaluation to
address key market progress tracking indicators, and then getting the surveys into the field quickly so that a
more lengthy data gap between the last cycle of data collected (2009) and the current cycle can be avoided.
As the team finalizes the scope of additional lighting process evaluation and market characterization studies,
the team could then add additional elements to the next quarterly data collection cycle within the specified
“fast-track” activity.
Other examples of possible fast-track data collection activities include:


Update on product availability, diversity, and pricing indices via shelf stocking information as reported
in the Advanced Lighting Baseline Study (currently being conducted by KEMA)
Assessment of nonresidential upstream lighting product distribution channels
We will use the results of this task to create work orders related to fast-track data collection efforts, and
separate subtasks under this task (4) will cover initiation of the fast-track data collection.
Deliverables:
 List of fast-track lighting process evaluation and market characterization data collection needs
Budget: $10,000
Task 5 – Develop Process Evaluation and Market Characterization Research Recommendations
The purpose of this task is to identify key study recommendations in collaboration with ED staff, contractors,
and IOUs. This task will identify all lighting-related process evaluation and market characterization activities to
be undertaken during this evaluation cycle. We will also coordinate with the lighting-related impact evaluation
teams who are identifying research needs as part of their research planning activities. The task will also
attempt to differentiate between process evaluation research priorities and market characterization research
priorities to set the stage for discussions regarding research assignments between the IOUs and ED to be held
as part of task 6 (Determine Research Assignments). A separate task in this work order will assign various
research activities for management by the IOUs or ED staff and contractors.
Prior lighting process evaluation and market characterization studies utilized in-depth interviews with lighting
market actors (such as representatives of distributors, manufacturers, and retailers) and IOU staff; detailed
lighting retail store shelf surveys; in-store customer intercept surveys; computer-aided telephone interviews
(“CATI surveys”) with the general population; and other research methods to address numerous topics
including:

California CFL supply chain, including where retailers obtain their CFL supplies from distribution
centers, manufacturers, etc.) and the frequency with which they obtain those supplies; delivery time
(from order to receipt); how shipment sizes for program-discounted CFLs are determined; problems
6
with delivery of program-discounted CFLs; how long it takes to sell through a shipment of programdiscounted CFLs; and retailer actions when selling program-discounted lighting products.

California CFL shopper characteristics, including CFL awareness, purchasing behavior, purchasing
motivations, and demand-side barriers to CFL purchases.

California CFL retail environment, including topics related to the relative availability and diversity of
program-discounted products to non-program-discounted products; relative availability of ENERGY
STAR® products; CFL lumen and wattage varieties; CFL lamp shapes; CFL package sizes; availability and
variety of specialty CFLs; CFL quality; CFL prices and retail pricing strategies; and CFL point-of-purchase
placement and promotional activities.

Preliminary indicators of program attribution and free ridership, including the relationship between
shoppers’ purchasing intentions and actual purchases; shopper awareness of CFL point-of-purchase
materials and the influence of these materials on their purchasing decisions; the effect of CFL multipacks on purchase quantities; the effect of CFL price on purchase quantities; shopper awareness of the
program discounts and the influence of these discounts; and store manager estimates of free ridership.

CFL disposition after the sale, including estimates of CFL leakage, residential versus non-residential use
of ULP-discounted CFLs; and CFL installation, storage, removal, and disposal rates and behaviors.

Satisfaction with the ULP and CFLs, including consumer satisfaction with the program;
recommendations for program improvement; and satisfaction with program–discounted CFLs.
It is anticipated that some of these same topics may be explored through similar research activities
undertaken as part of the 2010-2012 EM&V cycle (in addition to new research topics). The KEMA/Itron team
has participated in ongoing discussions regarding additional lighting process evaluation and market
characterization research activities that should be undertaken as part of the 2010-2012 EM&V cycle. In some
cases, these discussions have indicated a need for specific type of information (e.g., review of IOU’s minimum
equipment quality specifications, assessment of IOU’s allocations of lighting products by product type and
delivery channel, etc.) and in other cases, these discussions have identified a need for a potential EM&V study
(e.g., case study of market/customer behavior in response to new standards banning sale of incandescent
lamps, distribution network assessments for new/advanced lighting products, etc.). In this task, we will
continue the process of identifying research needs and potential study scopes. This task will also enable the
team to identify gaps in planned nonresidential upstream and all residential lighting process evaluation and
market characterization research.
As described above, tasks 2 (Program Tracking Data Review and Analysis) and 3 (Documentation Review and
Program Staff Interviews) will provide additional information/study needs. Other EM&V teams (e.g., impact
evaluation, market share tracking, residential and non-residential onsite saturation surveys, etc.) may identify
additional process evaluation and market characterization needs which will be incorporated as part of this
task.
As part of this task, the KEMA/Itron team will review the Program Performance Metrics (PPM) and Market
Transformation Indicators (MTI) related to lighting programs (see Tables 1 and 2 below) to identify research
required to address each PPM and MTI. The team will also recommend data collection/analysis approaches to
assess program progress with regard to each PPM and MTI.
7
Table 1. PPMs Related to Lighting Programs and Measures
Program
Multi-Family Energy
Efficiency Rebate
Advanced Lighting
Basic CFL
Lighting Market
Transformation
Technology
Assessment
PPM
1. Percentage of non-lighting measures as compared to the total EE measures adopted in the MFEER
program. (KWh for single-commodity IOU and BTU for mixed-commodity IOU.)
1. Percent kW/kWh/Qty of incented products under the Advanced Lighting program as compared to the
Basic Lighting program, by product type.
2. Percent of products incented under the Advanced Lighting Program by distribution channel and by hardto-reach zip-codes.
1. During 2010-2012, implement marketing efforts and/or campaign to encourage prompt installation of
CFLs as required in D-09-09-047 (Y/N metric)
2. Percent of products incented under the Basic Lighting Program by distribution channel and by hard-toreach (HTR) zip-codes.
3. Percent kW/kWh/Qty of incented products under the Basic CFL program as compared to the Advanced
Lighting program
1. Develop a lighting technology roadmap (i.e., what’s new and available by when (MM/YY), using
available information from all IOU and external parties) by January 2011 and update bi-annually. (Y/N
metric)
2. Develop a communication plan, by March 2011, to make the lighting technology roadmap, pipeline
plans, and technology resource information from this program available on the statewide MEO portal by
July 2011, and update annually. (Y/N metric)
3. Number of recommended projects initiated and completed, with findings and recommendations (i.e., this
is a tracking of lighting related projects for ET, Advanced Lighting and 3rd parties), by project type: (a)
work papers, (b) white paper, (c) pilot project (d) strategy document.
4. Number of EE lighting measures added, removed, or updated as a result of LMT activities and
influence, and reported in annual LMT June Report
1. Number of ETP measures which have undergone TA that are adopted* into the EE portfolio, including
but not limited to each of the following:
(a)—(b) [non-lighting]
(c) Advanced lighting technologies
Table 2. MTIs Related to Lighting Programs and Measures
Program
Advanced Lighting
Basic CFL
Lighting Market
Transformation
Codes and
Standards
MTI
1. The average lighting power density of residential and commercial lighting applications.
1. Number of basic CFLs sold annually in California (note: for entire market, not IOU-rebated CFLs)
2. Price of non-discounted MSB CFLs
3. Saturation of eligible sockets (MSB, non-dimming, interior) with CFLs or better
1. Percentage of total lighting sales comprised of Best Practice technologies (by sector)
2. Number of technologies (by sector) for which market transformation is achieved (as defined by program)
3. Number of technologies by sector that no longer require IOU program interventions
2. Number of utility incentivized EE measures that become part of the following code cycle (e.g. measures
incentivized in 2006-2008 would be part of 2011 or 2014 code) targeting the following:
(a)—(c) [non-lighting]
(d) Advanced Lighting
(e) [non-lighting]
This task will produce a full set of lighting-related research topics and study recommendations that capture
the perspectives of the KEMA/Itron team, ED staff and contractors, and IOUs with regard to study priorities.
We will develop this final list in close consultation with ED staff, consultants and the appropriate IOU staff.
This list will serve as the basis for discussions in task 6 (Determine Research Assignments), assist in identifying
the fast-track research elements associated with task 4 (Scoping for Fast-Track Data Collection Activities), and
provide input into the remaining tasks associated with this work order.3 The KEMA/Itron team anticipates that
this task will be completed in April/May, 2011.
3
Note that non-residential non-upstream research needs identified through this process will be handed over to the team managing
Work Order 12 (Nonresidential Overarching Process Evaluation and Market Characterization).
8
Deliverables:
 List of recommended lighting process evaluation and market characterization research activities
Budget: $35,000
Task 6 – Determine Research Assignments
Using the list of lighting process evaluation and market characterization research topics and study
recommendations developed in task 5 (Develop Process Evaluation and Market Characterization Research
Recommendations), the KEMA/Itron team will convene meetings with representatives of the IOUs, ED staff,
and consultants to agree upon the appropriate management structure (ED versus IOU) for these studies. It is
anticipated that these assignments will be discussed during an all-day working session with the appropriate
parties.
The outputs from this task will inform task 4 (Scoping for Fast-Track Data Collection Activities), task 7 (Scoping
and Oversight for IOU-managed Studies), and task 8 (Scoping and Implementation of ED-managed Studies).
The KEMA/Itron team anticipates that this task will be completed by late May, 2011.
Deliverables:
 List of lighting process evaluation and market characterization research activities with research
assignments
Budget: $5,000
Task 7 – Scoping and Oversight for IOU-managed Studies
The purpose of this task is to contribute to scope development for IOU-managed process evaluation and
market characterization studies pertaining to all residential lighting measures and to nonresidential upstream
lighting measures, and to provide oversight of those studies once they are implemented.
Note that this task will focus exclusively on collaboration to develop scopes and budgets for IOU-managed
studies (and providing oversight for those studies once undertaken) while task 8 (Scoping and Implementation
of ED-managed Studies) covers the development of scopes and budgets for studies lead by the ED.
We have assumed a budget of $50,000 for this task although more or less budget may be required depending
on the number and complexity of IOU-managed studies resulting from task 6 (Determine Research
Assignments).
Deliverables:
 Contributions to research plans for IOU-managed studies
Budget: $50,000
Task 8 – Scoping and Implementation of ED-managed Studies
9
The purpose of this task is to work with ED staff and consultants to develop research scopes and budgets for
lighting process evaluation and market characterization studies to be managed by ED.
In some cases, ED may wish to implement these studies via a direct award, in other cases, the work may be
assigned through a competitive bidding process. In the case of competitive bids, KEMA/Itron may be asked to
develop RFPs, manage the RFP distribution process, review proposals, participate in the subcontractor
selection process, manage and oversee subcontractors’ work, and so on.
As ED-managed studies are identified, new subtasks will be added to this task or separate work orders will be
issued. At this point, we have established a $50,000 for this task until such time as the actual study scopes and
budgets are determined.
Deliverables:
 Work orders for studies to be lead by ED or its consultants4
Budget: $50,000
4
The research plans resulting from these work orders will identify additional deliverables and will be addressed through subtasks
under Task 7.
10
Section 2.
Section 3.
Section 4.
Section 5.
Task
Number
Work Order Schedule
Work Order Personnel
Work Order Budget and Invoicing
Deliverables
Task Description
Deliverables
0
1
Develop Work Order
Work Order Management
2
Program Tracking Data Review & Analysis
3
Documentation Review and Program Staff Interviews
4
Scoping for Fast-Track Data Collection Activities
5
Develop Process Evaluation and Market
Characterization Research Recommendations
6
Determine Research Assignments
7
Scoping and Oversight for IOU-Managed Studies
8
Scoping and Implementation of ED-Managed Studies
 Final work order
 Monthly project status reports and invoices
 Memorandum summarizing key findings and
recommendations for additional research
 Draft and final interview guides
 Memorandum summarizing key findings from the
IOU staff interviews and documentation review
 List of fast-track lighting process evaluation and
market characterization data collection needs
 List of recommended lighting process evaluation
and market characterization research activities
 List of lighting process evaluation and market
characterization research activities with research
assignments
 Contributions to research plans for IOU-managed
studies
 Work orders for studies to be lead by ED or its
consultants
Section 6. Additional Sections
Attachment A – Resumes
If needed.
Attachment B – Staff Matrix
If needed. Contains a matrix that includes each proposed staff member’s years of experience, location, and
areas of expertise
11
Comments Tracking
Name
Section
Ralph Prahl
General
Mikhail
Haramati
Nikhil Gandhi
General
Nikhil Gandhi
Task 2 – Program Tracking Data Review and
Analysis
Task 2 - Program Tracking Data Review and
Analysis
Comment
Action Taken
I continue to wonder about the usefulness of combining the
two [residential and nonresidential lighting issues] given what I
would say are the relatively limited overlaps between the
markets for each of them.
(paraphrased) WO13 will be posted for public comment.
(by phone) Include a reference to the task of comparing 20062008 program results with 2010 results in terms of measure
mix, incentive levels, and delivery channels for the Upstream
Lighting Program in Task 2
NVG2 If this task is done here, it should cover the needs of the
impact WO for advanced lighting. I would prefer to see the
budget for this task come from the impact WO budget. It is a
single tabulation and analysis task for multiple purposes that
should feed into all WOs that need this data.
An overarching data QC WO also exists that Itron is managing.
Are they not looking at the quality of the tracking data in that
WO?
Nikhil Gandhi
Task 2
Nikhil Gandhi
Task 3 – Documentation Review and Program
Staff Interviews (comment on following text: “
(3) Conducting interviews with IOU program
managers and other staff to obtain more
nuanced information on program strategy and
their perspectives on process
evaluation/market characterization research
needs.”)
Task 5 – Develop Process Evaluation and
Market Characterization Research
Recommendations
Nikhil Gandhi
NVG3 See comment above. Is it likely that the impact WO
would need additional analysis that is not described here?
NVG4 I can see some value in conducting in-person interviews
for this purpose. However, what more is likely expected to be
uncovered that is not in the bottom up spreadsheets? That is
how will you build on and embellish the needs expressed in
those spreadsheets?
Shifted focus of work order from “all lighting” to all residential lighting as well as
nonresidential upstream lighting. Work order 12 (Overarching Nonresidential
Process Evaluation and Market Characterization) will address nonresidential nonupstream lighting
Modified scope and budget of Task 1 (Work Order Management) to include the
process of obtaining public comment on this work order.
Added text to make this explicit.
This task will be closely coordinated with the impact teams (many of the same staff
are on this and the relevant impact work orders) and much of the budget for this
task will be drawn from the impact budgets. Reduced budget for this task
accordingly.
Added mention of coordination with teams focused on data quality control to text
in this section, but note that the tracking data review included in the data quality
control work order is expected to be at a much higher level than required for this
work order, and the former is not expected to capture all of the nuances specific to
lighting programs and the upstream program elements in particular..
See comment above.
Recent communications with utility program staff suggests that actual program
implementation methods may differ somewhat from what was included in the PIPs
and other documentation, so in-depth interviews are expected to uncover
additional issues not addressed through the needs assessment process.
Again, these interviews should cover all impact related
questions too so we hit the program managers once.
Wouldn’t a gap analysis be appropriate to perform as part of
this task?
Added note to the text stating that these interviews will be coordinated with the
impact teams to the extent possible.
Agreed; modified text to make this point more explicit.
The intent of this text was not to suggest that the same research should necessarily
be conducted as part of the 2010-12 evaluation cycle but rather to provide an
indication of what research has been conducted recently and to suggest that some
of these topics may need to be explored again or explored further. The following
text in this section attempts to make this clear. Also added parenthetical statement
(“in addition to new research topics”) to clarify further.
Task 0 of this work order covers development of the work order itself and does not
include development of study scope for any of the research covered by the work
order, and Task 1 is for work order management. As such, Task 8 is necessary to
provide funding to develop scopes of work for ED-Managed Studies.
Nikhil Gandhi
Task 5 (comment on following text: “It is
anticipated that some of these same topics
may be explored through similar research
activities undertaken as part of the 2010-2012
EM&V cycle”
Why repeat? Should we not first justify why this needs to be
repeated for basic CFLs? I am not seeing the emphasis on
advanced lighting here.
Nikhil Gandhi
Task 8 – Scoping and Implementation of EDmanaged Studies (comment on the following
text: “The purpose of this task is to work with
ED staff and consultants to develop research
scopes and budgets for lighting process
evaluation and market characterization
studies to be managed by ED.”
Isn’t this be Task 0 and Task 1 of whatever WO is may result
in? Then why is Task 8 needed at all until specific WOs are
identified from research conducted as part of this WO?
12
13