Research Plan
Residential On-site Study / California Lighting and Appliance
Saturation Study (CLASS 2012) – WO021
Prepared by KEMA, Inc.
November 2011
Contact Information
Company
Address
City, State, Zip
Primary contact
Phone
Email
KEMA, Inc.
155 Grand Ave, Suite 500
Oakland, CA 94109
Jarred Metoyer
510-891-0446
Jarred.Metoyer@kema.com
Table of Contents
1.
2.
3.
4.
5.
6.
Introduction .......................................................................................................................... 4
1.1 Research Overview..................................................................................................... 4
1.2 Research Goals and Objectives .................................................................................. 4
1.3 Study Inputs and Results ............................................................................................ 5
1.4 Potential Plan Revisions ............................................................................................. 6
Residential Characteristics Needs........................................................................................ 7
2.1 Building Component Selection Rationale .................................................................... 8
Data Sources ..................................................................................................................... 11
3.1 Primary Data Sources ............................................................................................... 11
3.2 Secondary Data Sources .......................................................................................... 11
3.2.1 2009 Residential Appliance Saturation Study ............................................... 11
3.2.2 Third Party Housing Data ............................................................................. 12
3.2.3 Third Party Appliance Sales Data ................................................................. 12
Sample Design and Recruitment ........................................................................................ 14
4.1 Primary Sample Approach ........................................................................................ 15
4.2 Discussion of Bias .................................................................................................... 15
4.3 Recruitment Strategy ................................................................................................ 15
4.4 Monetary Incentive for On-site Survey Participants ................................................... 16
Survey Development and Implementation .......................................................................... 18
5.1 Telephone Surveys ................................................................................................... 18
5.2 On-site Surveys ........................................................................................................ 18
5.3 Surveyor Training ..................................................................................................... 19
5.3.1 Field Staff Coordination ................................................................................ 19
5.4 Digital Data Collection............................................................................................... 19
5.4.1 Quality Control
23
Analysis Methodologies ..................................................................................................... 25
6.1 Telephone Surveys ...................................................... Error! Bookmark not defined.
6.2
On-site Surveys ........................................................................................................ 25
6.2.1 Updating Efficiency Databases and Sources ................................................ 25
6.3
Residential Lighting and Appliance Analysis ............................................................. 25
6.3.1 Merge of Weights 25
6.3.2 Creation of Efficiency Categories ................................................................. 26
6.3.3 Weighting Adjustment for Unmatched Appliances ........................................ 26
Residential On-site Surveys
i
November 2011
Table of Contents
6.3.3.1 Model Number Matching ................................................................. 26
6.3.4 Analysis of Lighting and Appliance Data ...................................................... 28
6.4 Comparison of Results.............................................................................................. 28
6.5 Anticipated Results ................................................................................................... 29
7. Secondary Data Collection ................................................................................................. 35
7.1 Collected Information from Other Studies ................................................................. 35
7.2 Relevant Standard-Industry and -Practice Considerations ........................................ 35
8. Presentation of Research Results ...................................................................................... 36
8.1.1 Draft Report
36
8.1.1.1 Introduction and Executive Summary .............................................. 36
8.1.1.2 Study Methodology .......................................................................... 36
8.1.1.3 Characterization of Residential Appliances and Lighting Inventory .. 36
8.1.1.4 Comparison to Previous Study and Other Similar Work ................... 36
8.1.1.5 Database Development and Web-based Tool ................................. 37
8.1.1.6 Appendix ......................................................................................... 37
8.1.2 Final Report
37
8.1.3 Task 8.3: Public Presentation and Web-based Software Training ................ 37
8.2 Project Management ................................................................................................. 38
8.2.1 Monthly Status Reports ................................................................................ 38
9. Research Coordination ...................................................................................................... 39
9.1 Work Order and Other Evaluation Activity Coordination ............................................ 39
9.2 Communication with CPUC ED and DMQC ................. Error! Bookmark not defined.
9.3 Cross-Contractor Communications .............................. Error! Bookmark not defined.
9.4 Interaction with DEER Needs and Output DeterminationError! Bookmark not defined.
9.5 Methods Required for Non-load-impact Work .............. Error! Bookmark not defined.
10. Timeline ............................................................................................................................. 41
11. Work Plan and Budget ....................................................................................................... 42
11.1 Task-level Descriptions ............................................................................................. 42
11.1.0 Task 0: Development of Work Order ........................................................... 42
11.1.1 Task 1: Project Management....................................................................... 42
11.1.2 Task 2: Develop Detailed Residential On-site Research Plan ...................... 42
11.1.2.1 Needs assessment of residential baseline parameters, etc. ............ 42
11.1.2.2 Develop metering plan for systems or end-uses and sub-sampling . 43
Residential On-site Surveys
ii
November 2011
Table of Contents
A.
11.1.2.3 Develop testing plan for systems or end-uses and sub-sampling..... 44
11.1.2.4 On-site data collection plan ............................................................. 44
11.1.2.5 Sampling and Analysis Plan ............................................................ 44
11.1.2.6 Draft Overall Plan and Comment ..................................................... 45
11.1.2.7 Final Plan ........................................................................................ 46
11.1.2.8 Long Term Data Needs Summary ................................................... 46
11.1.3 Task 3: Scoping of RASS Data Mining and Activities .................................. 46
11.1.3.1 Scoping of RASS Tasks to Support Task 2 ..................................... 46
11.1.3.2 Execute Tasks to Support Task 2 .................................................... 47
11.1.3.3 Execute Longer Term Tasks............................................................ 47
11.1.4 Task 4: Sample Design ............................................................................... 47
11.1.5 Task 5: Preparation and Training ................................................................ 47
11.1.6 Task 6: CATI Telephone Surveys ................................................................ 47
11.1.7 Task 7: Recruiting 48
11.1.8 Task 8: On-Site Visits .................................................................................. 48
11.1.9 Task 9: Site Level Analysis ......................................................................... 48
11.1.10 Task 10: Reporting ...................................................................................... 48
11.2 Overall Budget and Accomplishments ...................................................................... 49
11.3 Task-level Budgets and Staffing Plans ...................................................................... 49
Web-based Analysis Tool................................................................................................. A-1
11.3.1 Tool Development A-3
11.3.2 Digital Data Collection ................................................................................ A-4
Residential On-site Surveys
iii
November 2011
1.
Introduction
The Residential On-site and End-use Metering Study is intended to gather information about
residential building characteristics that are best gathered through on-site observation,
measurement or metering. The research team endeavors to expand the body of knowledge
regarding the building characteristics of California residences gathered by the Energy
Commission’s Residential Appliance Saturation Survey (RASS), a mail and phone survey with
sample sizes an order of magnitude larger than for this on-site study. The results of this study
will be useful both to the Energy Division’s (ED) evaluation of residential programs and to the
planning of Investor Owned Utility (IOU) programs that require periodically updated baseline
data. While all the needs of ED and the IOUs cannot be met in the time frame and budget of this
study, the highest priority needs, in general, encompass those baseline building characteristics
that continue to contribute the greatest uncertainty to the current and next-cycle portfolios of
energy efficiency program savings.
1.1
Research Overview
The goal of the study is to gather baseline data on California residential building characteristics
in addition to the presence, efficiency, and usage of energy consuming devices found in
California households. Such data is best gathered through on-site observation, measurement,
or metering. These data will be used to update the baseline information upon which program
and portfolio planning and program evaluation rely. During the on-site surveys, we will also
gather information about the appliances and high-use, energy-consuming products in homes.
1.2
Research Goals and Objectives
To gather information about the California residential building characteristics as well as the
appliances and high-use, energy-consuming products in homes, the following steps are
planned:

Conduct on-site inspections at a sample of homes in California to characterize
residential building configurations (conditioned square footage, layout, room types)
and specific construction components (for example, attic insulation).

Characterize installed appliances and energy-consuming products during the on-site
surveys. Electric and gas-powered products with high unit energy consumption
(UEC), high on-peak demand, and those for which IOU rebate programs are
presently offered or anticipated will be emphasized.
Residential On-site Surveys
4
November 2011

Perform low-cost measurement of energy use at a large proportion of, if not all of, the
sample. Also, it may be decided to conduct high-cost or specialized measurements
at sub-samples—these would funded by other impact evaluations or studies.

Achieve greater statistical precision for residential building characteristics that are of
high interest, e.g. efficiencies of newer equipment types.

Mine the data gathered for the 2009 RASS and review the 2008 DEER prototypes to
inform the development of a comprehensive and robust plan that can be used to
enhance and supplement the periodically recurring CEC and CPUC studies.

Supplement and update the recent estimates of residential end use consumption,
although whether to emphasize UECs or load shapes is yet to be determined.
1.3
Study Inputs and Results
Study inputs include the 2009 RASS data, the 2005 California Lighting and Appliance Saturation
Survey (CLASS) findings, residential property data and IOU customer contact data. The sample
design process will consider and utilize variables that are included in the utilities’ billing systems
and that are routinely available for sampling. Also, if square footage can be obtained from
public property records or a third-party property data provider for most residential customers,
this information will be gathered and considered as a stratifying variable.
The following is a list of key expected study outputs for California residences:

Distribution of type, efficiency, size and age of equipment such as ACs, refrigerators
and furnaces

Distribution of installed power density (watts/square foot) of plug loads and lighting
by room type

Distribution of building characteristics such as square footage, room types and
window types

Distribution of household demographic characteristics such as number and ages of
occupants

At sub-samples of the study sample, distribution of equipment usage gathered by
end-use metering

Extent of market penetration for some building elements as needed by Work Order
059 regarding Overarching Residential Sector Market Assessment.
Residential On-site Surveys
5
November 2011
1.4
Potential Plan Revisions
This plan is expected to evolve as other Work Order evaluations determine any research needs
that could be addressed during the Residential On-site and End-use Metering Study. For
instance, it is anticipated that there will be interest in gathering metering data regarding
household lighting, plug loads, HVAC-Cooling, HVAC-Gas and domestic hot water.
Furthermore, some of the data gathered during the on-site surveys will likely be used to inform
the Residential Market Share Tracking (RMST) project that monitors the market penetration of
energy efficient measures in California.
Residential On-site Surveys
6
November 2011
2.
Residential Characteristics Needs
This study seeks to characterize building components that influence the energy consumption
across the current residential building population in California. A baseline for these building
components was established with the 2005 CLASS study and then partially refreshed in 2009
with the RASS study. Building component, lighting, and appliance characteristics have not been
inventoried at the site level by a trained field technician since 2005. Market trends and
technological advances have evolved since the last studies were performed. There is no doubt
the present day residential energy usage landscape likely reflects these changes. To what
extent is where the uncertainty lies.
Parties must be identified for study supplementation as well as review. Inviting key public-owned
utilities to join the study (SMUD, LADWP) is an important early step. The scoping will have ED,
Prime Management, DMQC, and Consultant review as well as proposed scope review by the
IOUs. The plan will receive public comment and may raise additional important scope
considerations.
The scoping will allow each party or individual to assign priority to some or all elements that will
be compiled and discussed. The project leadership will develop a final proposed list of data
collection and analysis needs as the deliverable of this task which will feed into the data
collection plan.
The following is a list of primary study outputs for all homes in the study:

Distribution (mean, median, mode, percentiles) of efficiency and size for equipment
such as AC (SEER/EER, tons), refrigerator (MEF, cubic feet), furnace (AFUE, kBTU
output).

Distribution of watts: Installed Power density (watts/square foot) and distribution by
room type for plug loads and lighting.

Age of equipment.

Building/Household characteristics – square footage, room types, number of
occupants and other demographics, configuration of HVAC, window type.

Usage patterns for appliances reported by the homeowners

Replacement practices on failed or outdated equipment as reported by the
homeowners
Residential On-site Surveys
7
November 2011
Elements included in the current study’s scope were included in recent studies. Selecting and
evaluating identical elements will enable a longitudinal or time series analysis comparing
previous studies analysis with the 2012 Study’s findings.
2.1
Building Component Selection Rationale
For each building characteristic of interest, at least one parameter has been identified as a
contributing factor to household energy consumption. To plan this study, a table of building
characteristic parameters was assembled that includes: on-site time needed to observe building
component and record data during on-sites, in minutes: instances when nameplate data will be
collected for appliances; instances when the location of the specified equipment will be
inventoried.
Building characteristic parameters will remain on said table unless all WO021 parties agree that
removal is appropriate. The building characteristics will be categorized based on like elements
as shown in Table 1 below.
Residential On-site Surveys
8
November 2011
Table 1: Building Characteristics Needs Planning Sample
Examples of Building
Element Factors, by
Category
Building Characteristics
and Demographics:
Lighting Installed
Inventory:
Data Collection
Time Accounting
for Saturation
(Minutes)
Manufacturer
Model Number
Full
Inventory
Locations
15
X
60
X
Primary Building
Equipment:
Space Cooling
6.5
X
Space Heating
10
X
Water Heating
7.5
X
Refrigerators/Freezer
7.5
X
8
X
Dishwasher
3.5
X
Range/oven
5
X
1.5
X
TV’s
15
X
X
Boxes/Entertainment
5
X
X
Personal Computers
4.5
X
X
Small Appliances
7.5
X
X
Appliances:
Clothes Washer/Dryer
Spa/Pool Equipment
Plug-in/Consumer
Electronics:
Total:
Residential On-site Surveys
156.5
9
November 2011
A cost structure will be developed such that the base cost of an on-site is the travel time and
expenses on average across the state so that all data collection options are incremental on a
site basis. The time to collect characteristics data via observation is based on time measured in
minutes. The time to look up or derive the rated performance from those data is also expressed
as labor. The costs for measurement and metering will be separated such that a metering
oversight group or a small group within this work order can develop detailed plans and provide a
cost per sample point to the larger planning group and parties. The metering samples will likely
be nested within the larger study and sample sizes for measurement of each element will be
independent of the standard site cost. All cost and labor estimates in the draft scoping
spreadsheet as well as data sources are to be refined under this task. Other Work Orders can
fund incremental data collection activities, but we expect that additions primarily pertain to
measurement and metering subsamples.
Residential On-site Surveys
10
November 2011
3.
Data Sources
Data sources for the 2012 CLASS Study will be three tiered. The primary data source will be
obtained during the on-site visits and from the CATI demographics and household consumer
surveys. Secondary data sources will be provided by the IOU’s. Tertiary data sources will come
in the form of past RASS studies, third party housing data, and third party consumer appliance
databases.
3.1
Primary Data Sources
The core of the data for the 2012 study will consist of the residential building characteristics
inventoried during the site visits. In addition, another substantial portion of the primary data will
come from 4,800 CATI Telephone surveys.
3.2
Secondary Data Sources
The report findings and reference databases created in the 2005 and 2000 CLASS Studies will
be utilized extensively for the 2012 study. KEMA is in possession of this data.
For the purposes of this study, KEMA has been provided with IOU data for the residential
population in each of the IOU service territories. In addition to customer contact information and
site location, the data provided for each customer includes:





3.2.1
Utility
Title 24 Climate zone
Rate type / Tier
Low income flags
Energy usage
2009 Residential Appliance Saturation Study
For this study, KEMA will make use of the data gathered for the 2009 RASS. Although the data
will need to be remapped from the forecast climate zones to the T24 Building Climate Zones,
the residential appliance saturation study results will be used to inform the sample design.
Residential On-site Surveys
11
November 2011
3.2.2
Third Party Housing Data
This study will purchase and make use of additional information typically provided in public
property records for most residential customers. This information will be gathered and, as
feasible, used as stratifying variables. The additional data to be sought include:





3.2.3
Square footage
Number of stories
Number of bedrooms
Number of bathrooms
Year built
Third Party Appliance Sales Data
The evaluation team will, upon approval, purchase the 2008-2011 sales data for the following
measures:

Room Air Conditioners

Refrigerators

Freezers

Washing Machines

Dishwashers

Clothes Dryers
According to the sales data provider, NPD, home appliance data is not available at the store
level for retail data. Consumer data for these measures is gathered through diary studies. For
these measures, data would be provided from Census region or state to determine an estimate
for the % sales for key appliances that have Energy Star certification. This sample size would be
too small to estimate total amount of sales. NPD would overlay this on the national sales data
to provide an estimate of % of sales in California that are Energy Star certified. Utilizing the
consumer data in this method, NPD will provide yearly estimates for California on the total
number of appliances sold in the categories listed above. Also, utilizing their national point of
sale (POS) data, NPD can then provide a monthly % that estimates how many of each of those
appliances is sold per month to account for the seasonality of purchases.
From the POS data, NPD can also provide the national percentage estimates of the various
energy related features depending on the device. This would be kWh for Refrigerators,
Freezers, Washing Machines and Dishwashers and EER rating for Room Air conditioners. NPD
Residential On-site Surveys
12
November 2011
does not currently have energy usage data on Dryers. It is possible the evaluation could create
categories (high/medium/low) for this appliance. According to NPD, by overlaying the
percentages of different features over the California sales totals, the evaluation team will get
usable data.
Residential On-site Surveys
13
November 2011
4.
Sample Design and Recruitment
This study is intended to gather information about the California residential building
characteristics as well as the appliances and high-use, energy-consuming products in homes.
This section is intended to discuss balancing the goals and budget of the study with the
logistical challenges and bias risks. Specific sample sizes are not included at this time since
assessment of the population data and estimating the expected precision of outputs are still
pending.
One of the key issues for the Residential On-site study, as is similar to past CLASS studies,
include the scope of work and sampling. The previous CLASS sampled approximately 1,000
homes and collected information on the building structure, the major appliances, and included a
comprehensive lighting inventory in a roughly one hour in-home visit. In addition to the IOUs,
SMUD was also included in the study and contributed to study funding. While all home types
were addressed, sample sizes for some market segments were fairly small.
The 2005 CLASS sample was proportionally allocated by utility and rate class, with the added
constrain that each rate class should have at least one sample point. SMUD joined the study
after the sample sites had been allocated to the utilities, which resulted in having a SMUD
sample size that is larger than proportional if compared to that of the three IOUs.
The team is considering the following changes to the sample design that was implemented in
2005:




Consider multi-family dwellings in the sample design
– 35% of sample (proportional allocation based on 2009 RASS)
– May be under-sampled for certain climate zones
– Inclusion of separate common area survey (19% of all residential customers are in
apartments with 5+ unit buildings)
Consider manufactured homes in the sample design in light of the need for information
for manufactured homes programs
– 4% of sample
Target new construction homes
– About 4% of sample
Conduct a consumer electronics / plug load inventory
– Inventory of non-measured items was part of a past plug load study (small sample of
75 homes)
Residential On-site Surveys
14
November 2011
4.1
Primary Sample Approach
KEMA plans to use a primary sample approach that will seek to obtain as many of the first set of
selected sample points as possible before recruiting sample replacements. Rather than
providing a list of potential sample participants that is a greater than the target completes at the
onset of the recruitment effort, sample replacements would be recruited only after a prescribed
list of attempts has been made to recruit the primary sample sites. The prescribed list of
attempts is yet to be developed. As examples, it could include features such as: at least seven
calls: three during morning weekdays, two during evening weekdays, and two on weekends; at
least one of the calls will leave a message with a phone number that the potential sample
participant can call to set up an appointment; etc.
Each of these primary sample points will have numbered replacements that are selected based
on characteristics beyond those determined by stratification. For example: if the sample design
was based on T24 zone, utility and level of consumption, primary sample sites would be
randomly selected within each stratum determined by each of these three factors.
Replacements for each of the primary sample sites would be determined by the three variables
and, in addition, be required to be in same zip code or to have a level of consumption that is
within 5% of the primary sample site that they are replacing, or other characteristics that
increase the similarities between the replacement site and the replaced primary sample site.
4.2
Discussion of Bias
Bias is a major concern with surveys in general and with on-site surveys in particular. On-site
surveys are time consuming and invasive. Households that agree to participate in these
surveys are more likely to include a person that is usually at home more (have higher daytime
occupancy), and/or to be more interested in energy use / energy efficiency than average. While
it would be nearly impossible and prohibitively expensive to eliminate bias, sampling techniques
and sampling analysis must aim to identify and minimize bias to the greatest extent possible.
The strategies proposed in this memo are intended to address and manage these risks without
undue burden on the study.
4.3
Recruitment Strategy
Recruitment of on-sites will be preceded by an introductory IOU mailing/e-mailing with study
sponsor contact information, an explanation of the study, and verification of KEMA’s
involvement. Recruitment for residential on-sites can be problematic because of a natural
Residential On-site Surveys
15
November 2011
distrust toward efforts to collect personal information. If potential participants are informed in
advance that the study is state-sponsored and in conjunction with the IOUs there will be less
suspicion and more willingness to participate.
We will send an explanatory letter that will give the potential participant both a direct and reliable
contact at the utility and provide a quick, easy, way (email and KEMA’s 800 number) to either
participate or decline. The utilities’ call centers will be informed of the study. These call centers
have hundreds of employees that deal with thousands of calls each day. The multiplicity of
issues that the call centers address, including many programs and studies, makes it difficult to
have these call centers be the main contact for participant questions regarding this study.
This mailing will be followed by a comprehensive CATI survey. From those that agree to an onsite survey, participants are screened for variables of interest and the selected participants are
called back to request on-site participation. The main benefit of the CATI survey is that it
provides a very good picture of the population prior to finalizing the sample design and/or the
cutoffs for the variables of interest.
The work group has agreed upon an approach that has been employed numerous times in the
past. It involves conducting a comprehensive CATI survey, then asking CATI participants
whether they are willing to participate in the on-site survey if called at a later date. From those
that agree to an on-site survey, participants are screened for variables of interest and the
selected participants are called back to request on-site participation.
The main benefit of this approach is that it provides a very good picture of the population prior to
finalizing the sample design and/or the cutoffs for the variables of interest. For example, a priori
the sample design may have considered stratifying by square footage and vintage of residence,
but the CATI may indicate that stratifying by square footage and climate zone may yield better
results.
4.4
Monetary Incentive for On-site Survey Participants
The team proposes offering a monetary incentive to on-site survey participants. When
combined with ample scheduling flexibility (for example, weekend or evening appointments),
incentives help reduce bias by increasing the number of survey participants that would have
otherwise declined the survey.
Residential On-site Surveys
16
November 2011
The team envisions tiered incentives that are proportional to the level of inconvenience and
effort of the survey participant. The incentive levels are not established yet – they are provided
as examples to illustrate the concept. For example:

Full on-site surveys may take up to two hours – an appropriate incentive may be in
the neighborhood of $100.

The Market Share Tracking group (WO023) suggests offering a short on-site to
people who refuse the full on-site, in order to verify one or two appliances, or plug
items recently purchased. These short on-site surveys may take approximately 20
minutes to complete. The monetary incentive could be substantially less than that of
the full on-site survey - $20 to $25.

Participants of the optional metering sub sample would qualify for additional
incentives – for example, $100 in addition to the incentive from participating in the
on-site survey. This incentive is offered in addition to the on-site survey with the
intention to allow participants of the short on-site survey to participate in the metering
sample.
The team will consider offering a higher level of incentive in a non-response follow-up for groups
that had a low response rate during the initial effort.
Residential On-site Surveys
17
November 2011
5.
Survey Development and Implementation
All of the demographics, building characteristics, appliance saturation, and lighting inventory
details included in the 2009 RASS study and the 2005 CLASS study will be included in one of
the two surveys administered for the 2012 study. In addition there will be a sub-section of
additional parameters addressed in the 2012 study not previously addressed in the past studies.
For example the technology and the available products for television viewing has evolved a
considerable amount since the last on-site CLASS study was performed. Additional data fields
will be collected for cable boxes, digital video recorders, gaming consoles, as well as the make
and model numbers of the televisions present in homes visited.
5.1
Telephone Surveys
After the utility mailing explaining the study, KEMA will contract with a CATI survey firm to make
the initial phone calls to potential participants. The purpose of these calls will be to gain
participation and verify certain key requirements. KEMA will use a bid process to determine the
CATI sub- contractor. KEMA will create an approximately 10 minute survey for the CATI subcontractor. The initial sample will be 5000 names with a goal of 2000 participants. There is no
incentive for this initial 10 minute survey. KEMA staff will monitor and QC the CATI progress to
insure adherence to the KEMA phone protocol. Lead time for delivery of willing participants will
be approximately one month from the time the initial and backup sample is given to the CATI
service. The scheduling of actual site visits from the names provided will be done by KEMA
staff. Scheduling staff will work directly with the field personnel and will utilize a real-time qualitycontrol schedule to insure accurate and complete data collection.
5.2
On-site Surveys
Accurately conducted onsite audits are vital to this project because they will:

Provide reliable information on lighting and appliance saturation and energy use in
the California residential market sector,

Provide first hand confirmation of key demographics questions asked during the
CATI surveys, and

Lay the groundwork for developing future residential programs aimed at transforming
the residential market towards energy efficiency.
Residential On-site Surveys
18
November 2011
5.3
Surveyor Training
KEMA will conduct a one-day training session for each auditor before on-site work begins. In
addition, each surveyor will receive a Training Material packet for reference in the field. The
information packet and training session will cover the following topics:
5.3.1

The purpose of the project,

The procedure for verifying the site visit with the homeowner,

The importance of being on time and courteous,

The protocols for dealing with unanticipated problems,

The procedure during the survey,

The best methods of collecting and recording the information,

How to operate and collect the data using the digital data collection device,

The procedure for transferring on-site data to master database, and

Any other relevant topics.
Field Staff Coordination
The trained auditors will conduct the onsite audits according to the schedule set by the recruiter.
KEMA will develop an on-line calendaring system that will allow auditors the ability to identify
the projects that they have been assigned. The recruiters will use this tool for scheduling
appointments and notifying customers of the auditor’s name that will be visiting them.
Each onsite visit consists of two elements: the customer interview and the walk-through
inventory. First, the auditor will conduct the interview with the occupants to verify select
demographic and behavioral questions covered in the CATI survey. Next, the auditor will
conduct the walk-through audit of the home and record the lighting and appliance data into the
digital data collection devices.
5.4
Digital Data Collection
As with the last CLASS Study in 2005, KEMA will conduct the data collection for the 2012
CLASS Study using a digital input format. KEMA has recently conducted a substantial amount
of internally-funded research to advance digital data collection technologies and procedures in
general. The KEMA developers and database managers have evaluated the latest hardware
Residential On-site Surveys
19
November 2011
and software options and are confident the following combination of hardware, servers, and
software will provide the most secure, reliable and capable digital data collection network. As
shown in Figure 1 and then detailed in the following section, the FileMaker Server Advanced
database network in conjunction with Apple iPads has the ability to meet all the requirements of
this large scale digital data collection effort. The process is explained below:

STEP 1: Digital site forms will be created in FileMaker Pro and hosted on a dedicated
KEMA server using the FileMaker Server Advanced software. (See Figure 2.)

STEP 2: As sites get recruited and assigned to field techs, recruiters or the data
collection task manager will assign a day or week’s worth of sites to a particular field
tech. Any special scheduling notes will be added to the site record’s information.

STEP 3: Field technicians will use a WiFi or 3G connection to connect to the FileMaker
Server and only access the records assigned to them in the master database. This will
be done prior to going on site when a reliable internet connection can obtained. The
geographic diversity of field work will require users to be able to work offline on local
versions of the database while on site.

STEP 4: Once a site visit or day of site visits have been completed the field tech can
then re-sync the updated data files to the master database when they have internet
access.

STEP 5: Completed site data will go through a QC/QA process and can be viewed by
authorized viewers who log in to view designated portions of the database through the
instant web publishing feature in FileMaker Server Advanced.
Residential On-site Surveys
20
November 2011
Figure 1: FileMaker Server Advanced Data Collection Network
Step1: KEMA Database Developers
Master
FileMaker
Database on
KEMA Network
create the Master Database and digital
site instruments using FileMaker Pro
Step 2 & 5:
Task Managers
and recruiters
assign sites and
QC completed
sites in the
Master DB
Backup
Database
automatically
saved on set
interval
Steps 3 & 4: Accessing Master DB with F5 app,
syncing assigned sites before and after site visits
Over 250 iPad’s w/FileMaker Go are capable of connecting to the master database with
FileMaker Server Advanced
Residential On-site Surveys
21
November 2011
Figure 2: Sample View of FileMaker Site Instrument
The FileMaker Server Network will eliminate post site visit data entry. It will enable authorized
viewers to access same day site data via a web portal, ensure all data is SSL encrypted,
provide the capability of working offline in remote locations, allow photos to be automatically
labeled and referenced to corresponding data fields, and ensure the highest level of accuracy
and consistency is maintained during data collection.
Maintaining secure data is an absolute necessity throughout this process. Therefore the master
database will always reside behind the KEMA firewall. In order for mobile devices to access
material behind the firewall they will use the F5 Networks BIG-IP Edge Portal application for iOS
devices. Details regarding the FileMaker Server Advanced, GoZync and the F5 Networks
application can be found in the appendices. Additionally, some, or all, of the master database
can be made viewable to authorized persons via a web portal display made possible by
FileMaker Server Advanced. Data from the master database will be output to a SQL server
Residential On-site Surveys
22
November 2011
database which is SAS-compatible. Initial queries and data analysis can also be done within
the FileMaker Pro software as well.
KEMA recently executed an extensive pilot study using the iPad/FileMaker combination for data
collection. The data collection and analysis has concluded and the feedback from all those
involved was extremely positive. KEMA developers created three different site instruments
which were deployed on the iPads using the application FileMaker Go. The developers and
managers learned a lot from the experience and are confident they can incorporate the lessons
into more successful data collection efforts utilizing the iPad/FileMaker combination in the
future.
Refer to the appendix for a detailed description of the different FileMaker database network
components and a more thorough account of the research and hardware considerations made
by KEMA prior to this research plan.
5.4.1
Quality Control
One of the primary benefits of the digital data collection method detailed in the previous section
is the ability to ensure the data collected is done so in the most consistent and accurate way
possible. By creating site instruments in FileMaker a litany of quality and consistency controls
are implemented which are not possible with paper forms. For example:

Over 90% of the survey instrument will be in the form of drop down choices

Validations on each page require every data field is completed before the form can be
submitted as complete

A notepad icon is located next to every data field that allows the field staff to document
any irregularities within the database in reference to a specific data field and not at the
bottom or in the margin of the page

A camera application allows for photo’s to be imbedded within the database with a
caption and in reference to a particular section of the survey instrument

Site data can be checked the same day by the task manager or project manager
Senior staff at KEMA will be available to review completed site visits on a daily basis. Senior
staff will review samples of uploaded survey data, hold conference calls with all surveyors to
discuss unforeseen issues that arise, and provide guidance and training on project efficiency.
Residential On-site Surveys
23
November 2011
The field supervisors will report to the KEMA Project Manager regularly so that all parties are
familiar with current findings and activities.
Site schedulers will schedule a one hour period for one on one QC conference calls with the
field supervisor after the second and third day for each of the field staff’s first week in the field.
They will also schedule a one hour QC for the second week and then on an as-needed basis.
These specific one on one QC sessions will be in addition to the weekly field staff conference
calls.
Residential On-site Surveys
24
November 2011
6.
Analysis Methodologies
A heavy emphasis will be placed on enabling the continuation of analysis performed in the
previous 2000 and 2005 CLASS studies. KEMA will refer to the findings of the previous studies
as a starting point when planning the analysis strategy for the 2012 study.
6.1
On-site Surveys
KEMA will develop and deliver two databases, an appliance database and a lighting database.
The parameters for these databases will match those used in the 2000 and the 2005 CLASS
studies. Additional data fields will be collected; however the structure and specifics of the data
fields established in the previous two studies will remain unchanged.
6.1.1
Updating Efficiency Databases and Sources
One key task required for completing the efficiency database is updating the efficiency
databases that were used in the past CLASS Studies for matching model numbers to efficiency
data. In the 2000 and 2005 studies several sources of efficiency databases were used in the
model number matching process. The following sources will be available for the 2012 Study
(see Error! Reference source not found. for a thorough discussion):

Association of Home Appliance Manufacturers (AHAM)

California Energy Commission Databases

Air-conditioning and Refrigeration Institute (ARI)

Carrier Blue Book

HVAC Partners

Preston’s Guide
6.2
Residential Lighting and Appliance Analysis
6.2.1
Merge of Weights
Once the sites are merged into the central database, a lookup table will provide the sample
design case weights for the analysis in each table in the database. Each site in a given stratum
will have a corresponding case weight that we define to be the number of sites in the population
Residential On-site Surveys
25
November 2011
that the site is thought to represent. The following formula defines the stratum weight to be the
ratio of the number of sites in the population in that stratum to the number of sites in the sample
in that stratum. Each stratum will have a corresponding weight, and accordingly, each site
within the stratum will be associated with that weight. These weights are used to expand the
sample to the population.
wh = Nh / nh , where h is the stratum number
6.2.2
Creation of Efficiency Categories
A second set of lookup tables will translate appliance type, size and energy efficiency into
efficiency categories – depending on the end use. KEMA will reuse the same queries
developed for the 2000 and 2005 studies which will not only save time but also result in a more
meaningful, or “apples to apples”, comparisons of the 2000 and 2005 findings to the 2012
findings. Since some new data is likely to be collected, it will be necessary to develop some
new efficiency categories. However, the bulk of this work is already completed and is ready to
be used for the 2012 Study.
Lastly, there will be numerous variables by which the data can be analyzed. In the previous
studies, we allowed the user the ability to “slice and dice” the data using numerous variables of
interest. For example, users could analyze the data by utility service area, household type (e.g.,
single family attached, multifamily, etc.), income range, climate zone, and much more. We
recommend using these same variables, and perhaps adding a few more if necessary.
6.2.3
Weighting Adjustment for Unmatched Appliances
An additional task that KEMA will perform as part of this project is a weighting adjustment to the
appliance data. The adjustment will be performed in order to remove the upward bias in
efficiency due to the lower matching rates for older models. Below, we discuss how model
number matching and unit degradation will eliminate the upward bias observed in the previous
study.
6.2.3.1
Model Number Matching
One of the study flaws identified in the 1999-00 Study was that new appliances were easier to
match to efficiency data than were older appliances. We have good reason to believe that these
uneven match rates produced more efficient overall baseline appliance efficiencies than is
actually the case.
Residential On-site Surveys
26
November 2011
To improve the findings of the study, we will calculate new weights for the appliance efficiency
data in order to account for the uneven match rates we expect to encounter.
The first step in calculating the new weights will be to group the efficiency data within age bins
by appliance. The appliance weight will be calculated by dividing the total number of appliances
in the sample (both matched and unmatched) by the number of matched appliances within each
age bin. Then this appliance weight will be multiplied by the sample weight to project the
appliance efficiency to the population. The appliance weight will ensure that each appliance
with efficiency within the given age ranges has the same proportional representation as the total
number of appliances within that age range with and without efficiency.
Take for example the findings from the 2000 Study for water heaters, as shown in Table 2Error!
Reference source not found.. As can be seen, the match rate for units manufactured between
1995 and 2000 was 56%, even though the saturation of units of this age in the population was
38%. Units manufactured between 1985 and 1989 had a match rate of 13%, even though this
age group represents 21% of the population. In a perfect situation, the match rates would more
closely resemble the saturation rates, as is the case for the 1990-1994 age bin shown in Table
2. Weighting of the data would serve as mathematical adjustment to the match rates so that
each matched appliance more accurately represents its contribution to the overall baseline
efficiency.
Table 2: Manufactured Date of Matched Water Heaters
Age
1995 -2000
1990 -1994
1985 -1989
1980 -1984
1979 or older
Total
Age Distribution of
Number of Percent of Units
all Water Heaters
Units Matched
Matched
(n = 754)
183
91
41
9
3
327
56%
28%
13%
3%
1%
100%
38%
29%
21%
7%
5%
100%
This task will be performed on the 2005 study data, and we will also go back to the 2000 study
data and perform the same task. This will produce an “apples to apples” comparison that would
not otherwise be possible if we did not re-weight the previous study data.
Residential On-site Surveys
27
November 2011
6.2.4
Analysis of Lighting and Appliance Data
Using the data structures and analysis queries that were developed for the previous studies,
KEMA will conduct the lighting and appliance analysis. In short, KEMA will analyze the data just
as it was analyzed for the previous studies, which met the majority of the project stakeholders’
expectations. By producing the same analysis, only using different input data, KEMA can
compare and contrast findings in the 2012 report by simply adding new columns to the data
tables presented in the report. Within the tables, KEMA will highlight findings that are statistically
different. The analysis will be conducted at the statewide level, and at the utility level, as was
done in the previous report. Furthermore, users of the web-based tool will find a plethora of
ways to analyze the data far beyond what is presented in the report, using a host of categorical
variables for “slicing and dicing” the data.
6.3
Comparison of Results
A central piece to the 2012 CLASS study will include a comparison of current results to the
previous results, in addition to a comparison to the Residential Market Share Tracking (RMST)
Study. For comparing the results to the previous study, we recommend producing a report with
all of the same content as the previous report. The results of the 2012 study would simply be
aligned next to the 2000 and 2005 study data, and where the variances are statistically different
we would shade or highlight the row of data. This allows for side-by-side comparisons across all
areas that were previously reported, and an easy way for readers to identify what is significantly
different, and what is not.
In terms of comparing the results to the RMST Study, KEMA will provide a brief section in the
final report that looks at this piece specifically. The RMST Study tracks the average efficiencies
and shares of highly energy efficient HVAC equipment, appliances, and lamps sold for use in
California’s residential sector. KEMA will analyze recent trends in energy efficiency lighting and
appliance sales to determine if there is a correlation to the trends identified by this project. For
example, trends in CFL sales in the state of California have gone from 1% market share in
2000, to more than 8% in 2001, and leveled off at around 5% in the last quarter of 2003. As a
result of this increase, we would expect to see an increase in the saturation of medium sized
CFLs in California Residences. Therefore, a comparison of RMST data to the 2004-05 data
would yield a better understanding of how well sales data and saturation data align.
For lighting and appliances, KEMA will produce a chapter of the report dedicated to comparing
and contrasting the 2012 Study findings to the trends identified by the 2005 and 2000 CLASS
Studies as well as the 2009 and 2004 RASS Studies. Where findings significantly contradict one
Residential On-site Surveys
28
November 2011
another, KEMA will try to assess the contributing factors to the differences identified through this
comparison analysis. Additionally, and as with the previous reports, KEMA will also produce
comparisons of equipment saturations of efficiency to current DOE standards and ENERGY
STAR standards for all major appliances and HVAC equipment.
6.4
Anticipated Results
The sample size for the 2012 CLASS study is primarily based on assumptions and data
obtained from the 2005 CLASS as well as the 2009 RASS. This data was downloaded from the
online California Residential Efficiency Saturation Tool (CALRESEST1). Table 10 (at the end of
this document) lists all the appliance types and parameters for which data queries can be run.
For all appliance categories highlighted in Table 10, key parameters were selected on which to
focus sample size calculations. Initially this data was used to calculate the necessary sample
size for each appliance type to achieve 10% error bounds where values ranged from about 200
to >20,000 homes. After considering this range, a sample size of 2,000 appeared to be
adequate and feasible for most key appliances.
In order to evaluate precision for the various appliances and parameter levels (n=2,000), the
data for the chosen parameters were organized by IOU and pre-determined bins. The data was
organized by three tables: “Results,” “Frequency Distribution,” and “Error Bounds.” For all of
appliance categories, relative precision was calculated using the following formula.
Equation _-1: Relative Precision
Relative Precision2005 =
Error Bounds2005
Results2005
Error ratios were calculated with the following formula using the 2005 frequency distribution,
relative precision and population (number of households), which was assumed to be 7,000,000.
Equation _-2: Error Ratio
1
CALRESEST, The California Residential Efficiency Saturation Tool. (calresest.kemainc.com)
Residential On-site Surveys
29
November 2011
Error Ratio2005 =
Relative Precision2005 √Frequency Distribution2005
(1.645)√1-
Frequency Distribution2005
Population
The methodology for selecting sample sizes per appliance type can be seen in Figure 3. The
total number of homes targeted for this study was assumed to be 2000. The saturation levels
per appliance were applied from the 2009 RASS study to predict the number of appliances that
are expected to be found, given the total sample size. RASS was used because it provided
more recent data and had a larger sample size compared to the 2005 CLASS study.
Figure 3: Overview of Frequency Distribution Process
The projected sample size was then stratified using the actual distribution from the 2005 CLASS
study for the IOUs within selected parameters.
Equation _-3: Precision
Error Ratio2005 ×1.645√1Precision2012 =
Frequency Distribution2012
Population
√Frequency Distribution2012
Key Examples
Cooling
Cooling efficiency is shown according to seasonal energy efficiency ratio (SEER), which is the
measure of air conditioning efficiency given in kBtu of cooing delivered per kWh of electrical
Residential On-site Surveys
30
November 2011
energy consumed.2 A total of 1300 homes out of 2000 are expected to have cooling units which
is based on a 65% saturation level from 2009 RASS.
Table 3: Cooling Efficiency Assessment of Average SEER
Sample Stratification & Precision Levels, combined n=2,000
Average
SEER
13 or
Higher
PG&E
571
43
123
48
223
55
32
48
SCE
531
33
58
46
195
108
38
53
SDGE
198
21
14
11
119
22
11
0
PG&E
1%
23%
14%
25%
9%
26%
28%
38%
SCE
1%
26%
19%
24%
11%
20%
27%
33%
SDGE
2%
33%
41%
42%
10%
38%
57%
0%
Precision Sample, n
IOU
12 12.99
11 11.99
10 10.99
9 to
9.99
8 to
8.99
Less
Than 8
Dishwasher
Dishwasher efficiency is characterized by energy factor (EF), which is the ratio of useful energy
output from an appliance to the total amount of energy delivered to it. The higher the EF, the
more efficient an appliance is.3 Based on a saturation of 67.3% from 2009 RASS, 1346 homes
are expected to have dishwashers and the breakdown per IOU along with associated precision
levels are shown in Table 4.
Table 4: Dishwasher Efficiency Assessment of Energy Factor
Sample Stratification & Precision
Levels, combined n=2,000
Precision Sample, n
IOU
Total
0.459
0.579
0.775
PG&E
678
76
520
82
SCE
470
67
329
74
SDGE
199
3
171
24
PG&E
-
21%
4%
23%
SCE
-
22%
6%
24%
SDGE
-
57%
6%
44%
2
RLW Analytics. 2005 California Statewide Residential Lighting and Appliance Efficiency Saturation
Study. August 23, 2005
3
http://www.energystar.gov/index.cfm?c=water_heat.pr_crit_water_heaters
Residential On-site Surveys
31
November 2011
Freezer
The number of freezers expected from a sample of 2000 is about 446 based on a 22.3%
saturation level. The distribution per IOU of these units according to unit energy consumption
(UEC), which is an estimate of average electricity consumption (kWh) per unit per year, can be
seen in Table 5 below.
Table 5: Freezer Efficiency Assessment of Unit Energy Consumption
Sample Stratification & Precision Levels, combined n=2,000
Precision Sample, n
IOU
PG&E
Annual
2001
Standard
Usage
225 to
Usage
425 to
Usage
625 to
Usage
825 to
Usage
1025 to
Usage
Usage
UEC
424.9
624.9
824.99
1024.99
1224.99
>1225
332
332
88
106
55
25
57
0
SCE
61
61
20
36
5
0
0
0
SDGE
52
52
12
5
19
17
0
0
PG&E
5%
3%
15%
14%
22%
38%
25%
0%
SCE
5%
7%
30%
19%
56%
0%
0%
0%
SDGE
8%
9%
39%
57%
33%
42%
0%
0%
Refrigerator
The RASS saturation for primary refrigerators is 100% so it is projected that we will find 2000.
The average annual nameplate UEC for refrigerator/freezers was obtained from the model
number matches to manufacturer data.
Residential On-site Surveys
32
November 2011
Table 6: Primary Refrigerator Assessment of Average Energy Consumption
Sample Stratification & Precision Levels, combined n=2,000
Precision Sample, n
IOU
350 to
<550
550 to
<750
750 to
<950
950 to
<1150
1150 to
<1350
1350
to
<1550
1550 to
<1750
1750
to
<1950
1950
to
2150
PG&E
251
443
166
84
54
28
10
15
10
SCE
170
390
98
54
17
0
0
0
0
SDGE
43
119
18
25
5
0
0
0
0
PG&E
10%
7%
13%
19%
36%
53%
80%
84%
80%
SCE
12%
6%
15%
21%
56%
0%
0%
0%
0%
SDGE
23%
10%
37%
30%
84%
0%
0%
0%
0%
Secondary Refrigerators
Using the RASS saturation of 23.9%, it is projected that 478 of the 2000 homes will have more
than one refrigerator. The parameter measured for secondary refrigerator was average
percentage above or below the 2001 standards for each unit which is calculated as follows:
% Relative to Std =
2001 Standard - UEC
2001 Standard
The 2001 Standard and UEC are measured in kWh/Year. For example, suppose the nameplate
annual energy consumption for a refrigerator is 550 KWh/Yr. The 2001 standard consumption
for this unit is 500 kWh/Yr. The percentage better or worse than 2001 standards is calculated as
follows:
500-550
50
== -10%
500
500
Thus, the annual energy consumption for this unit is 10% worse than 2001 standards.
Note, the secondary refrigerator sample size was very small in the 2005 CLASS study.4
4
RLW Analytics. 2005 California Statewide Residential Lighting and Appliance Efficiency Saturation
Study. August 23, 2005
Residential On-site Surveys
33
November 2011
Table 7: Secondary Refrigerator Assessment of Percent of NAECA Standard
90
0
48
8
0
0
4
SCE
170
0
47
65
38
10
10
0
0
0
0
SDGE
11
0
0
11
0
0
0
0
0
0
0
PG&E
-18%
28%
18%
23%
16%
0%
43%
50%
0%
0%
54%
SCE
-14%
0%
24%
17%
25%
49%
49%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
SDGE
Residential On-site Surveys
34
Pct Worse 175
to 199.9
40
Pct Worse 150
to 174.9
Pct Worse 25
to 49.9
73
Pct Worse 125
to 149.9
Pct Worse 0.01
to 24.9
35
Pct Worse 100
to 124.9
Pct Better 9.9
to 0
297
Pct Worse 75
to 99.9
Pct Better 35
to 10
PG&E
Pct Worse 50
to 74.9
IOU
pct of NAECA
standard
Precision Sample, n
Sample Stratification & Precision Levels, combined n=2,000
November 2011
7.
Secondary Data Collection
7.1
Collected Information from Other Studies
Data from the 2000 and 2005 CLASS Studies as well as the 2009 RASS Study. These studies
were all performed by KEMA so they will be free of cost or any wait period to obtain.
7.2
Relevant Standard-Industry and -Practice Considerations
The NPD Group can provide national sales data for the following appliances; room air
conditioners, refrigerators, freezers, washing machines, and dishwashers. The cost for this
would be $5,000 per category to a total of $25,000 because NPD does not have the energy data
for dryers.
Residential On-site Surveys
35
November 2011
8.
Presentation of Research Results
8.1.1
Draft Report
KEMA will produce a comprehensive draft final report that details the results of the study. The
major components of the draft final report will be the following:

A full characterization of the inventory for residential appliances and lighting at the
statewide level,

A summary of the saturation levels of appliances and lighting by efficiency levels at
the statewide level, and

A comparison of the previous study findings to the current study findings at the
statewide level.
The following is a brief outline of the draft report.
8.1.1.1
Introduction and Executive Summary
This section will serve as an overview of the approach and the findings. It will be suitable for
independent distribution to a non-technical audience as an executive summary of the study.
This section will contain only top-level findings and a brief overview of the methodology.
8.1.1.2
Study Methodology
This section will provide a detailed discussion of the methods employed in the study. This
section will assist the readers in understanding the approach and goals of the study. This
section will cover the methods used to develop the sample design, conduct customer recruiting
and on-site data collection, and analysis of the in-home data. This section will provide the
context needed for interpretation of the findings.
8.1.1.3
Characterization of Residential Appliances and Lighting Inventory
This section will present the study findings on the residential appliances and lighting inventory.
From this section, the reader will gain an understanding of the current state of residential
appliances and lighting.
8.1.1.4
Comparison to Previous Study and Other Similar Work
This section will present an analysis that will compare the saturation and efficiency levels to
those found in the 1999-00 Study and the 2004-05 Study. Findings will only be presented for
Residential On-site Surveys
36
November 2011
findings that are statistically different. The comparison analysis will be limited to the analysis
that was conducted for both the 1999-00 and 2004-05 Studies. KEMA will also review the
Market Share Tracking Studies to compare and contrast findings related to residential lighting
technologies. However, due to budget limitations, the emphasis of this task will be on a
comparison of the two lighting and appliance studies.
8.1.1.5
Database Development and Web-based Tool
This section will discuss the development of the database, the data sources and how that data
is analyzed by the Load Research System (LRS) tool. This section will also discuss the LRS tool
theory and how it should and should not be used. A section will be devoted entirely to the webbased tool, including user instructions, help information and disclaimers. This section will also
be presented on the website that hosts the tool, and will be in a format that can easily be
distributed as a stand-alone document.
Additional information about how the LRS system works can be found in the appendices.
8.1.1.6
Appendix
The appendix to the report will contain all supporting information, such as database
documentation, survey instruments, etc.
8.1.2
Final Report
KEMA will allow stakeholders ample time to review the final report material and provide
comments and input. KEMA will make reasonable changes to the report based on the
comments received. KEMA will produce a final report once all comments have been received.
KEMA will distribute as many bound copies of the report as requested by the client, not to
exceed 20.
8.1.3
Task 8.3: Public Presentation and Web-based Software Training
KEMA will formally present the results of the study to the project stakeholders at a time that is
convenient for the study group. During this meeting KEMA will also demonstrate the web-based
tool and provide training for those interested in this aspect of the presentation. KEMA will
arrange this as a web-based presentation to allow other parties the ability to participate in the
training activities.
Residential On-site Surveys
37
November 2011
Deliverable
Due Date
Draft Report
Final Report and Databases
Final Report Presentation and Webbased Tool Training
8.2
Project Management
8.2.1
Monthly Status Reports
The KEMA Team will provide monthly status reports that discuss the activities in the previous
month, for which the invoice represents, in addition to work planned for the upcoming month.
The KEMA Project Manager and the CPUC Project Manager will agree upon the format of the
monthly report and invoice detail early in the evaluation period.
The KEMA Team will also plan for bi-monthly conference calls (or as needed) to discuss project
activities. The KEMA Project Manager will provide agendas for these meetings no less than
three days in advance of the call. In addition, the KEMA Project Manager will summarize the
meeting events in project minutes that will be distributed to the project team no more than three
days following the call. For planning purposes, KEMA anticipates the cost of the conference
calls will be covered by the CPUC.
Deliverable
Due Date
Bi-monthly conference call agenda
Ongoing
Bi-monthly conference call minutes
Ongoing
Monthly Status Reports
Ongoing
Residential On-site Surveys
38
November 2011
9.
Research Coordination
9.1
Impact Work Order Coordination
The Residential On Site Metering study (WO21) will be coordinating it’s efforts with other work
orders in numerous ways:
Sample
Work orders 21 and 28 will utilize a shared sample design. From the general residential
population data KEMA will facilitate one large draw as the same intent is behind both studies.
CATI survey scripts and Outsourcing
By negotiating WO21 and WO28 CATI services KEMA will be able to negotiate a better price.
Also we will be able to utilize similar demographic questions on both CATI surveys. This will
allow us to find participants who are willing to allow us to do either the full WO21 site visit or the
abbreviated WO 28 lighting verification.
Scheduling staff and Field staff
Staff will be trained to do both work order field visits and sites will be scheduled for both work
orders for available sites that are in proximity. Coordinating the two field efforts will save a great
deal of money on field staff, scheduling staff and travel costs. KEMA will also be integrating a
separate Refrigerator metering study (WO35) into the WO21 site visits; 100 refrigerators will be
metered for a designated period of time. This will be an add on that will be offered in the field at
WO21 site visits. Participants will be offered an additional incentive to allow the refrigerator
metering. The work order for the refrigerator study will solely be responsible for the meter
removal costs.
9.2
Coordination with Process and Market Effects
Evaluations
The WO21 evaluation team is also coordinating with the Market Effects Market Transformation
team (WO54) and the Overarching Residential Process Evaluation team (WO11), as well as the
IOU’s related process and market studies teams. There are opportunities for each study to
collect data that can be utilized for the other study. It is also important to ensure that the same
Residential On-site Surveys
39
November 2011
market actors and end-use customers are not being contacted for multiple data collection
efforts.
9.2.1
Cross-Study Sample Coordination
To ensure that customers are not being over-contacted, all studies will submit their sample
frames to a centralized database that can track who is being contacted. In cases when multiple
studies are attempting to sample from the same customer segment which may be limited in
population, those studies will coordinate ahead of time to allocate the sample frame as needed.
9.2.2
Coordination with DEER Team
The DEER Residential Technology Group leads will be closely coordinated with the WO021
evaluation team as it relates to business and consumer electronics. These teams will continue
to work closely to support the DEER efforts on an ongoing basis.
9.2.3
Communication of Early Feedback to the IOUs
The IOUs and other stakeholders will be provided with the opportunity to review and provide
comments on this draft research plan. The timeline for this research plan has incorporated the
delivery of interim findings to provide early feedback to the IOUs. It is also planned to have
ongoing meetings and discussions with the IOUs to solicit feedback on approaches and the
need for interim results to support the IOUs’ needs. Through this process, the WO021 team will
attempt to meet the needs of the IOUs, with respect to providing timely and useful interim
results.
Residential On-site Surveys
40
November 2011
10.
Timeline
Task
Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12 Jan-13
Task 1: Project Management
Task 2: Research Plan Development
Task 3: Mine RASS Database
Task 4: Sample Design
Task 5: Preparation and Training
Task 6: CATI Surveys
Task 7: Recruiting
Task 8: On-Site Data Collection
Task 8.1 First half of sample
Task 8.1 Second half of sample
Task 9: Site Level Analysis/QC
Task 8.1 First half of sample
Task 8.1 Second half of sample
Task 10: Reporting
Residential On-site Surveys
41
November 2011
11.
Work Plan and Budget
11.1
Task-level Descriptions
11.1.0
Task 0: Development of Work Order
An initial work order was developed to facilitate planning. A subsequent work order amendment
will be developed one this research plan is approved to facilitate the implementation of the
study.
11.1.1
Task 1: Project Management
All project management of tasks will be tracked under this task and an associated set of
subtasks. Regular project meetings, invoicing and status reporting, and other fulfillment of
project management requirements will be separated by this task from the execution and
technical oversight of the following tasks. See discussion in Section 8.2 on project
management.
11.1.2
Task 2: Develop Detailed Residential On-site Research Plan
This study seeks to characterize building components that influence the energy consumption
across the current residential building population in California.
11.1.2.1
Needs assessment of residential baseline parameters, etc.
For each building characteristic of interest, at least one parameter has been identified as a
contributing factor to household energy consumption. For the purpose of planning this study, a
list of building characteristic variables will be assembled and, for each one, assigned a) a time
increment necessary to observe and record during on-sites in minutes, b) a priority rating
describing the importance to the WO021 parties on a scale of one to four (one being highest
priority), both short- and long-term, c) a quality rating of the best available information at present
on a scale of one to four (one having the least uncertainty), and 4) the estimated annual rate of
change for said variable. Building characteristic variables will be retained on said list unless all
WO021 parties agree that removal is appropriate. The building characteristics will be
categorized based on like elements as described in Section 2.1, Building Component Selection
Rationale.
Residential On-site Surveys
42
November 2011
A cost structure will be developed such that the base cost of an on-site is the travel time and
expenses on average across the state so that all data collection options are incremental on a
site basis. The time to collect characteristics data via observation is based on time measured in
minutes. The time to look up or derive the rated performance from those data is also expressed
as labor. The costs for measurement and metering will be separated such that a metering
oversight group or a small group within this work order can develop detailed plans and provide a
cost per sample point to the larger planning group and parties. The metering samples will likely
be nested within the larger study and sample sizes for measurement of each element will be
independent of the standard site cost. All cost and labor estimates in the draft scoping
spreadsheet as well as data sources are to be refined under this task. Other Work Orders can
fund incremental data collection activities but the structure is that primarily this pertains to
measurement and metering subsamples.
Parties must be identified for study supplementation as well as review. Inviting key public-owned
utilities to join the study (SMUD, LADWP) is a primary step. The scoping will have ED, Prime
Management, DMQC, and Consultant review as well as proposed scope review by the IOUs.
The plan will receive public comment and may raise additional important scope considerations.
The scoping will allow each party or individual to assign priority to some or all elements that will
be compiled and discussed. The project leadership will develop a final proposed list of data
collection and analysis needs as the deliverable of this task which will feed into the data
collection plan - Task 2.04.
11.1.2.2
Develop metering plan for systems or end-uses and sub-sampling
A separate small working group for metering as part of this Residential On-site study team will
discuss (and only develop if not redundant with crosscutting metering planning) preliminary
metering options. The overall M&V guidance team or crosscutting metering team may provide
information required for developing metering plans. Any plans funded by impact studies or
other sources should provide a memo outlining proposed metering and sample size to the
managers of this study planning team within a month of notification that this task has started.
The memo must outline the labor required to perform each step of metering for inclusion
including sub-sample design, installation, testing, removal, and any other on-site data collection
The deliverable is a set of detailed metering sections for all planned metering and information to
feed the data collection plan - Task 2.04 and sampling plan – Task 2.05.
Residential On-site Surveys
43
November 2011
11.1.2.3
Develop testing plan for systems or end-uses and sub-sampling
The Residential On-site study team will discuss the value and uncertainty of testing options and
propose high value test options. Testing and sub samples proposed by impact evaluation or
other teams will be part of the discussion and notification sent at the start of this task. Specific
one-time tests use different equipment, skills, and analysis methods than time series metering
under Task 2.02 and also may or may not be covered under any crosscutting measurement
guidance. These tests include but are not limited to assessment of building parameters or spot
power measurements. Some examples and considerations include:




Air Infiltration through pressurization/depressurization,
Duct Leakage through differential pressurization (longer duration methods should be
discussed as metering),
Solar exposure through Solar Pathfinder,
Spot power in each mode – On, Standby, Off-plugged in, etc.
– Useful for devices that are usually in standby, devices on timers, or always-on like
many set-top boxes
– Not informative for variable consumption end-uses like ACs, TVs, etc.
11.1.2.4
On-site data collection plan
The previous tasks will be compiled into a final on-site data collection plan inclusive of all
observations and measurements to be included. Elements not included will become part of a
long term summary of data needs - Task 2.08. Included elements will be part of the draft plan Task 2.06 and will outline characteristic and measurement details for each included element.
11.1.2.5
Sampling and Analysis Plan
The key issues for the Residential On-site study, as is similar to past CLASS studies, include
the scope of work and sampling. The previous CLASS sampled approximately 1,000 homes
and collected information on the building structure, the major appliances, and included a
comprehensive lighting inventory in a roughly one hour in-home visit. In addition to the IOUs,
SMUD was also included in the study and contributed to study funding. While all home types
were addressed, sample sizes for some market segments were fairly small. The following
adjustments to the previous CLASS sampling which was simple random with no intended
oversampling (SMUD add-on funding may have resulted in over sample of CZ 12) should be
considered:
Residential On-site Surveys
44
November 2011

Oversampling of multi-family dwellings
– 35% of random sample (based on 2010 RASS)
– May be under sampled for certain climate zones
– Inclusion of separate common area survey (19% 5+ unit apartments of random
sample)

Oversampling of manufactured homes to provide information for manufactured homes
programs
– 4% of random sample

Oversampling of new construction homes
– About 4% of random sample

Removing the lighting inventory component of the study since a comprehensive lighting
inventory was developed in the 06-08 evaluations and adding a consumer electronics /
plug load inventory
– Inventory of non-measured items was part of Past plug load study (small sample of
75 homes)
Subsampling options that were considered include:

Adding home infiltration (blower-door) and duct leakage (pressurization) testing to the
study, possibly in combination for the same sub-sample

Adding a metering component to the study for a sub-sample of homes, and if so, what
metering:
– Air conditioners
– Refrigerators
– Plug loads – mainly home office or entertainment equipment
– Metering Multiple end-uses to assess consumption for a large number of end-uses
utilizing Wi-Fi technology

Independent sub-samples that fit into multiple sites per day versus robust single on-site
per day with several additional components
11.1.2.6
Draft Overall Plan and Comment
This task covers all the contractor study team work that is directly related to the development
and approval of the detailed Research Plan and Project Schedule. The detailed planning effort
will identify the specific deliverables with their respective milestone schedules as the Research
Plan development proceeds. The study team will coordinate with the DEER Update
Residential On-site Surveys
45
November 2011
Management Team and the individual DEER Technology Group teams as they formulate and
evolve their prioritized list of updates, develop the measure technical specifications, and apply
the standardized measure naming convention. Additional research issues, questions, and
measures may emerge and add to the study’s scope as the detailed overarching Research Plan
is formulated and undergoes formal approval.
11.1.2.7
Final Plan
This task includes response to public comments on the plan and subsequent revision and
publication of the Research Plan.
11.1.2.8
Long Term Data Needs Summary
The research plan will also outline data needs that cannot be met based on current priorities
and budgets that should be studied in future years up to 2020. All the elements described in
Task 2.01-2.03 which are not included in the Final Plan will be listed in the long term needs
summary along with designation of high priority long term elements. The saturation rates of
many Residential Elements have been and will continue to be analyzed through RASS, these
and other elements may have been characterized in more depth through various other studies,
ranging from very recent to considerably older. The rate of change of each element in RASS
and past CLASS studies could be reviewed as well as estimates for other parameters from
other references for planning.
11.1.3
Task 3: Scoping of RASS Data Mining and Activities
11.1.3.1
Scoping of RASS Tasks to Support Task 2
Since a RASS was just completed in 2010, the activities around this study are limited to mining
the current data that was collected and to start planning for the next RASS. This task includes
the scope to plan data mining to support the Residential On-site Study. Additional planning can
be done in a new task to be added later. In general, the data mining activities would include:




Estimation of end-use saturations and equipment usage for sub-segments of the sample
Providing characterizations for sub-segments of the sample (such as large users)
Cross tabulating results of selected survey questions in response to focused analysis
questions
Providing weights to aggregate DEER results
Residential On-site Surveys
46
November 2011
A plan will be developed of short-term tasks that can be delivered before Task 2.04 and Task
2.05 are completed. Timing may be concurrent with DEER needs for some tasks. Longer
terms tasks will not be fully scoped until all planning to support Task 2 are completed in detail
and Task 3.02 is fully specified. Additional planning activities could include identifying the
appropriate time period between RASS studies; holding workshops to identify key issues that
could be addressed in the next RASS; developing methodologies to improve the RASS analysis
(such as incorporation of AMI data into the next RASS consumption analysis); and identifying
appropriate budgets that will be necessary to address all the issues that are typical to a RASS.
11.1.3.2
Execute Tasks to Support Task 2
TBD – Some options include:


Assessment of which types of data from the RASS are and are not likely to be reliable, in
order to inform the prioritization of CLASS activities
Population and sample frame development
11.1.3.3
Execute Longer Term Tasks
TBD
11.1.4
Task 4: Sample Design
In this task we will perform all the sample design activities presented above in Section 4.
11.1.5
Task 5: Preparation and Training
This task includes preparation of survey instruments (both CATI and digital on-site), recruiter
training, and on-site surveyor training. See Section 5 for a discussion of survey development
and training.
11.1.6
Task 6: CATI Telephone Surveys
KEMA will hire a subcontractor to conduct 5000 telephone surveys. See Section 5.1.
Residential On-site Surveys
47
November 2011
11.1.7
Task 7: Recruiting
KEMA will recruit 2000 customers for on-site surveys. Recruitment strategy is described in
Section 4.3. Recruiter coordination with field staff is discussed in Section 5.3.1.
11.1.8
Task 8: On-Site Visits
KEMA will conduct on-site surveys at 2000 homes. The various aspects of on-site data
collection are discussed in Section 5.
11.1.9
Task 9: Site Level Analysis
In this task the collected field data will be QC’d and analyzed. QC is discussed in Section 5.4.1.
Data analysis will include updating of efficiency look-up databases to include new equipment,
development of expansion weights, creating efficiency categories for the analysis, matching
appliances to model numbers and weighting to control for unmatched data, analysis of finalized
data sets, and comparisons to past studies. See Section 6 for a discussion of the analysis.
11.1.10 Task 10: Reporting
KEMA will prepare a draft report, a final report, public presentations, and software training as
discussed in Section 8.
Residential On-site Surveys
48
November 2011
11.2
Overall Budget and Accomplishments
11.3
Task-level Budgets and Staffing Plans
Task/Sub
task
Description
Task 0
Develop Work Order Overview
$20,000 Jarred Metoyer
X
Task 1
Prime Contractor PM and Coordination
$20,000 Jarred Metoyer
X
Task 2
Develop Detailed Research Plan
Task 3
RASS Data Mining
Task 4
Sample Design
Task 5
Preparation and Training
Task 6
CATI Surveys
Task 7
Recruiting
Task 8
On-Site Visits
Task 9
Site Level Analysis/QC
Task 10
Reporting
TOTAL
Residential On-site Surveys
Planned
Budget ($)
Task Lead
Task Support
Technical
Advisor
Brad Hoover;
$100,000 Jarred Metoyer Doug Norris
Fred Coito
Paula Ham-Su;
Paul Young; Paul
Higgins; Paolo
$60,000 Claire Palmgren Tanimoto
Metoyer
Paula Ham-Su;
Paul Young; Paul
Higgins; Paolo
$20,735 Claire Palmgren Tanimoto
Metoyer
$92,250 Brad Hoover
Jon Taffel; David
Cranford; Chris
Embry-Pelrine
Metoyer
$277,570 Claire Palmgren Geoff Barker
Amber Watkins;
$452,200 Doug Norris
Trudie Cabellero
David Cranford;
Chris Embry$1,525,835 Brad Hoover
Pelrine
Paula Ham-Su;
Paul Young; Paul
Higgins; Paolo
Tanimoto; Soofia
$190,000 Claire Palmgren Gilbert
$241,410 Claire Palmgren Fred Coito
Due
Date
Dec-11
Dec-11
Feb-12
Mar-12
Metoyer
Apr-12
Metoyer
Nov-12
Metoyer
Nov-12
Metoyer
Jan-13
Metoyer
Jan-13
$3,000,000
49
November 2011
Appendices
A.
Web-based Analysis Tool
The KEMA Load Research System (LRS) and its predecessor, the MBSS system, have been
used in many program evaluation studies. The methodology is described in The California
Evaluation Framework. Chapter 13 of that document describes the statistical methodology and
discusses a case study in substantial detail. Virtually all of the analysis discussed in the
Framework case study can be implemented in the KEMA System. See Chapter 5 of this
document.
In a typical program evaluation application, your goal is to assess the true savings of a specific
set of conservation projects implemented by your utility during a specified period of time. You
develop your sampling frame or population database by extracting a list of all projects from the
program tracking system. You include the estimate of the annual savings of each project that
was recorded in the tracking system as a suitable measure of the size of each project.
Using the KEMA System, you decide on the number of projects to be included in a sample,
develop a sample design that is stratified by the tracking estimate of savings, and select the
sample projects and the backups. Then you use an engineering team to assess the actual
savings of each sample project using appropriate project-specific measurement and verification
methods. Following evaluation principles, you may ask the team to assess both the gross
savings and the net savings.
Once the engineering team brings the results back to you for each sample project, you again
use the LRS to extrapolate the sample results to the program population. First you develop
case weights that reflect the final sample with complete data. Then you estimate the gross
realization rate and the net to gross ratio. Following evaluation principles, you know that the
gross realization rate is the ratio between the measured gross savings and the tracking savings,
so you use the stratified ratio estimation methods built into the LRS. Similarly, the net to gross
ratio is the ratio between the total net savings and the total gross savings, so again you can use
the stratified ratio estimation methods built into the LRS. You can treat this as a static study
within the LRS since, although the contractor may have used interval load data to help assess
the savings, your interest is on the gross and net savings and not on the underlying load data.
The final big challenge to the success of this project is to produce a user-friendly version of the
database and analysis engine for use on the Internet. In the previous study KEMA provided an
Access database that included an extensive number of analysis queries, which were used to
develop all results found in the statewide report. Additionally, we provided our MBSS Visual
Residential On-site Surveys
A-1
November 2011
Appendices
Basic tool. The MBSS VB tool interacts with the Access database for calculating ratio estimates,
saturation levels, sample sizes and error bounds. For this project we will provide the same
product, except that the tool would be accessed via the Internet by anyone wishing to use it, and
the user interface would be more user-friendly than the past application.
The only capability that would not be provided to the user is the ability to create new queries. In
the 1999-00 Study we trained users to create new queries for conducting their own “what-if”
analyses, but soon realized that this was a relatively difficult exercise for even advanced Access
and MBSS users. We also came to the realization that the queries developed by KEMA for
producing the findings in the report covered 90% or more of what users were hoping to access
the data for, therefore there was little need to create new queries. Based on this experience, we
believe that the queries used for the last project, and a few additional queries that KEMA will
add, will be sufficient for the great majority of users. For those few queries that will be
requested, KEMA will provide up to 5 added queries that will be updated on the Internet analysis
tool up to six months after the project report has been delivered.
Not having the need to train users on creating new analysis queries will greatly simplify the tool
KEMA will provide. Users will be able to “slice and dice” and drill down into the data far beyond
the data that will be provided in the statewide report. KEMA will produce easy to follow user
interfaces (web pages), on-line tutorial and training pages for using the analysis tool, in addition
to data disclaimers and other valuable information the user should know when using the
analysis tool.
The tailored application for this study will have the following capabilities:



Calculate ratio estimates, e.g., of the saturation level of a set of appliances, classified by
any available categorical variable such as climate zone, utility, residence type, or year.
Calculate the underlying sample sizes
Calculate the appropriate model-based error bounds
This software can be used to create one-way, two-way or multi-way tables categorizing the
market share of specified appliances and measures by any specified dimensions. The resulting
tables can be easily exported to Excel and displayed graphically.
Here are some examples of the type of statistics that one could obtain:


Average UEC of refrigerators
Saturation of refrigerators by age bins
This type of information can be developed for all sites, or for various classifications of
residences. Using the standard queries that we will provide in the database, the sites can be
Residential On-site Surveys
A-2
November 2011
Appendices
classified by any combination of the following variables (or any additional variables that are
deemed necessary):






11.3.1
Level of Efficiency
Utility Service Territory of Residence
Type of Residence
Size of Household
Square Footage
Income Level, etc.
Tool Development
Making the MBSS VB application and the lighting and appliance databases available to a large
number of users over the Internet will take some development time. There are also several
details that will need to be addressed before the scope of work can be fully defined. That said,
we outline our approach for making the MBSS/Access Tool web-based:
The current MBSS VB application is a stand-alone program compatible only with recent
versions of the Microsoft Windows operating system. Users are required to install the software
on their computers along with the two survey databases (one containing the appliance data, the
other containing lighting data). Because of the size of the two databases, distribution of the
software was often limited to direct distribution on CD-ROM. Results of the analyses were
stored back in the databases, requiring the user to open the files in Microsoft Access to view the
values.
This method of analysis required that users not only have the authority to install new software
on their machines (a privilege often restricted in many enterprise IT environments), but it also
required them have at least a rudimentary knowledge of MS Access in order to perform simple
investigations.
Our suggestion is to take the functionality of the original MBSS VB software and make it
available to users on the web. The current software will be modified to run as a CGI (Common
Gateway Interface) easily worked into a web site accessible from any standards compliant web
browser.
The main page will be simple enough to be easily incorporated into the site of the hosting
agency. Dynamic HTML will be used to allow the user to select which database, category and
analysis queries, grouping and summary variables they would like to evaluate as well as
Residential On-site Surveys
A-3
November 2011
Appendices
allowing them to specify any of the special options (Ratios for Each Site, Proportions of Total,
etc.).
Results will be calculated and displayed immediately in the user’s web browser, completely
eliminating the need for them to have access to any database software. Because the bulk of the
existing analysis engine will remain unchanged, a significant portion of the development time
(and expense) will be eliminated. Since the original application had a relatively simple user
interface, the adaptation to use within a web browser is expected to be straightforward.
Because the analysis engine will be implemented as a CGI, this will eliminate complexities
associated with the installation and configuration of specialized development languages such as
DreamWeaver, Perl or PHP. The only requirement for the web server will be that it be
configured properly to allow CGI scripting, and that it runs on a Microsoft Windows Server
platform (the use of a particular web server such as Apache or IIS is irrelevant as long as the
CGI requirement is met, which is the default configuration of most common web servers). The
actual database access will be performed by the CGI application and can be configured to use
any standard Windows database API such as ADO or DAO, according to the capabilities of the
server.
Changes to the user interface, such as the addition or removal of queries or variables, will be
easily implemented as simple updates to the underlying database and corresponding changes
to the front end web site.
11.3.2
Digital Data Collection
The primary goal of this procedure is to address the inefficiencies of data collection systems that
rely of paper survey instruments and data entry, as well as to improve the quality and
consistency of collected data. KEMA has pursued these kinds of advances in the past,
especially in residential projects, but the results fell short of expectations. Given the recent
influx of tablet computers, KEMA reassessed the potential of current technology.
KEMA explored a wide variety of options FileMaker on the Apple iPad 2 emerged as the best
option for several reasons shown in
Residential On-site Surveys
A-4
November 2011
Appendices
Table 8: Digital Data Collection Hardware Considerations
Devices:
Price:
I-pad-2
Motorola Xoom
$499
$599 unlocked
Operating System
Battery Life
Apple iOS 4.3
10 hrs
Android 3.0- Honeycomb
Does not support adobe flash
Panasonic H1 Field
$3,400
Windows 7 Professional
ASUS Eee Slate EP121
$1,050
Windows 7
$799
Windows 7
HP Slate 500
Samsung Galaxy
$299-$549w $29-$59/mo fee Android 2.2
Not available unlocked
HP Touchpad
Blackberry Playbook $450 16GB
Dell Streak 7
$350
$449 from Dell unlocked
Residential On-site Surveys
HP webOS
Blackberry Tablet Os
Android 2.2
A-5
Specs
Dual Core A5
16GB HD
10" screen
Photo and Video
10 hrs video playback 10.1" Screen
1 GHz dual core processor w/1GB of RAM
5 megapixel camera w/dual LED flash
6 hrs
Intel® Atom™ Processor Z540
Super Delux, Super Rugged too many features to list
3 hrs
12.1" Screen
Intel Core i5.470UM/1.33GHz
USB port
4 GB RAM
up to 5hrs
Intel Atom z540/1.86GHz
2GB RAM
USB
3 MP Camera
7 Hrs w/3g disabled Samsung S5PC110 1 Processor 512 MB RAM
1 sim spot and 1 micro SD
3 mp Camera w/flash
not listed
QUALCOMM Snapdragon APQ8060 1.2 GHz
10 hours
7" Screen
1 GHz dual core processor w/1GB of RAM
Not specified by Dell 1 GHz Dual-Core Processor w/1GB RAM
CNET was dissapointed7" screem
November 2011
Appendices
Table 9: FileMaker Go/Pro for iPad 2 WiFi Field Data Collection
Advantages
Disadvantages
Paperless
$500/unit cost for iPad 2 WiFi (16 GB)
Field data collection was as fast as paper
forms
$8/unit cost for FileMaker Go application
$300/per computer cost for FileMaker Pro
(only needed for form developers)
Small device with a long battery life- Currently
a single charge is enough for a full day of site
work with 20-30% to spare
Desirable device that might be prone to theft
KEMA-crafted survey instruments and
databases possible
The learning curve associated with the new
software- not enormous but still time
consuming
Very robust survey instrument allowing
validations, drop-down lists and skip patterns,
automated self QC
Can export files in a variety of well-established
file formats: xls, xlsx, csv, xml, htm, tab, dbf &
mer
The cost of the iPad 2 WiFi 16GB version is $499. The FileMaker Go software costs an
additional $40 per iPad 2. The cost of FileMaker Pro, the software designed to run on Windows
based PC’s, is $150 with the purchase of the FileMaker Go Application or $300 without. For
KEMA’s pilot project, a database was created using FileMaker Pro. The digital survey
instruments were also created using FileMaker Pro’s form creator utility. The form design utility’s
versatility allows digital instruments to range between very simple to extremely complex.
The KEMA team agreed several changes should be made to ensure the master database will
remain secure, reliable, and accessible to a large quantity and variety of possible authorized
users. The process implemented for a pilot study required only two to three people to access to
the master database. For the CPUC Class study there will be far more people needing to
access the database than the current database hosted on FileMaker Pro can handle.
Furthermore the pilot study process involved several manual steps of emailing, receiving, and
merging each surveyor’s daily site mini-database which can be automated with some network
changes.
According to the FileMaker website:
“FileMaker Server 11 securely hosts groups of FileMaker Pro users over a network or on
the web. It's fast, reliable, easy-to-use server software for securely managing your
Residential On-site Surveys
A-6
November 2011
Appendices
FileMaker Pro databases. When using FileMaker Go for iPhone and iPad, connect to
databases hosted on FileMaker Server for instant access to the most current
information. Any changes you make to your data while you’re on the go are immediately
accessible to your entire team. Connect to FileMaker Server using a local wireless
network or over the Internet using Wi-Fi or 3G. Choose FileMaker Server 11 Advanced
for Instant Web Publishing capabilities, ODBC/JDBC support, or to connect even larger
groups of FileMaker Pro users.
Info:







Setup & Administration Install FileMaker Server 11 in just 20 minutes or less.
Manage your databases remotely and automate administrative tasks.
Reliability & Availability Get anytime access to your data with 24/7 availability.
Protect your data with scheduled live back-ups.
Security Manage user access through external authentication via Active
Directory/Open Directory. Use SSL encryption for secure data transfer.
Web Publishing Create custom, data-driven websites using PHP or
XML/XSLT. Use the PHP Site Assistant for step-by-step help in creating PHP
websites.
Sharing NEW! FileMaker Server 11 Advanced does not limit the number of
FileMaker Pro clients. Connections are limited only by your hardware and
operating system.
Administrator Groups Create Administrator Groups and assign specific server
access privileges while retaining exclusive control of your databases.
Instant Web Publishing Easily publish and share databases over the web with
up to 100 concurrent web users.
ODBC/JDBC Connectivity Share data with other applications via server-based
ODBC/JDBC support. ODBC/JDBC clients can submit SQL queries to FileMaker
databases.”5
According to the SEEDcode website:
“GoZync by SEEDcode is a Push/Pull framework for FileMaker Go that lets you create
and edit records offline in FileMaker Go, and then push those records and edits to your
server when you have an internet connection: even a shaky 3G connection. GoZync lets
you deploy local files to your users: files that run ON the iPad or iPhone instead of from
FileMaker Server. Local files run faster (and open faster) than hosted files, making them
feel more like your users expect iOS apps to behave. With GoZync you can deploy
several smaller, single-purpose apps rather than asking your users to run the mother
ship system on their mobile devices.”6
5
http://www.filemaker.com/products/filemaker-server/, 8/17/2011
6
http://www.seedcode.com/cp-app/ste_cat/gozync, 8/17/2011
Residential On-site Surveys
A-7
November 2011
Appendices
According to the F5 Networks website:
“The BIG-IP Edge Portal app for iOS devices provides simple, streamlined access to
web applications that reside behind BIG-IP APM, without requiring full VPN access, to
simplify login for users and provide a new layer of control for administrators. Using BIGIP Edge Portal, users can access internal web pages and web applications securely, and
administrators can seamlessly add iPhone and iPad mobile device management to their
already existing BIG-IP infrastructure.
The BIG-IP Edge Client app provides not only full SSL VPN access from iPhones and
iPads, but also accelerated application performance when it’s used with BIG-IP Edge
Gateway. Administrators can maintain granular control with F5’s Visual Policy Editor,
and users experience fast downloads and quick web access with the integrated
optimization and acceleration technologies built into BIG-IP Edge Gateway. IT no longer
has to provision and manage multiple units to ensure their corporate applications are
available, fast, and secure to iPhone and iPad users.” 7
7
http://www.f5.com/pdf/white-papers/secure-iphone-access-tb.pdf, 8/17/2011
Residential On-site Surveys
A-8
November 2011
Appendices
Table 10: List of Queries Available for All Appliance Categories from 2005 CLASS
Appliance Category
Parameters
Sub Parameters
COMMON TO ALL
Utility
END-USES*
Type of Residence
Total People in Home
Total Adults in Home
Income
Home Age Range
Total Heated Floorspace
Rent or Own
Who Pays Electric
Who Pays Gas
Primary Language
Climate Zone
Cooling
Cooling Efficiency
Average age and bins-estimated
Type of System
Cooling Proportions
Central or Space
Primary System Type
Cooling Tons
Size in Tons
System Age
Space or Central
*
SEER bins
SEER Compared to Standards
Dishwasher
Dishwasher Efficiency
Average age and bins-estimated
System Age
Homes with dishwashers
*
Average energy factors
Energy factor bins
Dryer
Estimated Average age and bins
Fuel Type
Home with dryers
Envelope
Attic insulation R-value bins
Wall Construction Type
Floor insulation R-value
*
Low E
Pct of walls insulated
Pct With Slab on Grade
Wall construction type
Wall insulated
Wall Insulation R-value
Window frame type
Window frame type by number of frames
Residential On-site Surveys
A-9
November 2011
Appendices
Table 10: List of Queries Available for All Appliance Categories from 2005 CLASS
Appliance Category
Parameters
Sub Parameters
Freezer
Estimated avg age and bins
Freezer Age
Homes with 1 or 2 freezers
Freezer Type
Manufacture Matched Volume
Freezer Size
Size of Freezer
*
Type of Freezer
Freezer Efficiency
Average Unit Energy Consumption and Bins
Energy Standard Comparison
Energy Standard Comparison - Bins
General
Demographic proportions
Homes remodeled in the past 10 years
Homes with planned remodel in the next 2 years
Pays electric bill
Pays gas bill
Planned remodel type
Remodels involving hardwired lighting
Thermostat types
Type of remodel
Heating
Capacity bins
Fuel
Estimated age (bins)
Central or Space
Fuel Types
Type
Heating proportions
System Age
Number of heating systems
*
Primary System Type
Heating Efficiency
Average AFUE and bins
Pool
Fuel Types
Fuel Type
Pump Horsepower
*
Estimated Manufacture Date
Refrigerator Type
Manufacture Date
Refrigerator Size
Homes With Two or Three Refrigerators
Refrigerator Age
Refrigerator Types
*
Refrigerator
Volume
Volume (Bins)
Refrigerator Efficiency
Average Unit Energy Consumption
UEC Compared to Energy Star
UEC Compared to Federal Standard
UEC Compared to Federal Standard (Bins)
Residential On-site Surveys
A-10
November 2011
Appendices
Table 10: List of Queries Available for All Appliance Categories from 2005 CLASS
Appliance Category
Parameters
Sub Parameters
Unit Energy Consumption (Bins)
Secondary Refrigerator
Manufacture Date
Refrigerator Type
Manufacture Date - Estimated
Refrigerator Size
Type Of Refrigerator
Refrigerator Age
Volume
*
Volume (Bins)
Secondary Refrigerator
Average Unit Energy Consumption
Efficiency
UEC Compared to Energy Star
UEC Compared to Federal Standard
UEC Compared to Federal Standard (Bins)
Unit Energy Consumption (Bins)
Spa
Homes With Spas
*
Spa Fuel Type
Washing Machine
Average Age and Bins
Type of Washer
Homes With Washing Machines
System Age
Type of Washer
*
Washing Machine
Average Energy Factor
Efficiency
Energy Factor (Bins)
Water Heater
Average Age and Bins
Fuel
Average Size and Bins
Size
Fuel Type
Unit Type
Tank Wrap
System Age
Water Heater Proportions
*
Water Heater Efficiency
Average Energy Factor
Energy Factor Bins
Residential On-site Surveys
A-11
November 2011