Monitoring and Evaluation Plan for
California’s Weather Based
Smart Irrigation Controllers Program
Submitted to
Dr. Harry Cummings
Program Evaluation
University of Guelph
Guelph, ON N1G 2W1
Submitted by
Inem Chahal
April 2013
Table of Contents
1.0 Acronyms…………………………………………………………………………….3
2.0 Introduction……………………………………………………………………….. .4
2.1 Context and Background………………………………………………….4
2.2 The CWBSIC Project……………………………………………………...4
2.3 Goal of the Evaluation …………………………………………………….5
2.4 Objectives of the Evaluation………………………………………………5
3.0 Program Logic Model………………………………………………………………7
4.0 Evaluation Matrix………………………………………………………………….11
5.0 Evaluation Details………………………………………………………………….16
5.1 Type of Evaluation ……………………………………………………….16
5.2 Evaluation Design …………………………………………………………16
5.3 Evaluation Methods…………………………………………………….….17
5.4 Evaluation Team……………………………………………………….…..17
5.5 Evaluation Schedule……………………………………………………….18
5.6 Budget ...........................................................................................................18
6.0 Conclusion ….………………………………………………………………………19
References………………………………………………………………………………20
Monitoring and Evaluation Plan- CWBSICP
2
Acronyms
CDWR
-
California’s Department of Water Resources
CUWCC
-
California Urban Water Conservation Council
CWBSICP
-
California’s Weather Based Irrigation Controllers Program
CWSC
-
California Water Service Company
FG
-
Focus Group
KII
-
Key Informant Interview
WBIC
-
Weather Based Irrigation Controllers
Monitoring and Evaluation Plan- CWBSICP
3
2.0 Introduction
2.1 Context and Background
Water scarcity is defined as, “Water scarcity occurs where there are insufficient
water resources to satisfy long-term average requirements. It refers to long-term water
imbalances, combining low water availability with a level of water demand exceeding the
supply capacity of the natural system.” (Water Scarcity, 2008, p.1). This is an issue
because we are facing fresh water scarcity in many parts of the world, mainly because of
our high consumption levels. If the current consumption rates continue, then by 2025, the
demand for fresh water will exceed the supply (UN Assessment of Fresh Water
Resources, 1997).
The state of California is no exception to this, it is facing a water crisis of a fairly
epic proportion. In fact, by the year 2020, the demand for water will exceed the supply
(Comprehensive Resources Management Plan, n.d). More than eighty percent of the
water is used for agricultural purposes, the remaining twenty percent is allocated to
industrial and residential use. Furthermore, more than half of the water in residential
homes is used for outdoor purposes (i.e. water lawns, wash cars, etc.) (Water Use in the
California Residential Home, 2010). In the case of California, both ground water and
surface water levels are under threat. The water table in the aquifers is decreasing
because when surface water supplies are limited, people turn to pumping ground water.
In this region, aquifers will most likely be depleted within the next sixty to one hundred
years (Yu, 2011).
2.2 The CWBSICP Program
In response to this, CDWR introduced the installation of WBIC’s in Northern and
Southern California, in order to help households consume less water for outdoor
activities. The CDWR defines these controllers as, “Smart controllers estimate or
measure depletion of available plant moisture to operate an irrigation system that
replenishes water as needed while minimizing access. A properly programmed smart
Monitoring and Evaluation Plan- CWBSICP
4
controller makes irrigation adjustments throughout the season with minimal human
intervention” (Water Use in the California Residential Home, 2010, p.4).
Through “free exchange events” households in these two regions had the
opportunity to trade in their old controllers for these WBIC’s. Additionally, close to
three- quarters of the installed devices were free and the remaining twenty- five percent
received a rebate or voucher to cover the cost of the installed WBIC’s.
Households in Northern and Southern California are the main participants and
beneficiaries of this project, because reducing their demand for water will also help them
financially, provided that the price of water remains the same.
CDWR is the sole funding agent for this project. The total funding allocated for
Southern California is $1.8 million and $1.7 million for Northern California, a total of
$3.5 million USD (Water Use in the California Residential Home, 2010).
2.3 Goal of the Evaluation
This evaluation aims to thoroughly review and evaluate all aspects of this program and
ensure that the proposed program outcomes are being achieved. This evaluation will
provide a non- biased, independent assessment of the performance of the program. The
results of the evaluation will be presented to CDWR so it can build on the lessons learned
and improve future CDWR projects and programs. Additionally, the assessment will
demonstrate to citizens of California that CDWR is seriously committed to this project
and in ensuring that the program’s objectives are attained. The outcomes of this
evaluation will also provide evidence to donors, including CDWR that the funded
program is accomplishing the results that it was meant to achieve.
2.4 Objectives of the Evaluation
The specific objectives for the CWBSICP program evaluation are as follows:

To secure funding to further this program

Conclusions derived from the evaluation will help highlight the strengths and
weaknesses of the project

To measure the success rate of WBIC in helping households reduce their water
demand
Monitoring and Evaluation Plan- CWBSICP
5

Identify which methods of implementation were the most effective and
community success stories of the CWBSICP program

Efficient strategies and lessons learned from this project will serve as examples
for future projects funded by CDWR

Compare the results of the evaluation to the desired results indicated in the initial
project proposal and explain why there were differences, if any.
Monitoring and Evaluation Plan- CWBSICP
6
3.0 Program Logic Model
The program logic model is a diagram, which represents the program. It provides a
graphic explanation of the relationship between the program’s inputs, outputs, outcomes
(or impacts) with respect to the specified assumptions and risks (See Table 1)
(Cummings, 2007).

Inputs are resources, including personnel which contribute to the activities of the
program (Powell, 2005)

Activities are what the program does with the inputs to achieve its goal (Define
How a Program Will Work, 2012)

Outputs are the direct products of program activities (Define How a Program
Will Work, 2012)

Outcomes (or Impacts) are the benefits for participants during and after program
activities (Define How a Program Will Work, 2012)
The relationship between the above variables can be illustrated as follow: the inputs must
be in place for activities to occur. The activities must be finished before the outputs are
created. Finally, the outputs must be formed before the outcomes/ impacts are to be
achieved.
Inputs
Activities
Outputs
Outcomes
Impacts
(Source: Cummings, 2007)
Table 1: Assumptions and Risks
Inputs
Activities
Outputs
Outcomes
Impacts
 The Government of California continue to support this project through various water
management policies, low risk

California Department of Water Resources continues funding the project, medium risk

Households continue to accept the installation of the water smart irrigation controllers
[means that where rebates/ vouchers and free distributions are not applicable, households
are willing to buy these controllers with their own money], high risk
Monitoring and Evaluation Plan- CWBSICP
7
Table 2: Program Logic Model
*Logic Model Adapted from Azerbaijan Project Logic Model (Cummings, n.d.)
Inputs
 Funding provided by the
Activities
1. Project coordinators ensure proper
Outputs
1. Proper installation
California Department of Water
implementation of the project by
of these controllers in
demand for water in
California are
Resources
overseeing the installations of the
households will be
Northern and Southern
more aware of
California
the seriousness of
Communities are
water scarcity,

Outcomes
Reduced overall

Impacts
Communities in
 Alamada County Water District
controllers; Recruitment and training of assured
 Contra Costa Water District
the support staff to ensure proper
2. If households are
 Santa Clara Valley Water
installation of the irrigation controllers
successful in
better aware of the
ultimately
2. Monitoring the success of the water
conserving more
importance of water
leading to an
 Sonoma County Water Agency
smart irrigation controllers in reducing
water using the new
conservation within
increase in the
 City of Long Beach
water use in households1
controllers than
their communities
installation of
The community
these water smart
installation then
members will be more
irrigation
similar projects could
aware of the goals and
controllers,
be implemented in
objectives of the
reducing water
other states
project, leading to
wastage within
District
 Los Angeles Department of
water and Power
 City of Santa Monica
 San Diego County Water
Inputs
before their


Technique used to monitor the success involves comparing the following two components: one full year of pre- installation
water consumption versus one full year of post- installation water consumption
1
Authority
Activities
Outputs
Outcomes
Impacts
 City of Long Beach
3. Support staff to hold community
3. Households have a
greater acceptability of the state
 2 project Coordinators
meetings to talk about the project and
greater knowledge of
the water smart
 Support Staff
its significance
the goals and
irrigation controllers
behavior
 Community Members
4. Support staff and community
importance of the
Skills gained by the
(more water
volunteers to contact participating
project
support staff to
friendly) of
households to get their feedback on the
4. Thoughts and
achieve the goals of
households who
benefits and downsides of the project
opinions of
the project
previously did not
5. Set up free exchange events in a)
households on the
The California
take measures to
community centers b) community
project are obtained
Department of Water
conserve water
colleges and c) internet signup which
5. Households will be
Resources will be
within their homes
allows households to replace their old
able to obtain the
more aware of the
irrigation controllers with the water
new water smart
actual effectiveness of
smart irrigation controllers
irrigation controllers
the program and
6. Set up implementation methods for
6. Households will be
determine what kind
households (through free distributions ,
able to receive these
of changes should be
direct installation and rebates/
new controllers for
made
vouchers)
free (conditions
2
Inputs
2



Changed
apply)
If they exchange their old irrigation controllers with the new ones
Monitoring and Evaluation Plan- CWBSICP
9
Activities
Outputs
7. Send out customer service
7. Customer
satisfaction surveys via mail
satisfaction surveys
8. Evaluation of the California Water
received by
Smart Irrigation Controller Project
households
Outcomes
Impacts
8. A thorough
assessment of the
efficiency and
effectiveness of the
project is conducted
Monitoring and Evaluation Plan- CWBSICP
10
4.0 Evaluation Matrix
Evaluation matrix ensures that all relevant issues and covered. In table 3, the following issues will be covered:




Relevance: Refers to the extent to which the program caters to the goal of the organization (Cummings- matrix definitions,
2013).
Efficiency: Refers to the extent to which the program’s inputs and activities were planned to ensure that the outputs were
produced at the least cost (Cummings, 2013)
Effectiveness: Refers to the extent to which the outputs and outcomes catered to the overall program goal (Cummings, 2013).
Access/ Reach: Refers to whether everyone in the community had a fair chance of being part of and reaping the benefits of the
program (Cummings, 2013).
Table 3: Evaluation Matrix for the Process Evaluation
Issue
Relevance
Evaluation
Question
1. To what
extend does
the program
help the
CDWR
achieve its
goal of
reducing
water use
within
households?
2. Do the
agencies
that are part
of the
Indicators
Data Required
1.Determine if
the number of
days that saw
water shortage
changed or not
after the
implementation
of the program
1. Information
on the number
of days that saw
water shortage
before the
implementation
of CWBSICP
and the number
of days that saw
water shortage
after the
program
2a. Government
of California’s
budget for the
program
2. Number of
CDWR projects;
CDWR’s budget
for CWBSICP
Source of
Data
1. CDWR,
CUWCC,
CWSC,
households
2. CDWR,
Ministry
officials
Method of
Collection
1. KII with
CDWR
officials,
CUWCC,
CWSC,
households,
beneficiaries;
document
review
2. KII with
stakeholders
and
government
Analysis
Responsibility
Timing
1-3
KII’s and
FG’s will be
conducted
and
documents/
data will be
reviewed
and
compared
1-3
Document/
data review
will be
conducted by
the evaluation
team
1-3
Through
out the
project
lifetimeongoing
basis
KII’s and FG’s
will be
conducted by
the evaluation
team
May
2013
Issue
Relevance
(Continued)
Evaluation
Question
Indicators
program
think it’s a
priority or
not?
2b. CWBSICP
included in the
California Water
Plan
3. Is there
evidence
supporting
that the
introduction
of the
CWBSICP
addressed a
need in the
community?
3a.
Stakeholders,
including
households
agree that the
program is
essential in
addressing the
community’s
needs.
3b. Increased
access to smart
controllers and
less water
wasted in
household
irrigation
activities.
3c. Conduct
household
surveys to
determine if
there is a need to
install irrigation
controllers
Monitoring and Evaluation Plan- CWBSICP
Data Required
Source of
Data
Method of
Collection
Analysis
Responsibility
Timing
documents
review
3. List of
objectives met
by the
CWBSICP,
reduction of
outdoor water
use
3. CDWR,
households,
beneficiarie
s, master
plan and
baseline
data
3. KII with
CDWR
officials; FG
with
households,
beneficiaries
and document
review
12
Issues
Efficiency
Evaluation
Question
4. Were the
hired
personnel
on time?
Were there
any issues?
5. Were
there
enough
smart
controllers
to install?
Were they
delivered on
time? What
were some
of the
difficulties
encountered
, if any?
Indicators
Data Required
4.Organizational
structure and a
list of all
positions
recruited
4. Information on
recruited
positions,
organization chart
5a. # of water
smart
irrigation
controllers
available and
the # that are
actually needed
5b. # of
households
that could not
have the
controllers
installed
because of
delivery
failures
5. List of
suppliers and
contractors that
were responsible
for the delivery
and installation of
controllers
Monitoring and Evaluation Plan- CWBSICP
Source of
Data
4. CWBSICP
documents
and
monitoring
data
Method of
Collection
4. Document
data and
review; KII
with CDWR
officials
5. CWBSICP
documents
and
monitoring
data
5. Document
and data
review; KII
with
households,
suppliers and
contractors
Analysis
4-6
KII and FG
will be
conducted
Documents
and data
reviewed
and
compared
Responsibility
4-6
KII’s and
FG are to be
conducted
by the
evaluation
team
Documents
and data
will be
analyzed
and
compared
by the
evaluation
team
Timing
4-6
Through
out the
project
lifetimeongoing
basis
May
2013
13
Issues
Efficiency
(Continued)
Evaluation
Question
6. What was
the cost
associated
with the
CWBSICP?
Was it
comparable
with other
projects
conducted
by CDWR?
Indicators
6. Cost of
installing smart
controllers
versus the
installation of
water efficient
toilets (another
program by
CDWR)
Monitoring and Evaluation Plan- CWBSICP
Data Required
Source of
Data
6. Cost data from 6. CWBSICP
other programs
documents
funded by CDWR and
monitoring
data, CDWR
Method of
Collection
6. Document
data and
review; KII
with CDWR
officials
Analysis
Responsibility
Timing
14
Issues
Access
and Reach
Evaluation
Question
7. Are the
smart
controllers
accessible to
all
households?
Indicators
Data Required
7. Percentage of
people in the
community that
can access the
facilities which
supply these
controllers
7. Number of
households that
experienced no
difficulty in
locating a facility
that distributed
the controllers
8. Are the
benefits of
the project
available to
all members
of the
community?
8. Percentage of
beneficiaries
that come from
low income
households
8. Details of the
beneficiaries
economic status
9. Is the
CWBSICP
teaching the
intended
audience?
9. Actual
beneficiaries
reached vs. the
original planned
Monitoring and Evaluation Plan- CWBSICP
Source of
Data
7. Facilities set
up by CDWR,
households
Method of
Collection
7. KII with
households
and
beneficiaries
8. U.S. Census
(Zip code
statistics)
8. Postal code
ranking of
neighborhoods
according to
income per
year
Analysis
7-9
KII and
FG will be
conducted
Document
s and data
reviewed
and
compared
Responsibility
7-9
KII and FG
will be
carried out
by the
evaluation
team
Timing
7-9
May 2013
Documents
and data
will be
analyzed by
the
evaluation
team
15
5.0 Evaluation Details
5.1 Type of Evaluation
As decided by the evaluation team, the evaluation of the CWBSICP will be a summative
evaluation. This type of evaluation takes place during and after the implementation of the
project and tends to represent outcomes, as opposed to formative evaluation. It is appropriate for
this type of program because summative evaluation is associated with more objective and
quantitative methods. Quantitative and objective techniques are much more useful for this
situation because it is important to measure the amount of water saved through the use of these
smart controllers. Finally, summative evaluation will be used because the evaluation team would
like to assess whether the program has achieved its goals and whether there were any unforeseen
consequences. Through this type of evaluation, the team would also like to determine: What
lessons were learned? How can this program be improved? (Types of Evaluation, n.d.)
5.2 Evaluation Design
The evaluation plan was developed using a participatory approach because of the following
reasons (Participatory Methods, 2011):

Gives a voice to the marginalized population

Teachers people how to take more control over their lives

Provides a clearer picture of what works and what does not from the perspectives of
those directly involved

Provides a clearer picture of the needs of staff and beneficiaries
The evaluation will use a quasi- experimental approach because it will allow the evaluation
team to use comparison groups rather than randomly assigned control groups as the baseline.
These comparison groups will fall into any of the following three categories (QuasiExperimental Evaluation, n.d.):
1) Eligible non- participants in the same community
2) Individuals similar to the participants from an existing data base that contains the
outcome measures of interest
3) Individuals in a matched comparison site who would have been eligible for the
program if it was initiated in that location/ site
5.3 Evaluation Methods
This evaluation will use both qualitative and quantitative methods in order to assess the outputs
and outcomes of the CWBSIC program. The methods of the summative evaluation are listed in
Table 4 below:
Table 4: Evaluation Methods
Summative Evaluation

Document review

Data review

Key informant interview

Focus group discussion

Questionnaires for households

Budget analysis
5.4 Evaluation Team
*Required number of evaluators: 1

Minimum of a Bachelors degree in management or equivalent.

Strong written and oral communication skills in English is required

Previous experience in evaluation management

Strong track record in managing evaluations
*Required number of support staff to conduct telephone surveys, KII’s, focus groups/
questionnaires: 10

Graduate students whose research focuses on water conservation

Strong verbal and written communication skills

Previous experience in conducing surveys and dealing with the public is an asset
Monitoring and Evaluation Plan- CWBSICP
17
5.5 Evaluation Schedule
A general schedule is provided below for the summative evaluation. The schedule in table 5 may
be subject to minor chances throughout the duration of the evaluation. The total time dedicated to
this evaluation will be 5 months.
Table 5: Proposed Schedule for Evaluation
Timelines
March/ April 2013
Activities
Develop and finalize tools for Summative
Evaluation
July 2013
Summative Evaluation
-Document review
-Field Visit (10- 15 days)
-Analysis of Data
-Budget Analysis
-Submit Evaluation Report
5.6 Budget
Table 6: Budget
Proposed Budget Activities
Amount in $ (USD)
1. Document/ Data Review
$ 1500
2. Develop and Finalize the Evaluation Work
$2000
Plan
3. Develop Data Collection Tools/ Guides
$1200
4. Analyze/ Synthesize
$5000
5. Field Mission and Data Collection
$2000
6. Prepare Draft Report
$2000
7. Staff Salaries
$1000
8. Finalize/ Present Evaluation Report
Total
Monitoring and Evaluation Plan- CWBSICP
$10, 000
$24, 700
18
6.0 Conclusion
This evaluation plan includes all the necessary steps and information required when conducting
the assessment for the CWBSICP. The following information was covered in this report and is
necessary to take into account before the actual evaluation begins:

A brief background information on the issue of water scarcity in California

Introduction to the CWBSICP

The goal of the evaluation

The objectives of the evaluation

Program Logic Model

Evaluation Matrix- identifies the key issues

The type of evaluation that should be used for this program

The type of evaluation design that should be used for this program

Personnel required to conduct this evaluation

The schedule outlining when the main steps will be taken during the evaluation

The planned budget for this evaluation
Monitoring and Evaluation Plan- CWBSICP
19
References
(2010). Water use in the California residential home.
Cummings, H. (n.d.). Azerbaijan project. Unpublished raw
data, School of Environmental Design and Rural
Development, University of Guelph, Guelph,
Ontario, Canada. Available from CourseLink.
Retrieved from https://courselink.uoguelph.ca
Cummings, H. (2007). Monitoring and evaluation plan for
riwash.
Cummings, H. (2013, March). Evaluation matrix. Guelph, Ontario.
Define how a program will work: The logic model. (2012,
October 5). Retrieved from
http://nnlm.gov/outreach/community/logicmodel.html
Participatory methods. (2011). Retrieved from
http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/
EXTISPMA/0,,contentMDK:20190347~menuPK:412148~pagePK:148956
~piPK:216618~theSitePK:384329,00.html
Powell, E. (2005, march 21). Logic model: A framework for
program planning and evaluation. Retrieved from
courselink.uoguelph.ca
Quasi experimental evaluation. (n.d.). Retrieved from
http://www.le.ac.uk/oerresources/criminology/msc/unit8/page_05.htm
Types of evaluation. (n.d.). Retrieved from
http://evaluationtoolbox.net.au/index.php?
option=com_content&view=article&id=15&Itemid=19
Un assessment of freshwater resources. (1997, June 23).
Retrieved from
http://www.un.org/ecosocdev/geninfo/sustdev/waterrep.htm
Water scarcity. (2008, 11 26). Retrieved from http://www.eea.europa.eu/themes/water/featuredarticles/water-scarcity
Yu, J. (Director) (2011). Last call at the oasis [Web].
Retrieved from
http://www.cbc.ca/passionateeye/episode/last-call-atthe-oasis.html
Monitoring and Evaluation Plan- CWBSICP
20