Water Security initiative:
Customer Complaint
Surveillance Vendor
Integration Forum
Summary
Dallas, TX
February 28, 2011
Disclaimer
The Water Security Division of the Office of Ground Water and Drinking Water has reviewed and
approved this document for publication. Neither the United States Government nor any of its employees,
contractors, or their employees make any warranty, expressed or implies, or assume any legal liability of
responsibility for any third party’s use of or the results of such use of any information or process
described in this report, or represents that its use by such party would not infringe on privately-owned
rights. This document is not a substitute for applicable legal requirements, nor is it a regulation itself.
The word “should”, as used in this document, is intended solely to recommend or suggest and does not
connote a requirement. Mention of trade names or commercial products does not constitute endorsement
or recommendation for use.
Any mention of trade names, companies, products, or services in this guidance does not constitute an
endorsement by the U.S. Environmental Protection Agency of any non-federal entity, its products, or its
services.
Questions concerning this document or its application should be addressed to:
Brian Pickard
U.S. EPA Office of Ground Water and Drinking Water
U.S. EPA Headquarters, Ariel Rios Building
1200 Pennsylvania Ave., N.W.
Mail Code 4608T
Washington, D.C. 20460
Pickard.Brian@epamail.epa.gov
or
Nelson Mix
U.S. EPA Office of Ground Water and Drinking Water
U.S. EPA Headquarters, Ariel Rios Building
1200 Pennsylvania Ave., N.W.
Mail Code 4608T
Washington, D.C. 20460
Mix.Nelson@epamail.epa.gov
Table of Contents
SECTION 1.0: OVERVIEW ................................................................................................................................... 1
SECTION 2.0: CUSTOMER COMPLAINT SURVEILLANCE OVERVIEW AND APPROACH ............... 3
SECTION 3.0: IMPLEMENTING CUSTOMER COMPLAINT SURVEILLANCE ...................................... 7
SECTION 4.0: VENDOR BENEFITS ................................................................................................................. 11
SECTION 5.0: UTILITY CASE STUDIES ......................................................................................................... 13
5.1
CINCINNATI PILOT, GREATER CINCINNATI WATER WORKS (GCWW) ........................ 13
5.2
PHILADELPHIA PILOT, PHILADELPHIA WATER DEPARTMENT (PWD) ........................ 13
5.3
DALLAS PILOT, DALLAS WATER UTILITIES (DWU) ......................................................... 16
SECTION 6.0: LOOKING FORWARD .............................................................................................................. 17
SECTION 7.0: REFERENCES........................................................................................................................... 179
List of Figures
FIGURE 2-1.
CWS MONITORING COMPONENTS ................................................................................................... 3
FIGURE 2-2.
ARCHITECTURE OF A CWS .............................................................................................................. 4
FIGURE 2-3.
WS INITIATIVE PROGRAM OVERVIEW ............................................................................................. 5
FIGURE 3-1.
FUNNEL, FILTER, FOCUS .................................................................................................................. 7
FIGURE 3-2.
AET OUTPUT ................................................................................................................................... 8
FIGURE 3-3.
CINCINNATI HYDRA MAP DISPLAY ................................................................................................. 9
FIGURE 3-4.
CINCINNATI PILOT EXAMPLE ALERT NOTIFICATION..................................................................... 10
FIGURE 5-1.
PWD SYSTEM ARCHITECTURE ...................................................................................................... 14
FIGURE 5-2.
EXAMPLE OF PWD EDA MATRIX.................................................................................................. 14
FIGURE 5-3.
PWD DASHBOARD ......................................................................................................................... 15
FIGURE 5-4.
DWU CCS ARCHITECTURE ........................................................................................................... 16
v
List of Acronyms
AET
AWWA
CCS
CSC
CS/IMTech
CSR
CUSUM
CWS
DWU
EDA
EDS
EPA
GCWW
GIS
IT
IVR
PWD
SCADA
SME
SMS
WS initiative
Alarm Estimation Tool
American Water Works Association
Customer Complaint Surveillance
Computer Sciences Corporation
Customer Service/Information Management Technology
Customer Service Representative
Cumulative Sum
Contamination Warning System
Dallas Water Utilities
Event Detection Algorithm
Event Detection System
U.S. Environmental Protection Agency
Greater Cincinnati Water Works
Geographic Information System
Information Technology
Interactive Voice Response
Philadelphia Water Department
Supervisory Control Automated Data Acquisition
Subject Matter Expert
Short Message Service
Water Security initiative
vii
Section 1.0:
Overview
The Water Security (WS) initiative is a U.S. Environmental Protection Agency (EPA) program that
addresses the risk of intentional contamination of drinking water distribution systems. Initiated in
response to Homeland Security Presidential Directive 9, the overall goal is to establish recommendations
for the design and deployment of contamination warning systems for voluntary adoption by drinking
water utilities.
The EPA Customer Complaint Surveillance (CCS) Vendor Integration Forum was held on Sunday,
February 28, 2011 from 12:00–3:00 p.m. at the Intercontinental Hotel in Dallas, Texas. The Forum was
jointly hosted by EPA’s Office of Ground Water and Drinking Water and the American Water Works
Association (AWWA) in conjunction with the AWWA 2011 Customer Service/Information Management
Technology (CS/IMTech) conference. The purpose of this Forum was to meet with Vendors and
Integrators to speak with them regarding the WS initiative and CCS and to create an open dialogue for
product development.
The Forum was broken into two sessions. The first session provided an overview of the WS initiative and
the CCS component, including aspects of implementing a CCS system. The second session consisted of
presentations by three WS initiative Pilot Utility subject matter experts (SMEs), including a Question &
Answer session. The report below outlines the proceedings of this forum.
1
Section 2.0:
Customer Complaint Surveillance Overview
and Approach
After a brief introduction from Alan Roberson of AWWA, Brian Pickard, Team Leader, EPA Office of
Ground Water and Drinking Water, introduced the concept and reason behind the development of the WS
initiative, the nature of contamination threats, and the Contamination Warning System (CWS).
Mr. Pickard noted that CCS is important due to distribution system vulnerabilities, including accessibility
via commercial and residential service connections, fire hydrants, and finished water storage areas. He
also noted it is impossible to eliminate all access to a system, but key system components can be
hardened. Mr. Pickard stated that consequences of contamination include adverse impacts on public
health, loss of water for public safety uses, economic damages, and loss of consumer confidence. He also
stated that an attack using contaminants is likely to achieve multiple terror objectives. Such an attack
would not necessarily have to produce casualties in order to be considered successful by an organization,
reiterating the importance of having a CWS in place.
CWS design objectives include:






Detecting a broad spectrum of contaminant classes
Achieving spatial coverage of the entire distribution system
Detecting contamination in sufficient time for effective response
Reliably indicating a contamination incident with a minimum number of false-positives
Providing a sustainable architecture to monitor distribution system water quality
Emphasizing dual-use aspects
A CWS involves the active deployment and use of monitoring technologies and strategies and enhanced
surveillance activities. These activities collect, integrate, analyze, and communicate information in order
to provide a timely warning of potential water contamination incidents and to initiate response actions to
minimize public health and economic impacts. An integrated CWS will utilize water quality monitoring,
public health surveillance, sampling and analysis, enhanced security monitoring, and CCS. While CWS
requires this multi-pronged approach, as shown in Figure 2-1, CCS is the main focus of the Vendor
Integration Forum.
Figure 2-1. CWS Monitoring Components
3
A CWS is not merely a collection of monitors and equipment placed throughout a water system to alert of
intrusion or contamination. A CWS includes different information streams that are captured, managed,
analyzed, and interpreted to recognize potential contamination incidents, with time to respond effectively.
These data sources, when used concurrently, should support and augment each other such that the chances
of detecting a contamination incident are better than using any one information source on its own. While
a CWS will ultimately be designed by the utility, cooperation with partners, including vendors and
integrators, is vital to the successful adoption of the CWS. CWS operations can be broken down into two
phases: Phase I is Routine Monitoring and Surveillance; Phase II is Consequence Management and is
implemented only when there has been a possible determination of contamination. Figure 2-2 represents
this architecture.
Figure 2-2. Architecture of a CWS
Concerning Voluntary National Adoption of CWS, the WS initiative comprises work in three areas, or
phases:



Develop a conceptual design for a system that achieves timely detection and
appropriate response to drinking water contamination incidents to minimize public
health and economic impacts;
Demonstrate, test, and evaluate the conceptual design in CWS Pilots at drinking
water utilities;
Issue practical guidance and conduct outreach to promote voluntary national
adoption of effective and sustainable drinking water CWSs.
Figure 2-3 represents these three areas.
4
Figure 2-3. WS initiative Program Overview
EPA is now implementing an “inside-out” approach to utility adoption of CWSs. EPA is urging vendors
and integrators to implement this approach by including the required coding into established call
management and work management software programs and systems. This approach allows for a more
efficient approach to CCS adoption. The necessary mechanisms to track, transfer and analyze CCS data
are already built into many existing software systems. If utilities realize that they have this capability to
“plug-and-play”, it is more likely that they will adopt many, if not all, aspects of the CCS component.
5
Section 3.0:
Implementing Customer Complaint
Surveillance
This portion of the presentation was guided by the CSC team: Scott Weinfeld, a CSC Project Manager,
Adam Haas, Lead Analyst, and Doron Shalvi, Senior Application Developer and Technical Lead for the
Cincinnati Pilot.
Mr. Weinfeld began by giving a concise description of the features and characteristics of an ideal CCS.
According to EPA, a “CCS encompasses the customer complaint collection process, data management,
data analysis and anomaly detection of customer complaints, notification of anomalies, and investigation
procedures.” It is important to keep this definition in mind when developing new CCS systems and
making modifications to existing applications. Mr. Weinfeld also spoke regarding “Funnel, Filter, and
Focus”. Collecting complaints into a single location, or funneling them by using Interactive Voice
Response (IVR) and Asset Management Systems (including email, phone calls, and web forms) is a best
practice and AWWA standard. In filtering calls, he noted the importance to recognize and filter out nonsignificant calls, such as those relating to rusty or dirty water, cloudy water, or water pressure. The final
step is to “focus”: notify the appropriate utility personnel of detected anomalies. Figure 3-1 represents
this concept.
Figure 3-1. Funnel, Filter, Focus
Mr. Weinfeld described the characteristics of contamination that can be detected by CCS. A CCS system
is looking for anomalous patterns, aberrant signal-to-noise ratios, or deviations from the norm in either
time or space. These contamination events involve contaminants with aesthetic properties not associated
with a benign cause, and not associated with common utility water quality problems that may have similar
complaint descriptions, such as rusty or dirty water, cloudy water, or water pressure. Water
7
contamination will also display temporal clustering within minutes or hours of each other, and spatial
clustering, both linked by characteristics of the distribution system . Implementing a CCS system
provides the possible dual-use benefit of detecting anomalies in common water quality problems in
addition to those associated with possible contamination incidents.
The core functionalities that comprise a CCS component must include at a minimum:




A mechanism for a utility to baseline their data and establish alarm thresholds,
Near real-time analysis of data using automated surveillance algorithms and code,
Alert notification of anomalies, and most importantly, and
Easy integration with other utility systems.
This last point is especially important. EPA doesn’t want to re-invent the way utilities do business, but
rather integrate CCS into their existing business model. EPA thinks the best people to talk to drinking
water utility personnel are the vendor/integrator community who are deeply embedded in the existing
systems and processes already in place at these utilities.
Mr. Haas discussed creating a baseline for which utilities can establish a threshold. Baselining the data
involves reviewing and analyzing historic data, and identifying “normal” complaint volume and setting
thresholds for alarms. Mr. Haas spoke regarding how this process was completed in Cincinnati, and
about the Alarm Estimation Tool (AET). The AET is a simple tool created by EPA that assists
developers, integrators, and utilities in analyzing their historic water quality data and establishing
thresholds. Figure 3-2 shows an example output from the tool. The tool is free, and is available on
EPA’s website at: http://water.epa.gov/infrastructure/watersecurity/techtools/index.cfm.
Figure 3-2. AET Output
8
Once the utility has established their thresholds, they can build their Event Detection System (EDS).
Multiple automated algorithms running in parallel executed in near real-time can detect both temporal and
spatial clustering of water quality complaints. Once the utility has detected the anomalous water quality
complaints, they must be able to provide actionable notifications to appropriate personnel, and provide
procedure for closing out alarms. In implementing CCS at the Cincinnati Pilot, several activities
occurred:



Historical data was analyzed and reviewed,
Normal and anomalous complaint volumes were established, and
Data resources were identified, such as:
o Contact management systems (IVR, Customer Service Representative (CSR)
characterization, and work/asset management system),
o Algorithm code and servers, together comprising the EDS,
o Notification system (e-mail and text messaging), and
o Distribution system GIS.
Types of alarms were defined by analysis algorithms and data stream at the Cincinnati Pilot. The
algorithm code (Java) was designed to detect temporal anomalies at varying thresholds utilizing 1, 2, and
7 day scans and Cumulative sum (CUSUM) and spatial anomalies by monitoring neighborhoods, zip
codes, pressure zones, and other hydraulically significant areas. Spatial analysis, using Hydra MAP
Geographic Information System (GIS) application, is shown in Figure 3-3. It is important that all of the
different information available to the utility be displayed, for example as a dashboard.
Figure 3-3. Cincinnati Hydra Map Display
Mr. Shalvi spoke regarding notifications and notification content, investigation procedures, and
information technology (IT) considerations. He stated that immediate alert notifications must be provided
to appropriate personnel in near real-time. Possible communication channels are short message service
(SMS), email, auto-dialers, and interface pop-ups achieved through Supervisory Control Automated Data
9
Acquisition (SCADA). It is also important that the notification contains actionable information such as
details as to the type of alert, alert location, and investigation procedure instructions. At the Cincinnati
Pilot, CCS alarm notifications are received by the Water Quality department in email format. The
notification includes the algorithm that detected the anomaly, the dates, times, and locations of the
complaints and annotated information regarding each call that led to the alert. Figure 3-4 shows an
example of a notification received.
Figure 3-4. Cincinnati Pilot Example Alert Notification
Typically included on the e-mail (but not shown above) are the date, time, location and description of
each complaint to aid the investigation of the alarm. In Cincinnati, theses details were later added to the
e-mails, at the request of the investigating utility staff.
Investigative procedure may include:






Interviewing callers;
Reviewing call data and analyzing the location and nature of the complaint(s) to
determine if calls are clustered;
Reviewing plant and distribution system operations and assessing the potential for
operational changes to impact aesthetics of water;
Reviewing distribution system work: Breaks and repairs may impact color, turbidity,
and odor of the water;
Reviewing and analyzing available water quality data in the vicinity of the call cluster
to determine if there is a link;
Closing out the alert.
When integrating a CCS, there are many considerations to take into account. First, it is important to
design a flexible architecture using a standardized approach and ensuring that parameters, such as
notification templates, notification recipients, and thresholds, are configurable by managers. Second, it is
important to leverage existing systems and capabilities, including existing applications, the network
infrastructure, programming expertise, and security. Thirdly, it is important that alarm data is stored so
that the utility has the ability to go back and perform a retrospective analysis. And finally, it is important
to account for any time delay in receiving alerts.
10
Section 4.0:
Vendor Benefits
EPA wants to enlist vendors in a novel approach to providing CCS functionality to the water sector. EPA
is taking an approach in which AWWA and EPA drive demand by encouraging voluntary adoption of
CCS into their members’ operations while vendors and integrators supply the products and services that
make them easy to adopt. EPA & AWWA will create the demand; vendors and integrators create the
supply. Benefits are not only to the end users, AWWA and EPA, but also to the tool developers and
system integrators.
There are multiple documented cases in which customer complaints have been an early indicator
contamination incidents. In Milwaukee, Wisconsin, a telephone survey of 613 households provided
estimates on the total number of persons in Milwaukee experiencing mild, moderate, or severe illness as a
result of the cryptosporidiosis outbreak in 1993. When disease case estimates were adjusted for normal
background diarrheal disease rates, investigators estimated that 403,000 residents of the five-county area
experienced illness caused by the cryptosporidiosis outbreak. A high volume of customer calls could
have indicated problems with the distribution system, but there was no system in place at the time to alert
supervisors. Utilities will be looking for this functionality. An early warning saves a utility time and
money, and has the potential to save lives. A better knowledge of water quality complaint prevalence
also provides a dual-use benefit in that more knowledge can help improve normal operations. Off-theshelf solutions offer low implementation cost for utilities, and cost is key consideration when utilities are
upgrading or replacing a call or work management system.
As mentioned, AWWA and EPA are encouraging voluntary adoption of CCS, therefore creating demand.
In addition to this, there is a growing body of research and guidance. EPA has invested millions of
dollars into WS initiative Pilot CCS programs. Vendors and integrators stand to benefit from existing
pilot implementations in Cincinnati, Dallas, New York City, Philadelphia, and San Francisco. In addition
there is a dual-use, all hazards emphasis on improvements in monitoring and security, with updated
federal guidelines and recommendations. EPA has already issued guidance regarding the benefits of
implementing CCS, and AWWA and EPA have published articles on CCS implementation and benefits
(USEPA 2005). AWWA and EPA are committed to promoting voluntary adoption over the long term.
For tool developers, CCS functionality provides a more appealing product offering with market
differentiation, easy implementation (most features are already in existing tools), and easier
implementation for system integrators. For system integrators, CCS functionality provides consistent
implementations, efficiency of off-the-shelf solutions, and low implementation cost for the utility.
Adding CCS functionality to call and work management products will allow vendors to have their product
stand out from other products in the marketplace. Finally, products and services may already have the
desired features – but just need to be “re-visioned” with an eye to providing a CCS component.
11
12
Section 5.0:
5.1
Utility Case Studies
Cincinnati Pilot, Greater Cincinnati Water Works (GCWW)
The presentation began explaining how CCS implementation affected GCWW Operations. Mr. May
stated that historically, operational staff had to be constantly aware and tuned-in to notice data trends, but
now the capability to detect exists in a standardized system and there is a standard process for
investigation. A standardized system helps when there is employee turnover. The general benefits to
implementing the CCS are that the service-oriented architecture is accepted by GCWW, and GCWW was
able to “re-use” the web service. GCWW was Generation 1, or an ‘elementary algorithm’, but now CCS
is ingrained within the utility. However, if GCWW did it again, they would require more spatial
intelligence, and development of the platform in .net instead of Java. GCWW is also interested in
geographical RSS feeds for GIS.
Responding to a question regarding the GIS interface, Mr. May stated that work orders will automatically
show up on their GIS Hydra Map display.
5.2
Philadelphia Pilot, Philadelphia Water Department (PWD)
Charles Zitomer, PWD, spoke regarding the approach taken at PWD, in implementing CCS, CCS and
EDAs, and the PWD CWS dashboard. In approaching implementation of CCS, PWD opted to leverage
existing resources such as their GIS investment, skilled workforce, existing data systems, and business
procedures. PWD decided to improve and streamline customer response, conduct field investigations
more efficiently, and integrate new systems. A new work order management system (Cityworks), and a
CWS Dashboard, both of which are Web-based applications, allow remote access to assist with CCS
investigations.
In implementing CCS, PWD is using Cityworks as the work order management system. It is a Webbased application with an open architecture database and a spatially compliant data structure that allows
remote access. It was configured to maximize its benefits for the CWS. All water quality service
requests are funneled through the Call Center and captured in Cityworks. All work activities on the
distribution system will be captured in Cityworks, which will help with associating water quality alarms
with potentially related distribution system activities. Figure 5-1 represents the system architecture in
place at PWD.
13
Figure 5-1. PWD System Architecture
Use of EDAs allows PWD to get information to decision makers quickly, twenty four hours a day, seven
days a week. There is an emphasis on GIS and attribute data. EDAs are used to continuously process
water quality complaint information and determine whether a pre-defined complaint threshold has been
exceeded. An EDA’s purpose is to bring a potential water quality event to the attention of a human
investigator using real-time customer complaint information. At PWD, EDA thresholds were developed
based on water quality complaint type(s), spatial location or clustering, leveraging GIS, and date and
time. An example of an EDA, using 1-, 2-, and 7-Day Scan Windows, and monitoring multiple water
quality parameters, is shown in Figure 5-2.
Figure 5-2. Example of PWD EDA Matrix
Currently under development, the CWS Dashboard and the EDAs will assist the investigator by
automatically displaying only the relevant service request and work order types from the Cityworks data
and alerting them to activity occurring within a specific radius of the complaint(s). The CWS Dashboard
and the EDAs will also reduce the investigation and response time by promptly providing relevant
14
information during alarm conditions. The business processes for the PWD Dashboard are yet to be
implemented. There are currently issues with night crews that are ‘not so much into water quality’ and
can now receive training. The PWD Dashboard has the dual-use ability to show planned capital
improvement projects, such as line replacement and flushing, as well as customer complaints. An
example of the PWD dashboard is shown in Figure 5-3.
Figure 5-3. PWD Dashboard
Ultimately, PWD’s approach to leveraging existing GIS and Cityworks will allow the department to
capture all water quality complaint calls and work performed in the distribution system on the same map.
It will also allow PWD Operations to detect system issues and failures, improve customer service, and
provide PWD with the opportunity to standardize business processes and protocols for water quality
complaint investigation.
Responding to a question regarding CSR access to the Dashboard, Mr. Zitomer stated that CSR access to
the Dashboard is limited to because there is a concern that CSRs may give too much information back to
the customer about other complaints. Information regarding things such as main breaks may be made
available to the CSR.
Responding to a question regarding map information being available to the laboratory, Mr. Zitomer stated
that the laboratory would have a history of all the calls which would help with the issuance of
notifications occurring sooner.
Responding to a question regarding the algorithm, Mr. Zitomer stated that the work order system provides
customer complaint information which is then filtered based on type and is stored in the “warehouse”
until it is later, when an alert is issued and it is analyzed.
15
Responding to a question regarding time of occurrence of complaints and other issues, Mr. Zitomer stated
that all necessary information, including time, location, and nature of complaint, will be available on the
map. If a complaint was received before a main break, that information would be available to the user.
Responding to a question regarding resolution information, Mr. Zitomer stated that lab results from LIMS
are entered back into the service request in Cityworks for the complaint. There is a distinction between
311 call centers and an in-house call center.
5.3
Dallas Pilot, Dallas Water Utilities (DWU)
Cassia Sanchez, DWU, discussed DWU’s implementation of CCS. DWU is in the process of
implementing SAP and changing from one water quality category to approximately ten. Implementation
of this system should be completed by June 2011. DWU has shallow reservoirs. Because of this, DWU
switched to ozone for treatment in 2005 and eliminated the use of carbon for treatment of taste and odor
characteristics. This resulted in a significant reduction in the number of customer complaints received.
DWU is educating the 311 center CSRs, making changes in organization management, as well as with
water purification processes. For DWU, education coupled with system improvements represents a dualuse benefit. Figure 5-4 illustrates the CCS architecture currently in place at DWU.
Figure 5-4. DWU CCS Architecture
Responding to a question regarding system operation, Ms. Sanchez stated that the system has been
developed over the past year and was not fully up and running, though it would be integrated June of
2011.
Responding to a question regarding the characteristics that would be analyzed, Ms. Sanchez stated that
DWU would be analyzing complaints based on appearance, taste and odor, color, and other parameters.
Responding to a question regarding algorithm refinement, Ms. Sanchez stated that all algorithms would
be reviewed this summer, following full deployment of the system.
16
Section 6.0:
Looking Forward
Alan Roberson, AWWA, spoke to future collaborations, stating that looking forward there will be
collaboration with AWWA, EPA, utilities, and the private sector, with input from stakeholders and their
ideas. AWWA and EPA hope to start information sharing, and hold follow-up webinars in the future. As
WS initiative Expansion Pilot activities near completion, there will be additional information, lessons
learned and guidance generated from CCS efforts at DWU, PWD, the New York City Department of
Environmental Protection, and the San Francisco Public Utilities Commission. Finally, EPA and AWWA
believe that successful integration of CCS software by vendors will require a two-way conversation, and
so plan to start a dialogue with these private sector partners.
17
18
Section 7.0: References
U.S. Environmental Protection Agency. 2005. WaterSentinel System Architecture, EPA 817-D-05-003.
19