Customer Complaint Surveillance
Vendor Integration Forum
Jointly Hosted by the U.S. EPA and
American Water Works Association
February 27, 2011
12:00 - 3:00 p.m.
AWWA CS/IMTECH Conference
Dallas, TX
Presentation Outline
1. Overview and Approach of Customer Complaint
Surveillance
a. Program and approach
b. National adoption
2. Implementing Customer Complaint Surveillance
a. Features and characteristics
b. Integration and benefits
2
Overview and Approach of
Customer Complaint Surveillance
3
Distribution System Vulnerabilities
•
•
Large, complex, and accessible:
–
Commercial & residential service connections
–
Fire hydrants
–
Finished water storage
Impossible to eliminate all access, but …
–
•
key system components can be hardened
Difficult to contaminate an entire city via the
distribution system, but …
–
fairly easy to impact small sections or individual
buildings
4
Nature of Contamination Threats
•
•
Consequences of water contamination:
–
Adverse impacts on public health
–
Loss of water for public safety uses
–
Economic damage
–
Loss of consumer confidence
An attack using contaminants …
–
is likely to achieve multiple terror objectives
–
does not have to produce casualties to be successful
–
may be perceived as an especially serious threat by the
public
5
Water Security (WS) initiative
•
EPA’s WS initiative program addresses the risk of
intentional contamination of drinking water
distribution systems.
•
The initiative was established in response to
Homeland Security Presidential Directive
(HSPD) 9, under which the Agency must,
“develop robust, comprehensive, and fully
coordinated surveillance and monitoring
systems . . . for . . . water quality that provides
early detection and awareness of disease, pest,
or poisonous agents” (Jan 30, 2004).
6
CWS Design Objectives
•
Detection of a broad spectrum of contaminant
classes.
•
Achieve spatial coverage of the entire
distribution system.
•
Detect contamination in sufficient time for
effective response.
•
Reliably indicate a contamination incident with
a minimum number of false-positives.
•
Provide a sustainable architecture to monitor
distribution system water quality.
•
Growing emphasis on Dual-use aspects.
7
CWS Monitoring Components
Customer
complaint
surveillance
Water
quality
monitoring
Enhanced
security
monitoring
Integrated
Contamination
Warning System
Public
health
surveillance
Sampling
and
analysis
8
CWS Architecture
Phase II.
Consequence Management
Phase I.
Routine Monitoring & Surveillance
Online Water Quality
Possible
Determination
Credibility Determination Actions
confirm or rule out contamination
and may include:
· Site Characterization
Public Health
Sampling and
Analysis
Enhanced Security
· Outside data sources
· Event
Detection
· Laboratory confirmation
· Initial
Trigger
Validation
Response Actions protect public
health during the investigation
process and may include:
Remediation and Recovery
restores a system to normal
operations and may include:
· System characterization
· Remedial action
· Post-remediation activities
· Isolation
· Flushing
Customer Complaints
· Public alerts/notifications
9
National Adoption
The WS initiative comprises work in 3 areas:
1) Develop a conceptual design for a system that achieves
timely detection and appropriate response to drinking water
contamination incidents to minimize public health and
economic impacts;
2) Demonstrate, test, and evaluate the conceptual design in
contamination warning system pilots at drinking water
utilities;
3) Issue practical guidance and conduct outreach to promote
voluntary national adoption of effective and sustainable
drinking water contamination warning systems.
10
National Adoption
11
A New EPA Approach…
•
Implement an “Inside-Out” Approach
–
Integration of the required coding into established call
management and work management software would
allow for a more efficient “inside-out” approach to CCS
adoption
–
Necessary mechanisms to track, transfer and analyze
CCS data are already built into existing software
systems
–
Make utilities more likely to adopt aspects of a
contamination warning system – more like a plug-andplay system
12
Implementing
Customer Complaint Surveillance
13
Customer Complaint Surveillance
•
•
•
Features and Characteristics
Integrating Customer Complaint Surveillance
into Contact/Work Management Systems
Vendor Benefits
14
What is Customer Complaint Surveillance?
“Customer Complaint Surveillance (CCS)
encompasses the customer complaint collection
process, data management, data analysis and
anomaly detection of customer complaints,
notification of anomalies, and investigation
procedures.”
15
FUNNEL
Features & Characteristics
Primary Source:
Water Utility Contact Center
Secondary
Secondary Source:
Source:
Other Agencies
Other Utility
Departments
Non Water Quality
Related Complaints
FILTER
(As determined by
Customer)
Non Water Quality
Related Complaints
Customer
Complaint
System
Contact
Center
(As determined by
CSR)
Water Quality Complaints
Related to System
Operations
Operations
Staff
(Main Breaks,
FOCUS
Maintenance, etc.)
Water Quality Specialist
analyzes remaining
complaints for indications
of possible contamination
Water
Quality
Specialist
16
Features & Characteristics
•
What does water contamination look like?
–
Aesthetic (organoleptic properties) not associated with
a benign cause.
–
Not associated with common utility water quality
problems that may have similar complaint
descriptions.
 Rusty/dirty water
 Cloudy water
 Water pressure
–
Temporal clustering  complaints linked by time.
–
Spatial clustering  complaints linked by distribution
system characteristics.
17
Features & Characteristics
•
Characteristics of a Customer Complaint System
–
Provide a mechanism for a utility to “baseline” their
data and establish alarm thresholds
–
Near real-time analysis of data using automated
surveillance algorithms and code
–
Alarm notification of anomalies (alerts)
–
Allow for easy integration with other utility systems
18
Features & Characteristics
•
Provide a mechanism for a utility to “baseline”
their data and establish alarm thresholds
–
Review / analyze historic data
 Analysis of Trigger Values for GCWW paper
–
Identify “normal” complaint volume
–
Set thresholds for alarms
 Alarm Estimation Tool (AET)
19
Baselining Historic Data
http://water.epa.gov/infrastructure/watersecurity/techtools/index.cfm
20
Establish Alarm Thresholds
21
Features & Characteristics
•
Detection of Anomalies
–
Collect all complaints in a single location
–
Identify anomalous volume of water quality calls
 Multiple automated algorithms running in parallel
 Executes in near real-time
 Temporal clustering
 Spatial clustering
–
Provide actionable notifications to appropriate
personnel
–
Provide procedure for closing out alarms
22
Integrating CCS
•
Implementing CCS at GCWW
–
Reviewed / analyzed historical data
–
Established “normal” vs. anomalous complaint volume
–
Identified data resources
 Contact Management Systems
» Interactive Voice Response (IVR)
» CSR characterization (keystrokes)
» Work/Asset Management System
 Built Event Detection System (algorithms)
 Notification (email)
 GIS – distribution system (Hydra)
23
Features & Characteristics
•
Types of Alarms
–
Analysis Algorithms (at GCWW)

Algorithm code (Java)
» Temporal anomalies (GCWW thresholds)
 1 day scan
 2 day scan
 7 day scan
 CUSUM
» Spatial anomalies
 Neighborhoods
 Zip codes
 Pressure zones
 Other hydraulically significant areas
24
Spatial Analysis
GCWW - Hydra Map
25
Features & Characteristics
•
Provide notifications to appropriate personnel
–
–
–
Immediate alert notification (near real-time)
Possible communication channels
 Short Message Service (SMS)
 Email
 Auto-dialer
 User Interface pop-up (SCADA)
 Twitter
Notification contains actionable information
 Alert type details
 Location details
 Procedural instructions (investigations)
26
Features & Characteristics
•
CCS Alarm Notification
Water Quality Department receives an email alerting of a
CCS alarm and begins the investigation.
Notification includes:
• Algorithm that detected the anomaly
• Dates and times of complaints
• Locations of complaints
• Annotated information about the call
27
Features & Characteristics
•
Investigation Process:
–
–
Interview callers
Review call data
 Analyze the location and nature of the complaint to
determine if calls are clustered
–
Review plant and distribution system operations
 Assess the potential for operational changes to impact
aesthetics of water
–
Review distribution system work
 Breaks and repairs may impact color, turbidity, and odor
of the water
–
Review available water quality data
–
 Analyze water quality data in the vicinity of the call cluster
to determine if there is a link
Close out alarm
28
Integrating CCS
•
IT Considerations
–
Design flexible architecture
 Standardized approach
 Parameters configurable by managers
» Notification templates
» Notification recipients
» Thresholds
–
Leverage existing systems and capabilities




–
–
Existing systems and applications
Network infrastructure
Programming expertise
Security
Store alarm data for retrospective analysis
Account for time delay in receiving data
29
Vendor Benefits
•
End Users
•
AWWA/EPA
•
Tool Developers
•
System Integrators
30
Vendor Benefits
•
End Users
–
Utilities will be looking for this functionality





Saves time
Saves money
Saves lives
Dual-use for improving normal operation
Off-the-shelf solutions offer low implementation cost
for utilities
31
Vendor Benefits
•
AWWA/EPA
–
–
–
–
AWWA and EPA are encouraging voluntary adoption
(creating demand)
Growing body of research and guidance
Consistent offering from vendors provides the supply
Vendors benefit from existing pilot implementations
 Cincinnati, Dallas, New York City, Philadelphia,
San Francisco
–
Dual-use, All hazards emphasis
 Improvements in monitoring and security
 Federal guidelines / recommendations
32
Vendor Benefits
•
Tool Developers
–
CCS functionality provides:
 More appealing product offering
» Market differentiation
 Easy implementation – most features already in
existing tools
 Makes it easier for system integrators to implement
•
System Integrators
–
CCS functionality provides:
 Consistent implementations
 Efficiency of off-the-shelf solutions
 Low implementation cost for the utility
33
Utility Case Studies
•
Implementing Customer Complaint Surveillance
–
–
–
Bryan May, Greater Cincinnati Water Works
Charles Zitomer, PE, Philadelphia Water Department
Cassia Sanchez, PE, Dallas Water Utilities
34
Greater Cincinnati Water Works
35
Philadelphia Water Department
36
Philadelphia Water Department
Approach
•
Leverage Existing Resources
–
–
–
–
•
•
•
GIS Investment
Skilled Workforce
Existing Data Systems
Existing Business Processes
Improve and Streamline Customer Response
Conduct Field Investigations more Efficiently
Integrate with New Systems
–
–
Cityworks (Work Order Management System)
CWS Dashboard to assist with CCS Alarm
Investigations
 CWS Spatial Dashboard is a Web Application allowing
remote access
37
Philadelphia Water Department
System Architecture
38
Philadelphia Water Department
Implementing CCS
•
Using Cityworks as our Work Order
Management System
–
–
–
–
•
•
Open Architecture database
Spatially Compliant Data structure
Web Based Application
Configured Cityworks to maximize its benefits for CWS
All Water Quality Service Requests are funneled
through the Call Center and captured in
Cityworks
All Work activities on the Distribution System will
be captured in Cityworks
–
This will help with associating water quality alarms with
potentially related distribution system activities
39
Philadelphia Water Department
CCS and EDAs
•
EDAs ...
–
•
An EDA’s purpose is ...
–
•
Are used to continuously process
water quality complaint information
and determine whether a pre-defined complaint
threshold has been exceeded.
To bring a potential water quality event to the attention
of a human investigator using real-time customer
complaint information.
EDA Thresholds were developed based on:
–
–
–
Water Quality Complaint Type
Spatial Location (Clustering-leveraging GIS)
Date and Time
40
Philadelphia Water Department
CCS and EDAs
•
Example of an EDA
–
–
Using 1-, 2-, and 7-Day Scan Windows
Monitoring multiple water quality
parameters.
41
Philadelphia Water Department
CWS Dashboard
•
The CWS dashboard and the EDAs
will assist the investigator by:
–
–
•
Automatically displaying only the
relevant service request and work order types from the
Cityworks data
Alerting them to activity occurring within a specific
radius of the complaint(s).
The CWS Dashboard and the EDAs will also:
–
Reduce the investigation and response time by
promptly providing relevant information during alarm
conditions.
42
Philadelphia Water Department
Example Dashboard Application
43
Philadelphia Water Department
Conclusions
•
PWD’s approach to leveraging
existing GIS and Cityworks will:
–
–
–
–
Allow the department to capture all
water quality complaint calls and work performed in
the distribution system on the same map.
Allow PWD Operations to detect system issues and
failures.
Improve customer service.
Provide PWD with the opportunity to standardize
business processes and protocols for water quality
complaint investigation.
44
Dallas Water Utilities
45
Dallas Water Utilities
CCS
46
Looking Forward
•
Future Collaboration
–
–
–
–
•
Expansion Pilot Completions
–
•
AWWA, EPA, Utilities & Private Sector
Input from stakeholders and ideas
Start information sharing
Webinars
CCS Implementation & Evaluation
 Dallas Water Utilities
 Philadelphia Water Department
 New York City Department of Environmental
Protection
 San Francisco Public Utilities Commission
Real Benefits - Goal Accomplishment
47
Acknowledgements
•
•
•
•
American Water Works Association
US EPA WS initiative Pilot Cities:
Cincinnati, Dallas, New York City, Philadelphia,
and San Francisco
US EPA Water Security initiative Team
Computer Sciences Corporation
48