“To Ignore or Not to Ignore?”
Follow-up to Statistically
Significant Signals"
Reflections from San Diego County
Biosurveillance Information Exchange Working Group
2/23/06
Jeffrey Johnson, MPH
San Diego County
Health & Human Services Agency
SAN DIEGO COUNTY
• Nearly 3 million population
• International border
• Large military presence
• Biotechnology Hub
• 21 Emergency Departments
Early Event Detection in San Diego
• Evolving effort since pre - 9/11
• Data sources:
ER Visits, Paramedic transports, 911 calls,
school surveillance, OTC sales
• Systems:
Local SAS/Minitab system, ESSENCE, and
BioSense
• Statistical
Methods:
Descriptive, time series, CUSUM, EWMA, process
control methods (P&U Charts)
• Multiple syndromes
• Visualization and alerting
• Incident Characterization
• Follow-up to signals
County of San Diego
Health & Human Services Agency
If We Ignore A Signal……
• We take no action or follow-up
• Save staff resources
• Avoid bothering hospital staff yet again
• Another data source may signal
• “The Feds may pick it up”
• Might lose an earlier start to a response
• We might be dead wrong to ignore
If We Do Not Ignore a Signal……
• Will it be another “false alarm”
• May detect an event earlier
• Earlier response
• Continued interaction with the medical community
• Gain experience with follow-up
• Increased situational awareness
Characterization of Detections
• Detection Method
• Syndrome group
• % Admitted
• Deaths?
• Geographic cluster?
• Prior day’s level?
• Recent level?
• Age groups?
• Severe syndrome?
• Detections in other data sources?
• Other epidemiological intelligence?
• Other diagnostic information
Follow-up?
Action
or
No Action
or
Watch
Detection Follow-up
with Medical Community
What is the final diagnosis of Patients A, B, C?
Is there a common pattern among admitted patients?
Did any have lab test results that might suggest a larger event?
Among patients with a common zip code, was there a
shared living setting or common exposure?
Can we send someone out to review medical charts?
What is your facility’s assessment of the situation?
County of San Diego
Health & Human Services Agency
Routine Surveillance Activities
Aberration
detected
Ignore?
NO
IDENTIFY
YES
Rule out system error
Potential
false positive
“False Positive”
YES
Inform key departmental staff
NO
VERIFY
Preliminary evaluation
Describe initial results
Ignore?
True Positive
Ignore?
Inform key divisional staff
NOTIFY
Intensive
monitoring & surveillance
Ignore?
County of San
Diego
Health & Human Services Agency
Evaluate other
data sources
Cluster check
GI Syndrome Over Time (10/31/04 – 8/24/05)
90
80
70
60
ED
50
40
30
20
10
0
10/31/2004
11/30/2004
12/31/2004
1/31/2005
2/28/2005
3/31/2005
Gastrointestinal
4/30/2005
5/31/2005
6/30/2005
7/31/2005
7 Day Moving Average
30
25
20
911
15
10
5
0
10/31/2004
11/30/2004
12/31/2004
1/31/2005
2/28/2005
3/31/2005
Gastrointestinal
4/30/2005
5/31/2005
6/30/2005
7/31/2005
7 Day Moving Average
Paramedic
Runs
25
20
15
10
5
0
10/31/2004
11/30/2004
12/31/2004
1/31/2005
2/28/2005
3/31/2005
Gastrointestinal/Genitourinary
4/30/2005
5/31/2005
7 Day Moving Average
6/30/2005
7/31/2005
The Significant Aspects
of Syndromic Surveillance
• Statistical Significance
• Public Health Significance
• Significant Event
• Significant Public Awareness
• Significant Biological Agent Detection
Statistical significance vs. public health significance
HAZMAT FLAG – 12/04/2004
County of San Diego
Health & Human Services Agency
Statistical significance vs. public health significance
County of San Diego
Health & Human Services Agency
County of San Diego
Health & Human Services Agency
Significant event with statistically significance outcomes
Syndromic Surveillance for Natural Disasters
San Diego Wild Fires, 2003
San
Diego
County
Significant event with statistically significance outcomes
Syndromic surveillance for natural disasters
Significant Public Awareness
“The Clinton Effect”
September 4, 2004
San Diego County: Prehospital Transports and ED Visits
with "Chest Pain" as Chief Complaint (12/31/03 - 10/08/04)
While spikes in both datasets are apparent,
normalized counts show a relatively larger
increase in ED visits on Sept. 6, 2004.
60
50
San Diego County: Normalized Prehospital Transports and ED Visits with
"Chest Pain" as Chief Complaint (12/31/03 - 10/08/04)
5
30
4
20
3
10
0
12/31/03
1/31/04
2/29/04
3/31/04
4/30/04
ED Visits
5/31/04
6/30/04
Ambulance Runs
Normalized Count
Count
40
2
1
7/31/04
0
8/31/04
9/30/04
-1
-2
-3
12/31/03
1/31/04
2/29/04
3/31/04
4/30/04
ED Visits
5/31/04
6/30/04
Ambulance Runs
7/31/04
8/31/04
9/30/04
Significant Public Awareness
7/7/05
London Bombings
San Diego County
Paramedic
Transports
for
“Chest Pain”
Significant BT Agent Detection
Biowatch
BioWatch Detection
• Tells us agent, sensor site and date
• Plume plot may help us narrow surveillance on a
geographic area
Application of Syndromic Surveillance
Agent:
Syndrome categories
Specific word search in CC or DX fields
Sensor site:
Zip codes, population (schools)
Date:
Temporal based surveillance
New pre-detection baselines
Anatomy of a Detection
(a case example)
Daily Email Report
911 Call Data
Feb 5, 2006
Attached Table
911 Call Center - GI Syndrome Signal
Line listing
for review
Non-specific
call
complaints
911 Call Center - GI Syndrome Signal
21 Signals since
07/01/03
Various statistical
signals
The count for the
signals include a
consistent range
What did we do?
• Magnitude of cases
…... 14 vs mean of 7.8
• Which method(s) signaled?
…... CUSUM (2), P-Chart, U-Chart
• Check the other call centers
…… No signals
• Check the other data sources …… No Signals
(ED data, EMS transports)
• Review the line listing
……. No apparent pattern
• Our conclusion…..
>>>>> • Super Bowl Sunday
• Fewer trauma calls
• Smaller denominator (P-Chart)
• Traditional increase in GI on
this day
• Watch next day’s results
Case Example #2
Hospital 9 ED Data
Respiratory Syndrome
Hospital 9 - Daily Results Table
1/
1/
2
2/ 004
1/
3/ 2004
1/
2
4/ 004
1/
2
5/ 004
1/
2
6/ 004
1/
2
7/ 004
1/
2
8/ 004
1/
2
9/ 004
1/
10 200
/1 4
/
11 200
/1 4
/
12 20 0
/1 4
/2
1/ 004
1/
2
2/ 005
1/
3/ 200
1/ 5
2
4/ 005
1/
2
5/ 005
1/
2
6/ 005
1/
7/ 2005
1/
2
8/ 005
1/
2
9/ 005
1/
10 200
/1 5
/
11 200
/1 5
/
12 20 0
/1 5
/2
1/ 005
1/
2
2/ 006
1/
20
06
Hospital 9 Respiratory Syndrome
40
01/01/04 - 02/03/06
35
30
25
20
15
10
5
0
Count
Signal
Hospital 9 Respiratory Syndrome
• 24 signals over a 37 day period
• Count range: 11 – 34
• Over time an increasing mean
Greater Syndrome Specificity……
Hospital 9 Influenza-like-illness (ILI) Syndrome
• “ILI syndrome” has greater syndrome specificity than “Respiratory”
syndrome”
• 16 signals over a 37 day period
What We Have Learned
• S$gn&ls Happen!
• Make sure you see flames before yelling “Fire”
• CUSUM 2 & 3 STD may be too sensitive
• We lose precision with non-specific syndromes
• Everyone wants to know what’s going on all the time
• Increasing focus on situational awareness
• Further evaluation and testing required
Hype Cycle of Emerging
“Syndromic Surveillance” Technologies
Adapted from the Gartner Hype Cycle
9/11,
Anthrax
attacks
The
“magic
bullet”
Too many
signals?, IT Costs,
poor syndrome
specificity,
evaluation results
Dual use, situational
awareness,
appropriate signals
Prioritized
data sets,
protocols in
place,
Event
or
Technology
Trigger
Peak
of
Inflated
Expectations
Trough
of
Disillusionment
Slope
of
Enlightenment
Plateau
of
Productivity
Considerations
• More work in all areas of syndromic surveillance is
needed
• Knowledge requires responsibility
• The enemy is studying our efforts
• Current/future funding levels require reliability,
efficiency and sustainability of systems and
approaches
• The Future:
Neural networks and Artificial Intelligence (AI)?
• Are we ready?
Contact Information
Jeffrey Johnson
619-531-4945
jeffrey.johnson@sdcounty.ca.gov
Thank You