Detection and Analysis
Perspectives of Both Data
Monitors and Algorithm
Developers
Data Monitors
• Turbulence
– Turnover
– Training
•
•
•
•
Varied skill sets
Limited understanding of algorithms
Overworked and few
Feedback to providers critical
Developers
• Interaction and feedback from users is
important
– Iteration
• Critical for developers to know users
problems
• Limited understanding of public health
operations
– Expense of false positive to users
Available Detection Methods
• BioSense, Essence, RODS, Red Bat…
– Cusum, Smart Scores, RLS, EWMA, Wavelet,
…
• Spatial scan statistics, SatScan, zipcode
• Multiple detection algorithms – how many
are flagging?
• Multiple detections corroborate problem
• Real-time vs. batch?
Real-time Detection
• Data is unsettled when it arrives
– Pressure for real-time may exacerbate
existing problems
•
•
•
•
Is it sustainable?
Who will monitor it?
How valuable is it?
If not everyday – can we do it in a crisis?
Weaknesses of
Detection Algorithms
• Cusum, EWMA – most widely used
– Control chart has many assumptions
• Normally distributed
• Stationary assumption
– Are assumptions being met?
• Method must match data
– Getting false alarms that exceed rate that
would be expected may signal disconnect
between data and algorithm
Challenges (to name a few)
• Syndrome categorization
– More?
– Fewer?
– Subsetting?
• Statistical significance does not equal
public health significance
– “False” alarms
Issues for Users
•
•
•
•
•
•
•
False positives
Disconnect between developers and users
Difficulty evaluating what is a real alarm
Data quality issues
What user needs are being supported?
What are other uses for data?
Overall evaluation of syndromic
surveillance utility
Possible Approaches to Improving
Detection
• Explanation to user of why alarming
– Pop-up?
• Text-strings from physicians
• Phased alerting system
– Yelling, Anomalies, Alert
• Improving alert qualification (RODS in
Ohio)
Possible Approaches to Improving
Detection (continued)
• Better pre-processing of data
• Better post-processing of data
– De-duplication of data, etc.
• Failure analysis of false alarms
– What are major contributors?
Needs
• Refined data
• Improved algorithms
• Human expert required to make
determination
– Domain knowledge
– Local knowledge
– Statistical analysis is only a tool
• Good relationships among users
– Federal, state, local, facility levels
Summary
• Systems: Alarm too often
• Users: May become discouraged
• Statistical challenges
– Detection methods up to it?
– Data problems?
• Solutions:
– Better data, better processing, more money