Evaluation of a surveillance
system
Pawel Stefanoff
Evaluation of a surveillance system:
a cyclic process
Gaps between
needs and capacities
Monitoring and
evaluation
Planning for
improvements
Implementation
WHAT do we want to assess?
Surveillance system
Complexity
Disease specific
surveillance
Technique
The way of defining the problem  methods  solution.
Why evaluate?
• To assess the capacity of the system to meet
its purpose and objectives
• In order to
– improve its operation
– modify it to address:
• priorities
• epidemiology
– optimize the available resources
Plan of evaluation
• Description of the needs
– Country context
– Disease patterns
– Target audience
• Description of the surveillance system
– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Plan of evaluation
• Description of the needs
– Country context
– Disease patterns
– Target audience
• Description of the surveillance system
– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Description of surveillance needs
• Description of the country context
– Population
– Health system
– Political structure and administration
• Description of disease patterns:
– Disease characteristics (epidemic potential)
– Risk factors (country-specific?)
What’s the target audience of
surveillance?
• Public Health professionals?
• Clinicians / Microbiologists / Control of
Infection staff?
• Environmental Health professionals?
• Health service managers?
• Health educators / teachers?
• Government / Politicians?
• Public?
Plan of evaluation
• Description of the needs
– Country context
– Disease patterns
– Target audience
• Description of the surveillance system
– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Description of the surveillance system
• The description of the surveillance system is a
scientific process
– Poor methods will generate unreliable results
• When describing a surveillance system:
– Report your methods in the “methods” section
– Write the description in the “results” section
Methods that may be used to describe a
surveillance system
1. Review of documents
2. Review of products/ outputs
3. Qualitative interviews of key informants
4. Quantitative interviews of participants
1. Reviewing documents
• Consider:
– Work plans
– Guidelines
– Operation manuals
– Forms
• Generates information on the theoretical
surveillance system
2. Reviewing products/ outputs
• Consider:
– Records
– Data
– Feedback reports
• Generates information on the actual
surveillance system
Elements of the surveillance system
to describe (1/2)
• Diseases under surveillance
• Forms used
• Case definition
• Population under surveillance
– Total, sentinel or special
• Reporting mode
– Passive, stimulated or active
Elements of the surveillance system
to describe (2/2)
• Data structure
– Individual or aggregated
• Data transmission
– Flow chart
• Data analysis procedures
– Type, frequency
• Feedback
– Type, frequency
• Action taken
Define objectives of a surveillance
system
• Identify or help formulating the actual or
desirable objective(s) of the surveillance
system
– Identify the goal of the programme served
– Identify what decisions may need to be taken
• Review the attributes of the system
• Review whether the system’s attributes are
satisfactory to reach the objective(s)
Plan of evaluation
• Description of the needs
– Country context
– Disease patterns
– Target audience
• Description of the surveillance system
– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
System attributes
• Simplicity
• Flexibility
• Acceptability
• Data quality
• Sensitivity
• Positive predictive value
• Representativeness
• Timeliness
• Cost
Simplicity
• Structure
– information needed
– number and type of sources
– number of information users
• Ease of operation
– data transmission
– system maintenance
– data analysis
– information dissemination
Flowchart for surveillance of HIV/AIDS
in Norway
Reference
laboratory
Blood sample for HIV test
Primary
laboratory
HIV reporting form,
part 1
Lab report and HIV reporting form
HIV infection
Patient
AIDS
Death, emigration
Primary care
physician
Hospital physician
HIV reporting form, part 2
(Prompting if necessary)
AIDS reporting form
Semiannual check
Oral information
Semiannual check
National Institute
of Public Health
Flexibility
• Ability of the system to accommodate
changes with little additional time, persons or
allocated funds
– New event to follow-up
– New case definition
– New data about an event
– New sources of information
Acceptability
• Knowledge:
– Case definition, notification procedures
• Notification conditions
– Sufficient stock of notification forms
– Working charge
– Simplicity
• Motivation
– Understanding the importance
– Risk perception
– Existence of a regular feedback
Data quality
Completeness and validity of the data recorded
• Completenness of records
– blanks in the surveillance forms
• Errors when computing data
• Influenced by
– Simplicity of surveillance form
– Clarity of electronic surveillance forms
– Training
– Validation
Sensitivity
• Two aspects:
– proportion of cases of a disease or health condition detected
by the surveillance system.
– ability to detect outbreaks, to monitor changes (out of
outbreaks, how many were detected)
• Requires (in principle)
– to validate the information collected and
– to collect information outside of the system to determine the
frequency of the condition in the community.
Factors influencing case detection
• The probability of medical consultation
– Severity of symptoms
– Access to health care
– Social and cultural values and habits
• The probability of clinical diagnosis
– Pathogen causes a clinical condition also caused by other
pathogens
• The probability of lab diagnosis
– Demand for lab-testing
– Existence of laboratories with the specific technique
Quantitative estimates of the sensitivity
of a surveillance system
• Health care facility survey
– Proportion of cases seen in the health care facility
that are reported / captured by the surveillance
system
• Population survey
– Proportion of cases occurring in the population:
• Seen in health care facilities
• Reported / captured by the surveillance system
• Mathematical model
– Cases captured /expected
Sensitivity of measles surveillance in
Haora district, West Bengal, India
• Methods used
– Cluster sample with door to door case search
– Review of health care facility and surveillance records
• Key results:
– Of 240 cases identified in the population, 8 (3.3%) were
seen in public health care facilities and reported
• Conclusion:
– Measles surveillance is only 3.3% sensitive
• Recommendation:
– Promote vitamin A management in the population to
increase the use of health system in the case of measles
Positive Predictive Value
• PPV depends on
– sensitivity
– specificity, and the
– prevalence of the condition in the population
• Consequence of a low PPV
–
–
–
–
–
frequent "false-positive" report
inappropriate follow-up of non-cases
incorrect identification of epidemics (artifacts)
wastage of resources
inappropriate public concern (credibility)
Measles case definitions
• Suspect case
– rash and fever
• Probable case
– rash, fever, and (cough, coryza or conjunctivitis)
– epidemiological link to a confirmed case
• Laboratory confirmed
– saliva/serum IgM positive
Predictive value of notified measles
Effect of change in incidence
Non-measles
Genuine measles
100 %
100000
80 %
10000
60 %
1000
40 %
100
20 %
10
1
0%
Pre-vaccine
Low coverage High coverage Near elimination
PV+
Number of cases
1000000
Representativeness
• Representative = accurately describing the
distribution of a event health in the
population by place and time
• Related to
– data quality
– bias of data collection
– completeness of reporting
Timeliness
• Delay / speed between steps in surveillance
– onset
– diagnosis
– report
– data entry
– analysis
– interpretation
– intervention
• Chronic vs acute disease
Time
Median delay between notification
and intervention by public health services
Shigellosis, Delaware, 1991
Notification completed
(n = 216)
Notification received
(n = 264)
Intervention
(n = 255)
0
2
4
6
8
10
12
# days since onset of symptoms
14
Cost
• Cost is the estimated economic cost of the
system
• Critical evaluation:
– Could the system do an identical or better job
while using less resources?
Buehler's balance of systems attributes
Sensitivity
Representativeness
Predictive value positive
Timeliness
Acceptability
Flexibility
Simplicity
Cost
Plan of evaluation
• Description of the needs
– Country context
– Disease patterns
– Target audience
• Description of the surveillance system
– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Gaps between needs and capacities
• Is the system able to
– adapt to changing needs
– detect cases and outbreaks
– investigate and intervene
– disseminate and exchange information

identification of weaknesses
Plan of evaluation
• Description of the needs
– Country context
– Disease patterns
– Target audience
• Description of the surveillance system
– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Prioritisation of weaknesses
to be addressed
Weaknesses and
their
Prioritisation
causes
Objectives for improvement
-
Example 1
• Surveillance of HIV/AIDS in Spain
– notification on AIDS cases
– aggregate reporting by laboratories on new HIV
diagnoses
– sentinel clinics report on HIV testing
• Which attributes are relevant?
Example 2
• Surveillance for surgical site infections in a
hospital
– each patient after surgery is observed
– complications are recorded on the observation
card
• Which attributes are relevant?
Example 3
• Surveillance collecting information on viral
meningitis in Poland
– detection of enteroviral outbreaks
– investigation for poliovirus
– monitoring tick-borne encephalitis foci
– identification of new neurotropic viruses
• Which atributes are relevant?