Digital Survey Method
The use of ICTs in programme evaluation
(Tearfund HIV programme)
Gamos
May 2012
KAP Surveys
• Research
• Assessments
• Evaluation
Impact Surveys (evaluation)
• Accountability – finding out what works – and proving it!
• Two purposes of KAP surveys:
• To understand what’s going on (formative learning)
• To show changes that have taken place over duration of programme
• Key features of methodology:
• Large, randomly selected sample
• Internationally tested and accepted indicators
• Statistical tests for reliability
KAP Structure
Example KAP – baseline and endline KAP surveys for
Tearfund’s HIV programme in Ethiopia and Malawi.
Current Status
Poverty Descriptors
Knowledge/Attitude
HIV Knowledge
Behaviours
Outcomes
VCT
Household Descriptors
Individual Descriptors
Religious Involvement
Experience of Support
for the sick, dead
and orphans
Stigma reactions
PMTCT
Religious Stigma
Change Questions
Poverty Descriptors
OVC
Comment on
Religious Inst
Religious Care
Sexual Activity
Attitudes towards sex
Discuss HIV
Intentions
Engaged in AIDS
Activities
Experience of Support
for the sick, dead
and orphans
KAP Relationships
KAP Findings
Positive change in indicators linked
to exposure to HIV information
through Church partners
AIDS activities
antenatal care
sexual behaviour
talked about AIDS
testing
knowledge
stigma
attitudes to sex
womens rights
changes
support
Survey process (paper)
Revise
instruments
Recruitment
Training
Design
survey
Introductory
workshop
Create
questionnaires
& data entry
Field
preparations
Conduct
interviews
Data entry
Submit data
Analysis &
reporting
Review of ICTs for surveys
Technology can assist with data entry:
•
•
•
Devices for data capture (CAPI software) – eliminates need for data
transcription
Optical reading technologies – eliminate human errors in data entry
activity;
Computer assisted data entry – reduces human errors
ICTs for surveys – data capture
• Early experience with PDAs was not encouraging
• Netbooks require PC skills (that are rare)
• Mobile phones – good for data collection but
screen too small for CAPI.
• NOW tablets! Low cost, big screen, apps do not
need computing skills, mobile phone skills are
universal.
ICTs for surveys – data transcription
• Optical reading technologies – too expensive; better to give budget
to partners.
• Computer assisted data entry – Gamos created bespoke software
for each project.
Survey process - DigitalForms
Revise
instruments
Recruitment
Training
Design
survey
Introductory
workshop
Create
questionnaires
& data entry
Field
preparations
Conduct
interviews
Eliminates data entry
Data entry
Submit data
Analysis &
reporting
DigitalForms
• Tearfund & Gamos decided to invest in digital technology – HIV
IMPACT programme in Malawi
• Key decisions:
–
–
–
–
Operating system: Android v. iOS (Apple)
Bespoke software v. Freeware (Android)
Devices as capital (asset) or expenditure (disposable)
Automatic or manual management of data files
DigitalForms
Cost savings:
– Data entry software – eliminate need for computer assisted data entry;
– Data entry – no need to recruit data entry staff or to appoint supervisors.
– Printing costs – no need to copy paper questionnaires.
Additional costs:
– Hardware – handheld devices
– Writing software to run on the tablets
Overall – no cost premium
DigitalForms – the product
• Modules:
– Household & respondent (poverty &
social)
– Children & pregnancy (contraception,
ANC, delivery, PNC)
– HIV/AIDS awareness (exposure to
information, knowledge, attitudes)
– Testing & counselling (as part of ANC and
other)
– Response of churches (activities, support,
attitudes to PLWH)
– HIV status (consent and disclosure)
• Loops - household membership
roster; child birth history
DigitalForms – the product
• The kit: for enumerators:
–
–
–
–
–
Tablet
Mains charger
Micro SD card
Backup battery (not all kits)
Carrying case
• For supervisors:
– Car battery charger – enables 2 devices to be
charged up from the car battery;
– USB microSD card reader – for copying files
onto a central laptop.
Experience to date
• Training workshop – nothing to report; brief introduction to tap &
swipe, then people got into it – seems quite intuitive.
• Challenges from field experience:
– enumerators want the ability to edit records – Gamos created checking software
to help supervisors check (but not edit) submitted files;
– Questions need to be numbered so that teams can discuss issues / edits;
– field reports mentioned tablets running out of power (problems with backup
batteries?).
• Completion rates – only 1
question was ‘forced’,
otherwise non-completion
rates were typically around
2%.
Experience to date
• Downloading of updated versions – workshop used beta version and
demonstrated downloading software; teams downloaded release
version successfully (1 error).
• Android update - 4.30 on Friday afternoon (release date) Asus
upgraded to Android 4.0.3 – panic! Fortunately, rollout did not
extend to Malawi.
• Clean data!
Next steps…
•
•
•
•
•
Complete user review process
Automatic data uploading (to server)
More comprehensive data validation
Develop more modules
Cheaper devices
www.gamos.org
Supplementary slides
Baseline – endline analysis
• Difference-in-differences method
Beneficiaries (treatment)
Non-beneficiaries (control)
Before
after
• Takes account of errors:
•Beneficiaries may be different from others (selection bias);
•Other changes may take place over time
Exposure to HIV/AIDS
messaging
•Church based:
•Information – heard public talks (e.g. sermons), seen or heard
information at a religious institution relating to HIV;
•Participation in AIDS related activities organised by churches;
•Personal contact – talked about AIDS or multiple sexual partnerships
with church members or leaders.
•Others:
•Broadcast media – radio, TV
•Health services – antenatal care
•Health services - HIV tests (not antenatal care)
Definition of links (impact?)
Exposed group (treatment)
Non-exposed
group
(control)
No change
(or negative)
Positive
change
No change
(or negative)
Positive
change
-
Link flagged
-
-
Value of approach
Generated data on:
• Access to information on HIV/AIDS
• Responses of churches - HIV activities, stigma within church,
promote testing, policies
• HIV knowledge – protection, PPTCT
• Stigmatising attitudes
• Attitudes towards sex, women’s sexual rights,
• Access to ANC – male involvement
• Testing
• Sexual behaviour – age at debut, MCPs, condoms
• Outcomes – changes in behaviour, care and support, intentions