World Bank
Survey Solutions
COMPUTER-ASSISTED PERSONAL INTERVIEWING
Michael Lokshin
Average interval between surveys
The(years)
problem:
MENA
9
SSA
7
SA
6
EAP
5
World
4
ECA
3
LAC
2
0
2
4
6
8
10
Percentage of missing or invalid answers
problem:
(questionsThe
on learning
and literacy)
Laos
Ethiopia
Burundi
Madagascar
Nepal
Ghana
Uganda
Malawi
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
Only 11 out of 55 countries in Africa had comparable poverty data for
2005. The situation has not changed much since.
Background
• 2011: Comprehensive Assessment of CAPI software
products is commissioned to University of Maryland by
the WB.
… no existing software provides exactly the right mix
of features necessary for the sort of surveys
conducted by World Bank and its clients.
• 02.2012: LSMS and Computational Tools teams of
Research Department of the World Bank with support
from Global Strategy of FAO started development.
• 09.2013: First public version of Survey Solutions is
released
• 05.2015: Survey Solutions 4.4 is released
Pick 3: Fast, High Quality,
Cheap
Improve timeliness of data
collection
Ensure data quality and
comparability
Cost-effective, sustainable
solution for NSOs
Results:
Data processing
CAPI
Data cleaning
Data entry
Data collection
Policy
decisions
Data analysis
4-5
From data collection to policy: 1-2
months
years
CAPI System Requirements
• Simple yet flexible system for the non-expert users. Typical
clients – National Stat Offices
• Functionality for
– data capturing
– survey management
– data management
• Tablet-based with ability to display and navigate through multilevel large questionnaires.
• Support of panel surveys and complex validation algorithms.
• Cost effective system that can be used and supported by
NSOs without external TA.
Other CAPI products
• CSPro – free, closed source. resides in
Bureau of Census USA. Funded by
USAID. Windows and Android
• ODK – free, open source software
funded by USAID; UWashington, UC
Berkeley, UC Davis. A
– SurveyCTO, license fee
• Blaise – Statistics Netherlands, license
fee. W
• SurveyBe – closed source, license fee.
W
• SurveyToGo – closed source, license
fee. W
𝐷𝑄 = 𝛼𝐷𝐶 ∙ 𝛽𝐷𝑀 ∙ 𝛾𝑆𝑀
DQ –
DC –
DM –
SM –
data quality
data capturing
data management
survey management
Survey Solutions : α ≈ β ≈ γ
CAPI: Expert-centered approach
• Pros:
– A flexible system that can
accommodate complicated
surveys.
• Cons:
– Custom solutions cannot be
easily modified for different
environments
– High costs
– High requirements for the users.
– Questionable sustainability in
the high turnover environments
typical for NSO.
CAPI: User-centered approach
• Pros:
– High levels of
standardization, solutions
could be easily adapted
for different surveys in
different countries.
– Lower requirements for
the users, fast learning
curve
– High sustainability
• Cons:
– Less flexible compared to
custom solutions of
expert-centered software.
Survey Solutions: hybrid approach
• Sustainable, low-cost system for NSO
• Simple, flexible interface for questionnaire development
and testing.
• Tablet interface allows easy navigation through complex
questionnaires.
• Standardized survey management protocol based on the
best practices of data collection
• Intuitive, informative survey status reporting, survey
maps.
• Yet, powerful language for data validation and control of
questionnaire flow.
Survey Solutions
Competition
Survey Solutions: data capturing
• Large questionnaires: several thousand questions
• All standard types of questions:
– text, numeric, date; multi-choice; dynamic lists;
– Linked questions: Whose cow is it? User can select from the list of
household members.
– GPS location; Time; Barcode; Binary files (pictures)
– HQ/Supervisor-filled questions
• Rosters: can be generated from:
– Fixed lists; dynamic lists; numeric; multi-choice questions.
– Nested rosters with unlimited degree of nesting
• Interviewer Comments on a question and interview
• Question Instructions
GPS in action
Plots can be easily and
precisely measured by simply
walking around the area with
the tablets
Ability to measure distances
to banks, schools, roads,
wells. Maps used for optimal
routing
Sensors
Multi-purpose sensor
Water quality testing
Portable scales
Soil Fertility sensor
Vision tests
Wearable devices
Para Data: Adptive Survey Design
• Improve data quality by correcting survey process during
the field operation.
• System records all events with timestamps that happen on
a tablet: Data entry, Data correction, Responsibility
changes, etc.
• Analysis of time per interview, time per question, section.
• Changes in productivity over time, for different
interviewers, teams
• Quality control, monitoring and evaluation
Prices of tablets start from $60:
cheaper than paper questionnaires
Questionnaire delivery, Changes or corrections at no cost
Free software but not free survey
Software might be free but the TA support is very expensive:
Competitor #1: Proposal for a survey in Sri Lanka, 24,000 hhs –
$340K
Competitor #2: Actual cost of a survey in Uganda, 10,000 hhs $350K
Survey Solutions: Cost varies from about $5,000 to 0,000s for large
surveys.
Survey Solutions: system components
• Designer: online tool for questionnaire creation and
validation @ solutions.worldbank.org
• Tester: Android app connected to Designer to test
questionnaires in real time. @ Google play
• HQ: online tool for centralized survey management,
validation, data aggregation and reporting.
• Supervisor: online/offline tool to manage process of
data collection on a team supervisor level.
• Tablet CAPI: Android app for data capturing on a
tablet.
SURVEY PIPELINE: 50+ SURVEYS IN 30+ COUNTRIES
• Africa:
•
•
•
•
•
MENA:
SAR:
EAP:
ECA:
LAC:
Malawi, Uganda (4), Tanzania, Togo, Benin, Madagascar (3),
Niger, Cote D’Ivoire (4), Zambia (3), South Africa, Ghana, South
Sudan, Mozambique, Nigeria
Djibouti (4), Kuwait, Morocco, Tunisia
Bhutan (3), India, Pakistan
Thailand (2), Myanmar (7), Pacific Islands (4)
Armenia, Azerbaijan, Kirgizstan
St. Lucia (4), Paraguay, Belize
• Survey types: LSMS, HBS, LFS, Enterprise Survey, EDU, Health
• Largest survey:
• Largest questionnaire:
Malawi - 200,000 households
Tanzania – 3,000 questions
Researchers design
questionnaires using
visual tools and upload
them to the central
server
Questionnaires with no
errors are uploaded to
the
central server
HQ distributes
the
sample lists across
teams ofInternet
enumerators
Internet
Supervisors monitor the
submissions
Interviewers
Supervisors assign
synchronize
their
households to
devices
andinterviewers
upload
individual
completed
Enumerators
repeat
WiFi/USB
questionnaires
interviews if errors
WiFi/USB
are detected
Interviewers visit
households and collect
data
Main Website: www.worldbank.org/capi
Online questionnaire designer: solutions.worldbank.org
Youtube channel: Survey Solutions
Real time status of interviews
Map of the survey
Monitor the survey by checking the GPS location
of where and when the interview took place.