Short introduction of the Indonesian pilot of the CPRA.
In June and July of 2011, a pilot of the Child Protection Rapid Assessment (CPRA) tool was conducted
in Indonesia’s affected region of the Merapi Volcano eruption. The tool had previously been partially
piloted in Libya and Egypt. The goals of piloting the CPRA tool in Indonesia were twofold:
1)
2)
Piloting, developing and evaluating a set of globally developed rapid assessment tools (CPRA).
These goals were achieved by adapting the tool to the Indonesian context.
Building capacity on a national level. This was achieved, in cooperation with the Indonesian
government, by training a team of Indonesian researchers to use the assessment tool in order
to prepare for future emergencies.
One member of the team was tasked with observing and documenting the entire process. This was
done to ensure lessons learned would be captured and reported. This report is based on information
from the documenter, data captured, and additional input from the researchers and team leaders.
Context
The Merapi volcano erupted multiple times in late October and early November 2010, killing 141 people
and leaving over 279,000 people as Internally Displaced People (IDP’s). Victims were evacuated to
camps and temporarily shelters. The predicted affected population was estimated to be around 5,000
people, demanding a rapid reaction to accommodate the actual number of victims. This resulted in the
building of many IDP camps, many of which are still populated.
The government determined two districts to be most affected. The two areas were the Sleman and
Magelang districts, both located on the mountain. Sleman was mostly affected by the actual eruption of
the volcano, whereas Magelang was mainly affected by the lava streams.Whithin these areas …..sites
were visited, 17 sites in Sleman and ….. sites in Mangelang. The total amount of sites was
Preparation
The first phase of the Child Protection Rapid Assessment, namely the preparation phase, consisted of
several important elements. The following are summaries of lessons learned in the Indonesian pilot.
The budgeting should include top priorities.
Different emergencies will include different amounts of funding and various logistical challenges. While
the ideal budget would include enough resources to cover everything, many times there is not enough
money, and prioritization will need to occur. Below are a few of the budget items that were considered to
be priority in the Indonesian pilot:
1)
2)
3)
Transportation – both for the training, and for the actual fieldwork. Make sure to budget for fuel
and enough cars to transport the entire team. A few members of the team in Indonesia owned
motorcycles, which proved to be very useful in accessing more remote areas. The budget for
fuel and usage of these motorcycles was added to the budget after the start of fieldwork.
Training facilities – the ideal location for training is a central location that is easily accessed by
all researchers. If this is not possible, transportation arrangements should be made to ensure
full attendance.
Printing costs – the costs for printing should be priced according to the local cost. It is
important to plan for the printing of the training materials, the CPRA tool kit for every site, and
any additional logistical printing needs. In the pilot, there were a few communication errors
between team members that resulted in printing mistakes. The team was able to overcome
these mistakes by reallocating money from another category in the budget deemed less
important.
Ensuring a thorough selection of the team leader and researchers.
The importance of selecting the right team was very high. It is preferred that the researchers have child
protection experience, although it can be difficult to find qualified people during an acute emergency.
The researchers will be trained in the use of the tool during the training, but should be able to learn new
concepts rapidly, and should have personalities that work well with children.The Indonesian team was a
mix of social workers, government employees, university employees and NGO workers. This provided
an excellent forum for including many different ideas.
The team leader in Indonesia was employed by a national NGO, and had previously worked for both the
government and UNICEF. This history allowed him to use all of his networks to ensure successful
cooperation from all parties. The supervisor had excellent leadership skills and experience in the child
protection field. It is best if the supervisor knows the region where the assessment will take place. S/he
should be able to coordinate the planning of the sampling frame, the key informant selection, and have
a high level of cooperation with both national and local stakeholders, and other relevant organizations.
Intensive cooperation with stakeholders.
The Indonesian pilot was organized in coordination with the Indonesian government and the Child
Protection Working Group. The stakeholders included both national and regional government
departments, local and international NGOs, the University of Indonesia and Columbia University. All of
the stakeholders were contacted before the training began to ensure full participation. The government
and universities provided the information needed to conduct a complete desk review. Choosing
researchers from many different stakeholders ensured cooperation from all involved parties. This was
also done in order to build the capacity in Indonesia.
Choosing the sample frame of the assessment.
When choosing the sample frame of the assessment, different criteria can be used. In some
emergencies, there are different primary effects of the emergency, potentially necessitating multiple
sample frames. In the Indonesian pilot, the eruption of the Merapi volcano affected a range of villages,
whereas other sites were mainly affected by the lava flows that followed the eruption.
The Indonesian team discussed the ideal criteria for sample frames, and determined that the most rigid
and valid selection would include two sample frames. Namely, one sample frame, which included
villages and camps affected by the actual eruption of the volcano; and the other sample frame, which
included villages and camps effected by the lava flows.Using the information obtained in the desk
review, the team determined that the government had drawn two rings around the Volcano: Zone 1 and
Zone 2. These Zones represented the areas that were the most affected by the volcano. The definition
for the sample frames became “any village or camp that is completely or partially in Zone 1 or Zone 2 in
either the Sleman district or the Magelan district.”
Adaptation of the tools
Adaptation of the data collection toolkit
The CPRA toolkit consists of the following research tools, which are used for data collection, data entry
and data analysis:






Key informant interview tool
Direct observation tool
Desk review tool
Site report tool
Urgent action tool
Data entry and analysis tool
These are globally standardized tools, which need to be adapted to context specific circumstances. The
adaptation of the research tools is a time consuming and very important part of the CPRA. The following
are summaries of lessons learned in the Indonesian pilot regarding the adaptation of the research tools.
When adapting one of the tools, be consistent in the other tools
An important note for the adaptation of the toolkit and the data entry tool: If something is changed in one
of the tools, make sure that the other tools are consistent with the adaptations. For example, if an
answering category is added to a certain question in the key informant interview, the same category
should also be added in the corresponding question in the site report, as well as in the data entry tool.
When adapting the tools, be aware of the numbering of tool and the corresponding tools
When changes are made in one of the tools, one should be aware of the numbering of the tool, and of
the numbering of the corresponding tools. When deleting or adding questions to the key informant
interview, the numbering of the whole tool (and of the corresponding tools) has to be revised.
In order to make the adding and deleting of questions, or even chapters, in the tool easier, it is
recommended to number the tool independently per chapter. Only the numbering of that specific
chapter will have to be changed, instead of the numbering of the whole tool. When changing the
numbering of the site reports, these changes should also be made in the data entry tool in order,as
those two tools to correspond.
Process of adapting the data collection toolkit to the specific context
The adaption of the toolkit to the Merapi emergency contextwas a time consuming phase in the
preparation for the data collection. The first adaptations of the tool were made based on the information
collected during the desk review, from the (local) government, stakeholders and other relevant
organizations. The chapter on ‘Children and armed forces and groups’ was excluded from the toolkit, as
it did not appear to be relevant to the context and location of this assessment. However, the questions
regarding children and armed forces and groups could be relevant for futureemergencies in Indonesia.
This chapter should be included or excluded based on the contextual relevance, which could differ
between different locations, contexts and emergencies.
The second round of adapting the tool was done during the training. The participants of the training,
including the research team, provide input and information and weighed in on context specific answering
categories and the (culturally appropriate) wording of the questions.By practicing with key informant
interviews, direct observations and the compiling of site reports during the training, the participants were
triggered to rethink the wording and content of the questions.
One of the keyelements of the trainingwas a field test of the tool. During the field test, several key
informant interviews and direct observations were conducted in Klaten, one of the affected areas by the
Merapi volcano. Based on an evaluation of the field test with the research team, additional changes
were made in the toolkit. Most of the changes made after the field test were textual changes. The
questions of the key informant interviews appeared to contain long and difficult sentences, and the
wording of several questions was too difficult. The respondents seemed to have difficulties
understanding the questions. Therefore the wording of several questions was adapted after the field
test.
During the first week of data collection, additional changes weremade to the toolkit. During the first
phase of data collection, the answering categories of several questions wereagain adjusted. In several
cases, answering options were added because the respondents often responded in the category ‘other’.
Someanswering categories were also merged. For example, a question about the location with the
highest risks for children for violence contained the asnwering options of: ‘on their way to school’, ‘on
their way to the workplace’, ‘on their way to the market.’ The research team changed the answering
option to ‘on the street’ because there is usually one street in the villages and camps which leads to the
school, the workplace and the market.
Define the recall period used in the assessment
Usually, when conducting a CPRA after an emergency, the used time reference will be the emergency.
The assessment will be focused on the changes and protection issues after the onset of the emergency.
The Indonesian pilot was conducted 8 months after the eruption of the Merapi volcano. Using the
eruption as the time reference for the assessment would have possibly cause a bias in the answers due
to the long recall period. It was determined by select key informants and field testing that it would be too
difficult for respondents to remember what happened 8 months ago. In general, it is preferred to refer to
an event the respondents can all remember as a time frame. As there were no recent events or national
holidays the respondents would remember, and because of the importance of the same time reference
for all the respondents, the time reference of ‘2 months ago’ was chosen. In the Sleman district, there
was an appointment of a ‘guardian’ of Marapi that coincided with the 2 month recall period. The
reaserchers used that for reference if there was any clarfication needed for the key informant.
Move the sensitive subjects of the key informant interview to the end of the interview.
In the current version of the key informant interview, the sequence of the chapters is as follows:







General information
Separation from usual caregivers
Care for separated and unaccompanied children
Threats to children’s physical safety and security
Sexual violence
Psychosocial wellbeing
Support mechanisms
For the researcher to successfully discuss sensitive issues such as violence and sexual violence, it was
determined to be important that the interviewer and the interviewee had first established a rapport, or a
relationship. In the current tool, the sensitive issues are quite early in the interview discussed. The
interviewee might be hesitant to honestly answer these questions because they might not trust the
interviewer yet. It is therefore recommended to move the sensitive chapters, namely ‘Threats to
children’s physical safety and security’ and ‘sexual violence’ to the end of the interview. The proposed
sequence of the chapters is at follows:







General information
Separation from usual caregivers
Care for separated and unaccompanied children
Psychosocial wellbeing
Threats to children’s physical safety and security
Sexual violence
Support mechanisms
The editing of the data entry tool is a time consuming process, which requires someone with knowledge
of Excel
When editing the site report tool, the data entry and analysis tool should also be edited. The process of
editing the data entry and analysis tool was very time consuming. In order to edit the data entry tool, a
team member with advanced Excel knowledge was required.
Whereas deleting or renaming questions and answering categories were relatively easy, the addition of
new context specific questions and answering categories was more complicated. First, the new
questions or answering categories needed to be inserted in the data entry form, and new drop down
menus needed to be created. The data analysis sheet then needed to be adapted to the new questions
or answering categories.This required the existing formulas to be adapted and new formulas to be
written. Finally, the newly created answering categories or questions should be included in the graphs.
As the data entry sheet, the data analysis sheet and the graph sheet all correspond with each other, the
changes made in each sheet should be consistent with the other sheets.
Translation
The original CPRA tools were in English and needed to be translated before they could be used in the
field. After editing the content of the toolkit, the next phase was translating the materials in Bahasa
Indonesia. The following documents had to be translated:





The CPRA toolkit (Key informant interview tool, direct observation tool, desk review tool, site report
tool, urgent action tool)
The CPRA guidelines
The data management excel tool
The materials for the training: Powerpoint presentations and handouts.
The report of the analysis and results of the CPRA
The translation of the CPRA tools was a time consuming process, and required many stages of revision
in order to be successful.
The translation process of the CPRA tools additional materials took a lot of time. Although the translator
of the Indonesian pilot received the materials four weeks before the start of the research, more time for
translation was needed.
There are two reasons why the translation phase was time consuming. Firstly, several adaptations were
made in toolkit, which required a realtime updating of the Bahasa Indonesia version. Secondly, the
wording of the Bahasa Indonesia translation of the toolkit required many stages of revision, both by the
researchers and the teamleaders.
To minimalize the time spent on translating the tool, it is recommended to ensure that the toolkit and
other materials are sent to a translator as soon as possible. During an acute emergency, there will
probably be time constraints which will diminish the time that can be spend on translation. In time
pressured situations, it is recommended to use a team of translators in order for the translation
completed in a timely manner.
Collect information on which languages are widely used in the emergency area, when deciding which
language to use for the translation
In the Indonesian pilot, the chosen language for the CPRA was Bahasa Indonesia. A wide range of
languages and dialects is spoken in Indonesia, but the national language is Bahasa Indonesia. To be
able to reach as many respondents as possible, the documents and toolkit were translated in Bahasa
Indonesia.
Bahasa Indonesia is considered to be a very formal language. This resulted in some respondents being
uncomfortable speaking in Bahasa Indonesia. Several respondents preferred to speak in Javanese
(regional language). Several members of the Indonesia team were able to speak in Javanese and
managed to translate the questions on the spot. When implementing this tool in other regions or
countries, it is important to know which languages are spoken in the emergency area and to make sure
at least one of the research team members speaks the local tongue.
The translator and members of the research team should have a deep understanding of (the meaning
of) the CPRA tools.
One of the lessons learned from the Indonesian pilot is the importance of the translator and the research
team to have a deep understanding of the tool and the questions. The importance of the understanding
of the CPRA tools is twofold:


It enables the translator to translate the questions in culturally appropriate manner, instead of
sticking to a literal translation. Because of the complicated nature of linguistics, literal translations
will most of the time be less acurate because of word choice, grammer, and other translation
issues.
It enables the researchers to interpret and ask the questions in a culturally appropriate manner.
The sentences and the wording of the current English version of the tool are rather complex. Therefore,
the translation and interpretation of the tool was considered difficult by the translator and the
researchers. In the Indonesian pilot, several people who were involved in the development of the CPRA
tool were present, and could clarify questions from the research team. However, when there is no one
with CPRA experience present, it will be harder for the translator and researchers to get questions
clarified.
A recommendation is to develop an appendix in order to clarify the meaning of the questions. It is
proposed to structure this appendix per question, starting with Key InformantInterview question 1. The
concept and definitions that should be captured in that question should be listed, as well as an example
and a short explanation of the relevance of the question for the tool. This additional information will
make the translation and interpretation of the tools more accurate, as it would enable the translator to
use culturally appropriate wording.This would require a review by the team leader to ensure the
concepts of the questions are captured. The use of an appendix could also provide the researchers with
a deeper understanding of the question.
Training
Before the CPRA assessment began, the members of the research team were trained in the use of the
CPRA tool during a four-day-training. This training covered the following:







Background information on the tool and child protection in Indonesia
Ethical considerations
The use of the different CPRA tools
Exercises with the use of the CPRA tools
The methodology of the CPRA
Roles of the members of the research team
A field test of the tool
In order to build capacity in Indonesia, there were additional participants who were not part of the
research team for the Indonesian pilot. They were all stakeholders in the CPRA, and added immense
value to the training. After the completion of the CPRA training, an additional training for the research
team was conducted on the usage of Participatory Randking Method (PRM). In order to both train and
capture valid information, a PRM session was held with the researchers. The question presented to
them was “What was good about the CPRA training you just received?”
Below are the issues that were determined, in order of their perceived importance.
1)
Good team spirit
a. The team thought that the number one positive aspect of the training was how well
everyone worked together.
One participant stated that “the personality of the team defines thesuccess. And
creates the atmosphere for the participants to feelcomfortable.”
Personality/knowledge of the team leaders
a. “It’s important that the facilitator has a spirit to enable others to feelcomfortable.”
b. “It is important that the facilitator has experience with the tool andhas knowledge on
the cultural values, so he[she] can explain it well to theparticipants of the training.”
Time schedule of the training (prompted)
a. This was defined as how close the training was in proximity to the actual fieldwork.
The team appreciated being able to practice what they learned so quickly.
b. One participant stated that “It was good that straight after the training, we went into
the field.”
Spending time on the operationalization of words and concepts.
a. In order to ensure the research team had similar understandings of each question, a
process of operationalization was undergone during the CPRA training.
b. “It’s important to translate the documents for the use in practice. Thisis the core. We
want to collect information, but if we don’t understandthe language that is problematic.
Operationalisation is the core. But inorder to come to operationalisation, you first need
a good team spirit and teamwork.”
Including various stakeholders
a. “It’s important to involve the stakeholders because they can bring moreideas from
their experience.”
b. “It is important because, if there is an emergency, it is impossible for anagency to
handle the emergency alone. They will need to cooperate withother agencies. During
the training the need of cooperation became clear [to us].”
Excersises during training
a. The participants appreciated the exercises that were included in the CPRA training.
Specifically the Key Informant Interview excersise and determining the criteria for a
KII.
b. “[The training excersises] gave us a good understanding of the concepts that we will
use in the field.”
Field test during training
a. The main concept discussed for the positive aspects of field testing was “confidence.”
b. “It is important to get practise in the field through the field test to gainconfidence.”
b.
2)
3)
4)
5)
6)
7)
Highlight: the importance of a field test
After two days of training consisting of background information and exercises with the different CPRA
tools, a field test was conducted on day three. By conducting the field test on day three instead of day
four, there was one day of training left for the evaluation of the field test and adaptation of the CPRA
tools based on the field experience of the researchers.
Conducting a field test was important because it provided feedback on the tool and suggestions for
possible improvements of the tool. Besides the technical adjustments of the tool, the field test gave the
participants the opportunity to practice working with the tool and made the researchers feel comfortable
and confident in the practical usage of the tool.
One of the objectives of the Indonesian pilot was building capacity in country. By including a field test in
the training, all participants were able to get field experience even though some were not part of the
Indonesian assessment team.
Highlight: explanation of the methodology of the CPRA
A substantial amount of time was spent during the training on the explanation and clarification of the
methodology of the CPRA. The concepts of ‘sample frame’, ‘unit of measurement’ and ‘purposive
sampling’ appeared to be difficult. For future training, the explanation of these technical concepts could
be made easier by using a local example case to illustrate the explanation.
Highlight: explanation of the concept of the site report
One of the tools to be explained during the training is the site report. After explaining the concept of the
site report, it would be valuable for the participants to do an exercise with compiling a site report. It is
recommended to include a site report exercise for the team leader to practice with. The participants
could, for example, provide the team leader with three or four key informant interviews and direct
observations and the team could practice compiling the site report.
The indonesian pilot did not include a site report excersise. On the first day of fieldwork, the process of
compliing a site report revealed a few further adaptations and clarifications that needed to be changed
in the tools themselves. Had there been an escersise in the training, these clarifications could have
been done before the actual fieldwork.
Data collection
Over the course of four weeks, the Indonesian research team visited …… sites in the direct environment
of the Merapi volcano. A total of …… key informants were interviewed, approximately 3-4 key
informants per site.The same amount of direct observations were conducted per site. During daily
debriefings, the collected data were compiled into site reports by the research team and team leaders.
The research team finished the data collection efficiently and thouroughly in three weeks, which was
one week ahead of schedule. It was decided that with the extra time, an additional step of triangulation
would be added. The participatory ranking method with both children and key informants was chosen.
After finishing all the key informant interviews and direct observations, the research team returned to the
Sleman district a week later to conduct a PRM with members of the community.
The adult participants of the PRM were a combination of (several of) the prior interviewed key
informants and persons who qualified to be key informants but were not interviewed. Each PRM
included ….-….. participants. In addition to adding an additional step for triangulation, the PRM with the
key informants was also used as a “cross-validation” tool for the CPRA. The use of this methodology will
be thoroughly explained in the next chapter.
The following are summaries of lessons learned in the Indonesian pilot regarding the data collection.
As the assessment was conducted a relatively long time after the emergency, several sites did not
qualify anymore to be included.
The Indonesian pilot of the CPRA was conducted in an area affected by the eruption of the Merapi
volcano 8 months before. Because so much time had passed since the actual emergency, several sites
that were selected in the desk review did not exist anymore. Several camps and shelters were already
closed, and people moved back to their villages. The fact that time had passed since the emergency
diminished the total amount of sites visited. When the CPRA is used in the phase III timeframe after an
emergency, there would probably be more sites available for the assessment.
The engagement of local key figures facilitates the data collection process in several ways
In order for the data collection process to be smooth, the engagement of local key figures is essential. A
person who knows the key stakeholders and figures from different communities, (local) government
institutions and other relevant organizations can contribute to a smooth data collection process. During
the Indonesian assessment, several team members functioned as scouts during the data collection
process. As they were well informed on different communities, they had an important role in the
selection of reliable and qualified key informants and making appointments for interviews and PRMs.
Although during an acute emergency, there might not be time to send scouts to the sites to set up
interviews for the next day, the inclusion of key figures that can help contact (local) government bodies,
organizations and communities is highly recommended.
In some situations, it could be useful to split the research team up during the data collection
Although the CPRA guideline recommends the assessment to be conducted in a sweeping manner, in
specific situations it could be preferable to divide the team over different sites. During the Indonesian
assessment, 3-4 key informants were interviewed per site. The research team included 7 members, and
dividing the team in two enabled them to go to two sites in the same time. Splitting the team in two
turned out to be time saving in this specific situation. However, if the research team is smaller, dividing
the team over different sites is not recommended.
Data entry and analysis
After the data from the key informant interviews and the direct observations is collected, the team
leaders will compile this information in a site report in collaboration with the research team. The
compiled information of these site reports have to be entered into the data entry and analysis tool.
The following are summaries of lessons learned in the Indonesian pilot regarding the data entry.
Simultaneously with the data collection, the data should be entered in the data entry tool.
It is recommended to start the data entry right after a site report is compiled. Collecting and entering the
data simultaneously enables the researchers to clean and clarify ambiguities in the collected data.
A shortcoming of the current data entry tool is the absence of a ranking system in the answers of certain
questions.
While during the data collection, key informants are asked to rank their answers to certain questions;
these ranks cannot be entered in the data entry tool. The absence of a ranking system in the data entry
tool causes a great loss of collected, and potential valuable, information. It is therefore recommended
for future research to adapt the data entry tool, so that the researchers will be able to enter the ranking
of answers in the tool. This will make the analysis and graphs of the collected data even richer.
Methodology
The sample frames were initially defined by the request of the government to focus on the two districts,
Magelan and Sleman. The government mapped out two rings circling the volcano, outlining the most
affected areas. The research team decided to further define the sample frame by specifying that a
qualifying community must have been affected by the Marapi volcano and be at least partially in either
zone one or zone two.
The communities (sites) were villages, IDP camps, or shelters. The sites were selected based on the
criteria defined in the CPRA guide. In Magelan, many of the villages and camps had a population of less
than 20 citizens. These sites were not included in the sample frame, due to the lack of key informants.
The key informants were selected by meeting a list of criteria defined in the CPRA guide. This included,
but was not limited to, school teachers, social workers, government workers involved with children.
Researchers would conduct interviews with a minimum of 3 key informants. They would also walk
around the community and conduct a minimum of 3 direct observations as a form of triangulation.
At the end of each day, the team would gather together and begin compiling the site report for that
community. The senior researcher would take the information gathered from the key informant
interviews and direct observations and create the site report for that community. That site report would
then be entered into the database for analysis. The purpose of using different methods of information
gathering was to triangulate as much as possible. The PRM sessions were used order to test the
reliability of the CPRA tool.