Design and implementation of the Impact
evaluation of ISKUR training programs
June 15, 2011
This note describes the design and implementation of the impact evaluation of ISKUR’s vocational training
programs. The study is a joint effort by ISKUR and the World Bank aimed at identifying ways to improve
vocational training programs in the context of the rapid expansion of these programs in Turkey. In particular
the study is designed to answer the following questions: (1) What is the average impact of ISKUR training on
the labor market (as measured by the likelihood and quality of employment)? (2) Which trainees benefit the
most from training (in terms of gender, age, level of education and skills, work experience etc.)? (3) What are
mechanisms/processes through which training affects labor market outcomes (e.g. improved skills, reduced
search costs etc.) (4)What provider characteristics make training most effective?
The impact evaluation study started in the spring of 2010 and is expected to be completed in the spring of
2012. The first phase has been completed, including (i) the selection of provinces (23), training courses (130)
and participants (5,700) for the evaluation; and (ii) the collection of baseline data from about 5,300 evaluation
participants before the start of the training courses. Data was collected between September 2010 and January
2011 as training courses were rolled out. Data on training providers of the trainings selected for the
evaluation is being collected in June 2011. The final evaluation of ISKUR vocational training programs will be
mainly based on the follow up survey of evaluation participants in early 2012.
The evaluation relies on an experimental approach based on the excess demand for ISKUR’s vocational
training programs. Training providers interview all eligible and interested candidates. Participation in the
training programs is then randomly awarded among the best candidates into treatment (i.e. those receiving
training) and control (those not receiving the training) groups. Provided the randomization is successful (i.e.
treatment and control groups are statistically equivalent at the outset), the evaluation strategy is simply to
compare the labor market outcomes of treatment and control groups.
The note describes in detail the experimental approach used and the most important steps in the
implementation of the impact evaluation. Among others, it discusses the implementation of the experimental
design, the data collection activities and the main changes implemented in the ISKUR monitoring and
information system. The ultimate goal of this note is to provide hands-on information on how to conduct
impact evaluations of similar programs in Turkey and elsewhere, particularly in middle income countries
(MIC), as this is the first rigorous impact evaluation of a large-scale publicly provided training program in
MIC, and many in MIC face a similar excess demand for vocational training. This note accompanies another
note analyzing the profile, job-search behavior and expectations of ISKUR vocational trainees (based on the
baseline survey).
1
The rest of this note is organized as follows. Section 2 describes the evaluation design. Section 3 describes the
data collection needs, main counterparts involved and the different steps in the evaluation. In particular, it
describes the sampling and the course selection. Section 4 presents a detailed description of the sample
including, its geographical coverage, the power calculations and regional targets for the participants and the
set of vocational courses. It also discusses when and how the interviews take place. Section 5 discusses how
the treatment and control groups were selected and section 6 discusses some of the most important changes
in the ISKUR’s Monitoring and Information System.
1. Evaluation Methodology: Identifying the Treatment and Control Groups
The traditional assessment of the effectiveness of the ISKUR trainings, which is based on whether
the trainees are placed into jobs, is likely to lead to biased estimates. Most ISKUR provincial offices
still use the placement rates among trainees as a good proxy for the effectiveness of the trainings provided.
However, this is often difficult to implement due to the lack of data. ISKUR’s Management Information
System (MIS) does not keep up to up-to-date information on all the trainees nor does it collect information
on labor market outcomes after trainings. Furthermore, to isolate the effects of the trainings from the overall
economic conditions in the labor market, a control group is needed to establish a counterfactual (i.e. what
would have happened in the absence of training). It is important to carefully construct this control group.
ISKUR trainees are likely not identical to the average unemployed in Turkey. They meet very specific skills
and experience requirements that are usually set ex-ante by the training providers. All trainees also need to go
through an interview with the training providers and local ISKUR staff before being selected for training.1
This evaluation study is based on a sample of the best eligible applicants to the ISKUR trainings.
The sample frame includes a wide set of provinces where there is a large excess demand for the trainings. In
particular, the sampling frame, includes all the registered unemployed individuals who are eligible to take up
training (according to ISKUR basic criteria such as being registered to ISKUR) and are interested in taking up
the training, i.e., they have they applied for them. Latter, the training provider’s interview this group of
eligible and interested individuals and select the set of best candidates.
The evaluation study is based on an experimental design, comparing the labor market outcomes of
treatment and control individuals before and after the trainings. The participation into the trainings is
randomly awarded among the sample of the best applicants. The effects of the trainings in improving the
labor market outcomes are quantified by comparing the performance in the labor market, before and after the
trainings take place, for the two groups: treatment and control. Treatment and control groups together
World Bank (2011) reports the results for the baseline survey where the ISKUR beneficiaries are contrasted against the
sample of average unemployed and inactive individuals. Results show that the ISKUR applicants are more likely to be
women, younger and more educated. They also have less labor market experience.
1
2
account for the evaluation sample.2 The random assignment to treatment and control groups ensures that
both groups are as similar as possible (both in socio-economic background and in motivation). This is also
the fairest way possible to allocate a small number of slots among a larger list of applicants. To ensure that
the effects of the trainings will be identified separately for youth and women, the randomization is stratified
by these two characteristics.
2. Main “Steps” and Counterparts in the Evaluation
The evaluation involves a close collaboration among the evaluation team, World Bank Ankara office,
ISKUR central and provincial offices, the survey firm, and to smaller extent, some municipalities.
The field coordinator and researchers are part of the evaluation team. The baseline and follow up surveys are
implemented by an external agency (private firm). The firm is responsible for hiring and training of
enumerators. Some guidelines are provided to the agency and it presents a close monitoring of data quality is
ensured with the supervision of the field coordinator. The success of the evaluation lies in the strong
coordination across all these counterparts.
Figure 1 summarizes the main steps in the implementation of the evaluation. These include: the design
of the sample and the province and course selection; the evaluation pilot and field coordination; the main predata collection activities; the individual randomization, the baseline survey and the delivery of the training
programs; and, finally, the data analysis and planning of the follow up activities. Figure also shows that,
throughout the entire implementation process, there is a continuous improvement of the ISKUR’s
monitoring and information system. The figure reviews selected activities in each of these steps.
2
There are other trainees that take up the trainings but they are not part of the evaluation.
3
Figure 1: Process of the Evaluation
Sample Design and
Province/Course
Selection
Conduct power
calculations; Define regional
targets for the number of
participants in the
evaluation;

Select provinces included
in the evaluation;
Evaluation Pilot and
Coordination
Conduct a pilot in
Ankara;

Finalize the baseline
survey and adjust the
ISKUR MIS system;


Work together with
ISKUR regional offices on
the main procedures;

Select the vocational
courses included in the
evaluation.

Pre-Data Collection
Activities
Randomization, Baseline
Survey & Delivery Training
programs
Analysis and Follow up
Activities
Finalize baseline survey
and the provider’s
questionnaire with
counterparts;

Conduct the interviews
throughout the country;

Selection of the survey
firm.



Communication and
training of local ISKUR
staff and course providers
on the evaluation
protocols;

Analyze data;
Update contact
information for the
evaluation sample;

Randomly select the treatment
and control groups;

Schedule additional interviews
when no oversubscription is
attained;
Finalize and implement the
follow-up survey;


Collect
baseline data;
Analyze results;
Recommend permanent
changes in the MIS to ensure
high-quality data.

Delivery of the training
programs.

Public announcements
for the final set of
evaluation courses;

Finalize the
implementation calendar
with ISKUR regional
offices.

Improvement of ISKUR Monitoring and Information System
◦ For a stronger dissemination of the training courses to the target population
◦ For the development of an interface for the training providers (allowing applicants registration and update information)
◦ For the development and implementation of a randomization routine
◦ To publish periodic reports informing on the tracking of the treatment and control groups
Note: Some of the activities (for e.g. maintaining treatment and control groups) cover the entire duration of the evaluation and some are conducted simultaneously (for
e.g. preparing baseline survey and improvement of ISKUR's IT System).
4
Box 1: A Workshop on Impact Evaluation: Building capacity and support and working together on the design
of the impact evaluation
Before the beginning of the evaluation, the central ISKUR office and the evaluation team organized a large
workshop to discuss the design and the major implementation arrangements. The workshop took place in
April 2010 and gathered ISKUR staff from all the 23 participating provinces, reaching a total of 85
participants. The main objective of the workshop was to discuss the design of the evaluation and follow up
with the main counterparts on the most important implementation arrangements. It also aimed to gather
consensus and support in the most relevant, and possibly controversial, steps.
The first part of the workshop focused on the design of the evaluation and some of the technical details.
Main objective and expected outputs were described, the evaluation team was presented and the structure of
the impact evaluation design was also discussed. A wide range of issues was discussed, ranging from the
selection of provinces, regional target numbers, the importance of achieving oversubscription and the
randomization. The randomization process and selection of treatment and control groups were the focus of
the main part of the discussion. Finally, the role and responsibilities of the training providers and ISKUR
during the evaluation were also thoroughly discussed. The second part of the workshop focused on the
possible alterations in the ISKUR’s MIS, with the objective of supporting the evaluation and a high-quality
data collection process.
3. The Sample: Provinces, Courses and Power Calculations
3.1.
The provinces
The evaluation sample covers 23 provinces, which together are representative of the labor market
conditions all over Turkey. Figure 2 reports the final set of 23 provinces selected to be included in the
sample. Prior to the start of the study, ISKUR identified the set of provinces with at least two oversubscribed
courses during the period 2009/2010.3 Oversubscription in the training courses was defined as having at least
twice the number of applicants than training slots. The sample was stratified by the unemployment rate. A
sample of 20 provinces was randomly selected, of which 10 provinces had “high” unemployment rates and
ten provinces had a “low” rate. 4 Selection is proportional to percentage of individuals trained in 2009. Three
additional provinces (Antalya, Gaziantep, Diyarbakir) were proposed by ISKUR because of their importance
in representing varying labor market conditions across Turkey.
Table A2 in the annex lists the total of 39 provinces out of the 81 provinces in Turkey that met these criteria. It also
shows the strata they belonged to.
3
The provinces were classified into high and low unemployment provinces depending on whether the unemployment
rate was above or below 10%.
4
5
Figure 2: Provinces in the Evaluation Sample
3.2.
Power Calculations and Regional Targets for the Participants
The power calculations determined the optimal sample size to be able to detect a difference in job
placement rates across treatment and control group. The outcome indicator used in the power
calculations was the employment rate. Calculations also took into account observed past attrition from the
trainings (or the treatment). This was based on the analysis of previous drop-out rates from training courses.
Possible attrition rates also considered the possible non-response rates to the face-to face survey.5 The sample
size estimates accounted for an imperfect take-up of the training programs a power of 90% and an “effect
size” of 6%-7%.6
But the available information in the MIS may not lead to realistic assumptions on the share number
of applicants that may drop-out of the trainings. The regional ISKUR offices and course providers use
the information from the MIS to reach out to the registered unemployed and invite them to attend the
training programs. When a candidate rejects to participate in the trainings or drops-out of the trainings, some
The evaluation team is interested in determining the minimum sample size possible allowing the identification of the
effects of training on job placement.
5
We consider job placement rates of 20% and 30%. These numbers are defined using and analyzing the Social Security
data. The placement rate is low because only the formal sector jobs were considered.
6
6
of the individuals in a waitlist are invited to participate.7 Usually, the MIS records are updated based on the
paper records that are obtained after these slots are filled with the waitlisted candidates. Therefore, the “real”
share of drop-outs is not easily retrieved from the ISKUR records.
And, indeed, the pilot study showed that the initial assumption on the drop-out rates of the trainees
from the sample was heavily underestimated. The Ankara pilot and the analysis of retrospective data, led
to more realistic assumptions for (1) the share of trainees dropping out of treatment (also known as the
uptake of the treatment group) and for (2) the share of control individuals getting into the training programs
(also known as the uptake of control group). More realistic assumptions for the program take up and for the
contamination of the control until the end of the evaluation period led to revisions in sample size. Table 1
reports the optimal sample size under different assumptions for the effect size; take up of treatment and
control, base job placement rate.
Table 1: Optimal Sample Size8
Effect size
Treatment take-up
Control take-up
Base Job
Placement Rate
Optimal Sample
size
6%
6%
7%
7%
7%
7%
85%
85%
80%
80%
85%
80%
15%
20%
15%
20%
15%
15%
20%
20%
20%
20%
30%
30%
5390
6268
4645
5468
4973
5745
For reasonable assumptions, the optimal sample size is close to 5,700 individuals; the geographical
composition replicates the importance of each province in the national provision of the trainings.
Considering a 80%-85% uptake of treatment and 15%-20% uptake of control, sample attrition of 20% and a
base job placement rate of 20%-30% the optimal sample size is approximately 5,700 individuals. 50% of the
individuals are in the treatment and 50% in the control group. The sample is disaggregated in a representative
way across the country so that the number of evaluation participants in each province depends on the share
of vocational trainees in that province in 2009. There is a minimum number of trainees at province level of
100 to benefit from the economies of scale in the implementation and data collection. Table 2 reports the
target shares of each province in the overall sample and the number of trainees in 2009. Figure 3 reports the
actual number of individuals successfully interviewed at the baseline survey.
They may drop either because they change their minds and decide to take another course, find a job. Others still
cannot be reached.
8 The table assumes a power of 90%, an Alpha of 5%, and attrition rate of 20%.
7
7
Table 2: Sample size by Provinces
No
Province
Number
trainees
2009
1
ANKARA
2
ANTALYA
3
BAYBURT
4
DENIZLI
5
DIYARBAKIR
6
DUZCE
7
ELAZIG
8
ERZURUM
9
ESKISEHIR
10 GAZIANTEP
11 HATAY
12 ISPARTA
13 ISTANBUL
14 IZMIR
15 KAYSERI
16 KIRIKKALE
17 KOCAELI
18 MANISA
19 MUS
20 SAKARYA
21 TEKIRDAG
22 TRABZON
23 USAK
TOTAL
3,344
1,771
249
1,420
450
702
1,204
836
2,450
751
2,889
1,152
19,208
1,971
3,067
1,329
7,519
3,547
1,938
1,208
1,709
1,628
1,484
61,826
Source: Author’s calculations.
8
of Targets for the
in evaluation
participants
308
163
100
131
100
100
111
100
226
100
266
106
1,771
182
283
123
693
327
179
111
158
150
137
5,925
Figure 3: Provincial Breakdown of the Evaluation Sample at Baseline
3.3.
The selection of the vocational courses
The evaluation team worked closely with the regional ISKUR offices to determine the final set of
courses to be included in the evaluation. The sample was finalized by the ISKUR provincial directorates
in August 2010. The number of courses used in the evaluation sample is 130. The course capacities range
from 12 to 100 trainees. However, most of the courses in our sample have a capacity between 20 and 25
trainees. Several criteria dictate the final selection of courses:

Type of vocation: Even though ISKUR provides a wide set of training programs, including general
vocational training programs, job guaranteed internships and entrepreneurship courses - the
evaluation focuses only on general vocational training programs.9

Oversubscription: The evaluation focuses only on the training courses which are likely to be
oversubscribed. This is the only way to guarantee the random selection into treatment and control
among individuals in the evaluation sample. Therefore, only the most popular and more demanded
courses are selected.
These courses are not job guaranteed. Therefore they are suitable for our purpose of measuring the effect of courses
on finding jobs.
9
9

Course capacity: The evaluation prioritizes larger courses to simplify the implementation of the
randomization and the data collection activities.

Course providers: The evaluation prioritizes diversity in the training providers for the same
vocation. This will enable the analysis of the effectiveness of the trainings for providers of the same
courses with different qualities. A combination of public and private service providers is also
explicitly targeted.10

Timing of the courses: Evaluation sample covers courses that start between October and
December 2010. Preference was given to groups of courses starting at approximately the same time
but terminating before the end of March 2011.11 This geographical proximity promoted some savings
in the baseline survey which is conducted face-to-face. The latter is imposed to facilitate the
collection of follow-up data at approximately the same time after the trainings have been completed.
The evaluation team worked closely with all provincial offices to plan and disseminate the trainings
well in advance. In particular, an official letter explaining in detail the criteria was sent by the ISKUR general
directorate to all the participating provincial offices. There, the importance of oversubscription is explained in
detail. Provincial offices are also encouraged and supported to start to plan and announce courses as soon as
possible after these are selected. Moreover, because of the risks of low over subscription, the regional offices
were incentivized to select a set of reserve courses. Each province provide at least one reserve course whose
interview is scheduled to be later than the evaluation courses’ interviews.12 Some provinces anticipated more
challenges in achieving oversubscription of the courses than others. For example, the provinces of Istanbul,
Kayseri, Denizli and Gaziantep anticipated having more oversubscription problems than other provinces.
The ISKUR headquarters contacted periodically the heads of provincial offices and supported them to
advertise more strongly the courses to avoid the risk of not oversubscribing.
The final set of courses included the evaluation is quite similar to the most popular vocations in
previous years. Figure 4 refers to the final composition of trainings in the sample. The ISKUR provincial
offices are requested to submit a set of proposed courses during the summer 2009. After receiving the initial
proposals from all 23 provinces, the provincial course capacities are contrasted with the provincial targets
Inevitably, the timing of the evaluation affected the types of the training providers that are part of our sample. For
example, in some provinces, the courses provided by MoNE open only in summer.
11 Before the final selection of courses, the 23 provincial offices are informed about the criteria and their feedback is
requested. All over Turkey, the month of July was considered problematic to achieve a large oversubscription since new
employment fields are opening in summertime. For example in provinces with large agricultural sector the timing of the
agricultural activities and harvest are relevant. Agricultural activity typically increases during the summer when most of
the unemployed find jobs and may not be interested in training programs. The harvest time (September-October) also
likely takes up some applicants. Therefore, November-December seem a better timing for the oversubscription. The
provinces of Antalya, Bayburt, Denizli, Diyarbakir, Gaziantep, Isparta, Izmir, Kayseri, Kirikkale, Kocaeli, Sakarya,
Trabzon, Usak stated larger seasonal effects.
12 Some provincial offices faced unexpected budget constraints to open new courses. However, the ISKUR General
Directorate backed up the evaluation in a timely manner avoiding any disruption.
10
10
(reported in Table 2 above). Those provinces that did not meet the initial target numbers and/or did not
present any reserve courses, are contacted. Table A3, Table A4 and Table A5 in the annex show the trainings
which have the largest number of trainees in 2006, 2007 and 2008, respectively. In Table A6 in the annex, the
trainings with the largest number of trainees between Jan-Nov 2009 are listed.
Figure 4: The Set of Vocations included in the Evaluation Sample
Computer /Computer programming
3
2
2 1111
2 2
Accounting professionalist/Computerized Accounting
Babysitter
Cashier
24
Foreign Trade and Customs Professional
2
Fitter(natural gas)/Plumber
3
Old and Sick People Nurse
Welder, Gas Arc
3
4
Retailing and Merchandising/Salesperson
Cook
Weaver, carpet-ehram
5
Modelist/Stylist
Coiffeur and Hair care/Skin care and beauty
Operators (forklift/sewing machine)
7
3
38
5
Applied Basic Electronics/Electronic technicians
Finalcut
Waiter, service
6
9
3.4.
Medical Secretary
Human resources Management
Manufacturer, furniture
Moulder, grouting
6
Mechanic, maintenance
Scheduling the interviews
The interviews for the trainings were scheduled so that a member of the evaluation team could be
physically present to monitor the quality of the data. Provinces were grouped according to their
geographical proximity and the interviews for each group are set in a nearby date. Within each group, the
interviews were scheduled to be as close as possible. In practice, however, this was difficult to implement.
The team tried to be present in at least the first interviews of each group to identify and solve early on the
main problems. All courses were scheduled to have a minimum of 12 days between the interview and the
start of the courses. Although a larger period is better for the data collection purposes, it may raise attrition
from the treatment. For example, the trainees may be discouraged from not hearing the result and may apply
another course or be less motivated. As a result, the optimum number of days between the interviews dates
and the announcements (trainee list) are 10 to 15 days. The trainee lists must be announced 2 weekdays
11
before the starting dates of the courses. Special attention was given to scheduling of interviews to promote
the oversubscription of the courses. Below we highlight some important steps taken by the team regarding:

Days between public course announcements and the interview dates. The average time between
the course announcement and the interview dates is approximately one month. This period is considered
sufficient to allow the dissemination of the information and therefore expect a large awareness in the
target population.

Date of the course interviews and course starting dates. Within the same provinces, and to the
extent possible, the ISKUR provincial offices did not conduct interviews for different courses in the
same day. Moreover, if courses were very close in content, the team tried to maximize the numbers of
days as much as possible. The same applied to the course start date. This likely increased the number of
applicants for each interview thus improving the likelihood of oversubscription.

Scheduling of additional interview dates. For some courses, there was no oversubscription in the first
round of interviews. The provincial offices scheduled additional interviews for the same course to
achieve this. Team tried to balance two forces when deciding the number of days between the first and
the follow up interviews. On the one hand, it had to be long enough to allow for more time for
announcing. On the other hand, it had to be sufficiently close to keep interested successful candidates in
the first round of interviews. For most courses, the additional interviews took place within one week of
the first interview.
Even though, the oversubscription was not a problem in most provinces, box 2 illustrates that Istanbul
presented multiple challenges.
Box 2: The Fieldwork in Istanbul
The field work in Istanbul presented several challenges. However, the major challenge probably related with the low
initial oversubscription for most of the evaluation courses. Given its large size and geographical spread, Istanbul is a
unique province. Labor markets are characterized by a large and dynamic informal sector, which creates a diverse set of
job opportunities at decent earnings. Therefore, in Istanbul the ISKUR vocational trainings are possibly less attractive
as there are large opportunity costs for the trainee’s time.
Since the evaluation team faced very low over subscription rates in Istanbul, several actions were taken to minimize this
problem. First, evaluation team organized a longer and more detailed capacity building event with the training providers
in Istanbul. In addition, several follow up meetings with the multiple ISKUR offices in Istanbul were conducted to
involve them actively at all the states of the evaluation. Second, the ISKUR management (centrally and locally) became
more pro-active in Istanbul following a letter to the ISKUR central and provincial offices stating the importance of
Istanbul to the study and the needed additional effort/resources to improve the oversubscription. Reserve courses were
planned and additional interviews for problematic courses were scheduled. Finally, training providers and ISKUR
Istanbul provincial offices were requested to disseminate more aggressively the trainings.
12
4. Defining the Evaluation Sample:
4.1.
The most relevant groups in the evaluation
The training providers were requested to provide a larger than usual shortlist of the “best
interviewed candidates” for this study. In their standard screening of candidates, almost all eligible
applicants to the ISKUR courses are interviewed by the training providers; only a small number is selected to
participate. Most of the applicants who meet basic requirements for courses are usually interviewed by
training providers.13 The group of best candidates is selected and admitted into the program. A small number
of applicants are wait-listed while the remaining are rejected. However, for this impact evaluation, the
procedures were slightly adjusted. Training providers interview all candidates that are suitable and select a list
that is longer than usual. The list of the “best candidates” is now at least twice the size of the training slots.
Figure 5: Selection of ISKUR Trainees
All Applicants
Eligible Candidates
All Successful Candidates
Non-eligible Candidates
Waitlisted 2
The Best Candidates
Treatment
Group
Control
Group
Waitlisted
Training providers are requested to select a list of the best candidates that is at least 2.2 times the
number of training slots. Take for example, the case of a course with a total of 50 seats, this implies that the
training provider will select the best 120 applicants. Among these, 50 are randomly assigned to the treatment
group, 50 to the control group and 20 to the waitlisted group.14 When oversubscription implies a ratio of
applicants to training size, between 1.8 and 2.2, the size of the treatment and control groups together are
The course requirements are very diverse across the trainings. For example, applicants for a computer course may
need a secondary school degree, while other vocations are offered only for female participants.
13
14
In this case, the ratio of the number of best applicants to course capacity is 2.2 (=120/50).
13
smaller than the total course capacity. In our example above, assume that the training provider selects only a
list of the 100 best applicants. In this scenario, the evaluation team instructs that only 40 applicants are
randomly selected to be treatment and control. The remaining 10 training slots need to be filled with the
applicants that are waitlisted (i.e., they are trainees who are left outside the evaluation study). When the course
capacity is smaller than 20, the size of the list of the best applicants list is smaller than 1.8 times and the
treatment group is smaller than 10 individuals, the course is not included in the evaluation study. The reason
for the former was to minimize the dispersion in the number of courses considered.
After receiving the short list of the best candidates, ISKUR conducts an automatic randomization
among the treatment and control groups. Treatment and control groups are randomly selected among the
group of the best applicants (together with the waitlisted). The treatment group is the set of individuals who
will be offered a slot in the training program. The control group is the set of individuals who will not be
offered a slot in the training programs. The waitlisted group is the set of individuals who will not be offered a
slot to participate but are also not rejected. The participants in the evaluation consist only of those individuals
in the treatment and control groups. This is the only group who will be interviewed at baseline and follow up.
The implementation of the evaluation requires the selection and labeling in the ISKUR’s MIS of
some additional groups beyond the treatment and control group, including a second group of
waitlisted and the non-eligible trainees. The waitlisted are the set of randomly selected applicants among
the group of the “best candidates” defined during the interview process.15 This group is approximately 20%
of the list of all the successful candidates. The second waitlisted are the set of individuals who, although not
among the “best” applicants, are considered good enough by the training providers to participate in the
trainings. They are not randomly selected and account for approximately 10% of the number of training slots.
When an individual in the treatment group drops from the training (up to a 10th of the training period), the
training provider can replace the trainee first with a replacement from the waitlist and, upon exhaustion of the
first waitlist, they may access the second waitlist. Finally, the trainees not in the evaluation group are relevant
only when the oversubscription is not large enough and course capacity is small. In this case, a randomly
selected group among the most successful candidates will take up the trainings but will not be part of the
evaluation sample.16 Finally, the group of individuals who were interviewed but are considered not suitable to
take up the courses by the providers are also labeled accordingly by the MIS system.
All these groups are appropriately labeled in the ISKUR’s MIS. In sum, in the ISKUR MIS, each applicant is
classified into one of five categories - treatment, control, waitlisted, waitlisted 2, trainee not in the evaluation
and not suitable candidate17.
Figure 5 shows that this group is randomly selected together with the set of treatment and control groups.
In this case, the sum of treatment and control groups are together smaller than the total course capacity. The final size
depends on the size of the list of best candidates and on the course capacity.
17 The baseline and follow up data will only be collected for the set of individuals in the evaluation sample. This includes
only those that are either in the treatment or in the control groups.
15
16
14
Box 3: The Pilot: Testing Main Design and Implementation Features of the Evaluation
In September 2010, and before the trainings start nationwide, a small pilot was designed to test the
quality of the baseline questionnaire and the main features in the evaluation. The pilot study focused
on two vocational trainings: secretary and graphic designer. The pilot highlighted important facts with
implications for the scale up. Indeed, following the pilot there were also several adjustments in the
monitoring and information system.
First, the pilot highlighted a large number of dropouts. This implied that the providers have to select a
large enough list of the “best candidates”. Table 3 below shows the applicants’ dropout rate from the
pilot until the start of the course, and up to 10% of the training period:
Table 3: Learning from the Pilot: The Applicants’ Drop-out Rate in Course X
Number of applicants
Dropouts
after
the
announcement of the
trainees and until the start
of the course
Dropouts between the
start of the course and up
to 10%
Total dropouts:
(% applicants)
Treatment
50
Control
50
Waitlisted
10
Waitlisted 2
13
Total
123
8
0
3
1
12
1
0
0
0
1
11
0.22
0
0
3
0.30
1
0.08
15
0.12
Table shows that for the graphic designer course, 22% of the initially selected trainees never attended
or stopped attending the trainings early on. The numbers are even higher for the secretary course, and
the data is provided in Annex A1. The course providers are allowed to fill these slots with individuals
from the waitlist up to 10% of the course period. Furthermore, the table in the annex for the secretary
course also shows that some of individuals in the control group ended up being invited to the course.
These could not be replaced due to the small size of the initial waitlisted group and did not allow a full
replacement. Therefore, a second waitlist was created – waitlist 2 – based on a small set of second best
applicants.
Finally, the pilot also illustrated a number of other important points. First, the non-response to the
survey was close to 10%. Second, the interface for the electronic registration had to be improved and
an alternative plan defined for the areas with low internet access. Third, the interview process need to
emphasize the requirements and content of the trainings. For example, many applicants that were
interviewed knew little about the course and eventually dropped out. A stronger emphasis on the
content of the trainings, as well as the period and location of the trainings, was given during the
interviews and in the invitations directed to potential trainees (through SMS, emails).
15
4.2.
The short and medium term take up of the trainings by the nontrainees
Different strategies were discussed with ISKUR to discourage individuals in the control groups from
taking other trainings. Maintaining most of the control group without participating in any training during
the evaluation period (or approximately 18 months) is a very important dimension of the identification
design. Even though most courses are only offered once a year, applicants can apply to other training courses.
The ISKUR MIS tracks some of the registered unemployed overtime. In particular, it identifies when these
individuals that are not suitable or waitlisted became trainees some months following the completion of the
courses they initially applied for. Table 2 reports this information for six months and one year, respectively. 18
Based on this information it was decided to discourage the participation of the control group in trainings for
6 months.
Table 4: The short and medium term take up of the trainings by the non-trainees
Individuals who
became trainees
Total
Percentage of
individuals who became
trainees
Panel A: 6 months after application
Waitlisted candidates
Not suitable candidates
Panel B: 12 months after the application
93
306
703
2,692
13.2
11.3
Waitlisted candidates
109
703
15.5
Not suitable candidates
354
2,692
13.1
Initial Status
Source: Author’s based on the ISKUR MIS for the 23 provinces in the evaluation (2009).
Note: Panel A reports the number of “waitlisted” and “not suitable” candidates who became trainees 6 months after
applying. Panel B reports the number of “waitlisted” and “not suitable” candidates who became trainees 12 months after
the application. “Individuals who became trainees” are the applicants that become trainees after not being selected in the
initial interviews (when they were labeled waitlisted and not suitable in the system).
The evaluation team is always informed if individuals included in the control group apply to other
training programs. Although some exceptions may be granted and applicants in the control group can
become trainees, these are rare events and monitoring is strict. The evaluation team and ISKUR defined a
clear set of rules to guarantee the high quality of the evaluation. In particular, the evaluation team and ISKUR
committed not to accept any individual in the control group into the training course without the prior
approval of evaluation team. Until the final completion of evaluation, ISKUR provincial offices will also send
reports regarding the applications of control group members.
18
Data refers only to the 23 provinces that are part of the evaluation sample.
16
The ISKUR provincial offices in smaller provinces considered the program take up among the
control group, and especially among the most skilled, a greater concern. ISKUR provincial offices in
smaller provinces have more difficulties in maintaining the control group members since the individuals know
each other. In addition, courses with higher pre-requisites for the applicant’s skills – such as computerized
accounting, network specialist, airplane maintenance – are also more likely to have controls contaminated.
Since the skills are higher, individuals are more likely to apply several times for other courses.
5. Improving the ISKUR IT monitoring system
This section discusses the improvements in the ISKUR MIS with the objective of improving the
dissemination of the training courses, improving the training Providers Accounts to facilitate the collection of
information, the randomization code and the procedures that minimized the take up of the trainings from
applicants in the control group. Finally, we describe some of the periodical reporting issued by ISKUR
through their MIS. They automatically generated the lists of the most successful candidates among all the
applicants.
5.1.
Dissemination of the training programs at the local level
The announcement of the courses relied heavily on the information available through the ISKUR
MIS. Individuals may become aware of the courses through the announcements that sent to a group of
randomly selected individuals among the registered unemployed, and after defining other desirable
characteristics depending on the content of the vocations.19 In addition, individuals may also apply to a
course through the internet or at the local ISKUR office. All these individuals are labeled in the ISKUR M&E
system as "invited" and the IT system will automatically send emails or cell-phone messages to the group.
Some provincial labor offices also mail letters.
The content of the mails, emails and cell-phone messages is enriched to provide more specific
information about the content and location of the courses. For example, the ISKUR’s web address is
added to the content of the cell-phone messages. Also included, some information on the number of the
course and the date and address of the interview. In practice, there are large trade-offs between a more
intensive outreach and the saturation of the individual when the number of the messages increases.20 In
addition, information about course duration (days of week, hours a day - during the week and weekend
separately, total number of days, total number of months), the details on course provider (title, address,
The most common restrictions include minimum and maximum age restrictions and a minimum level of education.
Other constraints include the date of registration with ISKUR, location and size of the population.
19
ISKUR staff was concerned with the potential increase in costs – originating from sending two instead of one text
message – without necessarily improving the outreach. Some of the unemployed do not read the second message. The
emails and the webpage of the training courses on ISKUR's system also improved.
20
17
phone numbers, the address of their webpage), the rules (attendance, signing a commitment letter, restriction
on taking ISKUR courses in 6, 12 and 24 months, being unemployed, and etc), the minimum criterions to
apply for this course, the rights (stipend, insurance, unemployment payment – the trainees are allowed to
receive it while they are attending to a course) and employment consultancy were added to the contents of emails and webpage.
5.2.
Improving the training provider’s accounts in MIS
The training provider’s account in the MIS plays an important role in the evaluation. All the training
providers must be registered with ISKUR to offer a training course. The results of the interviews, including
the list of the best candidates, are recorded in the MIS in this section. The ISKUR staff has access to this
information confirming the applicants' current employment status and whether an applicant took a course
from ISKUR in 24 months. Updated contact information can also be collected. The successful applicants are
informed and invited to courses before their names are registered into the system. Before the evaluation, the
trainees were classified as such in the system only after some candidates drop out of the courses.21 This
change allows ISKUR to make a better assessment of the over subscription and drop-out rates. 22
The new interface is a friendly tool that updates applicants’ and providers’ data into the system. The
staff conducting the interviews can register the applicants and check whether they are currently working.
Through this interface, the provider can also have access to any information ISKUR already has on this
individual. In addition, the information can be filled or updated.23 Whenever the individuals are working, a
warning message is prompted. Those working are not allowed to take up the ISKUR trainings. The system
formally registers these individuals as “Working”. However, in practice, almost none of the currently
employed applicants are excluded from the interview. This happens because applicants may quit if accepted
into the trainings. In addition, there is no formal regulation on employment status of interviewees.
There could be different reasons for applicants to drop out of the courses. For example, some may change their minds
and decide to take another course, others may find a job.
22 During the interviews, the providers can either “Search” an existing profile of candidates or “Register” new applicants.
After entering the identity number of an applicant and clicking on "Search" button, course providers can see whether
this person is registered to ISKUR or not. If an applicant did not register to ISKUR, he/she is not allowed to interview.
Otherwise, he/she must enroll.
23 The headings on this screen are: 1) ID Number; 2) Name; 3) Surname; 4) Education level; 5) Home address; 6) Phone
number (home); 7) Cell phone no; 8) E-Mail; 9) Emergency contact Name - Surname;10) Emergency contact Phone
Number (home); 11) Emergency contact Phone Number (work); 12) Emergency contact Phone Number (cell);
13) Employment status (Yes/No: Recorded automatically after clicking on “Search” button). Note that both the edited
information and the original information are being kept on the system.
21
18
Although the system is very robust, in some provinces the electronic system failed and the training
providers recorded the relevant data in paper format. Course providers can record the information from
different interviews simultaneously. While recording the data, the provider can view the information for all
the applicants, recorded up to that time in another interface. The order of this list depends on the time of
recording. Each course provider can contact interviewees with the help of this list and the contact
information. In case the system fails, due to for example a power outage, a hard copy of the results needs to
be available. The training providers are responsible to upload this data electronically as soon as the system is
back to normal.
The system is important tool to enforce some of the pre-requisites needed to take up the training
courses. After registering or uploading the info on a new applicant, the system automatically checks some
requirement for the course, including age, education level, gender and whether person took a course over the
last 24 months. If providers fail to enter all the requested information, the interviewee will not be able to be
recorded as a trainee.
The information on whether each applicant is successful during an interview is recorded into the
MIS which latter randomizes among the group of the most successful candidates. The training
provider account, keeps the information on whether each applicant was successfully passed the interview.
The system also identifies the group of individuals who are not among the best applicants but are considered
good enough to be part of the waitlisted-2 group. Providers are requested to enter the outcome of the
interview into the system on the same day, or at most one day after the interview. This is important to ensure
a timely randomization. It is also important for evaluation purposes because the baseline data need to be
collected prior to the disclosure of the results and to the start of the courses. ISKUR provincial offices are
requested to complete controls on the applicant’s employment status and share the final randomized list of
applicants at most 2 days after the interviews.24
5.3.
The randomization code
The randomization was implemented through an automated code built into the MIS. The code
checks the validity of the main parameters and produces different results depending on the strength of the
oversubscription. The randomization is conducted by computer after the provider’s finalize the short list of
the most successful candidates. The computer code randomly divides the group of the most successful
interviewees into treatment, control and the waitlisted. As explained in section 4, the randomization is
different depending on the relative size of the list of the most successful candidates to the number of training
slots. Let p be the size of the successful list relatively to the course capacity. Then p= m/c, where m is the
number of most successful candidates and c is the course capacity. Depending on the value of c and on whether
24
This list is denoted “Questionnaire Report” in the MIS.
19
(1) p<1.8 or (2) p>=1.8 & p<2.2 or (3) p>=2.2, different treatment and control groups are constructed. In
addition, the randomization code also checks the validity of the threshold on the oversubscription.25 Finally,
the randomization is stratified by gender and age.
5.4.
Minimizing the take up of the trainings by the control group
The system is refined to prevent the take up of the trainings from those in the control group. On the
one hand, applicants selected into the control (and also in the treatment) group no longer receive invitations
for additional ISKUR courses. This is expected to minimize the likelihood of re-application, especially for
controls. On the other hand, the IT system prompt an error messages if any of the applicants in control
group is labeled in the future as “trainee”.26 There are exceptions that may be overruled by the evaluation
team, though. In the IT system, the labels of treatment and control that are initially allocated immediately
after the randomization remain constant for the remaining of the evaluation.
5.5.
The lists of the most successful applicants
A formal report with detailed information on all the individuals in the evaluation sample is produced
by the system and shared with all counterparts. Following the randomization, the system produces a list
with all the individuals in the evaluation sample. The report is accessible to the ISKUR staff involved in the
study, to the course providers and to the survey firm.27 The report gives priority to the control group
members so that they are interviewed earlier. In particular, the control group is labeled as “1st group to be
interviewed”, treatment group is labeled as “2nd group to be interviewed” and finally waitlisted will be labeled
as “3rd group – Reserve”. The survey firm receives this report after it is revised by the IE field coordinator.
If the list of successful applicants is not enough to constitute treatment and control groups, the code gives an error
and lead the user to reach and inform the evaluation team.
25
System gives an error saying that “This person is not allowed to be a trainee. Please contact with ISKUR General
Directorate”. Only one staff from ISKUR General Directorate can give permission for an individual in the control
group to take a training program.
26
The report is denoted “Questionnaire Report” and it contains the following information: a) ID Number; b) Name;
c) Surname (Recorded by the course provider); d) Surname (on the system records); e) Gender (on the system records);
f) Date of birth (on the system records); g) Education level (Recorded by the course provider); h) Education level (on
the system records); i) Home address (Recorded by the course provider); j) Home address (on the system records);
k) Phone number (home) (Recorded by the course provider); l) Phone number (home) (on the system records);
m) Phone number (cell) (Recorded by the course provider); n) Phone number (cell) (on the system records); o) E-Mail
(Recorded by the course provider); p) E-Mail (on the system records); q) Emergency contact Name-Surname (Recorded
by the course provider); r) Emergency contact phone number (home/work) (Recorded by the course provider);
s) Emergency contact phone number (cell) (Recorded by the course provider); t) Questionnaire status.
27
20
The report with the information on the waitlisted applicants is accessible to the training providers
only after the beginning of the course. The report produced through the system also includes contact
information of the trainees not suitable and on the waitlisted-2. This information is available on the electronic
format and in paper. Course provider can view this list two days before the beginning of the course through
their accounts. The system prevents to have access to this information earlier than this period. 28
6. Important Activities for Training Providers, ISKUR Provincial Offices,
and the Evaluation Team
This section summarizes the main activities and responsibilities of the ISKUR provincial directorates and for
the course providers. The evaluation involves the close collaboration between the ISKUR central and
provincial offices, the training providers, the World Bank and the survey firm. They differ substantially
depending on the status of the evaluation and thus they are divided into: (a) activities conducted before the
interviews, (b) activities conducted during the interviews, and (c) activities conducted following the
interviews. Some responsibilities are present throughout the whole evaluation. For example, the ISKUR
provincial offices cannot inform the participants that they are part of an evaluation study. In addition,
throughout the whole evaluation period, ISKUR is responsible for supporting and ensuring highest quality
possible for the activities led by the training providers.
Before the interviews
ISKUR Provincial offices and training providers collaborate closely to disseminate the information
on the training programs. Several communication channels were used, including announcements through
the web pages of the training providers and ISKUR, the network of training providers and previous trainees.
Cell-phone messages, billboards, colorful and attractive posters or flyers, advertisements on newspapers, local
magazines and TV were some of the other channels used. Often, this was conducted in close cooperation
with all the municipalities. The contact information on the potential trainees in the MIS systems was used to
send the announcements (including letters, SMSs). ISKUR also motivated the course providers to advertise
aggressively and prepare a new protocol including the requirements and duties related with the evaluation.
Finally, ISKUR senior staff received training and they trained their junior staff before the interviews so that at
least one ISKUR staff can attend each interview. This was important to ensure high quality.
The evaluation team is responsible for the overall coordination of the study, including drafting the
evaluation letters and for the follow up visits to all provincial offices. The evaluation team prepared two
important letters before the evaluation started. The letters informed on the selection criteria for the training
28 The course provider's interface with new screens and reports was piloted across the 23 provinces in August 2010.
21
courses selected to be part of the evaluation; and on the implementation of the evaluation. The ISKUR
general directorate distributed these letters to provincial offices. Letters give precise directions, state deadlines
and responsibilities. The evaluation team has followed up with all the provincial offices before the first
interviews took place.29
During the interviews
On the interview day, and before the start of the interview, training providers need to provide
detailed information about the courses to interviewees, including: (1) pre-requisites for the course, (2)
benefits received (e.g., stipend, unemployment wage, insurance) and conditions linked with being an ISKUR
trainee (e.g., registered with ISKUR, unemployed), (3) the commitment letter (to be signed by interviewees),30
(4) exclusion from taking up other courses supported by ISKUR after being a trainee ( e.g., prohibited to take
another ISKUR course within 24 months), (5) date and form of the disclosure of the results (6) course start
and end dates, (7) Course duration (e.g., days, hours), (8) a brief definition of the vocation and the possible
job opportunities, and (9) information about the survey study and the possibility of being invited to answer
the “face-to-face” survey. Training providers also need to ensure that they correctly assess the willingness of
the interviewees in taking and completing the trainings. This can be done with the help of a set of
standardized questions on the type of commuting and overlap of trainings with the school and other
activities.31
Informing the applicants about the main purpose of the evaluation study improves the willingness to
take-up the face-to-face survey. Some of the surveys were conducted in the interview place. Applicants
were informed that this study is being led by ISKUR and that their support is a key ingredient for its success.
It is disclosed that the ultimate objective of the study is to improve the services provided by ISKUR. This
clarification was important to gain the participants’ trust and support. It is clearly stated that participating in
the face to face survey does not affect in any way the outcome of the interview.
The meetings were conducted in two parts. In the first part, ISKUR staff is informed about the implementation of the
evaluation and their role during the process. In second part, ISKUR and course providers were trained on how to use
new interface in the MIS. The deadlines and the responsibilities of each party were also communicated in this session.
30 The copies of the commitment letter (which includes rules and rights) or any material related with the training can be
presented in waiting rooms.
31 “Why did you choose this course?” “How far away do you live from course location/how long did it take for you to
get here?” “What is your primary means of transport?” “Do you think you will be able to get here every day?” “Do you
have permission to from your relatives to attend the course?” “Will this course conflict with exams/school?” “Do you
have a job that starts soon or any other commitment that might prevent you from taking this course if accepted?”
29
22
After the interviews
The ISKUR provincial offices need to confirm the employment status of the most successful applicants. The
final short list is sent to the evaluation team. Those identified to be working need to be removed from the list
and replaced by candidates in the waitlist.
Whenever there is not enough oversubscription, the ISKUR provincial offices are responsible to schedule
additional interviews. This is done in close collaboration with the training providers. They are also responsible
to schedule additional courses – and invite new potential candidates – when the course in the evaluation
sample presents problems to the identification strategy.
Until the completion of the study, ISKUR provincial offices need to report to the evaluation team on the
number of re-applications from those in the control group. The evaluation team and the ISKUR general
directorate need to be informed whenever there are several attempts from any given candidate. Only the
evaluation team and ISKUR General directorate can overrule this restriction.
Provincial offices need to cooperate closely with the survey firm. Some baseline face-to-face interviews were
conducted in the provincial offices when the space was available. ISKUR can also clarify the objectives of the
study and inform on the follow up contacts by the survey firm.
7. Data Sources
The impact evaluation requires data collection in at least two points in time, before and after the
trainings take place. The baseline survey was collected face-to-face, before the training courses start. A
follow-up survey is planned to be also face-to-face and will be collected more than 18 months after the
trainings started. (possibly early in 2012). Between the baseline and the follow-up, the evaluation team is
planning a short exercise to track some of the individuals in the sample. This is important to minimize the
attrition from the sample, which is composed by relatively young and mobile individuals.
The baseline data collection started in September 2010 in Istanbul and was finalized in January/
February 2011. This data will capture detailed socio-economic information of each applicant as well their
labor market trajectories. Some psychological treats of applicants - relevant to measure their efforts and
persistency in searching for jobs - together with household characteristics and perceptions on training quality
was also collected. An analysis of this data is done in World Bank (2011).
23
Box 4: Looking into the Content of the Baseline Survey
The baseline survey collected detailed information on the training applicants. First, it collected current
and past labor market outcomes of applicants. The proposed set of indicators will inform on the
individual current employment status (employed, unemployed, searching), labor earnings as well as on
other indicators of job quality if employed (including number hours working, formality of the
employment contract, social security coverage).
Second, it gathered basic information on the characteristics of the potential trainees. Third, it also
identifies some of the channels (or intermediate outcomes) through which ISKUR trainings might
influence workers’ employment and earnings outcomes. Trainings may increase individual productivity
(or wages) through an increase in individual skills (either cognitive or non-cognitive skills), through
reduced costs of searching for a job, through greater effort (optimism/enthusiasm) or networking. A
special module with Raven’s test has also been included to capture an individual’s cognitive skills.
Forth, since the ISKUR trainings seem to be very useful for trainees we also propose to better
understand why individuals are not capable of investing in these trainings on their own. For this we
propose the collection of some information on individual assets and credit constraints, on the individual
degree of risk aversion - to human capital and other investments, on individual’s behavioral responses to
multi-year choices and on the accuracy of individual expectations on the importance of the trainings.
Finally, the evaluation will be looking at changes over time in the main outcomes of interest. Therefore,
the evaluation team is especially concerned with minimizing attrition firm the sample. Therefore,
detailed information was collected at baseline to track participants over time. This included information
on current and permanent address and the address and other contact information for three close
contacts (e.g. family, friends and neighbors). This information will enable the tracing of participants in
the follow up survey.
The baseline data collection was done through a face to face survey collecting comprehensive data
on treatment and control individuals. Special attention was given to (i) pre-program levels of the main
labor market outcomes of interest; (ii) pre-program measures of key control variables, including
characteristics which are expected to result in heterogeneous reactions to the treatment; and (iii) tracking data.
A list of 6,072 evaluation participants was shared with the survey firm. A total of 5,318 interviews were
completed. Table A7 in the annex reports the number of trainees in 2009, the number of courses in the
evaluation sample, and the number of registries shared with the survey firm for each of the provinces in
Turkey.
24
8. Annex
A1: The number of dropouts in pilot - Secretary Course
Pilot Course 2 : Secretary
Course
Number of applicants
Number of dropouts after the
announcement
of
the trainee list till the beginning
of the course
Number of dropouts after the
beginning of the course till the
1/10th of the course
TOTAL number of dropouts:
Percentages of dropouts:
Treatment
Control
Waitlisted
Waitlisted2
Total
25
25
3
NA
53
8
5
3
NA
16
0
2
0
NA
2
8
7
3
NA
18
0.32
0.28
1.00
NA
0.34
25
A2: Selection of provinces according to the unemployment rate and the number of
oversubscribed courses in 2009
Code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
Province
ADANA
AGRI
ANKARA
ANTALYA
AYDIN
BALIKESIR
BAYBURT
BINGOL
CANAKKALE
CORUM
DENIZLI
DIYARBAKIR
DUZCE
ELAZIG
ERZINCAN
ERZURUM
ESKISEHIR
GAZIANTEP
HATAY
ISPARTA
ISTANBUL
IZMIR
KASTAMONU
KAYSERI
KILIS
KIRIKKALE
KOCAELI
KUTAHYA
MANISA
MUS
NEVSEHIR
RIZE
SAKARYA
SAMSUN
TEKIRDAG
TOKAT
TRABZON
USAK
YOZGAT
Unemploymen
t Rate in 2009
20.5
6.5
11.8
9.7
12.4
7.9
4.7
14.8
7
7.5
9.4
15.7
10.2
15.5
6.1
6.2
9.9
16.8
17.7
8.1
11.2
11.8
5.3
11.1
10.9
11.1
11.6
6.5
9.8
13.2
9
5.2
8.5
7.8
9.1
5.9
6.1
9.2
12.2
Strata
2
1
2
1
2
1
1
2
1
1
1
2
2
2
1
1
1
2
2
1
2
2
1
2
2
2
2
1
1
2
1
1
1
1
1
1
1
1
2
Selected
Provinces
0
0
1
1
0
0
1
0
0
0
1
1
1
1
0
1
1
1
1
1
1
1
0
1
0
1
1
0
1
1
0
0
1
0
1
0
1
1
0
Note: Strata equals one if unemployment rate < %10, = 2 if unemployment rate > %10.
26
A3: ISKUR vocational training courses with the largest number of trainees in 2006
Number Of Trainees
Male
Female
Total
Order
Name of the Vocation
Number
of
Courses
1
2
3
4
5
6
7
8
9
10
Cable Network Assembly Operator
Computer
Production project for the disabled
Spot welding and Machine-tool assembly
Laborer, manufacturing/readymade
Fitter, natural gas
Welder
Manufacturer, furniture
Sewing Machine Operator
Cook
148
125
1
45
15
19
16
14
13
12
1884
1434
720
802
221
326
227
209
95
188
1085
444
720
0
226
3
37
20
108
2
2969
1878
1440
802
447
329
264
229
203
190
622
8603
3823
12426
TOTAL (Overall)
A4: ISKUR vocational training courses with the largest number of trainees in 2007
Order
Name of the Vocation
1
2
3
4
5
6
7
8
9
10
Computer operator
Computing software operator
Computerized Accountant
Sewing-machine operator (fabric)
Electric arc welder (with machines)
Furniture manufacturer
Security guard
Electronic computing operator
Sales person
Sewers by hand and with sewing machine
Number Of Trainees
Number of
Courses
Male
Female Total
TOTAL
27
117
70
45
28
19
16
11
13
10
8
1,346
741
473
161
309
241
195
191
94
76
271
249
398
336
2
1
28
23
84
97
1,617
990
871
497
311
242
223
214
178
173
1,200
16,463
6,371
22,834
A5: ISKUR vocational training courses with the largest number of trainees in 2008
Order
Name of the Vocation
1
2
3
4
5
6
7
8
9
10
Computer operator
Computing software operator
Computerized Accountant
Manuel Worker
Sewing-machine operator (fabric)
Sales person
Office clerk (general)
Security guard
Gas Arc Welder
Sewers by hand and with sewing machine
Number
of
Courses
183
114
69
45
49
28
1
26
24
18
1,806
TOTAL
Number Of Trainees
Male
Female
Total
1,831
1,119
601
777
152
266
250
371
392
77
474
446
548
296
859
266
250
72
1
272
2,305
1,565
1,149
1,073
1,011
532
500
443
393
349
20,586
11,341
31,927
A6: ISKUR vocational training courses with the largest number of trainees between Jan-Nov
2009
Trainees
Order
Name of the Vocation
1
2
3
4
5
6
7
8
9
10
Computerized accountant
Security officer
Computer manager
Babysitter
Sewing machine operator (fabric)
Security officer (armed)
Cook
Caregiver (elder and sick)
Slopworker/ Confection worker
Computer repairer (maintenance and repair )
Number
TOTAL
28
Percentage
8,658
7,707
7,552
4,125
3,311
3,255
3,243
3,026
2,854
2,828
6.5
5.7
5.6
3.0
2.4
2.4
2.4
2.2
2.1
2.1
133,284
100
A7: Sample Trainees, Courses in the Evaluation and Surveyed by the Firm
No
Province
1
ANKARA
2
ANTALYA
3
BAYBURT
4
DENIZLI
5
DIYARBAKIR
6
DUZCE
7
ELAZIG
8
ERZURUM
9
ESKISEHIR
10 GAZIANTEP
11 HATAY
12 ISPARTA
13 ISTANBUL
14 IZMIR
15 KAYSERI
16 KIRIKKALE
17 KOCAELI
18 MANISA
19 MUS
20 SAKARYA
21 TEKIRDAG
22 TRABZON
23 USAK
TOTAL
Number of
trainees in
2009
Target
interviews
Number of
courses in the
evaluation sample
Contacts sent to the
survey firm
3,344
1,771
249
1,420
450
702
1,204
836
2,450
751
2,889
1,152
19,208
1,971
3,067
1,329
7,519
3,547
1,938
1,208
1,709
1,628
1,484
61,826
308
163
100
131
100
100
111
100
226
100
266
106
1,771
182
283
123
693
327
179
111
158
150
137
5,925
9
3
3
4
2
2
2
2
7
3
8
3
29
5
5
4
15
6
1
4
5
4
4
130
300
150
150
152
100
52
100
100
340
120
500
110
1,314
190
282
120
604
290
200
118
150
180
150
6,072
29