Presented To
Uganda Evaluation Association & ECD Project
Kampala Evaluation Talk
Thursday, 2nd October 2014
1
•
•
•
•
•
•
•
•
Muhwana Wilberforce, MWSES CONSULT
Ntinda, Steka House, 1st Floor
Plot 3122 Ntinda – Kiwatule road
wamuhwana@gmail.com;
mwssconsult@rocketmail.com; 0772429679;
0701242652
Social Sector Development Consultancy
Education, Health, Community Livelihoods
Policy/project formulation, Baseline surveys, Mid and
End Line Evaluation, Impact Assessments, Tracking
studies
Government ministries, Institutions, Donor agencies,
NGOs
2






MUHWANA WILBERFORCE (LOCAL CONSULTANT) – Kampala,
Uganda
CLAUDIA MUMMENTHEY (INTERNATIONAL CONSULTANT) –
Cape Town, South Africa
Sourced by BTC separately
Teamed up to harmonise the methodology for the evaluation
Delivered a single inception report
Supported by 7 RAs and 3 data entrants
3
•
•
The Business, Technical, Vocational Education
and Training (BTVET) Subsector is under the
Ministry of Education and Sports (MoES), BTVET
Department and was established under TVET Act,
2008.
BTVET offers training under an overlapping
three- tier system:
– (i) P7 leavers to enroll in technical schools & community
polytechnics,
– (ii) ‘O’ leavers enroll in technical institutes & vocational
training institutes,
– (iii) ‘A’ leavers enroll in technical colleges; and
– (iv) recently non-formal short courses across the
various levels.
4




The sub-sector fell short of the overall goals and
objectives regarding the production of a ‘critical mass’
of graduates that could readily be absorbed into the
existing labour market.
Internally the sector and subsector was grappling with
issues of low quality of education and irrelevant
curricula (UNESCO, 2005). Externally the BTVET system
was not corresponding with the needs of the private
sector (ESAPR, 2010).
The private sector was not involved in curriculum
development thus there was no realignment of
knowledge and skills acquisition to the real needs of the
labour market.
There was widespread unfavourable perception and
recognition of the BTVET programs among the public
5



MoES embarked on developing a 10-year
BTVET Strategic Plan in 2011
Six studies had been conducted ex- ante
The Tracer Study conducted belatedly
◦ To generate data and information about the
effectiveness of different BTVET programmes in
Uganda as an input into the strategic planning
process of identifying potential improvement areas
for the overall system.
6
•
Overall objective: To provide an in-depth
analysis of the employment destinations of
graduates BTVET programmes.
– To undertake a comparative analysis of factors
affecting external effectiveness and relevance of
training provided by the public and private sectors,
– To identify employment destinations/career paths
of completers of BTVET programmes and factors
determining post – training careers
7
◦ Accurately assess the potential significant
differences in the external effectiveness of different
BTVET programmes
8

Tracing of a “reachable” (i.e. within the given
restrictions of data availability and time
constraints of the assignment) and
representative cohort of BTVET graduates
from the existing BTVET system using a
scientific approach.
◦ Effectiveness – ability to equip learners with the
skills and knowledge required in the labour market
thus leading to employment readily.
9
•
•
Consensus on study expectations
Methodology
– Industry model
– Scope
– Sample Structure
– Data collection tools
– Data collection
– Data recording and analysis
– Data Quality
10
•
•
•
•
Desk –based review of reports by strategy
development group
Debriefing by Strategy development group
Debriefing/interviews with key stakeholders –
MoES, WB, BTC, GTZ/PIVOT
On-site visits – BTVET institutions – public,
private, rural & urban
– Helped to understand expectations, quality of
BTVET programmes, available data; planning for the
study
11


Industry model:
Input
process
focus on:
◦
◦
◦
◦
output, with
Training competence and infrastructure (input)
training processes (processes)
Training outcome (output/outcome)
Overall satisfaction and image/reputation of BTVET
12
•
•
•
•
Scope:
GRADUATES (i.e. learners who had passed final examination;
learners who had completed the training but not yet received
certificate)
Current EMPLOYERS of graduates.
Contextual elements:
– Established the readiness of BTVET graduates being absorbed
into in the labour market (i.e. formally employed or selfemployed) and/or accessing further education and training
opportunities within the system.
– Identified and examined the factors and circumstances that
enhance and/or inhibit the placement of graduates into
employment and/or further education and training
opportunities.
– Included the evaluation of BTVET delivery processes,
interaction between BTVET institutions and employers and
expectations of graduates and employers, among others.
13

BTVET Graduate Cohort
◦ Multi dimensional & stratified:
(i) By institutions – public + private = 650
◦ arbitrary 10% representative sample – 65;
◦ Public – 33 + private – 32 = 65
(ii) By regional distribution - UGAPRIVI Classification
7 regions (i.e. Central (incl. Kampala), East, North,
South, South West, West Nile and Western
14
Region
%
region
al dstn
No.
Public
Inst.
Sample Sample
private public
inst
inst.
Central 124
24%
31
8
8
East
73
14%
18
4
4
North
73
14%
18
4
4
South
56
11%
15
3
4
South
West
72
14%
18
4
4
West
Nile
77
15%
20
6 9
6 5
West
43
8%
11
3 4
3
Total
518
100%
131
32
33

No.
Private
Inst.
15
•
BTVET Graduate Cohort
– (iii) By year of completion – distribution across different
years of completion (2007-2010)
– (iv) By learning pathways –formal, informal, non-formal
– (v) By exit examinations- UNEB, DIT
– (vi) By different occupational training areas
– (vii) By gender distribution – male- 80%, female – 20%
•
Target sample – telephone contacts for at least
60 graduates in each institution were compiled;
3629 contacts compiled; client’s target was
1500; 1415 graduates were interviewed by phone
16
•
Employers
– UBOS data on sector growth – agriculture, industry,
services (potential for employment)
– UBOS data on distribution of workforce by industry
– Kampala & suburbs were covered – good
concentration of businesses/companies
– Formal & informal
– Size of the enterprises
•
Target sample – contact details for 60
employers; 100 contacts compiled (more than
60 within Kampala); 63 interviewed, 12 not
responded, 25 not contacted
17

Structured questionnaires for graduates and
employers
◦ Inclusive of industry model focal areas
◦ 5- point Likert-type scales
◦ Qualitative responses
18
•
•
•
7 RAs deployed in 7 regions to collect bio data
(especially phone contacts) on all target graduates in
the selected institutions; used a fixed template
3 RAs deployed to collect profile data on employer
organisations (especially phone & e-mail contacts)
7 RAs administered the questionnaires to the
graduates using telephonic interviews (English and in
local languages)
– 7 weeks, 50 interviews per week/ 10 per day
•
2 RAs administered questionnaires to employers
physically, collected completed questionnaires, sent
via email – based on respondents’ preference
– 3 weeks, 20 interviews per week/4 per day
19
•
•
•
•
•
•
M&E web-based survey platform- Metior
Solutions.
3 data recorders trained for ½ day
Each data recorder was given account on the
platform
Entered each questionnaire and uploaded on the
platform
Quantitative and qualitative data was analysed
using platform’s own instant analysis tools
Descriptive statistics on quantitative data and
summary reports on qualitative data provided
instantly
20
•
•
•
•
•
Training of RAs for 3 days including
simulation & feedback
Initial supervised/observed telephonic
interviews – 5 interviews after training
Confirmation of interviews – 5 in every 50
Paper questionnaires thoroughly checked
Instant platform-based feedback on data
entry – incomplete questionnaires,
inconsistencies, errors in entries, etc
21
•
Overall satisfaction and BTVET image
– 59% of graduates were satisfied
– 48% of employers were satisfied
•
Training quality and delivery (efficiency)
– 55% of graduates provided a good rating
– 35% of employers rated private institutions as excellent
– 38% of employers rated public institutions as good
•
Overall BTVET outcome (effectiveness)
– 64% of graduates felt that skills acquired made them
ready for the labour market
– 63% of employers felt that graduates were ‘prepared’ for
applying the skills and knowledge but not up to the
required levels of employment
22





Establish effective BTVET Management
Information System
Upgrade, modernise and sufficiently resource
existing training infrastructure
Intensify and strengthen
cooperation/partnerships between training
institutions and industry
Enhance female participation
Focused BTVET programmes on local skills
demands
23
•
Lack of baseline data on both graduates and
employers
– A base population frame of contacts details had to be
established before sampling
•
EMIS BTVET data on graduates not disaggregated by
training programmes
– Called for prior field visit to public and private institutions
•
Lack of up-to-date contact data on public and private
institutions across 7 regions
– individual research approaches on institutions had to be
conducted
•
Obtaining employer interviews in the formal sector
was cumbersome as it involved formal approval
– Many were left out
24

Some employers required repetitive follow –
ups and/or solicitation for nominal incentives
◦ Discarded from the list after failure through
persuasion

Less eagerness for participation among
graduates in the central region
◦ opted for extra interviews in South and South West
to compensate
25




Tracer study periodically reaffirms quality,
relevance and effectiveness of BTVET sub
sector/related services
Tracer study adequately informs direction for
prioritisation
Telephonic interviews minimise time
consumed in ‘snowballing’
Metior platform facilitates quick delivery of
analysed data particularly in restrictive ‘time
bound’ situations
26