Annual Review - Summary Sheet This Summary Sheet captures the headlines on programme performance, agreed actions and learning over the course of the review period. It should be attached to all subsequent reviews to build a complete picture of actions and learning throughout the life of the programme. Title: Caribbean Inclusive Growth Programme Programme Value: £1.5 m Programme Code: 202435 Review Date: Jan 5-7, 2015 Start Date: July 2012 (ARIES approval), Sept 2013 (contract) End Date: March 2017 Summary of Programme Performance Jan Jan Year 2014 2015 Programme Score A C Medium High Risk Rating Summary of progress and lessons learnt since last review The main progress since the last Annual Review has been: Confirmed country coverage/footprint with an expanded list of directly participating countries, namely: 3 partnerships covering 6 countries – Jamaica, OECS partnership 1 (St. Lucia – lead, Antigua & Barbuda and Grenada) and OECS partnership 2 (St. Vincent & the Grenadines (lead) and Dominica) Confirmation and expansion of the sectors of focus for the training delivery Revised implementation plan, reflecting, inter alia, new country footprint, partners and timelines. Completion of procurement and selection process for the Canadian Colleges that will partner with the Caribbean counterparts to design and deliver training in each of the 3 partnerships. Planning for further on the ground missions by selected Canadian partners to inform substantive training design and delivery. However, delays and challenges in the confirmation of the country footprint for the intervention delayed the launch and implementation of the training partnerships, and the overall project schedule, affecting performance. Key lessons learnt are: The need to monitor contractual issues very closely, and engage with Procurement as early as possible to speed up the resolution process. Summary of recommendations for the next year In line with DFID’s mandatory SMART Rule for projects scoring C in an Annual Review: The DFID Caribbean (DFIDC) project team will identify of improvement measures for the programme, and reflect these in a delivery plan (by end February) The project team will make a submission to DFIDC management on the future of the project Other recommendations include: Speedy resolution of the contractual and funding issues that have arisen within the project affecting DFID’s funding share. Update of the programme logframe to reflect any programme changes and improve indicators as needed (once contractual issues are resolved). 1 A. Introduction and Context (1 page) DevTracker Link to Business Case: DevTracker Link to Log frame: 3565788 3755669; 4782974 Outline of the programme Under the Caribbean Inclusive Growth Programme, DFID is funding a Youth Skills Development Programme for disadvantaged youth. The programme provides demand-led training based on local labour market needs for youth, with vocational certification for each successful graduate. It also builds the capacity of national training agencies and training providers to reach disadvantaged groups and to design and deliver demand-led, competency-based training. At least 1,000 young people will benefit directly from this programme, though there will also be indirect beneficiaries. These young people will develop basic technical and critical life skills, which will enable them to seek jobs, become self-employed or access further training. The project is delivered through Colleges and Institutes Canada (CICan) who utilise selected Canadian colleges in their network to partner with the national training agencies (NTAs) and local training providers to design and deliver the training, and build capacity. DFID’s support is linked to a larger Caribbean regional skills development programme (Can $20 m) funded by Canada through DFATD, the CARICOM Education for Employment (C-EFE) programme. CEFE is working to improve the overall technical and vocational education and training (TVET) system in the Caribbean Community (CARICOM), working with regional and national agencies and training providers. C-EFE’s three project components focus on: strengthening regional capacity to coordinate, promote and conduct quality assurance for demanddriven TVET programming and certification; developing and managing appropriate systems to inform TVET planning, including labour market information systems and strategies for greater industry collaboration; and strengthening the capacity of TVET institutions to attract, train, and graduate citizens with relevant TVET skills, essentially focussing on the development and delivery of 16 TVET programmes following the demand-driven model, related to identified priority sectors, as well as 2 “pretechnology” programmes specifically targeted at disadvantaged groups. (All done through twinning with experienced, selected relevant Canadian colleges). DFID’s intervention builds on this programme, utilising the regional and national institutional relationships, strategic and capacity-building investments and the same executing agency (CICan). DFID’s contribution allows for greater reach to disadvantaged youth in this context. The programme responds to the major challenges in the region of high youth unemployment, shortages of key skills and the mismatch between the outputs of the education sector and the requirements of the labour market. It provides an avenue for a marginalised portion of the population to participate in and benefit from, economic activity. 2 B: PERFORMANCE AND CONCLUSIONS (1-2 pages) Annual outcome assessment Despite the delays on this project moving to full implementation (as explained later), there is still sufficient time for the project to deliver the outputs within the existing project timeframe. It is expected that the outcome of enhanced economic opportunities for disadvantaged groups can be achieved. Several key elements contribute to this assessment: the overall implementing partner has a strong track record in Education for Employment initiatives; there appears to be strong buy-in and commitment from the regional partners, there is good demand within the target groups in the six countries, the identified partner Canadian colleges are strong, there is already private sector engagement, and the sectors identified for work have reasonable prospects. Overall output score and description C – Outputs substantially did not meet expectation There has been progress on the project, but the delays relating to the confirmation and re-alignment of the country footprint for the project in turn delayed the launch and implementation of the training partnerships, and the overall project schedule. All the outputs are linked to training taking place, so the fact that training had not commenced in the review period meant that most of the milestones on the three project outputs could not be met or were only partially met. Specifically, the milestones on output 1 (re. youth trained) were not met, as the training did not commence in the review period. All the milestones on the capacity-building output (output 2) were also affected, as the partner local institutions could not actively participate in, and learn from the training design and delivery process without the commencement of the training. Similarly, on output 3 (private sector participation in training), work placements linked to the training could not be set up, meaning that one of the three milestones on this output could not be met. Key lessons The value of involving Procurement group early in the project and contract management process. (While intense efforts were made by the project team to clarify the proposed funding model and cost changes and to set out DFID’s position, earlier direct involvement by Procurement Group may have been helpful in speeding up a possible resolution). The value of deciding some key programme features up-front. (While flexibility and openness in deciding the country footprint for the project during inception had some merit, it would have been easier for project implementation if DFID had made a strategic decision up-front/ earlier on the preferred country footprint for the programme) Key actions In line with DFID’s mandatory SMART Rule for projects scoring C in an Annual Review: The DFIDC project team will identify improvement measures for the programme, and reflect these in a delivery plan (Timing: by end February) The project team will make a submission to DFIDC management on the future of the project Additional actions will also be taken forward, as outlined in the following table. Action Resolution of contractual and funding model issues Update of the programme logframe to reflect any programme changes and improve indicators as needed (once contractual issues are resolved). Ensure that capacity-building is set as a priority by Owner/Responsibilty Procurement (icw DFIDC) SDA and SPO icw with Results Adviser and CICan DFIDC (SDA and SPO) Timing Q1 2015 Within 1 month of resolution of contractual issues Ongoing 3 Action CICan once training design and delivery starts in earnest. Ensure that industry involvement remains a focus and is duly reflected in any performance improvement measures. Owner/Responsibilty & CICan DFIDC (SDA and SPO) & CICan Timing Ongoing Has the logframe been updated since the last review? No, not formally. Given the timing of the country footprint decision (May 2014), a logframe update was not done as planned in the first quarter of 2014. While a revised draft logframe was done in October to reflect the updated timelines (in line with a subsequent decision), it was decided to defer a formal logframe update until after the Jan 2015 review, so as to preserve the integrity of the Jan 2015 review process and to take on board any arising issues coming out of actual start-up on the ground. 4 C: DETAILED OUTPUT SCORING (1 page per output) Output Title Disadvantaged groups (youth) effectively trained in labour market relevant areas. Output number per LF 1 Output Score C Risk: High Impact weighting (%): 50 Risk revised since last AR? Y Impact weighting % revised N since last AR? Indicator(s) Milestones 1.1 No. of participants from 100 disadvantaged groups accessing training through the programme (cumulative, disaggregated by sex) 1.2 Proportion of trainees reporting 50% satisfaction with the training. (disaggregated by sex). Progress 0 – Milestone not met. The training has not yet commenced. N/A – Milestone not met. The training has not yet commenced. Key Points Since the last review, DFID Caribbean underwent some strategic changes, which affected the planned country footprint for the project. A set of countries was initially confirmed by DFID in March 2014. However, further changes became necessary because of a further corporate decision, and DFID finally approved a revised country footprint at the end of May 2014 (Jamaica, OECS partnership 1 – St. Lucia (lead), Antigua & Barbuda and Grenada); and OECS partnership 2 – St. Vincent & the Grenadines (lead) and Dominica). The issues with the country footprint stalled further progress on implementation, and as a result, the substantive training design by the Canadian partner colleges and launch of the training did not occur in the review period as originally envisaged in the contract. CICan had to do additional work for the inception work within the target countries, which included two new countries (St. Vincent & the Grenadines and Dominica). They selected and reconfirmed sectors of focus, engaged with and assessed proposed delivery agents and worked with the national training agencies across the six target countries. A revised implementation plan was developed and was submitted to DFID at the end of August 2014. The focus in the second half of the year was procurement of the partner Canadian colleges who would work with the regional partners to design and deliver programmes. This competitive tendering and evaluation process was completed in October 2014, and activities thereafter focussed on the practical arrangements for these partners to start their work in leading the design and delivery of the training. The start-up missions by the selected Canadian partner colleges were postponed from December due to logistical and timing challenges, and have been further delayed due to ongoing contractual issues that have arisen on the programme. Once these issues can be resolved, the first round of training will proceed. Summary of responses to issues raised in previous annual reviews (where relevant) The formal logframe review and update was deferred until after the Jan 2015 review in light of the delays in the country footprint decision. (See section B above for further detail) Recommendations Work with the CICan to resolve the contractual issues quickly and get the training plans on stream. Revise the logframe as needed to reflect the revised timing and country approach (once contract issues are resolved). 5 C: DETAILED OUTPUT SCORING (1 page per output) Output Title Enhanced capacity in selected countries to reach, identify, target and deliver skills training programmes for disadvantaged groups (youth and poor women). Output number per LF 2 Output Score C Risk: High Impact weighting (%): 20 Risk revised since last AR? Y Impact weighting % revised N since last AR? Indicator(s) Milestones 2.1 Number of national training agency 3 (NTA) approved training delivery agencies authorized to deliver programming for disadvantaged youth. 2.2 Proportion of participating programme training staff reporting enhanced understanding of how to address gender issues. Programme training staff in target countries directly engaged with training design and delivery. 2.3 Proportion of participating programme Programme training staff training staff reporting enhanced in target countries understanding of how to design and deliver directly engaged with programming for disadvantaged groups. training design and delivery. Progress Milestone not met However, at least 13 local delivery agents have been identified for the 3 country partnerships. Milestone not met However, programme training staff have been actively engaged with CICan in the inception process. Milestone not met However, programme training staff have been actively engaged with CICan in the inception process. Key Points Given delays in the start of the training, the opportunities to fully build capacity during the review period were constrained. As a result, planned capacity building in addressing gender issues and designing and delivering skills training programming for disadvantaged groups was limited to the exposure gained through the inception and the Canadian partner selection processes. The full hands-on element of the capacity-building for the local training providers will not take place until the substantive training design and delivery begins on the ground. Similarly, full authorisation and accreditation for disadvantaged youth programming will need to follow the actual commencement of the training. Representatives of selected national training agencies (NTAs) have also been fully engaged in the CEFE PSC, which includes a gender component, and outptuts from the Gender and TVET Technical Working Group, so the participating NTAs are also building their own awareness and understanding of, and sensitivity to, gender issues. Summary of responses to issues raised in previous annual reviews (where relevant) The formal logframe review and update was deferred until after the Jan 2015 review in light of the delays in the country footprint decision. (See section B above for further detail) Recommendations Ensure that capacity-building is set as a priority by CICan once training design and delivery starts in earnest. (DFID & CICan) 6 C: DETAILED OUTPUT SCORING (1 page per output) Output Title Private sector participates in design and delivery of skills training programme. Output number per LF 3 Output Score B Risk: High Impact weighting (%): 30 Risk revised since last AR? N Impact weighting % revised N since last AR? Indicator(s) 3.1 Extent to which private sector organisations are represented on overall Project Steering Committee for skills training programme. 3.2 No. of employers participating in field placements and internships in youth at risk programme. 3.3 No. of training delivery agents with established private sector partnerships providing for hosting trainees in practicum and hiring graduates on completion. Milestones 75% private sector attendance at PSC meetings 20 1 Progress 100% - Milestone exceeded. There has been full private sector/ industry attendance the PSCs held in 2014, with all designated representatives attending both PSC meetings. N/A* - (Milestone not met) *The training and related field placements and internships have not yet started. 2* - Milestone met. *There are at least 2 proposed training delivery agents with relationships with the private sector that would facilitate hosting of trainees as possible hiring. (These relationships are relevant to the project but are not necessarily fully/directy attributable). Key Points CICan has a strong track record on industry engagement in its Education for Employment work. There is standing representation from the Caribbean Association of Industry and Commerce and the Caribbean Tourism Organisation on the CARICOM Education for Employment Programme (C-EFE) Project Steering Committee, which also covers the DFID Youth Skills Development work. In this way, there is private sector input in the overall strategic direction and oversight of the programme. At the country level, efforts continue to be made to engage and involve the private sector. Private sector partners have been involved in the inception activities, and the programme has been interacting with key private sector bodies such as Chambers of Commerce, Employers’ Confederations and tourism associations. Good intial progress has been made in relation to identifying possible partners for the hands-on components of the training, particularly in the marine sector. For example, in St. Lucia there are linkages with Jus’ Sail and the marina to host trainees. The importance of strong industry engagement has been emphasised with the implementing partner CICan, and remains a programme priority. The progress made to date with private sector partners has largely met expectation, with the exception of the employers participating in field placements, which could not be realistically assessed since the training has not yet started. Summary of responses to issues raised in previous annual reviews (where relevant) The formal logframe review and update was deferred until after the Jan 2015 review in light of the delays in the country footprint decision. (See section B above for further detail) Recommendations Revise the logframe for this output as needed to reflect the revised timing and country approach and to clarify and improve the indicators (once contract issues are resolved). Ensure that industry involvement remains a focus and is duly reflected in any performance improvement measures. 7 D: VALUE FOR MONEY & FINANCIAL PERFORMANCE (1 page) Key cost drivers and performance While costs have been incurred by CICan for this review period, there has been no submission of invoices representing actual expenditure, despite repeated requests and queries. Invoices are currenty expected in January 2015. The key cost drivers identified in the business case remain (ie fees for curriculum development, training of instructors and participants, orientation of stakeholders, management); with the addition of travelrelated project expenses. However, the latest proposed estimates prepared by CICan represent some increases to specific line items (travel being the most significant), even though these are not currently proposed for payment by DFID. This along with other proposed funding model changes have been discussed extensively with CICan, and will be taken forward by Procurement within the contract management process. VfM performance compared to the original VfM proposition in the business case The overall value for money of the intervention was assessed in the business case with a simple costbenefit analysis, taking into account the cost of the maximum DFID investment (£1.5 mn), compared with the monetised benefits of the expected earnings of successful participants in the training who gained employment after training. At the last review, it was noted the original positive cost-benefit analysis from the project remained substantially valid, as the timing of the estimated flows with the implementation delays and that the DFID investment had decreased by 17% (due to the negotiated contract sum). The main changes over the current review period are the delays in implementation and the proposed new funding model for the project, which suggests higher overall project costs, and a related smaller DFID share of the costs and benefits. Even considering these developments, cost-benefit calculations give a positive NPV (ranging from £.16 m - £3.84 m) and an IRR above the 10% discount rate. We cannot assess effectiveness at this stage, since this relates to the trainees being able to get jobs or continue training, and the training has not yet commenced. Assessment of whether the programme continues to represent value for money Despite the delays, the programme can still deliver its outputs and outcome, and still continues to represent value for money. Based on our calculations, there is still a significant positive NPV, and the IRR is generally well above the discount rate, even when the proposed new funding model figures are taken into account. The perceived increase in overall project costs and its effect on the value of DFID’s investment is still of concern to DFID, so we will work through this issue in the contract management process, to ensure the final outcome gives good value for money. Quality of financial management Since the submission of the Inception Report in October 2013 CICan has submitted a revised Implementation plan and a Semi Annual Report which both record activities completed and provide an updated spending profile. The submission of these reports was consistent with the terms of the contract. However, as noted earlier, there have been no invoices submitted as yet for this financial year, despite several requests and queries from DFID. As these invoices reflect the actual fees and expenses incurred, we are not able to fully account for their performance against the forecast spending profile. Date of last narrative financial report Date of last audited annual statement 30 November 2014 N/A 8 E: RISK (½ page) Overall risk rating: High Overview of programme risk The risk rating from the last review has been raised. The project delays and the contractual challenges have raised the implementation risks around the project. These have superseded previous concerns including the varied capacity levels within the National Training Agencies and potential partner delivery agents; political risk; the potential difficulties in reaching the target beneficiaries and possible overshadowing of DFID programme by C-EFE. Mitigation around the implementation risks is to be put in place - taking forward a contract management process to address the funding model and delivery issues. Other programme delivery risks remain at medium with the political risk arising from the need to select a few countries from within the CARICOM group being mitigated through discussions in the PSC and DFID’s own mission visits to ODA-eligible OECS states. A Risk Register has been developed, shared and agreed between CICan and DFID, and contains risks identified in the business case and the Due Diligence Assessment. The DFID and CICan Sr. Programme officers have agreed to review it semi-annually and discuss any risk changes as they occur. The heightened implementation risks will be reflected in the Risk Register with the appropriate mitigation once the contract management process around this issue is fully in place. Outstanding actions from risk assessment There are no major outstanding issues from the risk assessment - key Due Diligence risks were all mitigated directly through conversations with CICan; and where necessary expert advice was sought from DFID HQ. The initial Due Diligence Assessment did not identify any critical or high weaknesses and mitigation/follow-up action has been identified for the medium and low priority areas for attention. F: COMMERCIAL CONSIDERATIONS (½ page) Delivery against planned timeframe There have been significant delays in the schedule for this project. As reported in the last Annual Review, after the initial procurement hurdles, a new implementation schedule was approved in the programme contract. However, following inception, challenges in DFID’s confirmation of the country footprint delayed the transition to implementation. A revised schedule was presented, and will need to be amended in light of some contractual issues that have arisen. The overall schedule for completion of all 3 cohorts of training is still largely on track, but completion of the first cohort of training is now delayed by about 6 months. These issues will be addressed in the contract management process and PIP. st Completion of 1 round of training Completion of 2nd round of training Completion of 3rd round of training Contract dates Apr 2015 Dec 2015 Jul 2016 Estimated new delivery dates* Oct 2015 Feb 2016 Sept 2016 * These dates reflect the dates from the new Implementation Plan, plus a 2-month lag for resolving the contractual issues. We judge that the delays have increased the costs of the intervention. While there has been no request to date to increase DFID’s contribution to the project, the additional costs seem to be absorbed by CICan and increased contributions from other sources. Performance of partnership (s) The main partnerships to date has been the partnership between CICan and the DFID. The working relationship has been generally constructive, in spite of challenges experienced . There have been some issues in interpretation and utilisation of the contract. There is scope to manage the contractual issues better and to increase DFID responsiveness; and CICan clarity. The other major partnerships are between CICan and the regional institutions. Project reports and feedback from DFID’s missions and from PSC meetings indicate that this is an effective and valued partnership. It is too early to comment on the partnership between the selected Canadian colleges and their local counterparts. Asset monitoring and control There are currently no programme assets from DFID funding. 9 G: CONDITIONALITY (½ page) Update on partnership principles (if relevant) The partnership principles are not directly relevant to this project, as it is implemented by a contracted agency from outside the Caribbean, CICan. There are no major issues within the target countries in the programme that would warrant concerns in implementation. The government-related National Training Agencies are involved in the project, and in this context, principles of country ownership and involvement are embedded in the governance and management arrangeements for the project, such as adequate country representation on the Project Steering Committee H: MONITORING & EVALUATION (M&E) (½ page) Evidence and evaluation We are not aware of any changes to the fundamental evidence base of the project. Since the DFID project is linked to the larger C-EFE programme, the M&E procedures for C-EFE are being extended to incorporate M&E of the youth skills development work. There is a consultancy on data-gathering tools and guidelines which is being expanded to include measurement of the youth skills project outputs and outcome. Basic baseline surveys have already been developed for the NTAs. The NTAs are to conducte tracer survery on participants to measure the programme impact on future training and employment. After the first round of training, each programme will be assessed, drawing on inputs from participants, trainers, employers and the NTAs. A forum will be convened to facilitate sharing, and necessary improvements will be made in the subsequent training rounds. Monitoring progress throughout the review period During the review period, DFID Caribbean: met with CICan at least quarterly (on 25 March, 28 May, 20 June, 20 August, 5 September, 9 October and 6 November 2014), in addition to several email exchanges; visited all 5 Eastern Caribbean states targeted by the intervention (St. Lucia, Grenada and St. Vincent & the Grenadines in June 2014, and Antigua & Barbuda and Dominica in October 2014); participated in the selection ceremony for the partner Canadian institutions on 24 October 2014; and participated in the November 2014 C-EFE Project Steering Committee Meeting. Feedback was received from representatives of the NTAs and some of the proposed local training delivery agents mainly through DFIDC’s scoping missions and through participation in partner selection process and C-EFE PSC meeting. The feedback indicated that the contributions to date were valued, and that they were pleased with the quality of personnel on the project, and saw expected good valueadded from the Canadian experts’ work in areas such as industry engagement, career counselling and addressing literacy and numeracy gaps. The feedback from a wider set of stakeholders in country through the scoping missions also emphasised the need for intervention in the area of skills and to address the needs of youth. The Annual Review was conducted as a desk exercise, drawing on information from project reports, project meetings with CICan, the November C-EFE PSC meeting, and the DFID Caribbean scoping missions.