Improving School Governance using an Action Learning Approach 527856-LLP-1-2012-1-PT-COMENIUS-CMP WP7 - Internal Project Evaluation and Quality Assurance Deliverable 7.2 – Report on Internal project evaluation & QA As stated in the Internal project evaluation & QA handbook (7.1), because IGUANA is a relatively small project with limited resources available for evaluation, the approach proposed provides for a simplified version which incorporates some of the procedures and tools that are based on the ‘PRINCE2’ approach and which concentrates on Quality Control. This encompasses three elements: internal Review of project deliverables (Peer Review) supplementary Review of a small number of designated deliverables (External Review) quality Register (a record of the how the deliverables have been reviewed) A fourth element was added to this strategy - the internal Iguana evaluation template/survey- in order to collect partner views, opinions and suggestions about all the project implementation. This tool was applied at the final stage of the project and was answered by all partners. 1 1. Review Process Most IGUANA project deliverables had the form of written documents and most of them deliverables required only ‘internal review’. As presented in the Internal project evaluation & QA handbook (7.1), Internal Quality Control was implemented through a ‘Peer Review’ system that worked as follows: for each Deliverable, the deliverable owner assigned a member of their team to run a first check on the deliverable. This is the ‘Primary Reviewer’. the consortium then assigned another reviewer (the Second Reviewer) to evaluate the deliverable. Though the handbook states that this would be the task of the deliverable owner, because of the need to be more efficient and get the project back on schedule, the partners had frequent online discussions and exchanges of e-mail; this allowed for more shared and efficient decisions on issues such as allocating second reviewers. The Peer Reviewers independently review their assigned Deliverable using the Quality Control Checklist (annex 1). Each deliverable has its own two checklists duly filled and signed by the reviewers. For the majority of written deliverables, the ‘peer review’ method carried out through the internal review process was sufficient for the Quality Control approach followed in IGUANA. Following the Quality Control Checklist, each written deliverable include a Deliverable Review History (annex 2) which documents the deliverable version number; its authors and reviewers; the dates authored and reviewed and the changes made. Table 1 lists the deliverables and their internal reviews, and supplementary reviews. 2 Table 1 – Internal and Supplementary Reviews Deliverable Internal review WP1: Management D1: Project Management Guidelines D1.2: Progress Report D1.3 Final Report WP2: Comparative Review D2.1 Report on school governance in the EU WP3: Methodology, Needs analysis and common governance model 3.1 Needs Mapping and action learning set methodology 3.2 Common training model, guidelines and standards 3.3 Training Needs Report WP4: IGUANA Platform and Web 2.0 tools 4.1 IGUANA web platform and tools WP5: Content Development 5.1 Emotional well-being Self-assessment tool 5.2 Draft Good Governance Training Module 5.3 Collaborative Learning Environment WP6: Delivery and Evaluation 6.1 Evaluation methodology and toolkit 6.2 Report on Evaluation outcomes and impacts WP7: Internal Project Evaluation and quality assurance D7.1 Quality Plan and internal project evaluation Guidelines D7.2: Report WP8: Reporting and Dissemination D8.1: Dissemination plan D8.2: Project pack D8.2: IGUANA Events D13: Dissemination report WP9: Exploitation D9.1: Exploitation plan D9.2: Final Exploitation and Sustainability Plan Suppleme ntary review The supplementary reviews were made by external parts, namely: Deliverable 3.2 Common training model, guidelines and standards - Around 100 school directors ran the Self Assessment Tool (pre-test). These school directors were involved in the Microsoft Innovative Leadership Programme (Ministry of Education and Science of Portugal and Microsoft Network). The idea was to test, refine and stabilize the Self Assessment Tool. 3 Deliverable 4.1 IGUANA web platform and tools; Deliverable 5.1 Emotional well-being Self-assessment tool; Deliverable 5.2 Draft Good Governance Training Module; Deliverable 5.3 Collaborative Learning Environment - IGUANA Web Platform was tested and used by all the school pilots involved in the project in the different countries. Other entities, part of the education sector i.e. ESHA members, but also entities out of the educational sector were also invited to test IGUANA resources i.e. Education Centres – institutions working with young offenders serving an ‘extreme’ custodial measure – to access and make use of IGUANA, concretely the SelfAssessment Tools (SAT) and the Iguana Learning Programme (ILP), The table 2 illustrates the quality procedures, namely in terms of the review process, and includes for each deliverable: The deliverable number and title The ‘assessment criteria’ that are specific to each deliverable and against which the deliverable is assessed (taken from Table 4 above) The Quality Method used The deliverable producers and reviewers Date of review Result of review Date deliverable finally approved Table 2 – Quality Register Quali ty activi ty No. Deliv erabl e N° Deliverable name Assessm-ent criteria Quality Metho d Deliv produc er Reviewer 1 D 1.1 Project Managemen t Guidelines No. of partners rating Guidelines as ‘very user friendly’ Intern al Revie w CEPC EP 2 D 2.1 Report on school Deliverable meets the 'acceptance Intern al Revie MEN ON Netw Delivery date Date Review Result Approve date Véroniqu 30/01/201 26/02/1 e Maes & 3 3 Joe Cullen Accept ed 26/02/ 13 Valentina 30/09/201 05/12/1 Dagiene; 3 3 Rodrigo Accept ed 06/09/ 13 4 governance in the EU Criteria' set out in the Quality Register w ork EEIG 3 D 3.1 Needs mapping and Action Learning Set methodolog y Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w EA 4 D 3.2 Common training model, guidelines and standards Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w EA 5 D 3.3 Training needs report Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w 6 D 4.1 IGUANA web platform and tools Deliverable meets the 'acceptance Criteria' set out in the Quality Register 7 D 5.1 Emotional well-being Selfassessment tool 8 D 5.2 9 D 5.3 Queiroz e Mello & Eleni Chelioti; Joseph Campbell Valentina 31/07/201 05/12/1 Dagiene; 3 3 Rodrigo Queiroz e Mello Accept ed 06/12/ 13 Valentina 30/11/201 29/11/2 Dagiene; 3 013 Rodrigo Queiroz e Mello Accept ed 30/11/ 13 EA Valentina 30/11/201 05/12/1 Dagiene; 3 3 Rodrigo Queiroz e Mello Accept ed 06/12/ 13 Intern al & Exter nal revie w Arcol a Accept ed 1/05/2 013 Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al & Exter nal revie w Conto ur Nikos ( 1/05/2013 Menon) Portugue se Principals (Lisbons) & ESHA members (Finland) Valentina 1/09/2013 12/12/2 Dagiene 014 & Eleni Chelioti Accept ed 17/12/ 2014 Draft Good Governance Training Module Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al & Exter nal revie w Conto ur Véroniqu 01/09/201 07/01/2 e Maes & 3 015 Joe Campbell Accept ed 28/02/ 2015 Collaborativ e Learning Environment Deliverable meets the 'acceptance Criteria' set Intern al & Exter nal Conto ur Joe 01/09/201 07/12/2 Campbell 3 014 & André Xavier de Accept ed 15/02/ 2015 5 out in the Quality Register revie w Carvalho 10 D 6.1 Evaluation methodolog y and toolkit Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w Arcol a Eleni 1/12/2013 04 & Chelioti & 07/12/1 Valentina 3 Dagiene Accept ed 30/11/ 13 11 D 6.2 Report on Evaluation outcomes and impacts Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w Arcol a Juliet Scott & Malcolm Kirsh 31/03/201 26/04/1 5 5 Accept ed 28/04/ 2015 12 D 7.1 Quality Plan and internal project evaluation Guidelines Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w UCP Joe Cullen & Malcolm 28/02/201 25/02/2 3 013 Accept ed 28/02/ 2013 13 D 8.1 Disseminatio n Plan Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w ESHA Véroniqu 1/05/2013 15-11e & Juliet 2013 Accept ed 15/11/ 13 14 D 8.2 Iguana information pack Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w ESHA Véroniqu 1/05/2013 30/11/1 e Maes & 3 Anita Juškevičie nė Accept ed 30/11/ 13 15 D 8.4 Disseminatio n Report Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w ESHA Anita 28/02/201 14/04/2 Juškevičie 5 015 nė & Vasilis Fasoulis Accept ed 16/04/ 2015 16 D 9.1 Exploitation plan Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w Conto ur Joe Campbel & Joe Cullen Accept ed 21/12/ 2013 1/07/2013 20/12/2 013 6 17 D 9.2 D9.2: Final Exploitation and Sustainabilit y Plan Deliverable meets the 'acceptance Criteria' set out in the Quality Register Intern al Revie w Conto ur Fred Verboon & Camila Child 20/04/201 26/04/2 5 015 Accept ed 2 – Internal Iguana evaluation template/survey This internal Iguana evaluation template/survey (annex 3) encompasses a set of questions part of the overall project level evaluation of IGUANA, aimed at getting partners views on how the project has been proceeding, what problems they have experienced and what changes or improvements they suggest might be useful to incorporate in the project. The purposes of the questionnaire was to conduct a ‘critical review’ of the project in terms of project management, communications within and outside the project, and progress in terms of objectives, milestones and content. For each question the respondent was invited to think about his experience of how: Project Management, Communication System and Scheduled Objectives have been carried out in IGUANA and for each aspect indicate any problems experienced; what changes or improvements would like to see and the of satisfaction with each particular aspect of the project by circling the number which applies and using the following scale: -2------------------ -1------------------ 0------------------- +1------------------ +2------------------------very dissatisfied dissatisfied neither satisfied satisfied very satisfied nor dissatisfied Above we present the results derived from the internal Iguana evaluation template/survey. This will be made firstly enlighten the quality data – views on project management issues (table 3); views on communication system (table 4) and 7 26/*0 4/201 5 comparison between scheduled objectives and results achieved (table 5) – presenting the identified problems and changes. Then we present the quantitative data considering the level of satisfaction applied scale (table 6). In this last exercise we have analyzed the results for each component and each aspect, presenting in table 6 the calculated median and average values. Table 3 - Views on Project Management Issues: Problems and Suggestions ASPECT (A) PROBLEMS (B) CHANGES Workplan matters (carrying out the tasks required) Support from the project coordinators with questions and concerns were not responsive. For example our difficulties in engaging the Belgium School and finding an alternative solution. More forums to raise concerns and troubleshoot in project meetings. Kick-off meeting in the first month (of the project) The delay in project beginning. Kickoff meeting was carried out only in February More commitment partners by Difficulties to conclude the deliverable review process in time Some delays in some tasks, namely Changes in the time-frame the pilot process for the deliverables of WP3, which were due to some delays in WP2 and Flow between WP2 and WP3. Any the delay in the kick-off problems were however resolved. meeting. Too little emphasis on exploitation More action oriented efforts. attitude towards execution of exploitation activities Not sufficiently detailed description of the tasks. Should be provided: More detailed explanation The project management was not of the tasks; strong enough , in my view, to guide Examples; 8 and make activities happen in time Clear explanation and with equal commitment of all expected results. partners of Financial matters UCP mostly provided timely notice for reporting tasks. Difficulties to have available the correct financial documentation from partners. The financial adequate Governance (decision-making and consultation) management was A more proactive governance would Early risk identification and helped the project in its risk then addressing with management. action plans. Some difficulties to get the same level of commitment to the project The consortium should by all partners. have acted more promptly on actions that were Some partners were more behind schedule. committed to the projects than others. There have been a few times when we have not received any response to Consultation. our questions. Coordinator should allocate a The governance was suitable as all responsible person for decisions were made together and quick response. partners consulted. Handling of co- Generally okay if a little sporadic. ordination and other meetings Communication tools Partners meetings Coordination could have been stronger and the handling of meetings with a more goal driven focus. In my view, more online meetings would have been an asset. Could have done with more regular telephone meetings Skype and Adobe were not particularly suitable for partner online meetings, because of bad internet connection: often it was difficult to hear what was said. 9 There have been a few meetings when the agenda was not enough structured. Table 4 - Communication System: Problems and Suggestions ASPECT (A) PROBLEMS Technical Some of the elements are not clear: matters how to upload content to the (e.g.using project Schools Sharing Space; how the Peer platform) Review Programme is accessed. (B) CHANGES Instructional videos may help to show how to navigate through the website and learning platform. The wiki space was not used by all partners in the same frequency. Availability of self-assessment tools and learning materials in Greek (translations have been provided) The WIKI for internal communication is suitable. Skype for online meetings on the other hand was never a perfect solution that could not be solved by having a professional tool such as Adobe connect or Webex Finding out about There is little information about Provide direct links to the things happening things happening within the Commissions websites and within the Commission. live feed updates. Commission Very little interactions here. Getting answers As above, I found the coordination from the project of this project very technical and by co-ordinator the book but not particularly enabling. A more proactive style, engaging all partners, creating an innovative culture. There have been a few 10 I do not think that was an issue. Questions by Emails. Deliverables production times when we have not received any response to our questions on time. Seemed to be to a good standard Rescheduling the delivery and mostly timely. of WP3 deliverables. Some delays on deliverables production. Some of them are dependent form other, and once started the delay it is difficult to recover. Some problems in the timing of the deliverables of WP3 in relation to WP2 were resolved. OK, however, we have under delivered on exploitation. The platform is good. It was always difficult for partners to stick to strict deadlines and as mentioned before the management team was sometimes absent. The big delay in organising the Kick-Off meeting did not help Communicating Skype was used for online meetings. Conference calling with other There were some minor problems software may improve and partners using this environment. provide more stable platforms. Mostly good though there could have been more learning shared, More opportunities to especially once piloting started. interact across the different work packages. Difficulties to obtaining answers from some partners. More ownership commitment. and Some partners just do not reply to mails at all. I really felt I had to beg Partners (including us) are really busy therefore we 11 for dissemination input. had some delay on writing research paper. Collaborative work No problems to communicate with the other partners. Some were less active and feed-back was sometimes very succinct. Table 5 - Comparison between scheduled objectives and results achieved ASPECT Keeping overall plan (A) PROBLEMS (B) CHANGES to The late start of the pilot proved Was made a rescheduling project difficult achieving all the in order to accelerate the requirements for data collection. process and recover time. I am not sure that strategies to keep A project extension was the project on course were made. addressed in due time. Overall the project achieved its research aims. Speeding up the WP2 work and rescheduling the Small delay in the kick-off meeting, following WP3 although the delay was initially deliverables. recovered, although more delays in crucial tasks (i.e. pilot process) had contributed to new delays. Ownership and commitment from the Small delay in the kick-off meeting. exploitation partner OK, except for exploitation. In general, there were only slight modifications of the plan, however it is very Modifications. advisable to take into account the current The overall plan was kept with some situation of each partner implementation understandable delays no harming plan stage. the quality of the deliverables. Project Difficulties to have available the There have been a few 12 Management (WP1) contributes/documentation for meetings when agendas progress and final report in useful were not clear and it time. would be useful to receive agendas earlier. Clear Agendas for meetings. Could have be stronger Comparative Review of school governance structures and approaches (WP2) This deliverable was late, could have An inter work package been more dialogue between this workshop between WP2 deliverable and the content. and WP5. Delay on the deliverable 2.1 Rescheduling the following WP3 deliverables. Some delay in the deliverable. Once it is decided to exclude or add something Review of Estonian and Latvian from/to the workplan – it literature exclusion. needs to be done right away and shared the revised version with all Understanding significance of the partners. review results and clear conclusions of all reviews. Clearly indicate main differences between After some hesitations on the governance structures and structure and harmonisation of the what they concludes. data, the partnership came up with solid Review. Methodology, Rescheduling the following Needs analysis A big effort was put into the exercise WP3 deliverables, after and common and a robust methodology was the delay in WP2. governance developed. model (WP3) IGUANA Platform Choice of platform and user Provide instructional and Web 2.0 friendliness. Difficulties with the videos to show how to use tools (WP4) peer learning platform. and navigate the website and portal. The IGUANA Portal has changed at the end of the project and became To encourage interaction less friendly and more confusing. and learning between IGUANA schools we could 13 The Platform which hosts the Open have offered Learning Space does not always webinars. work properly. facilitated Have available and Material available in other partners’ stabilized more early this languages. tools. The platform is containing great tools such as the quizzes on EI and OI and offers an comprehensive toolkit to transfer the methodology to any European school interested. Content Development (WP5) Accessibility of the content for the users, some language issues but also presentation of the website could be improved for exploitation. The content is relevant but the translation as so contextual is a harder side to organise. Delivery and Difficulties in engaging the schools in evaluation of particular, Belgium. Burden on the learning schools having to ask for additional environment and data. training programme Modifications of evaluation the (WP6) toolkit. An internal workshop on evaluating the pilots. The evaluation methodology and toolkit were modified after when the Lithuanian partner did the evaluation based on the primer handbook. It is very advisable to take The pilots were really the testimony into account the current that the partnership works well and situation of each partner the schools are demanding such pilot stage. training and workshops. Internal Evaluation QA (WP7) Lack of clarity on the processes for Internal evaluation should and identifying problems. have been a regular agenda item in project meetings. Difficulties to conclude the deliverables review process Dissemination (WP8) Very focussed on ESHA events. More focus at the project level on local dissemination plans. 14 Exploitation (WP9) Not linked enough to the evaluation Schools could have been work. considered stakeholders in the exploitation of IGUANA Bad performance during the project. No follow up on the meetings, no training offer. This caused a lot of Should be provided: problems within ESHA: Dissatisfied More detailed explanation members and numerous of WP9; complaints!!! Examples provided; Not clear description of WP9. Clear explanation expected results. of A lot of effort has been put into coming up with a future service delivery and a lot of opportunities are open to Iguana for the near future. A service delivery model has been discussed. Table 6 - Level of satisfaction Components & Aspects Median (for each aspect and between partners) Average (for each aspect and between partners) Views on Project Management Issues Workplan matters (carrying out the tasks required) 0 0,4 Financial matters 1 1,0 1 1,1 2 1,6 Governance (decision-making and consultation) Handling of co-ordination and other meetings Total average for this component Communication system Technical matters (e.g.using project platform) Finding out about things happening within the Commission Getting answers from the project co-ordinator Deliverables production Communicating with other partners 1,0 1 0,8 0,5 0,5 1 1,1 1 1,1 1 0,5 Total average for this component 0,8 Comparison between scheduled objectives and results achieved Keeping to overall project plan 0 0,6 15 Project Management (WP1) Comparative Review of school governance structures and approaches (WP2) Methodology, Needs analysis and common governance model (WP3) 1 1,2 1 0,8 1 1,2 IGUANA Platform and Web 2.0 tools (WP4) 1 0,9 Content Development (WP5) 1 1,3 1 1,3 1 1 1,0 1,4 1 0,8 Delivery and evaluation of learning environment and training programme (WP6) Internal Evaluation and QA (WP7) Dissemination (WP8) Exploitation (WP9) Total average for this component Scale: -2--------------------very dissatisfied -1--------------------dissatisfied 1,1 0--------------------neither satisfied nor dissatisfied +1-------------------satisfied +2-------------------very satisfied As can be observed in table 6 the average values round a satisfactory level – 1 – considering the scale. In the first component - Views on Project Management Issues – the average value is 1,0 what does it mean that partners are satisfied with this component. Still in this component, the aspect that which gathered lower score was “Workplan matters (carrying out the tasks required)” with an average value of 0.4. On the other hand “Handling of co-ordination and other meetings” was the aspect the aspect which the partners were more satisfied – with an average value of 1.6. Regarding the Communication System component the average value was 0,8 being the lowest value in “Finding out about things happening within the Commission aspect” and the higher one in the aspect “Deliverables production” and “Getting answers from the project co-ordinator”. Finally and regarding the last component - Comparison between scheduled objectives and results achieved – the mean value was again 1,1 which refers to a satisfactory level. The lowest satisfaction levels are in WP9 – Exploitation and the highest ones are in Dissemination. Regarding the aspect “Keeping to overall project plan” the mean value was 0,6. 16 Annex 1 – Quality Control Checklist Quality Control Check Y/N Reviewer recommendations/comments Generic Minimum Quality Standards Document Summary provided (with adequate synopsis of contents) IGUANA format standards complied with Language, grammar and spelling acceptable Objectives of Description of Work covered Work deliverable relates to adequately covered Quality of text is acceptable (organisation and structure; diagrams; readability) Comprehensiveness is acceptable (no missing sections; missing references; unexplained arguments) Usability is acceptable (deliverable provides clear information in a form that is useful to the reader) Deliverable specific quality criteria Deliverable meets the 'acceptance Criteria' set out in the Quality Register (see Table 5) For Supplementary Review Deliverables only Deliverable approved by external reviewers Checklist completed by Name: Signature: Date: 17 Annex 2 – Deliverable Review History Version Name Status * Date Summary of changes Annex 3- Internal Partner survey/template –Quality IGUANA Internal evaluation template This set of questions is part of the overall project level evaluation of IGUANA, aimed at getting partners views on how the project has been proceeding, what problems they have experienced and what changes or improvements they suggest might be useful to incorporate in the project. The purposes of the questionnaire are to conduct a ‘critical review’ of the project in terms of project management, communications within and outside the project, and progress in terms of objectives, milestones and content. Everyone involved in the project, including full partners, associate partners, sub-contractors and users group representatives are called to answer the questionnaire. We welcome ‘individual’ as well as ‘organisational’ views. NAME: _______________________________________ ORGANISATION:_______________________________ For each question please think about your experience of how Project Management, Communication System and Scheduled Objectives have been carried out in IGUANA and for each aspect indicated, write down: in Column A, any problems you have experienced; in Column B, what changes or improvements you would like to see; in Column C, your satisfaction with this particular aspect of the project (circle the number which applies, using the following scale): -2------------------- -1------------------- 0-------------------- +1------------------ +2------------------ 18 -- -- - -- -- very dissatisfied dissatisfied neither satisfied satisfied very satisfied nor dissatisfied 1. VIEWS ON PROJECT MANAGEMENT ISSUES: PROBLEMS AND SUGGESTIONS ASPECT (A) PROBLEMS (B) CHANGES (C) SATISFACTION Workplan matters (carrying out the tasks required) Financial matters Governance (decisionmaking and consultation) Handling of co-ordination and other meetings Are there any other points or suggestions on project management you would like to make? 19 2. COMMUNICATION SYSTEM: PROBLEMS AND SUGGESTIONS ASPECT (A) PROBLEMS (B) CHANGES (C) SATISFACTION Technical matters (e.g.using project platform) Finding out about things happening within the Commission Getting answers from the project coordinator Deliverables production Communicating with other partners 20 Are there any other points or suggestions on project management you would like to make? 21 3. COMPARISON BETWEEN SCHEDULED OBJECTIVES AND ACTUAL RESULTS ACHIEVED UP TO NOW ASPECT (A) PROBLEMS (B) CHANGES (C) SATISFACTION Keeping to overall project plan Project Management (WP1) Comparative Review of school governance structures and approaches (WP2) Methodology , Needs analysis and common governance model (WP3) IGUANA Platform and Web 2.0 tools (WP4) Content Development 22 (WP5) Delivery and evaluation of learning environment and training programme (WP6) Internal Evaluation and QA (WP7) Disseminatio n (WP8) Internal Evaluation and QA (WP8) Exploitation (WP9) Are there any other points or suggestions on project management you would like to make? 23