Improving School Governance using an Action Learning Approach 527856-LLP-1-2012-1-PT-COMENIUS-CMP WP7 - Internal Project Evaluation and Quality Assurance Deliverable 7.2 – Report on Internal project evaluation & QA IGUANA is a relatively small project with limited resources available for evaluation. Thus, following the Internal project evaluation and the Q&A handbook (7.1), the approach now proposed focuses on Quality Control. For this purpose we shall use a simplified version of the procedures and tools of the ‘PRINCE2’ approach. This approach encompasses three elements: internal review of project deliverables (peer review); supplementary review of a small number of designated deliverables (external review); quality register (a record of the how the deliverables have been reviewed). A fourth element was added to this tripod: the internal Iguana evaluation template/survey, a tool specifically designed to collect partners’ views, opinions and suggestions on the entire project implementation. This tool was applied at the final stage of the project and was answered by all partners. 1. Review Process Most IGUANA project deliverables assumed the form of written documents and most of the deliverables required only ‘internal review’. 1 As presented in the Internal project evaluation and the Q&A handbook (7.1), the Internal Quality Control was implemented through a ‘peer review’ system that worked as follows: For each Deliverable, the deliverable owner assigned a member of his/her team to run a first check on the deliverable: the ‘Primary Reviewer’. The consortium then assigned another reviewer (the Second Reviewer) to evaluate the deliverable. Though the handbook states that this would be the task of the deliverable owner, because of the need to be more efficient and get the project back on schedule, the partners undertook frequent online discussions and email exchanges; this continuous exchange allowed for more shared and greater efficiency in arriving at decisions on issues such as allocating second reviewers. The Peer Reviewers independently reviewed their assigned Deliverable using the Quality Control Checklist (annex 1). Each deliverable has its own two checklists duly filled and signed by the reviewers. For the majority of written deliverables, the ‘peer review’ method carried out through the internal review process was sufficient for the Quality Control approach followed in IGUANA. In line with the Quality Control Checklist, each written deliverable includes a Deliverable Review History (annex 2) documenting: (i) the deliverable version number, (ii) its authors and reviewers, (iii) the dates of the deliverable authoring and reviewing, and (iv) the changes made. Table 1 lists the deliverables, their internal reviews, and supplementary reviews. Table 1 – Internal and Supplementary Reviews Deliverable WP1: Management D1: Project Management Guidelines D1.2: Progress Report D1.3 Final Report WP2: Comparative Review D2.1 Report on school governance in the EU Internal review Suppleme ntary review 2 WP3: Methodology, Needs analysis and common governance model 3.1 Needs Mapping and action learning set methodology 3.2 Common training model, guidelines and standards 3.3 Training Needs Report WP4: IGUANA Platform and Web 2.0 tools 4.1 IGUANA web platform and tools WP5: Content Development 5.1 Emotional Well-being Self-assessment tool 5.2 Draft Good Governance Training Module 5.3 Collaborative Learning Environment WP6: Delivery and Evaluation 6.1 Evaluation methodology and toolkit 6.2 Report on Evaluation outcomes and impacts WP7: Internal Project Evaluation and quality assurance D7.1 Quality Plan and internal project evaluation Guidelines D7.2: Report WP8: Reporting and Dissemination D8.1: Dissemination plan D8.2: Project pack D8.2: IGUANA Events D13: Dissemination report WP9: Exploitation D9.1: Exploitation plan D9.2: Final Exploitation and Sustainability Plan The supplementary reviews were made conducted by external parties, namely: Deliverable 3.2 Common training model, guidelines and standards - Around 100 school directors ran the Self Assessment Tool (pre-test). These school directors were involved in the Portuguese Innovative Leadership Programme (Ministry of Education and Science of Portugal and Microsoft Network). The idea was to test, refine and stabilise the Self Assessment Tool. Deliverable 4.1 IGUANA web platform and tools; Deliverable 5.1 Emotional Well-being Self-assessment tool; Deliverable 5.2 Draft Good Governance Training Module; and Deliverable 5.3 Collaborative Learning Environment - IGUANA Web Platform; these four Deliverables were tested and used by all the project school pilots involved in the different countries. Other entities, within the education sector i.e. ESHA members, but also coming from outside the educational sector were equally invited to test IGUANA resources, i.e. Portuguese Education Centres – institutions working with young 3 offenders serving an ‘extreme’ custodial measure – to access, and to make use of IGUANA tools, namely the Self-Assessment Tools (SAT) and the Iguana Learning Programme (ILP), Table 2 illustrates the quality procedures, namely in terms of the review process, and includes for each deliverable: The deliverable number and title The ‘assessment criteria’ that are specific to each deliverable and against which the deliverable is assessed The Quality Method used The deliverable producers and reviewers Date of review Result of review Date deliverable was finally approved. Table 2 – Quality Register Quality activity No. Deliver able N° Deliverable name Assessment criteria Quality Method Deliv producer Review er 1 D 1.1 Project Managemen t Guidelines No. of partners rating Guidelines as ‘very user friendly’ Internal Review CEPCEP 2 D 2.1 Report on school governance in the EU Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal Review MENO N Networ k EEIG Delivery date Date Review Result Approval date Véron 30/01/201 26/02/1 ique 3 3 Maes & Joe Cullen Accept ed 26/02/ 13 Valent 30/09/201 05/12/1 ina 3 3 Dagie ne; Rodrig o Queir oz e Mello & Eleni Chelio ti; Josep h Camp Accept ed 06/09/ 13 4 3 D 3.1 Needs mapping and Action Learning Set methodolog y Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal Review EA 4 D 3.2 Common training model, guidelines and standards Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal Review EA 5 D 3.3 Training needs report Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal Review EA 6 D 4.1 IGUANA web platform and tools Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal & Externa l review Arcola 7 D 5.1 Emotional well-being Selfassessment tool Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal & Externa l review Contou r 8 D 5.2 Draft Good Governance Training Module Deliverable meets the 'acceptance Criteria' set out in the Quality Internal & Externa l review Contou r bell Valent ina Dagie ne; Rodrig o Queir oz e Mello Valent ina Dagie ne; Rodrig o Queir oz e Mello Valent ina Dagie ne; Rodrig o Queir oz e Mello Nikos ( Meno n) Portu guese Princi pals (Lisbo ns) & ESHA memb ers (Finla nd) Valent ina Dagie ne & Eleni Chelio ti 31/07/201 05/12/1 3 3 Accept ed 06/12/ 13 30/11/201 29/11/2 3 013 Accept ed 30/11/ 13 30/11/201 05/12/1 3 3 Accept ed 06/12/ 13 1/05/2013 Accept ed 1/05/2 013 1/09/2013 12/12/2 014 Accept ed 17/12/ 2014 Véron 01/09/201 07/01/2 ique 3 015 Maes & Joe Camp bell Accept ed 28/02/ 2015 5 Register 9 D 5.3 Collaborativ e Learning Environment Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal & Externa l review Contou r Joe 01/09/201 07/12/2 Camp 3 014 bell & André Xavier de Carval ho Eleni 1/12/2013 04 & Chelio 07/12/1 ti & 3 Valent ina Dagie ne Accept ed 15/02/ 2015 10 D 6.1 Evaluation methodolog y and toolkit Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal Review Arcola Accept ed 30/11/ 13 11 D 6.2 Report on Evaluation outcomes and impacts 12 D 7.1 Quality Plan and internal project evaluation Guidelines Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal Review UCP Joe 28/02/201 25/02/2 Cullen 3 013 & Malco lm Accept ed 28/02/ 2013 13 D 8.1 Disseminatio n Plan Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal Review ESHA Véron 1/05/2013 15-11ique 2013 & Juliet Accept ed 15/11/ 13 14 D 8.2 Iguana information pack Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal Review ESHA Véron 1/05/2013 30/11/1 ique 3 Maes & Anita Juškev ičienė Accept ed 30/11/ 13 15 D 8.4 Disseminatio n Report Deliverable meets the 'acceptance Criteria' set out in the Quality Register Internal Review ESHA Anita 28/02/201 14/04/2 Juškev 5 015 ičienė & Vasilis Fasoul is Accept ed 16/04/ 2015 16 D 9.1 Exploitation plan Deliverable meets the 'acceptance Criteria' set Internal Review Contou r Joe 1/07/2013 20/12/2 Camp 013 bel & Joe Accept ed 21/12/ 2013 Arcola 6 out in the Quality Register 17 D 9.2 D9.2: Final Exploitation and Sustainabilit y Plan Cullen Internal Review Contou r Fred Verbo on & Camil a Child 2 – Internal Iguana evaluation template/survey This internal Iguana evaluation template/survey (annex 3) encompasses a set of questions that are part of the overall project evaluation, aimed at getting partners views on how the project has been proceeding, what problems they have experienced and what changes or improvements they suggest might be useful to incorporate in the project. The purposes of the questionnaire was to conduct a ‘critical review’ of the project in terms of project management, communications within and outside the project, and progress in terms of objectives, milestones and content. For each question the respondent was invited to think about his experience on how Project Management, Communication System and Scheduled Objectives have been carried out in IGUANA and for each aspect indicate any problems experienced. Furthermore, respondents were asked to state what changes or improvements they would like to see and to express their satisfaction with each particular aspect of the project, by circling the number which applies and using the following scale: -2------------------ -1------------------ 0------------------- +1------------------ +2------------------------very dissatisfied Dissatisfied neither satisfied satisfied very satisfied nor dissatisfied Next, we present the results derived from the internal Iguana evaluation template/survey. This will be made firstly focusing on the quality data – views on 7 project management issues (table 3); views on communication system (table 4) and comparison between scheduled objectives and results achieved (table 5) – presenting the identified problems and changes. Then we shall present the quantitative data considering the level of satisfaction scale (table 6). In the latter case, we have analysed the results for each component and each aspect, presenting in table 6 the calculated median and average values. Table 3 - Views on Project Management Issues: Problems and Suggestions ASPECT (A) PROBLEMS (B) ISSUES , REMEDIES AND CHANGES Workplan matters Support from the project More forums to raise (carrying out the coordinators with questions and concerns and troubleshoot tasks required) concerns were not responsive. in project meetings. (for example our difficulties in engaging the Belgium School and finding an alternative solution). The delay in project beginning. Kick-off meeting in the first Kick-off meeting was carried out month (of the project) only in February Difficulties to conclude the More commitment deliverable review process in partners time by Changes in the time-frame Some delays in some tasks, for the deliverables of namely the pilot process WP3, which were due to some delays in WP2 and the delay in the kick-off Flow between WP2 and WP3. meeting. Any problems were however resolved. Too little emphasis exploitation efforts. Not sufficiently action-oriented on More attitude towards execution of exploitation activities detailed 8 description of the tasks. Should be provided: More detailed explanation of the tasks; Examples; Clear explanation of expected results. Financial matters UCP mostly provided timely notice for reporting tasks. Difficulties to have available the The project coordination correct financial documentation undertook financial from partners coercive measures Governance (decision-making and consultation) A more proactive governance Early risk identification would helped the project in its then addressed with action risk management. plans. Some difficulties to get the same The consortium should level of commitment to the have acted more promptly project by all partners. on actions that were behind schedule. Some partners were more There have been a few committed to the projects than times when we have not received any response to others. our questions. Coordinator should allocate a responsible person for Consultation. quick response. Handling of co- Generally okay if a little sporadic. ordination and other meetings 1. Communication tools 2. Partners meetings Could have done with more regular telephone meetings 1. Skype and Adobe were not particularly suitable for partner online meetings, because of bad internet connection: often it was difficult to hear what was said. 9 2. There have been a few meetings when the agenda was not enough structured. Table 4 - Communication System: Problems and Suggestions ASPECT Technical (e.g.using platform) (A) PROBLEMS matters Some of the elements are not project clear: how to upload content to the Schools Sharing Space; how the Peer Review Programme is accessed. (B) ISSUES, REMEDIES AND CHANGES Instructional videos may help to show how to navigate through the website and learning platform. The wiki space was not used by A final effort was all partners in the same undertaken to guarantee a frequency. maximum project accountability via the Wiki Availability of self-assessment tools and learning materials in Greek (translations have been provided) Finding out about There is little information about Provide direct links to the things happening things happening within the Commissions websites and within the Commission. live feed updates. Commission Very little interactions here. Getting answers As above, I found the from the project co- coordination of this project very ordinator technical and by the book but not particularly enabling. I do not think that was an issue. Questions by Emails. Deliverables production A more proactive style, engaging all partners, creating an innovative culture. There have been a number of requests by project coordination that were not given any timely response. Seemed to be to a good standard Rescheduling the delivery and mostly timely. of WP3 deliverables. 10 Some delays on deliverables production. Some of them are dependent form other, and once started the delay it is difficult to recover. Some problems in the timing of the deliverables of WP3 in relation to WP2 were resolved. OK, however, we have under delivered on exploitation. The platform is good. Communicating with other partners Skype was used for online meetings. There were some minor problems using this environment. Conference calling software may improve and provide more stable platforms. Mostly good though there could More opportunities to have been more learning shared, interact across the especially once piloting started. different work packages. Difficulties to obtaining answers More ownership from some partners. commitment. and Some partners just do not reply Partners (including us) are to mails at all. I really felt I had to really busy therefore we beg for dissemination input. had some delay on writing research paper. Work overload is a key problem Collaborative work for partners under heavy demand. 11 Table 5 - Comparison between scheduled objectives and results achieved ASPECT (A) PROBLEMS (B) ISSUES, REMEDIES AND CHANGES Keeping to overall The late start of the pilot proved An early and appropriate project plan difficult achieving all the rescheduling was done in requirements for data collection. order to accelerate the process and retrieve time. I am not sure that strategies to keep the project on course were A project extension was addressed in due time. Overall requested and granted. the project achieved its research aims. Small delay in the kick-off meeting, although the delay was initially recovered, although more delays in crucial tasks (i.e. pilot process) had contributed to new delays. Small delay meeting in the kick-off OK, except for exploitation. Modifications Speeding up the WP2 work and rescheduling the following WP3 deliverables. Most pilot implementation periods ended-up coinciding with the beginning of school recess and summer holidays, which entailed waiting for the beginning of the following academic year (2014-15) to accomplish all pilot tasks ascribed to each partner. Ownership and commitment from the exploitation partner was further requested In general, there were only slight modifications of the plan. However, better account of the current situation of each partner plan implementation stage could have been 12 undertaken. Project Management (WP1) Difficulties to have available the contributes/documentation for progress and final report in useful time. Clear Agendas for meetings. Comparative Review of school governance structures and approaches (WP2) Partners were too easygoing in providing the data for final reporting. A major effort was deployed at the latter part of the deadline to meet the final reporting requirements. It would have been better if partners were more cooperative in a timely and gradual collation of all information and data that was requested of them in due time. There have been a few meetings when agendas were not clear and it would be useful to receive agendas earlier. This deliverable was late, could An inter work package have been more dialogue workshop between WP2 between this deliverable and the and WP5. content. Delay on the deliverable 2.1 Rescheduling the following WP3 deliverables. Some delay in the deliverable. Once it is decided to exclude or add something from/to the workplan – it Review of Estonian and Latvian needs to be done right literature exclusion. away and the revised version shared with all partners in order to avoid Understanding significance of the further delays and review results and clear ambiguous conclusions of all reviews. interpretations. Clearly indicate main differences between governance structures and what they concludes. 13 Methodology, No significant problems. Needs analysis and common governance model (WP3) Rescheduling the following WP3 deliverables, after the delay in WP2. IGUANA Platform Choice of platform and user Provide instructional and Web 2.0 tools friendliness. Difficulties with the videos to show how to use (WP4) peer learning platform. and navigate the website and portal. The IGUANA Portal has changed at the end of the project and became less friendly and more confusing. To encourage interaction and learning between IGUANA schools we could have offered facilitated webinars. We lacked the resources to ensure an adequate portal evolution as project matured over time. The Platform which hosts the Have these tools made Open Learning Space does not available and stabilized always work properly. earlier on. Material available partners’ languages. in Full portal translations other were provided in Greek and Portuguese languages. However the WP responsible partner did not have the resources to localise the platform in these alternative languages. Content Development (WP5) Accessibility of the content for See previous the users, some language issues comments but also presentation of the website could be improved for exploitation. Delivery and evaluation of learning environment and training programme Difficulties in engaging the schools in particular, Belgium. Burden on the schools having to ask for additional data. point An internal workshop on evaluating the pilots. The evaluation methodology and toolkit were modified after when 14 (WP6) Modifications of evaluation the the Lithuanian partner did toolkit. the evaluation based on the primer handbook. It is very advisable to take into account the current situation of each partner pilot stage. Need of a better planning of pilot implementation in relation to school calendar and holiday seasons. Internal Evaluation Lack of clarity on the processes Internal evaluation should and QA (WP7) for identifying problems. have been a regular agenda item in project meetings. Difficulties to conclude the deliverables review process Dissemination (WP8) Very focussed on ESHA events. Greater focus project level on local dissemination plans. A better planning and advanced notice of papers under preparation would have helped in partners collaboration. Exploitation (WP9) Not linked enough evaluation work. to the Schools could have been considered stakeholders in the exploitation of IGUANA Bad performance during the project. No follow up on the meetings, no training offer. This caused a lot of problems within ESHA: Dissatisfied members and numerous complaints!!! Not clear description of WP9 The following elements should have been provided: More detailed explanation of WP9; Examples provided; Clear explanation of expected results; Better fit between exploitation meetings of wide information and project partners’ availabilities in order to avert the creation of 15 frustrating gaps between demand and supply of key project assistance to interested customers. Table 6 - Level of satisfaction\Components & Aspects Median (for each Average (for each aspect and aspect and between between partners) partners) Views on Project Management Issues Workplan matters (carrying out the tasks required) 0.5 0.6 Financial matters 1 1.0 Governance (decision-making and consultation) 1 1.1 Handling of co-ordination and other meetings 2 1.6 Total average for this component 1.1 Communication system Technical matters (e.g.using project platform) 0.5 0.8 Finding out about things happening within the Commission 0.5 0.5 Getting answers from the project co-ordinator 1 1.1 Deliverables production 1 1.3 Communicating with other partners 1 0.8 Total average for this component 0.9 Comparison between scheduled objectives and results achieved Keeping to overall project plan 0.5 0.6 Project Management (WP1) 1 1.4 Comparative Review of school governance structures and approaches (WP2) 1 0.8 1.5 1.3 1 0.9 Methodology, Needs analysis and common governance model (WP3) IGUANA Platform and Web 2.0 tools (WP4) 16 Content Development (WP5) 1 1.3 Delivery and evaluation of learning environment and training programme (WP6) 1 1.3 Internal Evaluation and QA (WP7) 1 1.1 1.5 1.5 1 0.6 Dissemination (WP8) Exploitation (WP9) Total average for this component Scale: -2--------------------very dissatisfied -1--------------------dissatisfied 1.1 0--------------------neither satisfied nor dissatisfied +1-------------------satisfied +2-------------------very satisfied As can be observed in table 6 the average values round a satisfactory level: 1. In the first component – Views on Project Management Issues – the average value is 1.1 meaning that partners are quite satisfied with this component. Still in this component, the aspect which gathered lower score was “Workplan matters (carrying out the tasks required)” with an average value of 0.6. On the other hand “Handling of co-ordination and other meetings” was the aspect for which partners showed a higher satisfaction, with an average value of 1.6. Regarding the Communication System component the average value was struck at 0.9; under this item the lowest value is in “Finding out about things happening within the Commission aspect” and the highest is on “Deliverables production”. Finally and regarding the last component – “Comparison between scheduled objectives and results achieved” – the mean value was again 1.1 which refers to a satisfactory level. The lowest satisfaction levels are registered in Exploitation and the highest are in Dissemination. 17 Annex 1 – Quality Control Checklist Quality Control Check Y/N Reviewer recommendations/comments Generic Minimum Quality Standards Document Summary provided (with adequate synopsis of contents) IGUANA format standards complied with Language, grammar and spelling acceptable Objectives of Description of Work covered Work deliverable relates to adequately covered Quality of text is acceptable (organisation and structure; diagrams; readability) Comprehensiveness is acceptable (no missing sections; missing references; unexplained arguments) Usability is acceptable (deliverable provides clear information in a form that is useful to the reader) Deliverable specific quality criteria Deliverable meets the 'acceptance Criteria' set out in the Quality Register (see Table 5) For Supplementary Review Deliverables only Deliverable approved by external reviewers Checklist completed by Name: Signature: Date: 18 Annex 2 – Deliverable Review History Version Name Status * Date Summary of changes Annex 3- Internal Partner survey/template –Quality IGUANA Internal evaluation template This set of questions is part of the overall project level evaluation of IGUANA, aimed at getting partners views on how the project has been proceeding, what problems they have experienced and what changes or improvements they suggest might be useful to incorporate in the project. The purposes of the questionnaire are to conduct a ‘critical review’ of the project in terms of project management, communications within and outside the project, and progress in terms of objectives, milestones and content. Everyone involved in the project, including full partners, associate partners, sub-contractors and users group representatives are called to answer the questionnaire. We welcome ‘individual’ as well as ‘organisational’ views. NAME: _______________________________________ ORGANISATION:_______________________________ For each question please think about your experience of how Project Management, Communication System and Scheduled Objectives have been carried out in IGUANA and for each aspect indicated, write down: in Column A, any problems you have experienced; in Column B, what changes or improvements you would like to see; in Column C, your satisfaction with this particular aspect of the project (circle the number which applies, using the following scale): -2------------------- -1------------------- 0-------------------- +1------------------ +2---------------------very dissatisfied dissatisfied neither satisfied satisfied very satisfied 19 nor dissatisfied 1. VIEWS ON PROJECT MANAGEMENT ISSUES: PROBLEMS AND SUGGESTIONS ASPECT (A) PROBLEMS (B) CHANGES (C) SATISFACTION Workplan matters (carrying out the tasks required) Financial matters Governance (decisionmaking and consultation) Handling of co-ordination and other meetings Are there any other points or suggestions on project management you would like to make? 20 2. COMMUNICATION SYSTEM: PROBLEMS AND SUGGESTIONS ASPECT (A) PROBLEMS (B) CHANGES (C) SATISFACTION Technical matters (e.g.using project platform) Finding out about things happening within the Commission Getting answers from the project coordinator Deliverables production Communicating with other partners 21 Are there any other points or suggestions on project management you would like to make? 22 3. COMPARISON BETWEEN SCHEDULED OBJECTIVES AND ACTUAL RESULTS ACHIEVED UP TO NOW ASPECT (A) PROBLEMS (B) CHANGES (C) SATISFACTION Keeping to overall project plan Project Management (WP1) Comparative Review of school governance structures and approaches (WP2) Methodology , Needs analysis and common governance model (WP3) IGUANA Platform and Web 2.0 tools (WP4) Content Development 23 (WP5) Delivery and evaluation of learning environment and training programme (WP6) Internal Evaluation and QA (WP7) Disseminatio n (WP8) Internal Evaluation and QA (WP8) Exploitation (WP9) Are there any other points or suggestions on project management you would like to make? 24