Date: April 9th, 2015
To : Alicia Mignone, CERT Chair
CC : - Ingrid Barnsley
- Carrie Pottinger
- Paolo Frankl
- REWP Cabinet
From : Martin Schoepe, REWP Chair
Reference: Request for Extension review process – REWP’s feedback
Dear Alicia,
Last December you shared your request for our WP to review the proposed draft revisions to the
RfE requirements during the WP plenary on 25-26th March 2015. As support for the discussions
were included a draft note from the Secretariat explaining the reflection process to date, together
with the draft proposed RfE revision. Deadline for the WP feedback was fixed for April 1st, and
later extended for REWP to April 8th 2015.
This note is the response to your request.
In addition, you will find two propositions to improve this process while respecting the original
calendar for mandatory new RfE questionnaire & process implementation in January 2017.
About REWP’s review
Following up from the first exchanges with the RfE task force last spring, REWP included an open
session in the September plenary meeting as to get the first reaction from delegates and IAs
chairs. That session provided some first feedback that was channelled through the task force by
our designated delegates, Andreas Indinger and Georgina Grenon.
After reception of your request last December, our cabinet took the initiative to start an in-depth
analysis of the proposed changes. Several recommendations and questions were discussed and
noted. It was also decided to include all REWP-related IAs in the process. In fact, after a thorough
strategic discussion that finished a year ago, our WP has identified a number of top priority
actions, including the improvement of the communication with the implementing agreements.
Given the importance of the RfE review task and the limited amount of time allocated between the
WP plenary and the deadline for feedback (only 2 weeks), it was decided to share the appropriate
note and proposed new RfE form with all REWP IAs, well in advance of the WP plenary. This was
meant to provide all participants with the necessary time to assimilate the documents and thus
ensure a meaningful discussion during our plenary session. All REWP IAs were invited to take part
on the plenary session, and 9 of them made it. A total of 1.5 hours was devoted to this topic, with
an initial introductory presentation by Carrie Pottinger that was followed up by an engaged
discussion.
REWP’s recommendations and new proposals to the RfE questionnaire & process
A number of recommendations and proposals were identified by the joint work of WP delegates
and IAs chairs on the proposed new RfE process and template. These proposed changes can be
grouped in three categories:
1. Further changes needed : Proposed new assessment format (S/I/na) not effective,
additional changes are seen as necessary
2. Opportunities for further improvement : proposed changes are seen as useful but could be
further improved with new recommendations
3. Minor issues : proposed changes could easily be fixed with new recommendations
1. Further changes needed : Proposed new assessment format (S/I/na) not effective,
additional changes are seen as necessary
The assessment format base on “Sufficient-Insufficient-NA” is not seen as useful and
constructive
Discussion:
• a lot of open questions, such as when to note as “insufficient“ and how to deal with it
• just “sufficient” is very discouraging after 5 years of hard work
• not much room for improvement
• Missing opportunity to give qualitative evaluation, as the space for “further/other feedback”
is minimal
• Missing opportunity for WPs delegates to provide qualitative responses to IAs requests for
support
REWP Proposal:
• without necessarily moving back to a “numerical” exercise, provide more meaningful
feedback such as, e.g.
– Exceeded expectations
– Delivered as expected
– Improvements recommended
– Improvements requested (the assessment should detail which improvements)
– Serious issue identified / show-stopper
– NA
• Provide some extra space for “further/other feedback” for WP delegates to use as they see
fit
• Provide some extra space for WP to provide answers to IAs requests for support
2. Opportunities for further improvement : proposed changes are seen as useful but could
be further improved with new recommendations
•
•
•
Main criteria categories remained unchanged from previous RfE format (Structural and
management, Performance-based, Value based criteria). However, several questions of the
“value” section could easily be absorbed in the performance section, thus providing the
same information in a shorter and more streamlined way (i.e. without having to repeat
identical information first on what it was and then on the “value” it created)
Summary page :
– Column b, lines 28 to 32 : should be open for scoring as a joint WP agreement
– Delegates to be invited to summarize their feedback on their summary form
Section a)
•
•
•
•
•
•
•
– Reference is made to the “work program”: where is it detailed now?
Section b)
– A question refers to the “barriers above”: which barriers does it refer to?
– A question refers to “IA legal text” : what does it refer to ? Hyperlink possible?
Section c)
– “When has a report been made/will be made to the relevant working party (ref.
Working Party requirements).” Which report?, why and how to evaluate that
information?
– “Is the Annual Report provided to the Secretariat a timely and complete manner (ref.
IEA Framework)?” REWP never gets these reports. Cannot be evaluated by REWP
– “Operations and procedures of the IEA” : where are they? Hyperlink possible?
Hyperlink to the “IEA framework” possible? // Idem for “WP requirements”
– Suggestion to include hyperlinks to “annual reports” and “two pagers”
Sections d), e) and f)
– Try to have more parallelism (respectively, contribution to technology, deployment
and policy)
– Example, if “three most significant” things are asked in one, they can be asked as
such on the other set of questions
Section e)
– Reference is made to “strategy”: it should be replaced by “strategic objectives” as
per new form.
– Opportunity to make link into lines 11 and 12
Section g) on environmental protection should be significantly adjusted
– Why addressing just costs of environmental protection?
– Can add effective identification of issues, studies to resolve, mitigate or address EP
in some way (these can reduce costs… but also increase them!)
Section i)
– Add “to partner countries” after “outreach efforts”
Section 4 :
– Insert back a split of budget into ExCo Support, Annexes and Communication.
Perception that having some budget for ExCo support (OA) is more efficient than
sole volunteered-based steering
– Remember (and remind!) that not all IAs are directly comparable… so quantitative
information should be used appropriately
3. Minor issues : proposed changes could easily be fixed with new recommendations
•
•
•
•
•
No place through the form to note whose feedback it is!
 need to add a (self-filling) cell that notes the name of country and delegate
Scoring columns through the form do not address past AND future (as do the summary
page)
 clarify/change?
No “summary rating” at the end of each session
 Include one that then gets picked up by summary page. For example in line with the
title…
Lots of references are made in the form
 Put a hyperlink into the “Energy Research & Technology Strategy” that gets referred
to in the form
 Include a cell to put hyperlink to the IAs website
 Note that references can be made to all these documents for further reading in
exceptional cases, but the “essence” has to be filled in the excel. The above listed
documents will not be evaluated or be part of the evaluation!
Several reporting documents are cited
•
•
 Clarify any distinction between:
• the “annual report”
• the “yearly report that is due to the secretariat”
• the “two-pagers”
The Excel format still not perceived as reader-friendly
 Clarify how the web-based format will provide more readibility
IAs would still need to run a strategy process to make a strategic plan, which is part of the
“old scheme”
 Provide elements/recommendations on how best to integrate both exercises
REWP’s propositions to the ongoing review process
As a result of the discussions, during and around the last REWP meeting, two important
recommendations emerge. They are aimed at improving this process while respecting the original
calendar for mandatory new RfE questionnaire & process in January 2017:
1) Further and clearer communications from the Secretariat to the IAs
It came out of the discussions that IAs still do not fully understand the objectives of the RfE
exercise nor are convinced about the results and expected outcome of the process. For example,
if one objective is simplifying the whole process, the current version of the questionnaire does not
come across as a much simpler one. Also, it is good to ask IAs to be accountable, but it has to be
clearly recognized and taken into account that IAs are voluntary agreements, can be very different
in nature, and therefore they cannot be compared across each other in an over-simplified manner.
REWP would like to request for the Secretariat to be clearer and more precise in communicating
with IAs about the objectives of this exercise and how its results will be used in the future.
2) Add a testing cycle
It is only after doing the exercise of filling in the modules/questionnaires that one can truly realize
the crucial points and major gaps. This was certainly the case with our deep discussion on the
“two pager” during our last REWP meeting, which contrasted with the more abstract discussion on
the RfE questionnaire.
There is an opportunity to include a first “trial period'' during which 1-2 IAs per WP make the
exercise of filling-in the revised questionnaires (including the changes requested/proposed by the
REWP and other WPs). After a live exercise like that, CERT and the Secretariat will be able to
finalise a questionnaire which is concise, focused and effective.
A suggestion for a new calendar including a testing cycle could be:
– June : CERT validates new proposed RfE questionnaire
– June : target "test" IAs are contacted to see if they would be willing to run this
questionnaire during the next cycle, at least one by WP (notice that for RfE
scheduled for autumn documents are due already this May)
– Winter/early spring : test IAs present their RfE based on the new questionnaires
– April : WPs send suggestions for improvements
– June : CERT approves final RfE questionnaire with mandatory use as of January
2017 (as originally planned)