Evaluation Process 2014
Geoff Callow
Director-Technology
Turquoise International Ltd
IMPART: July 2015
The evaluation process is:•
Published (the process, not the debates)
•
Secure
•
Structured and closely managed
•
Rigorous
•
Fair
•
Professional
•
Many-faceted and demanding to perform
•
Driven at every stage by the semantic tags
2
Your target tags and scores are:5
Excellent. The proposal successfully addresses
all relevant aspects of the criterion. Any
shortcomings are minor.
4
Very Good. The proposal addresses the
criterion very well, but a small number of
shortcomings are present.
Lower scores will not be funded in competitive calls
typical of Transport topics where 13 points or more
can be needed to secure funding.
3
Evaluation is a staged process:•
Eligibility check (Commission)
•
Individual evaluation report (alone, at home)
•
Consensus meeting (5 evaluators plus moderator
and rapporteur, Brussels)
•
Panel Review (Selected evaluators plus
Commission)
•
There is no negotiation stage. Successful
proposals are funded as written.
4
My approach:•
Read the abstract (Quality of proposition)
•
Note the partners and their geographical
distribution (Quality of consortium)
•
Note the duration, the budget and how it is shared
(Pace, value for money, all fully engaged?)
•
Having got the picture, begin reading the
proposal.
5
Evaluation Criteria:•
•
•
Excellence
•
clarity and pertinence of objectives
•
soundness of concept
•
extent of ambition, innovation beyond state-of-art
•
credibility of approach
Impact
•
outputs match programme expectations
•
enhance innovation capacity
•
strengthen competitiveness
•
environmental and social (additional to above)
•
effective measures to exploit and disseminate
Implementation
•
coherence and effectiveness of the work plan
•
complementarity of the participants
•
appropriate management plan
6
Consensus meeting:•
Obsessive attention to the wording of comments with no
“bleeding” between headings.
•
Often the experts converge rapidly but it can take 8 hours to
get consensus, by no means an averaging process. A
dissenting opinion can be recorded.
•
Moderator guides the process, Rapporteur writes down the
agreed words.
•
The agreed words are matched to the semantic tags, thus
attributing a score.
•
The Consensus Report is checked by a quality controller.
7
Final Panel (Panel Review):•
Evaluation results from all calls under the budget heading are listed
in descending order of overall mark.
•
The cut-off point of the available budget with a reserve margin is set
•
Above the line, the words and the marks are checked again.
•
Tie-breaker rules are applied.
•
The cut-off point of the budget is set and projects above the line are
“retained”.
•
The reserve list is noted in case of drop-outs.
•
The end. (ie no negotiation process)
8
In my opinion, you will have:•
All physically met for at least one full working day,
•
A detailed, wide-ranging risk register with intelligent mitigation actions and with
at least one high risk entry,
•
The consortium agreement done and dusted,
•
A background IPR catalogue showing you have “history” and a professional
search that supports freedom to exploit,
•
A detailed marketing plan for each partner for each deliverable, including
evidence of credible responses or relationships with potential customers,
•
Thought carefully about environmental and social benefits outside of your
consortium: trite answers are irksome,
•
Checked with the Commission that your interpretation of the Call is justified.
9
In my opinion, you will not have:•
Assumed that the text of the call means what you want it to
mean. Out of scope activities will be struck out along with
the associated budget,
•
Trotted out uncritically the expectations for global market
growth found by your favourite search engine and claimed
that to address only 1% of that will mean success,
•
Left it all too late to do justice to your idea,
•
Left it all to the consultant to write without checking that
their words convey , clearly and crisply, the controlled
passion that each of your partners has for the proposition.
10
And finally……….
•
Hold it in mind that everything associated with
your proposal has to lie between Very Good (4)
and Excellent (5). That includes the reputation of
all the partners, sub-contractors and the bid-writer.
It also includes the grammar, syntax, spelling and
style of the text. The non-UK evaluators expect
that of us.
•
Remember also that evaluation is strictly
structured. Just provide your responses under the
specified headings. That’s all you need to do!
11