Indicators
to measure the effectiveness
of the implementation
of the UNECE Strategy for ESD
Expert group Indicators for ESD:
Our methods, discussions and results
Steering Committee ESD, Geneva 4-5 December 2006
drs. R.M. van Raaij,
ministry LNV, the Netherlands, Dutch NAP “Learning for SD”
0
Outline :
- the context, mandate and limits of our work
- the process, approach and methodology
- the questions, discussions and struggles
- the reflection on comments and other ESD processes
- the set of Indicators
- working with this set of indicators
- further reflections and developments
1
The Strategy :
Strategy for ESD (Vilnius, March 2005):
To facilitate the introduction and promotion of education for
sustainable development (ESD) in the UNECE region to the
realization of our common vision.
The Aim of the Strategy:
To encourage UNECE member States tot develop and incorporate
ESD into their formal education systems, in all relevant subjects, and
in non-formal and informal education.
2
Objectives :
1) Ensure that policy, regulatory and operational
frameworks support ESD
2) Promotion SD through formal, non-formal and
informal learning
3) Equip educators with the competences to include
SD in their teaching
4) Ensure that adequate tools and materials for ESD
are available and accessible
5) Promote research on and development of ESD
6) Strengthen cooperation on ESD at all levels within
the UNECE region.
3
Establishment of Expert group following Vilnius
Mandate: “to develop indicators to measure the
effectiveness of the implementation of the
strategy”
-
Setting up a “framework”
-
Translating objectives into questions : what do we need to
know, what do we want to know, which data are available,
what methodology is available
-
Than: constructing indicators out of these questions, as far
as aggregation is possible and qualitative/quantitative data /
methods are available
4
Establishment of Expert group: Representatives from Austria,
Canada (e-mail), France, Germany, Greece, Italy, Lithuania,
the Netherlands, the Russian Federation, Slovenia, Sweden
and the United Kingdom; United Nations Educational,
Scientific and Cultural Organization (UNESCO), the
Intergovernmental Central Asia Working Group on
Environmental Education and Education for Sustainable
Development, the Environment and Schools Initiative
Network (ENSI) and European ECO-Forum, a coalition of
citizens’ environmental organizations , UNICEF (guest), IUCNCEC (guest), supported by UNECE
4 Meetings of 3 days work : September 2005 Ede, October 2005
Geneva, March 2006 Vienna, May 2006 Den Haag
Policymakers, scientists, educators, evaluators, statisticians,
researchers, practicians, NGO, governmental, universities, ..
5
We look for an evaluation model that covers :
a) the process of implementation
b) The effectiveness of the implementation (as a qualitative
feature of both the process and long-term effects of ESD).
There are several considerations given :
- Operate within the mandate of expert group and the Strategy
- stick to the text / objectives of the Strategy as such
- our work has to be understood in different countries, cultures,
educational systems, political systems, languages (!)
- Mostly based on existing data and methodology (Vilnius)
- not too much questions, keep it simple, look for aggregation
- UNECE is a political and policy-oriented process (international),
so this is the main audience
6
Developing the concepts, take into account :
-Sustainable Development is not a fixed goal, but a developing
concept, process oriented
-“Learning” is a broad concept, especially in ESD regarding to
non-formal and informal education. (For most non-educators it is
“a black box process”)
-Education refers to knowledge, but also to attitude, values,
skills, competences, behaviour. How to capture all ?
-Methodology may be demanding in some respect.
-Data are not always available, besides some in the field of
formal education.
- Education contributes to SD we trust and can be described in a
logical chain, but we can’t “proof” SD as a result from education
alone.
7
Background for our work : Evaluation model
Check List
……. Y/n
……. Y/n
TYPE 1
Type 0
Current
Situation
(baseline)
Policy Framework
TYPE 2
input
Direct /
Indirect
Throughput
activities
TYPE 3
Output
Effects,
Impact
On SD
Outcome TYPE 4 outcome
T=0
2015
8
The expert group gives into consideration to countries:
-With respect of difference between States, each state should
consider a baseline report of the situation in 2005, which is useful
to measure progress
-Into each country the possibility of a general ESD- monitoring
system should be determined. In national action plans the
objectives should be formulated as operational goals.
- observation: through the nature of the Strategy it selves most of
indicators seems to be “checklist” or “input” type ! We are aware,
but its not in our mandate to change that.
-This system of evaluation is not for benchmark or reporting only :
in line with the spirit of the Strategy we should celebrate progress,
share good practices and LEARN !
9
Some Reflection of Steering Committee 2005 :
-too much questions, please aggregate or skip, to have not too
much indicators
-Discussion on relations ESD and SD and how “measurable” this is
-Based on existing data
-Understandable in different educational systems, cultures, dealing
with different (federal) state systems
-Check with UNESCO process
10
Relevant Discussions in our Expert Group,
Some also discussed in Bath (Anglo-German research group), in
Hiroshima (UNESCO expert group) and Paris (UNESCO panel)
-the causality of ESD and SD
-Different views on education (instrumental – emancipator)
-Scale/level of use (international reporting – national – project- …)
-Who wants to know and for what: who asks for indicators en how the
results will be (mis)used
-Availability of data, methodology,evaluation systems, efforts to collect data
-Quantitative vs. Qualitative : in what parameters can we express SD, and
ESD, the use of scales to express indicators
-Focus on formal education, but what about non-formal and informal
learning ?
-“Learning as black-box-system” (esp. people without educational
Experience)
-Differences in educational systems, e.g. the role of curriculum, the power
of Governmental influences,…
-Differences in State organization, relation GO - NGO
11
Causality of ESD and SD a)
We are convinced that ESD is crucial for changes towards a sustainable
development , but can we prove this ?
Chain: Awareness  knowledge  attitude  behavior  changes in life?
Of course a good model, but in practice changes towards SD are also
driven by economy, feelings, culture, technical development, political
urgency, peers/family/ groups, More education =/= more sustainability
,but NO education is a disaster and leads to ignorance.
Our choice in UNECE is NOT to mingle up in SD-indicator discussions, the
outcome of ESD is learning and development of competences.
Compare: Panel for ESD (UK):
ESD enables people to develop the knowledge, value and skills to participate in decisions about
the way we do things individually and collectively (and locally and globally) that will
improve the quality of life now, without damaging the planet for the future.
12
Causality of ESD and SD b)
Who wants to know and what for ? To justify expenses on ESD ?
As causality is difficult, and very tricky, never promise to much, because
there are many other influences than education that conduct behaviour,
and of which your NAP in not in charge ! We notice differences in approach
in ministries of education and ministries of environment , and differences
between stakeholders! Always the danger of “wrong accountability” .
Different views on the role of education:
Instrumental (to support environmental policy / measures by government)
to emancipator: to empower people with competences so they can
participate.
Make as explicit as you can in your NAP ! different approaches ask for
different type of indicators.
13
Scale / level of monitoring:
-international reporting? (Mainly UNECE, but also EU, OESO,
UNESCO, …) (for benchmark or exchange of good practice ?)
-Monitoring progress in National Action plans (for accounability or
for learning ?)
-Reviewing of ESD programmes or projects (to justify the subsidies
or to evaluate the methods and results ?)
-Looking for effectiveness of learning process it selves (not only
WHAT we learned but also HOW we learned in the BEST WAY ?)
-Checking competence development (knowledge, skills, changing
attitudes) of individuals or groups ? (story writing, self-assessment,
other more qualitative methods, ….)
14
Who asks for indicators and who is working with results?
-international organisations as UN, UNECE, EU, IUCN, UNESCO, …
-National governments
-Politicians: minister, parliamentarians (cynical: to prove they do as
they promised in election time/ looking for short term success)
-Policymakers (at all levels)
-(to show improvement as result of their policy
-(to justify the efforts and money spend)
-(to prove that ESD is a “effective” instrument)
-Educators, professionals in (E)SD
-Developing ESD further (process oriented, double loop learning)
-Practitioners in ESD (to improve their performance)
-Participants in learning processes
-Teachers and learners/students; NGO’s / Private sector/ other
-organisations; Individuals / civil society (all different reasons)
-RESEARCHERS (they always want to do more research !)
15
Availability of data, methodology, resources for evaluation and
monitoring
1)
2)
3)
4)
5)
6)
Not everything that counts can be counted (Einstein) : don’t be afraid
for uncertainties and non-countable issue. Look for innovative ways.
Be creative with existing data: there is a lot in economic reporting,
social reports, environmental reports, even educational reports but
THE INTERDEPENDENCE nature of SD and ESD is a NEW
phenomena ! If no new data collected or research here is the main
difficulty.
Formulate the objectives in your NAP as SMART as possible, the
more chance to find quantitative output.
Differences in educational systems: ISCED is approved by the
ministries of Education
Start monitoring NOW, make a baseline-report, make reservation of
10-25 % for E and M ( because that’s also LEARNING)
Set up an E+ M group (participative approach, innovative, collection
of new data (esp. for non- and informal learning !)
16
Quantitative and Qualitative indicators:
- We found very little quantitative indicators.
- Questions “to what extend”…. are difficult to answer (objectively).
- Descriptive indicators are also indicators (but the sentiment for “sound
numbers” is always there !) let this not hold you !
The use of “scales” (only a few of our indicators has agreed to have a
scale)
-linear scale if countable numbers
-Relative scale in case of %
-Non-linear scale if the optimum is not the maximum
-Descriptive scales are difficult in interpretation ! and discutable !
-Not at all – seldom – some – mostly complete
-Not started – beginning – some experience – good practice – expert
-Increasing – stabile phase – decreasing of … (trends)
- (++; +; 0; -; --) or “smiles” or “traffic lights”
17
Focus on formal education ?!
-this we know best, it is in formal system, there is the role of
curriculum, there are existing data, …….
-But for ESD non- and informal learning is as much important !
-Input indicators are part of all actions under your NAP (Dutch: each
project has to fill in for subsidies in begin and end of project, giving
answers to questionnaires are part of the deal)
-Knowledge management / good-practices are systematically
collected
-Case writing and storytelling are acceptable as evaluation
-Social learning requires self-evaluation ! Participation is essential.
18
Final Results :
Products :
-2 reports of the Expert group
-List of indicators, with reference to character of indicator and sources
-Draft format for reporting + Addendum (= set of indicators)
-Informal guidance to support the use of indicators
6 objectives  18 indicators  48 sub-indicators/questions
Three levels of reflection:
• Yes/No mode
• Descriptive part (most qualitative, where possible some
quantitative), with template tables
• Summary and self-assessment
19
Remarks 1) content and character:
•this is a SET of indicators
•The use of this indicators and the preparation of a report should be a
participatory process
•This is the best possible result within mandate and limitations of existing
data, no big efforts in research and extra monitoring
•Own experience : fill in Yes/no + simple description : 3 hours, interactive
feedback from panel : 1 day, extra research for descriptive part : 2 days
•Each county could set up more detailed indicators, but that requires
further evaluation and monitoring system, new research and datacollecting. We recommend counties to do so as feasible in their National
Action Plan for ESD.
•This set of indicators is not for measuring but to look for progress and
LEARN.
20
Remarks 2) Compare with other processes
•This set of indicators is discussed in the Asia-Pacific Expertgroup on
indicators of UNESCO and IUCN and in other discourses.
•Different is the nature of the UNECE process (more in policy-level)
compared by the UNESCO draft plan for implementation, that calls on
UNESCO national focal points, NGO’s schools and governements.
•Another difference is that the work of UNESCO / IUCN will not deliver an
indicator system for international use, but a helpful guidance with
description of different kinds of indicators to set up indicator systems at
national level
•It is recommended to consider ESD in the specific context and needs of
the region. The concept of ESD is the same, but priorities can and will be
different in parts of the world. Diversity should be respected.
21
The use of the UNECE set of indicators :
-2007 : all countries Yes/No Mode; pilot countries full set of description
recommendation to provide baseline report/data and first
analyses of ESD implementation for Belgrade autumn 2007
-2010 : all countries full report
-2015 : all countries full report
Recommendation from Expert group to extend its mandate
- to reflect on the first round of pilot reporting and improve the templates for
indicators and draft format for reporting if necessary.
- to answer on the resolution of EECA: to develop criteria to asses successful
examples of implementation
22
Special thanks:
-To all members of the expert group
-To all criticasters from outside this group
-To UNESCO for cooperation
-To Austria, UNECE and Netherlands for hosting the meetings
-To UNECE secretariat for support the groups work,
esp. Ms. Ella Behlyarova and Ms. Angela Sochirca
23
Thank you for your attention,
Lots of success to us all !
24