World Meteorological Organization
WMO OMM
Working together in weather, climate and water
5.8 Monitoring and Evaluation of
Capacity Development
WMO
www.wmo.int
Linkage between the next SP, OP and RBB
SP
WMO OMM
OP (2012-15)
KOn11
Del
GSN
STn
ERn1
KOn12
Del
Del
KOn13
SECRETARIAT
RBB (2012-15)
Act
Budget
Act
Budget
Act
Budget
Act
Budget
Framework of the Results-based WMO Operating Plan
and Secretariat Budget 2012-2015
WMO OMM
(5)
ST 1
(8)
ER 1
KOs of RA I … VI
Indicators, Baselines,
Targets
Secretariat Departments
Activities and Budget
KOs of TC1 … TC8
Indicators, Baselines
Targets
ER 1
ER 2
ER 2
Strategic
level
For every ER, each
RA and TC will define specific Key
Outcomes together with Indicators,
Baselines and Targets; based on these
plans and in close coordination the
Secretariat develops deliverables,
activities and budget proposals.
Operational
level
KO’s , KPI’s and Priority Areas
WMO OMM
•
•
•
ST2: Advancing scientific research and application as well as development
and implementation of technology
ER4:Enhanced capabilities of Members to access, develop, implement and
use integrated and interoperable surface-based and space-based systems for
weather, climate and hydrological observations, as well as related
environmental observations, based on world standards set by WMO.
KO 4.1: WMO Integrated Global Observing System (WIGOS) is implemented
•
KPI
– Completion of WIGOS milestones according to WIGOS implementation
plan.,
– Increased availability of observations for users
•
Priority: WIGOS and WIS are priority areas
•
•
Total of 24KOs, most of them have not more than 2KPIs.
Additional 7 KOs on GFCS and DRR as major initiatives
WMO OMM
Strategic planning documents
1.
WMO Strategic Plan : High-level statement of strategic directions and
priorities expressed through Strategic Thrusts and organization-wide
Expected Results and Performance Indicators addressing the Global Societal
Needs;
2.
WMO Operating Plan: converts the organization-wide ERs into specific
outcomes/ deliverables, indicators and targets
3.
Secretariat Implementation Plan and budget: converts the organization-wide
ERs into Secretariat specific outcomes/ deliverables, indicators and targets
and links them to the planned activities and the Secretariat budget;
4.
WMO Strategic Executive summary: In a language appealing to those
outside WMO and those making decisions to fund NMHSs.
and
Tool/System for Monitoring & Performance Evaluation – Tracks
implementation and provides for control and corrective
actions for achieving the ERs.
WMO OMM
WMO Strategy for Capacity
Development : Why?
• Establish a CD Strategy to ensure, in a holistic approach, all actors in
Met/Hydro/Climate work towards the same overall objective :
Facilitating sustainable development of NMHS
Resolution Cg-XVI/B/WP 11.2
• Congress noted that a coordinated and cohesive approach for capacity
development would be needed to enhance capabilities of NMHSs in
developing countries to meet growing Societal Needs at different
levels.
• Congress noted that capacity building is a crosscutting activity and
contributes to all Expected Results (ER), especially ER 6 and
Strategic Thrust 3 of the 2012-2015 Strategic Plan.
WMO OMM
Definitions and Logic of
WMO strategic planning approach
GSNs : Global Societal Needs
STs : Strategic Thrusts
WMO
Strategic
Plan
ERs : Expected Results
KOs : Key Outcomes
Actions and Budget
WMO
OP
20122015
WMO OMM
WMO OMM
Strategic Thrust n3
Expected Result n6
WMO OMM
Tools to Monitor and Evaluate
Progress
• Types: Empirical, Qualitative, Index, breadbasket,
case histories, Indicators (available economic, social or
environmental)
• Example Collection tools:
–
–
–
–
–
Self reporting: Surveys, Visits of PR and national staff
Country assessments
National Records
Reports: OPAGS, Regional WGs, Fellowship Evaluations, project evaluation
Country Profile Data Base
WMO OMM
Conceptual Framework for
Monitoring and Evaluation
Feedback and learning
M&E
M&E
M&E
M&E
Inputs
Activities
Deliverables
Expected Results
Key Outcomes
WMO governance decisions
Activities as
stipulated in the
WMO Operating
Plan based on
contributions
of the
TCs, RA
and the
Secretariat
Deliverables as
stipulated in the
WMO Operating
Plan based on
contributions
of the
TCs, RAs
and the
Secretariat
Expected Results
as stipulated in the
WMO Strategic
Plan
Key Outcomes
as stipulated in the
WMO Operating
Plan
TCs and RAs human
resources
Financial & human
resources:
WMO
Secretariat
WMO Joint Programmes
Members
Partner Organizations’
contributions
Activities
of WMO Joint
Programmes
Activities of
Members
Deliverables
of WMO Joint
Programmes
Deliverables of
Members
WMO Secretariat enabling processes and coordination
Expected
Results
of WMO
Joint
Programmes
Key Outcomes
of WMO Joint
Programmes
WMO OMM
Capacity Development – and ER 6
Expected Result 6:
Enhanced capabilities of NMHSs, in
particular in developing and least developed
countries, to fulfil their mandates.
Looking at Key Outcomes (Kos) and
Key Performance Indicators (KPIs)
How do we measure?
WMO OMM
EC-WG/SOP Questionnaire
• The working group reviewed in 2011 a
standardized questionnaire to assess the
impacts of the achieved results.
• Such a survey could help create baselines and
targets.
WMO OMM
Draft Survey on WMO Impacts of
Achieved Results on Members
Survey Questions for Capacity Development,
ER 6
Key Outcomes 6.1-6.4
KO 6.1: NMHSs and Regional Centers are
improved, particularly in developing and
least developed countries
WMO OMM
KPI 6.1.1: Number of NMHSs and WMO Regional
Centers with defined roles in national and regional
socioeconomic development activities, in cooperation with
international and regional development agencies
1.
Has your NMHS been involved in national or regional socio-economic
development activities in cooperation with international and regional development
agencies?
Yes
2.
Does your NMHS have a defined role in the National Development Plan?
Yes
3.
No
No
If yes, please define your key role:
WMO OMM
KPI 6.1.2: Number of NMHSs with programmes and
networks for collaborating on service delivery at regional
levels
4.
During the past two years, has your NMHS carried out programmes or
become part of a network for collaborating on service delivery at regional level?
Yes
No
5.
Does your NMHS carry out programmes or participate in networks on service
delivery at Regional level? Yes
No
6.
If yes, please give details:
WMO OMM
• KO 6.2: Infrastructure and operational
facilities of NMHSs and Regional
Centres are improved, particularly in
developing and least developed
countries
WMO OMM
KPI 6.2.1: Number of NMHSs with improvement in
infrastructure and operational facilities to address
identified deficiencies
7.
During the last two years has your NMHS implemented or developed
improvements to its service delivery capacity?
Yes
No
8.
If yes,
i.
In the surface observing network?
Yes
No
ii.
In the upper-air observing network?
Yes
No
iii.
In equipment for meteorological/environmental satellite data?
Yes
No
iv.
In the data-processing/forecasting sector?
v.
In other operational sectors of your NMHS (please specify)
Yes
No
KPI 6.2.2: Number of NMHSs with modernized
capacities to develop and disseminate products to their
customers
WMO OMM
9.
What distribution systems do you use to get products and services to users?
i.
Telephone of facsimile
ii.
Media (printed, TV & Radio)
iii.
SMS
iv.
Cell Broadcast
v.
Internet (web pages, ftp, email etc.)
vi.
Social networking (twitter, face book etc)
vii.
Dedicated telecommunication links
viii.
Remote displays (electronic bill boards etc)
ix.
Staff attachment to users of centres of operation
x.
Remote briefing (video conference, telephone, help desk etc.)
xi.
Others (please list)
WMO OMM
• KO 6.3: Education and Training
Development activities at national and
regional levels are improved, especially
in developing and least developed
countries
WMO OMM
KPI 6.3.1: The number of RTCs providing education and
training support for GFCS related activities
• To be completed by WMO Secretariat using
other sources.
KPI 6.3.2: The degree to which Members are getting
value for money from the WMO Fellowship Programme
WMO OMM
10.
Has your NMHS contributed to or had students under the WMO Fellowship
Programme during the past two years?
Yes
11.
No
If yes,
In general terms, how would you rate the value your NMHS has been getting
from the Fellowship Programme?
1= very low; 5= very high
1
2
3
4
5
WMO OMM
KPI 6.3.3: The degree to which the RTCs support the
regional training demands in particular for students from
developing and least developed countries
12.
If you send staff to a WMO Regional Training Centre(s), please indicate by
inserting ‘X’ the WMO Region(s) where the RTC(s) are located:
WMO Regions
I
II
III
IV
V
VI
13.
From your perspective, how would you rate the degree of satisfaction with the
RTC(s) you use?
1= very low; 5= very high
1
2
3
4
5
WMO OMM
• KO 6.4: Capacities of NMHSs are
enhanced through cooperation and
partnerships with other national and
regional organizations
KPI 6.4.1: Development projects and activities funded
through voluntary and extra-budgetary resources
WMO OMM
• To be completed by WMO Secretariat using
other sources.
WMO OMM
Your inputs…ideas?
• Questions in Survey….
– Are we measuring what needs to be measured to
show the capacities of NMHS are being developed?
– Are the questions
• Too simple (yes/no) to be of use?
• Too difficult to answer with enough specificity to show
improvement in 4 years?
• About right
– What other tools does WMO need to develop/improve
(if any) to evaluate and monitor CD?
WMO OMM
Additional Sources
• Survey on the Role and Operation of
Meteorological Services (2011)
• Regional Surveys
• Programmatic Surveys
• Information from assessments done on bilateral
basis, other development agencies or at subregional levels as part of project development
WMO Strategic Plan 2012 – 2015
Key Performance Indicator(KPI) Reference Sheet
KPI:
WMO OMM
Expected Result:
Key Outcome:
UNIT OF MEASURE
PLAN FOR DATA ACQUISITION
Data collection method:
Annex 1
KPI reference
sheet to be
completed
Data source(s):
Timing/frequency of data acquisition:
Estimated cost of data acquisition:
Responsible department:
Responsible individual(s):
DATA QUALITY ISSUES
Data limitations (if any):
Actions planned to address data limitations:
PLAN FOR DATA ANALYSIS, REVIEW, & REPORTING
How the data will be analyzed:
How data will be presented:
Reporting date and to whom:
WMO Strategic Plan 2012 – 2015
Key Performance Indicator (KPI) Reference Sheet
5.2.1: Increased research on operations products and services
WMO OMM
Expected Result 5: Enhanced capabilities of Members to contribute to and draw
benefits from the global research capacity for weather, climate, water and
environmental science and technology development
Key Outcome 5.2: Research in the prediction of high-impact weather on time
scales of hours to seasons is enhanced
UNIT OF MEASURE
Assessment by JSC of WWRP on progress in high-impact weather.
Annex 2
Example
of a duly
completed KPI
Reference
Sheet
PLAN FOR DATA ACQUISITION
Data collection method: annual meetings of WWRP JSC
Data source(s): reports from WWRP working/expert groups/team
Timing/frequency of data acquisition: once per year
Estimated cost of data acquisition: minimal
Responsible department: RES/ARE/WWR
Responsible Individual(s): C/WWR
DATA QUALITY ISSUES
Data limitations (if any):
Actions planned to address data limitations:
PLAN FOR DATA ANALYSIS, REVIEW, & REPORTING
How the data will be analyzed: review of reports by JSC members, discussion at
annual meeting
How data will be presented: report from annual JSC/WWRP meeting
Reporting date and to whom: June each year; to CAS and WMO leadership