NC State University MEAS Climate Modeling Laboratory
Robert J. Mera
March 6, 2008
Marine, Earth and Atmospheric Sciences
North Carolina State University

NC State University MEAS Climate Modeling Laboratory
•
Objective and Motivation
•
Principles of Ensemble Forecasting
•
Traditional Relative Operating Characteristic
(ROC) and Economic Value (EV) analysis
•
Extended Relative Operating Characteristic
(EROC)
•
Conclusions
•
Applications and Future Work

NC State University MEAS Climate Modeling Laboratory
Climate prediction is becoming increasingly important for different sectors of the economy worldwide
Courtesy: El Universo A case of false alarm/miss

NC State University MEAS Climate Modeling Laboratory
• Ensemble Prediction System (EPS) forecasting is a method used to account for uncertainties and errors in the forecasting system (recall
Chaos theory)
• Through the ensemble approach one can generate probabilistic forecasts for assessing a future event such as excessive rains, droughts, etc.

NC State University MEAS Climate Modeling Laboratory
• A decision maker becomes a user of weather forecasts if he/she alters his/her actions based on forecast information
• A cost-loss analysis can be assessed based on a 2x2 matrix in which we evaluate the skill of a probabilistic forecast
No
No No cost
(  )
Yes False Alarm
(  )
Yes
Miss
(  )
Hit
(  )
Hit Rate: H= δ/(

+δ)
False Alarm Rate: F=

/(

+

)

NC State University MEAS Climate Modeling Laboratory
• In our contingency matrix we compute the hit rate and false alarm for an array of ensemble member groups
• Example: 15 ensembles used
– Hit rate and false alarm calculated for only 1 out of 15, 2/15 . . . n/N
• We can use this information to analyze the skill of a model

NC State University MEAS Climate Modeling Laboratory
The Relative Operating Characteristic (ROC)
•
The ROC method is widely used for estimating the skill of ensemble prediction systems (EPS)
(Marzban, 2004)
•
The closer a curve is to the upper-left-hand corner, the more skillful the forecast system is
•
A perfect forecast system would have a ROC area
(ROCA) of 1
•
A system with no capability of distinguishing in advance between different climate events has a score of 0.5, i.e. lying on the diagonal defined by (0,0) and (1,1)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
Using the information in our contingency matrix we can compute ROC
0.2
0.4
0.6
False Alarm Rate
0.8
Each point on the curve represents a group of ensembles (1/15, 2/15, etc.)
1

NC State University MEAS Climate Modeling Laboratory
• The ultimate utility of climate forecasts is economic and other benefits associated with their actual use in the daily decision-making process of individuals and different organizations
• Simplistically, users of climate forecasts either DO or DO NOT take action, but the
relative value of the forecasts varies with model performance (i.e. hit rate, false alarm)

NC State University MEAS Climate Modeling Laboratory
• The economic value (EV) graphical method provides a measure of the EPS performance in relative
economic terms for a specific hypothetical range of end users (C/L,) varying from 0 to
1
(Richardson 2000a,b)
• EV computes the relative economic value using the hit rate and false alarm rate, where a value of 1 translates to a perfect forecast.
0.3
0.15
0
0 0.33
C/L
For the sake of brevity, we will not discuss the mathematics involved.
0.66

NC State University MEAS Climate Modeling Laboratory
• Based on the ROC skill alone it is not possible to determine if a useful level of skill has been achieved for a specific end user.
• The EV method is more cumbersome to use and its approach could result in some oversimplification of the actual situation for many users because in reality they may have an infinite range of available mitigation options. Also, EV does not provide certain features of ROC that help to diagnose specific characteristics of EPS.
• Our goal was to develop an alternative procedure similar to the traditional implementation of the
ROC graphical method, but one that also provides evaluation for a specific end user.

NC State University MEAS Climate Modeling Laboratory
0.3
0.15
0
0 0.33
C/L
0.66
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0 0.2
0.8
0.4
0.6
False Alarm Rate
• ROC measures skill only for the C/L (μ) with the maximum value (V opt
)
• This effectively ignores any other user
1

NC State University MEAS Climate Modeling Laboratory
• Specific users are more interested in the economic value related their own mitigation options
• EROC allows us to build different base lines for different users
μ= V
(opt)
1
0.3
0.9
0.15
0
0
μ= 0.25
Two users
0.33
C/L
μ= 0.40
0.4
0.66
0.3
0.2
0.8
0.7
0.6
0.5
0.1
0
0 0.2
Base line for μ=ō(=0.33), Vmin=0
ROC
Base line for μ=0.25(<ō), Vmin=0
Base line for μ=0.40(>ō), Vmin=0
0.4
0.6
False Alarm Rate
0.8
1

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.7
0.6
0.5
0.4
0.3
1
0.9
0.8
0.2
0.1
0
0
NC State University
0.2
0.4
Base line for μ=ō(=0.33), Vmin=0
ROC
Base line for μ=0.25(<ō), Vmin=0
Base line for μ=0.40(>ō), Vmin=0
0.6
False Alarm Rate
0.8
1
MEAS Climate Modeling Laboratory
What if the end-user decides to use a forecast only if the V at a certain value?
min is
EROC allows for this.
0.3
Initial: V min
=0
0.15
0.2
Notice the shift
Base line for μ=ō(=0.33), Vmin=0
ROC
Base line for μ=0.25(<ō), Vmin=0.1
Base line for μ=0.40(>ō), Vmin=0.1
0.4
0.6
False Alarm Rate
0.8
1
0.3
0
0
0.15
0
0
V min
0.33
C/L
Initial: V min
=0.1
0.66
0.33
C/L
0.66

NC State University MEAS Climate Modeling Laboratory
•
Each curve in the EV plot represents a particular group of ensembles on the ROC plot (i.e. a point on the ROC curve)
• EROC preserves ROC’s ability to diagnose each ensemble group’s skill and its relative value for a specific user
•
This translates into a possibility of using a smaller number of ensemble members, say 3 or 4 model runs instead of 15
(i.e. a less expensive forecast)
1
0.3
0.9
0.15
0
0 0.33
C/L
0.66
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0 0.2
Base line for μ=ō(=0.33), Vmin=0
ROC
Base line for μ=0.25(<ō), Vmin=0
Base line for μ=0.40(>ō), Vmin=0
0.4
0.6
False Alarm Rate
0.8
1

NC State University MEAS Climate Modeling Laboratory
• An extended ROC (EROC) procedure has been developed from the traditional ROC and the EV graphical methods used for evaluating the performance of ensemble climate/weather prediction systems
• In the proposed EROC approach we recommend construction of user-specific baselines that provide us with an analysis of both skill and value of an
EPS forecast that is tailor-made for a specific user
• EROC allows for a clearer picture of minimum value and an ensemble group’s skill for a particular user

NC State University MEAS
• Implementation of a routine to calculate EROC and EV plots to the CML R statistical library
(currently in its final testing stage)
•
You can view the progress on http://climlab.meas.ncsu.edu/erocroutine.
html
•
An ongoing project in partnership with NCAR using WRF-DART simulations output for EROC implementation
Climate Modeling Laboratory

NC State University MEAS Climate Modeling Laboratory
http://climlab.meas.ncsu.edu
An Extended Procedure for Implementing the Relative Operating
Characteristic Graphical Method
Fredrick H. M. Semazzi, and Roberto J. Mera
Journal of Applied Meteorology and Climatology, Semptember 2006

NC State University MEAS Climate Modeling Laboratory

NC State University MEAS Climate Modeling Laboratory

NC State University MEAS Climate Modeling Laboratory
Decision to issue a forecast of an event (E) to occur is probabilistically based on the criteria: p

 
 p t
Where:
(N): size of the ensemble
(n): number of the runs in the ensemble for which (E) actually occurs
(p): probability given by the ratio (n/N)
This is the threshold fraction above which the event (E) is predicted to occur based on the model forecast