Predictive Skill, Predictive Capability and
Predictability in Ocean Forecasting
Allan R. Robinson
Patrick J. Haley, Jr.
Pierre F.J. Lermusiaux
Wayne G. Leslie
Division of Engineering and Applied Sciences
Department of Earth and Planetary Sciences
Harvard University
29 October 2002
Ocean Prediction
• Ocean prediction for science and operational
applications has been initiated on basin and
regional scales. Evaluation is now essential.
• Predictability limit is the time for two slightly
different ocean states to evolve into realistic but
entirely different states
• Predictive capability is ultimately limited by
predictability but errors in data, models and
methodology now limit prediction capability to
shorter times
Predictive Skill
• Qualitative and quantitative evaluation of ocean
forecasts by generic and regional-specific skill
metrics is essential
• Phase errors/structural errors, initial/BC/model
errors and their sources need to be identified
• Simple metrics from meteorology (root-mean square
error, anomaly pattern correlation coefficient) used
now but more sophisticated statistical metrics and
quantitative measures associated with underlying
dynamical processes are required.
CPSE/REA
Coastal Predictive Skill Experiment (CPSE)
• measures the ability of a forecast system to combine model results
and observations in coastal domains or regimes and to accurately
define the present state and predict the future state
– oversampling is required for rigorous quantitative verification
– provides the basis for optimal, efficient sampling for required accuracies
Rapid Environmental Assessment (REA)
• defined in the NATO naval environment as "the acquisition,
compilation and release of tactically relevant environmental
information in a tactically relevant time frame"
ASCOT-01
• Assessment of Skill for Coastal Ocean Transients
– 6-26 June 2001
– Massachusetts Bay and Gulf of Maine
– http://www.deas.harvard.edu/~leslie/ASCOT01/sci_plan.html
• Predictive Skill Experiment
–
–
–
–
quantitative skill evaluation
forecast system development
real-time at-sea forecasting
real-time adaptive sampling
• Coupled Physical-Biological Experiment
–
–
–
–
–
–
initialization surveys of Mass. Bay and Gulf of Maine
wind-induced events, e.g. upwelling and buoyancy circulations
Gulf of Maine inflow to Mass. Bay
Mass. Bay outflow to Gulf of Maine
MWRA diffuser dispersion
verification survey of Mass. May
• Multiple vessels
–
–
–
–
NRV Alliance – SACLANTCEN – La Spezia,Italy
RV Gulf Challenger – UNH – Portsmouth, NH
RV Lucky Lady – UMass-Dartmouth – New Bedford, MA
RV Neritic – UMass-Boston – Boston, MA
ASCOT-01 Data and Modeling Domains
6-26 June 2001
Forecast Skill Metrics
Skill of the operational forecasts is measured using the metrics, Root-Mean-Square
Error (RMSE) and Pattern Correlation Coefficient (PCC). These numbers are computed
model level by model level (1 to 16), and as a volume average. Perfect values of the
RMSE and PCC are, respectively, zero and one.
The metrics RSME and PCC are respectively defined by:
RMSE  (T f  Tˆ )T (T f  Tˆ )  T f  Tˆ
2
f
 T b )T (Tˆ  T b )
PCC 
f
T  T b Tˆ  T b
2
2
(T
where Tˆ denotes the true ocean, T its forecast, T b a background field vector (e.g.
large-scale field, climatological field, etc.), and || . ||2 the vector l2 norm.
f
A classic measure of skill is to compare the RMS and PCC of the forecast with that of
the initial conditions (IC) (persistence). If the RMSE of the forecast is smaller than that
of the IC, the forecast has RMS-skill or beats persistence. Similarly, if the PCC of the
forecast is larger than that of the IC, the forecast has PCC-skill or has better patterns
than persistence.
Observation Errors – ASCOT-01
19 June 2001
ASCOT-01 Skill Metrics
RMS (Temperature - Left; Salinity - Right)
PCC (Temperature - Left; Salinity - Right)
ASCOT-02
• Assessment of Skill for Coastal Ocean Transients
– 7-17 May 2002
– Tyhrrenian Sea, Ligurian Sea, Corsican Channel, Elba
– http://www.deas.harvard.edu/~leslie/ASCOT02/sci_plan.html
• Predictive Skill Experiment
–
–
–
–
–
–
quantitative skill evaluation
forecast system development
real-time at-sea forecasting
real-time adaptive sampling
rigorous test of distributed ocean prediction system
AUV exercise support
• Physics Experiment
–
–
–
–
–
initialization survey of Corsican Channel and Elba island region
flow between Corsica and Elba
anticyclone north of Elba
flow between Elba and the coast of Italy
reduce multi-variate forecast errors
• Multiple vessels
– NRV Alliance – SACLANTCEN – La Spezia,Italy
– AUVs – SACLANTCEN – La Spezia,Italy
– AUVs – MIT – Cambridge, MA
ASCOT-02 Data and Modeling Domains
7-17 May 2002
Observation Errors – ASCOT-02
15 May
16 May
ASCOT-02 Skill Metrics
RMS (Temperature - Left; Salinity - Right)
PCC (Temperature - Left; Salinity - Right)
General Adaptive
Sampling Objectives
• Go to dynamical “hotspots”
• Reduce error variance
 Reduce errors for tomorrow
 Maintain accurate forecast
• Maintain accurate synoptic picture
• Optimal sampling issues






Automate all 3 above quantitatively
Nonlinear and interdisciplinary impacts on the sampling
Optimal sampling can be highly dependent on objectives and metrics
Reducing error in analysis differs from reducing error in forecast
Minimal final time error differs from minimal time-averaged error
Minimize cost function containing 3 terms based on:
 forecasted model errors (ESSE),
 forecasted significant dynamical events (MS-EVA, pattern
recognition)
 maximum length of time an area can be left without updating
Motivations for Adaptive Sampling
Tracks – ASCOT-02/GOATS
• Sample in regions not yet covered to locate local structures
• Sample in regions not recently covered to understand evolution of
structures
• Determine strength and structure of anticyclone north of Elba
• Determine general nature of flow in vicinity of Procchio Bay (e.g.. is
it from north or result of flow through Corsican Channel from
Tyrrhenian turning around island?)
• Evaluate structure and evolution of flow between Corsica and Elba
• Determine impact of flow between Elba and coast of Italy
Adaptive sampling tracks
designed on a real-time basis.
NRV Alliance - Channel Domain
AUV - Procchio Bay
(Top left) Surface temperature after 4 days of model run. Overlaid on the
temperature field are the 50, 200 and 500m isobaths. (Right) Satellite sea
surface temperature. (Bottom Left) Surface current from the ocean model.
All fields are from 3 October.
Currents measured by NRV Alliance
with the ship-borne ADCP during the
first update surveys. Data of 4
consecutive nights are merged. The SE
current in the south-eastern corner is
due to high winds.
Currents measured by NRV Alliance
with the ship-borne ADCP during
update surveys in early October. The
anti-cyclonic eddy has shifted towards
the north.
HOPS ASCOT-01 Simulations (5 meters),
SeaWiFS Composite Imagery and in situ data
13 June
10-17 June
20 June
18-25 June
Conclusions
• It is critically important to interpret and evaluate
regional forecasts in order to establish usefulness
to scientific and applied communities
• Results from ASCOT highlight the dual use of
data for assimilation and skill evaluation and
demonstrate quantitative forecast skill
• Real-time forecast experiments can lead to
discoveries of regional features
• Multi-scale adaptive sampling is a fundamental
component of forecast systems
Issues in Multiscale
Adaptive Sampling
•
Uniformly sampled observations for initialization and assimilation as forecasts
advance in time



sampled uniformly over a predetermined space-time grid, adequate to resolve scales of interest
only a small subset of observations have significant impact on the accuracy of the forecasts
impact subset is related to intermittent energetic synoptic dynamical events
•
Adaptively sampled observations for initialization and assimilation as forecasts
advance in time




sampling scheme tailored to the ocean state to be observed
knowledge of ocean state from ongoing observations, nowcasts and forecasts
adaptive sampling targets observations of greatest impact
efficient adaptive sampling reduce observational requirements by one or two orders of magnitude
•
Subjective adaptive sampling and objective adaptive sampling



sampling can be based on environmental forecasts or error forecasts
forecast information combined with a priori experience to intuitively choose future sampling
objectively, forecast serves as input to a quantitative sampling criterion whose optimization predicts
the adapted sampling
automated objective adaptive sampling
