Ensemble Task
Isidora Jankov
Collaborators:
NOAA’s Environmental Modeling Center
NOAA’s Earth System Research Laboratory
1
Ensemble Task AOP12 Activities
Activity
HWRF ensemble for GSI-Hybrid Applications
methods
Establish a functionally similar environment of the
NCEP SREF
Status
Moved to the DA task
Accomplished
HMT Winter Experiment
Delayed
Evaluate ensemble products using CAPS, AFWA, and
NCEP algorithms
On hold
Ensemble Code Repository
Accomplished
Co-host semi-annual DTC Ensemble Task Workshop
Accomplished
Maintain ensemble group wiki page and web site
On-going
Work done late in AOP11, DTC contribution to SREF post-processing
2
Work performed late in AOP11
NAEFS Bias Correction and Downscaling effort
 High-resolution version of SREF (SREFx) was recently
implemented operationally at EMC.
 SREFx at 16km (NMM, NMMB)/17km (ARW) resolution
 Significant resolution upgrade, but still missing small scale
variability needed for NWS forecast grids
 Downscaling to the resolution of the NDFD (~5km) should
allow greater use of NCEP SREF output in the production
of forecasts
Courtesy of Brian Etherton, Jun Du and Tara Jensen
3
Bias Correction and Downscaling
FORECASTS
NMM
NMMB
ANALYSES
BC
NAM
ARW
RTMA
DS
Approach used in NAEFS
 Bias Correction accomplished by taking the mean forecast of each
model core (ARW, NMM, etc.) sub-ensemble of the NCEP SREF and
comparing it to the NAM analysis valid at the same time.
 Downscale by comparing the RTMA analysis (GRID 259, like GRID
197, 1073x689, 5.079km) with the NAM analysis valid at the same
time (10m wind,2m temperature and humidity).
Courtesy of Brian Etherton, Jun Du and Tara Jensen
4
Bias Correction and Downscaling
FORECASTS
NMM
NMMB
ANALYSES
BC
NAM
ARW
RTMA
DS
 Recursive averaging method is used (~30 day mean)
 Bias-Correction (in operations) and Downscaling (DTC testing) from NAEFS
 Bias corrected and downscaled fields for each member
 Also: downscaled forecasts of ensemble min, max, mean, spread, 10th,
50th, and 90th percentiles.
Courtesy of Brian Etherton, Jun Du and Tara Jensen
5
Implementation of downscaling at EMC
 Downscaling code used for
NAEFS ported to DTC
 Testing/evaluation of code
application to existing NCEP
SREF output showed forecast
improvement
 DTC code sent to EMC (Jun
Du) for application to the new
16km/17km NCEP SREF
 Code implemented by EMC in
approximately 2 days, with
downscaled forecasts being
produced since December
(Presentation at AMS Annual
Meeting )
Courtesy of Brian Etherton, Jun Du and Tara Jensen
6
Implementation of bias correction at EMC
3
2.5
2
1.5
SREFX_raw
1
SREFX_BC
0.5
0
3
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
-0.05
SREFX_raw
SREFX_BC
no skill
3
15 27 39 51 63 75 87
15 27 39 51 63 75 87
Ens mean fcst of T2m: RMSE
Prob fcst of T2m: RPSS (12km NAM as ref)
Raw SREFx vs Bias corrected SREFx (Nov. 10 – 30, 2011, against NDAS)
Courtesy of Jun Du, NOAA/EMC
7
Testing/Evaluation at DTC - Results
As implemented at
DTC, the bias
correction reduces
the RMSE of 2m T
forecasts & the
downscaling adds
further
improvement.
Courtesy of Brian Etherton,
Jun Du and Tara Jensen
ARW and NMM members of SREF 2011 – 0900 UTC Initialization
8
Compared to RTMA Analyses
Ensemble Task AOP12 Activities
Activity
HWRF ensemble for GSI-Hybrid Applications
methods
Establish a functionally similar environment of the
NCEP SREF
Status
Moved to the DA task
Accomplished
HMT Winter Experiment
Delayed
Evaluate ensemble products using CAPS, AFWA, and
NCEP algorithms
On hold
Ensemble Code Repository
Accomplished
Co-host semi-annual DTC Ensemble Task Workshop
Accomplished
Maintain ensemble group wiki page and web site
On-going
9
NCEP SREF Functionally Similar
Environment
Activity description: Define & establish scalable, flexible , highly portable
heterogeneous computational Functionally Similar Environment, (FSE) to NCEP SREF
operations.
Objectives and Accomplishments:
1.Develop FSE acceptance evaluation criteria in collaboration w/ NCEP
2.Establish portable integration of NCEP SREF components & DTC platforms
3.Participate in updates of NCEP NEMS code, SREF scripts & data cross
management
4.Continually support research, DET T&E workflow & future R2O
integrations
5.Analyze the differences & integrate NCEP operative & DET research
workflows
6.Work closely w/ EMC to deploy computational & research innovations.
Courtesy of Eugene Mirvis
10
Major Accomplishment:
• NCEP SREF v6.0.0 adapted by DTC
• DTC now has functionally similar
environment for 4-5 of 6 SREF
modules
11
New Generation of The Ensemble Testbed
1. DEVELOPMENT OF THE FSE ACCEPTANCE
CRITERIA
OPS
ENVIRON
-MENT
3. PARTICIPATE IN UPDATES,
OPS CODE & DATA
MANAGEMENT
Conventions
& Criteria
Of Functional
Similarity
6. ACADEMIA SUPPORT & INNOVATIONS
DTC/DE
T
TESTBED
ENVIRON
-MENT
5. MERGING OPS AND RESEARCH
WORKFLOWS
NOAA
ESRL
~ 60 UNIVERSITIES
2. PORTABLE INTEGRATION of OPS
&
Research Computing Platforms
4. SUPPORT RESEARCH, T&E
WORKFLOW
12
12
NCEP NEMS – DTC Ensemble Workflows Merging Analysis
External data
processing
Ensemble
Initialization
NEMS
Stochastic Forcing Perturbation Component
Alternative
Obs. data
NEMS
entry
RIP4
NEMS Apps Layer
BIAS
Correct
&
downscale
GSI
NPS
Standard
•Obs. data
Terrestrial
DATA
IC/LBC
perturb
Land
surface
perturb
YES
GLOB
?
MESO
Gridded Data:
NAM,GFS,
RUC, snow, ice,
SST
U
P
P
Cold
start
NEMS/WRF
Land
surface
init
NO
?
- - not used yet at NCEP
Ensemble
Post
Post
Visualize
& Verify
Shared Physics
NEMS I/O
GrADS
GEMPAK
NCL
Ensemble
post and product
generation
MET
Grid-vs-Grid
VSDB/FVS
Grid-vs-Obs
VSDB/FVS
13
Ensemble Task AOP12 Activities
Activity
HWRF ensemble for GSI-Hybrid Applications
methods
Establish a functionally similar environment of the
NCEP SREF
Status
Moved to the DA task
Accomplished
HMT Winter Experiment
Delayed
Evaluate ensemble products using CAPS, AFWA, and
NCEP algorithms
On hold
Ensemble Code Repository
Accomplished
Co-host semi-annual DTC Ensemble Task Workshop
Accomplished
Maintain ensemble group wiki page and web site
On-going
14
HMT Winter Experiment
 Activity description:
 Use HMTWinter Experiment real-time system to test the impacts of alternate
initial perturbation techniques (including the dynamical downscaling of global
perturbations)
 Test alternate ensemble perturbation techniques (cloud microphysical
parameterizations
 Presentation at NWA Annual.
 Accomplishments:
 Plans developed
 NWA presentation delivered
 Issues:
This task has been affected by changes in staffing which resulted in a significant
delay.
15
HMT Activities AOP12
HMT
real time
DTC 1
DTC 2
Microphysics
HMT
SREF (ARW)
HMT
Initial
Perturbations
HMT
HMT
SREF (ARW)
Grid Spacing
9
9
9
1
2
1. Evaluate the impact of changes in microphysics
2. Evaluate the impact of initial perturbation changes
2012 Tasks necessary to be completed for 2013 activities.
16
Impact of Microphysics on 2010-2011 HMT
NMM
 90% Intensity show over
forecast of precip for ARWFer and ARW-MY members
especially at higher
thresholds – which means
when it rains it pours in
these members
Color groups
Different
Microphysics
Ferrier
Milbrant-Yau
Thompson
Difference (Fcst-Obs) in Near Peak
Intensity in MODE Obj (mm)
Using Attributes from MODE
Objects
ARW
HMT
NAM
GFS
Optimal
Object Threshold (mm)
> 25.4
Intense Cores
Courtesy of Tara Jensen
17
Initial Perturbations Approach
Legend
Perturbation
Global Model Analysis
interpolated on LAM grid
Global Model Forecast
LAM Forecast
00Z
06Z
Forecast Time
XaL = XfL + (XaG – XfG)
18
Proposed Activities for 2013
19
Ensembles
AOP 2013 Proposed Activities
ID
HMT
EN3
EN4
Activity
Four tests related to implementation of the HMT
innovations into SREF
Evaluate options for optimal combination of reference
dynamic cores and corresponding analyses in SREF
(currently beyond 100%)
Evaluate the merits of using 2 vs. 3 dynamic cores in SREF
– using ARW subset available from HMT activity (currently
beyond 100%)
20
Ensemble Proposed AOP 2013 Activities
HMT CONTEXT
SREF CONTEXT
HMT
Real time
DTC 1
(AOP12)
DTC 2
(AOP12)
DTC 3
(AOP13)
NCEP
Operations
Microphysics
HMT
SREF (ARW)
HMT
SREF (ARW)
SREF
Initial
Perturbations
HMT
HMT
SREF (ARW)
SREF (ARW)
SREF
Grid Spacing
9
9
9
9
16
1
2&4
3
1. Evaluate impact of changes in initial conditions in context of SREF
2. Evaluate impact of HMT microphysics in context of SREF
3. Evaluate impact of increased resolution (16 vs. 9 km SREF)
4. Evaluate the impact of mixed microphysics by optimally combining
21
HMT and SREF runs.
Verification
Tressa L. Fowler
22
Verification AOP 2012 Activities
AOP 2012 Activity
Status
MET releases in July and January
Done / Slightly Delayed
Enhancement of software testing
Done, more in progress
New verification tools for T&E support Some done, more in
progress
METViewer GUI enhancements
Initial release of MET TC
LSM verification
In progress
Slated for 4.1 release
Delayed
Tutorial
Community presentations
Feb 4-5
Many completed
MET help
ongoing
23
Goals
Promote easy, consistent, correct verification across the
NWP community, especially in operations where there
is the greatest potential for impact.
Create a cohesive team of scientists and software
engineers engaged in challenging, interesting, rewarding
work.
24
Accomplishments
Software Development
 MET TC
 MADIS data support
 Ensemble spread skill
 GRIB2
 Series analysis tool
Testing and Evaluation support
 HMT verification
 MMET cases
Community support
25
Series Analysis Tool

Frequency Bias
 Gilbert Skill Score
Happy Users 
26
Verification Support of HMT
HMT Ensemble Mean
Capability was added to
METViewer:
User can constrain
aggregation using
Base Rate (aka, observed
relative frequency).
This feature helps users
focus on model skill for
meaningful events without
having to constrain on an
event-by-event basis
No constraint
Base Rate > 0.02
Increases analysis speed and
relevance
27
Verification Implementations
Confidence Tests
Box-and-whiskers
now on most FVS plots
(thanks to Keith Brill
of HPC)
Slide excerpted
from Geoff DiMego
EMC Verification Website (intranet only):
28
http://emc-ls-sand02.ncep.noaa.gov/EMC_VSDB_verif/index.cgi
Verification AOP 2013 Activities
ID
VX2
VX1
VX3
VX4
VX5
Activity Description
Verification support for T&E activities
Community Support: Release, tutorial, help
MET Development – External users
MET Development – Internal users
MET-TC Development (currently beyond 100%)
29
Next Year
Software Development – Internal
 Enhancements to MET TC in support of HWRF testing
 Series analysis tool testing and enhancements
 METViewer GUI redesign
Software Development – External
 Satellite data and cloud verification
 Complete MODE GUI
 Land Surface Model evaluation capability
Testing and Evaluation support
 HMT Evaluation
 Use of MET TC for HWRF evaluations
 Additional MMET cases
 Radiation scheme evaluation
30
Community Support
 Yearly tutorial, with HWRF tutorial if funded
 Updated online tutorial
 MET Help
 Verification lectures at conferences and workshops
31
MET TC Development
Hurricane T&E
development
Community TC
development
32
MODE GUI
Capability developed
outside DTC but will be
leveraged for inclusion in
community software!
33
Track Error Rank of Candidate Model vs. 3
Operational Models
34
Leveraging SURFRAD
 Development of SURFRAD ingest underway
 Useful for radiation scheme evaluations
 Land surface model verification
 Solar forecast evaluation for DOE project
35
Conclusions
 Some work will carry over to 2013, due to staffing changes.
 2013 will be a year of rebuilding.
 We accomplished much in 2012.
 Our 2013 planned work reaches into new areas
 More opportunities to assist with R2O
 More learning opportunities for our staff
36
Director’s Office & Visitor Program
Louisa Nance
37
Director’s Office AOP 2013 Activities
AOP 2012
AOP 2013
Director’s Office Staff
Director’s Office Staff
Non-salary expenses: travel,
external meetings, etc…
Non-salary expenses: travel, external
meetings, etc… (reduced budget by 20 K)
Director’s Reserve (65 K)
Director’s Reserve (65 K) – 85-100%
DTC Visitor Program
DTC Visitor Program – 85-100%
External communications (status:
newsletter is in progress)
External communications – below 100%
CO1: Investigate requirements ECMWFlike environment) – 85-100%
CO2: Post-doctoral Scientist Program –
85-100%
38
Director’s Office Staff
AOP 2012
AOP 2013
 National Director (Kuo) 0.5FTE
 Assistant Director (Nance) 0.55
 National Director (Kuo) 0.5FTE
 Assistant Director (Nance) 0.55
FTE
 JNT
 Management (Brown) 0.1 FTE
 Budget support (Jensen) 0.25 FTE
 Support staff for community
activities (Johnson) 0.5 FTE
 GSD
 Management (Toth) 0.1 FTE
 Budget support
(Etherton/Bernardet & Admin
Asst) 0.25 FTE
FTE
 JNT
 Management (Brown) 0.1 FTE
 Budget support (Jensen) 0.25 FTE
 Support staff for community
activities (Zarlingo) 0.3 FTE
 GSD
 Management (Toth) 0.3 FTE
 Budget support
(Etherton/Bernardet & Admin
Asst) 0.30 FTE
39
External Meetings
AOP 2012
 3 Workshops
 3 Tutorials
 4 Governance meetings
AOP 2013
 1 Workshop
 1 Tutorial
 4 Governance meetings
40
DTC Visitor Program
Framework for engaging the research community in projects
directed at improving NWP, with an eye towards eventually
improving operational forecasts
41
DTC Visitor Program
Supports visitors to work w/ the DTC to test new forecasting &
verification techniques, models & model components for NWP
 PI project – up to 2 months salary & travel & per diem
 Graduate student project - up to 1 yr temp living per diem stipend
& travel expenses for student to work w/ DTC &/or one of its
partners + travel & per diem for up to 2 2-wk visits to the location
of the student by project PI
42
Current DTC Visitors
PI
Institution
Project Title
NCAR
Development of an HWRF Diagnostics Module
to Evaluate Intensity & Structure Using
Synthetic Flight Paths Through Tropical
Cyclones
Adam Clark
U. of Oklahoma
Development & Application of 3-Dimensional
Object Algorithms to High Resolution
Forecasts
Delia Arnold &
Don Morton
Technical U. of
Catalonia / U. of
Alaska –Fairbanks
Moving The NEMS & NMM-B Into a Broader
Community Resource Environment
Robert Fovell /
Travis Wilson
UCLA
Improvements to modeling persistent surface
cold pools in WRF
Sai Ravela
MIT
Deploying the MIT Field Alignment System &
Test-bed (FAST) in DTC
Jonathan Vigh
43
Future DTC Visitors
PI
Institution
Thomas
Galarneau
NCAR
Diagnosing Tropical Cyclone Motion Forecast
Errors in HWRF
Robert Fovell /
Huang
UCLA
Improving HWRF Track and Intensity Forecasts Via
Model Physics Evaluation and Tuning
Man Zhang
CSU
Project Title
Impact Assessment of Cloud-Affected AMSU-A
Radiance Assimilation in TC inner-Core Region
using Hybrid Data Assimilation Approaches
Marion
Mittermaier
UKMET
Incorporating Observations Uncertainty to a Spatial
Probabilistic Verification Framework for km-scale
Models
Adam Clark
U. of
Oklahoma
Object-based Time-Domain Diagnostics for Highresolution Ensemble Forecasting and Evaluation in
NOAA/HWT Spring Forecasting Experiments
44
AOP 2012 Visitor Funds
 EC chair expressed concern over allocating all of the AOP 2012 funds
budgeted for visitor program due to AOP 2013 funding uncertainties
 NSF funds cannot be reprogrammed to cover other DTC activities, so
DTC moved forward with awarding 5 new visitor projects
 Estimate new projects will require only ~15-20 K from the NOAA AOP
2012 funding
 For continued NSF support, will be important to show NOAA matching,
so plan to pursue off-cycle visitor proposals consistent with DTC
priorities once we have a better picture of the AOP 2013 funding
 Demonstrating NOAA matching for visitor program will be important
for future proposals to NSF for both the visitor program and potential
Post-doctoral Scientist Program
45
Potential new activities
Following new activities currently fall in the 85-100 % scenario
46
ECMWF-like Visitor Tools ($128K)
Background: ECMWF has an
interactive meteorological
software application that allows
researchers to easily prepare &
execute research experiments.
Scope: Investigate current
capabilities & consider options
for building & expanding
capabilities to easily prepare &
run research experiments using
NCEP’s models.
Requirements analysis:
 Identify user community
(skills, needs)
 Identify system components
adaptable to these needs
 Identify hardware, software &
data needs & resources
 Identify documentation &
training needs
 Preliminary system design
Planning requires EMC staff participation in this info gathering stage!
47
Post-doctoral Scientist Program ($151K)
Motivation: Mechanism for engaging next generation scientists in
research focused on improving operational forecasts
Proposed format:
• Gather information from EMC on key areas for which they are
interested in contributions from the research community
• Applicants submit proposals for projects focused on an identified
key area
• Successful applicant would reside at EMC in order to work
closely with EMC staff and gain better understanding of
operational requirements
48
Summary
• Activities that engage the NWP community and keep the
community informed about DTC activities are important to fullfilling the DTC mission
• Need to make sure we don’t overlook the importance of these
activities and determine which types are most effective
49