U.S. EPA’s Evaluation of Long Range
Transport Models
Ralph E. Morris
ENVIRON International Corporation
Template
A&WMA CPANS Conference
University of Calgary
April 23-24, 2012
ENVIRON’s work for USEPA LRT Model
Evaluation Study
• Mesoscale Model Interface (MMIF) tool
– Pass WRF/MM5 meteorological model output directly to
CALPUFF with minimal modification (CALMET not needed)
• Document evaluation of CALPUFF and other LRT
dispersion models using tracer field study data
– EPA performed all of the model simulations
• Perform a Consequence Analysis analyzing CALPUFF and
•
CAMx for LRT AQ and AQRV
Evaluate CALPUFF, SCICHEM and CAMx using
atmospheric plume chemistry data
773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-899-0707
www.environcorp.com
2
MMIF Tool for CALPUFF Met Inputs
• CALPUFF needs
•
meteorological variables
defined at grid cell
centers
WRF uses an Arakawa C
grid configuration
– U and V winds defined on
cell edges
– Other met variables
defined on cell centers
– Impossible to pass-through
winds directly to CALPUFF
• Some level of interpolation
is needed
– MMIF elected to interpolate
winds to cell centers
Arakawa C
v, Ky
(i, j)
T, p, q, Kv
u, Kx
(i-1 , j)
(i, j)
u, Kx
(i, j)
(i, j-1 )
v, Ky
3
Inert Tracer Experiment Evaluation
• LRT transport and dispersion algorithms evaluated using
inert tracer field experiments
– Release known amount of inert tracer
– Measure at downwind receptor sites
– Evaluate LRT models ability to predict tracer concentrations
• Examples of past tracer experiment evaluation studies
–
–
–
–
–
1986 8-model study
1990 Rocky Mountain Acid Deposition Model Assessment
1998 EPA evaluates CALPUFF using 2 tracer experiments
1998 IWAQM Phase II recommendations
1994 European Tracer Experiment (ETEX)
 ATMES = real time model evaluation
 ATMES-II historical model evaluation
773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-899-0707
www.environcorp.com
4
USEPA LRT Tracer Test Evaluation
• EPA evaluated CALPUFF using five tracer test field study
measurements:
–
–
–
–
1975 Savannah River Laboratory (SRL75)
1980 Great Plains (GP80)
1983 Cross-Appalachia Tracer Experiment (CAPTEX)
1994 European Tracer Experiment (ETEX)
• Compare CALPUFF GP80/SRL76 with1998 EPA study
– Determine how advances in CALPUFF improve performance
• For some experiments compare multiple LRT dispersion
models:
– CALPUFF; HYSPLIT; FLEXPART; SCIPUFF; CAMx; CALGRID
773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-899-0707
www.environcorp.com
5
GP80 and SRL 75 Tracer Experiments
GP80
• Receptor arcs at 100 km and
•
•
600 km
Evaluate CALPUFF separately
on each arc using different
configurations
Compare with 1998 study
SRL75
• One 100 km receptor arc
• Evaluate CALPUFF using
•
different configurations
Compare with 1998 study
6
CAPTEX and ETEX
CAPTEX (CTEX3 & CTEX5)
ETEX
• Many CALMET configurations • Evaluate LRT models using
default configuration “out-of• Evaluate CALMET & MM5 &
•
CALPUFF
Evaluate LRT models using
default configuration
•
the-box”
Additional CAMx. HYSPLIT and
CALPUFF sensitivity tests
7
Number LRT Dispersion Model Runs
Test
CALPUFF
SCIPUFF
HYSPLIT
FLXPRT
CAMx
CALGRD
GP80
100km
36
0
0
0
0
0
GP80
600km
41
0
0
0
0
0
SRL75
100km
21
0
0
0
0
0
CAPTEX
CTEX3a
25
0
0
0
0
0
CAPTEX
CYEX5a
24
0
0
0
0
0
CAPTEX
CTEX3b
1
1
9
1
16
1
CAPTEX
CTEX5b
1
1
9
1
16
1
ETEX
8
1
9
1
16
0
Total
157
3
27
3
48
3
8
Cross Appalachian Tracer Experiment (CAPTEX)
• 5 tracer releases from
Dayton, OH or Sudbury,
Canada
– CTEX3 – Oct 2, 1983
– CTEX5 – Oct 25-26, 1983
• Six LRT Models Evaluated
– CALPUFF & SCIPUFF
– HYSPLIT & FLEXPART
– CAMx & CALGRID
• Meteorological Evaluation
– Evaluate MM5 and
CALMET
– Identify best performing
CALMET settings
– Used in to determine
USEPA’s recommended
CALMET settings
9
CTEX3: CALMET Sensitivity Tests Evaluation
• 31 CALMET Sens Tests : • 3 MMIF Sensitivity Tests:
– MM5 @ 80, 36 & 12 km
 How MM5 is used within
CALMET
– MM5 @ 36, 12 & 4 km
• CALMETSTAT to evaluate
wind speed and direction
– First Guess
– Step 1 Winds
– None
– CALMET @ 18, 12 & 4 km
– RMAX1/RMAX2 (OA)
Wind Speed
 A = 500/1000
Wind Direction
 B = 100/200
– EPA Recommended
 C = 10/100
 D = NOOBS
Temperature
Humidity
– Compare against common
benchmarks used to interpret
MM5/WRF evaluation
Root Mean Squared Error (RMSE)
Mean Normalized Bias (NMB)
Index of Agreement (IOA)
Mean Normalized Gross Error (MNGE)
Mean Normalized Bias (MNB)
Mean Normalized Gross Error (MNGE)
Mean Normalized Bias (NMB)
Index of Agreement (IOA)
Mean Normalized Gross Error (MNGE)
Mean Normalized Bias (NMB)
Index of Agreement (IOA)
≤ 2.0 m/s
≤ ±0.5 m/s
≥ 0.6
≤ 30°
≤ ±10°
≤ 2.0 K
≤ ±0.5 m/s
≥ 0.8
≤ 2.0 g/kg
≤ ±1.0 g/kg
≥ 0.6
10
WS & WD Bias/Error CALMET w/ 12 km MM5
EXP4 & EXP6 = 12 & 4 km CALMET
A,B,C,D: RMAX1/RMAX2 = 500/1000, 100/200, 10/100, 0/0
8
6
4
2
0
-2
-4
-6
-8
EXP 6D
EXP 6C
EXP 6B
EXP 6A
EXP 4D
EXP 4C
MM5_12km
WS Bias
WD Bias (deg.)
EXP 4B
MM5_12km
EXP 6D
EXP 6C
EXP 6B
EXP 6A
EXP 4D
EXP 4C
EXP 4B
EXP 4A
WSBias
BiasIOA
(ms^-1)
Gross
Error
(ms^-1)
WS
(ms^-1)
EXP 4A
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
WD Bias
WS Gross Error (ms^-1)
WD Gross Error (deg.)
35
2.5
30
2
25
1.5
20
1
15
10
0.5
5
MM5_12km
EXP 6D
WD Error
EXP 6C
EXP 6B
EXP 6A
EXP 4D
EXP 4C
EXP 4B
MM5_12km
EXP 6D
EXP 6C
EXP 6B
EXP 6A
EXP 4D
EXP 4C
EXP 4B
EXP 4A
WS Error
EXP 4A
0
0
11
Conclusions CTEX3 CALMET Model Performance
• CTEX3 MM5 and CALMET wind model performance:
– The EPA-recommended settings of RMAX1/RMAX2 (100/200,
“B” Series) produced best wind model performance.
– Use of 4 km grid resolution in CALMET tended to produce
better wind model performance than 12 or 18 km.
 Can’t say anything about finer grid resolution than 4 km.
– The CALMET wind model performance was better using the 12
and 36 km MM5 data as input than using the 80 km MM4 data.
• CTEX5 also found RMAX1/RMAX2 = 100/200 best wind
•
performance
Note: Not an independent evaluation since WS/WD obs
were also used as input to most CALMET tests
773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-899-0707
www.environcorp.com
12
CTEX3: CALPUFF/CALMET Sens Tests
• CALPUFF evaluation using •
25 CALMET sensitivity
tests
– CALPUFF runs for CTEX3
tracer release
– Evaluate CALPUFF using
ATMES-II 12 statistical
performance metrics
– Determine which CALMET
configuration results in
best CALPUFF performance
• CALPUFF evaluation using
3 MMIF met inputs
Evaluate using ATMES-II
statistical performance metrics:
– Spatial, Temporal & Global
– Bias and Error
– Scatter
– Correlation
– Cumulative Distribution
Statistical Metric
Perfect
Score
Definition
Spatial Statistics
AM  AP
FMS 
Figure of Merit in Space
(FMS)
AM  AP
 100 %
100%
Global Statistics
Normalized Mean Squared
Error (NMSE)
Pearson’s Correlation
Coefficient (PCC or R)
1
NMSE 
 P
i
N PM
 M
R 
i
M
 
 M  Pi  P
i

2
0%

1.0
i



 M
i
M

2



 P
i
P
P  M 
Fractional Bias (FB)
FB  2 B
Kolmogorov-Smirnov
(KS) Parameter
KS  Max C  M
k
  C  Pk 

2



0%
0%
13
)
R 
Two
i




2 
2 

Composite
Statistics
for
Ranking
Mi M
Pi  P Models








2), bias (FB),
• RANK combines statistics
for
correlation
(R
FB  2 B P  M 
spatial (FMS) and cumulative distribution (KS) with
 M4.0
  C  Pk 
KS  Max
k
perfect model giving
scoreCof
RANK  R
2
 1  FB / 2   FMS / 100  1  KS / 100 
 RANK statistics needs to be revised, propose replace FB w/ NMSE
• AVERAGE averages the N model’s rankings from 1 to N
across the 11 ATMES-II statistics with the best model
being model with lowest score closest to 1.0
– 1.0 = model is highest ranked model across all 11 ATMES-II
statistical metrics
14
CALPUFF MPE using 12 km MM5
EXP4 & EXP6 = 12 & 4 km CALMET
A,B,C,D: RMAX1/RMAX2 = 500/1000, 100/200, 10/100, 0/0
Normalized Mean Square Error
(NMSE)
Figure of Metric in Space (FMS)
(Perfect = 100%)
(Perfect = 0)
40%
35%
30%
25%
20%
15%
10%
5%
0%
12K MMIF
EXP 6D
EXP 6C
EXP 6B
EXP 6A
EXP 4D
EXP 4C
EXP 4B
EXP 4A
12K MMIF
EXP 6D
EXP 6C
EXP 6B
EXP 6A
EXP 4D
EXP 4C
EXP 4B
EXP 4A
60
50
40
30
20
10
0
Rank (RANK) (Perfect = 4)
Fractional Bias (FB)
1.6
(Perfect = 0)
1.4
1.2
1.4
1.2
1
0.8
0.6
0.4
0.2
0
1
0.6
(1-KS/100)
0.4
FMS/100
0.2
(1-FB/2)
R^2
12KM_MMIF
EXP6D
EXP6C
EXP6B
EXP6A
EXP4D
EXP4C
EXP4B
0
EXP4A
12K MMIF
EXP 6D
EXP 6C
EXP 6B
EXP 6A
EXP 4D
EXP 4C
EXP 4B
EXP 4A
0.8
15
CTEX3 CALPUFF Evaluation Conclusions
• The CALPUFF MMIF best performing for CTEX3
(worst for CTEX5).
• The CALPUFF/CALMET RMAX1/RMAX2 = 100/200
worst performing RMAX1/RMAX2 configuration
– Contrast to best performing CALMET configuration for
WS/WD
• CALPUFF/CALMET with higher MM5 resolution
(36 and 12 km) performs better than using 80 km
MM4 data.
• Generally, CALPUFF/CALMET using 4 km
resolution performs better
773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-899-0707
www.environcorp.com
16
CTEX3 Six LRT Model Evaluation
• Use common 36 km MM5 meteorology in all models
• Run 6 LRT models out-of-the-box using default
•
•
configuration
Evaluate using ATMES-II statistical metrics
6 LRT dispersion models evaluated:
– Two Lagrangian Puff Models
 CALMET (w/ MMIF, best performing for CTEX3) and SCIPUFF
– Two Lagrangian Particle Models
 FLEXPART and HYSPLIT
– Two Eulerian grid models
 CAMx and CALGRID
773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-899-0707
www.environcorp.com
17
CTEX3 Tracer Test LRT Model Evaluation
(best performing model has lowest value)
18
CTEX3 Tracer Test LRT Model Evaluation
(best performing model has highest value)
19
Conclusions of EPA LRT Evaluation (1 of 3)
• GP80 Tracer Field Experiment
– Using different valid CALMET configurations, the maximum
CALPUFF concentrations vary by factor of 3
 Need standardized model configuration for regulatory modeling
– Since less options, less variation using MMIF
– CALPUFF “SLUG” near-field option needed to reproduce “good”
model performance on 600 km arc from 1998 EPA study
 Raises question of validity of 1998 EPA study since SLUG typically not
used for LRT dispersion modeling
• SRL75 Tracer Field Experiment
– Fitted Gaussian plume evaluation approach can be flawed
 The a priori assumption that the observed plume has a Gaussian plume
distribution may not be valid at LRT distances
773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-899-0707
www.environcorp.com
20
Conclusions of EPA LRT Evaluation (2 of 3)
• CAPTEX Tracer Field Experiment
– RMAX1/RMAX2 = 100/200 (EPA recommendation)
produces best CALMET WS/WD but worst CALPUFF
tracer performance
 EPA needs help from modeling community to identify
regulatory CALMET settings
– CALPUFF/MMIF performs better than
CALPUFF/CALMET for CTEX3 but worst for CTEX5
– CTEX3: CAMx/SCIPUFF perform best followed by
CALPUF/FLEXPART with HYSPLIT/CALGRID worst
– CTEX5: CAMx/HYSPLIT performs best followed by
SCIPUFF/FLEXPART with CALPUFF/CALGRID worst
773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-899-0707
www.environcorp.com
21
Conclusions of EPA LRT Evaluation (3 of 3)
• ETEX Tracer Field Experiment
– Objective was to evaluate models run with default
configuration using common 36 km MM5 inputs
 Exception was to use more all hours puff splitting in
CALPUFF rather than default of once per day
• CAMx, HYSPLIT & SCIPUFF perform best
• FLEXPART & CALPUFF perform worst
– Sensitivity tests for CALPUFF, CAMx and HYSPLIT
 CALPUFF more aggressive puff splitting didn’t help
 CAMx found default configuration best performing
 HYSPLIT found some hybrid particle/puff configurations
perform better than particle-only default
773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-899-0707
www.environcorp.com
22
Questions?
• Full draft LRT tracer report available on EPA SCRAM website
(consequence analysis & plume evaluation to come)
773 San Marin Drive, Suite 2115, Novato, CA 94998 P: 415-899-0700 F: 415-899-0707
www.environcorp.com
23