Evaluating the Uncertainty of
Land-Use Regression Models
Halûk Özkaynak
US EPA, Office of Research and Development
National Exposure Research Laboratory, RTP, NC
Presented at the
CMAS Special Symposium on Air Quality
October 13, 2010
Land-Use Regression (LUR)
Models
• Widely-used methodology for estimating
individual exposure to ambient air pollution in
epidemiologic studies
Point
Sources
Area
Sources
Line
Sources
2
LUR Strengths
• Able to capture smaller-scale variability in
community health studies
• Less resource intensive –
– Easier to develop and apply compared with
other methods for measuring or estimating
subject-specific values (e.g., household
measurements, physical modelling)
• Land-use data widely available
3
LUR Limitations
4
• Inputs
– Require accurate monitoring data at large number of
sites - e.g., in highly industrialized urban areas with
many types of emission sources
• Application in health studies
– Not transferable from one urban area to another
– Do not address multi-pollutant aspects of air pollution
– Lack the fine-scale temporal resolution needed for
estimating short-term exposure to air pollution
– Often estimate ambient air pollution only versus
indoor and personal
• Lack the ability to connect specific sources of emissions
to concentrations for developing pollution mitigation
strategies
Analysis Goals: New Haven Case Study*
• Use air pollution predicted by coupled regional
(CMAQ) and local (AERMOD) scale air-quality
models
• Develop and evaluate land-use regression
models for:
– Benzene
– Nitrogen oxides (NOx)
– Particulate matter (PM2.5)
• Examine (in future) the implications of alternate
LUR development strategies on model efficacy
for multiple pollutants
Source: Johnson, M., Isakov, V., Touma, J.S., Mukerjee, S., and Özkaynak, H. (2010).
5
Evaluation
of Land Use Regression Models Used to Predict Air Quality Concentrations
in an Urban Area. Atmospheric Environment, Vol. 44, pp: 3660-3668.
Air Pollution Data
• Air pollution concentrations were predicted at 318
census block group sites in New Haven, Connecticut
using a coupled air quality model (Isakov et al., 2009)
• Predicted daily
concentrations for 2-month
periods in winter and
summer (2001) were used to
calculate seasonal average
concentrations for benzene,
NOx, and PM2.5 at each site
– July- August for summer
– January- February for winter
6
et al. 2009. Journal of the Air and Waste
Management Association; 59(4):461-472.
• Annual averages were based Isakov
on 365 daily means for 2001
LUR Model Structure and Inputs
Dependent Variables
Independent (Predictor) Variables
Pollutant Concentrations
Benzene, NOx, and PM2.5 Predicted
by Coupled Regional and Local
Scale Air Quality Models
=
Traffic
Intensity and
Proximity to
Roadways
+
• Traffic intensity near
the home (vpd/km2)
• Proximity (1/distance)
to major roadways
Proximity to
Ports and
Harbors
+
• Proximity (1/distance)
to seaports
• Proximity (1/distance)
to harbors
Population
and Housing
Density
+
• Population density in
census block group
• Housing density in
census block group
Proximity to
Industrial
Sources
• Proximity to
industrial emitters of:
–Benzene
–NOx
–PM2.5
• Multivariate linear regression models
• Initial pool of 60 potential predictors
• Eliminated variables based on
7
– High correlation (R-squared ~1.0) with other
selected predictors and/or
– Lack of interpretability
19 land-use variables included in model selection
Site Selection
• Sites
– Census block group centroids
• Training Sites
– Sites used to fit LUR models
– Selected from 318 census
block groups in the study
area
– Stratified random selection
among 4 census regions
• Test Sites
– Remaining sites withheld
from training set - minimum of
10%
used for independent model
evaluation
8
Model Development and Evaluation
9
• Variable selection
– Examined correlation structure for predictive
variables
• Model development
– All subsets with 3-7 independent predictors
– Model selection based on AIC, Mallow’s C(p),
adjusted r-squared, and variance inflation factor
• Model evaluation
– Cross-validation within training dataset
– Hold-out evaluation within test dataset
• Models for multiple pollutants and training sites
– Benzene, NOx, PM2.5
– 25, 50, 75, 100, 125, 150, 200, and 285
• Automated, iterative process
– Site selection -> model development
– Repeated 100x for each pollutant and number of
training sites
Model Performance in Test versus
Training Sites: Benzene
Proportion of Variance Explained (R2)
1.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
RSQ Predicted vs Observed Benzene in Test Dataset
0.1
RSQ LUR Models for Benzene in Training Dataset
0
0
25
50
75
100
125
150
175
200
Num ber of Sites in Training Dataset
10
225
250
275
300
Model Performance in Test versus
Training Sites: NOx
Proportion of Variance Explained (R2)
1.1
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
RSQ Predicted vs Observed NOx in Test Dataset
0.1
RSQ LUR Models for NOx in Training Dataset
0.0
0
25
50
75
100
125
150
175
200
Num ber of Sites in Training Dataset
11
225
250
275
300
Model Performance in Test versus
Training Sites: PM2.5
Proportion of Variance Explained (R2)
1.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
RSQ Predicted vs Observed PM2.5 in Test Dataset
0.1
RSQ LUR Models for PM2.5 in Training Dataset
0
0
25
50
75
100
125
150
175
200
Num ber of Sites in Training Dataset
12
225
250
275
300
LUR Prediction Errors: NOx
• Prediction error =
– Average (+/- SD) of
mean predicted minus
observed input values
– For 100 iterations - aka
100 LUR models
• Analyzed by low, medium,
and high NOx
concentration based on
total NOx distribution
– Low = 0 - 25th percentile
– Medium = 25th - 75th
– High = 75th - max
13
Rotterdam Area LUR versus Dispersion Model
(Hoek et al., 2010)
)
F
Dispersion Model
LUR Model
LUR Model Evaluation in Oslo from Hoek et al., 2010
Courtesy: Christian Madsen (Oslo)
F ull model
L OOC V
adj R
2
adj R
2
Validation
T raining s ets *
20 loc ations
40 loc ations
adj R
2
adj R
2
2005:
NO x
NO 2
NO
0.63
0.69
0.57
0.61 - 0.66
0.67 - 0.72
0.53 - 0.59
0.52 - 0.68
0.60 - 0.78
0.45 - 0.61
0.58 - 0.70
0.64 - 0.76
0.51 - 0.65
NO x
NO 2
NO
0.62
0.70
0.56
0.59 - 0.65
0.68 - 0.72
0.51 - 0.59
0.55 - 0.70
0.56 - 0.76
0.53 - 0.65
0.65 - 0.67
0.70 - 0.74
0.57 - 0.61
2008:
Comparison of Two LUR Models for Amsterdam
(Hoek et al., 2010)
Comparison of Two LUR Models for Amsterdam
Denoting Sites Impacted by Traffic or Urban Sources
(Hoek et al., 2010)
Summary and Conclusions
• We used air pollution concentrations predicted by coupled regional
and local scale AQ models to develop and evaluate LUR models in
New Haven, CT for benzene, PM2.5, and NOx
• Model performance and robustness improved as number of sites
used to build the models increased
– R-squares were inflated for models based on pollutant concentrations
from 25 trainings sites compared with models based on 100 -285
training sites
– R-squared for LUR model (training dataset) and R-squared predicted
versus observed (test dataset) converged as training sites increased
• It is critical to evaluate LUR performance using site-specific
independent measurement data sets
• Analysis suggests that coupled air quality models could provide a
useful tool for improving LUR estimates of exposure to ambient air
pollution in epidemiologic studies
18
• LUR model performance may be considerable poorer than
emissions based modeling results for urban environments with
complex sources and landscape characteristics
Acknowledgements*
•
•
•
•
•
•
19
Markey Johnson
Vlad Isakov
Joe Touma
Shaibal Mukerjee
Luther Smith (Alion Incorporated)
Ellen Kinnee (Computer Science Corporation)
*Although this work was reviewed by EPA and approved for publication, it may
not necessarily reflect official Agency policy
Additional Slides
20
Mean Contribution of Land-Use Factors in
Benzene Models
Proportion of Variability Explained
Benzene 25 Site Models
Proportion of Variability Explained
Benzene 100 Site Models
Models Based on
25 Training Sites
13%
2%
Models Based on
100 Training Sites
Intercept
10%
Traffic Intensity (vpd/km2)
8%
0%
12%
20%
11%
Proportion of Variability Explained
44%
Benzene 285 Site Models
Models
Based on
Proportion of Variability Explained
25 Site Models
285 Benzene
Training
Sites
15%
13%
5% 0%
2%
27%
Proximity
Proximity to Ports and
Harbors
Proximity to Industrial
Sources
20%
Population and
Housing
Density
Proximity
Harbors
Proximity
Sources
Populatio
Density
Traffic Intensity (vpd/km2)
Proximity to Roadways
20%
Proximity to Ports and
Harbors
Proximity to Industrial
Sources
Population and Housing
Density
11%5%
21
48%
Traffic Inte
Proximity to Roadways
Intercept
10%
22%
Intercept
38%
LEGEND
Intercept
Traffic Intensity (vpd/km2)
Proximity to Roadways
Proximity to Ports and Harbors
Proximity to Industrial Sources
Population and Housing Density
Mean Contribution of Land-Use Factors in
NOx Models
Proportion of Variability Explained
Models
Based on
NOx 25 Site Models
Proportion
of Variability
Explained
Models
Based
on
NOx 100 Site Models
25 Training Sites
100 Training Sites
1%
0%
Intercept
1%
Traffic Intensity (vpd/km2)
15%
21%
Intercep
1%
Traffic In
16%
28%
Proximity to Roadways
24%
38%
Proximity to Ports and
Harbors
Proximity to Industrial
Sources
17%
Population and Housing
Density
LEGEND
Proportion of Variability Explained
Benzene 25 Site Models
285 Training Sites
15%
11% 5%
0%
2%
10%
0%
32%
22
48%
Proximi
Harbors
Proximi
Sources
Populat
Density
38%
Proportion of Variability Explained
Models
Based
NOx 285 Site
Models on
13%
Proximi
Intercept
Intercept
Traffic Intensity (vpd/km2)
20%
Proximity to Roadways
Traffic Intensity (vpd/km2)
Proximity to Ports and
Harbors
Proximity to Industrial
Sources
Population and Housing
Density
Proximity to Ports and Harbors
Proximity to Roadways
Proximity to Industrial Sources
Population and Housing Density
Mean Contribution of Land-Use Factors in
PM2.5 Models
Proportion of Variability Explained
Models
Based
PM2.5 100 Site
Models on
Proportion
of Variability
Explained
Models
Based
on
PM2.5 25 Site Models
25 Training Sites
100 Training Sites
1%
Traffic Intensity (vpd/km2)
8%
5%
6%
9%
9%
Inter
0%
Intercept
2%
Traf
0%
Prox
Proximity to Roadways
7%
Proximity to Ports and
Harbors
Proximity to Industrial
Sources
Population and Housing
Density
73%
Proportion of Variability Explained
PM2.5 285
Models
Models
Based
on
Proportion
ofSite
Variability
Explained
LEGEND
285 Training Sites
Benzene 25 Site Models
13%
11%
11%
2%
0%
1% 5%
10%
Intercept
Traffic Intensity (vpd/km2)
20%
Proximity to Roadways
0%
23
83%
80%
Proximity to Ports and
Harbors
Proximity to Industrial
Sources
Population and Housing
Density
Intercept
Traffic Intensity (vpd/km2)
Proximity to Roadways
Proximity to Ports and Harbors
Proximity to Industrial Sources
Population and Housing Density
Prox
Harb
Prox
Sou
Pop
Den