Machine learning methods with
applications to precipitation and
streamflow
William W. Hsieh
Dept. of Earth, Ocean & Atmospheric Sciences,
The University of British Columbia
http://www.ocgy.ubc.ca/~william
Collaborators:
Alex Cannon, Carlos Gaitan & Aranildo
Lima
Nonlinear regression
Linear regression (LR):
Neural networks (NN/ANN):
Adaptive basis fns hj
2
Cost function J minimized to solve for the weights
Here J is the mean squared error.
Underfit and overfit:
3
Why is climate more linear than weather?
[Yuval & Hsieh, 2002. Quart. J. Roy. Met. Soc.]
4
Curse of the central limit theorem
Central limit theorem says averaging weather data
makes climate data more Gaussian and linear =>
no advantage using NN!
 Nowadays, climate is not just the mean of the
weather data, but can be other types of statistics
from weather data,

 E.g. climate of extreme weather
Use NN on daily data, then compile climate
statistics of extreme weather
=> Can we escape from the curse of the central
limit theorem?

5
Statistical downscaling

Global climate models (GCM) have poor spatial
resolution

(a) Dynamical downscaling: imbed regional climate
model (RCM) in the GCM

(b) Statistical downscaling (SD): Use statistical/machine
learning methods to downscale GCM output.

Statistical downscaling at 10 stations in
Southern Ontario & Quebec. [Gaitan, Hsieh & Cannon,
Clim.Dynam. 2014]

Predictors (1961-2000) from the NCEP/NCAR Reanalysis
interpolated to the grid (approx. 3.75° lat. by 3.75°
lon.) used by the Canadian CGCM3.1.
6
How to validate statistical
downscaling in future climate?
 Following Vrac et al. (2007), use regional climate model
(RCM) output as pseudo-observations.
 CRCM 4.2 provides 10 “pseudo-observational” sites
 For each site, downscale from 9 surrounding CGCM 3.1 grid
cells:
 6 predictors/cell : Tmax,Tmin, surface u, v, SLP, precipitation.
 6 predictors/cell x 9 cells = 54 predictors
 2 Periods:
1971 - 2000 (20th century climate: 20C3M run)
2041 – 2070 (future climate: SRES A2 scenario)
7
10 meteorological stations
8
Precipitation Occurrence Models
Using the 54 available predictors and a binary
predictand (precip./ no precip), we implemented
the following models:
Linear Discriminant classifier
Naive Bayes classifier
kNN (k nearest neighbours) classifier (45 nearest
neighbours)
Classification Tree
TreeEnsemble: Ensemble of classification trees.
ANN-C: Artificial Neural Network Classifier.
9
Peirce skill score (PSS) for downscaled precipitation occurrence:
20th Century (20C3M) and future (A2) periods.
Persistence Discriminant naïve-Bayes kNN
ClassTree TreeEnsem. ANN-C
10
Climdex Climate indices
Compute indices from downscaled daily data
11
Index of agreement (IOA) of climate indices
ANN-F
IOA
ARES-F
ARES-F
SWLR-F
SWLR-F
20C3M
A2
12
Differences between the IOA of
future (A2) and 20th Century (20C3M) climates
ANN-F
ARES-F
SWLR-F
13
Conclusion
Use NN on daily data, then compile climate
statistics of extreme weather
=> beat linear method
=> escaped from the curse of the central limit
theorem

14
Extreme learning machine (ELM): [G.-B. Huang]
ANN:
ELM: Choose the weights (wij and w0j) of the hidden
neurons randomly.
Only need to solve for aj and a0 by linear least squares.
ELM turns nonlinear regression by NN into a linear
regression problem!
15
Tested ELM on 9 environmental datasets [Lima,
Cannon and Hsieh, Environmental Modelling &
Software, under revision]
 Goal is to develop ELM into nonlinear updateable
model output statistics (UMOS).

16
Deep learning
17
Spare slides
18
Compare 5 models over 9 environmental datasets
[Lima, Cannon & Hsieh. Environmental Modelling & Software
(under revision)]
MLR = multiple linear regression
 ANN = Artificial neural network
 SVR-ES = Support vector regression (with
Evolutionary Strategy)
 RF = random forest
 ELM-S = Extreme Learning Machine (with scaled
weights)

 Optimal number of hidden neurons in ELM chosen over
validation data by a simple hill climbing algorithm.
Compare models in terms of:
RMSE skill score = 1 – RMSE/RMSEMLR
 t = cpu time

19
RMSE skill score (relative to MLR)
20
Cpu time
21
Conclusions
ELM turns nonlinear ANN into a multiple linear
regression problem, but with same skills as ANN.
 ELM-S is faster than ANN and SVR-ES in 8 of the 9
datasets and faster than RF in 5 of the 9 datasets.

 When dataset has both large no. of predictors and
large sample size, ELM loses its advantage over ANN.

RF is fast but could not outperform MLR in 3 of 9
datasets (ELM-S outperformed MLR in all 9
datasets).
22
Online sequential learning




Previously, we used ELM for batch learning. When new
data arrive, need to retrain model using the entire data
record => very expensive.
Now use ELM for “online sequential learning” (i.e. as
new data arrive, update the model with only the new
data)
For multiple linear regression (MLR), online sequential
MLR (OS-MLR) is straightforward. [Envir.Cda’s
updateable MOS (model output statistics), used to postprocess NWP model output, is based on OS-MLR]
Online sequential ELM (OS-ELM) (Liang et al. 2006,
IEEE Trans. Neural Networks) is easily derived from OSMLR.
23
Predict streamflow at Stave, BC at 1-day lead.
 23 potential predictors (local observations, GFS
reforecast data, climate indices) [same as Rasouli
et al. 2012].
 Data during 1995-1997 used to find the optimal
number of hidden neurons (3 for ANN and 27 for
ELM), and to train the first model.
 New data arrive (a) weekly, (b) monthly or (c)
seasonally. Validate forecasts for 1998-2001.
 5 models: MLR, OS-MLR, ANN, ELM, OS-ELM
 Compare correlation (CC), mean absolute error
(MAE), root mean squared error (RMSE) and cpu
time

24
25
26
Conclusion
 With
new data arriving frequently, OS-ELM
provides a much cheaper way to update &
maintain a nonlinear regression model.
Future research
 OS-ELM
retains the information from all the
data it has seen. If data are non-stationary,
need to forget the older data.
 Adaptive OS-ELM has an adaptive forgetting
factor.
27