Time Series Data
Statistical Methods in Data Mining
Conceição Amado
Instituto Superior Técnico
Lisboa
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Summary
1
Time Series Data
Introduction
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction
https://www.xenonstack.com/blog/time-series-deep-learning/
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction
Ordinarily, a time series is defined as some quantity that is measured
sequentially in time over some interval.
One defining characteristic of a time series is that its observations
are dependent or correlated, and the order of the observations is
therefore important
This correlation, introduced by the sampling of adjacent points in
time, restricts the applicability of statistical procedures and
techniques that rely on independence assumption and therefore
different methods are needed
Time-series data are of interest due to their ubiquity in various areas
ranging from science, engineering, business, finance, economics,
healthcare, to government.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction: examples
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction: examples
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction: examples
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction
A time series consists of a set of recordings {x1 , x2 , . . .} of a random
variable Xt arranged in the order they were collected over a certain
period of time.
Thus, we can look at a time series as a realization of a certain
stochastic process.
One simple method of describing a time series is the so-called
Classical (additive) Decomposition Model
Xt = mt + st + t ,
where mt is the trend component, st is the seasonal component with
known period d, and t is the random noise component.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction
Trend: is the increase and
decrease in the series over a
period of time, it persists
over a long period of time
Seasonality: regular pattern
of up and down
fluctuations. It is a short
term variation due to
seasonal factors
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction
Decomposition of a time series: example
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction
One first approach to model and forecast from a time series date is
estimate and extract mt and st so that t become stationary.
An alternative approach, developed by Box and Jenkins, consists in
differencing the data until a stationary time series is achieved.
In some instances, differencing may not solve all stationarity problems.
The Box-Cox transformation family is particularly effective at stabilizing
the variance.
Then we can use the theory of stationary processes to find a satisfactory
probabilistic model for t and to use it in conjunction with mt and st for
purposes of prediction.
Note: In the time-series context, we observe a single run of a stochastic process rather than
repeated runs of the stochastic process. We observe 1 long experiment rather than multiple,
independent experiments. We need stationarity and ergodicity so that observing a long run of a
stochastic process is similar to observing many independent runs of a stochastic process.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction
Roughly, a stationary time series is one whose properties do not depend on the
time at which the series is observed. Thus, time series with trends, or with
seasonality, are not stationary: the trend and seasonality will affect the value of
the time series at different times.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction: typical models
Moving Average Processes- MA(q)
A time series Xt is a moving average process of order q, written {Xt , t ∈ Z} ∼
MA(q), if it has the representation
Xt = µ + t +
q
X
θj t−j ,
t ∈ Z,
{t , t ∈ R} ∼ WN(0, σ2 ).
j=1
Autoregressive model - AR(p)
An Autoregressive model of order p ∈ N, often abbreviated to {Xt , t ∈ Z} ∼
AR(p), is of the form
Xt = c +
p
X
φj Xt−j + t ,
t ∈ Z,
{t , t ∈ R} ∼ WN(0, σ2 ).
j=1
The idea behind this is that current values of the process are linearly dependent
on the past p observed values Xt−1 , Xt−2 , . . . , Xt−p .
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction: typical models
Autoregressive Moving Average Processes
As the name suggests these processes constitute a combination of the AR and
MA models. They are particularly appreciated for producing more harmonious
models than their components.
An ARMA(p,q) process has the following representation
Xt = c +
p
X
j=1
φj Xt−j + t −
q
X
θj t−j ,
t ∈ Z, {t , t ∈ R} ∼ WN(0, σ2 )
j=1
Autoregressive Integrated Moving Average Processes
This class of models is an extension of the ARMA models that includes
non-stationary time series that can be reduced to ARMA processes though
repeated differentiation.
A process Xt is said to be an ARIMA process of orders p, d > 0, q ∈ N,
{Xt , t ∈ Z} ∼ ARIMA(p,d,q), if, for every t, ∇d Xt = (1 − B)d Xt is an
ARMA(p,q) process.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction: data preparation
Handling missing, unequally spaced, or unsynchronized values: eg.
using linear interpolation
Gross error removal - eg. perform a moving-Average smoothing and
after try to detect gross errors
Normalization:
Range-based normalization
Standardization
Data Transformation and Reduction (Discrete Wavelet Transform,
Discrete Fourier Transform, Symbolic Aggregate Approximation)
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Introduction: clustering time series
Clustering time series objects is particularly advantageous because it
leads to discovery of interesting patterns in time-series datasets.
As these patterns can be either frequent or rare patterns, several
research challenges have arisen such as: developing methods to
recognize dynamic changes in time-series, anomaly and intrusion
detection, process control...
Time-series clustering is the most-used approach as an exploratory
technique, and also as a subroutine in more complex data mining
algorithms, such as rule discovery, indexing, classification, and
anomaly detection
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Time-series clustering
Given a data set of n time-series data D = {TS1 , TS2 , · · · , TSn } the
process of unsupervised partitioning of D into C = {C1 , C2 , · · · , Ck } in
such away that homogeneous time-series are grouped together based on a
certain similarity measure, is called time-series clustering.
Then, Ci is called a cluster, where D = ∪ki=1 Ci and Ci ∩ Cj = ∅,
i, j = 1, 2, · · · , k.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Time-series clustering
A crucial question in cluster analysis is establishing what we mean by
“similar” data objects, i.e., determining a suitable
similarity/dissimilarity measure between two objects.
In the specific context of time series data, the concept of dissimilarity
is particularly complex due to the dynamic character of the series.
Dissimilarities usually considered in conventional clustering could not
work adequately with time dependent data because they ignore the
interdependence relationship between values.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Time-series clustering
There exist a broad range of measures to compare time series and the
choice of the proper dissimilarity measure depends largely on the nature of
the clustering, i.e., on determining what the purpose of the grouping is.
Once the dissimilarity measure is determined, an initial pairwise
dissimilarity matrix can be obtained and a conventional clustering
algorithm be then used to form groups of objects (e.g., a k-means or a
hierarchical clustering)
For a review: Saeed Aghabozorgi, Ali Seyed Shirkhorshidi, Teh Ying Wah,
Time-series clustering – A decade review, Information Systems, 53, 2015,
16-38, https://doi.org/10.1016/j.is.2015.04.007
For a review: Pablo Montero and José Vilar, TSclust: An R Package for
Time Series Clustering, Journal of Statistical Software, Articles, 62,1,
2014.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Cluster time series: illustration
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Cluster time series: illustration
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Similarity/dissimilarity measures in time-series
clustering
Many dissimilarity measures between time series have been proposed in
the literature.
We will briefly describe some dissimilarity measures.
Let the two real-valued time series {Xt , t ∈ Z} and {Yt , t ∈ Z} have
their partial realizations denoted by XT = (X1 , . . . , XT )T and
YT = (Y1 , . . . , YT )T respectively.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
A simple approach to measure the proximity between XT and YT is to
consider conventional metrics based on the closeness of their values at
specific points of time. Some commonly used raw-values-based
dissimilarity measures are introduced below.
Minkowski distance
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Fréchet distance
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Dynamic Time Warp
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Dynamic Time Warp
This distance employs a robust form of pattern detection by identifying similar
patterns in time series even if they do not coincide in time.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Euclidean and Dynamic Time Warp
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Dissimilarity Index Combining Temporal Correlation and Raw Values
Behaviours (CORT)
While the DTW distance takes only into account the shape of the data, the
CORT distance also factors in the temporal proximity of similar points of data.
The distance is given by
dCORT (XT , YT ) = Φ [cort(XT , YT )] δconv (XT , YT ),
(1)
where δconv (XT , YT ) denotes a conventional distance between the time series,
such as euclidean or DTW, and cort(XT , YT ) ∈ [−1, 1] represents a temporal
correlation coefficient with the formula
Pp−1
i=1 (Xi+1 − Xi ) (Yi+1 − Yi )
qP
cort(XT , YT ) = qP
.
(2)
p−1
2
p−1
2
(X
−
X
)
(Y
−
Y
)
i+1
i
i+1
i
i=1
i=1
A value of cort(XT , YT ) = 1(−1) signifies a simultaneous shift of the same
magnitude and in the same (opposite) direction on the part of both time series,
while cort(XT , YT ) = 0 denotes no relation in their stochastic behaviour.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Autocorrelation-based distances
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Autocorrelation-based distances
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Autocorrelation-based distances (cont.)
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
So far all metrics work in the time domain, but the frequency domain
approach also offers an interesting alternative to measure the dissimilarity
between time series. The key idea is to assess the dissimilarity between the
corresponding spectral representations of the series.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Periodogram-based distance
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Periodogram-based distance
This metric, introduced by Caiado et al. stems from periodogram analysis,
formulated by Schuster as a tool to identify latent periodicity in data. Consider
the Fourier representation of a set of n observations:
[n/2]
Zt =
X
(ak cosωk t + bk sinωk t)
k=0
wherein ωk = 2πk/n, k = 0, 1, . . . , [n/2] are Fourier frequencies and the
Fourier coefficients are given by
( P
n
1
Zt cosωk t,
k = 0, k = n2 if n is even
n−1 ak = n2 Pt=1
n
Z
cosω
t,
k
=
1,
2,
.
.
.
,
t
k
t=1
n
2
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Periodogram-based distance
and
n
2X
bk =
Zt sinωk t,
n t=1
n−1
.
k = 1, 2, . . . ,
2
The periodogram is given by the expression

2

k = 0,
na0 ,
I (ωk ) = n2 (ak2 + bk2 ),
k = 1, . . . , [(n − 1)/2]

 2
nan/2 ,
k = n2 if n is even
Caiado et al. introduce several versions of the periodogram-based distances,
the most robust of which takes the logarithm of the normalized periodogram,
and is given by
v
u[n/2]
uX
dLNP (x, y ) = t
[log (NPx (ωj )) − log (NPy (ωj ))]2 .
j=1
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-free approaches
Other dissimilarities:
Dissimilarity measures based on nonparametric spectral estimators
Dissimilarity measure based on the discrete wavelet transform
Dissimilarity based on the symbolic representation SAX
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-based approaches
Model-based dissimilarity measures assume that the underlying models are
generated from specific parametric structures.
The main approach in the literature is to assume that the generating
processes of XT and YT follow invertible ARIMA models. In such case,
the idea is fitting an ARIMA model to each series and then measuring the
dissimilarity between the fitted models.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-based approaches
Piccolo distance
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Model-based approaches
Maharaj distance
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Prediction-based approaches
Now we focus on a new dissimilarity notion governed by the performance of
future forecasts, i.e., two time series are similar if their forecasts at a specific
future time are close.
Conceição Amado
Statistical Methods in Data Mining
Time Series Data
Introduction
Cluster time series in R
Packages for time series clustering
TSclust: Package for Time Series Clustering. Montero, P and Vilar,
J.A. (2014) TSclust: An R Package for Time Series Clustering.
Journal of Statistical Software, 62(1), 1-43.
dtwclust: Time Series Clustering Along with Optimizations for the
Dynamic Time Warping (DTW) Distance.
https:github.comasardaesdtwclust
Conceição Amado
Statistical Methods in Data Mining