National and Kapodistrian
University of Athens
EVENT MANAGEMENT IN MULTIVARIATE
STREAMING SENSOR DATA
offline
online
Event Management in Sensor Network
What is an event?
• The term “event” is used to describe an alteration on one
or more variables monitored by the system
• Two kinds of processing modules with respect to an event
• Online event processing: focuses on real event detection,
identification of time dependent correlations and causalities
• Offline event processing: event storage, post-processing of
stored events and data -warehousing
Online event processing
Event streams
Sensor streams
0.15
s2
s3
250.0
8.0
248.3
251.0
23.0
21.4
342.1
22.9
e2
e1
•0
•1
•0
•0
e3
En
0.12
E2
150.0
E1
0.2
Event
detection
s1
•0
•1
•0
•0
•0
•0
•1
•0
t
0.8
0.6
0.5
e2  e3  e1
e1  e2
e2  e3
Probabilistic
Temporal Reasoning
Event
prediction
Adaptive
Filtering
Event
correlation
e2e3
e1
e2
e1e 2
e3
Dependency
structure
t
Event/Change Detection
• Sensor streams arrives as raw data that provide instant
measurements
• Generation of event streams over an existing set of
sensor streams
• The problem concerns both detecting whether or not a
change has occurred, or whether several changes might
have occurred, and identifying the times of any such
changes.
Event/Change detection algorithms
• Change detection algorithms
• Cumulative Sum (CUSUM)
• Shewhart Controller
• Multivariate Autoregressive Model (MAR)
CUSUM(1/3)
• The input parameters for the CUSUM algorithm are the
following:
• the target value μ
• the above-tolerance value 𝑘 +
• the below-tolerance value 𝑘 −
• the above-threshold value 𝑡ℎ𝑟𝑒𝑠ℎ+
• the below-threshold value 𝑡ℎ𝑟𝑒𝑠ℎ−
• The output parameters for the CUSUM algorithm are the
following:
• the above-detection signal 𝑠 + ∈ {0,1}
• the below-detection signal 𝑠 − ∈ {0,1}
CUSUM (2/3)
CUSUM (3/3)
• Experiment set up
• μ = 0.5
• 𝑡ℎ𝑟𝑒𝑠ℎ+ = 𝑡ℎ𝑟𝑒𝑠ℎ− = 1.3
  0.5,k  k   0.3,thresh  thresh   1.3
Positive sum P
Cumulative sums
Univariate time series xt
(Acceleration m/sec)
Negative sum N
Positive change
  0.5,k  k   0.3,thresh  thresh   1.3
Negative change
Time steps t
Time steps t
Shewhart Controller (1/3)
• In the Shewhart control chart, a variable 𝑥𝑡 is detected to
deviate at time t from its normality whenever exceeds one
of the control limits
• Control limits
• Upper Control Limit (UCL)
𝑈𝐶𝐿 = 𝑥𝑡 + 𝑘 ∙ 𝜎𝑡
• Lower Control Limit (LCL)
• 𝐿𝐶𝐿 = 𝑥𝑡 - 𝑘 ∙ 𝜎𝑡
Shewhart Controller (2/3)
Shewhart Controller (3/3)
k 3
Univariate time series xt
(Acceleration m/sec)
UCL
LCL
UCL and LCL
Detected change
Time steps t
Multivariate Autoregressive (MAR)
Multivariate Autoregressive (MAR)
thresh2
Detected change
x1,t
thresh2  7%
x1,t
 2
Relative Error e2t
Variable 1 estimation
(Luminance cd/m2)
e2,t
Time steps t
Time steps t
thresh1
Detected change
Relative Error e1t
e1,t
thresh1  5%
x2,t
 2
Variable 2 estimation
(Luminance cd/m2)
x2,t
Time steps t
Time steps t
Event Correlation
• Technique for making sense of a large number of events
and pinpointing the few events that are really important in
that mass of information
• Accomplished by looking for and analyzing relationships
between events.
• Implemented by a piece of software called “event
correlator”
Event correlation: step-by-step
• Event filtering
• consists in discarding events that are deemed to be irrelevant by
the event correlator
• Event aggregation
• a technique where multiple events that are very similar (but not
necessarily identical) are combined into an aggregate that
represents the underlying event data
• Event masking
• consists in ignoring events pertaining to systems that are
downstream of a failed system
• Root cause analysis
• It consists in analyzing dependencies between events, based for
instance on a model of the environment and dependency graphs, to
detect whether some events can be explained by others
Event Correlation Engine (ECE)
• Typical event correlation scheme (univariate data)
• A transition from object (i.e., event or sequence of events) A to
object B occurs if and only if B occurs immediately after A (i.e., not
within a time window).
• Only one object is considered at each step of the sequence (i.e.,
there are no objects occurring at the same time).
• Event correlation over multivariate sensor data
• an alerting situation or a malfunctioning system is expected to lead
to several events triggered at the same time step.
Correlation of Multivariate Event Data
• Stepwise correlation
• Based on a first order Markov chain
• Variable-order correlation of Multivariate Event Data
• Based on idea of partial matching [Fan et al. 1999]
• Event correlation based on sliding window
• Hybrid scheme that correlates events within a time window
Stepwise Correlation
A
B
C
1
0
1
0
1
1
0
0
0
1
0
1
0
1
0
0
1
0
0
0
0
1
0
0
0
1
0
PAC = 1
EV1  1,0,1
AC
EV3   0,0,0
PAC =
1
3
PAC =
1
2
PBC =
PAC,BC = 1
AC
PBC =
PAC,BC = 1
AC
EV4  1,0,1
PAC =
EV2   0,1,1
1
3
PBC =
PAC,BC = 1
AC
1
4
EV5   0,1,0
1
5
PAC,B =
B
1
2
PAC =
P =
PBC, = 1
1
3

P =
PBC, = 1
BC
2
5
PAC,BC =
AC
1
2
PBC =
1
5
EV8  1,0,0
1
4

P =
PBC, = 1
BC
1
5

P ,BC = 1
EV6   0,1,0
2
PB =
6
B
PB,B = 1
PAC,B =
1
2
PAC
2
=
6
AC
PAC,BC =
1
2
PBC
1
=
6
BC
PB =
PBC, = 1
1
P =
6

P ,BC = 1
2
7
PAC,B =
B
PB,B =
P ,BC = 1
PB =
EV7   0,0,0
BC
BC
2
4
1
2
A
B
1
2
PB,
PB =
0
1
0
1
1
0
0
0
1
0
1
0
1
0
0
1
0
0
0
0
1
0
0
0
1
0
PAC,B =
B
C
1
2
8
PB,B =
PB,
1
2
1
2
PAC =
2
7
PAC,BC =
AC
1
2
PBC =
1
7
PBC, = 1
2
7

BC
P ,BC = 1
1
=
2
1
2
P =
PAC =
2
8
PAC,BC =
AC
1
2
PBC =
1
8
P =
PBC, = 1

BC
P ,BC =
1
=
2
2
8
1
2
P ,A =
1
2
PA =
1
8
A
PB =
3
9
PAC,B =
B
PB,B =
1
2
EV9   0,1,0
PB,
1
2
PAC =
2
9
PAC,BC =
AC
1
2
PBC =
1
9
P =
PBC, = 1
BC
P ,BC =
1
=
2
1
2
P ,A =
PA,B = 1
2
9

1
2
PA =
A
1
9
Variable-order correlation
• Partial matching algorithm [Fan et al.199]
m  2,l  1
A
B
C
1
0
1
0
1
1
0
0
0
1
0
0
0
1
0
EV3   0, 0, 0
EV1  1,0,1
A/1
EV2   0,1,1
B/1
C/1
BC/1
C/2
B/1
A/1
AC/1
C/1
/1
AC/1
BC/1
B/1
C/1
C/1
BC/1
BC/1
B/1
BC/1
/1
/1
A/1
B/1
AC/1
C/2
B/1
C/1
BC/1
B/1
C/1
BC/1
/1
/1
/1
/1
/1
/1
/1
B/1
C/1
BC/1
/1
/1
/1
Variable-order correlation
EV4  1,0,0
/1
A/2
A/1
C/2
B/1
C/1
BC/1
B/1
C/1
BC/1
/1
/1
/1
/1
/1
/1
/1
A/1
EV5   0,1,0
/1
B/1
BC/1
/1
/1
A/1
A/1
B/2
BC/1
/1
/1
A/1
A/1
A/2
A/1
B/2
C/1
BC/1
B/1
/1
/1
/1
AC/1
C/2
B/1
C/1
BC/1
/1
B/1
C/1
BC/1
/1
/1
/1
A/1
/1
/1
/1
AC/1
B/1
C/1
BC/1
/1
/1
/1
m  2,l  1
A
B
C
1
0
1
0
1
1
0
0
0
1
0
0
0
1
0
Sliding window algorithm
• Address time dependencies among events within a specific
timeframe
• At each the algorithm the algorithm recalculates probability
values with respect to a sliding window taking into account
the new event vector arrived at the current time step t
• The algorithm has memory of exactly w time steps
• Directed graph G=(V, E) where V=P(I) is the power set of
I={𝑒1 … … . . 𝑒𝑛 }
• Graph Vertexes :
• Weighted transition edge:
Sliding window algorithm
• Frequency of each vertex, a – indicator
• For estimating the probabilities within two nodes, b -
indicator
• The b-indicator examines whether the event sets of two nodes
occur at two, possibly separate, time steps .
Sliding window algorithm
• Two steps
• First step: t < w
• Frequency of each vertex-node
• Probability of occurrence
• Frequency of v ∈ V within the occurrence of some node u ∈ V
• Conditional probability
Sliding window algorithm
• Second step: t > w
• Frequency of each vertex-node within the last w time
• Probability of occurrence
• Frequency of v∈V within the last w after the occurrence of some
node u ∈ V
• Conditional probability:
Sliding window algorithm
EV1  1,0,1
w3
A
1
B
0
C
PA1,w = 1
1
0
1
0
1
0
0
1
1
0
EV2   0,1,0
P
PC1,w = 1
1,w
PC,A
=1
A
2,w
A
1,w
1,w
PAC,A
= 1 PAC,C
=1
1,w
PA,AC
=1
1,w
PAC
=1
EV3  1,0,0
3,w
PB,A
=
3,w
PA,B
=
PA3,w =
2
3
3,w
C,A
P
2
=
3
PC3,w =
1
3
1
4
2,w
PA,C
=
1
2
2,w
PA,AC
=
1
2
2,w
C
P
C
PB2,w =
2,w
PC,B
=
1
2,w
1 PAC,C = P 2,w = 1
2 C,AC
2
2
2,w
PAC,B
=
AC
2,w
PAC
=
B
1
2
1
2
1
2
EV4  1,1,0
1
2
PB3,w =
C
1
3,w
1
3,w
PC,B
=
PA,C
=
3
4
2
1
3,w
3,w
1
3,w
PAC,A =
PA,A =
PC,AC
=
3
2
3
1
3,w
PAC,C =
3
1
3,w
PAC,B
=
1
3,w
3
PA,AC =
AC
4
1
3,w
PAC
=
3
A
1
2
2,w
PAC,A
=
1,w
PC,AC
=1
AC
2,w
PC,A
=
A
C
1,w
PA,C
=1
1
=
2
1
2
1
=
2
2,w
PA,B
=
1
3
PA4,w =
B
1
3
4,w
PA,B
=
1
2
4,w
PB,A
=
1
4
A
PB4,w =
B
4,w
PB,B
=
4,w
PA,A
=1
4,w
PAB,A
=1
4,w
PA,AB
=1
4,w
PAB,B
=1
4,w
PB,AB
=
AB
4,w
PAB
=
2
3
1
3
1
2
1
2
1
2
Event processing
• A method of tracking and analyzing (processing) streams
of information (data) about things that happen (events),
and deriving a conclusion from them
• Complex event processing, or CEP, is event processing
that combines data from multiple sources to infer events
or patterns that suggest more complicated circumstances
• Techniques for CEP
• Event-pattern detection
• Event abstraction
• Event filtering
• Event aggregation and transformation
• Modeling event hierarchies
CEP categories
• Two main categories
• Aggregation-oriented CEP: an aggregation-oriented CEP solution
is focused on executing on-line algorithms as a response to event
data entering the system. A simple example is to continuously
calculate an average based on data in the inbound events
• Detection-oriented CEP: focused on detecting combinations of
events called events patterns or situations. A simple example of
detecting a situation is to look for a specific sequence of events.
Adaptive filtering of rules
• Use of aging or decay function
•
  rt   f t 
f  t  Linear or exponential degradation
  ri   
  ri   exp  ki 
2k
 i  1  k  1
n 1
k 1
  ri 
k  0.8
  ri 
k  0.3
k  0.1
k  0.3
i
pr , t 
t
  r   p
i  t t 1
t
i
i  t t 1
i
k  0.06
i
r ,i
  r 
k  0.1
Rules probability
k  0.03