Commercial wireless communication
microwave links as an opportunistic
sensor network
for environmental monitoring
Hagit Messer
messer@eng.tau.ac.il
7 Nov. 2014
Empirical plot (ITU)
The Physics:
EM
attenuation
vs.
Rain-rate
relation
AdB  aR LEff
b
R [mm/h
Leff [km]
a, b= function
of (frequency, rain
temp., DSD)
In microwave frequency bands, a major source of
signal interruptions is precipitation (rainfall)
The opportunity: Microwave use in
Cellular Communications (10-40 GHz)
Base Station 1
User 1
Microwave Link (~75% in EU)
Base Station 2
User 2
Wired Line,
Fiber Optics (~25%)
About 75% of European backhaul
links are microwave links
The Idea: high resolution, low cost
meteorological monitoring
=> Measurements from EXISTING wireless networks
can be used for environmental monitoring.
The Result:
The available measurements (Israel)
Israel - area:
20,770 / 22,072 km2 (151st)
8,019 / 8,522 sq mi
Data Availability in Israel (2013) –
RSL time series from thousands of microwave links
New in
2012:
2014
Thanks!
Links’ Frequency Distribution
Red markers were used for one direction (antenna A to antenna B) and blue markers were
used for the second one (antenna B to antenna A). The plot included 3515 pair links
totalling in 7030 links (Cellcom).
Typical Data: Measured RSL links of
different length, geometry and frequency
link1,2:ramle-ramot_meir
link3,4:Ramle-Azarya
-36
-38
18.01.2010 Cellcom RSL
4:48
-37
-38
-38
-39
-39
0:00
-42
-40
-45
-50
-44
-46
Min
-55 -48
Max
-60 -50
-65 -52
-70
-41
-42
-40
-40
RSL [dBm]
9:36
RSL [dBm]
14:24
-40
RSL [dBm]
19:12
link7,8:Tachlifi-Ramle
-37
-40
RSL [dBm]
0:00
link5,6:ramle-neve_nof_lod
-39
-41
-42
-44
50
100
150
Time [14:27+..min]
0
-43
-44
-44
-45
-45
50
100
150
Time [14:27+..min]
-42
-43
-43
0
-41
0
-46
50
100
150
Time [14:27+..min]
0
50
100
150
Time [14:27+..min]
-75
link9,10:ramle-rasko_lod
-38
-40
link11,12:Eged_Ramle-Ramle
-42
-35
link13:ramle-bayt_arya2
-43
-40
-36
-45
-44
-44
-46
-48
-50
0
50
100
150
Time [14:27+..min]
-50
-45
RSL [dBm]
RSL [dBm]
Features:
• Quantization errors
• Variable dynamic range
• Variable zero level
• Non linear pre-processing
RSL [dBm]
-42
-46
-47
-37
-55
-60
-65
-48
-70
-49
-75
-50
link14:ramle-ramat_dan
-35
RSL [dBm]
-80
-38
-39
-40
0
50
100
150
Time [14:27+..min]
-80
0
20
40
Time [14:27+..min]
60
-41
0
50
100
150
Time [14:27+..min]
Commercial Microwave Network (CMN) as an
opportunistic sensor network for environmental
monitoring
Wireless Sensor
Network – main
characteristics
1.
low cost sensors;
2.
spatially distributed
sensors;
3.
autonomous
sensors.
RSL measurements of
Microwave Links from
communication systems:
1.
low cost sensors - YES
2.
spatially distributed
sensors - YES
3.
autonomous sensors YES
Problems:
1. Installation costs
2. Energy source
3. Communication and
data processing
4. Maintenance
5. Reliability
Our technique DOES
NOT suffer from
these difficulties
Two Parallel Research Directions:
o
o
o
o
Meteorological tools and
Theoretical Signal Processing
applications (mostly ad-hoc)
Questions (mostly related to
sensor networks)
Accurate, high tempo-spatial
resolution rainfall mapping
o Conditions for precipitation
reconstructability
Flood Prediction
o Optimal fusion of different
Study of other-than-rain
(in quality of measurements,
meteorological phenomena
sampling rate and spatial
(fog, moisture)
distribution) sensor networks
Precipitation classification and
robust estimation (rain, snow, o Multimodality: Optimal use of
co/prior information (e.g.,
sleet)
radar)
Rainfall measurements
Graham Upton
Department of Mathematical Sciences, University of Essex
Why? MANTISSA: To improve temporal resolution & to get better access
to specific areas of interest. US: all the above + to improve tempo-spatial
coverage and resolution and to improve accuracy.
Instantaneous Mapping – Radar vs Links
(synthetic example)
Estimation of rainfall fields using commercial microwave communication networks of variable density
Zinevich, A; Alpert, P; Messer, H
ADVANCES IN WATER RESOURCES Volume: 31 Issue: 11 Pages: 1470-1480, 2008
High Resolution Instantaneous Mapping –
Radar vs Links (real data)
Rain Rate Estimation Using Measurements From Commercial Telecommunications Links
Goldshtein, O; Messer, H; Zinevich, A
IEEE TRANSACTIONS ON SIGNAL PROCESSING Volume: 57 Issue: 4 Pages: 1616-1625, 2009
Dynamic reconstruction of rainfall
– results
r  x, y, t 
t
   υr  x, y, t   0
Frontal Rainfall Observation by a Commercial Microwave Communication Network Zinevich,
A; Messer, H; Alpert, P
JOURNAL OF APPLIED METEOROLOGY AND CLIMATOLOGY Volume: 48 Issue:
7 Pages: 1317-1334, 2009
Other mapping techniques
Space–time dynamics of 15-min rainfall depths (two panels per time step) from links (Left) and radars
plus gauges (Right) for September 10, 2011, 2030–2045 and 2230–2245 hours UTC (validation).
Overeem A et al. PNAS 2013;110:2741-2745
©2013 by National Academy of Sciences
ACCURATE RECONSTRUCTION OF RAIN
FIELD MAPS FROM COMMERCIAL
MICROWAVE NETWORKS USING
SPARSE FIELD MODELING
Yoav Liberman, Hagit Messer
Tel Aviv University
School Of Electrical-Engineering
Tel Aviv, Israel
Still open problems with rain fields
reconstruction
o
Coverage
o Accuracy (accumulated vs.
“instantaneous”): min/max, quantization
o Multimodality
o Variability of rain rate with height
o Challenging areas (e.g. urban, slopes)
o …..
Multimodality – preliminary results
Dr. Rana Samuels
Other than rain phenomena
The SP challenge: Attenuation is caused by
many different environmental factors
o The opportunity: DIVESITY! Many
spatially and temporally distributed sensors
of different sensitivity (ML length),
frequency.
o Some interesting results
o
Rain + other phenomena:
ML signal attenuation model
𝐴 𝑛, 𝐿 = 𝐴𝑝 𝑛, 𝐿 + 𝐴𝑤 𝑛 + 𝐴𝑣 𝑛, 𝐿 + 𝐴0 + 𝑁 𝑛 + 𝑞[𝑛] [𝑑𝐵]
Detect and classify these atmosphere phenomena
”Tree Type” Classification Algorithm
Rain
Other
methods
RSL
Statistical
Test:
MFLRT
Precipitation
Sleet
Snow
Other atmosphere
phenomena
Statistical
Test:
GLRT
Moist antenna
or fog
Water vapor
“Dew”
Fog
Characterization of link attenuation
Example: Fog detection
Predominant Fog and Visibility
Monitoring Techniques
Satellite
Systems
Visibility Meters Transmissometers
Human
Observer
Fog Induced attenuation in the microwave region:

F
- Fog induced attenuation (DB/Km)
LWC- Fog Liquid Water Content (gr/m3)

- Dependent on the temperature- T (K) and link frequency- f (GHz)
Units: ((dB/km)/(gr/m3)
Source: ITU-R P.840-6 (2013): Attenuation due to clouds and fog
N David, P Alpert, H Messer, The potential of commercial microwave
networks to monitor dense fog‐feasibility study Journal of Geophysical
Research: Atmospheres 118 (20), 11,750-11,761, Oct. 2013
Fog induced attenuation up to 100 GHz
2D Simulation test
• Goal: calculating the minimum detectable LWC using
measurements from multiple MLs at 20, 38, 80 GHz.
• How ?
• A fog modeled patch was swept across an area of the whole map
we wished to create.
• 696 microwave links (existing deployment)
• During each of the three simulations (20, 38, 80 GHz) the algorithm
was applied over an area of about 14,000 square km
Sensitivity tests at 20, 38 and 80 GHz (simulations)
Gultepe, I., Müller, M. D., and Boybeyi, Z.: A new visibility parameterization for warm-fog applications in numerical
weather prediction models, J. Appl. Meteor. Climatol., 45, 1469–1480, 2006.
Publications: review papers
O Sendik, H. Messer, A New Paradigm for Precipitation Monitoring,
IEEE Signal Processing Magazine, in press (to be published on May
2015)
o Messer, H. ; Zinevich, A. ; Alpert, P., Environmental sensor networks
using existing wireless communication systems for rainfall and wind
velocity measurements, IEEE Instrumentation & Measurement Magazine,
Volume: 15, Issue: 2 , pp 32 – 38, April 2012
o Messer, H.; Goldshtein, O.; Rayitsfeld, A.; Alpert, P.; Recent results of
rainfall mapping from cellular network measurements, Acoustics, Speech
and Signal Processing, 2008. ICASSP 2008. IEEE International
Conference on March 31 2008-April 4 2008 Page(s):5157 – 5160
o Messer, H.; Rainfall Monitoring Using Cellular Networks, Signal
Processing Magazine, IEEE, Volume 24, Issue 3, 2007 Page(s):144 – 142
o Messer, H; Zinevich, A; Alpert, P: Environmental monitoring by wireless
communication networks, SCIENCE, 312 (5774): 713-713 MAY 5 2006
o
Publications: Rainfall Mapping
o
o
o
o
o
o
Y. Liberman, Object Tracking Extensions for Accurate Recovery of Rainfall Maps Using
Microwave Sensor Network, EUSIPCO 2014
Y. Liberman, R. Samuels, P. Alpert, and H. Messer, New algorithm for integration
between wireless microwave sensor network and radar for improved rainfall
measurement and mapping, Atmos. Meas. Tech., 2014
Yoav Liberman, Hagit Messer H,, Accurate Reconstruction of Rain Field Maps from
Commercial Microwave Networks Using Sparse Field Modeling, ICASSP 2014
Zinevich, A; Messer, H; Alpert, P: Frontal Rainfall Observation by a Commercial
Microwave Communication Network, JOURNAL OF APPLIED METEOROLOGY
AND CLIMATOLOGY Volume: 48 Issue: 7 Pages: 1317- 1334, 2009
Goldshtein, O; Messer, H; Zinevich, A: Rain Rate Estimation Using Measurements
From Commercial Telecommunications Links, IEEE TRANSACTIONS ON SIGNAL
PROCESSING Volume: 57 Issue: 4 Pages: 1616-1625, 2009
Zinevich, A; Alpert, P; Messer, H: Estimation of rainfall fields using commercial
microwave communication networks of variable density, ADVANCES IN WATER
RESOURCES Volume: 31 Issue: 11 Pages: 1470-1480, 2008
Publications: Accuracy assessment
o
o
Asaf Rayitsfeld, Rana Samuels, Artem Zinevich, Uri Hadar, Pinhas
Alpert: Comparison of two methodologies for long term rainfall
monitoring using a commercial microwave communication system,
Atmospheric Research. Volumes 104–105, February 2012, Pages
119–127
Zinevich, A; Messer, H; Alpert, P: Prediction of rainfall intensity
measurement errors using commercial microwave communication
links , Atmospheric Measurement Techniques Atmos. Meas. Tech.,
3, Pages: 1385-1402, 2010
Publications: Other Meteorological Applications
o
o
o
o
Ostrometzky J. , Cherkasky, D. and Messer, Accumulated Mixed
Precipitation Estimation Using Measurements From Multiple Microwave
Links, Advances in Meteorology, Sept. 2014
Cherkasky, D. Ostrometzky J. and Messer, Precipitation Classification
Using Measurements from Commercial Microwave Links , IEEE
Transactions on Geoscience and Remote Sensing, Volume: 52 , Issue: 5,
May 2014
N David, P Alpert, H Messer, The potential of commercial microwave
networks to monitor dense fog - feasibility study, Journal of Geophysical
Research: Atmospheres 118 (20), 11,750-11,761, Oct. 2013
David, N; Alpert, P; Messer, H, The potential of cellular network
infrastructures for sudden rainfall monitoring in dry climate regions,
Atmospheric Research Volume 131, Pages 13–21, September 2013
More publications: Other Meteorological
Applications
o
o
o
David, N; Alpert, P; Messer, H: Humidity Measurements using
Commercial Microwave Links In Advanced Trends in Wireless
Communications, Editor: Mutamed Khatib. InTech, February 2011
David N.; Alpert P.; Messer H: Flash floods warning technique based on
wireless communication networks data, Geophysical Research Abstracts
Vol. 12, EGU2010-1169-3, 2010
David, N; Alpert, P; Messer, H: Technical Note: Novel method for water
vapour monitoring using wireless communication networks measurements,
ATMOSPHERIC CHEMISTRY AND PHYSICS Volume: 9 Issue: 7
Pages: 2413-2418, 2009
Publications: Signal Processing
o
o
o
o
o
J. Ostrometzky, H. Messer, Accumulated Rainfall Estimation Using
Maximum Attenuation of Microwave Radio Signal, SAM 2014
Y Broyde, M Livshitz, H Messer, ADAPTIVE STATISTICAL LEARNING
OF CELLULAR USERS BEHAVIOR, Signal Processing, Volume 93,
Issue 11, Pages 3151–3158, November 2013
O Harel, H Messer, Extension of the MFLRT to Detect an Unknown
Deterministic Signal Using Multiple Sensors, Applied for Precipitation
Detection, Signal Processing Letters, IEEE Volume: 20 , Issue: 10 , Oct.
2013
Sendik, O; Messer, H; On the reconstructability of images sampled by
random line projections, IEEE 27th Convention, ISRAEL 2012
Dani Cherkassky, Jonatan Ostrometzky and Hagit Messer: The Use of
Linear Feature Projection for Precipitation Classification Using
Measurements from Commercial Microwave Links, in Latent Variable
Analysis and Signal Separation Lecture Notes in Computer Science, 2012,
Volume 7191/2012, pp 511-519
Publications: Signal Processing
o
o
Elad Heiman and Hagit Messer, Parameter Estimation from
Multiple Sensors with Mixed Resolution of Quantization, Electrical
& Electronics Engineers in Israel (IEEEI), 2014 IEEE 28th
Convention of
Omry Sendik and Hagit Messer, On the achievable coverage of
rain field mapping using measurements from a given set of
microwave links, Electrical & Electronics Engineers in Israel
(IEEEI), 2014 IEEE 28th Convention of
International Impact
Africa!
Doumounia, Ali, et al. "Rainfall monitoring based on
microwave links from cellular telecommunication
networks: First results from a West African test bed ".
sretteL hcraeseR lacisyhpoeG (2014)
http://raincell01.sciencesconf.org/
Rain Cell Africa first Workshop
11-14 Nov 2014 Ouagadougou, Burkina Faso
Examples of Signal Processing
Challenges
o Classification
of atmospheric phenomena
o Parameter estimation from sensor
networks of different quantization levels
o Optimal rain rate estimation from
Min/Max measurements
Test statistic : MFLRT
max
0≤𝐿<𝑁
𝑙𝑔𝑙 𝑋 − 𝑙 𝑙𝑛
𝑙𝑔𝑙 𝑋 = 𝑙 𝑛
𝑙𝑔𝑙 𝑋
𝑙
+1
𝑢
𝑙𝑔𝑙 𝑋
𝑙
−1
𝐻𝑙
>
<𝛾
𝐻0
𝑃 𝑋 ; 𝜃𝑙 , ℋ𝑙
𝑃 𝑋 ; 𝜃0 , ℋ0
The motivation for this test is that it asymptotically minimizes the
Kullback-Leibler distance (divergence) between the true probability
density function (pdf) and the estimated one.
The Model :
𝑥𝑖 [𝑛] = 𝐿𝑖 𝐴𝑙 [𝑛] + 𝜇𝑖 + 𝑤𝑖 [𝑛]
𝐴𝑙 𝑛 - Unknown deterministic signal of unknown waveform
( 𝑙 – sparse signal )
Extension of the MFLRT to Detect an Unknown Deterministic
Signal Using Multiple Sensors, Applied for Precipitation
Detection
O Harel, H Messer
Signal Processing Letters, IEEE Volume:20 , Issue: 10 , Oct. 2013
Precipitation classification
Experimental setup
Where
Link
Frequency
[GHz]
Distance
[KM]
Antenna
Height [m]
Height from
sea level [m]
Nachla Sde-Moshe
Sde-Moshe –
K.Gat Glikson
18.958/17.948
6
22/20
126/154
18.820/17.810
4
20/25
154
Precipitation Detection - MFLRT VS GLRT
we can see that while the GLRT can serve for detection of precipitation
in a given time period, it is unable to estimate the signal order 𝑙 because, as
mentioned before, the GLRT always chooses the highest order of the signal
(i.e. assigns precipitation to all samples).
Waveform estimation
The usage of the MFLRT overcomes this limitation, and this detector,
in addition, can operate as an estimator for wet/dry samples
Precipitation coverage estimation
Problem definition
The MFLRT can also be used for
deriving the precipitation coverage
• We define the ”precipitation coverage” problem as a problem of
identifying, in a given network, what are the MLs that are
affected by precipitation at a particular time.
Frequency diversity
• The MFLRT decides, in a given network (region), whether or not
precipitation is present at a particular time, and in addition, which
are the MLs that are affected by precipitation (precipitation
coverage problem)
Experimental setup
Where
Results
Examples of Signal Processing
Challenges
o Classification
of atmospheric phenomena
o Parameter estimation from sensor
networks of different quantization levels
o Optimal rain rate estimation from
Min/Max measurements
Our proposed approach – assume a
parametric model
𝑓 𝑥, 𝑦; 𝜽
𝑧𝑖 = 𝑄𝑖 (𝑠𝑖 (𝑓 𝑥, 𝑦; 𝜽 ) + 𝑛𝑖 )
𝒛 = 𝑧1 , 𝑧2 , . . . , 𝑧𝑁
𝑠1 (𝑓 𝑥, 𝑦; 𝜽 )
𝑠2 (𝑓 𝑥, 𝑦; 𝜽 )
𝑇
𝑠3 (𝑓 𝑥, 𝑦; 𝜽 )
53
𝑠4 (𝑓 𝑥, 𝑦; 𝜽 )
27 July 2016
𝜽?
𝑧𝑖 = 𝑄𝑖 (𝑠𝑖 𝑓 𝑥, 𝑦; 𝜽
+ 𝑛𝑖 )
θml = arg max 𝐿(𝒛; 𝜽)
𝜽
𝑁
𝐿 𝒛; 𝜽 = log 𝑃(𝒛; 𝜽) =
log(𝑃 𝑧𝑖 ; 𝜽 )
𝑖=1
𝑚𝑖 = 𝑠𝑖 𝑓 𝑥, 𝑦; 𝜽
Δi
Δi
𝑃 𝑧𝑖 ; 𝜽 = 𝑃(𝑚𝑖 ∈ 𝑧𝑖 − , 𝑧𝑖 + )
2
2
+ ni )
𝑁
θml = arg max
𝜽
log(
𝑖=1
∆
𝑧𝑖 + 2𝑖
∆
𝑧𝑖 − 2𝑖
54
− 𝑚−𝑠𝑖 𝑓 𝑥,𝑦;𝜽
𝑒
2𝜎𝑖 2
2
𝑑𝑚)
27 July 2016
Real data for rain mapping
1
−
2 1−𝜌2
𝑓 𝑥, 𝑦; 𝜽 =R ∙ e
∗[
2
2𝜌 x−μx y−μy
x−μx 2 y−μy
+
−
2
2
𝜎𝑥 𝜎𝑦
σx
σy
55
]
27 July 2016
Real data
18/01/2010, 10 events every 5 min
56
27 July 2016
13:30
13:35
57
27 July 2016
Pelephone only
Pelephone+Cellcom
𝛒
SD
𝛒
SD
13:20
0.4408
3.501
0.5238
3.58
NML
0.4421
3.54
0.5268
3.54
ML
Pelephone only
𝝆𝒄𝒐𝒓𝒓
SD
Pelephone+Cellcom
𝝆𝒄𝒐𝒓𝒓
SD
13:40
0.65
2.64
0.7
1.67
NML
0.66
3.31
0.75
1.43
ML
Pelephone only
𝝆𝒄𝒐𝒓𝒓
SD
Pelephone+Cellcom
𝝆𝒄𝒐𝒓𝒓
SD
14:00
0.452
1.5
0.512
1.4
NML
0.452
1.5
0.512
1.4
ML
Examples of Signal Processing
Challenges
o Classification
of atmospheric phenomena
o Parameter estimation from sensor
networks of different quantization levels
o Optimal rain rate estimation from
Min/Max measurements
60
The Rain Rate Statistics
; Best for heavy storms (>50[mm/hr]).
; Best for light to moderate storms.
Example of
a Log-Normal
(-0.5,1.4)
random process
Rain-gauge
measurements
Tel Aviv,
December 2013
The IMS rain-gauges cannot detect rainrate intensity lower than 0.6 [mm/hr]!
61
Generalized Extreme Value Distribution
Gamma maxima
asymptotically converge
into Type I
Log-Normal
maxima asymptotically
converge into Type II
62
ACCUMULATED RAIN ESTIMATION
FROM MIN/MAX RSL DATA SERIES:
DIRECT MLE FROM THE EXTREME
STATISTICS
63
The Estimation Process
yi
Max RainRate
Calculation
Maximum
Likelihood
Estimation
Trans. to
rain-rate
statistics
Accumulated
rainfall
calculation
• The available set of measurements is the minimum and
maximum RSL data series:
64
The Estimation Process
yi
Max RainRate
Calculation
Maximum
Likelihood
Estimation
Trans. to
rain-rate
statistics
Accumulated
rainfall
calculation
• The minimum and maximum rain-rates are calculated
from the maximum and minimum RSL’s:
• A(tn) is the minimum or maximum attenuation, and is
being calculated directly from the maximum and minimum
RSL data series.
65
The Estimation Process
yi
Min/Max
Rain-Rate
Calculation
zi
Maximum
Likelihood
Estimation
Trans. to
rain-rate
statistics
Accumulated
rainfall
calculation
• The set of min/max rain rates {zi} continue to the maximum
likelihood estimation, which finds the GEV parameters:
66
The Estimation Process
yi
Min/Max
Rain-Rate
Calculation
zi
Maximum
Likelihood
Estimation
^
^^
ε,μ,σ
Trans. to
rain-rate
statistics
Accumulated
rainfall
calculation
• Then, the GEV parameters are transformed into the rain-
rate Probability Distribution Function parameters:
67
The Estimation Process
yi
Max RainRate
Calculation
zi
Maximum
Likelihood
Estimation
^
^^
ε,μ,σ
Trans. to
rain-rate
statistics
^^
κ,Θ
Accumulated
rainfall
calculation
• From The rain-rate PDF parameters, the accumulated
rainfall can be found simply by using the known expected
value formula:
; Gamma PDF expected value
; Log-Normal PDF expected value
68
The Experimental Setup:
69
Results
70
Comparison with other method
The Research Team (2013)
o
Prof. Hagit Messer-Yaron –
o Prof. Pinhas Alpert –
meteorology
signal processing
Graduate Students (2013):
o
Yoni Ostromtzky
Ori Auslender
Yoav Liberman
Elad Heiman
Oz Harel
o
o
o
o
Noam David
o Ronen Radian
Post-doc: Dr. Rana Samuels
o
Graduated:
Dr. Atrem Zinevich (Ph.D)
Oren Goldstein, Asaf Rayitsfeld, Uri Hadar, Yoni Broyde, Lior Palti, Dani Cherkassky,
Yoni Ostromtzky, Omry Sendik (M.Sc.)
Dionis Teshler (B.Sc.), (Anat Neumann)
Acknowledgments
This project could not be done without the
collaboration of Cellcom & Pelephone, and
Partner who provided the data.
o
o
o
In Pelephone, we would like to thank: A. Shilo, N.
Dvela, A. Hival and Y. Shachar.
In Cellcom, we would like to thank: Y. Eisenberg, Y.
Dagan, Y. Koriat, I. Inbar. Shahar Shilian,
Eli Levi
In Partner we would like to thank H. Ben Shabat, A
Shor, H Mushvilli and Y. Bar Asher.