Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Table of contents
1
General................................................................................................................................ 4
1.1.
General ........................................................................................................................................... 4
1.2.
Workspace...................................................................................................................................... 4
1.3.
Supported fields in database .......................................................................................................... 5
1.4.
Widgets .......................................................................................................................................... 6
1.4.1. Filters for all widgets ............................................................................................................ 7
1.4.2. Widget controls ................................................................................................................... 9
1.4.3. Colours of fields in histograms and tables ......................................................................... 11
1.5.
2
Supported browsers ..................................................................................................................... 11
File Manager ..................................................................................................................... 11
2.1.
Accessing and using the File Manager ......................................................................................... 11
2.2.
Functionality of the File Manager ................................................................................................ 12
2.3.
Workspace definitions.................................................................................................................. 14
3
Filter Panel ........................................................................................................................ 14
3.1.
FLOAT and DOUBLE field types .................................................................................................... 14
3.2.
CLASSIFIER and LIST fields ............................................................................................................ 15
3.3.
LONG (ID) fields ............................................................................................................................ 17
3.4.
DateTime fields ............................................................................................................................ 18
3.5.
Latitude-longitude combo field .................................................................................................... 18
3.6.
General ......................................................................................................................................... 18
4
Map Widget ...................................................................................................................... 19
4.1.
Working with heatmap layers ...................................................................................................... 19
4.2.
Filtering ........................................................................................................................................ 20
4.3.
Heatmap colour bar ..................................................................................................................... 21
4.3.1. Heatmap bin sizes .............................................................................................................. 21
4.3.2. Filter by N in each heatmap bin ......................................................................................... 21
4.3.3. Colour bar .......................................................................................................................... 22
4.3.4. Filter and colour bar interaction ........................................................................................ 24
4.4.
Geofence ...................................................................................................................................... 24
4.5.
Cell symbols.................................................................................................................................. 25
4.6.
Two functions of the click on a map............................................................................................. 26
Reach-U 2021
1/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
4.7.
Map layers .................................................................................................................................... 28
4.8.
Further help ................................................................................................................................. 29
5
Map Algebra Widget ......................................................................................................... 30
5.1.
Work principle .............................................................................................................................. 30
5.2.
Getting started ............................................................................................................................. 30
5.3.
More details ................................................................................................................................. 30
6
Table widget ...................................................................................................................... 32
6.1.
The first column ........................................................................................................................... 32
6.2.
Adding more columns .................................................................................................................. 34
6.3.
Hiding NULL values ....................................................................................................................... 34
6.4.
Using table to define filter for other widgets ............................................................................... 35
6.5.
Deep dive into UNIQUE in the table ............................................................................................. 36
6.6.
Other table controls ..................................................................................................................... 37
6.7.
Further help ................................................................................................................................. 38
7
Crosstable Widget ............................................................................................................. 38
7.1.
Hiding NULL values ....................................................................................................................... 39
7.2.
Deep dive into UNIQUE in the crosstable..................................................................................... 42
7.3.
Further help ................................................................................................................................. 43
8
Histogram widget .............................................................................................................. 43
8.1.
Selecting field for the histogram .................................................................................................. 43
8.2.
Histogram controls ....................................................................................................................... 44
8.3.
Further help ................................................................................................................................. 45
9
2D Histogram widget ........................................................................................................ 45
9.1.
Selecting fields ............................................................................................................................. 45
9.2.
2D Histogram controls.................................................................................................................. 46
9.3.
Further help ................................................................................................................................. 47
10 Data types in Sparrow LDB4 database .............................................................................. 48
10.1. Handling of NULL by database engine and API............................................................................. 48
10.2. Supported aggregations in API ..................................................................................................... 49
10.3. Supported operations in API ........................................................................................................ 49
10.4. Extending data types for Map Explorer GUI ................................................................................. 49
11 Data Ingestion Layer Configurability ................................................................................. 49
11.1. Lookup-Translation (LTF) files ...................................................................................................... 49
11.2. LTF file function in a nutshell ....................................................................................................... 50
Reach-U 2021
2/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
11.3. More details of LTF file ................................................................................................................. 50
11.4. Handling of values that are NOT described in LTF file .................................................................. 51
11.5. Using LTF files ............................................................................................................................... 51
11.6. Dropdowns in Map Explorer GUI.................................................................................................. 51
11.7. Bitmap .......................................................................................................................................... 52
11.8. Unsupported cases....................................................................................................................... 52
12 User Interface Configurability ........................................................................................... 52
12.1. Field names, field units, groups and group order ........................................................................ 52
12.2. Color settings for heatmap........................................................................................................... 53
12.3. Color settings for tables and 1D histograms ................................................................................ 55
12.4. Default and available aggregations .............................................................................................. 56
12.5. Default settings ............................................................................................................................ 57
Reach-U 2021
3/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
1 General
1.1. General
Map Explorer is a user interface to expose capabilities of Reach-U Geo Spotlight database engine. Geo
Spotlight is unique in its capability to aggregate geospatial information very fast: the heatmap that uses
150,000,000 records is returned within seconds, the database from where 150M records are extracted
can be in terabytes. Database engine and GUI can render maps, histograms, tables, multi-dimensional
aggregations and tables. More precisely, the database engine returns array of numbers and it is up to
the user interface if it is rendered as table, map or histogram. Geo Spotlight is schema-less database
and can serve wide range of data, as long as time and location (and attributes) are provided.
1.2. Workspace
The layout of the interface is controlled by user in great extent – what widgets with what data, in what
display settings, and which location in screen are shown:
User can edit, save, load and organise workspaces in file manager:
More detailed description is available in file manager help section.
Reach-U 2021
4/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
The same layout may look good for one screen resolution, but bad for other (lower) resolution. This
may happen when switching between video projectors. To adapt the workspace to low resolution, use
browser zoom. Depending on your browser, refresh may be necessary to adapt screen widgets to the
new zoom setting.
(the above screenshot from Chrome, other browsers handle it differently).
1.3. Supported fields in database
The database supports following data types:
FLOAT
DOUBLE
LONG that can store also content that has special handling in GUI layer, e.g. nominal numbers
(guiType=ID) or UNIX timestamps (guiType=dateTime))
CLASSIFIER (can be also called “varchar”)
LIST (data type that can store more than one classifier value in one field (equivalent to N-M
relation) in relational databases)
BOOLEAN
All data types support NULL in the database, but the way user interface handles NULL differs - filtering
by NULL is supported only for CLASSIFIER, LIST, BOOLEAN and LONG (ID) fields. In addition, LONG (ID)
supports !NULL (NOT NULL) as a filter.
User interface configuration permits (this configuration is not accessible for regular users of Map
Explorer user interface):
Converting NULL to text that is more meaningful for given field. It can be "Unknown", "Not
applicable" etc.
guiType=ID numeric values can be shown in GUI as human-readable aliases.
Following operations are supported:
Lazy count for all field types (returns count of rows in database that meets filter criteria)
Breakdown for CLASSIFIER, LIST, BOOLEAN fields (includes count for NULL as one item in the
response)
Histogram for FLOAT and DOUBLE fields (null values are ignored)
Reach-U 2021
5/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Following aggregations are supported, but each of them can be hidden for every field separately in the
system settings (that user cannot change):
Aggregation
CLASSIFIER, LIST,
BOOLEAN
FLOAT, DOUBLE
LONG
LONG
ID
dateTime
MIN, MAX, AVG
-
yes
-
-
COUNT
yes
yes
yes
-
SUM
-
yes
-
-
UNIQUE
yes
-
yes
-
Each user interface widget exposes fields and functionality that depends on the database field type and
user interface settings.
1.4. Widgets
Lower right corner provides "Add New Widget" button:
Following widget types are supported:
Reach-U 2021
6/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
For more details, refer to help in each widget type. Some widgets may be in beta phase. The list of
available widgets may vary – it depends on authorities that particular user has and product
configuration.
Geo Spotlight database can support many data tables. Which data tables are available to users depends
on the product configuration. Every widget runs on one database table and user must select the table
when creating a new widget.
When user logs in, the default workspace provides selected widgets and defaults to the last 15 minutes
of data in the database (the same happens when user creates new workspace). The layout and fields in
widgets are defined in product configuration.
1.4.1. Filters for all widgets
The content of each widget is generated for subset of database table that is within the defined:
Time period,
Geographic area that is defined by geofence and
Satisfies the given database table filter:
Every widget runs on its own table, and every table can have different available data range, geographic
extent and field structure.
Time period
Time period defines the start and end of the time that each record must fall between (time is a
mandatory field in the database) to be shown in the widget. Click on the watch icon to open the start
date picker:
Time period definition has two modes: absolute time and relative time (the latter is default).
Absolute time defines query in terms from and until of concrete datetime. Whenever user opens
the workspace, always the same time is used in the query.
Relative time defines the query in relation to the available date range in the database (not the
user time). For example, "last 15 minutes" is of the data in the database, not "since the time
user logged in". Every time new data is deployed to the database (this typically happens every
Reach-U 2021
7/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
15 minutes), queries that use "last X" most likely return the different result. The same happens
with queries that use "first X" after deletion of first partitions of data.
The actual data in the database typically does not start from full hour or exactly at 0:00. First and last
presets snap the query period to the nearest logical time unit (e.g. day, hour, quarter hour etc.)
The timezone is defined in the system configuration. This means that all data is converted to this
particular timezone, irrespective what the timezone of the data origin was.
The date picker provides relative time shortcuts that start counting from the oldest data in the database
or newest data in the database. All other dates outside of the available date range are greyed out and
not selectable.
Start date and time of absolute time can be edited also inline in the header. Hover mouse over date
area:
Time period length field supports following abbreviations: m (minutes), h (hours) and d (days).
The example above will give the information for March 8 05:25 to 05:40, and when user clicks on
highlighted "+1h" button, the query will be run for March 8 6:25 to 6:40.
Time period must be always given, for all widgets.
Geofence
Geofence defines the geographic area of the data that is shown in a widget, and is mandatory for all
widgets. Geofence can be defined and edited via geofence icon in the header of the widget.
Location (latitude and longitude) and time are mandatory fields, they are present on all records in the
database. All other fields can include NULL values.
Reach-U 2021
8/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
For new widgets, the geofence defaults to the geographic extent that includes approximately 85% of all
data in the database within the default time period. User can extend the size of the geofence.
Database field filters
Database field filter panel opens from the header (unlike geofence and time, this filter is optional).
For the thorough help on filtering, open the filter panel on any of widgets and click on help icon in the
header of the filter panel.
Apply to all
Filters have two modes: "Apply to this widget" and "Apply to all widgets". When "Apply to all widgets"
is selected, all changes in this widget propagate to all other widgets of the workspace.
Apply to all is indicated in the header with inverted button background:
Every widget can use different database table and every database table can have its own structure. In
the "apply to all" context:
If the database field name is the same, apply to all applies to other widgets. Note that the
technical field name in the database is compared, not the human-friendly name that can be
changed in Manage Fields dialog.
If the field name does not exist in the table that other widget(s) use, then the apply to all does
not apply on that widget.
1.4.2. Widget controls
Use dotted area in upper left corner to move the location of the widget:
Click on the widget name in upper left corner to rename the widget:
Grab from watch, filter or geofence symbol and drag to any other widget to apply the same criteria
(release anywhere inside the widget, the widget boundary turns green when the drop is accepted). Pay
attention to the Apply/Apply to all setting of the destination widget (this can be seen in the header) –
when a new filter is dragged to the "apply to all" widget, the change propagates to the whole workspace,
it is not limited to the destination widget. Drag-drop will not function if the field does not exist in the
table that the destination widget uses.
They work independently – to use exactly the same data (subset of records of the database) in other
widgets, drag each of three:
Reach-U 2021
9/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Drag a selection in the first column of the table and drop to any other widget to apply these field values
as a filter:
Reach-U 2021
10/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Hover the mouse over the field name in table, map or histograms for the tooltip that describes the
content of the field:
Hint for compare function: set time, geofence and filters for one widget, make a copy of the widget and
then change one or two components to have side-by-side comparison. Copy button is provided in upper
right corner of each widget:
1.4.3. Colours of fields in histograms and tables
Each field can have its own optional colour in system settings. When defined, this colour will be used
for the same field in tables and histograms. These colours are not user-selectable.
1.5. Supported browsers
User interface is tested to work with Chrome 87.0 and Firefox 83.0.
2 File Manager
File manager is a tool for saving and organising workspaces and other files in user private account.
2.1. Accessing and using the File Manager
Use the dropdown that opens when hovering over the workspace name. The new state uses workspace
name "My Workspace 1", user can change this with Save As function:
Reach-U 2021
11/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Save (and Save as...) store workspace definitions in the server under user account. Other users do not
have an access to them.
Workspaces are files that have specific internal structure that application can interpret when building
the user interface.
File manager window is re-sizeable and moveable and hovers above the active workspace, and user can
continue using the application at the same time:
Teet
File manager has two modes: tree mode and search mode, user can toggle these modes in the header
of the file manager. Sort and filter of items is shown below the search bar, they are applicable only in
search mode.
2.2. Functionality of the File Manager
In general, the functionality is very similar to what operating systems provide, with certain differences
that are described below:
Run (or double click) opens and runs the saved workspace. File Manager window is not closed
after hitting "Run". User can move the File Manager window to the side and use it for running
other workspaces.
Reach-U 2021
12/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
The workspace that is currently loaded is highlighted with dark background in the file tree.
The workspace that is under the mouse and where user-selectable commands apply is
highlighted in the file tree.
Workspaces and all other files can be organised into subfolders.
Items can be renamed, deleted and moved to another folder. Use drag and drop for moving.
The highest level folders (user home folder, Trash) cannot be renamed and deleted.
Unlike file managers, file names do not need to be unique within folder (file manager uses
internal unique ID), and user can drag and re-arrange the order of items within folder manually.
Deletion of file moves it to Trash. To delete permanently, delete the file from the Trash folder.
To put file back, select Restore in the Trash or drag the file back to the desired folder. Double
click in Trash works as Restore.
Search finds matching files (not folders).
Use three dots at the right end of each item. Note that different set of functions is applicable
for files, folders and the root folder:
Files can be renamed, downloaded, locked and deleted
Subfolders can be renamed and deleted, it is possible to upload new file and add new
subfolder
Root folder cannot be deleted or renamed, it is only possible to upload new file and add new
subfolder
Write protect function is useful for defining template workspaces that should not be overwritten
accidentally, these are indicated with "lock" on the icon behind the file name:
Write protected items cannot be saved, all other operations (delete, rename, move, save as,
etc.) are permitted.
Currently active workspace can be deleted by user (i.e. works differently than Windows file
system where open file is locked). When user deletes currently active workspace in File
manager, Save functionality for this workspace is not possible – use Save As.
In addition to workspaces, other files (smaller than 5MB) can also be uploaded to file manager.
Workspaces of other applications are also visible (depends on the platform configuration).
Use Download for sharing workspaces with other users. Workspace is downloaded as JSON file
that can be forwarded to other users or stored on shared directory. To use the workspace, other
users need to use Upload a file function in File manager.
How downloaded workspace JSON files are saved on disk depends on browser settings of the
user. Different browsers handle it differently, below is the screenshot of Chrome:
Reach-U 2021
13/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
2.3. Workspace definitions
Workspaces are JSON files that are editable outside of Map Explorer application. Experienced users can
use workspace JSON as an example and build their own workspace JSON with external tools.
Workspaces do not contain any data, only the definition of queries and how the query result is to be
shown in user interface.
3 Filter Panel
Any combination of filters on all fields is permitted.
When more than one fields have a criteria defined, AND operator is applied between them. For example,
the example below means RSRP <-120 AND RSRQ < -13:
When more than one value is defined in the same field, OR operator is applied. For example, the
example below means Triggering mode is eNB or MME:
User interface of filters depends on the type of the data – is it numeric or text, or more precisely, if the
data field includes qualitative or quantitative, discrete or continuous data.
Numeric data comes in three kinds:
Continuous. FLOAT and DOUBLE field types are used in Geo Insights to store this data.
Discrete. Data of this type should be stored in Geo Insights as LONG (ID) or CLASSIFIER (FLOAT
and DOUBLE technically also work, but not optimally). Each of them is presented in GUI
differently and which mathematical operations are available depend also on the database field
type.
Nominal. Data of this type is usually stored as LONG (ID) or sometimes CLASSIFIER.
3.1. FLOAT and DOUBLE field types
These fields support:
<, <=, >=, > operators
Reach-U 2021
14/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
For equals (=), enter the number into the search field (e.g. 12. Notation '=12' is also supported.
Due to the way floating numbers are handled by processors, equals may not work precisely,
except for =0, that works always correctly
For a range, use two dots, e.g. "1 .. 12" (this means 1<=x<=12)
Filter by NULL (that may be translated in GUI to "No information", depending on the field) is not
available.
To assist user with the range of the data, the hint is provided in the search field. The data range is
calculated over all data in the selected time period. For example, when the time period (set in the
header of the widget) is:
, then the available range is:
Whereas if the time period is changed to:
, available data range also changes to:
If the selected time period has only NULL values for given field, "No data" is displayed (some other time
periods may include data):
Range hint uses multipliers to shorten the display (k for 1000, M for 1,000,000 etc., also G/B
(giga/billion), T (tera), P (peta), E (exa) are supported). The shortening inevitably introduces rounding,
and therefore range hint is somewhat approximate (e.g. the max in data may be 3141,567.89, but the
range hint shows ".. 3.2k").
User can use these multipliers when entering the filter - i.e. instead of <42000, user can enter 42k.
3.2. CLASSIFIER and LIST fields
These fields open dropdown that presents all possible values. User can pick a value (or more than one),
or start typing:
Reach-U 2021
15/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
If more than one value is entered into the search box, OR operation is applied.
Semi-fuzzy search is used to suggest options that match the typed string: pre-completion works as
"starts with" on every word in the data and translation.
User is not bound to use only what the dropdown provides. There are two options:
Paste one or more values into search field. Pasting is supported also from external applications
(e.g. Excel).
Type the value and press enter. If this value does not exist in database, dropdown will not find
a match, but the filter can still be stored. Database content is constantly in change, maybe the
value will be there after next import.
Classifier and list field values can be translated to labels in the application layer in LTF files without
changing the database content. When translated, resting a mouse over the entry in the dropdown opens
tooltip that explains the meaning of the entry:
The order of selections in dropdown is determined by platform settings, with LTF files more frequently
used selections can be arranged to top, less frequently used to the bottom etc.
The logic of populating of the dropdown and how pre-completion works is:
1. All data that exists in the database table is presented to user (in the whole database table, it is not
time period dependent). Different widgets can use different database table that can have the same
field name, but the content can be different.
Reach-U 2021
16/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Dropdowns show the first 10 items that match the user typed string among first 2000 that match
this string in the database. If the field includes more than 2000 unique values typing in more
characters may be necessary to find the desired string
2. Before presenting to user, the system checks if the field in this database table is translated with help
of LTF files.
3. If it is, the database content is translated with the help of LTF files. If the LTF file is not complete
(i.e. the data includes actually more values that LTF file translates), then those values are also
presented to user.
In short – user has access to all data, but some values may be translated to human-friendly form.
Filter IS NULL is supported. "NULL" can be translated in different ways, this depends on LTF file and
platform configuration. User can always type into search field NULL, dropdown provides this option of
the field does not include too many values.
This can be overridden in platform settings, and NULL can carry different text per field (e.g. "Not
applicable" etc.).
3.3. LONG (ID) fields
When numbers carry nominal data content (for example, all fields that contain some sort of numeric
ID), they are stored as LONG in Geo Spotlight database and the following applies:
No range hint is given for these fields:
Mathematical comparisons <, > etc are not meaningful for ID-s and they are not supported. User
can enter (or paste) one or more value into the search field (as for classifiers).
Dropdown and fuzzy search are not available.
Reach-U 2021
17/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Filter by NULL (that may be translated in GUI to "No Information", depending on the field) is
available.
Filter by NOT NULL (!NULL) is also permitted entry.
Platform administrator can define aliases for LONG (ID) field. In this case @ symbol is shown at the end
of the filter field. Alias is a human-readable name for particular number. The sample below shows that
the alias of 3788203102664 is "Test". Both can be entered into the filter field. If user enters
"3788203102664", then the result is converted into alias ("Test" in the case below).
3.4. DateTime fields
Datetime fields are stored in database as LONG integers (UNIX timestamps) and defined as dateTime in
the GUI settings. These fields are presented in filter panel as time range selectors:
3.5. Latitude-longitude combo field
It is possible to combine latitude and longitude fields into combined filter field that defines a box round
the centre with given side length. This is useful when database contains more than one
latitude;longitude;time triplet. For databases with one triplet, filters in the header of each widget are
more ergonomic:
3.6. General
CTRL+ENTER (or CMD+Enter on Mac) is a shortcut for "Apply".
Reach-U 2021
18/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Arrow button adjacent to Apply button opens "Apply to all widgets" button.
"Show applied" shows only these fields that have a filter criteria set:
CLASSIFIER, LIST and LONG (ID) field can have many values in the filter (OR applies between them), for
example:
To copy one value, double-click on value and press CTRL+C (557009 is ready for copy in the image
below):
To copy all values, place a mouse cursor anywhere in the field and press CTRL+C.
4 Map Widget
4.1. Working with heatmap layers
Heatmap supports one layer of data at a time. Layer selector functionality depends on the field type in
the database (FLOAT/DOUBLE, LONG (ID), CLASSIFIER/LIST/BOOLEAN).
For each FLOAT/DOUBLE field, five aggregation types can be used:
Reach-U 2021
19/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Aggregation is performed on non-NULL values in the database (i.e. NULL is different value than 0 when
calculating AVG, MIN and MAX). Heatmap shows these aggregated values for each bin on the map. What
aggregators are available for which field is determined by system settings (for example, SUM is not
meaningful for signal strength and therefore can be configured not to be available).
If a field type is ID (stored as LONG number in the database), or CLASSIFIER/LIST, then only COUNT and
UNIQUE aggregations are available (as AVG, MIN, MAX are not applicable for these data types):
For COUNT, user must select at least one value for the field that is being mapped:
When more than one value is selected, OR operation is performed between them.
4.2. Filtering
The map is generated for subset of database that is within
the time period,
the geographic area that is defined by geofence and
satisfies the given filter:
Reach-U 2021
20/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Filter and heatmap layer are independent - filtering can be done on the same field or any other field (or
field combinations). For example, heatmap of Average Uplink Throughput for situations when RSRP<X
can be done.
4.3. Heatmap colour bar
Colour bar describes the content of the map and gives to user variety of methods to control how the
selected data is rendered on a map.
4.3.1. Heatmap bin sizes
The bottom row determines the heatmap resolution (bin size). Allowable resolution range depends on
how large area was used for heatmap calculation (refer to geofence explanation below). Larger areas
can be mapped at lower resolution, smaller areas in higher resolution. Changing the heatmap resolution
can change the heatmap - large map bin usually includes larger number of records, and therefore SUM,
COUNT will almost definitely go up. How MIN, MAX, AVG react on bin size change depends on the data.
Heatmap bins use Google S2 http://s2geometry.io geometry, not the national grid of the country. S2 is
global coordinate system, therefore inevitably the north direction and size of bins varies, depending on
the latitude and longitude. Therefore, heatmap resolution labels are approximate and can differ 1.5
times from the real dimension on the ground.
4.3.2. Filter by N in each heatmap bin
N is the number of records that was used when calculating the aggregated value for each bin. For
example, average of particular map bin can be 100, but this can be calculated from 2 or 200 or 2000
records. Average that is calculated from very small number can be misleading. For addressing this, N>=
filter is provided. For example,
less than 5 records in the database.
removes from the map all bins that were calculated from
tells how many records are inside the most populated map bin (this determines the height of
the highest bin in 3D heatmap, the bin that has one record has no height and the height of other bins is
determined proportionally between them). NMAX gives the maximum useful value for N>= filter.
Use 3D map to see both aggregated values and how many non-NULL values were used for aggregation
per map bin, as a multi-variate map. Higher peaks show where more data was available for calculating
Reach-U 2021
21/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
the heatmap, whereas colours of map show the result of the aggregation (Average of Session RSRPServing Cell in the case of screenshots below).
Filter by N effectively removes these "low" areas from the map (and "low" in this case means "not
sufficient data to calculate the aggregation reliably":
4.3.3. Colour bar
Colour bar explains and controls heatmap appearance:
Colour bar range behaviour for maps that use MIN, MAX or AVG aggregation is configurable in system
settings (and the configuration can be different for each field in the database):
Reach-U 2021
22/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Auto fit. If the data includes outliers, then heatmap colour bar applies auto fit method and
selects colours based on approximately 85% of data in the database (in the database, not on the
map! Map can be zoomed into certain area and shows aggregated values, whereas database
content is not dependent what is shown in map widget). All low values (outside of 85% band)
will be coloured with "minimum colour", and all high values with "maximum colour". When user
applies filters or changes time on map, colour bar is not changed. But when new data is ingested,
data range may change and this will automatically recalculate "auto fit" range.
When auto fit method is used, !-button appears at the right end of the colour bar. Hover the
mouse over the i-button to get the description of the data range.
Preset. The system is configured to use always the same range, no matter what the content of
the database is.
Colour bar range behaviour for maps that use COUNT, SUM or UNIQUE aggregation is different: it always
fits the colour range to the result of the query. When user applies any filter, or changes time, or changes
heatmap boundaries, or heatmap resolution, max value (NMAX) changes.
Colour bar shows also the range of aggregated values on the map between dotted lines.
The example above tells that the heatmap has values between -124...-66, but the colour bar is
configured to support values between -141...-44.
Sliders on the colour bar are provided to "knock off" heatmap above or below certain value. This can be
used to highlight "hotspots", or extremes. The example above shows that values that are larger than 77.1 are removed from the map.
The label under the right end of the colour bar tells what aggregation is applied on the map and the unit
for the field being mapped.
Reach-U 2021
23/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
4.3.4. Filter and colour bar interaction
Filter applies before aggregation (on raw input data rows), heatmap shows data after aggregation. Refer
to the example screenshot below:
Only these rows where Throughput<80M are used for generating heatmap.
On a heatmap, no bins have values larger than 40M (but as the histogram shows, there is still
9% of data between 40...80M, but those got averaged per map bin by other records).
User has filtered out from the map all bins that have value less than 40M, but this is again postaggregation filter. Heatmap may use some datapoints that have Throughput for example 80M,
provided that they fall into map bins that have average Throughput less than 30M.
tells from how many records in the database the heatmap was generated (that meet time,
geofence and filter criteria and are not NULL). Therefore, applying filter by N (
change the NSUM.
) does not
4.4. Geofence
Heatmap can be generated only for defined area (use geofence create and edit tools).
Reach-U 2021
24/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Alternatively, use "set geofence to match map window" function.
Geofence that is defined on the map can be applied also on other widgets (histograms, tables and also
other maps). For this, drag the geofence symbol in the heatmap widget to any other widget.
4.5. Cell symbols
This layer may be not functional, it depends on the database structure. This feature is being redesigned
and will change in next releases.
Use "Toggle cells" icon to turn on the layer that shows the celldata layer. Each cell sector is shown with
its azimuth, omnidirectional cells are shown as circles.
Reach-U 2021
25/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
To reduce clutter, when map is zoomed out, individual cells are hidden and map shows locations of
eNodeBs:
The styling (size and colour) of cell symbols is user-selectable from the icon that appears after hovering
on the cell layer toggle icon:
Cell symbol colours and size is per-user setting and applies to all map widgets and all workspaces of
user.
4.6. Two functions of the click on a map
1. Value of the heatmap bin
Click on a heatmap bin to get a precise reading of the post-aggregated value (that determines the
colour) and number of datapoints used during aggregation (that determines the height of 3D map).
Reach-U 2021
26/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Hint for compare function: create readings on a map, make a copy of the map widget and then change
time or filter or field that is being mapped.
Clicking on particular heatmap bin on 3D map may be obstructed by other high 3D bins. Rotate the map,
change the tilt angle or switch to 2D map:
2. Cell/eNodeB information
This layer may be not functional, it does not work if the database does not include cells/eNodeB-s. This
feature is being redesigned and will change in next releases.
Click on the cell layer button to show/hide eNodeB locations and cell sectors (omnicells are shown as
rings). Clicking on cell symbol opens the info window that shows eNodeB number and ECIs of all cells
within this eNodeB.
Reach-U 2021
27/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
4.7. Map layers
Map widget uses different layers:
The lowest layer is the base map. Available options are defined by system configuration. User can select
the base map style (or turn it off):
The next layer is heatmap. User can turn off heatmap and toggle the appearance between 2D and 3D
perspective:
The uppermost layer is reference cell symbol point layer, user can turn this layer on and off and change
the appearance (this function may not work if the database does not include cells).
Reach-U 2021
28/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
User can select 3D drawing that will be part of the base map, e.g. building floorplan.
If the GLTF file does not georeferenced (does not have geographic coordinates), user can define the
corner point and orientation of the drawing:
This is one-time task, this data will be stored along with the GLTF file in File Manager.
Very big GLTF layer can slow down the user interface significantly, this depends the complexity of the
model and user computer performance. When creating GLTF layers, simplicity is strongly suggested.
4.8. Further help
Refer to the general and database field filter help for explanation of functionality of map widget that is
common for all widget types.
Reach-U 2021
29/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
5 Map Algebra Widget
5.1. Work principle
Map algebra creates a new layer by applying a formula that takes input from existing map layer(s). Map
algebra works on raster maps, each layer must have the same resolution and pixels must align (they
must be "on top of each other").
Formula example is below:
5.2. Getting started
1. Click the Fx button to open algebra function.
2. Define map layers. Use + button to add new layers and garbage bin button to remove a particular
layer.
3. Enter the formula in Fx row and click Apply and show.
The above will make a heatmap of average bid requests per user.
5.3. More details
a) What is permitted on formula bar
Refer to:
https://github.com/silentmatt/expr-eval#unary-operators
Reach-U 2021
30/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
https://github.com/silentmatt/expr-eval#pre-defined-functions
https://github.com/silentmatt/expr-eval#operator-precedence
All these should work, although all is not tested thoroughly.
Use upper case for variables in formula and lower case for functions (e.g. max(A,B), not MAX(a,b). Error
handling and informing user is in infancy in the beta version.
b) How to define map layers
Each layer requires definition of field to be used, aggregation, filter and time like other widgets. Refer
to help of crosstable, filters and general help, these instructions are not repeated here.
To simplify definition, drag and drop of filters and time work between other widgets and inside the Fx
window also:
c) Inspect the formula
Click on the heatmap to understand how the result of each heatmap bin of map algebra result is
calculated.
The formula uses the last column ("Value)".
d) Eliminate some data from calculations
To ensure the representativeness of the result, it is possible to define limit for each map layer if the map
pixel can be calculated or not:
Reach-U 2021
31/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
"Value filter" makes it possible to eliminate 1/2=0.5, 10/20=0.5 etc from the map, but keep
100/200=0.5.
"filter by N" works similarly, but on raw records in the database. For example, if average is calculated
from 1-2 records, this average will be not used.
Eliminated pixel behaves similarly to non-existing map pixel – the resulting map will be empty.
e) Other reasons for empty map or map pixel
If formula includes an error (division by zero), map pixel will be empty.
If one of the heatmaps has empty pixel, the whole formula will be empty.
f)
Use "parameters/names" toggle to add descriptive names for each layer:
6 Table widget
6.1. The first column
The table defaults to view that shows how many records the database contains for each value in the
selected field in selected time period, filter and geofence.
Reach-U 2021
32/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
User can change the field that is being used for forming rows in the table (click on selector icon in the
header of the first column):
If the query response would generate the table with more than 500 rows, the table is clipped and
respective indication is shown:
Reach-U 2021
33/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
(The example above shows that 2 676 records in the database matched time, database field and
geofence filter, breakdown over sTMSI was requested and 1575 unique sTMSI values were found. The
table would include 1576 rows where 1 row is for NULL, but it is clipped and only first 500 are shown).
All rows are still available for sorting and copying to clipboard.
The second column shows the number of non-NULL records per each breakdown row, total row shows
how many records in the database match time, database field and geofence filter.
The selector for the first column of table includes only fields that contain discrete data (CLASSIFIER, LIST,
LONG(ID) database field type). For continuous data fields (FLOAT, DOUBLE), use histogram widgets.
If the query response would include more than 100,000 rows in the table, the query is halted, the error
message is shown and user needs to narrow selection criteria (make time period shorter or geofence
smaller or apply other filters).
6.2. Adding more columns
User can add more columns to the table by clicking on "Add column" icon.
Additional columns are different than the first column – float and id-type fields can be selected here.
Aggregation for the selected field is calculated for each row in the table. Aggregation options are MIN,
MAX, AVG, SUM, COUNT, UNIQUE, but some of them may not be available depending on the content
of the field (e.g. average on numeric field that contains identifiers has no analytical sense, and therefore
AVG, MIN, MAX, SUM are not possible for these).
The table includes aggregations, not raw data (individual records) from the database. NaN is presented
when all records that met the query filter criteria were NULL.
6.3. Hiding NULL values
The situation where data field contains NULL values is not uncommon. This can carry very valuable
information, but sometimes can be annoying. For example, if the field contains "City of residence", then
Reach-U 2021
34/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
for rural people the NULL is perfectly valid value. As a result, rural people may pop up in tables and
diagrams among first columns. To address this, user can hide NULL values from table (and crosstable).
This has impact also on the summary row of the table (showing 29 cities of residence and 30th line in
the table for "rural" that is stored as City==NULL. In this case "the table includes 30 rows" and "the table
includes 29 cities" and "the table includes 29 cities plus one for non-city" are all valid statements).
Hide Null values can hide a row from the table, but does not change the value of UNIQUE, because NULL
is not a number and not used when calculating the unique.
Important: "hide NULL" from the output and "filter by NOT NULL" are two very different things, both
are supported in Map Explorer. For example:
Use "handset model NOT NULL" into the filter and make table of throughput per cell: in this case
data from handsets that have no model name are not used in the throughput analysis (this may
not be a wise filter, because TAC can be known and the throughput data reliable).
Use "hide null from output" just hides the result from the output.
6.4. Using table to define filter for other widgets
User can select fields in the first column, and drag them to any other widget. These will become a filter
there. It is also possible to drag this filter to the same table widget to display only selected rows in the
table.
Drag to select a range, and CTRL+click to pick selection one by one.
Other widgets may have their own filters, they will be combined with new filter criteria as follows:
If the destination widget is using the filter on the same field, it will be overwritten
Reach-U 2021
35/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
If the destination widget is using filter(s) on other fields, they stay untouched.
6.5. Deep dive into UNIQUE in the table
If the table uses column with UNIQUE aggregation, the click on the table cell opens the "mini-table" that
lists all values that were used to calculate the unique.
If the field contains also NULL values they will be highlighted (but not counted as part of unique). This
is why the sample above shows 5 unique values, but mini-table shows 6 items - 5 rows for each value
and 6h for NULL.
Use "hide null" to eliminate "no information" item from the mini-table.
Reach-U 2021
36/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
The mini-table has the same feature set as the table widget: drag-drop to filter, copy to clipboard and
sort.
6.6. Other table controls
The header of each column gives variety of controls to the user. For accessing these controls, hover
mouse over column header:
Explanation of icons from left to right:
Settings icon to delete the column or change content/settings of the column. This includes also
number of decimals that user can change between 0 and 10. First two columns of the table
widget cannot be deleted.
Sort ascending/descending button to sort the table. Tables render max 500 rows, but sort works
on full data that the query from the database returned. To access all data, not only 500 rows,
use "Copy table to clipboard, paste to Excel" button in the upper right corner of the table.
Eye icon shows or hides colour bars. Colour bar of the largest value in the column is the longest,
and other values are proportionally shorter. Colour bars for positive values are left aligned, for
negative values right aligned
Left and right arrow buttons move the column in table. SHIFT+click moves the column to the
first or last position in the table.
Copy table to clipboard, paste to Excel function in upper right corner copies the all data behind the table
(not only top 500 rows that GUI widget shows).
If the table presents pointer symbol in front of the first column
, then these objects can be dragged
from table to map. Map will focus and zoom to the selected object.
Reach-U 2021
37/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
6.7. Further help
Refer to the general and database field filter help for explanation of functionality of table widget that is
common for all widget types.
7 Crosstable Widget
Crosstable permits any LIST/CLASSIFIER or LONG guiType:ID field for rows and columns, and userselectable aggregation for calculating the value for each table cell.
Explanation for usage of NaN and - based on first two cells in the example table above:
1. As the first step time, field and geofence filters are applied:
Reach-U 2021
38/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
2. After filters, no records in the database contained ECI=142600734 and Band Class=EUDD.
Therefore, table cell shows "-".
3. The database contained rows where ECI=142606868 and Band Class=EUDD, but after filters no
records were found where RSRP was not NULL, therefore aggregation value could not be computed
and the table cell has "NaN".
Aggregation field selector gives two filters:
Filter by N (works similarly to heatmap N filter). When applied, value is shown in table only when
more than N records are found in the database for calculating the aggregation in the table cell
VALUE FILTER can be used to eliminate too small or large values from the table, this applies on
aggregated values.
In other words: filter by N is applied before the table content is calculated, value filter is applied after
calculation and before rendering the table widget. User can paste value from table to value filter. In
addition, range filter is also supported.
Crosstable cells that contained value that N or VALUE filter removed, are shown as "-". If the N or VALUE
filter remove all values in crosstable row or column, the whole column/row will be removed. This means
that "empty" columns and rows are not shown.
7.1. Hiding NULL values
The situation where data field contains NULL values is not uncommon. This can carry very valuable
information, but sometimes can be annoying. For example, if the field contains "City of residence", then
for rural people the NULL is perfectly valid value. As a result, rural people may pop up in crosstable (and
other widgets) tables and diagrams among first columns:
Reach-U 2021
39/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
To address this, user can hide NULL values from crosstable (and table).
With this setting, origin-destination matrix will be much more readable.
Hide Null values works both on rows and columns on of the crosstable.
Important: "Hide NULL" from the output and "Filter by NOT NULL" are two very different things, both
are supported in Map Explorer. For example:
Use "Handset model NOT NULL" into the filter and make table of throughput per cell: in this
case data from handsets that have no model name are not used in the throughput analysis (this
may not be a wise filter, because TAC can be known and the throughput data reliable).
Use "Hide null from output" just hides the result from the output.
Reach-U 2021
40/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Crosstable can be sorted: click on a row or column header. Click once more to toggle between ascending
and descending order. The row/column that is used for sort is shown in blue:
Numerical values are sorted first and then NaN and "-" cells follow. There is no sorting inside NaN and
"-" groups.
Cells in cross table are coloured from the largest to smaller. The largest value is in the darkest
background. Depending on how the table is sorted, dark cells may not fit into the widget (they are
behind the corner, user may need to scroll the table or increase the widget size).
Values in crosstable are rounded. For the precise reading, click on the table cell. Copy and paste to Excel
gives also precise numbers.
Depending on the number of unique values in fields that are used for crosstable, the query response
can potentially include huge number of values. To protect the server, maximum limit of 100 000 values
in the response is applied (i.e. the maximum size can be 100x1000 or 315x315 or 10x10000 etc).
Furthermore, in GUI, the crosstable widget size is limited to 150 rows and 150 columns. For example,
the server response may the array of 50 columns and 300 rows, in this case all columns are displayed,
but only first 150 rows. The sorting, however, works on full data array, not within 150x150 limit.
When the limit of 150 is exceeded, indicator is shown behind the field name:
Reach-U 2021
41/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
If a column in crosstable has empty values for all rows, the column will not be shown in the table. The
same applies to rows that have empty values in all columns.
Copy table to clipboard, paste to Excel function in upper right corner copies the all data behind the
crosstable (not only top 150 rows/columns that GUI widget shows).
7.2. Deep dive into UNIQUE in the crosstable
If the crosstable uses column with UNIQUE aggregation, the click on the table cell opens the "mini-table"
that lists all values that were used to calculate the unique.
If the field contains also NULL values, these will be highlighted (but not counted as part of unique). Use
"hide null" to eliminate "no information" item from the mini-table.
Reach-U 2021
42/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
The mini-table has the same feature set as the table widget: drag-drop to filter, copy to clipboard, and
sort are available.
7.3. Further help
Refer to the general and database field filter help chapters for explanation of functionality of crosstable
widget that is common for all widget types.
8 Histogram widget
8.1. Selecting field for the histogram
Histogram supports FLOAT and DOUBLE fields only. Click on selector icon under the X axis of the
histogram to open the field picker:
Histogram uses two Y axis:
Left Y axis shows the range for columns on the histogram and shows COUNT of records in
particular histogram bin.
Right Y axis shows the range for the cumulative line (and is therefore always from 0 to 1).
Reach-U 2021
43/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
8.2. Histogram controls
Use mouse to get a reading (height) of particular bar or cumulative value (the sample below tells that
80% of records have value less than 25.5M).
Click on the widget preserves the location of the crosshair (this is useful for print screen).
Histogram X axis is user-selectable:
Preset uses the same min-max setting that is used for limiting the range of heatmap colour bar.
This is shown only when preset min-max are defined for given field in system settings (user
cannot change these settings).
Global fits to all data in database. If the database has even a single outlier record, the histogram
will be condensed as shown below:
Auto attempts to fit 85% of the data on the X axis. Hover the mouse over !-button to see how
many datapoints are cut off:
In addition, Auto feature determines the number of bins that is appropriate for given field based
on how many unique values does the field have in the database. Without this, the following
could happen:
Reach-U 2021
44/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
The illustration above shows the histogram with 50 bins of data that has only 39 unique values
in the database. This introduces gaps in the visualization that actually do not exist in data. Auto
feature avoids this happening.
Behind the X axis label is shown the following information:
The number of records that have non-NULL value in the query response in the field that is being
plotted on histogram – i.e. based on how many datapoints the histogram was drawn.
The number of records that meet the query criteria.
Percentage of non-NULL values.
In the example above, on 23.6% of records RSRP data was populated and NULL for 66.4% of records.
8.3. Further help
Refer to the general and database field filter help for explanation of functionality of histogram widget
that is common for all widget types.
9 2D Histogram widget
9.1. Selecting fields
Click on selector icon in front of the X or Y axis label (only FLOAT and DOUBLE fields are available for 2D
histogram):
Reach-U 2021
45/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
2D Histogram shows the COUNT of non-NULL records in the database in each 2D histogram bin.
9.2. 2D Histogram controls
Use mouse to get a reading of the number of records in particular 2D bin. Click the crosshair to preserve
its location (this is useful for print screen).
2D Histogram axis are user-selectable:
Preset uses the same min-max setting that is used for limiting the range of heatmap colour bar.
This is shown only when preset min-max are defined for both fields in system settings (user
cannot change these settings).
Global fits to all data in database. Should the database have even a single outlier record, the
histogram will be condensed as shown below:
Reach-U 2021
46/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Auto attempts to fit 85% of the data on the axis. Hover the mouse over !-button to see how
many datapoints are cut off:
In addition, Auto feature determines the number of bins that is appropriate for given field based on
how many unique values does the field have in the database. Without this, the following could happen.
Behind the X and Y axis label is shown the following information:
The number of records that have non-NULL value in the query response in the field that is being
plotted on histogram axis.
The number of records that meet the query criteria.
Percentage of non-NULL values.
Note that for being plotted on 2D histogram, BOTH X and Y field must have non-NULL value.
9.3. Further help
Refer to the general and database field filter help for explanation of functionality of histogram widget
that is common for all widget types.
Reach-U 2021
47/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
10 Data types in Sparrow LDB4 database
Sparrow LDB4 is a database engine that exposes data over API. Northbound application(s) add their own
capabilities/restrictions that is important to keep in mind when deciding the data type for each column.
Sparrow LDB4 database supports following data types:
FLOAT
DOUBLE
LONG (that can store various content, e.g. nominal numbers or UNIX timestamps)
CLASSIFIER (can be also called “varchar”)
LIST (data type that can store more than one classifier value in one field (N-M relation))
BOOLEAN
All data types above support also NULL.
In API, CLASSIFIER, LIST, and BOOLEAN behave the same (the same set of features is exposed. Therefore,
metadata API endpoints and this document refers to all these three data types as CLASSIFIER).
Note that Sparrow LDB4 API has more capabilities than currently exposed in Map Explorer or any other
northbound application.
10.1. Handling of NULL by database engine and API
For FLOAT, DOUBLE, LONG fields:
Storing NULL and 0 are both supported.
Number of NULL values in response can be obtained with lazy count minus count operation.
When aggregation is done on only NULL values, “NaN” is returned (i.e. average of always NULL
values is not 0, but “NaN”).
For CLASSIFIER fields
Much more flexibility is available in ETL. In addition to NULL, “undefined”, “missing”, “N/A” can
be used, depending on the context.
In breakdowns, NULL is one of values in the response (shown in sample below as “not defined”,
but this can be changed with LTF files).
Reach-U 2021
48/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
IS NULL and IS NOT NULL can be used as filter criteria for CLASSIFIER, LIST, BOOLEAN and LONG
guiType:ID fields.
10.2. Supported aggregations in API
Sparrow LDB4 supports 6 aggregations, but not all are available for all data types:
Aggregation
CLASSIFIER, LIST,
BOOLEAN
FLOAT, DOUBLE
LONG
LONG
guiType: ID
guiType: dateTime
MIN, MAX, AVG
-
yes
-
-
COUNT
yes
yes
yes
-
SUM
-
yes
-
-
UNIQUE
yes
-
yes
-
10.3. Supported operations in API
Lazy count (returns count of rows in database that meets filter criteria)
Breakdown for CLASSIFIER, LIST (includes count for NULL as one item in the response)
Histogram for FLOAT and DOUBLE (null values are ignored)
Histogram and breakdowns can be multi-dimensional and mixed (e.g. first dimension is breakdown over
CLASSIFIER and second dimension is any aggregation for FLOAT) etc.
10.4. Extending data types for Map Explorer GUI
Map Explorer application uses LDB4 data type to decide what can be done with particular field. In
addition, in Map Explorer configuration certain modifiers can be used to determine how field is handled
and fields can be also grouped:
FLOAT, DOUBLE
LONG guiType ID
LONG guiType dateTime
CLASSIFIER (that includes also LIST and BOOLEAN)
LOCATION (that combines two float fields containing latitude and longitude)
11 Data Ingestion Layer Configurability
11.1. Lookup-Translation (LTF) files
LTF files are a powerful tool to convert enumerated data into classifiers in data, and provide userfriendly text and tooltips in graphical user interface. The same file is used in ETL layer (on Node 1) and
GUI layer (on Node 2). ETL Orchestrator module ensures that LTF files on both nodes are in sync.
Reach-U 2021
49/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
11.2. LTF file function in a nutshell
The function explained based on the example:
ASR
1
RDB
SUCCESS
Label
Successful record
Tooltip
This record has all data correct
In ETL layer:
If the input data has value 1 (NB! numeric value 1), then in Geo Insights database it will be stored as
SUCCESS (NB! Not a numeric value any more, but CLASSIFIER).
In GUI layer:
SUCCESS is never shown in the user interface, it is converted to text “Successful record”. This is used in
dropdowns, filter fuzzy search and tables.
When user rests mouse over label “Successful record”, tooltip is displayed that reads “This record has
all data correct”.
11.3. More details of LTF file
ASR
1
4
2
8
3
5
NULL
RDB
SUCCESS
CLASS_4
FAIL
FAIL
MISSING
UNKNOWN
NULL
Label
Successful record
Second best
Failed record 1
Failed record 2
Missing from data
Unknown
Not applicable
Tooltip
This record has all data correct
Demonstration only
Something completely failed somewhere
Something failed somewhere 2
The value was not here \nbut should have been here
We do not know the value
Not applicable for this field
The order of rows in LTF file determines the order of items in dropdown (NB! Not in table!). This
is useful to show most frequently used options the first in the list
Note that NULL is a special row in LTF file. First, as LTF file describes how numbers in input get
converted into RDB and LDB4, then “NULL” may not be in input at all. Some other number (like
MININT or 999 or any other specified value) may be used, this is defined in input data
specification.
The row that describes NULL must have “NOT_DEFINED” in the RDB column, because LDB4 API
returns the string “NOT_DEFINED” for NULL values.
Handling NULL in LTF file is not mandatory. If not given, the GUI reverts to the string that is
defined in application language file.
To make long text in tooltips more readable, use \n as line break symbol.
Values in ASR columns must be unique (non-unique would mean that two methods of mapping
for the same value is provided, this is not a good practice)
If more than one row in RDB column of LTF has the same value, this introduces mapping several
input values into one database value. Map Explorer GUI uses label and the tooltip from the last
Reach-U 2021
50/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
line in LTF file of RDB value group. Other applications that use Sparrow API may behave
differently.
Refer to the example above: both values 2 and 8 in input are converted into value “FAIL” in RDB.
User interface uses “Failed record 2” in filters and breakdowns, and “Something failed
somewhere 2” as a tooltip.
LTF must be tab separated text file with .tsv extension
11.4. Handling of values that are NOT described in LTF file
Refer to the example above. If the real data includes value that is not 1..5, 8 or NULL, for example 99,
the following happens:
Number 99 is converted to string “99” and inserted into the database
Data Quality issue is reported to Elasticsearch. Number of records that have value 99 is available
there to assess the severity of the impact
Breakdown queries use “99” normally, as it handles values “SUCCESS”, “FAIL” etc.
Dropdown in GUI shows “99” among other options that are described in LTF file.
When LTF file is improved by adding row for describing the mapping for input value 99, and a
key is introduced in RDB column (for example “NEW_KEY”), then starting from the next data
ingestion, 99 gets converted to this key. Old data is not recalculated and stores “99”, new data
stores “NEW_KEY”). This needs to be kept in mind when analyzing RDB file content and running
queries in LDB4 that span across data partitions that use “old LTF file” and “new LTF file”.
11.5. Using LTF files
LTF file is optional feature for CLASSIFIER and LIST type fields in LDB4. LTF file must have the same name
as the field in RDB schema (e.g. myfield.tsv).
When LTF file is missing for given field, numbers in input get converted to strings (e.g. 1 to “1”). IF LTF
file is available for given field, 1 gets converted to SUCCESS (refer to the example above).
LTF files apply on all fields that ETL creates and writes into RDB. It is not important that the field exists
in input data feed. This means that also synthetic (new) fields that ETL creates are also driven by LTF
files.
Changing ETL code logic is a product code change, but LTF files are configuration files that can be
changed per deployment without interrupting the service, no downtime is required. Refer to ETL
documentation for instructions.
Note that changing the value in RDB column will introduce backwards incompatibility in data, because
the value in RDB column is stored in the Sparrow LDB4 database (not the value in ASR column). Old data
in database will not be recalculated when value mapping in LTF file is changed.
11.6. Dropdowns in Map Explorer GUI
When particular field does not have the LTF file, filter dropdown presents to user the selection of
available values in the database, but user can enter/paste into search criteria via dropdown a value that
Reach-U 2021
51/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
does not exist in data (this will subsequently return 0 records, but this is by design. After next data
ingestion, this filter may already return records).
When particular field does have the LTF file, filter dropdown presents to user only these rows in LTF file
that exist in the data, in the order of LTF file. This means that user cannot enter into search criteria via
dropdown a value that does not exist in data.
When particular field does have the LTF file, and the real data includes values that in are not specified
in LTF file, filter dropdown presents to user all values in Label column in LTF file plus untranslated real
data. This means that user can still access all data despite the fact that some values is not described in
LTF.
11.7. Bitmap
LDB4 does not support bitmap data type, but LIST type implements most features of the bitmap. For
bitmaps, LTF file has slightly different structure:
BIT
NONE_SET
1
2
NULL
RDB
NONE
SUCCESS
BIT_2
NULL
Label
No bit set
Successful record
Second best
Not applicable
Tooltip
No bit is set in bitmap
This record has all data correct
Demonstration only
Not applicable here
Note that numbering of bits starts from 1 (not from 0), and 1 is the least significant bit.
The keyword “NONE_SET” is introduced to capture the situation where all bits are 0.
Explanation of the example above: if the input has binary value 11 (or decimal 3), then LIST type field in
database will carry values “SUCCESS” and “BIT_2”.
11.8. Unsupported cases
If more than one row in ASR/BIT column of LTF has the same value, the last row in LTF file will
be used by ETL, Map Explorer GUI behavior is not predictable.
LTF files are designed to convert enumerated numbers (integers) into Classifiers/List in database
and human-readable text in GUI, however it should work also when the input is of string type.
This is not tested feature.
12 User Interface Configurability
Items below are not necessary for storing data in LDB4 but representing data that applications fetch
from LDB4. Each application can have its own settings, this chapter describes settings for Map Explorer.
12.1. Field names, field units, groups and group order
It is common that most frequently used fields pop up first, without excessive scrolling and fields with
similar meaning are grouped together etc.
Reach-U 2021
52/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Map Explorer provides GUI for managing this. The GUI is available for platform users with “Data
Manager” authority. Changes take effect immediately.
12.2. Color settings for heatmap
Every field that can be plotted on map needs to use some default color. These are described in the
platform configuration.
Color configuration string example:
0:AA300000;0.5:AAfcc143;0.6:AAfeff04;0.7:AA96ff59;0.8:AA54fe00;
0.8:AA00c600;0.9:AA00c600;1:AA008200
This string works follows:
The leftmost end of color scale uses color #30000, as it is behind “0:”
The rightmost end of color scale uses color #008200, as it is behind “1:”
The middle of the color scale uses color #008200, as it is behind “0.5:”
Reach-U 2021
53/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
The left half of the color scale changes color gradually from #30000 to #008200, refer to
substring “0:AA300000;0.5:AAfcc143”.
The right half of the color scale gradual color scale between 0.5 to 0.6, from 0.6 to 0.7, from 0.7
to 0.8 and from 0.9 to 1.0.
Section 0.8:AA54fe00; 0.8:AA00c600;0.9:AA00c600 introduces the sudden jump of color at 0.8,
and the color stays the same between 0.8 and 0.9.
AA in front of each color scale defines the alpha (transparency). Hex values between 00 and FF
are permitted. Transparency is currently ignored by Map Explorer GUI, will be supported in
future versions as follows: 2D heatmaps use alpha for each color inflection point, 3D heatmaps
use the alpha of the first color definition.
The color string defines colors between 0..1 relative scale. The real data, however, has much different
real scale. The logic is that min and max value is found for all heatmap bins in map widget. For minimum
left end is used, for maximum the right end is used. This means that color scale is “stretched to fit the
data” each time the heatmap is rendered. This means that actual values on heatmap and colors of
heatmap can shift every time heatmap is recalculated.
This is sometimes inevitable, sometimes not desired.
Example for not desired change of colors are RSRP and throughput. It is understandable that user wants
heatmap to use always the same color for the same value. For example, RSRP<120 is “weak” and should
be red, no matter what filters or location is used on the map. The same applies for throughput.
To achieve this, two configuration parameters can be used to fix relative scale to real scale:
"globalMin": -21,
"globalMax": -2,
The result will be:
No matter what the query on heatmap is, -9.5 is always shown yellow in the example above.
globalMin and globalMax make colors consistent, but have some inevitable limitations/downsides:
Data may contain outliers and in this case outliers will go “off the scale” and will not be visible
on a map. It may be very difficult to find globalMin and globalMax that work both for normal
data and at the same time for outliers.
Determining globalMin and globalMax for each data field is usually manual data analysis work,
and entering these values for hundreds of fields of database takes time.
globalMin and globalMax are not applicable for SUM, COUNT and UNIQUE aggregations, they
are applicable only for AVG, MIN, MAX.
Note that heatmap shows aggregated values per map bin, not the raw original values in database (unlike
histograms that have no concept of aggregation).
Reach-U 2021
54/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Each aggregation for the same field can have its own color string (it does not make usually sense to use
different color scale for AVG, MIN and MAX, but configuration does not make any attempt to forbid
this):
colorStrings": {
"AVG": "0:AAE6F7FF;1.0:AA006600",
"MIN": "0:AAE6F7FF;1.0:AA006600",
"MAX": "0:AAE6F7FF;1.0:AA006600",
"COUNT": "0:AAE68A00;1.0:AAE6E600",
"SUM": "0:AAE6E600;1.0:AAE68A00",
"UNIQUE": "0:AAE68A00;1.0:AAE6E600"
}
Only SUM is applicable for heatmaps of CLASSIFIER and LIST type fields.
The colorString list above can be different for each field in database. Hundreds of fields times 6
aggregations gives over 1000 different colors – much more than human eye can cope with. Therefore
platform configurators need to take decisions – what really deserves its own color and what does not.
Depending on field type, the right side of the color bar can mean “better” or “worse”, this determines
the choice of colors, which in turn depends also on the culture (e.g. is red used usually to carry the
meaning of “good” or “bad”).
12.3. Color settings for tables and 1D histograms
For each field, home color can be defined:
"colorStrings": {
"SUM": "0:AAece7f2;1:AA2b8cbe",
"COUNT": "0:AAe5f5e0;1:AA31a354",
"home": "008200"
}
This determines the color that is used for given field in Table, 1D histogram, Crosstable (and 2D
histogram). Heatmaps do not support home color.
There is only one home color, this cannot be defined per on aggregation like on heatmap (although
table supports aggregations). The purpose of home color is to tie together histograms and tables, but
for histograms aggregation concept is not applicable. This is why the table uses also only one color per
field, for all aggregations.
Reach-U 2021
55/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
Home color does not support transparency, as there is no background that needs to be seen through
(but heatmaps need to let the base map to visible).
For 2D histogram widgets, the color of the field on X axis is used.
Crosstable widget uses home color (not colors scale per aggregation).
12.4. Default and available aggregations
Sparrow LDB4 supports 6 aggregations, but not all are available for all data types, refer to 10.2
Supported aggregations in API). Data type (CLASSIFIER, LIST; FLOAT, LONG etc) is defined in database
schema and cannot be changed in configuration, but how LONG is interpreted in GUI can be configured.
Supported options are:
"guiType": "dateTime"
"guiType": "ID"
dateTime fields can be used in filters, and GUI applies time period selector:
dateTime fields cannot be used in tables, histograms nor heatmap.
ID fields can be used everywhere. In filtes, >, < etc operators are not permitted.
Only permitted aggregations for given field are shown in table and heatmap controls:
What is shown as a default selection in dropdown is defined here:
"aggregation": "avg",
The default for guiType:ID fields is UNIQUE, and for float fields is AVG. If you not want to change these,
setting “aggregation” for each field is not necessary.
Sometimes the data type is still not sufficient to determine what aggregations make sense and what do
not, for example SUM of RSRP in map bin or for eNodeB make no sense. This can be driven with:
"aggregationsOverride": ["MIN", "MAX", "AVG", "COUNT"],
Reach-U 2021
56/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
12.5. Default settings
To avoid configuring the same value for all fields, “defaults” object is provided.
"defaults": {
"extraInfoMap": {
"colorStrings": {
"COUNT": "0:AAf1f0f8;1:AA756bb1",
"UNIQUE": "0:AADDEBF7;1:AA034e7b",
"AVG": "0:AA300000;1:AA008200"
},
"baseColor": "ff0000"
}
}
The priority order is:
“attributeID”
“defaults”
When both are missing, system-wide hardcoded defaults are used. This means the platform
works also without any configuration, although the appearance may not be optimal.
12.6. Default screen configuration
When user creates a new workspace:
The default layout is applied. Layout includes what widgets are shown, their sizes and locations. The
default workspace does not need to expose all capabilities of Map Explorer.
For each widget, the default field(s) are defined (e.g. what is the default layer on map, field used for
making histogram etc). These are defined also for widgets that are not shown in the default workspace.
Reach-U 2021
57/58
Geo Spotlight Map Explorer Pro Help
Copy of online help version 8.1
All settings of the default workspace are configurable.
12.7. Multitable support
Map Explorer gives access to all LDB4 tables that are deployed in the system. Each LDB4 table can have
a different structure and therefore all settings described in this chapter can be defined per LDB4 table.
Reach-U 2021
58/58