Juniper Secure Analytics
Reference Data Collections
Release 2014.5
Juniper Networks, Inc.
1194 North Mathilda Avenue
Sunnyvale, CA 94089
USA
408-745-2000
www.juniper.net
Published: 2016-02-04
Copyright Notice
Copyright © 2016 Juniper Networks, Inc. All rights reserved.
Juniper Networks, Junos, Steel-Belted Radius, NetScreen, and ScreenOS are registered trademarks of Juniper Networks, Inc. in the United States and
other countries. The Juniper Networks Logo, the Junos logo, and JunosE are trademarks of Juniper Networks, Inc.
The following terms are trademarks or registered trademarks of other companies:
JavaTM and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.
All other trademarks, service marks, registered trademarks, or registered service marks are the property of their respective owners.
All specifications are subject to change without notice. Juniper Networks assumes no responsibility for any inaccuracies in this document or for any
obligation to update information in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication
without notice.
FCC Statement
The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A
digital device, pursuant to part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the
equipment is operated in a commercial environment. The equipment generates, uses, and can radiate radio-frequency energy and, if not installed and
used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential
area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense. The following
information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it
is not installed in accordance with Juniper Networks’ installation instructions, it may cause interference with radio and television reception. This
equipment has been tested and found to comply with the limits for a Class B digital device in accordance with the specifications in part 15 of the FCC
rules. These specifications are designed to provide reasonable protection against such interference in a residential installation. However, there is no
guarantee that interference will not occur in a particular installation. If this equipment does cause harmful interference to radio or television reception,
which can be determined by turning the equipment off and on, the user is encouraged to try to correct the interference by one or more of the following
measures: Reorient or relocate the receiving antenna. Increase the separation between the equipment and receiver. Consult the dealer or an
experienced radio/TV technician for help. Connect the equipment to an outlet on a circuit different from that to which the receiver is connected.
Caution: Changes or modifications to this product could void the user's warranty and authority to operate this device.
Disclaimer
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT, SUBJECT TO THE MODIFICTAIONS SET FORTH
BELOW ON THIS PAGE, ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED
HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR
JUNIPER NETWORKS REPRESENTATIVE FOR A COPY.
Reference Data Collections
Release 2014.5
Copyright © 2016, Juniper Networks, Inc.
All rights reserved. Printed in USA.
Revision History
December 2016 — Reference Data Collections
The information in this document is current as of the date listed in the revision history.
END USER LICENSE AGREEMENT
The Juniper Networks product that is the subject of this technical documentation consists of (or is intended for use with) Juniper Networks software. Use
of such software is subject to the terms and conditions of the End User License Agreement (“EULA”) posted at http://www.juniper.net/support/eula.html,
as modified by the following text, which shall be treated under the EULA as an Entitlement Document taking precedence over any conflicting provisions
of such EULA as regards such software:
As regards software accompanying the STRM products (the “Program”), such software contains software licensed by Q1 Labs and is further
accompanied by third-party software that is described in the applicable documentation or materials provided by Juniper Networks.
2
For the convenience of Licensee, the Program may be accompanied by a third party operating system. The operating system is not part of the Program,
and is licensed directly by the operating system provider (e.g., Red Hat Inc., Novell Inc., etc.) to Licensee. Neither Juniper Networks nor Q1 Labs is a
party to the license between Licensee and the third party operating system provider, and the Program includes the third party operating system “AS IS”,
without representation or warranty, express or implied, including any implied warranty of merchantability, fitness for a particular purpose or
non-infringement. For an installed Red Hat operating system, see the license file: /usr/share/doc/redhat-release-server-6Server/EULA.
By downloading, installing or using such software, you agree to the terms and conditions of that EULA as so modified.
3
4
CONTENTS
1
REFERENCE DATA COLLECTIONS
About Reference Data Collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Creating a Reference Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Utility Command Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1
REFERENCE DATA COLLECTIONS
Use the ReferenceDataUtil.sh utility to make complex reference data collections,
such as a Reference Map, Reference Map of Sets, and Reference Map of Maps.
This technical note applies to Juniper Secure Analytics (JSA) and Log Manager.
About Reference
Data Collections
Reference data collections enable the storage, retrieval and testing of complex
data structures. You can create the following reference data collection types:
•
Reference Map - In a Reference Map, data is stored in records that map a key
to a value. For example, to correlate user activity on your network, you can
create a reference map that uses the Username parameter as a key and the
user’s global ID as a value.
•
Reference Map of Sets - In a Reference Map of Sets, data is stored in records
that map a key to multiple values. For example, to test for authorized access to
a patent, you can create a Map of Sets that uses a custom event property for
Patent ID as the key and the Username parameter as the value to populate a
list of authorized users.
•
Reference Map of Maps - In a Reference Map of Maps, data is stored in
records that map one key to another key, which is then mapped to single value.
For example, to test for network bandwidth violations, you can create a Map of
Maps that uses the Source IP parameter as the first key, the Application
parameter as the second key, and the Total Bytes parameter as the value.
For each reference data collection, you specify what the element type is for the
collection. You can specify the following element types:
•
Alphanumeric
•
Alphanumeric Ignore Case
•
IP
•
Numeric
•
Port
•
Date
Reference Data Collections
8
REFERENCE DATA COLLECTIONS
Creating a
Reference Data
Collection
Using the ReferenceDataUtil.sh utility, you can create a reference data collection.
Before you begin
If you plan to load an external file containing data elements, ensure that the file is
in Comma Separated Value (CSV) format. Also ensure that you have copied the
file to your JSA system.
The file must follow the format in the following examples reference data
collections:
Example 1
#
# ReferenceMap
#
key1,data
key1,value1
key2,value2
Example 2
#
# ReferenceMapOfSets
#
key1,data
key1,value1
key1,value2
Example 3
#
# ReferenceMapOfMaps
#
key1,key2,data
map1,key1,value1
map1,key2,value2
The # symbol in the first column indicates a comment line.The first non-comment
line is the column header and identifies the column name (ie., key1, key2, data).
Then each non-commented line after that is a data record that gets added to the
map. Keys are alphanumeric strings.
About this task
See Utility Command Reference for a list of all commands and parameters you
can use to manage your reference data collections. You can also type
./ReferenceDataUtil.sh and press Enter to access a list these commands
Reference Data Collections
Creating a Reference Data Collection
9
Procedure
Step 1 Using SSH, log in to JSA as the root user:
Username: root
Password: <password>
Step 2 Create a reference data collection:
a
To change to the /opt/qradar/bin directory, type the following command:
cd /opt/qradar/bin
b
To create the reference data collection, type the following command:
./ReferenceDataUtil.sh create <name> <count> [MAP | MAPofSETS
| MAPofMAPS] [ALN | NUM | IP | PORT | ALNic | DATE]
[timeout_type] [timeToLive]
Step 3 To populate the map with data from an external file, type the following command:
./ReferenceDataUtil.sh load <name> <filename> [-encoding=...]
[-sdf=" ... "]
What to do next
Log in to the JSA user interface to create rules that add data to your reference data
collections or rule tests that detect activity from elements in your reference data
collection. For more information on creating rules and rule tests, see the Juniper
Secure Analytics Users Guide.
Reference Data Collections
10
REFERENCE DATA COLLECTIONS
Utility Command
Reference
Use the following commands to manage your reference data collections:
Table 1-1 Command Reference
Command
Parameters
create
<name> is the name of the reference data collection.
<count> is the maximum number of elements that the reference
data collection can contain.
[timeout_type] specifies whether the timeToLive is from the time
the element was inserted (0) or last seen (1).
[timeToLive] specifies the amount of time reference data
collection elements remain in the collection.
[MAP | MAPofSETS | MAPofMAPS] specifies the type of
reference data collection.
[ALN | NUM | IP | PORT | DATE] is the type of data in the
reference set, where:
update
•
ALN specifies a reference data collection of alphanumeric
values. This data type supports IPv4 and IPv6 addresses.
•
ALNic specifies a reference data collection of alphanumeric
values but tests ignore the case. This data type supports IPv4
and IPv6 addresses.
•
NUM specifies a reference data collection of numeric values.
•
IP specifies a reference data collection of IP addresses. This
data type supports only IPv4 address.
•
PORT specifies a reference data collection of PORT
addresses.
•
DATE specifies a reference data collection of DATE values.
<name> is the name of the reference data collection.
<count> is the maximum number of elements that the reference
data collection can contain.
[timeout_type] specifies whether the timeToLive is from the time
the element was inserted (0) or last seen (1).
[timeToLive] specifies the amount of time reference data
collection elements remain in the collection.
add
<name> is the name of the reference data collection.
<value key1 [key2]> specifies the values you want to add.
Key1 is required for MAP, MAPofSETS and key2 is required
for MAPofMAPS. Keys are alphanumeric strings.
Reference Data Collections
Utility Command Reference
11
Table 1-1 Command Reference
Command
Parameters
delete
<name> is the name of the reference data collection
<value key1 [key2]> specifies the values you want to delete.
Key1 are required for MAP, MAPofSETS and key2 is required
for MAPofMAPS
[-sdf=" ... "] specifies the Simple Date Format string used to
parse the date data.
remove
<name> is the name of the reference data collection.
purge
<name> is the name of the reference data collection.
get
<name> is the name of the reference data collection
[loadElements] displays all elements in the specified
reference data collection.
getall
[loadElements] displays all elements in all reference data
collections.
load
<name> is the name of the reference data collection
<filename> is a fully qualified filename to be loaded, with each
line in the file representing a record to be added to the
reference data collection
[-encoding=...] specifies encoding to use when reading the
file.
[-sdf=" ... "] specifies the Simple Date Format string used to
parse the date data.
Reference Data Collections