Configuring the log file protocol
The log file protocol retrieves event files that are stored from hosts to process events stored in remote locations.
About this task
The log file protocol is intended for systems that write daily event logs. It is not appropriate to use the log file
protocol for devices that appended information to their event files.
Log files are retrieved one at a time to be processed. The log file protocol can manage plain text, compressed
files, or file archives. Archives must contain plain-text files that can be processed one line at a time. When the log
file protocol downloads an event file, the information received in the file updates the Log Activity tab. If more
information is written to the file after the download is complete, the appended information is not processed.
Table 1. Log file protocol parameters
Parameter
Description
Log Source Name
Type a unique name of the log source.
Log Source Description
Optional. Type a description for the log source.
Log Source Type
From the list, select the type of log source to add.
Protocol Configuration
From the list, select Log File.
Log Source Identifier
Type an IPv4 address or host name to identify the log source
that created the events.
If the remote source contains multiple devices, such as a file
repository, administrators must specify the IP address of the
device that created the event.
Unique identifiers ensure that events are associated to the
correct device in the network, instead of identifying the event
for the management console or file repository.
Service Type
From the list box, select the protocol to use when retrieving
log files from a remove server. The options include:
 SFTP - Secure file transfer protocol
 FTP - File transfer protocol
 SCP - Secure copy protocol
The default is SFTP.
The server that is specified in the Remote IP or
Hostname field must have the SFTP subsystem enabled to
retrieve log files with SCP or SFTP.
Remote IP or Hostname
Type the IP address or host name of the device that contains
the event log files.
Table 1. Log file protocol parameters
Parameter
Description
Remote Port
Type the port that is used to communicate with the remote
host. The valid range is 1 - 65535. The options include:



FTP - TCP Port 21
SFTP - TCP Port 22
SCP - TCP Port 22
If the remote host uses a non-standard port number,
administrators must adjust the port value to retrieve events.
Remote User
Type the user name necessary to log in to the host that
contains the event files.
Remote Password
Type the password necessary to log in to the host.
Confirm Password
Confirm the password necessary to log in to the host.
SSH Key File
Type the path to the SSH key, if the system is configured to
use key authentication.
When an SSH key file is used, the Remote Password field is
ignored.
Remote Directory
Type the directory location on the remote host from which
the files are retrieved. The directory path is relative to the
user account that is used to log in.
Note:
For FTP only. If the log files are in the remote user’s home
directory, you can leave the remote directory blank. A blank
remote directory field supports systems where a change in the
working directory (CWD) command is restricted.
Recursive
Select this check box to enable the file pattern to search sub
folders. By default, the check box is clear.
This option is ignored for SCP file transfers.
FTP File Pattern
Type the regular expression (regex) required to identify the
files to download from the remote host. All files that match
the regular expression are included in the download.
This field applies to the SFTP or FTP file transfers.
SCP Remote File
For SCP file transfers, type the name of the file on the remote
host.
Table 1. Log file protocol parameters
Parameter
Description
FTP Transfer Mode
From the list box, select the transfer mode for the log source:

Binary - Select this option for log sources that require
binary data files or compressed archive files.
 ASCII - Select ASCII for log sources that require an
ASCII FTP file transfer.
Administrators must select NONE in the Processor field
and LINEBYLINE in the Event Generator field for ASCII
transfers over FTP.
Start Time
Type the time of day for the log source to start the file
import.
This parameter functions with the Recurrence value to
establish when and how often the Remote Directory is
scanned for files.
Recurrence
Type a time interval to determine how frequently the remote
directory is scanned for new event log files. The minimum
value is 15 minutes.
The time interval can include values in hours (H), minutes
(M), or days (D). For example, a recurrence of 2H scans the
remote directory every 2 hours.
Run On Save
Select this check box to start the log file import immediately
after the administrators saves the log source.
After the first file import, the log file protocol follows the
start time and recurrence schedule that is defined by the
administrator.
When selected, this check box clears the list of previously
downloaded and processed files.
EPS Throttle
Type the number of Events Per Second (EPS) that the
protocol cannot exceed.
The valid range is 100 - 5000.
Processor
If the files on the remote host are stored in an archive format,
select the processor that is required to uncompress the event
log.
Table 1. Log file protocol parameters
Parameter
Ignore Previously Processed
File(s)
Description
Select this check box to track files that were processed by the
log source.
This option prevents duplicate events from files that are
processed a second time.
This check box applies to FTP and SFTP file transfers.
Change Local Directory?
Select this check box to define the local directory on
the Target Event Collector to store event logs before they
are processed.
Administrators can leave this check box clear for more
configurations.
Local Directory
Type the local directory on the Target Event Collector. This
option is used with the Change Local Directory field.
The directory must exist before the log file protocol attempts
to retrieve events.
Event Generator
From the Event Generator list box, select one of the
following options:





LineByLine - Each line of the file is processed as a
single event. For example, if a file has 10 lines of
text, 10 separate events are created.
HPTandem - The file is processed as a HPTandem
NonStop binary audit log. Each record in the log file
(whether primary or secondary) is converted into text
and processed as a single event. HPTandem audit logs
use the following file name pattern: [aA]\d{7}.
WebSphere Application Server - Processes event logs
for WebSphere Application Server. The remote
directory must define the file path that is configured
in the DSM.
W3C - Processes log files from sources that use the
w3c format. The header of the log file identifies the
order and data that is contained in each line of the file.
Fair Warning - Processes log files from Fair Warning
devices that protect patient identity and medical
information. The remote directory must define the file
path to the event logs that are generated by the Fair
Warning device.
Table 1. Log file protocol parameters
Parameter
Description





File Encoding
DPI Subscriber Data - The file is processed as a DPI
statistic log produced by a Juniper Networks MX
router. The header of the file identifies the order and
data that is contained in each line of the file. Each line
in the file after the header is formatted to a tabdelimited name=value pair event.
SAP Audit Logs - Process files for SAP Audit Logs
to keep a record of security-related events in SAP
systems. Each line of the file is formatted to be
processed.
Oracle BEA WebLogic - Processes files for Oracle
BEA WebLogic application log files. Each line of the
file is formatted to be processed.
Juniper SBR - Processes event log files from Juniper
Steel-belted RADIUS. Each line of the file is
formatted to be processed.
ID-Linked Multiline - Processes multiline event logs
that contain a common value at the start of each line
in a multiline event message. This option uses regular
expressions to identify and reassemble the multiline
event in to single event payload.
From the list box, select the character encoding that is used
by the events in your log file.
Folder Separator
Type the character that is used to separate folders for your
operating system. The default value is /.
Most configurations can use the default value in Folder
Separator field.
This field is intended for operating systems that use a
different character to define separate folders. For example,
periods that separate folders on mainframe systems.
Enabled
Select this check box to enable the log source.
When this check box is clear, the log source does not collect
events and the log source is not counted in the license limit.
Credibility
Select the credibility of the log source. The range is 0
(lowest) - 10 (highest). The default credibility is 5.
Credibility is a representation of the integrity or validity of
events created by a log source. The credibility value assigned
to a log source can increase or decrease based on incoming
Table 1. Log file protocol parameters
Parameter
Description
events or adjusted as a response to user created event rules.
The credibility of events from log sources contributes to the
calculation of the offense magnitude and can increase or
decrease the magnitude value of an offense.
Target Event Collector
Select the Event Collector to use as the target for the log
source. When a log source actively collects events from a
remote source, this field defines which appliance polls for the
events.
This enables administrators to poll and process events on the
target event collector, instead of the Console appliance. This
can improve performance in distributed deployments.
When an administrator verifies firewall ports
between QRadar® and the remote database, the firewall must
allow communication between the target event collector and
the remote database.
Coalescing Events
Select this check box to enable the log source to coalesce
(bundle) events.
Coalescing events increase the event count when the same
event occurs multiple times within a short time interval.
Coalesced events provide administrators a way to view and
determine the frequency with which a single event type
occurs on the Log Activity tab.
When this check box is clear, the events are displayed
individually and the information is not bundled.
New and automatically discovered log sources inherit the
value of this check box from the System
Settingsconfiguration on the Admin tab. Administrators can
use this check box to override the default behavior of the
system settings for an individual log source.
Store Event Payload
Select this check box to enable the log source to store the
payload information from an event.
New and automatically discovered log sources inherit the
value of this check box from the System
Settingsconfiguration on the Admin tab. Administrators can
use this check box to override the default behavior of the
system settings for an individual log source.
Table 1. Log file protocol parameters
Parameter
Description
Log Source Language
Select the language of the events generated by the log source.
The log source language helps the system parse events from
external appliances or operating systems that can create
events in multiple languages.
Log Source Extension
Optional. Select the name of the extension to apply to the log
source.
This parameter is only available after a log source extension
is uploaded. Log source extensions are XML files that
contain regular expressions, which can override or repair the
event parsing patterns defined by a device support module
(DSM).
Extension Use Condition
From the list box, select the use condition for the log source
extension. The options include:


Groups
Parsing enhancement - Select this option when most
fields parse correctly for the log source.
Parsing override - Select this option when the log
source is unable to correctly parse events.
Select one or more groups for the log source.
Procedure
1.
Click the Admin tab.
2.
Click the Log Sources icon.
3.
Click Add.
4.
Configure the parameters for your log source. The DSM Configuration Guide provides step-by-step
instructions to configure each log source.
5.
Click Save.
6.
On the Admin tab, click Deploy Changes.