How QRadar SIEM Collects Security Data Luis Latas WW Technical Sales Enablement QRadar Data Flow - Overall From an Appliance Perspective Event Collector Capabilities Event Collector (ecs-ec-ingress) Event Collector (ecs-ec) From an Appliance Perspective Event/Flow Processor Capabilities Event Collector (ecs-ec-ingress) + Event Collector (ecs-ec) Event Processor (ecs-ep) From an Appliance Perspective AIO/Console Capabilities Event Collector (ecs-ec-ingress) + Event Collector (ecs-ec) Event Processor (ecs-ep) Magistrate High-level component architecture and data stores Flow and event data is stored in the Ariel database on the event processors Identities Assets Offenses Configuration – If accumulation is required, accumulated data is stored in Ariel accumulation data tables – As soon as data is stored, it cannot be changed (tamper proof) Console services User interface Magistrate Reporting Flows Events Accumulations – Data can be selectively indexed Offenses, assets, and identity information are stored in the master PostgreSQL database on the Console Event processor Flow collector Event collector Network packet interface, sFlow, and 3rd party Events from log sources – Provides one master database with copies on each processor for backup and automatic restore Secure SSH communication between appliances in a distributed environment is supported 6 QRadar Data Flow - Overall IBM Security / © 2020 IBM Corporation 7 Collecting and Normalizing raw events An event is a record from a device that describes an action on a network or host. QRadar SIEM normalizes the varied information found in raw events. – Normalizing means to map information to common field names, for example: • SRC_IP, Source, IP, and others are normalized to Source IP. • user_name, username, login, and others are normalized to Username. – Normalized events are mapped to high-level and low-level categories to facilitate further processing. – After raw events are normalized, it is easy to search, report, and cross-correlate these normalized events. Event data pipeline Event data is sent to or pulled by QRadar Event Data Event Collector Ingress – Responsible for collecting data at all times (zero event loss) Data is collected and buffered during patch and deploys and processed once the operation is complete Protocols – Reads or pulls raw data from network devices (e.g: Windows Servers, Firewalls, etc) Throttle Filter - Licensing - On a second-bysecond basis, slows down the incoming rate so it does not exceed the license on the appliance. Protocols Event Collector – Ingress (ecsec-ingress) Throttle Filter Licensing Event Collector (ecs-ec) Events are sent to ecs-ec-parse to be parsed IBM Security / © 2020 IBM Corporation 9 Event data pipeline Event Collector – Ingress (ecsec-ingress) Event data is received from the ecs-ec-ingress Parsing – DSMs / LSX / CEP – take the raw data and normalize it into a common structure. Coalescing - “Event Compression”. Find nearly identical events and delete one and increase the event count on the record. Key is: source IP, dest IP, dest port, QID, username Forwarding - Applies routing rules for the system, such as sending event data to offsite targets, external Syslog systems, JSON systems, and other SIEMs. Log Only/Data Store supports the storage of an unlimited number of logs without counting against the EPS License Events are then sent to the Event Processor component and pass through the Custom Rules Engine (CRE). IBM Security / © 2020 IBM Corporation Parsing (DSM, LSX, CEP) Event Collector (ecs-ec) Coalescing Forwarding Log Only/Data Sore Event Processor 10 Events not counted against the EPS licenses - The list of log source types that do not incur EPS hits are as follows: - System Notification - Custom Rule Engine (CRE) - SIM Audit - Anomaly Detection Engine - Asset Profiler - Search Results from scheduled searches - Health Metrics - Sense DSM - Risk Manager questions, Simulations and internal logging - Log Only/Data Store - Supports the storage of an unlimited number of logs without counting against the EPS QRadar SIEM license - Enables an organization to build custom apps and reports based on this stored data to gain deeper insights into IT environments. Event Coalescing - Event Coalescing is a method of reducing the data going through the pipeline. - As data arrives in the pipeline QRadar will attempt to group like events together into a single event. - Coalescing occurs after licensing and parsing - Coalescing is indexed by Log Source, QID, Source IP, Destination IP, Destination Port and Username. - If more than 4 events arrive within a 10 second window with these properties being identical any additional events beyond the 4th will be collapsed together. - Coalesced events can be identified by looking at the Event Count column in the log viewer, if the Event Count is >1 the event has been coalesced. - Coalescing can be turned on or off per log source or by changing the default setting in the system setting page. QRadar Data Flow - Overall © Copyright IBM Corporation 2017 13 Flow collection and processing A flow is a communication session between two hosts QFlow Collectors read packets from the wire or receive flows from other devices QFlow Collectors convert all gathered network data to flow records similar normalized events; they include such details as: – when, who, how much, protocols, and options. Flow pipeline The QFlow component collects and creates flow information from internal and external flow sources Flows Event Collector – Responsible for parsing and normalizing incoming flows Qflow Asymmetric recombination - Responsible for combining two sides of each flow when data is provided asymmetrically Deduplication - Flow deduplication is a process that removes duplicate flows when multiple Flow Collectors provide data to Flow Processors appliances. Flow Governor - Monitors the number of incoming flows to the system to manage input queues and licensing. Custom flow properties – extracts any properties defined in the Custom Flow Properties Forwarding - Applies routing rules for the system, such as sending flow data to offsite targets, external Syslog systems, JSON systems, and other SIEMs. Flows are then sent to the Event Processor component and pass through the Custom Rules Engine (CRE). They are tested and correlated against the rules that are configured Asymmetric Recombination De-Duplication Event Collector Flow Governor (Licensing) CFP Parsing Flow Forwarding Event Processor QRadar Data Flow - Overall IBM Security / © 2020 IBM Corporation 16 Event & Flow Correlation and Processing After Events and Flows are normalized they are then sent to the Event Processor for processing Event Collector Licensing is applied again on ingress to the EP The CRE or Custom Rules Engine Applies the correlation rules that were created in the UI. Flow data is then sent to the Ariel Database for storage. Host Profiling – Also called passive profiling or passive scanning. Watches flows on the network in order to make educated guesses about which IPs/assets exist and what ports are open. Streaming – Responsible for the “real time (streaming)” view in User Interface If an event matches a rule, the Magistrate component generates the response that is configure in the custom rule Licensing CRE Event Processor Ariel Storage and Indexing Host Profiling Real Time streaming Magistrate Asset Profiler QRadar Data Flow - Overall IBM Security / © 2020 IBM Corporation 18 Magistrate • The Magistrate creates and stores offenses in the PostgreSQL database; these offenses are then brought to the analyst’s attention in the interface • The Magistrate instructs the Ariel Proxy Server to gather information about all events and flows that triggered the creation of an offense • The Vulnerability Information Server (VIS) creates new assets or adds open ports to existing assets based on information from the EPs • The Anomaly Detection Engine (ADE) searches the Accumulator databases for anomalies, which are then used for offense evaluation Ariel Components Ariel Proxy Server Event Processor Ariel Query Server Ariel Ariel Historical Correlation Offline Forwarder Accumulator Report Runner Ariel Components Ariel Components Console Ariel Proxy Server Ariel Ariel query Server Ariel query Server Managed Host Ariel Managed Host Ariel Asset and Vulnerability Flow ecs-ec (event collector Identity Data Asset Profiler Passive Profiling Scanners/QVM/3rd Party vis (Vulnerability integration service) ecs-ep Event Processor POSTGRES Gathering asset information Active scanners QRadar Vulnerability Manager scanner, Nessus, Nmap, Qualys, and others Provide: • List of hosts with risks and potential vulnerabilities • IP and MAC addresses • Open ports • Services and versions • Operating system Pros • Detailed host information • Policy and compliance information Cons • • • • Out of date quickly Full network scans can take weeks Active scanners cannot scan past firewalls User can hide from active scans Passive detection Flows from QFlow, or other flow sources in accounting technologies such as IPFIX/NetFlow, sFlow, and others Provide: • IP addresses in use • Open ports in use Pros • Real-time asset profile updates • Firewalls have no impact • End system cannot hide • Policy and compliance information Cons • Not as detailed as active scans • Does not detect installed but unused services or ports The Remainder Hostcontext “Owns” the host it is responsible for starting and stopping processes and for overall system health and backups. Reporting Executor A stopwatch responsible for keeping track of reports and when they should run and then instantiating the report runner Report Runner The process that actually generates the reports, querying postgres, Ariel, etc. Tomcat Process that drives our web UI and serves up web pages. Historical Correlation Processor Process that is responsible for historical correlation. Runs a specified search, runs the results through CRE rules (based on QRadar time or device time) and generates offenses QRadar Data Flow - Overall Thank you Follow us on: ibm.com/security securityintelligence.com ibm.com/security/community xforce.ibmcloud.com @ibmsecurity youtube.com/ibmsecurity © Copyright IBM Corporation 2020. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty, of any kind, express or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others. Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks on others. No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access. IBM systems, products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products or services to be most effective. IBM does not warrant that any systems, products or services are immune from, or will make your enterprise immune from, the malicious or illegal conduct of any party.
