Building a high availability
ArcSight solution
Paul Brettle – Presales Manager, Americas Pacific Region
#HPProtect
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
What is high availability?
High availability system design approach and associated service implementation
that ensures a prearranged level of operational performance will be met during a
contractual measurement period.
1. Elimination of single points of failure. This means adding redundancy to the system so that failure of a
component does not mean failure of the entire system.
2. Reliable crossover. In multithreaded systems, the crossover point itself tends to become a single point of
failure. High availability engineering must provide for reliable crossover.
3. Detection of failures as they occur. If the two principles above are observed, then a user may never see a
failure. But the maintenance activity must.
3
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
What is disaster recovery?
Disaster recovery (DR) involves a set of policies and procedures to enable the
recovery or continuation of vital technology infrastructure and systems following
a natural or human-induced disaster.
• [1] Disaster recovery focuses on the IT or technology systems supporting critical business functions
Critical differentiation
• What do I need?
• How do I approach it?
• What is the minimum that I will accept?
4
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
But what is high availability?
Understand what is required, approach and differences
•
•
•
•
•
Data
Systems
Usage
Resilience
Processing
Understand differences between hot, warm, and cold!
5
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Prioritize and organize
What are the drivers for this?
•
•
•
•
Regulation?
Legislation?
Compliance?
Good governance/best practice?
Start examining the critical components
Look at systems, processes and models to assist you
• More on this later!
6
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
What do I get by default?
Communications
• Reliable communications
Cache
• Built in once collected for all SmartConnectors
Commit
• Commit model for storage of data (SmartConnector -> ESM)
Recovery
• Archive files
Hardware
• Dual power supply, reliable hardware, hot swap components and storage
7
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
ArcSight Architecture
ArcSight Logger Instance
Analysts will leverage the
ArcSight Console or a web
browser to access ESM,
Logger, and CA.
Logger
SAN
(Optional)
Enriched events from ESM will be forwarded
to Logger for long-term event storage.
ArcSight ESM Instance
Analysts
Events from all
SmartConnectors will be
forwarded to the ESM Instance.
Manager
Database
Connector
Appliance
(Optional)
SAN
All SmartConnectors are
managed remotely via the
ArcSight Connector Appliance
or ESM Manager.
ArcSight
ArcSight
ArcSight
SmartConnectors
8
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
ArcSight Architecture
ArcSight ESM Instance
AUP Master
Analysts will leverage the
ArcSight Console or a web
browser to access ESM,
Logger, and CA.
Manager
Database
SAN
Events of interest will be forwarded from Logger to ESM
for real-time correlation. Correlated events will be
forwarded back to Logger for long-term storage.
ArcSight Logger Instances (2+)
Logger
Analysts
Connector
Appliance
Logger
Loggers are configured in a Peer Network.
Events from all
SmartConnectors will be
forwarded to separate Loggers
for load balancing purposes.
ArcSight
AUP Master
9
ArcSight
ArcSight
SmartConnectors
ArcSight
ArcSight
ArcSight
SmartConnectors
ArcSight
ArcSight
ArcSight
SmartConnectors
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
All SmartConnectors are
managed remotely via the
ArcSight Connector Appliance.
ArcSight Architecture
Globally correlated and base events will be
forwarded from the Global ESM Instance to
Logger for long-term storage.
Analysts will leverage the ArcSight Console
or a web browser to access the Global or
Regional ESM and Logger Instances.
All SmartConnectors are managed remotely
via the ArcSight Connector Appliance.
Global ESM Instance
ArcSight Logger Instances
Logger (Optional)
Analysts
Manager
Database
SAN
Correlated and the base events will be forwarded
from each Regional ESM Instance to the Global ESM
Instance for Global Correlation.
Loggers can be configured in a Peer Network for
a holistic view of all events in the environment.
Regional ESM Instance
Manager
Database
Regional ESM Instance
Events of interest
will be forwarded
from Logger to ESM
for real-time
correlation.
Correlated events
will be forwarded
back to Logger for
long-term storage.
SAN
ArcSight Logger Instances
Loggers
ArcSight
ArcSight
ArcSight
SmartConnectors
Connector Appliance
ArcSight
ArcSight
Manager
SmartConnectors
Database
Regional ESM Instance
Events of interest
will be forwarded
from Logger to ESM
for real-time
correlation.
Correlated events
will be forwarded
back to Logger for
long-term storage.
SAN
ArcSight Logger Instances
Events from all
SmartConnectors
will be forwarded to
the Regional ESM
Instances.
ArcSight
Connector Appliance
(Optional)
ArcSight
Loggers
ArcSight
ArcSight
SmartConnectors
Connector Appliance
ArcSight
ArcSight
Manager
SmartConnectors
SAN
ArcSight Logger Instances
Events from all
SmartConnectors
will be forwarded to
the Regional ESM
Instances.
ArcSight
Database
ArcSight
Loggers
ArcSight
ArcSight
SmartConnectors
Connector Appliance
ArcSight
ArcSight
ArcSight
SmartConnectors
Connector layer
ArcSight Logger/Express/ESM
•
•
•
•
Push connector type
Load balanced
Needs consistency
Typically used for
–
–
Each SmartConnector forwards on the
encrypted, compressed and processed
events to the ArcSight solution.
Syslog
Large volumes
Node 1
Node 2
Session information
shared for load
balancing only.
The source devices send their logs
and events directly to the load
balancing IP address using their
native protocol, such as Syslog.
11
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Here a two-node load
balancing solution can be
deployed. The load balancing
system can be used to spread
the load between two or more
nodes for processing. There is
no need for clustering here as
we simply want to process the
logs and events and this
represents the most efficient
method to do this.
Connector layer
ArcSight Logger/Express/ESM
• Pull connector type
• Log messages not lost
• Active HA needed
–
SmartConnector forwards on the
encrypted, compressed and processed
events to the ArcSight solution.
Require consistency
• Typically not implemented
Active
node
Passive
node
Shared disk
SmartConnector connects to
the sources directly from the
active node. All processing is
done by the active node but
state information stored on
shared drive.
12
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Two-node active/passive
cluster for the
SmartConnector. Should the
active node fail for any
reason, the passive node can
continue where it left off.
Since the shared disk is used,
all current events are
processed with no loss
or duplication.
Log storage layer
Logger DR site
Main Logger
• Dual feed strategy
• Duplicate in two Loggers
• No replication needed
Connector receives/pulls the
events and forwards on to
configured Loggers.
Connector
Devices send/receive their logs and events to
and from the Connector in their native formats
as required. Typically this will be via Syslog,
which uses UDP.
13
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Log storage layer
Storage device used for
archived daily logs. Secondary
Logger can retrieve archives
as necessary.
Logger DR site
•
•
•
•
Warm standby model
Backup configuration
Access archives
Provide cache at connectors
Configuration restored to
access stored data and
assume role of main Logger.
Main Logger
Devices send/receive their logs and events to
and from the shared IP in their native formats
as required. Typically this will be via Syslog,
which uses UDP.
14
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Log storage layer
• Most effective solution
• Dual feed and dual archive
–
–
–
Easy to restore
Little impact
No replication needed
• Be aware of network
Logger DR site
Main Logger
Loggers auto-archive to
storage system for resilient
long-term storage.
Connector receives/pulls the
events and forwards on to
configured Loggers.
Connector
Connector
Devices send/receive their logs and events to
and from the Connector in their native formats
as required. Typically this will be via Syslog,
which uses UDP.
15
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Remote site
Correlation layer
Primary
Manager
• ESM with Oracle
–
–
–
Simple fail-over to single DB
Use commercial solutions
Tried and trusted
• Replicate database
–
Several technologies available
• Fail-over manager starts
• Console re-connects
Heartbeat
ArcSight Console
Oracle
database
Fail-over
Manager
16
Here a primary Manager is used
as the single processing server
for the correlation etc. of the ESM
solution. All communications to
the database come from the
single primary Manager.
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Correlation layer
• ESM with CORRe
–
–
–
• Consider ESM/Express
• Look at options
• Work out difference
–
Primary
Manager
No one single DB
Need to replicate DB
Consider options
Replication
ArcSight Console
HA or DR
Fail-over
Manager
17
CORR
database
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CORR
database
Options?
Hardware
• Power
• Disk
• Network
Software
• HA/fail-over/cluster software
Operating system
• HA/fail-over/cluster software
Virtualization
• Don’t forget what you can get here
• Usually a cost option
18
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Summary
Lots of options
• Consider what is needed and how to address
HA deployed at a lot of customers
• Using in-built and external technologies
Only as strong as weakest link
Plan and understand issues
19
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Please give me your feedback
Session TT3058 Speaker Paul Brettle
Please fill out a survey.
Hand it to the door monitor on your way out.
Thank you for providing your feedback, which
helps us enhance content for future events.
20
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.