The Pros and Cons of Collecting Performance Data
using Agentless Technology
Dima Seliverstov
John Tavares
Tianxiang Zhang
BMC Software, Inc.
Why Agentless
› Installing and administrating agents is a pain
–Agents have to be distributed across the enterprise
–When agents break they have to be patched
–Each agent you install has its own file system and
network port requirements
–Agents are not always built to network or security
standards
© Copyright BMC Software, Inc.
Blue Sky Model
› Wouldn’t it be great if operating systems came with
a standards based agent installed and configured
for monitoring and capacity planning?
© Copyright BMC Software, Inc.
Weather Report
› Identify which operating systems come with
performance collection installed
› Identify the metrics available and how often to
collect them
› Network and security issue
› Identify an alternate way of getting the system
performance data without having to install an agent
© Copyright BMC Software, Inc.
Overview of the Presentation
› Agentless data collection is a powerful technology
which has its advantages and disadvantages.
–System performance metrics availability
–Agentless monitoring components installed as part of
the operating system
–Security and network issue
› Present our experience with Windows agentless
technology
© Copyright BMC Software, Inc.
Agentless Implementations Considered
› SNMP (Simple Network Management Protocol)
› WMI (Windows Management Interface)
› Windows Remote Registry
› WBEM (Web Based Enterprise Management)
› Agentless monitoring by sending system
commands over the network
© Copyright BMC Software, Inc.
WBEM (Web Based Enterprise Management)
Introduction
› Developed by DMTF (Distributed Management Task Force)
› Built on top of the Common Information Model (CIM)
– Hierarchical object oriented representation of management information such
as computer system, network devices and applications
– Compiled textual representation known as MOF (Managed Object Format)
© Copyright BMC Software, Inc.
WBEM Communication Mechanism
› WBEM Client
– Issues CIM Operation requests and receives and
processes CIM Operation responses
› WBEM Server
– Uses CIMOM (CIM Object Manager) to communicate
with clients
– Receives and processes CIM Operation requests and
issues CIM Operation responses
– Uses CIM Repository as persistent store containing CIM
data in a namespace
– The root of the namespace is root\cimv2
› WBEM Provider
– Process CIM Operations on one or more managed
resources and maps the resource specific interface to a
CIM interface
© Copyright BMC Software, Inc.
WBEM Security
› Windows implementation of WBEM
– WMI (Windows Management Instrumentation)
– Encryption and authentication on per call or a per packet basis
› UNIX WBEM
– Encryption and authentication support via public key certificates via https
© Copyright BMC Software, Inc.
Windows Remote Registry
› In addition to WMI, Microsoft provides a Remote Registry interface to
getting system information from the performance registry remotely
› Available since Windows NT
› Unfortunately, there have been several security warning regarding
the use of remote registry service
› The protocol for the remote registry is RPC (Remote Procedures Call)
and introduces challenges in terms of network connectivity and
platform independent client tools
© Copyright BMC Software, Inc.
WBEM Availability
› Windows
– WMI installed as part of the core operating system since Windows 2000
› Solaris
– Solaris WBEM Services has been installed since Solaris 2.9
› HP-UX
– HP WBEM Services has been installed since HPUX 11.0
› AIX
– WBEM is supported for via the AIX 5L Expansion Pack and Web Download
Pack for AIX 5.2 and higher
› Linux
– RedHat and SUSE support Open Pegasus package Linux install packages
© Copyright BMC Software, Inc.
SNMP Overview
› Standard primarily used for network management
› SNMP stores elements in a Management Information Base (MIB )
– Extensible
– MIB stores information in a tree format
– Extended for system performance data
• RFC 2790 Host Resource MIB
rfc1213.mib
RFC1213-MIB
[Types]
iso (1)
org (3)
dod (6)
internet (1)
mgmt (2)
mib-2 (1)
system (1)
interfaces (2)
at (3)
ip (4)
icmp (5)
tcp (6)
udp (7)
egp (8)
transmission (10)
snmp (11)
© Copyright BMC Software, Inc.
SNMP Communication Mechanism
› Management information is maintained by the SNMP agent and
queried by a SNMP manager
› SNMP agent and the SNMP manager must have access to the
Management Information Base
› The SNMP agent and SNMP manager communicate via UDP (User
Datagram Protocol) port 161
› The SNMP manager gets the data from the agent via the get-request
and get-next-request
get-response
get-response-next
SNMP Manager
© Copyright BMC Software, Inc.
SNMP Agent
UDP Port 161
SNMP Security
› There are 3 major versions of SNMP
–SNMP v1
• Unencrypted traffic
• IP address-based access lists and community
strings
–SNMP v2
• Encryption
–SNMP v3
• Encryption
• Authentication
© Copyright BMC Software, Inc.
SNMP Agent Availability
› Usually not installed as part of the operating system
› Open source SNMP agent called net-snmp is
available to gather the data for the Host Resource
MIB
› A Host Resource MIBs provides a limited set of data
–The data availability will be shown in the slides to
follow
© Copyright BMC Software, Inc.
Metric Availability
› Windows provides the best data set
› UNIX WBEM implementations provide a good data
set with a limited number of exceptions
› SNMP provides a limited amount of performance
configuration data and almost no statistical data
© Copyright BMC Software, Inc.
CPU
CPU Configuration
Model
Clock Rate
CPU Statistics
User Time
System Time
Idle Time
Wait Time
Run Queue Length
Solaris
HPUX
AIX
Linux
Windows
SNMP
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
No
No
No
No
No
© Copyright BMC Software, Inc.
Memory
Total Physical
Memory
Free Physical Memory
Solaris
HPUX
AIX
Linux
Windows
SNMP
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
© Copyright BMC Software, Inc.
Disk and File System
Disk Configuration
Model
Capacity
Disk Statistics
Bytes Read
Bytes Written
I/O Count
I/O Time
Queue Length
File System Type
Mount Point
Block Size
Capacity
Free Space
Solaris
HPUX
AIX
Linux
Windows
SNMP
No
Yes
Yes
Yes
Yes
Yes
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
No
No
No
No
No
AIX
Yes
Yes
Yes
Yes
Yes
Linux
Yes
Yes
Yes
Yes
Yes
Windows
Yes
Yes
Yes
Yes
Yes
SNMP
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Solaris
Yes
Yes
Yes
Yes
Yes
No
No
No
No
HPUX
Yes
Yes
Yes
Yes
Yes
© Copyright BMC Software, Inc.
Network
Network Configuration
Type
Network Statistics
Bytes Received
Bytes Transmitted
Packets Received
Packets Transmitted
Solaris
HPUX
AIX
Linux
Windows
SNMP
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
No
No
No
No
No
No
No
No
No
No
Yes
Yes
Yes
Yes
No
No
No
No
© Copyright BMC Software, Inc.
Process
Command Name
Real User ID
Process ID
Start Time
Priority
Nice Value
Scheduling Class
Working Set Size
Resident Set Size
Shared Resident Size
Resident Data Size
Resident Text Size
User Time
System Time
Major Page Faults
Minor Page Faults
Bytes Read
Bytes Written
Solaris
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
No
Yes
Yes
Yes
Yes
Yes
Yes
HPUX
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
No
No
No
No
AIX
Yes
Yes
Yes
Yes
No
No
No
No
No
No
No
No
Yes
Yes
No
No
No
No
© Copyright BMC Software, Inc.
Linux
Yes
Yes
Yes
Yes
Yes
Yes
No
No
No
No
No
No
Yes
Yes
No
No
No
No
Windows
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
SNMP
Yes
No
Yes
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
UNIX Agentless Monitoring by Sending System Commands
over the Network
› Log in remotely to multiple agentless computers and collect the
necessary performance metrics
› Use Secure Shell
– Supports certificate based encryption and authentication
Metric Type
Command
CPU
sar,uptime
Memory
sar, vmstat
Process
ps
I/O
netstat,iostat,vmstat,netstat,nfsstat
Disk
df,iostat
© Copyright BMC Software, Inc.
Downside the Approach
› The commands vary from operating system to operating
system
› The output of the command must be parsed specifically for
each operating system
› Secure shell may not be installed on all computers of
interest
› Metrics are usually instantaneous and not cumulative and
must be sampled frequently for a reliable system estimates
© Copyright BMC Software, Inc.
Issues with Agentless Data Collection
› Network Ports
› Network Bandwidth
› Data Sampling
› Patching
© Copyright BMC Software, Inc.
Network Ports
› Windows WMI and Remote Registry use dynamic
ports
–This wide range of ports used, can play havoc with
firewall rules
Infrastructure
Ports Used
SNMP
161
WMI
135,137,139,445 1024-65536
Remote Registry
WBEM
5988,5989
© Copyright BMC Software, Inc.
Network Bandwidth
› SNMP agents use considerably/exponentially less
bandwidth then WBEM
– WBEM uses an XML based protocol
› If you lose network connectivity from the client to the
agent-less server, you will lose data
© Copyright BMC Software, Inc.
Network Bandwidth continued
› No batch transfer
›
– Data is collected through out the day
– It’s not possible to schedule batch data transfer at a particular
time when the network utilization is low
No summarization
– Data is not summarized at the agentless computer
– Every data point requested is transmitted over the network
© Copyright BMC Software, Inc.
Data Sampling
› Cumulative Counters and Instantaneous Gauges
– Cumulative Counters need to be sampled less frequently because they
represent the difference in a system between two points in time
› WMI and WBEM provide many metrics that are cumulative
› SNMP and sending system commands over the network approach will
provide mostly instantaneous metrics
› SNMP and some UNIX WBEM implementations CPU and Process
Metrics are provided as an average over a 1 minute time interval
› In general, to achieve good representation of system activity for
capacity planning a sample rate should be 1 – 2 minutes for
instantaneous values
© Copyright BMC Software, Inc.
Patching agentless components
› Isn’t the whole point of going to agentless data
collection to avoid patching?
› Agentless really means that someone already
installed an agent for you
› Agentless components do break and need to be
patched
› It can be challenging to identify and address a
problem due to a variety of tools and debugging
techniques for each platform
© Copyright BMC Software, Inc.
Experience with Windows Agentless
Technology
› WMI requires 'Enable Account' and 'Remote Enable' permissions for
the Root/CIMV2 for the WMI namespace
› Configuring user accounts on individual computers can be
challenging
– Use domain administrator account
– Use regular user domain account to configure agent-less computers with a
regular user with 'Enable Account' and 'Remote Enable' permissions for the
Root/CIMV2 for the WMI namespace
• You will need to create the permission on each individual computer
› Windows WMI encountered frequent security events when running
using a non-administrator user account on Windows 2003 machines
› Network administrators were concerned about the constant network
traffic during the day
© Copyright BMC Software, Inc.
Final Thoughts
› WBEM is extremely promising
› Software vendors including Hewlett-Packard, IBM,
Microsoft, RedHat and SUSE are beginning to roll
out their products with WBEM support enabled
› WBEM provides a common and secure framework
for collecting system performance data
› A strong development, open source community and
a standards organization are supporting a variety of
implementations for clients, servers and providers
© Copyright BMC Software, Inc.