Systems Management in
an Untrusted Network
Dealing with backups,
monitoring, administration, and
logging in the DMZ
Cory L. Scott, Lead Security Consultant
Securify, Inc.
Introduction
Implementing systems management components in
untrusted or semi-trusted networks is difficult…
…if you are concerned about security!
Outline of today’s talk:
Example of threats
DMZ Network Architectures in the Real World™
Two core designs and advanced design issues
System and network configuration for systems management
The threat is out there…
 SNMP
– Multiple Vendor SNMP World Writeable Community
Vulnerability
– NAI Sniffer Agent SNMP Buffer Overflow
Vulnerability
 Sniffers
– Microsoft Network Monitor Multiple Buffer Overflow
Vulnerabilities
– Solaris snoop (print_domain_name) Buffer
Overflow Vulnerability
Specific Vulnerability Titles courtesy of SecurityFocus
The threat is out there…
 Remote Control Software (besides its
intended functionality)
– AT&T VNC Weak Authentication Vulnerability
– PCAnywhere32 Denial of Service Vulnerability
 Administrative Interfaces (over intended
functional protocols)
– Allaire ColdFusion Server 4.5.1 Administrator
Login Password DoS Vulnerability
– Cisco 7xx Series Router DoS Vulnerability
– Cisco 675 Web Administration Denial of Service
Vulnerability
Specific Vulnerability Titles courtesy of SecurityFocus
The threat is out there…
 System logging
– Age-old attacks:
• log flood
• log erase
• selective log edit
– Linux syslogd Denial of Service Vulnerability
– Solaris syslogd Unresolvable Address Remote
Denial of Service Vulnerability
 Backup
– Unauthorized restore/delete, unencrypted backups
– Veritas Backup Denial of Service Vulnerability
Specific Vulnerability Titles courtesy of SecurityFocus
The Purpose of the DMZ
 But… I’m filtering System Management Protocols
at the perimeter! Isn’t that enough?
 No.
 Why?
 Two words: Aggravated Penetration.
Want two more? Privilege Escalation.
More? Insider attack.
 DMZ hosts are bastion hosts or perimeter service
hosts. Why do we spend all this time hardening
our DNS servers and then leave a poor password
on the ssh service listening on an untrusted
interface?
The Purpose of the DMZ
 The DMZ exists to mitigate risk by
isolating certain services and functions in
a separate segment of the network.
 Segmentation by isolation is generally not
enough. Ingress filtering is usually
deployed, but typical designs need work
past the “crunchy” outside layer. Defense
in depth, along with proper protection of
internal hosts from the DMZ, is required.
The Purpose of the DMZ
 Example 1. Bastion hosts in a DMZ
Segment.
The Purpose of the DMZ
 Example 2. Perimeter service hosts in a
flat network.
The Purpose of the DMZ
If it were only that simple…
The Purpose of the DMZ
 Example 3. Segmented DMZs.
The Purpose of the DMZ
 Example 4. Colocated DMZs.
The Purpose of the DMZ
 Example 5. Partner DMZs.
The Purpose of the DMZ
 Other problems in the DMZ
– Constant change
– Too many hands in the pot
– Service protocols not designed with security in
mind
– Systems management protocols not designed
with security in mind
– Scalability mechanisms create additional
separation and obfuscation of a clean network
design
– Collusion of disparate types of traffic going
through the DMZ
System Management – Composite Sketch
 Common sighting – the status quo:
– No centralized logging.
– SSH inbound from the internal network; often from
external network, too.
– PCAnywhere, VNC, or SMS accessible from some
management hosts or worse...
– Backup system non-existent or backups batch copied to
internal hosts.
– Default administrative protocols and interfaces left
accessible within the DMZ and from the internal
network:
• SNMP on routers, web interfaces on servers
• Openview or other monitoring system “pinging” from
the inside.
Dealing with the problem…
 Securing Systems Management
components requires a combination of
network architecture and system
configuration.
Two Core Designs
DMZ Network Architecture for System Management
 Example 1. Combination of Management and
Production Traffic on the same untrusted
segment.
 Definition of untrusted segment: Where
untrusted users and/or processes can place
packets on the segment.
 Advantages:
– Simple to manage, do not have deal with multiple
interfaces
– Easier firewall rulesets and router ACLs to manage
DMZ Network Architecture for System Management
 Example 1. Combination of Management and Production
Traffic on the same untrusted wire.
 Disadvantages:
– Bandwidth utilization
– Failure to segment different types of traffic introduces security
risks
– Must place loghosts, monitoring consoles, and control
components on the internal network to keep isolation
– Harder to monitor for policy violations
– Untrusted segment behind firewall will advertise management
services
– For services that listen for input, must configure host-based
inclusion rather than interface/network inclusion
– Compromised host on segment could spoof management
connection
DMZ Network Architecture for System Management
 Example 2. Separate Management LAN.
 Advantages:
– Protects bandwidth on untrusted network segment
– Introduces another hurdle for intruders to jump
interfaces, which can be locked down more
aggressively
– Ability to monitor for violations in both segments
improved
– Can place loghosts, monitoring hosts, and control
components in management LAN with less risk,
reducing internal network exposure and reliance
– Allows for more flexibility with private address
space and less border firewall concerns
DMZ Network Architecture for System Management
 Example 2. Separate Management LAN.
 Disadvantages:
– Need to make sure that forwarding is disabled;
routing must be configured correctly on each host;
additional configuration and equipment needed
– Management LAN can still be used as a conduit to
attack hosts if not properly secured and monitored
– Adds complexity to segmented DMZs and potential
bypass mechanism between segments
Advanced Design Issues
DMZ Network Architecture for System Management
 Example 3. Management Aggregation Points
based on natural segregation of the
segmented DMZ.
 Advantages:
– Works well in segmented DMZs
– Reduces management LAN bandwidth
– All of the advantages of segmented DMZs
 Disadvantages:
– More equipment and more routes
– Need to maintain ACLs and rulesets between
Management LANs
– Additional points of failure
– All of the disadvantages of segmented DMZs
DMZ Network Architecture for System Management
 Example 4. Pushing data versus pulling data.
 Pushing data from internal network to the DMZ/Admin
LAN. Good – but how much do you trust your internal
users?
 DMZ/Admin LAN pulling data from the internal network.
Bad.
 Degrees of push:
– File / Data one-way with or without validation
– Interactive transfer with restricted privilege
– Remote control administration with full interactivity
 Use the minimum amount of push whenever possible.
 When DMZ hosts need to push data for administrative
purposes, aggregate in the same trust boundary. Then
pull from a more trusted environment.
 Never have DMZ hosts pull or push from the Internet
without appropriate risk analysis and mitigation.
The Need for Systems Management
 Backup
 Diagnostic information and availability
monitoring
 Remote administration
 System logging
Backup Solutions
 Risks
–
–
–
–
–
–
Bandwidth utilization
Unauthorized restore / backup
Capture of backup traffic
Agent vulnerabilities – authentication
Procedures for restore offsite
Local backup devices unmanageable or
difficult to scale
– Backup clients not necessarily designed with
security in mind
Backup Solutions
Securing Backup Solutions
 Protect the backup server at all costs
– Place behind another firewall / filter
– Backup server should initiate all backup / restore requests
to eliminate inbound connections
– Consider the physical security of the server and the media
– Implement tight security controls on server.
 Encryption – examine the risks / benefits
– Is the wire insecure? If so, client has burden of encrypting
the data.
– Store the data encrypted or not? How is key management
performed? What happens if the key is lost?
– Encrypt both on-site and off-site media?
Backup Solutions
Securing Backup Solutions
 Administrative LAN segment very beneficial for backup
solutions
 Implementing a Storage Area Network may provide
another means for backup that doesn’t use the LAN
 One example of a hard-to-secure product:
– Legato:
• Server uses default ports 7937-9936/TCP&UDP
• Client uses default ports 10001-30000/TCP&UDP
• Runs its own portmapper
• Ports can be restricted
• Authentication client/server unclear
• NAT not supported
• Unable to determine which interface it listens on
Monitoring Solutions
SNMP
 Assume that anything sent over SNMP is readable by all.
 Community strings should be changed.
 If possible, limit the hosts that can query SNMP on the
queried device itself.
 Examine the type of information that your device gives via
SNMP – it may surprise you.
 Determine the criticality of the information when deciding
whether or not to use SNMP.
 Never allow reconfiguration of devices via SNMP. Disable
write privileges on any SNMP device.
 Traps should be used sparingly and there should be a
dedicated receiver in the DMZ.
 NT SNMP giveaway.
 Oftentimes, it is the lesser of another evil.
Monitoring Solutions
ICMP
 Echo reply/request is fine on an internal
interface.
 If possible, throttle your ICMP response
queue.
Remote Administration Solutions
Console access
 Target devices include: routers, switches, load
balancers, some UPSes, firewalls, and servers
 Aggregate console connections into a terminal server
 Can use a hardware terminal server with a serial or
network interface to a PC that maintains access
 Alternatively, many newer terminal servers support
direct network connections via SSH, with RADIUS
support and IP filtering
 Can also connect out-of-band via dial-up modem with
callback feature
Remote Administration Solutions
Console access
 Advantages:
– Console messages can be logged to a terminal server
– Central point of authentication into console
management
– Provides the ability to turn off telnet and other
administrative clear-text protocols on network
equipment
– If ssh or other interactive interface fails to respond,
administrator can directly connect to console without
physically going to the DMZ
Remote Administration Solutions
Console access
 Disadvantages:
– The unintentional <BREAK> problem
– Additional hardware and cabling
– Authentication and logging for console use (once
the user has accessed the terminal server) is
difficult to implement with a hardware device
Remote Administration Solutions
SSH bastion gateway
 One (hardened) point of entry via SSH to
other hosts
 Can use ssh-agent to eliminate interactivity
on the gateway, while maintaining only a
single host that can SSH to the endpoints
 Use RSA identity files
 Disable password authentication
 Disable rhosts authentication and root login
 Bind ssh only to admin LAN interface
 Watch your patch levels – ssh is a popular
target
Remote Administration Solutions
Windows GUI – 2 popular options:
– PCAnywhere
– Windows Terminal Services
Remote Administration Solutions
Windows GUI – PCAnywhere
 Risks
– Runs on well-known port – juicy target for attackers
– Previous versions have been vulnerable to DoS
attacks and weak password encryption
– Typical configuration binds to all interfaces
– Should avoid exposing on an untrusted network
segment
– Typical configuration bypasses Windows login
mechanism
Remote Administration Solutions
Windows GUI – PCAnywhere
 Securing PCAnywhere
–
–
–
–
–
–
–
–
–
–
–
–
Make use of the allowed IP addresses feature – limit admin hosts
Enable TCPIPHostBindMode to only listen on admin interface
Change default port
Make sure the Windows NT user is logged off after session
disconnect (normal and abnormal)
Enable event logging and session recording (if disk space permits)
Utilize Symmetric encryption / Deny lower-level
If possible, use X.509 for host authentication
Disable response to PCAnywhere query broadcasts
Configure clients to only use TCP to connect (rather than a UDP
query – reduces firewall ruleset)
Use separate user account for each admin with strong passwords
Limit login attempts
Only use PCAnywhere user with PCAnywhere privileges
Remote Administration Solutions
Windows GUI – Terminal Services
 Risks
– Utilizes Windows authentication method
– Runs on a well-known port
– Should avoid exposing on an untrusted
network segment
Remote Administration Solutions
Windows GUI – Terminal Services
 Securing WTS (for administration use)
– Bind only to the administrative segment interface
– Force all configuration parameters at the server level
– Use a separate WTS login from Windows login and give each
administrator unique login with strong password
– Take Administrators group out of connection permissions
– Enable security auditing
– Remove TsInternetUser account
– Utilize High Encryption for RDP
– Disconnect idle/broken connections aggressively
– For those who are paranoid, change the WTS port.
Logging solutions
Log types
Type
Mode
Syslog – UNIX and
Network Devices
Write to local filesystem or send
over network (UDP 514)
Windows NT/2000
Event Logs
Write to local filesystem (network
support for syslog available from 3rd parties)
Application / Service Syslog, NT Event Log, flat file, binary
Log Files
file, database entry
The system management need is to centralize
logs for analysis.
Logging solutions
Network syslog
 If possible, limit which machines can send log
entries to a host.
 Heartbeat creation and detection is absolutely
imperative.
 Flood detection is also imperative.
 Syslog servers should sit on administrative LANs if
at all possible.
 Make sure that clients are sending the messages
over the administrative LAN interface.
 Initiatives are out there for secure syslog – not close
to implementation yet:
– Log signing
– Encrypted transfer
– Insertion / deletion attacks
 Take a look at syslog-ng. http://www.balabit.hu
Logging solutions
NT/2000 Event Log
 Need to get those logs off each server posthaste.
 Two major options:
– Agent-based forwarding
• Syslog
• Commercial solutions
– Batch retrieval
• Can use common resource kit utilities to pull logs out in binary
and text format
• To push or to pull?
• If log is cleared by an intruder, you better know about it! (Use a
perl script to check for Event ID 517.)
 See my SANS NetSec 2000 presentation for many more
details!
Logging solutions
Flat file logs
 Can always “syslog” them
– tail -f /var/log/mylog | logger
– Ok… maybe not! 
 Need to get them off of the originator as soon as
possible.
 If they are too big and/or cumbersome, consider
culling them during the push or pull process.
 How often to push/pull? Determine the criticality of
the logs and analyze the worst case scenario: where
the attacker blows away the local copy of the log file
and your mission is to figure out what happened.
 When log disappears, you better know about it!
General tips
Watch out for administrative interfaces.
Follow best practices, especially in regards
to:
Resource utilization
Segmentation
Authentication
Integrity