Centralized Logging: Why Would I Want to Do That? Garrett Lanzy Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/ Outline • Problems • Centralized Logging at Metropolitan State – Architecture and Configuration – Other Considerations • • • • Real-Life Scenario – How Does Logging Help? Future Directions Conclusions Q&A Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/ Problems: Logging Issues • Logs spread across servers and network devices • Different devices use different log formats • Default settings often omit important information • Log data retention varies • An “event” can involve numerous different devices – hard to search/correlate Key Questions • • • • What should be logged? How can this data be gathered? How long should this data be retained? How can searching be simplified? An Answer: Centralized Logging Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/ Centralized Logging at Metropolitan State University • Single central logging server (axe) – Linux (SUSE Linux Enterprise Server) – syslog-ng logging daemon – 30GB partition for logs • Stores 65 days worth of logs for 60+ devices (including firewall logs for 2 campuses) – Physical server – Same server used for network monitoring (Nagios) What is syslog? • Logging architecture for Unix/Linux • Developed since 1980s • Provides services for local & centralized logging (single or multi-tier) • Communication protocol standardized by IETF – RFC 3164 - “classic” BSD Unix syslog (UDP only) – RFC 5424 – TLS transport support – Neither defines format of log messages What is syslog-ng? • • • • • “Next generation” syslog Open-source & commercial versions Supports UDP, TCP, and TLS transports Flexible configuration options Many options for storing log data – Simple files – Structured files (CSV, etc) – Database interfaces syslog Message Facilities • syslog defines 24 types of sources for logs (called facilities) – common ones: – – – – – – – kernel – messages from OS kernel user – user-level messages mail – e-mail related messages system – messages from system daemons/services auth – security/authorization messages syslog – messages generated by syslog itself local0 – local7 – implementation-defined message • Metro design creates separate files per facility for each device syslog Message Severities • Each message assigned a severity by creator • Can be used to help filter by importance Severity Description 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages Metro Centralized Logging Design • Based on Novell Cool Solutions article Centralized Syslogging with Syslog-NG on SUSE Linux by Scott Flowers http://www.novell.com/coolsolutions/feature/18044.html • Supports classic syslog on UDP port 514 & syslog over TCP using TCP port 5140 • Simpler directory structure for easier searches • Added automated compression via logrotate Logging server syslog-ng config • Default SuSE configuraton contains logging source src for messages originating on local server • Metro design adds logging source nethosts for logs coming form other servers • Adds logging destination byhost which writes logs for each host to a specific directory /var/log/HOSTS/hostname • Log files named facilityname.log • Local logs for central logging server not changed syslog-ng.conf for nethosts source # define source for messages from network hosts Source nethosts { # # process log messages from network: # udp(ip(“0.0.0.0") port(514)); tcp(ip(“0.0.0.0") port(5140) keep-alive(yes) max_connections(100)); }; syslog-ng.conf for byhost destination # separate out hosts into individual log directories destination byhost { file("/var/log/HOSTS/$HOST/$FACILITY.log" owner(root) group(root) perm(0640) dir_perm(0750) create_dirs(yes) ); }; log { source(src); source(nethosts); destination(byhost); }; Configuring other devices/servers • Use TCP over UDP when possible – Exception: Cisco ASA devices stop forwarding if syslog over TCP configured & not working • Check server/device logging settings to make sure appropriate information is logged – Example: some servers only log failed login attempts, best to also log successful ones • Make sure to adjust firewall configuration on centralized logging server Configuring Linux Server - syslog • syslog always sends via UDP (default port 514) • Sample config to send messages for all facilities of severity informational or higher to central log server – add lines to syslog.conf: # send all logs of severity info or higher to logging server *.info @x.x.x.x Configuring Linux Server - syslog-ng • Define destination loghost for TCP port 5140 on logging server, send all messages there • Add following lines to syslog-ng configuration: destination loghost { tcp("x.x.x.x" port(5140)); }; log { source(src); destination(loghost); }; Configuring Network Devices • Most managed network devices support logging to central syslog server • Many configurable via web/GUI interfaces • Sample Cisco IOS configuration: Core-6500#conf term Enter configuration commands, one per line. End with CNTL/Z. Core-6500(config)#logging source-interface Vlan1 Core-6500(config)#logging x.x.x.x Core-6500(config)#end Configuring ASA Firewall Logging • MnSCU-owned ASA firewalls can be configured to log to campus logging servers contact MnSCU network support team: hostmaster@mnscu.edu • For campus-owned devices, easy to set up with Cisco ASDM: – Configuration -> Device Management -> Syslog Servers – Configuration -> Device Management -> Syslog Setup Configuring ASA Firewall Logging • MnSCU-owned ASA firewalls can be configured to log to campus logging servers contact MnSCU network support team: hostmaster@mnscu.edu • For campus-owned devices, easy to set up with Cisco ASDM: – Configuration -> Device Management -> Syslog Servers – Configuration -> Device Management -> Syslog Setup Configuring Cisco ACS Logging • Can configure logging for each “report name” • To start: from web admin interface, select System Configuration -> Logging • Select Configure link for syslog column of desired “report” • Click Enable Logging checkbox • Scroll down, enter IP address of server, set port to 514, and max bytes to 1024 • Submit Sample ACS Config (pt. 1) Sample ACS Config (pt. 2) Sample ACS Config (pt. 3) Windows Servers • Windows event logging traditionally has had no support for centralized logging • Several add-on products available to send Windows events to syslog server – Snare for Windows – open-source product • Microsoft has added capability for centralized event collection in Windows 2008 – for info, see: http://www.windowsecurity.com/articles/Centralized-Auditing-here-FREE.html Snare Agent for Windows • Snare Agent for Windows is open-source, free software (GNU Public License) • Available at: http://www.intersectalliance.com/projects/SnareWindows/ • 2 versions: – Snare for Windows: supports NT, 2000, XP, 2003 – Snare for Windows Vista: supports Vista, 2008, Windows 7 • Either will run on 32 or 64-bit installations Configuring Snare Agent Windows Event Log Settings • Security audit settings – clean Win2008 install A better idea? Other Considerations • • • • Clock synchronization Log rotation/space utilization Backups Network Monitoring Clock Synchronization • System/device clocks need to be synchronized in order to correlate log data • Any state/MnSCU router can be used as an NTP time source • Most network devices can sync to NTP • Windows standalone servers or domain controllers should be synchronized to NTP http://support.microsoft.com/kb/816042 Linux NTP Configuration • Linux VMs: try kernel option clock=pit • Remove any references to local unsynchronized clock (127.0.0.1) • Simple sample ntp.conf driftfile /var/lib/ntp/drift/ntp.drift # path for drift file logfile /var/log/ntp server 172.16.255.238 server 156.98.1.1 Log Rotation • logrotate – Linux utility to rotate and compress log files • Creates new log files when run and deletes after specified interval • Metro configuration: – Rotate files daily – Delay compression for 1 day – Keep 65 days worth of logs Log server logrotate.conf additions # rotate logs in /var/logs/HOSTS tree /var/log/HOSTS/*/*.log { daily compress delaycompress dateext missingok notifempty rotate 65 maxage 90 sharedscripts create 640 root root postrotate /etc/init.d/syslog reload endscript } Real-Life Scenario – Have your received one of these? > Subject: Notice ID: 22-79162500 Notice of Unauthorized Use of Paramount Pictures Corporation Property > Date: Fri, 23 Apr 2010 10:07:58 -0700 > From: paramount-no-reply@copyright-compliance.com > Reply-To: paramount-no-reply@copyright-compliance.com > To: abuse@mnscu.edu > -----BEGIN PGP SIGNED MESSAGE----> Hash: SHA1 > Notice ID: 22-79162500 > Notice Date: 23 Apr 2010 16:58:09 GMT > Minnesota State University System > Dear Sir or Madam: > > BayTSP, Inc. ("BayTSP") swears under penalty of perjury that Paramount > Pictures Corporation ("Paramount") has authorized BayTSPto act as > its non-exclusive agent for copyright infringement notification. > BayTSP'ssearch of the protocol listed below has detected > infringements of Paramount's copyright interests on your IP > addresses as detailed in the below report. … And to the Relevant Information: > Evidentiary Information: > Notice ID: 22-79162500 > Initial Infringement Timestamp: 23 Apr 2010 16:23:27 GMT > Recent Infringement Timestamp: 23 Apr 2010 16:23:27 GMT > Infringers IP Address: 199.17.228.252 > Protocol: BitTorrent > Infringed Work: She's Out of My League > Infringing File Name: She's Out of My League 2010 CAM-lu > Infringing File Size: 608227413 > Bay ID: d116ad77ba844d11a350055c9227e2b0d12a29d9|608227413 > Port ID: 45739 > Infringer's DNS Name: > Infringer's User Name: How to proceed? • 199.17.228.252 is a NAT address for the student wireless network on the Midway campus • Search the Midway ASA firewall log to determine the local IP address • Then search the ACS appliance log to find the user logged in …Um, but wait…. • Logs older than 1 day old are compressed, so need to decompress before they can be searched axe:~ # cd /var/log/HOSTS/mwpix/ axe:/var/log/HOSTS/mwpix # bzcat local4.log-20100423.bz2 >/tmp/mwpix.log axe:/var/log/HOSTS/mwpix # cd /var/log/HOSTS/172.16.210.49/ axe:/var/log/HOSTS/172.16.210.49 # bzcat auth.log-20100423.bz2 >/tmp/acs.log axe:/var/log/HOSTS/172.16.210.49 # What does the ASA log look like? Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic TCP translation from inside:10.15.38.69/49523 to outside:199.17.228.252/20192 Apr 23 09:00:00 mwpix %ASA-5-304001: 10.15.38.69 Accessed URL 65.55.17.27:http://www.msn.com/ Apr 23 09:00:00 mwpix %ASA-6-302020: Built inbound ICMP connection for faddr 207.171.116.193/0 gaddr 199.17.231.41/0 laddr 199.17.231.41/0 Apr 23 09:00:00 mwpix %ASA-6-302021: Teardown ICMP connection for faddr 207.171.116.193/0 gaddr 199.17.231.41/0 laddr 199.17.231.41/0 Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic UDP translation from inside:10.15.38.67/62560 to outside:199.17.228.252/49397 Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic UDP translation from inside:10.15.38.67/60379 to outside:199.17.228.252/41373 Apr 23 09:00:00 mwpix %ASA-4-313005: No matching connection for ICMP error message: icmp src inside:10.15.38.67 dst outside:199.17.241.241 (type 3, code 3) o n inside interface. Original IP payload: udp src 199.17.241.241/53 dst 10.15.38.67/60379. Apr 23 09:00:00 mwpix %ASA-6-305011: Built dynamic UDP translation from inside:10.15.35.98/50165 to outside:199.17.228.252/38489 Apr 23 09:00:00 mwpix %ASA-7-609001: Built local-host outside:74.63.145.159 We’re looking for…. • … a line that contains: 199.17.228.252/45739 • … at about 16:23:27 GMT (11:23:27 CDT) grep is your friend! • grep – utility to search by regular expression • Some basic regular expression syntax: – Most characters “match” themselves – Exceptions: [\^$.|?*+() – \ is an escape character; meaning varies by context, always suppresses special meaning of [\^$.|?*+() – . matches any single character (except the line break character) – * repeats the previous item 0 or more times – …. So .* matches “anything” http://www.regular-expressions.info/reference.html grep in action axe:/tmp # grep 199.17.228.252/45739 mwpix.log Apr 23 08:49:17 mwpix %ASA-6-305011: Built dynamic TCP translation from inside:10.15.34.45/1701 to outside:199.17.228.252/45739 Apr 23 08:49:47 mwpix %ASA-6-305012: Teardown dynamic TCP translation from inside:10.15.34.45/1701 to outside:199.17.228.252/45739 duration 0:00:30 Apr 23 11:26:23 mwpix %ASA-6-305012: Teardown dynamic TCP translation from inside:10.15.38.87/50372 to outside:199.17.228.252/45739 duration 0:03:00 Looks like we have a match - a connection of duration 3 min. which ended at 11:26:23 – i.e., was initiated at 11:23:23 (vs. 11:23:27) Now check the ACS log (note: ACS timestamps are GMT, not local time) axe:/tmp # grep 10.15.38.87 acs.log Apr 23 16:19:06 172.16.210.49 CisACS_01_PassedAuth y2gjm36g 1 0 Message-Type=Authen OK,User-Name=vvvvvv,NAS-IPAddress=172.16.212.7,Caller-ID=10.15.38.87,NAS-Port=vvvvvv,GroupName=Default Group,Filter Information=No Filters activated., Apr 23 16:19:06 172.16.210.49 CisACS_03_RADIUSAcc y2gjm36h 1 0 UserName=vvvvvv,NAS-IP-Address=172.16.212.7,NAS-Port=29,GroupName=Default Group,Framed-IP-Address=10.15.38.87,Calling-StationId=10.15.38.87,Acct-Status-Type=Start,Acct-SessionId=4bd1c873/00:24:2b:ae:79:c1/38738, Apr 23 18:05:44 172.16.210.49 CisACS_03_RADIUSAcc xjgh1n40 1 0 UserName=vvvvvv,NAS-IP-Address=172.16.212.7,NAS-Port=29,GroupName=Default Group,Framed-IP-Address=10.15.38.87,Calling-StationId=10.15.38.87,Acct-Status-Type=Stop,Acct-InputOctets=495945068,Acct-Output-Octets=1553689408,Acct-SessionId=4bd1c873/00:24:2b:ae:79:c1/38738,Acct-Session-Time=6398,AcctInput-Packets=1155133,Acct-Output-Packets=1692907, axe:/tmp # Determination • User vvvvvv made the connection outlined in the e-mail and downloaded 1.5GB – this case will be handled as inappropriate use • Actual time to perform the log search process (including reading the e-mail, logging in to the logging server, decompressing the logs, and searching): less than 5 minutes Future Directions • Centralized logging for Windows workstations • Use TLS connections instead of TCP – Makes snooping much more difficult – Prevents insertion of “bogus” logs • Automated log monitoring/proactive notification • Log to database instead of stream files? – Disk space is cheap, right? Conclusions • Centralized logging is not that difficult to set up • Significant benefits for both normal operations and incident handling • Can save large amounts of time for network/server/security administrators Questions? Garrett Lanzy Through 5/11/10: Garrett.Lanzy@metrostate.edu 5/12/10 & After: Garrett.Lanzy@csu.mnscu.edu Access more information security training for campus technical staff and earn CEUs: its.mnscu.edu/security/training/