“Rooting Out” Rootkits David Taylor & John Lupton ISC Information Security Security-SIG, 15 December 2005 ISC/Information Security rootkit: (n) • A collection of software “tools” - utilities, scripts, data files, etc. • Installed on a target computer following compromise (usually remote, but locally possible as well) • Used not only for operations on that machine, but also as a “stash” to retrieve when breaking into other machines Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu rootkit (n): (cont.) • Originally a (mostly) Unix/Linux threat – “rooted” (recompiled) versions of common utilities, e.g. ls, ps, netstat – Re-written to hide presence and activity of other rootkit files – Usually cleverly hidden in file system • Windows rootkits have (surprise!) become much more common in recent years – Structure and operation different than U/L rootkits, but still do essesntially the same sorts of things: Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu rootkit activity • • • • • • Hide files, processes, network connections Wipe logs (“cover your tracks”) Install backdoors Sniff networks Replace binaries and executables And??…Whatever else the attackers wants! Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu rootkit evolution • As operating system kernels have evolved, so have the ways rootkits are written to take advantage – Linux: LKM’s (Loadable Kernel Modules) – A compromised kernel means the machine is “0wn3d” to its very foundation • Windows rootkits often create and install a specialized system driver and configuration files that access API “hooks” allowing attacker to name the process and determine where and how to hide it. Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Example: Hacker Defender Typical configuration file for Hacker Defender: [H<<<idden T>>a/"ble] h"xdef"* r|c<md\.ex<e:: /[/H/idd\en Ser:vi"ces]Ha>:ck"er//Def\ender *[Set/tin/:\gs] / P:assw\ord=hxdef-rulez Ba:ckd:"oor"Shell=hxdef$.exe Fil:eMappin\gN/ame=_.-=[Hacker Defender]=-._ Serv:iceName=HackerDefender100 Se|rvi:ceDisp<://la"yName=HXD Service 100 Ser>vic:eD||escr<ip:t"ion=powerful NT rootkit Dri<ve\rN:ame=HackerDefenderDrv100 D:riv>erFileNam/e=hxdefdrv.sys Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu rootkit vs. anti-virus • Sometimes, if the hacker is careless, the rootkit will be caught and quarantined by anti-virus software • For a knowledgeable hacker, this is only a temporary setback – Can usually turn A-V on and off at will – Don’t think deleting it out of quarantine will solve the problem - ”I’ll be back…” (The Terminator) Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Rootkit Detectors • Like anti-virus software and firewalls…useful and effective up to a point • Can detect many well-known, widely distrubuted rootkits • Many rootkits are known only to one person - the one who wrote it Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Rootkit Detectors: Dave T’s Picks Blacklight (Free Beta) http://www.f-secure.com/blacklight/ Free RootkitRevealer http://www.sysinternals.com/Utilities/RootkitRevealer.html Rkdetector http://www.rkdetector.com/ UnHackMe http://www.greatis.com/unhackme/ Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu “Be vewwy, vewwy quiet…I’m hunting wootkits” • Before you start: – Have on hand statically re-compiled versions of common operating system utilities • If you suspect the presence of a rootkit, you cannot trust any element of the file system on the machine • You might also want to check the MD5 hash of your “trusted” copies against a pre-written list and/or known good copies on other machines • Keep these trusted utilities on CD-ROM, stored in a secure place until needed – Decide whether this is a “live” patient or an “autopsy” Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu “Is it live, or is it Memorex?…” • On a live system, you can check: – Active processes – Open files – Changes in file sizes, attributes and access times – Active network connections – Sniff the network for traffic to and from the “patient” Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu “Autopsy” • If circumstances dictate the system be taken down and rebuilt immediately – Use dd or similar utility to make image file – rootkit presence can still be found by examining files, directories, attributes, “metadata”, etc – Can be done post facto, at leisure Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Document what you do • Whether “live” or “autopsy” mode, keep a log of what you do, when you do it and what you find – May come in handy if situation arises again – You may find evidence of a crime • Might not even relate to rootkit, e.g. presence of child pornography Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Check available logs • • • logins, ssh, sendmail, ftp, http, etc… On target system, are likely to be wiped, but not always Many systems configured to use remote logging utilities – • “wiped” logs may exist elsewhere Look for anomalies, e.g.: – user ‘davet’ shows up running ftp sessions, and you know he: a) Doesn’t know what ftp is b) Isn’t smart enough to use it if he did c) Is dead, and you forgot to delete the account Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Check cron jobs • cron runs processes, programs, scripts, etc. at predetermined times and intervals • Similar to Scheduled Tasks in Windows • Typical location is /var/spool/cron • Anything there that looks unfamiliar or suspicious? Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Use ps -auxww or -elf to see what processes are running [root@dobro bin]# ps -elf F S UID PID PPID 4 S root 1 0 1 S root 2 1 1 S root 3 1 1 S root 4 3 1 S root 5 3 1 S root 34 3 1 S root 35 1 1 S root 46 3 1 S root 47 3 1 S root 49 3 1 S root 48 1 1 S root 122 1 1 S root 192 3 1 S root 194 1 1 S root 195 1 1 S root 208 3 1 S root 209 3 1 S root 217 1 1 S root 1134 1 0 S root 1141 1 1 S root 1569 1 1 S root 2009 3 1 S root 2060 1 1 S root 2563 1 5 S root 2567 1 5 S rpc 2586 1 5 S rpcuser 2606 1 1 S root 2639 1 1 S root 2710 1 5 S ntp 2804 1 Rooting Out Rootkits 15 December 2005 C PRI NI ADDR SZ 0 76 0 - 1187 0 94 19 0 0 65 -10 0 0 71 -10 0 0 74 -10 0 0 65 -10 0 0 75 0 0 0 75 0 0 0 75 0 0 0 67 -10 0 0 75 0 0 0 84 0 0 0 65 -10 0 0 85 0 0 0 85 0 0 0 66 -10 0 0 66 -10 0 0 75 0 0 0 75 0 0 0 69 -10 900 0 76 0 0 0 66 -10 0 0 75 0 0 0 76 0 906 0 76 0 633 0 75 0 - 1186 0 81 0 - 1449 0 76 0 - 4963 0 79 0 634 0 76 0 - 4638 WCHAN ksofti worker worker worker worker hub_th pdflus pdflus worker kswapd serio_ worker worker worker kjourn kaudit kjourn syslog - STIME Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 Dec11 Nov15 Nov15 Nov15 Nov15 Nov15 Nov15 TTY ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? TIME 00:00:00 00:00:01 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:02 00:00:00 00:00:07 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:10 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:03 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 ISC/Information Security security@isc.upenn.edu CMD init [5] [ksoftirqd/0] [events/0] [khelper] [kacpid] [kblockd/0] [khubd] [pdflush] [pdflush] [aio/0] [kswapd0] [kseriod] [ata/0] [scsi_eh_0] [scsi_eh_1] [kmirrord] [kmir_mon] [kjournald] [khpsbpkt] udevd [knodemgrd_0] [kauditd] [kjournald] syslogd -m 0 klogd -x portmap rpc.statd rpc.idmapd /usr/sbin/acpid ntpd -u ntp:ntp -p lsof can show which files a process has open… [root@dobro bin]# lsof -p 2563 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME syslogd 2563 root cwd DIR 253,0 4096 2 / syslogd 2563 root rtd DIR 253,0 4096 2 / syslogd 2563 root txt REG 253,0 37992 5144639 /sbin/syslogd syslogd 2563 root mem REG 253,0 105080 42975258 /lib64/ld-2.3.4.so syslogd 2563 root mem REG 253,0 1489097 42975260 /lib64/tls/libc-2.3.4.so syslogd 2563 root mem REG 253,0 56791 42975257 /lib64/libnss_files-2.3.4.so syslogd 2563 root 0u unix 0x000001012afb0e00 5893 /dev/log syslogd 2563 root 2w REG 253,0 430682 30002610 /var/log/messages syslogd 2563 root 3w REG 253,0 0 30002360 /var/log/secure syslogd 2563 root 4w REG 253,0 1248 30002361 /var/log/maillog syslogd 2563 root 5w REG 253,0 190062 30002364 /var/log/cron syslogd 2563 root 6w REG 253,0 0 30002362 /var/log/spooler syslogd 2563 root 7w REG 253,0 0 30002363 /var/log/boot.log Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Check ifconfig In most installations, the running mode is normally MULTICAST… eth0 Link encap:Ethernet HWaddr 00:12:3F:64:7A:DA inet addr:192.168.0.2 Bcast:255.255.255.255 Mask:255.255.255.0 inet6 addr: fe80::212:3fff:fe64:7ada/64 Scope:Link UP BROADCAST RUNNING PROMISCUOUS MTU:1500 Metric:1 RX packets:247288 errors:0 dropped:0 overruns:0 frame:0 TX packets:382125 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:143347639 (136.7 MiB) TX bytes:59167774 (56.4 MiB) Base address:0xdcc0 Memory:dfee0000-dff00000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:8626 errors:0 dropped:0 overruns:0 frame:0 TX packets:8626 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:6840040 (6.5 MiB) TX bytes:6840040 (6.5 MiB) …there may be a sniffer running on eth0 Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Use netstat to check network connections [root@dobro bin]# netstat -atup Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address tcp 0 0 *:32769 *:* tcp 0 0 *:mysql *:* tcp 0 0 *:netbios-ssn *:* tcp 0 0 *:sunrpc *:* tcp 0 0 *:auth *:* tcp 0 0 localhost.localdomain:ipp *:* tcp 0 0 192.168.0.2:ipp *:* tcp 0 0 *:microsoft-ds *:* tcp 0 1728 192.168.0.2:ssh 24.168.97.666:35424 udp 0 0 *:32768 *:* udp 0 0 172.16.213.1:netbios-ns *:* udp 0 0 172.16.245.1:netbios-ns *:* udp 0 0 192.168.0.2:netbios-ns *:* udp 0 0 *:netbios-ns *:* udp 0 0 172.16.213.1:netbios-dgm *:* udp 0 0 172.16.245.1:netbios-dgm *:* udp 0 0 192.168.0.2:netbios-dgm *:* udp 0 0 *:netbios-dgm *:* udp 0 0 *:662 *:* udp 0 0 *:bootpc *:* udp 0 0 *:sunrpc *:* udp 0 0 *:ipp *:* udp 0 0 192.168.0.2:ntp *:* udp 0 0 localhost.localdomain:ntp *:* udp 0 0 *:ntp *:* raw 0 0 *:icmp *:* Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu State LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN LISTEN ESTABLISHED Inodes • Key part of file system “metadata” structure • Sequentially numbered “container” that contains file name, permissions, and location(s) in file system (i.e., disk) • Term “inode” most commonly applied to Unix/Linux, but same principle used in Windows/NTFS Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Inode numbering [root@dobro bin]# ls 16433200 -rwxr-xr-x 16433242 -rwxr-xr-x 16433221 -rwxr-xr-x 16433184 -rwxr-xr-x 16433231 -rwxr-xr-x -li l* 1 root 1 root 1 root 1 root 1 root [root@dobro bin]# ls 30409514 -rw-rw-r-30409513 -rw-rw-r-30409511 -rw-rw-r-30409274 -rw-rw-r-30409510 -rw-rw-r-30409327 -rw-rw-r-30408835 -rw-rw-r-- -li /home/lupton/*.* 1 lupton lupton 1 lupton lupton 1 lupton lupton 1 lupton lupton 1 lupton lupton 1 lupton lupton 1 lupton lupton Rooting Out Rootkits 15 December 2005 root 20088 Jun 20 root 31880 Jun 20 root 100952 Jun 15 root 28024 Sep 14 root 87608 Jun 20 987 581 1160 0 1746 4535 1656 07:45 07:45 2004 04:48 07:45 Dec Dec Dec Oct Dec Dec Nov link ln loadkeys login ls 14 14 14 31 14 14 21 12:16 12:16 12:15 11:52 12:10 11:46 10:49 /home/lupton/ifconfig.txt /home/lupton/ifconfig.txt~ /home/lupton/lsof-p2563.txt /home/lupton/mandolin.iso /home/lupton/netstat-a--inet.txt /home/lupton/pself.txt /home/lupton/upd.txt ISC/Information Security security@isc.upenn.edu Inode behavior • Inode number remains same when file is edited • If file is deleted, and new file with same name is written to disk, will usually retain Inode number • Inode number changes when file is replaced or overwritten by new file Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu MAC Times • “MAC”: “Modified/Accessed/Changed” • “M-time”: date/time file contents last modified • “A-time”: date/time file was last accessed • “C-time”: date/time inode information last changed (chmod, new blocks written, defragmentation, etc.) Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu MAC Implications • Hackers like to use rootkits to hide things AND plant “rooted” versions of standard binaries • If the M- or C-times of standard utilities (e.g. ls, ps, netstat) have been altered, it may indicate a bogus version • Similarly, if the inode number appears to have changed, it may be a “rooted” version Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Follow the changing inode… [root@dobro bin]# ls -li /home/lupton/inode_test.txt 30409516 -rw-rw-r-- 1 lupton lupton 77 Dec 14 13:24 /home/lupton/inode_test.txt [root@dobro bin]# cat /home/lupton/inode_test.txt This is the first file, before deletion. To be saved as "inode_test.txt"... [root@dobro bin]# rm /home/lupton/inode_test.txt rm: remove regular file `/home/lupton/inode_test.txt'? Y Now, I write and save a new file with the same name… [root@dobro bin]# ls -li /home/lupton/inode_test.txt 30409516 -rw-rw-r-- 1 lupton lupton 56 Dec 14 13:27 /home/lupton/inode_test.txt [root@dobro bin]# cat /home/lupton/inode_test.txt This is the second version, after deleting the first... Next, we overwrite the second version with another file… [root@dobro bin]# mv /home/lupton/inode_bogus.txt /home/lupton/inode_test.txt mv: overwrite `/home/lupton/inode_test.txt'? y [root@dobro bin]# ls -li /home/lupton/inode_test.txt 30409285 -rw-rw-r-- 1 lupton lupton 28 Dec 14 13:32 /home/lupton/inode_test.txt [root@dobro bin]# cat /home/lupton/inode_test.txt This is the "bogus" version [root@dobro bin]# Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu MAC Timeline • Almost always a major part of a full forensic examination • Correlates filenames and the dates/times their M, A and/or C were altered • Usually lengthy and time consuming to look through, but can often reveal exactly when and how a rootkit was installed Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Basic MAC evaluation using ls • List inode # with M-time: ls -li • List inode # with A-time: ls -luti • Long listing with C-time: ls -lci Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu ls -li, -luti, -lci [root@dobro bin]# ls 16433200 -rwxr-xr-x 16433242 -rwxr-xr-x 16433221 -rwxr-xr-x 16433184 -rwxr-xr-x 20767391 -rwxr-xr-x -li l* 1 root 1 root 1 root 1 root 1 root [root@dobro bin]# ls 16433231 -rwxr-xr-x 16433200 -rwxr-xr-x 16433242 -rwxr-xr-x 16433221 -rwxr-xr-x 16433184 -rwxr-xr-x -luti l* 1 root root 87608 Dec 14 13:20 ls 1 root root 20088 Dec 9 04:02 link 1 root root 31880 Dec 9 04:02 ln 1 root root 100952 Dec 9 04:02 loadkeys 1 root root 28024 Dec 9 04:02 login [root@dobro bin]# ls 16433200 -rwxr-xr-x 16433242 -rwxr-xr-x 16433221 -rwxr-xr-x 16433184 -rwxr-xr-x 16433231 -rwxr-xr-x -lci l* 1 root root 20088 Nov 1 root root 31880 Nov 1 root root 100952 Nov 1 root root 28024 Nov 1 root root 87608 Nov Rooting Out Rootkits 15 December 2005 root 20088 Jun root 31880 Jun root 100952 Jun root 28024 Sep root 90654 Aug 20 20 15 14 4 4 4 4 4 4 07:45 link 07:45 ln 2004 loadkeys 04:48 login 2005 ls 12:27 12:25 12:24 12:25 12:22 link ln loadkeys login ls ISC/Information Security security@isc.upenn.edu What’s wrong with this picture? [root@dobro bin]# ls -a . dd .. df .. dmesg alsaunmute dnsdomainname arch doexec ash domainname ash.static dumpkeys aumix-minimal echo awk ed Rooting Out Rootkits 15 December 2005 igawk ipcalc kbd_mode kill ksh link ln loadkeys login nisdomainname pgawk ping ping6 ps pwd red rm rmdir ISC/Information Security security@isc.upenn.edu tar tcsh touch tracepath tracepath6 traceroute traceroute6 true umount Take a closer look… [root@dobro]# ls -a . dd .. df .. dmesg igawk ipcalc kbd_mode nisdomainname pgawk ping How can there be two ‘..’ directories?… Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu tar tcsh touch How did this happen? [root@dobro bin]# mkdir ..\ This is actually: mkdir <space><dot><dot><backslash><space><enter> It creates a directory actually named “dot-dot-space” Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu What’s in this “mystery” directory? [root@dobro [root@dobro total 24 -rw-r--r--rw-r--r--rw-r--r--rw-r--r--rw-r--r--rw-r--r-- Rooting Out Rootkits 15 December 2005 bin]# cd ..\ .. ]# ls -l 1 1 1 1 1 1 root root root root root root root root root root root root 0 0 0 0 0 0 Dec Dec Dec Dec Dec Dec 15 15 15 15 15 15 12:19 12:19 12:19 12:19 12:19 12:19 ISC/Information Security security@isc.upenn.edu rootkit_file_01 rootkit_file_02 rootkit_file_03 rootkit_file_04 rootkit_file_05 rootkit_file_06 What happens when… [root@dobro bin]# mkdir ..\ \ \ \ \ (i.e., dot-dot with 5 spaces) Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu What does ls show? [root@forensic_laptop bin]# ls -a . date hostname .. dd igawk .. df ipcalc .. dmesg kbd_mode nice nisdomainname pgawk ping Yet another ‘..’ directory… Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu sync tar tcsh touch Anything in it? [root@dobro [root@dobro total 24 -rw-r--r--rw-r--r--rw-r--r--rw-r--r--rw-r--r--rw-r--r-- Rooting Out Rootkits 15 December 2005 bin]# cd ..\ \ \ \ \ .. ]# ls -l 1 1 1 1 1 1 root root root root root root root root root root root root 0 0 0 0 0 0 Dec Dec Dec Dec Dec Dec 15 15 15 15 15 15 12:23 12:23 12:23 12:23 12:23 12:23 ISC/Information Security security@isc.upenn.edu evilroot_01 evilroot_02 evilroot_03 evilroot_04 evilroot_05 evilroot_06 Where do you find rootkit files? • Sometimes, right under your nose – Watch out for “stupid tricks” with ‘..’ directories and filenames beginning with ‘.’ – Use both -l and -a flags when using ls • -l does not list ‘.’ and ‘..’ entries: [root@forensic_laptop bin]# ls -l total 6864 -rwxr-xr-x 1 root root 15528 Jul -rwxr-xr-x 1 root root 2812 Sep -rwxr-xr-x 1 root root 98356 Jun -rwxr-xr-x 1 root root 522116 Jun -rwxr-xr-x 1 root root 12964 Jun lrwxrwxrwx 1 root root 4 Oct -rwxr-xr-x 1 root root 13068 Jun Rooting Out Rootkits 15 December 2005 19 14 15 15 15 31 20 07:34 04:52 2004 2004 2004 13:10 07:52 alsaunmute arch ash ash.static aumix-minimal awk -> gawk basename ISC/Information Security security@isc.upenn.edu Look for things that are odd, out of place, or you don’t recognize •Hackers usually aren’t going to name the files “rootkit_01”, etc. •They have an entire file system to hide them in Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu OK, I found a rootkit - or I’m pretty sure there’s one in there somewhere… What do I do?? Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu OK, I found a rootkit - or I’m pretty sure there’s one in there somewhere… What do I do?? REBUILD!!! Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu You cannot trust a system that has a rootkit installed. Rebuilding is not just the best option… IT’S THE ONLY OPTION See: www.microsoft.com/technet/community/columns/secmgmt/sm0504.mspx By Jesper Johansson (Microsoft Security Manager) Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu Questions? Comments? Rooting Out Rootkits 15 December 2005 ISC/Information Security security@isc.upenn.edu