2-UNIX LIVE RESPONSE
John P. Abraham
Professor
University of Texas Pan American
UNIX LIVE RESPONSE








Almost identical to windows
The only difference will be the commands that we’ll use to acquire the
evidence.
We will be running the same netcat commands to transfer the live
Response to the forensic workstation.
First, on the forensic workstation, we will run the following:
 Nc –v –l –p 10000 > command.txt
Now, any data sent to TCP port 10,000 on our forensic workstation
will be saved to command.txt.
On the victim computer, you will want to run a command to collect
Live Response data.
The output of the command will be sent over our TCP channel on port
10,000 in this case.
 Command | nc forensic_workstation_ip_address 10000
After these commands have completed, you will press ctrl-c to break
the netcat session, and our file command.txt will contain all the
data from the command we executed. A simple MD5 checksum of
command.txt can be calculated so that you may prove it s
authenticity.
ANALYZING VOLATILE DATA




The System Date and Time
 Date command
Current Network Connections
 Netstat –an
 We find an intrusion on Port 515 which is the port on
which the printerdaemon typically listens.
 We go to http://www.securityfocus.com to find that this
is a vulnerable TCP port. This may have been the
initial point of intrusion.
Open TCP or UDP Ports
 Netstat –an command
 Through examining the open ports, we hope to discover
any backdoors the attacker may have established.
Netstat –anp will list the process number that opened the
port also. This only works with Linux and will not work on
UNIX
EXECUTABLES OPENING TCP OR UDP
PORTS






IN UNIX we use lsof for list open files. It displays the processes that open
ports, but also lists the files that the process has open. It is the single most
powerful tool in our live response toolkit for UNIX systems.
Lsof –n produces lengthy output
The fourth bolded line on page 52 shows that file descriptor #2 (aka the
standard out) is redirected to /dev/null. This indicates that the errors from this
program will not be sent to the console but instead will be thrown away.
The next bolded line shows that the “printer” port is involved in an active
connection for this process.
The last two lines show an open TCP port (3287) and an established
connection. It would be pretty safe to say at this point that the IP address of
94.90.84.93 is probably the intruder’s origin.
Because we brought a trusted copy of lsof from an uncompromised machine,
we have to surmise that the mechanism the intruder used to hide the open port
must exist at the kernel level in addition; we will see that /tmp/.kde does not
show up on the file system when we list the time and date stamps. That
solidifies our theory that the kernel is probably trojaned, possibly with a
loadable kernel module (LKM). A good resource:

Malware: Fighting Malicious Code
CONTINUED..

Running processes
 To view the open processes:



Ps –aux
This command lists all the running processes on the system. And the users
running them.
Open Files
 Use


Lsof
To view open files
Once we run on our victim machine, we realize that the first
process has a current working directory of /tmp/.kde. The second
process has a current working directory of /tmp/.kde/brute/john1.6/run. This is not good! Someone may be running John the
Ripper on BRJDEF. John the Ripper is a common password
cracking program that attackers employ to learn users’
passwords.
The Internal Routing Table
 We obtain the routing table




Netstat –rn
In this case, the routing table looks intact.
CONTINUED..


Loaded Kernel Modules
 Because we suspect that the kernel may have been
trojaned, we will want to review all the loaded kernel
modules. This is done with the lsmod command. When
we dump the loaded kernel modules, we see the
information on page 57.
 There exist ways to hide a module after it is loaded.
After a module is hidden there is no way (in the Live
Response process) we can detect that it is installed.
Furthermore, when we investigate Phrack magazine, we
see there are other ways to infect the kernel without
even loading a module.
Mounted File Systems
 The mounted file systems can be obtained from issuing
either the mount command or the df command.

Output of the df command on bottom of page 57.
ANALYZING NONVOLATILE DATA

we would like to obtain several key pieces of information
while the machine is still running. The type of data we
will discuss in this section is nonvolatile. This means we
could also retrieve this information from a forensic
duplication if we so desired, but that option may be
difficult or impossible to achieve. Some of the information
we would like to acquire is as follows:









system version and patch level
file system time and date stamps
file system MD5 checksum values
Users concurrently logged on
A history of logins
Syslog logs
User accounts
User history files
Suspicious files
System Version and Patch Panel
 When we issue the command uname –a we
receive all the available operating system
version information.
 The patch level information for a Linux
machine is a little more difficult to retrieve
because it is dependent on the Linux
distribution used to create the system.
 In this case, it is RedHat 7.0 so we issue the
rpm –qa command
 File System Time and date stamps.
 File system time and date stamps can be
obtained using the same command for
windows

CONTINUED..

The following command we can print out the file
permissions, last access date, last access time,
modification date, modification time, inode change date,
inode change time, user ownership, group ownership,
file size, and full path of the file for the compete file
system:


Find / -printf “%m;%Ax;%AT; %Tx;%TT;%Cx;%CT;%U;%G;%s;%p\n”
We will sort all the files by the time the inode was last
changed (also known as the “ctime”). We do this
because there is no creation date on Unix file systems.
The change date is as close as we can get to the creation
time. After ruling out the other files changed on
September 8, 2003, we have subset of files that we
cannot explain. Pg 60.
It seems as if someone installed the Net::Telnet module.
Perhaps this was a dependency to one of the tools that the
intruder ran on our system.
 We also see that the /etc/passwd/ and /etc/shadow were
altered. This is usually done when an intruder adds a
backdoor in the form of a valid user on the victim machine.
 The other thing we see is two versions of lpd. The first
instance in the table is probably the real lpd program.

FILE SYSTEM MD5 CHECKSUM VALUES





MD5 Checksum- is a 128-bit mathematical
fingerprint of the contents in a file, for every
file on the file system.
After we calculate the MD5s, we can compare
them to databases of known MD5s.
www.hashkeeper.org
There is also the NSRL software reference
library. These databases will help us eliminate
the known files and leave us with only
unknown files to examine.
To calculate the MD5 checksum for every file
on the system, you would issue the following
command:
Find / -type f –xdev –exec md5sum –b {} \;
 The results for this sweep are on the DVD.

USERS CURRENTLY LOGGED ON
The users who are currently logged on are saved in
the /var/run/utmp log. We could reconstruct this
information from a forensic duplication, but it is
much easier to acquire this information with the w
command. Output on page 62.
 We see that root, Curtis, and lpd are all logged in.
the most important line that should be evident her
is the lpd login lp is a valid system account, and
perhaps the attacker hooped that lpd would blend
in.
 We also see that the attacker’s IP address is
94.90.84.93.
 Even though these logs are in binary format, the
format is well known. If the attacker chose to run
a program called zap2 he could have easily zapped
the lpd login that we see here.

A HISTORY OF LOGINS
A history of logins is saved in the /var/log/wtmp binary log, so we
can reconstruct this information form a forensic duplication. We
can view this information easily during our Live Response.
Syslog logs
 Unix systems have a syslog facility
 The syslog daemon listens for messages from either local
programs or other servers on the Internet and logs them according
to the /etc/syslog.conf configuration file.
 The only logs that will be relevant to our investigation are
/var/log/messages and /var/log/secure.
 Using netcat we transfer relevant logs to our forensic
workstation for further analysis.
 In the messages log, we see that on September 8, at
approximately 14:35, a series of events begins that does not make
much sense (and that is difficult to even copy to this page). The
message looks like binary data written to the log. This is typically
an indication of a buffer overflow attack. When buffer overflows
occur, they break valid programs. When programs break, garbage
is typically generated in the log. In this scenario, we cannot see
what program was attached. Other overflows will report the
name of the service that wrote to the log, such as the rpc.statd
buffer overflow, which reports that rpc.statd wrote to the log.
 On the log we notice that kjones failed at login and then
successfully log in as matt shortly after. Dead end as explained
on top of page 66.




User Accounts (page 66)
 We know the attacker is creating backdoors on BRJDEV
at this point. Let us examine the /etc/passwd file to see
whether the intruder has added any rogue user
accounts.
 Once we see it, the normal system and user accounts
make up every line except the last. We see that there e
is a new account, called lpd. It has a root directory of/
and, more importantly, a user ID of zero. This means
that when lpd logs in, he will have root privileges, even
though he will have a different password than root.
User History Files
 History files are commands the user typed at the
prompt. The files may contain commands that failed, or
we may discover the hacker’s methodology. The one
account we know the attacker used was the lpd account.
When we examine lpd’s home directory (/), we do not
find a history file. We also know that the attacker used
the Richard account because it initiated the
investigation. We do, however, find a bsh history file for
Richard. Pg 67. The bolded lines show activity
that Richard did not recognize.
SUSPICIOUS FILES

If we were not acquiring a forensic duplication of BRJDEV, we
could transfer any suspicions file with our netcat.


Because BRJDEV’s kernel was probably trojaned, we would
collect any of the suspicions files on the file system through the
forensic duplication process to make sure we have a pristine copy
of the tools.
 There is a way we can acquire copies of running processes. In
windows, an executable cannot be deleted while it is running in
memory. The operating system locks the file and it cannot be
removed. That is a benefit for us during an investigation. In
Unix, an attacker can run a file, such as datapipe, and delete the
original binary. If we were to shut down the machine in a
graceful manner, we would not have a copy of this file for analysis
later. This is where the /proc file system comes into play
The /proc file system does not actually exist on the hard drive. It
exists in memory and references running processes and other system
information. By entering the /proc directory, you see directories
named after integers, such as 1348.


Nc –v –l –p 10000 > filename
Cat filename | nc forensic_workstation_ip_adress 10000