The Linux Audit
Framework
Gary Smith, EMSL, Pacific Northwest National
Laboratory
A Little Context
Cyber Security is all about managing risk.
How do you think about managing risk?
The Five Golden Principles of Security
Know your system
Principle of Least Privilege
Defense in Depth
Protection is key but detection is a must.
Know your enemy.
2
Introduction
Linux audit helps make your system more secure by
providing you with a means to analyze what is happening
on your system in great detail.
It does not, however, provide additional security itself—it
does not protect your system from code malfunctions or
any kind of exploits.
Instead, Audit is useful for tracking these issues and
helps you take additional security measures, like
SELinux, to prevent them.
Audit consists of several components, each contributing
crucial functionality to the overall framework.
The audit kernel module intercepts the system calls and
records the relevant events.
3
Introduction (cont.)
The auditd daemon writes the audit reports to disk.
Various command line utilities take care of displaying,
querying, and archiving the audit trail.
4
Linux Audit Framework Capabilities
Audit enables you to do the following:
Associate Users with Processes
Audit maps processes to the user ID that started them.
This makes it possible for the administrator or security officer
to exactly trace which user owns which process and is
potentially doing malicious operations on the system.
Review the Audit Trail
Linux audit provides tools that write the audit reports to disk
and translate them into human readable format.
Review Particular Audit Events
Audit provides a utility that allows you to filter the audit reports for
certain events of interest.
5
Linux Audit Framework Capabilities (1)
You can filter for:
User
Group
Audit ID
Remote Hostname
Remote Host Address
System Call
System Call Arguments
File
File Operations
Session
Success or Failure
6
Linux Audit Framework Capabilities (2)
Apply a Selective Audit
Audit provides the means to filter the audit reports for events of
interest and also to tune audit to record only selected events.
You can create your own set of rules and have the audit daemon
record only those of interest to you.
Prevent Audit Data Loss
Audit provides several mechanisms to prevent the loss of audit
data in the event of a loss of system resources.
7
The Components of Linux Audit
8
Configuring The Linux Audit Framework
Before you can actually start generating audit logs and
processing them, you must configure the audit framework.
Julius Caesar said, “Gallia est omnis divisa in tres partes”,
and just like Gaul, the configuring the audit framework is
divided into three parts:
The Audit Daemon Configuration
The Audit Rules
The Audispd Daemon Configuration
9
/etc/audit/auditd.conf
The /etc/audit/auditd.conf configuration file determines
how the audit system functions once the daemon has
been started.
For most use cases, the default settings shipped with the
package should suffice.
Let’s take a look at a sample auditd configuration file.
10
A Sample auditd.conf
log_file = /var/log/audit/audit.log
log_format = RAW
log_group = root
priority_boost = 4
flush = INCREMENTAL
freq = 20
num_logs = 5
dispatcher = /sbin/audispd
disp_qos = lossy
name_format = NONE
##name = mydomain
max_log_file = 6
max_log_file_action = ROTATE
space_left = 75
space_left_action = SYSLOG
action_mail_acct = root
admin_space_left = 50
admin_space_left_action = SUSPEND
disk_full_action = SUSPEND
11
Setting Up Audit Rules
We’ve given auditd its marching orders; now we have to
define what we are interested in auditing.
Audit rules are used to specify which components of your
system are audited.
There are three basic types of audit rules:
Basic audit system parameters
File and directory watches
System call audits
Before creating an audit rule set and before rolling it out
to your system, carefully determine which components to
audit.
Extensive auditing can cause a substantial logging load.
Remember: First match wins!
12
Setting Up Audit Rules (1)
Make sure that your system provides enough disk space
to store large audit logs and test your audit rule set
extensively before rolling it out to production.
Audit rules can either be passed to the audit system by
the command line using auditctl or bundled into a rules
file located under /etc/audit/audit.rules that is read
during the start of the audit daemon.
13
A Sample audit.rules
# basic audit system parameters
-D
-b 8192
-f 1
-e 1
# some file and directory watches
-w /etc/audit/auditd.conf -p rxwa
-w /etc/audit/audit.rules -p rxwa
-w /var/log/audit/
-w /etc/passwd -p rwxa
-w /sbin/auditctl –p x
# an example system call rule
-a entry,always -S umask
14
File Watches Caveats
Directory watches produce less verbose logs than exact
file watches.
When in need of detailed file-related records, enable
separate file watches for all files of interest.
Pathname globbing of any kind is not supported by audit.
Always use the exact pathnames.
Auditing can only be performed on existing files.
Any files added while the audit daemon is already running
are ignored until the audit rule set is updated to watch the
new files.
15
Assigning Keys to Rules
Assigning keys to your audit rules helps you to identify
any records related to this rule in the logs. An example
rule plus key:
-w /etc/selinux -k MAC-Policy
You may use the same key on different rules in order to
be able to group rules when searching for them.
It is also possible to apply multiple keys to a rule
-w /sbin/auditctl -p x -k privileged -k ids-exec-info
Using the ausearch log analyzer, you can easily filter for
any events related to this particular rule.
16
Auditing the Execution of Setuid/Setgid Binaries
Let’s say that as a matter of compliance, you have to
audit the execution of setuid/setgid binaries on your
system.
How do you do set that up?
First, run a script like this at boot time from /etc/rc.local
sending the output to a temp file, /tmp/snorf, for example.
17
Auditing the Execution of Setuid/Setgid Binaries (1)
#!/bin/bash
# Find all the file systems that are locally mounted
for i in `/bin/egrep '(ext4|ext3|ext2)' /etc/fstab | /bin/awk '{print $2}'`
do
# Find all the files on the file system found above and print out
# and audit rule for it
/usr/bin/find $i -xdev -type f \( -perm -4000 -o -perm 2000 \) -print | \
/bin/sort | /bin/awk '{ print "-a always,exit -F path=" $1 " -F perm=x \
-F auid>500 -F auid!=-1 -k privileged -k ids-exec-high" }'
done
18
Auditing the Execution of Setuid/Setgid Binaries (2)
And you get something like this (YMMV depending on
what’s installed).
-a always,exit -F path=/bin/fusermount -F perm=x -F auid>500 -F auid!=-1 -k
privileged -k ids-exec-high
-a always,exit -F path=/bin/ping -F perm=x -F auid>500 -F auid!=-1 -k
privileged -k ids-exec-high
-a always,exit -F path=/bin/ping6 -F perm=x -F auid>500 -F auid!=-1 -k
privileged -k ids-exec-high
-a always,exit -F path=/bin/su -F perm=x -F auid>500 -F auid!=-1 -k
privileged -k ids-exec-high
-a always,exit -F path=/usr/bin/chage -F perm=x -F auid>500 -F auid!=-1 -k
privileged -k ids-exec-high
-a always,exit -F path=/usr/bin/chfn -F perm=x -F auid>500 -F auid!=-1 -k
privileged -k ids-exec-high
-a always,exit -F path=/usr/bin/chsh -F perm=x -F auid>500 -F auid!=-1 -k
privileged -k ids-exec-high
-a always,exit -F path=/usr/bin/crontab -F perm=x -F auid>500 -F auid!=-1 -k
privileged -k ids-exec-high
19
Auditing the Execution of Setuid/Setgid Binaries (3)
Then, point auditctl at the temp file to add the newly
created audit rules.
The auditctl program is used to control the behavior, get
status, and add or delete rules into the kernel’s audit
system.
/sbin/auditctl –R /tmp/snorf
A couple of things about auditctl:
auditctl is not a filter, so output cannot be piped into it.
Rules files for auditctl must be owned by root.
20
System Call Auditing
System call auditing lets you track your system's behavior
on a level even below the application level.
The audit subsystem supports an ample collection of
events, to include the tracing of arbitrary system calls
identified by system call name, or by system call number.
The audit subsystem can also filter by PID, UID, system
call success, system call argument, and many other
possibilities.
When designing these rules, consider that auditing a
great many system calls may increase your system load
and cause you to run out of disk space.
Remember: First match wins!
21
System Call Audit Rules Examples
-a always,exit –S settimeofday
-a always,exit -F arch=b64 -S chmod
-a always,exit -F arch=b64 -S chown -F exit=-EACCES
-a always,exit -F arch=b64 -S creat -F exit=-EACCES –F
uid>=500
-a always,exit -F arch=b64 -S unlink -F auid>=500 -F
auid!=-1
-a always,exit -F arch=b64 -S init_module -S
delete_module –k modules –k ids-sys-info
-a always,exit -S execve –F auid>500 -F uid=0 –k root-exe
-a never,exit -F path=/somefile.dat -S unlink -S unlinkat
-S rename -S renameat
For details on setting up audit rules, see auditctl(8).
22
Creating Audit Reports
The audit records are stored in /var/log/audit/audit.log.
grep is your friend and you can pull stuff out of the audit
log and get stuff like this:
type=SYSCALL msg=audit(1365719016.212:333043): arch=c000003e
syscall=171 success=yes exit=0 a0=7fff86310c37 a1=6 a2=d
a3=7fff8630f3b0 items=0 ppid=22300 pid=22311 auid=0 uid=0 gid=0
euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=37491
comm="domainname" exe="/bin/hostname"
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
key=73797374656D2D6C6F63616C65016964732D7379732D6C6F77
Maybe grep isn’t your friend, after all.
The raw audit data auditd stores in the /var/log/audit
directory is quite complex.
To find what you want, you might have to sift through
bazillions of other events before you locate the one that
you want.
23
Creating Audit Reports (1)
Use aureport to create concise, human-readable reports.
Some of the useful options are:
--summary
--failed
--start and --end (aureport understands today, yesterday, now,
recent, this-week, this-month, and this-year)
--auth, --avc, --login, --user, --executable, --syscall
To get started, do aureport –summary and you get
something like this:
24
Creating Audit Reports (2)
Summary Report
======================
Range of time in logs: 03/01/2013 08:17:01.765 - 03/11/2013 16:07:49.252
Selected time for report: 03/01/2013 08:17:01 - 03/11/2013 16:07:49.252
Number of changes in configuration: 669
Number of changes to accounts, groups, or roles: 10
Number of logins: 11
Number of failed logins: 2941
Number of authentications: 38
Number of failed authentications: 5
Number of users: 3
Number of terminals: 10
Number of host names: 6
Number of executables: 20
Number of files: 597
Number of AVC's: 347
Number of MAC events: 11
Number of failed syscalls: 76
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 11882
Number of keys: 20
Number of process IDs: 17444
Number of events: 120152
25
Creating Audit Reports (3)
Lets look at some of the failed logins with aureport –l –
failed:
Login Report
============================================
# date time auid host term exe success event
============================================
1. 03/07/2013 15:00:37 root 192.168.1.101 ssh /usr/sbin/sshd no 178116
2. 03/07/2013 15:01:52 root 192.168.1.101 ssh /usr/sbin/sshd no 178135
3. 03/07/2013 15:01:54 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178146
4. 03/07/2013 15:01:54 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178147
5. 03/07/2013 15:02:01 root 192.168.1.101 ssh /usr/sbin/sshd no 178169
6. 03/07/2013 15:02:02 (invalid user) 192.168.1.101 ssh /usr/sbin/sshd no 178184
7. 03/07/2013 15:02:02 (invalid user) 192.168.1.101 ssh /usr/sbin/sshd no 178187
8. 03/07/2013 15:02:02 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178199
9. 03/07/2013 15:02:04 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178203
10. 03/07/2013 15:02:04 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178205
11. 03/07/2013 15:02:04 (invalid user) 192.168.1.101 ssh /usr/sbin/sshd no 178210
12. 03/07/2013 15:02:06 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178220
13. 03/07/2013 15:02:06 (invalid user) 192.168.1.101 ssh /usr/sbin/sshd no 178230
14. 03/07/2013 15:02:33 ftp 192.168.1.101 ssh /usr/sbin/sshd no 178332
26
Drilling Deeper with ausearch
Using aureport lets you to create overall summaries of
what is happening on the system, but if you want to drill
deeper into the details of a particular event, ausearch is
the tool to use.
ausearch allows you to search the audit logs using
special keys and search phrases that relate to most of the
flags that appear in event messages in
/var/log/audit/audit.log
A methodology to use is find an event class of interest
with aureport and then drill down into the nitty-gritty with
ausearch.
For instance, you use aureport –syscall –failed to see
the failed system calls. Use ausearch and one of the
event ids to get more information.
27
Drilling Deeper with ausearch (1)
From aureport –syscall –fail we get:
Syscall Report
=======================================
# date time syscall pid comm auid event
=======================================
1. 04/09/2013 16:19:15 87 4006 semodule 25016 53039
From ausearch –i –a 53039 we get:
type=PATH msg=audit(04/09/2013 16:19:15.564:53039) : item=0
name=/etc/selinux/targeted/modules/tmp/disable_dontaudit inode=394090 dev=fd:00
mode=dir,700 ouid=root ogid=root rdev=00:00
obj=unconfined_u:object_r:semanage_store_t:s0
type=CWD msg=audit(04/09/2013 16:19:15.564:53039) : cwd=/root
type=SYSCALL msg=audit(04/09/2013 16:19:15.564:53039) : arch=x86_64 syscall=unlink
success=no exit=-2(No such file or directory) a0=0x7f7e0a429c60 a1=0x0 a2=0x2
a3=0x7fff08fa5450 items=1 ppid=20218 pid=4006 auid=blotto uid=root gid=root euid=root
suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=6353 comm=semodule
exe=/usr/sbin/semodule subj=unconfined_u:unconfined_r:semanage_t:s0-s0:c0.c1023 key=MACpolicy key=ids-sys-low key=ids-file-info
28
Drilling Deeper with ausearch (2)
A useful feature, if you tagged your audit rules with keys,
is to search for events based on those keys.
For instance, ausearch -i -k identity –start recent and
we get:
type=PATH msg=audit(03/10/2013 17:28:14.821:326340) : item=0 name=/etc/shadow
inode=786095 dev=fd:00 mode=file,000 ouid=root ogid=root rdev=00:00
type=CWD msg=audit(04/11/2013 17:28:14.821:326340) : cwd=/home/gorgo
type=SYSCALL msg=audit(04/11/2013 17:28:14.821:326340) : arch=x86_64 syscall=open
success=no exit=-13(Permission denied) a0=0x7fffe185972e a1=O_WRONLY|O_TRUNC a2=0x0
a3=0x7fffe18571a0 items=1 ppid=22814 pid=25045 auid=gorgo uid=gorgo gid=users euid=gorgo
suid=gorgo fsuid=gorgo egid=users sgid=users fsgid=users tty=pts0 ses=46582 comm=cp
exe=/bin/cp key=identity key=ids-file-info
29
Visualizing Audit Data
Neither the data trail in /var/log/audit/audit.log nor the
different report types generated by aureport, provide an
intuitive reading experience to the user.
The aureport output is formatted in columns and thus
easily available to any sed, perl, or awk scripts that users
might connect to the audit framework to visualize the
audit data.
A solution: mkbar and mkgraph were created by Steve
Grubb at Red Hat. They are available from
http://people.redhat.com/sgrubb/audit/visualize/.
Note: These scripts need gnuplot and graphviz to create
their visualizations.
30
Visualizing Audit Data (1)
Create a plot of events with aureport -e -i --summary |
mkbar events
31
Visualizing Audit Data (2)
Create a summary of syscall events with aureport -s -i -summary | mkbar syscall
32
Visualizing Audit Data (3)
To create a summary chart of successful or failed events
of any of the above event types, just add the --success or
--failed option to the respective aureport command.
To cover a certain period of time only, use the -ts and -te
options on aureport.
Any of these commands can be tweaked further by
narrowing down its scope using grep or egrep and
regular expressions.
33
Visualizing Audit Data Relationships
To illustrate the relationship between different kinds of
audit objects, such as users and system calls, use the
script mkgraph.
Graphs can also be combined to illustrate complex
relationships.
See the comments in the mkgraph script for further
information and an example.
34
Visualizing Audit Data Relationships (1)
To graph the syscalls to programs, do aureport -s -i |
awk '/^[0-9]/ { printf "%s %s\n", $6, $4 }' | sort | uniq |
mkgraph syscall-vs-program
35
Visualizing Audit Data Relationships (2)
To graph to successful programs to files, do LC_ALL=C
aureport -f -i --success | awk '/^[0-9]/ { print $7" "$4 }' |
sort | uniq | mkgraph program-vs-file
36
Resources
The Audit Manual Pages
There are several man pages installed along with the audit tools
that provide valuable and very detailed information:
auditd(8) The Linux Audit daemon
auditd.conf(5) The Linux Audit daemon configuration file
auditctl(8) A utility to assist controlling the kernel's audit
system
autrace(8) A program similar to strace
ausearch(8) A tool to query audit daemon logs
aureport(8) A tool that produces summary reports of audit
daemon logs
audispd.conf(5) The audit event dispatcher configuration file
audispd(8) The audit event dispatcher daemon talking to
plugin programs.
37
Resources (1)
http://people.redhat.com/sgrubb/audit/index.html The
home page of the Linux audit project. This site contains
several specifications relating to different aspects of Linux
audit, as well as a short FAQ.
/usr/share/doc/audit The audit package itself contains a
README with basic design information and sample .rules
files for different scenarios:
capp.rules: Controlled Access Protection Profile (CAPP)
lspp.rules: Labeled Security Protection Profile (LSPP)
nispom.rules: National Industrial Security Program Operating
Manual Chapter 8(NISPOM)
stig.rules: Secure Technical Implementation Guide (STIG)
38
T-t-t-t-that’s all, folks!
Gary Smith
Information System Security Officer, Molecular Science
Computing, EMSL, Pacific Northwest National
Laboratory
Richland, WA
gary.smith@pnnl.gov
39