ccTLD name server training Server management and security September 10, 2002 Ko, YangWoo yw@mrko.pe.kr Note • Contents are NOT mine. Most of them are from the wonderful book “Practical Unix and Internet Security” and “Real World Linux Security”. • Others are extracted from various good resources including; – Linux Security FAQ – Solaris Security FAQ – Sun Solaris / HP-UX / Tru64 Unix man pages Table of contents • • • • • • Before we start Security basics Unix / Linux sever security System setup guide Detection Recovery Module 1 : Before we start Welcome to wild Internet ! • Quote from Crypto-Gram (June 15, 2001 ) A random computer on the Internet is scanned dozens of times a day. The life expectancy of a default installation of Red Hat 6.2 server, or the time before someone successfully hacks it, is less than 72 hours. A common home user setup,with Windows 98 and file sharing enabled, was hacked five times in four days. Systems are subjected to NetBIOS scans an average of 17 times a day. And the fastest time for a server being hacked: 15 minutes after plugging it into the network. No system is ever perfectly secure. But, still we need security. • Any number of toolkits exist that allow total amateurs to become holy terrors. • The good news is that if you can beat the popular intrusion toolkits, 90 percent of the bad guys will go bother somebody else who's less secure. System security in a page • The Seven Most Deadly Sins – Weak Passwords – Open Network Ports – Old Software Version – Poor Physical Security – Insecure CGIs – Stale and Unnecessary Accounts – Procrastination Module 2 : Security basics Security requirements • • • • • • • Confidentiality Integrity Authentication Non-repudiation Availability Access control Combined – User authentication used for access control – Non-repudiation combined with authentication Some terminologies • System security / network security • Passive attack / active attack – sniffing / spoofing • Two models – Access control • discretionary access control vs. mandatory access control – Audit Security policy • Simple and generic policy for system which users can readily understand and follow. • Starting point : – That which is not permitted is prohibited. • Setup steps (1) Identify what you are trying to protect. (2) Determine what you are trying to protect it from. (3) Determine how likely the threats are. (4) Implement measures which will protect your assets in a cost-effective manner. (5) Review & improve the process continuously Security policy (continued) • References – rfc2196 : Site Security Handbook • Samples – ftp://coast.cs.purdue.edu/pub/doc/policy Module 3 : Unix / Linux server security • • • • • • • • Password Superuser File system Account Integrity Log and Audit Programmed threats TCP/IP Module 3-1 : Password Bad passwords • Your name, spouse’s name, partner’s name, pet’s name, child’s name, friends’ name, boss’s name • Operating system, hostname, username • Phone number, license plate number, birth date, social security number • Words in the dictionary • Simple patterns of letters on the keyboard (qwerty) • Passwords of all the same letter • Any of above spelled backwards • Any of above followed or prepended by a single digit Password Good passwords • Have both uppercase and lowercase letters. • Have digits and/or punctuation characters as well as letters. • May include some control characters and/or spaces. • Are easy to remember, so they do not have to be written down. • Are seven or eight characters long. Password The Thompson Test • Devised by Ken Thompson • Cracking algorithm – One to six ASCII characters – Seven or eight lowercase letters – Any word from a large dictionary such as hangman-words, or a word spelled backward or with the digit “1” instead of the letter “l”, with the digit “0” instead of the letter “o”, or with the digit “3” instead of the letter “e”. – Any pair of words from a large dictionary or words spelled backwards. Password Module 3-2 : Superuser Who is superuser ? • UID of 0 • Any username can be the superuser. • Normal security checks and constraints are ignored for the superuser. • Superuser is not for casual use. – Do not login as superuser, use ‘/bin/su’ with “-” option instead. Superuser Simple trap to steal superuser • Premise • Set a trap – Root’s PATH starts with “.” • Contents of shell script ‘ls’ #!/bin/sh cp /bin/sh ./junk/.ss chmod 4555 ./junk/.ss rm –f $0 exec /bin/ls ${1+”$@”} % cd % chmod 700 . % touch ./-f • To do is just say to administrator. “I have a funny file in my directory I can’t seem to delete.” Superuser Several tricks for superusers • Test complex commands in a non-destructive way before running it. – rm foo*.bar “after” echo foo*.bar • alias rm=‘rm –i’ • Only become root to do single specific task. Stay normal user shell until you are sure what needs to be done by root. • Command path – Minimum and trusted directories only – Never include “.” – No writable directories Superuser Several tricks for superusers (continued) • Never use r-utilities (e.g. rlogin, rsh). Never create .rhosts for for the root. • No login from the remote – Linux, HPUX : /etc/securetty • file which lists ttys from which root can log in – Solaris : /etc/default/login • CONSOLE=/dev/console • Always be slow and deliberate running as root. Think before you type. Superuser Module 3-3 : File system File permission File type - : plain file d : directory c : character device (tty, printer) b : block device (disk, CD-ROM) l : symbolic link s : socket =, p : FIFO Access granted to others -rwxr--r-Access granted to owner r : read / w : write / x : execute Access granted to group member File system SUID/SGID/sticky bits • SUID (set uid) – Processes are granted access to system resources based on user who owns the file. • SGID (set gid) – (For file) Same with SUID except group is affected. – (For directory) Files created in that directory will have their group set to the directory's group. • sticky bit – If set on a directory, then a user may only delete files that the he owns or for which he has explicit write permission granted, even when he has write access to the directory. (e.g. /tmp ) File system File system tips • Finding SUID and SGID Files # find / \( -local -o -prune \) \( -perm -004000 -o perm -002000 \) -type f -print ( xdev can be used in place of local/prune) • Files without associated owner/group can be a signal of compromise. # find / -nouser –o –nogroup –print • Users are not allowed to have .rhosts file. # find /home –name .rhosts -print File system File system tips (continued) • Turning off SUID / SGID in mounted file system – use nosuid (and nodev if possible) when mounting remote file system or allowing users to mount floppies or CD-ROMs • Device file can be created as a backdoor after compromise. # find / \( -local -o -prune \) \( -type c -o -type b \) -exec ls -l {} \; File system Critical system files • These files should be backed up and compared with saved version frequently. – – – – – – – – – – /etc/passwd, /etc/shadow, /etc/group /etc/rc* /etc/ttys, /etc/ttytab, /etc/inittab /usr/lib/crontab, /usr/spool/cron/crontabs/, /etc/crontab /usr/lib/aliases /etc/exports, /etc/dfs/dfstab /etc/netgroups /etc/fstab, /etc/vfstab /etc/inetd.conf UUCP related files File system Module 3-4 : Account Dangerous accounts • Accounts without passwords # cat /etc/passwd | awk -F: 'length($2)<1 {print $1}' • Default accounts – Just remove them ! • Shared accounts – Less accountability, less security. – Create several accounts in a group. • e-mail ID and account – Do not use e-mail ID as an account, utilized alias feature instead. Account Dormant account • Risks – Intruder can use dormant account without being noticed. – Owner of dormant account cannot follow your policy or order. (e.g. Dear every users, please change your passwords right now.) • How to handle – Disabling dormant account automatically (SVR4) • usermod –f 10 newcat (locked if no login in 10 days) – Freeze it • Put “*” in password field • chmod 0 /home/newcat • find / -user newcat -ls Account Dormant account (continued) • How to find #!/bin/sh PATH=/bin:/usr/bin;export PATH umask 077 THIS_MONTH=`date | awk ‘{print $2}’` /bin/last | /bin/grep $THIS_MONTH | awk ‘{print $1}’ | /bin/sort –u > /tmp/users1$$ cat-passwd | /bin/awk –F: ‘{print $1}’ | /bin/sort –u /tmp/users2$$ /bin/comm –13 /tmp/users[12]$$ /bin/rm –f /tmp/users[12]$$ Account Module 3-5 : Integrity Simple examples • By metadata # cat /usr/adm/filelist | xargs ls -ilds > /tmp/now # diff -b /usr/adm/savelist /tmp/now • By checksum # find `cat /usr/adm/filelist` -ls -type f -exec md4 {}\; > /tmp/now # diff -b /usr/adm/savelist /tmp/now Integrity Tripwire • Tripwire is a tool that checks to see what has changed on your system. The program monitors key attributes of files that should not change, including binary signature, size, expected change of size, etc. • Where is it ? – Commercial version : http://www.tripwire.com/ – For Linux user : http://www.tripwire.org/ – For Unix user : ftp://coast.cs.purdue.edu/pub/COAST/Tripwire/trip wire-1.2.tar.Z Integrity Tripwire tutorial in a slide • Initial setup – download / build / install it – modify policy file (e.g. remove unnecessary files) # vi /etc/tripwire/twpol.txt – generate policy file # twadmin –create-polfile /etc/tripwire/twpol.txt – build initial database # tripwire –init • check periodically # tripwire –check – reconcile differences (e.g. software installation) # tripwire –update –accept-all –twrfile report_file Integrity Module 3-6 : Log and audit Basics • Consider remote logging to secure log data. • List of log files – – – – – – – – – acct / pacct : Commands run by users aculog : Dial-out modem (acu : automatic call unit) lastlog : Most recent login success/fail times loginlog : Bad login attempts messages : Console / syslog facility sulog : su command utmp / utmpx : Each user currently logged in wtmp / wtmpx : Login/out, shutdown/startup xferlog : FTP access Log and audit Files and commands • lastlog file – lastlog (Linux only) • Displays last login time and location. • u/wtmp file – last • Displays login and logout information about users and terminals • acct/pacct file – (Solaris 5.8) /usr/lib/acct/[startup , shutacct] • Starts or stop accounting. – (Solaris 5.8) acctcom, lastcom • Displays the recent commands executed. Log and audit Monitoring logs • logcheck (logsentry) – Extracts anything that might indicate a security violation or other abnormality, and informs via e-mail. – http://www.psionic.com/products/logsentry. html Log and audit Module 3-7 : Programmed threats Basic terms • Bug vs. malware (or malicious software) • Kinds of malwares – Security tools and toolkits – Back doors and trap doors – Logic bombs – Viruses – Worms – Trojan horses – Bacteria and rabbits Programmed threats Against programmed threats • Back door – Do regular integrity check. – Install software only from well-known sources. – Separate test bed and production system. • Trojan horse – Never execute anything until you’re sure of program or inputs to program. – Never run anything as root unless you absolutely must. Programmed threats Against programmed threats (continued) • Viruses – Use same techniques used against back doors and Trojan horse. – Don’t include nonstandard directories (including .) in your PATH. – Don’t leave common binary directories unprotected and set permission of commands to 555 or 511. – Make sure your own directories are writable only by you not by your group or world. Programmed threats Against programmed threats (continued) • Worm – Prevention • If an intruder can enter your machine, so can a worm program. – If under attack, • Call computer incident response center to se if other sites have made similar reports. • Isolate your server to prevent spread. Programmed threats Module 3-8 : TCP/IP Vulnerabilities • ftp – Passwords are sent in plain text. – /etc/ftpusers • List of accounts that are NOT allowed to use ftp. • telnet – Passwords are sent in plain text. – Attacker can hijack the session. TCP/IP Vulnerabilities (continued) • smtp (sendmail) – Must be upgraded 8.9.3 or higher. Current version is 8.12.6. – Check permission of /var/spool/mqueue, sendmail.cf, /etc/aliases*, /etc/mail/mailertable* (owned by root, writable by owner only) TCP/IP Vulnerabilities (continued) • Sun RPC portmapper – Assigns the TCP/UDP ports used for RPC. – To improve security, turn it off if possible. Or, • Replace it with Wietse Venema’s version. • Block packets on port 111. • rexec, rsh, rlogin – Executes remote program or login. – rexec transmits plain text password and rsh/rlogin use “trusted host/user” concept. – Disable rexec, and replace rsh/rlogin with ssh. TCP/IP Vulnerabilities (continued) • web – Yet another BIG topic. See references; • Lincoln D. Stein’s WWW Security FAQ – http://www-genome.wi.mit.edu/WWW/faqs/wwwsecurity-faq.html • Paul Phillips CGI security FAQ – http://www.primus.com/staff/paulp/cgi-security • NCSA’s CGI security documentation – http://hoohoo.ncsa.uiuc.edu/cgi/security.html TCP/IP Vulnerabilities (continued) • NFS – Limit exported and mounted file systems – Export read-only and use root ownership – Remove group-write permission for files and directories – Do not export server executables and home directories – Do not allow users to log into server – Use fsirand and set the portmon variable – Use showmount –e – Use secure NFS TCP/IP Vulnerabilities (continued) • tftp (UDP 69) – No security at all. • finger ( 79 ) – Provides user information. • POP ( 109, 110 ) – Username/password is sent in plain text. TCP/IP Module 4 : System setup guide Useful links for system setup • Solaris – Solaris/Unix Security Checklist Version 1.0 • http://www.geocities.com/losttoy2000/solarissec .rtf – The Solaris Security FAQ • http://www.itworld.com/Comp/2377/security-faq/ • Linux – Securing Debian Manual • http://www.debian.org/doc/manuals/securingdebian-howto/ System setup steps (1/2) 1. 2. 3. 4. 5. Disconnect system from network. Install a minimal Operating System. Install the recommended patches. Use BIOS/EEPROM security. Securing root account – Force root to login through su. – Check environments • default mask (027), PATH 6. Apply hardening script if available. 7. Direct syslog to loghost System setup steps (2/2) 8. Create minimal accounts and disallow login. 9. Let minimal services run; – /etc/rc*, /etc/inet.d 10. Use tcpwrapper for network services. 11. Install Secure Shell and encourage its use. 12. Install integrity checker (e.g. Tripwire). 13. Test it periodically – e.g. Nessus, COPS, Tiger, … 14. Monitor it forever – Check logs, login/outs, commands Module 5 : Detection • Monitoring • Scanning • Handling Monitoring (1/2) • Log (logcheck) – Propagate it using loghost and e-mail. – Check it. • Network port (netstat) – Trojan horse may use network ports. – http://www.glocksoft.com/trojan_port.htm • Network (tcpdump) Monitoring Monitoring (2/2) • Process (ps) – Check suspicious processes, e.g. compiler. – Record typical size of daemons and important programs to detect Trojan horse. • Load (uptime) Monitoring Scanning • Find suspicious files. • Run Tripwire. • Detect promiscuous network interfaces. – (see next page) Scanning Perl script to detect sniffer #!/usr/bin/perl my $ifconfig = “/sbin/ifconfig”; my $recips = admin@my.admin.host; my %PROMISC = (); my $interface = “”; open( IFCONFIG, “$ifconfig|” ) || die( “Error: cannot run ifconfig!” ); while( <INCONFIG> ) { $interface = $1 if m/^(\S+)/; $PROMISC{$interface} = 1 if m/promisc/I; } close( IFCONFIG ); if( %PROMISC ) { open( MAIL, “|Mail –s ‘Promisc mode’ $recips” ) || die( “Error: cannot send mail” ); print MAIL “Interfaces in Promisc mode: “, join( “ “, sort keys %PRMISC), “\n”; close MAIL; } Scanning Handling incidents • Don’t panic – Is it really a security incident ? – Was any damage really done ? – Evidence or normal operation, that is the question. • Document – Write down everything you find, always noting the date and time. • Plan ahead !!! Handling Module 6 : Recovery • Regaining control of system • Finding and repairing the damage • Tracing attacker Regaining control of system • Operate as an unprivileged user. • Check integrity of commands used. • Have stealth version of crucial commands (ps / ls / tar / …) – Build from open source. Or, – Rename from existing binary • • • • cd /home/larry/bin cp /bin/ls monthly cat text_file >> monthly (echo ls is monthly; md5sum monthly) | lpr • Process must be kill by –9. Regaining TCP/IP control of system Analyze Trojan horse • Save suspicious executables on (removable) media. • Analyze – strings Trojan – file Trojan • if not stripped – nm Trojan (see function names, syscalls) – run debugger (see stack trace) – Check files opened by Trojan • (Linux) /proc/pid/fd • (Solaris) pfile pid Regaining control of system Prevent further damage 1. Drop connection (unplug LAN, modem) 2. Shutdown abruptly 1. Close database 2. Run sync (from non privileged user) 3. Press reset (or power) button 3. Boot again 1. Remove the system disk from the compromised system and connect it as second disk to a secure system. (Or, boot from secure boot floppy.) 2. Run fsck 3. Before coming up multi-user mode, check cracker generated email. Regaining control of system Checking logs • Log files – – – – – /var/log/* Shell history files (esp. for root) Mailboxes (mbox, /?/spool/mail, /?/spool/mqueue) Firewall logs, ISP’s log tcpwrapper log (denied log only) • Other files – /tmp/* – Hidden directories (e.g. /home/*/.??*) – Other files started with “.” Finding and repairing the damage Finding cracker-altered files • Use file integrity tools (e.g. Tripwire) • Compare file system with backups. – GNU tar “-d” option is very useful. • Rename any Trojan horse found something obvious. – mv /mnt2/tmp/ls /mnt/tmp/ls-CRACKED – chmod 0 /mnt/tmp/ls-CRACKED • Find normal files hidden in /dev – find /dev –type f –ls • Find set UID programs Finding and repairing the damage Useful commands • With IP address (A.B.C.D) – nslookup –type=any D.C.B.A.in-addr.arpa – dig –x A.B.C.D • With domain name – whois • Using ping – See the distance • Using traceroute Tracing hacker Module 7 : D.I.Y. • Requirement • Analysis • Plan and Do What assets do I have ? • Classification of assets – Hardware • Server / PC / Storage device / Printer – Network • Network distribution component (e.g. router, hub, switch) • Network service host (e.g. directory, NMS) • Network connection / Cabling – – – – Data (e.g. database, agreement, policy, guideline) Software Human Environment (e.g. UPS, air conditioner, cabinet) Requirement How valuable they are ? (1/4) • Review documentations – List of all servers – List of all security products in place – Operation guidelines • Interview with operational personnel • Valuation methods – CIA • Confidentiality / Integrity / Availability – Cost of loss Requirement How valuable they are ? (2/4) • Confidentiality – 5 : Top secret – 4 : Secret – 3 : Limited – 2 : Limited within organization • Ordinary documents – 1 : Open Requirement How valuable they are ? (3/4) • Integrity – 5 : Critical damage to operation – 1 : No (or very least) damage to operation • Availability – 5 : Non stop – 4 : Recovery within 4 hours – 3 : Recovery within 8 hours – 2 : Recovery within 12 hours – 1 : Recovery within 24 hours Requirement How valuable they are ? (4/4) • Cost of loss – 5 : Serious loss (e.g. Bankruptcy) – 4 : Major loss (e.g. Discontinuance of some businesses) – 3 : Significant loss (e.g. Discontinuance of some tasks) – 2 : Loss (e.g. < U$ 10,000) – 1 : Trivial loss (e.g. < U$ 1,000) Requirement Define analysis areas • Network / system security – Service daemons – Backdoors, vulnerable files – Misuse by users – User accounts – Log management – Network configuration – Network device management – Database security • Physical security • Security management Requirement – Compliance assessment – Security policy assessment – Contingency planning Analysis • Automated analysis – e.g. Nessus • Manual analysis – OS checklists Analysis Sample results • Service daemons – Problems • Some old-version daemons have buffer overflow vulnerabilities. • Unnecessary daemons are running. – To do • Remove unnecessary daemons. • Keep necessary daemon up to date. • Run security scanner periodically. Analysis Sample results • Backdoors, vulnerable files – Problems • Backdoor is not found, but there is no counter measure for future backdoors. – To do • Install and run Tripwire periodically. Analysis Sample results • Misuse by users – Problems • Sendmail’s vulnerability can lead to root compromise. – To do • Remove if unnecessary. • Keep it up to date if necessary. Analysis Sample results • User accounts – Problems • Super user accounts are shared by administrators and developers. • Weak passwords are found. – To do • Define each systems’ usages clearly. • Define each users’ role according to usage of system. • Apply password control (including aging). Analysis Sample results • Log management – Problems • No log management. – To do • Setup a loghost, and all logs are configured to be sent to it. • Write a log management guideline and apply it. Analysis Sample results • Network configuration – Problems • Database servers are exposed to Internet. – To do • Set up a DMZ. • Put external service servers at DMZ. • Put Database servers at internal network Analysis Categories of reaction • Configuration issue – Issues are solved by configuring servers and network equipments properly. – Usually done within a week. • Infra structure issue – Issues are solved by investing on infrastructure. – Usually outsourced in long time period. • Management issue – Several units within organization work together to handle these issues. – Plan => Do => See cycle Plan Categorize To Dos • Configuration issue – Remove unnecessary daemons. – Apply password control (including aging). • Management issue – Write a log management guideline and apply it. – Define each systems’ usages clearly. – Define each users’ role according to usage of system. • Infra structure issue – Run security scanner periodically. – Set up a DMZ. Plan