Lecture 3: Unix installation&startup Guntis Barzdins Girts Folkmanis Juris Krumins Mājas Darbs #1 A. Izvēlēties iespēju robežās no citiem atšķirīgu *nix distribūciju, to patstāvīgi instalēt un īsumā aprakstīt galvenos soļus, grūtības un tml. Instalācijas procesa ekranattēlu iegūšanai ieteicams lietot VirtualBox, Vmware vai citu emulatoru. B. Instalētajā *nix versijā panākt lai darbojas C kompilātors; tajā uzrakstīt, nokompilēt un palaist vienkāršu "Sveika, pasaule!" programmiņu. C. (Neobligātā daļa atzīmes 10 iegūšanai): nodemonstrēt buffer overflow ielaušanos datorā (Unix shell piekļuvi vai kādas programmas izpildi) caur jūsu "Sveika, pasaule!" programmiņu, kas satur scanf vai līdzīgu nedrošu C bibliotēkas funkciju. Pašu ielaušanos veikt tikai programmas ievadā, bez ievainojamās programmas "pašuzlaušanas". Praktiskas lietas Video lekcijas – padziļinātai izpratnei (angliski) Kristaps Džonsons http://students.mii.lu.lv/Lekcijas/OperetajsistemaUNIX/ Mājasdarba C-daļai skatīt 11stack1.mov (no 24-46 min) 12stack2.mov (no sākuma) Ieteicamās virtualiācijas vides majas darbam VirtualPC (Win) VMWare (Win, Mac) VirtualBox (Sun, Oracle) Parallels (Mac) KVM (Linux) Linux Distributions Debian GNU/Linux (Xandros, Knopix,...) a free operating system that provides almost 4000 software packages for six architectures (Intel, Motorola, Alpha, SPARC, PowerPC, ARM) and is developed entirely by volunteers over the Internet Fedora/RedHat a Red-Hat-sponsored and community-supported open source project. The goal of The Fedora Project is to work with the Linux community to build a complete, general purpose operating system exclusively from free software. Development will be done in a public forum Mandrake Linux a friendly Linux Operating System which specializes in ease-of-use for both servers and the home/office. It is freely available in many languages throughout the world Slackware a long running Linux distribution that descends from the SLS Distribution SuSE Linux an extremely popular distro that features excellent documentation, thousands of applications on a DVD/CD-ROM set Gentoo Linux an optimized GNU/Linux distribution featuring 5 different pgcc-optimized builds, including Intel Pentium, Pentium Pro/II/III and AMD K6 optimized BSD Versions FreeBSD - Focusing on features and ease of use Darwin (OS X) - Focusing on the desktop and multimedia with use of the Apple's Aqua Interface NetBSD - Focusing on portability OpenBSD - Focusing on security Citi brīvi pieejami Unix Solaris x64/x86 SPARC Minix Unix Instalacija no CD/DVD Installing NetBSD The main menu of the installation program Installing NetBSD Choosing the partitioning scheme Installing NetBSD fdisk Installing NetBSD Installing the boot selector Installing NetBSD The disklabel editor System startup System initialization Overview of the PC Boot Process When a PC is powered on, the BIOS (Basic Input-Output System) runs first, followed by a boot loader and finally the operating system initialization routine. System initialization The BIOS When power is initially applied to the computer this triggers the RESET pin on the processor. This causes the processor to read from memory location 0xFFFFFFF0 and begin executing the code located there. This address is mapped to the ReadOnly Memory (ROM) containing the BIOS. The BIOS must poll the hardware and set up an environment capable of booting the operating system. BIOS functionality can be broken into three areas: Power On Self Test (POST), Setup and Boot. The last action of the BIOS is to execute the 19h interrupt, which loads the first sector of the first boot device. Since this is the location of the boot loader, execution of the 19h interrupt transfers control to the boot loader. Hard Drive Construction Master Boot Record (MBR) Track 0, Sector 1 (512bytes): • Number of partitions and their location (max 4 primary partitions or 3 primary + 1 Extended + 12 Logical) • Pointer to partition with bootloader struct master_boot_record { char bootinst[446]; /* space to hold actual boot code */ char parts[4 * sizeof (struct fdisk_partition_table)]; ushort signature; /* set to 0xAA55 to indicate PC MBR format */ }; System initialization The Boot Loader Once the BIOS loads the first sector of the boot device into RAM, the boot loader begins execution. In the case of a hard drive, this first sector is referred to as the Master Boot Record (MBR). The MBR contains the partition table describing the partitions defined on the hard drive. It also contains a program, the boot loader, which will load the first sector of the partition marked as active into RAM and execute it. The size of the MBR is limited to one sector on disk or 512 bytes, since it is located within the first sector of the drive at cylinder 0, head 0, sector 1. Typically boot loaders have been highly integrated with the operating system that they support. This integration cuts down on the operations a boot loader must perform, making a 512 byte boot loader feasible. When more functionality is required, a multi-stage boot loader may be used. A multi-stage boot loader provides more function and flexibility by working around the 512 byte size limitation. Rather than consisting of a single program which loads the operating system directly, multi-stage boot loaders divide their functionality into a number of smaller programs that each successively load one another. Boot Block The system’s primary disk unit contains a boot block that contains the bootstrapping program that loads the OS to memory. This program is invoked by the computer’s minimal bootstrap program in ROM. This boot block is often called the Master Boot Record (MBR). Different operating systems treat the MBR in very different ways. Some are flexible enough to install a boot loader in the MBR, so that the disk can contain different OS in different disk partitions. The loader for each OS is then stored at the beginning of its own partition. Examples: Windows NT/2000/xp boot loader, Linux lilo and grub. A “bootable” disk is one on which a boot block has been installed. How computer startup? Booting is a bootstrapping process that starts operating systems when the user turns on a computer system A boot sequence is the set of operations the computer performs when it is switched on that load an operating system 20 Booting sequence 1. 2. 3. 4. 5. 6. Tern on CPU jump to address of BIOS (0xFFFF0) BIOS runs POST (Power-On Self Test) Find bootale devices Loads and execute boot sector form MBR Load OS 21 BIOS (Basic Input/Output System) BIOS refers to the software code run by a computer when first powered on The primary function of BIOS is code program embedded on a chip that recognises and controls various devices that make up the computer. BIOS on board BIOS on screen 22 Boot loader MBR (Master Boot Record) OS is booted from a hard disk, where the Master Boot Record (MBR) contains the primary boot loader The MBR is a 512-byte sector, located in the first sector on the disk (sector 1 of cylinder 0, head 0) After the MBR is loaded into RAM, the BIOS yields control to it. 24 MBR (Master Boot Record) 25 MBR (Master Boot Record) The first 446 bytes are the primary boot loader, which contains both executable code and error message text The next sixty-four bytes are the partition table, which contains a record for each of four partitions The MBR ends with two bytes that are defined as the magic number (0xAA55). The magic number serves as a validation check of the MBR 26 Extracting the MBR To see the contents of MBR, use this command: # dd if=/dev/hda of=mbr.bin bs=512 count=1 # od -xa mbr.bin **The dd command, which needs to be run from root, reads the first 512 bytes from /dev/hda (the first Integrated Drive Electronics, or IDE drive) and writes them to the mbr.bin file. **The od command prints the binary file in hex and ASCII formats. 27 Boot loader Boot loader could be more aptly called the kernel loader. The task at this stage is to load the Linux kernel Optional, initial RAM disk GRUB and LILO are the most popular Linux boot loader. 28 Other boot loader (Several OS) bootman GRUB LILO NTLDR XOSL BootX loadlin Gujin Boot Camp Syslinux GAG 29 GRUB: GRand Unified Bootloader GRUB is an operating system independant boot loader A multiboot software packet from GNU Flexible command line interface File system access Support multiple executable format Support diskless system Download OS from network Etc. 30 GRUB boot process 1. The BIOS finds a bootable device (hard disk) and transfers control to the master boot record 2. The MBR contains GRUB stage 1. Given the small size of the MBR, Stage 1 just load the next stage of GRUB 3. GRUB Stage 1.5 is located in the first 30 kilobytes of hard disk immediately following the MBR. Stage 1.5 loads Stage 2. 4. GRUB Stage 2 receives control, and displays to the user the GRUB boot menu (where the user can manually specify the boot parameters). 5. GRUB loads the user-selected (or default) kernel into memory and passes control on to the kernel. 31 Example GRUB config file 32 LILO: LInux LOader Not depend on a specific file system Can boot from harddisk and floppy Up to 16 different images Must change LILO when kernel image file or config file is changed 33 LILO v.s. GRUB LILO Run LILO to modify mini-bootloader in the MBR Cannot read file system itself GRUB Multistage loader Can read file-system itself Parameter passing (runlevel, init) to kernel Actually hacking – modifies address and name inside kernel for the process to start Boot Terminology Loader: Program that moves bits from disk (usually) to memory and then transfers CPU control to the newly “loaded” bits (executable). Bootloader / Bootstrap: Program that loads the “first program” (the kernel). Boot PROM / PROM Monitor / BIOS: Persistent code that is “already loaded” on power-up. Boot Manager: Program that lets you choose the “first program” to load. LILO: LInux LOader A versatile boot manager that supports: Choice of Linux kernels. Boot time kernel parameters. Booting non-Linux kernels. A variety of configurations. Characteristics: Lives in MBR or partition boot sector. Has no knowledge of filesystem structure so… Builds a sector “map file” (block map) to find kernel. /sbin/lilo – “map installer”. /etc/lilo.conf is lilo configuration file. Example lilo.conf File boot=/dev/hda map=/boot/map install=/boot/boot.b prompt timeout=50 default=linux image=/boot/vmlinuz-2.2.12-20 label=linux initrd=/boot/initrd-2.2.12-20.img read-only root=/dev/hda1 LILO interface Once LILO has successfully loaded, you will see a LILO prompt. You may let LILO time out (after 10 seconds), which will boot the default Linux partition. Second, you can press the TAB key, which will list a selection of operating systems to boot from. For example we would get "Linux" and "WindowsXP" as our options. Typing either of these will load up that OS. GRUB (GRand Unix Bootloader) Master Boot Record Partition 1: 100M EXT3 (Linux) /boot MBR Passes control to the /boot partition GRUB GRUB then boots to an OS (Windows or Linux) Partition 2: 10GB NTFS (Windows XP) Partition 3: 512MB Linux Swap Partition 4: 10GB EXT3 (Linux) / Grub interface Kernel is loaded into memory GRUB decompresses the Linux kernel and loads it into memory /boot vmlinuz-2.6.12-1447_FC4 Memory GRUB also passes “kernel command line” (255bytes) to the loaded Linux kernel grub.conf # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this\ file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,1) # kernel /vmlinuz-version ro root=/dev/hdb3 # initrd /initrd-version.img #boot=/dev/hdb default=0 timeout=10 splashimage=(hd0,1)/grub/splash.xpm.gz title Linux Fedora (2.6.5-1.358smp) root (hd0,1) kernel /vmlinuz-2.6.5-1.358smp ro root=LABEL=/ rhgb quiet initrd /initrd-2.6.5-1.358smp.img title Linux Fedora-up (2.6.5-1.358) root (hd0,1) kernel /vmlinuz-2.6.5-1.358 ro root=LABEL=/ rhgb quiet initrd /initrd-2.6.5-1.358.img title Windows 2000 rootnoverify (hd0,0) chainloader +1 Specifies the default boot image will be the first hard entry Grub will wait for 10 seconds for input from the user before continuing to boot. The root partition is the second partition on the first hard drive. Startup Sequence Power-on Boot loader LILO Initialisation Scripts Create init process Linux hardware rc.sysinit Load Kernel runlevel 0-6 rc.local Ready Changing runlevel at boot time LILO: append the runlevel to the boot command : LILO: linux 3 or LILO: linux 5 GRUB: press the `e' key to edit the boot configuration append the runlevel to the end of the boot command as shown: kernel /vmlinuz ro root=/dev/hda1 5 Kernel starts the init process Kernel passes desired runlevel to the init pocess as argument Partition 4: 10GB EXT3 (Linux) / If runlevel not specified, then /etc # Default runlevel id:5:initdefault: (Configuration files) /etc/init.d /etc/inittab Resource (RC) scripts are stored here and are called depending on the system runlevel acpid* anacron* apmd* atd* auditd* ... /etc/rc.d rc5.d Linux Run Levels Linux defines 7 run levels Each run level defines a set of commands that are run to stop and start processes. The actual commands are held in /etc/init.d directory The run level directories rc0.d, rc2.d…rc6.d contain links to the actual commands Each command is prefixed with S or K and a number 00-99 S prefix means that a process should be started e.g. S10network K prefix means that a process should be stopped (killed) K70syslog The numbers determine the order in which the commands are run from lowest first to highest last Linux Run Levels 0-2 Runlevel 0 Directory /etc/rc.d/rc0.d Actually shutdown sequence Kill all processes Turn off virtual memory system (i.e. swap partition) unmounts swap and file systems Runlevel 1 /etc/rc.d/rc1.d single user mode used for maintenance by system administrators when they need sole control of machine, e.g. reconfiguring hardware installing software Runlevel 2 /etc/rc.d/rc2.d Multi-user + networking (minus NFS) Linux Run level 3-6 Runlevel 3 /etc/rc.d/rc3.d Default run level multi-user + NFS Runlevel 4 /etc/rc.d/rc4.d not defined ( available for customization) Runlevel 5 /etc/rc.d/rc5.d Same as 3 under Redhat this but includes starting X windows Runlevel 6 /etc/rc.d/rc6.d reboot similar to 0 but allow allows option to shutdown (halt) or reboot Shutdown shutdown allows... users to be warned the systems is going down the contents of disk caches to be written disk file systems to be marked as having been closed properly ( avoid file system check on next startup) Access to the shutdown command is restricted (it is in /sbin) eg shutdown -h now h = halt r = reboot HD identification IDE drives dev/hda dev/hdb ... Partition identification: SCSI dev/sda dev/sdb ... brw-rw---- 1 root disk 3, 0 May 5 1998 hda brw-rw---- 1 root disk 8, 0 May 5 1998 Note that /dev is not a real directory for accessing the contents of disks – it is rather a naming space for all hardware devices (and their drivers) present in the system, including keyboard, screen etc. The contents of disks is made available in other part of root / hierarchy. File System Layout Bootloader can be installed in different locations (if bootloader not in MBR, the BIOS interprets MBR to find boot partition) Mandatory partitions root /dev/hda1 (recommended) This is partition for the actual root / of the UNIX filesystem swap /dev/hda2 (recommended) 4GB (max with 32bit x86 CPU) RAM SIZE * 2 (recommended) This partition is not visible in root / and is used only by kernel for virtual memory page swapping (“raw” sectors, no structure) Theoretically swap partition is optional, if enough RAM Optional partitions or separate HD Typically on separate partitions /home, /boot, /tmp, /var, /usr Reasons for separate partitions logical or physical damage reinstallation (affects only some partitions) limit overflow effects better performance shared via network # df Filesystem 1K-blocks /dev/hda1 /dev/hda3 none Used Available Use% Mounted on 9070728 4701180 3908768 24950428 683412 22999584 257236 0 257236 55% / 3% /home 0% /dev/shm File System Types VFS UFS / FFS NFS RFS S5FS VxFS JFS CFS EXT2/EXT3 GFS XFS CXFS QFS HFS NTFS FAT16/ FAT32 GPFS .... Virtual File System UNIX File System – used interchangeably with FFS (Fast File System) Network File System – developed by Sun Remote File System – developed by AT&T Original System V file system Veritas Journaling File System Journaled File System – AIX Cluster file system – Tru64 2nd/3rd Extended File System – Linux systems Global File System – Linux Extended File System – SGI and others Clustered Extended File System – SGI 64-bit very large file system – Sun Solaris Hierarchical File System – S/390 UNIX, OS/2, MacOS X Windows NT/2000/2003 Windows – several versions IBM File System Support Linux can interpret many file system types, including; EXT2, EXT3, UFS/FFS, FAT16, FAT32 NTFS (read-only), HPFS ISO9660, UDF, UFS, etc. Win32 can interpret a few file system types, including; FAT16, FAT32 NTFS Creating Partitions: GNU Parted Notes: 1. 2. 3. 4. 5. For ext2, ext3: the start of the partition must stay fixed. For ext2, ext3: the partition you copy to must be bigger or exactly the same size as the partition you copy from. For ext2 and ext3: the checking is limited to ensuring the resize and copy commands will be ok. For fat: the size of the new partition after resizing or copying is restricted by the cluster size. Parted can shrink the cluster size, so you can always shrink your partition. However, if you can't use FAT32 for some reason, you may not be able to grow your partition. Parted supports both FAT16 and FAT32. Parted can convert file systems between FAT16 and FAT32, if necessary. Linux Partition Types Primary Partitions Swap Partitions Foreign Partition Types Logical Partitions Linux Device names IDE controller naming convention drive name drive controller drive number /dev/hda /dev/hdb /dev/hdc /dev/hdd 1 1 2 2 1 2 1 2 Linux partition names drive name drive controller drive number partition type partition number /dev/hda1 1 1 Primary 1 /dev/hda2 1 1 Primary 2 /dev/hda3 1 1 Primary 3 /dev/hda4 1 1 Swap NA /dev/hdb1 1 2 Primary 1 /dev/hdb2 1 2 Primary 2 /dev/hdb3 1 2 Primary 3 /dev/hdb4 1 2 Primary 4 Linux Logical Partitions drive name drive controller drive number partition type partition number /dev/hdb1 1 2 primary 1 /dev/hdb2 1 2 extended NA /dev/hda5 1 2 logical 2 /dev/hdb6 1 2 Logical 3 passwd, shadow, group files unix etc # ls -l passwd shadow group -rw-r--r-- 1 root root 705 Sep 23 15:36 group -rw-r--r-- 1 root root 1895 Sep 24 18:20 passwd -rw------- 1 root root 634 Sep 24 18:22 shadow unix etc # unix root # more /etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/bin/false daemon:x:2:2:daemon:/sbin:/bin/false adm:x:3:4:adm:/var/adm:/bin/false lp:x:4:7:lp:/var/spool/lpd:/bin/false sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt ... guest:x:405:100:guest:/dev/null:/dev/null nobody:x:65534:65534:nobody:/:/bin/false girtsf:x:1000:100::/home/girtsf:/bin/bash dima:x:1001:100::/home/dima:/bin/bash guntis:x:1002:100::/home/guntis:/bin/bash students:x:1003:100::/home/students:/bin/bash unix root # unix root # more /etc/shadow root:$1$VlYbWsrd$GUs2cptio.rKlGHgAMBzr.:12684:0::::: halt:*:9797:0::::: ... guest:*:9797:0::::: nobody:*:9797:0::::: girtsf:$1$u6UEWKT2$w5K28n2iAB2wNWtyPLycP1:12684:0:99999:7::: dima:$1$BQCdIBdV$xzzlj4s8XT6L9cLAmcoV50:12684:0:99999:7::: guntis:$1$fiJF/0BT$Py9JiQQL6icajjQVyMZ7//:12684:0:99999:7::: students:$1$wueon8yh$nLpUpNOKr8yTYaEnEK6OJ1:12685:0:99999:7::: unix root # unix root # more /etc/group root::0:root bin::1:root,bin,daemon daemon::2:root,bin,daemon sys::3:root,bin,adm adm::4:root,adm,daemon tty::5:girtsf disk::6:root,adm lp::7:lp mem::8: kmem::9: wheel::10:root,girtsf floppy::11:root mail::12:mail ... users::100:games,girtsf nofiles:x:200: qmail:x:201: postfix:x:207: postdrop:x:208: smmsp:x:209:smmsp slocate::245: portage::250:portage utmp:x:406: nogroup::65533: nobody::65534: unix root # Advanced Boot Concepts Initial ramdisk (initrd) – two-stage boot for flexibility: First mount “initial” ramdisk as root. Execute linuxrc to perform additional setup, configuration. Finally mount “real” root and continue. See Documentation/initrd.txt for details. Also see “man initrd”. Net booting: Remote root (Diskless-root-HOWTO). Diskless boot (Diskless-HOWTO). UUID A Universally Unique Identifier (UUID) is an identifier standard used in software construction, standardized by the Open Software Foundation (OSF) as part of the Distributed Computing Environment (DCE). The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. Thus, anyone can create a UUID and use it to identify something with reasonable confidence that the identifier will never be unintentionally used by anyone for anything else. A UUID is a 16-byte (128-bit) number. In its canonical form, a UUID consists of 32 hexadecimal digits, displayed in 5 groups separated by hyphens, in the form 8-4-4-4-12. UUIDs are often used to identify filesystems. UUID • • $ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-2.6.32-24-generic-pae root=UUID=32fffc527a96-4acb-b94d-d743617ab0c1 ro quiet splash • • • $ sudo blkid /dev/sda6: UUID="32fffc52-7a96-4acb-b94d-d743617ab0c1" TYPE="ext4" /dev/sda7: UUID="f0a604b1-bfd4-4100-a963-401b0f2bdd82" TYPE="swap" • • • $ cat /etc/fstab UUID=32fffc52-7a96-4acb-b94d-d743617ab0c1 / ext4 errors=remount-ro 0 1 UUID=f0a604b1-bfd4-4100-a963-401b0f2bdd82 none swap sw 0 0 • • $ uuidgen 8cd52053-1bdd-4192-8a90-788f9e972c88 LVM Logical volume management provides a higher-level view of the disk storage on a computer system than the traditional view of disks and partitions. This gives the system administrator much more flexibility in allocating storage to applications and users. Storage volumes created under the control of the logical volume manager can be resized and moved around almost at will, although this may need some upgrading of file system tools. The logical volume manager also allows management of storage volumes in user-defined groups, allowing the system administrator to deal with sensibly named volume groups such as "development" and "sales" rather than physical disk names such as "sda" and "sdb". http://tldp.org/HOWTO/LVM-HOWTO/ LVM • • # pvcreate /dev/sdd1 Physical volume "/dev/sdd1" successfully created • • # vgcreate vgrupa2 /dev/sdd1 Volume group "vgrupa2" successfully created • • # lvcreate -l100%FREE -n data2 vgrupa2 Logical volume "data2" created • • # mkfs.ext3 /dev/vgrupa2/data2 ... • pvdisplay, vgextend, lvresize, ... md Software RAID devices are so-called "block" devices, like ordinary disks or disk partitions. A RAID device is "built" from a number of other block devices - for example, a RAID-1 could be built from two ordinary disks, or from two disk partitions. RAID modes: linear, 0, 1, 4, 5, 6. mdadm - manage MD devices aka Linux Software RAID. http://raid.wiki.kernel.org/ md • # mdadm --create /dev/md1 --level=raid1 --raid-devices=2 /dev/sdb /dev/sdc • • • # cat /proc/mdstat md1 : active raid1 sdb[0] sdc[1] 39070016 blocks [2/2] [UU] • # mdadm --stop /dev/md1 • # mdadm --assemble /dev/md1 /dev/sdb /dev/sdc Upstart • Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running. • Known Users: – – – – – – – * Ubuntu 6.10 and later * Fedora 9 and later * Debian (as an option) * Nokia's Maemo platform * Palm's WebOS * Google's Chromium OS * Google's Chrome OS Upstart Tasks and Services are started and stopped by events. Events are generated as tasks and services are started and stopped. Events may be received from any other process on the system. Services may be respawned if they die unexpectedly. Supervision and respawning of daemons which separate from their parent process. Communication with the init daemon over D-Bus.