Dr. Michael Perzl - IBM Migration Factory, Germany October 2013 Open Source Software Solutions and Best Practices for AIX (pOX669) © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Trademarks The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both. Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market. Those trademarks followed by ® are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States. For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml: *, AS/400®, e business(logo)®, DBE, ESCO, eServer, FICON, IBM®, IBM (logo)®, iSeries®, MVS, OS/390®, pSeries®, RS/6000®, S/30, VM/ESA®, VSE/ESA, WebSphere®, xSeries®, z/OS®, zSeries®, z/VM®, System i, System i5, System p, System p5, System x, System z, System z9®, BladeCenter® The following are trademarks or registered trademarks of other companies. Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce. * All other products may be trademarks or registered trademarks of their respective companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography. 2 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Important reminder Please note: The opinions given in this presentation are my own and do not necessarily represent IBM‘s position. PS: I still like my job... 3 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Agenda Open source software – what are we talking about ? Customer expectation and a look at the competition A historical view – 12 years ago... Current open source software availability for AIX My personal repository: http:///www.perzl.org/aix Technical Part: AIX 32-bit physical address space Recommended compile environment Shared libraries on AIX GCC on AIX Compatibility issue with the original AIX Toolbox for Linux Applications AIX V6.1 and V7.1 specifics Summary 4 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Open source software – what are we talking about ? 5 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Open source software – what are we talking about ? Description of Open Source Software (OSS): Not only literally availability of source code but also defines a boundary on the usage, modification and redistribution of open source software Software licenses grant rights (on usage, modification and redistribution) to users which would otherwise be prohibited by copyright Properties of open source software: Licensed through one of the many available open source licenses May be licensed under more than one license type, commercial and open source – Good example here is Nokia‘s (formerly Trolltech) QT GUI-framework Used license may restrict usage for certain – typically commercial – purposes For our considerations: Software where the source code is (freely) available and licensed through one of the open source licenses 6 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Selected open source licenses Public domain vs. licensed software From: http://en.wikipedia.org/wiki/Comparison_of_free_software_licences Apache License Berkeley Database License BSD license Eclipse Public License IBM Public License GNU General Public License (GPL) GNU Lesser General Public License (LGPL) MIT license / X11 license Mozilla Public License Sun Public License and many more... 7 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Some prominent examples of open source software 8 Apache - many more besides web server OpenOffice.org Emacs PHP Firefox PostgreSQL Gaim – instant messenger RDist – file distribution Ganglia RSync GCC – GNU Compiler Collection Samba Gimp – image manipulation program Squid – proxy server GNOME (alternative GUI) Thunderbird gzip - file compression utility VIM (VI improved) JBOSS VLC Media Player KDE (alternative GUI) VNC LibreOffice (OpenOffice fork) GNU wget Net-SNMP Wireshark (previously called ethereal) Munin XEmacs MySQL .... and many more... © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Customer expectation and a look at the competition 9 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Why are customers using open source software ? Some reasons: Because most of this software is pretty good Because it provides additional functionality not present in the native UNIX implementation Different native implementations behave differently (e.g., differences in the Solaris, HP-UX and AIX implementation) They want to use the same tools across platforms 10 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Customer expectation Customers who have used open source software before have a certain expectation: – They are used to having open source tools easily and readily available (just download and install) • For our UNIX competitors (Oracle, HP) this is possible • Especially customers who also use Linux fall into that category – Typically they want the latest version or a rather up-to-date version of that software Linux and its software availability is clearly the measuring stick for UNIX here ! 11 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX A look at the UNIX competition* * Not really a competition anymore when looking at Unix market share! Oracle – URL: http://www.sunfreeware.com/ – Claims: • 19 years as the #1 site for Free Software Packages for Solaris • Currently offering more than 60 Gigabytes of packages for Solaris 2.5 – 2.10 • Used by over 528,000 unique IPs in 220 countries/territories – My personal remarks: • Huge selection of Open Source Software • Very much up-to-date – No official SUN support but “private project“ of Steven M. Christensen (and Associates, Inc.) HP – Porting and Archive Centre for HP-UX • URL UK: http://hpux.connect.org.uk/ – My personal remarks: • Huge selection of Open Source Software • Very much up-to-date – No official HP support but “private project“ of Connect Internet Solutions Limited 12 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX A historical view – 12 years ago... 13 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Do you still remember back in 2001… ? Next generation of AIX AIX 5L is the new name New AIX logo signals: – Leadership technology advancement – Openness and strong Linux affinity 14 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 5L: Linux affinity, why ? Benefits Expands the AIX application portfolio Allows Linux developers to use familiar tools on AIX – Common source code, tools, development environment Scalability and enhanced RAS features of AIX for Linux applications Allows Linux developers to easily reach the AIX install base – AIX the easiest and best UNIX platform to port to Increases the commonality between Linux and AIX AIX Linux 15 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 5L: Linux affinity implementation Implemented in two Phases First phase: Release of the AIX Toolbox for Linux Applications Popular "Linux - GNU" tools, utilities, look & feel on AIX For AIX 4.3.3 and AIX 5L 5.1 (shipped with every media) Linux Linux AIX AIX Second phase: Integration of Linux compatible APIs and header files into AIX 5L (header files and libraries) Application Programming Interface (API) approach to provide Linux source application interoperability ( actually not more than Linux Affinity on Application Level ! Not an environment or additional layer/wrapper to run Linux applications Application recompile necessary, recompiled application is native AIX application Tighter integration of application to OS than with Application Binary Interface (ABI) approach possible 16 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 5L: Linux affinity API approach (1/3) Definition of a common Linux base: Linux Distributions which were considered: – – – – RedHat 6.2 SuSE Linux 6.4 Debian GNU/Linux 2.25 Caldera OpenLinux 2.4 and compared to: – AIX 4.3.3 which included a development environment Determination of differences based upon: Common Linux files missing from AIX Common Linux library symbols missing from AIX 17 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 5L: Linux affinity API approach (2/3) Linux system calls: Employed documents – – – – – AIX 4.3.3 Linux kernel 2.2.14 GNU libc 2.1.3 Open Group Single UNIX Specification V2 (SUSv2) Linux Standards Base specification 0.2pre Taken from the system call table (syscalls.h) in the i386 port of Linux 2.2.14 Determine use frequency of incompatible Linux system calls and LSB subroutines – compared against the external symbols referenced during the building of over 500 RPM packages on RedHat 6.2 Determine use frequency of non-AIX Linux system calls and LSB subroutines – compared against the external symbols referenced during the building of over 500 RPM packages on RedHat 6.2 18 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 5L: Linux affinity API approach (3/3) Different groups of APIs: Linux APIs compatible with AIX – most system calls fall into this category Linux APIs not available on AIX – typically Linux specific system calls, e.g., vm86() Linux affinity on application source code level ! – Linux APIs introduced in AIX 5L Version 5.1 – see Redbook "Linux Applications on pSeries" for a detailed list Linux APIs available on AIX but not 100% compatible (less than 10) – Dual-semantic functions, plenty of #ifdef and typedef additions – Use preprocessor macro -D_LINUX_SOURCE_COMPAT flag to obtain wanted functionality Redbook resource: "Linux Applications on pSeries" (SG24-6033) web site: http://www.redbooks.ibm.com 19 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX Toolbox for Linux Applications Application Development – gcc, g++, gdb, rpm, cvs, automake, autoconf, libtool, bison, flex, gettext Desktop Environments – Gnome, KDE GNU base utilities – gawk, m4, indent, sed, tar, diffutils, fileutils, findutils, textutils, grep, sh-utils Programming Languages – guile, python, Tcl/Tk, rep-gtk System Utilities – emacs, vim, bzip2, gzip, git, elm, ncftp, rsync, wget, lsof, less, samba, zip, unzip, zoo Graphics Applications – ImageMagick, transfig, xfig, xpdf, ghostscript, gv, mpage Libraries – ncurses, readline, libtiff, libpng, libjpeg, slang, fnlib, db, gtk+, Qt System Shells – bash2, tcsh, zsh Window Managers – enlightenment, sawfish 20 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX IBM support for AIX Toolbox Applications IBM provides the GNU tools and other software in the AIX Toolbox for Linux Applications on an "as-is" basis. Linux Affinity version of GNU tools usually better integrated into AIX than if you downloaded the GNU software and compiled it yourself ! All patches made to the source code available Support via the Linux Affinity web sites (back then): http://www.ibm.com/servers/aix/products/aixos/linux/ – AIX Toolbox for Linux Applications homepage http://www.ibm.com/servers/aix/products/aixos/linux/faq.html – Frequently Asked Questions about the AIX Toolbox for Linux Applications http://oss.software.ibm.com/developerworks/projects/aixtoolbox/ – Project homepage within the IBM DeveloperWorks web hierarchy – Most useful site with regards to the AIX Toolbox for Linux Applications 21 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX Toolbox for Linux Applications over the years... The AIX Toolbox for Linux Applications was intended to give customers exactly what they want. In the beginning and first years this worked really well and was well received by our customers. However, legal issues have significantly limited the usefulnesss of the AIX Toolbox for Linux Applications the last 7 to 8 years. Current situation: – – – – Only bug fixes (security issues) are added to the AIX Toolbox packages Hardly any new packages have been released the last couple of years Hardly any new versions of packages have been released the last couple of years Important packages, e.g., Samba, are only available in “pre-historic“ versions In its current state the AIX Toolbox is not very usable anymore ! 22 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX The AIX Toolbox for Linux Applications is outdated... As of 08/2004: Please also check the posting of David Clissold (back then IBM maintainer of the Toolbox): http://sourceforge.net/p/aixtoolbox/mailman/message/6711692/ The issue with our distributing open source packages really isn't the building or packaging of the images; it is a legal question. We cannot distribute open source packages -- whether just a version update or (even rarer) a new package -- without extensive review and approval from the legal group. So having it built externally doesn't help get it onto our own site, but thanks for the offer. Note that this doesn't stop anyone from building packages themselves and sharing them with each other directly (though I know that's not what you were asking). Or even making their own images publicly available somewhere, from an independent site of their own. Bull AIX freeware site: UCLA AIX freeware site: 23 © 2013 IBM Corporation http://www.bullfreeware.com/ http://aixpdslib.seas.ucla.edu/ pOX669 - Open Source Software Solutions and Best Practices for AIX Current open source software availability for AIX 24 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Open source repositories for AIX (1/4) AIX Toolbox for Linux Applications – URL: http://www-03.ibm.com/systems/power/software/aix/linux/index.html Bull open source software archive for AIX – URL: http://www.bullfreeware.com/ Bull's continued version of the AIX Toolbox for Linux Applications – URL: http://www.bullfreeware.com/aixtoolbox/index.html Darren Tucker’s OpenSSH for AIX website: – Older versions of OpenSSH for AIX – URL: http://www.zipworld.com.au/~dtucker/openssh/ 25 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Open source repositories for AIX (2/4) Up to date web browsers for AIX, based on GTK+ and GTK2 – Firefox, Seamonkey and Thunderbird for AIX – URL: http://www.linkitup.de/ForAIX/ – No Firefox beyond version 3.6.25! Alternative desktop environments (instead of default AIX CDE environment) – KDE v3.4.3 • Unfortunately not downloadable from the KDE FTP Server anymore – was: ftp://ftp.kde.org/pub/kde/stable/3.4.3/AIX/ • You can still download a copy from my website: – http://oss4aix.org/download/KDE/ – GNOME for AIX5L (v2.14, v2.16): • Downloadable from Bull Open Source Software Archive for AIX – URL: http://www.bullfreeware.com/ – Direct URL: http://gnome.bullfreeware.com/new_index.html 26 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Open source repositories for AIX (3/4) RRDtool versions by Zoran Majcenic – http://oss.oetiker.ch/rrdtool/download.en.html – http://www.inet.hr/zmp/ibm/rrdtool Recent Python versions for AIX V6.1 by Sascha Thomas Spreitzer – https://spreitzer.name/python-2-6-5-for-ibm-aix-6-1 27 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Open source repositories for AIX (4/4) Hudson Valley Community College open source software for AIX – Provides packages in BFF format – URL: http://pware.hvcc.edu/ – Required AIX levels: • AIX 5.3 TL-09-06 or AIX 5.3 TL-10-04 or AIX 5.3 TL-11-04 or AIX 5.3 TL-12-02 • AIX 6.1 TL-03-06 or AIX 6.1 TL-04-06 or AIX 6.1 TL-05-02 or AIX 6.1 TL-06-01 • AIX 7.1 TL-00-01 – Notes: • AIX 5.2 not supported anymore • The AIX 5.3 code base no longer supported since April 30, 2012 Samba for AIX – http://us1.samba.org/samba/ftp/Binary_Packages/AIX/ – Note: They just link to Hudson Valley Community College – Just get it from my website! 28 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX My personal website: http://www.perzl.org/aix 29 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX My personal AIX open source repository (1/3) URL: http://www.perzl.org/aix/ Started 12/2005 because I was clearly frustrated with the current situation of the AIX Toolbox for Linux Applications Goal: – Completely replace the AIX Toolbox for Linux Applications • Not much missing anymore ☺ – Stay 100% upgrade-compatible to the AIX Toolbox for Linux Applications • Sometimes difficult Software availability: – – – – Currently more than 920 different packages available Total number of downloadable packages ≥ 16000 Total downloadable volume: 40+ GB of RPM packages Growing constantly… This is the “de-facto” AIX Toolbox now! 30 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX My personal AIX open source repository (2/3) So far user feedback has been superb! Number #1 ranking on Google (7 years in a row now!) Just enter “aix open source” on Google Available download protocols: FTP, HTTP, RSYNC I do this strictly in my free time – No relation to IBM on purpose (please see next slide) If you have a request for a specific package let me know and we’ll see what we can do… 31 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX My personal AIX open source repository (3/3) http://www.perzl.org/aix runs on an Intel x86_64 VServer (CentOS 6.2) Hosted by Strato (German ISP) with 100 GB disk space Unlimited traffic (otherwise I would go bankrupt) – I personally pay for the ISP services Website uses PmWiki as CMS (Content Management System) Personal plea: If this website has been – – – – useful to you convinced you to stay on AIX convinced you to buy even more AIX systems etc. please let me or IBM know. 32 © 2013 IBM Corporation Feedback is always very welcome ! pOX669 - Open Source Software Solutions and Best Practices for AIX Some download statistics from http://www.perzl.org/aix 33 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Technical part 34 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Technical part – Agenda Best practices: AIX 32-bit Physical Address Space Recommended Compile Environment Shared Libraries on AIX GCC on AIX Compatibility issue with the original AIX Toolbox for Linux Applications AIX V6.1 and V7.1 Specifics 35 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 32-bit physical address space 36 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 32-bit physical address space (1/3) Virtual memory: hardware provides continuous range of virtual memory to access data from 0x0000000000000 to 0xFFFFFFFFFFFFF total addressable space of more than 1,000 TB 52-bit representation required memory access instructions generate 32-bit addresses Consequence: segment::offset mechanism implemented to provide mapping of virtual to physical addresses 37 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 32-bit physical address space (2/3) 4 28 segment offset 32-bit program address segment register 38 24 28 segment offset © 2013 IBM Corporation 52-bit virtual memory address pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 32-bit physical address space (3/3) segment::offset mechanism addressing scheme provides access to 16 segments of up to 256 MB (= 228 bytes) each each segment register contains a 24-bit segment ID that becomes a prefix to the 28-bit offset, which together form the virtual memory address Process space is 32-bit address space programs use 32-bit pointers each process or interrupt handler can address only the system-wide virtual memory space (segment) whose segment IDs are in the segment register 39 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 32-bit address space layout (1/3) Segment number 0x0 0x1 0x2 0x3 : : : : 0xC 0xD 0xE 0xF 40 Kernel Process text Process private shmat / mmap 256 MB 256 MB 256 MB Available number 1 1 1 (attached with explicit compiler options) 2.56 GB 10 Shared library text shmat / mmap Shared library data 256 MB 256 MB 256 MB 1 1 1 Use © 2013 IBM Corporation Size pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 32-bit address space layout (2/3) Remarks: the kernel (0x0) and shared-library (0xD) text segment are shared by all processes and have restricted access when executing in the user protection domain the process text segment is held by segment 1 the process private segment is held by segment 2 and contains: – – – – user block (ignored from now on, small and fixed-size) kernel stack (ignored from now on, small and fixed-size) user stack user data regions the segments 3 through 12 may be loaded (using interfaces such as shmat or mmap) to provide access to files or shared memory data these conventions constrain the size of the user data and user stack regions (single shared 256 MB segment #2) 41 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 32-bit address space layout (3/3) user data initialized variables & constants in /etc/security/limits: fsize = core = cpu = data = rss = stack = 0x2000 0000 DATA uninitialized variables BSS malloc allocated space HEAP u.ulimit[CUR].data u.ulimit[MAX].data unused space u.ulimit[MAX].stack u.ulimit[CUR].stack user stack kernel stack user block 42 © 2013 IBM Corporation 0x2FF2 3000 0x2FF3 B400 pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 32-bit large address space layout (1/3) Large program address space model is supported to handle programs with requirements for large amounts of uninitialized data the process user data is relocated to start at segment 3 for as much data as the program needs up to a maximum of eight segments – the program can therefore have up to 2 GB of data – maximum attachable shared memory: 11 segments (2.75 GB) to enable the large address space model, use the -bmaxdata flag of the ld command Other aspects of the program address space remain unchanged: the user stack, kernel stack and user block continue to reside in segment 2 as a result of this organizational scheme, the user stack is still limited by the size of segment 2 43 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 32-bit large address space layout (2/3) 0x0 kernel 0x1 process text 0x2 process private user data shmat/mmap 0x3 0x4 segment 2: 0x2000 0000 unused space 0x5 0x6 0x7 0x8 u.ulimit[MAX].stack 0x9 0xA 0xB 0xC 0xE shared library text shmat / mmap 0xF shared library data 0xD 44 © 2013 IBM Corporation user stack kernel stack user block 0x2FF2 3000 0x2FF3 B400 pOX669 - Open Source Software Solutions and Best Practices for AIX AIX 32-bit large address space layout (3/3) -bmaxdata option of the ld command to link a program that will have the maximum eight segments reserved to it, the following command line could be used: cc sample.o -bmaxdata:0x80000000 the number 0x80000000 is the number of bytes, in hex format, equal to eight 256 MB segments use -bmaxdata option if the program needs large data areas use only multiples of 0x10000000 the maxdata value is stored in the o_maxdata field of the XCOFF (Extended Common Object File Format) header (AIX uses XCOFF as binary format) 45 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Check and set the o_maxdata field (1/2) Check the setting of the o_maxdata field root@aix:/> dump -ov <binary> ..... maxSTACK 0x00000000 maxDATA 0x80000000 SNbss magic 0x0003 0x010b modtype 1L Script to set the o_maxdata field to 0x80000000 for any binary: # patch 32-bit binary to use the maximum number of 8 x 256MB data segments f=<binary> if [[ "`/usr/bin/file ${f} | awk '{ print $2 }'`" = "executable" ]] ; then /usr/bin/echo '\0200\0\0\0' | \ /usr/bin/dd of=${f} bs=4 count=1 seek=19 conv=notrunc fi Alternatively, set the "LDR_CNTRL=MAXDATA=0x80000000" environment variable 46 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Check and set the o_maxdata field (2/2) You can also modify an existing program so that it will use the large or very large address-space model. To set the maxdata value in an existing 32-bit XCOFF program, a.out, to 0x80000000, use the following command: /usr/ccs/bin/ldedit -bmaxdata:0x80000000 a.out 47 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Recommended compile environment 48 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX My compile environments (1/2) Compiling on AIX5L v5.1 – AIX 5.1 ML 04 plus some selected ML 05 filesets – IBM XL C/C++ compiler version 7 with all applicable patches for AIX 5.1 ML 04 Compiling on AIX5L v5.2 – AIX 5.2 ML 01 – IBM XL C/C++ compiler version 7 with all applicable patches for AIX 5.2 ML 01 Compiling on AIX5L v5.3 – AIX 5.3 ML 01 – IBM XL C/C++ compiler version 7 with all applicable patches for AIX 5.3 ML 01 Compiling on AIX v6.1 – AIX 6.1 TL 00 – IBM XL C/C++ compiler version 9 with all applicable patches for AIX 6.1 TL 00 Compiling on AIX v7.1 – AIX 7.1 TL 00 – IBM XL C/C++ compiler version 11.1 with all applicable patches for AIX 7.1 TL 00 49 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX My compile environments (2/2) Remarks: I always use the BASH shell if possible Main development system is AIX5L v5.1 For the different AIX versions I use the environment settings listed on the following slides If any different settings are required they are explicitely specified in the RPM spec file Strategy: Try to build every library as a shared library – You can always statically link a shared library in AIX! Build always 32-bit and 64-bit versions 50 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Environment settings for IBM XL C/C++ (1/3) AIX5L v5.1: export CONFIG_SHELL=/opt/freeware/bin/bash export CONFIG_ENV_ARGS=/opt/freeware/bin/bash export CC=cc export CFLAGS="-qmaxmem=16384 -DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 -D_AIX51 \ -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include" export CXX=xlC export CXXFLAGS=$CFLAGS export F77=xlf export FFLAGS="-qmaxmem=16384 -O -I/opt/freeware/include" export LD=ld export LDFLAGS="-L/opt/freeware/lib -Wl,-bmaxdata:0x80000000" export PATH=/usr/bin:/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/vac/bin: \ /usr/vacpp/bin:/usr/ccs/bin:/usr/dt/bin:/usr/opt/perl5/bin:/opt/freeware/bin: \ /opt/freeware/sbin:/usr/local/bin:/usr/lib/instl 51 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Environment settings for IBM XL C/C++ (2/3) AIX5L v5.2: Same as for AIX5L v5.1 except the definition of $CFLAGS export CFLAGS="-qmaxmem=16384 -DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 \ -D_AIX51 -D_AIX52 \ -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include" AIX5L v5.3: Same as for AIX5L v5.1 except the definition of $CFLAGS export CFLAGS="-qmaxmem=16384 -DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 \ -D_AIX51 -D_AIX52 -D_AIX53 \ -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include" 52 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Environment settings for IBM XL C/C++ (3/3) AIX v6.1: Same as for AIX5L v5.1 except the definition of $CFLAGS export CFLAGS="-qmaxmem=16384 -DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 \ -D_AIX51 -D_AIX52 -D_AIX53 -D_AIX61 \ -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include" AIX v7.1: Same as for AIX5L v5.1 except the definition of $CFLAGS export CFLAGS="-qmaxmem=16384 -DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 \ -D_AIX51 -D_AIX52 -D_AIX53 -D_AIX61 -D_AIX71 \ -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include" 53 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Environment settings for GCC (1/3) AIX5L v5.1: export CONFIG_SHELL=/opt/freeware/bin/bash export CONFIG_ENV_ARGS=/opt/freeware/bin/bash export CC=gcc export CFLAGS="-DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 -D_AIX51 \ -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include" export CXX=g++ export CXXFLAGS=$CFLAGS export F77=g77 export FFLAGS="-O -I/opt/freeware/include" export LD=ld export LDFLAGS="-L/opt/freeware/lib -Wl,-bmaxdata:0x80000000" export PATH=/usr/bin:/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/vac/bin: \ /usr/vacpp/bin:/usr/ccs/bin:/usr/dt/bin:/usr/opt/perl5/bin:/opt/freeware/bin: \ /opt/freeware/sbin:/usr/local/bin:/usr/lib/instl 54 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Environment settings for GCC (2/3) AIX5L v5.2: Same as for AIX5L v5.1 except the definition of $CFLAGS export CFLAGS="-DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 -D_AIX51 \ -D_AIX52 \ -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include" AIX5L v5.3: Same as for AIX5L v5.1 except the definition of $CFLAGS export CFLAGS="-DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 -D_AIX51 \ -D_AIX52 -D_AIX53 \ -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include" 55 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Environment settings for GCC (3/3) AIX v6.1: Same as for AIX5L v5.1 except the definition of $CFLAGS export CFLAGS="-DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 -D_AIX51 \ -D_AIX52 -D_AIX53 -D_AIX61 \ -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include" AIX v7.1: Same as for AIX5L v5.1 except the definition of $CFLAGS export CFLAGS="-DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 -D_AIX51 \ -D_AIX52 -D_AIX53 -D_AIX61 -D_AIX71 \ -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include" 56 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Large file (≥ 2GB) support for 32-bit AIX binaries Must define the following CPP (C preprocessor) symbol: -D_LARGE_FILES Sometimes in source code you find also defines like -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE but they are only Linux-centric and not necessary for AIX! Recommended settings: export CC="xlc_r -D_LARGE_FILES" 57 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Shared libraries on AIX 58 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Some facts about shared libraries on AIX (1/5) Objects files are defined by XCOFF (eXtended Common Object File Format) on AIX. AIX provides two different development environments: – 32-bit – 64-bit Object files on AIX have two different bit modes: – 32-bit object files – 64-bit object files To build executable files, all object files and archive members must be in the same bit mode, either 32 or 64. To distinguish the bit mode of object files, use the file command: $ file *.o foo1.o: executable (RISC System/6000) or object module not stripped foo2.o: executable (RISC System/6000) or object module not stripped foo3.o: 64-bit XCOFF executable or object module not stripped In this example, foo3.o is a 64-bit object, whereas foo1.o and foo2.o are 32-bit. AIX object files are always position independent code (PIC), i.e., no absolute jumps, only relative jumps within the code segment. The GCC "–fPIC" option is not necessary on AIX. 59 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Some facts about shared libraries on AIX (2/5) Multiple object files can be archived into a single library archive file (libfoo.a). Hybrid mode library archives An AIX library archive can contain both the 32- and 64-bit object modules as its members. This is called a hybrid mode library archive. Most AIX system libraries are hybrid mode. Utility commands Utility commands that deal with object files, such as ar, dump, and nm, have been enhanced with the -X option in order to support the 64-bit XCOFF format object format. Possible options (with or without space): -X32 -X64 -X32_64 Alternatively set the “OBJECT_MODE“ variable 60 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Some facts about shared libraries on AIX (3/5) Difference between shared and static objects on AIX On many UNIX operating systems, a shared object file normally has a file name extension ".so" (stands for shared object) and a static object file has normally a file name extension ".o" (stands for object). However, an object normally has a file name extension ".o" regardless of shared or static on AIX. Therefore, you cannot determine whether an object is shared or regular static object from the file name extension on AIX. To determine whether the object file is shared or static, use the dump command. Note: AIX also supports shared objects with a file name extension ".so" like other UNIX operating systems do, which are used for run-time linking. 61 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Some facts about shared libraries on AIX (4/5) Difference between the terms shared library and shared object on AIX: Shared object – A shared object is a single object file that has the SHROBJ flag in the XCOFF header. – A shared object normally has a name of the form name.o on AIX. Shared library – A shared library refers to an ar format archive library file, where one or more of the archive members is a shared object. – Note that the library can also contain regular, non-shared object files, which are handled in the normal way by the linker. – A shared library normally has a name of the form libname.a on AIX. – The magic number of the file is used by the linker to determine whether the file is valid object file or not. Note: Shared libraries (also often referred to as dynamic link libraries or DLL) on other UNIX operating systems are actually shared objects, whereas multiple shared objects can be contained in a single library on AIX. 62 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Some facts about shared libraries on AIX (5/5) Shared objects comparison with Linux: AIX: – /usr/lib/libfoo.a contains the shared object foo.so.1 – For different versions of foo.so the library archive /usr/lib/libfoo.a must contain these different versions, e.g., foo.so.1, foo.so.2, foo.so.3 Linux: – /usr/lib/libfoo.so.1 is the shared object, compiled with "-fPIC" – Different versions of foo.so are just part of the file system, i.e., /usr/lib/libfoo.so.1 /usr/lib/libfoo.so.2 /usr/lib/libfoo.so.3 /usr/lib/libfoo.so –> libfoo.so.3 – /usr/lib/libfoo.a contains the statically compiled (without "-fPIC") members of foo 63 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Hardcoded shared library search path To check use the “dump -Hv <binary>“ command: root@aix51:/> dump -Hv <binary> ...... ***Import File Strings*** INDEX PATH BASE 0 /opt/freeware/lib:/usr/vac/lib:/usr/lib:/lib 1 libc.a 2 /opt/freeware/lib libintl.a MEMBER shr.o libintl.so.1 To hardcode it in your 32-bit binary use: export LDFLAGS="-L/opt/freeware/lib \ -Wl,-blibpath:/opt/freeware/lib:/usr/lib:/lib \ -Wl,-bmaxdata:0x80000000" To hardcode it in your 64-bit binary use: export LDFLAGS="-L/opt/freeware/lib64 -L/opt/freeware/lib \ -Wl,-blibpath:/opt/freeware/lib64:/opt/freeware/lib:/usr/lib:/lib \ -Wl,-bmaxdata:0x80000000" 64 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Recommended further reading IBM Redbook SG24-5674: Developing and Porting C and C++ Applications on AIX URL: http://www.redbooks.ibm.com/abstracts/sg245674.html IBM Redbook SG24-6034: AIX 5L Porting Guide URL: http://www.redbooks.ibm.com/abstracts/sg246034.html 65 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX GCC on AIX 66 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX GCC availability chart (from my website) (1/2) AIX5L V5.1 AIX5L V5.2 AIX5L V5.3 AIX V6.1 AIX V7.1 GCC V4.8.2 – – yes yes yes GCC V4.8.1 – – yes yes yes GCC V4.8.0 – – yes yes yes GCC V4.7.3 – – yes yes yes GCC V4.7.2 – – yes yes yes GCC V4.7.1 – – yes yes yes GCC V4.7.0 – – yes yes yes GCC V4.6.4 – soon yes yes yes GCC V4.6.3 – yes yes yes yes GCC V4.6.2 – yes yes yes yes GCC V4.6.1 – yes yes yes yes GCC V4.6.0 – yes yes yes – continued... 67 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX GCC availability chart (from my website) (2/2) 68 AIX5L V5.1 AIX5L V5.2 AIX5L V5.3 AIX V6.1 AIX V7.1 GCC V4.5.4 yes yes yes yes yes GCC V4.5.3 yes yes yes yes yes GCC V4.5.2 yes yes yes – – GCC V4.4.7 yes yes yes yes yes GCC V4.4.6 yes yes yes yes yes GCC V4.4.5 yes yes yes yes – GCC V4.3.6 yes yes yes yes yes GCC V4.3.5 yes yes yes yes – GCC V4.2.4 yes yes yes yes – GCC V4.2.3 yes yes yes yes – GCC V4.2.2 yes yes yes yes – © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX GCC error message "gcc -g ..." fails with an error message similar like that: ld: 0711-596 SEVERE ERROR: Object xxx.o An RLD for section 2 (.data) refers to symbol 111, but the storage class of the symbol is not C_EXT or C_HIDEXT. The source file contains implicitly initialized global symbols. “Interim fix“: # cd /usr/ccs/bin # mv as as.ORIG # cp <Your working assembler copy from a previous AIX level, here V6.1 TL00> as.61TL00 # ln -s as.61TL00 as 69 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX List of APARs with fixes available now: AIX V7.1 TL00: IZ99107 – – IZ99107: ASSEMBLER GENERATES INVALID OBJECT FILE WITH GCC-PRODUCED FILES fixed in AIX V7.1 TL01 SP0, i.e., must upgrade to TL01 (7100-01-00-1140, bos.adt.base.7.1.1.0) until AIX V7.1 TL00 SP4 is available AIX V6.1 TL06: IZ98226 – – IZ98226: ASSEMBLER GENERATES INVALID OBJECT FILE WITH GCC-PRODUCED FILES fixed in AIX V6.1 TL06 SP6 (6100-06-06-1140, bos.adt.base.6.1.6.16) AIX V6.1 TL05: IZ98732 – – IZ98732: ASSEMBLER GENERATES INVALID OBJECT FILE WITH GCC-PRODUCED FILES fixed in AIX V6.1 TL05 SP7 (6100-05-07-1140, bos.adt.base.6.1.5.3) AIX V6.1 TL04: IV00150 – – IV00150: ASSEMBLER GENERATES INVALID OBJECT FILE WITH GCC-PRODUCED FILES fixed in AIX V6.1 TL04 SP11 (6100-04-11-1140, bos.adt.base.6.1.4.4) AIX V6.1 TL03: IZ98861 – – IZ98861: ASSEMBLER GENERATES INVALID OBJECT FILE WITH GCC-PRODUCED FILES as AIX V6.1 TL03 is not supported anymore, no fix for TL03 is available, instead you have to upgrade to a later TL AIX5L V5.3 TL12: IZ98134 – – – IZ98134: ASSEMBLER GENERATES INVALID OBJECT FILE WITH GCC-PRODUCED FILES APPLIES TO AIX 5300-12 fix available at https://www-304.ibm.com/support/docview.wss?uid=isg1fixinfo123037 fix contained in AIX5L V5.3 TL12 SP4 AIX5L V5.3 TL11: IZ98477 – – – IZ98477: ASSEMBLER GENERATES INVALID OBJECT FILE WITH GCC-PRODUCED FILES APPLIES TO AIX 5300-11 fix available at https://www-304.ibm.com/support/docview.wss?uid=isg1fixinfo123037 fix contained in AIX5L V5.3 TL11 SP7 AIX5L V5.3 TL10: IZ98385 – – – 70 IZ98385: ASSEMBLER GENERATES INVALID OBJECT FILE WITH GCC-PRODUCED FILES APPLIES TO AIX 5300-10 fix available at https://www-304.ibm.com/support/docview.wss?uid=isg1fixinfo123037 fix contained in AIX5L V5.3 TL10 SP7 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Compatibility issue with the original AIX Toolbox for Linux Applications 71 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Compatibility issue with the original AIX Toolbox for Linux Applications (1/3) I strive to keep my RPM packages 100%-compatible with the RPM packages of the AIX Toolbox for Linux Applications. However, there are some things beyond my control. Whenever the rpm.rte AIX LPP gets updated – for whatever reason, e.g., a technology level update – problems may arise if you have installed newer versions of the following packages from my website: – – – – – 72 bzip2 gettext info popt zlib © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Compatibility issue with the original AIX Toolbox for Linux Applications (2/3) This update of rpm.rte blindly overwrites any newly installed versions of those RPM packages in /opt/freeware/lib. I consider this behavior broken but there is nothing in my power to change that. Newer versions of rpm.rte at least issue a statement during installation like that: Warning: the contents of the following RPM packages will be overwritten. If these packages did not come from the AIX Toolbox for Linux Applications, it may be necessary to reinstall them. bzip2-1.0.5-1 gettext-0.17-1 info-4.12-1 zlib-1.2.3-5 73 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Compatibility issue with the original AIX Toolbox for Linux Applications (3/3) Most problems arose with updated versions of the gettext package. Therefore, if you get the following error message: ... 0509-150 Dependent module /opt/freeware/lib/libintl.a(libintl.so.8) could not be loaded. 0509-152 Member libintl.so.8 is not found in archive ... you have two options (number #1 is recommended): 1. Please update to a later version of the RPM package as I have since then recompiled all my packages to only use the basic versions of the AIX Toolbox. 2. Please re-install the current gettext-0.17 RPM from my website again with the following command as root user: rpm -Uvh --force gettext-0.17*rpm Note: 74 Please do not update the gettext RPM! © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX V6.1 and V7.1 specifics 75 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX AIX V6.1 and V7.1 specifics (1/2) Situation: The configure script does not detect the shared library capability on AIX V6.1/7.1 Check the following items: Check that your have autoconf, automake, libtool, m4 versions are not older than January, 2008! – Older versions still have explicit checks like “aix4* | aix5*)“ in these scripts, thus neglecting AIX V6.1/7.1 while the newer versions have checks like “[[aix4-9]]*)“ Check that the config.guess script knows about AIX V6.1 and V7.1! – config.guess tries to “guess“ the correct value for the OS • Wrong: rs6000-ibm-aix • Correct: powerpc-ibm-aix6.1.0.0 – A “grep AIX config.guess“ must return the following output: *:AIX:*:[45]) • Wrong: • Correct: *:AIX:*:[4567]) or *:AIX:*:[4-9]) Check if the situation after an “autoreconf“ has improved? Still no solution in sight? 76 © 2013 IBM Corporation contact me pOX669 - Open Source Software Solutions and Best Practices for AIX AIX V6.1 and V7.1 specifics (2/2) AIX V6.1 and V7.1 specific versions are not necessary – i.e., all RPM packages compiled on previous AIX versions should be fine due to the excellent AIX binary compatibility – except when: – newer X11 libraries are required, as AIX V6.1/7.1 has rather recent X11 libraries e.g., libXcursor, libXpm, libXrender etc. – certain AIX features – e.g., AME (Active Memory Expansion) – are required. – platform-specific and OS level-specific header files are required, e.g., for GCC, lsof. 77 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Summary 78 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Summary Open Source Software is important to AIX as our clients are using it A viable alternatives to the AIX Toolbox for Linux Applications exists please use my personal website now Building Open Source Software on AIX has become rather easy nowadays – The AIX APIs are more "mainstream" than in earlier years – The Open Source community in the meantime has learnt do deal with the "quirks" and peculiarities of AIX 79 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Questions ? Thank you for your attention ! 80 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Backup charts 81 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX The GNU build system – aka autotools 82 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Building open source software yourself Advantages: It is in many cases really easy to do, e.g., GNU software only requires $ ./configure $ make $ make install Disadvantages: Requires ressources, e.g., application development environment and compiler No automatic software dependency checking You have to do the "regression testing" Not easily transferable from one system to another without using a package management system 83 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX The GNU build system – aka autotools (1/2) Properties of the GNU build system – aka autotools A suite of programming tools produced by the GNU project These tools are designed to assist in making various source code packages portable to many Unix-like systems. The GNU build system is part of the GNU toolchain and is widely used in many free software and open source packages. While the tools comprising the GNU build system themselves are GPL-ed free software, there are no restrictions in using them in making non-free software portable. 84 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX The GNU build system – aka autotools (2/2) Tools included in the GNU build system Autoconf Automake Libtool Other related tools frequently used with the GNU build system are GNU make GNU gettext pkg-config GNU Compiler Collection, also called GCC 85 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX GNU autoconf Autoconf processes files (configure.in or configure.ac) with the help of GNU m4 to generate a configure script. When running the generated configure script, other template files, customarily ending in ".in", such as Makefile.in, are processed to create their final output, in this case a Makefile. Autoconf is used to attempt to work around the quirks found in various Unix-like operating systems. – For example, some Unix-like systems may have certain facilities that are known to be broken or missing entirely. – Autoconf creates a shell script which can detect these, and the program can work around them. The final output of the Autoconf utility is the configure script. Autoconf comes with several auxiliary programs designed to ease the creation of configure.ac: – autoheader, which is used to help manage C header files – autoscan, which can create an initial input file for autoconf – ifnames, which can list C pre-processor identifiers used in the program. 86 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX GNU automake and libtool GNU automake: Automake aims to allow the programmer to write a makefile in a higher-level language, rather than having to write the whole makefile manually. Automake helps to create portable Makefiles, which are in turn processed with the make utility. It takes its input as Makefile.am and turns it into Makefile.in, which is used by the configure script to generate the file Makefile output. GNU libtool: Libtool helps manage the creation of static and dynamic libraries on various Unix-like operating systems. Libtool accomplishes this by abstracting the library creation process, hiding differences between various systems (e.g., Linux vs. Solaris, AIX etc.). 87 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX What if my software has no configure script ? One option is to execute a series of commands at a shell prompt: $ $ $ $ aclocal autoconf autoheader automake Another common convention is to provide a script, often named autogen.sh, that runs all the needed pre-build tools. In some cases, one can also use: $ autoreconf which automatically invokes the above commands as needed. 88 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Flow diagram of autoconf and automake autoscan editor configure.ac Makefile.am aclocal autoheader automake aclocal.m4 config.h.in Makefile.in Developer autoconf configure Makefile make 89 © 2013 IBM Corporation User pOX669 - Open Source Software Solutions and Best Practices for AIX Building RPM packages on AIX 90 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Building RPM packages on AIX The RPM package manager Introduction and overview Best practices General documentation about RPM available at: http://wiki.rpm.org/Docs http://www.rpm.org/RPM-HOWTO/ 91 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Building RPM packages on AIX – the RPM package manager 92 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX The RPM package manager RPM is the RPM Package Manager. It is an open packaging system available for anyone to use. It maintains a database of all packages and their files that can be used for verifying packages and querying for information about files and/or packages. Originally developed by Red Hat, Inc., it has become the package manager for most Linux distributions. RPM on AIX: RPM on AIX is installed via the BFF package rpm.rte AIX still uses version 3 of RPM while all Linux distributions are now at version 4 RPM version 4 has split the rpm command into rpm and rpmbuild. The directory prefix for all RPM software on AIX is /opt/freeware. 93 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Directory structure of RPM Directory structure under /opt/freeware/src/packages: BUILD RPMS noarch ppc SOURCES SPECS SRPMS 94 © 2013 IBM Corporation here the RPM package is extracted and compiled generated architecture-independent RPMs generated binary RPMs for AIX all sources and patches all spec files generated source RPMs pOX669 - Open Source Software Solutions and Best Practices for AIX Useful RPM commands (1/2) Query RPM information: Show all installed RPM files rpm -qa List contents of a specfic RPM file rpm -qpl <rpm-file> Show information about an RPM file rpm -qpi <rpm-file> Show package requirements of an RPM file rpm -qp --requires <rpm-file> To which RPM file belongs the file /usr/bin/abc123 ? rpm -qf /usr/bin/abc123 Show pre/post install scripts of an RPM file rpm -qp --scripts <rpm-file> Show changelog history of an RPM file rpm -qp --changelog <rpm-file> 95 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Useful RPM commands (2/2) Installation and Removal of RPM files: Install an RPM file rpm -ivh <rpm-file> Update an RPM file (typically better than the „-ivh" option) rpm -Uvh <rpm-file> Update (Freshen) only installed RPM files rpm -Fvh *.rpm Remove an RPM file rpm -e <rpm-file> Verification of RPM files: Verify complete RPM database (did you erase a file by accident ?) rpm -Va 96 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Building RPM packages on AIX – introduction and overview 97 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Building RPM packages Basic procedure to build an RPM: Get the source code you are building the RPM for to build on your system. Make a patch of any changes you had to make to the sources to get them to build properly. – RPM by design uses the pristine sources from the original web site – Any necessary changes are to be supplied via patch files Make a spec file for the package. Make sure everything is in its proper place. Build the package using RPM. 98 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX The Spec File – “dissected" The spec file contains the build instructions for the package: Header %prep %build %install %clean optional pre and post install/uninstall scripts – – – – %pre %post %preun %postun %files %changelog 99 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX The RPM header (1/5) Header standard fields that must be filled: Summary: – This is a one line description of the package. Name: – This must be the name string from the rpm filename you plan to use. Version: – This must be the version string from the RPM filename you plan to use. Release: – The release number for a package of the same version (i.e., if we make a package and find it to be slightly broken and need to make it again, the next package would be release number 2). Copyright: – This line tells how a package is copyrighted. – You should use something like GPL, BSD, MIT, public domain, distributable, or commercial. 100 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX The RPM header (2/5) Group: – This line is used to tell high level installation programs where to place this particular program in its hierarchical structure. – The group tree currently looks something like this: Amusements/Games Amusements/Graphics Applications/Archiving Applications/Communications Applications/Databases Applications/Editors Applications/Emulators Applications/Engineering Applications/File Applications/Internet Applications/Multimedia Applications/Productivity Applications/Publishing Applications/System Applications/Text 101 © 2013 IBM Corporation Development/Debuggers Development/Languages Development/Libraries Development/System Development/Tools Documentation System Environment/Base System Environment/Daemons System Environment/Kernel System Environment/Libraries System Environment/Shells User Interface/Desktops User Interface/X User Interface/X Hardware Support pOX669 - Open Source Software Solutions and Best Practices for AIX The RPM header (3/5) Source: – This line points at the HOME location of the pristine source file – It is used if you ever want to get the source again or check for newer versions – Note: • The filename in this line must match the filename you have on your system, i.e., don't download the source file and change its name – You can also specify more than one source file using lines like: Source0: foo−0.tar.gz Source1: bar−1.tar.gz Source2: abc.tar.gz These files would go in the SOURCES directory 102 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX The RPM header (4/5) Patch: – This is the place you can find the patch if you need to download it again – Note: • The filename here must match the one you use when you make your patch. • You may also want to note that you can have multiple patch files much as you can have multiple sources. – You would have something like: Patch0: foo−0.patch Patch1: bar−1.patch Patch2: abc.patch These files would go in the SOURCES directory 103 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX The RPM header (5/5) BuildRoot: – This line allows you to specify a directory as the "root" for building and installing the new package. – You can use this to help test your package before having it installed on your machine. %description – It's not really a header item, but should be described with the rest of the header. – You need one description tag per package and/or subpackage. – This is a multi−line field that should be used to give a comprehensive description of the package. 104 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX %prep (1/3) Second section in the spec file Used to get the sources ready to build Here you need to do anything necessary to get the sources patched and setup like they need to be setup to do a make. Things to note: Each of these sections is really just a place to execute shell scripts. You could simply make an sh script and put it after the %prep tag to unpack and patch your sources. Macros exist to help you. 105 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX %prep (2/3) The %setup macro In its simplest form (no command line options), it simply unpacks the sources and cd's into the source directory. Options: −n name • will set the name of the build directory to the listed name. • The default is %{name}−%{version}. −c • will create and cd to the named directory before doing the untar. −b # • will untar Source# before cd'ing into the directory (and this makes no sense with −c so don't do it). • Only useful with multiple source files −a # • will untar Source# after cd'ing into the directory −T • This option overrides the default action of untarring the source and requires a −b 0 or −a 0 to get the main source file untarred • You need this when there are secondary sources −D • Do not delete the directory before unpacking • Only useful where you have more than one setup macro • Should only be used in setup macros after the first one (but never in the first one) 106 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX %prep (3/3) The %patch macro This macro helps automate the process of applying patches to the sources. Options: # • will apply Patch# as the patch file −p # • specifies the number of directories to strip for the GNU patch(1) command −p • The default action is to apply Patch (or Patch0). This flag inhibits the default action and will require a 0 to get the main source file untarred. This option is useful in a second (or later) %patch macro that required a different number than the first macro. −b extension • will save originals as filename.extension before patching. – You can also do %patch# instead of doing the real command: %patch # −p After those macros you can do any other setup you need to do via sh type scripting. Anything you include up until the %build macro (discussed in the next slide) is executed via sh. 107 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX %build There aren't really any macros for this section. You should just put any commands here that you would need to use to build the software once you had untarred the source, patched it, and cd'ed into the directory. This is just another set of commands passed to sh, so any legal sh commands can go here (including comments). Your current working directory is reset in each of these sections to the toplevel of the source directory, so keep that in mind. – You can cd into subdirectories if necessary. The variable RPM_OPT_FLAGS is set using values in /usr/freeware/lib/rpm/rpmrc. – Look there to make sure you are using values appropriate for your system or – simply don't use this variable in your spec file as it is optional. – Instead define your compile settings. 108 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX %install There aren't really any macros for this section. You basically just want to put whatever commands here that are necessary to install. If you have make install available to you in the package you are building, put that here. If not, you can either patch the makefile for a make install and just do a make install here, or you can hand install them here with sh commands. You can consider your current directory to be the toplevel of the source directory. The variable RPM_BUILD_ROOT is available to tell you the path set as the BuildRoot: in the header. – Using build roots are optional but are highly recommended because they keep you from cluttering your system with software that isn't in your RPM database. 109 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX %clean It's a good idea to always make sure there is a clean build root before building a package a second time on a system. The %clean macro will help with that. Simply put the proper commands there to remove a former build root. It is suggested to test that RPM_BUILD_ROOT wasn't set to / before doing something this volatile: %clean [ "${RPM_BUILD_ROOT}" != "/" ] && rm -rf ${RPM_BUILD_ROOT} 110 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Optional pre and post Install/Uninstall Scripts You can put scripts in the spec file that are run before and after the installation and uninstallation of binary packages. Macros for each of the scripts: – – – – %pre %post %preun %postun is the macro to do pre−install scripts is the macro to do post−install scripts is the macro to do pre−uninstall scripts is the macro to do post−uninstall scripts The contents of these sections should just be any sh style script, though you do not need the #!/bin/sh. 111 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX %files (1/2) Here you must list the files for the binary package. RPM has no way to know what binaries get installed as a result of make install. Available macros: %doc used to mark documentation in the source package that you want installed in a binary install – The documents will be installed in /opt/freeware/doc/%{name}−%{version}−%{release}. – You can list multiple documents on the command line with this macro, or you can list them all separately using a macro for each of them. %config used to mark configuration files in a package – This includes files like sendmail.cf, passwd, etc. – If you later uninstall a package containing config files, any unchanged files will be removed and any changed files will get moved to their old name with a .rpmsave appended to the filename. – You can list multiple files with this macro as well. 112 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX %files (2/2) %dir marks a single directory in a file list to be included as being owned by a package – By default, if you list a directory name WITHOUT a %dir macro, EVERYTHING in that directory is included in the file list and later installed as part of that package. %defattr allows you to set default attributes for files listed after the defattr declaration – The attributes are listed in the form (mode, owner, group) where the mode is the octal number representing the bit pattern for the new permissions (like chmod would use), owner is the username of the owner, and group is the group you would like assigned. – You may leave any field to the installed default by simply placing a − in its place. %files −f <filename> allows you to list your files in some arbitrary file within the build directory of the sources – This is nice in cases where you have a package that can build it's own filelist. – You then just include that filelist here and you don't have to specifically list the files. The biggest caveat in the file list is listing directories. If you list /usr/bin by accident, your binary package will contain every file in /usr/bin on your system. 113 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX %changelog Log of what changes occurred when the package is updated If you are modifying an existing RPM it is a good idea to list what changes you made here. Simple format: Start each new entry with a line with a '*' followed by the date, your name, and your email address. The date should appear in the same format that is output by: date +"%a %b %d %Y" The rest of the section is a free text field, but should be organized in some coherent manner. 114 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Build the Package with RPM (1/2) Once you have a spec file, you are ready to try and build your package. The most useful way to do it is with a command like the following: rpm −ba mypackage−1.0−1.spec Other useful options with the −b switch: p just run the %prep section of the spec file l do a list check on %files c do a %prep and %build i do a %prep, %build, and %install b do a %prep, %build, %install, and %build a binary package only a build it all (both source and binary packages) 115 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Build the Package with RPM (2/2) There are several modifiers to the −b switch: −−short−circuit – will skip straight to a specified stage (can only be used with c and i) −−clean – removes the build tree when done −−keep−temps – will keep all the temp files and scripts that were made in /tmp – You can see what files were created in /tmp using the −v option −−test – does not execute any real stages, but does keep−temp 116 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Building RPM packages on AIX – Best Practices 117 © 2013 IBM Corporation © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Best Practices First, compile the software before starting with the SPEC file Eliminate any compile errors, i.e., produce appropriate AIX patches If the software permits – most GNU software does – run a "make check" Do a "test-installation" into some temporary directory – Do a "./configure ––prefix=<tmp-dir-location>" Then, start with a SPEC file template For libraries, do a 32-bit build first – Once this works, try the 64-bit build – Then combine the two From the "test-installation" you can already determine what gets installed and where – Put this information into the %files section Check if a "rpm –bb <spec-file>" is successful ? – If yes, then also build the source RPM 118 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Some more hints... Useful RPM macro: – %ifos – possible values include: aix5.1, aix5.2, aix5.3, aix6.1, aix7.1 %ifos aix5.1 %define buildhost powerpc-ibm-aix5.1.0.0 Requires: AIX-rpm >= 5.1.0.0 %endif Sometimes GNU make is required instead of AIX make %{_tmppath} on AIX resolves to /var/tmp How to extract the abc-1.2.3-1.src.rpm source RPM file into a local directory tmp ? mkdir tmp cd tmp rpm2cpio <path-to-SRPM>/abc-1.2.3-1.src.rpm | /opt/freeware/bin/cpio –idm You need GNU cpio as AIX cpio produces I/O errors. 119 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Spec File Example (1/7) Name: t1lib Version: 5.1.2 Release: 1 Summary: PostScript Type 1 font rasterizer Group: Applications/Publishing License: LGPL URL: ftp://sunsite.unc.edu/pub/Linux/libs/graphics Source0: ftp://sunsite.unc.edu/pub/Linux/libs/graphics/%{name}%{version}.tar.gz # Patches taken from Debian and adapted for AIX Patch0: %{name}-5.1.1-manpages.patch Patch1: %{name}-5.1.1-xglyph-env.patch Patch2: %{name}-5.1.1-t1libconfig.patch BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root BuildRequires: patch %description T1lib is a rasterizer library for Adobe Type 1 Fonts. It supports rotation and transformation, kerning underlining and antialiasing. It does not depend on X11, but does provides some special functions for X11. AFM-files can be generated from Type 1 font files and font subsetting is possible. The library is available as 32-bit and 64-bit. 120 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Spec File Example (2/7) %package devel Summary: Header files and static libraries for %{name} Group: Development/Libraries Requires: %{name} = %{version}-%{release} %description devel This package contains header files and static libraries for %{name}. If you are compiling a 32-bit program, no special compiler options are needed. If you are compiling a 64-bit program, you have to compile and link your application with "cc -q64" or "gcc -maix64". %prep %setup -q export PATH=/opt/freeware/bin:$PATH %patch0 -p0 %patch1 -p1 %patch2 -p0 121 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Spec File Example (3/7) %build # setup environment for 32-bit and 64-bit builds export AR="ar -X32_64" export NM="nm -X32_64" # first build the 64-bit version export CC="cc -q64" export CXX="xlC -q64" export CFLAGS="-O2" ./configure \ --prefix=%{_prefix} \ --enable-shared --enable-static make %{?_smp_mflags} without_doc cp lib/.libs/libt1.so.5 . cp lib/.libs/libt1x.so.5 . make distclean # now build the 32-bit version export CC="cc" export CXX="xlC" ./configure \ --prefix=%{_prefix} \ --enable-shared --enable-static make %{?_smp_mflags} without_doc 122 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Spec File Example (4/7) # add the 64-bit shared objects to the shared library containing already the # 32-bit shared objects ${AR} -q lib/.libs/libt1.a ./libt1.so.5 ${AR} -q lib/.libs/libt1x.a ./libt1x.so.5 %install [ "${RPM_BUILD_ROOT}" != "/" ] && rm -rf ${RPM_BUILD_ROOT} make DESTDIR=${RPM_BUILD_ROOT} install /usr/bin/strip ${RPM_BUILD_ROOT}%{_bindir}/* || : # setup environment for 32-bit and 64-bit builds export AR="ar -X32_64" # add the 64-bit shared objects to the shared library containing already the # 32-bit shared objects ${AR} -q ${RPM_BUILD_ROOT}%{_libdir}/libt1x.a ./libt1x.so.5 mkdir -p ${RPM_BUILD_ROOT}%{_sysconfdir}/t1lib cp examples/FontDataBase ${RPM_BUILD_ROOT}%{_sysconfdir}/t1lib/ cp lib/t1lib.config ${RPM_BUILD_ROOT}%{_sysconfdir}/t1lib/ chmod 644 ${RPM_BUILD_ROOT}%{_sysconfdir}/t1lib/* 123 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Spec File Example (5/7) mkdir -p ${RPM_BUILD_ROOT}%{_sbindir} cp debian/t1libconfig ${RPM_BUILD_ROOT}%{_sbindir}/ chmod 755 ${RPM_BUILD_ROOT}%{_sbindir}/t1libconfig mkdir -p ${RPM_BUILD_ROOT}%{_mandir}/man1/ mkdir -p ${RPM_BUILD_ROOT}%{_mandir}/man5/ mkdir -p ${RPM_BUILD_ROOT}%{_mandir}/man8/ cp debian/type1afm.1 ${RPM_BUILD_ROOT}%{_mandir}/man1/ cp debian/xglyph.1 ${RPM_BUILD_ROOT}%{_mandir}/man1/ chmod 644 ${RPM_BUILD_ROOT}%{_mandir}/man1/* cp debian/FontDataBase.5 ${RPM_BUILD_ROOT}%{_mandir}/man5/ chmod 644 ${RPM_BUILD_ROOT}%{_mandir}/man5/* cp debian/t1libconfig.8 ${RPM_BUILD_ROOT}%{_mandir}/man8/ chmod 644 ${RPM_BUILD_ROOT}%{_mandir}/man8/* ( cd ${RPM_BUILD_ROOT} for dir in bin include lib do mkdir -p usr/${dir} cd usr/${dir} ln -sf ../..%{_prefix}/${dir}/* . cd done ) 124 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Spec File Example (6/7) %clean [ "${RPM_BUILD_ROOT}" != "/" ] && rm -rf ${RPM_BUILD_ROOT} %files %defattr(-,root,system,-) %doc Changes LGPL LICENSE README.t1lib-5.1.2 %dir %{_sysconfdir}/t1lib %config %{_sysconfdir}/t1lib/t1lib.config %config %{_sysconfdir}/t1lib/FontDataBase %{_bindir}/* %{_sbindir}/* %{_mandir}/man*/* %{_libdir}/*.a /usr/bin/* /usr/sbin/* /usr/lib/*.a %files devel %defattr(-,root,system,-) %doc doc/t1lib_doc.pdf %{_includedir}/* %{_libdir}/*.la /usr/include/* /usr/lib/*.la 125 © 2013 IBM Corporation pOX669 - Open Source Software Solutions and Best Practices for AIX Spec File Example (7/7) %changelog * Fri Feb 01 2008 Michael Perzl <michael@perzl.org> - updated to version 5.1.2 * Thu Jan 03 2008 Michael Perzl <michael@perzl.org> - included both 32-bit and 64-bit shared objects * Tue Aug 14 2007 Michael Perzl <michael@perzl.org> - updated to version 5.1.1 * Wed Jan 04 2006 Michael Perzl <michael@perzl.org> - first version for AIX V5.1 and higher 126 © 2013 IBM Corporation - 5.1.2-1 - 5.1.1-2 - 5.1.1-1 - 5.1.0-1