Mobile Operating Systems and Applications (CSCIE-65) Assignment 0 - Solution About this assignment The purpose of this assignment is to get you acquainted with the kind of challenges you will face in CSCIE-65 in the spring semester of 2013, and hopefully open your eyes to the less documented sides of mobile operating systems. This assignment will not be graded, and solutions will be posted by the first lecture. We strongly recommend, however, that you do not wait for the solutions – but, rather, try to tackle it yourself. You might even learn a thing or two, especially about the similarities and differences of the major three. 0. Prerequisite Step: VMWare or other hypervisor (e.g. VirtualBox) Get VMWare, if you don’t already have it. If you’re on a Mac, you need VMWare Fusion 5. If you’re on a Windows box, you need VMWare Workstation 8 or 9. For the off chance you run Linux as your desktop, you’ll need VMWare Workstation for Linux. For either flavor, you can get VMWare as a trial download, though in particular Fusion is well worth the $49.99 academic price. Note: If you would rather use VirtualBox (which is free), that’s also a possibility. All in all, however, VMWare certainly has numerous advantages (especially on a Mac). The instructor is working on getting you all an academic license (which means you won’t have to pay a cent, and still get to you use this fantabulous product for 200-300 days, for this course and beyond). An update will be posted as soon as we have it. Until then, just use the evaluation: 30 days should buy us all enough time. 1. Android a. Setup Download a Linux ISO (preferably Ubuntu, as this will give you the least GLIBC conflicts), and set up a VM. Choose to install everything, and prefer a DVD to a CD image (20GB should be more than enough). On the VM, install both the Android SDK (r21, now available as the “ADT Bundle”) and the Android NDK (version r8d). You will also need to make sure you have the Java Development Kit (JDK), version 1.6 or later (most recent is 1.7). The download links are easily Google-able, though you can start at http://developer.android.com/sdk/index.html. Why Linux? Though you are all probably using another OS in your daily lives, Linux is the closest “relative” of Android, and is therefore extremely useful as a host platform for Android devices. The Android emulator (which is part of the SDK) and the devices themselves contain many of the Linux commands (in one “toolbox”), but the commands are of limited functionality. Using Linux as the host enables you to pipe the output of the command from the device or emulator onto the host, and run host commands (such as grep, sed, sort and their ilk) to process and filter the output. Set up the SDK and NDK. The two will install in /usr/local. Find the “adb” and “android” commands. As root, create a symbolic link to them (using ln –s) in /usr/bin. This will make your life easier as you will from time to time need to fire up the emulator (android &) and the Android Debugger Bridge (adb) to communicate with either the emulator or – if you have one – your Android phone. Alternatively, Android now comes in the “adt-bundle”. The process to install in /usr/local would be as follows: # # First verify you actually have the download, usually in ~/Downloads or elsewhere: # Note “x86_64” is the 64-bit version: Otherwise this would be “i386” for 32-bit version # root@Forge:~# ls ~/Downloads/adt-bundle-linux-x86_64.zip /home/morpheus/Downloads/adt-bundle-linux-x86_64.zip # # Now change to /usr/local (or any other directory) for unzipping: # root@Forge:~# cd /usr/local root@Forge:/usr/local# unzip /home/morpheus/Downloads/adt-bundle-linux-x86_64.zip Archive: /home/morpheus/Downloads/adt-bundle-linux-x86_64.zip inflating: adt-bundle-linux-x86_64/sdk/system-images/android-17/armeabi-v7a/ramdisk.img inflating: adt-bundle-linux-x86_64/sdk/system-images/android-17/armeabi-v7a/userdata.img … inflating: adt-bundle-linux-x86_64/eclipse/about_files/webkit-bsd.txt inflating: adt-bundle-linux-x86_64/eclipse/artifacts.xml inflating: adt-bundle-linux-x86_64/eclipse/eclipse.ini inflating: adt-bundle-linux-x86_64/eclipse/notice.html Creating the symbolic link is a matter of convenience. You don’t HAVE to do it. If you want to just be able to run the commands with the path, you can do the following: # # Before: neither android nor adb can be found: # root@Forge:/usr/local# which android # # Add to path: We use “export” (to set the variable), and append to the existing path # (this is what PATH=$PATH:... does: it assigns to the PATH variable its present value, # followed by “:...” - : is the path separator) # root@Forge:/usr/local# export PATH=$PATH:/usr/local/adt-bundle-linux-x86_64/sdk/platformtools:/usr/local/adt-bundle-linux-x86_64/sdk/tools # # Et voila! Both commands are visible: # root@Forge:/usr/local# which android /usr/local/adt-bundle-linux-x86_64/sdk/tools/android root@Forge:/usr/local# which adb /usr/local/adt-bundle-linux-x86_64/sdk/platform-tools/adb If you want the symbolic link, you would do: # # Create symbolic link to adb (similar process for “android”, but in “tools”) # root@Forge:/usr/local# ln -s /usr/local/adt-bundle-linux-x86_64/sdk/platform-tools/adb /usr/bin/adb root@Forge:/usr/local# which adb /usr/bin/adb root@Forge:/usr/local# ls -l /usr/bin/adb lrwxrwxrwx 1 root root 57 2013-02-14 09:32 /usr/bin/adb -> /usr/local/adt-bundle-linuxx86_64/sdk/platform-tools/adb To stave off permission problems (the zip bundle permissions are bad) , it is recommended to # Make all directories cd’able linux-xroot@Forge:/usr/local/adt-bundle-linux-x86_64# find . -type "d" -exec chmod 755 {} \; also do: # Make all files executable root@Forge:/usr/local/adt-bundle-linux-x86_64# find . -type "f" -exec chmod 777 {} \; b. First steps This will fix an emulator permission bug which could result in “PANIC” when starting an AVD. If you have an Android phone or tablet, you can do the following on the actual device. If not, you can simply use the emulator (which you can start up from anywhere as “android &”, assuming you have set up the symbolic link. Recall the “&” means starting it in the background). If you are using the emulator, start an instance of Ice Cream Sandwich (4.0.x or later), or Jelly Bean. If you are using a device, find and enable “USB Debugging” (any Android version you have will be fine – fear not - we’re not asking you to upgrade your device). Remember that, over VMWare, you need to allow the VM to obtain the USB connection, rather than the host. Once the device is connected (or the emulator is running), set it aside. This is the last time you’ll need the GUI for a while. Instead, fire up “adb”. Make sure the device/emulator is connected and ready by trying “adb devices” (remembering that if you have more than one device/emulator instance you will need to specify “-s” with the serial number). Then, spawn a shell, using “adb shell”. # # With the emulator and a device # morpheus@Forge:~$ adb devices List of devices attached emulator-5554 device 5d091005 device morpheus@Forge:~$ adb -s emulator-5554 shell root@android:/ # 1) Peek around the Android file system. What are its similarities and differences from the Linux file system of the host? The Android file system is similar, but not exactly the same as the host: Android has: - - /system: where most of the OS is installed. Binaries are installed in /system/bin and /system/xbin (where the ‘x’ are for debug/sysadmin commands). There is also a /system/app, and /system/framework, for the Android built-in apps and frameworks, respectively. /system is mounted read-only, and is writable only to root. /data: This is the writable partition, meant to store runtime and application data. We’ll talk more about this later on. 2) Try ‘ls –l’ to find your favorite commands (ls, df, mv, etc..), which are all in /system/bin. What do you see? These commands are all symbolic links to “toolbox”: # # Checking some unix favorites: # root@android:/ # ls -l /system/bin/ls /system/bin/df /system/bin/umount lrwxr-xr-x root shell 2012-11-09 01:46 ls -> toolbox lrwxr-xr-x root shell 2012-11-09 01:46 df -> toolbox lrwxr-xr-x root shell 2012-11-09 01:46 umount -> toolbox # # Running toolbox with no arguments – Toolbox doesn’t know what to do # root@android:/ # toolbox Toolbox! # # Toolbox emulates the tool given as argument, or through symbolic link # root@android:/ # toolbox umount umount <path> # # Given a wrong tool name: # root@android:/ # toolbox blah blah: no such tool 3) Grab an executable file from the emulator – some binary from /system/bin or /system/xbin will do. Use the Linux “file” command to identify its type, and compare it with the output of “file” on the same file, but from the host. What do you see? # # Grab /system/bin/ls (actually, toolbox) from emulator to /tmp/ls # morpheus@Forge:~$ adb -s emulator-5554 pull /system/bin/ls /tmp/ls 988 KB/s (134976 bytes in 0.133s) morpheus@Forge:~$ file /tmp/ls /tmp/ls: ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked (uses shared libs), stripped # # Compare to host’s /bin/ls – Notice both are identical, with the exceptions of 32/64, # and “ARM” vs. “x86” (Intel) # morpheus@Forge:~$ file /bin/ls /bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, stripped 4) Try “cat /proc/cpuinfo” on both your device/emulator, and the Linux host. What do you see? How does this corroborate what you just saw in (3)? # # The host: an Intel machine # morpheus@Forge:~$ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 58 model name : Intel(R) Core(TM) i7-3667U CPU @ 2.00GHz stepping : 9 cpu MHz : 2492.781 ... # # The Emulator # morpheus@Forge:~$ adb -s emulator-5554 shell cat /proc/cpuinfo Processor : ARMv7 Processor rev 0 (v7l) BogoMIPS : 478.41 Features : swp half thumb fastmult vfp edsp neon vfpv3 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xc08 CPU revision : 0 # # The Device (Samsung Galaxy S3, notice 2 cpus (dual core). Also notice # more advanced “Features” (e.g. tls, vfpv4) # morpheus@Forge:~$ adb -s 5d091005 shell cat /proc/cpuinfo Processor : ARMv7 Processor rev 0 (v7l) processor : 0 BogoMIPS : 13.53 processor BogoMIPS : 1 : 13.53 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 CPU implementer : 0x51 CPU architecture: 7 CPU variant : 0x1 CPU part : 0x04d CPU revision : 0 2. iOS For iOS, there is no need to jailbreak your i-Device (though it’s perfectly legal and harmless to do so). In fact, at the time of writing a Jailbreak for 6.x isn’t publicly available yet. We will therefore resort to using the iOS emulator bundled with XCode… But not yet: this exercise will focus on the iOS software images, which are available for download. There is a catch, however : iOS images are encrypted, and so just downloading them is not enough – only the devices can decrypt and use them. Enter: http://theiphonewiki.com/. This website, which was started by well known hacker GeoHot (credited with an early iOS jailbreak as well as the Sony PlayStation 3), serves as a vast repository for all things iOS – and also enables you to get the decryption keys for the iOS images. You can download the decryption utility (VFDecrypt or any one of its clones/GUIs) from http://theiphonewiki.com/wiki/Vfdecrypt/. The keys can be downloaded from http://theiphonewiki.com/wiki/Vfdecrypt_keys/. In particular, keys are available for the “Old Boot ROM” devices (also known as “A4” devices, for the Apple SoC they contain). Once you have the image (the .IPSW file), unzip the root filesystem (usually the first, and largest .DMG file it contains). Proceed to decrypt it, so you end up with a valid DMG (i.e. “Disk Image”, in OS X parlance). You can tell if the DMG has been successfully decrypted by running the “file” command – if it reports “data”, the decryption has been unsuccessful. As an example, consider the following output (on a Mac OS X system) (~)$ unzip ./IPSW/iPod4,1_6.0.1_10A523_Restore.ipsw Archive: ./IPSW/iPod4,1_6.0.1_10A523_Restore.ipsw inflating: 038-7009-012.dmg 038-7009-012.dmg Extracting the root file system image.. (~)$ file 038-7009-012.dmg 038-7009-012.dmg: data Unencrypted image: appears as “data” (~)$ ./Tools/vfdecrypt -h Help is on the way. Stay calm. Usage: vfdecrypt -i in-file [-p password] [-k key] -o out-file (~)$ ./Tools/vfdecrypt -i 038-7009-012.dmg <decryption output omitted..> -k <key> -o 6.0.1.dmg Using ”vfdecrypt” to decrypt the image It’s not a VAX COFF – but it’s certainly not data, either! (~)$ file 6.0.1.dmg 6.0.1.dmg: VAX COFF executable not stripped - version 376 (~)$ open 6.0.1.dmg OS X only: Equivalent to double clicking on the DMG (i.e mounts it and opens Finder) If you got this far, you’re almost there! You should be able to answer the following questions: 1) Which image did you use for this experiment? You could use any of the images for iOS which are for an A4 device (1st gen iPad, iPod <= 4th, iPhone <= 4). 2) Why are there no keys available for the iPhone 5, iPad 2 and later, and iPod Touch 5G? All of the A4 devices have a vulnerability in their boot ROM (firmware) which enables the retrieval of the image encryption key. A5 devices (iPad2, iPod >= 5th, iPhone >= 4S) are not vulnerable to any known vulnerability, so their keys are still secret. 3) What are the similarities to OS X system directories? # # ls on the iOS filesystem, from an iOS 6.0.1 image, mounted under /Volumes: # bash-3.2# ls -F /Volumes/Sundance10A523.N81OS/ .Trashes/ Applications/ Library/ bin/ dev/ private/ .file Developer/ System/ cores/ etc@ sbin/ # # # # # tmp@ usr/ var@ ls on the host. Notice iOS doesn’t have /Volumes (for mounted external storage), /Network (for NFS/AFS mounts) or /Users (as it is essentially a single user system, unlike OS X). It also doesn’t have the kernel image (/mach_kernel) lying around (it’s encrypted on disk elsewhere in “kernelcache” form. bash-3.2# ls -F / .DocumentRevisions-V100/ .PKInstallSandboxManager/ .Spotlight-V100/ .Trashes/ .file .fseventsd/ .vol/ Applications/ Developer@ Library/ Network/ System/ User Information@ Users/ Volumes/ bin/ cores/ dev/ etc@ home/ mach_kernel 4) What is the main difference between Android’s “/System/bin” ,“/System/xbin” and iOS’s /bin and /sbin? The iOS /bin is virtually empty (containing only /bin/launchctl)! Since it doesn’t expect to host user shell sessions, there’s no need for any binaries. /sbin has some daemons in it, but really only minimal commands. 5) Go to /Applications and pick one of the default built-in Apps (alternatively, look through your PC or Mac to find “.ipa” files on the hard drive, if you have synced an i-Device recently, and unzip the .ipa). Irrespective of which .app you have chosen, they all follow the same general structure. What is it? mnt/ net/ private/ sbin/ tmp@ usr/ .ipa files are zip files (like Android APKs). Unlike Mac OS X Apps, which are neatly bundled, OS X apps throw all their data in one directory (the .app). There is always an Info.plist file, and the binary is the same name as the “.app” (minus the extension, of course) 6) Examine one of the application binaries (which have the same name as the .app you have selected) using OS X’s “otool” command (using the –l and -h switches), or – if you want a deeper inspection, try the “jtool” command from http://www.newosxbook.com/ with the same switches. What can you say about the binary format of iOS? How does it relate to that of OS X? # # Analyzing an iOS binary , with jtool or otool # bash-3.2# jtool -h templerun2 Processing templerun2: Magic: 32-bit Mach-O Type: executable CPU: ARMv7 Cmds: 46 size: 5072 bytes Flags: 8085 bash-3.2# otool -hV templerun2 templerun2: Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags MH_MAGIC ARM V7 0x00 EXECUTE 46 5072 NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES # # Analyzing an OS X binary , with jtool or otool # bash-3.2# ~/Documents/Work/JTool/jtool -h /bin/ls Processing /bin/ls: Magic: 64-bit Mach-O Type: executable CPU: x86_64 Cmds: 18 size: 1880 bytes Flags: 0x200085 bash-3.2# otool -hV /bin/ls /bin/ls: Mach header Magic cputype cpusubtype caps filetype ncmds sizeofcmds flags MH_MAGIC_64 X86_64 ALL LIB64 EXECUTE 18 1880 NOUNDEFS DYLDLINK TWOLEVEL PIE Note the similarity to Android system binaries: again, the ARM vs. X86_64 case, but otherwise the binaries are the same structure (“Mach-O”). Using –l will reveal the iOS binaries are actually further encrypted – but more on that later on in the course. Extra: Working with XCode You won’t need XCode for a while, but now is the time to make sure you can secure access to a Mac OS X system with XCode on it. Getting Xcode is easy – it can be now downloaded freely from the Mac App Store. Bear in mind - you will need to register as a developer at http://developer.apple.com/ The tricky part, however, may be getting OS X itself. Your options: a. Get yourself a Mac! You may already have a Macbook, Macbook pro, or Macbook Air. In that case, all you’ll really need is to make sure it can support OS X 10.7, a.k.a. “Lion”. If you can do Mountain Lion (10.8) even better b. Use the Extension school Mac lab – if you’re a Cantabrigian or in the general Boston Area Once you have XCode, familiarize yourself with the interface, especially in building a sample OS X and/or iOS project (There are many, many tutorials and websites, as well as books on the subject – and we will very briefly try to cover as much as we can in class, as well). Familiarize yourself with the SDK at the filesystem level – that is, create a symbolic link to XCode’s Developer/ directory (in XCode.app/Contents/Developer) and look for /Developer/Platforms/iPhoneOS.platform, especially in Developer/ and DeviceSupport/. 3. Windows Phone a. Setup Download the evaluation version of Windows 8 (Enterprise edition is fine) from http://msdn.microsoft.com/en-us/evalcenter/jj554510.aspx. Prepare a VM. Then, continue to also download Visual Studio 2012. You might also want to download “Process Explorer” from http://www.sysinternals.com/ (now part of Microsoft Technet), though you can get to the same conclusions using Windows’ built-in task manager (CTRL-SHIFT-ESC). b. First steps 1) Which process is responsible for the so called “Metro UI”? dwm 2) Using Visual Studio 2012’s “Dumpbin.exe” command (and the “/headers” switch), which you can start from the “Visual Studio Command Prompt” (somewhere under “tools”)), compare the “kernel32.dll” files you can find in C:\windows\system32, C:\windows\sysWOW64, and the file found in http://www.technologeeks.com/e65/hw/kernel32.dll. What are the similarities and differences between the files? Again, as was the case with iOS and Android – the binaries have the same format – PE (Portable Executable). In Windows’ case, these are PE, or PE32+ (for 64-bit systems), targeting Intel. In Windows Phone/WinRT Tablets – this would be straight PE, but – for ARM. 3) Research the web for the differences between “WinRT APIs” and the “Win32 APIs”. Make sure you understand the concept of the latter – we will go over the former (and explain why they are dependent on the latter, after all) in class! Explained in Class Epilog We won’t grade this assignment, but you can grade yourself: If you managed to perform all the tasks in this assignment, the only challenges you will see in this course are the programming challenges. You will need to write code in this course. Java should be ok for most, though we will also work with some Objective-C and C++. Fortunately, the course is built in such a way so as to allow you to “specialize” in one of the OSes – and you can pick your OS based on your favorite device-type and language. If the UNIX commands are rusty, or you’re not fully following all of the flow, don’t despair – this is the level of operating system work you will need in this course. Any good book on UNIX (or a course on using UNIX, such as Bruce Molay’s great “Unix/Linux Systems Programming, CSCIE-215) should bring you up to speed on what you need to know. Note, that while we would want to, we simply don’t have time to dwell on the UNIX commands and syntax in this course. We only have a semester, and it’s far more important to teach you the stuff that is not in the books and not as documented, rather than spend even more time on the basics. Programming experience is much more important in this course than command line capabilities. That said, bolstering your UNIX skills is more important than ever, now that (thanks to mobile OSes) UNIX is once more the dominant operating system type in the world. Either way, if you are intrigued by this assignment, you are definitely set for an exciting course. There are many courses on mobile application development out there (not to mention a myriad of books), but few (if any) take the low-level, operating system perspective we will take.