USB Automatic Installation System Networked Software Systems Laboratory Eyal Koren & Leeor Langer 11/11/2009 1 Contents 1. Introduction 5 1.1. Abstract 5 1.2. Project Goal 5 2. Technology Overview 5 2.1. Unattended installation and quiet installation 5 2.2. Windows Automated Installation Kit (WAIK) 5 2.3. Windows preinstallation environment (WinPE) 6 2.4. HTML Application (HTA) 6 2.5. Cascading Style Sheets (CSS) 6 2.6. SysPrep 7 2.7. Visual Basic Scripting (VBscripts) 7 2.8. Windows Management Instrumentation (WMI) 2.9. DiskPart 2.10. Images capture, Windows imaging format, ImageX and deployment 2.11. USB 2.0 8 2.12. Windows PE utility (Wpeutil) 8 Design 9 3. 8 3.1. Basic Design Requirements 9 3.1.1. USB Data Source 9 3.1.2. Dynamic system 9 3.1.3. Flexible system 9 3.1.4. Minimum user's interference 9 3.1.5. GUI 9 3.2. General design 10 3.2.1. GUI 11 3.2.5. Data Base 12 3.3. Specific design 12 3.3.1. Gui general scheme 12 3.3.2 GUI detailed scheme: 13 4.1. GUI Stages 14 3.5. Data Base 18 3.5.1. Database levels For OS: 18 6.1.1. Database levels For Applications: 19 4. 7 Administrator's related code (exposed code) 19 4.1. OS for unattended installation 19 4.2. OS for Image deployment 20 8 4.3. Applications. 5. 5.1. End User Guide Troubleshooting: 6. Project Evolution 21 23 23 24 6.3. Acquaintance with the Technologies 24 6.1.1. Manual Installation of Operating Systems 24 6.1.2. WAIK 25 6.1.3. Unattended Installations 25 6.1.4. WinPE 25 6.1.5. HTML 25 6.1.7. DiskPart 25 6.1.8. VBScript 25 6.1.9. Quiet Application Installation 25 6.3. Framework Design and Implementation: 26 6.4. Comprehensive Implementation: 27 6.5. Debug: 28 6.1.1. Capturing an image 28 6.1.2. Drive query 29 6.1.3. Unattended Installation Process 29 6.1.4. Partitioning Tool 29 6.1.5. Rebooting WinPE 29 6.1.6. Applications 30 2. Figures Figure1 ...................................................................................................... 10 Figure2 ...................................................................................................... 11 Figure3 ...................................................................................................... 12 Figure4 ...................................................................................................... 12 Figure5 ...................................................................................................... 13 Figure6 ...................................................................................................... 14 Figure7 ...................................................................................................... 14 Figure8 ...................................................................................................... 16 Figure9 ...................................................................................................... 17 Figure10 .................................................................................................... 18 Figure11 .................................................................................................... 18 Figure12 .................................................................................................... 20 Figure13 .................................................................................................... 21 Figure14 .................................................................................................... 22 Figure15 .................................................................................................... 23 Figure 16 ..................................................................................................... 26 Figure17 .................................................................................................... 26 Figure 18 ..................................................................................................... 28 1. Introduction 1.1. Abstract The USB AUTOMATIC INSTALLATION SYSTEM provides the user a fast and easy way to create a full working station set in minimum effort and time, using only a USB device. 1.2. Project Goal The goal of the project is to build a dynamic automatic working station installation system, that can work from a portable USB. In additional to that, the system should work unattended, and by that we mean that after supplying the system needed data and user's preferences, setting the working station would be fully automatic. The system's main target audience are computer farms and lab's technicians, thou it can also be used by private computer owners. 2. Technology Overview 2.1. Unattended installation and quiet installation Unattended installation refers to booting an operating system installation in an automatic way, with all the relevant data supplied by the user at the beginning of process. Quiet installation refers to automatic installation of applications with no user's actions required through all of the process. 2.2. Windows Automated Installation Kit (WAIK) The WAIK is a collection of tools and technologies produced by Microsoft designed to assist in the boot and deployment of Windows operating systems. The WAIK was first introduced with Windows Vista. The main components of WAIK are the WinPE, SysPrep and ImageX. Example: Creating an unattended file for Vista. 2.3. Windows preinstallation environment (WinPE) WinPE is one of the components of the Windows Automated Installation Kit (WAIK). WinPE is a lightweight version of the common operating systems, and it is used as significant preinstallation tool. WinPE main roles are to prepare the computer for windows installation, to copy disk images from source and to initiate windows setup. WinPE can be launched from a variety of media, such as CD's and USB devices. WinPE has the powerful capability of booting from a RAM disk, Which enables it to make the necessary disk related operations. The WinPE can be customized by selecting the packages added to it from a variety of packages, like HTA package, scripting package etc. WinPE can also be customized by changing the booting prefrences. Example: Used as a "launch pad" for running our GUI + environment. 2.4. HTML Application (HTA) HTA is a Microsoft Windows application written with HTML and Dynamic HTML. HTA was introduced by microsoft. In addition to the attributes of HTML, HTA supports files operations (creating, editing and removing) and registry entries. HTML applications do not require compilation and are supported in custom WinPE packages. Example: This is our choice of technology for creating our GUI. 2.5. Cascading Style Sheets (CSS) CSS is a style sheet language which controls the presentation features (looks and formatting) of the web page (in our case the HTA). CSS is designed primarily to enable the separation of document content (written in HTA) from the document's presentation. This separation provides more flexibility and control in the specification of presentation characteristics. It also simplifies the code by separating the elements in the code which are functional and those which are graphical. Example: Used for our GUI's graphical capabilities. 2.6. SysPrep SysPrep is one of the components of the Windows Automated Installation Kit (WAIK). SysPrep is used to prepare the operating system for capture disk and disk deployment. SysPrep also "generalized" the unique characteristics of each computer, such as computer name and security identifiers, that they can be changed after deployment. Example: Used before capturing a system image for deployment. 2.7. Visual Basic Scripting (VBscripts) VBscript is a scripting language, developed by Microsoft. VBscript's syntax reflects its origin as variation of Microsoft's Visual Basic programming language. A VBscript script must be executed within a host environment, such as Windows Script Host (WSH) and Internet Explorer (IE). VBscripts enables writing executable functions, also in HTML/HTA pages. The use of VBscripts in HTA enables the accessibility of the scripting to files and folders operations, such as creating, deleting, reading and editing. The advantage of using in the project the VBscripts instead of other languages (java script, TCL …), is the compatibility to Microsoft's components and full support in custom WinPE and its powerful scripting capabilities. Example: Our choice of technology for scripting functionality. This refers to all processes and functions used by the GUI for enabling the various unattended installation procedures. 2.8. Windows Management Instrumentation (WMI) WMI is a Management component that defines a set of environment-independent specifications which allow management information to be shared between management applications. For example, WMI allows scripting languages like VBScript to manage Microsoft Windows personal computers and servers. WMI also provides set of functions that can be used with scripting languages. Example: Used for low level function calls such as querying HD information. 2.9. DiskPart Diskpart is a command-line hard disk partitioning utility. In addition to creating the new partitions, Diskpart can also format, clean, activate and assign partition letter. DiskPart supports the use of scripts to automate it's usage. Example: Used for formatting, partitioning and more. 2.10. Images capture, Windows imaging format, ImageX and deployment Image capture refers to copying a disk, bit by bit and. The capture is stored in Windows Imaging Format (WIM), and can Be deployed later on other disk. ImageX is the command-line tool used to create, edit and deploy Windows disk images. ImageX is a part of the WAIK. Images can be mount to add or remove data and then be unmount to have a new image. The deployment process is based on the Hardware abstraction layer (HAL), which refers to the seperation between the physical hardware and the software running on the computer. The main advantage of the HAL is hiding diffrences in hardware, from the operating system's kernel. The HAL was first introduced in windows vista. Example: Used for capture and deploy. 2.11. USB 2.0 USB 2.0 is high speed data transfer protocol. Using USB device as the data source of the system is possible due to the USB 2.0 protocol speed. 2.12. Windows PE utility (Wpeutil) The Windows PE utility (Wpeutil) is a command-line tool that enables you to run various commands in a Windows PE session. For example, you can shut down or restart Windows PE, enable or disable a firewall, set language settings, and initialize a network. Example: Used for rebooting. 3. Design 3.1. Basic Design Requirements 3.1.1. USB Data Source The source of all data of the project and the system itself are all located on the USB. 3.1.2. Dynamic system The system must be dynamic so new operating systems and applications can be added to data source (USB) and be ready for automatic installation with the minimum adaption needed and in elegant way. The system must recognize the installation options in a dynamic way and present to the UI the options for the full working station set. 3.1.3. Flexible system The system must support operations needed for future expandability. For example, adding new applications or OS options that require changing the code minimally, in specific predefined sections, in a user-friendly way, without knowing the full system code. Parts of the code must be exposed to the System Administrator in order to support future OS and applications releases and updates. 3.1.4. Minimum user's interference The system should be fully independent so that after entering in the beginning of the process the needed data such as OS type, wanted applications, computer's name and password (in the unattended process), the system will work independently until the creation of the full working station. 3.1.5. GUI The system's user interface must be user friendly, and easy to use. the GUI should take the user each step at a time in designing his wanted workstation and must be clear to first time user as well as to the expert user. 3.2. General design The general design of the system is composed of 4 main blocks, which interact one with the other. The dynamic and flexible features of the system force all the 4 blocks to interact with each other. Figure 1 3.2.1. GUI In order to create a clean and organized code, we split 3 functionalities of the GUI to 3 areas in the code. 3.2.2. HTML\HTA Framework: Our GUI is an HTML application in itself. HTML is the "backbone" of our project. 3.2.3. Graphics: The user controls and graphics of our GUI are implemented in CSS. 3.2.4. Scripting: All the actions performed by the GUI are implemented in VBScript. This includes temporary file creation, partitioning etc. Framework HTA GUI CSS Graphics User Figure 2 Scripting VBScripts Actions performed by GUI 3.2.5. Data Base Data base contains all the data needed for the installations and how to call this data. The GUI's directory hierarchy enables us an orderly method for sourcing the appropriate OS or applications. Level 1 Level 2 Figure 3 3.3. Specific design 3.3.1. Gui general scheme The system's GUI is composed of several stages. During each stage the user much choose between a few options. After pressing the "next" arrow the next stage is instantiated. This stage may be instantiated in different ways, depending on choices previously made. The GUI may be divided to the following stages: Reboot Figure 4 MS installion\deployment process 3.3.2 GUI detailed scheme: Welcome Screen Partition OS Specific Choice OS General choice Unattended Installation Image installation Unattended File Data No user data User Info step application for selection No applications Applications step ReBoot Figure 5 4.1. GUI Stages 3.4.1. Welcome screen: The Welcome screen contains basic system information and instructions. 3.4.2. Partition: In this step, the user is given the data of the disks (name, sizes, current partitioning) and select the partitions wanted and partitions options (formatting method, file system etc.). The partition choices made in this step are used as input for creating PartitionScript.txt. This temporary script is called by the DiskPart utility after the user completes filling in the GUI. All the partition related files are located in the Partition folder. Figure 6 The next scheme, describes how our partitioning works. After the user supplies the data, a temporary script is created from the template script. The temporary script contains all the relevant information supplied by the user. After the user completes filling in the GUI form, diskpart is called with the temporary script. DiskPart Figure 7 This same process applies to the Clean, Create and Delete options. The difference is that these 3 occur on-the-fly and not adjacent to the installation process. Refresh function is called after each partition action. The ‘Current Partition Table’ is composed of parsed WMI calls for system information. 3.4.3. OS general choice: All the OS offered for installation in level 1 (as shown in figure X) are loaded to the current page, for the user's selection. The OS options are dynamically loaded from the OS folder by the same name. 3.4.4. OS specific choice: According to level 1 selection, the level 2 relevant data options are loaded to the current page, for the user's selection. As in last step, the options are loaded dynamically. Both general and specific OS selections are concaternated to a global variable called strOS. This variable contains the path of the specified OS and will be used in the installation\deployment process. 3.4.5. User info: In this step, the user provides, if asked, data such as computer name, password, product key etc. The data is asked only when the OS installation chosen was unattended. If the installation is a deployment one, no data is required from user. The data from user is used to create a temporary unattended file. The unattended file template is located in the OS folder. After completing the GUI, the unattended installation call is used with an "unattended" flag and the temporary unattended file is sourced also. 3.4.6. Applications: If the process is unattended installation, the system will offer the user a variety of applications to choose from for installation. This collection of application are found in the "Apps" directory and is loaded dynamically, like the rest of the GUI stages. If the process is image deployment, the application selection will not be possible. In this stage, the selection of applications will generate a batch file (all_inst.bat) that contains all the command lines of each application installation. The all_inst.bat file will later be executed and will install all the applications in serially, without contention. The system currently has 3 options for applications. The addition of new applications for installment is described in the administrator section, but it's worth mentioning that adding applications is easy and intuitive due to the flexibility demands of the project mentioned above. Figure 8 3.4.7. Reboot: After all steps are completed, the system starts several processes and upon completion, the WPEutils reboot function is called. This step is different for unattended installation and for deployment. Unattended installation: Launch unattended Unattended Process starts : This function calls all the functions needed for unattended installation. unattended file creation Unattended file is created with the user's inputs. (winnt.sif) Diskpart Temporary partition script is created. Diskpart launches with this Partition script. Launch Installation script This script calls an executable file named winnt32.exe, which initiates the installation. Reboot Wpeutils reboots the system. Image deployment: Launch Deployment Deployment Process starts : This function calls all the functions needed for deployment. Figure 9 Diskpart Temporary partition script is created. Diskpart launches with this Partition script. Image Operations Operations related to Image chosen for deployment. Launch Deployment script ImageX utility is called with the appropriate flags for the current image. Reboot Wpeutils reboots the system. 3.5. Data Base As described before, the database is composed of data levels as shown in scheme : OS Database Level1 - XP Vista unattended Level2 ImageXP XP ImageVista1 windows7 ImageVista2 Figure 10 applications Level1 Level2 Winrar Winrar Setup command Winzip Winrar setup Winzip Setup command Winzip setup Figure 11 3.5.1. Database levels For OS: Level1: contains all OS options for the user (XP,Vista, Windows7…). Level2: contains all versions of the Level1 OS choices (SP1,SP2,images…). Windows7 Image 6.1.1. Database levels For Applications: Level1: contain all applications available for installation. Level2: contain all application setup files (usually an executable) and setup commands for quiet installations (bat file with appropriate flags). 4. Administrator's related code (exposed code) The system design must support easy addition of new OS and applications with no code modification. Here we will detail the steps for adding new components to the system. 4.1. OS for unattended installation The distinction between OS Image and OS for unattended installation is that every OS for unattended installation contains the phrase "unattended" in its name. This is the naming convention we used to make a distinction between unattended installation and image deployment. Unattended installation is an old technology that simulates the installation process without the need for a user during the process. Deployment of an image is possible only after capturing this image.Before Vista, image capture was hardware dependant, so an image from a certain computer cannot be deployed to another computer. Vista introduced the HAL- Hardware Abstraction Layer. This layer removes the HW-SW(OS) dependency. This means that we can deploy a vista image on many different computers, thus making the unattended process obsolete. In addition, the difference between unattended files of the existing OS and the future OS, make the unattended installation complicated for future assimilation. Because of the above reasons, we support only unattended installation of XP, and don’t support unattended installation of Vista or future OS (not to say that this is not possible). For new application addition (only versions of XP): A. Create in the XP folder a new folder which its name is the OS option that will appear in the GUI. The OS name must contain the phrase "unattended". B. In the new folder, create two files: the first is the the unattended file, named winnt.sif. The second is a bat file that activates the execute file winnt32.exe for unattended installation. The bat file must have the same name like the folder it located in, and it must have the required flags for unattended installation. unattended file path Installation flags Figure 12 4.2. OS for Image deployment As described in 3.4.1, the preferred installation method is image deployment. For new Image addition : A. capture an image using imageX. B. create a new folder for the Image. If the image is a version of an existing OS (like a new image with the latest ServicePack), put the folder under the matching OS folder. If the Image is a new OS that doesn't exist in the system at all, create under the OS folder, a new folder with the new OS name, and put the image in the new folder. C. copy the new image to the folder (by same name), and create the batch file with the same Image name. Deployment the VistaImage1 in drive C: Image File Figure 13 4.3. Applications. Every application in level 3 has a relevant batch file in level 4, and the setup file in the same folder. The batch file contain the command line for setup has the relevant flags for quick installation. For new application addition : A. Create a new folder in the apps folder. Rename the folder by a name suitable for identification in the application step. B. copy the application's setup file into the folder and rename it the same as the folder. C. Create a batch file with the command line executing the setup file. Be aware that the command line must include the setup file path and the quiet installation flags. For example, lets take a look on the two installations. The folders and file has a leveled structure, as can be shown in the winrar installation: The installation command line is : Setup file path Quiet installation flag Figure 14 The winzip installation files are leveled in the similar way: Figure 15 The installation command line is: Setup file path Other installation flags 5. End User Guide 5.1. Troubleshooting: Which FileSystem to use? Answer: choose ntfs for current MS OS. You may do this by entering 'ntfs' in lowercase letters (no need to enter the ''!). Which format flags to use? Answer: No need to enter any flags for default installations. You may use 'quick' for debug purposes. Which letters to choose for my partitions? Answer: Use 'c' for your OS installation logical drive only. Use other letters ('d,e,f...') for different partitions. Do not enter 'x:' as your letter, only 'x' (where 'x' refers to all letters!). Which installation methods to use? Currently the system supports unattended installation for XP only. Image deployment is supported for Vista OS and Win7. Quiet installation flag These technologies are the most suitable for the OS above (see Vista\Win7 HAL for explanation of image deployment...). How to configure my BIOS to support USB boot? 1. Check that booting from USB is supported. 2. Boot order must be Hard disk first and USB second. 3. Boot from USB only when initiating the GUI. How to change\upgrade my WinPE image? See the walkthrough in the ReadMe documentation or check out Microsoft TechNet for step by step guides. 6. Project Evolution This project is not programmatically difficult. Most of the code is simple and the GUI itself is an HTA. The difficulties of this project are different than standard project at the Network Software Lab. Understanding how to work with the different technologies and integrating them together into a complete product is the main difficulty. We will explain in detail what we accomplished and the phases of development that we completed. 6.3. Acquaintance with the Technologies Before starting the actual coding of the GUI, we had to familiarize ourselves with the many technologies in order to understand and experiment with all the different options ahead. This phase is important because of two reasons: First, we had to choose which technology is best for our needs. Second, we had to gain a deep understanding of what we will be doing and what to expect. 6.1.1. Manual Installation of Operating Systems Installing XP and Vista on our technician computer. 6.1.2. WAIK Creating manually unattended files for unattended installation. 6.1.3. Unattended Installations Using unattended files for unattended installation. This was done on XP and Vista OS. 6.1.4. WinPE Installing WinPE on our USB HD and learning to work with the WinPE command prompt. Later, we also had to customize our WinPE installation in order to support all the different technologies. 6.1.5. HTML Learning to work with HTML from youtube videos. 6.1.6. ImageX Learning how to capture and deploy images with this tool (and the sysprep tool). 6.1.7. DiskPart Learned how to work with this tool in order to properly format the HD. 6.1.8. VBScript Studying this language and learning how to work with it alongside HTML in order to create dynamic elements, calls utilizing WMI, text and file manipulation etc. 6.1.9. Quiet Application Installation Using different tools to accomplish this goal (AutoIt scripts, msi files, etc.) 6.2. Basic Design and Technology decisions: After completing our acquaintance with all the different technologies, we started planning which features we want to include and how we will implement each feature. There are many ways to accomplish each task, but we needed to make smart decisions to avoid using the wrong technology for each feature. In addition, we planned the "look and feel" of the GUI and how we should implement each feature. Example: Should we use system variables or parse our scripts? Should we use AutoIt for quiet installations or not? OS + App DataBase Scripting Engines WinPE GUI HTML application Figure 16 6.3. Framework Design and Implementation: The objective of this phase of the project was a "preparation stage" for the final product. We planned a basic, non-dynamic system in which each phase of the GUI was a standalone application. After planning the functionality in each stage, we implemented (in different HTA files) the different stages of the GUI. In each file, we coded the basic functionality of the relevant stage. Partition OS choice Definitions Applications •Calling ext utility •Create HTML input fields and buttons •Passing variables using VBScript •Writing variables to files •Calling scripting elements (bat files, WMI, etc.) Figure 17 Example: In the "OS choice" phase we created fixed buttons (not created dynamically) and learned how to read the input fields and pass the inputs to variables. 6.4. Comprehensive Implementation: During this phase of the project we put together all the elements needed in order to implement the final product. We soon learned that only a small part of the framework was reusable. For example, creating dynamic elements (such as buttons) is very different from creating fixed elements. We redesigned the code in one complete file (our HTA) and added dynamic areas on our form that we clear after each stage of GUI usage (after pressing "next" we clear the form and instantiate the next stage). In addition, we separated the graphical code from the functionality using CSS. This changed dramatically the "look and feel" of the GUI. Global variables which contain data that is needed for different stages of the GUI were added. We added coordination and synchronization between the different stages of the GUI. Also, a progress bar was added. In summary, during this phase of the project we merged all the different technologies while reusing some of the framework code. Here is a screenshot of the complete GUI: Progress bar for user indication HTA body Graphics implemented by CSS Figure 18 6.5. Debug: This phase of the project was very time consuming. Some of the challenges in implementing a USB Automatic Installation System are that some of the Microsoft technologies are mainly used internally by Microsoft and were not intended for massive public users. This leads to very poor documentation and so we needed to rely on external documentation and a lot of "try and experiment". Another difficulty in this project is that in order to debug our system, we need to wait for a complete format and installation before we can see if our debug was successful. The following were the most notable and time consuming bugs we encountered: 6.1.1. Capturing an image Our captured images were not working correctly. Deploy process was not working properly. Solution – Using sysprep before capturing an image. 6.1.2. Drive query One of the first procedures the GUI runs is a query to determine which drive letter our USB receives. This letter changes from computer to computer. We found that the Windows Scripting Host (WSH) query was not working on all computers. Solution – We implemented this query with Windows Management Instrumentation (WMI). 6.1.3. Unattended Installation Process The first problem with this process was that it was not clear when the process ends! The process seemed to terminate prematurely. This was not properly documented at all. Solution – After the computer reboots this process continues. In order to make the process automatic we added a reboot function. The second problem was that after completing the installation, a lot of files were unpacked on to our USB HD. This included some files being overwritten, including the "bootmgr" file in the root of the USB HD. This restrains our system to be able to work only once, since it cannot be booted again. Solution – Using the proper flags to initiate the unattended installation process (see figure X). 6.1.4. Partitioning Tool We planned our system to work with an external tool in order to partition the HD. This tool was not able to format the HD when launched from WinPE. Solution – We designed our own partitioning tool using the DiskPart utility (with script) and WMI queries to show the current drives and partitions on the computer. 6.1.5. Rebooting WinPE In order to reboot WinPE we can simply type "exit" at the command prompt. But, when calling a batch file with the "exit" command this only causes a new window to open and the immediately close. Solution – Using "wpeutils" reboot function. 6.1.6. Applications When installing multiple applications we encountered a synchronization problem. Calling multiple batch files in order to install multiple applications caused contention. Solution – Merge all the needed batch files into one batch file (AllInstall.bat).