Remote Access Controller (RAC) Field Repair Utility Dell OpenManage™ STEP BY STEP By Brian Brelsford and Jon McGary ESG Software Development September 2002 Contents Executive Summary .................................................................................................................... 3 Introduction ................................................................................................................................. 4 Overview of the Software Field Repair Utility ..................................................................... 5 Detect When to Use It ................................................................................................................ 6 DRAC III................................................................................................................................ 6 ERA, ERA/O, DRAC III/XT ................................................................................................ 6 How to Use Software Field Repair Utility ............................................................................. 7 Overview ............................................................................................................................... 7 Webpack Instructions ......................................................................................................... 8 Field Repair Utility Syntax ....................................................................................................... 9 NAME..................................................................................................................................... 9 SYNOPSIS ............................................................................................................................. 9 REQUIREMENTS/DESCRIPTION .................................................................................. 9 INPUT .................................................................................................................................... 9 OUTPUT .............................................................................................................................. 10 DRAC III Field Repair Utility LED Indicators ................................................................... 11 Competitive Analysis .............................................................................................................. 12 Conclusions ............................................................................................................................... 13 Figures Figure 1: Initiating the Field Repair Utility By Inserting a Floppy Diskette........................................ 6 Tables Table 1: Competitive Comparison of RAC versus Compaq Lights Out .............................................12 Revised September 2002 Page 2 Dell Enterprise Systems Group 1 Section Executive Summary This Dell OpenManage™ Step-By-Step paper provides a summary look at the software Field Repair Utility provided with the Dell Remote Access Card (DRAC III), DRAC III/XT, Dell Embedded Remote Access (ERA), and the Dell Embedded Remote Access Option (ERA/O) system management solutions. Collectively, these solutions are known as remote access controllers (RACs). This utility is used for: Reprogramming corrupted RAC firmware Upgrading / downgrading operational RAC firmware Revised September 2002 Page 3 Dell Enterprise Systems Group Section 2 Introduction The RAC provides a complete hardware and software solution that is installed into Dell PowerEdge servers to provide remote systems management to the server. Some of the key functions provided by the RAC include: High performance console redirection Inspection of hardware sensors such as temperature, voltage, and fans Access to hardware and alert logs Ability to generate alerts, even when the system is down Ability to power up and power down the system Provides remote floppy boot operations The RAC software Field Repair Utility is a powerful tool to reprogram the RAC firmware without having to remove the hardware and return it to Dell in the event of a software failure. This can help Administrators achieve productivity gains made by having the ability to reprogram the RAC without having to endure the downtime of waiting for repacement hardware. Although this utility is currently supported only on PowerEdge 1600SC, 1650, 2600, 2650, 4600, 6600, 6650 servers, this feature can be used on any 32-bit server that boots to a DOS partition. Current implementation does not support systems that boot to a non-DOS partition (i.e., systems that boot to EFI – Extensible Firmware Interface). This paper describes the software Field Repair Utility including: Overview of Field Repair Utility How To Detect When To Use It How To Use the Field Repair Utility Brief Competitive Analysis Revised September 2002 Page 4 Dell Enterprise Systems Group Section 3 Overview of the Software Field Repair Utility The Dell Field Repair Utility command allows the user to totally re-flash the RAC firmware. This operation can be done on: A working RAC A RAC that has corrupted flash images and is no longer operational The Dell Field Repair Utility is a software tool that is distributed on a set of DOSbootable floppy diskettes. The diskettes have the necessary programs and files to locate the RAC device in the system, install programs to communicate, and reprogram the card over the PCI bus. The entire process is designed to take less than five minutes and requires minimal interaction from the user, involving inserting the floppy diskettes into the managed node that contains the RAC, rebooting the system, and confirming that the RAC should be reprogrammed. As the system boots, the boot diskette unzips all of the required files and runs the racflash.exe command, which performs the programming functions. As the command executes, it displays status to the user on the console. The filename is displayed as each file is downloaded to the RAC and programmed. Large files display a byte count as a progress indicator. When programming starts, a seconds counter is displayed to also indicate to the user that programming is taking place. Once complete, the RAC device resets. It is important to note that this recovery tool overwrites all of the files on the RAC device. Any information that the user has configured in the user database will be lost. The user database is reset to factory defaults. After flashing the RAC, the user removes the repair diskette and reboots the managed node. Each RAC hardware device (DRAC III, ERA, ERA/O, DRAC III/XT) may have a unique repair utility package that contains RAC hardware specific firmware. To prevent the wrong firmware from being programmed into the RAC device, the utility will prompt the user to confirm the correct firmware is being installed as well as automatically verify the firmware matches the hardware installed in the system. If the user confirms an incorrect firmware to be installed on the RAC hardware, the repair utility will print an error message indicating that the firmware version is incorrect for the target RAC hardware detected and abort without updating the RAC. Revised September 2002 Page 5 Dell Enterprise Systems Group Section 4 Detect When to Use It DRAC III A DRAC III that has a corrupt flash image usually demonstrates that corruption by the amber LED flashing at a ½ second rate, and the DRAC III being unresponsive The DRAC III has two visual LEDs on the card’s edge – one green, one amber – that enables a user to visually monitor the health of the card. In normal operation, the green LED blinks at a 1 second rate to indicate to the user good health. In the event the card becomes corrupt or unusable, the green LED is turned off and the amber LED flashes. If the amber LED is flashing and continues to flash after the DRAC III card has been reset, an administrator might assume that this means that the card needs to be sent back to the factory. Not with the Dell DRAC III card. A system administrator can use the Dell software Field Repair Utility to repair the firmware and be back on line in a matter of minutes in most cases, see Figure 1. ERA, ERA/O, DRAC III/XT Unlike the DRAC III, the ERA, ERA/O, DRAC III/XT do not have a visible LED to assist the user in determining the operational state of the hardware. Some symptoms a RAC device is corrupt include: Unable to log into RAC GUI over LAN connection, Unable to communicate using racadm CLI commands, BIOS delays system boot with console message trying to initialize the RAC device. The system will continue to boot after a short delay stating that the RAC failed to initialize, During system boot, the text warning message “Embedded Systems Management is not present” is displayed on the console. Floppy Disk with software to repair RAC Server with RAC Figure 1: Initiating the Field Repair Utility By Inserting a Floppy Diskette Revised September 2002 Page 6 Dell Enterprise Systems Group Section 5 How to Use Software Field Repair Utility Overview Use the Field Repair Utility by following the steps below. 1. Extract the software Field Repair Utility zip file to any local system from Dell Support. 2. When the webpack package is executed a banner message is displayed with the firmware version, systems supported, and instructions to create the diskette set and reprogramming the RAC device. The program prompts the user to insert two diskettes on which the utility is copied, automatically creating a boot disk for the user. Refer to instructions below. 3. Insert the first boot floppy disk into the system that contains the RAC and reboot the system. 4. Once the repair program starts, a banner message is displayed indicating the firmware version and systems supported. The user is prompted to confirm that the PowerEdge system and firmware versions are correct before reprogramming the RAC hardware. 5. If confirmed, the user is prompted to insert the second diskette and press any key. 6. The necessary files on the floppy are automatically unzipped to the target platform. 7. The utility verifies that the repair utility matches the RAC hardware found in the system and reprograms the RAC over the PCI bus. 8. Once the RAC has been flashed, remove the disk and reboot the system. Revised September 2002 Page 7 Dell Enterprise Systems Group Webpack Instructions The following instructions are an example of what is displayed to the user when the webpack package is executed to create the repair utility diskettes. DELL Remote Access Controller (RAC) Firmware Repair Utility Firmware Version X.Y Instructions to create Repair Diskettes: --------------------------------------------------------------1. This utility will require 2 formatted diskettes. 2. Insert the first diskette 3. Press the Continue button. 4. When the copy is complete, remove the diskette and insert the second diskette. 5. Press the Continue button. 6. When the copy is complete, press the Finish button. Supported PowerEdge Systems: --------------------------------------------------------------PowerEdge XXX RAC Repair Utility Instructions: --------------------------------------------------------------1. Insert the first update diskette into the target system with RAC installed and reboot the system. You are prompted to confirm reflashing the RAC. 2. After you confirm the update, you are prompted to insert the second update diskette. The RAC Flash update will start automatically, displaying status information on the screen. 3. After the update has successfully completed,remove the update diskette and reboot the system. NOTE: This utility reprograms the RAC to preinstalled settings. All user modified settings will be erased. If you are upgrading a system and you want to save/restore any modified RAC configurations, see the racadm getconfig/config commands in the "Dell Remote Access Controllers Installation and Service Guide" for instructions. Revised September 2002 Page 8 Dell Enterprise Systems Group Section 6 Field Repair Utility Syntax NAME racflash - Reflash a good, or a corrupted RAC device. SYNOPSIS racflash [–p <repair file path>] racflash -v REQUIREMENTS/DESCRIPTION The racflash command allows the user to totally re-flash the RAC firmware. This operation can be done on: A working RAC A RAC that has corrupted flash images and is no longer operational A DRAC III that has a corrupt flash image is usually indicated by the amber LED flashing at a ½ second rate, and the DRAC III being unresponsive. This recovery tool overwrites all of the files on the card. It does not matter what the previous version was, or the new version that is being installed on the card. An ERA, ERA/O, DRAC III/XT will not be responsive to user communication via the LAN or racadm CLI commands. Note: Any information that the user has configured in the user database will be lost. The user database is reset to factory defaults. Note: This repair process is only available on DOS. INPUT -v The version option will display the firmware version only. No flashing takes place. -p The path option allows the user to specify where all of the update files are located. Revised September 2002 Page 9 Dell Enterprise Systems Group OUTPUT The filename is displayed as each file is downloaded to the RAC and programmed. Large files display a byte count as a progress indicator. When programming starts a seconds counter is displayed to also indicate to the user that programming is taking place. Revised September 2002 Page 10 Dell Enterprise Systems Group Section 7 DRAC III Field Repair Utility LED Indicators The DRAC III LEDs indicate to the user which of the five modes the DRAC III hardware is in currently. Mode: LEDs: Description: Operating Mode Green Flashing Amber OFF The DRAC III FW is operating normally. Mode: LEDs: Repair Mode Green OFF Amber: Flashing at a 1/2 second rate Description: The DRAC III FW has detected an incorrect/corrupt flash image. The DRAC III is unavailable for normal operation and the Field Repair Utility MUST be used to reflash the DRAC III. Mode: LEDs: Load Mode Green ON Amber Flashing Description: The DRAC III loader code has been uploaded successfully and is in a mode to be able to receive flash programming data from the utility. Mode: LEDs: Program Mode Green OFF Amber ON Description: The DRAC III loader has received a flash image and is now programming/updating the flash. Mode: LEDs: Init Mode Green Flashing at a 1/2 second rate Amber OFF Description: This mode is only present for a very short duration (maybe a second or two). The DRAC III HW is being setup to receive the DRAC III loader. Once the DRAC III. loader software is programmed and executing, the card will transition into "Program Mode" as defined above. Revised September 2002 Page 11 Dell Enterprise Systems Group Section 8 Competitive Analysis Table 1 shows a competitive comparison of the RAC Field Repair Utility with the Compaq Lights Out remote management board. Information is current as of September 18, 2002. Competitive analysis is based on information provided on the Compaq web site and Dell assessment based on internal research results. Function RAC Compaq Lights Out Software Repair Utility (free) Yes No Blink code diagnostic to troubleshoot board Yes No Table 1: Competitive Comparison of RAC versus Compaq Lights Out Compaq does not provide a field repair utility with its Lights Out card. If the Compaq Lights Out card suffers a failure, it must be replaced. Revised September 2002 Page 12 Dell Enterprise Systems Group Section 9 Conclusions The RAC software Field Repair Utility provides an administrator with a userfriendly solution to repairing the RAC hardware instead of replacing it. The RAC software Field Repair Utility has been designed to be easy-to-use and quick to deploy, as the user simply has to insert two floppy diskettes and reboot the system for the utility to initiate a repair of the hardware. Please refer to Dell Remote Access Solutions INFOBrief at http://www.dell.com/downloads/global/topics/openmanage/RemoteMgmtCardsI NFOBrief.doc for more information on other RAC features. The features of the RAC include: In-band configuration using Server Administrator, racadm command line interface, and the Option ROM Out-of-band management capabilities using NIC, modem (DRAC III only), VT100 (DRAC III only) Fully integrated with other OpenManage products Constantly monitors system health Alerts system events using alpha or numeric paging (DRAC III only), email, SNMP traps Console Redirection (graphical & text) Dead Server Management Access to SEL and POST logs Embedded web server supporting IE and Netscape Support for 32- and 64-bit platforms (DRAC III only) Operation from PCI power, external AC power adapter (DRAC III only), and/or battery power(DRAC III only) Revised September 2002 Page 13 Dell Enterprise Systems Group This White Paper is for informational purposes only. DELL MAKES NO WARRANTIES, EITHER EXPRESS OR IMPLIED, IN THIS WHITE PAPER. Dell cannot be responsible for errors in typography or photography. Dell, OpenManage, and PowerEdge are trademarks of Dell Computer Corporation. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others. ©Copyright 2002 Dell Computer Corporation. All rights reserved. Reproduction in any manner whatsoever without the express written permission of Dell Computer Corporation is strictly forbidden. For more information, contact Dell. Information in this document is subject to change without notice. Revised September 2002 Page 14 Dell Enterprise Systems Group