MGX 8250 and MGX 8850 (PXM1) − Boot Code and Firmware Graceful Upgrade Script Document ID: 6934 Contents Introduction Before You Begin Conventions Prerequisites Components Used Background Task Detail Stage 1: Planning Stage 2: Network Preparation Stage 3: The Upgrade Appendix A − Network Health Check Related Information Introduction This document describes the Cisco recommended 28−step process for an MGX 8850 Edge Switch graceful upgrade. Before You Begin Conventions For more information on document conventions, see the Cisco Technical Tips Conventions. Prerequisites Graceful upgrades cause little or no service disruption and are recommended when upgrading: • To a compatible firmware version. • To a compatible database / Management Information Base (MIB) structure. • To a redundant MGX 8850 with two Processor Switch Modules (PXMs). The MGX 8850 graceful upgrade uses the following commands. All commands are case sensitive. Command Switch Software Upgrade Equivalent install first loadrev to new version newrev runrev to new version Function Loads the new firmware version. Executes the new firmware version. Results in a switchcc from the Active PXM / Primary Service Module to the Standby PXM / Secondary Service Module. commit abort Completes the upgrade to the second loadrev new firmware version. Graceful to new version downgrade to original firmware version is lost. loadrev to old version Restores the PXM to the original firmware version. Must be issued prior to the commit command. Not supported for Service Module firmware. The MGX 8850 firmware provides redundancy by providing support for hot insertion and removal of the PXM module, as well as 1:1 hot standby redundancy for high availability of the MGX 8850. The active and standby PXM have exactly the same database in the local memory at any given time. The active PXM is responsible for updating the standby PXM whenever the database is changed. When the active PXM fails, the standby PXM will switchover in 100 milliseconds (msec). The switchover is transparent to the RPM and service modules. In some cases, older firmware versions are incompatible with newer versions due to incompatible database structures or incompatible MIB structures, and the MGX 8850 Boot Code and Firmware Upgrade Script for Non−Redundant Switches script should be used. To determine compatibility, please refer to the Release Notes for the desired firmware. The tasks listed in this document are recommended for redundant MGX 8850 firmware upgrades using two PXMs. The tasks were verified in the order shown in a laboratory test of a redundant MGX 8850 upgrade from release 1.1.21 to release 1.1.24. To maintain database integrity an interim PXM runtime firmware upgrade to release 1.1.23 was required. The path of the graceful upgrade was: • 1.1.21 −> 1.1.23 −> 1.1.24. This document lists the minimum required steps, and then addresses each step in some detail. The MGX 8850 is based on the same platform as the MGX 8220, and it is recommended that the MGX 8220 Upgrade and Downgrade Matrices, Concepts and Definitions be reviewed to familiarize the reader with general upgrade concepts. The screen displays used to illustrate the tasks were taken from laboratory equipment and are in no way intended to specify Internet Protocol (IP) addressing or naming schemes. Caution: • Only one image must be loaded onto the PXM per Trivial File Transfer Protocol (TFTP) session. • Multiple TFTP sessions are required to load boot code and firmware images onto a PXM. • If multiple firmware images are loaded in one TFTP session, all files copied after the initial image will be corrupted. • This document is intended to be used as an aid for conducting successful firmware upgrades, but is not a substitute for proper planning with your Cisco Sales Engineer, Systems Engineer, or Account Manager. Components Used The information in this document is based on the software and hardware versions below. • Graceful PXM runtime firmware upgrades are not supported from release 1.1.21 to release 1.1.24. This document includes an interim PXM runtime firmware upgrade to 1.1.23, which ensures database integrity and user traffic continuity. • No graceful downgrade is supported from release 1.1.24 or later to version 1.1.21 or below due to MIB changes. Background This section explains IP addressing on the MGX 8850 shelf in general. There are three separate IP addresses for on MGX 8850 shelf with two PXMs. • One cnfifip IP address, also known as the shelf IP address • Two bootChange IP addresses, also known as the PXM IP address The cnfifip IP address or the shelf IP address is the live IP Address of the Active PXM Ethernet port on the MGX 8850. It is the IP address used to manage the MGX 8850 shelf. If a switchcc occurs, the new MAC address of the Standby PXM card is automatically broadcast out and takes over the cnfifip IP address. To verify the existing IP address, issue the dspifip command. The dspifip output also displays the ATM and SLIP addresses assigned to the MGX 8850 shelf. • The ATM address is used for inband IP routing (NWIP) management of the MGX 8850 shelf. • The SLIP address is a legacy assigned to the MGX 8850. The SLIP interface does not support statistics collection. The cnfifip and bootChange IP addresses are retained after the clrallcnf command is issued. bootChange is a Service−level command that is used as necessary for MGX 8850 bring up when the PXMs have no run time firmware. The bootChange IP address or the PXM IP address should be different than the cnfifip IP address. The bootChange IP address of the Active PXM should also be different than the bootChange IP address of the Standby PXM. The bootChange IP address is active only when the PXM is in the boot mode or when the PXM is in the Standby mode and is used to load firmware and boot code directly into the PXM. Refer to Bringing up the PXM with no runtime firmware for more information. Once the PXM is booted up, the cnfifip IP address is active. The bootChange gateway address specifies the next hop that allows the shelf to communicate with a laptop (PC) or Cisco WAN Manager (CWM) station on a different LAN segment while the MGX 8850 is in the boot mode. To view the bootChange IP address of the PXM when the MGX 8850 shelf is using runtime firmware, issue the version command. sj_core.1.7.PXM.a > bootChange '.' = clear field; '−' = go to previous field; boot device : lnPci processor number : 0 host name : solwandbg1 file name : inet on ethernet (e) : 10.1.2.15:ffffff00 inet on backplane (b): host inet (h) : gateway inet (g) : 10.1.1.1 user (u) : autoprog ftp password (pw) (blank = use rsh): flags (f) : 0x0 target name (tn) : pxm−7 startup script (s) : other (o) : ^D = quit sj_core.1.7.PXM.a > dspifip Interface −−−−−−−−−−−−−−− Ethernet/lnPci0 SLIP/sl0 ATM/atm0 Flag −−−− UP DOWN DOWN IP Address −−−−−−−−−−−−−−− 10.1.2.44 0.0.0.0 0.0.0.0 Subnetmask −−−−−−−−−−−−−−− 255.255.255.0 255.255.255.0 255.255.255.0 Broadcast Addr −−−−−−−−−−−−−−− 10.1.1.1 (N/A) 0.0.0.0 sj_core.1.7.PXM.a > To assign a bootChange IP address to the Standby PXM, issue the Service level shellCon command and the bootChange command. The Ethernet port of the Standby PXM must be cabled to a hub or similar network device to load files using the bootChange IP address. Cisco recommends using two LAN connections when loading the ComMat.dat file onto the Active and Standby PXMs. If you only use one LAN connection, move the cable from the Active PXM to the Standby PXM to download the ComMat.dat file. sj_core.1.7.PXM.a >cc 8 (session redirected) sj_core.1.8.PXM.s >shellCon −> bootChange To abort the command use Ctrl−C. To exit from the shellCon mode issue quit. Task Detail Stage 1: Planning The following summarizes the planning steps that are necessary for a successful upgrade. All steps should be completed irrespective of network size. 1. Evaluate known anomalies in the selected release. Some anomalies may require additional preparation in order to ensure a smooth upgrade. This may mean: ♦ Additional upgrade steps ♦ Parameter changes ♦ Workarounds 2. Review release notes for upgrade steps specific to this release. As in Task 1, this task may result in: ♦ Additional upgrade steps ♦ Parameter changes ♦ Workarounds 3. Write scripts, which is an optional task to aid the parameter changes required in certain sections of Stage 3. Writing and testing scripts will: ♦ Make the parameter change process easier to execute ♦ Highlight any commands that have changed in the new firmware release. There are various products that can be used to aid in setting parameters in preparation for a network upgrade. Stage 2: Network Preparation The following summarizes the network preparation steps that are necessary for a successful upgrade. All steps should be completed irrespective of network size. Note: This stage needs to be completed one week before firmware upgrade. 1. Network health check. See Appendix A. 2. Monitor network closely until time of upgrade. Step 1 should highlight any existing network issues, but it is prudent to monitor the network for new firmware errors and card errors right up to the time of the upgrade. Report recurring errors to Cisco TAC. See Appendix A for details on checking for firmware errors and card errors. 3. Verify network management connectivity to network nodes. Ensure that every network MGX 8850 shelf can be connected to using Out of Band access. Using TELNET, connect to each MGX 8850 in the network. 4. Verify the CardState of both PXMs. Verify that one PXM is Active and the other Standby. Issue the dspcds command to verify the state of both PXMs. If the PXM states are not Active and Standby, do not proceed with the upgrade. A sample dspcds output that displays the correct state of both PXMs is provided below. Note that for this document, only the first page of dspcds output is provided. jet.1.7.PXM.a > dspcds Slot −−−− 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.18 1.19 CardState −−−−−−−−−−− Active Active Active Active Empty Empty Active Standby Empty Active Active Empty Empty Empty Empty Empty Empty Empty Empty CardType −−−−−−−− FRSM−2E3 FRSM−2CT3 FRSM−2E3 VISM−8T1 PXM1−OC3 PXM1−OC3 RPM VISM−8E1 CardAlarm −−−−−−−−− Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Type <CR> to continue, Q<CR> to stop: 5. Verify bootChange address configuration on each of the PXMs. Redundancy −−−−−−−−−−− Use the Service level bootChange command to assign a unique IP address to each PXM in the MGX 8850 shelf. The bootChange IP address is used to load runtime firmware onto a PXM. The bootChange IP address must also be different from the IP address assigned to the MGX 8850 shelf using the cnfifip command. jet.1.7.PXM.a > bootChange '.' = clear field; '−' = go to previous field; ^D = quit boot device : lnPci processor number : 0 host name : solwandbg1 file name : inet on ethernet (e) : 192.168.1.65:ffffff00 inet on backplane (b): host inet (h) : gateway inet (g) : 192.168.1.1 user (u) : autoprog ftp password (pw) (blank = use rsh): flags (f) : 0x0 target name (tn) : pxm−7 startup script (s) : other (o) : To verify the bootChange IP address of the Active PXM issue the version command. jet.1.7.PXM.a > version VxWorks (for POPEYE) version 5.3.1. Kernel: WIND version 2.5 Made on Mar 30 1999, 12:20:01. Boot line: lnPci(0,0)solwandbg1: e=192.168.1.65 g=192.168.1.1 u=autoprog tn=pxm−7 PXM firmware version : 1.0.00 Boot Image version : 1.0.00Dc1 To assign bootChange IP address to the Standby PXM, issue the Service level shellCon command and then use the bootChange command. jet.1.7.PXM.a >cc 8 (session redirected) jet.1.7.PXM.s >shellCon −> −> bootChange bootChange '.' = clear field; '−' = go to previous field; boot device : lnPci processor number : 0 host name : solwandbg1 file name : inet on ethernet (e) : 192.168.1.30:ffffff00 inet on backplane (b): host inet (h) : gateway inet (g) : 192.168.1.1 user (u) : autoprog ftp password (pw) (blank = use rsh): flags (f) : 0x0 target name (TN) : pxm−7 startup script (s) : other (o) : value = 0 = 0x0 ^D = quit −> quit quit (session resumed) jet.1.8.PXM.s > version VxWorks (for POPEYE) version 5.3.1. Kernel: WIND version 2.5. Made on Jun 6 2000, 23:05:55. Boot line: lnPci(0,0)solwandbg1: e=192.168.1.30:ffffff00 g=192.168.1.1 u=autoprog TN=pxm7 PXM firmware version : 1.1.21 Boot Image Version : 1.1.21 Issue the cnfifip command to assign the IP address used to connect to the MGX 8850 shelf. The IP address assigned by the cnfifip command is the IP address used to connect to the MGX 8850 when the shelf is in a normal operating state. jet.1.7.PXM.a > cnfifip 26 192.168.1.23 255.255.255.0 192.168.1.255 To verify the shelf IP address issue the dspifip command. jet.1.7.PXM.a > dspifip Interface Flag −−−−−−−−−−−−−−− −−−− Ethernet/lnPci0 UP SLIP/sl0 DOWN ATM/atm0 DOWN IP Address −−−−−−−−−−−−−−− 192.168.1.23 0.0.0.0 0.0.0.0 Subnetmask −−−−−−−−−−−−−−− 255.255.255.0 255.255.255.0 255.255.255.0 Broadcast Addr −−−−−−−−−−−−−−− 192.168.1.255 (N/A) 0.0.0.0 The ATM address is used for in−band management of the MGX 8850 shelf over the feeder trunk to the Cisco BPX 8600 Series Switch. Stage 3: The Upgrade The following summarizes the steps that are necessary for a successful upgrade. All steps should be completed irrespective of network size. 1. Provisioning freeze starts. Halt provisioning of new services until completion of upgrade. 2. As a precautionary step, save MGX 8850 PXM and Service Module (SM) configuration. Save a snapshot of the MGX 8850 configuration on a CWM (SV+) workstation. If the MGX 8850 configuration is not saved, the entire configuration must be manually re−entered. jet.1.7.PXM.a > saveallcnf jet.1.7.PXM.a > ll C:/CNF size date time −−−−−−−− −−−−−− −−−−−− 512 MAY−21−1999 17:46:12 DIR> 512 MAY−21−1999 17:46:12 182762 JUL−06−2000 15:33:45 182762 JUL−06−2000 15:33:48 In the file system : total space : 819200 K bytes free space : 712933 K bytes name −−−−−−−− . < .. <DIR> jet_1533000602.zip jet.zip From the TFTP server issue the following commands to save the configuration file to the server. The TFTP server can be a Unix workstation or CWM workstation. unix−prompt>tftp 192.168.1.23 tftp>bin tftp>get CNF/jet_1533000602.zip Received 182762 bytes in 2.4 seconds tftp>quit 3. View and record card errors and clear all error log files. On all nodes to be upgraded record the card errors and clear the card errors using the following commands on the respective cards: dspcderrs on the PXM, FRSM, AUSM, VISM, CESM. clrcderrs on the FRSM, AUSM. clrerr on the PXM. clrlog on the PXM. 4. Load new revision into CWM (SV+) stations. Load new firmware version into CWM (SV+) stations. Verify that the images have loaded successfully by comparing file sizes to those listed in the Firmware Release Notes. 5. Remove the cause of all MAJOR alarms and if possible all MINOR alarms. Ideally, the network should be alarm free at the time of the firmware upgrade. If this is not possible, at least the reason for all major alarms should be identified and noted, and then suitable reconfiguration should be made in order to remove the alarm. Verify connection totals by issuing the dsptotals command as described in Appendix A. Any minor alarms should be noted so that, after the upgrade, a comparison can be made. 6. Load target boot code revision into PXM. Upload the new PXM boot code to the MGX 8850 using the TFTP process and verify the checksum. The byte count and checksum below is just an example. It will be different for different images. For this test, the intermediate PXM boot code version of 1.1.23 is not required. unix−prompt>tftp 192.168.1.23 tftp>bin tftp>put pxm_bkup_1.1.24.fw POPEYE@PXM.BT Sent 1274256 bytes in 7.2 seconds tftp>quit jet.1.7.PXM.a > Program length = 1274256 Calculated checksum = 0xb5fb283e stored checksum = 0xb5fb283e Fw checksum passed The PXM executes boot code sequentially, so if there is an older image loaded, the PXM will execute the oldest image. To avoid this problem, either delete the existing boot code image or rename the filename with a .old extension. If the existing boot code image is renamed, the FW directory contents will have two boot code files, one with a .old extension. A sample FW directory is provided below. To view the contents of the FW directory; from the C: drive issue the cd FW command and then the ll command. The current boot code file and two old boot code files are highlighted. jet.1.7.PXM.a > ll size date −−−−−−−− −−−−−− 512 JUL−21−2000 time −−−−−− 17:13:30 name −−−−−−−− . <DIR> 512 JUL−21−2000 17:13:30 .. <DIR> 2105328 JUL−20−2000 14:30:12 pxm_1.1.11_fw.old 620368 JUL−20−2000 16:49:48 sm90.fw 799440 MAY−11−2000 18:53:24 sm35.fw 1178168 MAY−11−2000 18:54:40 sm50.fw 934356 JUL−21−2000 11:47:08 sm130.fw 1246872 JUL−20−2000 15:54:40 pxm_bkup_1.1.12.old 21 JUL−24−2000 15:58:44 ComMat.dat 1265620 JUL−24−2000 10:36:14 pxm_bkup_1.1.21.old 1253388 NOV−16−1999 06:42:38 pxm_bkup_1.1.13.fw 1246872 OCT−20−1999 11:07:28 pxm_bkup_1.1.12.old 2105328 OCT−20−1999 11:58:34 pxm_1.1.11.fw 644624 OCT−20−1999 12:07:38 pxm_bkup_1.1.01.old 2006664 OCT−20−1999 12:02:16 pxm_1.1.01.fw 2117676 NOV−16−1999 06:45:22 pxm_1.1.12.fw 1274256 JUL−24−2000 13:42:42 pxm_bkup_1.1.24.fw 2183088 JUL−24−2000 13:47:42 pxm_1.1.24.fw 2182548 JUL−24−2000 14:45:18 pxm_1.1.23.fw In the file system : total space : 819200 K bytes free space : 727272 K bytes Note: The firmware files displayed using the ll command are a superset of the firmware files displayed by the dspfwrev command. jet.1.7.PXM.a > dspfwrevs Card Type Date Time −−−−−−−−−−− −−−−−−−−−−−−−−−−−−− CESM−8T1E1 07/20/2000 16:49:48 FRSM−8T1E1 05/11/2000 18:53:24 AUSM−8T1E1 05/11/2000 18:54:40 FRSM−VHS 07/21/2000 11:47:08 PXM1 07/24/2000 11:21:48 VISM−8T1E1 07/24/2000 12:04:34 PXM1 07/24/2000 13:42:42 PXM1 07/24/2000 13:47:42 PXM1 07/24/2000 14:45:18 Size −−−−−−−− 620368 799440 1178168 934356 2147060 1315400 1274256 2183088 2182548 Version −−−−−−−−−−−−−−−−−−− 10.0.04 10.0.11 10.0.11 10.0.11 1.1.21 1.0.02 1.1.24 1.1.24 1.1.23 File Name −−−−−−−−−−−−−−−−−− sm90.fw sm35.fw sm50.fw sm130.fw pxm_1.1.21.fw sm150.fw pxm_bkup_1.1.24.fw pxm_1.1.24.fw pxm_1.1.23.fw The newly uploaded firmware files will be automatically replicated on to the Standby PXM in few seconds. To verify the files on the Standby PXM, issue the following commands: a. cc <card_number> b. CD FW c. ll The listing of firmware images residing on the Standby PXM in slot 8 is provided below. jet.1.8.PXM.s > ll size date −−−−−−−− −−−−−− 512 MAY−12−2000 512 MAY−12−2000 2105328 JUL−20−2000 620368 JUL−20−2000 799440 MAY−11−2000 1178168 MAY−11−2000 934356 JUL−21−2000 1265620 JUL−24−2000 2147060 JUL−24−2000 21 JUL−24−2000 1246872 JUL−20−2000 1315400 JUL−24−2000 1274256 JUL−24−2000 time −−−−−− 00:03:16 00:03:16 14:30:12 16:49:48 18:53:24 18:54:40 11:47:08 10:36:14 11:21:48 15:58:44 15:54:40 12:04:34 13:42:42 name −−−−−−−− . <DIR> .. <DIR> pxm_1.1.11_fw.old sm90.fw sm35.fw sm50.fw sm130.fw pxm_bkup_1.1.21.old pxm_1.1.21.fw ComMat.dat pxm_bkup_1.1.12.old sm150.fw pxm_bkup_1.1.24.fw 2183088 2182548 JUL−24−2000 JUL−24−2000 13:47:42 14:45:18 pxm_1.1.24.fw pxm_1.1.23.fw In the file system : total space : 819200 K bytes free space : 682019 K bytes jet.1.8.PXM.s > 7. Load intermediate and target runtime firmware versions into PXMs. Upload the intermediate and target runtime firmware versions to the MGX 8850 using the TFTP process and verify the checksum. The byte count and checksum below is shown for illustration and the values will be different for other images. Note that for this test, both 1.1.23 and 1.1.24 versions of runtime firmware are loaded. Storing multiple versions of runtime firmware can be accomplished as long as the order of firmware upgrade steps is followed. unix−promt>tftp 192.168.1.23 tftp>bin tftp>put pxm_1.1.23.fw POPEYE@PXM.FW Sent 2182548 bytes in 10.4 seconds tftp>quit jet.1.7.PXM.a > Program length = 2182548 Calculated checksum = 0xa65cb14f stored checksum = 0xa65cb14f Fw checksum passed unix−promt>tftp 192.168.1.23 tftp>bin tftp>put pxm_1.1.24.fw POPEYE@PXM.FW Sent 2182548 bytes in 10.4 seconds tftp>quit jet.1.7.PXM.a > Program length = 2182548 Calculated checksum = 0xcb8h24ac stored checksum = 0xcb8h24ac Fw checksum passed To verify the uploaded versions on each of the PXMs issue the dspfw command. jet.1.7.PXM.a > dspfw PXM FW versions: "1.1.21" in pxm_1.1.21.fw "1.1.24" in pxm_1.1.24.fw "1.1.23" in pxm_1.1.23.fw jet.1.7.PXM.a > cc 8 (session redirected) jet.1.8.PXM.s > dspfw PXM FW versions: "1.1.21" in pxm_1.1.21.fw "1.1.24" in pxm_1.1.24.fw "1.1.23" in pxm_1.1.23.fw 8. Install the intermediate version ComMat.dat file on both the PXMs The ComMat.dat file contains compatibility matrix data that specifies the firmware version ranges that support graceful upgrades. Different versions of the ComMat.dat file can not be stored on the PXM. Each version of the ComMat.dat file will need to uploaded prior to each installation of runtime firmware. Upload the 1.1.23 ComMat.dat file and then copy to the C:/FW directory of the Active PXM. UNIX−prompt>tftp 192.168.1.23 tftp>bin tftp>put ComMat.dat Sent 21 bytes in 0.3 seconds tftp>quit jet.1.7.PXM.a > pwd C: jet.1.7.PXM.a >mv ComMat.dat C:/FW/ComMat.dat To upload the ComMat.dat file to the Standby PXM, use the bootChange IP address for the TFTP. The bootChange IP address is functional when the PXM is in the Standby state. Copy the ComMat.dat file to the C:/FW directory of the Standby PXM. UNIX−prompt>tftp 192.168.1.30 tftp>bin tftp>put ComMat.dat Sent 21 bytes in 0.3 seconds tftp>quit jet.1.8.PXM.s > pwd C: jet.1.8.PXM.s > MV ComMat.dat C:/FW/ComMat.dat 9. If network has been stable for 30 minutes after the successful firmware downloads, install the boot code into the PXM flash. Issue the install bt command to upload the boot code file into the PXM flash memory. This command will download the boot code to both the Active and Standby PXM. jet.1.7.PXM.a > install bt "1.1.24" writing pxm_bkup_1.1.24.fw to flash... Board recognised as a PXM1B board ... Checksum size is 1274256 ... Erasing the flash .... FLASH erase complete Downloading C:/FW/pxm_bkup_1.1.24.fw into the flash ... verifying flash contents .... Flash ok .... Flash download completed ... copying pxm_bt_1.1.24.fw to standby... writing flash on other card... command completed OK on both pxms. The new boot code will be used after the next reset 10. Upgrade to the intermediate PXM runtime firmware version using the install, newrev, and commit commands. Issue the install 1.1.23 command to install the intermediate PXM runtime firmware. The Standby PXM will reset and go into the Hold state. This will take a few seconds. jet.1.7.PXM.a > this may take a install command please wait for install 1.1.23 while ... completed OK the other card to enter the hold state. jet.1.7.PXM.a > dspcds Slot CardState CardType −−−− −−−−−−−−−−− −−−−−−−− 1.1 Active FRSM−2E3 1.2 Active FRSM−2CT3 1.3 Active FRSM−2E3 1.4 Empty 1.5 Empty CardAlarm Redundancy −−−−−−−−− −−−−−−−−−−− Clear Clear Clear Clear Clear 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.18 1.19 Empty Active Hold Empty Active Active Empty Empty Empty Empty Empty Empty Empty Empty Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear PXM1−OC3 PXM1−OC3 RPM VISM−8E1 Type <CR> to continue, Q<CR> to stop: Issue the newrev 1.1.23 command after the Standby PXM is in the Hold state. After the newrev 1.1.23 command is issued, the Active PXM will reset and go to Hold state and the Standby PXM will be Active. jet.1.7.PXM.a > newrev 1.1.23 reset type: 0x00000002 pio input: 0xf00f5771 Error EPC: 0x800c6e70 Status Reg: 0x3040ff05 Cause Reg: 0x00000000 CacheErr Reg: 0xb0000000 Reset L2 cache... DRAM size: 0x08000000 Reset L1 cache... Backup Boot Version: 1.1.24 Verify Checksum... Valid jumping to romStart ............................................. ......................................... To verify the PXM state, login to the console port of the PXM in slot 8. Login: card going active.. SM Feature Bit Map is = 0 SM Feature Bit Map is = 0 After the newrev command is issued, the output of the dspcd command on the PXM in slot 8 will show the interim firmware version. The MGX 8850 is now running the interim firmware and health and status as wells as user traffic should be verified. jet.1.8.PXM.a > dspcd ModuleSlotNumber: FunctionModuleState: FunctionModuleType: FunctionModuleSerialNum: FunctionModuleHWRev: FunctionModuleFWRev: FunctionModuleResetReason: LineModuleType: LineModuleState: SecondaryLineModuleType: SecondaryLineModuleState: mibVersionNumber: 8 Active PXM1−OC3 SCK03160179 A0 1.1.23 Upgrade Reset PXM−UI Present MMF−4−155 Present 0.0.00 configChangeTypeBitMap: cardIntegratedAlarm: cardMajorAlarmBitMap: cardMinorAlarmBitMap: BkCardSerialNum: TrunkBkCardSerialNum: FrontCardFabNumber: No changes Clear Line Alarm Line Statistical Alarm SBK02420284 SAK0320005M 800−05086−03 After the PXM in slot 7 is reset and successfully enters the Hold state, issue the commit 1.1.23 command. The commit 1.1.23 command completes the runtime firmware upgrade on both PXMs and the PXM in slot 7 will now enter the Standby state mgx1.1.8.PXM.a > commit 1.1.23 this may take a while ... commit command completed OK 11. Verify the intermediate version and the CardState of each MGX 8850 PXM. To verify the CardState of the PXMs issue the dspcds command. Note that the PXM which was previously in the Standby state is now Active. Issue the version command to verify the firmware version on each of the PXMs. jet.1.8.PXM.a > dspcds Slot −−−− 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.18 1.19 CardState −−−−−−−−−−− Active Active Active Active Empty Empty Standby Active Empty Active Active Empty Empty Empty Empty Empty Empty Empty Empty CardType −−−−−−−− FRSM−2E3 FRSM−2CT3 FRSM−2E3 VISM−8T1 PXM1−OC3 PXM1−OC3 RPM VISM−8E1 CardAlarm −−−−−−−−− Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Redundancy −−−−−−−−−−− Type <CR> to continue, Q<CR> to stop: 12. Verify PXM functionality. To verify PXM functionality, issue the switchcc command. After the command is executed, the Active PXM will be in slot 7 and the Standby PXM will be in slot 8. Report any alarms incurred during the switchcc command to the Cisco TAC. 13. Install the target version ComMat.dat file into PXMs. The ComMat.dat file contains compatibility matrix data that specifies the firmware version ranges that support graceful upgrades. Different versions of the ComMat.dat file can not be stored on the PXM. Each version of the ComMat.dat file will need to uploaded prior to each installation of runtime firmware. Upload the 1.1.24 ComMat.dat file and then copy to the C:/FW directory of the Active PXM. unix−prompt>tftp 192.168.1.65 tftp>bin tftp>put ComMat.dat Sent 21 bytes in 0.3 seconds tftp>quit jet.1.7.PXM.a > pwd C: jet.1.7.PXM.a >mv ComMat.dat C:/FW/ComMat.dat To upload the ComMat.dat file to the Standby PXM, use the bootChange IP address for the TFTP. The bootChange IP address is functional when the PXM is in the Standby state. Copy the ComMat.dat file to the C:/FW directory of the Standby PXM. UNIX−prompt>tftp 192.168.1.30 tftp>bin tftp>put ComMat.dat Sent 21 bytes in 0.3 seconds tftp>quit jet.1.8.PXM.s > pwd C: jet.1.8.PXM.s > MV ComMat.dat C:/FW/ComMat.dat 14. If network has been stable for 30 minutes after the successful upgrade to the intermediate firmware version, upgrade to the target PXM runtime firmware version using the install, newrev, and commit commands. Repeat steps 9 and 10 in Stage 3 in order to upgrade the PXM runtime firmware from 1.1.23 to 1.1.24. Replace occurrences of 1.1.23 with 1.1.24 in each command. 15. Load target Service Module boot code and firmware versions into the PXM. The PXM evaluates all firmware on the MGX 8850 Service Modules. If the PXM detects any incompatibilities between the PXM and Service Module runtime firmware versions an error or Mismatch condition will result. If the new firmware version does not require a Service Module boot code upgrade, omit the boot code step. Upload the target firmware and boot code for each Service Module to the shelf. Note that the checksum results are only displayed for firmware uploads. Service Module boot code must be loaded per slot. Service Module firmware is copied onto the MGX 8850 PXM hard drive into the /FW directory. If no slot is specified when loading Service Module firmware by using the 0, any Service Module can be inserted into a valid slot and retrieve necessary firmware from the PXM. Loading Service Module firmware without specifying a slot will overwrite the old version of firmware if it exists on the hard drive. Boot code and firmware files will be automatically replicated to the Standby PXM a few seconds after they are loaded onto the Active PXM. To upload the new Service Module boot code : unix−prompt>tftp 192.168.1.23 tftp> bin tftp>put frsm_vhs_VHS_BT_1.0.02.fw POPEYE@SM_1_1.BOOT Sent 457988 bytes in 14.2 seconds tftp>quit Syntax of the put command is put <backup boot> popeye@SM_1_<slot#>.BOOT To upload the new firmware so that it applies to all Service Modules of the same model: unix−prompt>tftp 192.168.1.23 tftp> bin tftp>put frsm_vhs_10.0.12.fw POPEYE@SM_1_0.FW Sent 913360 bytes in 18.3 seconds tftp>quit jet.1.7.PXM.a > Program length = 913360 Calculated checksum = 0xe2f5ca1b stored checksum = 0xe2f5ca1b Fw checksum passed The syntax of the put command to apply firmware to all Service Modules of the same model is: put <firmware_filename> POPEYE@SM_1_0.FW 16. Upgrade Service Module boot code and firmware version. Install the uploaded Service Module firmware for each service module. For ungraceful upgrades associated with non−redundant Service Modules, issue the resetcd <card_number> command from the Active PXM. The resetcd <card_number> command forces the Service Module to execute the new boot code and firmware. The resetcd <card_number> command will cause service interruption to the connections for approximately five minutes as there is no redundant Service Module. For graceful Service Module upgrades, redundancy must be configured and used. The redundant Service Module firmware upgrade uses the same steps as the redundant PXM firmware upgrade, except the abort command is not supported. The MGX 8850 offers 1:1 and 1:N redundancy depending on the Service Module. For this document, 1:1 redundancy is addressed. To configure 1:1 redundancy a secondary Service Module must be available to back−up the primary Service Module. The primary and secondary Service Modules must be the same model, type, and use the same Line Module or back card. To activate 1:1 redundancy between the Service Modules in 2 slots, issue the addred command from the Active PXM. The redundant slots do not need to be contiguous, but a disperse configuration makes cable management and troubleshooting difficult. To identify redundancy on an MGX 8850, issue the dspred command from the Active PXM. Once a Service Module is configured as secondary in a 1:1 redundant scenario, the state changes from Active to Standby. The state change indicates that many commands will not work when issued directly on a Service Module in the Standby state. Commands that do not work on a Service Module in the Standby state include install, newrev, and commit. mgx1.1.8.PXM.a > dspred Primary Primary Primary Secondary Secondary SlotNum Type State SlotNum Type −−−−−−− −−−−−−− −−−−−−− −−−−−−−−− −−−−−−−−− 1 FRSM−2E3 Active 3 FRSM−2E3 Secondary State −−−−−−−−− Standby Red. Type −−−− 1:1 Red.Slot Cover −−−−−−−− 0 Issue the install bt sm <slot_number> <boot_code_version> to execute the target version of boot code. Issue the following commands to execute the target version of Service Module firmware: Command install Syntax install smslot_number> <firmware_version> to new Function Loads the new firmware version. newrev commit abort Executes the new firmware version. Results in a newrev smslot_number> switchcc from the <firmware_version> Primary Service Module to the Secondary Service Module. commit smslot_number> <firmware_version> Completes the upgrade to the new firmware version. Graceful downgrade to original firmware version is lost. The abort command is not supported for graceful Service Module firmware upgrades. jet.1.7.PXM.a > install sm 1 10.0.12 Do you want to proceed (Yes/No)? yes jet.1.7.PXM.a > newrev sm 1 10.0.12 Do you want to proceed (Yes/No)? yes jet.1.7.PXM.a > dspcds Slot CardState CardType CardAlarm Redundancy −−−− −−−−−−−−−−− −−−−−−−− −−−−−−−−− −−−−−−−−−−− 1.1 Boot FRSM−2E3 Clear Covered by slot 3 1.2 Active FRSM−2CT3 Clear 1.3 Active FRSM−2E3 Clear Covering slot 1 1.4 Active VISM−8T1 Clear 1.5 Active VISM−8T1 Clear 1.6 Empty Clear 1.7 Active PXM1−OC3 Clear 1.8 Standby PXM1−OC3 Clear 1.9 Empty Clear 1.10 Active RPM Clear 1.11 Active VISM−8E1 Clear 1.12 Empty Clear 1.13 Empty Clear 1.14 Empty Clear 1.15 Empty Clear 1.16 Empty Clear 1.17 Empty Clear 1.18 Empty Clear 1.19 Empty Clear Type <CR> to continue, Q<CR> to stop: jet.1.7.PXM.a > commit sm 1 10.0.12 Do you want to proceed (Yes/No)? yes 17. Let network settle and run customer specific validation tests. After 10 minutes, login to the target node and verify health using the following commands: ♦ dsplog ♦ dsperr −en ♦ dsptotals This period provides an ideal time to run tests to check that the new firmware is functioning correctly. Interrogate all external management systems that are used to manage any routers that are connected to the MGX 8850 network. This interrogation is done to ensure that all devices are reachable. If possible, end users should be contacted and asked to check that all network connections are in proper working order. Note: In the unlikely event that a decision is taken to revert back to the previous firmware revision, Cisco TAC should be contacted prior to switching to the old revision. Important information as to why the new firmware is not functioning correctly will be lost after switching back to the old revision. 18. Network health check. See Appendix A 19. Save MGX 8850 PXM and Service Module (SM) configuration. See step 2 of Stage 3. 20. Provisioning freeze ends. Appendix A − Network Health Check Follow these steps to check the network health: 1. Audit the parameters within the following commands. Settings should be consistent across all nodes of the same type within the network. Document differences and any variations from the default values. dsptotals dsplog dspalms dspshelfalm 2. Audit network for recent errors (active and standby controller cards), card errors, load model inconsistencies and alarms. Use the following commands to accomplish these tasks: dsperr −en dsplog s dsplog printlog dspcderrs or the dspcderrs <slot #> dspalms 3. Investigate the following: ♦ Recent firmware errors: Any nodes that continually log errors or have logged recent errors should be reported to Cisco TAC. ♦ Card errors: Cards that are logging failures or have a history of hardware errors should be investigated by Cisco TAC. ♦ Any trunks that are logging errors: Should be fixed for the duration of the upgrade. ♦ All alarms should be accounted for. The real purpose of this check is to make sure that there are no alarms that will require special intervention before the upgrade. 4. Ensure that any necessary corrections are made before the start of the upgrade. Related Information • MGX 8220 Upgrade and Downgrade Matrices, Concepts and Definitions • Configuring PXM Cards • 1.1.00 Version Software Release Notes Cisco WAN MGX 8850 Software • MGX 8850 Boot Code and Firmware Graceful Upgrade Script • Cisco WAN Switching Solutions − Cisco Documentation • Guide to New Names and Colors for WAN Switching Products • Software Center − WAN Switching Software • Technical Support − Cisco Systems Contacts & Feedback | Help | Site Map © 2014 − 2015 Cisco Systems, Inc. All rights reserved. Terms & Conditions | Privacy Statement | Cookie Policy | Trademarks of Cisco Systems, Inc. Updated: Apr 17, 2009 Document ID: 6934