VCS UPGRADE NOTES – v2.2
VCS Staggered Upgrade Notes
Additional Information for aVCS staggered upgrade procedures
1 – Overview
This document assumes you have a working knowledge of aVCS. Before attempting any upgrade it is recommended
you consult the appropriate System Administration Guide, and that you are familiar with Management and Data
plane separation.
ACOS Virtual Chassis System (aVCS) enables you to manage multiple ACOS devices as a single, virtual chassis. One
ACOS device in the virtual chassis is the virtual master (vMaster). The other ACOS devices are virtual blades (vBlades)
within the virtual chassis, and are managed by the vMaster.
aVCS, as a management tool, provides high availability functionality on the ACOS device with the help of VRRP-A
and High Availability across two or more ACOS devices.
In an aVCS chassis, there is a single point of management, and the vMaster acts as a controller for vBlades. The
vMaster controller provides centralized storage of the entire ACOS device configuration. Any configuration changes
from the vMaster controller are automatically propagated to the vBlades. This ensures consistency in the event of a
failure of the vMaster.
Depending on the ACOS series model, with the help of VRRP-A, aVCS can support a maximum 7 additional blades.
aVCS requires that all devices in the same virtual switch have the same number of CPUs and are the same ACOS
device model.
2 –Network Topologies
The topology of the network should be considered when using aVCS, not just for staggered upgrade mode, but to
help ensure all features function fully as intended. VCS uses IP Multicast packets (224.0.0.210 default) to communicate
and all aVCS devices should be on the same Layer 2 Network segment. The recommended topology is to always to
cable the VCS back-to-back when using 2 devices with a static or dynamic LACP trunk for port level redundancy.
www.a10networks.com
1
VCS UPGRADE NOTES – 2.2
In cases where VCS has more devices than 2, then a Layer 2 switch will be required. It is still recommended to use a
trunk for interface level resiliency. VCS blades in the same chassis must always be in the same Layer 2 Broadcast
Domain. Having an isolated VLAN dedicated to the VCS / VRPP topology is recommended.
It is of course possible to run VCS chassis over large distances such as separate data centers and even different
countries without issue. Whilst different topologies will not cause a problem. If a vBlade does not receive a heartbeat
message within a specified amount of time (heartbeat dead interval), the vBlade changes its state from vBlade to
vMaster-candidate, and engages in the vMaster election process with the other devices that are still up. The default
heartbeat time is 3 seconds. The default heartbeat dead interval is 10 seconds. Both parameters are configurable.
This introduces some elements that need to be considered when thinking about VCS installation and its operation. In
particular give thought to any devices or elements that could interfere with Multicast and network convergence
times, especially over larger distances.
- Spanning Tree Protocols – These can delay or break VCS Election processes after upgrades / reloads
- Edge Ports - Port Fast should be enabled is STP protocol MUST be used between the ACOS devices
- VRRP Slow Start etc. – Other protocols in the network, which may be slow to converge when a device reloads
- Latency / RTD Times – There is no real restriction here but it should be as low as possible - QOS / Multicast – Any
protocols that may use similar multicast, any mechanisms which may delay/drop multicast.
3 - VCS Upgrade Methods
There are several valid methods to upgrading ACOS in a VCS environment. This document covers Staggered (with
VRRP) and Manual (with VRRP). For all methods and further instruction, please refer to the ACOS release notes.
Staggered (with VRRP-A) – Uses the staggered upgrade feature of VCS. Easier.
Manual (with VRRP-A) – Disables VCS and manually upgrades the devices. More control.
Full Chassis Upgrade – All devices are upgraded and rebooted. Service outage.
Staggered (no VRRP-A) – This is used for older devices
4 – Selecting an upgrade method
If you are certain there is nothing in the network that can delay or drop IP multicast packets after reload (such as STP
protocols), then you can use the staggered upgrade method. See section 1 for more detail on this.
As discussed earlier in the notes, when using staggered upgrade it is important that there is no delay to the multicast
packets reaching the vBlade after it reloads. Failure to ensure this may result in the vBlade declaring itself vMaster (as
www.a10networks.com
2
VCS UPGRADE NOTES – 2.2
it has not received hello from the real vMaster in time). This can clear the staggered upgrade flag on the device and
cause codes to re-sync automatically.
It can also be he case that the vBlade declares itself as vMaster even for a very short time period, until it hears again
from the real vMaster and election takes place. It can appear that the staggered upgrade process is failing and only
a careful analysis of the logs and timeline will show that it became vMaster incorrectly and the flag was cleared.
If you do have more than a few hops switches between the ACOS devices you should consider the manual upgrade
method. This method has a few more steps, but gives you complete control over the process.
NOTE: To assist with this network delay you can configure ‘vcs force-wait-interval seconds’. Number of seconds to wait
following a reload or reboot to start aVCS.
If you have attempted staggered upgrade and it failed, then it is recommended to try the manual approach instead,
this may be an easier option than trying to change the intermediate network.
Comparison of upgrade methods
Complexity
Steps
Reliability
Impact
Staggered
Lower
Less
Lower
Low
Manual
Higher
More
Higher
Low
VCS is a management plane feature, and will not impact the data plane of the A10 device. The only impact to this
approach is the change of VRID. Where session sync features are used it is possible to perform full upgrade without
any packet loss.
5 – Saving configuration and backing up the system
Before any code upgrade you should always backup the system. Refer to the release notes for further details though
it can be run form the CLI with the following commands. Make sure to also save configuration on all devices and
partitions.
backup system [use-­‐mgmt-­‐port] url write memory all The url specifies the file transfer protocol, username (if required), directory path, and filename. The following types of
URLs are supported:
tftp://host/file ftp://[user@]host[:port]/file scp://[user@]host/file rcp://[user@]host/file sftp://[user@]host/file www.a10networks.com
3
VCS UPGRADE NOTES – 2.2
You can enter the entire URL on the command line or press Enter to display a prompt for each part of the URL. If you
enter the entire URL and a password is required, you will still be prompted for the password.
The use-mgmt-port option uses the ACOS device’s management port as the source interface. Otherwise, a data
interface is used.
6 – Staggered upgrade method
In this procedure, the vBlades are upgraded first, followed by the vMaster.
NOTE: These steps assume that when you begin the procedure, the vMaster is also the active VRRP-A device for all
VRIDs. If this is not the case you should change the vMaster using VCS Device Priority. It also assumes that you are NOT
connecting to the VCS Floating IP, but directly to the Management IP address on each device.
NOTE: It is also assumed that all devices in the chassis are running on the same HD (Primary or Secondary) and there is
no mismatch i.e vMaster is running primary and vBlade is running secondary.
NOTE: If you are using VCS VRRP-A Affinity then this should be disabled.
Perform step 1 through step 5 on the vMaster:
1) - On the vMaster, verify the currently running software version and the image area currently in use. List out your
partitions.
show bootimage show version show partition All devices in the virtual chassis use the same image area (primary or secondary). For example, if the software running
on the vMaster is in the primary image area, all the vBlades also are running their software from the primary image
areas on those devices.
2) - Save the configuration to the other image area:
write memory {primary | secondary} [all-­‐partitions]
NOTE: Make sure to use the all-partitions option, if RBA/L3V private partitions are configured.
3) - Upgrade the vBlade, by loading the new software image into the image area currently in use by the vBlade:
upgrade hd {pri | sec} [use-­‐mgmt-­‐port] url staggered-­‐upgrade-­‐mode device DeviceID The device DeviceID specifies the vBlade’s aVCS deviceID.
The url specifies the file transfer protocol, username and password (if required), directory path, and filename.
The use-mgmt-port option uses the ACOS device’s management port as the source interface. Otherwise, a data
interface is used. This step reboots the vBlade. The vMaster continues to operate.
4) - For each VRID that is active on the device, failover from the vMaster to the vBlade using priority.
www.a10networks.com
4
VCS UPGRADE NOTES – 2.2
vrrp-­‐a vrid {num | default} This command changes to the configuration level for the VRID. At this level, use the following command:
priority 255 device DeviceID or vrrp-­‐a force-­‐self-­‐standby {all-­‐partitions} NOTE: If you use the vrrp-a force-self-standby command, do not proceed to step 6 until you are satisfied all services
are working correctly.
NOTE: Remember if you have multiple partitions these will also need to be failed over.
5) - Validate that the load-balanced services are working. (The show commands or other techniques depend on your
deployment. The show slb virtual-server command is useful in almost any deployment.) Perform step 6 on the vBlade,
to take over vMaster role:
6) - On the vBlade that is running the new software image, enter the following command:
At the Privileged EXEC level (AX#), use the following command to force the vBlade to take over the vMaster role. Do
not attempt this until you are satisfied that traffic is working correctly on the vBlade with the new software Image.
vcs vmaster-­‐take-­‐over 255 During failover, the vBlade becomes the vMaster, and the vMaster becomes a vBlade. The new vMaster will detect
that the vBlade device is running old software, and it will upgrade the vBlade. As part of this upgrade, the vMaster will
reboot the vBlade. If you used vrrp-a force-self-standby remember that when the old vMaster reboots it will fail back
the VRIDs after it restarts.
(Optional) Perform step 7 on the new vBlade (former vMaster), to resume the vMaster role and again become the
active device for the VRID:
7) - Optionally, force failover back to the original vMaster.
At the Privileged EXEC level (AX#), use the following command to take over the vMaster role:
vcs vmaster-­‐take-­‐over 255 For each VRID, use the following commands to reset the VRRP-A priority to its previous value.
vrrp-­‐a vrid {num | default} priority previous-­‐value device DeviceID
See the release notes for a full example of the staggered upgrade process with VRRP-A.
www.a10networks.com
5
VCS UPGRADE NOTES – 2.2
7 – Manual upgrade method
In this procedure VCS will be disabled first on all devices and then the upgrade will be made. There will be minimum
impact the same as the staggered upgrade method (the only impact is to change VRRP VRID between devices the
same as the staggered method).
NOTE: These steps assume that when you begin the procedure, the vMaster is also the active VRRP-A device for all
VRIDs. If this is not the case you should change the vMaster using VCS Device Priority. It also assumes that you are NOT
connecting to the VCS Floating IP, but directly to the Management IP address on each device.
It is also assumed that all devices in the chassis are running on the same HD (Primary or Secondary) and there is no
mismatch i.e vMaster is running primary and vBlade is running secondary.
1) - On the vMaster, verify the currently running software version and the image area currently in use. List out your
partitions.
show bootimage
show version
show partition
2) – Take a note of the VCS Configuration from each device. As VCS disable can remove some lines, it is best to keep
a note of the complete configuration from each device in the chassis.
sh run | section vcs
vcs enable vcs vMaster-­‐id 2 vcs chassis-­‐id 1 vcs floating-­‐ip 10.158.1.7 /24 vcs multicast-­‐ip 224.0.0.210 vcs device 1 priority 200 interfaces management enable vcs device 2 priority 150 interfaces management enable vcs local-­‐device 1 2) – Disable VCS on the vMaster
At the Privileged EXEC level (AX#), use the following command to disable VCS.
vcs disable sh vcs summary 3) – Disable VCS on the vBlades
www.a10networks.com
6
VCS UPGRADE NOTES – 2.2
vcs disable sh vcs summary 4) - For each VRID that is active on the Ex vMaster device, failover VRID(s) from the vMaster to the vBlade using
priority.
vrrp-­‐a vrid {num | default} This command changes to the configuration level for the VRID. At this level, use the following command:
priority 255 device DeviceID or vrrp-­‐a force-­‐self-­‐standby {all-­‐partitions} NOTE: Remember if you have multiple partitions these will also need to be failed over. NOTE: If you are using Session Synchronization with Layer 4 vPorts you should also check these are in Sync before
failover. Use the ‘sh session’ command on the standby device.
5) – Upgrade the old vMaster
upgrade hd {pri | sec} [use-­‐mgmt-­‐port] url The url specifies the file transfer protocol, username and password (if required), directory path, and filename.
The use-mgmt-port option uses the ACOS device’s management port as the source interface. Otherwise, a data
interface is used. This step reboots the device. Wait until the upgrade and reboot completes successfully before
proceeding.
6) - For each VRID that is active on the vBlades, failback from the vBlades to the vMaster using priority.
sh vrrp-­‐a status
Make sure that the vMaster has formed VRRP-A adjacency before proceeding.
vrrp-­‐a vrid {num | default} This command changes to the configuration level for the VRID. At this level, use the following command:
priority 255 device DeviceID NOTE: Do not use the vrrp-a force-self-standby command.
7) – Re-enable VCS on the old vMaster
Add and check the device specific configuration for VCS on the device (see step 1) and perform a VCS Reload.
conf t vcs reload www.a10networks.com
7
VCS UPGRADE NOTES – 2.2
The device will take over as vMaster.
8) – Enable again VCS on the vBlades
Add and check the device specific configuration for VCS on the devices (see step 1) and perform a VCS Reload.
conf t vcs reload 9) - Verify VCS operation
Connect to the VCS Floating IP and verify the operational status of aVCS
sh vcs status sh vcs statistics www.a10networks.com
8
VCS UPGRADE NOTES – 2.2
About A10 Networks
A10 Networks is a leader in application networking, providing a range of high-performance application networking
solutions that help organizations ensure that their data center applications and networks remain highly available,
accelerated and secure. Founded in 2004, A10 Networks is based in San Jose, California, and serves customers
globally with offices worldwide. For more information, visit: www.a10networks.com
Corporate Headquarters
Worldwide Offices
A10 Networks, Inc
North America
Taiwan
To learn more about the A10 Thunder
3 West Plumeria Ave.
sales@a10networks.com
taiwan@a10networks.com
Application Service Gateways and how it
San Jose, CA 95134 USA
Europe
Korea
can enhance your business, contact
Tel: +1 408 325-8668
emea_sales@a10networks.com
korea@a10networks.com
A10 Networks at:
Fax: +1 408 325-8666
South America
Hong Kong
www.a10networks.com/contact
brazil@a10networks.com
HongKong@a10networks.com
or call to talk to an A10 sales
Japan
South Asia
representative.
jinfo@a10networks.com
SouthAsia@a10networks.com
China
Australia/New Zealand
china_sales@a10networks.com
anz_sales@a10networks.com
www.a10networks.com
©2014 A10 Networks, Inc. All rights reserved. A10 Networks, the A10 Networks logo, A10 Thunder, Thunder, vThunder, aCloud, ACOS, and aGalaxy are trademarks or registered
trademarks of A10 Networks, Inc. in the United States and in other countries. All other trademarks are property of their respective owners. A10 Networks assumes no responsibility
for any inaccuracies in this document. A10 Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.
www.a10networks.com
9