Customer Driven Innovation
Compact Training – A10 Thunder Fundamentals
José Luis Serrano
Sr. Systems Engineer, Spain & Portugal
Do not distribute/edit/copy without the
written consent of A10 Networks
1
Agenda
„
Thunder ADC Series Overview
„
Device Management
„
Basic Device Setup
„
Basic SLB Configuration
„
VRRP-A High Availability
„
aVCS Clustering
„
Troubleshooting
„
Tech Support Procedure
„
Additional Online Resources
„
Q&A
2
Gama Thunder
3
A10 Product Portfolio Overview
Product Lines
ADC
Application
Delivery Controller
TPS
CGN
Threat Protection
System
Carrier Grade
Networking
Application Networking Platform
ACOS Platform
Dedicated
Network
Managed
Hosting
§ ADC – Application Acceleration & Security
§ CGN – IPv4 Extension / IPv6 Migration
§ TPS – Network Perimeter DDoS Security
§
§
§
§
Cloud IaaS
Performance
Scalability
Extensibility
Flexibility
IT Delivery Models
4
ACOS Scalable Symmetrical Multi-Processing
Efficient &
Accurate Memory
Architecture
64-Bit Multi-Core
Optimized
Optimized
Flow Distribution
Shared Memory
Shared
MemoryArchitecture
Architecture
Application
Acceleration
1
2
3
Flexible Traffic Accelerator
N
Application
Security
Application
Availability
Switching and Routing
5
ACOS: FTA Models
Efficient &
Accurate Memory
Architecture
64-Bit Multi-Core
Optimized L4-7
Processing &
Security
Shared Memory Architecture
CPU
0
Mgmt. CPU
CPU
1
CPU
2
CPU
3
CPU
12
Compression
Data CPUs
SSL
Hardware Assisted
Flow Distribution
Broadcom ASIC Chip
for High Performance
Switching
Flexible Traffic Accelerator (FPGA Matrix)
Switching and Routing ASIC (Broadcom)
6
ACOS: Non FTA Models
Efficient &
Accurate Memory
Architecture
64-Bit Multi-Core
Optimized L4-7
Processing &
Security
Shared Memory
Architecture
CPU
0
Mgmt. CPU
CPU
1
CPU
4
High Performance Driver (HPD)
CPU
5
CPU
11
Compression
Data CPUs
SSL
Software Optimized
Flow Distribution
Flexible Traffic Acceleration by HPD (non-ASIC)
Intel 82599 Chip for
High Performance
Switching
Switching and Routing by HPD (non-ASIC)
7
Thunder ADC Hardware Appliances – Entry & Mid Range
Price
30 Gbps (L4&L7)
750k L4 CPS
3M RPS (HTTP)
14k SSL CPS 2k
30 Gbps (L4&L7)
1,5M L4 CPS
7.5M RPS (HTTP)
52k SSL CPS 2k
FTA DDoS (55M Syn/s)
42 Gbps (L4&L7)
2,5M L4 CPS
12M RPS (HTTP)
75k SSL CPS 2k
FTA DDoS (55M Syn/s)
Thunder 3430S ADC
Thunder 3230S ADC
10 Gbps (L4&L7)
450k L4 CPS
2M RPS (HTTP)
7.5k SSL CPS 2k
Thunder 3030S ADC
5 Gbps (L4&L7)
200k L4 CPS
1 M RPS (HTTP)
400 SSL CPS 2k
Thunder 1030S ADC
Thunder 930 ADC
Performance
8
Price
Thunder ADC Hardware Appliances – High End
38 Gbps (L4&L7)
2.7M L4 CPS
11M RPS (HTTP)
84k CPS SSL
FTA DDoS (112M Syn/s)
79/78 Gbps (L4/L7)
3.7M L4 CPS
20M RPS (HTTP)
110k CPS SSL
FTA DDoS (112M Syn/s)
79/78 Gbps (L4/L7)
6M L4 CPS
32.5M RPS (HTTP)
172k CPS SSL
DDoS (100M Syn/s)
150/145 Gbps (L4/L7)
5.3M L4 CPS
31M RPS (HTTP)
130k CPS SSL
FTA DDoS (212M Syn/s)
150/145 Gbps (L4/L7)
7.1M L4 CPS
38M RPS (HTTP)
173k CPS SSL
FTA DDoS (223M Syn/s)
Thunder 6630 ADC
Thunder 6430(S) ADC
Thunder 5630 ADC
Thunder 5430(S)-11 ADC
Thunder 4430(S) ADC
Performance
9
vThunder – Appliance Virtual
vThunder (Perpetual Licensing)
§
§
High-performance
8 Gbps
Precio
§
§
200 Mbps to 8 Gbps
VMware, KVM, Hyper-V & Xen
hypervisors
Dynamic provisioning, faster roll out
Scale up or down on-demand
High-performance
4 Gbps
Entry Level/Lab
1 Gbps
Entry Level/Lab
200 Mbps
Lab Edition
Rendimiento
10
Thunder 930 ADC
11
Blade Front View
• 2 x 10GE Fiber (SFP+)
• Management Interfaces:
• 1 x Console Port
• 1 x Ethernet Port
• 6 x 1GE Copper
• 2 x 1GE Fiber (SFP)
• 1 x USB Port
12
Blade Rear view
• 4 x Hot-Swap Smart
Fans
• Power Switch
• 2 x Hot-Wap PSU
• 76W Max Consumption
• 80Plus Platinum Eficiency (90%
Eficiency min)
13
A10 Feature Set
†
†
Application Delivery & Acceleration
¿
Comprehensive IPv4/IPv6 Support
¿
Advanced Layer 4/Layer 7 Server Load Balancing
¿
HTTP Acceleration & Optimization
¿
aFleX – for customizable, application-aware switching
¿
Advanced Health Monitoring
¿
Spam Filter Support
¿
FWLB, GSLB, TCS, Link Load Balancing (LLB), Diameter AAA
Load Balancing, Database Load Balancing
Security
¿
Web Application Firewall (WAF)
¿
Next-generation DDoS protection
¿
Application Access Management (AAM)
¿
DNS Application Firewall (DAF)
¿
SSL-- SSL Intercept (SI), SSL Acceleration, SSL Session ID
Reuse
¿
Connection Rate Limiting/Connection Limiting
†
High Performance, Scalable Platform
†
Management
¿
Industry-standard Command Line Interface
¿
Web-based Graphical User Interface (GUI) with Language
Localization
¿
REST-style XML API (aXAPI)
†
Networking
¿
Integrated Layer 2/Layer 3
¿
Routing – Static Routes, IS-IS (v4/v6), OSPF v2/v3, BGP4+
¿
VLAN (802.1Q), Trunking (802.1AX), LACP
¿
Access Control Lists (ACLs)
¿
IPv4-->IPv4 NAT/NAPT & IPv6-->IPv6 NAPT
†
IPv6 Migration/IPv4 Preservation
¿
Full native IPv6 management and feature support
¿
SLB-PT (Protocol Translation), SLB-64 (IPv4<->IPv6, IPv6<>IPv4)
†
Virtualization
¿
aVCS (Virtual Chassis System)
†
¿
Multi-tenancy with Application Delivery Partitions (ADPs)
¿
NVGRE
¿
VXLAN
Carrier-grade Hardware
¿
Advanced hardware architecture
¿
Smart Fans (hot swap)
¿
Hot swap Redundant Power Supplies (AC and DC)
¿
Solid-state drive (SSD)
¿
High Port Density
14
A10 Licensing
†
No extra licenses required for performance or features
†
Each A10 is offered with full scalability and benefits
15
Device Management
16
ACOS Management Access
†
CLI
¿
†
Web
¿
¿
†
HTTP (configurable ports - disabled by default)
HTTPS (configurable ports)
API
¿
†
Console (RS-232 connection / 9600, 8, N, 1)
Telnet (disabled by default)
SSHv2
aXAPI: a REST like API
User Authentication
¿
¿
¿
CLI: Login ID/Password and Enable ID/Password
Web: Admin roles (read-write / read-only)
Modes: Local (default)/RADIUS/TACACS+/LDAP
17
CLI: Privilege Levels
Official name Common Prompt name Purpose User EXEC Level user > • Monitor SLB & CGN, do backups, use simple diagnosAc uAliAes • From this level user cannot affect the funcAoning of the device or change configuraAon Privileged EXEC Level enable # • (same as user) + Manage system but not SLB or CGN configuraAon • Monitor system Privileged EXEC Level -­‐ Config Mode config (config)# • (same as enable) + Configure SLB or CGN. AcAons which could affect SLB or CGN configuraAon are also accessible only from here, like config restore • Enable-­‐level commands can be executed here by prepending them with “do” 18
CLI: Additional Prompt Indicators
†
HA/VRRP-A
¿
¿
†
aVCS
¿
¿
†
ACOS-Active-vMaster[7/1]>
ACOS-Standby-vBlade[7/2]>
Packet capture
¿
†
ACOS-Active>
ACOS-Standby>
ACOS(axdebug)#
Hostname
¿
¿
ACOS(config)#hostname MyThunder1
MyThunder1(config)#
19
CLI: Help
†
List options
¿
†
Option disambiguation
¿
†
ACOS>show health monitor ? WORD<length:1-­‐31> Name all-­‐partitions All partition configurations partition Per-­‐partition configurations | Output modifiers ACOS>show ic? icmp
Display ICMP statistics icmpv6 Display ICMPv6 statistics Tab completion
¿
ACOS>show rad<tab> ACOS>show radius-­‐server 20
CLI: Usability
†
Commands can be abbreviated
¿
#show run
°
¿
†
#show running-config
Commands are case insensitive
¿
#show run
°
¿
†
instead of:
equals:
#SHOW RUN
Defined Items are case sensitive
¿
#show slb server s1
°
¿
is not the same as:
#show slb server S1
†
Commands typed take affect immediately
†
Show commands can be run within configuration mode as well
21
CLI: Undo
†
Commands are undone by prepending ‘no’
¿
ACOS(config)#ip nat pool nat1 10.0.2.15 10.0.2.16 netmask /24 ACOS(config)#show ip nat pool Total IP NAT Pools: 1 Pool Name Start Address End Address Mask Gateway HA Group Vrid nat1 10.0.2.15 10.0.2.16 /24 0.0.0.0 0 default ¿
ACOS(config)#no ip nat pool nat1 ACOS(config)#show ip nat pool Total IP NAT Pools: 0 22
CLI: Disabling Configuration Elements
†
On configuration elements, ‘no enable’ has the same effect as
command ‘disable’
¿
ACOS# show run | sec slb server s1 10.0.2.18 ¿
ACOS(config)#slb server s1 ACOS(config-­‐real server)#no enable
¿
ACOS# show run | sec slb slb server s1 10.0.2.18 disable 23
CLI: Filtering Output (section & include)
†
ACOS supports filtering by piping output to section and include ¿
¿
section retrieves configuration elements containing regex
°
ACOS#show run | sec slb °
slb server s1 10.0.2.18 port 80 tcp slb service-­‐group http tcp member s1:80 include retrieves lines containing regex
°
ACOS#show run | inc slb °
slb server s1 10.0.2.18 slb service-­‐group http tcp 24
CLI: OR
†
To use ‘|’ symbol as OR in inc or sec, escape it with ‘\’ with no spaces
around it
°
ACOS#show run | inc tacacs\|radius °
tacacs-­‐server host 1.0.0.100 secret (encrypted_secret) port 49 timeout 12 radius-­‐server host 1.0.0.100 secret (encrypted_secret) 25
CLI: Exiting Current Level
†
Exit command takes CLI one level down
¿
†
End command exits out of config
¿
†
ACOS(config-­‐slb vserver-­‐vport)#exit ACOS(config-­‐slb vserver)#exit ACOS(config)#exit ACOS#exit ACOS> ACOS(config-­‐slb vserver-­‐vport)#end ACOS#exit ACOS> Ctrl-C is a keyboard shortcut for exit in config mode, Ctrl-Z is a
shortcut for end 26
CLI: Workflow
†
†
With CLI, build your configuration from bottom up
¿
System (IP/VLAN/…etc.)
¿
Redundancy + clustering (VRRP-A/aVCS)
¿
Servers
¿
Service Groups
¿
NAT pools
¿
Templates
¿
Virtual server
¿
Virtual server port
Then apply pre-configured elements on virtual server port (vPort)
¿
To use programming analogy, configuration elements are like functions. Those
functions have to be called from vPort before they take effect.
27
WebUI: Privilege Levels
†
Monitor Mode
¿
Equivalent to CLI User EXEC Level (user)
28
WebUI: Privilege Levels
†
Config Mode
¿
Equivalent to CLI Privileged EXEC Level - Config Mode (config)
29
CLI vs. WebUI
†
CLI benefits
¿
¿
¿
†
Structured, enhances understanding
Excellent for troubleshooting – can display multiple configuration items at the same
time
Can be very fast with some familiarity
WebUI benefits
¿
Flexible workflow
¿
Easy admin role definition
¿
Familiar interface
¿
Excellent for monitoring – graphical display
30
aXAPI Architecture †
aXAPI uses a REST like request/response model to exchange data over HTTPS
Admin Authentication
The aXAPI uses the same admin authentication resources as those
configured for CLI and GUI access. For example, if the A10 device is
configured to use RADIUS first to authenticate admins, RADIUS will be
used first when authenticating an admin for an aXAPI session.
Session ID
The first request from the third-party application sends the authentication
method along with a valid A10 admin username and password. If the
username and password are valid, the A10 device replies with a session
ID. The third-party application must present the session ID with all future
requests during that session. The session ID is valid until the third-party
application sends a session close request or the session times out.
Encoding
The aXAPI expects all data to be UTF-8 encoded, and it checks for valid
UTF-8 sequences. If an invalid sequence is found, the aXAPI assumes
that the data is ISO-8859-1 encoded and converts it to UTF-8. The aXAPI
discards data that is sent in any other format.
31
aXAPI Request Format – Header The request header is a URL in the following format:
https://<AX-IPaddr:port>/services/rest/<aXAPI Version>/?session_id=<session ID>&method=<aXAPI method name>&format=<data format>
Host name or IP address of the A10
device (IPv4 or IPv6), & the HTTPS
service port on A10 device. By
default, the port number is 443 for
HTTPS, and can be omitted.
The aXAPIversion to be
used.
The string returned by the authentication
method. For the authentication method,
omit the following parameter
(&session_id=<session id>), since you
may not have the session ID at that time.
The aXAPI method to
be invoked. The aXAPI
is organized according
to a series of methods
and their corresponding
data structure.
https://192.168.2.2/services/rest/V2/?session_id=308528f465597c7be6631533c4c315&method=system.time.get
• A properly formatted request to the aXAPI is a URI
request header and a request body.
• The request body can be a URI-based or JSON*-based
data structure.
• The request can be sent as an HTTP or HTTPS GET or
POST action.
The data format you wish to use in the aXAPI request and response. aXAPI has the
following formats:
• url: (default) url-based data for requests and XML-based data for responses
• json*: json-based data for both requests and responses
• xml: (not currently supported) XML-based data for both requests and responses
You can leave the data format field empty when using the ‘default’ data format (xml)
32
Named configuration profiles
†
Benefits of named profiles
¿
¿
¿
¿
†
Create new profile
¿
†
ACOS#write memory <new_profile> ACOS(config)#copy <existing_profile> <new_profile> See all profiles
¿
†
Maintain multiple configurations
Link startup configuration per partition to a named profile
Copy and edit profiles without disrupting normal operations
Maintain single configuration for both physical partitions
ACOS#show startup-­‐config all Link startup config to profile
¿
ACOS(config)#link startup-­‐config <profile_name> [primary|secondary] 33
ACOS System Backup & Restore
†
†
†
ACOS full system backup
¿
WebUI: Config > System > Maintenance > Backup > System ¿
CLI: ACOS(config)#backup system […] ACOS full system restore
¿
WebUI: Config > System > Maintenance > Restore > System ¿
CLI: ACOS(config)#restore […] Note: Supported upload protocols: FTP, SFTP, SCP, RCP, TFTP, and
HTTPS (via WebUI)
34
ACOS Software Location
†
ACOS software is stored on
¿
Two disk partitions: primary and secondary
°
¿
Two Compact Flash partitions: primary and secondary
°
†
Second partition is designed for easy software rollback
CF is designed for emergency recovery
Note: Each storage location has its own software and A10
configuration
35
ACOS Software Upgrade Options
†
Check the ACOS running partition
¿
¿
†
†
Upgrade A10 device’s other partition
¿
WebUI: Configuration > System > Maintenance > Upgrade ¿
CLI: ACOS(config)# upgrade […]
Copy running configuration to the other partition or link existing profile to it
¿
¿
†
WebUI: Monitor > Overview > Summary > System Information CLI: ACOS# show bootimage ACOS# write memory [primary|secondary] ACOS(config)# link startup-­‐config <profile_name> [primary|secondary] Set boot source to the other partition
¿
¿
WebUI: Configuration > System > Settings > Boot CLI: ACOS(config)# bootimage hd [primary|secondary] 36
A10 Initial Deployment & Configuration
1) Initial Configuration
•
•
•
•
Rack
Power
Cooling
Cabling
• Connect Console
• Assign
Management IP
Address
• Software Update
• Management Tasks
• Users
• Syslog
• SNMP
• VLANS
• VE Interfaces
• IP Addresses
• Routing
• Static
• Protocols
2) Application Load
Balancing
• Servers
• Server Ports
• Health Checks
• Match Application
• Service Groups
• TCP/UDP
• LB Algorithm
• Server Members
• Health Checks
• Virtual IP (VIP)
• Application Ports
• Service Groups
• NAT (Optional)
• SSL
• Templates
3) Advanced Load
Balancing
• Scripts
• Customer Health checks
• Content Inspection
• Modify traffic Content
• GSLB Configuration
• Rate Limiting
• Security Features
• HTTP Compression
• RAM Caching
• API Programming
37
ACOS Initial Configuration
†
First Step configuration
¿
Connect to the A10 console (9600 baud - 8 bits – no parity - 1 stop bit)
°
°
°
°
°
¿
Default user/password: admin/a10
Management IP address: 172.31.31.31 /24
SSH Enabled (telnet disabled)
HTTP redirected to HTTPS
All Data Ports Disabled
Configure the management interface, and it’s default Gateway
°
Finish the A10 configuration via CLI (SSH) or WebUI (HTTPS)
² Configure Production interfaces (vlan, Ethernet/ve interfaces)
² Enable production interfaces
² (optional) Configure routing (static/dynamic)
² (optional) Configure specific management rights
² Configure Servers / Service Groups / Virtual Servers / etc.
38
ACOS Initial Configuration - Example
AX# AX#conf t AX(config)#interface management AX(config-­‐if:management)#ip address 192.168.2.2 /24 AX(config-­‐if:management)#ip default-­‐gateway 192.168.2.1 AX(config-­‐if:management)#end AX#wr mem Building configuration... Write configuration to default startup-­‐config [OK] AX# 39
Sample ACOS L2/3 Configuration
vlan 11 tagged ethernet 1 router-­‐interface ve 11 vlan 12 tagged ethernet 1 router-­‐interface ve 12 interface ethernet 1 enable interface ve 11 ip address 100.0.1.11 255.255.255.0 interface ve 12 ip address 100.0.0.11 255.255.255.0 40
Server Load-Balancing Basics
41
Server Load Balancing (SLB)
†
Share load among multiple servers (load balancing)
†
Provide high availability of services
42
Server Load Balancing
†
ACOS SLB configuration has three core elements
¿
Servers
¿
Service Groups
¿
Virtual Servers (VIPs)
Service Group -­‐ Web Web DNS SMTP VIP Server Web DNS SMTP Server 43
SLB: Server
†
Minimum configuration
¿
Name
IP address (can use DNS name)
¿
Ports
¿
†
†
Server configuration
¿
WebUI: Config > SLB > Service > Server
¿
CLI: Thunder(config)# slb server <name> […]
Server status and statistics
¿
WebUI: Monitor > Service > SLB > Server
¿
CLI: Thunder# show slb server […]
Web DNS SMTP VIP Server Web DNS SMTP Server Sample Configuration
slb server S1 100.0.0.201
port 80 tcp
slb server S2 100.0.0.202
port 80 tcp
44
SLB: Service Group
†
†
Minimum configuration
¿
Name
¿
Type (TCP/UDP)
¿
LB Algorithm
¿
At least one Server/Port
Service Group -­‐ Web Service Group status and statistics
Web DNS SMTP VIP ¿
WebUI: Monitor > SLB > Service > Service Group
¿
CLI: Thunder# show slb service-group […]
Server Web DNS SMTP Server Sample Configuration
slb service-group http1 tcp
member S1:80
member S2:80
45
Load Balancing Algorithms
†
Service group – load balancing algorithms
¿
Round Robin
¿
Fastest Response Time
¿
Least Connection
¿
Least Request
¿
Service Least Connection
¿
Round Robin Strict
¿
Weighted Round Robin
¿
Stateless
¿
Weighted Least Connection
¿
And more…..
¿
Service Weighted Least
Connection
46
SLB: Virtual Server
†
†
Minimum configuration
¿
Name
¿
IP address (accessed by end users)
¿
Virtual server ports (usually)
¿
Service Groups
Virtual Server status and statistics
¿
WebUI: Monitor > SLB > Service > Virtual Server
¿
CLI: Thunder# show slb virtual-server […]
Service Group -­‐ Web Web DNS SMTP VIP Server Web DNS SMTP Server Sample Configuration
slb virtual-server "VIP1" 100.0.0.10
port 80 http
service-group http1
47
Source IP Persistence
†
When to use Source IP Persistence?
¿
Source IP persistence must be used when clients must have their future
connections/traffic terminated on the same server
Connection 1
Connection 2
48
Source IP Persistence Template
†
Create Source IP Persistence Template
¿
Name Type
°
°
°
¿
¿
¿
†
Port (persistence per VIP:Port)
Server (persistence per VIP)
Service-Group (persistence per URL or Host)
Timeout: How long inactive entries are saved (default = 5 minutes)
Don't Honor Conn Rules: Ignore connection limits defined on Servers and Server Ports
and connect new clients' connections to the Server (default = disabled)
Netmask: Granularity of Client IP address hashing (default = 255.255.255.255 for the
most granularity)
Assign the Source IP Persistence Template to the Virtual Server
Port
Sample Configuration
slb template persist source-ip srcip
49
SLB Source NAT
†
Create IP Source NAT Pool
¿
Name
°
Name of the template
°
Start IP address (can be the ACOS interface IP)
°
End IP address (can be the same as Start IP)
Note: If the "Start" and "End IP address" are the same, the ACOS will NAT with one unique IP address
and can NAT up to 64k flows
†
¿
Netmask (used by "IP Source NAT – Group" when servers are on different subnets)
¿
(optional) Gateway: Specify a gateway to use to reply to the clients' requests
¿
(optional) "HA Group": Specify the HA group to tie to the SLB source NAT pool
Assign the SLB Source NAT Pool to the Virtual Server Port
Sample Configuration
ip nat pool sNAT1 100.0.0.50 100.0.0.50 netmask /24
50
Health-checks
†
Service availability is checked using health monitors (HMs)
†
Health monitors can be applied to
¿
¿
¿
†
¿
¿
¿
Web DNS Health monitors can test server availability
¿
†
Service Group -­‐ Web Server
Server:Port
Service Group
VIP On Layer 3: ping (ICMP)
On Layer 4: TCP, UDP
On Layer 7 (application):
HTTP. HTTPS. FTP, SMTP, POP3, DNS, RADIUS, LDAP, RSTP, NTP, SIP
Via manually created scripts
SMTP Server Web DNS SMTP Server Multiple L3/L4/L7 tests can also be combined in a Boolean expression (and/or/
not)
Sample Configuration
health monitor http-hm
method http
51
Applying a Health Monitor
†
Physical server health monitor
¿
If HM fails, that server is considered down and service groups configured with that specific
server stop using it for load balancing
Note: Default Server HM type is ICMP
†
Physical server port health monitoring
¿
If HM fails, that server port is considered down and service groups configured with that
specific Server:Port stop using it for load balancing
Note: Default TCP server port HM type is TCP handshake
†
Service group health monitor
¿
If HM fails for a specific member, the service group stops using this member for load
balancing
Note: By default, no HM is configured on Service Group
52
Sample ACOS SLB Configuration
ip nat pool sNAT1 100.0.0.50 100.0.0.50 netmask /24
health monitor http-hm
method http
slb server S1 100.0.0.201
port 80 tcp
slb server S2 100.0.0.202
port 80 tcp
slb service-group http1 tcp
health-check http-hm
member S1:80
member S2:80
slb template persist source-ip srcip
slb virtual-server "VIP1" 100.0.0.10
port 80 http
service-group http1
source-nat pool sNAT1
template persist source-ip srcip
53
Topology: One-Armed L2 (Switched) Mode
VIP = 100.0.0.10 100.0.0.0/24 SNAT = 100.0.0.50 100.0.0.0/24 200.0.0.1 100.0.0.[100-­‐200] Source IP
Dest IP
Source IP
Dest IP
200.0.0.1
100.0.0.10
100.0.0.50
100.0.0.101
Source IP
Dest IP
Source IP
Dest IP
100.0.0.10
200.0.0.1
100.0.0.101
100.0.0.50
54
Topology: One-Armed L2 (Switched) Mode
VIP = 100.0.0.10 SNAT = 100.0.0.50 100.0.0.0/24 100.0.0.0/24 200.0.0.1 †
100.0.0.[100-­‐200] Benefits:
¿
¿
¿
No change required on clients
or servers
Easy to test
Clients can be in servers’
subnet
†
Points to keep in mind:
¿
¿
Servers lose Client IP visibility (can
be partly remedied by IP header
insertion in HTTP/TCP)
Requires Source NAT on SLB if the
servers don’t point to the A10 for
their default gateway.
55
Topology: L3 (Routed) Mode with SNAT
VIP = 100.0.0.10 100.0.0.0/24 SNAT = 100.0.1.50 100.0.1.0/24 200.0.0.1 100.0.1.[100-­‐200] Source IP
Dest IP
Source IP
Dest IP
200.0.0.1
100.0.0.10
100.0.1.50
100.0.1.101
Source IP
Dest IP
Source IP
Dest IP
100.0.0.10
200.0.0.1
100.0.1.101
100.0.1.50
56
Topology: L3 (Routed) Mode with SNAT
VIP = 100.0.0.10 SNAT = 100.0.1.50 100.0.0.0/24 100.0.1.0/24 200.0.0.1 †
100.0.1.[100-­‐200] Benefits:
¿
¿
No change required on clients or
servers
Easy to test
†
Points to keep in mind:
¿
¿
Servers lose Client IP visibility
(can be partly remedied by IP
header insertion in HTTP/TCP)
Requires Source NAT (SNAT) on
SLB
57
Topology: L3 (Routed) Mode without SNAT
VIP = 100.0.0.10 100.0.0.0/24 100.0.1.0/24 200.0.0.1 100.0.1.[100-­‐200] Source IP
Dest IP
Source IP
Dest IP
200.0.0.1
100.0.0.10
200.0.0.1
100.0.1.101
Source IP
Dest IP
Source IP
Dest IP
100.0.0.10
200.0.0.1
100.0.1.101
200.0.0.1
58
Topology: L3 (Routed) Mode without SNAT
VIP = 100.0.0.10 100.0.0.0/24 100.0.1.0/24 200.0.0.1 †
100.0.1.[100-­‐200] Benefits:
¿
No change required on clients
Provides additional layer of
security
†
Points to keep in mind:
¿
Configure SLB as default gateway
on servers
59
Topology: Direct Server Return (DSR) Mode
VIP = 100.0.0.10 Loopback IP = VIP = 100.0.0.10 100.0.0.0/24 100.0.0.0/24 200.0.0.1 100.0.0.[100-­‐200] Source IP
Dest IP
Source IP
Dest IP
200.0.0.1
100.0.0.10
200.0.0.1
100.0.0.10
Source IP
Dest IP
100.0.0.10
200.0.0.1
60
Topology: Direct Server Return (DSR) Mode
VIP = 100.0.0.10 Loopback IP = VIP = 100.0.0.10 100.0.0.0/24 100.0.0.0/24 200.0.0.1 †
100.0.0.[100-­‐200] Benefits:
¿
Highly scalable (SLB processes
only incoming traffic)
†
Points to keep in mind:
¿
¿
Can’t use any Layer 7 features
(aFleX can still be applied at virtual
port level)
Configure VIP IP as loopback on
servers
61
VRRP-A High-Availability
62
VRRP-A
†
VRRP-A (Virtual Router Redundancy Protocol) provides redundancy for up to
8 devices or L3V partitions
†
Assigns Virtual MAC address for each VRID
¿
VRRP-A assigns a virtual MAC address to each VRID with the format 021f.a000.nnnn. The last 2
bytes of the address (nnnn) indicate the partition ID, set-id, and VRID.
†
While server/application failure is covered by Health Monitors, VRRP-A
covers A10 device or network Element or Link failures
†
VRRP-A supports arbitrary N+M deployments where N is the number of
active, and M is the number of standby devices
†
VRRP-A was introduced in release 2.6 to replace Legacy HA
¿
Legacy HA is still supported for backwards compatibility but can't run in parallel with VRRP-A
63
VRRP-A: Selection of Active VRRP-A device
Devices boot Weights equal? No Device with highest weight is elected acAve Yes PreempAon disabled OR prioriAes equal? Device with lowest ID is elected acAve Yes No Device with highest priority is elected acAve 64
VRRP-A: Design Options
†
Active-Standby mode
¿
1 Active A10 and 2 or more Passive AXs
AX1
AX1: Standby#1 for all VIPs
AX2: Active for all VIPs
AX3: Standby#2 for all VIPs
AX2
AX3
65
VRRP-A: Design Options (cont.)
†
Active-Active mode: All AXs are active for some services (VIPs)
N+1 deployment
N+M deployment
Note: N+M deployments means M boxes standby for higher availability
AX1
AX1
AX2
AX2
AX3
AX3
AX1: Active for VIPs-Group1
AX2: Active for VIPs-Group2
AX3: Standby#1 for VIPs-Group1
Standby#1 for VIPs-Group2
AX1: Active for VIPs-Group1
Standby#1 for VIPs-Group2
Standby#1 for VIPs-Group3
AX2: Active for VIPs-Group2
Standby#1 for VIPs-Group1
Standby#1 for VIPs-Group3
AX3: Active for VIPs-Group3
Standby#1 for VIPs-Group1
Standby#1 for VIPs-Group2
66
VRRP-A: Active–Standby Mode
†
Active-Standby Mode
¿
Active A10 processes all production traffic
¿
Standby A10 does not process any production traffic
¿
Standby A10 mirrors all session information from Active AX
°
In case of "N Standby" deployments, only the primary
standby mirrors the sessions
¿
One VRID (default) is sufficient to implement Active-Standby
¿
Reliability is scaled but not performance
VIPs
Floating IP
SNAT IP
Active
Standby
VIPs
Floating IP
SNAT IP
67
VRRP-A: Active–Standby Failover
†
Active-Standby Failover
¿
¿
¿
¿
¿
Peer A10 elected as active
Gratuitous ARPs for virtual, floating and NAT IPs are
sent
Existing mirrored sessions are picked up by newly
elected active AX
New sessions are served by newly elected active AX
In case of "N Standby" deployments, the secondary
standby becomes primary standby and mirrors the active
sessions from the new Active AX
Failed
New Active
VIPs
Floating IP
SNAT IP
VIPs
Floating IP
SNAT IP
68
VRRP-A: Active–Standby Configuration
†
VRRP-A Active–Standby Mode – configuration steps
1.
Configure VRRP-A Set ID
The Set ID is a unique identifier for all participating devices. All devices must be in
the same layer 2 broadcast domain
¿ AX(config)# vrrp-a set-id 1
Note: Each VRRP-A/aVCS cluster in an L2 domain must have a unique set-id
¿
2.
Configure VRRP-A Device ID
¿
¿
3.
The Device ID is a unique device identifier within the VRRP-A set
AX(config)# vrrp-a device –id (AX1 = 1, AX2 = 2, etc)
Enable VRRP-A
¿
AX(config)# vrrp-a enable
69
VRRP-A: Active–Standby Configuration (cont.)
4.
Configure VRRP-A group options (called VRID)
¿
All functional resources not explicitly assigned to user-created VRIDs are
automatically assigned to default VRID
Default VRID number is 0. That number cannot be used to create a custom VRID
¿
Recommended settings:
¿
°
¿
Optional settings: (Recommended values in "italic“)
°
°
°
°
¿
¿
Floating IP (VRRP IP Address used as gateway by servers/routers)
Preempt ("enabled ", default = enabled)
Preempt Delay (“vrrp-a preemption-delay 60”)
Priority ("AX-Active=200 / AX-Standby=199", default = 150)
Tracking
² Gateway ("default gateway IP Address", no default)
² Interface ("production interfaces", no default)
Deployment scenarios with more than one active device require at least as many
VRIDs as active devices (including default)
AX(config)# vrrp-a vrid default
AX(config-vrid-default)# …
70
VRRP-A: Active–Standby Configuration (cont.)
5.
Configure VRRP-A settings for VIPs
¿
No Configuration is required if using VRID default
¿
Optional settings
°
°
¿
Enable HA Connection Mirroring on the VIP ports: To synchronize SLB session table (available for
TCP, UDP, RTSP, FTP, MMS and SIP VIP types)
Note: For HTTP/HTTPS VIP types, the client session is terminated on the A10 device. HA Connection
Mirroring is not available for these VIP types.
AX(config)# slb virtual-server <name>
AX(config-slb vserver)# port <#> tcp
AX(config-slb vserver-vport)# ha-conn-mirror
Optional settings – Not recommended
°
Enable Dynamic Server Weight: Reduce the A10 VRRP-A priority when a server is down
71
VRRP-A: Active–Standby Configuration Example
AX01-Active#
AX02-Standby#
slb server gw 10.0.1.1
!
vrrp-a device-id 1
vrrp-a set-id 1
vrrp-a enable
vrrp-a vrid default
priority 200
floating-ip 10.0.2.10
tracking-options
interface ethernet 1 priority-cost 2
interface ethernet 2 priority-cost 2
gateway 10.0.1.1 priority-cost 2
!
slb virtual-server vip1 10.0.1.12
port 80 tcp
...
ha-conn-mirror
port 21 ftp
...
ha-conn-mirror
slb server gw 10.0.1.1
!
vrrp-a device-id 2
vrrp-a set-id 1
vrrp-a enable
vrrp-a vrid default
priority 199
floating-ip 10.0.2.10
tracking-options
interface ethernet 1 priority-cost 2
interface ethernet 2 priority-cost 2
gateway 10.0.1.1 priority-cost 2
!
slb virtual-server vip1 10.0.1.12
port 80 tcp
...
ha-conn-mirror
port 21 ftp
...
ha-conn-mirror
72
VRRP-A: Active–Active Mode
†
Active-Active Mode
¿
¿
¿
All A10 units process the production traffic
Sessions and state information are
mirrored between Active & peer units for
each Group-ID
Performance is scaled in addition to
reliability
VIPs – Group1
Floating IP – Group1
SNAT IP – Group1
VIPs – Group1
Floating IP – Group1
SNAT IP – Group1
Active
Active
VIPs – Group2
Floating IP – Group2
SNAT IP – Group2
VIPs – Group2
Floating IP – Group2
SNAT IP – Group2
73
VRRP-A: Active–Active Failover
†
Active-Active Failover
¿
Peer A10 is elected active for VIPs-group 2 and sends
gratuitous ARPs for virtual IPs, floating IPs, and NAT
IPs
¿
Existing mirrored sessions are picked up by peer AX
¿
Peer A10 serves requests for both VIPs groups
¿
In case of "N Standby" deployments, the secondary
standby becomes primary standby and mirrors the
active sessions from the new Active AX
Failed
Active
VIPs – Group1+2
Floating IP – Group1+2
SNAT IP – Group1+2
VIPs – Group1+2
Floating IP – Group1+2
SNAT IP – Group1+2
74
VRRP-A: Active–Active Configuration
†
VRRP-A Active-Active Mode – configuration steps
1.
Configure VRRP-A
°
2.
3.
Same as Active/Standby
Configure VRRP-A group options (called VRID)
°
Same as Active/Standby (configured for each VRRP-A VRID)
°
Requires a unique VRID for each Group-ID
Configure VRRP VRID for SLB-VIPs + NAT
°
Associate the SLB-VIPs + NAT with a VRID
Note: By default the SLB-VIPs + NAT are in the default VRID
75
VRRP-A: Active–Active Configuration Example
AX01#
AX02#
slb server gw 10.0.1.1
!
vrrp-a device-id 1
vrrp-a set-id 1
vrrp-a enable
vrrp-a vrid 1
floating-ip 10.0.2.2
priority 200
tracking-options
interface ethernet 1 priority-cost 2
interface ethernet 2 priority-cost 2
gateway 10.0.1.1 priority-cost 2
vrrp-a vrid 2
floating-ip 10.0.2.3
priority 199
tracking-options
interface ethernet 1/1 priority-cost 2
interface ethernet 1/2 priority-cost 2
gateway 10.0.1.1 priority-cost 2
!
slb virtual-server vip1 10.0.1.12
vrid 1
...
slb virtual-server vip2 10.0.1.13
vrid 2
…
slb server gw 10.0.1.1
!
vrrp-a device-id 2
vrrp-a set-id 1
vrrp-a enable
vrrp-a vrid 1
floating-ip 10.0.2.2
priority 199
tracking-options
interface ethernet 1 priority-cost 2
interface ethernet 2 priority-cost 2
gateway 10.0.1.1 priority-cost 2
vrrp-a vrid 2
floating-ip 10.0.2.3
priority 200
tracking-options
interface ethernet 1/1 priority-cost 2
interface ethernet 1/2 priority-cost 2
gateway 10.0.1.1 priority-cost
!
slb virtual-server vip1 10.0.1.12
vrid 1
...
slb virtual-server vip2 10.0.1.13
vrid 2
…
76
VRRP-A: Troubleshooting
†
VRRP-A status
¿
†
VRRP-A statistics
¿
†
CLI-only: AX# show vrrp
CLI-only: AX# show vrrp detail
VRRP-A manual failover
¿
A10 failover
°
¿
CLI-only: AX(conf)# vrrp force-self-standby
A10 specific group (vrid) failover
°
CLI-only: AX(conf)# vrrp force-self-standby vrid <#>
77
Virtual Chassis System - aVCS
78
aVCS
†
aVCS (Virtual Chassis System) is a centralized configuration management layer
†
aVCS can be combined with VRRP-A or legacy HA
†
Combined with redundancy, aVCS facilitates clustering of ACOS devices
Virtual Chassis aVCS vMaster vBlade vBlade vBlade VRRP-­‐A 79
aVCS: Benefits
†
Centralized point (single IP) for management of all aVCS devices
¿
L4/7 configuration changes are automatically propagated to all devices
¿
L2/3 device configuration can be performed using device-specific ID
°
A1-­‐Active-­‐vMaster[1/1](config)#vlan 2/300 A1-­‐Active-­‐vMaster[1/1](config-­‐vlan:2/300)#tagged ethernet 3 °
†
OR A1-­‐Active-­‐vMaster[1/1](config)#router device-­‐context 2 All the following router configuration will go to device 2 A1-­‐Active-­‐vMaster[1/1](config)#router ospf 1 Adding new devices to aVCS chassis is largely automated
¿
vMaster checks and upgrades vBlade if necessary
¿
vMaster pushes configuration to vBlade
80
aVCS: Requirements
†
†
Topology
¿
aVCS uses Link Local UDP multicast for heartbeat messages
¿
Heartbeat messages are sent via multicast to all vBlades
°
Multicast IP: 224.0.0.210
°
UDP Port: 41217
¿
vMaster transfers data to vBlades (configuration, status, image files) via Unicast TCP
¿
Interfaces selected for aVCS need to be in the same layer 2 broadcast domain
Software and hardware
¿
Devices should be the same model number and hardware and capable of running the
same version of ACOS
81
aVCS: vMaster and vBlade
†
vMaster
¿
¿
†
All configuration is performed from here
vMaster uses floating IP, so admin always logs in to the same management IP -- even
after failover
vBlade
¿
Device which acts as a blade in a virtual chassis
¿
Config privilege level is disabled
¿
vBlade can become vMaster when the device hosting the latter loses connectivity or
admin forces vMaster takeover
82
aVCS: Device ID and Chassis ID
†
Device ID
¿
Unique device identifier within the virtual chassis. It can be set using vrrp-­‐a command
°
†
A1(config)#vrrp-­‐a device-­‐id 1 Chassis ID
¿
Unique chassis identifier. It can be set using vrrp-a command
°
A1(config)#vrrp-­‐a set-­‐id 1
83
aVCS: vMaster Election Process
No Yes 84
aVCS: Overriding vMaster Election
†
vMaster device priority
¿
†
vMaster affinity
¿
†
A1-­‐Active-­‐vMaster[1/1](config)#vcs device 1 A1-­‐Active-­‐vMaster[1/1](config-­‐vcs-­‐dev)#affinity-­‐vrrp-­‐a-­‐vrid <vrid> vMaster takeover
¿
†
A1-­‐Active-­‐vMaster[1/1](config)#vcs device 1 A1-­‐Active-­‐vMaster[1/1](config-­‐vcs-­‐dev)#priority ? <1-­‐255> Device priority A2-­‐Standby-­‐vBlade[1/2]#vcs vmaster-­‐take-­‐over ? <1-­‐255> vMaster take over priority Note: Default vMaster device priority is 0
85
aVCS: Configuration
†
Device 1
¿
vrrp-­‐a device-­‐id 1 vrrp-­‐a set-­‐id 1 vcs enable vcs floating-­‐ip <ip_add> /<netmask> vcs device 1 interface <interface_id> interface <interface_id> enable vcs reload †
Device n
¿
vrrp-­‐a device-­‐id n vrrp-­‐a set-­‐id 1 vcs enable vcs device n interface <interface_id> interface <interface_id> enable vcs reload
Note: You must issue a ‘vcs reload’ after each aVCS configuration change.
Note: Use ‘vcs reload disable-merge’ for un-configured device (RMA)
Note: You should configure more than one aVCS interface for redundancy
86
aVCS: Troubleshooting
†
aVCS summary and status
¿
†
A1-­‐Active-­‐vMaster[1/1]#show vcs summary aVCS statistics
¿
A1-­‐Active-­‐vMaster[1/1]#show vcs stat °
†
Check vcs running config
¿
†
(over 100 lines of output per device in the chassis)
A1-­‐Active-­‐vMaster[1/1]#show run | sec vcs Check ACOS versions
¿
A1#show bootimage ¿
A1#show version 87
aVCS: Best practices
†
Configure aVCS before VRRP-A
†
Set up redundant aVCS paths (data ports/trunk and management
port)
†
Use vcs vMaster-­‐maintenance <seconds> mode when making config
changes on production networks in order to preserve integrity of the
original configuration during maintenance
†
Set vcs failure-­‐retry-­‐count -­‐1 to prevent aVCS timeouts
†
Use staggered upgrade to install new ACOS on devices in an aVCS
chassis
88
Troubleshooting
89
A10 Troubleshooting – Bottom Up Approach
†
†
†
†
†
Basic Network – L1 / L2 / L3
¿
L1; power, cabling, system core, logs
¿
L2/3: ping / traceroute / interface status / routing
Server Status
¿
Health Checks,
¿
Server/Service-group status
TCP / UDP - L4
¿
SYN/SYN ACKs
¿
Ports Available / NAT
HTTP ( HTTPS ) - L7
¿
aFleX ?
¿
Rewrite Rules / Redirection / Response Codes
¿
Compression / Caching?
Sessions / Persistence L4/7
¿
†
Cookies / Source IP / Ports / LB Metric?
Performance
¿
CPS / L4-7 / SSL
¿
Total Connections established
90
Log
†
ACOS logs many informational, warning, and error messages.
show log is the first place to check when experiencing issues.
¿
¿
¿
¿
¿
¿
¿
†
Port/Interface up/down messages
L2 loop detection warnings
Unicast/Multicast/Broadcast packet limit warnings
MAC address movement warnings
Duplicate IP warnings
Server & service port up/down messages
Application-specific error messages: SLB, PBSLB, HTTP, HA, AFLEX, […]
Monitoring
¿
¿
WebUI: Monitor > System > Logging > Logging
CLI: ACOS# show log [ | inc <reg_ex> ] 91
Audit Log
†
ACOS logs administrative actions with username, date, and time
stamp. It also logs new administrative sessions.
¿
Example
Sep 30 2013 12:21:04 [admin] web: add Source IP Persistence template [pers1] successfully. Sep 30 2013 11:41:54 [admin] cli: vcs device-­‐context device 2 Sep 30 2013 12:29:28 A web session[1] opened, username: admin, remote host: 10.254.102.12 †
Monitoring
¿
WebUI: Monitor > System > Logging > Audit
¿
CLI: ACOS# show audit [ | inc <reg_ex> ] 92
Examining running config
†
Examine running config with the following tools
¿
ACOS# show run [ | sec ^[0-­‐z] ] ↑ the optional element at the end of this command strips blank lines from the output
¿
ACOS# show run | sec <config_element> ¿
ACOS# show run slb […] ↑ statistics for each configuration element
¿
ACOS# show ha [config] ¿
ACOS# show vrrp-­‐a [ config | detail ] ¿
ACOS# show vcs [ summary | message-­‐buffer ] 93
Correlating log to audit log
†
Use built-in include and section utilities to find corresponding lines
in log, audit log, and running config
¿
Thunder# show log :45 Warning [ACOS]:Duplicated IP 10.0.1.1 MAC 000c.
2976.5904 from Port 1 VLAN 3 detected ¿
ThunderX# show audit | inc °
Sep 24 2013 09:56:46 [admin] cli: port 80 http Sep 24 2013 09:56:28 [admin] cli: slb virtual-­‐server vip1 10.0.1.1 ¿
ThunderX(config)# show run | sec 10.0.1.1 °
ip route 0.0.0.0 /0 10.0.1.1 slb virtual-­‐server vip1 10.0.1.1 port 80 http 94
Server Health Check
†
Display health check statistics
ACOS# show health stat [long list of statistics] IP address Port Health monitor Status Cause(Up/Down) Retry PIN 10.0.2.18 default UP 11 /0 @0 0 0 /0 0 10.0.2.19 80 default UP 20 /0 @0 0 0 /0 0 10.0.2.18 80 web UP 10 /0 @0 0 0 /0 0 10.0.2.19 80 web UP 10 /0 @0 0 0 /0 0
(see CLI Reference manual for codes)
†
Show running health monitors
ACOS #show health monitor Idle = Not used by any server In use = Used by server Monitor Name Interval Retries Timeout Up-­‐Retries Method Status ping 5 3 5 1 ICMP In use web 5 3 5 1 HTTP In use 95
axdebug
†
†
axdebug
¿
Captured files are in pcap format (Wireshark / tcpdump)
¿
Able to see every detail of the packets the A10 receives & sends
axdebug is session based
¿
If one packet matches filter, dump all the following packets in the same session
Packet 1
Client:
200.0.0.1
Src: 200.0.0.1
Dst: 100.0.0.10
Src: Port 35525
Dst: Port 80
Packet 2
AX-VIP:
100.0.0.10
NAT Pool:
100.0.0.100
Src: 100.0.0.100
Dst: 100.0.0.201
Src Port: 35525
Dst Port: 80
Server:
100.0.0.201
96
axdebug filters
†
Build filters to fine tune your capture
¿
†
axdebug example
¿
¿
†
ACOS# axdebug ACOS(axdebug)# count 3000 ACOS(axdebug)# filter 1 ACOS(axdebug-­‐filter:1)# ip 1.2.3.4 /32 ACOS(axdebug-­‐filter:1)# exit ACOS(axdebug)# capture save <file_name> NOTE: (make sure to use caution when printing output to the screen on a production system. Limiting
the count number is good practice) Stop axdebug trace
¿
†
Multiple conditions within a filter are ANDed, multiple filters are ORed.
ACOS# no axdebug Export axdebug trace
¿
ACOS# export axdebug <filename> [use-­‐mgmt-­‐port] <destination> 97
Session Filtering
†
Fine-tune session monitoring by using filters
¿
†
ACOS(config)# session-­‐filter <filter_name> […] Example
ACOS(config)# session-­‐filter c1 source-­‐addr 10.0.1.161 dest-­‐addr 10.0.1.12 dest-­‐port 80 ACOS# show session filter c1 Prot Forward Source Forward Dest Reverse Source Reverse Dest Age Hash Flags Type Tcp 10.0.1.161:36690 10.0.1.12:80 10.0.2.18:80 10.0.2.16:14075 0 1 NSe1 SLB-­‐L7 Tcp 10.0.1.161:36660 10.0.1.12:80 10.0.2.18:80 10.0.2.16:14045 0 1 NSe1 SLB-­‐L7 98
Layers 1-4
†
Layer 1-2
¿
†
†
ACOS# show int […] Layer 3
¿
ACOS# show arp ¿
ACOS# show ip route ¿
ACOS# show access-­‐list ¿
ACOS# show run | sec router Layer 4
¿
ACOS# show slb l4 ¿
host# telnet <ip> <port> ¿
ACOS# axdebug 99
Layer 7: HTTP Troubleshooting
†
†
Show enabled L7 features
¿
ACOS# show run | sec slb ¿
Try without the advanced features first (compression, connection reuse, and so on)
Packet trace
¿
ACOS# axdebug °
Is server receiving the request sent by the ACOS device?
°
Any standard HTTP header missing? (host, method, … and so on)
°
Do all of the HTTP headers have desired values?
°
Response Code from server’s response?
°
Size of request / response payload?
Is it taking a long time to process the request?
°
What are the cookies?
°
100
Layer 7: HTTP (cont.)
ACOS# show slb http debug
DP0
DP1
Total
----------------------------------------------------Fwd
req fail - buff
0
0
0
Fwd req fail - rport
0
0
0
Fwd req fail - route
0
0
0
Fwd req fail - persist
0
0
0
Fwd req fail - server
0
0
0
Fwd req fail - tuple
0
0
0
L4 switching (succ)
0
0
0
L4 switching (enQ)
0
0
0
Cookie switching (succ) 0
0
0
Cookie switching (enQ)
0
0
0
aFleX switching (succ)
0
0
0
aFleX switching (enQ)
0
0
0
URL switching (succ)
0
0
0
URL switching (enQ)
0
0
0
Host switching (succ)
0
0
0
Host switching (enQ)
0
0
0
Normal LB switching
0
63
63
Normal LB switch. (succ) 0
63
63
Normal LB switch. (enQ) 0
0
0
Client RST
0
3
3
Client RST - request
0
3
3
Client RST - connecting 0
0
0
Client RST - connected
0
0
0
Client RST - response
0
0
0
Server RST
0
0
0
Request 1.0
Request 1.1
Method GET
Method HEAD
Method PUT
Method POST
Method TRACE
Method OPTIONS
Method CONNECT
Method DELETE
Method UNKNOWN
Resp 1.0
Resp 1.1
Resp content len
Resp chunk encoding
status code 1XX
status code 2XX
status code 3XX
status code 4XX
Resp <= 1K
Resp <= 2K
Resp <= 4K
Resp <= 8K
Status code 1XX
Status code 100
Status code 101
Status code 102
Status code 2XX
Status code 200
Status code 201
Status code 202
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
63
63
0
0
0
0
0
0
0
0
0
63
63
0
0
63
0
0
63
0
0
0
0
0
0
0
63
63
0
0
0
63
63
0
0
0
0
0
0
0
0
0
63
63
0
0
63
0
0
63
0
0
0
0
0
0
0
63
63
0
0
101
Layer 7: HTTP (cont.)
DP0 DP1 Total
-----------------------------------------------------------------Fwd req fail
0
0
0
Fwd req fail - buff
0
0
0
Fwd req fail - rport
0
0
0
Fwd req fail - route
0
0
0
Fwd req fail - persist
0
0
0
Fwd req fail - server
0
0
0
Fwd req fail - tuple
0
0
0
L4 switching
0
0
0
L4 switching (succ)
0
0
0
L4 switching (enQ)
0
0
0
Cookie switching
0
0
0
Cookie switching (succ)
0
0
0
Cookie switching (enQ)
0
0
0
aFleX switching
0
0
0
aFleX switching (succ)
0
0
0
aFleX switching (enQ)
0
0
0
URL switching
0
0
0
URL switching (succ)
0
0
0
URL switching (enQ)
0
0
0
Host switching
0
0
0
Host switching (succ)
0
0
0
Host switching (enQ)
0
0
0
Normal LB switching
0
34
34
Normal LB switch. (succ)
0
34
34
Normal LB switch. (enQ)
0
0
0
Connecting RST retran
0
0
0
Connecting RST ofo
0
0
0
Connecting ACK
Packets retrans
Packets ofo
Stale sess
Server re-select failed
Large cookies
Large cookie headers
Huge cookies
Huge cookie headers
Parse cookie fail
Parse set-cookie fail
Assemble cookie fail
Asm cookie header fail
Assemble set-cookie fail
Asm set-cookie hdr fail
Invalid header
Too many headers
Line too long
Header name too long
Wrong response header
Header insert
Header delete
Insert client IP
Negative request remain
Negative response remain
Retry on 503
aFleX http retry
aFleX lb reselect
aFleX lb reselect (succ)
Request 1.0
Request 1.1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
60
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
60
102
Layer 7: HTTP (cont.)
ACOS# show slb http-proxy
Total
-----------------------------------------------------------------Curr Proxy Conns
0
Total Proxy Conns
63
HTTP requests
63
HTTP requests(succ)
63
HTTP req (cache succ)
0
No proxy error
0
Client RST
3
Server RST
0
No tuple error
0
Parse req fail
0
Server selection fail
0
Fwd req fail
0
Fwd req data fail
0
Req retransmit
0
Req pkt out-of-order
0
Server reselection
0
Server premature close
0
Server conn made
63
Source NAT failure
0
Tot data before compress 0
Tot data after compress 0
Request over limit
0
Request rate over limit 0
103
Layer 7: HTTPS/SSL Troubleshooting
†
Show enabled features
¿
ACOS# show run | sec slb °
†
Show SSL stats
°
°
†
show slb ssl stat show slb ssl cert Packet trace
¿
ACOS# axdebug °
Is client able to finish SSL Handshake with VIP?
Is ACOS device able to finish SSL Handshake with server?
°
Analyze packet pcap in protocol analyzer tool.
°
Any issues pertaining to redirect?
°
†
Are client-ssl and server-ssl templates applied on vport?
Decrypted trace
¿
Are there any absolute links in Javascripts / Links / Images (http://xxx)?
104
Session details
#show session
Traffic Type
Total
------------------------------------------------TCP Established
1
TCP Half Open
10
UDP
0
Non TCP/UDP IP sessions
0
Other
0
Reverse NAT TCP
0
Reverse NAT UDP
0
Curr Free Conn
2031556
Conn Count
1387
Conn Freed
1354
TCP SYN Half Open
0
Conn SMP Alloc
0
Conn SMP Free
0
Conn SMP Aged
0
Conn Type 0 Available
3866622
Conn Type 1 Available
1933300
Conn Type 2 Available
966644
Conn Type 3 Available
483305
Forward Source: Client IP address when connecting to a VIP.
• For DNS sessions, the client’s DNS transaction ID is shown instead of a protocol port
number.
• The output for connection-reuse sessions shows 0.0.0.0 for the forward source and forward
destination addresses.
Forward Dest: VIP to which the client is connected.
Reverse Source: Real server’s IP address.
Reverse Dest: IP address to which the real server responds.
• If source NAT is used for the virtual port, this address is the source NAT address used by
A10 device when connecting to the real server.
• If source IP NAT is not used for the virtual port, this address is the client IP address.
Age: Number of seconds since the session started.
Hash: CPU ID.
Flags: This value is used by A10 Technical Support.
Prot Forward Source
Forward Dest
Reverse Source
Reverse Dest
Age Hash Flags
---------------------------------------------------------------------------------------------------------------------------------------------------------Tcp 192.168.4.1:60456
192.168.4.200:80
192.168.3.100:80
192.168.4.50:2344
0 1
NSe1
Tcp 192.168.4.1:60447
192.168.4.200:80
0.0.0.0
0.0.0.0
0 1
NSe1
…..
If 0.0.0.0 then connection has not been established yet (half-open)
Total Sessions:
11
105
ACOS Performance
†
Show memory utilization
¿
ACOS# show memory [ system ] System Memory Usage: Total(KB) Free Shared Buffers Cached Usage 16456546 8224340 0 2420 159084 49.0% ↑ Memory is pre-allocated based on system resource configuration. †
Show cpu utilization / Slb usage
¿
ACOS# show cpu [ interval […] ] ¿
ACOS# show slb performance [ interval […] ] ↑ shows utilization per cpu for the past minute. Customizable “interval” triggers continuous updates.
†
Show resource limits
¿
ACOS# show system resource-­‐usage ↑ shows minimum, maximum, default, and currently set limits for configuration items
106
ShowTech
†
ShowTech is a comprehensive collection of output from many
troubleshooting utilities
¿
†
WebUI: generate new file and save to laptop
¿
†
Monitor > System > Diagnosis > Show Techsupport
WebUI: view and save previously generated files
¿
†
When contacting A10 Tech Support you will be asked to generate one
Monitor > System > Diagnosis > ShowTech File
CLI: generate and export file to a remote server or view on the screen
¿
ACOS# show techsupport [export] [use-­‐mgmt-­‐port] [<remote_destination>] 107
The Power of Show Tech and Backup Logs
†
What is ‘show tech’ and why you need it?
¿
¿
†
Configuration, logging, crash, version, uptime, memory, and real time snapshot of
various l2-l7 statistics of the system.
Getting a diff of two show tech snapshots while a problem is happening can help
identify underlying problems in the platform.
¿
Can greatly aid in offline debugging.
¿
CLI : (ACOS# sh tech export)
¿
GUI : Monitor > System > Diagnosis > ShowTech File
Backup logs may also contain valuable information into the cause
of the problem.
¿
Could be extremely helpful in post-network outage troubleshooting.
¿
CLI : (‘ACOS# export log’)
¿
GUI : Config > System > Maintenance > Backup
108
Useful Troubleshooting Commands– from Bottom-Up
L1/System
¿
¿
¿
¿
¿
show interface eth <number>, show int stat
show stat interface eth <number>
show core, show version, show log, show tech
show cpu, show hardware, show slb performance
show mem, show mem system
L2
L7
¿
°
°
°
¿
¿
¿
HTTP
SSL/HTTPS
°
show arp, show mac, show ipv6 neigh
show switch mac (FPGA units)
°
°
L3
°
¿
¿
¿
¿
show slb switch
show ip route, sh ip fib (sh ipv6 route, sh ipv6 fib)
debug packet l3-protocol <ip/ipv6> <ip address>
capture tools (Axdebug, Debug Monitor)
L4
¿
¿
¿
¿
¿
¿
¿
¿
show session, Show session persist
show slb server, Show slb virtual, show slb service
show slb l4 <det>
show ip nat pool stat
show ip nat trans
show slb persist
debug tcp stack, show slb tcp stack
debug packet l4-protocol <tcp/udp> <port>
capture tools (Axdebug, Debug Monitor)
show slb ssl stat
show slb ssl cert
debug ssl
x.509 aFlex logging
¿
Capture tools (Axdebug, Debug Monitor)
¿
aFlex:
°
¿
show slb http debug
show slb http-proxy
debug http-proxy
°
°
°
debug aflex,
show aflex debug
show aflex <aflex name> debug
(aflex TCL logging)
Helathcheck
°
°
°
show health stat
show health monitor
debug hm
109
A10 Tech Support Resources
110
Experienced & Focused Organization
†
Qualified Support Staff
¿
†
Engineering Background with industry experience
Support Organization
¿
Japan and China Support
°
¿
Tier 2 and Tier 3 Support Engineers
°
¿
Case manager -> Support engineers
Support QA Engineers
°
°
°
¿
Local language support available in Japan and China
Patch and Maintenance testing
Recreating CFD (Customer Found Defects)
Verifying CFD fixes using customer profile
Sustaining Engineers
°
Integrated into Support
111
2014 Global Support and RMA Depots
Netherlands
San Jose, USA
§
4 Technical Support Centers
providing 24 x 7 x 365 support.
§
60 Support Resources
§
Toll free numbers and local language
support
§
China
Dubai
Tokyo,
Japan
Support Center
35 RMA depots worldwide and
growing (99% OTD)
112
Hardware RMA Centers
†
RMA Depots
¿
4 hour Advance RMA
°
¿
¿
US & Canada
Next Business Day Advance Replacements
°
US & Canada
°
Japan
°
Taiwan
°
Hong Kong
°
EU Countries
°
Australia
By Q4 2014
°
Colombia, Chile, New Zealand, China, South Korea, Singapore, Turkey, Saudi Arabia, Dubai,
Switzerland
113
A10 Support Contact Guidelines
†
If there is a network emergency or time-critical issue – Call the
A10 Networks TAC:
¿
¿
¿
†
+1-888-TACS-A10 (888-822-7210)
+1 (408) 325-8676
900 804 766 (Spain Toll Free)
If you have a critical question on “How do I….” Contact A10 TAC via:
Phone: 1-888-TACS-A10 (888-822-7210)
Email: support@a10networks.com
Support Web: http://a10networks.com/support
†
You may refer to the following document for A10 TAC procedures:
¿
<https://www.a10networks.com/resources/files/A10-BR-Support.pdf>
114
Information Gathering
†
When requesting A10 TAC assistance, be prepared to provide the
following:
¿
¿
¿
¿
¿
¿
¿
¿
¿
Product Serial number
Customer contact information
Partner/Reseller name
Description of the problem in detail
Priority level and impact of the problem
Indication of the activity that was being performed when the problem occurred
Software version
Configuration and/or network topology information
Show techsupport (output that provides the whole configuration and statistics)
°
°
WebUI: Monitor > System > Logging > Show Techsupport
CLI: AX# show techsupport
115
Additional Information required for RMA request
†
For RMA requests, include the shipping information.
¿
Company
¿
Ship-to Address
¿
City, State, ZIP code
¿
Country (if outside of US)
¿
Contact person
¿
Contact phone number
116
Online Tech Support
†
A10 TAC offers two online resources
¿
Support Web Portal: <https://www.a10networks.com/support-axseries/index.php>
¿
A10 User Community Forum: <https://www.a10networks.com/vadc/>
117
Severity Level
Priority 1:
Priority 3:
Network
Down
Performance Impact,
Installation Issue
Priority 2:
Priority 4:
Serious
Performance
Degradation
Information
request
Priority 1 and 2 issues should be reported via 1-888-TACS-A10
118
Service Level Agreement Response Time
119
Escalation Metrics
Level 1
Level 2
(after 1hr)
Level 3
(after 4hrs)
Level 4
(after 24hrs)
Priority 1,
Critical
TAC Engineer/
Manager
Director,
Technical Support
VP, Engineering/
Sales
CEO
Priority 2,
High
TAC Engineer
TAC Manager
Director,
Technical Support
VP, Engineering/
Sales
CEO
Priority 3,
Medium
TAC Engineer
TAC Engineer
TAC Engineer
TAC Manager
Flagged
Priority 4,
Low
TAC Engineer
TAC Engineer
TAC Engineer
TAC Engineer
Flagged
(after 14 days)
Escalation
Level 5
(>7days)
120
Additional Online Resources
121
Online Resources
†
A10 Support Portal:
¿
†
A10 Online Community:
¿
†
https://www.a10networks.com/resources/training.php
A10 Deployment Guides:
¿
†
https://www.a10networks.com/support-axseries/faqs/category/121/0/10/aFleX/
A10 Training Programs:
¿
†
https://www.a10networks.com/vadc/
A10 A-Flex Support:
¿
†
https://www.a10networks.com/support-axseries/index.php
http://www.a10networks.com/resources/deployment_guides.php
A10 A10 ADC Resources:
¿
http://www.a10networks.com/products/axseries_adc.php
122
Thank You
www.a10networks.com
Do not distribute/edit/copy without the written consent of A10 Networks
123
A10 Software Release Cycle
124
Software Release Types
†
Engineering Releases
¿
†
New features
Support Releases
¿
Maintenance release
125
Engineering Release Type
†
Architecture Release : 1.x , 2.x, 3.x
¿
¿
†
Major Release: x.1.y to x.2.y
¿
¿
†
Architectural Change
Scheduled on average around 2 years
Release for customer features and internal enhancements.
Scheduled on average between 12-14 months.
Minor Release: x.y.1 to x.y.2
¿
¿
Periodic bug fix release, with some minor feature enhancement.
Schedule on average every 4 – 6 months.
126
Support Release Type
†
Patch Release: x.y.z-P1
¿
¿
¿
¿
¿
†
Release for bug fixes (mostly customer reported bugs)
May include minor changes for supportability and reliability
Scheduled between 2 – 6 Months
Include previous patch releases’ bug fixes
Signed off by Sustaining, QA, and Support
Special Patch Release: x.y.z.-P1-SP1
¿
¿
¿
¿
Emergency release targeted for a specific customer.
Full automation regression testing 2-3 days
Limited manual functional testing
Signed off by Engineering, Sustaining, QA, and Support
127
Software Release Type
†
Gold Releases
¿
2.6.1-GR1 as the First SLB Gold Release
°
¿
Released in February, 2012
2.6.6-GR1 as the First LSN/IPv6 Gold Release
°
To be released in 2Q2013
¿
Supported for a minimum of 4 years
¿
Additional QA resources for extended test cases
¿
Thorough code reviews on all code check-in
¿
Based on proven released branch with field exposure
¿
No major enhancement added
°
Supportability, Compliance, and MIB changes may be added
128
Software Release Branch Diagram
2.6.6 p2
2.6.6-GR1
LSN 2.6.6
2.6.1 GA
2.6.1-GR1
2.6.1 p4
2.6.1
Main Trunk
2.7.0
2.4.3
....
2.4.3-p14
129