Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
Sangfor CCOM
vSTA Mirrored Traffic Troubleshooting
Guide
Product Version
CCOM 3.0.60
Document Version
01
Released on
Jun. 9, 2022
Version 01 (Mar.24, 2021)
Confidentiality: Public in Company
1
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
Copyright © Sangfor Technologies Inc. 2022. All rights reserved.
Unless otherwise stated or authorized, Sangfor Technologies Inc. (hereinafter referred
to as "Sangfor") and its affiliates reserve all intellectual property rights, including but
not limited to copyrights, trademarks, patents, and trade secrets, and related rights to
text, images, pictures, photographs, audio, videos, charts, colors, and layouts as
presented in or concerning this document and content therein. Without prior written
consent of Sangfor, this document and content therein must not be reproduced,
forwarded, adapted, modified or displayed or distributed by any other means for any
purpose.
Disclaimer
Products, services or features described in this document, whether wholly or in part,
may be not within your purchase scope or usage scope. The products, services or
features you purchase must be subject to the commercial contract and terms as
agreed by you and Sangfor. Unless otherwise provided in the contract, Sangfor
disclaims warranties of any kind, either express or implied, for the content of this
document.
Due to product version upgrades or other reasons, the content of this document will
be updated from time to time. Unless otherwise agreed, this document is used for
reference only, and all statements, information, and recommendations therein do not
constitute any express or implied warranties.
Version 01 (Jul.07, 2021)
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
Technical Support
For technical support, please visit: https://www.sangfor.com/en/aboutus/contact-us/technical-support
Send information about errors or any product related problem to
tech.support@sangfor.com.
Version 01 (Jul.07, 2021)
1
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
About This Document
This document describes the troubleshooting guide for vSTA mirrored traffic of
Sangfor Cyber Command(CCOM).
Intended Audience
This document is intended for:
⚫
Network design engineers
⚫
O&M personnel
Note Icons
English Icon
Description
Indicates an imminently hazardous situation which, if not avoided,
will result in death or serious injury.
Indicates a potentially hazardous situation which, if not avoided,
could result in death or serious injury.
Indicates a hazardous situation, which if not avoided, could result
in minor or moderate injury.
Indicates a hazardous situation, which if not avoided, could result
in settings failing to take effect, equipment damage, or data loss.
NOTICE addresses practices not related to personal injury.
Calls attention to important information, best practices, and tips.
NOTE addresses information not related to personal injury or
equipment damage.
Change Log
Date
Jun. 09, 2022
Change Description
This is the first release of this document.
Version 01 (Jul.07, 2021)
2
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
Contents
Technical Support ................................................................................................................... 1
Change Log .............................................................................................................................. 2
1 Common Issue ..................................................................................................................... 4
2 Use tcpdump to Capture Packet ....................................................................................... 8
Version 01 (Jul.07, 2021)
3
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
1 Common Issue
Issue 1: vSTA is unable to see any inbound traffic in the mirrored interface.
Checklist 1 :
1.
Check the mirrored traffic on the core switch.
2.
The different brand core switch has a different configuration guide. You can
confirm with the core switch vendor, or you may also confirm the mirrored
traffic are correct. The traffic must be bidirectional (inbound and outbound).
In order to verify the mirrored traffic, you may use the Wireshark tool on
your PC by connecting your PC to the mirrored port and open Wireshark to
check the mirrored traffic.
3.
To confirm the bidirectional traffic, you may check the connection on the
Wireshark. For example, normal TCP traffic must have 3 ways handshake,
SYN, SYN-ACK, and ACK. In these 3 packets, we know the SYN and ACK are
outbound packets, and SYN-ACK is inbound packets.
If the traffic you captured on Wireshark is SYN and ACK packet for one
connection only, you are required to check the configuration on the core
switch to ensure whether the bidirectional traffic has been mirrored
correctly or not.
Version 01 (Jul.07, 2021)
4
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
Checklist 2:
Check the hardware platform and the following information first:
1.
Does the ad_appd running?
ps aux |grep ad_appd
2.
Check
the
CPU
model.
Does
it
support
AVX
command?
lscpu | grep avx or you may check the details on the Intel website.
If it has the AVX instruction set, the CPU can support DPDK. If the CPU
does not support the AVX instruction set, it will cause issues for the DPDK
to receive the mirrored traffic. The server CPU used by Vmware can check
the model on the Intel official website. Below is the example of E7-4850 v1
and E7-4850 v2 versions:
Version 01 (Jul.07, 2021)
5
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
Checklist 3:
Ad_appd shows a NUMA error when running the ad_appd with
/usr/sbin/ad_appd -af as shown below:
Reason:
ad_appd huge memory configured for the process doesn’t enough
Try run usr/sbin/ad_appd -af for debugging.
How to change?
Run the command below:
cat /proc/meminfo |grep -i huge
AnonHugePages:
534528 kB
HugePages_Total:
388
HugePages_Free:
149
HugePages_Rsvd:
0
HugePages_Surp:
0
Hugepagesize:
2048 kB
The HugePages_Total need to be bigger than HugePages_Free. You may try to
use 1000, 1500, or 2000 to enlarge the hugepage memory, then command as
Version 01 (Jul.07, 2021)
6
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
below:
echo x > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
x is the example to enlarge 1000 in hugepage memory, you can run it like this:
echo 1000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
After that, restart the ad_appd by executing systemctl restart ad_appd to
ensure the ad_appd process is running well.
At Last, modify the value and force write the value into the boot-up scripts.
As the image above, edit the /usr/sbin/netconfig/hugemem.sh, change the
line 54 and add the new value in line 55, for example if enlarge 1000 can make
the ad_appd running without issue, you may insert the following line command
into line 54: echo 1000 > /sys/kernel/mm/hugepages/hugepages2048kB/nr_hugepages
For the next reboot, the vSTA will read the configuration and run it.
Version 01 (Jul.07, 2021)
7
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
2 Use tcpdump to Capture Packet
Use tcpdump to check whether the VMware configuration is correct or not.
This is because the VMware is using DPDK and does not support the tcpdump
command.
1.
Stop ad_appd:
systemctl stop ad_appd
ps aux |grep ad_appd
Ensure that all ad_appd processes are stopped.
2. Check the NIC pciid bound by dpdk and the previous driver:
/usr/local/dpdk/usr/local/share/dpdk/usertools/dpdk-devbind.py –status
/usr/local/dpdk/usr/local/share/dpdk/usertools/dpdk-devbind.py --statusdev net
3. Unbind DPDK on the mirror port that needs to be unbound, such as
unbound 0000:01:00.0:
/usr/local/dpdk/usr/local/share/dpdk/usertools/dpdk-devbind.py -u
0000:01:00.0
4. Re-bind the system network card driver, and fill in the bind parameter with
the unused parameter in the figure above:
Version 01 (Jul.07, 2021)
8
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
vsta: /usr/local/dpdk/usr/local/share/dpdk/usertools/dpdk-devbind.py -bind=vmxnet3 0000:01:00.0
5. run ifconfig -a to see the mirror port.
6. After the mirroring port is bound to the system driver, it must be manually
up the network interface by executing ifconfig eth1 up.
7. Reboot the vSTA to use the DPDK driver.
Version 01 (Jul.07, 2021)
9
Sangfor Cyber Command V3.0.60 vSTA Mirrored Traffic Troubleshooting Guide
Version 01 (Jul.07, 2021)
1