Document Code Product Name 5G RAN Intended Audience INTERNAL Version 5G RAN V100R002 Prepared By 5G&SRAN I&V Dept, WN Document Version V1.0.0 5G Troubleshooting Guide – NSA Access Prepared By Xue Ruiqi and Cui Qingxia Date 2018-9 Reviewed By Qi Haofeng, Qian Jin, and Qin Yan Date 2018-9 Approved By Qi Haofeng Date 2018-9 Huawei Technologies Co., Ltd. All Rights Reserved. 5G Troubleshooting Guide – NSA Access INTERNAL Change History Date Version 2019-04 2019-06 2019-10 2023-05-02 Description Author V1.0 Completed the draft. Xue Ruiqi and Cui Qingxia V1.1 Added the contention Msg4 analysis method. Cui Qingxia V1.2 Section Added the VIP complaint handling method. Added the analysis guidance when a UE fails to measure cells and synchronization fails due to RF causes. Cui Qingxia Huawei confidential. No spreading without permission. ii 5G Troubleshooting Guide – NSA Access INTERNAL About This Document Overview This document describes the user access process and common fault demarcation methods, helping users to preliminarily isolate and troubleshoot user access problems. Precautions When using this guide to quickly troubleshoot problems, some user data such as Internet Protocol (IP) addresses may be used. You are obligated to take considerable measures, in compliance with the laws of the countries concerned and the user privacy policies of your company, to ensure that the personal data of users is fully protected. You are advised to abide by local laws and regulations to execute the task and take measures to fully protect personal data. For example, delete the folder for saving tracing records after the processing. Intended Audience This document is intended for: Maintenance engineers Onsite engineers Symbol Conventions The symbols that may be found in this document are defined in the following table. Symbol Description Indicates an imminently hazardous situation which, if not avoided, will result in death or serious injury. Indicates a potentially hazardous situation which, if not avoided, could result in death or serious injury. 2023-05-02 Huawei confidential. No spreading without permission. iii 5G Troubleshooting Guide – NSA Access Symbol INTERNAL Description Indicates a potentially hazardous situation which, if not avoided, may result in minor or moderate injury. Indicates a potentially hazardous situation which, if not avoided, could result in equipment damage, data loss, performance deterioration, or unanticipated results. NOTICE is used to address practices not related to personal injury. Calls attention to important information, best practices and tips. NOTE is used to address information not related to personal injury, equipment damage, and environment deterioration. 2023-05-02 Huawei confidential. No spreading without permission. iv 5G Troubleshooting Guide – NSA Access INTERNAL Contents Change History ................................................................................................................................ ii About This Document ................................................................................................................... iii 1 Basic Principles .............................................................................................................................. 1 1.1 Basic Concept ............................................................................................................................................................... 1 1.1.1 Networking Modes .................................................................................................................................................... 1 1.1.2 Network Topology ..................................................................................................................................................... 2 1.2 Interaction Between NEs .............................................................................................................................................. 3 1.2.1 Interaction Process Between NEs .............................................................................................................................. 3 1.3 NSA-related Terms ....................................................................................................................................................... 4 2 Basic Check..................................................................................................................................... 5 2.1 Checking Version Mapping........................................................................................................................................... 5 2.2 Checking Operations, Alarms, Faults, and External Events ......................................................................................... 5 2.3 Checking Parameters .................................................................................................................................................... 8 2.3.1 Key Configurations Related to NSA Access .............................................................................................................. 8 2.3.2 Automatic Check of Key Configurations Related to NSA Access ............................................................................. 9 2.3.3 Checking Basic UE Configurations ......................................................................................................................... 10 2.4 Checking SIM Card Registration ................................................................................................................................ 10 2.5 Checking RF Channels ............................................................................................................................................... 10 3 Quick Troubleshooting of NSA Access Problems ............................................................... 12 3.1 Overall Instruction Map .............................................................................................................................................. 12 3.2 Quick Troubleshooting of NSA Access Problems ...................................................................................................... 12 3.3 Common Access Logs ................................................................................................................................................ 13 4 NSA Access Problem Locating Guide ..................................................................................... 14 4.1 Procedures on the LTE Side ........................................................................................................................................ 14 4.1.1 Access Failure on the LTE Side ............................................................................................................................... 14 4.1.2 NR Measurement Is Not Delivered .......................................................................................................................... 15 4.1.3 NR Cells Cannot Be Measured ................................................................................................................................ 21 4.1.4 B1 Measurement Result Is Reported But No SgNB Addition Procedure Is Initiated .............................................. 28 4.2 Access Preparation Process ......................................................................................................................................... 31 4.2.1 The gNodeB Does Not Respond with SGNB_ADD_REQ_ACK ........................................................................... 31 4.2.2 The gNodeB Returns SGNB_ADD_REQ_REJ ....................................................................................................... 33 2023-05-02 Huawei confidential. No spreading without permission. v 5G Troubleshooting Guide – NSA Access INTERNAL 4.2.3 The eNodeB Does Not Return an SGNB_RECFG_CMP Message. ........................................................................ 36 4.3 NR Air Interface Access Failure ................................................................................................................................. 41 4.3.1 The UE Does Not Initiate Access After Receiving a Reconfiguration Message ...................................................... 41 4.3.2 The RAR Message over the Air Interface Times Out .............................................................................................. 43 4.3.3 Msg3 Failure ............................................................................................................................................................ 53 4.3.4 Contention Msg4 Failure ......................................................................................................................................... 58 5 Methods of Analyzing KPI-related Problems in Traffic Statistics (NSA) ....................... 64 5.1 Access KPI Definition ................................................................................................................................................ 64 5.2 Signaling Procedures and Measurement Points .......................................................................................................... 65 5.3 Method of Analyzing Access KPI-related Problems ................................................................................................... 67 5.3.1 Action 1: Determining the Problem Type ................................................................................................................ 67 5.3.2 Action 2: Time Trend Analysis ................................................................................................................................ 67 5.3.3 Action 3: Traffic Statistics Cause Analysis .............................................................................................................. 67 5.3.4 Action 4: Top N Analysis ......................................................................................................................................... 68 5.3.5 Action 5: Associated KPI Analysis .......................................................................................................................... 69 5.3.6 Action 6: Operations and External Events ............................................................................................................... 72 5.3.7 Action 7: Detailed Analysis of Top Sites or Typical Sites........................................................................................ 72 6 VIP Complaint Handling .......................................................................................................... 78 6.1 Overview .................................................................................................................................................................... 78 6.2 Complaining User Information Collection ................................................................................................................. 79 6.3 User Complaint Analysis Methods ............................................................................................................................. 79 6.4 Complaints Analysis ................................................................................................................................................... 80 2023-05-02 Huawei confidential. No spreading without permission. vi 5G Troubleshooting Guide – NSA Access INTERNAL 1 Basic Principles 1.1 Basic Concept NSA introduces 5G access into existing LTE networks where E-UTRAN and EPC serve as the anchors for mobility management and network coverage. For details about the evolution paths of the 5G architecture, see 3GPP TR 38.801. 1.1.1 Networking Modes The current version supports two network architectures: SCG split bearer (option 3x) and MCG split bearer (option 3). Option 3 (MCG split bearer) The user-plane data received from the core network is transmitted to the PDCP layer of the eNodeB. Then, the PDCP layer of the eNodeB distributes the data to the RLC layer of the gNodeB through the X2 interface. UE eNB_PDCP MME/S-GW gNB RRC CONNECTION S1-C S1-U X2-C X2-U UE 2023-05-02 eNB_PDCP MME/S-GW gNB Option 3x (SCG split bearer) Huawei confidential. No spreading without permission. 1 5G Troubleshooting Guide – NSA Access INTERNAL The user-plane data is first transmitted from the core network to the PDCP layer of the gNodeB. Then, the PDCP layer of the gNodeB distributes the data to the RLC layer of the eNodeB through the X2 interface. UE eNB MME/S-GW gNB_PDCP RRC CONNECTION S1-C X2-C X2-U S1-U UE eNB MME/S-GW gNB_PDCP 1.1.2 Network Topology The following figure shows the protocol architecture of NSA networking. Lab NSA E2E networking: 2023-05-02 Huawei confidential. No spreading without permission. 2 5G Troubleshooting Guide – NSA Access INTERNAL Maintenance PC Shielding cabinet A7 rack Switch O&M switch Serial port server Serial cable Core network Service switch Feeders CAT 6 cable Feeders Maintenance cable X2 interface 1.2 Interaction Between NEs This section describes the interaction between the UE and the eNodeB/gNodeB. It helps you to understand how a UE accesses the NR network from the LTE network. 1.2.1 Interaction Process Between NEs Figure 1-1 Interaction between NEs in Option 3 (MCG split bearer) and Option 3x (SCG split bearer) 2023-05-02 Huawei confidential. No spreading without permission. 3 5G Troubleshooting Guide – NSA Access INTERNAL 1.3 NSA-related Terms 2023-05-02 Acronym and Abbreviation Full Name NSA Non-Stand Alone NR New Radio LTE Long Term Evolution DC Dual Connectivity EN-DC LTE-NR DC eNodeB Evolved NodeB gNodeB gNodeB (supporting NR and connectivity to NGC) MeNB Master eNodeB SgNB Secondary gNodeB MN Master Node SN Secondary Node CA Carrier Aggregation MCG Master Cell Group SCG Secondary Cell Group PCell Primary Cell SCell Secondary Cell PSCell Primary SCell PCC Primary Component Carrier SCC Secondary Component Carrier PDCP Packet Data Convergence Protocol LNR Long Term Evolution and New Radio Huawei confidential. No spreading without permission. 4 5G Troubleshooting Guide – NSA Access INTERNAL 2 Basic Check 2.1 Checking Version Mapping Ensure that the versions of the NR base station, LTE base station, TUE, CPE, U2020, and CN NEs are recommended ones and are compatible. For details about the version mapping, see 5G 18B/19A/19B recommended version policies. 2.2 Checking Operations, Alarms, Faults, and External Events The operation, alarm, and fault logs of the base station can be obtained from the U2020 and one-click logs and can be opened using the FMA. Operation logs are used to check for operations affecting UE access and whether the problem occurrence time is related to the operation time. For alarms and faults, check whether there are uncleared alarms at the time when the fault occurs. In addition, check whether there are important events based on the time when the problem occurs, and then perform event association analysis. The following table lists the alarms and events that need to be focused on. Alarm ID Alarm Name Alarm Impact ALM-25954 User Plane Fault The X2-U or S1-U link fault causes access failure. ALM-25952 User Plane Path Fault The automatically established X2-U or S1-U link fault causes access failure. ALM-29841 NR Cell Unavailable The NR cell where this alarm is reported cannot provide services. ALM-29840 gNodeB Out of Service All cells where this alarm is reported cannot provide services. ALM-29842 NR Cell Blocked The NR cell where this alarm is reported cannot provide services. ALM-29800 gNodeB X2 Interface Fault The base station cannot continue to support the EN-DC services with the corresponding base stations. ALM-25888 SCTP Link Fault The SCTP link cannot process signaling messages. 2023-05-02 Huawei confidential. No spreading without permission. 5 5G Troubleshooting Guide – NSA Access INTERNAL Alarm ID Alarm Name Alarm Impact ALM-25889 SCTP Link Congestion The services are disrupted because the data cannot be transmitted due to insufficient space of the sending buffer. ALM-25955 SCTP Link Unreachable If the two configured destination IP addresses of the SCTP link are both unreachable, the upper-layer services will be interrupted. ALM-25954 User Plane Fault The user plane cannot carry services. ALM-25886 IP Path Fault Services carried on the IP Path cannot be normally processed. ALM-25952 User Plane Path Fault User plane services cannot be processed. ALM-29204 X2 Interface Fault When the neighboring base station is gNodeB, the gNodeB cannot continue to support the EN-DC services with the corresponding base station. ALM-29225 Maximum Number of X2 Interfaces Reached The EN-DC service cannot be initiated with the corresponding gNodeB. IP Address Choose 5G FMA > Tools > Brdlog Unpack Tool to decompress the one-click logs of the main control board to obtain the alarm logs and fault logs of the base stations. Then, import the logs into the FMA for quick analysis and troubleshooting. Figure 2-1 Operation alarms and fault logs in one-click logs 2023-05-02 Huawei confidential. No spreading without permission. 6 5G Troubleshooting Guide – NSA Access INTERNAL Figure 2-2 Operation logs parsed by the FMA Figure 2-3 Fault logs parsed by the FMA Figure 2-4 Alarm logs parsed by the FMA 2023-05-02 Huawei confidential. No spreading without permission. 7 5G Troubleshooting Guide – NSA Access INTERNAL 2.3 Checking Parameters Parameter check is one of the basic actions for network performance optimization. In the current phase, many access problems are caused by configuration problems. Therefore, checking parameters is the most convenient and efficient way to troubleshoot access problems. 2.3.1 Key Configurations Related to NSA Access Base Station Key Configuration Item MML eNodeB Adding a neighboring NR frequency ADD NRNFREQ: LocalCellId=21, DlArfcn=636666, UlArfcnConfigInd=NOT_CFG, SsbOffset=0, SsbPeriod=20MS, SubcarrierSpacing=30KHZ; Configuring the frequency band for a neighboring NR frequency whose NR-ARFCN falls within the following ranges: 158200-160600, 285400-303400, 386000398000, 402000-404000, 422000-434000, 514000537999, 620000-653333, or 2054166-2084999 Setting the frequency band to which the neighboring NR frequency belongs ADD NRMFBIFREQ: FrequencyBand=n78; Adding an external NR cell ADD NREXTERNALCELL: Mcc="262", Mnc="01", GnodebId=255, CellId=1, DlArfcn=636666, UlArfcnConfigInd=NOT_CFG, PhyCellId=1, Tac=1, NrNetworkingOption=NSA; Adding a neighbor relationship with the NR cell ADD NRNRELATIONSHIP: LocalCellId=21, Mcc="262", Mnc="01", GnodebId=1, CellId=7; Adding a candidate PCC ADD PCCFREQCFG: PccDlEarfcn=1500; Adding a candidate SCC to an NR SCG ADD NRSCGFREQCONFIG: PccDlEarfcn=1500, ScgDlArfcn=636666, ScgDlArfcnPriority=6, NsaDcB1ThldRsrp=-105, NrB1TimeToTrigger=40MS; Modifying NSA DC management configuration MOD NSADCMGMTCONFIG: LocalCellId=21, NsaDcAlgoSwitch=NSA_DC_CAPABILITY_SWITCH-1; MOD CELLQCIPARA: LocalCellId=21, Qci=9, NsaDcDefaultBearerMode=SCG_SPLIT_BEARER; MOD QCIPARA: Qci=9, NsaDcRlcPdcpParamGroupId=223; MOD RLCPDCPPARAGROUP: RlcPdcpParaGroupId=223, RlcMode=RlcMode_AM, AmPdcpSnSize=AmPdcpSnsize_18bits; Modifying the downlink data split mode for NR cells MOD GNBPDCPPARAMGROUP: PdcpParamGroupId=1, DlDataPdcpSplitMode=SCG_ONLY; gNodeB 2023-05-02 Huawei confidential. No spreading without permission. DlArfcn=636666, 8 5G Troubleshooting Guide – NSA Access Base Station INTERNAL Key Configuration Item MML Modifying the settings of Uplink Data Split Primary Path and Uplink Data Split Threshold MOD GNBPDCPPARAMGROUP: PdcpParamGroupId=1, UlDataSplitPrimaryPath=SCG, UlDataSplitThreshold=INFINITY, DlPdcpSnSize=BITS18; Setting NR cell algorithm switches MOD NRCELLALGOSWITCH: NsaDcSwitch=ON; Modifying the bearer configuration of an NR cell with RLC Mode set to AM and AmPdcpParamGroupId set to the same value as PdcpParamGroupId MOD NRCELLQCIBEARER: NrCellId=7, AmPdcpParamGroupId=1, RlcMode=AM; NrCellId=7, Qci=9, 2.3.2 Automatic Check of Key Configurations Related to NSA Access Use the NSA Parameter Check tool (FMA > Tools > Parameter Check) to automatically check NSA parameters. http://3ms.huawei.com/hi/group/3402967/thread_6894907.html?mapId=8584899&for_statisti c_from=all_group_forum&l=en (FMA download hyperlink) 1. Check NSA DC configurations, including whether the external NR cells, frequencies, and neighbor relationships are correctly configured, and whether the NSA DC switch is turned on. 2. Check whether the X2 link configuration is correct and whether the number of X2 links has reached the maximum. 3. Check whether PCI conflict exists in an LTE cell and its neighboring NR cells and whether PCI conflict exists in cells served by the same NR base station. 4. Check whether the SpecUeIdentifySwitch is turned on and check PDCP parameter groups. 5. Check GNBIDLENGTH consistency. 6. Check the NR architecture option setting. 7. Check 19BC10 NRMFBIFREQ. 8. Check frame offset settings in the 2.6 GHz band. 2023-05-02 Huawei confidential. No spreading without permission. 9 5G Troubleshooting Guide – NSA Access INTERNAL Figure 2-5 Parameter check in the FMA tool 2.3.3 Checking Basic UE Configurations Currently, no commercial 5G UEs are available. 5G UEs are basically Huawei TUEs and CPEs. Check the basic configurations by referring to the TUE Product White Paper or CPE Commissioning Guide. The reference link is as follows: http://3ms.huawei.com/hi/group/3402967/thread_7528635.html?mapId=9300715&for_statisti c_from=all_group_forum&l=en 2.4 Checking SIM Card Registration Confirm with CN engineers to ensure that the SIM card is registered properly, can normally access the LTE and 5G networks, and will not be rejected by the CN. 2.5 Checking RF Channels When interference signals exist in a cell, the uplink and downlink services of the cell are affected. In severe cases, UEs cannot access the network from the cell. In this case, you need to analyze the interference. You can choose interference tracing by choosing Tracing Monitor > NR > Cell Performance Monitoring on the U2020. For details about how to determine the interference, see the Problem Location Guide-Channel and Interference. 2023-05-02 Huawei confidential. No spreading without permission. 10 5G Troubleshooting Guide – NSA Access INTERNAL Figure 2-6 U2020 interference detection tracing 2023-05-02 Huawei confidential. No spreading without permission. 11 5G Troubleshooting Guide – NSA Access 3 INTERNAL Quick Troubleshooting of NSA Access Problems 3.1 Overall Instruction Map 3.2 Quick Troubleshooting of NSA Access Problems The checklists in the following attachment provide the quick troubleshooting methods: 5G Access Fault Quick Isolation Guide_2.1_0416.xlsx 2023-05-02 Huawei confidential. No spreading without permission. 12 5G Troubleshooting Guide – NSA Access INTERNAL 3.3 Common Access Logs All access logs of the 5G network are mainly based on X2 signaling. The records of the received X2 messages in the debug log are as follows: CU_UEM_LOG_CODE(0x00AE) "[FILE]cuuem_x2_agent.cpp [FUNCTION]CuUemX2Agent::OnMsg X2agent receive X2 msg.msg_type=17, x2_link_id=0" Common messages over the X2 interface: msg_type=14: msg_type=15: msg_type=16: msg_type=17: msg_type=18: msg_type=19: msg_type=20: msg_type=21: msg_type=22: msg_type=23: msg_type=24: msg_type=25: msg_type=26: msg_type=27: msg_type=28: msg_type=29: msg_type=30: msg_type=31: msg_type=32: msg_type=33: msg_type=34: msg_type=35: msg_type=36: msg_type=37: msg_type=38: msg_type=39: msg_type=40: msg_type=41: 2023-05-02 X2AP_SGNB_ADD_REQ X2AP_SGNB_ADD_ACK X2AP_SGNB_ADD_REJ X2AP_SGNB_RECFG_CMP X2AP_SGNB_MOD_REQ X2AP_SGNB_MOD_ACK X2AP_SGNB_MOD_REJ X2AP_SGNB_MOD_REQUIR X2AP_SGNB_MOD_CONF X2AP_SGNB_MOD_REFUSE X2AP_SGNB_CHANGE_REQUIR X2AP_SGNB_CHANGE_CONF X2AP_SGNB_CHANGE_REFUSE X2AP_SGNB_REL_REQ X2AP_SGNB_REL_REQ_ACK X2AP_SGNB_REL_REQ_REJ X2AP_SGNB_REL_REQUIR X2AP_SGNB_REL_CONF X2AP_SGNB_COUNTER_CHK_REQ X2AP_RRC_TRANSFER X2AP_HANDOVER_REQ X2AP_HANDOVER_REQ_ACK X2AP_HANDOVER_PREPARATION_FAIL X2AP_SN_STATUS_TRANSFER X2AP_UE_CONTEXT_REL X2AP_HANDOVER_CANCEL X2AP_LOAD_INFORMATION X2AP_ERR_IND Huawei confidential. No spreading without permission. 13 5G Troubleshooting Guide – NSA Access 4 INTERNAL NSA Access Problem Locating Guide 4.1 Procedures on the LTE Side 4.1.1 Access Failure on the LTE Side 4.1.1.1 Basic Concepts In NSA networking, UEs need to access the LTE network first before establishing connection to NR. When 5G signals are unavailable, the problem may not be caused by the 5G network. Instead, UEs may fail to access the LTE network. 4.1.1.2 Identification Method A UE fails to access the LTE network in either of the following scenarios: 1. The UE does not initiate an access procedure on the LTE network. The L3 Message tracing results show that the UE does not send any access message. 2. The UE initiates an attach procedure on the LTE network and is rejected by the CN. The L3 Message tracing results show that the UE receives an Attach Reject message after initiating a connection request to the LTE network. 4.1.1.3 Problem Location 1. UEs do not initiate access to the LTE network. The common problems are as follows: 2023-05-02 Huawei confidential. No spreading without permission. 14 5G Troubleshooting Guide – NSA Access INTERNAL The CPE does not initiate access to the LTE network because the LTE device-pipe synergy switch is turned off. This parameter can be directly checked by using the NSA Parameter Check tool in the parameter check phase. The LTE cell status configuration is abnormal. (Analyze the value of the Cell Bar IE in the SIB message.) Problem Identification Method Rectification Method The SpecUeIdentifySwitch option of the UECOOPERATIONPARA.SpecUserCooperationSwitch parameter is deselected for an LTE cell, and the CPE does not initiate access in the cell. Run the following command to select the SpecUeIdentifySwitch option: MOD UECOOPERATIONPARA: LocalCellId=X, SpecUserCooperationSwitch=SpecUeIdentifySwitch-1; The cellBarred:notBarred (1) and cellReservedForOperatorUse:reserved (0) are carried in the SIB1. According to 3GPP TS 36.304, a UE reads the AC information in the SIM card to determine whether to access the network. If the AC is not 11 or 15, the UE determines that the cell is in the barred state and does not initiate network access. Run the following command to modify the configuration on the LTE side: MOD CELLOP: LocalCellId=XX, TrackingAreaId=XX, CellReservedForOp=CELL_NOT_RESERVED_FOR_OP; 2. The UE initiates an attach procedure on the LTE network and is rejected by the CN. The attach problem is irrelevant to the radio access network. Contact CN engineers and terminal engineers to locate the problem. For details about other LTE access failures, see X Solutions for Access and Paging Performance Optimization in the LTE Network. The link is as follows: http://3ms.huawei.com/hi/group/9261/file_12805859.html?for_statistic_from=my_group_file &l=en 4.1.1.4 Cases N/A 4.1.2 NR Measurement Is Not Delivered 4.1.2.1 Basic Concept The NR measurement message is carried in the RRC reconfiguration message over the air interface. An NR measurement control message contains the following IEs (key information includes the NR frequency, bandwidth, and B1 RSRP threshold): 2023-05-02 Huawei confidential. No spreading without permission. 15 5G Troubleshooting Guide – NSA Access INTERNAL 4.1.2.2 Identification Method Check the Event List in the UE's Probe signaling to determine whether the eNodeB delivers a B1 measurement control message. Alternatively, check whether a B1 measurement control message is sent over the Uu interface by referring to section 4.1.2.1. 4.1.2.3 Problem Location 4.1.2.3.1 Signaling & Configuration Analysis After a UE accesses the LTE network, the LTE network determines whether to deliver the NSA B1 measurement to the UE based on the following conditions: 2023-05-02 Huawei confidential. No spreading without permission. 16 5G Troubleshooting Guide – NSA Access INTERNAL The UE capabilities, reported through the RRC_UE_CAP_INFO message, contain endc-r15. That is, the ueCapacityRAT-Container IE with rat-Type set to eutra carries the irat-ParametersNR-r15 field, which contains en-DC-r15:supported. The NSA capability of the UE is not restricted on the core network. The core network sends an S1AP_INITIAL_CONTEXT_SETUP_REQ message containing the nRestriction IE to restrict NSA services. UEs do not use the QCIs specific to LTE services. Specific QCIs vary with versions. For details, see NSA Networking based on EPC Feature Parameter Description of the corresponding version. 2023-05-02 Huawei confidential. No spreading without permission. 17 5G Troubleshooting Guide – NSA Access INTERNAL Check the QCI of the established bearer in the S1AP_INITIAL_CONTEXT_SETUP_REQ message. The NSA DC switch and neighboring NR frequencies are correctly configured on the LTE side. Refer to the forward check -> parameter check. You can use the NSA Parameter Check tool to perform an automatic check. The following provides the results obtained by an automatic check of the NSA Parameter Check tool. The tool compares the LTE's neighboring NR cell configuration with the actual NR configuration to check whether the configuration is correct, whether the frequency configuration is correct, and whether the NSA DC switch is turned on. When a configuration exception occurs, an error message is displayed. LTE cells support NSA. Some LTE boards do not support NSA. The following table lists the hardware support for Option 3 and Option 3x. In actual projects, there may be special specifications archived for tier-1 operators. If hardware that does not meet requirements is used, contact the corresponding DPM or MO for a desired customer to obtain the hardware requirements of the corresponding network, and then assess the hardware requirements. 2023-05-02 Huawei confidential. No spreading without permission. 18 5G Troubleshooting Guide – NSA Access INTERNAL In version 19B, UE MR-DC capability supports the combination of candidate PCC and NR SCG frequency. If the UE capabilities carry the MR-DC capability, the base station determines whether the LTE-NR band combinations in the MR-DC capability include the combination of the candidate PCC and NR SCG frequency. As shown in the following figure, the MR-DC capability can be viewed from the UE capabilities. A UE may support multiple LTE-NR band combinations. If the B1 measurement control message is not delivered, check whether the supported BandCombinations contain the candidate PCC and NR SCG frequency configured on the LTE side. 2023-05-02 Huawei confidential. No spreading without permission. 19 5G Troubleshooting Guide – NSA Access INTERNAL For details about common problems and troubleshooting methods, see the access fault tree. CHR-based Problem Location As shown in the following figure, if the LTE CHRs in version 19B contain the CHRs circled in blue, NR measurement is not delivered. Select NSADC_EVENT in the Type column. The event is printed once an hour. Based on the test time, find the event in the corresponding time. For example, if the test time is 16:00, check the event occurred at 16:34:55 (the time segment is 15:34:55–16:34:55). 2023-05-02 Huawei confidential. No spreading without permission. 20 5G Troubleshooting Guide – NSA Access INTERNAL 4.1.2.4 Cases For details, see 4.1.2.3 "Problem Location." 4.1.3 NR Cells Cannot Be Measured 4.1.3.1 Basic Concept The air interface message carrying the NR measurement report is an RRC measurement report. The key IEs in the NR measurement report are as follows (the NR measurement result carries the r15 feature field): 4.1.3.2 Identification Method CHR-based Identification As shown in the following figure, the LTE CHRs in version 19B contain the CHRs circled in blue. Based on the circled CHRs, you can determine whether NR cells can be measured. Select NSADC_EVENT in the Type column. The event is printed once an hour. Based on the test time, find the event in the corresponding time. For example, if the test time is 16:00, check the event occurred at 16:34:55 (the time segment is 15:34:55–16:34:55). 2023-05-02 Huawei confidential. No spreading without permission. 21 5G Troubleshooting Guide – NSA Access INTERNAL 4.1.3.3 Problem Location Possible Cause After the eNodeB delivers a B1 measurement configuration, the UE sends a measurement report if the threshold conditions are met. If the UE does not send a B1 measurement report within 3s, the eNodeB deletes the B1 measurement configuration. If the B1 measurement report is not sent to the eNodeB, the possible causes are as follows. NE Check Item Troubleshooting Method Recovery Measure LTE Incorrect NR frequency configurations Check whether the B1 threshold is too high. Check whether ScgDlArfcn is consistent with the gNodeB SSB frequency. Correct the NR frequency configurations. Decrease NsaDcB1ThldRsrp. Weak coverage RF Check whether the test point is within the coverage of the NR cell. NsaDcB1ThldRsrp The actual signal strength does not reach the B1 threshold. Check engineering parameters and check whether the NR cell covers the test point. Check whether coverage parameters are set properly. Check whether the AAU cable connections and AAU antenna space are the same as planned. Abnormal NR cell status The value of the Cell Status field in the DSP NRCELL command output is not Normal. Abnormal AAU transmit power Choose cell performance monitoring > AAU channel power monitoring to monitor the AAU transmit power. The output power of the AAU channel is less than 50 dBm according to the cell performance monitoring result. NR 2023-05-02 Optimize coverage. Modify improper parameters and connections. Ensure that the cell status is normal. For details, see NR Cell Troubleshooting Guide. 1. Increase the transmit power of the AAU and check whether the problem is resolved by running the following command: MOD NRDUCELLTRP: NrDuCellTrpId=0, MaxTransmitPower=XX; 2. Increase the power of the PBCH and synchronization signals and check whether the UE can search for the cell by running the following command: MOD Huawei confidential. No spreading without permission. 22 5G Troubleshooting Guide – NSA Access AAU channel calibration failure Run the DSP NRDUCELLCHNCALIB: NrDuCellId=x; command and find that the latest channel calibration results indicate failures. When the base station is close to the TUE, the downlink power is oversaturated. As a result, the TUE fails to detect cells. The distance between the UE and the base station and the AAU power are used for judgment. Manual determination is required. Cell search failure due to interference from neighboring 5G cells Interference from neighboring 5G cells causes the cell search failure. Use the frequency scanner or TUE spectrum scanning function to check whether downlink neighboring cell interference exists. UE No NR cell is found due to UE problems. If the problem is not caused by LTE and NR base stations, check whether the cell search failure is caused by UE problems. E2E There are too many interfrequency measurement objects. As a result, the UE measurement takes a long time and the timer expires after 3s. Check the number of inter-frequency and inter-RAT measurement objects and the number of measurements configured for the UE through signaling. INTERNAL NRDUCELLCHNPWR: PbchPowerOffset=2000, SchPowerOffset=2000; 3. If the problem persists, collect one-click logs of the AAU/BBP/MPT and send the logs to Huawei headquarters for analysis. See the 5G channel fault location guide. Increase the attenuation on the TUE to prevent access failures due to oversaturated uplink and downlink power. Eliminate the downlink interference source or modify the center frequency. Use other terminals to isolate the problem. Use blind configuration to avoid this problem. On the LTE side, the following parameters must be configured and a license is required: NSA blind configuration switch of neighbor relationships: MOD NSADCMGMTCONFIG: LocalCellId=xx, NsaDcAlgoSwitch = NSA_BLIND_SCG_ADDITION_ SWITCH-1; NR blind configuration indicator: MOD NRNRELATIONSHIP: LocalCellId=xx, Mcc="xx", Mnc="xx", GnodebId=xx, CellId=xx, BlindConfigIndicator=TRUE; Disable inter-frequency MRs using the inter-frequency MR blacklist and check whether the problem is resolved. Abnormal AAU Transmit Power 1. Check whether L2 sends the MIB properly based on CHR logs. Check L2CellChrBlkInfo > L2_TRPCHR_BLK_CCHP_INFO > ChrProcessRecord > CCHPSendPBCHCnt. This event is measured every 10 minutes by cell. The following figure shows 30000 times for 10 minutes when the NRDUCELL.SsbPeriod parameter is set to 20 ms. This indicates that the L2 sends the MIB properly. If the number of MIB sending times is too small, contact L2 personnel for processing. 2023-05-02 Huawei confidential. No spreading without permission. 23 5G Troubleshooting Guide – NSA Access INTERNAL If L2 sends the MIB properly, go to the next step. 2. Check whether the MIB is sent over the L1 egress of the base station. Obtain AAU one-click logs. The logs need to be parsed by the baseband tool. You can contact L1 to view the logs. Check the antenna power in the DLPWRTRACE. The normal transmit power is about 20,000 and the non-transmit power is about 7000. The following figure shows the normal transmit power of the AAU. The following figure shows an abnormal case: 2023-05-02 Huawei confidential. No spreading without permission. 24 5G Troubleshooting Guide – NSA Access INTERNAL If the egress does not transmit power, you may check the following items: Check whether the cell is set up successfully. Check whether related alarms are reported on the MPT alarm console. Check whether the optical fiber is properly connected and whether the AAU experiences wrong positioning. Check whether L1 experiences abnormal verification (contact L1 personnel). If the MIB is sent properly over the L1 egress, go to the next step. 3. Monitor the AAU transmit power to check whether the AAU transmits power properly. If the AAU transmit power is abnormal, perform the following operations: Check whether the AAU transmit power exceeds the AAU power range. Contact AAU engineers for further check. No NR Cells Measured Due to RF Causes For the problem that the UE cannot measure NR cells due to RF causes, frontline engineers need to provide engineering parameters and test locations. (For Huawei UEs, Probe logs can be collected. For other UEs, the test locations and surrounding NR cells are marked on the map.) If the AAU transmit power is normal and reaches the maximum, but the UE still cannot measure any NR cell, check whether the NR cells to be measured are proper based on the test locations and engineering parameters, in terms of the distance, and horizontal and vertical coverage areas of the antenna. On the base station side, check the following parameters: 2023-05-02 Configure a proper beam scenario by using NRDUCELLTRPBEAM.CoverageScenario based on the test result. − When better horizontal coverage is required, SCENARIO_1, SCENARIO_6, or SCENARIO_12 is recommended. With the setting, cell edge UEs can obtain higher beam gains and cell edge coverage is improved. − If there are any fixed interference sources at the cell edge, SCENARIO_2, SCENARIO_3, SCENARIO_7, SCENARIO_8, or SCENARIO_13 can be used to narrow down the horizontal coverage scope and avoid the interference. − When there are only isolated buildings, SCENARIO_4, SCENARIO_5, SCENARIO_9, SCENARIO_10, SCENARIO_11, SCENARIO_14, SCENARIO_15, or SCENARIO_16 is recommended to provide small horizontal coverage. Huawei confidential. No spreading without permission. 25 5G Troubleshooting Guide – NSA Access INTERNAL Check whether the downtilt configuration (NRDUCELLTRPBEAM.Tilt) is proper based on the test point. Check whether the azimuth configuration (NRDUCELLTRPBEAM.Azimuth) is proper based on the test point. Check whether the SSB power offset configuration (NRDUCELLCHNPWR.MaxSsPbchPwrOffset) is proper. If the preceding parameters are correctly set, check the physical connections of the AAU to avoid the situation that NR cells do not cover the test points due to incorrect connections of the AAU. 4.1.3.4 Cases The UE Cannot Report the B1 Measurement Report Because of Too Small Transmit Power On December 22, 2018, the NR carrier configuration on the AAU5613 was set to 20 dBm at a site. As a result, the UE cannot access the network. The reported power monitoring value was N/A. The calibration was normal. The problem was resolved after the carrier power configuration was set to 34.9 dBm on December 25, 2018. 2023-05-02 Huawei confidential. No spreading without permission. 26 5G Troubleshooting Guide – NSA Access INTERNAL When 20 dBm is configured, the AAU has pilot power. When the NR carrier configuration is 20 dBm, the calibration is normal, indicating that the AAU channel is normal. On the baseband side, power is delivered. The chip records on the AAU indicate that the input power can be detected. According to the log statistics on the baseband side, the power is always delivered to the AAU. The following figure shows the pilot power (unit: 0.01 dbfs) at the ingress of the 6219E chip measured by the AAU5613. 2023-05-02 Huawei confidential. No spreading without permission. 27 5G Troubleshooting Guide – NSA Access INTERNAL According to the AAU alarms, no new AAU alarms were reported after 16:00 on December 22. The reported value of AAU power monitoring is N/A, which is the AAU feedback power. AAU power monitoring is used on the monitoring interface. The reported value is the AAU feedback power. The AAU solution for power statistics requires that the feedback power must be greater than or equal to -47 dbfs. If the feedback power is less than -47 dbfs, the feedback power is discarded and N/A is displayed on the monitoring interface. The carrier configuration is 20 dBm, and the AAU pilot power is too small. When the intersite distance between outdoor sites is 100 m, the UE may not be within the main lobe beam range. As a result, the UE cannot search for the AAU transmit signals. After you increase the power and enable the UE to access the network, the N/A information displayed on the monitoring interface does not indicate that the AAU does not transmit power. 4.1.4 B1 Measurement Result Is Reported But No SgNB Addition Procedure Is Initiated 4.1.4.1 Basic Concept After the UE reports the B1 measurement result, the LTE network initiates an SgNB addition procedure to the cell corresponding to the PCI carried in the B1 measurement result through the X2 interface. 4.1.4.2 Identification Method On the Uu interface, find the Call ID corresponding to the 5G B1 measurement report. Then, check whether the SGNB_ADD_REQ message corresponding to the Call ID exists on the X2 interface. 2023-05-02 Huawei confidential. No spreading without permission. 28 5G Troubleshooting Guide – NSA Access INTERNAL 4.1.4.3 Problem Location MML Query For LTE 19B SPC170 and later versions, the DSP SCGADDFAILUREINFO command can be executed to check the reason why SgNB Add Req is not initiated after measurement reports are reported. You can determine the specific cause: missing configuration of neighboring cells, PCI conflict between neighboring cells, or X2 fault. The following figure helps you find the specific LTE cell, NR cell, and the addition failure cause. Configuration Analysis After the B1 measurement is reported, the LTE side determines that an SCG addition can be initiated if all the following conditions are met: Neighboring NR cells are correctly configured (can be checked using the NSA Parameter Check tool). There is no PCI conflict. Refer to the forward check -> parameter check. You can use the NSA Parameter Check tool to perform an automatic check. The following provides the results obtained by an automatic check of the NSA Parameter Check tool. The tool compares the LTE's neighboring NR cell configuration with the actual NR configuration to check whether the configuration is correct, whether the frequency configuration is correct, and whether the NSA DC switch is turned on. When a configuration exception occurs, an error message is displayed. Manual check is required for missing neighboring cells. Check whether the PCI in the measurement report is configured with neighbor relationship on the LTE side. //External NR cell: MO NREXTERNALCELL //Neighbor relationship with an NR cell: MO NRNRELATIONSHIP 2023-05-02 Huawei confidential. No spreading without permission. 29 5G Troubleshooting Guide – NSA Access INTERNAL The X2 link is normal. (Use the NSA Parameter Check tool to check the X2 configuration. Check whether the X2 link is normal based on alarms.) Run the following commands to query the status of the X2 link between the eNodeB and the gNodeB. If the X2 link is abnormal, the eNodeB does not send the SGNB_ADD_REQ message. Run the DSP X2INTERFACE command on the eNodeB. Run the DSP GNBCUX2INTERFACE command on the gNodeB. CHR-based Problem Location As shown in the following figure, the LTE CHRs in version 19B contain the CHRs circled in blue. Based on the circled CHRs, you can determine whether the eNodeB has sent the SGNB_ADD_REQ message. Select NSADC_EVENT in the Type column. The event is printed once an hour. Based on the test time, find the event in the corresponding time. For example, if the test time is 16:00, check the event occurred at 16:34:55 (the time segment is 15:34:55–16:34:55). 2023-05-02 Huawei confidential. No spreading without permission. 30 5G Troubleshooting Guide – NSA Access INTERNAL 4.1.4.4 Cases For details, see maintenance and test methods in the problem location section. 4.2 Access Preparation Process 4.2.1 The gNodeB Does Not Respond with SGNB_ADD_REQ_ACK 4.2.1.1 Basic Concept After receiving an SGNB_ADD_REQ message from the eNodeB, the gNodeB starts to prepare for the access on the NR side. The following figure shows the signaling procedure. The following figure shows the signaling tracing over the X2 interface. After the UE reports the B1 measurement report, the eNodeB sends an SGNB_ADD_REQ message through the X2 interface. This message carries the bearer setup message. 2023-05-02 Huawei confidential. No spreading without permission. 31 5G Troubleshooting Guide – NSA Access INTERNAL Upon reception of the SGNB_ADD_REQ message, the gNodeB responds with an SGNB_ADD_REQ_ACK message. This message informs the eNodeB of the address allocated by the gNodeB and carries the parameters required for the UE's Uu interface access. 2023-05-02 Huawei confidential. No spreading without permission. 32 5G Troubleshooting Guide – NSA Access INTERNAL 4.2.1.2 Identification Method If the gNodeB receives an SGNB_ADD_REQ message over the X2 interface but does not respond with an SGNB_ADD_REQ_ACK message, the problem is caused by the 5G side. 4.2.1.3 Problem Location Step 1 Run MML commands to check whether links are abnormal. Check Item MML Command Version X2INTERFACE (LTE side) LST X2INTERFACE (Whether the link is abnormal) 19A EPGROUP DSP EPGROUP (Whether the link is abnormal) 19A Step 2 Analyze why the gNodeB does not return an SGNB_ADD_REQ_ACK message based on one-click logs. ----End 4.2.1.4 Cases No such problem is found. 4.2.2 The gNodeB Returns SGNB_ADD_REQ_REJ 4.2.2.1 Basic Concept After receiving an SGNB_ADD_REQ message, the NR replies with an SGNB_ADD_REQ_REJ message to reject the access request if checking the IEs in the SGNB_ADD_REQ message fails or the gNodeB status is abnormal. 4.2.2.2 Identification Method The X2 signaling tracing results show that the gNodeB responds with an SGNB_ADD_REQ_REJ message after receiving an SGNB_ADD_REQ message. This message contains the failure cause. 2023-05-02 Huawei confidential. No spreading without permission. 33 5G Troubleshooting Guide – NSA Access INTERNAL 4.2.2.3 Problem Location Known Issue Check If the gNodeB replies with an SGNB_ADD_REQ_REJ message, see the access fault tree. CHR Analysis In CHRs, filter the Type column by L3IChrCuSgnbAddRej to obtain all AddRej data. In CHR events, you can view the latest 30 signaling messages, including the cause value carried in the Add Rej message, User TMSI, CallID, CRNTI, and information about the peer anchor site. You can summarize all the information to find the rule. TrackListInfo records some key dbg information about the user rej, which can be used to help further analyze the rej cause. 2023-05-02 Huawei confidential. No spreading without permission. 34 5G Troubleshooting Guide – NSA Access INTERNAL The 5G FMA supports statistical analysis of SgNB Add Rej. Based on this, you can perform further analysis on DCInnerCause statistics of Add Rej, whether top UEs exist, and whether top abnormal IP addresses exist. 4.2.2.4 Cases 4.2.2.4.1 The NR Side Returns a Setup Request Rejection in NSA Networking. Symptom In NSA networking, when the UE initiates an NR access, the X2 interface tracing results show that a rejection message is returned after an SGNB_ADD_REQ message is transmitted. 2023-05-02 Huawei confidential. No spreading without permission. 35 5G Troubleshooting Guide – NSA Access INTERNAL Analysis The X2 interface tracing results show that a rejection message follows an SGNB_ADD_REQ message. This indicates that the processing on the NR side is abnormal. According to the analysis of logs, an error occurs when the data of the uplink cell is queried. As a result, an exception occurs. According to the confirmation with test personnel, the uplink and downlink decoupling switch is turned on but the uplink cell is not configured. Therefore, this problem occurs. In this case, the uplink and downlink decoupling switch is turned off and users can access the network. Conclusion The uplink and downlink decoupling switch is turned on, but the uplink cell is not configured. As a result, the uplink cell data cannot be found in the CU processing procedure and the rejection procedure is performed. 4.2.3 The eNodeB Does Not Return an SGNB_RECFG_CMP Message. 4.2.3.1 Basic Concept The UE reconfiguration is completed when the eNodeB sends an X2AP_SGNB_RECFG_CMP message to the gNodeB through the X2 interface. 2023-05-02 Huawei confidential. No spreading without permission. 36 5G Troubleshooting Guide – NSA Access INTERNAL 4.2.3.2 Identification Method The X2 signaling tracing on the NR side indicates that the gNodeB does not receive an SGNB_RECFG_CMP message after sending an SGNB_ADD_REQ_ACK message. The gNodeB or eNodeB initiates a release procedure. 5G does not receive the SGNB_RECFG_CMP message. After the timer expires, 5G initiates a release procedure. SGNB_RECFG_CMP: The LTE initiates a release procedure. 2023-05-02 Huawei confidential. No spreading without permission. 37 5G Troubleshooting Guide – NSA Access INTERNAL 4.2.3.3 Problem Location Known Issue Check If LTE does not respond with an SGNB_RECFG_CMP message, check whether the SCG addition reconfiguration message is delivered over the LTE Uu interface. If yes, check whether the UE experiences a reestablishment or SCGFAIL occurs. 1. If the UE experiences a reestablishment, LTE does not send an SGNB_RECFG_CMP message to 5G. 2. The reconfiguration completion message does not contain the scg-ConfigResponseNR-15 IE. As a result, LTE does not send an SGNB_RECFG_CMP message to 5G. The following figure shows a normal reconfiguration completion message. 2023-05-02 Huawei confidential. No spreading without permission. 38 5G Troubleshooting Guide – NSA Access INTERNAL Troubleshoot the reestablishment and the problem that the reconfiguration completion message does not contain the r15 IE. For details about common problems and troubleshooting methods, see the access fault tree. CHR Analysis Use the statistics function of the FMA to collect statistics on the three DC fields of the L3ChrCuAbnormalRel event, as shown in the following figure. You can check whether an exception occurs in the ADD phase by viewing the siglist of each DC. 2023-05-02 Huawei confidential. No spreading without permission. 39 5G Troubleshooting Guide – NSA Access INTERNAL For example, in the case of DCInnerCause=CU_UEM_SGNBADD_PROC_ONKILL, LTE does not respond with an SgNB_Recfg_CMP message. You can further check the cause by using the LTE CHR or signaling. The eNBID and tmsi information in the UEPublicInfo can help find the corresponding eNodeB and UE. 2023-05-02 Huawei confidential. No spreading without permission. 40 5G Troubleshooting Guide – NSA Access INTERNAL 4.2.3.4 Cases N/A 4.3 NR Air Interface Access Failure 4.3.1 The UE Does Not Initiate Access After Receiving a Reconfiguration Message 4.3.1.1 Basic Concept After receiving a reconfiguration message from the SCG, the UE initiates air interface access only after the following steps are complete: 1. The UE verifies the IEs in the reconfiguration message to check whether the UE supports the channel resources allocated by the eNodeB. If not, the UE sends an SCG_FAIL_INFO message carrying the cause value "scg-reconfigFailure" to the eNodeB. Alternatively, the UE sends an RRC_CONN_REESTAB_REQ message carrying the cause value "reconfigFailure". 2. The UE searches for the 5G cell indicated in the SCG reconfiguration message and synchronizes it with the 5G cell. If the synchronization fails, the UE sends an SCG_FAIL_INFO message carrying the cause value "synchReconfigFailureSCG" to the eNodeB. 4.3.1.2 Identification Method 2023-05-02 Uu message tracing Huawei confidential. No spreading without permission. 41 5G Troubleshooting Guide – NSA Access INTERNAL The Uu interface tracing results show that after receiving an RRC reconfiguration message for 5G SCG configuration, the UE immediately sends an SCG_FAIL_INFO message carrying the cause value "scg-reconfigFailure/synchReconfigFailureSCG" to the eNodeB. Probe tracing In the Probe tracing results of the UE, you can view the printing of the reconfiguration failure from the Event List and view the possible causes of the reconfiguration failure in the Key Event List. 4.3.1.3 Problem Location The possible causes of the reconfiguration failure are as follows: The 5G cell search fails because the cell that the UE accesses is not the strongest cell or the inter-cell interference is severe. The parameters in the SCG reconfiguration message fail to be verified on the UE side. In this case, it is recommended that terminal engineers participate in the fault locating. For details about common problems and troubleshooting methods, see the access fault tree. 4.3.1.4 Cases 4.3.1.4.1 Cell Search Fails Because the UE Accesses a Non-strongest Cell. Symptom In 19A TR6, the 5G SCG access fails after an LTE handover in a 5G mobility drive test. 2023-05-02 Huawei confidential. No spreading without permission. 42 5G Troubleshooting Guide – NSA Access INTERNAL Analysis According to KeyEventList, the 5G access failure is caused by cell search failures. Check the B1 measurement results. The cell with PCI 152 is in the third place, and the RSRP of this cell is 13 dB lower than that of the best cell with PCI 435. This indicates that the access failure cause is that the cell search fails because the UE accesses a non-strongest cell. Further analysis shows that the neighbor relationships between the LTE cell and the 5G cells with PCIs 435 and 436 are not configured. As a result, the problem occurs. Conclusion The root cause of this problem is that the neighbor relationships between the LTE cell and the 5G cells with PCIs 435 and 436 are not configured. After the neighbor relationships between the LTE cell and the 5G cells with PCIs 435 and 436 are added, the problem is solved. 4.3.2 The RAR Message over the Air Interface Times Out 4.3.2.1 Basic Concept The first step of a random access procedure is that the UE sends a random access preamble. The preamble is used to inform the base station of a random access request, so that the base 2023-05-02 Huawei confidential. No spreading without permission. 43 5G Troubleshooting Guide – NSA Access INTERNAL station can estimate the transmission delay between the base station and the UE, calibrate the uplink timing, and send the TA adjustment information to the UE through an RAR. After sending the preamble, the UE monitors the PDCCH in the RAR time window (RA response window) to receive the RAR of the corresponding RA-RNTI (in this case, the measurement gap may not be considered). If no RAR message is received from the eNodeB within the RAR time window, the random access procedure fails. In NSA networking, non-contention-based random access is used. The DCI of the RAR is carried in the common search space of the PDCCH. The content of the RAR is carried on the PDSCH and occupies one RBG. 4.3.2.2 Identification Method The Uu interface tracing results show that, after receiving an RRC reconfiguration message for the 5G SCG configuration, the UE returns an SCG_FAIL_INFO message with the cause value ScgAccessFailure or scg-ChangeFailure to the eNodeB at an interval specified by T304. In the Probe tracing result of the UE, the access failure information is displayed in the Event List, and the RAR timeout information is displayed in the Key Event List. 4.3.2.3 Problem Location RAR timeout scenarios are as follows: The UE sends a Preamble but the base station does not receive it. The base station sends an RAR message but the UE does not receive it. Basic Configuration Check The configuration that may affect the preamble reception function is as follows: MOD NRDUCELLPRACH:RootSequenceIndex=0; 2023-05-02 Huawei confidential. No spreading without permission. 44 5G Troubleshooting Guide – NSA Access INTERNAL Root sequence index used by the preamble. The root sequence index allocated to the UE by higher layers and the root sequence index configured for the base station must be consistent. Otherwise, preamble parsing fails. MOD NRDUCELL:CellRadius=9000; Cell radius configuration. Cell radius affects the NCS parameter used for preamble sequence generation. If this parameter is set to a small value, UEs in the middle and far points cannot access the network. MOD NRDUCELLRSVDPARAM:RsvdSwParam4_bit5=0; If this parameter is set to 1, PRACH resources are allocated to the PUSCH to increase the uplink peak rate. This switch must be configured after UEs access the network. Otherwise, UEs cannot access the network. CHR Analysis Use the L2 CHRs to analyze the RAR timeout and, for the non-contention-based preamble, find the L2USerChr at the time when the problem occurs. The l2_user_chr_preamble_info event block is used to analyze the RAR timeout. The event block contains the following events: The method for analyzing whether the preamble is correctly received is as follows: 2023-05-02 Obtain the dedicated PreambleId received by L2 from L3, the reception time, and the CRNTI of the UE, from the L3Alloc event block. Huawei confidential. No spreading without permission. 45 5G Troubleshooting Guide – NSA Access 2023-05-02 INTERNAL Analyze the L1Rpt event block to determine whether the base station receives the preamble, the reception time, and PreambleId. Compare the PreambleId with the PreambleId in the L3Alloc event block. If the event block contains the time when the base station receives the preamble and the PreambleId is the same as the PreambleId in the L3Alloc event block, the base station has received the preamble. Huawei confidential. No spreading without permission. 46 5G Troubleshooting Guide – NSA Access INTERNAL Analyze the L2sendRar event block to determine whether the base station delivers an RAR message. The event block records the time when the base station delivers the RAR message, as well as the CRNTI, TA value, Msg3 Ulgrant information, and CommonDCI content of the RAR message. If the base station receives the preamble but the L2sendRar event block does not contain the RAR delivery record, further analyze the following uplink and downlink scheduling events to determine whether the RAR scheduling failure is caused by uplink scheduling exceptions or downlink scheduling exceptions. CellDT Analysis The air interface access failure on the NR side can be isolated by using the No. 1 tracing of L2 TTI tracing. L2 TTI tracing uses the No. 1 tracing to locate the access problem. (No. 1 tracing is the access process TTI tracing set, which contains a maximum of dozens of tracing numbers in the entire access process.) The method for isolating an access failure in TTI tracing is as follows: The No. 212 tracing result in the No. 1 tracing task shows that L3 configures the dedicated preamble for L2 and L1 reports the dedicated preamble. 2023-05-02 Huawei confidential. No spreading without permission. 47 5G Troubleshooting Guide – NSA Access INTERNAL The key information in the No. 212 tracing result is ucL3IndToSusrFlag. ucL3IndToSusrFlag=1 indicates that the dedicated preamble is allocated by L3. ucL3IndToSusrFlag2 and ucL3IndToSusrFlag=4 indicate that the dedicated preamble is released by L3. ucL3IndToSusrFlag=5 indicates that the baseband reports an invalid preamble ID. If the time of ucL3IndToSusrFlag=2 and ucL3IndToSusrFlag=4 is 2s later than ucL3IndToSusrFlag=1, the access fails. In this case, perform the following operations to demarcate the failure phase: Preamble phase failure In the No. 1 tracing task, there is no record in the No. 210 tracing result (L2 receives the preamble stream tracing result reported by L1). Alternatively, there are records in the No. 210 tracing result, but the reported preamble ID is different from the ID of the dedicated preamble allocated by L3. In this case, the access fails in the preamble phase. RAR phase failure The No. 201 tracing result contains the process from the time L1 reports the preamble to the time L2 assembles and delivers RAR packets. If L2 has assembled and delivered an RAR message to L1 but the UE does not receive the RAR message, the access fails in the RAR phase. RAR delivery process (indicated by the ulRaStatus field): 2023-05-02 CCHP_RA_PREAMBLE_PROC indicates that the UE is obtained from the preamble chain to be scheduled. CCHP_RA_ADD_DLSCH_RARNTI indicates that the preamble UE is added to the downlink scheduling chain. CCHP_RA_ADD_ULSCH_RARNT indicates that the preamble UE is added to the uplink scheduling chain. CCHP_RA_DLSCH_INIT indicates that scheduling the preamble UE in the downlink starts. Huawei confidential. No spreading without permission. 48 5G Troubleshooting Guide – NSA Access INTERNAL CCHP_RA_DLSCH_SCH indicates that scheduling the preamble UE in the downlink ends. CCHP_RA_ULSCH_INIT_SNAP indicates that scheduling the preamble UE in the uplink starts. CCHP_RA_ULSCH_SCH indicates that scheduling the preamble UE in the uplink ends. CCHP_RA_PACK_RAR indicates that RAR packets are delivered. If RAR scheduling fails, determine whether the failure is caused by uplink or downlink scheduling failure based on the preceding procedure. If uplink scheduling fails, check the No. 408 tracing. If downlink scheduling fails, check the No. 502 tracing. The tracing result records the error code of the scheduling failure. Contact R&D engineers of the corresponding module to locate the fault. After the preceding process is completed, further analyze whether the No. 205 tracing has corresponding content (the No. 205 tracing carries the beam information when the RAR is delivered and the code stream of the RAR). In the No. 205 tracing, there are two frame numbers and subframe numbers, which are the system frame number and subframe number of the RAR packet and the frame number and subframe number that is expected to be delivered by the RAR over the air interface. The latter subframe number is represented by SubframeNo and SlotNo, the actual slot number is SubframeNo x 2 + SlotNo. 4.3.2.4 Cases 4.3.2.4.1 The TUE Fails to Access the Network in RF Direct Connection Mode and Cannot Receive the Preamble. Symptom In B078, the TUE fails to access the network. The L2 tracing result shows that the TUE does not receive the preamble (no data is traced in the 210 Preamble code stream tracing). Contact the HQ for further processing. 2023-05-02 Huawei confidential. No spreading without permission. 49 5G Troubleshooting Guide – NSA Access INTERNAL Analysis The ingress can detect signals but cannot decode valid preamble data. Therefore, the IF/RF and TUE L1 must be used together to locate the fault. The preamble is periodically sent in the TUE L1 odd frames, and the IF/RF module is used to collect data. Based on the data collected by the IF/RF module, it is found that the data is transmitted to the IF/RF for a time offset. Analysis results show that an initial TA value is configured for the TUE. The TUE data sent to the IF/RF module is beyond the receive window. As a result, the preamble is truncated and baseband demodulation fails. After the initial TA of the TUE is changed to 0, the problem is solved. The following figure shows the signals received by the IF/RF module before and after the modification. Conclusion An initial TA value is configured for the TUE, which causes the problem that the TUE data sent to the IF/RF module is beyond the receive window. As a result, the preamble is truncated and baseband demodulation fails. The problem is resolved after the initial TA of the TUE is changed to 0. 4.3.2.4.2 The TUE Fails to Access the Network in RF Direct Connection Mode and Cannot Receive the RAR Message. Symptom In the CID environment, the 5G TUE fails to access the network. The TUE OMT shows that T304 times out. 2023-05-02 Huawei confidential. No spreading without permission. 50 5G Troubleshooting Guide – NSA Access INTERNAL Analysis The base station L2 tracing result shows that the preamble is successfully reported and the RAR has been scheduled and delivered to the base station L1. The simulation result of the base station L1's samplings shows that the RAR message has been sent successfully and the signals are normal. Symbol 0 is a PDCCH, symbol 6 and symbol 9 are DMRSs, and signal energy is higher. The samplings of TUE L1 at the corresponding frame number and subframe number indicate that there is no valid signal on the baseband ingress of the TUE. Currently, the base station's L1 sends normal signals but the TUE's L1 does not receive the signals. The next step is to analyze whether the channel between the base station's L1 and the TUE's L1 is normal. 2023-05-02 Huawei confidential. No spreading without permission. 51 5G Troubleshooting Guide – NSA Access INTERNAL In order for the IF/RF module to collect data, the base station needs to periodically send an RAR message. Then, the base station and TUE use the fixed offset of the frame interrupt to check whether the intermediate channel is normal. The stub test method is as follows: 1. Use the TUE to send preambles at a fixed subframe number. T304 does not expire. 2. After receiving the preamble, the base station returns an RAR every time, without requiring L3 to allocate dedicated preambles. The UE is not released by the base station L3. The IF/RF module collects data. The signal characteristics are consistent with those of RAR signals. The frequency domain occupies 20 MHz bandwidth (48 RBs in BWP0). The RAR uses the PDSCH of one RBG (4 RBs) for transmission, corresponding to the 1-2 MHz bandwidth (4 RBs = 30 kHz x 12 x 4 = 1.44 MHz) in the figure. The TUE IF/RF module collects data. The signal characteristics are the same as those of the RAR signal. IF/RF entry on the TUE: The following figure shows the signals transmitted from the TUE RF output port to the TUE baseband. 2023-05-02 Huawei confidential. No spreading without permission. 52 5G Troubleshooting Guide – NSA Access INTERNAL This problem occurs between the TUE IF/RF and the TUE baseband. Conclusion This problem occurs between the TUE IF/RF and the TUE baseband. The TUE R&D engineers analyze the problem and find that the TUE OM is abnormal. As a result, the data link between the TUE IF/RF and the TUE baseband is abnormal. 4.3.3 Msg3 Failure 4.3.3.1 Basic Concept Different from the access in the LTE network or the access in SA networking, the Msg3 in NSA networking is an MCE (carrying BSR information) but not RRC signaling. When the uplink MAC demultiplexing receives the MCE from the UE, the UE successfully accesses the network through the air interface. 4.3.3.2 Identification Method The Uu interface tracing results show that, after receiving the RRC reconfiguration message for the 5G SCG configuration, the UE returns an SCG_FAIL_INFO message with the cause value ScgAccessFailure or scg-ChangeFailure to the eNodeB at an interval specified by T304. In the Probe tracing result of the UE, the access failure information is displayed in the Event List, and the RAR timeout information is displayed in the Key Event List. 2023-05-02 Huawei confidential. No spreading without permission. 53 5G Troubleshooting Guide – NSA Access INTERNAL 4.3.3.3 Problem Location Known Issue Check For details about common problems and troubleshooting methods, see the access fault tree. CHR Analysis Use the L2 CHRs to analyze the Msg3 scheduling or demodulation failure and find the L2UserChr at the time when the problem occurs. Use the L2_USERCHR_ULSCH_ACCESS_INFO event block to analyze the cause for Msg scheduling or demodulation failures. 2023-05-02 Huawei confidential. No spreading without permission. 54 5G Troubleshooting Guide – NSA Access INTERNAL The event block shows the results of Msg3 scheduling, reception time, and demodulation result. As shown in the preceding figure, Msg3 scheduling is successful, but CRC errors occur during the first transmission and five retransmissions. This indicates that the access failure is caused by the Msg3 CRC error. CellDT Analysis Msg3 phase failure The TUE OMT shows that the TUE has sent Msg3. However, the No. 324 tracing on the base station side (DMAC demultiplexing result tracing) shows that the TBSize of the decoded MAC PDU is 0. This indicates that Msg3 demodulation fails. After Msg3 is successfully scheduled, L2 sends the management frame of Msg3 to L1. As long as Msg3 fails to be demodulated, the L2 tracing result shows that Msg3 CRC is incorrect. The cause may be either of the following: The TUE does not send Msg3, or the TUE sends Msg3, but the baseband fails to demodulate the message. It is recommended that the No. 1 tracing be performed on the base station side and L2 tracing be performed on the TUE side. Step 1 Check whether the TUE has sent Msg3. If the TUE has decoded the RAR but does not send Msg3, continue to locate the TUE problem. In the No. 490 tracing result, check whether the base station successfully receives Msg3. Preamble in the bit05UsrType column indicates Msg3 scheduling. The bit02aucCrcResultTB0 column indicates the CRC value of Msg3. The value 0 indicates that the base station demodulates Msg3 successfully. The value 1 indicates Msg3 demodulation fails. 2023-05-02 Huawei confidential. No spreading without permission. 55 5G Troubleshooting Guide – NSA Access INTERNAL In the No. 777 tracing, check the DMRS RSRP at the time when Msg3 is scheduled. If the signal is good, the UE sends Msg3. If the signal is close to the background noise, the UE does not send Msg3. Further check the DCI sent by L2 to the UE and the management frame sent by L2 to L1. Step 2 Check whether the DCI carried in the RAR code stream is consistent with the content of the management frame sent by L2 to L1. The No. 404 tracing result records the Ulgrant code stream of Msg3, and the content of the management frame can be viewed in the No. 455 tracing result. The key information to be checked is as follows: frame number, subframe number, allocated frequency domain resources, DMRS type, DMRS Port, and codec information (MCS, rank, and so on). If the DCI information is the same as that of the management frame, check whether the DCI sent by the TUE is consistent with the DCI delivered by the base station's L2. Step 3 If the scheduling and codec information contained in the base station's L2, management frame, and TUE L2 are consistent, the TUE L1 and the base station's L1 need to perform simulation. Check whether the data sent by the TUE L1 contains the correct Msg3 information and whether the data collected by the entry of the base station's L1 can be used to decode the correct data. Contact R&D personnel to locate the cause of the problem. ----End 4.3.3.4 Cases 4.3.3.4.1 UE Access Fails Because the Management Frame of Msg3 Is Verified by Layer 1. Symptom In B110, the TUE B092 fails to access the network. Msg3 has been sent according to the TUE display. 2023-05-02 Huawei confidential. No spreading without permission. 56 5G Troubleshooting Guide – NSA Access INTERNAL Analysis Capture the tracing results of the No. 1, No. 105, and No. 455 tracing tasks. It is found that there is no 323 and 324 tracing in the No. 1 tracing result. That is, L2 does not receive the CRC result of Msg3. Further analyze the scheduling and management frames of Msg3. According to the No.404 tracing result, Msg3 has been successfully scheduled. Check whether Msg3 management frames are framed and delivered. The No.455 tracing result shows that the management frames of Msg3 are delivered (including one initial transmission and four retransmissions). 2023-05-02 Huawei confidential. No spreading without permission. 57 5G Troubleshooting Guide – NSA Access INTERNAL The No.105 tracing result shows that the management frames are always delivered to L1. After L1 analysis, it is found that the CRC result is not generated and sent to L2 because there is no response to the message sent by the FPGA to the DSP at L1. Conclusion There is no response to the message sent by the FPGA to the DSP at L1. As a result, the CRC result is not generated and sent to L2, and Msg3 fails to be received. 4.3.4 Contention Msg4 Failure 4.3.4.1 Basic Concept If the UE initiates contention-based random access, the base station needs to deliver Msg4 to the UE to complete the random access. The following figure shows a contention-based random access procedure. 2023-05-02 Huawei confidential. No spreading without permission. 58 5G Troubleshooting Guide – NSA Access INTERNAL The base station does not allocate non-contention-based preambles to UEs in the following scenarios: The non-contention-based preambles of the base station have been used up. The UE does not report the SSBindex in the measurement report or LTE does not send the SSBindex to 5G. The UE triggers resynchronization access. 4.3.4.2 Identification Method The SCG addition reconfiguration message on the UE side or the Add Req Ack message on the 5G side contains the CRNTI and non-contention-based preamble. If the message contains the ra-PreambleIndex field, the base station allocates a noncontention-based preamble to the UE. If the field is not contained, the base station does not allocate a non-contention-based preamble to the UE, and the UE initiates contention-based random access. 2023-05-02 Huawei confidential. No spreading without permission. 59 5G Troubleshooting Guide – NSA Access INTERNAL The message also contains the CRNTI allocated by the base station to the UE. In the following figure, newUE-Identity indicates the allocated CRNTI. Both the CHR and CellDT on the base station side can use the CRNTI to align with the UE. In the Probe tracing result of the UE, the access failure information is displayed in the Event List, and the NR MAC receive content timeout information is displayed in the Key Event List. 4.3.4.3 Problem Location Known Issue Check For details about common problems and troubleshooting methods, see the access fault tree. 2023-05-02 Huawei confidential. No spreading without permission. 60 5G Troubleshooting Guide – NSA Access INTERNAL CHR Analysis Use the L2 CHRs to analyze the Msg3 failure and find the L2UserChr at the time when the problem occurs. To analyze the Msg4 access failure, use the L2_USER_KEYCHR_ACCESS_DLSCH_INFO to check the Msg4 scheduling. 2023-05-02 Huawei confidential. No spreading without permission. 61 5G Troubleshooting Guide – NSA Access INTERNAL CellDT Analysis 1. In the No. 490 tracing result, check whether the base station delivers Msg4 scheduling after receiving Msg3. Msg4 is one false SR scheduling triggered by the base station. After receiving it, the UE considers that the contention resolution succeeds. Sort the DE column lastSchTti in ascending order. Filter the time points to be analyzed, CRNTI, and UsrId. The frame number and slot number must correspond to Air indicated by columns L and M. You can check with those on the UE side. The I column bit02aucCrcResultTB0 indicates the uplink CRC result. The value 0 indicates correct while the value 1 indicates incorrect. Preamble in the O column bit05UsrType indicates Msg3. For non-contention-based random access, Msg3 corresponds to the C-RNTI. For contention-based random access, the C-RNTI corresponding to Msg3 is the TC-RNTI. You can map the C-RNTI and TCRNTI through the No. 716 tracing. The following figure shows the No. 716 tracing result. You can map the CRNTI and TCRNTI in one call by checking columns T and Y. 2. If the No. 490 tracing does not contain Msg4 scheduling and the preceding known issues are excluded, you are advised to contact R&D engineers of the corresponding module to locate the fault. 3. If the No. 490 tracing result shows that Msg4 has been scheduled but the CRC result always indicates incorrect, check the DCI traced in the No. 602 tracing result and compare it with the DCI received by the UE. 2023-05-02 Huawei confidential. No spreading without permission. 62 5G Troubleshooting Guide – NSA Access Sort the I column ttiJobCycle in ascending order. Filter the time points to be analyzed, CRNTI, and UsrId. The BD column is bit04DciFormat, indicating a format. The AN column is bit08DciLength. Observe the DCI length. INTERNAL 4. Check the uplink signal quality through the No. 777 tracing. Sort the K column rptTti in ascending order. Filter the time points to be analyzed, CRNTI, and UsrId. The O column l1MeasRptTag allows you to view information about the DMRS RSRP, DMRS SINR, SRS RSRP, SRS SINR, and SRS TA. For the U column DmrsAvgRsrp, the measured value must be divided by 64. For the V column DmrsFlow0Sinr, the measured value must be divided by 256. 4.3.4.4 Cases N/A 2023-05-02 Huawei confidential. No spreading without permission. 63 5G Troubleshooting Guide – NSA Access 5 INTERNAL Methods of Analyzing KPI-related Problems in Traffic Statistics (NSA) 5.1 Access KPI Definition The KPIs are defined as follows: LTE Counter NR Counter Counter ID Counter Name L.NsaDc.SgNB.Add.A tt Total number of SgNB addition attempts Total number of successful SgNB additions L.NsaDc.SgNB.Add.S ucc SgNB addition success rate = L.NsaDc.SgNB.Add.Succ/L.NsaDc.SgNB.Add.Suc c Counter ID Counter Name N.NsaDc.SgNB.Add.Att Number of SgNB addition requests N.NsaDc.SgNB.Add.Succ Number of successful SgNB additions SgNB addition success rate = N.NsaDc.SgNB.Add.Succ/N.NsaDc.SgNB.Add.Att N.RA.Contention.Att N.RA.Contention.Resolution.Succ Number of random preamble receptions Number of successful contention resolutions N.RA.Dedicated.Att Number of dedicated preamble receptions N.RA.Dedicated.Msg3 Number of Msg3 receptions during non-contention based random access RA success rate = (N.RA.Contention.Resolution.Succ + N.RA.Dedicated.Msg3)/(N.RA.Contention.Att + N.RA.Dedicated.Att) Non-contention RA success rate = N.RA.Dedicated.Msg3/N.RA.Dedicated.Att Contention RA success rate = N.RA.Contention.Resolution.Succ/N.RA.Contention.Att Both the LTE and NR counters measure the SgNB addition success rate. However, the values of the two counters may be different. For example, if the SgNB Add Reject is caused by cell unavailability, the cell counter object is not activated on the 5G side. As a result, the N.NsaDc.SgNB.Add.Att counter on the 5G side cannot measure this. 2023-05-02 Huawei confidential. No spreading without permission. 64 5G Troubleshooting Guide – NSA Access INTERNAL In addition, for NSA, N.NsaDc.SgNB.Add.Succ does not mean access success. It only indicates the completion on the signaling plane, after which the random access procedure starts. Due to false detection of preambles, the contention resolution success rate is usually low. For NSA, the access is generally non-contention-based. The non-contention-based RA success rate can be more concerned. 5.2 Signaling Procedures and Measurement Points L.NsaDc.SgNB.Add.Att/L.NsaDc.SgNB.Add.Succ N.NsaDc.SgNB.Add.Att/N.NsaDc.SgNB.Add.Succ As shown at point A in the following figure, the L.NsaDc.SgNB.Add.Att counter is incremented each time the eNodeB sends an SgNB Addition Request message to the gNodeB. As shown at point B in the following figure, the L.NsaDc.SgNB.Add.Succ counter is incremented each time the eNodeB sends an SgNB Reconfiguration Complete message to the gNodeB. As shown at point A in the following figure, the N.NsaDc.SgNB.Add.Att counter is incremented by one each time the gNodeB receives an SgNB Addition Request message from the eNodeB. As shown at point B in the following figure, the N.NsaDc.SgNB.Add.Succ counter is incremented by one each time the gNodeB receives an SgNB Reconfiguration Complete message from the eNodeB. The values of these counters are accumulated in the primary secondary cell (PSCell) specified by the gNodeB. N.RA.Contention.Att/N.RA.Contention.Resolution.Succ As shown at point A in the following figure, the N.RA.Contention.Att counter is incremented by 1 each time the gNodeB receives a random preamble from a UE. As shown at point C in the following figure, in a contention-based random access procedure, the N.RA.Contention.Resolution.Succ counter is incremented by 1 each time the gNodeB successfully delivers a Contention Resolution message to a UE performing contention-based access. 2023-05-02 Huawei confidential. No spreading without permission. 65 5G Troubleshooting Guide – NSA Access INTERNAL N.RA.Dedicated.Att/N.RA.Dedicated.Msg3 As shown at point B in the following figure, the N.RA.Dedicated.Att counter is incremented by 1 each time the gNodeB receives a dedicated preamble from a UE. As shown at point D in the following figure, the N.RA.Dedicated.Msg3 counter is incremented by 1 each time the gNodeB receives Msg3 from a UE during non-contention based random access. 2023-05-02 Huawei confidential. No spreading without permission. 66 5G Troubleshooting Guide – NSA Access INTERNAL 5.3 Method of Analyzing Access KPI-related Problems 5.3.1 Action 1: Determining the Problem Type KPI problems are classified into the following four types. When frontline personnel report problems, confirm the problem type and objective. KPIs deteriorate suddenly or in some time segments. KPIs change slowly and gradually deteriorate. The current KPIs do not meet the requirements and need to be improved to certain target values. The KPIs in a specific area are worse than those in other areas. For problems of type 1, mainly perform actions 4 and 6. For problems of type 3 and type 4, skip action 6. Action 5 for type 4 problems is the analysis focus. By identifying the differences between other KPIs in two areas, you can directly or indirectly prove the differences in the network structure and air interface quality to quickly locate the root problem cause or isolate the influencing factors. 5.3.2 Action 2: Time Trend Analysis Time trend analysis is used to analyze the change trend of each counter involved in the KPI formula and check whether the deterioration time is regular. First check how the total number of handover attempts and the number of successful handovers change. Based on the changes in the numerator and denominator, check whether the problem is related to the change in the number of UEs. Check whether the deterioration time is regular. Specifically, check whether the KPI value decreases gradually or abruptly, or whether the KPI value restores after a decrease in a period of time and then decreases again. In the event of abrupt decrease or repeated decrease, check whether it always occurs in certain hours of each day, or on a certain day of each week, or at the beginning or end of each month. When comparing KPIs at the day level, compare them with those of other weeks at the same time, such as weekends. Pay attention to the change of the number of cells during the analysis of traffic statistics. This prevents incorrect judgment regarding KPI changes caused by incomplete traffic statistics reported by frontline engineers and significant changes in the number of cells. If the number of cells differs greatly, contact frontline engineers to check whether the feedback data is complete and whether the number of cells and sites meet expectations. If the feedback data is correct, the live network is gradually adding sites or closing sites, which causes KPI changes. The check belongs to external event check. 5.3.3 Action 3: Traffic Statistics Cause Analysis If the 5G side responds with an Add Rej message, the following counters are measured on the 5G side to determine whether the failure is caused by radio or transmission faults. Counter ID Counter Name Counter Description Version 1911817854 N.NsaDc.SgNB.Add.Fail.Radio Number of SgNB addition failures caused by radio faults in LTE-NR NSA DC scenarios 19B SPC150 2023-05-02 Huawei confidential. No spreading without permission. 67 5G Troubleshooting Guide – NSA Access INTERNAL Counter ID Counter Name Counter Description Version 1911817855 N.NsaDc.SgNB.Add.Fail.Radio.No Res Number of SgNB addition failures caused by the unavailability of radio resources in LTE-NR NSA DC scenarios 19B SPC150 1911817856 N.NsaDc.SgNB.Add.Fail.TNL Number of SgNB addition failures caused by transmission faults in LTENR NSA DC scenarios 19B SPC150 The N.NsaDc.SgNB.Add.Fail.TNL counter is incremented by 1 each time the gNodeB receives an SgNB Addition Request message sent by the eNodeB and the gNodeB responds with an SgNB Addition Reject message. If the cause value indicates a radio fault, the N.NsaDc.SgNB.Add.Fail.Radio counter is incremented by 1. If the cause value is "No Radio Resources Available", the N.NsaDc.SgNB.Add.Fail.Radio.NoRes counter is incremented by 1. The values of these counters are accumulated in the primary secondary cell (PSCell) specified by the gNodeB. In later versions, the number of Add Ack responses will be added. If 5G responds with an Add Ack message but the LTE does not respond with a CMP message, the Add AckN.NsaDc.SgNB.Add.Succ counter can be used for measurement. 5.3.4 Action 4: Top N Analysis Then, check whether the problem occurs in top cells or on the entire network. Top cell-level problems: After top N cells with the lowest access success rate and the largest number of access failures are excluded (the value of N depends on the cell scale, for example, 100), if the access success rate of the entire network improves significantly and is basically the same as that before deterioration (or reaches the target value), the problem is defined as a top cell-level problem. Entire network-level problems: If the access success rate of the entire network does not improve after the top N cells with the lowest access success rate and the largest number of access failures are excluded, the problem is an entire network-level problem. When selecting top N cells, exclude the cells with a high access failure rate caused by a small number of access attempts and a small number of access failures. The number of access attempts must be greater than or equal to 50% of the average value based on the traffic level on the live network. Alternatively, the number of access attempts in a single cell within an hour must be greater than or equal to 1000. For a top cell-level problem, if operation causes are excluded, check whether the top N cells are distributed regularly based on the geographical location. If the problem is related to transmission, check whether the transmission topology of the site where the top N cells are located is regular. For example, check for the joint connection to a transmission subnode. 2023-05-02 Huawei confidential. No spreading without permission. 68 5G Troubleshooting Guide – NSA Access INTERNAL 5.3.5 Action 5: Associated KPI Analysis If the failure is caused by the air interface according to the isolation result of traffic statistics counters, you need to analyze the changes of counters related to the air interface channel quality and network load. Performing analysis of associated KPIs can directly or indirectly prove network changes and differences. Confirm network limitation factors through comparison of associated KPIs of different networks. Rapidly locate root causes or complete initial isolation through coupling analysis of target KPIs and associated KPIs. Common associated KPIs are as follows: number of UEs, uplink interference, PRB usage, cell status, uplink and downlink bit error rates (BERs), and average CQI. Number of UEs: The number of UEs indicates the network load. You can analyze it together with the PRB usage and interference level trend. Generally, when the number of UEs increases, the KPIs of a network decrease. The causes are as follows: 1. As the number of UEs increases, the PRB usage increases and the load between neighboring cells increases, which is the main cause. 2. As the number of UEs increases, the number of abnormal UEs on the network may increase. Generally, a small number of abnormal UEs can 指标ID 指标名称 of failures. 指标描述 contribute to a large number 1911816771 N.User.RRCConn.Avg 小区内处于RRC连接态的平均用户数 1911816772 N.User.RRCConn.Max 小区内处于RRC连接态的最大用户数 1911817042 N.User.RRCConn.Max.Operator 小区运营商的最大用户数 1911817043 N.User.RRCConn.Avg.Operator 小区运营商的平均用户数 Average uplink interference The average value is used for comparison. The maximum value, which is related to the network load, is an instantaneous sampling result and therefore is for reference only. If continuous strong external interference exists, the minimum value may be significantly different. The interference granularity of the traffic statistics is large. Therefore, when the traffic statistics indicate that interference exists, you can infer that interference probably exists. However, if the traffic statistics cannot indicate that interference exists, interference may or may not exist. For top N cells, you can further check whether interference exists through U2000 interference monitoring. Average access distance The TA value of random access indirectly reflects the distance between the UE and the base station. The average TA value can be used to reflect UE distribution. The larger the average value is, the more the UEs distributed at the cell edge. In this case, the related channel conditions and KPIs are worse theoretically. Average TA value: (0 x N.RA.TA.UE.Index0 + 1 x N.RA.TA.UE.Index1 + ... + 12 x N.RA.TA.UE.Index12)/(N.RA.TA.UE.Index0 + N.RA.TA.UE.Index0 + ...+ N.RA.TA.UE.Index12) 2023-05-02 Huawei confidential. No spreading without permission. 69 5G Troubleshooting Guide – NSA Access INTERNAL Initial transmission/retransmission bit error The following table lists the counters that indicate the bit errors in the uplink and downlink. When comparing coverage in multiple areas, you can also evaluate the coverage difference between two areas indirectly through QPSK ratios. 2023-05-02 Huawei confidential. No spreading without permission. 70 5G Troubleshooting Guide – NSA Access INTERNAL Downlink IBLER: (N.DL.SCH.QPSK.ErrTB.Ibler + N.DL.SCH.16QAM.ErrTB.Ibler + N.DL.SCH.64QAM.ErrTB.Ibler + N.DL.SCH.256QAM.ErrTB.Ibler)/(N.DL.SCH.QPSK.TB + N.DL.SCH.16QAM.TB + N.DL.SCH. 64QAM.TB + N.DL.SCH. 256QAM.TB) x 100% Downlink retransmission rate: (N.DL.SCH.QPSK.TB.Retrans + N.DL.SCH.16QAM.TB.Retrans + N.DL.SCH.64QAM.TB.Retrans + N.DL.SCH.256QAM.TB.Retrans)/(N.DL.SCH.QPSK.TB + N.DL.SCH.16QAM.TB + N.DL.SCH. 64QAM.TB + N.DL.SCH. 256QAM.TB) x 100% Downlink RBLER: (N.DL.SCH.QPSK.ErrTB.Rbler + N.DL.SCH.16QAM.ErrTB.Rbler + N.DL.SCH.64QAM.ErrTB.Rbler + N.DL.SCH.256QAM.ErrTB.Rbler)/(N.DL.SCH.QPSK.TB + N.DL.SCH.16QAM.TB + N.DL.SCH. 64QAM.TB + N.DL.SCH. 256QAM.TB) x 100% Uplink retransmission rate: (N.UL.SCH.HalfPiBPSK.TB.Retrans + N.UL.SCH.QPSK.TB.Retrans + N.UL.SCH.16QAM.TB.Retrans + N.UL.SCH.64QAM.TB.Retrans + N.UL.SCH.256QAM.TB.Retrans)/(N.UL.SCH.HalfPiBPSK.TB + N.UL.SCH.QPSK.TB + N.UL.SCH.16QAM.TB + N.UL.SCH. 64QAM.TB + N.UL.SCH. 256QAM.TB) x 100% Uplink RBLER: (N.UL.SCH.HalfPiBPSK.ErrTB.Rbler + N.UL.SCH.QPSK.ErrTB.Rbler + N.UL.SCH.16QAM.ErrTB.Rbler + N.UL.SCH.64QAM.ErrTB.Rbler + N.UL.SCH.256QAM.ErrTB.Rbler)/(N.UL.SCH.HalfPiBPSK.TB + N.UL.SCH.QPSK.TB + N.UL.SCH.16QAM.TB + N.UL.SCH. 64QAM.TB + N.UL.SCH. 256QAM.TB) x 100% PRB usage The PRB usage is used to evaluate the network load. High PRB usage indicates greater interference between cells in the uplink and downlink. Downlink PRB usage = N.PRB.DL.Used.Avg/N.PRB.DL.Avail.Avg x 100% Uplink PRB usage = N.PRB.UL.Used.Avg/N.PRB.UL.Avail.Avg x 100% 2023-05-02 Cell status Average CQI Huawei confidential. No spreading without permission. 71 5G Troubleshooting Guide – NSA Access INTERNAL The CQI reflects the downlink channel quality and interference level and is measured by a single codeword. Average CQI: (0 x N.ChMeas.CQI.SingleCW.0 + 1 x N.ChMeas.CQI.SingleCW.1 + ... + 15 x N.ChMeas.CQI.SingleCW.15)/(N.ChMeas.CQI.SingleCW.0 + 指标描述 指标名称 指标ID N.ChMeas.CQI.SingleCW.1 + ... + N.ChMeas.CQI.SingleCW.15) 1911816595 N.ChMeas.CQI.SingleCW.0 单码字全带宽CQI为0的上报次数 1911816596 N.ChMeas.CQI.SingleCW.1 单码字全带宽CQI为1的上报次数 1911816597 N.ChMeas.CQI.SingleCW.2 单码字全带宽CQI为2的上报次数 1911816598 N.ChMeas.CQI.SingleCW.3 单码字全带宽CQI为3的上报次数 1911816599 N.ChMeas.CQI.SingleCW.4 单码字全带宽CQI为4的上报次数 1911816600 N.ChMeas.CQI.SingleCW.5 单码字全带宽CQI为5的上报次数 1911816601 N.ChMeas.CQI.SingleCW.6 单码字全带宽CQI为6的上报次数 1911816602 N.ChMeas.CQI.SingleCW.7 单码字全带宽CQI为7的上报次数 1911816603 N.ChMeas.CQI.SingleCW.8 单码字全带宽CQI为8的上报次数 1911816604 N.ChMeas.CQI.SingleCW.9 单码字全带宽CQI为9的上报次数 1911816605 N.ChMeas.CQI.SingleCW.10 单码字全带宽CQI为10的上报次数 1911816606 N.ChMeas.CQI.SingleCW.11 单码字全带宽CQI为11的上报次数 1911816607 N.ChMeas.CQI.SingleCW.12 单码字全带宽CQI为12的上报次数 1911816608 N.ChMeas.CQI.SingleCW.13 单码字全带宽CQI为13的上报次数 1911816609 N.ChMeas.CQI.SingleCW.14 单码字全带宽CQI为14的上报次数 1911816610 N.ChMeas.CQI.SingleCW.15 单码字全带宽CQI为15的上报次数 5.3.6 Action 6: Operations and External Events Network operations and external events include but are not limited to: Site reconstruction (continuous new site deployment or site disconnection in some periods of time), network adjustment (such as frequency refarming, RF optimization, and parameter modification on the entire network), reconstruction of neighboring NEs (such as LTE NEs, core network, and transmission), major festival activities, launch of new UEs, and version upgrade of mainstream UEs, and changes in operators' tariff policies. These operations and events cause changes in the network topology, UE distribution, and user behavior. As a result, the network load and interference level change, affecting KPIs. For example, capacity expansion, reconstruction, and frequency refarming may cause UE migration. In this case, reduce the load of a frequency or cell to improve KPIs. Otherwise, KPIs deteriorate given that site disconnection and damaged coverage occur due to transmission, power supply, and other problems and the load increases due to UE splitting. In practice, the problem analysis depends on the actual situation. This guide is for reference only. 5.3.7 Action 7: Detailed Analysis of Top Sites or Typical Sites If the root cause cannot be located based on traffic statistics, associated counters, and external events, and top sites exist, collect signaling tracing data of the sites for a period of time and analyze the data together with CHRs. If the problem is not caused by top sites, select several typical sites with more failures and collect data for detailed analysis. If different types of problems exist during traffic statistics cause analysis, select several typical sites for detailed data collection and analysis for each type of problem cause. 2023-05-02 Huawei confidential. No spreading without permission. 72 5G Troubleshooting Guide – NSA Access INTERNAL After obtaining the signaling tracing and CHR data of typical sites, use the FMA to collect statistics on logs and check the failure scenarios. Analyze each failure scenario by referring to chapters 1–4. Analyzing signaling over the standard interface: The following describes how to collect statistics on signaling failures over the standard interface. Use the FMA to open the 4G Uu/S1/X2 signaling or 5G X2 signaling. Rightclick it and choose Diagnosis from the shortcut menu. In the displayed dialog box, select analysis scenarios. The tool automatically processes these scenarios. Figure 5-1 Navigation path to the automatic diagnosis function for standard interface signaling on the FMA The following figure shows the analysis result. The upper part of the window displays the number and percentage of each release scenario. Click a scenario. The lower part displays detailed information about each time and categorized statistics on each key field. 2023-05-02 Huawei confidential. No spreading without permission. 73 5G Troubleshooting Guide – NSA Access INTERNAL Figure 5-2 Result display of the automatic diagnosis function for standard interface signaling on the FMA CHR analysis − For Add rej, you can directly filter L3IChrCuSgnbAddRej in the Type column. You can summarize the cause value carried in the Add Rej message, User TMSI, CallID, CRNTI, and information about the peer anchor site to find the rule. TrackListInfo records some key dbg information about the user rej, which can be used to help further analyze the rej cause. 2023-05-02 Huawei confidential. No spreading without permission. 74 5G Troubleshooting Guide – NSA Access 2023-05-02 INTERNAL − If LTE does not respond with an SGNB_RECFG_CMP message, you can use the Statistic function on the FMA for CHRs and collect DC statistics by using L3ChrCuAbnormalRel. − CHRs related to random access Huawei confidential. No spreading without permission. 75 5G Troubleshooting Guide – NSA Access INTERNAL (1) Use the cell-level CHR event L2CellChrBlkInfo to check whether the preamble is received, whether the RAR is sent, and the CRC result of Msg3. This event is measured every 10 minutes, including both contention-based and non-contentionbased access. You can check the preamble reception, RAR delivery, and Msg3 reception status based on L2_CellCHR_BLK_CCHP_INFO and L2_CELLCHR_BLK_ULSCH_INFO. 2023-05-02 Huawei confidential. No spreading without permission. 76 5G Troubleshooting Guide – NSA Access INTERNAL (2) For UE-level CHRs, check the non-contention-based preamble, RAR, and Msg3 status by referring to sections 4.3.2 to 4.3.3, and Msg4 status by referring to section 4.3.4 "Contention Msg4 Failure." 2023-05-02 Huawei confidential. No spreading without permission. 77 5G Troubleshooting Guide – NSA Access 6 INTERNAL VIP Complaint Handling 6.1 Overview This chapter describes basic methods of complaint handling, involving complaining user information collection, location data collection, and basic problem analysis. It provides guidance for quickly closing complaints. Routine maintenance Perform routine network performance monitoring through daily performance reports, weekly performance reports, performance alarms, and proactive performance care. Perform routine network health check through parameter check, interference check, and user experience test. In this way, problems of the network, some base stations, and some UEs can be detected in time, and hidden troubles that affect the network performance can be eliminated, thereby reducing complaints as much as possible. Complaint handling Routine monitoring and check cannot exclude all problems. Therefore, customer complaints still arise. Generally, adhere to the following principles: − Record the symptoms clearly. It is recommended that you record the original words of the customer to avoid misunderstandings during the subsequent information transmission. In addition, submit a trouble ticket in the iCare system. − Respond to critical customer complaints in a timely manner, and immediately initiate customer visits to resolve and avoid problems, pacify customers, and ease customer complaints. Then, analyze root causes of problems and provide solutions to avoid similar problems. − For the complaint about a general problem, keep the environment intact. Capture sufficient location data, analyze the root cause, provide a solution to problems of the same type as soon as possible, and then close the complaint. Pay enough attention to complaints about general problems to avoid the problem resolution delay. A too long delay will escalate the complaints to critical complaints. − If complaints from multiple customers are received simultaneously, you need to prioritize the complaints and analyze them one by one. − The essence of complaint handling is not to solve individual problems of customers, but to summarize common problems through such individual problems and provide solutions. Customer return visit − 2023-05-02 Customer return visits are mandatory for critical customer complaints. For complaints about general problems that cannot be remotely resolved, customer return visits are Huawei confidential. No spreading without permission. 78 5G Troubleshooting Guide – NSA Access INTERNAL required to collect more information for problem locating, find out the root causes, and provide solutions. − For problem complaints that are remotely solved, pay telephone return visits based on the problem importance and customer importance to confirm that the problems are solved. 6.2 Complaining User Information Collection By categorizing the complaints, the operator first filters out the problems about network quality, and then provides the detailed information about the complaint user, including time, place, problem description, mobile phone number of the complaining user, complaint type, and estimated time for a solution. The problems can be better located and analyzed if such information as the types of complaining UEs, the users' activity range and habits, types of the users' business and service packages. The IMSI and serving cell are mandatory for user complaint analysis. Engineers need to further obtain them based on the information provided by the customer. The IMSI can be obtained from the mobile number (MSISDN) or the customer. The serving cell information can be obtained in either of the following ways: Based on the geographic location information of the complaining user, engineering parameters, and maps, find the cells closest to the user. Then, use these cells as possible serving cells and collect data related to complaint analysis. If the Nastar is deployed at a site, you can use the VIP analysis function to query the list of cells where the user has accessed the network, as shown in the following figure. 6.3 User Complaint Analysis Methods Historical data analysis For sites where the Nastar has been deployed, preferentially use the complaint analysis support function on the Nastar to analyze historical data. The Nastar must be deployed at a site to support complaint handling. The complaints must be related to accessibility, retainability, and mobility. In addition, the Nastar must subscribe to data of eNodeBs in the complaint area. Then, you can use the complaint analysis support function on the Nastar. If customer complaints are generated, the Nastar analyzes the known problematic cell and the IMSI and MSISDN about complaining user, and quickly retrieves the user's call records related to the problematic cell. This function helps locate and resolve complaints by analyzing causes for call setup failures, handover failures, and abnormal service drops. The complaint analysis support function on the Nastar can display the historical signaling procedures, uplink RSRP/SINR, and release causes of a user on the interface and provide real-time data collection and analysis. If the problem persists with the help of historical data, collect data in real time to obtain related location data. 2023-05-02 Huawei confidential. No spreading without permission. 79 5G Troubleshooting Guide – NSA Access − INTERNAL eNodeB/gNodeB data collection Virtual UE trace and CellDT − EPC data collection (Huawei EPC) − EPC data collection (non-Huawei EPC) − If the EPC is not provided by Huawei, the customer needs to coordinate the MME of another vendor to create and provide a trace reference number based on the IMSI. If the trace reference number cannot be provided, real-time user monitoring cannot be performed. The interconnection with the MMEs of Ericsson, Cisco, and Nokia Siemens Networks for end-to-end signaling tracing has been completed Remarks: The number of tracing or monitoring tasks and the maximum tracing duration are limited. The specific limitation depends on the type and version of the U2020. For details, see the U2020 online help. Problem reproduction You are advised to perform the following operations for problem reproduction: − When the problem is reproduced, the scenario must be consistent with that when the user complains about the problem, including the time, location, UE, and service type. − Record related test data and trace data, including the UE logs, real-time tracing of U2020 users, CellDT, and one-click logs after problem reproduction. 6.4 Complaints Analysis View the exception cause, release cell, and release time. The prerequisite for complaint analysis support is that the network side (mainly the core network side) obtains the user identity information (such as IMSI/IMEI) through the user authentication procedure. The authentication procedure is triggered after the RRC connection is set up. If the RRC connection setup on the LTE side fails, the network side cannot obtain the user identity information, and the Nastar cannot obtain the corresponding call records for complaint analysis support. Therefore, the Nastar cannot analyze access failures in the LTE RRC phase. Signaling analysis For failure signaling in virtual UE tracing or on the Nastar, refer to the introduction in chapter 4 and analyze the root cause based on signaling before and after the current one Geographic display The geographic display shows the cells where UEs fail to access the network and provides the network topology. It can be combined with the air interface quality and signaling to help find the root cause. CHR analysis on the base station Based on the preceding signaling, obtain the time when the UE experiences an exception, TMSI/C-RNTI information, and 4G/5G cell information. Then, obtain the CHR logs of the corresponding site on the base station side for further analysis. For detailed operations, see CHR analysis operations of accessibility problems in chapter 4. In the following figure, you can find the time when a UE encounters an exception in CHRs. View the TMSI and C-RNTI, and get matched to those of VIP users. 2023-05-02 Huawei confidential. No spreading without permission. 80 5G Troubleshooting Guide – NSA Access INTERNAL Retesting and detailed signaling tracing If the root cause cannot be found through CHR analysis and the problem persists, collect real-time tracing data for further analysis. The real-time tracing data related to accessibility includes UE logs, Uu/S1/X2 signaling tracing, and CellDT tracing (No. 1). 2023-05-02 Huawei confidential. No spreading without permission. 81