LTE THROUGHPUT TROUBLESHOOTING GUIDELINE 1. How to use this Guideline.............................................................................................................................................2 2. Ask yourself and the customer......................................................................................................................................3 3. Throughput Considerations...........................................................................................................................................3 4. 5. 6. 3.1. Data Rate......................................................................................................................................................3 3.2. HARQ based outer-loop................................................................................................................................4 Isolation and Troubleshooting.......................................................................................................................................4 4.1. Between the Ue and eNodeB:......................................................................................................................4 4.2. Between eNodeB and MPG:.........................................................................................................................4 Basic Health Checks.....................................................................................................................................................5 General Checks outside of the node.............................................................................................................................5 6.1. Traceroute from PC attached to Ue to media-server....................................................................................5 6.2. CPRI cables..................................................................................................................................................5 6.3. Ue and RF conditions...................................................................................................................................6 6.4. IP Settings.....................................................................................................................................................6 6.5. General Checks inside eNodeB:...................................................................................................................7 6.6.1. 6.6. Check MAC and RLC Settings:....................................................................................................................7 Case specific trouble shooting and traces in eNodeB..................................................................................8 6.6.1. Traces Summary DL:....................................................................................................................................8 6.6.2. Trace Validation DL:......................................................................................................................................8 6.6.3. Traces Summary UL:....................................................................................................................................8 6.6.3.1. Traces to see DTX, TA, RxPower PRB allocation RankIndication (RI) :.................................................8 6.6.3.2. Trace UL grant:........................................................................................................................................9 6.6.3.3. The trace below shows if there is enough data in the buffer to be scheduled:.......................................9 6.6.3.4. The trace below shows the reasons for packets discards by eNB:........................................................9 6.6.3.5. RxPower at eNB (too high or too low?):..................................................................................................9 6.6.3.6. GINR on DL:..........................................................................................................................................10 6.6.3.7. NACKs received from UE:.....................................................................................................................10 6.6.3.8. UL Scheduler Checks:...........................................................................................................................12 6.6.3.9. Other Checks:........................................................................................................................................12 6.6.3.9.1. CRC errors detected by UL L1.........................................................................................................12 6.6.3.9.2. The following pmCounters may be used to identify a low data rate from core to eNB:..................13 6.6.4. Troubleshooting on MPG (2010B CP04):...................................................................................................14 References.............................................................................................................................................................................15 1. PLM LTE Wiki Page....................................................................................................................................................15 2. Vzn LTE-SAE Throughput Troubleshooting:...............................................................................................................15 How to use this Guideline Page 1 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE Slow Throughput issue reported One eNodeB No Subs affected? Many or All How many eNodeB s affected ? Multiple eNodeBs Commo n point of failure? Yes Isolate and trou blshoot equipmen t Yes Engage other Core Groups No All eNodeBs No Subs affecte d? Multiple Is this considere d an Outage? No Check eNodeB for issues One Implement troubl shooting guideli ne Check the Ue config or laptop config Figure 1: Troubleshooting Flow Chart Page 2 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE Ask yourself and the customer 1. Is the issue happening with 1 Ue or multiple Ue’s? 2. If multiple Ue’s is it the same vendor/model/rev of Ue? 3. Is the issue happening at 1 enodeB, several enodeB’s or all enodeB’s? (traceroute from SGW S1-U context to enodeB S1 interface IP can help determine what network devices are common) 4. What is the observed download/upload speeds and what is expected? 5. How is download/upload speeds determined? (ftp, speed test, other) 6. When was the last time the throughput was observed to be ok? 7. What has changed since the last time throughput was observed to be ok? 8. Does the issue occur during certain times of day (i.e. busy hour) 9. Does the issue occur during certain times of the week (i.e. During local event) 1. Throughput Considerations The UE estimates SINR (Signal to Interference Noise Ratio) based on the PSD (Power Spectral Density) of the downlink Reference Signals (RS) and PSD offset between Physical Downlink Shared Channel (PDSCH) and RS. The SINR is converted to Channel Quality Indicator (CQI) and reported to the RBS in the Channel Feedback Report (CFR). The CQI indicates the radio quality, and is used by the link adaptation function to select the transport format matching the channel conditions. This leads to improved radio resource use. CFRs contain CQI, and Rank Indicator (RI). RI is used only when a Multiple Input Multiple Output (MIMO) channel is present. CFRs are transmitted when the RBS triggers one over the Physical Uplink Shared Channel (PUSCH) based on downlink data activity and the age of the earlier received CFR. The RBS performs an adaptive adjustment of the SINR derived from CQI to compensate for errors and mismatches, and fulfills the targeted operating point. Mapping of CQI to MCS (Modulation Coding Schemes) is performed by DL Link Adaptation, Included are guiding tables as defined in 3GPP (36213-920). Since UE will report lower CQI values when using MIMO as opposed to SIMO in same RF environment (SINR), UE will typically use lower Modulation/MCS. This is due to that the UE will take the inter-stream interference into account when reporting CQI and for SIMO transmission no such interference will exist and the CQI will typically be higher. For example when the UE is configured in SIMO it will report CQI 15 but when it is configured in MIMO it will report CQI 10. Note, further fine tuning of MCS’s is performed with HARQ based Outer Loop Link Adaptation (maintain a certain Block Error Rate). 1.1. Data Rate The transport format for the transmission, and the required set of PRB (Physical Resource Block) resources out of the candidate set, are determined based on the channel measurements provided by the UE in CQI reports and the available number of bits. Link adaptation will use the smallest amount of PRBs that can empty the buffer. If the source (SGW and northbound all the way up to the server) is not able to keep the DL RLC buffers full, primarily the transmission bandwidth (number of PRBs) will be decreased and not the MCS. Most common reason for not keeping up data rates is: TCP and UDP configuration in client and server. IP configuration in the network, like package size, fragmentation, etc. Transport network problems like routing, lack of bandwidth Page 3 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE 1.2. HARQ based outer-loop The HARQ outer-loop will adjust the CQI, based on HARQ ACK and NACK, to Link Adaptation in order to maintain same BLER after first transmission. For example if UE is reporting to good CQI when running the MIMO this feature will see this is too many NACK’s which will trigger decrease of CQI. Therefore BLER should be same for SIMO and MIMO. Change (decreases) is in the Transport Block Size (TBS). 2. Isolation and Troubleshooting As packets travel from the Ue towards the MPG and there are more than one device in common in the path. Isolation of the problem can help to quickly find the problematic device. There are several areas where we can observe low throughput in the EPS network: 1. Between Ue and enodeB 2. Between enodeB MPG 3. Between MPG and Internet Note: Please note that MME is handling the signaling part of the network and does not have direct contribution to throughput. 2.1. Between the Ue and enodeB: This segment of the network is the air interface between the mobile and the enodeB. Various tools can be used at the Ue to capture the messaging and RF conditions between the Ue and the enodeB. If these tools are not available a good place to start is with a Wireshark sniffer capture on the laptop connected to the Ue. One of the tools is LLDM with LG phone. The tool to decode the LLDM logs can be downloaded from: \\erinas1.lmc.ericsson.se\LMCY\yr\Integration\LTE\MPCS\Mops and Workshops\UE 2.2. Between enodeB and MPG: This segment of the network is made up of routers and switches and may use third party transport products to carry the traffic. We can ping and traceroute on the enodeB and MPG to help isolate the devices in this segment. If a sniffer capture is possible that will also help. A sniffer capture at the enodeB and at the MPG is ideal. Whether or not a sniffer capture is available, use the steps below to troubleshoot this segment of the network and help determine the problematic device: 1. Traceroute from the SGW VRF-S1-U context to the enodeB S1 interface IP 2. From the SGW VRF-S1-U context ping the S1 interface IP of the enodeB 3. Check for delay on the ping response. 4. If the delay is large, backup 1 hop in the traceroute list and ping again. Keep doing this until the ping response time is ok. This will isolate the link that is introducing the delay. 5. Ping with different size packets see how large the packet can be before it fails. The MTU setting in the network is 1500 so a ping of 1470 should work fine. 6. If a large packet ping does not work, there may be MTU size issues. 7. Determine the maximum size packet that can be used to ping the enodeB S1 interface. 8. From the traceroute list back up 1 hop and see if you can ping with larger size packets. 9. Keep backing up through the network to see where the pings start to succeed, this will isolate the device with the incorrect MTU setting. If we perform a packet capture on this network segment below is the list of areas we need to monitor and we should ask for PCAP format of the packet capture which can be analyzed by Wireshark: 1. At Ue 2. S1/X2 on enodeB 3. S1-U on SGW 4. S5/S8 on SGW 5. Any network device in the path 6. S1-MME on the MME (optional*) 7. S11 on the MME (optional*) Page 4 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE 8. S11 on SGW (optional*) Note: The items marked optional above are signaling interfaces and not directly involved in user payload. It may be necessary to sniff on these interfaces if there are issues with QOS or other signaling that is not working properly. 3. Basic Health Checks 1. Check MME for alarms #gsh list_alarms 2. Check SGW for alarms #show system alarms 3. Check SGW S1 and S5/S8 ports/interfaces/connections for errors or dropped packets #show port counters detail (from appropriate context) #show ip interface all-context 4. Check SGW eps control packet counters #show eps statistics control sgw 5. Check SGW eps traffic packet counters #show eps statistics traffic sgw 6. Check the SGW to ensure the Ue has the correct bearers and QCI #show eps ue imsi <IMSI> d 7. Check enodeB for alarms >ala 8. Check the S1-MME interfaces on the enodeB >st mme 9. Check enodeB S1 interface for dropped packets >EtHostMo_getPmCounters -h 1 -t 1 10. Check the MME to ensure the Ue has the correct bearer’s setup with the correct QCI and it matches the SGW. #gsh get_subscriber -imsi <IMSI> -dl 2 11. Check the mobility logs and isp logs on the MME # /tmp/DPE_COMMONLOG/isp.log # /tmp/OMS_LOGS/mobility_event_log/ready/ 4. General Checks outside of the node 4.1. Traceroute from PC attached to Ue to media-server 1. Attach Ue 2. Perform a traceroute from PC attached to Ue to media-server, e.g. FTP-server, HTTP-server where throughput is suspected to be bad: In a Windows command prompt: tracert -d 10.74.3.17 3. 4.2. When media-server is located close to EPC core - Round-Trip-Time is around 30-40 milliseconds. CPRI cables Check that CPRI (Common Public Radio Interface) cables are correctly connected (not crossed between sectors). Effect of crossing cables can be transmitting of incorrect Physical Cell Identity (PCI). Correct mounting of CPRI cables: Page 5 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE Figure 2 CPRI Connections 4.3. Ue and RF conditions Requirements for MTU size and TCP Window size settings at laptop 1. MTU size: 1360 2. TCP Window size: 512 Kb See document “Characteristics Requirements for LTE Backhaul” (5/100 56-HSC 105 50/1 Uen F) in CPI for additional details. Optimal RF conditions: 1. RSSI/RSRP: minimum -90 dBm (typical, but SINR is more relevant) 2. SNR: maximum DL 15M (if its MIMO: SINR around 8, For Tx div. around 10); UL 7M (it depends on UL SINR, but approx around 5db DL) 3. CQI: minimum 7 (minimum required for 16QAM) 4. Rank Information: We need to make sure the terminal is in MIMO (no TXD). This information can be obtained by having LLDM logs from LG phone under the coverage of the site. The Tool can be downloaded from this link to decode the LLDM logs: \\erinas1.lmc.ericsson.se\LMCY\yr\Integration\LTE\MPCS\Mops and Workshops\UE 4.4. IP Settings 1. Are there any limitations on the FTP Server? 2. Receiving Window Settings 3. MTU Size Settings: 1360 Page 6 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE 4.5. General Checks inside enodeB: 6.6.1. Check MAC and RLC Settings: O52A7X> get MACConfiguration|DataRadioBearer 111102-11:47:52 172.26.132.57 8.0y ERBS_NODE_MODEL_B_1_53_COMPLETE stopfile=/tmp/20370 ================================================================================= ================================ 681 ENodeBFunction=1,RadioBearerTable=default,DataRadioBearer=1 ================================================================================= ================================ DataRadioBearerId 1 dlMaxRetxThreshold 8 dlPollPDU 4 headerCompression 0 (NULL) pelr 1 rlcMode 0 (AM) tPollRetransmitDl 50 tPollRetransmitUl 50 tReorderingDl 35 tReorderingUl 35 tStatusProhibitDl 25 tStatusProhibitUl 25 ulMaxRetxThreshold 8 ulPollPDU 4 ENodeBFunction=1,RadioBearerTable=default,MACConfiguration=1 ================================================================================= ================================ MACConfigurationId 1 dlMaxHARQTx 4 dlPathlossChange 3 tPeriodicBSRTimer 5 tPeriodicPHRTimer 200 tProhibitPHRTimer 200 tTimeAlignmentTimer 5120 ulMaxHARQTx 4 ================================================================================= ================================ Total: 2 Mos In the printout above check Page 7 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE SignalingRadioBearer dlMaxRetxThreshold and ulMaxRetxThreshold - Maximum number of RLC re-transmissions in DL MACConfiguration dlMaxHARQTx and ulMaxHARQTx - If set 1 then HARQ disabled. 4.6. Case specific trouble shooting and traces in enodeB - Are there enough data in the buffer to be scheduled? - Is data being discarded by eNB? - What is the received power at eNB? - What is the GINR (Gain to interface and noise ratio) on the DL? - Are there a lot of NACK’s received from the UE? Traces which can be activated to answer above questions: 6.6.1. Traces Summary DL: lhsh gcpu01024 te e trace4 UpcDlMacCeFt_DL_SCHEDULER lhsh gcpu00768 te e all UpDlPdcpPeFt_DISCARD <-- PDCP discards lhsh gcpu00768 te e all UpDlRlcPeFt_DISCARD <-- RLC discards lhsh gcpu00768 te e all UpDlRlcPeFt_RETRANSM lhsh gcpu00768 te e trace1 UpDlL1* <-- Problems on L1, CRC errors, etc lhsh gcpu01024 te e trace1 UpcDlMac* <-- TRAFFIC_ABNORMAL on MAC (scheduler) mtd peek -ta dlRlcPeBl -si UP_DLRLCPE_FI_STATUS_FOR_DL_TRAFFIC_IND <-- peek on the RLC status reports DL lhsh gcpu01024 te e trace4 UpcDlMacCeBl_Fl1Trans lhsh gcpu00768 te e trace1 UpDlL1PeFt_DEADLINE_MISSED lhsh gcpu00768 te e trace1 UpDlMacPeFt_DEADLINE_MISSED lhsh gcpu00768 te e trace1 UpDlRlcPeFt_DEADLINE_MISSED mtd peek -ta ulMacPeBl -signal UP_ULMACPE_CI_UL_L1_MEAS2_DL_IND -pe 2 -rep 65535 mtd peek -ta dlMacPeBl -si UP_DLMACPE_CI_DL_UE_CANDIDATES_IND -di 1 -rep 65535 6.6.2. Trace Validation DL: From a DL UPC perspective, the validatorFO is the software entity that orders the SE's for transmission based on weight. In a HARQ config, the order would be SI - retx - newtx. The following traces can be used to verify the validatorFO behaviour/decision making. lhsh gcpu00768 te e all UpDlRrcCPeBl_Ieic lhsh gcpu01024 te e all UpcDlMacCeFt_UE lhsh gcpu01024 te e all UpcDlMacCeFt_DL_VALIDATION 6.6.3. 4.6.3.1. Traces Summary UL: Traces to see DTX, TA, RxPower PRB allocation RankIndication (RI) : lhsh gcpu00256 te e bus_send UpUlL1PeBlMaster_Ieis lhsh gcpu00256 te e all UpUlPdcpPeFt_DISCARD lhsh gcpu00256 te e all UpUlRlcPeFt_DISCARD lhsh gcpu00256 te e trace1 UpUlMacPeFt_DEADLINE_MISSED lhsh gcpu00256 te e trace1 UpUl* lhsh gcpu01024 te e trace5 bus_send trace1 UpcUlMacCeFt_UL_SCHEDULER Page 8 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE mtd peek -ta ulMacCeBl -si UP_ULMACPE_CI_UL_UE_ALLOC_IND -di OUTGOING mtd peek -ta dlMacPeBl -si UP_DLMACPE_CI_UL_HARQ_ALLOC_IND -di 1 mtd peek -ta ulMacCeBl -si UP_ULMACPE_CI_UL_MAC_CTRL_INFO_IND -di 3 mtd peek -ta ulMacPeBl -signal UP_ULMACPE_CI_UL_L1_MEAS2_UL_IND -pe 2 -rep 65535 lhsh gcpu01024 te e trace1 UpcUlMac* mtd peek -ta dlRlcPeBl -si UP_DLRLCPE_FI_STATUS_FOR_UL_TRAFFIC_IND 4.6.3.2. Trace UL grant: lhsh gcpu01024 te e trace5 trace6 UpcUlMacCeFt_UL_SCHEDULER 4.6.3.3. The trace below shows if there is enough data in the buffer to be scheduled: lhsh gcpu01024 te e trace4 UpcDlMacCeFt_DL_SCHEDULER Example trace output 1970-01-05 13:51:50.776] 0x5326278f /UpcDlMacCeFt_DL_SCHEDULER LEVEL2 cellId=2 subframeNr=8 : Assigned SE PQ: rnti=284 bbUeRef=33554656 PQ(lcid=1 assignableBits=0 minPduSize=56) [1970-01-05 13:51:50.776] 0x53262806 /UpcDlMacCeFt_DL_SCHEDULER LEVEL2 cellId=2 subframeNr=8 : Assigned SE PQ: rnti=284 bbUeRef=33554656 PQ(lcid=2 assignableBits=0 minPduSize=56) [1970-01-05 13:51:50.776] 0x5326287d /UpcDlMacCeFt_DL_SCHEDULER LEVEL2 cellId=2 subframeNr=8 : Assigned SE PQ: rnti=284 bbUeRef=33554656 PQ(lcid=3 assignableBits=3171648 minPduSize=56) Note: lcid=1, lcid=2 are signaling data lcid=3 is the actual data in the buffer 4.6.3.4. The trace below shows the reasons for packets discards by eNB: lhsh gcpu00768 te e trace1 UpDlRlcPeFt_DISCARD Example trace output [1970-01-02 04:12:06.596] 0xe7067317=(bfn:3696, sfn:624, sf:6.87, bf:49) /UpDlRlcPeFt_DISCARD TRAFFIC_ABNORMAL Received DL_DATA_IND. Free CM 0 bytes is below threshold 300 bytes. Discarding msg. payloadLength=698 bytes incl GTP-U header 4.6.3.5. RxPower at eNB (too high or too low?): lhsh gcpu00256 te e bus_send UpUlL1PeBlMaster_Ieis Example Trace Output (shows -69.5 dBm): UpUlMacPeCiUlL1Meas2UlIndS { sigNo = 430 header { cellId = 12 sfn = 537 subFrameNo = 9 } nrOfPuschReports = 1 nrOfPucchSrReports = 0 totalNrOfReports = 1 Page 9 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE reportList[0] { puschReport { meas2DlUlReportType = 2 padding0 = 0 bbUeRef = 201326912 isDtx { isDtx = 0, padding0 = 0 } rxPower { prbListStart = 1, prbListEnd = 4, rxPowerReport = -695, padding0 = 2908, sinr = 1078027011 } } pucchSrReport { meas2DlUlReportType = 2 padding0 = 0 bbUeRef = 201326912 } } } 4.6.3.6. GINR on DL: lhsh gcpu01024 te e trace4 UpcDlMacCeFt_DL_SCHEDULER Example Trace Output: [1970-01-05 13:51:50.772] 0x532445ec /UpcDlMacCeFt_DL_SCHEDULER LEVEL2 cellId=2 subframeNr=6 : Assigned SE: rnti=284 seSesDataS(seWeight=15695394 nrOfPqs=5 pqWeight={0 0 0 15695394 0} dciFormat=6 timingAlignmentBits=0 tacOctet=65535 nrofCce=8 startCceIdx=16 dlGinrOuterLoopAdj=SQ8_15(0xfff612f0)) Note: A high negative value for dlGinrOuterLoopAdj means that the outer loop adjustment has reduced the output. This will lead to SINR used for calculating the MCS to be always 0. A high negative value for dlGinrOuterLoopAdj could be due to a lot of NACK’s received from the UE on DL transmissions. 4.6.3.7. NACK’s received from UE: Example trace output UpUlMacPeCiUlL1Meas2DlIndS { sigNo = 429 header { cellId = 12 sfn = 537 subFrameNo = 8 } nrOfPuschReports = 0 Page 10 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE nrOfPucchReports = 1 totalNrOfReports = 1 reportList[0] { puschReport { meas2DlUlReportType = 1 padding0 = 0 bbUeRef = 201326912 isDtx { isDtx = 0, padding0 = 0 } dlHarqInfo { dlHarqValid = 1, detectedHarqIndication = 0, dlHarqProcessId = 2, nrOfTb = 2, swapFlag = 0, padding0 = 0 } timingAdvanceError { timingAdvanceError = 0, padding0 = 0 } cfrPusch { ElibBbBaseCfrInfoS cfrInfo { ri = 7, cfrLength = 118, cfrFormat = 1, cfrValid = 0, cfrExpected = 1, cfrCrcFlag = 0, padding0 = 0 }, cfr[] = [0, 0, 0, 0] as hex: [00 00 00 00 00 00 00 00] } } pucchReport { meas2DlUlReportType = 1 padding0 = 0 bbUeRef = 201326912 isDtx { isDtx = 0, padding0 = 0 } dlHarqInfo { dlHarqValid = 1, detectedHarqIndication = 0, dlHarqProcessId = 2, nrOfTb = 2, swapFlag = 0, padding0 = 0 } rxPower { prbListStart = 0, prbListEnd = 0, rxPowerReport = -630, padding0 = 0, sinr = 0 } timingAdvanceError { timingAdvanceError = 0, padding0 = 0 } cfrPucch { ElibBbBaseCfrInfoS cfrInfo { ri = 0, cfrLength = 0, cfrFormat = 0, cfrValid = 0, cfrExpected = 0, cfrCrcFlag = 0, padding0 = 0 }, cfr[] = [0, 0] as hex: [00 00 00 00] } } } } Note: Check which report (pucch or pusch) is valid: nrOfPuschReports = 0 nrOfPucchReports = 1 totalNrOfReports = 1 The above print outs shows pucch report is valid. Page 11 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE Check if dlHarq is valid: dlHarqValid = 1 If dlHarqValid is valid, check detectedHarqIndication: detectedHarqIndication = 4 DTX Two codewords (1 cw/2 cw): detectedHarqIndication = 0 NACK/NACK detectedHarqIndication = 1 NACK/ACK detectedHarqIndication = 2 ACK/NACK detectedHarqIndication = 3 ACK/ACK One codeword: detectedHarqIndication = 0 NACK detectedHarqIndication = 1 ACK 4.6.3.8. UL Scheduler Checks: lhsh gcpu01024 te e trace3 trace4 trace5 trace6 trace7 UpcUlMacCeFt_UL_SCHEDULER lhsh gcpu01024 te e trace1 trace3 trace4 trace7 trace9 UpcUlMacCeFt_UL_VALIDATION lhsh gcpu01024 te e trace3 trace4 trace5 UpcUlMacCeFt_UL_LINKADAPTATION lhsh gcpu01024 te e trace5 UpcUlMacCeBl_SseSession lhsh gcpu01024 te e trace4 UpcDlMacCeFt_DL_SCHEDULER lhsh gcpu00256 te e all UpUlPdcpPeFt_DISCARD lhsh gcpu00256 te e all UpUlRlcPeFt_DISCARD lhsh gcpu00768 te e all UpDlPdcpPeFt_DISCARD lhsh gcpu00768 te e all UpDlRlcPeFt_DISCARD lhsh gcpu00768 te e all UpDlRlcPeFt_RETRANSM lhsh gcpu00768 te e trace1 UpDlL1* lhsh gcpu01024 te e trace1 UpcDlMacCeBl mtd peek -ta ulMacPeBl -signal LPP_UP_ULMACPE_CI_UL_L1_MEAS2_DL_IND -pe 2 -rep 65535 mtd peek -ta ulMacPeBl -signal LPP_UP_ULMACPE_CI_UL_UE_ALLOC_IND -pe 1 -rep 65535 mtd peek -ta ulMacPeBl -signal LPP_UP_ULMACPE_CI_UL_L1_MEAS2_UL_IND -pe 2 -rep 65535 mtd peek -tar ulMacPeBl -sig LPP_UP_ULMACPE_CI_UL_MAC_CTRL_INFO_IND -dir 1 Note! HiCap tracing needed 4.6.3.9. Other Checks: 4.6.3.9.1. CRC errors detected by UL L1 lhsh gcpu00256 te e trace1 UpUl* Example trace output [xxxx-xx-xx xx:xx:xx.xxx] 0x2ff06c39=(bfn:767, sfn:767, sf:0.40, bf:195) ULMA1/UpUlMacPeBl_Smac TRAFFIC_ABNORMAL DEMUX: cellId=7 bbUeRef=0x70000c0 systemFrameNr=766 subframeNr=9: Demux result :MAC PDU received with TB CRC error. TB CRC is equal to 0xe00130, expected = 0xa1915. PDU length=15 bytes (without CRC) mtd peek -ta ulMacCeBl -si UP_ULMACPE_CI_UL_MAC_CTRL_INFO_IND -di 3 harqInfo = 1 (UpUpCommonMacCommonMacCtrlElemHarqFeedbackAck) is the interesting part Page 12 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE [xxxx-xx-xx xx:xx:xx.xxx] 0x82f7f969=(bfn:2095, sfn:47, sf:8.47, bf:150) ULMA0/UpcUlMacCeBl_Imtdi BIN_REC : UP_ULMACPE_CI_UL_MAC_CTRL_INFO_IND (432) <= UNKNOWN (sessionRef=0xfffff f) UpUlMacPeCiUlMacCtrlInfoIndS { sigNo = 432 header { cellId = 7 sfn = 47 subFrameNo = 7 } payloadSize = 17 nrOfUeUlMacCtrlInfo = 1 ueUlMacCtrlInfo[0] { header { sessionRef = 117440512 harqInfo = 1 (UpUpCommonMacCommonMacCtrlElemHarqFeedbackAck) isDtx = 0 prbListStart = 1 prbListEnd = 2 ulHarqProcessId = 5 nrOfSduInfos = 0 nrOfMacCtrlElements = 1 size = 17 } dummyNrOfSduInfos = 0 dummyNrOfMacCtrlElements = 0 macCtrlElementList[0] { type = 7 (UpUpCommonMacCommonMacCtrlElemLongBsr) powerHeadroomReport { type = 7 (UpUpCommonMacCommonMacCtrlElemLongBsr), powerHeadroom = 0 } cRnti { type = 7 (UpUpCommonMacCommonMacCtrlElemLongBsr), crnti = 0 } truncatedBSR { type = 7 (UpUpCommonMacCommonMacCtrlElemLongBsr), bufferSize = 0 } shortBSR { type = 7 (UpUpCommonMacCommonMacCtrlElemLongBsr), bufferSize = 0 } longBSR { type = 7 (UpUpCommonMacCommonMacCtrlElemLongBsr), bufferSizeNr1Nr2 = 0, bufferSizeNr3Nr4 = 0 } } } } 4.6.3.9.2. The following pmCounters may be used to identify a low data rate from core to eNB: pmIfInOctetsLink1Hi [GigaBitEthernet] pmIfInOctetsLink1Lo [GigaBitEthernet] Page 13 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE By sampling them at, e.g. 10 sec intervals, and calculating the difference the low input can be identified. In such cases the operator must be notified that its transport network needs corrections. For more details please see PRIMUS solution SCS1100666: [http://e-support.ericsson.se/reader_iview/ui/eserver.asp?ID=SCS1100666]. 6.6.4. Troubleshooting on MPG (2010B CP04): Check MPG for alarms : > #show system alarms Display alarms on the system. Check S1 interface (enodeB and MME/MPG) and S5/S8 (SGW/PGW) ports/interfaces/connections for errors or dropped packets: > #show interfaces > #show interfaces terse: Display summary information about interfaces. > #show interfaces gc-fpc/pic/port <brief | detail | extensive > none—Display information about all interfaces. gc-fpc/pic/port—Name of a GGSN-C interface. brief—(Optional) Display brief interface information. detail—(Optional) Display detailed interface information. extensive—(Optional) Display very detailed interface information. Similarly, this command can be used to check details on GGSN-U interface. The application alarms can only be reached by Simple Network Management Protocol (SNMP) > #show snmp mib walk 1.3.6.1.4.1.10923.1.1.1 Display the value of all objects of the entire GGSN MIB by using the above command. > #show snmp mib get 1.3.6.1.4.1.10923.1.1.1.1.1.3.2.0 Display the total number of active PDP contexts by using the above command. #show snmp mib walk 1.3.6.1.4.1.10923.1.1.1.1.1.5 Display the statistics for all APN’s, including the indices used for identifying particular APN’s by using the above command. #show snmp mib get 1.3.6.1.4.1.10923.1.1.1.1.1.5.1.3.1 Display the number of active PDP contexts of the particular APN with index 1 by using the above command. Tracing operations provide more detailed information about the operation of the GGSN, its routing-protocols, and services. It is used as a troubleshooting tool rather than as a tool for continuous information collection Tracing can be configured to collect general routing information, interface, protocol, or service specific information. All trace files are created in /var/log unless otherwise specified. The symptoms of a problem in the network are usually quite obvious, such as the failure to reach a remote host. Solution: To identify the symptoms of a problem on the network, start at one end of network and follow the routes to the other end, entering all or one of the following user@host> ping (ip-address | host-name) user@host> show route (ip-address | host-name) user@host> traceroute (ip-address | host-name) Page 14 2/5/2015 LTE THROUGHPUT TROUBLESHOOTING GUIDELINE References : 1. PLM LTE Wiki Page http://lte-plm.rnd.ki.sw.ericsson.se/lte_trsh_wiki/L12A/index.php?n=UseCases.UserThroughputDegraded 2. Vzn LTE-SAE Throughput Troubleshooting LTE-SAE Slow Throughput_Rev_0.1.doc Page 15 2/5/2015