High RSSI Troubleshooting Case 1 – High RSSI on one branch This case is valid when RSSI values on one branch are very high, but the other branch is not at all affected. Values before issue: - Branch 1: -101.13 dBm - Branch 2: -100.72 dBm - Imbalance: 0.41 dBm Values during issue: - Branch 1: -100.55 dBm - Branch 2: -72.37 dBm - Imbalance: 28.18 dBm Values after locking 2nd branch: - Branch 1: -100.43 Case 2 – High RSSI on both branches, imbalance >= 6dBm This case is valid when RSSI values on one branch are very high, and the other branch is also affected, but it can be clearly seen that there is at least 6 dBm difference between the RSSI values of those two. That branch is most likely affected by the issue of the faulty one. If that is the case, the branch with lower RSSI should normalize when the faulty one is locked. Values before issue: - Branch 1: -101.65 dBm - Branch 2: -101.51 dBm - Imbalance: 0.14 dBm Values during issue: - Branch 1: -73.57 dBm - Branch 2: -86.74 dBm - Imbalance: 13.17 dBm Values after locking 1st branch: - Branch 2: -102.36 Case 3 – High RSSI on both branches, imbalance <= 6dBm This case is valid when RSSI values on both branches are high, and the imbalance between them is less than 6 dBm. This as a high probability to be caused by external interference. Values during issue: - Branch 1: -82.79 dBm - Branch 2: -78.27 dBm - Imbalance: 4.53 dBm Values after locking 2nd branch: - Branch 1: -82.62 dBm Case 4 – Intermittent high RSSI spikes This case is valid when RSSI values on either one or both branches have intermittent high RSSI. The spikes can occur randomly or on a regular basis (around same time of day). Random spikes could show hardware problem, whereas spikes that happen during the same time of day could be a sign of external interference, where a device sends data on a regular basis and interferes with Net1 frequency. Troubleshooting procedures* Procedures are divided according to RSSI issue case type (1-4) and eNB installation type (RRH on ground or top of the mast). A. B. C. D. Troubleshooting cases 1, 2, 3 – RRH at ground level Troubleshooting cases 1, 2, 3 – RRH on top of the mast (straight and cross-connect) Troubleshooting case 4 – RRH at ground level Troubleshooting case 4 – RRH on top of the mast (straight <D.I.> and cross-connect<D.II.>) Important! - RSSI readings are done in RRH port, no matter in which antenna port it is connected - Before on-site intervention, remember to unlock and measure RSSI - Before any tech intervention, remember to lock the RRH - After each step in the guide below, RSSI values are measured - All RSSI values after each step are recorded to easier understand the issue * All scenarios will be presented with high RSSI issue present on Sector A, branch 2 Installation types RRH at ground level RRH on top – straight, split RRH on top – cross-connect Legend Scenario A - Troubleshooting cases 1, 2, 3 - RRH at ground level 1. Reset RRH There have been cases when resetting RRH restored RSSI to normal values - Select eNB from list -> Right click -> Properties - Go to Equipment view (1) -> Right click on RRH (2) -> Actions -> Reset - Check box (3) -> Click Yes (4) Did the issue clear? - Yes: RRH fault solved by RRH reset - No: continue troubleshooting This is service affecting for the sector for ~2 minutes 2. Lock branch with highest RSSI value Locking worst branch often improves values on the other branch of the sector, or even RSSI values on the other sectors - Select eNB from list -> Right click -> Properties Go to Components view (1) -> Select AntennaPort (2) Uncheck “rxUsed” and “txUsed” boxes (3) Click Apply (4) -> Yes (5) This is service affecting for all sectors for ~2 minutes CpriRadioEquipment (for non-cross-connect eNB): 0 – sector A 1 – sector B 2 – sector C 3. Dispatch technician on site Technicians should take the following tools/spares with them: - 2 x 3 meter jumper cables (this is for RRH on ground level case) - 2 x 1/2”-7/8” connectors - 2 x 1/2“-1 1/4” connectors - Dummy loads (40 W / 46 dBm) - RRH 4. Check bottom connections - Check faulty branch connections if too loose / tight / corroded - Tighten / loosen / replace accordingly - Check the other branch's connections if too loose / tight / corroded - Tighten / loosen / replace accordingly 5. Swap bottom jumpers at RRH-jumper level - Disconnect jumper from RRH port 1 - Disconnect jumper from RRH port 2 - Connect jumper from port 1 to port 2 - Connect jumper from port 2 to port 1 Did the high RSSI values stay on same branch as last step? - Yes: bad RRH – replace RRH - No: continue troubleshooting ! Observation: If troubleshooting continues, the swapped jumpers will remain as they are for the next step 6. Swap bottom jumper at jumper-feeder level From the setup at step 5 (swapped jumpers at RRH-jumper level), now swap the jumpers at jumper-feeder level. Observe that the jumpers are fully swapped (branch 1 jumper on branch 2 and vice versa) - Disconnect jumper from feeder port 1 - Disconnect jumper from feeder port 2 - Connect jumper from port 1 to port 2 - Connect jumper from port 2 to port 1 Did the high RSSI values stay on same branch as last step? - Yes: bad jumper – replace jumper - No: continue troubleshooting 7. Swap back jumpers in initial position (step 4) Or label jumpers accordingly Before / after jumpers’ swap 8. Check top connections - Check faulty branch connections if too loose / tight / corroded - Tighten / loosen / replace accordingly - Check the other branch's connections if too loose / tight / corroded - Tighten / loosen / replace accordingly 9. Swap top jumper at feeder-jumper level - Disconnect jumper from feeder port 1 - Disconnect jumper from feeder port 2 - Connect jumper from port 1 to port 2 - Connect jumper from port 2 to port 1 Did the high RSSI values stay on same branch as last step? - Yes: bad feeder – escalate to Net1 - No: continue troubleshooting ! Observation: If troubleshooting continues, the swapped jumpers will remain as they are for the next step 10. Swap top jumper at jumper-antenna level From the setup at step 9 (swapped jumpers at feeder-jumper level), now swap the jumpers at jumper-antenna level. Observe that the jumpers are fully swapped (branch 1 jumper on branch 2 and vice versa) - Disconnect jumper from feeder port 1 - Disconnect jumper from feeder port 2 - Connect jumper from port 1 to port 2 - Connect jumper from port 2 to port 1 Did the high RSSI values stay on same branch as last step? - Yes: bad jumper – replace jumper - No: continue troubleshooting 11. Swap back jumpers in initial position (step 8) Or label jumpers accordingly Before / after jumpers swap 12. Install dummy load on bad branch Dummy load installation should give a value of ~-104 dBm, considering there is no loss / hardware imperfections RSSI values should read OK at this point, as all parts up to dummy load were checked. Possible issues: - Faulty antenna - External interference Escalate to Net1 for further troubleshooting Scenario B - Troubleshooting cases 1, 2, 3 - RRH on top of the mast a. Straight connection 1. Reset RRH Same procedure as in scenario A 2. Lock branch with highest RSSI value Same procedure as in scenario A 3. Dispatch technician on site Technicians should take the following tools/spares with them: - Dummy loads (40 W / 46 dBm) - RRH - 20 meters jumper (if available) 4. Check jumper connection in RRH - Check faulty branch connections if too loose / tight / corroded - Tighten / loosen / replace accordingly - Check the other branch's connections if too loose / tight / corroded - Tighten / loosen / replace accordingly 5. Swap jumpers at RRH-jumper level Same jumper swap procedure as in scenario A Did the high RSSI values stay on same branch as last step? - Yes: bad RRH – replace RRH - No: continue troubleshooting ! Observation: If troubleshooting continues, the swapped jumpers will remain as they are for the next step 6. Swap jumper at jumper-antenna level From the setup at step 5 (swapped jumpers at RRH-jumper level), now swap the jumpers at jumper-antenna level. Observe that the jumpers are fully swapped (branch 1 jumper on branch 2 and vice versa) - Disconnect jumper from feeder port 1 - Disconnect jumper from feeder port 2 - Connect jumper from port 1 to port 2 - Connect jumper from port 2 to port 1 Did the high RSSI values stay on same branch as last step? - Yes: bad jumper – replace jumper - No: continue troubleshooting 7. Swap back jumpers in initial position (step 4) Or label jumpers accordingly Before / after jumpers swap 8. Install dummy load on bad branch RSSI values should read OK at this point, as all parts up to dummy load were checked. Possible issues: - Faulty antenna - External interference Escalate to Net1 for further troubleshooting Scenario B - Troubleshooting cases 1, 2, 3 - RRH on top of the mast b. Cross-connected jumpers 0. Cross-connections RRH1 port1 –> Antenna 1 port 1 RRH1 port2 –> Antenna 2 port 2 Cross-connect solution is used in difficult to access sites for redundancy reasons. One RRH can be lost without losing coverage for one full sector. RRH2 port1 –> Antenna 2 port 1 RRH2 port2 –> Antenna 3 port 2 Example of cross-connect redundancy RRH3 port1 –> Antenna 3 port 1 RRH3 port2 –> Antenna 1 port 2 - RRH1 is lost (fiber to RRH / RRH issue) - Antenna 1 still up and running due to connection with RRH3 port 2 - Antenna 2 loses DL capacity due to connection with faulty RRH1 port 2 Cross-connect redundancy trades capacity for two sectors for the gain of full coverage in case of RRH fault Sites with cross-connect (all high-masts, RRH on top) DK_BHO3006 SE_DAL2054 SE_DAL2073 SE_GOT1067 SE_JAM2109 SE_KAL4680 SE_KRO4007 SE_NOR2290 SE_NOR2305 SE_NOR2315 SE_NOR4677 SE_OST4003 SE_VAB2237 SE_VAG1007 SE_VAL2077 If high RSSI on –> Reset Cell A port 1 –> RRH 1 Cell A port 2 –> RRH 3 Cell B port 1 –> RRH 2 Cell B port 2 –> RRH 1 Cell C port 1 –> RRH 3 Cell C port 2 –> RRH 2 1. Reset RRH Same procedure as in scenario A When resetting RRH, keep in mind the following 2. Lock branch with highest RSSI value Same procedure as in scenario A When locking branch, keep in mind the following If high RSSI on -> Lock Cell A port 1 –> CPRI 0 AntPort 1 Cell A port 2 –> CPRI 2 AntPort 2 Cell B port 1 –> CPRI 1 AntPort 1 Cell B port 2 –> CPRI 0 AntPort 2 Cell C port 1 –> CPRI 2 AntPort 1 Cell C port 2 –> CPRI 1 AntPort 2 3. Dispatch technician on site Technicians should take the following tools/spares with them: - Dummy loads (40 W / 46 dBm) - RRH - 20 meters jumper (if available) 4. Check jumper connection in RRH - Check faulty branch connections if too loose / tight / corroded - Tighten / loosen / replace accordingly - Check the other branch's connections if too loose / tight / corroded - Tighten / loosen / replace accordingly 5. Swap jumpers at RRH-jumper level Same jumper swap procedure as in scenario A Did the high RSSI values stay on same branch as last step? - Yes: bad RRH – replace RRH - No: continue troubleshooting ! Observation: If troubleshooting continues, put back the jumpers as they were initially If high RSSI on –> Swap jumpers in Sector A port 1 –> RRH 1 Sector A port 2 –> RRH 3 Sector B port 1 –> RRH 2 Sector B port 2 –> RRH 1 Sector C port 1 –> RRH 3 Sector C port 2 –> RRH 2 6. Swap jumper at jumper-antenna level From the setup at step 5 (swapped jumpers at RRH-jumper level), now swap the jumpers at jumper-antenna level. Observe that the jumpers are fully swapped (branch 1 jumper on branch 2 and vice versa) - Disconnect jumper from feeder port 1 - Disconnect jumper from feeder port 2 - Connect jumper from port 1 to port 2 - Connect jumper from port 2 to port 1 Did the high RSSI values stay on same branch as last step? - Yes: bad jumper – replace jumper - No: continue troubleshooting ! Observation: If troubleshooting continues, put back the jumpers as they were initially ! Important for cross-connected sites ! When swapping at RRH-jumper level Issue moves from –> To Sector A port 1 –> Sector B port 2 Sector A port 2 –> Sector C port 1 When swapping at jumper-antenna level Issue moves from –> To Sector A port 1 –> Sector A port 2 Sector A port 2 –> Sector A port 1 Sector B port 1 –> Sector C port 2 Sector B port 2 –> Sector A port 1 Sector B port 1 –> Sector B port 2 Sector B port 2 –> Sector B port 1 Sector C port 1 –> Sector A port 2 Sector C port 2 –> Sector B port 1 Sector C port 1 –> Sector C port 2 Sector C port 2 –> Sector C port 1 7. Swap back jumpers in initial position (step 4) 8. Install dummy load on bad branch RSSI values should read OK at this point, as all parts up to dummy load were checked. Possible issues: - Faulty antenna - External interference Escalate to Net1 for further troubleshooting Scenario C - Troubleshooting case 4 - RRH at ground level Due to the intermittent nature of the issue, these cases require at least 2 dispatches in order to find the fault. Most of the time the issue will not be present, so there is a high chance it will not be when the techs are on site. In some cases, the high RSSI spikes come at regular time intervals (usually same time of day). This indicates that it’s a possible external interference, but it is also easier to predict next spike, which helps troubleshooting decisions. Example of predictable RSSI spikes: Example of unpredictable RSSI spikes: 1. Identify faulty branch and reset RRH Same procedure as in scenario A, B for RRH reset. To identify the faulty branch, simply 2. Monitor RSSI for the next pre-determined interval, OR until next RSSI spike The interval is calculated based on the longest time interval between 2 spikes, as in the example below. If the interval 3 days, the issue should be monitored 3 x highest interval (9 days in this case). For lower intervals, the monitor period is calculated in the same way, for at least 7 days. Is any high RSSI values are observed during the monitor period? No: case solved by RRH reset or issue source not present any more Yes: continue troubleshooting 3. Dispatch technician on site Technicians should take the following tools/spares with them: - 2 x 3 meter jumper cables - 2 x 1/2”-7/8” connectors - 2 x 1/2“-1 1/4” connectors 4. Replace bottom jumper + jumper-feeder connector This is done to eliminate the issue in jumper and jumper-feeder connector 5. Swap RRH running the faulty branch with another RRH on site In the example case, RRH swap is done between current RRH 1 and current RRH 2. The RRHs will not be moved from the position in the rack mounting. The swap is done by swapping the following: Step a – Fibers in d2U (eCCM) between port 1 and port 2. By doing that RRH 1 becomes RRH2 and vice versa (on the logical level, RRH will identify its number according to the eCCM port it is connected to) After Step a Step b – Swap both jumpers between RRHs at RRH –jumper level Jumper in RRH/port –> Goes to RRH 1, port 1 –> RRH 2, port 1 RRH 1, port 2 –> RRH 2, port 2 RRH 2, port 1 –> RRH 1, port 1 RRH 2, port 2 –> RRH 1, port 2 This finalizes the RRH swap. If after this step the high RSSI spikes are detected in another sector (in this case it would move from A to B), the fault lies in the RRH. If the issue stays on same sector, issue is further up, starting with feeder. After Step b Step 8 6. Swap jumpers between branches in RRH 1 (former RRH 2) at jumper-feeder level 7. Replace top jumper and top feeder-jumper connector 8. Swap jumpers at jumper-antenna level After step 8, by detecting on which branch the RSSI spike occurs, it can be narrowed down to feeder issue (issue moves on the other branch) or antenna/ext. interference (issue stays on the same branch). Before After !!! Everything must be properly labeled so it can easily be reverted at the next visit !!! 9. Monitor for RSSI spikes. When detected, proceed to next step 10. Escalate to Net1 for decision Step 6 Setup after tech visit “Untangled” RRH1 and RRH2 for better overview “Untangled” RRH1 and RRH2 for better overview Scenario D.I. - Troubleshooting case 4 - RRH on top of the mast (straight) 1. Identify faulty branch and reset RRH Same procedure as in scenario A, B for RRH reset. To identify the faulty branch, simply 2. Monitor RSSI for the next pre-determined interval, OR until next RSSI spike Is any high RSSI values are observed during the monitor period? No: case solved by RRH reset or issue source not present any more Yes: continue troubleshooting 3. Dispatch technician on site Technicians should take the following tools/spares with them: - 2 x 7/16 DIN connectors – these are the connectors at the ends of the jumpers, plugging in the RRH and antenna After Step a 4. Swap RRH running the faulty branch with another RRH on site In the example case, RRH swap is done between current RRH 1 and current RRH 2. The RRHs will not be moved from the position in the rack mounting. Step a – Swap fibers and SFPs between RRH 1 and RRH 2. SFP swap is required as each port uses different wavelength Step b – Swap both jumpers between RRHs at RRH –jumper level Jumper in RRH/port –> Goes to RRH 1, port 1 –> RRH 2, port 1 RRH 1, port 2 –> RRH 2, port 2 RRH 2, port 1 –> RRH 1, port 1 RRH 2, port 2 –> RRH 1, port 2 After Step b 5. Swap jumpers between branches in RRH 1 at jumper-feeder level 6. Replace connectors at the ends of the jumper This is to avoid replacing whole jumper, as for RRH on top solution, jumpers are usually over 20 meters long. Start with connector at RRH end. This is where the work at RRH side is complete and work at antenna side begins. Replace the connector at antenna end. 7. Swap jumpers at jumper-antenna level After step 7, by detecting on which branch the RSSI spike occurs, it can be narrowed down to jumper issue (issue moves on the other branch) or antenna/ext. interference (issue stays on the same branch). !!! Everything must be properly labeled !!! For a better overview Setup after tech visit “Untangled” RRH1 and RRH2 Scenario D.II. - Troubleshooting case 4 - RRH on top of the mast (cross-connect) Thank you!