UTRAN TROUBLE-SHOOTING EXERCISES 1 2 3 4 5 6 7 8 9 Synchronization problem .......................................................................................................................................................................................................... 4 1.1 Symptom: ......................................................................................................................................................................................................................... 4 1.2 Find the fault: .................................................................................................................................................................................................................... 4 1.3 Correct the fault ................................................................................................................................................................................................................ 5 Missing devices ........................................................................................................................................................................................................................ 8 2.1 Symptom: ......................................................................................................................................................................................................................... 8 2.2 Find the fault: .................................................................................................................................................................................................................... 8 2.3 Correct the fault ................................................................................................................................................................................................................ 8 Aal2 problem, missing aal2pathes ............................................................................................................................................................................................ 9 3.1 Symptom: ......................................................................................................................................................................................................................... 9 3.2 Find the fault: .................................................................................................................................................................................................................. 10 3.3 Correct the fault .............................................................................................................................................................................................................. 11 Aal2 problem, locked Aal2Pathes ........................................................................................................................................................................................... 12 4.1 Symptom: ....................................................................................................................................................................................................................... 12 4.2 Find the fault: .................................................................................................................................................................................................................. 12 4.3 Correct the fault .............................................................................................................................................................................................................. 13 Aal2 problem, Aal2 pathes are disabled ................................................................................................................................................................................. 14 5.1 Symptom: ....................................................................................................................................................................................................................... 14 5.2 Find the fault: .................................................................................................................................................................................................................. 14 5.3 Correct the fault .............................................................................................................................................................................................................. 16 Aal2 problem, bad Aal2 routing .............................................................................................................................................................................................. 17 6.1 Symptom: ....................................................................................................................................................................................................................... 17 6.2 Find the fault: .................................................................................................................................................................................................................. 17 6.3 Correct the fault .............................................................................................................................................................................................................. 18 Cell problem, parameter mismatch ......................................................................................................................................................................................... 19 7.1 Symptom: ....................................................................................................................................................................................................................... 19 7.2 Find the fault: .................................................................................................................................................................................................................. 19 7.3 Correct the fault .............................................................................................................................................................................................................. 21 Cell problem, transmission configuration in RNC .................................................................................................................................................................... 22 8.1 Symptom: ....................................................................................................................................................................................................................... 22 8.2 Find the fault: .................................................................................................................................................................................................................. 22 8.3 Correct the fault .............................................................................................................................................................................................................. 24 Cell problem, transmission fault in RXI ................................................................................................................................................................................... 27 9.1 Symptom: ....................................................................................................................................................................................................................... 27 Finn Magnusson 2006-06-26 rev PA1 (1/38) 9.2 Find the fault: .................................................................................................................................................................................................................. 27 9.3 Correct the fault .............................................................................................................................................................................................................. 28 10 Iu Problem, MTP3 link deactivated ..................................................................................................................................................................................... 29 10.1 Symptom: ....................................................................................................................................................................................................................... 29 10.2 Find the fault: .................................................................................................................................................................................................................. 29 10.3 Correct the fault .............................................................................................................................................................................................................. 31 11 Iu Problem, transmission problem ...................................................................................................................................................................................... 32 11.1 Symptom: ....................................................................................................................................................................................................................... 32 11.2 Find the fault: .................................................................................................................................................................................................................. 33 11.3 Correct the fault .............................................................................................................................................................................................................. 34 Finn Magnusson 2006-06-26 rev PA1 (2/38) All Cell/Iub States (strt) RNC2> strt Following 65 sites are up: --------------------------------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS --------------------------------------------------------------------------------------------------------------------------1 2060070 2060070-1/2/3 11111 11111 11111 1111 MS-24-1:vp4 MS-25-1:vp32 1 2060090 2060090-1/2/3 1111 1111 1111 1111 MS-24-1:vp3 1 2030040 2030040-1/2/3 1111 1111 1111 1111 MS-24-1:vp14 1 2065090 2065090-1/2/3 1111 1111 1111 1111 MS-24-1:vp19 1 2060060 2060060-1/2/3 1111 1111 1111 1111 MS-24-1:vp8 MS-25-1:vp31 1 2010090 2010090-1/2/3 1111 1111 1111 1111 MS-24-1:vp9 2 2088030 2088030-1/2/3 1111 1111 1111 1111 ES1-26-1:vp4 2 2090010 2090010-1/2/3 1111 1111 1111 1111 ES1-26-1:vp2 ES1-2-1:vp20 3 2000020 2000020-1/2/3 1111 1111 1111 1111 ES1-26-1:vp43 3 2060030 2060030-1/2/3 1111 1111 1111 1111 ES1-2-1:vp4 ES1-26-1:vp19 ............................. -------------------------------------------------------------------------------------------------------------------------Following 14 sites are totally or partially unavailable: --------------------------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS --------------------------------------------------------------------------------------------------------------------1 2000250 2000250-1/2/3/4/5/6 1111 1111 0000 1111 MS-24-1:vp40 MS-25-1:vp39 2 9000000 9000000-0 L 2 2007040 2007040-1/2/3/4/5/6 1111 1011 1111 1111 ES1-2-1:vp3 ES1-26-1:vp16 2 2060050 2060050-1/2/3 10001 10001 10001 1111 ES1-26-1:vp6 3 2060080 2060080-1/2/3/4/5/6 1111 1000 1111 1111 ES1-2-1:vp6 ES1-26-1:vp17 4 2000290 2000290-1/2/3/4/5/6 0000 0000 0000 1111 ES1-2-1:vp12 ES1-26-1:vp13 4 2000790 2000790-1/2 0000 0000 0011 ES1-2-1:vp7 4 2000090 2000090-1/2/3/4/5/6 11110 11110 0000 1111 ES1-2-1:vp40 ES1-26-1:vp32 4 2009020 2009020-1/2/3/4/5/6 1111 1111 0000 1111 ES1-26-1:vp33 ES1-26-1:vp48 5 2037030 2037030-1/2/3/4/5/6 0000 1111 1111 1111 ES2-2-1:vp7 ES2-2-1:vp8 5 2000220 2000220-1/2/3/4/5/6 1000 1111 1111 1111 ES2-2-1:vp47 ES2-2-1:vp18 5 2110030 2110030-1/2/3 L000 L000 L000 1111 ES2-2-1:vp25 5 2000150 2000150-1/2/3/4/5/6 1101 1101 1101 1111 ES2-2-1:vp43 ES2-2-1:vp32 10 2000300 2000300-1/2/3 L000 1111 1111 1111 ES1-2-1:vp20 ----------------------------------------------------------------------------------------------------------------------- Comments on “strt”: CELL: the first digit is the state of the cell. the three following digits are the state of the common channels (fach/rach/pch). the fifth digit (if present) represents the state of the hsdpa channel (hsdsch). the sixth digit (if present) represents the state of the enhanced uplink channel (eul) CD12: the first digit is the state of nbapCommon the second digit is the state of nbapDedicated Finn Magnusson 2006-06-26 rev PA1 (3/38) the final two digits are the states of NodeSynchTp 1 and 2. States: L means Locked, 0 means Disabled, and 1 means Enabled. If many sites are down, check whether they are all connected to the same AtmPort Possible causes for sites down - nbap down usually means there is a communication problem between the rnc and rbs. could be faulty transmission or faulty configuration. - nodesynchtp down usually means there is a problem in the rnc TU board(s) or synchronisation source. - all common channels down and cell up usually means there is an aal2 problem, check aal2 configuration on both sides as well as end-to-end transmission of aal2path - rach common channel down and cell up usually means there is a problem in the rbs rax board - cell down and nbap up usually means that there is a problem in the rbs hw or configuration mismatch between rbs/rnc (output power, localcellid, etc) - see moshell command “h lazy” for more info. 1 Synchronization problem 1.1 Symptom: all cells down in “str” 1.2 Find the fault: //ALL NODESYNCHTP ARE DOWN RNC10> str ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 10 3012-8/9/0 00000 00000 00000 1100 1 11 3013-1/2/3 0000 0000 0000 1100 1 12 3013-4/5/6 00000 00000 00000 1100 1 13 3013-7/8/9 0000 0000 0000 1100 1 14 3014-0/1/2 00000 00000 00000 1100 1 15 3014-3/4/5 0000 0000 0000 1100 1 16 3014-6/7/8 0000 0000 0000 1100 Finn Magnusson 2006-06-26 rev PA1 (4/38) //THE CLOCKS ARE FREE RUNNING //BECAUSE NO VALID SYNCHRONIZATION REFERENCE IS DEFINED IN SYNCHRONIZATION MO RNC10> get synchroniz ================================================================================================================= 1171 TransportNetwork=1,Synchronization=1 ================================================================================================================= SynchronizationId 1 degradationIsFault 0 (degrNotFault) syncRefActivity i[8] = 1 1 1 1 1 1 1 1 syncRefPriority i[8] = 1 0 0 0 0 0 0 0 syncRefStatus i[8] = 0 0 0 0 0 0 0 0 syncReference [8] = >>> syncReference = Equipment=1,Subrack=MS,Slot=4,PlugInUnit=1,TimingUnit=1,TuSyncRef=1 >>> syncReference = >>> syncReference = >>> syncReference = >>> syncReference = >>> syncReference = >>> syncReference = >>> syncReference = systemClockA 4 (freeRunningMode) systemClockB 4 (freeRunningMode) userLabel ================================================================================================================= Total: 1 MOs RNC10> lst Subrack=MS,Slot=4,PlugInUnit=1,TimingUnit=1,TuSyncRef=1 =================================================================================== Proxy Adm State Op. State MO =================================================================================== 473 1 (UNLOCKED) 0 (DISABLED) Equipment=1,Subrack=MS,Slot=4,PlugInUnit=1,TimingUnit=1,TuSyncRef=1 =================================================================================== Total: 1 MOs 1.3 Correct the fault //ADD A VALID SYNCHRONIZATION REFERENCE, FOR INSTANCE AN OS155 CONNECTING TO THE IU INTERFACE RNC10> lk ranap=iuc =================================================================================== Proxy Adm State Op. State MOs linked to 4505:Ranap=Iuc2300 =================================================================================== Finn Magnusson 2006-06-26 rev PA1 (5/38) 2216 TransportNetwork=1,Aal2RoutingCase=Iu1 4505 0 (DISABLED) RncFunction=1,Ranap=Iuc2300 2306 TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpGlobalTitle=Iuc-1-2300 1994 TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iu1,Aal2PathDistributionUnit=Iu1-2 1995 TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iu1,Aal2PathDistributionUnit=Iu1-1 1993 0 (DISABLED) TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iu1 2307 TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpEntitySet=Iuc-1-2300 2311 0 (DISABLED) TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpApRemote=Iuc-1-2300 1714 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bAp=Iuc-1-2300-3 1718 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bAp=Qaal2-1-2300-12 1720 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iuc-1-2300-3,Mtp3bSr=1 1717 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3,Mtp3bSlItu=1 1716 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3 1719 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iuc-1-2300-3 2273 1 (ENABLED) TransportNetwork=1,NniSaalTp=Iuc-1 1698 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=Iuc-1 2240 1 (ENABLED) TransportNetwork=1,AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc34 =================================================================================== RNC10> get AtmPort=MS-6-1 ================================================================================================================= 2223 TransportNetwork=1,AtmPort=MS-6-1 ================================================================================================================= AtmPortId MS-6-1 availabilityStatus 0 (NO_STATUS) hecCorrectionMode true operationalState 1 (ENABLED) userLabel MS-6-1 uses Subrack=MS,Slot=6,PlugInUnit=1,ExchangeTerminal=1,Os155SpiTtp=1,Vc4Ttp=1 valueOfLastCellWithUnexp Struct{3} >>> 1.vpi = 2 >>> 2.vci = 50 >>> 3.pti = 0 ================================================================================================================= //WE CHOOSE SYNCPRIORITY=2 BECAUSE 1 WAS ALREADY TAKEN RNC10> acc synch addsyncrefresource ================================================================================================================= Proxy MO Action Nr of Params ================================================================================================================= 1171 Synchronization=1 addSyncRefResource 2 Parameter 1 of 2, syncReference (moRef-ManagedObject): Enter mo LDN: Subrack=MS,Slot=6,PlugInUnit=1,ExchangeTerminal=1,Os155SpiTtp=1 Parameter 2 of 2, syncPriority (long): 2 >>> Return value = null Finn Magnusson 2006-06-26 rev PA1 (6/38) ================================================================================================================= Total: 1 MOs attempted, 1 MOs actioned //NOW THE CLOCKS ARE IN LOCKED MODE RNC10> get synchroniz ================================================================================================================= 1171 TransportNetwork=1,Synchronization=1 ================================================================================================================= SynchronizationId 1 degradationIsFault 0 (degrNotFault) syncRefActivity i[8] = 1 2 1 1 1 1 1 1 syncRefPriority i[8] = 1 2 0 0 0 0 0 0 syncRefStatus i[8] = 0 3 0 0 0 0 0 0 syncReference [8] = >>> syncReference = Equipment=1,Subrack=MS,Slot=4,PlugInUnit=1,TimingUnit=1,TuSyncRef=1 >>> syncReference = Equipment=1,Subrack=MS,Slot=6,PlugInUnit=1,ExchangeTerminal=1,Os155SpiTtp=1 >>> syncReference = >>> syncReference = >>> syncReference = >>> syncReference = >>> syncReference = >>> syncReference = systemClockA 2 (lockedMode) systemClockB 2 (lockedMode) userLabel ================================================================================================================= Total: 1 MOs //THE NODESYNCHTP MOs ARE UP RNC10> str ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 10 3012-8/9/0 00000 00000 00000 1111 1 11 3013-1/2/3 0000 0000 0000 1111 1 12 3013-4/5/6 00000 00000 00000 1111 1 13 3013-7/8/9 0000 0000 0000 1111 1 14 3014-0/1/2 00000 00000 00000 1111 1 15 3014-3/4/5 0000 0000 0000 1111 Finn Magnusson 2006-06-26 rev PA1 (7/38) 2 Missing devices 2.1 Symptom: all cells down in “str” 2.2 Find the fault: //CHECK THE TE LOG READ IN MODULE MP RNC10> lh mod1 te log read [2006-07-03 06:23:34.520] rnhCellRoC[36] ../src/RnhCellRoC.cpp:6139 unlock cell, reason : = 16002 (CellNoDrhResources) [2006-07-03 06:23:34.520] rnhCellRoC[37] ../src/RnhCellRoC.cpp:6139 unlock cell, reason : = 16002 (CellNoDrhResources) [2006-07-03 06:23:34.520] rnhCellRoC[38] ../src/RnhCellRoC.cpp:6139 unlock cell, reason : = 16002 (CellNoDrhResources) [2006-07-03 06:23:34.520] rnhCellRoC[39] ../src/RnhCellRoC.cpp:6139 unlock cell, reason : = 16002 (CellNoDrhResources) [2006-07-03 06:23:34.520] rnhCellRoC[40] ../src/RnhCellRoC.cpp:6139 unlock cell, reason : = 16002 (CellNoDrhResources) INFO:rnhCellRoC[cellFroId 36, iubLinkFroId 12] failed to INFO:rnhCellRoC[cellFroId 37, iubLinkFroId 12] failed to INFO:rnhCellRoC[cellFroId 38, iubLinkFroId 12] failed to INFO:rnhCellRoC[cellFroId 39, iubLinkFroId 13] failed to INFO:rnhCellRoC[cellFroId 40, iubLinkFroId 13] failed to //WE SEE THAT MANY CELLS CANNOT BE UNLOCKED DUE TO MISSING DEVICES (DRH = DEVICE RESOURCE HANDLER) //CHECK THE STATE OF THE DEVICES //ALL CC DEVICES ARE LOCKED RNC10> std --------------------------------------------------------------------------DEVICE STATE MOD LNH TYPE DEV/SPM DEV-ID SPM-ID --------------------------------------------------------------------------1 001900/sp1.lnh CC L1 MS/19/CC/1 MS/19/2 1 002000/sp1.lnh CC L1 MS/20/CC/1 MS/20/2 8 002200/sp1.lnh CC L1 MS/22/CC/1 MS/22/2 8 002300/sp1.lnh CC L1 MS/23/CC/1 MS/23/2 --------------------------------------------------------------------------- 2.3 Correct the fault //UNLOCK ALL CCDEVICES Finn Magnusson 2006-06-26 rev PA1 (8/38) RNC10> deb ccdevice ============================================================================================================ Id MO AdministrativeState Result ============================================================================================================ 54 Subrack=MS,Slot=23,PlugInUnit=1,SpbDeviceGroup=1,SpbDeviceSet=Cc,CcDevice=1 1 >>> 101 Subrack=MS,Slot=22,PlugInUnit=1,SpbDeviceGroup=1,SpbDeviceSet=Cc,CcDevice=1 1 >>> 133 Subrack=MS,Slot=20,PlugInUnit=1,SpbDeviceGroup=1,SpbDeviceSet=Cc,CcDevice=1 1 >>> 145 Subrack=MS,Slot=19,PlugInUnit=1,SpbDeviceGroup=1,SpbDeviceSet=Cc,CcDevice=1 1 >>> ============================================================================================================ Total: 4 MOs attempted, 4 MOs set Set. Set. Set. Set. //MOST CELLS CAME UP ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 10 3012-8/9/0 11111 11111 11111 1111 1 11 3013-1/2/3 1111 1111 1111 1111 1 12 3013-4/5/6 11111 11111 11111 1111 1 13 3013-7/8/9 1111 1111 1111 1111 1 14 3014-0/1/2 11111 11111 11111 1111 1 15 3014-3/4/5 1111 1111 1111 1111 1 16 3014-6/7/8 1111 1111 1111 1111 1 17 3014-9/0/1 1111 1111 1111 1111 1 18 3015-2/3/4 1111 1111 1111 1111 3 Aal2 problem, missing aal2pathes 3.1 Symptom: all common channels down in a site (iublink=2) RNC10> str -i 2 Following 1 sites are totally or partially unavailable: ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 2 3010-4/5/6 10001 10001 10001 1111 ---------------------------------------------------------------------------------------------------- Finn Magnusson 2006-06-26 rev PA1 (9/38) 3.2 Find the fault: //IF ALL 3 COMMON CHANNELS FACH/RACH/PCH ARE DOWN IT USUALLY MEAN THAT THERE IS AN AAL2 PROBLEM //THIS IS BECAUSE THESE CHANNELS REQUIRE AN AAL2 PATH FOR CARRYING THE FRAME PROTOCOL //(SINCE THE UTRANCELL MO IS UP, IT PROBABLY MEANS THAT NBAP COMMUNICATION AND RBS HW ARE OK) //FIND OUT THE AAL2 MOS ASSOCIATED WITH THIS SITE RNC10> lk iublink=2$ =================================================================================== Proxy Adm State Op. State MOs linked to 4682:IubLink=2 =================================================================================== 3736 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,UtranCell=30104 3724 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,UtranCell=30105 3712 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,UtranCell=30106 4684 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=2,NbapCommon=1 4687 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=2,NbapDedicated=1 4686 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=2,NodeSynchTp=1 4685 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=2,NodeSynchTp=2 4683 RncFunction=1,IubLink=2,NodeSynch=1 4682 1 (ENABLED) RncFunction=1,IubLink=2 1913 TransportNetwork=1,Aal2RoutingCase=Iub2 1990 TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iub2,Aal2PathDistributionUnit=Iub2 1989 1 (ENABLED) TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iub2 1728 1 (UNLOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub2-1 1727 1 (UNLOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub2-2 1873 1 (ENABLED) TransportNetwork=1,UniSaalTp=10 1871 1 (ENABLED) TransportNetwork=1,UniSaalTp=11 1869 1 (ENABLED) TransportNetwork=1,UniSaalTp=12 1809 1 (ENABLED) TransportNetwork=1,UniSaalTp=7 1807 1 (ENABLED) TransportNetwork=1,UniSaalTp=8 1805 1 (ENABLED) TransportNetwork=1,UniSaalTp=9 2153 1 (ENABLED) TransportNetwork=1,Aal0TpVccTp=5 2152 1 (ENABLED) TransportNetwork=1,Aal0TpVccTp=6 1956 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=10 1955 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=11 1954 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=12 1959 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=7 1958 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=8 1957 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=9 1681 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=2,VpcTp=1,VclTp=vc34 1680 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=2,VpcTp=1,VclTp=vc35 1679 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=2,VpcTp=1,VclTp=vc36 1678 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=2,VpcTp=1,VclTp=vc37 1677 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=2,VpcTp=1,VclTp=vc38 1676 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=2,VpcTp=1,VclTp=vc39 1675 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=2,VpcTp=1,VclTp=vc40 1674 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=2,VpcTp=1,VclTp=vc43 1673 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=2,VpcTp=1,VclTp=vc44 Finn Magnusson 2006-06-26 rev PA1 (10/38) 1672 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=2,VpcTp=1,VclTp=vc45 =================================================================================== Total: 38 MOs //DO A GET ON THE AAL2 MOS : Aal2Ap, Aal2RoutingCase, Aal2PathDistributionUnit, Aal2PathVccTp //WE NOTICE THAT THE Aal2PathDistributionUnit does not have contain any Aal2PathVccTp MOs RNC10> get Aal2PathDistributionUnit=Iub2$ ================================================================================================================= 1990 TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iub2,Aal2PathDistributionUnit=Iub2 ================================================================================================================= Aal2PathDistributionUnitId Iub2 aal2PathVccTpList [0] = rpuId ReliableProgramUniter=ans_aal2cpsrc_12_1 userLabel userLabel ================================================================================================================= 3.3 Correct the fault //ADD THE AAL2PATHES INTO THE AAL2PATHDISTRIBUTIONUNIT //IN THIS CASE, WE DON’T HAVE TO CREATE NEW AAL2 PATHES BECAUSE THEY ALREADY EXIST, IT’S JUST THAT THEY WERE NOT ADDED INTO THE PATHDISTRIBUTIONUNIT RNC10> acc Aal2PathDistributionUnit=Iub2$ addpath ================================================================================================================= Proxy MO Action Nr of Params ================================================================================================================= 1990 Aal2Sp=1,Aal2Ap=Iub2,Aal2PathDistributionUnit=Iub2 addPath 1 Parameter 1 of 1, aal2PathVccTpId (sequence-moRef-Aal2PathVccTp): Enter LDN, or sequence of LDNs (separated by spaces): Aal2PathVccTp=Iub2-1 Aal2PathVccTp=Iub2-2 >>> Return value = null ================================================================================================================= Total: 1 MOs attempted, 1 MOs actioned //NOTE: COULD ALSO BE DONE WITH THE SET COMMAND INSTEAD RNC10> set Aal2PathDistributionUnit=Iub2$ aal2PathVccTpList Aal2PathVccTp=Iub2-1 Aal2PathVccTp=Iub2-2 ============================================================================================================ Id MO aal2PathVccTpList Result ============================================================================================================ 1990 Aal2Sp=1,Aal2Ap=Iub2,Aal2PathDistributionUnit=Iub2 1728,1727 >>> Set. ============================================================================================================ Total: 1 MOs attempted, 1 MOs set Finn Magnusson 2006-06-26 rev PA1 (11/38) 4 Aal2 problem, locked Aal2Pathes 4.1 Symptom: all common channels down in a site (iublink=3) RNC10> str -i 3 Following 1 sites are totally or partially unavailable: ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 3 3010-7/8/9 10001 10001 10001 1111 ---------------------------------------------------------------------------------------------------- 4.2 Find the fault: //IF ALL 3 COMMON CHANNELS FACH/RACH/PCH ARE DOWN IT USUALLY MEAN THAT THERE IS AN AAL2 PROBLEM //THIS IS BECAUSE THESE CHANNELS REQUIRE AN AAL2 PATH FOR CARRYING THE FRAME PROTOCOL //(SINCE THE UTRANCELL MO IS UP, IT PROBABLY MEANS THAT NBAP COMMUNICATION AND RBS HW ARE OK) //FIND OUT THE AAL2 MOS ASSOCIATED WITH THIS SITE RNC10> lk iublink=3$ ................................................................................... =================================================================================== Proxy Adm State Op. State MOs linked to 4676:IubLink=3 =================================================================================== 3700 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,UtranCell=30107 3688 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,UtranCell=30108 3676 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,UtranCell=30109 4678 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=3,NbapCommon=1 4679 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=3,NbapDedicated=1 4681 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=3,NodeSynchTp=1 4680 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=3,NodeSynchTp=2 4677 RncFunction=1,IubLink=3,NodeSynch=1 4676 1 (ENABLED) RncFunction=1,IubLink=3 1911 TransportNetwork=1,Aal2RoutingCase=Iub3 1988 TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iub3,Aal2PathDistributionUnit=Iub3 1987 1 (ENABLED) TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iub3 1701 0 (LOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub3-1 1700 0 (LOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub3-2 1699 0 (LOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub3-3 1867 1 (ENABLED) TransportNetwork=1,UniSaalTp=13 1864 1 (ENABLED) TransportNetwork=1,UniSaalTp=14 Finn Magnusson 2006-06-26 rev PA1 (12/38) 1861 1 (ENABLED) TransportNetwork=1,UniSaalTp=15 1859 1 (ENABLED) TransportNetwork=1,UniSaalTp=16 1858 1 (ENABLED) TransportNetwork=1,UniSaalTp=17 1857 1 (ENABLED) TransportNetwork=1,UniSaalTp=18 2150 1 (ENABLED) TransportNetwork=1,Aal0TpVccTp=10 2151 1 (ENABLED) TransportNetwork=1,Aal0TpVccTp=9 1953 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=13 1952 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=14 1951 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=15 1950 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=16 1949 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=17 1948 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=18 1669 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=3,VpcTp=1,VclTp=vc34 1668 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=3,VpcTp=1,VclTp=vc35 1667 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=3,VpcTp=1,VclTp=vc36 1666 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=3,VpcTp=1,VclTp=vc37 1665 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=3,VpcTp=1,VclTp=vc38 1664 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=3,VpcTp=1,VclTp=vc39 1663 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=3,VpcTp=1,VclTp=vc40 1662 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=3,VpcTp=1,VclTp=vc43 1661 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=3,VpcTp=1,VclTp=vc44 1660 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=3,VpcTp=1,VclTp=vc45 =================================================================================== Total: 39 MOs //WE SEE STRAIGHT AWAY THAT THE AAL2PATHES ARE LOCKED 4.3 Correct the fault //UNLOCK THE AAL2PATHES RNC10> deb aal2pathvcctp=iub3============================================================================================================ Id MO AdministrativeState Result ============================================================================================================ 1699 Aal2PathVccTp=Iub3-3 1 >>> Set. 1700 Aal2PathVccTp=Iub3-2 1 >>> Set. 1701 Aal2PathVccTp=Iub3-1 1 >>> Set. ============================================================================================================ Total: 3 MOs attempted, 3 MOs set //SITE IS NOW UP RNC10> str –i 2 ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 3 3010-7/8/9 11111 11111 11111 1111 ---------------------------------------------------------------------------------------------------Finn Magnusson 2006-06-26 rev PA1 (13/38) 5 Aal2 problem, Aal2 pathes are disabled 5.1 Symptom: all common channels down in a site (iublink=6) Following 1 sites are totally or partially unavailable: ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 6 3011-6/7/8 1000 1000 1000 1111 ---------------------------------------------------------------------------------------------------- 5.2 Find the fault: //IF ALL 3 COMMON CHANNELS FACH/RACH/PCH ARE DOWN IT USUALLY MEAN THAT THERE IS AN AAL2 PROBLEM //THIS IS BECAUSE THESE CHANNELS REQUIRE AN AAL2 PATH FOR CARRYING THE FRAME PROTOCOL //(SINCE THE UTRANCELL MO IS UP, IT PROBABLY MEANS THAT NBAP COMMUNICATION AND RBS HW ARE OK) //FIND OUT THE AAL2 MOS ASSOCIATED WITH THIS SITE RNC10> lk iublink=6$ ................................................................................... =================================================================================== Proxy Adm State Op. State MOs linked to 4658:IubLink=6 =================================================================================== 3592 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,UtranCell=30116 3580 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,UtranCell=30117 3568 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,UtranCell=30118 4659 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=6,NbapCommon=1 4660 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=6,NbapDedicated=1 4663 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=6,NodeSynchTp=1 4662 1 (UNLOCKED) 1 (ENABLED) RncFunction=1,IubLink=6,NodeSynchTp=2 4661 RncFunction=1,IubLink=6,NodeSynch=1 4658 1 (ENABLED) RncFunction=1,IubLink=6 1908 TransportNetwork=1,Aal2RoutingCase=Iub6 1982 TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iub6,Aal2PathDistributionUnit=Iub6 1981 1 (ENABLED) TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iub6 2424 1 (UNLOCKED) 0 (DISABLED) TransportNetwork=1,Aal2PathVccTp=Iub6-1 2423 1 (UNLOCKED) 0 (DISABLED) TransportNetwork=1,Aal2PathVccTp=Iub6-2 1828 1 (ENABLED) TransportNetwork=1,UniSaalTp=31 1826 1 (ENABLED) TransportNetwork=1,UniSaalTp=32 1824 1 (ENABLED) TransportNetwork=1,UniSaalTp=33 1822 1 (ENABLED) TransportNetwork=1,UniSaalTp=34 1820 1 (ENABLED) TransportNetwork=1,UniSaalTp=35 Finn Magnusson 2006-06-26 rev PA1 (14/38) 1819 1 (ENABLED) TransportNetwork=1,UniSaalTp=36 2145 1 (ENABLED) TransportNetwork=1,Aal0TpVccTp=21 2144 1 (ENABLED) TransportNetwork=1,Aal0TpVccTp=22 1935 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=31 1934 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=32 1933 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=33 1932 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=34 1931 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=35 1930 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=36 1633 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc34 1632 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc35 1631 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc36 1630 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc37 1629 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc38 1628 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc39 1627 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc40 1626 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc43 1625 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc44 1624 1 (ENABLED) TransportNetwork=1,AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc45 =================================================================================== Total: 38 MOs //WE SEE STRAIGHT AWAY THAT THE AAL2PATHES ARE DISABLED //WE COMPARE ONE OF THE DISABLED PATH WITH ANOTHER PATH FROM A WORKING SITE RNC10> st aal2path =================================================================================== Proxy Adm State Op. State MO =================================================================================== 1205 1 (UNLOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub4-2 1206 1 (UNLOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub4-1 1207 1 (UNLOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub33-2 1208 1 (UNLOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub33-1 1210 1 (UNLOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub15-2 1211 1 (UNLOCKED) 1 (ENABLED) TransportNetwork=1,Aal2PathVccTp=Iub15-1 RNC10> diff 1205 2423 ===================================================================================================== MO Aal2PathVccTp=Iub4-2 Aal2PathVccTp=Iub6-2 ===================================================================================================== vclTpId AtmPort=MS-26-1,VplTp=4,VpcTp=1,VclTp=vc40 AtmPort=MS-26-1,VplTp=6,VpcTp=1,VclTp=vc40 reservedBy Aal2Sp=1,Aal2Ap=Iub4,Aal2PathDistributionUnit=Iub4 Aal2Sp=1,Aal2Ap=Iub6,Aal2PathDistributionUnit=Iub6 operationalState 1 (ENABLED) 0 (DISABLED) Aal2PathVccTpId Iub4-2 Iub6-2 continuityCheck false true remoteBlockingState 0 (UNDEFINED) 2 (REMOTELY_BLOCKED) availabilityStatus 0 (NO_STATUS) 1056 (DEPENDENCY_FAILED) Finn Magnusson 2006-06-26 rev PA1 (15/38) aal2PathId 440 640 ===================================================================================================== //WE SEE THAT ONE OF THE DIFFERENCES BETWEEN THE WORKING AND THE NON-WORKING AAL2PATH IS THE CONTINUITY CHECK //IF THE CONTINUITY CHECK IS SET DIFFERENTLY ON EACH END OF THE LINK THEN IT CANNOT COME UP 5.3 Correct the fault //CHANGE THE CONTINUITY CHECK //IN THIS CASE THE VALUE USED WAS FALSE BUT IN GENERAL IT IS RECOMMENDED TO USE TRUE ACROSS THE NETWORK RNC10> set Aal2PathVccTp=Iub6 continuitycheck false ============================================================================================================ Id MO continuityCheck Result ============================================================================================================ 2423 Aal2PathVccTp=Iub6-2 false >>> Set. 2424 Aal2PathVccTp=Iub6-1 false >>> Set. ============================================================================================================ //LOCK/UNLOCK THE IUBLINK MOS RNC10> lbl iublink=6, ============================================================================================================ Id MO AdministrativeState Result ============================================================================================================ 4659 IubLink=6,NbapCommon=1 0 >>> Set. 4660 IubLink=6,NbapDedicated=1 0 >>> Set. 4662 IubLink=6,NodeSynchTp=2 0 >>> Set. 4663 IubLink=6,NodeSynchTp=1 0 >>> Set. ============================================================================================================ Total: 4 MOs attempted, 4 MOs set RNC10> ldeb iublink=6, ============================================================================================================ Id MO AdministrativeState Result ============================================================================================================ 4659 IubLink=6,NbapCommon=1 1 >>> Set. 4660 IubLink=6,NbapDedicated=1 1 >>> Set. 4662 IubLink=6,NodeSynchTp=2 1 >>> Set. 4663 IubLink=6,NodeSynchTp=1 1 >>> Set. ============================================================================================================ Total: 4 MOs attempted, 4 MOs set //SITE IS NOW UP RNC10> str –i 6 ---------------------------------------------------------------------------------------------------Finn Magnusson 2006-06-26 rev PA1 (16/38) MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 6 3011-6/7/8 1111 1111 1111 1111 ---------------------------------------------------------------------------------------------------- 6 Aal2 problem, bad Aal2 routing 6.1 Symptom: all common channels down in a site (iublink=5) RNC10> str -i 5 Following 1 sites are totally or partially unavailable: ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 5 3011-3/4/5 10001 10001 10001 1111 ---------------------------------------------------------------------------------------------------- 6.2 Find the fault: //IF ALL 3 COMMON CHANNELS FACH/RACH/PCH ARE DOWN IT USUALLY MEAN THAT THERE IS AN AAL2 PROBLEM //THIS IS BECAUSE THESE CHANNELS REQUIRE AN AAL2 PATH FOR CARRYING THE FRAME PROTOCOL //(SINCE THE UTRANCELL MO IS UP, IT PROBABLY MEANS THAT NBAP COMMUNICATION AND RBS HW ARE OK) //IN THIS CASE, WE CHECKED THE MOS BUT DIDN’T FIND AN OBVIOUS FAULT //WE TRY TO SEE IF THE ANS_AAL2NCC2_PROC TRACE9 CAN GIVE MORE INFORMATION //WE ENABLE IT IN THE MODULE MP OF OUR SITE, IN THIS CASE MOD1 RNC10> lh mod1 te e trace9 Ans_aal2ncc2_proc ============================================================================================= $ lhsh 001400 te e trace9 Ans_aal2ncc2_proc $ RNC10> lh mod1 te log read ============================================================================================= $ lhsh 001400 te log read ..... 0014: [2006-06-14 20:35:36.900] Ans_aal2ncc2_proc outPVfilter.c:417 TRACE9:rejectType = 1,rejectCause = 50331712 (LOCALREJECT_CONFIGURATION_FAULT_NO_MATCH_FOR_AESA) $ Finn Magnusson 2006-06-26 rev PA1 (17/38) //IN THE LOG WE SEE THAT THERE IS NO MATCH FOR AESA, PROBABLY AN AAL2 ROUTING PROBLEM //CHECK THE AAL2ROUTING NUMBER DIRECTION IN RNC AND AESA IN RBS (AAL2SP MO) RNC10> get routingcase=iub5$ ================================================================================================================= 1909 TransportNetwork=1,Aal2RoutingCase=Iub5 ================================================================================================================= Aal2RoutingCaseId Iub5 numberDirection 901520004301999 routeList [1] = >>> routeList = TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iub5 routePriorityList i[1] = 1 userLabel ================================================================================================================= Total: 1 MOs //WE SEE THAT THE NUMBERDIRECTION IS INCORRECT 6.3 Correct the fault //USE THE RSET COMMAND BECAUSE THE ATTRIBUTE IS RESTRICTED RNC10> mom routin numberdir ################################################################################################## MO Class Attribute Type Flags ################################################################################################## Aal2RoutingCase numberDirection string restricted,noNotification,mandatory -------------------------------------------------------------------------------------------------RNC10> rset routingcase=iub5$ numberDirection 901520004301005 .... //SITE IS NOW UP RNC10> str -i 5 Following 1 sites are up: ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 5 3011-3/4/5 11111 11111 11111 1111 ---------------------------------------------------------------------------------------------------- Finn Magnusson 2006-06-26 rev PA1 (18/38) 7 Cell problem, parameter mismatch 7.1 Symptom: All cells and common channels down in one site (iublink=1) RNC10> str -i 1 Following 1 sites are totally or partially unavailable: ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 1 3010-1/2/3 00000 00000 00000 1111 ---------------------------------------------------------------------------------------------------- 7.2 Find the fault: //WE TRY TO LOCK AND UNLOCK THE FAULTY CELLS RNC10> bl cell=3010[123]$ ============================================================================================================ Id MO AdministrativeState Result ============================================================================================================ 3751 UtranCell=30103 0 >>> Set. 3763 UtranCell=30102 0 >>> Set. 3775 UtranCell=30101 0 >>> Set. ============================================================================================================ Total: 3 MOs attempted, 3 MOs set RNC10> deb cell=3010[123]$ ============================================================================================================ Id MO AdministrativeState Result ============================================================================================================ 3751 UtranCell=30103 1 >>> Set. 3763 UtranCell=30102 1 >>> Set. 3775 UtranCell=30101 1 >>> Set. ============================================================================================================ Total: 3 MOs attempted, 3 MOs set //WE CHECK THE TE LOG READ IN MODULE MP RNC10> lh mod1 te log read 0014: [2006-07-03 06:48:00.204] rnhCellRoC[0] ../src/RnhCellRoC.cpp:9232 INFO:rnhCellRoC[cellFroId 0, iubLinkFroId 0]: An entry for localCellId 9991 expected from RBS but not found in nbapAuditRsp message Finn Magnusson 2006-06-26 rev PA1 (19/38) //THIS SEEMS TO SHOW THAT THERE IS A MISMATCH IN THE LOCALCELLID PARAMETER BETWEEN RNC AND RBS //WE PUT THE NBAP/AAL2 TRACE IN THE RBS //IN CASE OF SIMULATED ENVIRONMENT (CORBEN), LOCK ALL NBAPCOMMON EXCEPT THE ONE FOR THAT SITE AND TAKE THE TRACE IN MODULE MP RBS1> te e bus_send bus_receive NBAP* $ te e bus_send bus_receive NBAP* $ RBS1> te e bus_send bus_receive Ans_aal2ap_proc $ te e bus_send bus_receive Ans_aal2ap_proc $ RBS1> te log read | decoder.pl | flow.pl [19:11:17.792] <<---<< [19:11:17.792] <<---<< [19:11:17.800] >>--->> (cell=1003) (maxDLpwr=500) [19:11:17.800] >>--->> (cell=1003) (maxDLpwr=500) (NBAP) (NBAP) (NBAP) AuditRequest (transID=2735L) AuditRequest (transID=2735L) AuditResponse (transID=2735L) (cell=1001) (maxDLpwr=500) (cell=1002) (maxDLpwr=500) (NBAP) AuditResponse (transID=2735L) (cell=1001) (maxDLpwr=500) (cell=1002) (maxDLpwr=500) //WE SEE THAT THE RNC IS NOT EVEN SENDING OUT ANY CELL SETUP REQUEST MESSAGE //USUALLY THIS HAPPENS WHEN THE CELLS ARE REPORTED AS DISABLED BY THE RBS (IN THE AUDIT RESPONSE MESSAGE), OR IF THERE IS SOME MISMATCH BETWEEN THE CELL PARAMETERS IN RNC AND RBS, E.G. MAXTXPOWERDL, LOCALCELLID, ETC. //IN THIS CASE THE LOCALCELLIDs IN RBS ARE 1001, 1002, 1003 //LET’S CHECK WHAT ARE THEIR VALUES IN RNC RNC10> get cell=3010[123] localcell ================================================================================================================= MO Attribute Value ================================================================================================================= UtranCell=30103 localCellId 9993 UtranCell=30102 localCellId 9992 UtranCell=30101 localCellId 9991 ================================================================================================================= Total: 3 MOs //WE SEE THAT THERE IS A MISMATCH BETWEEN THE LOCALCELLID OF THE CELLS IN RNC AND THE CELLS IN RBS //THIS IS WHY THE RNC IS NOT SENDING THE CELL SETUP MESSAGE AND THE CELLS ARE DISABLED Finn Magnusson 2006-06-26 rev PA1 (20/38) 7.3 Correct the fault //SET THE LOCALCELLID TO THE CORRECT VALUE ACCORDING TO NETWORK PLAN RNC10> set cell=30101 localcellid 1001 ============================================================================================================ Id MO localCellId Result ============================================================================================================ 3775 UtranCell=30101 1001 >>> Set. ============================================================================================================ Total: 1 MOs attempted, 1 MOs set RNC10> set cell=30102 localcellid 1002 ============================================================================================================ Id MO localCellId Result ============================================================================================================ 3763 UtranCell=30102 1002 >>> Set. ============================================================================================================ Total: 1 MOs attempted, 1 MOs set RNC10> set cell=30103 localcellid 1003 ============================================================================================================ Id MO localCellId Result ============================================================================================================ 3751 UtranCell=30103 1003 >>> Set. ============================================================================================================ Total: 1 MOs attempted, 1 MOs set //SITE IS NOW UP RNC10> str –i 1 ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 1 3010-1/2/3 11111 11111 11111 1111 ---------------------------------------------------------------------------------------------------- Finn Magnusson 2006-06-26 rev PA1 (21/38) 8 Cell problem, transmission configuration in RNC 8.1 Symptom: All cells and common channels down in one site (iublink=1) RNC10> str -i 4 Following 1 sites are totally or partially unavailable: ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 4 3011-0/1/2 0000 0000 0000 1111 ---------------------------------------------------------------------------------------------------- 8.2 Find the fault: //WE PUT THE NBAP/AAL2 TRACE IN THE RBS //IN CASE OF SIMULATED ENVIRONMENT (CORBEN), LOCK ALL NBAPCOMMON EXCEPT THE ONE FOR THAT SITE AND TAKE THE TRACE IN MODULE MP RBS4> te e bus_send bus_receive NBAP* $ te e bus_send bus_receive NBAP* $ RBS4> te e bus_send bus_receive Ans_aal2ap_proc $ te e bus_send bus_receive Ans_aal2ap_proc $ RBS4> te log read | decoder.pl | flow.pl [19:32:03.312] [19:32:03.312] <<---<< <<---<< (NBAP) (NBAP) AuditRequest (transID=2775L) AuditRequest (transID=2775L) //WE SEE THAT THE RNC KEEPS SENDING AUDIT REQUEST BUT NEVER GETS ANY RESPONSE //THIS POINTS TO SOME KIND OF TRANSMISSION PROBLEM ON NBAP //WE CAN ALSO PUT THE rnhCellRoC and rnhRbsC traces to see UtranCell/IubLinks events RNC10> fro iublink=4$ ================================================================================================================= MO FroName FroId ================================================================================================================= Finn Magnusson 2006-06-26 rev PA1 (22/38) IubLink=4 froId 3 ================================================================================================================= RNC10> fro cell=3011[012]$ ================================================================================================================= MO FroName FroId ================================================================================================================= UtranCell=30112 froId 11 UtranCell=30111 froId 10 UtranCell=30110 froId 9 ================================================================================================================= RNC10> lh mod1 teall rnhRbsC[3] $ lhsh 001400 te e check error enter return info trace1 trace2 trace3 trace4 trace5 trace6 trace7 trace8 trace9 state_change bus_send bus_receive rec_sig send_sig param interface object user1 user2 user3 user4 rnhRbsC[3] RNC10> lh mod1 teall rnhCellRoC[9] $ lhsh 001400 te e check error enter return info trace1 trace2 trace3 trace4 trace5 trace6 trace7 trace8 trace9 state_change bus_send bus_receive rec_sig send_sig param interface object user1 user2 user3 user4 rnhCellRoC[9] RNC10> lh mod1 te log read 0014: [2006-07-03 08:34:05.204] rnhRbsC[3] ../src/RnhRbsC.cpp:4822 ERROR:Failed to remove nbapAuditReq with orderNo=3446 from orderNoTable 0014: [2006-07-03 08:34:05.204] rnhRbsC[3] ../src/RnhRbsC.cpp:1351 ENTER:startAuditReqSendTimer 0014: [2006-07-03 08:34:05.204] rnhRbsC[3] ../src/RnhRbsC.cpp:4877 TRACE3: RnhRbsRoC: audit timeot or asn1failed, message nbapAuditFaultInd sent to RnhCellRoC on portIndex = 0 0014: [2006-07-03 08:34:05.204] rnhRbsC[3] ../src/RnhRbsC.cpp:4877 TRACE3: RnhRbsRoC: audit timeot or asn1failed, message nbapAuditFaultInd sent to RnhCellRoC on portIndex = 1 0014: [2006-07-03 08:34:05.204] rnhRbsC[3] ../src/RnhRbsC.cpp:4877 TRACE3: RnhRbsRoC: audit timeot or asn1failed, message nbapAuditFaultInd sent to RnhCellRoC on portIndex = 2 0014: [2006-07-03 08:34:05.204] rnhRbsC[3] ../src/RnhRbsC.cpp:5858 TRACE5:In nbapAudit state, choicepoint rbsCAuditReady 0014: [2006-07-03 08:34:05.204] rnhCellRoC[9] ../src/target/Cello/RTActor/logMsg.cc:76 REC SIG:Signal:nbapAuditFaultInd, Port:rnhRbsNbapDecodedCellP, Sender:rnhRbsC 0014: [2006-07-03 08:34:05.204] rnhCellRoC[9] ../src/target/Cello/RTActor/logMsg.cc:128 PARAM:Signal:nbapAuditFaultInd, Data:void: 0014: [2006-07-03 08:34:05.204] rnhCellRoC[9] ../src/RnhCellRoC.h:533 TRACE3:rnhCellRoC[cellFroId 9]: signal nbapAuditFaultInd received on port rnhRbsNbapDecodedCellP. Current state WaitForActivation 0014: [2006-07-03 08:34:05.204] rnhCellRoC[9] ../src/RnhCellRoC.h:534 TRACE4:rnhCellRoC[cellFroId 9]: signal data: void 0014: [2006-07-03 08:34:05.204] rnhCellRoC[9] ../src/RnhCellRoC.cpp:1280 ENTER:CreateCellFault(eventNr = 16004, severity = 1) 0014: [2006-07-03 08:34:05.204] rnhCellRoC[9] ../src/RnhCellRoC.cpp:1321 TRACE5:Cancel repetition of faultInd. //THE TRACES SEEM TO POINT TO NBAP FAULT //LET’S CHECK THE PROTOCOL STACK ON NBAP RNC10> llk iublink=4,nbap Finn Magnusson 2006-06-26 rev PA1 (23/38) Command not supported on MO class NbapCommon. Use lko/llko instead. RNC10> llko iublink=4,nbap RNC10> llko iublink=4,nbap 060626-21:36:57 137.58.194.88 6.1zu RNC_NODE_MODEL_G_1_4_COMPLETE stopfile=/tmp/29417 =================================================================================== MOs linked to 4674 RncFunction=1,IubLink=4,NbapDedicated=1 (U,E) =================================================================================== activeUniSaalTpRef IubLink=4,NbapDedicated=1 ---> (-,E) UniSaalTp=19 standbyUniSaalTpRef IubLink=4,NbapDedicated=1 ---> (-,E) UniSaalTp=20 aal5TpVccTpId UniSaalTp=19 ---> (-,E) Aal5TpVccTp=19 uniSaalProfileId UniSaalTp=19 ---> (-,-) UniSaalProfile=win30 aal5TpVccTpId UniSaalTp=20 ---> (-,E) Aal5TpVccTp=20 processorId Aal5TpVccTp=19 ---> (U,E) Subrack=MS,Slot=14,PlugInUnit=1 vclTpId Aal5TpVccTp=19 ---> (-,E) AtmPort=MS-26-1,VplTp=4,VpcTp=1,VclTp=vc36 processorId Aal5TpVccTp=20 ---> (U,E) Subrack=MS,Slot=15,PlugInUnit=1 vclTpId Aal5TpVccTp=20 ---> (-,E) AtmPort=MS-26-1,VplTp=4,VpcTp=1,VclTp=vc43 atmTrafficDescriptorId AtmPort=MS-26-1,VplTp=4,VpcTp=1,VclTp=vc36 ---> (-,-) AtmTrafficDescriptor=U3P1000M80 =================================================================================== MOs linked to 4678 RncFunction=1,IubLink=4,NbapCommon=1 (U,E) =================================================================================== activeUniSaalTpRef IubLink=4,NbapCommon=1 ---> (-,E) UniSaalTp=21 standbyUniSaalTpRef IubLink=4,NbapCommon=1 ---> (-,E) UniSaalTp=22 aal5TpVccTpId UniSaalTp=21 ---> (-,E) Aal5TpVccTp=21 uniSaalProfileId UniSaalTp=21 ---> (-,-) UniSaalProfile=win100 aal5TpVccTpId UniSaalTp=22 ---> (-,E) Aal5TpVccTp=22 processorId Aal5TpVccTp=21 ---> (U,E) Subrack=MS,Slot=14,PlugInUnit=1 vclTpId Aal5TpVccTp=21 ---> (-,E) AtmPort=MS-26-1,VplTp=4,VpcTp=1,VclTp=vc37 processorId Aal5TpVccTp=22 ---> (U,E) Subrack=MS,Slot=15,PlugInUnit=1 vclTpId Aal5TpVccTp=22 ---> (-,E) AtmPort=MS-26-1,VplTp=4,VpcTp=1,VclTp=vc44 atmTrafficDescriptorId AtmPort=MS-26-1,VplTp=4,VpcTp=1,VclTp=vc37 ---> (-,-) AtmTrafficDescriptor=U3P1000M80 =================================================================================== //WE SEE THAT NBAPCOMMON IS USING VC37/44 AND NBAPDEDICATED IS USING VC36/43 //BY CONVENTION, USUALLY THE OPPOSITE IS USED //WE CAN DOUBLE-CHECK ON RBS TO SEE WHAT VC’S ARE USED BY NBAPCOMMON AND NBAPDEDICATED //WHAT IS HAPPENING IS THAT RNC IS SENDING NBAPCOMMON MESSAGES ON THE VC USED BY NBAPDEDICATED ON RBS SIDE, AND VICE-VERSA 8.3 Correct the fault //CHANGE THE PARAMETERS ACTIVEUNISAALTPREF AND STANDBYUNISAALTPREF IN RNC RNC10> u+ Finn Magnusson 2006-06-26 rev PA1 (24/38) RNC10> ldel iublink=4,nbap ================================================================================================================= MO Attribute Value ================================================================================================================= RncFunction=1,IubLink=4,NbapDedicated=1 NbapDedicatedId 1 RncFunction=1,IubLink=4,NbapDedicated=1 activeUniSaalTpRef UniSaalTp=19 RncFunction=1,IubLink=4,NbapDedicated=1 administrativeState 1 (UNLOCKED) RncFunction=1,IubLink=4,NbapDedicated=1 availabilityStatus 0 (NO_STATUS) RncFunction=1,IubLink=4,NbapDedicated=1 l2EstablishReqRetryT 5 RncFunction=1,IubLink=4,NbapDedicated=1 operationalState 1 (ENABLED) RncFunction=1,IubLink=4,NbapDedicated=1 standbyUniSaalTpRef UniSaalTp=20 RncFunction=1,IubLink=4,NbapDedicated=1 userLabel ================================================================================================================= Total: 1 MOs ================================================================================================================= MO Attribute Value ================================================================================================================= RncFunction=1,IubLink=4,NbapCommon=1 NbapCommonId 1 RncFunction=1,IubLink=4,NbapCommon=1 activeUniSaalTpRef UniSaalTp=21 RncFunction=1,IubLink=4,NbapCommon=1 administrativeState 1 (UNLOCKED) RncFunction=1,IubLink=4,NbapCommon=1 availabilityStatus 0 (NO_STATUS) RncFunction=1,IubLink=4,NbapCommon=1 l2EstablishReqRetryT 5 RncFunction=1,IubLink=4,NbapCommon=1 operationalState 1 (ENABLED) RncFunction=1,IubLink=4,NbapCommon=1 standbyUniSaalTpRef UniSaalTp=22 RncFunction=1,IubLink=4,NbapCommon=1 userLabel ================================================================================================================= Total: 1 MOs Deleting ... =================================================================================== 4674 RncFunction=1,IubLink=4,NbapDedicated=1 >>> Mo deleted 4678 RncFunction=1,IubLink=4,NbapCommon=1 >>> Mo deleted =================================================================================== Total: 2 MOs attempted, 2 MOs deleted RNC10> uTo undo, execute command: run /home/eanzmagn/moshell_logfiles/logs_moshell/undo/undo_RNC10_060626-214039.mos RNC10> l cat $undocommandfile #### Undo Script #### confb+ gs+ cr RncFunction=1,IubLink=4,NbapCommon=1 UniSaalTp=21 #activeUniSaalTpRef UniSaalTp=22 #standbyUniSaalTpRef Finn Magnusson 2006-06-26 rev PA1 (25/38) lset RncFunction=1,IubLink=4,NbapCommon=1$ userLabel lset RncFunction=1,IubLink=4,NbapCommon=1$ administrativeState 1 lset RncFunction=1,IubLink=4,NbapCommon=1$ l2EstablishReqRetryT 5 cr RncFunction=1,IubLink=4,NbapDedicated=1 UniSaalTp=19 #activeUniSaalTpRef UniSaalTp=20 #standbyUniSaalTpRef lset RncFunction=1,IubLink=4,NbapDedicated=1$ userLabel lset RncFunction=1,IubLink=4,NbapDedicated=1$ administrativeState 1 lset RncFunction=1,IubLink=4,NbapDedicated=1$ l2EstablishReqRetryT 5 confbgs//WE SWAP THE ATTRIBUTE VALUES OF THE ACTIVE/STANDBY UNISAALTPREF IN THE UNDOCOMMANDFILE, SO THAT NBAPCOMMON USES UNISAALTP=19/20 AND NBAPDEDICATED USES UNISAALTP=21/22 //WE THEN RUN THE UNDOCOMMANDFILE WITH THE RUN COMMAND RNC10> run $undocommandfile RNC10> #### Undo Script #### RNC10> confb+ RNC10> gs+ RNC10> RNC10> cr RncFunction=1,IubLink=4,NbapCommon=1 Attribute 1 of 2, activeUniSaalTpRef (moRef:UniSaalTp): Enter mo LDN: UniSaalTp=19 #activeUniSaalTpRef Attribute 2 of 2, standbyUniSaalTpRef (moRef:UniSaalTp): Enter mo LDN: UniSaalTp=20 #standbyUniSaalTpRef >>> [Proxy ID = 5204] MO name :ManagedElement=1,RncFunction=1,IubLink=4,NbapCommon=1 ........etc................ //SITE IS NOW UP RNC10> str –i 4 ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 4 3011-0/1/2 1111 1111 1111 1111 ---------------------------------------------------------------------------------------------------- Finn Magnusson 2006-06-26 rev PA1 (26/38) 9 Cell problem, transmission fault in RXI 9.1 Symptom: All cells and common channels down in one site (iublink=7) RNC10> str -i 7 Following 1 sites are totally or partially unavailable: ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 7 3114-1/2/3 00000 00000 00000 0111 ---------------------------------------------------------------------------------------------------- 9.2 Find the fault: //WE SEE THAT THE NBAPCOMMON IS DOWN //LET’S CHECK THE STATUS OF THE NBAPCOMMON STACK RNC10> llko iublink=7,nbapc =================================================================================== MOs linked to 805 RncFunction=1,IubLink=7,NbapCommon=1 (U,D) =================================================================================== activeUniSaalTpRef IubLink=7,NbapCommon=1 ---> (-,E) UniSaalTp=b7ca standbyUniSaalTpRef IubLink=7,NbapCommon=1 ---> (-,E) UniSaalTp=b7cb aal5TpVccTpId UniSaalTp=b7ca ---> (-,E) Aal5TpVccTp=b7ca uniSaalProfileId UniSaalTp=b7ca ---> (-,-) UniSaalProfile=win30 aal5TpVccTpId UniSaalTp=b7cb ---> (-,E) Aal5TpVccTp=b7cb processorId Aal5TpVccTp=b7ca ---> (U,E) Subrack=MS,Slot=14,PlugInUnit=1 vclTpId Aal5TpVccTp=b7ca ---> (-,E) AtmPort=MS-26-1,VplTp=vp6,VpcTp=1,VclTp=vc36 processorId Aal5TpVccTp=b7cb ---> (U,E) Subrack=MS,Slot=15,PlugInUnit=1 vclTpId Aal5TpVccTp=b7cb ---> (-,E) AtmPort=MS-26-1,VplTp=vp6,VpcTp=1,VclTp=vc43 atmTrafficDescriptorId AtmPort=MS-26-1,VplTp=vp6,VpcTp=1,VclTp=vc36 ---> (-,-) AtmTrafficDescriptor=U3P1000M80 RNC10> lacc AtmPort=MS-26-1,VplTp=vp6,VpcTp=1,VclTp=vc(36|43) eteloopback ================================================================================================================= Proxy MO Action Nr of Params ================================================================================================================= 2080 AtmPort=MS-26-1,VplTp=vp6,VpcTp=1,VclTp=vc43 eteLoopBack 0 >>> Return value = false 2084 AtmPort=MS-26-1,VplTp=vp6,VpcTp=1,VclTp=vc36 eteLoopBack 0 Finn Magnusson 2006-06-26 rev PA1 (27/38) >>> Return value = false ================================================================================================================= Total: 2 MOs attempted, 2 MOs actioned //WE SEE THAT THE LOOPBACK TEST HAS FAILED //LET’S CHECK THE CROSS-CONNECTS IN RXI RXI1> stc b7 ========================================================================================================================= CCId STATE VclTpA VclTpB Vp/VcA Vp/VcB TD-A TD-B UserLabel ========================================================================================================================= b7sa 11111 MS-26-1/vp6/vc34 MS-3-ima1/vp1/vc34 6/34 1/34 C1P5 C1P5 AtmCrossConnection b7sa b7sb 11111 MS-26-1/vp6/vc35 MS-3-ima1/vp1/vc35 6/35 1/35 C1P5 C1P5 AtmCrossConnection b7sb b7da 11111 MS-26-1/vp6/vc37 MS-3-ima1/vp1/vc37 6/37 1/37 U3P1000M80 U3P1000M80 AtmCrossConnection b7da b7db 11111 MS-26-1/vp6/vc44 MS-3-ima1/vp1/vc44 6/44 1/44 U3P1000M80 U3P1000M80 AtmCrossConnection b7db b7ia 11111 MS-26-2/vp1/vc54 MS-3-ima1/vp1/vc32 1/54 1/32 U4 U4 AtmCrossConnection b7ia b7ib 11110 MS-26-2/vp1/vc55 MS-3-ima1/vp1/vc33 1/55 1/33 U4 U4 AtmCrossConnection b7ib ------------------------------------------------------------------------------------------------------------------------//WE SEE THAT THERE ARE NO CROSS-CONNECTS FOR THE NBAPCOMMON 9.3 Correct the fault //CREATE SOME CROSS CONNECTS IN RXI FOR THE NBAPCOMMON RXI901> cr TransportNetwork=1,AtmCrossConnection=b7ca Attribute 1 of 2, vclTpAId (moRef:VclTp): Enter mo LDN: AtmPort=MS-26-1,VplTp=vp6,VpcTp=1,VclTp=vc36 #vclTpAId Attribute 2 of 2, vclTpBId (moRef:VclTp): Enter mo LDN: AtmPort=MS-3-ima1,VplTp=vp1,VpcTp=1,VclTp=vc36 #vclTpBId >>> [Proxy ID = 135] MO name :ManagedElement=1,TransportNetwork=1,AtmCrossConnection=b7ca RXI901> cr TransportNetwork=1,AtmCrossConnection=b7cb Attribute 1 of 2, vclTpAId (moRef:VclTp): Enter mo LDN: AtmPort=MS-26-1,VplTp=vp6,VpcTp=1,VclTp=vc43 #vclTpAId Attribute 2 of 2, vclTpBId (moRef:VclTp): Enter mo LDN: AtmPort=MS-3-ima1,VplTp=vp1,VpcTp=1,VclTp=vc43 #vclTpBId >>> [Proxy ID = 136] MO name :ManagedElement=1,TransportNetwork=1,AtmCrossConnection=b7cb Finn Magnusson 2006-06-26 rev PA1 (28/38) //THE SITE IS NOW UP RNC10> str -i 7 Following 1 sites are totally or partially unavailable: ---------------------------------------------------------------------------------------------------MOD IUBLINK CELLNAMES CELL1 CELL2 CELL3 CD12 ATMPORTS ---------------------------------------------------------------------------------------------------1 7 3114-1/2/3 11111 11111 11111 1111 ---------------------------------------------------------------------------------------------------- 10 Iu Problem, MTP3 link deactivated 10.1 Symptom: A Ranap is disabled RNC10> st ranap=iuc =================================================================================== Proxy Adm State Op. State MO =================================================================================== 4508 0 (DISABLED) RncFunction=1,Ranap=Iuc2300 =================================================================================== Total: 1 MOs 10.2 Find the fault: //WE CHECK THE RANAP PROTOCOL STACK RNC10> lk ranap=iuc ......................... =================================================================================== Proxy Adm State Op. State MOs linked to 4508:Ranap=Iuc2300 =================================================================================== 2218 TransportNetwork=1,Aal2RoutingCase=Iu1 4508 0 (DISABLED) RncFunction=1,Ranap=Iuc2300 2309 TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpGlobalTitle=Iuc-1-2300 1995 TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iu1,Aal2PathDistributionUnit=Iu1-2 1996 TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iu1,Aal2PathDistributionUnit=Iu1-1 1994 0 (DISABLED) TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iu1 2310 TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpEntitySet=Iuc-1-2300 2314 0 (DISABLED) TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpApRemote=Iuc-1-2300 1714 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bAp=Iuc-1-2300-3 Finn Magnusson 2006-06-26 rev PA1 (29/38) 1718 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bAp=Qaal2-1-2300-12 1720 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iuc-1-2300-3,Mtp3bSr=1 1717 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3,Mtp3bSlItu=1 1716 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3 1719 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iuc-1-2300-3 2276 1 (ENABLED) TransportNetwork=1,NniSaalTp=Iuc-1 1698 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=Iuc-1 2243 1 (ENABLED) TransportNetwork=1,AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc34 =================================================================================== Total: 17 MOs //WE SEE THAT VCLTP IS ENABLED, HOWEVER THIS DOES NOT NECESSARILY MEAN THAT TRANSMISSION IS OK SINCE VCLTP IS ALWAYS UP AS LONG AS THE PHYSICAL LAYER IS UP //WE DO AN END TO END LOOPBACK ON THE VCLTP RNC10> lacc TransportNetwork=1,AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc34 eteloopback ================================================================================================================= Proxy MO Action Nr of Params ================================================================================================================= 2243 AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc34 eteLoopBack 0 >>> Return value = true ================================================================================================================= Total: 1 MOs attempted, 1 MOs actioned //LOOPBACK IS OK, MEANS WE HAVE CONTINUITY ON THE VC LAYER //WE NOW CHECK THE MTP3 LAYER RNC10> get 1719 1716 1717 1720 ================================================================================================================= 1719 TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iuc-1-2300-3 ================================================================================================================= Mtp3bSrsId Iuc-1-2300-3 autoReroute false congestedLevel 0 (notCongested) destPointCode 2300 operationalState 0 (DISABLED) reservedBy [3] = >>> reservedBy = TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3 >>> reservedBy = TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bAp=Iuc-1-2300-3 >>> reservedBy = TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bAp=Qaal2-1-2300-12 userLabel ================================================================================================================= 1716 TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3 ================================================================================================================= Mtp3bSlsId Iuc-1-2300-3 availabilityStatus 1056 (DEPENDENCY_FAILED) cLinkAnsi false Finn Magnusson 2006-06-26 rev PA1 (30/38) mtp3bSrsId Mtp3bSpItu=Iu1,Mtp3bSrs=Iuc-1-2300-3 operationalState 0 (DISABLED) periodicLinkTestFlag true reservedBy [1] = >>> reservedBy = TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iuc-1-2300-3,Mtp3bSr=1 usageState 0 (IDLE) userLabel ================================================================================================================= 1717 TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3,Mtp3bSlItu=1 ================================================================================================================= Mtp3bSlItuId 1 autoStartLink false availabilityStatus 128 (NOT_INSTALLED) linkCongestLevel 0 (level_0) linkState 32 (DEACTIVATED) operationalState 0 (DISABLED) prioBeforeSio false proceduralState 0 (NOT_INITIALIZED) signLinkCode 2 tpId NniSaalTp=Iuc-1 usageState 0 (IDLE) userLabel ================================================================================================================= 1720 TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iuc-1-2300-3,Mtp3bSr=1 ================================================================================================================= Mtp3bSrId 1 RouteBlockingState 0 (NOT_BLOCKED) availabilityStatus 1056 (DEPENDENCY_FAILED) linkSetM3uId Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3 operationalState 0 (DISABLED) priority 1 usageState 0 (IDLE) userLabel ================================================================================================================= Total: 4 MOs //WE SEE THAT THE MTP3B SIGNALING LINK (MTP3BSLITU) HAS BEEN MANUALLY DEACTIVATED 10.3 Correct the fault //ACTIVATE THE MTP3BSLITU RNC10> acl 1717 ================================================================================================================= Proxy MO Action Nr of Params ================================================================================================================= 1717 Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3,Mtp3bSlItu=1 activate 0 1717 Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3,Mtp3bSlItu=1 deactivate 0 1717 Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3,Mtp3bSlItu=1 emergencyActivate 0 Finn Magnusson 2006-06-26 rev PA1 (31/38) 1717 Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3,Mtp3bSlItu=1 localInhibit 0 1717 Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3,Mtp3bSlItu=1 localUninhibit 0 ================================================================================================================= RNC10> acc 1717 activate ================================================================================================================= Proxy MO Action Nr of Params ================================================================================================================= 1717 Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3,Mtp3bSlItu=1 activate 0 >>> Return value = null ================================================================================================================= Total: 1 MOs attempted, 1 MOs actioned //RANAP STACK IS NOW ENABLED RNC10> lk ranap=iuc =================================================================================== Proxy Adm State Op. State MOs linked to 4508:Ranap=Iuc2300 =================================================================================== 2218 TransportNetwork=1,Aal2RoutingCase=Iu1 4508 1 (ENABLED) RncFunction=1,Ranap=Iuc2300 2309 TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpGlobalTitle=Iuc-1-2300 1995 TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iu1,Aal2PathDistributionUnit=Iu1-2 1996 TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iu1,Aal2PathDistributionUnit=Iu1-1 1994 1 (ENABLED) TransportNetwork=1,Aal2Sp=1,Aal2Ap=Iu1 2310 TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpEntitySet=Iuc-1-2300 2314 1 (ENABLED) TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpApRemote=Iuc-1-2300 1714 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bAp=Iuc-1-2300-3 1718 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bAp=Qaal2-1-2300-12 1720 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iuc-1-2300-3,Mtp3bSr=1 1717 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3,Mtp3bSlItu=1 1716 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iuc-1-2300-3 1719 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iuc-1-2300-3 2276 1 (ENABLED) TransportNetwork=1,NniSaalTp=Iuc-1 1698 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=Iuc-1 2243 1 (ENABLED) TransportNetwork=1,AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc34 =================================================================================== Total: 17 MOs 11 Iu Problem, transmission problem 11.1 Symptom: A Ranap is disabled Finn Magnusson 2006-06-26 rev PA1 (32/38) RNC10> st ranap=iup =================================================================================== Proxy Adm State Op. State MO =================================================================================== 4508 0 (DISABLED) RncFunction=1,Ranap=Iup2810 =================================================================================== Total: 1 MOs 11.2 Find the fault: //WE CHECK THE RANAP PROTOCOL STACK RNC10> lk ranap=iup =================================================================================== Proxy Adm State Op. State MOs linked to 4912:Ranap=Iup2810 =================================================================================== 4912 0 (DISABLED) RncFunction=1,Ranap=Iup2810 2311 TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpGlobalTitle=Iup-2-2810 2312 TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpEntitySet=Iup-2-2810 2306 0 (DISABLED) TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpApRemote=Iup-2-2810 1715 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bAp=Iup-2-2810-3 1724 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iup-2-2810-3,Mtp3bSr=1 1722 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iup-2-2810-3,Mtp3bSlItu=1 1721 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iup-2-2810-3 1723 0 (DISABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iup-2-2810-3 2205 1 (ENABLED) TransportNetwork=1,NniSaalTp=Iup-2 1202 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=Iup-2 2242 1 (ENABLED) TransportNetwork=1,AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc50 =================================================================================== Total: 12 MOs //WE SEE THAT VCLTP IS ENABLED, HOWEVER THIS DOES NOT NECESSARILY MEAN THAT TRANSMISSION IS OK SINCE VCLTP IS ALWAYS UP AS LONG AS THE PHYSICAL LAYER IS UP //WE DO AN END TO END LOOPBACK ON THE VCLTP RNC10> lacc TransportNetwork=1,AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc50 eteloopback ================================================================================================================= Proxy MO Action Nr of Params ================================================================================================================= 2243 AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc50 eteLoopBack 0 >>> Return value = false ================================================================================================================= Total: 1 MOs attempted, 1 MOs actioned Finn Magnusson 2006-06-26 rev PA1 (33/38) //LOOPBACK IS FALSE, MEANS WE DON’T HAVE CONTINUITY ON THE VC LAYER //LET’S CHECK THE VCLTP MO RNC10> get 2242 RNC10> get 2242 ================================================================================================================= 2242 TransportNetwork=1,AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc50 ================================================================================================================= VclTpId vc50 atmTrafficDescriptorId AtmTrafficDescriptor=U3P4500M3000 availabilityStatus 0 (NO_STATUS) externalVci 500 operationalState 1 (ENABLED) reservedBy Aal5TpVccTp=Iup-2 userLabel ================================================================================================================= Total: 1 MOs //WE SEE THAT THE VCLTP IS USING THE VCI VALUE 500, WHEREAS IT WAS MOST LIKELY MEANT TO USE THE VCI VALUE 50 SINCE THIS IS WHAT IS GIVEN IN THE VCLTPID 11.3 Correct the fault //CHANGE THE VCI VALUE, NEED TO USE RSET SINCE THIS IS A RESTRICTED ATTRIBUTE RNC10> set 2242 externalvci 50 ============================================================================================================ Id MO externalVci Result ============================================================================================================ 2242 AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc50 50 !!!! Processing failure :se.ericsson.cello.moframework.NoSuchAttributeException TAG :"NoSuchAttribute" VARIABLES :"$attrName" = "externalVci", "$moTypeName" = "VclTp" ============================================================================================================ Total: 1 MOs attempted, 0 MOs set RNC10> mom vcl externalv ################################################################################################## MO Class Attribute Type Flags ################################################################################################## VclTp externalVci long restricted,noNotification,mandatory -------------------------------------------------------------------------------------------------Virtual Channel Idenitifier, VCI, value for this Virtual Channel Link, VCL. Range: 32 - 65535. Range: 32 to 65535 ************************************************************************************************** Finn Magnusson 2006-06-26 rev PA1 (34/38) RNC10> rset 2242 externalvci 50 Set restricted attribute externalvci on following MOs ? =================================================================================== 2242 TransportNetwork=1,AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc50 =================================================================================== Are you Sure [y/n] ? y etc.... //RANAP STACK IS OK NOW RNC10> lk ranap=iup =================================================================================== Proxy Adm State Op. State MOs linked to 4912:Ranap=Iup2810 =================================================================================== 4912 1 (ENABLED) RncFunction=1,Ranap=Iup2810 2311 TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpGlobalTitle=Iup-2-2810 2312 TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpEntitySet=Iup-2-2810 2306 1 (ENABLED) TransportNetwork=1,SccpSp=Iu1,SccpScrc=1,SccpApRemote=Iup-2-2810 1715 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bAp=Iup-2-2810-3 1724 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iup-2-2810-3,Mtp3bSr=1 5209 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iup-2-2810-3,Mtp3bSlItu=1 1721 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSls=Iup-2-2810-3 1723 1 (ENABLED) TransportNetwork=1,Mtp3bSpItu=Iu1,Mtp3bSrs=Iup-2-2810-3 5208 1 (ENABLED) TransportNetwork=1,NniSaalTp=Iup-2 5207 1 (ENABLED) TransportNetwork=1,Aal5TpVccTp=Iup-2 5206 1 (ENABLED) TransportNetwork=1,AtmPort=MS-6-1,VplTp=1,VpcTp=1,VclTp=vc50 =================================================================================== Total: 12 MOs 12 BOARD AND NODE RESTARTS 12.1 Restart Ranks •Cold (in CPP3 = ”restart with forced reload”) –HW tests, HW reset, SW reloaded and restarted, non-persistent data is lost, T&E/llog is lost –•Refresh –HW reset, SW reloaded and restarted, non-persistent data is lost, T&E/llog is kept –•Warm –No HW reset, SW conditional reloaded, non-persistent data is lost –•Hot –No HW reset, SW conditional reloaded, State Data will survive (MP) Finn Magnusson 2006-06-26 rev PA1 (35/38) Rank Domain Reason HW reset HW tests Cold Node, Processor Total or partial Fully Refresh Node, Processor Node, Processor Program Warm Hot Power on, Reset button, HW watch dog or Operator. Operator or Function SW reload Novram Dynamic CLI Config. conmemory tainer Forced Cleared Deleted Deleted BSS/ initiated Cleared/ Initiated Total or partial Operator or None Function Small Forced Cleared Deleted Deleted Cleared/ Initiated None Condi- Pretional served Deleted Deleted Cleared/ Initiated Operator or None Function None Condi- Pretional served Preserved Preserved Cleared/ Initiated Node restart PIU (MP/BP) restart Processor SP/DP/FPG A restart Program restart Cold Refres h Warm Hot 12.2 Restart Commands from MO interface FIXME: mom man restart acc 0 restart Finn Magnusson 2006-06-26 rev PA1 (36/38) mom plug restart mom restartrank mom restartcause acc 001400 restart acc mod1 restart acc 001900/sp0.lnh restart acc cc1 restart lacc ...auxplug restart 12.3 Restart Commands from OSE shell FIXME: restartObj me reload (difference between restartObj and reload) 12.4 Checking Restart Information FIXME: llog –l dump list –a dump info lgsvm cabr lgd restartcollector REFERENCES Can be downloaded from https://psp.uab.ericsson.se/cpp/deliveries/cpp4/doc_4.htm: - FS load and start - FD file system - FS Fault Tolerant Core - UG cello - Description cello Finn Magnusson 2006-06-26 rev PA1 (37/38) - Cello Core presentations part 2 and 3 (http://utran01.epa.ericsson.se/utranwiki/KnowledgeTransferPresentations) 12.5 Moshell command line The command line uses the Readline library from bash. Here are some of the supported function keys: move forward one character right arrow or Ctrl-f move backward one character left arrow or Ctrl-b previous command in history buffer up arrow next command in history buffer down arrow delete one character backward backspace delete one character forward Ctrl-d or <del> go to beginning of line Ctrl-a or <home> go to end of line Ctrl-e or <end> erase all characters backward Ctrl-u erase all characters forward Ctrl-k move forward one word Alt-f move backward one word Alt-b copy to clipboard select or select + ctrl-<insert> paste from clipboard <insert> or shift-<insert> Useful: by typing the beginning of a command and then use the up/down arrow key, the command completion will only match the commands from history that start with this string Support commands hi : to print the command history l+/l- : to log the output to a logfile (l+m: “mute” ---> no output to screen, only to file. l+o: “overwrite” --> logfile is overwritten) l : to run a command locally on the workstation without having to exit from moshell or open a new window ip2d/d2ip, h2d/d2h : to convert to/from decimal to hex or ip Finn Magnusson 2006-06-26 rev PA1 (38/38)