System Diagnostic Procedures
Diagnosis of system performance can most effectively be accomplished using pagers and
ReFLEX25 protocol decoder such as Advanced Signal SignalPro or Grayson PageTracker.
Remote diagnostics of system are somewhat less definitive, but by observing behavior of
devices and system logs, it is possible to derive information on what the system is doing. The
sequence of events beginning with a pager’s registration may be tracked and used to verify
system performance. The following is a description of “typical” operation:
Pager is powered upa)
Pager scans for channel (“Storing messages” is displayed)
b)
Pager detects forward channel (“Basic Service” or “New Messages Only” is
displayed)
c)
Pager sends a Registration Request packet which is forwarded to terminal from
RFC.
d)
Terminal sends a Registration Grant to pager
e)
Pager receives Registration Grant and changes to “Registered” state (“Full
Service” or “New and Stored Messages” is displayed)
If any of these steps are missed, there are troubleshooting “points” in the system. Observing
the pager’s behavior may effectively point to the problem area.
1) Pager continues to display “Storing Messages”. Pager cannot decode forward channel.
Check RFB for data being accepted, GPS sync. (A 192 OPPM, A 208)
Check RFO for correct frequency and power cutback/ shutdown. (A 83)
2) Pager continues to display “Basic Service”. Pager unable to register.
Check pager for programming of correct Home Index
Check RFA(s) for IP connection, correct frequency, and correct ID. Receiver may be
“traced” via FIPS port and RFC snoop to observe packets.
Check RFC for correct Home Index-to-Terminal mapping
Check RFC and Terminal for valid WMTP connection, check database in Terminal
for pager’s indicated active zone/ service area.
Check pager programming in Terminal (capcode & service areas or zones allowed)
If these diagnostic steps fail to uncover the problem, it may be necessary to examine pager
and terminal configuration more carefully. Log files from RFC and any test equipment
(Grayson/ Advanced Signal) should be collected if possible. Also, attempt to use more than
one type of pager to prevent any confusion arising from device-specific problems.
RF-O!/ RF-A! Procedures
To telnet to one of the stations to check diagnostics, you first need to telnet to
the terminal server then go to the port on the t/s that is connected to the device
i.e. the Baton should be connected to port 7, RFA should be connected to port
M
3/8/2016 1:50 AM
5, (or port 6 in systems planned for only 1 receiver to be installed), OCM
should be connected to port 8. This procedure may be used to monitor for
outbound traffic and to check for alarms, diagnostics, etc.
To connect to one of a station’s “FIPS” ports, use the pull down command tool or shell tool
from the “Utilities” menu on Sun workstation (Choreographer) as described below:
<<command tool>>
#
# telnet 10.1.16.99
<RETURN>
connect to port <6, 7, or 8> (by typing 5,6,7,or 8)
then, To view RF-B/ RF-O/ RF-A alarm:
FIPS ( default password for FIPS is 6000, can be altered with R/W 707)
a 99 read all alarms
a 110 read software errors
a 104 read station errors
To trace RF-B/ RF-O/ RF-A events:
(from FIPS):
RF-B!:
a 208 to check generic GPS receiver status
a 192 OPPM 4 to monitor data arriving at Baton! from RFC
a 192 PCRM 4 to monitor PDM’s and psos, launch timing
a 192 GPSC 8 to monitor GPS satellite tracking (Motorola GPS)
a 192 SASM 34 to monitor state of GPS synchronization (after GPSC 8,
“state 4” is necessary for launch of PDM) (Motorola GPS)
a 193 xxxx # to disable monitor/ tracing
a 215 to check time with Motorola GPS
r 704 to check “color code” in transmitter (Baton!)
r 705 to check Baton! ip address
r 706 to check Baton! ip subnet mask
r 708 to check Baton! ip address of “default gateway”
RF-O!:
a 192 RFCQ 1 to monitor forward channel control data as sent from Baton!
a 193 RFCQ 1 to disable monitor
a 192 SPCQ 64 to monitor station power readings each sample period
a 193 SPCQ 64 to disable power reading monitor
a 79 2 to check forward power at transmitter output (external wattmeter)
a 79 3 to check reflected power at transmitter output
M
3/8/2016 1:50 AM
RF-A!:
w 953 1 to enable tracing of received packets and time stamp
w 953 0 to disable tracing
w 551 1 to enable “null packets” forwarded to RFC
w 551 0 to disable “null packets”
a 198 to check time
a 196 ? to check and configure ip parameters of RFA
a 196 19200, 10.6.35.100,10.6.35.99,255.255.255.224, ,1
where 10.6.35.100 is the ip address of the receiver
& 10.6.35.99 is the gateway (terminal server address)
This command also gives the capability to check PPP status.
To clear RF-B/ RF-O/ RF-A alarms:
a 103 clear all alarms
a 113 clear software error log
a 111 clear station error log
a 117 RESETS device- for all devices
exit to get out of FIPS
^} (control+“right bracket”) to exit telnet session
RF-C! Procedures
To ascertain that paging traffic is being sent through the RF-C correctly, it may
be useful to employ a UNIX utility (“snoop”) to observe packets into and out of
the RF-C. In the event of an alarm in the RF-C that causes a cessation of traffic
to a given site, or in the event of congestion in the RF-C, a reset of the RF-C
may be necessary.
To trace data into and out of the RF-C:
<<command tool>>
# telnet rfcctl
login: root (or other password as set in UNIX/ Solaris administration)
password: motorola
[rfcctl]# snoop -d qe1 port 12000 or port 5001 :monitor both inbound and outbound traffic
To trace the event log in the RF-C:
<<command tool>>
# telnet rfcctl
M
3/8/2016 1:50 AM
login: root
password: motorola
[rfcctl]# cd /shared/rfc/current
[rfcctl]# pwd
[rfcctl]# /home/rfc/v2.2.2a (or appropriate version number of installed software, e.g., v2.3.7,
v2.4.2, v3.7.0 etc.)
[rfcctl]# tail -f rfc_event.log :monitor “events” through RFC, specifically of concern are
“congestion warning” or “congestion reject” messages.
To reset the RF-C Application:
simply access the “HA” menu and initiate a “takeover”
This will stop the application on the active side and complete its initiation on the backup
side.
To stop the RF-C Application:
<<command tool>>
# telnet rfcctl
[rfcctl]# cd shared/rfc
[rfcctl]# pwd
[rfcctl]# /shared/rfc
[rfcctl# ./RFC_shutdown
REDUNDANT off-->Will STOP the RF-C! application
It may also be necessary to stop execution of the “HA” application to recover from a corrupt
configuration file or disk error.
To stop HA:
<<command tool>>
# telnet rfcctl
[rfcctl]# cd shared/rfc/current/killHaRfc
(Should see “HA services stopped” and “Failed to un-mount shared disk” messages.)
Repeat “killHaRfc” commmand on new active side of RFC.
Login to either side of RFC and re-mount shared disk:
[rfcactl]# umount /shared
[rfcactl]# mount /dev/dsk/c1t0d0s6 /shared
Changes to /home/rfc/current directory files can be made.
Restart RFC, remounting shared drive as read/writable on active side only by issuing:
[rfcactl]# shutdown –y –g0 –i6 command on both sides.
M
3/8/2016 1:50 AM
The “Tracer” application is a development tool that may be used to track the status of the
RFC and to assist diagnosing problems with GPS receivers or terminal connection. It is not
intended to be a complete test facility. Tracer should be run from the inactive side of the
RFC or from the system administration Sun workstation (Choreographer).
To initiate “Tracer” application:
<<command tool>>
#./tracer “ip address of active RFC on network that Choreographer is connected to”
Commands for "Tracer" application:
RF-C! Tracer >>> (127.0.0.1) help
>
*** Remember to 'trace off' when doing Load Test ***
*** Tracer may cause RF-C! to core dump
***
Valid commands for RF-Conductor! are:
clientstatus
emstatus
flexQstatus
gpsstatus
help
nsconfig
reflexQstatus
setdest
setlevel
tasks
trace
zonestatus
Enter "quit" to exit Tracer.
Example of "clientstatus" command output:
RF-C! Tracer >>> (127.0.0.1) clientstatus
>
*** Remember to 'trace off' when doing Load Test ***
*** Tracer may cause RF-C! to core dump
***
IP Address Client ID Index
Port State OUT Queue Size
---------------------------------------------------------------------10.1.0.133
3000 OPEN
0
(10.1.0.133 | 10.1.0.133)
8001
1
ASSOC_UP
-------------------------------------10.10.0.130
3000 CLOSED
0
(10.10.0.130 | )
43004011
0
ASSOC_DOWN
--------------------------------------
M
3/8/2016 1:50 AM
RF-C! Tracer >>> (127.0.0.1) quit
NOTE:
More detail on each of these commands may be obtained by typing help <command name>.
The most useful commands are "tasks"- to determine that all processes are up and running,
"gpsstatus"- to check status of gps receiver, "clientstatus"- to check that the configured
terminals have established a WMTP link to the controller, and "trace"- to examine packets
that are going out from the controller and coming back from pagers & receivers. Note that
“trace” will cause “rfc_event.log” to be written to, nothing shows up in “tracer” window.
M
3/8/2016 1:50 AM
WMG
This is the current lay-out of the PAGENET DFW WMG:
“UCC1” is the upper UCC, “UCC2” is the lower UCC.
Alarms from the ALARM VIEWER will indicate “1001” for alarms on UCC2 and “1002” on
UCC1.
UCC2/ 1001/ UCC1 on line config screens (all application layer)
--> 972 area code (972)330-0003
UCC1/ 1002/ UCC2 on line config screens (all application layer)
--> 817 area code
To RESET the UCCs
alarms 1001 or 1002 could represent a UCC problem, RESET the UCC that is alarmed, the
GUI will show that UCC1 is alarmed (1001 as indicated on AlarmViewer), this is actually
UCC2 in the hardware/ IP structure of the WMG. Specifically, the alarm that will cause
UCC problems is “DSP Resource Not Available”. When this alarm appears, the system
should be checked to see that it gives a voice prompt (“Leave a message after the tone, press
pound when finished”) without excessive delay. If the number rings, gives a busy tone, or
does not pick up the call, the UCC should be reset.
$ rlogin ucc1c -l root to access UCC1 (you can access from desktop pop-up
menu by choosing Xterm UCC1 or Xterm
UCC2)
Password: motorola
# sync <---start here if login from desktop pop-up menu
# sync
# init 6
$ rlogin ucc2c -l root to access UCC2
Password: motorola
# sync <---start here if login from desktop pop-up menu
# sync
# init 6
** After finish the UCC# reset proceedure type: when reset the UCC#:
# ping -s ucc2c
or
# ping -s ucc1c , depending on which was RESET
(this checks to see if the UNIX OS has booted up)
# ps -ef |grep mgr
and look for /hs/ucc/bin/resc-mgr/fs/hs 100 process
(if this process is present, the UCC has re-started resource manager and should shortly be
accepting calls.)
M
3/8/2016 1:50 AM
To check the status of the “site” pagers:
The following procedure will create a log file of the pages received for each of the sites at
which test pagers are located. This should be executed at least twice daily.
cd/ $WMGHOME
cd / config/Utilities/pagerReport
pwd
$/usr/local/wmg/config/Utilities/pagerReport
$ tx_site_report >XX.out (NEED to hit the RETURN twice)
the “>“ symbol sends the data to the filename placed after the symbol
the “|” symbol will pipe the file somewhere, e.g., a printer (lp)
To check for messages being submitted to the WMG and forwarded to the RFC:
The following command will execute a trace of events through the “Message Server” subprocess in the WMG. This verifies that the WMG is able to accept calls from a UCC and
able to communicate with the RFC and its operational zone. During a “quiet” or non-busy
period, a call to the system should give a “Page Request delivered SUCCESSFULLY:
holding PENDING ACK(s)” message on the Message Server screen.
$cd /tmp
$Ccclient msgServer@#####
<debug3>
Other Useful UNIX Commands:
man UNIX Help
ls List Directory
pwd List path
ls -al Lists all files
mp filename print
lp filename print
M
3/8/2016 1:50 AM