Vodafone MachineLink 3G Logging Architecture Feature Spotlight 1 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification The MachineLink 3G logging system • The MachineLink 3G router automatically keeps a log of system events. • By default, the router logs to volatile memory which means that if power is lost or the router is rebooted for some reason, the logs will be lost. • Loss of log data can be avoided by configuring the router to log to non-volatile memory or by configuring a remote syslog server. • The logging system in its default state logs up to 256 kilobytes of data meaning that events that happened some time ago (depending on activity) may not show in the logs. 2 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Enabling log to non-volatile memory Flash memory generally has a finite number of program-erase operations that it can perform to the blocks of memory. While this number of operations is quite high, we recommend that you only enable log to non-volatile memory when actually troubleshooting a problem. 3 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Local Syslog server Vs. Remote syslog server A syslog server helps to alleviate the problem of losing logs due to limited or volatile memory on the router. A syslog server is a piece of software which runs remotely i.e. not on the router, and accepts log data from the router. The log data can then be saved or manipulated however you please. The syslog can be running on a computer connected directly to the MachineLink 3G router or on a machine on the other side of the world. The choice is yours. You may prefer to use a syslog server locally when: • your mobile broadband data plan is limited • you have the capacity to run a computer near the MachineLink 3G router • you are not too far from the MachineLink 3G router • you want use a syslog server temporarily to troubleshoot an issue You may prefer to use a syslog server remotely when: • you do not have to worry about bandwidth or traffic concerns on your mobile broadband plan • the MachineLink 3G router is operating in a place where you can’t run a computer near it 4 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Using a remote syslog server 5 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Using a remote syslog server - Locally Open your syslog server. In this example we are using the free “Tftpd32” application which also acts as a syslog server. The server interface is 192.168.1.190 which is the IP address assigned to the network card on the machine running the syslog server. As changes occur on the router, the log is updated. 6 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Using a remote syslog server - Remotely To send log data to a syslog server which is not connected to the router directly, enter the WAN IP address of the syslog server. 7 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Using a remote syslog server - Remotely The log data is displayed on the remote syslog server. 8 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Troubleshooting problems When you experience a problem on the MachineLink 3G, it is important to analyse the logs to identify the cause of the problem. When you send log files to NetComm for analysis, you should also include as much information as possible about the problem, such as: • When the problem occurs – please specify as accurately as you can • Does the problem occur regularly or does it seem to be random? • What was happening on the router at the time, if you know • Is the problem reproducible? 9 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Troubleshooting problems Other tips for troubleshooting problems: • Try to capture the logs of a working example and a problem example to provide a contrast • Make sure that the logs capture the problem event • It is important that the time is set accurately on the router so that when logs are analysed, the problem can be identified at the precise time it occurred. Also ensure that the time stamp in the log is accurate. • It helps us a lot if you can annotate the log file to identify when certain events happened e.g. At timestamp 13:41:09 the Ethernet cable was connected, at 13:45:17 a dropout occurred. • When you send log files to NetComm Wireless, it is very helpful if you include a configuration backup of the router settings. This helps in reproducing the problem or identifying configuration issues. • Sometimes a connection diagram showing the device scenario/environment can assist in diagnosis. 10 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Example problems in the log file SIM card not inserted Forbidden carrier selected Incorrect PIN entered 11 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Log filter There are 5 levels that you can use to filter the log displayed in the user interface of the router. The filters do not change the level of logs that are kept but assist in removing unnecessary items from view when you are trying to troubleshoot a problem. 12 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification The MachineLink 3G log The log is displayed on the user interface of the router and may also be downloaded as a file and viewed using a text editor. If you download the log file for viewing, remember that it is stored in Linux format (CR only). When opening the log file on Windows system, ensure that you use a file viewer that can display “CR only” files correctly. A good example of this is the open source text file editor “Notepad++”. 13 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification The MachineLink 3G log The log is displayed on the user interface of the router and may also be downloaded as a file and viewed using a text editor. 14 Field Description Date & Time The date and time that the event occurred Machine The machine logging the event Level The level (severity) of the event Message A description of the event Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Logging levels There are 8 levels of events in the log: 1. 2. 3. 4. 5. 6. 7. 8. Emergency – System is unusable Alert – action must be taken immediately Critical – a critical condition has occurred Error – an error condition has occurred Warning – this message is a warning Notice – a normal but significant condition Information – an informational message Debug – debug-level message The levels in red above are available as filters in the user interface. When you select a logging level to display in the user interface view of the log, only the levels above the chosen one will be displayed. For example, selecting Notice displays events 1 through 6 in the user interface view. 15 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification Facilities The facility argument is used to specify the type of program that is logging the message. The facility usually appears in the log as a prefix to the level, e.g. user.notice. The following facilities are available in the MachineLink 3G: 1. 2. 3. 4. 5. 6. 7. 8. 9. Auth – Security/Authorization messages. Authpriv – Security/Authorization messages (private) Cron – clock daemon Daemon – system daemons without separate facility value ftp – FTP daemon Kern – Kernel messages (these can’t be generated by user processes) Local0 through Local7 – reserved for local use Syslog – messages generated internally by syslogd User – generic user-level messages 16 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification AT Command logging The following commands permanently change debug level of port manager root:~# rdb_set -p wwan.0.debug 8 root:~# reboot_module.sh turning off the phone module... sleeping for 3 seconds... turning on the phone module... To collect all the AT commands, use the following command root:~# logread -f | grep 'at_read\|at_send' Jan 1 10:01:45 vdf_nwl10 local5.info simple_at_manager[2826]: at_send_with_timeout: --> [AT] (timeout = 1 sec) Jan 1 10:01:45 vdf_nwl10 local5.info simple_at_manager[2826]: at_read: <-- [AT] Jan 1 10:01:45 vdf_nwl10 local5.info simple_at_manager[2826]: at_read: <-- [OK] Jan 1 10:01:45 vdf_nwl10 local5.info simple_at_manager[2826]: at_send_with_timeout: --> [ATE1] (timeout = 5 sec) Jan 1 10:01:45 vdf_nwl10 local5.info simple_at_manager[2826]: at_read: <-- [ATE1] Jan 1 10:01:45 vdf_nwl10 local5.info simple_at_manager[2826]: at_read: <-- [OK] Jan 1 10:01:45 vdf_nwl10 local5.info simple_at_manager[2826]: at_send_with_timeout: --> [AT+CSAS=0] (timeout = 2 sec) Jan 1 10:01:45 vdf_nwl10 local5.info simple_at_manager[2826]: at_read: <-- [AT+CSAS=0] To clear the debugging level configuration back to default, perform factory reset or run the following command root:~# rdb_set -p wwan.0.debug "" root:~# reboot_module.sh 17 Integrated M2M Terminals Introduction Vodafone MachineLink 3G v1.0 C3 confidentiality classification