The SmartServer is based on the Linux operating system and many common Linux commands can be used to access the SmartServer through the SmartServer console or by using SSH. This section covers some of the most commonly used console/SSH commands for troubleshooting the SmartServer. Most commands can be issued using the apollo login. In some cases, you will need to add sudo to the beginning of the command for higher privileges. Always use the apollo user login and not root. Newer versions of the SmartServer restrict or prevent root login.
Many commands (like top) will continuously run until you exit the command, and therefore, should be run in an SSH session instead of the SmartServer console. A SSH session will disconnect when the PC is no longer able to access the SmartServer, and therefore stopping the continuous command. You can exit continuous commands by entering CTRL-C.
The table below describes and provides examples of console/SSH commands. To eliminate hidden characters, copy the commands to a text editor (e.g., notepad), and then copy the commands from the text editor and paste them into the SSH session.
The commands shown in the table below assume that apollo login is used.
| Command | Description |
|---|---|
| cat | Allows you to read a file. This command is most useful for reading small files. For longer files, use more. Displays the contents of a log file $cat /var/log/supervisor/lte.log Displays the contents of a large log file $more /var/log/supervisor/lte.log |
| date | Displays the SmartServer platform current local time (time changes are not propagated to CMS). The SmartServer CMS and the SmartServer platform time and timezone may differ. See also timedatectl below. Example: date $ date Mon Dec 12 15:10:56 PST 2022 |
| df | Displays system disk space usage. If your system disk space usage is close to 100%, then you may experience performance problems. Example: df df -h Example: df (healthy system) $ df -h Filesystem Size Used Avail Use% Mounted on /dev/root 2.9G 2.3G 496M 83% / devtmpfs 846M 0 846M 0% /dev tmpfs 1007M 20K 1007M 1% /dev/shm tmpfs 202M 4.8M 197M 3% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 1007M 0 1007M 0% /sys/fs/cgroup /dev/mmcblk3p1 16M 14M 2.4M 85% /boot /dev/mmcblk3p5 675M 33M 593M 6% /var/apollo /dev/loop0 275M 320K 254M 1% /var/core /dev/mmcblk2p1 30G 554M 28G 2% /media/sdcard A /dev/root of 85% is acceptable; however, if it shows 100%, or another item shows close to 100%, then use the du command to determine which file is using the excessive memory. |
| du | Allows you to determine which files are using a lot of disk space. If the df -h command shows anything close to 100% for any of the items listed, then you can use this du command to help identify which files are the cause. Some files are expected to be large while others are expected to be small. Go to the directory that shows 100%, or close to 100%, and search for files over 5MB. Example: Looks for directories over 1 MB sudo du -t 1M -x -s -h * A large file does not mean it is a problem. You should compare the results with another working SmartServer. If you find a file on a SmartServer that is much larger than the same file on another SmartServer, then you should look at that file. For example, if /dev/root shows 100% usage, change to the root directory run the du command. $ cd / $ sudo du -t 1M -x -s -h * 14M boot 1.1E etc 191M home du: cannot access 'proc/831/task/831/fd/3': No such file or directory du: cannot access 'proc/831/task/831/fdinfo/3': No such file or directory du: cannot access 'proc/831/fd/3': No such file or directory du: cannot access 'proc/831/fdinfo/3': No such file or directory du: cannot read directory 'proc/32610/task/32610/net': Invalid argument du: cannot read directory 'proc/32610/net': Invalid argument 26M root 1.1M run 1.8G usr 264M var Lines starting with du: cannot access can be ignored. If a file appears to be too large, then go to that directory and run the du command again. You can then issue the ls -lh command to see which files use a lot of space. |
| echo | This command can be used to determine the SmartServer software version. The SmartServer software version will also be displayed when you log into the SmartServer console/SSH. Example: Determines SmartServer version echo $APOLLO_VERSION Example: echo $ echo $APOLLO_VERSION 3.60.016 |
| grep | Displays any line in a file that has the specified text. Example: Searches for any indication of EMI? sudo grep "EMI?" /var/log/syslog |
| history | Displays the previously issued SSH commands. This list is not cleared when the console/SSH session is terminated. Example: Displays history history Example: Clears history history -c |
| ifconfig | Displays the SmartServer hardware interfaces, like Ethernet IP Address, U60/U70 interfaces, and IOX. Example: ifconfig ifconfig Example: eth0 is the LAN port, eth1 is the WAN port, IP70 is the SmartServer backbone, lon# (lon0) refers to a U60/U70 $ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.92.3 netmask 255.255.255.0 broadcast 192.168.92.255
inet6 fe80::200:ff:fe00:5 prefixlen 64 scopeid 0x20<link>
ether 00:d0:71:0a:e3:3c txqueuelen 1000 (Ethernet)
RX packets 11630 bytes 845821 (845.8 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11644 bytes 785215 (785.2 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.113 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 fe80::200:ff:fe00:5 prefixlen 64 scopeid 0x20<link>
ether 00:d0:71:0a:e3:3d txqueuelen 1000 (Ethernet)
RX packets 263942 bytes 66959105 (66.9 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 292136 bytes 183708154 (183.7 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ip70: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
inet 172.24.25.128 netmask 255.255.255.0 broadcast 172.24.25.255
inet6 fe80::2d0:71ff:fe0a:e341 prefixlen 64 scopeid 0x20<link>
ether 00:d0:71:0a:e3:41 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 223936 bytes 11910231 (11.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 7081807 bytes 1530678946 (1.5 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7081807 bytes 1530678946 (1.5 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lon0: flags=67<UP,BROADCAST,RUNNING> mtu 1280
inet 44.111.199.120 netmask 255.255.255.0 broadcast 44.111.199.255
unspec 00-D0-71-10-08-AE-30-30-00-00-00-00-00-00-00-00 txqueuelen 10 (UNSPEC)
RX packets 112368 bytes 2040716 (2.0 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 225088 bytes 1491920 (1.4 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
If all of your LON devices are down, then use ifconfig to look for lon# or check the System Configuration page to see if the U60/U70 appears. To see traffic on a U60/U70, determine the lon# (e.g., lon0) and use the ifconfig command to look at TX and RX packets. Re-issue the command a couple of seconds later. Example: ifconfig lon0 ifconfig lon0 ifconfig lon0 Example: ifconfig lon0 results $ ifconfig lon0
lon0: flags=67<UP,BROADCAST,RUNNING> mtu 1280
inet 44.137.199.120 netmask 255.255.255.0 broadcast 44.137.199.255
unspec 00-D0-71-14-BF-43-00-00-00-00-00-00-00-00-00-00 txqueuelen 10 (UNSPEC)
RX packets 258937 bytes 4487024 (4.4 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 535757 bytes 3206752 (3.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
$ ifconfig lon0
lon0: flags=67<UP,BROADCAST,RUNNING> mtu 1280
inet 44.137.199.120 netmask 255.255.255.0 broadcast 44.137.199.255
unspec 00-D0-71-14-BF-43-00-00-00-00-00-00-00-00-00-00 txqueuelen 10 (UNSPEC)
RX packets 259032 bytes 4488670 (4.4 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 535947 bytes 3208090 (3.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
|
| lscpu | Displays the number of CPUs that are available with the SmartServer. See CPU(s) in the command output. This information is used to determine whether you have Quad Core or Dual Core SmartServer. Example: lscpu $ lscpu Architecture: armv7l Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Vendor ID: ARM Model: 10 Model name: Cortex-A9 Stepping: r2p10 CPU max MHz: 792.0000 CPU min MHz: 396.0000 BogoMIPS: 6.00 Flags: half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 |
| lsusb | Determines if you are using a U60/U7 as indicated with Echelon Co. Example: Device 003 indicates a U60/U70 $ lsusb Bus 001 Device 004: ID 0781:5571 SanDisk Corp. Cruzer Fit Bus 001 Device 003: ID 0920:5550 Echelon Co. USB2.0 Hub Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 002: ID 1618:9113 Redpine Signals, Inc. Wireless USB Network Module Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub |
| mosquitto_sub | Subscribes to an MQTT topic (see IAP/MQ for more details). These commands should only be run using SSH and NOT using the console. Enter CTRL-C to exit. Example: Views all devices mosquitto_sub -v -t glp/0/+/fb/dev/+/+/cfg Example: Views all events mosquitto_sub -v -t glp/0/+/ev/data/# Example: Views all MQTT traffic (with timestamp) mosquitto_sub -v -t "#" -F "%I %t\n %p" Example: Saves all MQTT traffic (with timestamp) to a file - only let run temporarily as may use up a lot of disk space mosquitto_sub -v -t "#" -F "%I %t\n %p" > /var/apollo/data/mqtt.log |
| mosquitto_pub | Writes using an MQTT topic (see IAP/MQ for more details). Use caution with this command and the topics to which you write. Example: mosquitto_pub mosquitto_pub -t glp/0/17q3awh/rq/dev/lon/5/if/lamp/0 {"nviValue":{"value":{"value":75,"state":1}}}
|
| pmic_checker.sh | Used to check the SmartServer PMIC version. Older SmartServer software version may not support newer SmartServer hardware version since the CPU, PMIC, or memory may have changed and the older software is not aware of the new hardware. Using the wrong software version may cause the SmartServer to stop working and be unrecoverable. PMIC:
Determining the SmartServer PMIC sudo /root/.setup/pmic/pmic_checker.sh Example determine the SmartServer PMIC sudo /root/.setup/pmic/pmic_checker.sh [sudo] password for apollo: DA9063 |
| reboot | Reboots the SmartServer. You can use this method if you cannot reboot the SmartServer using the System Configuration page. Once you issue the command, there is a short delay before the SmartServer reboots. Example: reboot sudo reboot |
| re-image.sh | Starts the re-image script. You first have to plug in a USB flash drive with the SmartServer flash drive image (unzipped files), and then run this command. See also Updating or Re-imaging the SmartServer with the SmartServer Console. Example: re-image script sudo /media/usb0/re-image.sh |
| smartserver-secure | With SmartServer 3.5 and higher, this command is used to enable/disable enhanced security and is the same as using the Enable enhanced security setting on the System Configuration page. Enhanced security defaults to being enabled. Enable Enhanced Security sudo smartserver-secure +all Disable Enhanced Security sudo smartserver-secure -all |
| su | Switches to higher permissions so that you do not have to add sudo before console/SSH commands. Example: su, switches to higher permissions $ su Password: Example: exit, exits higher permissions exit |
| smartserverctl | For SmartServer 4.0 and higherChecks the status of SmartServer services and shows built-in services, like bacnet, which is used for BACnet, and any internal device application. Example: smartserverctl sudo smartserverctl status Example: smartserverctl status $ smartserverctl status alarm ACTIVE 1h 4min bacnet ACTIVE 1h 4min core ACTIVE 1h 4min datapoint-connection-manager ACTIVE 1h 4min datapoint-controller ACTIVE 1h 4min datapoint-get ACTIVE 1h 4min enocean-driver ACTIVE 1h 4min formatter ACTIVE 1h 4min housekeeper ACTIVE 1h 4min iap-controller ACTIVE 1h 4min init ACTIVE 1h 4min lim ACTIVE 1h 4min loader ACTIVE 1h 4min logger ACTIVE 1h 4min lon ACTIVE 1h 4min lte ACTIVE 1h 4min ltx ACTIVE 1h 4min lwd ACTIVE 1h 4min modbus ACTIVE 1h 4min monitoring ACTIVE 1h 4min mosquitto ACTIVE 1h 4min node-red ACTIVE 1h 4min pyro4-ns ACTIVE 1h 4min query ACTIVE 1h 4min reboot-manager ACTIVE 1h 4min resource-publisher ACTIVE 1h 4min scheduler ACTIVE 1h 4min secmount ACTIVE 1h 4min services ACTIVE 1h 4min smartserver ACTIVE 1h 4min storage-manager ACTIVE 1h 4min apollo@smartserver-17qam88:~$ Most services have roughly the same uptime. If the uptime for one service is very different from others, then this may mean that the service was restarted. LTE and LTX services may have different uptimes than the rest of the services. These services are re-started automatically each time you import or re-import an XIF or resource file set XML files. They may also get restarted due to other normal processing. The example below shows how to restart both the LTE and LTX services using a single command. Example: Restarts both LON services (LTE and LTX) sudo smartserverctl restart lon Example: Restarts all services sudo smartserverctl reload Example: Disables automatic start on boot for a given services or groups # smartserverctl disable service [service ...] Disable service command Issuing only the disable service command does not stop the service from running. The service will continue to run and will only be stopped after a reboot or by issuing the stop command as well. If you want to stop the service from running, then issue the following commands: # smartserverctl stop service [service ...] |
| supervisorctl | For SmartServer 3.6 and prior releases. For SmartServer release 4.0 and higher look at smartserverctl. Checks the status of SmartServer services and shows built-in services, like echbacnet, which is used for BACnet, and any internal device application. Example: supervisorctl sudo supervisorctl status Example: supervisorctl status $ sudo supervisorctl status apollo-init EXITED Dec 15 06:57 AM core:echhousekeeper RUNNING pid 1183, uptime 2:54:05 core:echlim RUNNING pid 1189, uptime 2:54:05 echbacnet RUNNING pid 1167, uptime 2:54:05 echmodbus RUNNING pid 1259, uptime 2:54:03 echopcua-server STOPPED Not started lon:echlte RUNNING pid 3497, uptime 2:52:29 lon:echltx RUNNING pid 3496, uptime 2:52:29 ready EXITED Dec 15 06:57 AM services:echalarm RUNNING pid 1239, uptime 2:54:04 services:echapollo-rp-launch RUNNING pid 1250, uptime 2:54:03 services:echconnection RUNNING pid 1241, uptime 2:54:04 services:echdatapointcontrol RUNNING pid 1244, uptime 2:54:04 services:echdatapointget RUNNING pid 1248, uptime 2:54:04 services:echformatter RUNNING pid 1249, uptime 2:54:04 services:echiap-controller RUNNING pid 1255, uptime 2:54:03 services:echloader RUNNING pid 1238, uptime 2:54:04 services:echlogger RUNNING pid 1212, uptime 2:54:04 services:echmonitoring RUNNING pid 1224, uptime 2:54:04 services:echquery RUNNING pid 1232, uptime 2:54:04 services:echscheduler RUNNING pid 1252, uptime 2:54:03 Most services have roughly the same uptime. If the uptime for one service is very different from others, then this may mean that the service was restarted. LTE (lon:echlte) and LTX (lon:echltx) services may have different uptimes than the rest of the services. These services are re-started automatically each time you import or re-import an XIF or resource file set XML files. They may also get restarted due to other normal processing. The example below shows how to restart both the LTE and LTX services using a single command. Example: Restarts both LON services sudo supervisorctl restart lon: Example: Restarts all services sudo supervisorctl reload |
| systemctl | For SmartServer 3.6 and prior releases. Determines if the CMS has started, even if you cannot log into the CMS. The CMS is typically accessible within five minutes after the Change Password button becomes enabled on the System Configuration page. If the SmartServer is been up for 15 minutes, then most likely the CMS is up. If you can access the Configuration pages and console/SSH, but not the CMS, then issue the following command. Example: systemctl sudo systemctl status karaf The following command shows if the Karaf service is up. If you see active (running) in green, then this means that the service is up. However, it does not guarantee that Karaf (CMS) is fully working. If you cannot reach the SmartServer Configuration pages (nginx) and other services, then enter the following commands: sudo systemctl status sudo systemctl list-units --state=failed |
| tail | Displays the end of a file and also displays all new entries. This command should only be used with SSH. Enter CTRL-C to exit. Example: Shows the last 10 lines in a file and shows all new updates tail -10f /var/log/supervisor/lte.log tail -f /var/log/supervisor/lte.log |
| tar | Creates tape archive files (i.e., a Linux zip file). Once you tar-up a file, you can then copy it to your PC and un-tar it with an application like 7-Zip. For SmartServer 4.0 and higher, use the SmartServer System Configuration page Logs button to download the SmartServer log files. Most of the driver logs are included in the SmartServer 4.0 and higher /var/log/syslog. For SmartServer 3.6 and prior, use the following commands to get the SmartServer log files. To retrieve the SmartServer log files (most, but not all, built-in log files), use the commands shown in the example below to tar and copy the /var/log and then save into /var/data//var/apollo/data/logs.tar.gz. The chown command is used to change the tar file so that you can copy the tar file with the apollo login. Example: Retrieves SmartServer files sudo tar -czvf /var/apollo/data/logs.tar.gz /var/log sudo chown apollo /var/apollo/data/logs.tar.gz Use an SFTP application (like WinSCP) to save the files to your PC. You can then use 7-Zip (or a similar application) to extract the files. |
| timedatectl | Displays or sets the SmartServer platform time (time changes are not propagated to CMS). The SmartServer CMS and the SmartServer platform time and timezone may differ. See also date above. Example: timedatectl $ timedatectl Local time: Mon 2022-12-12 14:52:58 PST Universal time: Mon 2022-12-12 22:52:58 UTC RTC time: Mon 2022-12-12 22:52:58 Time zone: America/Los_Angeles (PST, -0800) System clock synchronized: yes NTP service: active RTC in local TZ: no |
| top | Provides a dynamic, real-time view of a running SmartServer system. Enter CTRL-C to exit. Idle time id shows the idle time. A large number (65.1 in the example below) for idle time indicates that the SmartServer has a lot of time extra processing time. This command can be used to determine whether the SmartServer is performing a software update using the GLPO file. When a GLPO update is in progress, you will see wget, followed by unzip, and then mender using the most resources. The CPU will go down as the update nears completion. Example: top $top
top - 14:11:24 up 7:04, 1 user, load average: 3.45, 2.77, 2.40
Tasks: 183 total, 4 running, 178 sleeping, 0 stopped, 1 zombie
%Cpu(s): 21.6 us, 12.2 sy, 0.0 ni, 65.1 id, 0.3 wa, 0.0 hi, 0.8 si, 0.0 st
KiB Mem : 2060972 total, 560516 free, 1259012 used, 241444 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 707228 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1549 apollo 20 0 756796 519496 6832 S 5.2 25.2 86:18.16 java
6649 postgres 20 0 209740 60356 53252 S 4.1 2.9 1:17.62 postgres
1 root 20 0 36624 9032 4612 R 2.7 0.4 36:20.92 systemd
1372 apollo 20 0 89500 5704 4128 S 2.4 0.3 39:01.07 echbacnet
499 message+ 20 0 6096 2836 2348 R 2.3 0.1 25:53.07 dbus-daem+
2978 apollo 20 0 181160 14216 4548 S 2.0 0.7 23:35.19 runner
733 apollo 20 0 7460 4860 3500 S 1.4 0.2 15:24.15 mosquitto
1522 apollo 20 0 22092 3620 2728 S 1.4 0.2 26:14.44 echmodbus
1947 apollo 20 0 198984 58696 11908 S 1.4 2.8 14:47.22 echltx
674 apollo 20 0 197464 57460 9060 S 1.2 2.8 11:07.83 node-red
508 root 20 0 12952 4268 3604 S 1.1 0.2 12:04.99 systemd-l+
837 apollo 20 0 32332 2876 2052 S 0.4 0.1 6:28.37 redis-ser+
1484 apollo 20 0 185992 49016 9080 S 0.4 2.4 9:19.66 echalarm
12377 apollo 20 0 4716 2212 1712 R 0.4 0.1 0:00.80 top
1006 root 20 0 6524 2308 1924 S 0.3 0.1 2:39.87 ledscript+
1501 apollo 20 0 173396 33868 7804 S 0.2 1.6 3:50.14 echdpget
10 root 20 0 0 0 0 R 0.2 0.0 2:39.28 rcu_preem+
|
| ufw | Displays which ports are open, and also opens/closes ports. Example: Displays port status $ sudo ufw status Status: active To Action From -- ------ ---- 22 ALLOW Anywhere 80 ALLOW Anywhere 5353 ALLOW Anywhere 2541/udp ALLOW Anywhere 8883 ALLOW Anywhere 1883 ALLOW Anywhere 47808/udp ALLOW Anywhere 47809/udp ALLOW Anywhere 41797/udp ALLOW Anywhere 443 ALLOW Anywhere 47808/udp ALLOW OUT Anywhere 47809/udp ALLOW OUT Anywhere 49152:65535/udp ALLOW OUT Anywhere Example: Opens a port sudo ufw allow 1883 sudo ufw allow 50012/tcp Example: Closes the port that was opened sudo ufw delete allow 50012/tcp |