Many issues are resolved by updating to the latest SmartServer software. Review the SmartServer release notes before upgrading or re-imaging to a different software version. For issues that require diagnostics, this section provides some helpful hints to troubleshoot various issues that you may experience using the SmartServer.

In addition, a list of useful console/SSH commands for troubleshooting and checking the SmartServer health are provided (see Using Console / SSH commands). For help outside the scope of these sections, go to the Knowledge Base (KB). See Contacting Support below to open a support support ticket.

Starting with software release 4.2 and higher, you may see a "WARNING - POTENTIAL BREACH" message when try to log into SSH or SFTP (this warning message is expected and can be usually be ignored, see KB1458 for more details).

Before starting the warranty return process for a SmartServer, or if the warranty has run out, first try re-imaging the SmartServer since this operation may fix SmartServer issues. In some cases, in addition to re-imaging you may need to do a reset database or restore to factory to resolve some issues. Backup the SmartServer settings or database prior to upgrading or re-imaging.

SmartServer software versions are hardware specific. Before you contact the Support team, consider upgrading to the latest SmartServer version to ensure that you are running the latest SmartServer features and fixes. See Determining SmartServer Software Version Compatibility before upgrading to verify compatibility for your update.

Device capacity and maintenance licenses may be required for SmartServer 3.3 and higher depending on the application. Not all applications require licenses. Maintenance License are required to upgrade the SmartServer to a newer software version. See Manage Licenses and Frequently Asked Questions - Licensing for more information.

This section consists of the following:

Warnings

The following list provides use-case specific warnings for the SmartServer. These warnings are in addition to the warnings listed for your SmartServer release in the SmartServer Release Notes.

BACnet – Use SmartServer 3.6 Update 3 or higher. SmartServer 3.6 Update 3 includes fixes for BACnet broadcast message handling.
LoRaWAN – Use SmartServer 4.2 or higher.
LON IP-852 routing – Use SmartServer 3.5 or higher. SmartServer 3.5 improves support for disconnected or unreliable LAN or WAN Ethernet connections.
LON U60 and U70 network interface noise issues – If you are using a LON U60 or U70 network interface that is attached to a LON FT, TP-1250, or PL-20 channel with very high levels of electrical noise, then the noise can cause USB communication failures in the SmartServer. One symptom of electrical noise is an EMI? error in the /var/log/kern.log file. You can reduce the electrical noise that enters a SmartServer from the LON channel by installing a ferrite clamp on core filter onto the U60 or U70 USB cable. An example filter is available at https://www.mouser.com/ProductDetail/623-0431164281.
LON PL-20 power line networks using the U70 network interface – Use SmartServer 3.6 Update 5, 4.1 (Beta), or newer, with U70 Firmware 2.0. If your U70 has U70 Firmware 1.0, then you can update the firmware as described in Updating the U70 Firmware. The SmartServer and U70 must have compatible versions as described in the U70 Firmware release notes. U70 Firmware 2.0 improves the reliability of the communication between SmartServer and U70 and fixes the issue that causes the U70 to stop working and show a CENELEC error on the SmartServer Configuration page LON tab.

Troubleshooting Tools

The following tools are useful for SmartServer troubleshooting:

A console/ssh application, like putty.
1. Use an USB cable to access the SmartServer console to determine the SmartServer IP addresses and to watch the boot process.
2. Use SSH for most other console/SSH commands.
An SFTP application, like WinSCP, to copy and read files on the SmartServer.
MQTT Explorer to monitor MQTT traffic on the SmartServer platform.
1. Requires IAP/MQ ports to be open (SmartServer Configuration page Features tab) before you can view MQTT traffic.
2. Enhanced Security (available with SmartServer 3.5 and higher, System Configuration page) defaults to these ports being disabled.
LON devices only – LonScanner Protocol Analyzer to monitor channel bandwidth and packets.
1. You can download the LonScanner Protocol Analyzer for free. Use the LonScanner 5 link.
  1. Most LON channels are half duplex; therefore, maintain a sustained bandwidth below 40% for optimal performance.
  2. For SmartServers using power line repeating.
    1. LonScanner Protocol Analyzer allows you to see all packets for devices in which the SmartServer can directly communicate to the destination device.
    2. Packets between repeaters, or between a repeater and the destination device, are typically not shown in LonScanner.
  3. For SmartServer networks where you also have a U10/U60 or U20/U70, you can open two LonScanner windows to check communication at both the SmartServer and a problem device.
    1. With LonScanner Protocol Analyzer, you can see if network traffic makes it through the network and whether the destination device responds.
      1. If you do not see a response either there is an issue with the channel (e.g., FT-10 there is a short or broken connection) or the device is down
      2. If you see repeated packets and a delayed response this may mean the channel bandwidth is too high, the device is taking too long to respond back or there are impairment on the Lontalk channel.
      3. Some channel impairment, like motors running may cause a temporary interruption in communication.
    2. To see packets at both ends (at the SmartServer and at the edge device) you will need to take your laptop near the problem device and use a U10/U60 or U20/U70 to look at traffic at the device.
      1. Open a LonScanner window for the SmartServer LonScanner RNI.
        Verify that the Protocol Analyzer is enabled on the SmartServer LON Configuration page and that a LonScanner RNI is created in the control panel LonWorks Interface applet.
      2. Open another LonScanner window for the U10/U60 or U20/U70.
2. NodeUtil diagnostic tool allows you to determine whether you can communicate to a device.
  1. Example: if you are using a SmartServer RNI "x.default.176"
    nodeutil -dx.default.176

Hardware LEDs and Buttons

SmartServer IoT LEDs and Buttons – to determine if the SmartServer hardware is operating properly.
U60 LEDs
U70 LEDs and Buttons

Troubleshooting Scenarios

1. I cannot determine the SmartServer IP address, or I am having IP connection issues.

To find the SmartServer IP address, connect your laptop to the SmartServer console port and run ifconfig, or if your PC is on the same IP subnet as the SmartServer, then try accessing it using the SmartServer host name (e.g., for host name 17q2eyh, use https://smartserver-17q2eyh.local). See Step 2 - Connect to Your SmartServer IoT. When using the ifconfig command from the console, either log in as root or enter sudo ifconfig.

When you first access your SmartServer, follow the procedure in the Configuring the SmartServer IoT IP Address and Hostname section to change the LAN/WAN ports that you are using from Startup Mode to Static IP Address or DHCP. Failure to do so may result in flaky Ethernet connections especially if you unplug and re-plug in the SmartServer, or if there are issues on the Ethernet connection. This may happen because startup mode looks for DHCP, and if not found, it falls back to a static IP address. Therefore, if you experience issues on the Ethernet channel, or if you unplug and re-plug in the Ethernet cable, then the SmartServer may not know whether it is using DHCP or static IP addresses. Changing the Ethernet ports to Static IP Address or DHCP when you first access the SmartServer eliminates this problem.

2. The SmartServer requires that I change the password.

Starting with SmartServer 3.5, enhanced security was added as a default (System Configuration page). If you update the SmartServer software, then enhanced security is disabled if the older software did not support enhanced security, or if enhanced security was disabled. If you re-image the SmartServer to 3.5 or higher, then enhanced security is enabled by default. Perform one of the following:

Change the password and continue using enhanced security.
Change the password and disable enhanced security (using SmartServer Configuration Page System tab).
Disable enhanced security, or disable the enhanced security password checking so you do not have to change the password. Perform one of the following using console/SSH:
Disable enhanced security from console/ssh
```
sudo smartserver-secure -all
```
Disaable only password checking but use all other enhanced security features
```
sudo smartserver-secure -pwd
```
Once you have disabled enhanced security, close the existing web page that is pointing to the login web page and open a new web page to access the SmartServer.

3. I forgot the SmartServer password, or have password issues.

There are two types of user logins: apollo, the primary user that is used to configure the SmartServer (e.g., configure the SmartServer IP Addresses), and end-user.

End-user
1. End-user passwords can be recovered by clicking Forgot Password on the SmartServer CMS login page, which causes a reset password message to be sent to the user, if SMTP is configured.
2. An owner user type can click action button for the user that is having a problem from the Users widget and select the Reset Password action, which causes a reset password message to be sent to the user, if SMTP is configured.
3. If SMTP is not configured, then you may need to remove the user using the CMS Users widget, and then add the user with a new password.
Apollo user
1. When you change the apollo password, the password needs to be changed in three areas: SSH/console/SFTP, the Configuration page, and the CMS.
  1. When you change the apollo password using the System Configuration page, then all three areas are changed. It is recommended that you only change the apollo password using System Configuration page.
    1. If the CMS does not accept the new apollo password, then try logging in with the old password. If that works, then use the Users widget to change the apollo password to the new password.
  2. If you change the apollo password using the Users widget, then only the CMS gets changed and the SSH/console and Configuration page will still use the old password.
2. If the apollo password works for SSH/console, SFTP, and the Configuration page, but not for the CMS web page, and you just changed the password using the System Configuration page Change Password button, then perform the following:
  1. Reboot the SmartServer.
  2. Log into the CMS using the old password. If this is successful, then go to the Users widget and change the password to the new password.
3. Use password recovery, if enabled on the System Configuration page to convert the password back to the original password.
  1. If you restored a Full System image, then the original password is the password of the SmartServer from which you made the backup.
4. If the password problem still persists, then you will need to change the SmartServer password and perform a restore to factory settings, which will delete your current configuration (i.e., CMS and IP address settings).
  1. If you changed the apollo password, then restore the password using the console port from the U-boot prompt. Contact the EnOcean support team for instructions.
  2. Next do a restore to factory settings using the System Configuration page Reset To Defaults button.

4. I cannot access the SmartServer Configuration page.

The SmartServer Configuration page allows you to specify the SmartServer IP address and other hardware configurations.

Verify that you are you using https://<smartServer IP address>. http will not work.
Does the web page show an error similar to NET:ERR_CERT_DATE_INVALID when using signed certificates and when entering the following in the URL: https://<smartserver hostname>.echelon.cloud?
1. This error can indicate that the certificate timed out. The SmartServer will try to automatically renew the certificate, but may not be able to do so.
2. To continue using signed certificates, manually re-new the certificate using KB1455.
3. Alternatively, you can use the SmartServer IP address instead of the SmartServer hostname.
You may see not be able to access the SmartServer Configuration page and CMS if the HTTP and HTTPS ports changed from the defaults.
1. In this case, you can ping the SmartServer, but not access the SmartServer Configuration page or CMS.
2. Issue the following console command: sudo grep "listen" /etc/nginx/sites-enabled/apollo.conf. The line with ssl should show 443 and there should be two lines showing 80.
3. If 443 and 80 are not shown then you may need to change the apollo.conf file to use the default ports. See Managing SmartServer IoT Ports and Services.
Verify whether the SmartServer is constantly rebooting by looking at the SmartServer console port.

5. I cannot access the SmartServer CMS.

Verify that you are you using https://<smartServer IP address>. http will not work.
Determine whether you can log into the SmartServer Configuration page.
1. If you are on a VPN and you are using the SmartServer hostname to access the SmartServer, then clicking the CMS tab on the SmartServer Configuration page will not work.
  1. Changing the hostname to the SmartServers IP address in the web page URL should allow you to access the CMS.
2. Go to the Network Configuration page and make sure you are not in startup mode for the Ethernet ports that you are using.
  1. If startup mode is assigned, then you may see flaky Ethernet connections.
  2. Change the IP address to static or DHCP and then reboot the SmartServer.
Does the web page show an error similar to NET:ERR_CERT_DATE_INVALID when using signed certificates and when using something like https://smartserver-17qehie.echelon.cloud/cms?
1. This error may mean that the certificate timed out. The SmartServer will try to automatically renew the certificate, but may not be able to do so.
2. To continue using signed certificates, manually re-new the certificate using KB1455.
3. Alternatively, you can use the SmartServer IP address instead of the hostname.
Determine whether you can ping the SmartServer from your PC.
1. Open a Windows command prompt.
2. Issue a ping command with your SmartServer IP address (e.g., ping 10.0.0.113).
Determine whether you can you log into the SmartServer Configuration page.
1. You may not be able to access the SmartServer Configuration page and CMS if the HTTP and HTTPS ports changed from the defaults HTTP (80) and HTTPS (443).
  1. You should be able to ping the SmartServer, but not access the SmartServer Configuration page and CMS.
  2. Issue the following console command: sudo grep "listen" /etc/nginx/sites-enabled/apollo.conf. The line with ssl should show 443 and there should be two lines showing 80.
  3. If 443 and 80 are not shown then you may need to change the apollo.conf file to use the default ports. See "Changing HTTP and HTTPS Ports" for details.
Determine whether you can use SSH or SFTP (e.g., using WinSCP) to the SmartServer.
If you are using signed certificates, then check the cert file permissions after cert renewal (should be apollo apollo) using WinSCP or SSH
1. Using WinSCP or SSHSSH issue the following command and make sure the last updated file has apollo apollo:
  Make sure files are assigned to apollo apollo
```
sudo ls -l /var/apollo/data/certs/smartserver-*.echelon.cloud/fullchain-*
```
2. Workaround: if files show root as owner.
  Workaround signed certificate assigned wrong user
```
sudo chown apollo:apollo /var/apollo/data/certs/smartserver-*.echelon.cloud/fullchain-*.pem
sudo chown apollo:apollo /var/apollo/data/certs/ssiot.crt
```
  Reboot the SmartServer after making changes.
If you are using self-signed certificates, some SmartServer versions try to automatically renew these certificates.
1. Whether the certificates are renewed or not
  1. Distinguishing between a valid self-signed certificate and an expired one is not very obvious. Both are reported as not secure by browsers.
2. Manually renewing self-signed certificates
  SmartServer 3.5 and 3.6 (does not work on newer versions)
```
sudo ./sbin/update-sscert
```
If you are connected to the SmartServer console, then issue ifconfig to determine the IP address.
1. When using the ifconfig command from the console, either log in as root or enter sudo ifconfig.
2. Determine whether the SmartServer is constantly rebooting.
Verify that another device is not using the same IP address.
1. If another device or PC is using the same IP address, then accessing some pages, SSH, or SFTP may work and the other may not.
2. Unplug the Ethernet cable from the SmartServer and ping the SmartServer IP address to see if another device responds.
  1. Ping response may be disabled by the IT department or the other device may not respond back to a ping.
Unplug and plug back in the Ethernet cable to the SmartServer.
Check if any space in the SmartServer has 100% usage, as this could cause problems.
1. Run df -h.
For SmartServer 4.3 and prior, check if the SmartServer CMS (Karaf) service is running.
1. Run sudo systemctl status karaf using console/SSH. If you see a green response, then most likely the CMS service is running.
For SmartServer 4.3 and prior, try restarting the karaf.
1. Run sudo systemctl restart karaf using console/ssh.
2. If the restart works, then wait at least 5 minutes and then reboot the SmartServer to make sure the CMS works after a reboot.
Rebuild the SmartServer configuration.
1. If you can log into the Configuration page using the apollo username and password, but cannot log into the CMS using the same password and the apollo username, then perform the following:
  1. Rebuild the SmartServer; backup the SmartServer settings.
  2. Delete the current SmartServer database using one of the following:
    1. Restore to factory settings using the System Configuration page Reset To Defaults button.
    2. Re-image the SmartServer.
  3. Rebuilt your SmartServer configuration.
If you need to contact the Support team, then have the following information available (see the Contacting Support section below for details):
1. All /var/log files.
2. Results from sudo systemctl status karaf (for SmartServer 4.3 and prior).
3. Karaf log files (for SmartServer 4.3 and prior).
Check if the SmartServer is constantly rebooting by looking at the SmartServer console port.

6. The SmartServer CMS shows incorrect information.

In most cases, if the SmartServer CMS shows incorrect information, then this problem is related to cached data in the web browser. To clear cached data, perform one or more of the following:

Click the Refresh button () on the CMS web page / widgets.
Enter CTRL-F5 in the web browser.
Open a private window in the web browser.
Close all browser windows that are accessing this SmartServer.
Reopen one web browser window.
1. Enter CTRL-F5.
Optionally clear the cached in the web browser.
Check the Alarms and Events widget for warning or error messages that are related to what you are doing.

7. SSH, SFTP (WinSCP), and MQTT no longer work.

If you re-image the SmartServer to 3.5 or higher, then the SmartServer defaults to enhanced security, which automatically disables SSH, SFTP, and MQTT ports. You need to manually re-enable these features using the SmartServer Firewall Configuration page.

8. The SmartServer licenses disappear (device capacity and maintenance licenses).

Device and maintenance licenses are not affected (deleted or modified) by software upgrades (GLPO file) or re-image.

If you perform a software upgrade or re-image to SmartServer 3.6 and higher, and you did not have a valid maintenance license, then the SmartServer will disable any device related processes even if there is a valid device capacity license. Activating a maintenance license will resolve this issue.

If the SmartServer had been previously licensed and all the SmartServer Licenses disappear, then perform the following steps:

Re-enter the customer ID and re-activate the device capacity and maintenance licenses.
1. If you try to re-enter customer ID, but the dialog just seems to spin:
  1. Check if the customer ID is correct.
    1. If the customer ID is correct and this is a lab SmartServer, or a SmartServer that you can re-image, then try re-imaging the SmartServer.

Retrieve the license specific log (/var/apollo/data/lic_data*) and setup files using the following console/SSH commands:

sudo tar -czvf /var/apollo/data/lic_data-${APOLLO_INSTALL_CODE}-$(date +"%Y%m%d%H%M%S").tar.gz /var/.slm /var/apollo/data/certs/lic/.data/lservrc /var/log/supervisor/lim.log /var/log/supervisor/supervisord.log /var/log/syslog

sudo chown apollo:users /var/apollo/data/lic_data*

Get MQTT data.
Get MQTT data with ssh/console
```
mosquitto_sub -v -t glp/0/+/fb/lic/cfg -t glp/0/+/fb/lic/sts --retained-only
```
If you have devices that are purple but you have sufficient device credit, then run this MQTT request shown below and send Support the log file starting with /var/apollo/data/retained_mqtt /var/apollo/data.
Sufficient device credit, but seeing purple devices
```
mosquitto_sub -t "glp/#" -F "%J" --retained-only > /var/apollo/data/retained_mqtt_${APOLLO_INSTALL_CODE}-$(date +"%Y%m%d%H%M%S")_log.txt
```
Use an SFTP application, like WinSCP, to get the tar file /var/apollo/data/lic_data*.
Look at the following license log for any issues: /var/log/supervisor/lim.log.
1. Use one of the following:
  1. Console/SSH and enter the following command:
    cat /var/log/supervisor/lim.log
  2. WinSCP to unzip the tar file.
2. If you see Failed with status 510046 in lim.log, then perform the following:
  Console/ssh workaround (only applies to the 510046 failure)
```
sudo supervisorctl stop core:echlim
su
rm -r /var/.slm/
sudo supervisorctl restart core:echlim
sudo supervisorctl reload
```
3. If you still have a problem, then reboot the SmartServer.
4. If you are still not able to resolve this issue, then contact Support as described below.
If you need to contact the Support team, provide the the SmartServer install code that is displayed on the SmartServer System Configuration page or the label on back of SmartServer.

9. The SmartServer shows up as unprovisioned in CMS Devices widget or the System Configuration page.

This problem can occur for the following reasons:

The signed or unsigned certificate has timed out.
1. Using ssh, enter the command shown below to see if the SmartServer ID is returned.
  
  You should see the SmartServer ID in the response. If you do not see the SmartServer ID, then the certificate may have timed out.
```
mosquitto_sub -t glp/0/././#
"17qam88"
```
2. Click Ctrl-C to exit the command.
Workaround:
1. Select the Signed Certificates option on the System Configuration page. Wait for the change to take effect (the checkbox changes from greyed-out to active, or wait 15 seconds), and then click the option again. The system interprets this process as switching from your current mode of operation (either signed or not signed) and then back again. Finally, reboot the SmartServer.

10. IzoT CT or IzoT Net Server tool do not see Service messages (e.g., IP-852 router) from the LON SmartServer LON Configuration page.

All IP-852 and U60/U70 routers have a service/connect button in the LON Configuration page that send a service pin message.

Routing is only available when using IMM (DMM is the default).
1. Make sure the SmartServer is configured for IMM.
  1. Look at the System Configuration page (LON Network Management Mode shows Independent or CMS shows IMM).
  2. To change to IMM, go to the CMS Devices widget and select the Segment Controllers tab. Then click the SmartServer's Action button () and select the Switch to LON independent Management Mode action.
2. The LON Configuration page allows you to send service pin messages for the routers.
  1. IzoT CT and IzoT Net Server neuron IDs are called MAC addresses on the LON Configuration page.
3. Every time you re-image, reset the database, or restore to factory defaults, these operations change the MAC addresses. With these operations, re-commission the SmartServer in the LNS database with the new neuron IDs.
4. If you are using enhanced security (System Configuration page), then outgoing ports that are used for routing are closed and you will need to do one of the following using ssh:
  1. Disable enhanced security.
    Disabling enhanced security
    sudo smartserver-secure -all
  2. Open any outgoing ports.
    Check which ports open
    sudo ufw status verbose
    Look at the Config Server to see which ports you need to open (if you are using the default 1628 port).
    sudo ufw allow out 1628/udp
5. Some of the neuron IDs (called MAC IDs on the SmartServer) are displayed on the LON Configuration page.
6. If you do not see service pin messages in the integration tool, refer to the following steps:
  1. Workaround:
    1. You may need to perform a restore-to-factory for the IP-852 service pin messages to work properly after performing a re-image.
    2. Try to manual add the neuron ID using the router MAC address that is displayed in the LON Configuration page.
  2. Check the following to see if there is something mis-configured on the SmartServer using console/SSH.
    ssh/console - send to support team
    1. Show ethaddr_space=32: sudo fw_printenv | grep "ethaddr_space" 2. Verify that the assignedmacs owner is apollo: ls -l /var/apollo/data/router/ -rw-r--r-- 1 apollo users 0 Feb 9 11:04 assignedmacs --- needs to be apollo -rw-r--r-- 1 apollo apollo 21 Feb 9 06:14 lonNi.conf 3. List the available mac addresses: cat /var/apollo/conf.d/macaddrs 4. List the mac assignment: cat /var/apollo/data/router/assignedmacs

11. SmartServer routers stop working in an IzoT CT or IzoT Net Server network after a SmartServer reboot.

This problem usually indicates that the SmartServer is configured for DMM (the default).

In DMM, the SmartServer is the network integration tool. Therefore, if you commissioned the SmartServer routers in an IzoT CT/IzoT Net Server network, and the SmartServer reboots, then the SmartServer will see that its router configuration is not what is expected for DMM and it will re-commission it for DMM. As a result, your IzoT CT or IzoT Net Server network will lose all access to the SmartServer routers and edge devices that are connected to the SmartServer U60/U70 channels.

Changing back and forth between IMM and DMM is not supported. Therefore, you must first discern which mode of operation (DMM or IMM) your application requires and then maintain that mode. For IzoT CT and IzoT Net Server networks, when you are using the SmartServer routers, you will first need to change to IMM before commissioning the SmartServer routers in IzoT CT or IzoT Net Server. See the workaround below if you mistakenly used DMM.

Add a ferrite clamp onto the U60/U70 USB cable to reduce LON channel noise (similar to one used for the SmartPhone as seen here: https://www.usbfirewire.com/parts/rr-fh0-500b.html).

Only SmartServer 3.50.026 and higher can be used for IP-852 routing. In prior releases, unplugging or re-plugging the Ethernet cable and possibly IP infrastructure issues could stop communication to external devices through U60s/U70s.

Workaround

Backup any settings or files that are on the SmartServer. If you are only using the SmartServer for routing or RNI, then you do not need to backup anything.
Click the Reset To Defaults button (System Configuration page).
After the reset is done, optionally do another reboot of the SmartServer using the System Configuration page. This reboot will cause the SmartServer router neuron IDs to change.
Go to the CMS Devices widget, Segment Controller tab.
Select the Switch to LON Independent Management Mode (IMM) action.
Verify that the SmartServer is IMM mode.
1. The CMS shows the IMM mode in the upper left-hand corner.
2. The System Configuration page shows the LON Network Management Mode as Independent (IMM).
In IzoT CT/IzoT Net Server, decommission the SmartServer routers and do not save the neuron IDs.
In IzoT CT/IzoT Net Server, commission the SmartServer routers with the new neuron IDs using the SmartServer LON Configuration page.

12. Other SmartServer LON routers issues.

With SmartServer 4.0, IP-852 service pin message do not work in with enhanced security.

Workaround: verify that the ports you are using are open.

13. My LON devices are all RED (down) in the CMS using U60s or U70, or I am seeing communication issues to the LON devices.

If the SmartServer Configuration page LON tab shows "The lon1 device has failed a CENELEC protocol query. Device likely needs to be reset to restore communication" then you need to use SmartServer 3.65 or 4.1 (Beta) and higher, and update the U70 software. For power-line repeating networks, update the SmartServer and U70 software even if do not see the CENELEC error.

If you experience problems with the U60 or U70, then add a USB 2.0 ferrite clamp onto the USB cable to reduce LON channel noise. See https://www.usbfirewire.com/parts/rr-fh0-500b.html

See the steps below for more information and diagnostics.

Verify that the U60 or U70 is up using one of the following methods:

Use the SmartServer Configuration page LON tab to see if the U60/U70 appears under LON interfaces.
1. Use the console/SSH command ifconfig and check the lon#. There should be one lon# per U60/U70. When using the ifconfig command from the console, either log in as root or enter sudo ifconfig.
If the U60 or U70 do not appear, then perform the following:
1. Verify whether these devices are plugged in.
2. Check for USB noise by looking for EMI? in the SmartServers.
  1. For SmartServer release 4.0 and higher (most of the important logs are now in the /var/log/syslog):
    1. Go to the System Configuration Web page and click on the Logs button which will download the file to your PC (usually downloads folder).
    2. Extract the *.tgz - on windows you can use 7-zip to extract the files.
      1. During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
    3. Extract the *.tar file.
      1. During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
      2. As some of the files in the *.tgz are symbolic links which may not be supported by Windows, wait until the progress bar is completely green and the click the Close button.
      3. Look for EMI? in /var/log/syslog, using console/SSH.
        
        sudo grep "EMI?" /var/log/syslog
  2. For Smartserver 3.6 and prior releases:
    1. Look at /var/log sys.log, kern.log, and error files using the commands below. If you see EMI? in the log files, then you need to add a filter, like a USB 2.0 ferrite bead clamp onto the U60/U70 USB cable.
      Console/ssh commands to look for looking for EMI?
      
      sudo grep "EMI?" /var/log/kern.log sudo grep "EMI?" /var/log/error sudo grep "EMI?" /var/log/syslog
3. Reboot the SmartServer.
  If the U60/U70 continuously stops working after power cycles, then perform the following:
  1. For SmartServer 3.6 only, enter the following SSH command (this command may allow the U60/U70 to come up, but it is not a fix):
    sudo /sbin/apollo-usbreset start
  2. For all SmartServer versions, add a USB 2.0 ferrite bead clamp onto the U60/U70 USB cable.

Verify that the LTE and LTX services are running.

For SmartServer release 4.0 and higher:

Check if LTE and LTX services are running. The LTE and LTX may get restarted if you import or make changes to an XIF file or re-import Resource File Set XML file.

Check the current status of these services using the following ssh commands. The example below shows the U60 working.

$ sudo smartserverctl status
alarm                              ACTIVE              1h 4min
bacnet                             ACTIVE              1h 4min
core                               ACTIVE              1h 4min
datapoint-connection-manager       ACTIVE              1h 4min
datapoint-controller               ACTIVE              1h 4min
datapoint-get                      ACTIVE              1h 4min
enocean-driver                     ACTIVE              1h 4min
formatter                          ACTIVE              1h 4min
housekeeper                        ACTIVE              1h 4min
iap-controller                     ACTIVE              1h 4min
init                               ACTIVE              1h 4min
lim                                ACTIVE              1h 4min
loader                             ACTIVE              1h 4min
logger                             ACTIVE              1h 4min
lon                                ACTIVE              1h 4min
lte                                ACTIVE              1h 4min
ltx                                ACTIVE              1h 4min
lwd                                ACTIVE              1h 4min
modbus                             ACTIVE              1h 4min
monitoring                         ACTIVE              1h 4min
mosquitto                          ACTIVE              1h 4min
node-red                           ACTIVE              1h 4min
pyro4-ns                           ACTIVE              1h 4min
query                              ACTIVE              1h 4min
reboot-manager                     ACTIVE              1h 4min
resource-publisher                 ACTIVE              1h 4min
scheduler                          ACTIVE              1h 4min
secmount                           ACTIVE              1h 4min
services                           ACTIVE              1h 4min
smartserver                        ACTIVE              1h 4min
storage-manager                    ACTIVE              1h 4min
apollo@smartserver-17qam88:~$

You can also restart these services as they may get out of sync (for example, you may have deleted a device in the CMS, but it did not get cleared from MQTT) using the following command:
```
sudo smartserverctl restart lon
```

For SmartServer 3.6 and prior releases:

Check if lon:echlte and lon:echltx services are running. The LTE and LTX may get restarted if you import or make changes to an XIF file or re-import Resource File Set XML file.

Check the current status of these services using the following ssh commands. The example below shows the U60 working.

sudo supervisorctl status


~$ sudo supervisorctl status
apollo-init                      EXITED    Dec 19 06:29 AM
core:echhousekeeper              RUNNING   pid 1159, uptime 4:21:35
core:echlim                      RUNNING   pid 1163, uptime 4:21:35
echbacnet                        RUNNING   pid 1155, uptime 4:21:35
echmodbus                        RUNNING   pid 1240, uptime 4:21:34
echopcua-server                  STOPPED   Not started
lon:echlte                       RUNNING   pid 3627, uptime 4:19:57
lon:echltx                       RUNNING   pid 3626, uptime 4:19:57
ready                            EXITED    Dec 19 06:29 AM
services:echalarm                RUNNING   pid 1208, uptime 4:21:35
services:echapollo-rp-launch     RUNNING   pid 1236, uptime 4:21:34
services:echconnection           RUNNING   pid 1214, uptime 4:21:35
services:echdatapointcontrol     RUNNING   pid 1216, uptime 4:21:35
services:echdatapointget         RUNNING   pid 1227, uptime 4:21:35
services:echformatter            RUNNING   pid 1229, uptime 4:21:35
services:echiap-controller       RUNNING   pid 1239, uptime 4:21:34
services:echloader               RUNNING   pid 1185, uptime 4:21:35
services:echlogger               RUNNING   pid 1170, uptime 4:21:35
services:echmonitoring           RUNNING   pid 1174, uptime 4:21:35
services:echquery                RUNNING   pid 1180, uptime 4:21:35
services:echscheduler            RUNNING   pid 1237, uptime 4:21:34

You can also restart these services as they may get out of sync (for example, you may have deleted a device in the CMS, but it did not get cleared from MQTT) using the following command (include the : at the end of the command):
```
sudo supervisorctl restart lon:
```

Use NodeUtil to see if device status "s" works.
1. Using NodeUtil: Make sure that an RNI is set up for the U60 or U70. Next make sure you set up a RNI in the PC Windows Control panel > LonWorks Interfaces applet.
  1. Start NodeUtil (e.g., nodeutil -dx.default.215) using a DOS command prompt.
  2. Add the device neuron Id (from the CMS Devices Widget) to NodeUtil using the "a" add command. Next click "g" followed by return and then "s".
2. If NodeUtil device status works, then:
  1. Make sure the device is configured for the correct domain Id.
    1. Press "d" for domain in nodeUtil which will show you the device's domain ids. You can look at the SmartServer Configuration Page LON tab for the current domain id.
  2. If the domain ids are correct, then restart the LON drivers as shown above.

14. BACnet is not working.

Verify that BACnet is enabled as described in Add a BACnet Interface.

SmartServer 4.1 and prior example

SmartServer 4.2 and higher example

Verify that both BACnet enable checkboxes (shown above) are checked, that the Router Configuration device instance is unique (defaults to 20039), and that the Server Configuration (network number that defaults to 8888) are unique in the BACnet network. If you are using a BACnet tool to look at LON and Modbus devices, then the Server Configuration Starting Device Instance needs to be unique in the BACnet network.

Verify that the Router Configuration table Network Numbers match what is used for the BACnet networks. This Network Number may not be used for device discovery. Some BACnet features may not work if this is set up incorrectly.

If BACnet was not enabled prior to importing BAC, BTM, or DTP files with BAC and BTM files, then you need to enable BACnet and may need to re-import these files

Verify that the BACnet service is running by issuing the following console/SSH command (depends on SmartServer version):

For SmartServer release 4.0 and higher, look for bacnet.

SmartServer release 4.0 and higher

sudo smartserverctl status

For SmartServer 3.6 and prior releases, look for echbacnet.

SmartServer 3.6 and prior releases

sudo supervisorctl status

Instructions are based on BACnet Client or BACnet Server:

BACnet Client – real BACnet devices and you use the SmartServer CMS to read and write to BACnet devices.

Verify that BACnet is enabled.

If not, installing BAC files (or DTP files that include BAC files) prior to enabling BACnet will cause the files to never get processed. You will need to re-import the BAC files after enabling the SmartServer for BACnet.

If you manually imported the BAC files, then determine whether these files worked before.

If you are using CMS device discovery, then it may take 10-30 minutes after the CMS Devices widget Discover button () indicates that device discovery completed before all the background processing of new devices is fully done. Therefore, you may not see some devices appearing immediately after device discovery, but appearing later.

Discovered BACnet device BAC files are located in the following folder: /var/apollo/data/bacnetrouter/res. The number of BAC files depends on the number of devices and whether the devices have unique datapoint lists.
1. Copy any *.bac to your PC in case you need to contact EnOcean Support.
Determine whether the number of BAC files in the folder above matches the number BACnet device types you see in the CMS Device Types widget.
Check if any BACnet devices show up using MQTT console/ssh command. Press CTRL-C to exit.
mosquitto_sub -v -t glp/0/+/fb/dev/bacnet/+/if/#

To verify that BACnet datapoints are getting updated properly, refer to the following:

To verify if datapoint events are working using MQTT console/ssh commands:
(press CTRL-C exit command)

Example datapoint : "dev/bacnet/4/if/Temperature/0/temperature"

a. Show all datapoint (any protocol) with datapoint name "temperature"

Example SSH/console command to see "Temperature" datapoint updates

mosquitto_sub -v -t glp/0/+/ev/data/# | grep "temperature/value"

b. Show all datapoints for specific device device blockName/blockIndex

Example SSH/Console command to see same BACnet datapoint from a single device

mosquitto_sub -v -t glp/0/+/ev/data/# | grep "dev/bacnet/4/if/Temperature/0"

BACnet Server – use a BACnet tool to read and write to LON and Modbus devices that are connected to a SmartServer.

Verify that you imported BACnet Type Map (BTM) files for each device type (one per program ID) after you enabled BACnet in the BACnet Configuration page. You must enable BACnet before importing the BTM files.
If you do not see LON/Modbus devices in the BACnet tool, then it is possible that the BTM file is incorrect. You can use the BTM creation tool (a web page that creates BTM files). Once these files are created, you can edit the file and then import the new file into the SmartServer CMS.

15. Modbus is not working.

Verify that the RS-485 Configuration page is configured for the Modbus RS-485 port. Only one RS-485 port can be configured for Modbus.

Verify that the Modbus service is running by issuing the following console/SSH command (depends on SmartServer version):
1. For SmartServer 4.0 and higher, look for modbus.
  SmartServer release 4.0 and higher
```
smartserverctl status
```
2. For SmartServer 3.6 and prior, look for echmodbus.
  SmartServer 3.6 and prior releases
```
sudo supervisorctl status
```
In most cases, if you can add a device and the datapoints are not showing up properly, then it is likely that there is a problem with the .MOD file.
Verify that datapoints are being updated.
Example SSH/console command to see "temperature" datapoint updates
```
mosquitto_sub -v -t glp/0/+/ev/data/# | grep "temperature/value"
```
Example SSH/Console command to see same datapoint from a single device
```
mosquitto_sub -v -t glp/0/+/ev/data/# | grep "dev/bacnet/4/if/Temperature/0"
```
Log Modbus operations (logging should only be enabled for a short period of time due to high CPU and memory demand).
1. Enable Modbus logging using the following SSH command (not the console):
```
mosquitto_pub -t idl/rq/modbus/debugMode -m 1
```
2. View real-time data using the following command:
  SmartServer release 4.0 and higher
```
sudo tail -f /var/log/syslog | grep "modbus"
```
  SmartServer 3.6 and prior releases
```
tail -f /var/log/supervisor/modbus.log
```
3. Disable Modbus logging using the following command:
```
mosquitto_pub -t idl/rq/modbus/debugMode -m 0
```
4. View the entire modbus log using the following command:
  SmartServer release 4.0 and higher
```
sudo grep "modbus" /var/log/syslog
```
  SmartServer 3.6 and prior releases
```
cat /var/log/supervisor/modbus.log
```

16. EnOcean devices are not working.

Verify that the EnOcean Configuration page has the EnOcean driver enabled.

EnOcean datapoint values are pushed from the EnOcean device. To get all updates, set up event-driven updates in the CMS Datapoint Properties widget.

Verify that the EnOcean service is running by issuing the following console/SSH command (depends on SmartServer version). For SmartServer 4.0 and higher, look for enocean.
SmartServer release 4.0 and higher
```
smartserverctl status
```
In most cases, if you can add a device and the datapoints are not showing up properly, then it is likely that there is a problem with the .eno file.
Verify that you see datapoint updates from the EnOcean device.
Example SSH/Console command to see any datapoint from an EnOcean device
```
mosquitto_sub -v -t "glp/0/+/ev/data" | grep /dev/enocean
```
Example SSH/Console command to see same datapoint from a single device
```
mosquitto_sub -v -t glp/0/+/ev/data/# | grep "dev/enocean/4/if/Temperature/0"
```
Example SSH/console command to see "temperature" datapoint updates
```
mosquitto_sub -v -t glp/0/+/ev/data/# | grep "temperature/value"
```
If you are not seeing datapoints updates, or they are sporadic, verify that the CMS Datapoint Properties widget shows that event-driven monitoring is enabled.
1. If so, then check MQTT to see if the EnOcean driver also shows that the event-driven monitoring is enabled.
  Example SSH/Console command to check if event driven enabled
```
mosquitto_sub -v -t "glp/0/+/fb/dev/enocean/+/if" | grep '"event":'
```
  Look for "event": in red. True indicates that event-driven monitoring is enabled and false indicates that event-driven monitoring is disabled. Either de-provision the device and re-provision it using the CMS Devices widget, or in the CMS Datapoint Properties, select polling and then select event-driven, and then click the update button.
View SmartServer logs for EnOcean devices operations.
1. View real-time data using the following command:
  SmartServer release 4.0 and higher
```
sudo tail -f /var/log/syslog | grep "enocean"
```
2. View all EnOcean log entires using the following command:
  SmartServer release 4.0 and higher
```
sudo grep "enocean" /var/log/syslog
```

17. LoRaWAN is not working.

Verify that the LoRaWAN Configuration page has LoRaWAN enabled, and that ChirpStack is enabled and is configured properly.

LoRaWAN datapoint values are pushed from the LoRaWAN device. Some devices only push data once every 24 hours. Many sensors will send once an hour. To get all updates, set up event-driven updates in the CMS Datapoint Properties widget.

Verify that the LoRaWAN service (called lwd) is running by issuing the following console/SSH command (depends on SmartServer version). For SmartServer release 4.2 and higher, look for lwd.
SmartServer release 4.2 and higher
```
smartserverctl status lwd
```
If devices do not show up, then datapoints are being seen or datapoint updates are not happening. Some devices only send updates once a day.
1. Go to the LoRaWAN gateway and make sure it is set up correctly.
  1. Make sure that the LoRaWAN devices are working.
    1. Look at device data and check for logs.
2. Check the Chirpstack setup in the SmartServer.
  1. Verify that the gateway active.
  2. Verify whether devices are added.
  3. Verify whether you are receiving data from the devices.
3. If see data then check the *.lorawan file.
4. Check the SmartServer CMS.
  1. Verify whether LoRaWAN devices appear in the Devices widget (they are automatically added after the *.lorawan file has been imported and after the next datapoint update from the device which could full day).
  2. Verify whether you can see the datapoints in Datapoints widget.
  3. Verify whether the Datapoint Properties widget shows that event-driven updates are enabled.

View real-time data using the following command:
SmartServer release 4.2 and higher
```
sudo tail -f /var/log/syslog | grep "smartserver-lwd"
```
View the entire LoRaWAN log using the following command:
SmartServer release 4.2 and higher
```
sudo grep "smartserver-lwd" /var/log/syslog
```

18. I used device discovery but my devices are not showing up.

When the Discover button () shows that device discovery is complete, it means that the SmartServer is finished the process of looking for devices. However, it will still take some time for the SmartServer to fully add all of the devices. This process can take more than 10 minutes, after the Discover button has a line though it, for all the devices to appear in the CMS Devices widget.

You may also need to run device discovery multiple times in order to discover all of the devices (particularly in the case of BACnet networks).

BACnet
1. Make sure you enabled BACnet using the BACnet configuration page prior to running a device discovery.
  1. Each time you run device discovery, the discovery process will create new devices for unknown devices, and check existing BACnet devices to make sure that the datapoint information has not changed. If it has, then a new device type will be created for these devices.
Modbus

Modbus protocol does not have a built-in way to discover Modbus devices, but the SmartServer can discover then if you specify a marker value in the MOD file.
Verify that you configured the Modbus using the RS-485 Configuration page prior to running a device discovery.
Verify that you imported the MOD file prior to running device discovery.

LON devices
1. Verify that you imported all device XIF files and any necessary Resource File Set XML files prior to running device discovery.
2. Devices that appear in the CMS but are not working are generally devices in which the device XIF, or necessary Resource File Set XML files, were not imported prior to running device discovery. These devices are not usable.
  1. Delete these devices and the corresponding device types.
  2. Import the device XIF, or any necessary Resource File Set XML files, and re-run device discovery.

LON devices only - DMM device discovery - caution about discovering previously installed LON devices

If you discover and provision a previously installed LON device when using DMM, then the SmartServer may modify the configuration of the device. If the device is part of an operational system, then the system may no longer function as expected due to the change in the LON device configuration.

Discovering and/or provisioning a discovered device may change the device's subnet/node IDs, and set the existing datapoint values to default value, and delete all existing connections to devices.

To restore the site back to normal operation, perform one of the following:

Restore the network using integration tool that was originally used to create the site network.
1. Reconfigure discovered LON devices using the SmartServer CMS.

LON devices only - DMM device discovery - caution about deleting LON devices

Deleting a discovered LON device (using the Remove action in the Devices widget) will cause the SmartServer to try to decommission the device and set the subnet and device ID to 0 before deleting the device from the SmartServer. This process may cause a previously working device to stop working.

To delete a discovered LON device, first make sure the device is no longer connected to the network (e.g., FT channel), or disconnect the SmartServer from the network (such as disconnect the U60), and then delete the device in the CMS Devices widget.

Doing a reset database will not affect the existing devices.

19. Device discovery never seems to stop.

Device discovery can take up to 45 minutes. If device discovery is not complete (i.e., the Devices widget discovery button is still greyed-out) after 1.5 hours, then it is possible that one of drivers restarted and interfered with the SmartServer being able to complete the device discovery process. In this case, reboot the SmartServer.

20. LON IzoT Net Export (inex) issues.

inex is used to export a subset or complete IzoT CT drawing or IzoT Net Server database into the SmartServer in order to access device datapoints. inex only works with CT or IzoT Net Server 4.2. Use inex 1.50.016+ (make sure the SmartServer is supported; check the SmartServer release notes).

Change the SmartServer to IMM prior to running inex. Failure to change to IMM prior to running inex will cause the IP-852 and U60/U70 routing and datapoint access to stop working after a reboot.

Any required Resource File Sets must be converted to XML files and then imported prior to importing device XIF files or running inex. A SmartServer IoT device needs to be added to CT for each SmartServer IoT device you are going to use. The SmartServer IoT device needs to be added to a IP-70 channel (each SmartServer needs its own IP-70 channel), which may require you to add additional IP-70 or U60 channel. Only devices in the CT subsystem or child subsystem will be exported to the SmartServer.

Issues:

Did you change the SmartServer to IMM before using inex?
1. If not, then reset the SmartServer database, change to IMM using the SmartServer CMS Devices widget, and go through the inex import process again.
inex cannot find the SmartServer IoT device in CT or IzoT Net Server.
1. Did you add a SmartServer IoT device to a SmartServer IP-70 channel in your CT drawing or IzoT Net Server database?
2. If not, then add the device.
inex cannot reach the SmartServer.
1. Make sure that the MQTT ports are open; by default these may be closed. Go to the SmartServer Configuration page Firewall tab and open the ports that you are using with the inex tool.
Datapoints in the SmartServer are correctly formatted or are not working properly.
1. Did you import any required resource file set prior to running inex? Resource XML files need to be imported manually into the SmartServer CMS prior to running inex.
  1. Workaround: remove all devices and device types, then import the Resource file Set XML files, and finally run inex.
2. If you ran inex using /skipfile and did not import the XIF manually or import the device XIF into the SmartServer CMS prior to running inex, then this could cause the issue.
  1. Workaround: remove all devices and device types, import any required Resource file Set XML files, optionally import the XIF files, and run inex with /skipfiles if XIF files were already imported, or without /skipfiles for inex to export the XIF files.
Routing or datapoint access stop working after a SmartServer reboot.
1. This usually means that you ran inex while the SmartServer is in DMM.
  1. Workaround: reset the SmartServer database, change to IMM, and reimport the Resource File Set XML and XIF files, and then run inex.

21. The SmartServer constantly reboots in the U-boot prompt, or is stuck in the U-boot.

This issue can occur when re-imaging using a lower-quality USB flash drive. In this case, if the SmartServer is stuck in the U-boot, plug in a high-quality USB flash drive that you previously used for re-image, and then reboot the SmartServer. In many cases, the SmartServer will then complete the re-image process. Use quality flash storage such as a Samsung flash storage device.

If the SmartServer constantly reboots in the U-boot prompt, then you may be able to recover the SmartServer by performing the workarounds described below. If the SmartServer gets stuck in the U-boot, then try to stop the U-boot boot process.

U-boot Instructions

Connect to the SmartServer console port (SSH cannot be used).
Press and hold down any key on your keyboard (e.g., the space bar) until the boot process stops. Once it is stopped, you will have less than 30 seconds to enter commands. You can press any key to extend the timeout for another 30 seconds.
If the boot process stops, then re-save the environmental variables by issuing the following commands as one copy and paste. Doing so fixes most rebooting issues.
```
env default -a
saveenv
```
Once the SmartServer comes up, perform another reboot using the System Configuration page.
If the previous workaround does not work, then re-image the SmartServer from the U-boot prompt by following KB1438.
If you cannot stop the boot process, then contact Support as described below.

22. SmartServer does not recognize a USB flash drive.

This issue may occur when doing a re-image using lower quality USB flash drive, or when the USB flash drive is not formatted properly. Use a quality flash storage such as a Samsung flash storage device.

To resolve this problem, perform the following:

Try rebooting the SmartServer with the USB drive plugged in.
1. In older SmartServer version you will need to unplug the U60/U70, plug in the USB flash driver and reboot the SmartServer before the USB flash drive is recognized.
2. Using SSH/console, enter the following command:
```
sudo mount /dev/sda1 /media/usb0
```
3. If the problem persists, then enter the following commands:
```
sudo mkdir -p /media/usb0
sudo mount /dev/sda1 /media/usb0
```
Try using a different USB flash drive.

23. The SmartServer power line repeating does not work, or cannot communicate with power line devices.

If the SmartServer Configuration page LON tab shows "The lon1 device has failed a CENELEC protocol query. Device likely needs to be reset to restore communication" then you need to use SmartServer 3.65 or 4.1 (Beta) and higher, and update the U70 software.

For power line repeating networks, update the SmartServer and U70 software even if do not see the CENELEC error.

If you experience problems with the U60 or U70, then add a USB 2.0 ferrite clamp onto the USB cable to reduce LON channel noise. See https://www.usbfirewire.com/parts/rr-fh0-500b.html.

Once you enable repeating static/dynamic, you cannot revert to non-repeating. Doing so requires that you perform a database reset using the System Configuration page that deletes all of your settings and devices. Once repeating is enabled, you can change between static and dynamic.

You typically will enable dynamic repeating to cause the SmartServer to set up repeating chains based on power line communication. In most cases, you will use dynamic repeating so that the SmartServer periodically checks the network and sets up the best repeating path based on current channel impairments.

After you change the power line communications setting in the SmartServer LON Configuration page and press the Power Line Communications Update button, you need to reboot the SmartServer.
1. Devices appear red in the CMS – if you are using repeating (static or dynamic), then you can use the DCI console commands to see the devices' current state and repeater chains. If the CMS Devices widget shows red devices, but the dcxshowdevices shows that devices are up, then use the CMS Devices widget to test the device. This test will determine whether communication is truly down.
2. Check the SmartServer Configuration Page LON Tab to see if the U70 is working.
  1. If U70 is not working you many need to upgrade the U70 software (and SmartServer software), and add a filter. See CENELEC note above for upgrading U70 software, and U60 and U70 USB 2.0 Ferrite Bead filter note. for filter details.
    1. Even there is no issue with the U70, you may want to add these enhancements to prevent future issues.
3. Using NodeUtil – when power line repeating is enabled, NodeUtil can only be used to communicate edge devices, which can be reached directly by the SmartServer. NodeUtil cannot be used to communicate to edge devices that require a repeater since there is no way to specify a repeater path with NodeUtil.
4. Using LonScanner – LonScanner can only see traffic locally; therefore, you will only see traffic that the SmartServer can hear. LonScanner sees packets that go to devices that the SmartServer can communicate directly to (1 hop). For devices that require repeaters (2 or more hops), in most cases, LonScanner will not show the packets in between repeaters and the destination device.

24. I am having connections issues.

You can create connections with both the CMS Connections widget and using MQTT. MQTT created connections do not show up in the Connections widget. If you make changes using the Connections widget but do not see the updates, then click the Refresh button () to see the updates.

Only LON devices using DMM – why am I not able to create a peer-to-peer between edge devices or an internal device and an edge device?
1. Not all datapoints on the LON device can have a peer-to-peer connection. This applies to both input and output datapoints. Look at the device's XIF to determine if the datapoint can have a peer-to-peer connection. Each datapoint has a poll flag, which when set, means that the datapoint can only be polled. If a datapoint is poll only, see the datapoint section for details.
  1. Workarounds:
    1. Internal devices – you can change the XIF you are using such that the datapoint poll flag in the XIF file is set to a 0 and then re-import in the XIF file. You may need to change the program ID in the XIF file if you were using this same XIF file for real devices.
    2. Edge devices – you will either need to use point-to-point connections (which may slow down sending updates to sources datapoints) or compile new code for the devices with the poll flags set to 0.

25. Scheduled events astronomical sunrise and sunset times are off.

The SmartServer uses Civil Dawn and Civil Dusk for sunrise and sunset, respectively, as Civil times are reportedly more accurate for street light applications. The Calendar widget only shows a fixed sunrise and sunset time.

Determine actual sunrise (Civil Dawn) and sunset (Civil Dusk) times.
1. With SmartServer 3.62 and higher, the /var/log/supervisor/scheduler.log shows each day, or after power up, the daily Civil Dawn and Civil Dusk in local time.
2. If the times are off, check latitude and longitude using CMS Devices widget. Click the View All Properties action to see latitude and longitude.
  1. Using the CMS:
    1. Assign the device to a context.
    2. Export devices, modify the devices.csv to include latitude or longitude, and re-import the file.
  2. Using REST or MQTT API to add latitude and longitude.
3. Verify that the SmartServer is in the correct timezone by looking CMS Devices widget → Segment Controllers tab.
4. Using console/SSH, check current time by entering timedatectl.

26. I am having issues with the SmartServer data log.

The SmartServer data log is accessible through the CMS Datapoints widget, or through REST/WebSocket APIs. MQTT does not have access to the data log data.

If the SmartServer does not appear to be logging all data, then this usually means that you may be using the default logging setup. This setting is not correct and will usually mean gaps in log data. If you are using Value for Log Minimum Delta Value, then you must specify a numeric value.

Example of an incorrect setting: default logging setup on the Datapoint Properties widget Monitoring and Logging Configuration tab (this configuration may result in gaps of data log data):

Example of a correct setting: configuration for a numeric scalar datapoint (e.g, only logging temperature data if temperature value changes by one degree since the last logged data value):
1. Log settings:
  1. Structured datapoint – Log Minimum Delta Value must be changed to Always or On Any Change
  2. Scalar datapoints
    1. Number datapoint – Log Minimum Delta Value can be Value, Always or On Any Change
      1. If you are using Value, then you must specify a numeric value as shown above.
        If you do not specify a numeric value, then you will get inconsistent or no log data.
    2. Enumerated datapoints (datapoints that are strings, such as HVAC_Auto) – Log Minimum Delta Value must be changed to Always or On Any Change

27. For LON devices only, I deleted a device, and now I cannot add a new device with the same name.

Only for LON devices using DMM.

You can create connections with both the CMS Connections widget and using MQTT. MQTT created connections do not show up in the Connections widget.

This problem may indicate that the LON drivers LTE and LTX crashed or are out of sync.

Restart only the LON services using SSH/console. Doing so typically allows you to add the new device.

SmartServer release 4.0 and higher

sudo smartserverctl restart lon

Restarting LON services LTE and LTX for SmartServer 3.6 and prior releases

sudo supervisorctl restart lon:

For SmartServer 3.6 and prior, include the colon : at the end of the command.

28. SmartServer internal device stops working after upgrading to SmartServer 4.0 or higher.

Perform the following steps to troubleshoot problems with edge devices:

With SmartServer 4.0 and higher, to access services and install internal devices, note the following:
1. supervisorctl is replaced with smartserverctl. For SmartServer 3.6 and prior, supervisorctl is used.
2. Update any scripts or code that use supervisorctl to use smartserverctl (the format has changed for some commands); create unit files.

29. Troubleshooting edge device issues.

Perform the following steps to troubleshoot problems with edge devices:

Go to the CMS Devices widget and check the current state.
1. If device appears to be down, but is expected to be up, then do one of the following:
  1. Click the device's Action button () and select the Test action to verify communications to the device.
  2. Go to the CMS Datapoints widget and filter on at least one of the device's datapoints, and either have the polling interval greater than 0 or click the Refresh button ().
  3. Use MQTT explorer to check device health (e.g., topic glp/0/17qam88/fb/dev/lon/2/sts health parameter).
2. For LON devices only that are using power line repeating:
  1. The CMS Datapoints widget polling interval should be set either to 0 or a very large number. Do not leave the polling interval at the default 30 seconds since this can affect power line performance.
If data logging is enabled for a datapoint on the device, then check the data log using Datapoints widget and filter on logged data.
1. Each data log entry will report whether the device is running (normal) or down.
Go to the CMS Alarms and Events widget and look for warning or error messages.
1. Check the device information, typically available in the alarm event Details or Description.
  1. Manually review the information in these columns; there is no filter capability for the columns.
  2. Filter on Raising local time column to reduce the number of events that are displayed.
    1. Raising local time is when the alarm/event occurs (based on the SmartServer local time).
  3. Export the alarms and use Excel to search for the device name.
For LON devices only:
1. Use LonScanner.
  1. Use LonScanner to check whether the device is responding to write or read requests.
  2. Use the SmartServer's built-in LonScanner RNI to look at LON traffic.
    1. To fully check a LON channel, use a laptop with its own network interface and place it near the problem device, and open LonScanner with two log windows, one for the laptop network interlace and one at the SmartServer. In this way, you can determine if a request from the SmartServer reaches the edge device, and whether the edge device response is received by the SmartServer. You can also determine the degree of delay in responses.
  3. A delayed response, due to heavy traffic on the channel or noise, can cause the device to be reported as down even though there is communication to the device.
  4. Sustained traffic on a LON FT-10 or XF-1250 channel should be 40% or lower, since this half duplex channel needs to allow some bandwidth for important traffic to get through without delay.
    1. If the SmartServer is originating all of the communication (it is the only one sending request packets), then you can have a higher sustained bandwidth value.
2. Use NodeUtil to communicate to the edge device using the SmartServer RNI or another network interface.
  1. Check whether you can communicate to the device by:
    1. Getting node status information.
    2. Looking at the device's domain/subnet/node IDs.
    3. Reading back network variable values.
  2. For power line repeating channels, you can only communicate to devices with direct communication to the SmartServer.

30. I cannot export files using the SmartServer CMS Import/Export button.

When you export settings or create backups using the SmartServer CMS Import/Export Button and click the download button, the SmartServer will open a new web page that uses the SmartServer hostname in the URL. If you do the export remotely through a VPN, or the hostname to IP address resolution does not work, then you will get an error similar to the one shown below.

Perform one of the the following:

Change the URL to use the SmartServer IP address instead of the hostname. For example:
1. URL using the SmartServer hostname that returns an error: https://smartserver-17qap31/iap/cms/storage/download/e9xbVBKle7
2. Replace the hostname with the SmartServers IP address and then click the enter key: https://10.100.16.31/iap/cms/storage/download/e9xbVBKle7
  1. If you change the HTTPS port (say from 443 to 1443) you will also need to add that: https://10.100.16.31:1443/iap/cms/storage/download/e9xbVBKle7
If the SmartServer always uses a fixed IP address, change the /etc/hosts file to resolve the hostname to IP address. See Backing up and Restoring the SmartServer for more information.

Contacting Support

Before you contact the Support team, consider upgrading to the latest SmartServer version to ensure that you are running the latest SmartServer features and fixes.

If you need to contact support, please provide the following information:

Provide the SmartServer software version. The software version appears on the Configuration page System tab, the CMS dashboard, and when you log into an SSH/console session.
1. Provide the SmartServer serial number.
  1. The SmartServer datecode is part of the SmartServer serial # is found on the SmartServer label.
    1. Only use the label serial number for RMAs.
    2. You can also see the serial # on the SmartServer Configuration Page System tab and when you ssh/console in.
      1. Note, these values are changed when restoring a backed-up or cloned system image; cannot be used for RMAs.
  2. Format: 44yywwrxxxxx, where ww is the week of the year, yy is the year, and r is hardware revision. xxxxx indicates other characters in the serial #.
    Example Serial # 442031F00100: Datecode = 2031F which means 31st week in year 2020 with Rev "F" hardware (Quad core). Rev A-D are dual core, Rev F+ are quad core.
2. For license issues also provide the SmartServer install code (Look at System Configuration page or label on back of SmartServer).
3. Provide additional information, such as:
  1. Identify when the problem started.
    1. Example: the problem happened after a GLPO update (i.e., an update using the CMS Devices widget, Update action), re-image, or restoring a previous backup.
  2. Identify whether this a new problem, or whether this problem always existed.
  3. Provide screen captures or videos of error messages, web pages, or console/SSH commands.
4. Provide SmartServer log files.
  1. For SmartServer release 4.0 and higher - Most of the important logs are now in the /var/log/syslog.
    1. Go to the System Configuration Web page and click on the Logs button which will download the file to your PC (usually downloads folder).
    2. Extract the *.tgz - on windows you can use 7-zip to extract the files.
      1. During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
    3. Extract the *.tar file.
      1. During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
      2. As some of the files in the *.tgz are symbolic links which may not be supported by Windows, wait until the progress bar is completely green and the click the Close button.
    4. The /var/log/syslog contains the most of the important log information.
      1. In older Smartserver version (prior to release 4.0) each driver (like lte, ltx, or BACnet) had their own log file. Starting with 4.0 these logs are now combined in syslog.
  2. For SmartServers 3.6 and prior, some log files cannot be copied with the apollo login.
    1. You can retrieve the log files in one of the following ways:
      1. Use WinSCP with root login (not supported if enhanced security, available with SmartServer 3.5 or higher, is enabled; see Enhancing Security) to copy all files and directories in
        /var/log
      2. If you are using SmartServer 3.5 or higher, and enhanced security is enabled, then go to the System Configuration page and temporarily disable Enhancing Security. Use WinSCP with root login to copy all files and directories in /var/log, and then re-enable enhanced security using the System Configuration page.
        /var/log
      3. Create a log tar file (like a windows zip file) using the apollo login for both SSH and WinSCP.
        Run the following two SSH commands (these commands can be used with enhanced security enabled or disabled), and use an SFTP application (e.g., WinSCP) to copy the file /var/apollo/data/logs.tar.gz.
        
        sudo tar -czvf /var/apollo/data/logs.tar.gz /var/log sudo chown apollo /var/apollo/data/logs.tar.gz
    2. For SmartServer 4.3 and prior, if you have any issues with the CMS, then provide the Karaf log files.
      1. The location of the files is dependent on the SmartServer version number.
        /home/apollo/echelon-container-devel-xx.xx.xxx/data/log/karaf.log
        where xx.xx.xxx is the SmartServer version number
        For example, SmartServer 2.61.005:
        /home/apollo/echelon-container-devel-2.61.005/data/log/karaf.log
      2. To read the last 2000 lines in the file, use the following ssh/console command:
        tail -2000 /home/apollo/echelon-container-devel-2.61.005/data/log/karaf.log
5. Provide an Export of the CMS Settings:
  1. For SmartServer 3.5 and higher, use the Import/Export button ().
  2. For SmartServer 3.4 and prior, use the CMS Settings button ().
6. Before processing any warranty returns (RMA) or if the SmartServer is out of warranty
  1. Try re-imaging the SmartServer to the latest supported software for the hardware as a re-image fixes most issues.
  2. In some cases in addition to the re-image you may need to do additional reset database or restore to factory to correct an issue.
7. Open a Support ticket:
  1. Open a support ticket at https://support.enocean.com/#login. This usually provides the fastest response time from our support team.
  2. Send an email to edge.support@enocean.com.

EnOcean Edge Portal

SmartServer Troubleshooting

Warnings

Troubleshooting Tools

Hardware LEDs and Buttons

Troubleshooting Scenarios

1. I cannot determine the SmartServer IP address, or I am having IP connection issues.

2. The SmartServer requires that I change the password.

3. I forgot the SmartServer password, or have password issues.

4. I cannot access the SmartServer Configuration page.

5. I cannot access the SmartServer CMS.

6. The SmartServer CMS shows incorrect information.

7. SSH, SFTP (WinSCP), and MQTT no longer work.

8. The SmartServer licenses disappear (device capacity and maintenance licenses).

9. The SmartServer shows up as unprovisioned in CMS Devices widget or the System Configuration page.

10. IzoT CT or IzoT Net Server tool do not see Service messages (e.g., IP-852 router) from the LON SmartServer LON Configuration page.

11. SmartServer routers stop working in an IzoT CT or IzoT Net Server network after a SmartServer reboot.

12. Other SmartServer LON routers issues.

13. My LON devices are all RED (down) in the CMS using U60s or U70, or I am seeing communication issues to the LON devices.

14. BACnet is not working.

15. Modbus is not working.

16. EnOcean devices are not working.

17. LoRaWAN is not working.

18. I used device discovery but my devices are not showing up.

19. Device discovery never seems to stop.

20. LON IzoT Net Export (inex) issues.

21. The SmartServer constantly reboots in the U-boot prompt, or is stuck in the U-boot.

U-boot Instructions

22. SmartServer does not recognize a USB flash drive.

23. The SmartServer power line repeating does not work, or cannot communicate with power line devices.

24. I am having connections issues.

25. Scheduled events astronomical sunrise and sunset times are off.

26. I am having issues with the SmartServer data log.

27. For LON devices only, I deleted a device, and now I cannot add a new device with the same name.

28. SmartServer internal device stops working after upgrading to SmartServer 4.0 or higher.

29. Troubleshooting edge device issues.

30. I cannot export files using the SmartServer CMS Import/Export button.

Contacting Support