Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Many issues are resolved by updating to the latest SmartServer software. Review the SmartServer release notes before upgrading or re-imaging to a different software version. For issues that require diagnostics, this section provides some helpful hints to troubleshoot various issues that you may experience using the SmartServer.

In addition, a list of useful console/SSH commands for troubleshooting and checking the SmartServer health are provided (see Using Console / SSH commands). For help outside the scope of these sections, go to the EnOcean Support Portal Knowledge the  Knowledge Base (KB). See Contacting Support below to open a support support ticket.

Starting with software release 4.2 and higher, you may see a "WARNING - POTENTIAL BREACH" message when try to log into SSH or SFTP (this warning message is expected and can be usually be ignored, see KB1458 for more details).

Before starting the warranty return process for a SmartServer, or if the warranty has run out, first try re-imaging the SmartServer since this operation may fix SmartServer issues. In some cases, in addition to re-imaging you may need to do a reset database or restore to factory to resolve some issues. Backup the SmartServer settings or database prior to upgrading or re-imaging.

...

  • BACnet – Use SmartServer 3.6 Update 3 or higher. SmartServer 3.6 Update 3 includes fixes for BACnet broadcast message handling.

  • LoRaWAN – Use SmartServer 4.2 or higher.

  • LON IP-852 routing – Use SmartServer 3.5 or higher. SmartServer 3.5 improves support for disconnected or unreliable LAN or WAN Ethernet connections.

  • LON U60 and U70 network interface noise issues – If you are using a LON U60 or U70 network interface that is attached to a LON FT, TP-1250, or PL-20 channel with very high levels of electrical noise, then the noise can cause USB communication failures in the SmartServer.  One symptom of electrical noise is an EMI? error in the /var/log/kern.log file. You can reduce the electrical noise that enters a SmartServer from the LON channel by installing a ferrite clamp on core filter onto the U60 or U70 USB cable.  An example filter is available at https://www.mouser.com/ProductDetail/623-0431164281.

  • LON PL-20 power line networks using the U70 network interface – Use SmartServer 3.6 Update 5, 4.1 (Beta), or newer, with U70 Firmware 2.0.  If your U70 has U70 Firmware 1.0, then you can update the firmware as described in Updating the U70 Firmware. The SmartServer and U70 must have compatible versions as described in the U70 Firmware release notes.  U70 Firmware 2.0 improves the reliability of the communication between SmartServer and U70 and fixes the issue that causes the U70 to stop working and show a CENELEC error on the SmartServer Configuration page LON tab.

...

  1. A console/ssh application, like putty.
    1. Use an USB cable to access the SmartServer console to determine the SmartServer IP addresses and to watch the boot process.
    2. Use SSH for most other console/SSH commands.

  2. An SFTP application, like WinSCP, to copy and read files on the SmartServer.

  3. MQTT Explorer to monitor MQTT traffic on the SmartServer platform.
    1. Requires IAP/MQ ports to be open (Features SmartServer Configuration page Features tab) before you can view MQTT traffic. 
    2. Enhanced Security (available with SmartServer 3.5 and higher, System Configuration page) defaults to these ports being disabled. 

  4. LON devices only – LonScanner Protocol Analyzer to monitor channel bandwidth and packets.
    1. You can download the LonScanner Protocol Analyzer for free. Use the LonScanner 5 link.
      1. Most LON channels are half duplex; therefore, maintain a sustained bandwidth below 40% for optimal performance.
      2. For SmartServers using power line repeating.
        1. LonScanner Protocol Analyzer allows you to see all packets for devices in which the SmartServer can directly communicate to the destination device.
        2. Packets between repeaters, or between a repeater and the destination device, are typically not shown in LonScanner. 
      3. For SmartServer networks where you also have a U10/U60 or U20/U70, you can open two LonScanner windows to check communication at both the SmartServer and a problem device.
        1. With LonScanner Protocol Analyzer, you can see if network traffic makes it through the network and whether the destination device responds. 
          1. If you do not see a response either there is an issue with the channel (e.g., FT-10 there is a short or broken connection) or the device is down
          2. If you see repeated packets and a delayed response this may mean the channel bandwidth is too high, the device is taking too long to respond back or there are impairment on the Lontalk channel.
          3. Some channel impairment, like motors running may cause a temporary interruption in communication.  
        2. To see packets at both ends (at the SmartServer and at the edge device) you will need to take your laptop near the problem device and use a U10/U60 or U20/U70 to look at traffic at the device.
          1. Open a LonScanner window for the SmartServer LonScanner RNI.
            1. Verify that the Protocol Analyzer is enabled on the SmartServer LON Configuration page and that a LonScanner RNI is created in the control panel LonWorks Interface applet.
          2. Open another LonScanner window for the U10/U60 or U20/U70.
    2. NodeUtil diagnostic tool allows you to determine whether you can communicate to a device.   
      1. Example: if you are using a SmartServer RNI "x.default.176"
        nodeutil -dx.default.176

...

To find the SmartServer IP address, connect your laptop to the SmartServer console port and run ifconfig or if your PC is on the same IP subnet as the SmartServer then try accessing it using the SmartServer host name (e.g., for host name "17q2eyh"  use "https://smartserver-17q2eyh.local").See Finding the SmartServer IP Address See Step 2 - Connect to Your SmartServer IoT.

When you first access your SmartSever, follow the procedure in the Configuring the SmartServer IoT IP Address and Hostname section to change the LAN/WAN ports that you are using from Startup Mode to Static IP Address or DHCP. Failure to do so may result in flaky Ethernet connections especially if you unplug and re-plug in the SmartServer, or if there are issues on the Ethernet connection. This may happen because startup mode looks for DHCP, and if not found, it falls back to a static IP address. Therefore, if you experience issues on the Ethernet channel, or if you unplug and re-plug in the Ethernet cable, then the SmartServer may not know whether it is using DHCP or static IP addresses. Changing the Ethernet ports to Static IP Address or DHCP when you first access the SmartServer eliminates this problem.

...

  1. Change the password and continue using enhanced security.

  2. Change the password and disable enhanced security (using SmartServer Configuration Page System tab).

  3. Disable enhanced security, or disable the enhanced security password checking so you do not have to change the password. Perform one of the following using console/SSH:

    Code Block
    titleDisable enhanced security from console/ssh
    sudo smartserver-secure -all


    Code Block
    titleDisaable only password checking but use all other enhanced security features
    sudo smartserver-secure -pwd


    Once you have disabled enhanced security, close the existing web page that is pointing to the login web page and open a new web page to access the SmartServer.

...

  1. End-user
    1. End-user passwords can be recovered by clicking Forgot Password on the SmartServer CMS login page, which causes a reset password message to be sent to the user, if SMTP is configured. 
    2. An owner user type can click action button for the user that is having a problem from the Users widget and select the Reset Password action, which causes a reset password message to be sent to the user, if SMTP is configured. 
    3. If SMTP is not configured, then you may need to remove the user using the CMS Users widget, and then add the user with a new password.

  2. Apollo user 
    1. When you change the apollo password, the password needs to be changed in three areas: SSH/console/SFTP, the Configuration page, and the CMS.
      1. When you change the apollo password using the System Configuration page, then all three areas are changed. It is recommended that you only change the apollo password using System Configuration page.
        1. If the CMS does not accept the new apollo password, then try logging in with the old password. If that works, then use the Users widget to change the apollo password to the new password.
      2. If you change the apollo password using the Users widget, then only the CMS gets changed and the SSH/console and Configuration page will still use the old password.
    2. If the apollo password works for SSH/console, SFTP, and the Configuration page, but not for the CMS web page, and you just changed the password using the System Configuration page Change Password button, then perform the following:
      1. Reboot the SmartServer.
      2. Log into the CMS using the old password. If this is successful, then go to the Users widget and change the password to the new password. 
    3. Use password recovery, if enabled on the System Configuration page to convert the password back to the original password.
      1. If you restored a Full System image, then the original password is the password of the SmartServer from which you made the backup. 
    4. If the password problem still persists, then you will need to change the SmartServer password and perform a restore to factory settings, which will delete your current configuration (i.e., CMS and IP address settings).
      1. If you changed the apollo password, then restore the password using the console port from the U-boot prompt. Contact the EnOcean support team for instructions.
      2. Next do a restore to factory settings using the System Configuration page Reset To Defaults button.

...

  1. Verify that you are you using https://<smartServer IP address>. http will not work.

  2. Determine whether you downgraded the SmartServer software from release 4.0 and higher to 3.xx.
    1. Starting with SmartServer 4.0, certificates and keys are now encrypted to improve security, but the older software versions do not know what to do with the encrypted certificates and keys. Consequently, a re-image from SmartServer 4.0 and higher to 3.xx will partially fail without any error message.
      1. If you have experience this issue, after downgrading, you will be able to access the SmartServer Configuration pages and the Mosquito service will not work; however, you will be able to access the SmartServer CMS.  
      2. To remove SmartServer 4.0 and higher certificates (this can be done before or after doing a re-image), perform one of the following using console/SSH commands:
        1. Using script:

          Code Block
          titleRemoving 4.0+ certificates
          sudo cd to /var/apollo/data/certs
          sudo chmod 755 prep_for_3.x.sh
          sudo ./prep_for_3.x.sh


        2. Using remove commands (you will need to modify the command based on the SmartServer hostname):

          Code Block
          titleFormat: Replace <hostname> with your SmartServer's hostname
          sudo rm /etc/ssl/certs/message_broker/
          sudo rm /var/apollo/data/certs/<hostname>.echelon.cloud


          Code Block
          titleExample: SmartServer with the hostname SmartServer-17qehie
          sudo rm /etc/ssl/certs/message_broker/
          sudo rm /var/apollo/data/certs/SmartServer-17qehie.echelon.cloud


  3. Does the web page show an error similar to NET:ERR_CERT_DATE_INVALID when using signed certificates and when using something like https://smartserver-17qehie.echelon.cloud?
    1. This may mean that the certificate timed out. The SmartServer will try to automatically renew the certificate, but may not be able to do so.
    2. To continue using signed certificates, manually re-new the certificate using KB1455.
    3. Alternatively, you can just use the SmartServer IP address.

  4. You may see not be able to access the SmartServer Configuration page and CMS if the HTTP and HTTPS ports changed from the defaults.
    1. You should be able to ping the SmartServer, but not access the SmartServer Configuration Page and CMS Web page.
    2. Issue the following console command: sudo grep "listen" /etc/nginx/sites-enabled/apollo.conf. The line with ssl should show 443 and there should be two lines showing 80.
    3. If 443 and 80 are not shown then you may need to change the apollo.conf file to use the default ports. See "Changing HTTP and HTTPS Ports" for details.

  5. Check if the SmartServer is constantly rebooting by looking at the SmartServer console port.

...

  1. Verify that you are you using https://<smartServer IP address>. http will not work.

  2. Determine whether you can log into the SmartServer Configuration page.
    1. If you are on a VPN and you are using the SmartServer hostname to access the SmartServer, then clicking the CMS tab on the SmartServer Configuration page will not work.
      1. Changing the hostname to the SmartServers IP address in the web page URL should allow you to access the CMS.
    2. Go to the Network Configuration page and make sure you are not in startup mode for the Ethernet ports that you are using.
      1. If startup mode is assigned, then you may see flaky Ethernet connections.
      2. Change the IP address to static or DHCP and then reboot the SmartServer. 

  3. Does the web page show an error similar to NET:ERR_CERT_DATE_INVALID when using signed certificates and when using something like https://smartserver-17qehie.echelon.cloud/cms?
    1. This error may mean that the certificate timed out. The SmartServer will try to automatically renew the certificate, but may not be able to do so.
    2. To continue using signed certificates, manually re-new the certificate using KB1455.
    3. Alternatively, you can use the SmartServer IP address instead of the hostname.

  4. Determine whether you can ping the SmartServer from your PC.
    1. Open a Windows command prompt.
    2. Issue a ping command with your SmartServer IP address (e.g., ping 10.0.0.113).

  5. Determine whether you can you log into the SmartServer Configuration page.
    1. You may not be able to access the SmartServer Configuration page and CMS if the HTTP and HTTPS ports changed from the defaults HTTP (80) and HTTPS (443).
      1. You should be able to ping the SmartServer, but not access the SmartServer Configuration page and CMS. 
      2. Issue the following console command: sudo grep "listen" /etc/nginx/sites-enabled/apollo.conf. The line with ssl should show 443 and there should be two lines showing 80.
      3. If 443 and 80 are not shown then you may need to change the apollo.conf file to use the default ports. See "Changing HTTP and HTTPS Ports" for details.

  6. Determine whether you can use SSH or SFTP (e.g., using WinSCP) to the SmartServer.

  7. If you are using signed certificates, then check the cert file permissions after cert renewal (should be apollo apollo) using WinSCP or SSH
    1. Using WinSCP or SSHSSH issue the following command and make sure the last updated file has apollo apollo:

      Code Block
      titleMake sure files are assigned to apollo apollo
      sudo ls -l /var/apollo/data/certs/smartserver-*.echelon.cloud/fullchain-*


    2. Workaround: if files show root as owner.

      Code Block
      titleWorkaround signed certificate assigned wrong user
      sudo chown apollo:apollo /var/apollo/data/certs/smartserver-*.echelon.cloud/fullchain-*.pem
      sudo chown apollo:apollo /var/apollo/data/certs/ssiot.crt

      Reboot the SmartServer after making changes.

  8. If you are using self-signed certificates, some SmartServer versions try to automatically renew these certificates. 
    1. Whether the certificates are renewed or not
      1. Distinguishing between a valid self-signed certificate and an expired one is not very obvious.  Both are reported as not secure by browsers.
    2. Manually renewing self-signed certificates

      Code Block
      titleSmartServer 3.5 and 3.6 (does not work on newer versions)
      sudo ./sbin/update-sscert


  9. If you are connected to the SmartServer console, then issue ifconfig to determine the IP address.
    1. Determine whether the SmartServer is constantly rebooting.

  10. Verify that another device is not using the same IP address.
    1. If another device or PC is using the same IP address, then accessing some pages, SSH, or SFTP may work and the other may not.
    2. Unplug the Ethernet cable from the SmartServer and ping the SmartServer IP address to see if another device responds.
      1. Ping response may be disabled by the IT department or the other device may not respond back to a ping.

  11. Unplug and plug back in the Ethernet cable to the SmartServer.

  12. Check if any space in the SmartServer has 100% usage, as this could cause problems.
    1. Run df -h.

  13. Check if the SmartServer CMS (Karaf) service is running.
    1. Run sudo systemctl status karaf using console/SSH. If you see a green response, then most likely the CMS service is running.

  14. Try restarting the karaf.
    1. Run sudo systemctl restart karaf using console/ssh.
    2. If the restart works, then wait at least 5 minutes and then reboot the SmartServer to make sure the CMS works after a reboot.

  15. Rebuild the SmartServer configuration.
    1. If you can log into the Configuration page using the apollo username and password, but cannot log into the CMS using the same password and the apollo username, then perform the following:
      1. Rebuild the SmartServer; backup the SmartServer settings. 
      2. Delete the current SmartServer database using one of the following:
        1. Restore to factory settings using the System Configuration page Reset To Defaults button.
        2. Re-image the SmartServer. 
      3. Rebuilt your SmartServer configuration.

  16. If you need to contact the Support team, then have the following information available (see the Contacting Support section below for details):
    1. All /var/log files.
    2. Results from sudo systemctl status karaf.
    3. Karaf log files.

  17. Check if the SmartServer is constantly rebooting by looking at the SmartServer console port.

...

  1.  Re-enter the customer ID and re-activate the device capacity and maintenance licenses.
    1. If you try to re-enter customer ID, but the dialog just seems to spin:
      1. Check if the customer ID is correct.
        1. If the customer ID is correct and this is a lab SmartServer, or a SmartServer that you can re-image, then try re-imaging the SmartServer.

  2.  Retrieve the license specific log (/var/apollo/data/lic_data*) and setup files using the following console/SSH commands:

    Code Block
    sudo tar -czvf /var/apollo/data/lic_data-${APOLLO_INSTALL_CODE}-$(date +"%Y%m%d%H%M%S").tar.gz /var/.slm /var/apollo/data/certs/lic/.data/lservrc /var/log/supervisor/lim.log /var/log/supervisor/supervisord.log /var/log/syslog
    
    sudo chown apollo:users /var/apollo/data/lic_data*


  3. Get MQTT data.

    Code Block
    titleGet MQTT data with ssh/console
    mosquitto_sub -v -t glp/0/+/fb/lic/cfg -t glp/0/+/fb/lic/sts --retained-only

    If you have devices that are purple but you have sufficient device credit, then run this MQTT request shown below and send Support the log file starting with /var/apollo/data/retained_mqtt   /var/apollo/data.

    Code Block
    titleSufficient device credit, but seeing purple devices
    mosquitto_sub -t "glp/#" -F "%J" --retained-only > /var/apollo/data/retained_mqtt_${APOLLO_INSTALL_CODE}-$(date +"%Y%m%d%H%M%S")_log.txt


  4. Use an SFTP application, like WinSCP, to get the tar file  /var/apollo/data/lic_data*.

  5. Look at the following license log for any issues: /var/log/supervisor/lim.log

    1. Use one of the following:

      1. Console/SSH and enter the following command:

        Code Block
        cat /var/log/supervisor/lim.log


      2. WinSCP to unzip the tar file.

    2. If you see Failed with status 510046 in lim.log, then perform the following:

      Code Block
      titleConsole/ssh workaround (only applies to the 510046 failure)
      sudo supervisorctl stop core:echlim
      su
      rm -r /var/.slm/
      sudo supervisorctl restart core:echlim
      sudo supervisorctl reload


    3. If you still have a problem, then reboot the SmartServer.

    4. If you are still not able to resolve this issue, then contact Support as described below.

  6. If you need to contact the Support team, provide the the SmartServer install code that is displayed on the SmartServer System Configuration page or the label on back of SmartServer.

...

Changing back and forth between IMM and DMM is not supported. Therefore, you must first discern which mode of operation (DMM or IMM) your application requires and then maintain that mode. For IzoT CT and IzoT Net Server networks, when you are using the SmartServer routers, you will first need to change to IMM before commissioning the SmartServer routers in IzoT CT or IzoT Net Server. See the workaround below if you mistakenly used DMM.

Add a ferrite clamp onto the U60/U70 USB cable to reduce LON channel noise (similar to one used for the SmartPhone as seen here: https://www.usbfirewire.com/parts/rr-fh0-500b.html).

...

If you experience problems with the U60 or U70, then add a USB 2.0 ferrite clamp onto the USB cable to reduce LON channel noise. See https://www.usbfirewire.com/parts/rr-fh0-500b.html

See the steps below for more information and diagnostics. 

  1. Verify that the U60 or U70 is up using one of the following methods:
    1. Use the LON SmartServer Configuration page to LON tab to see if the U60/U70 appears under LON interfaces.
      1. Use the console/SSH command ifconfig and check the lon#. There should be one lon# per U60/U70.
    2. If the U60 or U70 do not appear, then perform the following:
      1. Verify whether these devices are plugged in.
      2. Check for USB noise by looking for EMI? in the SmartServers.

        1. For SmartServer release 4.0 and higher (most of the important logs are now in the /var/log/syslog): 
          1. Go to the System Configuration Web page and click on the Logs button which will download the file to your PC (usually downloads folder).
          2. Extract the *.tgz - on windows you can use 7-zip to extract the files.
            1. During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
          3. Extract the *.tar file.
            1. During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
            2. As some of the files in the *.tgz are symbolic links which may not be supported by Windows, wait until the progress bar is completely green and the click the Close button.
            3. Look for EMI? in  /var/log/syslog, using console/SSH. 

              Code Block
              sudo grep "EMI?" /var/log/syslog


        2.  For Smartserver 3.6 and prior releases:
          1.  Look at /var/log sys.log, kern.log, and error files using the commands below. If you see EMI? in the log files, then you need to add a filter, like a USB 2.0 ferrite bead clamp onto the U60/U70 USB cable.

            Code Block
            titleConsole/ssh commands to look for looking for EMI?
            sudo grep "EMI?" /var/log/kern.log
            
            sudo grep "EMI?" /var/log/error
            
            sudo grep "EMI?" /var/log/syslog


      3. Reboot the SmartServer.

        If the U60/U70 continuously stops working after power cycles, then perform the following:
        1. For SmartServer 3.6 only, enter the following SSH command (this command may allow the U60/U70 to come up, but it is not a fix):

          Code Block
           sudo /sbin/apollo-usbreset start


        2. For all SmartServer versions, add a USB 2.0 ferrite bead clamp onto the U60/U70 USB cable.

    3. Verify that the LTE and LTX services are running.
      1. For SmartServer release 4.0 and higher:
        1. Check if LTE and LTX services are running. The LTE and LTX may get restarted if you import or make changes to an XIF file or re-import Resource File Set XML file.
        2. Check the current status of these services using the following ssh commands. The example below shows the U60 working.

          Code Block
          $ sudo smartserverctl status
          alarm                              ACTIVE              1h 4min
          bacnet                             ACTIVE              1h 4min
          core                               ACTIVE              1h 4min
          datapoint-connection-manager       ACTIVE              1h 4min
          datapoint-controller               ACTIVE              1h 4min
          datapoint-get                      ACTIVE              1h 4min
          enocean-driver                     ACTIVE              1h 4min
          formatter                          ACTIVE              1h 4min
          housekeeper                        ACTIVE              1h 4min
          iap-controller                     ACTIVE              1h 4min
          init                               ACTIVE              1h 4min
          lim                                ACTIVE              1h 4min
          loader                             ACTIVE              1h 4min
          logger                             ACTIVE              1h 4min
          lon                                ACTIVE              1h 4min
          lte                                ACTIVE              1h 4min
          ltx                                ACTIVE              1h 4min
          lwd                                ACTIVE              1h 4min
          modbus                             ACTIVE              1h 4min
          monitoring                         ACTIVE              1h 4min
          mosquitto                          ACTIVE              1h 4min
          node-red                           ACTIVE              1h 4min
          pyro4-ns                           ACTIVE              1h 4min
          query                              ACTIVE              1h 4min
          reboot-manager                     ACTIVE              1h 4min
          resource-publisher                 ACTIVE              1h 4min
          scheduler                          ACTIVE              1h 4min
          secmount                           ACTIVE              1h 4min
          services                           ACTIVE              1h 4min
          smartserver                        ACTIVE              1h 4min
          storage-manager                    ACTIVE              1h 4min
          apollo@smartserver-17qam88:~$


        3. You can also restart these services as they may get out of sync (for example, you may have deleted a device in the CMS, but it did not get cleared from MQTT) using the following command: 

          Code Block
          sudo smartserverctl restart lon


      2. For SmartServer 3.6 and prior releases:
        1. Check if lon:echlte and lon:echltx services are running. The LTE and LTX may get restarted if you import or make changes to an XIF file or re-import Resource File Set XML file.
        2. Check the current status of these services using the following ssh commands. The example below shows the U60 working.

          Code Block
          sudo supervisorctl status
          
          
          ~$ sudo supervisorctl status
          apollo-init                      EXITED    Dec 19 06:29 AM
          core:echhousekeeper              RUNNING   pid 1159, uptime 4:21:35
          core:echlim                      RUNNING   pid 1163, uptime 4:21:35
          echbacnet                        RUNNING   pid 1155, uptime 4:21:35
          echmodbus                        RUNNING   pid 1240, uptime 4:21:34
          echopcua-server                  STOPPED   Not started
          lon:echlte                       RUNNING   pid 3627, uptime 4:19:57
          lon:echltx                       RUNNING   pid 3626, uptime 4:19:57
          ready                            EXITED    Dec 19 06:29 AM
          services:echalarm                RUNNING   pid 1208, uptime 4:21:35
          services:echapollo-rp-launch     RUNNING   pid 1236, uptime 4:21:34
          services:echconnection           RUNNING   pid 1214, uptime 4:21:35
          services:echdatapointcontrol     RUNNING   pid 1216, uptime 4:21:35
          services:echdatapointget         RUNNING   pid 1227, uptime 4:21:35
          services:echformatter            RUNNING   pid 1229, uptime 4:21:35
          services:echiap-controller       RUNNING   pid 1239, uptime 4:21:34
          services:echloader               RUNNING   pid 1185, uptime 4:21:35
          services:echlogger               RUNNING   pid 1170, uptime 4:21:35
          services:echmonitoring           RUNNING   pid 1174, uptime 4:21:35
          services:echquery                RUNNING   pid 1180, uptime 4:21:35
          services:echscheduler            RUNNING   pid 1237, uptime 4:21:34


        3. You can also restart these services as they may get out of sync (for example, you may have deleted a device in the CMS, but it did not get cleared from MQTT) using the following command (include the : at the end of the command):

          Code Block
          sudo supervisorctl restart lon:


  2.  Use NodeUtil to see if device status "s" works.

    1. Using NodeUtil: Make sure that an RNI is set up for the U60 or U70. Next make sure you set up a RNI in the PC Windows Control panel > LonWorks Interfaces applet. 

      1. Start NodeUtil (e.g., nodeutil -dx.default.215) using a DOS command prompt.

      2. Add the device neuron Id (from the CMS Devices Widget) to NodeUtil using the "a" add command. Next click "g" followed by return and then "s".

    2. If NodeUtil device status works, then:
      1. Make sure the device is configured for the correct domain Id.
        1. Press "d" for domain in nodeUtil which will show you the device's domain ids. You can look at the SmartServer Configuration Page LON tab for the current domain id.
      2. If the domain ids are correct, then restart the LON drivers as shown above. 

...

Instructions are based on BACnet Client or BACnet Server:

BACnet Client – real BACnet devices and you use the SmartServer CMS to read and write to BACnet devices.

Verify that BACnet is enabled.

...

16. EnOcean devices are not working.

Verify that the  EnOcean Configuration page has the EnOcean driver enabled.

...

Note
titleLON devices only - DMM device discovery - caution about deleting LON devices

Deleting a discovered LON device (using the Remove action in the Devices widget) will cause the SmartServer to try to decommission the device and set the subnet and device ID to 0 before deleting the device from the SmartServer. This process may cause a previously working device to stop working. 

To delete a discovered LON device, first make sure the device is no longer connected to the network (e.g., FT channel), or disconnect the SmartServer from the SmartServer. This process may cause a previously working device to stop working. 

To delete a discovered LON device, first make sure the device is no longer connected to the network (e.g., FT channel), or disconnect the SmartServer from the network (such as disconnect the U60), and then delete the device in the CMS Devices widget.

Doing a reset database will not affect the existing devices.

...

the network (such as disconnect the U60), and then delete the device in the CMS Devices widget.

Doing a reset database will not affect the existing devices.

19. Device discovery never seems to stop.

Device discovery can take up to 45 minutes. If device discovery is not complete (i.e., the Devices widget discovery button is still greyed-out) after 1.5 hours, then it is possible that one of drivers restarted and interfered with the SmartServer being able to complete the device discovery process. In this case, reboot the SmartServer.

20. LON IzoT Net Export (inex) issues.

inex is used to export a subset or complete IzoT CT drawing or IzoT Net Server database into the SmartServer in order to access device datapoints. inex only works with CT or IzoT Net Server 4.2. Use inex 1.50.016+ (make sure the SmartServer is supported; check the SmartServer release notes). 

Change the SmartServer to IMM prior to running inex. Failure to change to IMM prior to running inex will cause the IP-852 and U60/U70 routing and datapoint access to stop working after a reboot. 

Any required Resource File Sets must be converted to XML files and then imported prior to importing device XIF files or running inex. A SmartServer IoT device needs to be added to CT for each SmartServer IoT device you are going to use. The SmartServer IoT device needs to be added to a IP-70 channel (each SmartServer needs its own IP-70 channel), which may require you to add additional IP-70 or U60 channel.  Only devices in the CT subsystem or child subsystem will be exported to the SmartServer.

Issues:

  1. Did you change the SmartServer to IMM before using inex?
    1. If not, then reset the SmartServer database, change to IMM using the SmartServer CMS Devices widget, and go through the inex import process again. 
  2. inex cannot find the SmartServer IoT device in CT or IzoT Net Server.
    1. Did you add a SmartServer IoT device to a SmartServer IP-70 channel in your CT drawing or IzoT Net Server database?
    2. If not, then add the device.
  3. inex cannot reach the SmartServer.
    1. Make sure that the MQTT ports are open; by default these may be closed. Go to the SmartServer Configuration page Firewall tab and open the ports that you are using with the inex tool.
  4. Datapoints in the SmartServer are correctly formatted or are not working properly.
    1. Did you import any required resource file set prior to running inex? Resource XML files need to be imported manually into the SmartServer CMS prior to running inex.
      1. Workaround: remove all devices and device types, then import the Resource file Set XML files, and finally run inex.
    2. If you ran inex using /skipfile and did not import the XIF manually or import the device XIF into the SmartServer CMS  prior to running inex, then this could cause the issue. 
      1. Workaround: remove all devices and device types, import any required Resource file Set XML files, optionally import the XIF files, and run inex with /skipfiles if XIF files were already imported, or without /skipfiles for inex to export the XIF files.
  5. Routing or datapoint access stop working after a SmartServer reboot.
    1. This usually means that you ran inex while the SmartServer is in DMM.
      1. Workaround: reset the SmartServer database, change to IMM, and reimport the Resource File Set XML and XIF files, and then run inex. 

21. The SmartServer constantly reboots in the U-boot prompt, or is stuck in the U-boot.

If the SmartServer constantly reboots in the U-boot prompt, then you may be able to recover the SmartServer by performing the workarounds described below. If the SmartServer gets stuck in the U-boot, then try to stop the U-boot boot process. While this issue is rare, it mostly occurs when re-imaging to an older image (prior to SmartServer 3.5) or when upgrading an older image.

...

  1.  Connect to the SmartServer console port (SSH cannot be used).

    1. Press and hold down any key on your keyboard (e.g., the space bar) until the boot process stops. Once it is stopped, you will have less than 30 seconds to enter commands. You can press any key to extend the timeout for another 30 seconds.
    2. If the boot process stops, then re-save the environmental variables by issuing the following commands as one copy and paste. Doing so fixes most rebooting issues.

      Code Block
      env default -a
      saveenv


    3. Once the SmartServer comes up, perform another reboot using the System Configuration page.

    4. If the previous workaround does not work, then re-image the SmartServer from the U-boot prompt by following KB1438.
    5. If you cannot stop the boot process, then contact Support as described below.

...

22. SmartServer does not recognize a USB flash drive.

This issue may occur when doing a re-image using lower quality USB flash drive, or when the USB flash drive is not formatted properly. Use a quality flash storage such as a Samsung flash storage device.

...

  1. Try rebooting the SmartServer with the USB drive plugged in.
    1. In older SmartServer version you will need to unplug the U60/U70, plug in the USB flash driver and reboot the SmartServer before the USB flash drive is recognized.
    2. Using SSH/console, enter the following command:

      Code Block
      sudo mount /dev/sda1 /media/usb0


    3. If the problem persists, then enter the following commands:

      Code Block
      sudo mkdir -p /media/usb0
      sudo mount /dev/sda1 /media/usb0


  2. Try using a different USB flash drive.

...

23. The SmartServer power line repeating does not work, or cannot communicate with power line devices.

If the SmartServer Configuration page LON tab shows "The lon1 device has failed a CENELEC protocol query. Device likely needs to be reset to restore communication" then you need to use SmartServer 3.65 or 4.1 (Beta) and higher, and update the U70 software.

...

If you experience problems with the U60 or U70, then add a USB 2.0 ferrite clamp onto the USB cable to reduce LON channel noise. See https://www.usbfirewire.com/parts/rr-fh0-500b.html.

  1.  Once you enable repeating static/dynamic, you cannot revert to non-repeating. Doing so requires that you perform a database reset using the System Configuration page that deletes all of your settings and devices. Once repeating is enabled, you can change between static and dynamic. 

    You typically will enable dynamic repeating to cause the SmartServer to set up repeating chains based on power line communication. In most cases, you will use dynamic repeating so that the SmartServer periodically checks the network and sets up the best repeating path based on current channel impairments.

    After you change the power line communications setting in the SmartServer LON Configuration page and press the Power Line Communications Update button, you need to reboot the SmartServer.

    1. Devices appear red in the CMS – if you are using repeating (static or dynamic), then you can use the DCI console commands to see the devices' current state and repeater chains. If the CMS Devices widget shows red devices, but the dcxshowdevices shows that devices are up, then use the CMS Devices widget to test the device. This test will determine whether communication is truly down.
    2. Check the SmartServer Configuration Page LON Tab to see if the U70 is working.
      1. If U70 is not working you many need to upgrade the U70 software (and SmartServer software), and add a filter. See CENELEC note above for upgrading U70 software, and U60 and U70 USB 2.0 Ferrite Bead filter note. for filter details.
        1. Even there is no issue with the U70, you may want to add these enhancements to prevent future issues.
    3. Using NodeUtil – when power line repeating is enabled, NodeUtil can only be used to communicate edge devices, which can be reached directly by the SmartServer. NodeUtil cannot be used to communicate to edge devices that require a repeater since there is no way to specify a repeater path with NodeUtil.
    4. Using LonScanner – LonScanner can only see traffic locally; therefore, you will only see traffic that the SmartServer can hear. LonScanner sees packets that go to devices that the SmartServer can communicate directly to (1 hop). For devices that require repeaters (2 or more hops), in most cases, LonScanner will not show the packets in between repeaters and the destination device. 

...

24. I am having connections issues.

You can create connections with both the CMS Connections widget and using MQTT. MQTT created connections do not show up in the Connections widget. If you make changes using the Connections widget but do not see the updates, then click the Refresh button () to see the updates.

  1.  Only LON devices using DMM – why am I not able to create a peer-to-peer between edge devices or an internal device and an edge device?
    1. Not all datapoints on the LON device can have a peer-to-peer connection. This applies to both input and output datapoints. Look at the device's XIF to determine if the datapoint can have a peer-to-peer connection. Each datapoint has a poll flag, which when set, means that the datapoint can only be polled. If a datapoint is poll only, see the datapoint section for details.
      1. Workarounds:
        1. Internal devices – you can change the XIF you are using such that the datapoint poll flag in the XIF file is set to a 0 and then re-import in the XIF file. You may need to change the program ID in the XIF file if you were using this same XIF file for real devices.
        2. Edge devices – you will either need to use point-to-point connections (which may slow down sending updates to sources datapoints) or compile new code for the devices with the poll flags set to 0.

...

25. Scheduled events astronomical sunrise and sunset times are off.

The SmartServer uses Civil Dawn and Civil Dusk for sunrise and sunset, respectively, as Civil times are reportedly more accurate for street light applications. The Calendar widget only shows a fixed sunrise and sunset time. 

  1. Determine actual sunrise (Civil Dawn) and sunset (Civil Dusk) times. 
    1. With SmartServer 3.62 and higher, the /var/log/supervisor/scheduler.log shows each day, or after power up, the daily Civil Dawn and Civil Dusk in local time.
    2. If the times are off, check latitude and longitude using CMS Devices widget. Click the View All Properties action to see latitude and longitude.
      1. Using the CMS:
        1. Assign the device to a context.
        2. Export devices, modify the devices.csv to include latitude or longitude, and re-import the file.
      2. Using REST or MQTT API to add latitude and longitude.
    3. Verify that the SmartServer is in the correct timezone by looking CMS Devices widget → Segment Controllers tab.
    4. Using console/SSH, check current time by entering timedatectl.

...

26. I am having issues with the SmartServer data log.

The SmartServer data log is accessible through the CMS Datapoints widget, or through REST/WebSocket APIs. MQTT does not have access to the data log data.

  1. If the SmartServer does not appear to be logging all data, then this usually means that you may be using the default logging setup. This setting is not correct and will usually mean gaps in log data. If you are using Value for Log Minimum Delta Value, then you must specify a numeric value.

    Example of an incorrect setting: default logging setup on the Datapoint Properties widget Monitoring and Logging Configuration tab (this configuration may result in gaps of data log data):



    Example of a correct setting: configuration for a numeric scalar datapoint (e.g, only logging temperature data if temperature value changes by one degree since the last logged data value):



    1. Log settings:
      1. Structured datapoint – Log Minimum Delta Value must be changed to Always or On Any Change
      2. Scalar datapoints
        1. Number datapoint – Log Minimum Delta Value can be ValueAlways or On Any Change
          1. If you are using Value, then you must specify a numeric value as shown above.
            1. If you do not specify a numeric value, then you will get inconsistent or no log data.
        2. Enumerated datapoints (datapoints that are strings, such as HVAC_Auto) – Log Minimum Delta Value must be changed to Always or On Any Change

...

27. For LON devices only, I deleted a device, and now I cannot add a new device with the same name.

Info

Only for LON devices using DMM.

...

For SmartServer 3.6 and prior, include the colon at the end of the command.

...

28. SmartServer internal device stops working after upgrading to SmartServer 4.0 or higher.

Perform the following steps to troubleshoot problems with edge devices:

  1. With SmartServer 4.0 and higher, to access services and install internal devices, note the following:
    1. supervisorctl is replaced with smartserverctl. For SmartServer 3.6 and prior, supervisorctl is used.
    2. Update any scripts or code that use supervisorctl to use smartserverctl (the format has changed for some commands); create unit files.

...

29. Troubleshooting edge device issues.

Perform the following steps to troubleshoot problems with edge devices:

  1. Go to the CMS Devices widget and check the current state.
    1. If device appears to be down, but is expected to be up, then do one of the following:
      1. Click the device's Action button () and select the Test action to verify communications to the device.
      2. Go to the CMS Datapoints widget and filter on at least one of the device's datapoints, and either have the polling interval greater than 0 or click the Refresh button ().
      3. Use MQTT explorer to check device health (e.g., topic glp/0/17qam88/fb/dev/lon/2/sts health parameter).
    2. For LON devices only that are using power line repeating:
      1. The CMS Datapoints widget polling interval should be set either to 0 or a very large number. Do not leave the polling interval at the default 30 seconds since this can affect power line performance.

  2. If data logging is enabled for a datapoint on the device, then check the data log using Datapoints widget and filter on logged data.
    1. Each data log entry will report whether the device is running (normal) or down.
       
  3. Go to the CMS Alarms and Events widget and look for warning or error messages.
    1. Check the device information, typically available in the alarm event Details or Description.
      1. Manually review the information in these columns; there is no filter capability for the columns.
      2. Filter on Raising local time column to reduce the number of events that are displayed.
        1. Raising local time is when the alarm/event occurs (based on the SmartServer local time). 
      3. Export the alarms and use Excel to search for the device name. 

  4. For LON devices only:
    1. Use LonScanner.
      1. Use LonScanner to check whether the device is responding to write or read requests.
      2. Use the SmartServer's built-in LonScanner RNI to look at LON traffic.
        1. To fully check a LON channel, use a laptop with its own network interface and place it near the problem device, and open LonScanner with two log windows, one for the laptop network interlace and one at the SmartServer. In this way, you can determine if a request from the SmartServer reaches the edge device, and whether the edge device response is received by the SmartServer. You can also determine the degree of delay in responses. 
      3. A delayed response, due to heavy traffic on the channel or noise, can cause the device to be reported as down even though there is communication to the device.
      4. Sustained traffic on a LON FT-10 or XF-1250 channel should be 40% or lower, since this half duplex channel needs to allow some bandwidth for important traffic to get through without delay.
        1. If the SmartServer is originating all of the communication (it is the only one sending request packets), then you can have a higher sustained bandwidth value. 
    2. Use NodeUtil to communicate to the edge device using the SmartServer RNI or another network interface.
      1. Check whether you can communicate to the device by:
        1. Getting node status information.
        2. Looking at the device's domain/subnet/node IDs.
        3. Reading back network variable values.
      2. For power line repeating channels, you can only communicate to devices with direct communication to the SmartServer.

...

30. I cannot export files using the SmartServer CMS Import/Export button.

When you export settings or create backups using the SmartServer CMS Import/Export Button and click the download button, the SmartServer will open a new web page that uses the SmartServer hostname in the URL. If you do the export remotely through a VPN, or the hostname to IP address resolution does not work, then you will get an error similar to the one shown below.

...

  1. Provide the SmartServer software version. The software version appears on the Configuration page System tab, the CMS dashboard, and when you log into an SSH/console session.
    1. Provide the SmartServer serial number.
      1. The SmartServer datecode is part of the SmartServer serial # is found on the SmartServer label.
        1. Only use the label serial number for RMAs. 
        2. You can also see the serial # on the SmartServer Configuration Page System tab and when you ssh/console in.
          1. Note, these values are changed when restoring a backed-up or cloned system image; cannot be used for RMAs.
      2. Format: 44yywwrxxxxx, where ww is the week of the year, yy is the year, and r is hardware revision. xxxxx indicates other characters in the serial #.
        Example Serial # 442031F00100:  Datecode = 2031F which means 31st week in year 2020 with Rev "F" hardware (Quad core). Rev A-D are dual core, Rev F+ are quad core.
    2. For license issues also provide the SmartServer install code (Look at System Configuration page or label on back of SmartServer).
    3. Provide additional information, such as:
      1. Identify when the problem started.
        1. Example: the problem happened after a GLPO update (i.e., an update using the CMS Devices widget, Update action), re-image, or restoring a previous backup.
      2. Identify whether this a new problem, or whether this problem always existed.
      3. Provide screen captures or videos of error messages, web pages, or console/SSH commands.
    4. Provide SmartServer log files.
      1. For SmartServer release 4.0 and higher - Most of the important logs are now in the /var/log/syslog
        1. Go to the System Configuration Web page and click on the Logs button which will download the file to your PC (usually downloads folder).
        2. Extract the *.tgz - on windows you can use 7-zip to extract the files.
          1. During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
        3. Extract the *.tar file. 
          1. During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
          2. As some of the files in the *.tgz are symbolic links which may not be supported by Windows, wait until the progress bar is completely green and the click the Close button.
        4. The /var/log/syslog contains the most of the important log information.
          1. In older Smartserver version (prior to release 4.0) each driver (like lte, ltx, or BACnet) had their own log file. Starting with 4.0 these logs are now combined in syslog. 
      2. For SmartServers 3.6 and prior, some log files cannot be copied with the apollo login. 
        1. You can retrieve the log files in one of the following ways:
          1. Use WinSCP with root login (not supported if enhanced security, available with SmartServer 3.5 or higher, is enabled; see Enhancing Security) to copy all files and directories in 
            /var/log
          2. If you are using SmartServer 3.5 or higher, and enhanced security is enabled, then go to the System Configuration page and temporarily disable Enhancing Security. Use WinSCP with root login to copy all files and directories in /var/log, and then re-enable enhanced security using the System Configuration page.
            /var/log
          3. Create a log tar file (like a windows zip file) using the apollo login for both SSH and WinSCP.
            Run the following two SSH commands (these commands can be used with enhanced security enabled or disabled), and use an SFTP application (e.g., WinSCP) to copy the file /var/apollo/data/logs.tar.gz.

            Code Block
            sudo tar -czvf /var/apollo/data/logs.tar.gz /var/log
            sudo chown apollo /var/apollo/data/logs.tar.gz


        2. If you have any issues with the CMS, then provide the Karaf log files. 
          1. The location of the files is dependent on the SmartServer version number. 
                /home/apollo/echelon-container-devel-xx.xx.xxx/data/log/karaf.log
                    where xx.xx.xxx is the SmartServer version number
            For example, SmartServer 2.61.005:
                /home/apollo/echelon-container-devel-2.61.005/data/log/karaf.log   
          2. To read the last 2000 lines in the file, use the following ssh/console command:
                tail -2000 /home/apollo/echelon-container-devel-2.61.005/data/log/karaf.log
    5. Provide an Export of the CMS Settings:
      1. For SmartServer 3.5 and higher, use the Import/Export button ().
      2. For SmartServer 3.4 and prior, use the CMS Settings button ().

    6. Before processing any warranty returns (RMA) or if the SmartServer is out of warranty
      1. Try re-imaging the SmartServer to the latest supported software for the hardware as a re-image fixes most issues.
      2. In some cases in addition to the re-image you may need to do additional reset database or restore to factory to correct an issue.

    7. Open a Support ticket:
      1. Open a support ticket at https://support.enocean.com/help#login. This usually provides the fastest response time from our support team.
      2. Send an email to edge.support@enocean.com.