Many issues are resolved by updating to the latest SmartServer software. Review the SmartServer release notes before upgrading or re-imaging to a different software version. For issues that require diagnostics, this section provides some helpful hints to troubleshoot various issues that you may experience using the SmartServer.
...
- BACnet – Use SmartServer 3.6 Update 3 or higher. SmartServer 3.6 Update 3 includes fixes for BACnet broadcast message handling.
- LoRaWAN – Use SmartServer 4.2 or higher.
- LON IP-852 routing – Use SmartServer 3.5 or higher. SmartServer 3.5 improves support for disconnected or unreliable LAN or WAN Ethernet connections.
- LON U60 and U70 network interface noise issues – If you are using a LON U60 or U70 network interface that is attached to a LON FT, TP-1250, or PL-20 channel with very high levels of electrical noise, then the noise can cause USB communication failures in the SmartServer. One symptom of electrical noise is an EMI? error in the /var/log/kern.log file. You can reduce the electrical noise that enters a SmartServer from the LON channel by installing a ferrite clamp on core filter onto the U60 or U70 USB cable. An example filter is available at https://www.mouser.com/ProductDetail/623-0431164281.
- LON PL-20 power line networks using the U70 network interface – Use SmartServer 3.6 Update 5, 4.1 (Beta), or newer, with U70 Firmware 2.0. If your U70 has U70 Firmware 1.0, then you can update the firmware as described in Updating the U70 Firmware. The SmartServer and U70 must have compatible versions as described in the U70 Firmware release notes. U70 Firmware 2.0 improves the reliability of the communication between SmartServer and U70 and fixes the issue that causes the U70 to stop working and show a CENELEC error on the SmartServer Configuration page LON tab.
...
When you first access your SmartSever, follow the procedure in the Configuring the SmartServer IP Address and Hostname (Release 3.4 to 4.1) section to change the LAN/WAN ports that you are using from Startup Mode to Static IP Address or DHCP. Failure to do so may result in flaky Ethernet connections especially if you unplug and re-plug in the SmartServer, or if there are issues on the Ethernet connection. This may happen because startup mode looks for DHCP, and if not found, it falls back to a static IP address. Therefore, if you experience issues on the Ethernet channel, or if you unplug and re-plug in the Ethernet cable, then the SmartServer may not know whether it is using DHCP or static IP addresses. Changing the Ethernet ports to Static IP Address or DHCP when you first access the SmartServer eliminates this problem.
...
Changing back and forth between IMM and DMM is not supported. Therefore, you must first discern which mode of operation (DMM or IMM) your application requires and then maintain that mode. For IzoT CT and IzoT Net Server networks, when you are using the SmartServer routers, you will first need to change to IMM before commissioning the SmartServer routers in IzoT CT or IzoT Net Server. See the workaround below if you mistakenly used DMM.
Add a ferrite clamp onto the U60/U70 USB cable to reduce LON channel noise (similar to one used for the SmartPhone as seen here: https://www.usbfirewire.com/parts/rr-fh0-500b.html).
...
If you experience problems with the U60 or U70, then add a USB 2.0 ferrite clamp onto the USB cable to reduce LON channel noise. See https://www.usbfirewire.com/parts/rr-fh0-500b.html
See the steps below for more information and diagnostics.
- Verify that the U60 or U70 is up using one of the following methods:
- Use the LON Configuration page to see if the U60/U70 appears under LON interfaces.
- Use the console/SSH command ifconfig and check the lon#. There should be one lon# per U60/U70.
- If the U60 or U70 do not appear, then perform the following:
- Verify whether these devices are plugged in.
Check for USB noise by looking for EMI? in the SmartServers.
- For SmartServer release 4.0 and higher (most of the important logs are now in the /var/log/syslog):
- Go to the System Configuration Web page and click on the Logs button which will download the file to your PC (usually downloads folder).
- Extract the *.tgz - on windows you can use 7-zip to extract the files.
- During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
- Extract the *.tar file.
- During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
- As some of the files in the *.tgz are symbolic links which may not be supported by Windows, wait until the progress bar is completely green and the click the Close button.
Look for EMI? in /var/log/syslog, using console/SSH.
Code Block sudo grep "EMI?" /var/log/syslog
- For Smartserver 3.6 and prior releases:
Look at /var/log sys.log, kern.log, and error files using the commands below. If you see EMI? in the log files, then you need to add a filter, like a USB 2.0 ferrite bead clamp onto the U60/U70 USB cable.
Code Block title Console/ssh commands to look for looking for EMI? sudo grep "EMI?" /var/log/kern.log sudo grep "EMI?" /var/log/error sudo grep "EMI?" /var/log/syslog
- For SmartServer release 4.0 and higher (most of the important logs are now in the /var/log/syslog):
Reboot the SmartServer.
If the U60/U70 continuously stops working after power cycles, then perform the following:For SmartServer 3.6 only, enter the following SSH command (this command may allow the U60/U70 to come up, but it is not a fix):
Code Block sudo /sbin/apollo-usbreset start
For all SmartServer versions, add a USB 2.0 ferrite bead clamp onto the U60/U70 USB cable.
- Verify that the LTE and LTX services are running.
- For SmartServer release 4.0 and higher:
- Check if LTE and LTX services are running. The LTE and LTX may get restarted if you import or make changes to an XIF file or re-import Resource File Set XML file.
Check the current status of these services using the following ssh commands. The example below shows the U60 working.
Code Block $ sudo smartserverctl status alarm ACTIVE 1h 4min bacnet ACTIVE 1h 4min core ACTIVE 1h 4min datapoint-connection-manager ACTIVE 1h 4min datapoint-controller ACTIVE 1h 4min datapoint-get ACTIVE 1h 4min enocean-driver ACTIVE 1h 4min formatter ACTIVE 1h 4min housekeeper ACTIVE 1h 4min iap-controller ACTIVE 1h 4min init ACTIVE 1h 4min lim ACTIVE 1h 4min loader ACTIVE 1h 4min logger ACTIVE 1h 4min lon ACTIVE 1h 4min lte ACTIVE 1h 4min ltx ACTIVE 1h 4min lwd ACTIVE 1h 4min modbus ACTIVE 1h 4min monitoring ACTIVE 1h 4min mosquitto ACTIVE 1h 4min node-red ACTIVE 1h 4min pyro4-ns ACTIVE 1h 4min query ACTIVE 1h 4min reboot-manager ACTIVE 1h 4min resource-publisher ACTIVE 1h 4min scheduler ACTIVE 1h 4min secmount ACTIVE 1h 4min services ACTIVE 1h 4min smartserver ACTIVE 1h 4min storage-manager ACTIVE 1h 4min apollo@smartserver-17qam88:~$
You can also restart these services as they may get out of sync (for example, you may have deleted a device in the CMS, but it did not get cleared from MQTT) using the following command:
Code Block sudo smartserverctl restart lon
- For SmartServer 3.6 and prior releases:
- Check if lon:echlte and lon:echltx services are running. The LTE and LTX may get restarted if you import or make changes to an XIF file or re-import Resource File Set XML file.
Check the current status of these services using the following ssh commands. The example below shows the U60 working.
Code Block sudo supervisorctl status ~$ sudo supervisorctl status apollo-init EXITED Dec 19 06:29 AM core:echhousekeeper RUNNING pid 1159, uptime 4:21:35 core:echlim RUNNING pid 1163, uptime 4:21:35 echbacnet RUNNING pid 1155, uptime 4:21:35 echmodbus RUNNING pid 1240, uptime 4:21:34 echopcua-server STOPPED Not started lon:echlte RUNNING pid 3627, uptime 4:19:57 lon:echltx RUNNING pid 3626, uptime 4:19:57 ready EXITED Dec 19 06:29 AM services:echalarm RUNNING pid 1208, uptime 4:21:35 services:echapollo-rp-launch RUNNING pid 1236, uptime 4:21:34 services:echconnection RUNNING pid 1214, uptime 4:21:35 services:echdatapointcontrol RUNNING pid 1216, uptime 4:21:35 services:echdatapointget RUNNING pid 1227, uptime 4:21:35 services:echformatter RUNNING pid 1229, uptime 4:21:35 services:echiap-controller RUNNING pid 1239, uptime 4:21:34 services:echloader RUNNING pid 1185, uptime 4:21:35 services:echlogger RUNNING pid 1170, uptime 4:21:35 services:echmonitoring RUNNING pid 1174, uptime 4:21:35 services:echquery RUNNING pid 1180, uptime 4:21:35 services:echscheduler RUNNING pid 1237, uptime 4:21:34
You can also restart these services as they may get out of sync (for example, you may have deleted a device in the CMS, but it did not get cleared from MQTT) using the following command (include the : at the end of the command):
Code Block sudo supervisorctl restart lon:
- For SmartServer release 4.0 and higher:
- Use the LON Configuration page to see if the U60/U70 appears under LON interfaces.
Use NodeUtil to see if device status "s" works.
Using NodeUtil: Make sure that an RNI is set up for the U60 or U70. Next make sure you set up a RNI in the PC Windows Control panel > LonWorks Interfaces applet.
Start NodeUtil (e.g., nodeutil -dx.default.215) using a DOS command prompt.
Add the device neuron Id (from the CMS Device Devices Widget) to NodeUtil using the "a" add command. Next click "g" followed by return and then "s".
- If NodeUtil device status works, then:
- Make sure the device is configured for the correct domain Id.
- Press "d" for domain in nodeUtil which will show you the device's domain ids. You can look at the SmartServer Configuration Page LON tab for the current domain id.
- If the domain ids are correct, then restart the LON drivers as shown above.
- Make sure the device is configured for the correct domain Id.
...
Instructions are based on BACnet Client or BACnet Server:
BACnet Client – real BACnet devices and you use the SmartServer CMS to read and write to BACnet devices.
Verify that BACnet is enabled.
...
If you experience problems with the U60 or U70, then add a USB 2.0 ferrite clamp onto the USB cable to reduce LON channel noise. See https://www.usbfirewire.com/parts/rr-fh0-500b.html.
Once you enable repeating static/dynamic, you cannot revert to non-repeating. Doing so requires that you perform a database reset using the System Configuration page that deletes all of your settings and devices. Once repeating is enabled, you can change between static and dynamic.
You typically will enable dynamic repeating to cause the SmartServer to set up repeating chains based on power line communication. In most cases, you will use dynamic repeating so that the SmartServer periodically checks the network and sets up the best repeating path based on current channel impairments.
After you change the power line communications setting in the SmartServer LON Configuration page and press the Power Line Communications Update button, you need to reboot the SmartServer.- Devices appear red in the CMS – if you are using repeating (static or dynamic), then you can use the DCI console commands to see the devices' current state and repeater chains. If the CMS Devices widget shows red devices, but the dcxshowdevices shows that devices are up, then use the CMS Devices widget to /wiki/spaces/TEMP/pages/1478027 test the device. This test will determine whether communication is truly down.
- Check the SmartServer Configuration Page LON Tab to see if the U70 is working.
- If U70 is not working you many need to upgrade the U70 software (and SmartServer software), and add a filter. See CENELEC note above for upgrading U70 software, and U60 and U70 USB 2.0 Ferrite Bead filter note. for filter details.
- Even there is no issue with the U70, you may want to add these enhancements to prevent future issues.
- If U70 is not working you many need to upgrade the U70 software (and SmartServer software), and add a filter. See CENELEC note above for upgrading U70 software, and U60 and U70 USB 2.0 Ferrite Bead filter note. for filter details.
- Using NodeUtil – when power line repeating is enabled, NodeUtil can only be used to communicate edge devices, which can be reached directly by the SmartServer. NodeUtil cannot be used to communicate to edge devices that require a repeater since there is no way to specify a repeater path with NodeUtil.
- Using LonScanner – LonScanner can only see traffic locally; therefore, you will only see traffic that the SmartServer can hear. LonScanner sees packets that go to devices that the SmartServer can communicate directly to (1 hop). For devices that require repeaters (2 or more hops), in most cases, LonScanner will not show the packets in between repeaters and the destination device.
...
- Go to the CMS Devices widget and check the /wiki/spaces/TEMP/pages/73367553 current state.
- If device appears to be down, but is expected to be up, then do one of the following:
- Click the device's Action button () and select the /wiki/spaces/TEMP/pages/73367553Test action to verify communications to the device.
- Go to the CMS /wiki/spaces/TEMP/pages/73433089 Datapoints widget and filter on at least one of the device's datapoints, and either have the polling interval greater than 0 or click the Refresh button ().
- Use MQTT explorer to check device health (e.g., topic
glp/0/17qam88/fb/dev/lon/2/sts
health parameter).
- For LON devices only that are using power line repeating:
- The CMS Datapoints widget polling interval should be set either to 0 or a very large number. Do not leave the polling interval at the default 30 seconds since this can affect power line performance.
- The CMS Datapoints widget polling interval should be set either to 0 or a very large number. Do not leave the polling interval at the default 30 seconds since this can affect power line performance.
- If device appears to be down, but is expected to be up, then do one of the following:
- If data logging is enabled for a datapoint on the device, then check the data log using /wiki/spaces/TEMP/pages/73433089 Datapoints widget and filter on logged data.
- Each data log entry will report whether the device is running (normal) or down.
- Each data log entry will report whether the device is running (normal) or down.
- Go to the CMS Alarms and Events widget and look for warning or error messages.
- Check the device information, typically available in the alarm event Details or Description.
- Manually review the information in these columns; there is no filter capability for the columns.
- Filter on Raising local time column to reduce the number of events that are displayed.
- Raising local time is when the alarm/event occurs (based on the SmartServer local time).
- Export the alarms and use Excel to search for the device name.
- Check the device information, typically available in the alarm event Details or Description.
- For LON devices only:
- Use LonScanner.
- Use LonScanner to check whether the device is responding to write or read requests.
- Use the SmartServer's built-in LonScanner RNI to look at LON traffic.
- To fully check a LON channel, use a laptop with its own network interface and place it near the problem device, and open LonScanner with two log windows, one for the laptop network interlace and one at the SmartServer. In this way, you can determine if a request from the SmartServer reaches the edge device, and whether the edge device response is received by the SmartServer. You can also determine the degree of delay in responses.
- A delayed response, due to heavy traffic on the channel or noise, can cause the device to be reported as down even though there is communication to the device.
- Sustained traffic on a LON FT-10 or XF-1250 channel should be 40% or lower, since this half duplex channel needs to allow some bandwidth for important traffic to get through without delay.
- If the SmartServer is originating all of the communication (it is the only one sending request packets), then you can have a higher sustained bandwidth value.
- Use NodeUtil to communicate to the edge device using the SmartServer RNI or another network interface.
- Check whether you can communicate to the device by:
- Getting node status information.
- Looking at the device's domain/subnet/node IDs.
- Reading back network variable values.
- For power line repeating channels, you can only communicate to devices with direct communication to the SmartServer.
- Check whether you can communicate to the device by:
- Use LonScanner.
...
- Provide the SmartServer software version. The software version appears on the Configuration page System tab, the CMS dashboard, and when you log into an SSH/console session.
- Provide the SmartServer serial number.
- The SmartServer datecode is part of the SmartServer serial # is found on the SmartServer label.
- Only use the label serial number for RMAs.
- You can also see the serial # on the SmartServer Configuration Page System tab and when you ssh/console in.
- Note, these values are changed when restoring a backed-up or cloned system image; cannot be used for RMAs.
- Format: 44yywwrxxxxx, where ww is the week of the year, yy is the year, and r is hardware revision. xxxxx indicates other characters in the serial #.
Example Serial # 442031F00100: Datecode = 2031F which means 31st week in year 2020 with Rev "F" hardware (Quad core). Rev A-D are dual core, Rev F+ are quad core.
- The SmartServer datecode is part of the SmartServer serial # is found on the SmartServer label.
- For license issues also provide the SmartServer install code (Look at System Configuration page or label on back of SmartServer).
- Provide additional information, such as:
- Identify when the problem started.
- Example: the problem happened after a GLPO update (i.e., an update using the CMS Devices widget, Update action), re-image, or restoring a previous backup.
- Identify whether this a new problem, or whether this problem always existed.
- Provide screen captures or videos of error messages, web pages, or console/SSH commands.
- Identify when the problem started.
- Provide SmartServer log files.
- For SmartServer release 4.0 and higher - Most of the important logs are now in the /var/log/syslog.
- Go to the System Configuration Web page and click on the Logs button which will download the file to your PC (usually downloads folder).
- Extract the *.tgz - on windows you can use 7-zip to extract the files.
- During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
- Extract the *.tar file.
- During the extraction process you may see a Confirm File Replace dialog pop up. If you see this dialog click the Auto Rename button.
- As some of the files in the *.tgz are symbolic links which may not be supported by Windows, wait until the progress bar is completely green and the click the Close button.
- The /var/log/syslog contains the most of the important log information.
- In older Smartserver version (prior to release 4.0) each driver (like lte, ltx, or BACnet) had their own log file. Starting with 4.0 these logs are now combined in syslog.
- For SmartServers 3.6 and prior, some log files cannot be copied with the apollo login.
- You can retrieve the log files in one of the following ways:
- Use WinSCP with root login (not supported if enhanced security, available with SmartServer 3.5 or higher, is enabled; see /wiki/spaces/TEMP/pages/209397555Enhancing Security) to copy all files and directories in
/var/log - If you are using SmartServer 3.5 or higher, and enhanced security is enabled, then go to the System Configuration page and temporarily disable /wiki/spaces/TEMP/pages/209397555Enhancing Security. Use WinSCP with root login to copy all files and directories in /var/log, and then re-enable enhanced security using the System Configuration page.
/var/log Create a log tar file (like a windows zip file) using the apollo login for both SSH and WinSCP.
Run the following two SSH commands (these commands can be used with enhanced security enabled or disabled), and use an SFTP application (e.g., WinSCP) to copy the file /var/apollo/data/logs.tar.gz.Code Block sudo tar -czvf /var/apollo/data/logs.tar.gz /var/log sudo chown apollo /var/apollo/data/logs.tar.gz
- Use WinSCP with root login (not supported if enhanced security, available with SmartServer 3.5 or higher, is enabled; see /wiki/spaces/TEMP/pages/209397555Enhancing Security) to copy all files and directories in
- If you have any issues with the CMS, then provide the Karaf log files.
- The location of the files is dependent on the SmartServer version number.
/home/apollo/echelon-container-devel-xx.xx.xxx/data/log/karaf.log
where xx.xx.xxx is the SmartServer version number
For example, SmartServer 2.61.005:
/home/apollo/echelon-container-devel-2.61.005/data/log/karaf.log - To read the last 2000 lines in the file, use the following ssh/console command:
tail -2000 /home/apollo/echelon-container-devel-2.61.005/data/log/karaf.log
- The location of the files is dependent on the SmartServer version number.
- You can retrieve the log files in one of the following ways:
- For SmartServer release 4.0 and higher - Most of the important logs are now in the /var/log/syslog.
- Provide an Export of the CMS Settings:
- For SmartServer 3.5 and higher, use the Import/Export button ().
- For SmartServer 3.4 and prior, use the CMS Settings button ().
- Before processing any warranty returns (RMA) or if the SmartServer is out of warranty
- Try re-imaging the SmartServer to the latest supported software for the hardware as a re-image fixes most issues.
- In some cases in addition to the re-image you may need to do additional reset database or restore to factory to correct an issue.
- Open a Support ticket:
- Open a support ticket at https://support.enocean.com/help. This usually provides the fastest response time from our support team.
- Send an email to edge.support@enocean.com.
- Provide the SmartServer serial number.