Stability of LoPy in nano gateway mode?

[lollisoft]

Hi,

I am running two LoPy devices since some days and one is the nano gateway. The first time I was able to run the gateway about a day and the second time I was able to run it about three days.

Then it stopped. The blue LED did no more blink, the telnet server was no more reachable. After a reset all is fine again.

I run the last firmware or the second last one on the gateway. I think the second last and the last on my node.

Is this a known issue?

What is about opening UDP port for everyone? That may be the cause and I should open it only for the ttn network :-)

Thanks, Lothar

[harix]

@lollisoft Thanks for this information! I'll try this!

[lollisoft]

@harix I have initially opened the udp port in my firewall and thus, it might be any hacking trials coming in to the port. After removing the rule, all worked well since some months. So be aware of this and it is probably to the limited amount of RAM available in the device. The firewall knows what incoming IP addresses are allowed to pass due to prior outgoing traffic to that server and thus allows udp packages from that server only flow back. For me that is a reasonable explanation.

[harix]

I am experiencing the same problem. After around 3000 packets received (sender sends every 2 sec) the nano gateway stops working. I have modified the nano-gateway code to run without ACK each packet received. Even a "soft reboot" did not solve the issue. A power off / power on cycle was necessary to get it receiving again. I have tested with the latest firmware 1.7

[lollisoft]

@jmarcelino Maybe it was the open UDP port. Actually, with a closed port I still receive Pull Ack and Push Ack messages. So I learned a bit more about UDP and firewalls :-)

If it runs stable, then there was some foreign traffic, the system could not work with and probably freezed.

I'll see.

[jmarcelino]

@lollisoft
Sadly that is a MPSSE cable, I think - but not 100% sure - the FTDI chip inside is set to MPSSE mode (JTAG,I2C,etc) mode and so the UART mode you need is disabled.

[lollisoft]

@jmarcelino I think, I can use my FPGA ZPU programming cable (C232HM-DDHSL-0). I used that for the CPU programming within my FPGA and that is a 3V3 Xilinx Spartan 6 based chip. I am pretty sure, that this is usable.

http://www.ftdichip.com/Support/Documents/DataSheets/Cables/DS_C232HM_MPSSE_CABLE.pdf

[jmarcelino]

@lollisoft
You can find USB Serial TTL adapters for about 5 Euros, try looking on Ebay for example. Look for CP2104 for example (CP2102 also works but not as good). Make sure it supports 3.3V

Then just set to 3.3V connect RX, TX, GND

I really recommend having something to monitor the serial port of the Pycom boards, many errors happen at the low level (ESP-IDF) and outside the control of the MicroPython. If not the baseboard (easiest) then one of these adapters.

[lollisoft]

@jmarcelino said in Stability of LoPy in nano gateway mode?:

@lollisoft
Did you add the suggested capacitors to your 78S05? That said I don't think power is your issue especially as you're powering from 5V which gets regulated again.

In front I have a 220 uF and on both sides I have also 100nF. I may add some more to the LoPy as the breadboard is likely the case for the spikes (long copper wires and two breadboards).

Here is my crappy work: https://twitter.com/lollisoft/status/856193461434953728

You should monitor the serial console for errors.

No base board. I have only one. I need a plain dil chip to enable usb to serial functionality. It is a bit too much prototype :-)

I also don't think the LoPy Nanogateway is "bulletproof" code, it's more proof of concept and the first version. I remember Daniel writing here they were working on improvements.

At least I tried to improve a bit the memory consumption by moving buffer creation into the udp_lock code block.

I also tried to use the WDT class, but that is not available at the LoPy. Will it be available and if, when?

Also I think of buying a ttn gateway, if that makes sense (more users, reach and ROI).

I am planning a project around code generating the python code and infrastructure code around an IoT solution as a practical reference project for my open source project, I regularly post about in twitter :-)

That's the ROI factor.

As for opening UDP, the LoPy initiates the UDP connection to port 1700 at TTN and TTN replies back from that same IP and source port. This is kept active by the frequent PULL_DATA requests from the LoPy with status data.

Most NAT firewall will be able to persist this UDP mapping without needing to open UDP to the world, but of course your configuration may be different.

I am using a Fritz Box router and I am not aware of the firewall capabilities but now trying without that open udp port. I'll see it soon or later.

[jmarcelino]

@lollisoft
Did you add the suggested capacitors to your 78S05? That said I don't think power is your issue especially as you're powering from 5V which gets regulated again.

You should monitor the serial console for errors.

I also don't think the LoPy Nanogateway is "bulletproof" code, it's more proof of concept and the first version. I remember Daniel writing here they were working on improvements.

As for opening UDP, the LoPy initiates the UDP connection to port 1700 at TTN and TTN replies back from that same IP and source port. This is kept active by the frequent PULL_DATA requests from the LoPy with status data.

Most NAT firewall will be able to persist this UDP mapping without needing to open UDP to the world, but of course your configuration may be different.

[lollisoft]

Here is the powersupply measured with my DSM203. The toggling signal is my blinker from a CMOS 4060 circuit.
The signal seems to get some spikes, that also may cause a freeze :-)

[lollisoft]

I need to tell more about my simple gateway implementation - hardware wise.

I have implemented a power supply with a simple 78S05 and feeding the 5V into the corresponding inputs of the LoPy board. I have only one wire from the 3V3 open to save boot occasionally. Else this wire is open. The obvious next connection is the LoRa antenna. That's it.

My question: Could it be that any remaining pin on my breadboard must be applied with a proper GND or 3V3 level or with the help of some pull ups? Can it be that I make the LoPy stable by not doing that correctly?

I have bought the twin pack and the base board is running my node and maybe the base board keeps the containing LoPy stable.

[lollisoft]

@RobTuDelft Actually I have updated the gateway and it still freezes. Meanwhile I have monitored the free memory for some minutes and sometimes it drops down to below 5000 bytes. At the last freeze it did not dropped that low, but that is not a save answer to what actually happened.

The gateway code does not only watching for LoRa nodes sending data, but also sends a stat package every minute and also a pull data alarm every 25 seconds.

These two alarms may be the cause to drop the free memory as they may be running in paralell at some point in time.

Can this be optimized or is that exthaustly tested to not be a problem?

[lollisoft]

@livius Yes, meanwhile I have found these functions in micro python documentation.

I found that my gateway has an average of free memory between 30000 and 50000 bytes. But sometimes it drops below 10000 and even 5000.

I have inspected the code and I did not yet found any suspectible code I could try to improve. Even the UDP receive port gets only 1024 bytes and could not be a cause for a memory flood from any hacker in the world trying to send oversized UDP packets to my port 1700. I'll restrict my firewall to only ttn servers in the future.

How could I forward the memory stats as a node (of the gateway itself)?

I have seen, that every 60 seconds a stats packet is send. Can I attach any values as extra fields, such as free_mem?

[livius]

@lollisoft
Memory monitoring you can do by calling

import gc
print(str(gc.mem_free()))

you also can collect garbages by:

gc.collect()

[lollisoft]

@RobTuDelft Seems to be at least a candidate. I'll do that at the weekend. Thanks

[RobTuDelft]

@lollisoft Start by upgrading the firmware to latest versions on both devices I would say. Latest version is 1.6.12.b1.

Release notes here

[lollisoft]

No, I was not debugging the device yet at all. Running on Mac OS X most of the time and if programming at LoPy I jump to my Windows machine and connect with Atom and PyMakr plugin via telnet.

I have two ideas: One is the open port worldwide that is not a good idea but for the start the easyest. And a memory issue. The memory issue is not anymore likely the cause, because the second run was the three day run.

I probably log something to a unix log server and also monitor the memory consumption via ttn.

How can I implement both (logging and memory monitoring)?

I am really new to python.

[RobTuDelft]

Can you still see what's happening during the freeze via the usb-com connection? Maybe add some debugging statements in your code and see what's happening.