Pygate - ethernet reliability issues
-
@jcaron @gijs @peterp
I tried another network on an (much) older sitecom 10/100mbps switch connected to my computer on a different ethernet port.
That network does not have the noise which is present on my "normal" office lan.I am now seeing this:
64 bytes from 192.168.0.2: icmp_seq=386 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=387 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=388 ttl=255 time=1.90 ms 64 bytes from 192.168.0.2: icmp_seq=389 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=390 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=391 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=392 ttl=255 time=1.92 ms 64 bytes from 192.168.0.2: icmp_seq=393 ttl=255 time=1.96 ms 64 bytes from 192.168.0.2: icmp_seq=394 ttl=255 time=1.92 ms # <== until here it worked without any of the previous problems; 64 bytes from 192.168.0.2: icmp_seq=395 ttl=255 time=1.92 ms # <== at this point in time I attach my other LAN to the sitecom switch 64 bytes from 192.168.0.2: icmp_seq=413 ttl=255 time=1.29 ms # <== here I disconnect that cable 64 bytes from 192.168.0.2: icmp_seq=414 ttl=255 time=1.90 ms 64 bytes from 192.168.0.2: icmp_seq=415 ttl=255 time=1.90 ms 64 bytes from 192.168.0.2: icmp_seq=416 ttl=255 time=2.14 ms 64 bytes from 192.168.0.2: icmp_seq=417 ttl=255 time=1.69 ms 64 bytes from 192.168.0.2: icmp_seq=418 ttl=255 time=1.91 ms 64 bytes from 192.168.0.2: icmp_seq=419 ttl=255 time=1.91 ms 64 bytes from 192.168.0.2: icmp_seq=420 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=421 ttl=255 time=1.91 ms 64 bytes from 192.168.0.2: icmp_seq=422 ttl=255 time=1.92 ms 64 bytes from 192.168.0.2: icmp_seq=423 ttl=255 time=1.91 ms 64 bytes from 192.168.0.2: icmp_seq=424 ttl=255 time=1.30 ms 64 bytes from 192.168.0.2: icmp_seq=425 ttl=255 time=1.94 ms # <== at this point in time I attach my other LAN to the sitecom switch again 64 bytes from 192.168.0.2: icmp_seq=434 ttl=255 time=1.31 ms # <== here I disconnect that cable again 64 bytes from 192.168.0.2: icmp_seq=435 ttl=255 time=2.10 ms 64 bytes from 192.168.0.2: icmp_seq=436 ttl=255 time=1.93 ms ^C --- 192.168.0.2 ping statistics --- 436 packets transmitted, 372 received, +36 errors, 14.6789% packet loss, time 438231ms rtt min/avg/max/mdev = 1.186/3.154/419.886/21.812 ms, pipe 3 [jaap@jaap ~ ]$This makes me believe for at least 90% this is actually a software issue somewhere in the pygate/pycom module related to incoming network traffic.
Doesn't this still ring a bell? I could potentially debug this, but there is currently not enough datasheet information available about components used, especially the POE module. I couldn't even find information regarding the required POE voltage and typical current usages. But that's a bit offtopic for now.PS 15 minutes later without the other lan:
[jaap@jaap ~ ]$ ping 192.168.0.2 PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. 64 bytes from 192.168.0.2: icmp_seq=1 ttl=255 time=129 ms 64 bytes from 192.168.0.2: icmp_seq=2 ttl=255 time=1.22 ms 64 bytes from 192.168.0.2: icmp_seq=3 ttl=255 time=1.88 ms 64 bytes from 192.168.0.2: icmp_seq=4 ttl=255 time=1.92 ms 64 bytes from 192.168.0.2: icmp_seq=5 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=6 ttl=255 time=1.95 ms 64 bytes from 192.168.0.2: icmp_seq=7 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=8 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=9 ttl=255 time=1.91 ms 64 bytes from 192.168.0.2: icmp_seq=10 ttl=255 time=1.96 ms 64 bytes from 192.168.0.2: icmp_seq=11 ttl=255 time=1.92 ms 64 bytes from 192.168.0.2: icmp_seq=12 ttl=255 time=1.92 ms 64 bytes from 192.168.0.2: icmp_seq=13 ttl=255 time=1.91 ms 64 bytes from 192.168.0.2: icmp_seq=14 ttl=255 time=1.91 ms 64 bytes from 192.168.0.2: icmp_seq=15 ttl=255 time=1.92 ms 64 bytes from 192.168.0.2: icmp_seq=16 ttl=255 time=1.92 ms 64 bytes from 192.168.0.2: icmp_seq=17 ttl=255 time=1.91 ms 64 bytes from 192.168.0.2: icmp_seq=18 ttl=255 time=1.92 ms 64 bytes from 192.168.0.2: icmp_seq=19 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=20 ttl=255 time=1.62 ms 64 bytes from 192.168.0.2: icmp_seq=21 ttl=255 time=1.93 ms 64 bytes from 192.168.0.2: icmp_seq=22 ttl=255 time=1.94 ms 64 bytes from 192.168.0.2: icmp_seq=23 ttl=255 time=1.94 ms 64 bytes from 192.168.0.2: icmp_seq=24 ttl=255 time=1.94 ms ^C --- 192.168.0.2 ping statistics --- 24 packets transmitted, 24 received, 0% packet loss, time 23043ms rtt min/avg/max/mdev = 1.219/7.172/128.881/25.378 ms [jaap@jaap ~ ]$@jcaron said in Pygate - ethernet reliability issues:
Otherwise it could be quite a long debug. As stated above, I would start by trying to determine if the issue is on the RX or TX side, by either switching on Ethernet on the WiPy or using a simple UDP echo sever with logging, to see if the delay (and loss) happens on one side or the other.
Do we have some Pycom/python examples at hand to try within 30 seconds?
-
@Jaap-Crezee This looks like a good starting point. You probably want to add a loop and send a counter in the client, and print the counter on both client (TX and RX) and on the server, with timestamps.
It's regular Python but I don't think there's much to change.
-
I am on a noisy network. I could also try moving to a less noisy network (although I would argue that a device should not get slower by receiving some broadcasts; we are talking about 0.5-1kbyte/s here.). And we are not always in control of the networks we should be connecting to.
-
@jcaron said in Pygate - ethernet reliability issues:
Meanwhile, I investigated a bit further:
It seems RX related.
I come to this conclusion because of the following:
I have a few Lora devices. They can only join the LoraWAN network just after a reset of the Pygate. For joining a LoraWAN network it seems the Lora device should be able to receive a packet back. That stops functioning after a while.
This correlates with ICMP pings getting slower and slower until they timeout and no longer work (and will never work again).
It seems packets no longer arrive at the Pygate, but it can still send data from a (wireless) Lora device to the LoraWAN (TTN) backend. So TX remains operational. I also see this (packet wise) behavior while sniffing the network. I also notice the >>> console getting very slow if this problem is present.With this in mind, I understand people state that the Pygate will remain operational because Lora packets keep arriving at the LoraWAN backbend (TTN for example), but I think this claim is not true in some noisy networks because packets from the backend are not getting back through the Pygate gateway to wireless LoraWAN devices.
Unfortunately, I do not have time to investigate myself in more detail at the moment, but - if desirable - I could produce some excerpts of the broadcast packets which are normally present on my network and which could trigger this problem. -
@schmelpe said in Pygate - ethernet reliability issues:
Hi,
ok. I can confirm that the PyGate with WiPy3 stops responding to PING requests a few minutes after startup with PoE.
But I can't confirm that the PyGate didn't work anymore. The Gateways status within the TTN Console is updated every 30 Seconds or so for the last 15 days.Hi,
ok. I can confirm that the PyGate with WiPy3 stops responding to PING requests a few minutes after startup with PoE.
But I can't confirm that the PyGate didn't work anymore. The Gateways status within the TTN Console is updated every 30 Seconds or so for the last 15 days. -
Hi,
my 2 cents concerning this issue...
I was using an Ethernet cable with the POE adapter, without providing power(POE), and also using a USB cable to power up the PYGATE, .... I had all kind of problems.
Since I removed the USB cable, and replaced it with a POE injector over the Ethernet cable only, .... storm is over.
Regards.
Richard.
-
Interesting... I'll have to try that for myself, thanks! (also tagging @peterp in this)
-
@Richard-Martineau said in Pygate - ethernet reliability issues:
I was using an Ethernet cable with the POE adapter, without providing power(POE), and also using a USB cable to power up the PYGATE, .... I had all kind of problems.
Since I removed the USB cable, and replaced it with a POE injector over the Ethernet cable only, .... storm is over.If this helps, I still think it should be fixed without the Po. I do not want to use PoE I just need the E from it..
-
The only issue I can think of here right now, is that the USB 'communication' is the potential issue here (I cannot see a reason how it would be related to the power distribution of the PoE and USB devices, but you probably know about Murphy's law). Could you perhaps try a USB cable without communication link (or like connected to a powerbank?)
-
Hi @Richard-Martineau
Over the last weeks we did some tests with the method you describe, but unfortunately no resolution there :( -
This post is deleted!
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login