LoPy Nano-Gateway


  • Pycom Pyoneers

    **NOTE: THIS EXAMPLE HAS BEEN UPDATED AND I RECOMMEND USING THE NEW CODE THAT CONTAINS SEVERAL IMPROVEMENTS

    GO TO https://forum.pycom.io/post/3242

    Here are some code samples to put the LoPy in nano-gateway mode. This is just a code demo and you will need to change it to meet your needs.

    For this demo we are connecting 2 LoPys (nodes) to 1 LoPy in Nano-Gateway mode

    nano_gateway.py

    import socket
    import struct
    from network import LoRa
    
    # A basic package header, B: 1 byte for the deviceId, B: 1 byte for the pkg size, %ds: Formated string for string
    _LORA_PKG_FORMAT = "!BB%ds"
    # A basic ack package, B: 1 byte for the deviceId, B: 1 bytes for the pkg size, B: 1 byte for the Ok (200) or error messages
    _LORA_PKG_ACK_FORMAT = "BBB"
    
    # Open a LoRa Socket, use rx_iq to avoid listening to our own messages
    lora = LoRa(mode=LoRa.LORA, rx_iq=True)
    lora_sock = socket.socket(socket.AF_LORA, socket.SOCK_RAW)
    lora_sock.setblocking(False)
    
    while (True):
        recv_pkg = lora_sock.recv(512)
        if (len(recv_pkg) > 2):
            recv_pkg_len = recv_pkg[1]
    
            device_id, pkg_len, msg = struct.unpack(_LORA_PKG_FORMAT % recv_pkg_len, recv_pkg)
    
    # If the uart = machine.UART(0, 115200) and os.dupterm(uart) are set in the boot.py this print should appear in the serial port
            print('Device: %d - Pkg:  %s' % (device_id, msg))
    
            ack_pkg = struct.pack(_LORA_PKG_ACK_FORMAT, device_id, 1, 200)
            lora_sock.send(ack_pkg)
    

    The _LORA_PKG_FORMAT is used to have a way of identifying the different devices in our network
    The _LORA_PKG_ACK_FORMAT is a simple ack package as response to the nodes package

    node_1.py

    import os
    import socket
    import time
    import struct
    from network import LoRa
    
    # A basic package header, B: 1 byte for the deviceId, B: 1 bytes for the pkg size
    _LORA_PKG_FORMAT = "!BB%ds"
    _LORA_PKG_ACK_FORMAT = "BBB"
    DEVICE_ID = 0x01
    
    
    # Open a Lora Socket, use tx_iq to avoid listening to our own messages
    lora = LoRa(mode=LoRa.LORA, tx_iq=True)
    lora_sock = socket.socket(socket.AF_LORA, socket.SOCK_RAW)
    lora_sock.setblocking(False)
    
    while(True):
        # Package send containing a simple string
        msg = "Device 1 Here"
        pkg = struct.pack(_LORA_PKG_FORMAT % len(msg), DEVICE_ID, len(msg), msg)
        lora_sock.send(pkg)
        
        # Wait for the response from the gateway. NOTE: For this demo the device does an infinite loop for while waiting the response. Introduce a max_time_waiting for you application
        waiting_ack = True
        while(waiting_ack):
            recv_ack = lora_sock.recv(256)
        
            if (len(recv_ack) > 0):
                device_id, pkg_len, ack = struct.unpack(_LORA_PKG_ACK_FORMAT, recv_ack)
                if (device_id == DEVICE_ID):
                    if (ack == 200):
                        waiting_ack = False
                        # If the uart = machine.UART(0, 115200) and os.dupterm(uart) are set in the boot.py this print should appear in the serial port
                        print("ACK")
                    else:
                        waiting_ack = False
                        # If the uart = machine.UART(0, 115200) and os.dupterm(uart) are set in the boot.py this print should appear in the serial port
                        print("Message Failed")
    
        time.sleep(5)
    

    The node is always sending packages and waiting for the ack from the gateway.

    To adapt this code to your needs you might:

    1. Put a max waiting time for the ack to arrive and resend the package or mark it as invalid
    2. Increase the package size changing the _LORA_PKG_FORMAT to "BH%ds" the H will allow to keep 2 bytes for size (for more information about struct format go here)
    3. Reduce the package size with bitwise manipulation
    4. Reduce the message size (for this demo a string) to something more useful for you development

  • Pycom Pyoneers

    @Colateral
    Thanks for the quick response, i'll try a blocking approach to see if i can reproduce the error and check whats the root of it.



  • @Roberto HI Roberto. Thanks. I ran the last script and is running fine. Regarding the memory leak, I believe is related with socket blocking approach. If we change the script and move to non blocking (as your script does) everything works fine.


  • Pycom Pyoneers

    Hi @Colateral
    As i wrote in my last message, there is a new example of the nano-gateway with a non-blocking loop approach.
    Regarding the memory issues we will look into this. I left the code on the other post running for 3 hours without any problems. I will leave it running today on a test bench for 24+ h to see if there are any problems and monitor the memory while it transmits. Will let you know the results.

    The link to the new code is
    https://forum.pycom.io/post/3242



  • @Roberto Your example it is a good start but looping and blocking might not be a good approach.

    We built another script that is using threads with blocking socket ... and run the script on a minimal Lora "star" network: one GW and 2 NODES. For each device you have to configure tx,rx and device id in cfg file . and upload the script and cfg file on it and run the script in Putty (not Pymakr). We didn't faced with this script clashing issues.. but we faced other issues.

    You will see that after awhile on GW, the script is ending with a memory leak... but this is never happening on the nodes.
    If you run the script from 3 Pymakr instances ... sometimes is working for hours and after that the GW is stop sending.

    We conclude that is something wrong on the stack.

    The issue might be related only to rx_iq configuration.
    https://drive.google.com/drive/folders/0BzRE8IpnMMnLNHBseDJWblN4VGM?usp=sharing


  • Pycom Pyoneers

    Hi @feabhas
    yeah, that sounds about right. Change that please. Ill edit the post accordingly Thanks



  • @Roberto I have successfully got a 2 node and 1 gateway setup working based on your code, thanks very much. One minor query, In the gateway you use

    _LORA_PKG_FORMAT = "!BB%ds"
    

    but in the node_1.py you miss off the !, e.g.

    _LORA_PKG_FORMAT = "BB%ds"
    

    Shouldn't they be consistent?


  • Pycom Pyoneers

    Hi @Colateral,

    Yes, i have noticed that sometimes some messages arrive malformed or with bytes missing. This could be due to LoRa message collision between devices. If two devices send at the same time in the same band this can happen.

    There is a new blog here in the forum with a new code suggestion that includes.

    1. Message length check on the nano-gateway
    2. Message retry on the nodes
    3. Max timeout in the nodes

    Number 3 will help with the fact that you have to have the nano-gateway on before you start the nodes. In this code, after sending a message the node waits in an infinite loop for the ack. The problem with this approach is that if no ack is received (the nano-gateway was not connected yet) the node will stay in that loop until reseted.

    Keep in ming that both this and the new code are just examples for specific usages and they require adaptation depending on your needs.
    The link for the new post is LoPy Nano-Gateway Extended (Timeout and Retry)

    Best,
    Roberto



  • @Roberto We setup a test bench with 3 LoPy following the idea of your code. We did the test with blocking socket and thread and non blocking sockets. The result is the same.

    2 are the nodes configured as tx_iq and and one is the Gw (rx_iq).

    The script is simple: each device (node and GW) is sending 29 bytes message size (_MSG_FORMAT = "!HHLBB%ds") each 2 seconds. In the msg data we are printing a counter (that varies from device). That counter is also in the header (see above)

    We noticed that sometimes if you start the gw after the nodes... randomly some of the nodes are not receiving the data, despite the fact the GW is receiving the data from the nodes. And we observed also that sometimes when we start the node after the GW, the GW is not receiving the data from the nodes.

    You need to reset the node or GW in order to get the broadcasted data from to the others... and this hazardous and not at all good in real deployment.

    BUT THE MOST PROBLEMATIC is the fact that the 29bytes are partially received, sometimes you are getting 15bytes or whatever number less than 29bytes.

    Have you experience this connection issue related with nodes/gw starting order?
    Does LoPy stack is guarantee the receiving of msg in one chunk? ( if you send 29bytes you got 29bytes)

    Here are some samples:
    good msg receive

    from: 800
    to: 0
    Id: 3058
    Flags: 0
    PayloadSize: 18
    Payload: b'3058- 1234567890 '
    <<< beat msgId 3066
    bufferSize 29
    pkgSize 28
    data_size 18
    remaining buffer 0

    bad msg receive

    from: 800
    to: 0
    Id: 3059
    Flags: 0
    PayloadSize: 18
    Payload: b'3059- 1234567890 '
    bufferSize 15
    buffer too small <<< ustruct exception



  • @gertjanvanhethof I'm also interrested to use LoPy as a LoRaWan gateway. Looking forward to a solution.



  • This post is deleted!


  • Hi,

    On "Nano-Gateway" with the latest firmware 1.3.0.b1 i'm still getting "ValueError: buffer too small" after hour of two running posted sample code. Is this problem with the sample code or is it firmware problem?



  • This post is deleted!


  • Just run the Nano-Gateway with the latest firmware 1.3.0.b1. All problems are resolved :D. Thanks Pycom team.



  • The distance and the power do not seem to be the problem. I tried running the test over 100 meters. If I put a time.sleep(0.2) after lora_sock.recv(512) and lora_sock.send(pkg) on both server side and client side, the code runs. In previous firmware the code run straight out of the box. I think the OSError:[Enro 11] is caused by a cue in the send buffer. Sometimes there are still receive messages left in the buffer, even after a soft reset. I use the code

    recv_dumpbuffer = lora_sock.recv(512)
    time.sleep(0.2)
    

    to make sure the receivebuffer is empty.
    the 200ms waits in between Lora commands are hardly ideal. On previous firmware versions the code run straight out of the box.



  • @Feiko Are both devices close? Try lowering the transmission power to the minimum (2 dBm) and move them several meters apart (or use an attenuator).

    I have been doing tests between a LoPy and a Raspberry with a Dragino Lora/GPS hat (SX1276 based) and with just 2 dBm I can receive packets reliably with the RPi and the LoPy in different flats and at the opposite sides of the building. Depending on the antenna positioning I even registered an average SNR of -8 dB, which is amazing.



  • @Roberto Thanks for your reply.

    I've tried to give the program enough time to receive. According to documentation time.time() should be in seconds not milliseconds. However

    import time
    from machine import Timer
    
    chrono = Timer.Chrono()
    
    chrono.start()
    timeout = time.time() + 20
    while time.time() < timeout:
        pass
    chrono.stop()
    print(chrono.read())
    

    This outputs 3.062622 so 3 seconds.
    I've even tried increasing this to 10 seconds, no luck.

    I used the deviceId as a package identifier. Bad practice, but it made for an easy test scenario in which I could identify the package beinig acknoledged.

    using time.sleep(0.2) before sending the ACK package fixed the nanogateway crash. However the client still receives the wrong response message (DEVICE_ID -1).

    You used the new firmware is this 1.2.2.b1 or the upcoming release?

    Could you look into reproducing the problem?


  • Pycom Pyoneers

    Hello @Feiko,
    This is an answer to your post here New release New firmware release 1.2.2.b1:
    I'm answering in the blog since this will allow us to keep a more organised discussion

    Regarding your problem with the NanoGateway code, here are a few remarks from your post:

    1. In your code you are sending a message and changing the ID every time you do it, the original code had this Device ID as an ID of the device not as a message Id witch is the way you are using it here. It was meant as a way of identifying 2 or mode LoPys connected to the same gateway. I actually tested this same code with 10 devices some days after the original post.

    2. Even with 1 in mind you are just giving the device time.time() + 20 (that is 20ms) to
      a. Pack the message
      b. Send the message using LoRa
      c. 2nd LoPy receive the message using LoRa
      d. Message unpack and processing
      e. Sending ACK over LoRa

      The problem with this approach is that in LoRa packages (specially several byte long ones) can take hundreds of milliseconds to transmit.
      Since you are checking almost immediately after you send the message and in a while loop with no delays you get that no ACK received output.
      My recommendation would be to
      a. Extend the amount of time you are going to way for a message (this might change depending on your application). Lets say 20 seconds for testing
      b. make the while(waiting_ack and timeout > time.time()) delay at least 100 ms before checking for messages again

    3. Regarding the Ack with wrong device_id: 1 message. A little more of the same as point 2. To illustrate here is a little time span of your problem
      a. Message ID 1 is created and send
      b. Node waits 20 ms for ack
      c. Message arrives at gateway in ms: 25
      d. Node already ditched the package as non responding and sends Message ID: 2
      e. Node wauts 20 ms for ack on pkg 2
      f. Gateway responds to ack pkg 1 in ms 26
      g. Node gets to ms 40 and trashes the pack 2 as non responding to ack and sends message ID: 3
      g. Message with ack of message 1 arrives at node in ms 48.
      h. At this point if (device_id == DEVICE_ID) will check if the ack that jsut arrived (1) matches the current pkg (3). NO. node prints to screen
      Ack with wrong device_id

      Adding extra time (like in point 2) would fix this issue but it will give you a temporal solution only, since the error will present itself if the message takes more than 20 secs (unprovable). The full solution would require an implementation of a retry message sending process with a message queue of messages that where sent and are waiting for an ACK.
      There is a lot of documentation regarding this kind of algorithms out there and it was not implemented in the original NanoGateway post since it escapes the scope of the demonstration code.

    4. Regarding the OSError: [Errno 11] EAGAIN i'll check into this but i tried the original code posted in the blog with the new firmware and was not able to reproduce it.


  • administrators

    @this-wiederkehr if you go to the docs: https://docs.pycom.io/pycom_esp32/library/network.LORA.html?highlight=lora#network.lora.init

    You can see the default values:

    lora.init(mode, *, frequency=868000000, tx_power=14, bandwidth=LoRa.868000000, sf=7, preamble=8, coding_rate=LoRa.CODING_4_5, power_mode=LoRa.ALWAYS_ON, tx_iq=false, rx_iq=false, adr=false, public=true, tx_retries=1)
    

    Cheers,
    Daniel



  • I see, this uses some default settings for lora (frequency, tx_power, bandwidth, sf, preamble, coding_rate). But what are the default values? Is there a way to get the default values programatically?


Log in to reply
 

Looks like your connection to Pycom Forum was lost, please wait while we try to reconnect.