Pygate with Gpy lost connection to ttn



  • Hello,

    I have a Pygate with a Gpy attached, and connect the Pygate to TTN via LTE (1nce - LTE Cat-NB1).

    There is a radio mast about 100m air-line distance at the same height as the Pygate, the Gpy has full network connection throughout and no breaks.

    It works very well until after an undefined time (between 15min and 12 hours) the connection to the TTN network is lost and the gateway is offline in the TTN console (webapp and ttnctl), in the 1nce web interface the Gpy is listed as connected.

    (Nodes can send or receive data to the TTN network during this time without any problems)

    I would be very happy about help, it is unfortunately very frustrating, because I have been looking for a long time for the error but can't find it.

    The code and the config are attached

    Subquestion:
    Is it possible to test programmatically if the connection to ttn is active? (like e.g. lte.is_connected())

    Used hardware:
    Pygate & Gpy

    Firmware Version:
    Pycom MicroPython 1.20.2.rc11 [v1.11-d574024] on 2020-08-25; GPy with ESP32

    from network import LTE
    import time
    import machine
    from machine import RTC
    import pycom
    
    
    off = 0x000000
    white = 0xffffff
    red = 0x8a0000
    green = 0x007f00
    orange = 0xff7300
    blue = 0x035afc
    yellow = 0xffff00
    purple = 0xf700ff
    
    
    print('\nStarting LoRaWAN concentrator')
    
    
    pycom.heartbeat(False)
    
    
    def machine_cb(arg):
        evt = machine.events()
        print("-------------------------")
        print(evt)
        print("-------------------------")
        if (evt & machine.PYGATE_START_EVT):
            pycom.rgbled(green)
        elif (evt & machine.PYGATE_ERROR_EVT):
            pycom.rgbled(red)
        elif (evt & machine.PYGATE_STOP_EVT):
            pycom.rgbled(off)
    
    
    # register callback function
    machine.callback(trigger=(machine.PYGATE_START_EVT |
                              machine.PYGATE_STOP_EVT | machine.PYGATE_ERROR_EVT), handler=machine_cb)
    
    
    # setup LTE
    lte = LTE()
    
    if not lte.isconnected():
        # Modem does not connect successfully without first being reset.
        print("Resetting LTE modem ... ", end='')
        lte.send_at_cmd('AT^RESET')
        print("OK")
        time.sleep(1)
    
        if not lte.isattached():
            print("Attaching to LTE network ", end='')
            lte.attach(band=8, apn="iot.1nce.net")
            arc = 0
            while(True):
                if arc >= 60:
                    machine.reset()
    
                if lte.isattached():
                    pycom.rgbled(purple)
                    time.sleep(0.1)
                    pycom.rgbled(off)
                    print(" OK")
                    lte.send_at_cmd("AT+CSQ")
                    break
    
                print('.', end='')
                arc += 1
                time.sleep(1)
    
        if not lte.isconnected():
            print("Connecting on LTE network ", end='')
            lte.connect()
            crc = 0
            while(True):
                if crc >= 30:
                    machine.reset()
    
                if lte.isconnected():
                    pycom.rgbled(orange)
                    time.sleep(0.1)
                    pycom.rgbled(off)
                    print(" OK")
                    break
    
                print('.', end='')
                crc += 1
                time.sleep(1)
    
    
    # Sync time via NTP server for GW timestamps on Events
    print('Syncing RTC via ntp...', end='')
    rtc = RTC()
    rtc.ntp_sync(server="pool.ntp.org")
    
    while not rtc.synced():
        print('.', end='')
        time.sleep(.5)
    print(" OK\n")
    
    
    # Read the GW config file from Filesystem
    fp = open('/flash/config.json', 'r')
    buf = fp.read()
    
    
    # Start the Pygate
    print("------ start paygate ------", end="\n\n")
    
    machine.pygate_init(buf)
    
    # disable degub messages
    # machine.pygate_debug_level(1)
    
    
    # while True:
    #     time.sleep(1)
    #     if not lte.isconnected():
    #         pycom.rgbled(red)
    #         time.sleep(5)
    #         machine.pygate_deinit()
    #         machine.pygate_reset()
    
    {
    	"SX1301_conf": {
    		"lorawan_public": true,
    		"clksrc": 1,
    		"antenna_gain": 0,
    		"radio_0": {
    			"enable": true,
    			"type": "SX1257",
    			"freq": 867500000,
    			"rssi_offset": -164.0,
    			"tx_enable": true,
    			"tx_freq_min": 863000000,
    			"tx_freq_max": 870000000
    		},
    		"radio_1": {
    			"enable": true,
    			"type": "SX1257",
    			"freq": 868500000,
    			"rssi_offset": -164.0,
    			"tx_enable": false
    		},
    		"chan_multiSF_0": {
    			"enable": true,
    			"radio": 1,
    			"if": -400000
    		},
    		"chan_multiSF_1": {
    			"enable": true,
    			"radio": 1,
    			"if": -200000
    		},
    		"chan_multiSF_2": {
    			"enable": true,
    			"radio": 1,
    			"if": 0
    		},
    		"chan_multiSF_3": {
    			"enable": true,
    			"radio": 0,
    			"if": -400000
    		},
    		"chan_multiSF_4": {
    			"enable": true,
    			"radio": 0,
    			"if": -200000
    		},
    		"chan_multiSF_5": {
    			"enable": true,
    			"radio": 0,
    			"if": 0
    		},
    		"chan_multiSF_6": {
    			"enable": true,
    			"radio": 0,
    			"if": 200000
    		},
    		"chan_multiSF_7": {
    			"enable": true,
    			"radio": 0,
    			"if": 400000
    		},
    		"chan_Lora_std": {
    			"enable": true,
    			"radio": 1,
    			"if": -200000,
    			"bandwidth": 250000,
    			"spread_factor": 7
    		},
    		"chan_FSK": {
    			"enable": true,
    			"radio": 1,
    			"if": 300000,
    			"bandwidth": 125000,
    			"datarate": 50000
    		},
    		"tx_lut_0": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_1": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_2": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_3": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_4": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_5": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_6": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_7": {
    			"pa_gain": 0,
    			"mix_gain": 6,
    			"rf_power": 11,
    			"dig_gain": 3
    		},
    		"tx_lut_8": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 13,
    			"dig_gain": 2
    		},
    		"tx_lut_9": {
    			"pa_gain": 0,
    			"mix_gain": 8,
    			"rf_power": 14,
    			"dig_gain": 3
    		},
    		"tx_lut_10": {
    			"pa_gain": 0,
    			"mix_gain": 6,
    			"rf_power": 15,
    			"dig_gain": 2
    		},
    		"tx_lut_11": {
    			"pa_gain": 0,
    			"mix_gain": 6,
    			"rf_power": 16,
    			"dig_gain": 1
    		},
    		"tx_lut_12": {
    			"pa_gain": 0,
    			"mix_gain": 9,
    			"rf_power": 17,
    			"dig_gain": 3
    		},
    		"tx_lut_13": {
    			"pa_gain": 0,
    			"mix_gain": 10,
    			"rf_power": 18,
    			"dig_gain": 3
    		},
    		"tx_lut_14": {
    			"pa_gain": 0,
    			"mix_gain": 11,
    			"rf_power": 19,
    			"dig_gain": 3
    		},
    		"tx_lut_15": {
    			"pa_gain": 0,
    			"mix_gain": 12,
    			"rf_power": 20,
    			"dig_gain": 3
    		}
    	},
    
    	"gateway_conf": {
    		"gateway_ID": "XXX",
    		"server_address": "router.eu.thethings.network",
    		"serv_port_up": 1700,
    		"serv_port_down": 1700,
    		"keepalive_interval": 10,
    		"stat_interval": 30,
    		"push_timeout_ms": 100,
    		"forward_crc_valid": true,
    		"forward_crc_error": false,
    		"forward_crc_disabled": false
    	}
    }
    

    First log of the Pymakr Console, after the TTN connection is lost

    [3453045039] lorapf: INFO_ [main] report
    2020-11-15 21:30:46 GMT #####
    [UPSTREAM] ###
    RF packets received by concentrator: 0
    CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
    RF packets forwarded: 0 (0 bytes)
    PUSH_DATA datagrams sent: 1 (111 bytes)
    PUSH_DATA acknowledged: 0.00%
    [DOWNSTREAM] ###
    PULL_DATA sent: 3 (100.00% acknowledged)
    PULL_RESP(onse) datagrams received: 0 (0 bytes)
    RF packets sent to concentrator: 0 (0 bytes)
    TX errors: 0
    TX rejected (collision packet): 0.00% (req:4, rej:0)
    TX rejected (collision beacon): 0.00% (req:4, rej:0)
    TX rejected (too late): 0.00% (req:4, rej:0)
    TX rejected (too early): 0.00% (req:4, rej:0)
    [JIT] ###
    [jit] queue is empty
    [GPS] ###
    GPS sync is disabled
    END #####
    [3453045124] lorapf: WARN_ [up  ] ignored out-of sync PUSH_ACK packet buff_ack[0:109] != token[0:110]
    [3453045175] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 1
    

    New, different log, with a lost ttn connection

    [3584179962] lorapf: INFO_ [main] report
    ##### 2020-11-17 09:56:21 GMT #####
    ### [UPSTREAM] ###
    # RF packets received by concentrator: 0
    # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
    # RF packets forwarded: 0 (0 bytes)
    # PUSH_DATA datagrams sent: 1 (111 bytes)
    # PUSH_DATA acknowledged: 0.00%
    ### [DOWNSTREAM] ###
    # PULL_DATA sent: 3 (0.00% acknowledged)
    # PULL_RESP(onse) datagrams received: 0 (0 bytes)
    # RF packets sent to concentrator: 0 (0 bytes)
    # TX errors: 0
    ### [JIT] ###
    [jit] queue is empty
    ### [GPS] ###
    # GPS sync is disabled
    ##### END #####
    [3584180778] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 0
    [3584181528] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 1
    


  • Today :)



  • @Gijs

    Ahhh, many thanks!

    Regarding the at ping command i had tomatoes on my eyes, i could have thought of that myself :D

    Do you know when the new firmware will be released?



  • The Pygate reset feature you discuss will be available in the next pygate release (and comes with a pygate firmware update as well). It allows for a hard-reset of the whole module, which can come in handy for an LTE modem lockup.

    The push timeout you discuss is indeed a solution.

    And last, that is a typo of me, it should be 'AT+PING="8.8.8.8"' (watch the quotes, theyre very specific about that)

    Gijs



  • @Gijs Thanks for the fast answer, you are awesome!

    Since the gateway should be placed in a place that is not so easily accessible, I follow the principle, double is better than once :)

    I have already implemented the callback and want to deinitialize and reset the pygate in the handler, unfortunately i get the following error, when I try to call machine.pygate_reset() (as described in the docs)...

    'module' object has no attribute 'pygate_reset'
    

    available methods

    ['__class__', '__name__', 'main', 'ADC', 'BROWN_OUT_RESET', 'CAN', 'DAC', 'DEEPSLEEP_RESET', 'HARD_RESET', 'I2C', 'PIN_WAKE', 'PWM', 'PWRON_RESET', 'PWRON_WAKE', 'PYGATE_ERROR_EVT', 'PYGATE_START_EVT', 'PYGATE_STOP_EVT', 'Pin', 'RMT', 'RTC', 'RTC_WAKE', 'SD', 'SOFT_RESET', 'SPI', 'Timer', 'Touch', 'UART', 'ULP_WAKE', 'WAKEUP_ALL_LOW', 'WAKEUP_ANY_HIGH', 'WDT', 'WDT_RESET', 'callback', 'deepsleep', 'disable_irq', 'enable_irq', 'events', 'flash_encrypt', 'freq', 'idle', 'info', 'mem16', 'mem32', 'mem8', 'pin_sleep_wakeup', 'pygate_cmd_decode', 'pygate_cmd_get', 'pygate_debug_level', 'pygate_deinit', 'pygate_init', 'remaining_sleep_time', 'reset', 'reset_cause', 'rng', 'secure_boot', 'sleep', 'temperature', 'unique_id', 'wake_reason']
    

    regarding the PUSH_ACK timeout I have adjusted the current config, which improves the problem a bit.

    • push_timeout_ms set to 2000 (i know 2 seconds is a long time)
    • autoquit_threshold set to 6 (if 6 packets are lost the gateway will be restarted automatically)
    {
    	"SX1301_conf": {
    		"lorawan_public": true,
    		"clksrc": 1,
    		"antenna_gain": 0,
    		"radio_0": {
    			"enable": true,
    			"type": "SX1257",
    			"freq": 867500000,
    			"rssi_offset": -164.0,
    			"tx_enable": true,
    			"tx_freq_min": 863000000,
    			"tx_freq_max": 870000000
    		},
    		"radio_1": {
    			"enable": true,
    			"type": "SX1257",
    			"freq": 868500000,
    			"rssi_offset": -164.0,
    			"tx_enable": false
    		},
    		"chan_multiSF_0": {
    			"enable": true,
    			"radio": 1,
    			"if": -400000
    		},
    		"chan_multiSF_1": {
    			"enable": true,
    			"radio": 1,
    			"if": -200000
    		},
    		"chan_multiSF_2": {
    			"enable": true,
    			"radio": 1,
    			"if": 0
    		},
    		"chan_multiSF_3": {
    			"enable": true,
    			"radio": 0,
    			"if": -400000
    		},
    		"chan_multiSF_4": {
    			"enable": true,
    			"radio": 0,
    			"if": -200000
    		},
    		"chan_multiSF_5": {
    			"enable": true,
    			"radio": 0,
    			"if": 0
    		},
    		"chan_multiSF_6": {
    			"enable": true,
    			"radio": 0,
    			"if": 200000
    		},
    		"chan_multiSF_7": {
    			"enable": true,
    			"radio": 0,
    			"if": 400000
    		},
    		"chan_Lora_std": {
    			"enable": true,
    			"radio": 1,
    			"if": -200000,
    			"bandwidth": 250000,
    			"spread_factor": 7
    		},
    		"chan_FSK": {
    			"enable": true,
    			"radio": 1,
    			"if": 300000,
    			"bandwidth": 125000,
    			"datarate": 50000
    		},
    		"tx_lut_0": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_1": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_2": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_3": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_4": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_5": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_6": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 9,
    			"dig_gain": 3
    		},
    		"tx_lut_7": {
    			"pa_gain": 0,
    			"mix_gain": 6,
    			"rf_power": 11,
    			"dig_gain": 3
    		},
    		"tx_lut_8": {
    			"pa_gain": 0,
    			"mix_gain": 5,
    			"rf_power": 13,
    			"dig_gain": 2
    		},
    		"tx_lut_9": {
    			"pa_gain": 0,
    			"mix_gain": 8,
    			"rf_power": 14,
    			"dig_gain": 3
    		},
    		"tx_lut_10": {
    			"pa_gain": 0,
    			"mix_gain": 6,
    			"rf_power": 15,
    			"dig_gain": 2
    		},
    		"tx_lut_11": {
    			"pa_gain": 0,
    			"mix_gain": 6,
    			"rf_power": 16,
    			"dig_gain": 1
    		},
    		"tx_lut_12": {
    			"pa_gain": 0,
    			"mix_gain": 9,
    			"rf_power": 17,
    			"dig_gain": 3
    		},
    		"tx_lut_13": {
    			"pa_gain": 0,
    			"mix_gain": 10,
    			"rf_power": 18,
    			"dig_gain": 3
    		},
    		"tx_lut_14": {
    			"pa_gain": 0,
    			"mix_gain": 11,
    			"rf_power": 19,
    			"dig_gain": 3
    		},
    		"tx_lut_15": {
    			"pa_gain": 0,
    			"mix_gain": 12,
    			"rf_power": 27,
    			"dig_gain": 3
    		}
    	},
    
    	"gateway_conf": {
    		"gateway_ID": "XXX",
    		"server_address": "router.eu.thethings.network",
    		"serv_port_up": 1700,
    		"serv_port_down": 1700,
    		"keepalive_interval": 10,
    		"stat_interval": 30,
    		"push_timeout_ms": 2000,
    		"autoquit_threshold": 6,
    		"forward_crc_valid": true,
    		"forward_crc_error": false,
    		"forward_crc_disabled": false
    	}
    }
    

    When I try to run AT+CPING in a while loop at the end of the file, I only get "ERROR", I have no idea why this happens...

    while True:
        lte.pppsuspend()
        print(lte.send_at_cmd('AT+CPING="8.8.8.8"'))
        lte.pppresume()
        time.sleep(5)
    

    Maybe it helps, the Gpy has a LTE signal strenght of:
    +CSQ: 12,99

    Value 	RSSI dBm 	Condition
    2 	-109 	        Marginal
    3 	-107 	        Marginal
    4 	-105 	        Marginal
    5 	-103 	        Marginal
    6 	-101 	        Marginal
    7 	-99 	        Marginal
    8 	-97 	        Marginal
    9 	-95 	        Marginal
    10 	-93 	        OK
    11 	-91 	        OK
    12 	-89 	        OK
    13 	-87 	        OK
    14 	-85 	        OK
    15 	-83 	        Good
    16 	-81 	        Good
    17 	-79 	        Good
    18 	-77 	        Good
    19 	-75 	        Good
    20 	-73 	        Excellent
    21 	-71 	        Excellent
    22 	-69 	        Excellent
    23 	-67 	        Excellent
    24 	-65 	        Excellent
    25 	-63 	        Excellent
    26 	-61 	        Excellent
    27 	-59 	        Excellent
    28 	-57 	        Excellent
    29 	-55 	        Excellent
    30 	-53 	        Excellent
    

    ... maybe it is a bit less, i will get a bigger LTE antenna within the next few days, maybe this will help a little bit.



  • There could be many causes for a sudden loss in connection that can both me client, networks side initiated, or somewhere in between. Now the trick is to discover this and reconnect when it happens.
    There are several ways of doing this:
    First is to ping a known online server every once in a while (you can use lte.send_at_cmd('AT+CPING="8.8.8.8"') for example, or make a http request to something). There is also a ping library available from micropython, but that appears to have a memory leak, making it less suitable.
    If it fails, you can suspect the connection is offline and reset the device.

    Next, you can use the lte callback described on this page: https://docs.pycom.io/tutorials/networks/lte/. It is not foolproof and might not catch all disconnection events (yet)

    The thing with 'PUSH_ACK receive timeout' is not as reliable, as slow networks might always give this message, even though the messages are pushed through (we wait for a maximum of 15 milliseconds if I remember correctly)


Log in to reply
 

Pycom on Twitter