Need Help-Reliable connection to TTN.

[sachindayma]

I am using two boards first being a Pygate with Lopy4 and the second is pysense with lopy4. I belong to IN865 region which is not explictly defined in the pygate config. Hence to comply to th efrequqency band requirements of this region, I modified the default config example of pygate under documentation. The current pygate config is attached.[0_1597223028818_config.json](Uploading 100%)
I dont have any issues connecting pygate to TTN
My pysense conde is pretty much using the default code. All i am trying to done is have the pysense connect to TTN, send data to TTN every 5 minutes. I can see that the Pysense wakes up accurately every 5 minutes and the HasJoined function of lora also returns true but the payload I want to send on the socket is not reliably recieved by TTN.

The pysense code block is below

# initialize LoRa in LORAWAN mode.
# Please pick the region that matches where you are using the device:
# Asia = LoRa.AS923
# Australia = LoRa.AU915
# Europe = LoRa.EU868
# United States = LoRa.US915
lora = LoRa(mode=LoRa.LORAWAN, region=LoRa.EU868, tx_retries = 3)

# create an OTA authentication params
dev_eui = binascii.unhexlify('70B3D5499D087953')
app_eui = binascii.unhexlify('70B3D57ED003268F')
app_key = binascii.unhexlify('345493A61547B7120452809FCB0C017F')

# set the 3 default channels to the same frequency (must be before sending the OTAA join request). The frequencies below
# are specific to IN865 frequency Plan
lora.add_channel(0, frequency=865062500, dr_min=0, dr_max=5)
lora.add_channel(1, frequency=865402500, dr_min=0, dr_max=5)
lora.add_channel(2, frequency=865985000, dr_min=0, dr_max=5)

lora.join(activation=LoRa.OTAA, auth=(dev_eui,app_eui, app_key), timeout=0, dr=config.LORA_NODE_DR)
print("Lora Join request sent")
timeout=0
while not lora.has_joined() and timeout<20:
    time.sleep(2.5)
    print('Not joined yet...')
    timeout=timeout+1

sensorData = ReadSensor()

#Initiate Lora COnnection
if lora.has_joined():
    pycom.rgbled(0x008000)
    print("Lora Connected")

    #Read Sensor Data and create a content to send over Lora
    contentStr = ""
    contentStr = str(sensorData.readTemperature())+"," + str(sensorData.readHumidity())
    print(contentStr) 

    # create a LoRa socket
    newLoraSocket = socket.socket(socket.AF_LORA, socket.SOCK_RAW)
    # set the LoRaWAN data rate
    newLoraSocket.setsockopt(socket.SOL_LORA, socket.SO_DR, config.LORA_NODE_DR)
    newLoraSocket.setsockopt(socket.SOL_LORA, socket.SO_CONFIRMED, False)  
    # make the socket non-blocking
    newLoraSocket.setblocking(False)

    bytesToSend = b''+contentStr
    newLoraSocket.send(bytesToSend)    
    time.sleep(120)
    machine.deepsleep(300000)   
else:
    pycom.rgbled(0xFF0000)
    print("ConnectionFailed")
    machine.deepsleep(240000)

[sachindayma]

@Gijs
Hip Hip Hurray. I have a much reliable connection now with almost no packet loss. I had used that same file to get the gateway configuration but I am not sure where the error was. I just looked into the gateway config file which I had uploaded in the first post and I realized that the gateway was still looking at the eu router rather than the as one. The rest of the settings were mapped correctly.

Thanks.

[Gijs]

@sachindayma said in Need Help-Reliable connection to TTN.:

The issue definately is about frequency Plan

I was suspecting this as well, but you did not provide the pygate json frequency settings or location. There are also some peculiarities with TTN and their routers. I believe the EU router only supports EU available frequency plans. Did you select the correct router for your area in both the pygate settings and the correct frequency plan in TTN? I want to provide more working frequency plan settings, but my experience is limited..

You can grab the correct frequency plan json from here: https://github.com/TheThingsNetwork/gateway-conf for your region (note that you need to add the items at the bottom, I will make a guide on how to do this correctly

		"gateway_ID": "XXXX",
		"keepalive_interval": 10,
		"stat_interval": 30,
		"push_timeout_ms": 100,
		"forward_crc_valid": true,
		"forward_crc_error": false,
		"forward_crc_disabled": false

. Make sure you select the accompanying router in TTN and frequency plan in your OTAA node

[sachindayma]

@Gijs
Thank you for your time. I have two wifi connections and I have tried this on both of them. I don't think internet reliability is a question because the connection request reaches pretty consistently.
As you rightly annoted, the pygate receives the packet from the node(pysense in my case). But they dont reach TTN for sure. The data log on TTN does not show any thing other than the activation request(screen shot below). My suspicion is that the packet is lost in transition but not sure where. Is there any way I can enable more logs out of pygate?

The issue definately is about frequency Plan. For the same setup, if I change gateway frequency plan as per the EU(using the default config file), the data is delivered reliably to TTN.

[sachindayma]

This post is deleted!

[sachindayma]

@Gijs
Making the socket Blocking and unblocking didn't help much either on the pysense. The only patterns I have noticed are

1. If the packet is received on TTN, it always is with a 10s delay with respect to join/activation request on TTN.
2. The data when received always mentions retry next to port number (image below)
   
3. a lot of warnings are on the pygate which mention "ignored out-of sync PUSH_ACK packet buff_ack[0:10] != token[0:11]". The buff_ack and token are always off by 1.

[Gijs]

It seems from the limited logs, the Pygate is receiving the packets fine by this section: (I annotated it)

### [UPSTREAM] ###
# RF packets received by concentrator: 2 <--- 2 packets received 
# CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00% <--- CRC is good
# RF packets forwarded: 2 (46 bytes)
# PUSH_DATA datagrams sent: 3 (523 bytes) <--- 2 LoRa packets and 1 'gateway online' packet
# PUSH_DATA acknowledged: 0.00%. <--- None are acknowledged

but not forwarding them to the TTN server. How is your Pygate connected to the internet? Through WiFi or Ethernet? Make sure the communication is reliable

You should be able to verify this by checking the Pygate logs in combination with the TTN logs. A received packet will look like this in the terminal: (sorry I will add this tomorrow when Im able to!)

And with a blue arrow in TTN:

Best,
Gijs

[sachindayma]

Below is the gateway log when my pysense tried to send data to TTN and the data was not received by TTN.

[3811533543] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 0
[3811533593] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 1
[3811533639] lorapf: WARN_ [up  ] ignored out-of sync PUSH_ACK packet buff_ack[5:210] != token[5:211]
[3811533689] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 1
[3811540891] lorapf: INFO_ jitqueue: Current concentrator tv_sec=1546 time_us=1546739590, pkt_type=0, packet->count_us=1544297558, (c-t)=4292525264/-2442032 us = 4292/4292 s
[3811540896] lorapf: WARN_ jitqueue: IGNORED: not REJECTED, already too late to send it (current=1546739590, packet=1544297558, type=0)
[3811553444] lorapf: INFO_ [up  ] received pkt from mote: D003268F (fcnt=46037/B3D5), RSSI -59.0
[3811553499] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 0
[3811553549] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 1
[3811557603] lorapf: INFO_ jitqueue: Current concentrator tv_sec=1563 time_us=1563451618, pkt_type=0, packet->count_us=1564288494, (c-t)=836876/836876 us = 0/0 s
[3811557607] lorapf: WARN_ jitqueue: IGNORED: not REJECTED, already too late to send it (current=1563451618, packet=1564288494, type=0)
[3811563498] lorapf: INFO_ [main] report
##### 2020-08-12 15:00:30 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 2
# CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
# RF packets forwarded: 2 (46 bytes)
# PUSH_DATA datagrams sent: 3 (523 bytes)
# PUSH_DATA acknowledged: 0.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 3 (66.67% acknowledged)
# PULL_RESP(onse) datagrams received: 2 (426 bytes)
# RF packets sent to concentrator: 1 (66 bytes)
# TX errors: 0
# TX rejected (collision packet): 0.00% (req:85, rej:0)
# TX rejected (collision beacon): 0.00% (req:85, rej:0)
# TX rejected (too late): 0.00% (req:85, rej:0)
# TX rejected (too early): 0.00% (req:85, rej:0)
### [JIT] ###
[jit] queue contains 1 packets:
[jit] queue contains 0 beacons:
 - node[0]: count_us=1544297558 - type=0
### [GPS] ###
# GPS sync is disabled
##### END #####
[3811563568] lorapf: WARN_ [up  ] ignored out-of sync PUSH_ACK packet buff_ack[5:212] != token[5:213]
[3811563623] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 1

I also kept on receiving the below logs on my pygate after my pysense went to deepsleep.

[3811653844] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 0
[3811653894] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 1
[3811683761] lorapf: INFO_ [main] report
##### 2020-08-12 15:02:30 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 0
# CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
# RF packets forwarded: 0 (0 bytes)
# PUSH_DATA datagrams sent: 1 (111 bytes)
# PUSH_DATA acknowledged: 0.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 3 (100.00% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
# TX rejected (collision packet): 0.00% (req:85, rej:0)
# TX rejected (collision beacon): 0.00% (req:85, rej:0)
# TX rejected (too late): 0.00% (req:85, rej:0)
# TX rejected (too early): 0.00% (req:85, rej:0)
### [JIT] ###
[jit] queue contains 1 packets:
[jit] queue contains 0 beacons:
 - node[0]: count_us=1544297558 - type=0
### [GPS] ###
# GPS sync is disabled
##### END #####
[3811683851] lorapf: WARN_ [up  ] ignored out-of sync PUSH_ACK packet buff_ack[5:216] != token[5:217]
[3811683901] lorapf: WARN_ [up  ] PUSH_ACK recieve timeout 1

[Gijs]

Hi,
It seems your pygate config did not upload correctly, can you reupload that? (In the background we are working on adding pygate config files for all regions but we struggle in the testing procedure..)
Next, is there any notice in the pygate logs? It seems the activation works fine, but your data packets are not coming through somehow. it would be interesting to see if they arrive at the pygate?

For your pysense node, I suggest you use a blocking socket to send data, and a nonblocking socket to receive, like so:

# make the socket blocking
# (waits for the data to be sent and for the 2 receive windows to expire)
s.setblocking(True)

# send some data
s.send(bytes([0x01, 0x02, 0x03]))

# make the socket non-blocking
# (because if there's no data received it will block forever...)
s.setblocking(False)

# get any data received (if any...)
data = s.recv(64)
print(data)