uart.readline() not working reliably for long strings?

SciWax

Hello everybody,

I want to send a JSON-formatted string with measurement data from my Arduino to my GPy, so that the GPy sends the data to my database. It works for a while, but then sometimes uart.readline() doesn't read the full string and I get only a tiny amount of the String the Arduino sends. I would be open for other suggestions, if people recommend me to not use uart.readline(). I have no idea, if it's just unreliable.

The code:

import os

import pycom
import machine
import time
import urequests
from network import WLAN  # https://docs.micropython.org/en/latest/library/network.WLAN.html
wlan = WLAN(mode=WLAN.STA)
# Configure first UART bus to see the communication on the pc
uart = machine.UART(0, 115200)

# Configure second UART bus on pins P3(TX1) and P4(RX1)
uart1 = UART(1, baudrate=115200) # 9600 baudrate also causes the same problem
# Init WLAN
try:
  wlan.connect(ssid="your_ssid", auth=(WLAN.WPA2, "your_password"))
  while not wlan.isconnected():
    time.sleep_ms(50)
    print(wlan.ifconfig())

  print("WLAN connection on ...")
except:
  print("[WLAN] try() (except). reset board.")


while True:
  dataRec = uart1.readline()
  time.sleep(5)
  if dataRec is not None:
    dataRec = dataRec.decode("utf-8")[:-2] # to remove /r/n
    print("")
    print(dataRec)
    dataRec = eval(dataRec)
    res = urequests.post("https://website.com/data.php", json=dataRec) 
    res.close()

The output till it crashes (The first segment shows an example of a correct JSON formatted string, which is send from the Arduino and is received by the GPy via Serial. I've printed it out for demonstration purposes):

{"Id":131,"Y":2192,"M":12,"D":31,"Hr":14,"Mi":59,"B1":4.2,"B2":4.3,"T1":13.33,"H1":13.33,"T2":27.36,"H2":44.45,"O1":100.00,"O1g":200.25,"O1t":27.00,"O1h":56.00,"N1":111.00,"N1g":222.25,"N1t":27.00,"N1h":56.00,"C1":10000.00,"C1g":20.25,"C1t":27.00,"C1h":56.00,"O2":24300,"N2":26000,"C2":21000,"C3":22000,"P1":100.11,"P2":130.11,"P4":160.11,"P10":200.11}

{"Id":131,"Y":2196,"M":12,"D":31,"Hr":14,"Mi":59,"B1":4.2,"B2":4.3,"T1":13.33,"H1":13.33,"T2":27.36,"H2":44.45,"O1":10
Traceback (most recent call last):
  File "<stdin>", line 38, in <module>
  File "<string>", line 2
SyntaxError: invalid syntax

eval() can't convert the received string into a dictionary object, so that's why the error happens. It makes sense, because not the whole string was received and therefore, the string hasn't the right format for the conversion:

[https://www.tutorialspoint.com/How-to-convert-a-String-representation-of-a-Dictionary-to-a-dictionary-in-Python](link url)

I would be glad about any help or suggestions.

Kind regards

SciWax

robert-hh

@robert-hh with uart.readline() you have to be sure, that the receive buffer is sufficiently large. You can increase that when creating the UART object with the parameter rx_buffer_size=nnn. Since the sample JSON object of your post is 352 bytes, you could try to use a buffer size of 1024 or even larger, which will cope with delays during forwarding the content.
Edit: The default size is 512.

SciWax

@robert-hh

I added your suggestion to my code. Thanks!

Do you think this would help with uart.readline()? Last night, I've tested my old script with uart.readline() for 4 hours reading a string sent by my arduino every 10 seconds via a serial connection between my arduino and GPy. 10 times in 4 hours, uart.readline() didn't read the full json-formatted string from the arduino completely.

robert-hh

@SciWax If you set the timeout to 1 second and call read(1), if the uart buffer has at least one character, then it will return immediately and not wait for a timeout.

SciWax

@robert-hh

Thank you for that information. Do you would recommend to me using that? I have alot of characters so in the worst case, it could take ages if anything goes wrong. For instance reading byte by byte, with 200 bytes to read and a timeout of 1 second, in the worst case maybe I would have to wait 200 seconds. Please correct me, if I'm wrong.

robert-hh

@SciWax the UART object can also be created with a timeout_chars=x parameter. In that case it will wait at least x character times for a value to arrive. The default is 0, which means: do not wait.
see: https://docs.pycom.io/firmwareapi/pycom/machine/uart/#app

SciWax

I think I've got it now:

def serialrecv(uartport):
  #read the data, byte by byte and decode it
  length_str = ""
  char = uart1.read(1)
  if char is not None:
    while char is not None:
      print("inside the while loop")
      length_str += char.decode("utf-8")
      char = uart1.read(1)
  else:
    return print("Waiting for datastring") 
  return length_str

Didn't throw any errors so far. I will test my new code and will give feedback about my experiences with the change.

SciWax

I think I'm making some progress:

def serialrecv(uartport):
  #read the length of the data, letter by letter until we reach EOL
  length_str = ""
  char = uart1.read(1)
  if char is not None:
    while char.decode() != "\r":
      print("inside the while loop")
      length_str += char.decode("utf-8")
      char = uart1.read(1)
  else:
    return print("Waiting for datastring") 
  return length_str

while True:
  dataRec = serialrecv(uart1)
  print(dataRec)

The output:

None
Waiting for datastring
None
Waiting for datastring
None
inside the while loop
inside the while loop
inside the while loop
.....
inside the while loop
inside the while loop
inside the while loop
{"Id":131,"Y":2148,"M":12,"D":31,"Hr":14,"Mi":59,"B1":4.2,"B2":4.3,"T1":13.33,"H1":13.33,"T2":27.36,"H2":44.45,"O1":100.00,"O1g":200.25,"O1t":27.00,"O1h":56.00,"N1":111.00,"N1g":222.25,"N1t":27.00,"N1h":56.00,"C1":10000.00,"C1g":20.25,"C1t":27.00,"C1h":56.00,"O2":24300,"N2":26000,"C2":21000,"C3":22000,"P1":100.11,"P2":130.11,"P4":160.11,"P10":200.11}
inside the while loop
Traceback (most recent call last):
  File "<stdin>", line 49, in <module>
  File "<stdin>", line 32, in serialrecv
AttributeError: 'NoneType' object has no attribute 'decode'

I don't really get the error. Something should only be decoded, if a char is not None. How does a char with None gets into my while loop?

SciWax

@robert-hh

thanks for your suggestions. I have a hard time at the moment figuring out how to make the testing work, when I have to test for single bytes. For the code in my first post, it's pretty easy.

Basically at the end, the GPy is only there for listening to the Arduino and sending the data. I kinda have to tell the GPy, if uart.read(1) is None, jump back to the beginning of the function till uart.read(1) finally reads something, which isn't None.

I've tried to realize it like that, but no success:

def serialrecv(uartport):
  #read the length of the data, letter by letter until we reach EOL
  length_str = ''
  char = uart.read(1)
  if char is not None:
    while char.decode() != '\r':
      length_str += char.decode("utf-8")
      char = uartport.read(1)
  else:
    return print("Waiting for datastring") 
  return length_str

The output for this is this:

None
Waiting for datastring
None
Waiting for datastring
None
Waiting for datastring
None

robert-hh

@SciWax UART read is not blocking. It returns immediately, and if there is nothing to read, it returns None. So you have to test for the retirn value, or call read only if uart.any returns, zhst sufficiently many characters are present.

SciWax

If I try the other method from the user vruizvil, I run into the problem, that the function wants to decode chars of the NoneType class, which doesn't work:

Traceback (most recent call last):
  File "<stdin>", line 44, in <module>
  File "<stdin>", line 29, in serialrecv
AttributeError: 'NoneType' object has no attribute 'decode'

I've changed

length_str += char.decode()

to

length_str += char.decode("utf-8")

def serialrecv(uartport):
  #read the length of the data, letter by letter until we reach EOL
  length_str = ''
  char = uartport.read(1)
  while char.decode() != '\r':
    length_str += char.decode("utf-8")
    char = uartport.read(1) 
  return length_str

Can I put an identifier somewhere, so that the function doesn't decode chars, which are belong to the NoneType class. Maybe with an if statement?

The string I'm interested in starts with {. Can I use that somehow to start from there decoding my chars?

SciWax

There is another solution by the user vruizvil:

https://forum.pycom.io/topic/882/arduino-sends-data-seriell-how-to-get-them-into-wipy-solved/6?_=1596645777419

My question about this answer is, if there will be any issue with Heap-Fragmentation, because I'm using a string variable and sending data every minute?

Originally I've planned to nest everything like that, so I don't use a string variable in my code:

res = urequests.post("https://website.com/data.php", json=eval(dataRec.decode("utf-8")[:-2]))

Explore Pybytes | Official Documentation | Report a Firmware Bug/Issue | GitHub

uart.readline() not working reliably for long strings?

Pycom on Twitter