UART Reading Problem

[Christian Storm]

Hey, I am using FiPy with DP Firmware, connected to a external device, reading data via UART1.
I do have a problem while reading data. I expect exactly 2881 bytes, parted by '\n' charater. But the uart connection gives me sometimes 2880 or 2879 or even 2865 etc... Sometime it is correct, and I get 2881. I am not sure, where the problem is. If I reading the external device with an FDTI USB TTL RS232 3v3 device, I always get 2881 bytes. So the external device sending data correct.

for UART Init I use:

uart.init(19200, bits=8, parity=None, stop=1, timeout_chars=1, pins=('P22','P23'), rx_buffer_size=4096)

for reading I use this:

def ReadlineSerial(self):
   rl = ""
   while (self.uart.any() > 0):
       tmp = self.uart.readline()
       _tmp = str( tmp.decode() ).rstrip()
       rl = rl + _tmp
       break #this stops while for sure, so only reading one LINE
   return rl


rcv = self.ReadlineSerial()
while (rcv != ""):
    _p = self.Parse(rcv)
    if not _p == False:
       parsed.append(_p)
    rcv = self.ReadlineSerial()
    time.sleep(0.1)

I also tried:

def ReadAllSerial(self):
    buffer = ""
    available = self.uart.any()
    while (available > 0):
        b = self.uart.read(available)
        buffer = buffer + str( b.decode() )
        time.sleep(0.1)
        available = self.uart.any()
    return buffer;

rcv = self.ReadAllSerial()
if (len(rcv) > 0):
    _ps = rcv.split('\n')
    for n in range(0, len(_ps)):
       _p = self.Parse(_ps[n])
       if not _p == False:
          parsed.append(_p)

What I am doing wrong? I rely on getting save 2881 bytes

Thanxx for feedback!

[jcaron]

@Christian-Storm what kind of data is it? ASCII text? UTF-8 text? Binary?

I’m not sure I understand what you are doing with decode and str (and of course rstrip)…

[robert-hh]

@Christian-Storm readline reads data up to a newline or up to a timeout, whatever occurs first. Try to increase the timeout.

[Christian Storm]

Hey @jcaron, data is ASCII, but uart read return bytes. Thats why I use decode.

[Christian Storm]

Hey @robert-hh, I did try increase timeout_chars up to 255. But than I miss more bytes. Even if I use uart.readline or uart.read

[Christian Storm]

For me it looks like, that these posts:
https://forum.pycom.io/topic/2595/uart-rx-buffer-issue-after-buffer-overflow
https://forum.pycom.io/topic/6228/uart-readline-not-working-reliably-for-long-strings
are still unsolved. Well, in the meantime I found an other solution

[robert-hh]

@Christian-Storm That's true. There seem to be no interest at Pycom to fix bugs, even if the fix is trivial. For a while I supplied PRs for bug fixes, and that worked to some extend. But no one seems to care about that since a while. It is also not clear what the actual 'master' branch of their firmware is, and whether the published one is the recent firmware. People at Pycom seem to change frequently, so there is no continuous building up and keeping of knowledge about the existing code base.

[SciWax]

@Christian-Storm

Hi Christian,
welcome to the forum. You linked a thread where I've asked a question about UART and how to parse certain strings. I actually made it work and it works for nearly 2 years now thanks to the help of @robert-hh, who helped me getting on the right track.

Actually I was sure, that I edited my first post in the thread with a solution, but doesn't seem like it.

Tomorrow, I will have access to my PC with a working solution. I will try posting a minimal example for you!

[Christian Storm]

Hi @robert-hh, that sounds not very good. Since I'm almost done with my project, about 95% achieved. The last meters are so important. I do need a reliable uart communication on few Bytes and many Bytes. I solved it with a kind of file buffer. So I save the "normal" data from my external UART device (btw without any option of flowcontrol) which have about 45bytes. And if I am requesting the whole data, so I get it from file buffer. Thanxx for your help and time

[Christian Storm]

Hi @SciWax, would be great to hear from you and your solution!

[SciWax]

@Christian-Storm

Hi, sorry for the late reply. My week was really turbulent.

In my case, I basically designed the format of the strings myself the Pycom board receives. I had more freedom, therefore I've created a start marker and end marker, which nicely surround my strings.

I've created json formatted strings, which begin with { as a start marker and end with } as an end marker. My reference is a tutorial in the official Arduino Forum

So basically I read my input strings like this embedded in a while loop:

        dataRec = serialrecv(uart1) # collect single bytes
        if dataRec is not None:
            if dataRec.startswith('{') and dataRec.endswith('}\r\n'):
                print(dataRec)

where dataRec is the string received and parsed by

# Configure UART on pins P3(TX1) and P4(RX1)
uart1 = UART(1, baudrate=9600, timeout_chars=10000) 

def serialrecv(uartport):
  #read the data, letter by letter until we reach EOL
  dataString = ""
  char = uart1.read(1)
  if char is not None:
      print("")
      print("Reading string from Serial:")
      while char is not None:
          dataString += char.decode("utf-8")
          char = uart1.read(1) 
  return dataString

[Christian Storm]

Hi @SciWax, thanks for your reply. I rely on the communication protocoll of the external device, which have only an end marker "\r\n". The normal dataflow, about 45 bytes are ok with readline and even with read by byte (uart.read(1)). The Problem occurs one data larger than the overflow. If I try to read more than 2000 Bytes at once, it makes no sence. In between the "received" data block there are missing data, and thats the mess.