Does pulses_get function have a 200 bit limitation when reading?

[robert-hh]

@jcaron The test code used for that was;
LoPy4 test code:

from machine import UART
from time import ticks_us, ticks_diff, sleep_ms
msg = "cccccccc\n"
def run():
    uart=UART(1, baudrate=1200, pins=("P3", None))
    sleep_ms(5000)
    for _ in range (3):
        uart.write(msg)
        print(uart.wait_tx_done(int((len(msg) * 8.333)) + 10))
        uart.init(baudrate=1200, pins=(None, "P3"), timeout_chars=20)
        print(uart.readline())
        uart.init(baudrate=1200, pins=("P3", None))

run()

PC test code (Lopy4 connected to PC with a USB/Serial adapter, both RX and TX of that adapter connected to LoPy P3:

import serial

ser = serial.Serial("/dev/ttyUSB2", 1200)

while (True):
    while True:
        line = ser.readline()
        print(line)
        if (b"ccc" in line):
            break
    print("Send Echo")
    ser.write(b"Response\n")

Edit: That test code works fine. Trimming is required for the wait time after sending the command.
Edit2: Actually it's not that difficult. The specified wait time just has to be long enough, because the call returns as soon as the message has been sent (or after the wait time, if that is earlier).

[jcaron]

@robert-hh sorry, I meant using the same pin as both TXD and RXD when configuring the UART. Probably not allowed by the internal mux, but you never know.

[robert-hh]

@jcaron I just tried that. Even if you get no error message, no data is sent.

[Pesien]

@robert-hh I got it working with time.sleep_us, but I have a new problem. When I read the sensor 6-7 times the timers stop working properly, and the more I read it, the worse it gets. It feels like the timers are not reseting when the board does.
However, manually reseting the board or forcing at the end of the code a power reset fixes it.
Here is the code, and I'll really appreciate any help you can give me about this problem.
(The code works reading the pin every 833 us, which is the duration for 1 bit, and storing the value in a list to translate it later. The timer at the end is not 833 because I had to take into acount the time the chip needs to run the code before it.)

        self.pin_unico=Pin(Pin_SDI12, mode=Pin.IN, pull=Pin.PULL_DOWN)
        self.lista_bits=[]
        self.lista_bytes=[]
        while True:
            if self.pin_unico()==1:
                break
        self.lista_bits.append(1)
        time.sleep_us(833)

        while True:
            self.lista_bits.append(self.pin_unico())

            if len(self.lista_bits)==10:
                if self.lista_bits==[0,0,0,0,0,0,0,0,0,0]:
                    #self.lista_bytes.append(self.lista_bits)
                    self.lista_bits=[]
                    break
                else:
                    self.lista_bytes.append(self.lista_bits)
                    self.lista_bits=[]
            time.sleep_us(777)

The correct answer for the identification command for this sensor is:
013DECAGON MPS-6 386889208069\r\n
(I get this the first 6-7 times)
Then I get this results:
013DECAGON MPS-6 38688920807FSFENQ (7th time)
013DECAGON MPS-6 3868[FSYCAN[CANESCFSFENQ (8th time)
013DECAGONP&()SYNESCPEM[ESC[[NlLnLMNcB (9th time)

And so on. As you can see the reading is incorrect because there is some kind of problem with the timers, but the only way I found to solve this was to power reset the board.

[robert-hh]

@Pesien Long long ago i made a UART read by pit probing. As far as I recall, the software:

• waited for the start bit transition, and then
• probed the value in the middle of the following bit frames.

I see no reason why your approach would deviate over time. Expept that the time for append increases. So for the 7 bits of a byte I would preallocate a bytearray of 7 bytes, or just shift that into a single int value. Something like:

        EOL = const(0x0a) # end of line
        self.lista_bytes=b''
        
        while True:
            bits = 0
            while True: # wait for the start bit
                if self.pin_unico()==1:
                    break
            time.sleep_us(1200) # wait for a 1.5 bit times
        
            for bit in range (8): # get the next 8 bits
                bits = (bits << 1) | (1 - self.pin_unico())
                time.sleep_us(800) # to be determined, about a bit time
            bits >>= 1
            self.lista_bytes += bits.to_bytes(1, "little")
            if bits == EOL: # maybe \n or 0x0a
                break
# now msg contains a list of bytes with the response

Edit: If the lowest order bit is sent fist, then the inner loop may be:

            for bit in range (8): # get the next 8 bits
                bits = (bits >> 1) | ((1 - (self.pin_unico()) << 8)
                time.sleep_us(800) # to be determined, about a bit time

and the shift after the loop can be omitted.

[Pesien]

@robert-hh Thank you. I'll try to preallocate an array of bytes before reading and the 1.5 initial bit time and will update with the results.

[robert-hh]

@Pesien Yesterday evening I though I coudl try if my test function works and indeed got it working for 1200 baud, 7 data bits. What cause me some time to overcome was a glitch in the firmware, which I will analyze later: you cannot have reliable sleep_us(x) with x > 1000, means, sleep_us() longer than 1 ms. The resulting sleep time is sometimes much shorter. The code below uses a low for mark, when reading form the pin. If you want to adapt it, you have to invert the level. At the moment it also generates reference pulses at P11 to see, if the read timing matches the data stream.
In the target, you would pre-allocate buffer.
Still, using the built-in UARt class with the method discussed below seems more robust to me.

from time import sleep_us, sleep_ms
import gc
from machine import Pin, disable_irq, enable_irq
t = Pin("P11", Pin.OUT)
t(1)
pin = Pin("P9", Pin.IN)


def read_ser(pin_unico):
    EOL = const(0x0d)  # maybe \n or \r
    buffer = bytearray(100)
    buffer_size = len(buffer)
    bytes = 0
    gc.collect()

    while True:
        bits = 0
        while True:  # wait for the start bit
            if pin_unico() == 0:
                break
        sleep_us(390)  # wait for a 1.5 bit times
        sleep_us(390)  # since there is a bug for wait time > 1 ms
        sleep_us(390)  # three shorter sleeps are used.

        for bit in range(7):  # get the next 7 bits
            bits = (bits >> 1) | (pin_unico() << 7)
            t(0)
            sleep_us(20)
            t(1)
            sleep_us(780)  # to be determined, about a bit time
        bits >>= 1
        bits &= 0x7f
        # print(hex(bits), chr(bits))
        buffer[bytes] = bits
        bytes += 1
        if bits == EOL or bytes >= buffer_size:
            return buffer[:bytes]
# now msg contains a list of bytes with the response

[Pesien]

@robert-hh I'm afraid I am not able to make your code work. The SDI-12 sensors have 10 bits/byte, and the 7 data bits are in inverted USASCII code. You also have the starting high bit, the parity bit and the (low) ending bit.
Trying to adapt your code to these settings has failed, probably because of my own inexperience and the fact that I don't know what some of the things there do, but it might be because of the inverted 7 data bits.
I'll keep trying.

[robert-hh]

@Pesien The 10 bits are 1 start, 7 data, 1 parity, 1 stop.
my code ignores the start bit (the initial delay skips it) and the parity & stop bit. The latter should always be ignored, to get more time to store the data. Adapting to the reverse polarity and parity should be like the code below. But I will try this evening with reverse polarity.

from time import sleep_us, sleep_ms
import gc
from machine import Pin, disable_irq, enable_irq
t = Pin("P11", Pin.OUT)
t(1)
pin = Pin("P9", Pin.IN)


def read_ser(pin_unico):
    EOL = const(0x0d)  # maybe \n or \r
    buffer = bytearray(100)
    buffer_size = len(buffer)
    bytes = 0
    gc.collect()

    while True:
        bits = 0
        while True:  # wait for the start bit
            if pin_unico() == 1:
                break
        sleep_us(390)  # wait for a 1.5 bit times
        sleep_us(390)  # since there is a bug for wait time > 1 ms
        sleep_us(390)  # three shorter sleeps are used.

        for bit in range(8):  # get the next 8 bits including parity
            bits = (bits >> 1) | ((1 - pin_unico()) << 8)
            t(0)
            sleep_us(20)
            t(1)
            sleep_us(780)  # to be determined, about a bit time
        bits >>= 1
        bits &= 0x7f # strip off parity
        # print(hex(bits), chr(bits))
        buffer[bytes] = bits
        bytes += 1
        if bits == EOL or bytes >= buffer_size:
            return buffer[:bytes]
# now msg contains a list of bytes with the response

[robert-hh]

@Pesien I wonder why you tell that the data is inverted. From the SDI12 spec I cannot tell that it is. But besides that, the spec is not very detailed about that topic, and I have no real unit for testing.

[Pesien]

@robert-hh I have seen it with an oscilloscope. The sensor returns the data as I said in my last post.
Edit: In order to translate the pulses into characters, I created my own dictionary with all the characters, so translation is not needed, only pulse storage.

[robert-hh]

@Pesien I found another spec which confirma that. It had a more detailed timing diagram. OK. That explains why you cannot use the standard UART. It would need an automatic bidirectional inverting driver.

[Pesien]

@robert-hh The thing is that I had it working in two different ways, but both have some kind of limitation. Using the pulses_get function was perfect and robust, but it had the 128 entry limit. The timers worked fine but only the first 6-7 times.
I also tried to adapt my own code with the timers to wait more time and to preallocate memory but failed. The timing solution seems the only one for me, if I can make it work...

[robert-hh]

@Pesien I'll checkl this night. I am confident that I g´can get the bit-bang version working,but it will always due to fail. If during receving there is an interrupt, the timing will be disturbed. Using a timer interrupt for the bit probing of a byte is also no reliable solution, since the PyCom devices have a largely varying ISR-response latency, reaching from 10 to 900 µs.

[robert-hh]

@Pesien Reading through the data sheet and API of the ESP32 I noticed, that the ESP32 UART can operate with inverted signals. This may not be supported by the Micropython API, but it should not be too difficult to add. I'll try that too. If that works, the UART can be used.

[Pesien]

@robert-hh My teacher has it working with the UART and two pins. Since my project is to create an UART I don't think I can use it anyways.

[robert-hh]

@Pesien With two pins and the UART, you can use external inverters. With just one pin, you have to switch the UART to inverted mode. But I understand your task.

[robert-hh]

@Pesien This is the code for the inverted data, which also reads and ignores the parity bit. It reads from P9 and still has the timing marks on P11. If you drop these, you have to add a few (~4) µs to the loop's sleep_us. It is working here on a LoPy4. It returns a bytearray. If you need a string, you can convert that with bytes(return_value).decode()

from time import sleep_us, sleep_ms
import gc
from machine import Pin
t = Pin("P11", Pin.OUT)
t(1)
pin = Pin("P9", Pin.IN)


def read_ser(pin_unico):
    EOL = const(0x0d)  # maybe \n or \r
    buffer = bytearray(100)
    buffer_size = len(buffer)
    bytes = 0
    gc.collect()

    while True:
        bits = 0
        while True:  # wait for the start bit
            if pin_unico() == 1:
                break
        sleep_us(390)  # wait for a 1.5 bit times
        sleep_us(390)  # since there is a bug for wait time > 1 ms
        sleep_us(390)  # three shorter sleeps are used.

        for bit in range(8):  # get the next 8 bits
            t(0) # sync-marker
            bits = (bits >> 1) | ((1 - pin_unico()) << 7)
            t(1)
            sleep_us(810)  # to be determined, about a bit time
        bits &= 0x7f  # mask 7 bit
        buffer[bytes] = bits
        # print(hex(bits), chr(bits))
        bytes += 1
        if bits == EOL or bytes >= buffer_size:
            return buffer[:bytes]
# now buffer contains a list of bytes with the response

I called it a few times in a row with supplying a ~100 charactes line, and that is the result. Obviously, if you feed lines too fast, it gets out of sync:

bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')

Below is a logic analyser snap shot of a read, which shows, that the read markers match the bit location centers.

[Pesien]

@robert-hh Thank you again for all your help. I'll try it tomorrow and see if it works finally.

[robert-hh]

@Pesien Just for interest I tried to set up an interrupt based version where:

• the start bit would trigger a pin interrupt callback, which then
• fires a series of n timer callbacks to read the bits of each byte.

Even if the software itself worked, that attempt failed because:

• the callback latency of the pin interrupt is greatly varying between 100µs and ~1000µs, making it impossible to sync on the first bit.
• the shortest period of a sequence of timer interrupts is about 2 ms. 0.883 ms would be needed for 1200 baud.

Even if the timer based bit probing is replaced by the active waiting method, the initial pin interrupt latency prevents reliable reading.