Does pulses_get function have a 200 bit limitation when reading?

[Pesien]

Hello, I'm a new programmer and I'm working on a library to comunicate with SDI-12 sensors using a GPIO pin in a WiPy 3.

In order to read the pulses I get from the sensor I use the pulses_get function, but it seems to have a limitation on 200 bits even though it is not mentioned in the documentation.

After sending the bits ,the code for reading is:

pin=Pin('P11', mode=Pin.IN)
pulses=pycom.pulses_get(pin, 10000)

And the output I get:
[(1, 4146), (0, 1659), (1, 1656), (0, 835), (1, 827), (0, 830), (1, 2487), (0, 1661), (1, 828), (0, 1663), (1, 829), (0, 1658), (1, 1657), (0, 1659), (1, 1657), (0, 833), (1, 2486), (0, 831), (1, 2487), (0, 828), (1, 828), (0, 834), (1, 828), (0, 830), (1, 828), (0, 830), (1, 2487), (0, 2494), (1, 828), (0, 1661), (1, 3315), (0, 2494), (1, 829), (0, 830), (1, 4146), (0, 830), (1, 828), (0, 834), (1, 828), (0, 2490), (1, 2488), (0, 830), (1, 829), (0, 833), (1, 830), (0, 3318), (1, 1658), (0, 2492), (1, 1658), (0, 2489), (1, 1657), (0, 830), (1, 828), (0, 834), (1, 4974), (0, 831), (1, 829), (0, 1662), (1, 829), (0, 829), (1, 828), (0, 1660), (1, 1657), (0, 830), (1, 828), (0, 835), (1, 4146), (0, 829), (1, 829), (0, 831), (1, 828), (0, 834), (1, 828), (0, 1659), (1, 1657), (0, 830), (1, 829), (0, 830), (1, 828), (0, 834), (1, 828), (0, 830), (1, 828), (0, 1659), (1, 829), (0, 830), (1, 1657), (0, 834), (1, 1657), (0, 1659), (1, 828), (0, 1659), (1, 1657), (0, 834), (1, 4975), (0, 829), (1, 829), (0, 1663), (1, 828), (0, 1659), (1, 1658), (0, 1658), (1, 1658), (0,834), (1, 3316), (0, 2489), (1, 829), (0, 1663), (1, 1657), (0, 1659), (1, 829), (0, 1658), (1, 1658), (0, 834), (1, 3317), (0, 2489), (1, 828), (0, 1663), (1, 3317), (0, 2489), (1, 829), (0, 1663), (1, 830), (0, 829), (1, 1658), (0, 2489), (1, 1656), (0, 834)]

Translated it is:
013DECAGON MPS-6 386889
I know the reading is incomplete because it should end with \r\n

I tried it with multiple sensors and commands but I always get the same bit limit.
Does pulses_get have a limit? If it has, why is it not mentioned in the documentation?

[robert-hh]

@Pesien @mfalkvidd
I mage a PR for inverted UART signals, based on the release candidate version. The ony changes file is mods/moduart.c, so it's easy to use it for the master branch too:
https://github.com/pycom/pycom-micropython-sigfox/pull/310

[robert-hh]

@Pesien Congratulations. I did not try yet to set the UART in inverted polarity mode. Having already located the spots to change, I'll try that next week. Even if it is not needed for your project, it may be useful.

[Pesien]

@robert-hh FINALLY I managed to make it work. I'll upload the full library in a few weeks here in the forums, because I know there are a lot of people that will like to use it. It comunicates with SDI12 sensors using only one GPIO pin for sending and receiving data.

If it wasn't for your help, I wouldn't have been able to make it. I really appreciate the effort you made to help me and I'm really grateful for it.
Thank you so much.

[robert-hh]

@Pesien Yes. You should get some testing equipment to tell, whther the signal level arer those that you expect, and if, that you can determine in your code the high and low levels. For such kind of analysis, an oscilloscope works better than a logic anayzer, because it show also possible troubles with analog levels and transition times.

[Pesien]

@robert-hh Even if the timing is not correct, it should detect any of the following possitive bits, but it doesn't. And with the code you made, that while doesn't work even for the first bit, so I think that there is something else happening that I don't see.

[robert-hh]

@Pesien said in Does pulses_get function have a 200 bit limitation when reading?:

The other thing is about how pythone deals with objects like a list. The statement self.lista_bytes[y]=self.lista_bits does not copy the list, but just copies a reference. If you change lista_bits later on, the entry in lista_bytes will be changed too. At the end, all entries in lista_bytes will be the same. So instead of self.lista_bytes[y]=self.lista_bits, you must write:
self.lista_bytes[y]=self.lista_bits[:], which will create a copy. In any case self.lista_bits=[0,0,0,0,0,0,0,0,0,0] is not needed.

[robert-hh]

@Pesien The timing is essential. You have to wait for the next byte in the time frame between the last bit of the previous byte and the start bit of the next byte. That time frame may be as short a one bit duration (883 µs). If you miss the transision of the start bit, all timing is disturbed for a byte and all subsequent bytes. The code you have at the end of a byte seems to be too slow.
Yind you have an oscilloscope, you can try to monitor the state of your code in relation to the incoming data, liek I did with the monitor pulses. A sequence debug_pin(1);debug_pin(0) takes about 5 µs and will leave a visible time marking in the oscilloscope. Without such analysis capabilities it's hard to get along. I'm using a check logic analyzer tool for that, like this one: https://www.amazon.com/HiLetgo-Analyzer-Ferrite-Channel-Arduino/dp/B077LSG5P2/ref=sr_1_3?crid=3S1O51OGJW1Q2&keywords=logic+analyzer+usb&qid=1558955763&s=gateway&sprefix=logic+anal%2Caps%2C242&sr=8-3, which works nice together with the saleae logic analyzer software.

[Pesien]

@robert-hh Finally I got some time to work on this. I tried your solution, but for some reason it simply does not work.
However, I managed to get it working with the next code:

def read(self):

        
        self.pin_unico=Pin(Pin_SDI12, mode=Pin.IN, pull=Pin.PULL_DOWN)
        self.lista_bits=[0,0,0,0,0,0,0,0,0,0]
        self.lista_bytes=[None]*50
        y=0



        while True:
            while True:
                if self.pin_unico()==1:
                    break
            time.sleep_us(390)
            time.sleep_us(390)
            time.sleep_us(390)
            self.lista_bits[0]=1


            for x in range (0, 8):
                self.lista_bits[x+1] = self.pin_unico()
                time.sleep_us(814)


            self.lista_bits[9]=0
            #print(self.lista_bits)

            if self.lista_bits==[1,1,0,1,0,1,1,1,1,0]:
                self.lista_bytes[y]=self.lista_bits
                self.lista_bits=[0,0,0,0,0,0,0,0,0,0]
                break
            else:
                self.lista_bytes[y]=self.lista_bits
                self.lista_bits=[0,0,0,0,0,0,0,0,0,0]
                y+=1
                #print(self.lista_bytes)

        #print(self.lista_bytes)
        
        return(self.lista_bytes)

The thing is that it only works the first time (first byte) and it gets stucked here the second time:

while True:
    if self.pin_unico()==1:
        break

I don't know why this is happening. It makes no sense to me.
Edit: I just tried it and your code has the same issue but I cant make it work even the first time.

[robert-hh]

@Pesien I tried this interrupt mechanism with the microypthon.org branch of MicroPython. That works better. Only the first character in a message is disturbed after a longer (few seconds) break between messages. This glitch is caused by the timer burst delayed by ~500µs. The reason for that should be the relevant code not being in the memory cache and having to be imported from serial flash. This code page fetch takes at least 300µs.

[robert-hh]

@Pesien Just for interest I tried to set up an interrupt based version where:

• the start bit would trigger a pin interrupt callback, which then
• fires a series of n timer callbacks to read the bits of each byte.

Even if the software itself worked, that attempt failed because:

• the callback latency of the pin interrupt is greatly varying between 100µs and ~1000µs, making it impossible to sync on the first bit.
• the shortest period of a sequence of timer interrupts is about 2 ms. 0.883 ms would be needed for 1200 baud.

Even if the timer based bit probing is replaced by the active waiting method, the initial pin interrupt latency prevents reliable reading.

[Pesien]

@robert-hh Thank you again for all your help. I'll try it tomorrow and see if it works finally.

[robert-hh]

@Pesien This is the code for the inverted data, which also reads and ignores the parity bit. It reads from P9 and still has the timing marks on P11. If you drop these, you have to add a few (~4) µs to the loop's sleep_us. It is working here on a LoPy4. It returns a bytearray. If you need a string, you can convert that with bytes(return_value).decode()

from time import sleep_us, sleep_ms
import gc
from machine import Pin
t = Pin("P11", Pin.OUT)
t(1)
pin = Pin("P9", Pin.IN)


def read_ser(pin_unico):
    EOL = const(0x0d)  # maybe \n or \r
    buffer = bytearray(100)
    buffer_size = len(buffer)
    bytes = 0
    gc.collect()

    while True:
        bits = 0
        while True:  # wait for the start bit
            if pin_unico() == 1:
                break
        sleep_us(390)  # wait for a 1.5 bit times
        sleep_us(390)  # since there is a bug for wait time > 1 ms
        sleep_us(390)  # three shorter sleeps are used.

        for bit in range(8):  # get the next 8 bits
            t(0) # sync-marker
            bits = (bits >> 1) | ((1 - pin_unico()) << 7)
            t(1)
            sleep_us(810)  # to be determined, about a bit time
        bits &= 0x7f  # mask 7 bit
        buffer[bytes] = bits
        # print(hex(bits), chr(bits))
        bytes += 1
        if bits == EOL or bytes >= buffer_size:
            return buffer[:bytes]
# now buffer contains a list of bytes with the response

I called it a few times in a row with supplying a ~100 charactes line, and that is the result. Obviously, if you feed lines too fast, it gets out of sync:

bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')
bytearray(b'Far away behind the word mountains, far from Vokalia and Consonantia, there live the blind texts.\r')

Below is a logic analyser snap shot of a read, which shows, that the read markers match the bit location centers.

[robert-hh]

@Pesien With two pins and the UART, you can use external inverters. With just one pin, you have to switch the UART to inverted mode. But I understand your task.

[Pesien]

@robert-hh My teacher has it working with the UART and two pins. Since my project is to create an UART I don't think I can use it anyways.

[robert-hh]

@Pesien Reading through the data sheet and API of the ESP32 I noticed, that the ESP32 UART can operate with inverted signals. This may not be supported by the Micropython API, but it should not be too difficult to add. I'll try that too. If that works, the UART can be used.

[robert-hh]

@Pesien I'll checkl this night. I am confident that I g´can get the bit-bang version working,but it will always due to fail. If during receving there is an interrupt, the timing will be disturbed. Using a timer interrupt for the bit probing of a byte is also no reliable solution, since the PyCom devices have a largely varying ISR-response latency, reaching from 10 to 900 µs.

[Pesien]

@robert-hh The thing is that I had it working in two different ways, but both have some kind of limitation. Using the pulses_get function was perfect and robust, but it had the 128 entry limit. The timers worked fine but only the first 6-7 times.
I also tried to adapt my own code with the timers to wait more time and to preallocate memory but failed. The timing solution seems the only one for me, if I can make it work...

[robert-hh]

@Pesien I found another spec which confirma that. It had a more detailed timing diagram. OK. That explains why you cannot use the standard UART. It would need an automatic bidirectional inverting driver.

[Pesien]

@robert-hh I have seen it with an oscilloscope. The sensor returns the data as I said in my last post.
Edit: In order to translate the pulses into characters, I created my own dictionary with all the characters, so translation is not needed, only pulse storage.