Despite heavy investment in FiPy/GPy, not possible to use board as anything more than LTE modem (and even that's problematic)



  • As a software developer who runs my own support forum, I am well aware of the injustice of ignorant people making false attacks against a digital product. I promise I am not guilty of that, and this post is 100% valid and very important.

    My mind is boggled as to why this isn't being talked about.

    Cut to the chase:

    Critical Problem 1

    1. lte = LTE() often/sometimes causes irreversible hang after calling deinit() (currently happening to a GPy whereas it seemed to not for a while with a FiPy)
    2. Cannot use deepsleep mode satisfactorily unless you deinit() the LTE modem
    3. Power consumption is absolutely unacceptable with these boards unless you can do this, as I only upload every 10/20 mins
      4. Therefore, all FiPy/GPy boards are utterly useless for embedded purposes where I will be deploying a remote surveillance tower to upload sensor data 24/7 - until the LTE issues are fixed

    Before you say it:

    1. Yes, I updated to the most recent stable firmware on the Fipy/Gpy
    2. Yes, I updated the Sequans modem to the July 10th firmware upgrade (good thing I backed that up, as the new frustrating website link changes have utterly removed this link on Pycom support. Everything is 404 now.)

    As I write this post, I have a GPy flashing heartbeat currently stuck on the second line:

    writelog("System booting: "+str(time.ticks_ms()));
    lte = LTE(carrier=MyLTECarrier)         #it's been minutes and the board is stuck here!
    writelog('LTE created')
    

    I have acquired an Adafruit ItsyBitsy M4 Express controller - fractional power consumption - to run as my 24/7 reliable, dependable, master MCU instead (it takes only 10mA). I need to at least be able use my GPy/FiPy to upload over LTE, communicating with M4 board over UART.

    At this rate, I am not going to be able to use the deepsleep mode (irreversible LTE hang persists until you hard-power-cycle the board, deepsleep DOESN'T reset it, nor does the reset button - only unplug USB/plug-in). Instead, I am literally going to need to regulate the GPy/FiPy power supply with a transistor connected to my ItsyBitsy M4 pin, and manually power on/off the device, because of the issue above.

    Sound good? Well, another mission-critical disastrous occurrence happened for me during testing:

    Critical problem #2

    1. I was testing my FiPy with solar charger installation
    2. I came to it one day and discovered the program (main.py and boot.py) had been completely reset - no explanation whatsoever
    3. I posted on this forum and a helpful guy told me it was likely from power down while writing to files. Conclusion: Arduinos don't do this; the possibility of losing your entire program if an unexpected power-down should occur is unacceptable. I will have to drive 3 hours to my remote tower just to reprogram the board with my code should this happen! Until this is fixed, all Pycom products are not suitable for embedded usage.

    Do any of you vets have some tips for me? The Pycom boards have so much potential and they are so promising. But right now, the only way I can get consistent LTE 24/7 upload with them is to leave LTE connected and attached in perpetuum, and do hard machine.reset()s when connection finally drops, which uses 0.25A/250ma constantly and is absolutely unacceptable.

    I have purchased probably around 10 of these boards.

    Please help. Thank you in advance.



  • @aaronkondziela How do I set the LTE disabled on boot flag? Now when the board powers ON, I need to call LTE init() and deinit() to reduce the power consumption.

    BTW, I am also having the LTE hang issue: https://forum.pycom.io/topic/3129/lte-lte-getting-stuck-after-reset-fw-1-17-3-b1-on-fipy/4

    It will be very helpful if you could tell me how to set the LTE disabled on boot flag. Thanks in advance.



  • @aaronkondziela Are you on the July 10th modem firmware or the newest one? If the newest one, your report may suggest the lte = LTE() infinite zombie state is an ongoing, unsolved issue in the new firmware. But I myself still must test this.



  • @dan @Fred Thank you for the response. I will do another round of profiling/exact reproduction steps with the brand new firmware. I did deploy my station yesterday which uses GPy just as LTE modem (power on every 10 mins for 1 min max), and it is coded to do a hard machine.reset() suicide if execution goes beyond 50 seconds (i.e., the attach fails by going on ad infinitum, even when there's perfect signal). I am pretty sure the machine.reset() multithreaded suicide will properly reset the LTE modem - but time will tell. Currently uploading no-problems for 36 hours. This was with the new firmware so I will get back to you and see if there is a definitive improvement with the new firware, testing on one of my other GPys. Might take a few days until I can get to this, but thank you for recognizing the importance!



  • I'm also seeing this issue. And, almost universally, a second call to lte=LTE() or lte.deinit() will lock the board up.

    Possibly related, calling machine.deepsleep(n) works once only; the second time around it locks the board up. I mention it here since I know there is some relation between LTE.deinit and deepsleep, so it could be related. The deepsleep problem happens regardless of LTE calls. Note, I have lte disabled on boot flag set, and I slapped an old, non-active SIM in because I saw other users mention that LTE gets grumpy and slow with no SIM installed, even if not in use. This also happened without the SIM.

    FiPy board, and Pysense board, both with firmware updated about seven days ago.


  • administrators

    Hi @PaulM and @RWall

    Could you upgrade to the latest stable firmware and run import sqnsupgrade and sqnsupgrade.version() after resetting your module in safe-boot mode (CTRL-F)? Also, after using any of the sqnsupgrade commands it is necessary to reset the board to use the LTE class again.


  • administrators

    @paulm Thank you for giving us a good overview of the issue you are experiencing. I have asked one of our cellular experts to replicate the issue. Can you email me on fred@pycom.io so we put forward a resolution.


  • administrators

    Hi @PaulM and @RWall

    I'm sorry to hear that you're experiencing this issue. We'll look into it. Thank you for your patience!



  • @paulm- I too am surprised this is not getting more traction. Maybe you could edit your original posts and add more tags (modem, LTE M1, LTE Roaming, Cat M1, ...) than just the one.

    in answer to your specific question- we are doing a couple things.
    - We are looking at a different solution from Link Labs. Mostly waiting for management to get on board for the switch.
    - While we wait on management, I am taking a much deeper dive into the modem command set and may end up bypassing various Pycom software modules. The modem has more AT commands than are documented.

    I don't have a definitive solution to share at the moment, but I am slowly gaining more insight into the black box.



  • @rwall Thanks for posting. Have you made any breakthroughs in this question?

    HOW is this not being talked about more. Earth to Pycom: can someone from the team PLEASE comment on the irreversible lte = LTE() hang? The whole point of the Pycom GPy/FiPy boards is to have consistent, reliable LTE connection - which is currently impossible.

    Pycom's flagship product is merely a theoretical abstraction until this crucial issue is fixed!



  • @PaulM - I too am experiencing a consistent and similar "irreversible hang" that has us reluctantly looking at a completely different hardware platform. A hard reset appears to be the only way to get the pycom cellular modem operating again which, as you pointed out, precludes field deployment.

    We are using a GPy/PyTrack based platform with ALL firmware updated, including the latest sequans modem update released a few days ago.

    Thanks for broaching the topic.



  • Further, in relation to Critical Issue #1,

    1. The end-of-universe lte = LTE() eternal hang does not just occur after lte.deinit()

    2. It occurs also after I reset/deepsleep wakeup (but not power cycle) after timing-out of a lte.attach() or lte.connect() that takes too long, which brings me to another problem:

    3. lte.attach() and lte.connect() randomly go into eternal non-completion at occasional times, when signal is just fine and identical to when they pick up connection in seconds

    4. So, in order to deal with this strange behavior, I need to timeout/reset when it happens. But then that also triggers the LTE zombie state where calling lte = LTE() after reset causes eternal hang.


 

Pycom on Twitter