Couldn't connect to Modem (modem_state=disconnected)

[kjm]

lte modem on a fipy ((1.20.2.rc9 with LR5.2.1.0-48829)) has gone AWOL. sqnsupgrade_info says "Cannot determine modem state!" Tried pycom.lte_modem_en_on_boot(1) as an act of desperation, didn't work.

I googled the problem but no obvious solution. Can someone please suggest a method to get the modem back? I presume the problem is the internal uart interface between the fipy & the modem?

[Gijs]

To clarify, this did not happen during the update process right?

Im assuming you already tried to disconnect power and reconnect it again, and that lte.factory_reset() is also not working for you. Could you try sqnsupgrade.info(debug=True) and let me know of the output

[kjm]

I finally got to the bottom of this. On this device with these firmwares the at+cfun=0 cmd (supposedly for switching the lte radio off) actually disconnects the lte modem uart. The only fix is to either stop using that particular cmd or recycle the power.

[Gijs]

@kjm Could you send an example code or some steps that we can take to reproduce this? Its not something that should happen I think, but I have no clear reproduction on what causes it

[kjm]

@Gijs This

import os, network, machine, time; begin=time.ticks_ms()
def _t(): t=(time.ticks_ms()-begin)//1000; now='0'+str(t) if t<10 else str(t); return now
def _end(reason): print(_t(), reason); machine.reset()

print(_t(), os.uname()[1], os.uname()[2])
print(_t(), 'lte'); lte=network.LTE()
print(_t(), 'attach', end=' '); lte.attach()
for i in range(9):
  time.sleep(1); print(i, end=' ')
  if lte.isattached(): print(); break
else: print(); _end('attachment timeout')
print(_t(), 'connect', end=' '); lte.connect()
for i in range(2):
  time.sleep(1); print(i, end=' ')
  if lte.isconnected(): print(); break
else: print(); _end('connect timeout')
print(_t(), 'switch off radio'); lte.pppsuspend(); lte.send_at_cmd('at+cfun=0')
_end('finish')

reliably sends the fipy modem awol here after the first run

00 FiPy 1.20.2.rc9
00 lte
03 attach 0 1 
10 connect 0 
12 switch off radio
17 finish

>>> 
>>> 
00 FiPy 1.20.2.rc9
00 lte
Traceback (most recent call last):
  File "<stdin>", line 9, in <module>
OSError: Couldn't connect to Modem (modem_state=disconnected)

There also seems to be some wifi issues this firmware as well in that the lopy is very reluctant to connect to the office router here, a problem we didn't have before we upgraded to 1.20.2.rc9

This firmware doesn't flash the rgb led blue every 4s either (like earlier firmwares do when not running user code) so it's difficult to know what state the device is in too .

[Gijs]

Hi,
Thanks for sending that through, I'll be testing your example, thanks a lot!

You can re-enable the heartbeat (4s. flash) by using
import pycom; pycom.heartbeat_on_boot(True).

Im not familiar with the wifi issues you mention, we do sometimes have issues with our router though, where it stops broadcasting the 2.4GHz WiFi network, but other than that Im not aware of issues with devices and connecting. Perhaps you could elaborate? Though I have no clue where the issue would be, if there is one.

[kjm]

@Gijs I'm perusing a hunch that maybe the fipy is deafer to wifi than our gpys because of the inability to switch off the lora radio, maybe caused by some local oscillator leakage? The fipy definitely reports lower wifi signal strengths than the gpys in the same location.

[Gijs]

That is quite a hunch you got there :). To be honest I'd have to check that in more detail (In my quick test here using this example code, I was not able to spot a noticeable difference over time between a Lopy4, Fipy or Gpy sitting right next to each other):

from network import WLAN
import time
import machine
wlan = WLAN(mode=WLAN.STA)

while True:
  nets = wlan.scan()
  for net in nets:
    if(net.ssid == 'ssid'):
      print(net.ssid, net.rssi)
  time.sleep(10)

Do you perhaps have different devices to test your hunch with?
For the original issue, Im currently having some reception issues from within the office, but It's still on my list!

[kjm]

@Gijs Got another uncommunicative sequans modem Gijs

>>> import sqnsupgrade
>>> sqnsupgrade.info(debug=1)
<<< Welcome to the SQN3330 firmware updater [1.2.6] >>>
>>> GPy with firmware version 1.20.1.r1
The current delay is 1000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) None
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) None
The current delay is 2000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) None
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) None
The current delay is 3000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) None
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) None
The current delay is 4000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) None
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) None
The current delay is 5000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) None
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) None
Modem state: None

The lte.factory_reset() is not option since an uncommunicative modem means I can't create an lte. Any other ideas I can try?

[Gijs]

Either the modem is talking on a wrong baudrate, or wrong UART, or bootrom, or you have an issue with the communication pins (perhaps you attached some device to it, or an internal issue in the esp32), or you need a power cycle. As a last resort, the LTE modem could be fryed.

To solve the first two cases, you can check different baudrates using:

from machine import UART
import time
# Open serial connection to LTE modem
serial = UART(1, baudrate=921600, pins=('P20', 'P18', 'P19', 'P17'), timeout_chars=1)
# For example write AT to modem
serial.write("AT" + '\r\n')
# Read back the response from the modem
time.sleep(3)
print(serial.read())
 # Deinit uart to lte modem
serial.deinit()

Next to that, safebooting might help

The wrong UART port is a bit more complicated (using testpads on the bottom of the Fipy), if needed I can elaborate.

[kjm]

@Gijs Response was None, no joy from safe booting. The only event suffered by this gpy is the lipo powering it went flat due failed solar panel, so hoping the modem is recoverable. Underside test pads is sounding fiddly, how likely is it that the modem would be on the wrong baud or port? Wouldn't the modem firmware have only 1 value for each, namely the value appropriate for the gpy? Maybe I should be writing code to deepsleep everything once the lipo hits 3.4v if permanent modem loss is going to be a by product of low battery voltage operation?

Also your suggested code gives the same result (None) on a gpy with a working modem, what baud should the gpy & the sequans chat on?

[Gijs]

I'm not sure what the chance is of the modem talking on a different UART, or whether that is what is the case here. Though there exist some AT commands that change the UART settings, and there is some recovery bootloader mode that uses a different UART that should always be accessible. If you send me a message over at support@pycom.io I can talk you through reading the other UART ports (it is indeed somewhat fiddly).

Also your suggested code gives the same result (None) on a gpy with a working modem, what baud should the gpy & the sequans chat on?

For a Gpy, the pins are different. I thought we were discussing the Fipy. For the Gpy, you should use ('P5', 'P98', 'P7', 'P99').

[kjm]

@Gijs said in Couldn't connect to Modem (modem_state=disconnected):

('P5', 'P98', 'P7', 'P99')

Tnx for gpy appropriate pin nums Gijs. Sadly the gpy with the uncommunicative sequans has become obstinate itself

rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2))
waiting for download

is all I can get out of it now. Erased the flash & reloaded the firmware but no change. Any tips on recovering dead GPYs?

[pottendo pottendo]

@Gijs said in Couldn't connect to Modem (modem_state=disconnected):

To clarify, this did not happen during the update process right?

Im assuming you already tried to disconnect power and reconnect it again, and that lte.factory_reset() is also not working for you. Could you try sqnsupgrade.info(debug=True) and let me know of the output

hi,
I have the same problem - on a Gpy, running:
Pycom MicroPython 1.20.2.r4 [v1.11-ffb0e1c] on 2021-01-12; GPy with ESP32

Here's my output:

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff8020,len:8
load:0x3fff8028,len:2128
load:0x4009fa00,len:19760
entry 0x400a05bc
Smart Provisioning started in the background
See https://docs.pycom.io/smart for details
Pycom MicroPython 1.20.2.r4 [v1.11-ffb0e1c] on 2021-01-12; GPy with ESP32
Pybytes Version: 1.6.1
Type "help()" for more information.
>>> import sqnsupgrade; sqnsupgrade.info()
<<< Welcome to the SQN3330 firmware updater [1.2.6] >>>
>>> GPy with firmware version 1.20.2.r4
Cannot determine modem state!
>>> sqnsupgrade.info(debug=True)
<<< Welcome to the SQN3330 firmware updater [1.2.6] >>>
>>> GPy with firmware version 1.20.2.r4
The current delay is 1000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) None
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) None
The current delay is 2000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) None
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) None
The current delay is 3000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) None
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) None
The current delay is 4000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) None
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) None
The current delay is 5000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) None
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) None
Modem state: None
Cannot determine modem state!

This state was entered when using the Linux cmd-line to flash the CATM1 Firmware via USB after having the NB-IoT installed some weeks ago using the same method.

Python 3.9.5 (default, May 11 2021, 08:20:37) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqnsupgrade
>>> sqnsupgrade.run('/dev/ttyACM0', '/work/src/arduino-play/pycom/CATM1-5.2-48829/CATM1-5.2-48829-1.dup', load_fff=False)
<<< Welcome to the SQN3330 firmware updater [1.2.7] >>>
Attempting AT wakeup...
Session opened: version 1, max transfer 8192 bytes
Sending 416448 bytes: [########################################] 100%
Waiting for modem to finish the update...
<<<=== DO NOT DISCONNECT POWER ===>>>
Resetting.....................................................................................................................................................................................................................................................................................................................................................Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/work/src/arduino-play/pycom/pycom-libraries/lib/sqnsupgrade/sqnsupgrade.py", line 1076, in run
    sqnup.upgrade_ext(port=port, ffile=ffile, mfile=mfile, resume=resume, debug=debug, pkgdebug=False, verbose=verbose, load_fff=load_fff, force_fff=force_fff)
  File "/opt/work/src/arduino-play/pycom/pycom-libraries/lib/sqnsupgrade/sqnsupgrade.py", line 948, in upgrade_ext
    if self.__run(file_path=ffile, resume=True if mfile is not None else resume, direct=False, port=port, debug=debug, pkgdebug=pkgdebug, verbose=verbose, load_fff=load_fff, fc=fc, force_fff=force_fff):
  File "/opt/work/src/arduino-play/pycom/pycom-libraries/lib/sqnsupgrade/sqnsupgrade.py", line 719, in __run
    self.wait_for_modem(send=False, echo_char='.', expected=b'+SYSSTART')
  File "/opt/work/src/arduino-play/pycom/pycom-libraries/lib/sqnsupgrade/sqnsupgrade.py", line 150, in wait_for_modem
    raise OSError('Timeout waiting for modem to respond!')
OSError: Timeout waiting for modem to respond!

I hope the modem can be somehow recovered.
If not I fear the Gpy is not suitable for our projects... :-(

thx, pottendo

[Tarmo Saluste]

@Gijs said in Couldn't connect to Modem (modem_state=disconnected):

sqnsupgrade.info(debug=True)

I have the same problem.

The current delay is 1000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) b'\x00'
Response (AT #4) None
Response (AT #1 @ 115200) None
Response (AT #2 @ 115200) b'\x00'
The current delay is 2000
Response (+++ #1): None
Response (AT #1) None
Response (AT #3) b'\x00'
Response (AT #4) b'\x00'
Response (AT #1 @ 115200) b'\x00'
Response (AT #2 @ 115200) b'\x00'
The current delay is 3000
Response (+++ #1): None
Response (AT #1) b'\x00'
Response (AT #2) None
Response (+++ #2): None
Response (AT #3) b'\x00'
Response (AT #4) b'\x00\x00'
Response (AT #1 @ 115200) b'\x00'
Response (AT #2 @ 115200) b'\x00'
The current delay is 4000
Response (+++ #1): b'\x00'
Response (AT #1) None
Response (AT #3) b'\x00\x00'
Response (AT #4) b'\x00'
Response (AT #1 @ 115200) b'\x00'
Response (AT #2 @ 115200) b'\x00\x00'
The current delay is 5000
Response (+++ #1): None
Response (AT #1) b'\x00'
Response (AT #2) None
Response (+++ #2): None
Response (AT #3) b'\x00\x00'
Response (AT #4) b'\x00\x00\x00'
Response (AT #1 @ 115200) b'\x00\x00'
Response (AT #2 @ 115200) b'\x00\x00'
Modem state: None
Cannot determine modem state!