Method To Take Control of Cell Modem

tlanier

I seemed to have hit a major problem using the AT+SQNHTTPSND modem command. Although the manual does not state this, the maximum size of the data posted is apparently 1500 bytes.

If anyone has ideas of how to increase this, please let me know. Unfortunately my app requires posting messages over 3000 bytes.

Disappointed...

tlanier

I've now converted my code to use the Sequans modem socket commands in command mode (not on-line mode). I first tried to use the on-line mode, but ran into issues (no good way I could find to prevent overrunning tx buffer). This code resolves the 1500 byte limitation I ran into using the HTTP modem commands. There's still some cleanup to do to make the code more robust; however, it is working with large transmit and receive data blocks now.

During development while I had coding errors and the program would terminate abnormally, I did put the modem in a mode that I couldn't recover from without power cycling the GPy. The modem would not response to AT commands, Trying to do lte = LTE() would generate the following error (OSError: Couldn't connect to Modem (modem_state=disconnected)). Doing machine.reset() would not correct the problem.

Link to discussion of similar problem on Github

Hopefully whatever I did to cause the problem will not occur in debugged code. There needs to be a true hardware reset of the modem. It seems that a GPIO pin should be tied to a physical reset line going to the modem that we could pulse if needed. I'm also of the opinion that the watchdog timer in the GPy should have been a hardware circuit not dependent on software to work.

So far I have not seen any unexplained crashes while running the code.

This code has been added to my modem.py module.

#####################################################################
#                       Socket Close          
#####################################################################
def socket_close():
    # socket close
    # connID = 1 --> socket connection identifier
    a = at('AT+SQNSH=1', 'OK')
    if len(a) == 0:
        log("error closing connection")
        return


#####################################################################
#                       Socket Connect          
#####################################################################
def socket_connect(ip, port, timeout_secs=10):
    # returns: True if connection successful
    log("connecting to ip - {0}, port - {1}".format(ip, port))

    # make sure socket is closed
    socket_close()

    # socket configuration
    # connID = 1 --> socket connection identifier
    # cid = 1 --> PDP context identifier
    # pktSz = 1500 --> packet size
    # maxTo = 10 --> exchange timeout in secs
    # connTo = 600 --> connection timeout in msecs
    # txTo = 3 --> data sending timeout in hundreds of msecs
    # a = at('AT+SQNSCFG=1,1,1500,10,600,3', 'OK')
    a = at('AT+SQNSCFG=1,1,1500,{0},600,3'.format(timeout_secs), 'OK')
    if len(a) == 0:
        log("error connecting")
        return False
    
    # socket extended configuration
    # connID = 1 --> socket connection identifier
    # srMode = 1 --> SQNSRING URC data amount mode
    # recvDataMode = 1 --> data in hex
    # keepalive = 0 --> unused
    # listenAutoRsp = 0 --> deactivated
    # sendDataMode = 1 --> data in hex
    a = at('AT+SQNSCFGEXT=1,1,1,0,0,1', 'OK')
    if len(a) == 0:
        log("error connecting")
        return False

    # socket dial
    # connID = 1 --> socket connection identifier
    # txProt = 0 --> TCP protocol
    # rPort = <port> --> remote host port to contact
    # IPaddr = <ip> --> address of remote host
    # closureType = 0 --> local host closes immediately when remote host has closed
    # lPort = 0 --> UDP port (not used)
    # connMode = 1 --> command mode connection
    # acceptAnyRemote = 0 --> disabled 
    a = at('AT+SQNSD=1,0,{0},"{1}",0,0,1,0'.format(port,ip), 'OK')
    if len(a) == 0:
        log("error connecting")
        return False

    log("** connection successful **")
    return True
    

#####################################################################
#                       Socket Info         
#####################################################################
def socket_info():
    # modem response: ['+SQNSI: 1,0,0,0,0', 'OK']
    # connId: 1
    # sent: total bytes sent
    # received: total bytes received
    # buff_in: total bytes not yet read (buffered)
    # ack_waiting: total bytes sent but not yet acknowledged
    a = at('AT+SQNSI=1')


#####################################################################
#                       Socket Receive Data         
#####################################################################
def socket_rx():
    # receives up to 1500 bytes maximum
    log("check for rx data")
    a = at('AT+SQNSRECV=1,1500')
    # no data ready --> modem response: ['+CME ERROR: operation not supported']
    # data ready --> modem response: ['+SQNSRECV: 1,3', '010203', 'OK']
    b = b''
    for i in range(len(a)):
        if a[i].startswith('+SQNSRECV:'):
            # next line is the data
            b = unhexlify(a[i+1])

    log("rx {0} bytes".format(len(b)))
    return b


#####################################################################
#                       Socket Status         
#####################################################################
def socket_status():
    # modem response: ['+SQNSS: 1,0', '+SQNSS: 2,0', '+SQNSS: 3,0', '+SQNSS: 4,0', '+SQNSS: 5,0', '+SQNSS: 6,0', 'OK']
    # connID
    # state: 
    #   0-socket closed
    #   1-socket with active data transfer connection
    #   2-socket suspended
    #   3-socket suspended with pending data
    #   4-socket listening
    #   5-socket with incoming connection
    #   6-socket in opening process
    a = at('AT+SQNSS?')
    return a


#####################################################################
#                       Socket Transmit Data         
#####################################################################
def socket_tx(tx, n=0):
    # transmits up to 1500 bytes maximum
    # returns: True if successful
    if n == 0:
        n = len(tx)
    
    log("tx {0} bytes".format(n))
    a = at('AT+SQNSSENDEXT=1,{0}'.format(n), '> ')
    hex = hexlify(tx)
    uart.write(hex)
    a = at_response('OK')
    if len(a) == 0:
        log("error transmitting data")
        return False

    return True

This code is in my http.py module.

import modem
import sys
import ubinascii
import uos
import ustruct
import util
from util import log
from utime import sleep_ms, ticks_ms, ticks_diff

BUFFER_SIZE = const(1500)
CR = const(0x0d)
LF = const(0x0a)
MAX_HEADER_SIZE = const(128)
MAX_TX_SIZE = const(1500)
HTTP_TIMEOUT_MSECS = const(15000)
# HTTP_TIMEOUT_MSECS = const(300000)  # TESTING

http_error = False      # HTTP communication error: True/False
http_size = 0           # HTTP POST message size
rx_cnt = 0              # no. bytes in rx_data buffer
rx_data = b''           # rx data buffer
rx_pt = 0               # pointer into rx data buffer
tx_cnt = 0              # no. bytes in tx_data buffer
tx_data = bytearray(MAX_TX_SIZE)    # tx data buffer
tx_flag = False         # controls tx_byte 

#####################################################################
#                       Close HTTP Socket
#####################################################################
def close_http():
    modem.socket_close()


#####################################################################
#                       Open HTTP Socket
#####################################################################
def open_http(host_ip, host_port):
    global http_error, rx_cnt, rx_data, rx_pt, tx_cnt
    
    http_error = False
    rx_cnt = 0
    rx_data = b''
    rx_pt = 0
    tx_cnt = 0

    r = modem.socket_connect(host_ip, host_port)
    if r == False:
        log("TCP socket connect error")
        http_error = True


#####################################################################
#                    Receive BLOCK From Response
#####################################################################
def rx_block(cnt):
    b = bytearray(cnt)
    i = 0
    while i < cnt:
        b[i] = rx_byte()
        i = i + 1

    return b


#####################################################################
#                    Receive BYTE From Response
#####################################################################
def rx_byte():
    global http_error, http_size, rx_cnt, rx_data, rx_pt

    t0 = ticks_ms()

    while rx_cnt == 0:
        rx_data = modem.socket_rx()
        rx_cnt = len(rx_data)
        rx_pt = 0
        if rx_cnt > 0:
            break

        t1 = ticks_ms()
        tx = ticks_diff(t1, t0)
        if (tx > HTTP_TIMEOUT_MSECS):
            http_error = True
            return 0

        sleep_ms(200)

    b = rx_data[rx_pt]
    rx_cnt = rx_cnt - 1
    rx_pt = rx_pt + 1
    http_size = http_size - 1

    # print("rx byte: {0}".format(hex(b)))
    return b


#####################################################################
#                    Receive DWORD From Response
#####################################################################
def rx_dword():
    dw1 = rx_byte()
    dw2 = rx_byte()
    dw3 = rx_byte()
    dw4 = rx_byte()

    return (dw1 + (dw2<<8) + (dw3<<16) + (dw4<<24))


#####################################################################
#                    Receive One HTTP Header
#####################################################################
def rx_http_header():
    # returns header string
    global http_error

    n = 0
    header = bytearray(MAX_HEADER_SIZE)
    s = ""

    while True:
        b = rx_byte()
        # print("byte = {0}".format(b))
        if http_error:
            return ""

        if b == CR:
            continue

        if b == LF:
            break

        header[n] = b
        n = n + 1

        if n >= MAX_HEADER_SIZE-1:
            http_error = True
            return ""

    if n > 0:
        # print("n = {0}, header = {1}".format(n,header))
        s = header[0:n].decode('utf-8')
    
    return s

        
#####################################################################
#                    Receive HTTP Headers
#
# HTTP/1.1 200 OK
# <headers>
# Content-Length: nnn
# <headers>
# <crlf>
#
# Updates: http_size, http_error
#####################################################################
def rx_http_headers():
    global http_error, http_size

    headers = []      # array of header lines
    http_size = 0

    while True:
        s = rx_http_header()
        log("length = {0}, header = {1}".format(len(s), s))
        if http_error:
            return headers

        if s == "":
            break

        headers.append(s)
 
    if len(headers) < 2:
        http_error = True
        return headers
    
    a = headers[0].split(' ')
    if (a[1] != "200") or (a[2]) != "OK":
        http_error = True
        return headers

    for s in headers:
        if s.startswith("Content-Length:"):
            a = s.split(' ')
            http_size = int(a[1])

    return headers


#####################################################################
#                    Receive String From Response
#####################################################################
def rx_string(max_len):
    rx = bytearray(max_len)
    i = 0
    while i < max_len:
        b = rx_byte()
        if b == 0:
            break

        rx[i] = b
        i = i + 1

    if i == 0:
        return ""

    s = rx[0:i].decode('utf-8')

    return s


#####################################################################
#                    Receive WORD From Response
#####################################################################
def rx_word():
    dw1 = rx_byte()
    dw2 = rx_byte()

    return (dw1 + (dw2<<8))


#####################################################################
#                       Transmit Block
#####################################################################
def tx_block(b):
    for x in b:
        tx_byte(x)


#####################################################################
#                       Transmit Byte
#####################################################################
def tx_byte(b):
    global http_size, tx_cnt, tx_data, tx_flag

    if tx_flag == False:
        http_size = http_size + 1
        # print("tx byte: {0}, http size: {1}".format(hex(b), http_size))
        return

    # print("tx byte: {0}".format(hex(b)))

    tx_data[tx_cnt] = b
    tx_cnt = tx_cnt + 1
    if tx_cnt >= MAX_TX_SIZE:
        tx_flush()


#####################################################################
#                       Transmit DWORD
#####################################################################
def tx_dword(dw):
    tx_word(dw & 0xffff)
    tx_word((dw >> 16) & 0xffff)


#####################################################################
#                       Transmit Flush
#####################################################################
def tx_flush():
    global tx_cnt, tx_data

    if tx_cnt == 0:
        return

    r = modem.socket_tx(tx_data, tx_cnt)
    if not r:
        http_error = True
        return

    tx_cnt = 0


#####################################################################
#                       Transmit HTTP Headers
#####################################################################
def tx_http_headers(http_post, host_name, host_port):
    # example:
    # POST <http_post> HTTP/1.1
    # Host: <host_name>:<host_port>
    # Content-Length: <http_size>
    # Content-Type: application/octet-stream
    # <crlf>
    global http_error, http_size

    log("tx HTTP headers")

    if http_error:
        return

    s = "POST {0} HTTP/1.1\r\n".format(http_post)
    tx_string(s)

    if host_port != 80:
        s = "Host: {0}:{1}\r\n".format(host_name, host_port)
    else:
        s = "Host: {0}\r\n".format(host_name)

    tx_string(s)
    s = "Content-Length: {0}\r\n".format(http_size)
    tx_string(s)
    s = "Content-Type: application/octet-stream\r\n"
    tx_string(s)
    s = "\r\n"
    tx_string(s)


#####################################################################
#                       Transmit String
#####################################################################
def tx_string(s):
    b = s.encode('utf-8')

    for x in b:
        tx_byte(x)
    

#####################################################################
#                       Transmit WORD
#####################################################################
def tx_word(w):
    tx_byte(w & 0xff)
    tx_byte((w >> 8) & 0xff)

tlanier

We now have our program running completely independent of the LTE module. 26 units have been running for multiple days with no crashes. I now think we have found a reliable solution. The Sequans modem AT socket commands are used to do the HTTP posts. We reliably receive 90K files and transmit 1-5K of data with no issues. Pycom MicroPython V1.20.0.rc13 is the OS version we are using.

Currently we are using whatever version of the modem firmware (41065, 43818, and 47510 have been seen) that comes with the GPy. The version of the OS we are using has the ability to "downgrade" the modem firmware to 41065 if we decide it is needed. Some newer versions of the OS do not have this ability.

One note of importance is we did discover that in our case the default stack size was marginal when receiving large files. Yes, I did go back and check the old program and verified that the stack issue was not the cause of the original crash problems.

#####################################################################
#                          Initialize Tasks
#####################################################################
def init_tasks():
    n = 8192    # 4096 is marginal
    _thread.stack_size(n)
    log("initializing tasks with stack size: {}".format(n))
    
    _thread.start_new_thread(com.com_task, ())
    _thread.start_new_thread(cycle.cycle_task, ())

ruok4t

@tlanier I really like the sound of what your doing here. I've been struggling with Pycom LTE module for months now. Really don't like the way it blocks other threads and randomly locks up for no apparent reason. Are you possibly in a position to share a complete "Hello LTE World" example. I'm sure I wouldn't be the only one around here who would be very interested. Hopefully someone from Pycom might also take an interest and finally admit they need to revise their code.

tlanier

@ruok4t Sorry I didn't see your request earlier.

The software below is now working in production units. We ran about 20 units testing in house for two weeks with no crashes before beginning deployment with the new version. We are using MicroPython V1.20.0.rc13 with multiple versions of the modem firmware. It's a shame we had to resort to this, but it seems to work reliably.

I've had to break the post up into 3 parts because of limitations of this forum.

Part 1 of 3

Here's the modem.py module:

import globals as g
import gpio
from machine import UART
import micropython
import sys
import util
from ubinascii import hexlify, unhexlify
from util import log, mem_info
from utime import sleep_ms, ticks_ms, ticks_diff

uart = None

#####################################################################
#                     Send AT Command To Modem
#####################################################################
def at(cmd, filterx=""):
    try:
        # util.mem_info("modem.at")

        if uart.any() > 0:
            # get unsolicited modem response 
            a = at_response()

            for s in a:
                at_process(s)

        if gpio.G():
            log("  modem command: {}".format(cmd))

        # send command to modem
        n = uart.write(cmd + '\r')
        if n != len(cmd) + 1:
            if gpio.G():
                log("error sending command to modem: n = {}".format(n))
            return []

        a = at_response(filterx)
        return a

    except Exception as ex:
        sys.print_exception(ex)
        a = []

    return a    # return modem response lines in array


#####################################################################
#                     Filter Modem Response
#
#####################################################################
def at_filter(a, filterx):
    # a is passed as reference value
    # filter response

    if filterx == "":
        return

    n = len(filterx)
    i = len(a)
    while i > 0:
        i = i - 1

        if not a[i].startswith(filterx):
            # remove lines that do not start with filter value
            at_process(a[i])
            a.pop(i)


#####################################################################
#                     Process Unsolicited Response
#####################################################################
def at_process(s):
    if s == 'OK':
        return
    
    if s.startswith("ERROR"):
        return
    
    if s.startswith("+CME ERROR:"):
        return

    # process unsolicited response here
    if gpio.G():
        log("  unsolicited modem response: {}".format(s))


#####################################################################
#                       Get AT Command Response
#####################################################################
def at_response(filterx=""):
    try:
        t0 = ticks_ms()
        
        while True:
            if uart.any() > 0:
                # insure entire response has been received
                sleep_ms(50)
                break

            t1 = ticks_ms()
            tx = ticks_diff(t1, t0)

            if tx > 10000:
                log("timeout error - no modem response after 10 secs")
                return []

            sleep_ms(100)

        b = uart.read()
        s = str(b, 'utf-8')
        a = s.split('\r\n')
        a = list(filter(None, a))

        if gpio.G():
            log("    modem response: {}".format(a))
        
        at_filter(a, filterx)

    except Exception as ex:
        sys.print_exception(ex)
        a = []

    return a    # return modem response lines in array


#####################################################################
#                       Check Registration
#####################################################################
def check_registration():
    # return 0-5, 99-unknown
    # response: ['+CEREG: 2,4', 'OK']
    # response: ['+CEREG: 2,1,"2C29","0B54400F",7', 'OK']
    # 2,0 - not registered, not searching
    # 2,1 - registered, home network
    # 2,2 - not registered, searching
    # 2,3 - registration denied
    # 2,4 - unknown (out of coverage)
    # 2,5 - registered, roaming

    try:
        while True:
            log("check registration")

            a = at('AT+CEREG?','+CEREG:')
            if len(a) == 0:
                break

            a = a[0].split(",")

            if (a[0] != "+CEREG: 2"):
                break
            
            reg = int(a[1])
            if reg == 1:
                log("*** registered - home network ***")
            elif reg == 5:
                log("*** registered - roaming ***")
            else:
                log("*** NOT registered ***")

            return reg

    except Exception as ex:
        log("check registration exception")
        sys.print_exception(ex)

    # unknown registration
    log("error checking registration")
    return 99


#####################################################################
#                       Check SIM Status
#####################################################################
def check_sim_status():
    try:
        log("checking SIM card status")
        a = at('AT+CPIN?', '+CPIN: READY')
        if len(a) > 0:
            return True

    except Exception as ex:
        sys.print_exception(ex)

    log("error SIM card not ready")
    return False


#####################################################################
#                       Disable LTE Modem
#####################################################################
def disable_modem():
    try:
        log("disable modem")
        a = at('AT+CFUN=0','OK')     # minimum functionality
        if len(a) == 0:
            log("error disabling modem")

    except Exception as ex:
        sys.print_exception(ex)


#####################################################################
#                       Enable Modem
#####################################################################
def enable_modem():
    try:
        log("enabling modem")
        a = at('AT+CFUN=1','OK')     # full functionality
        if len(a) == 0:
            log("error enabling modem")

    except Exception as ex:
        sys.print_exception(ex)


#####################################################################
#                Get CFUN - Modem Level of Functionality
#####################################################################
def get_cfun():
    # 0 - minimum functionality
    # 1 - full functionality
    # 4 - radios disabled but access to SIM allowed
    # -1 - unknown
    # note: if modem does +SHUTDOWN/+SYSSTART, CFUN will be reset to 0
    try:
        while True:
            a = at('AT+CFUN?','+CFUN:')
            # response: ['+CFUN: 1', 'OK']

            if len(a) == 0:
                break

            cfun = int(a[0][7])
            return cfun

    except Exception as ex:
        sys.print_exception(ex)

    return -1


#####################################################################
#                           Get SMS Mode
#####################################################################
def get_sms_mode():
    # return 0-PDU mode or unknown, 1-text mode

    try:
        log("get SMS mode")
        a = at('AT+CMGF?','+CMGF:')
        # response: ['+CMGF: 1', 'OK']
        if len(a) == 0:
            return 0    # mode unknown

        mode = int(a[0][7])
        return mode

    except Exception as ex:
        sys.print_exception(ex)

    return 0    # mode unknown


#####################################################################
#                     Get SIM Card ICCID Number
#####################################################################
def iccid():
    # CFUN must be 1 or 4
    # SIM status must be ready 
    try:
        log("get SIM card ICCID")
        a = at('AT+SQNCCID?', '+SQNCCID:')
        # example response: ['+SQNCCID: "89014103271203065543",""', 'OK']
        if len(a) == 0:
            return ''

        a = a[0].split(",")
        n = len(a[0])
        r = a[0][11:n-1]

    except Exception as ex:
        sys.print_exception(ex)
        r = ''

    log("ICCID = {0}".format(r))
    return r    # ICCID as string


#####################################################################
#                       Get Modem IMEI Number
#####################################################################
def imei():
    try:
        log("get modem IMEI number")
        a = at('AT+CGSN=1','+CGSN:')
        # example response: ['+CGSN: "354347094028575"', 'OK']
        if len(a) == 0:
            return ''

        n = len(a[0])
        r = a[0][8:n-1]

    except Exception as ex:
        sys.print_exception(ex)
        r = ''

    log("IMEI = {0}".format(r))
    return r    # IMEI as string


#####################################################################
#                   Modem Firmware Info
#####################################################################
def info():
    log("get modem firmware version")
    a = at('ATI1')
    return a


#####################################################################
#                       Initialize Modem
#####################################################################
def init_modem():
    try:
        log("initializing modem")

        if uart == None:
            init_uart()

        info()

        set_sms_mode()

        # enable +CME ERROR: <err> result code and use verbose <err> values
        at('AT+CMEE=2')

        enable_modem()

        log("wait for SIM card ready")
        n = 0
        while True:
            log("checking SIM card status")
            a = at('AT+CPIN?', '+CPIN: READY')
            if len(a) > 0:
                break

            n = n + 1
            if n > 20:
                util.reset(13, "No SIM card")
                
            sleep_ms(200)

        set_apn()

    except Exception as ex:
        sys.print_exception(ex)


#####################################################################
#               Initialize Uart Connected To LTE Modem
#####################################################################
def init_uart():
    global uart
    # pins=(TXD, RXD, RTS, CTS)
    # FiPy: pins=('P20', 'P18', 'P19', 'P17')
    # GPy:  pins=('P5', 'P98', 'P7', 'P99')
    log("initializing UART")
    # rx_buffer_size: must be > 127, 200000 works, 250000 crashes
    uart = UART(1, baudrate=921600, bits=8, parity=None, stop=1, timeout_chars=2, pins=('P5', 'P98', 'P7', 'P99'), rx_buffer_size=512)


#####################################################################
#           Is Cellular Modem Attached To Packet Domain Service
#####################################################################
def isattached():
    # returns: True/False
    try:
        a = at('AT+CGATT?','+CGATT:')
        # example response: ['+CGATT: 0', 'OK']
        if len(a) == 0:
            return False

        if a[0][8] == '1':
            return True

    except Exception as ex:
        sys.print_exception(ex)

    return False


#####################################################################
#                           Reset Modem
#####################################################################
def reset_modem():
    try:
        if uart == None:
            init_uart()

        log("reseting modem...")
        a = at('AT^RESET', '+SHUTDOWN')
        # generates +SHUTDOWN ... +SYSSTART responses
        if len(a) == 0:
            log("error resetting modem - no +SHUTDOWN")

        a = at_response('+SYSSTART')
        if len(a) == 0:
            log("error resetting modem - no +SYSSTART")

    except Exception as ex:
        sys.print_exception(ex)


#####################################################################
#                  Check Modem For Restart/Crash
#####################################################################
def restarted():
    # return True if modem has crashed / restarted
    try:
        # if CFUN != 1, modem has crashed / restarted
        cfun = get_cfun()
        if cfun != 1:
            return True

        # if SMS text mode is reset to zero, modem has crashed / restarted
        mode = get_sms_mode()
        if mode != 1:
            return True

        return False

    except Exception as ex:
        sys.print_exception(ex)

    return True


#####################################################################
#                           Set APN
#####################################################################
def set_apn():
    try:
        log("set APN")

        # ICCID is only valid if SIM status is ready 
        r = iccid()[0:5]
        if r == "89014":
            apn = g.oIni.apn_89014   # Kore AT&T SIM
        elif r == "89012":
            apn = g.oIni.apn_89012   # Kore T-Mobile SIM
        elif r == "89445":
            apn = g.oIni.apn_89445   # Hologram SIM
        else:
            apn = g.oIni.apn_89014   # Kore AT&T SIM

        g.oIni.apn = apn
        log("APN: {0}".format(apn))

        # try to reduce wear of non-volatile memory
        a = at('AT+CGDCONT?', '+CGDCONT:')
        # response --> ['+CGDCONT: 1,"IP","10569.mcs",,,,0,0,0,0,0,0', ...]
        if len(a) > 0:
            a = a[0].split(",")
            if (a[0] == '+CGDCONT: 1') and (a[1] == '"IP"') and (a[2] == '"'+apn+'"'):
                return

        # apn needs setting
        cmd = 'AT+CGDCONT=1,"IP","{0}"'.format(apn)
        a = at(cmd, 'OK')
        if len(a) == 0:
            log("error setting APN")

    except Exception as ex:
        sys.print_exception(ex)


#####################################################################
#                       Set SMS Mode To Text
#####################################################################
def set_sms_mode():
    try:
        log("set SMS mode to text")
        a = at('AT+CMGF=1', 'OK')     # set SMS mode to text
        if len(a) == 0:
            log("error setting SMS mode to text")

    except Exception as ex:
        sys.print_exception(ex)


#####################################################################
#                       Get Signal Strength
#####################################################################
def signal_strength():
    # 0-31, 99-unknown
    try:
        while True:
            log("check signal strength")
            a = at('AT+CSQ', '+CSQ:')
            # example response: ['+CSQ: 18,99', 'OK'] where 18 is signal strength
            if len(a) == 0:
                break

            n = len(a[0])
            s = a[0][6:n-1]
            a = s.split(",")
            cell_signal = int(a[0])

            g.oStatus.cell_signal = cell_signal
            log("signal strength = {0}".format(cell_signal))
            return cell_signal

    except Exception as ex:
        sys.print_exception(ex)

    # unknown signal strength
    g.oStatus.cell_signal = 99
    log("signal strength unknown")
    return 99


#####################################################################
#             Delete Read, Sent, and Unsent SMS Messages
#####################################################################
def sms_delete():
    try:
        log("delete all READ, SENT, and UNSENT SMS messages")
        a = at('AT+CMGD=1,3', 'OK')
        if len(a) == 0:
            log("error trying to delete read, sent, and unsent SMS messages")

    except Exception as ex:
        sys.print_exception(ex)


#####################################################################
#                     Delete All SMS Messages
#####################################################################
def sms_delete_all():
    # doesn't work until some time after attach() is called
    try:
        log("delete all SMS messages")
        a = at('AT+CMGD=1,4', 'OK')
        if len(a) == 0:
            log("error trying to delete all SMS messages")
        
    except Exception as ex:
        sys.print_exception(ex)


#####################################################################
#                     Process SMS Messages
#####################################################################
def sms_process(r):
    # example response:
    # ['+CMGL: 1,"REC READ","+1xxxxxxxxxx",,"19/01/07,09:16:53-20"', 'Test3', '+CMGL: 2,"REC READ","+1xxxxxxxxxx",,"19/01/07,11:12:30-20"', 'Test4', 'OK']
    
    try:
        for m in r:
            if (m[0:5] == "+CMGL"):
                continue
            
            a = m.split("=")
            cmd = a[0].upper()

			# process message here

    except Exception as ex:
        sys.print_exception(ex)


#####################################################################
#                      Read Unread SMS Messages
#####################################################################
def sms_read():
    # response example: ['+CMGL: 1,"REC UNREAD","+1xxxxxxxxxx",,"19/01/07,09:16:53-20"', 'Test3', 'OK']
    log("read unread SMS messages")
    a = at('AT+CMGL="REC UNREAD"')
    return a
    

#####################################################################
#                       Read All SMS Messages
#####################################################################
def sms_read_all():
    # response example:
    # ['+CMGL: 1,"REC READ","+1xxxxxxxxxx",,"19/01/07,09:16:53-20"', 'Test3', '+CMGL: 2,"REC READ","+1xxxxxxxxxx",,"19/01/07,11:12:30-20"', 'Test4', 'OK']
    log("read all SMS messages")
    a = at('AT+CMGL="ALL"')
    return a


#####################################################################
#                       Socket Close          
#####################################################################
def socket_close():
    # socket close
    # connID = 1 --> socket connection identifier
    log("closing socket")
    a = at('AT+SQNSH=1', 'OK')
    if len(a) == 0:
        log("error closing connection")


#####################################################################
#                       Socket Connect          
#####################################################################
def socket_connect(ip, port, timeout_secs=10):
    # returns: True if connection successful
    log("connecting to ip - {0}, port - {1}".format(ip, port))

    # make sure socket is closed
    socket_close()

    # socket configuration
    # connID = 1 --> socket connection identifier
    # cid = 1 --> PDP context identifier
    # pktSz = 1500 --> packet size
    # maxTo = 10 --> exchange timeout in secs
    # connTo = 600 --> connection timeout in msecs
    # txTo = 3 --> data sending timeout in hundreds of msecs
    # a = at('AT+SQNSCFG=1,1,1500,10,600,3', 'OK')
    a = at('AT+SQNSCFG=1,1,1500,{0},600,3'.format(timeout_secs), 'OK')
    if len(a) == 0:
        log("error connecting")
        return False
    
    # socket extended configuration
    # connID = 1 --> socket connection identifier
    # srMode = 1 --> SQNSRING URC data amount mode
    # recvDataMode = 1 --> data in hex
    # keepalive = 0 --> unused
    # listenAutoRsp = 0 --> deactivated
    # sendDataMode = 1 --> data in hex
    a = at('AT+SQNSCFGEXT=1,1,1,0,0,1', 'OK')
    if len(a) == 0:
        log("error connecting")
        return False

    # socket dial
    # connID = 1 --> socket connection identifier
    # txProt = 0 --> TCP protocol
    # rPort = <port> --> remote host port to contact
    # IPaddr = <ip> --> address of remote host
    # closureType = 0 --> local host closes immediately when remote host has closed
    # lPort = 0 --> UDP port (not used)
    # connMode = 1 --> command mode connection
    # acceptAnyRemote = 0 --> disabled 
    a = at('AT+SQNSD=1,0,{0},"{1}",0,0,1,0'.format(port,ip), 'OK')
    if len(a) == 0:
        log("error connecting")
        return False

    log("** connection successful **")
    return True
    

#####################################################################
#                       Socket Info         
#####################################################################
def socket_info():
    # modem response: ['+SQNSI: 1,0,0,0,0', 'OK']
    # connId: 1
    # sent: total bytes sent
    # received: total bytes received
    # buff_in: total bytes not yet read (buffered)
    # ack_waiting: total bytes sent but not yet acknowledged
    log("socket info")
    a = at('AT+SQNSI=1')


#####################################################################
#                       Socket Receive Data         
#####################################################################
def socket_rx():
    # receives up to 1500 bytes maximum
    log("check for rx data")
    a = at('AT+SQNSRECV=1,1500')
    # no data ready --> modem response: ['+CME ERROR: operation not supported']
    # data ready --> modem response: ['+SQNSRECV: 1,3', '010203', 'OK']
    b = b''
    for i in range(len(a)):
        if a[i].startswith('+SQNSRECV:'):
            # next line is the data
            b = unhexlify(a[i+1])

    log("rx {0} bytes".format(len(b)))
    return b


#####################################################################
#                       Socket Status         
#####################################################################
def socket_status():
    # modem response: ['+SQNSS: 1,0', '+SQNSS: 2,0', '+SQNSS: 3,0', '+SQNSS: 4,0', '+SQNSS: 5,0', '+SQNSS: 6,0', 'OK']
    # connID
    # state: 
    #   0-socket closed
    #   1-socket with active data transfer connection
    #   2-socket suspended
    #   3-socket suspended with pending data
    #   4-socket listening
    #   5-socket with incoming connection
    #   6-socket in opening process
    log("socket status")
    a = at('AT+SQNSS?')
    return a


#####################################################################
#                       Socket Transmit Data         
#####################################################################
def socket_tx(tx, n=0):
    # transmits up to 1500 bytes maximum
    # returns: True if successful
    if n == 0:
        n = len(tx)
    
    log("tx {0} bytes".format(n))
    a = at('AT+SQNSSENDEXT=1,{0}'.format(n), '> ')
    b = hexlify(tx)
    uart.write(b)
    a = at_response('OK')
    if len(a) == 0:
        log("error transmitting data")
        return False

    return True

tlanier

@tlanier
Part 2 of 3

Here's our com.py module. It is more application dependent but it may give you some ideas of how to use the modem module.

import globals as g
import modem
from poll import poll
import pycom
import sys
import uos
import util
from util import log
from utime import sleep_ms
import wifi

_reset = True    # modem reset

#####################################################################
#                         Communication Task
#####################################################################
def com_task():
    global _reset

    try:
        sleep_ms(500)
        log("com_task: started")
        
        g.poll_host = True      # triggers poll at startup

        # this opens the door for continuous resets to cause high data usage
        util.set_ppp_errors(0)

        wifi.deinit()

    except Exception as ex:
        sys.print_exception(ex)

    while True:
        try:
            _reset = False
            g.modem_state = g.MS_RESET_MODEM
            modem.reset_modem()

            g.modem_state = g.MS_INIT_MODEM
            modem.init_modem()

            while _reset == False:
                # util.mem_info("com_task")

                if modem_ready():
                    g.modem_state = g.MS_REGISTERED
                    if ready_to_poll():
                        transmit()
                else:
                    g.modem_state = g.MS_REGISTERING

                sleep_ms(2000)

        except Exception as ex:
            sys.print_exception(ex)


#####################################################################
#                           Modem Ready
#####################################################################
def modem_ready():
    global _reset

    try:
        log("** check for modem ready")

        # get modem signal strength
        g.oStatus.cell_signal = modem.signal_strength()
        
        # check modem registration
        r = modem.check_registration()
        if (r == 1) or (r == 5):
            # registered
            g.com_secs = 0

            # handle SMS messages
            r = modem.sms_read()
            if len(r) > 1:
                modem.sms_process(r)
                modem.sms_delete()

            return True
        
        if g.com_secs > (10*60):
            # unable to attach in 10 mins, try resetting modem
            _reset = True
            
        if modem.restarted():
            # modem has crashed / restarted, try resetting modem
            _reset = True

        return False

    except Exception as ex:
        sys.print_exception(ex)
    
    return False


#####################################################################
#                           Ready To Poll
#####################################################################
def ready_to_poll():
    # return: True/False
    try:
        log("--- time since last poll: {0} secs ---".format(g.oNvram.last_poll_secs))

        if (g.ppp_error_secs > 0):
            log("*** poll throttling: {0} secs ***".format(g.ppp_error_secs))
            return False
            
        if (g.oIni.intra_poll_secs > 0):
            # contact server every <oIni.intra_poll_secs> seconds
            if (g.oNvram.last_poll_secs > g.oIni.intra_poll_secs):
                return True
        
        if (g.oSetup.intra_cycle_count > 0):
            # contact server every <oSetup.intra_cycle_count> cycles
            if (g.oNvram.cycle_count >= g.oSetup.intra_cycle_count):
                return True

        # g.poll_host is set at startup and by SMS POLL_HOST message
        if (g.poll_host == True):
            return True

    except Exception as ex:
        sys.print_exception(ex)

    return False


#####################################################################
#                         Transmit
#####################################################################
def transmit():
    try:
        util.lcd("Transmitting", "")
        g.modem_state = g.MS_CONNECTING

        r = poll()

        g.modem_state = g.MS_DISCONNECT
        if (r == False):
            util.lcd("Transmit failed", "")
            sleep_ms(2500)
            util.lcd("","")
            return

        util.lcd("Successful", "")
        sleep_ms(2500)
        util.lcd("","")

        g.poll_host = False
        g.oNvram.last_poll_secs = 0
        g.poll_secs = 0

        if (g.update_flag == True):
            # done after disconnect
            try:
                if util.file_exists("REMOTE.TAR"):
                    log("remove old REMOTE.TAR")
                    util.remove("REMOTE.TAR")
                
                log("rename TEMP to REMOTE.TAR")
                uos.rename("TEMP", "REMOTE.TAR")

                # create file to signal boot.py that update is available
                log("create UPDATE signal file")
                with open("UPDATE", "w") as f:
                    f.write("UPDATE")
            
                log("** New REMOTE.TAR ready to update")
            
            except Exception as ex:
                sys.print_exception(ex)

        if (g.reboot_after_poll == True):
            log("SMS message changed INI setting, rebooting")
            util.reset(6, "INI changed")

        if (g.oHost.erase == True):
            log("Erase command received, rebooting")
            g.oNvram.erase()
            util.reset(7, "Erase NVRAM")

        if (g.oHost.reset == True):
            log("Reset command received, rebooting")
            util.reset(8, "Reset command")

    except Exception as ex:
        sys.print_exception(ex)

tlanier

@tlanier
Part 3 of 3

Here's the http.py module:

Note that the modem socket commands have a 1500 byte limitation. The HTTP module overcomes that by breaking longer messages up into smaller blocks.

import modem
import sys
import ubinascii
import uos
import ustruct
import util
from util import log
from utime import sleep_ms, ticks_ms, ticks_diff

# note: HTTP errors raise exceptions

BUFFER_SIZE = const(1500)
CR = const(0x0d)
LF = const(0x0a)
MAX_HEADER_SIZE = const(128)
MAX_TX_SIZE = const(1500)
HTTP_TIMEOUT_MSECS = const(15000)
# HTTP_TIMEOUT_MSECS = const(300000)  # TESTING

http_size = 0           # HTTP POST message size
rx_cnt = 0              # no. bytes in rx_data buffer
rx_data = b''           # rx data buffer
rx_pt = 0               # pointer into rx data buffer
tx_cnt = 0              # no. bytes in tx_data buffer
tx_data = bytearray(MAX_TX_SIZE)    # tx data buffer
tx_flag = False         # controls tx_byte 

#####################################################################
#                       Close HTTP Socket
#####################################################################
def close_http():
    modem.socket_close()


#####################################################################
#                       Open HTTP Socket
#####################################################################
def open_http(host_ip, host_port):
    global rx_cnt, rx_data, rx_pt, tx_cnt

    rx_cnt = 0
    rx_data = b''
    rx_pt = 0
    tx_cnt = 0

    r = modem.socket_connect(host_ip, host_port)
    if r == False:
        raise Exception("TCP socket connect error")


#####################################################################
#                    Receive BLOCK From Response
#####################################################################
def rx_block(cnt):
    b = bytearray(cnt)
    i = 0
    while i < cnt:
        b[i] = rx_byte()
        i = i + 1

    log("rx_block: {} bytes".format(len(b)))
    
    return b


#####################################################################
#                    Receive BYTE From Response
#####################################################################
def rx_byte():
    global http_size, rx_cnt, rx_data, rx_pt

    t0 = ticks_ms()

    while rx_cnt == 0:
        rx_data = modem.socket_rx()
        rx_cnt = len(rx_data)
        rx_pt = 0
        if rx_cnt > 0:
            break

        t1 = ticks_ms()
        tx = ticks_diff(t1, t0)
        if (tx > HTTP_TIMEOUT_MSECS):
            raise Exception("HTTP timeout in rx_byte")

        sleep_ms(200)

    b = rx_data[rx_pt]
    rx_cnt = rx_cnt - 1
    rx_pt = rx_pt + 1
    http_size = http_size - 1

    # print("rx byte: {0}".format(hex(b)))
    return b


#####################################################################
#                    Receive DWORD From Response
#####################################################################
def rx_dword():
    dw1 = rx_byte()
    dw2 = rx_byte()
    dw3 = rx_byte()
    dw4 = rx_byte()

    return (dw1 + (dw2<<8) + (dw3<<16) + (dw4<<24))


#####################################################################
#                    Receive One HTTP Header
#####################################################################
def rx_http_header():
    # returns header string
    n = 0
    header = bytearray(MAX_HEADER_SIZE)
    s = ""

    while True:
        b = rx_byte()
        # print("byte = {0}".format(b))

        if b == CR:
            continue

        if b == LF:
            break

        header[n] = b
        n = n + 1

        if n >= MAX_HEADER_SIZE-1:
            raise Exception("Maximum header size error in rx_http_header")

    if n > 0:
        # older versions of MicroPython do not implement decode on bytearrays
        a = bytes(header[0:n])
        s = a.decode('utf-8')
        log("header = {}".format(s))
    
    return s

        
#####################################################################
#                    Receive HTTP Headers
#
# HTTP/1.1 200 OK
# <headers>
# Content-Length: nnn
# <headers>
# <crlf>
#
# Updates: http_size
#####################################################################
def rx_http_headers():
    global http_size

    headers = []      # array of header lines
    http_size = 0

    while True:
        s = rx_http_header()
        # log("length = {0}, header = {1}".format(len(s), s))

        if s == "":
            break

        headers.append(s)
 
    if len(headers) < 2:
        raise Exception("Not enough headers received in rx_http_headers")
    
    a = headers[0].split(' ')
    if (a[1] != "200") or (a[2]) != "OK":
        raise Exception("Invalid first header line: {0}".format(headers[0]))

    for s in headers:
        if s.startswith("Content-Length:"):
            a = s.split(' ')
            http_size = int(a[1])

    return headers


#####################################################################
#                    Receive String From Response
#####################################################################
def rx_string(max_len):
    rx = bytearray(max_len)
    i = 0
    while i < max_len:
        b = rx_byte()
        if b == 0:
            break

        rx[i] = b
        i = i + 1

    if i == 0:
        return ""

    # older versions of MicroPython do not implement decode on bytearrays
    a = bytes(rx[0:i])
    s = a.decode('utf-8')

    return s


#####################################################################
#                    Receive WORD From Response
#####################################################################
def rx_word():
    dw1 = rx_byte()
    dw2 = rx_byte()

    return (dw1 + (dw2<<8))


#####################################################################
#                       Transmit Block
#####################################################################
def tx_block(b):
    for x in b:
        tx_byte(x)


#####################################################################
#                       Transmit Byte
#####################################################################
def tx_byte(b):
    global http_size, tx_cnt, tx_data, tx_flag

    if tx_flag == False:
        http_size = http_size + 1
        # print("tx byte: {0}, http size: {1}".format(hex(b), http_size))
        return

    # print("tx byte: {0}".format(hex(b)))

    tx_data[tx_cnt] = b
    tx_cnt = tx_cnt + 1
    if tx_cnt >= MAX_TX_SIZE:
        tx_flush()


#####################################################################
#                       Transmit DWORD
#####################################################################
def tx_dword(dw):
    tx_word(dw & 0xffff)
    tx_word((dw >> 16) & 0xffff)


#####################################################################
#                       Transmit Flush
#####################################################################
def tx_flush():
    global tx_cnt, tx_data

    if tx_cnt == 0:
        return

    r = modem.socket_tx(tx_data, tx_cnt)
    if not r:
        raise Exception("Error transmitting in tx_flush")

    tx_cnt = 0


#####################################################################
#                       Transmit HTTP Headers
#####################################################################
def tx_http_headers(http_post, host_name, host_port):
    # example:
    # POST <http_post> HTTP/1.1
    # Host: <host_name>:<host_port>
    # Content-Length: <http_size>
    # Content-Type: application/octet-stream
    # <crlf>
    global http_size

    log("tx HTTP headers")

    s = "POST {0} HTTP/1.1\r\n".format(http_post)
    tx_string(s)

    if host_port != 80:
        s = "Host: {0}:{1}\r\n".format(host_name, host_port)
    else:
        s = "Host: {0}\r\n".format(host_name)

    tx_string(s)
    s = "Content-Length: {0}\r\n".format(http_size)
    tx_string(s)
    s = "Content-Type: application/octet-stream\r\n"
    tx_string(s)
    s = "\r\n"
    tx_string(s)


#####################################################################
#                       Transmit String
#####################################################################
def tx_string(s):
    b = s.encode('utf-8')

    for x in b:
        tx_byte(x)
    

#####################################################################
#                       Transmit WORD
#####################################################################
def tx_word(w):
    tx_byte(w & 0xff)
    tx_byte((w >> 8) & 0xff)

ruok4t

@tlanier Thanks, will download all parts and take a look.

SciWax

@tlanier
Thank you for compiling all this information for the community. I really appreciate it. It could become handy for my project.

Scott Basgaard

@tlanier did you ever put this in a repo?

tlanier

@Scott-Basgaard Sorry, I have not.

Petteri Salo

@tlanier Hey, I started looking into the three files you posted (http.py, com.py, modem.py) and it is looking very usefull! However, both com.py and modem.py have a line:
"import globals as g"
and I'm guessing it is a file you have, but I get an "ImportError: no module named 'globals'"

Perhaps you could upload that file too? After a quick look throuh all "g.*" I could probably deduce most of it, but things like :

g.modem_state = g.MS_RESET_MODEM

would probably be usefull and save us some time.

Anyway, Thanks for helping the community at large with your efforts!!

Cheers,
Petteri

Scott Basgaard

@Petteri-Salo agree! Would be great @tlanier if you could share anything else here. Really struggling to get LTE working stable and any further insight for us others would be greatly appreciated. Digging into this more now and will let you know how far I get as well. Thanks for sharing!

tlanier

@Petteri-Salo The g.modem_state global variable is used in a timer interrupt routine to control the blinking and color of the LED on the GPY. This gives the user a visual indication of what state the modem is in. This variable is not critical to using the supplied code.

Unfortunately the code I have published is just snippets of my production code. It was not designed to be a general purpose library (although I believe it comes close). Without publishing the entire application, there will be pieces that are missing.

Your message has motivated me to add a few random notes.

We had production, shipping GPy based systems that were experiencing reliability issues. Upgrades to newer versions of the OS seemed to make the problems worse. There was considerable evidence that the problems were caused by the LTE module. For months I waited hoping that Pycom would make the existing product more robust. Instead they seemed to be going in all directions introducing new products but not improving the fundamental building blocks of the GPy. I was especially concerned when we purchased a new batch of chips and they came by default with a version of the OS which would not run our application for more than a few minutes without crashing. In addition Pycom seemed to be moving in a direction requiring our code in the future to be dependent on a cloud based system (Pybytes). This seemed to be making everything even more complex. We couldn't even get the simple things to work reliably.
In principle our application was not complex. We just needed to transmit a few packets of data to a web site periodically over the LTE network reliably without crashing the system. It was also desired that the communication would not interfere with the real time task of sampling data.
After looking at the C/C++ source code to the LTE module, it seemed way too complex. I had no good way to make changes, improvements, or bug fixes (assuming I could find the problem). If we moved the modem code to Python, we could bypass the usage of the LTE module and use the TCP/IP stack included in the Sequans modem. Maybe by pushing the communication load to the Sequans modem, the MicroPython would be more reliable and responsive to real time tasks. Maybe by publishing the code more people could observe and help to improve the code. We desperately needed a reliable method of communication.
Since converting our code to controlling the cell modem directly in Python code, I can now say the random crashes have been solved. The real time task is more responsive.
I hope that someday the LTE module will be improved so that it can be used again. Having a TCP/IP stack that is usable by both WIFI and LTE is certainly an advantage. Can it be made reliable? Can the ESP32 chip smoothly multitask while using LTE methods?
I wish I knew of a method to marry our higher level Python calls for modem management with the TCP/IP stack of MicroPython. I think such a technique could make the LTE module code simpler.
I think the dream of most application developers is to have simple reliable software methods to connect/disconnect to the network. The fact that we are connected using the cell network, WIFI, or whatever should be mostly transparent.
We all want Pycom to be successful. They have developed a good product. We did what we did because their priorities were not ours at the time.

Explore Pybytes | Official Documentation | Report a Firmware Bug/Issue | GitHub

Method To Take Control of Cell Modem

Pycom on Twitter