Method To Take Control of Cell Modem
-
@peterp Sorry to take so long to respond. I wanted to finish updating my posts showing the new code that seems to be working well. Soon I should be able to post the entire refined code. Maybe it will help others with the same problems we have had.
The reason I went down this development path was as described originally:
"I currently have a program working in production using Pycom MicroPython V1.20.0.rc13 on many units. When I try to run the same program on a newly purchased GPy running Pycom MicroPython 1.20.2.r3, the program crashes (20 sec watchdog timeout) when I do an AT command ('AT+CEREG?') after disconnecting from a successful transmission. How do I troubleshoot such a crash?"
Our original reason for picking Pycom and GPy was the thinking that all the modem complexity would be handled by the high level LTE module. It's only a guess but our thinking that since the crashes always happened during times communicating with the modem, the LTE module was the likely source of the problem.
The new method seems to be superior from the standpoint of interference with multiple tasks. Since we always call non-blocking uart commands, the code should never get hung in the C code for any period. A simple example of the LTE module code freezing multi-tasking is to do a simple modem reset. Reading and writing to the file storage also has a minor affect. The real reason for the development was to try to eliminate program crashes. Only longer term testing will show whether we have achieved that goal.
One concern I do have is whether I am properly initializing the cell modem. Any input from anybody out there that is more familiar with properly setting up the LTE modem would be appreciated. It seems to work fine at this point; however, I'm not sure whether it's working because the GPy I'm testing with has run the LTE initialization at some point in the past and that has possibly setup the non-volatile memory of the modem to the proper settings that are needed.
One drawback of using this method of transmission is it is not compatible with WIFI connections. That is not an issue for us at this time. Maybe later we can research how to do some form of communication that is compatible with both WIFI and direct AT commands. I assume that involves using the socket commands of the modem, but this is an area I'm not sure about at this time.
-
@tlanier Admire your efforts squire. I've been struggling with lte for a year or so now. It's not perfect (the lte.isattached() gets confused sometimes) but I got it robust enough to be usable. I use the requests library for data transfer & my main complaint is that it can't be timed out. I modified it with machine.WDT so that I can abort in a timely manner but I'd like a better solution. Am I right in thinking your wait_rx(n, 10000) allows you to quit if http post/get times out after 10s?
-
@kjm I think I remember having to modify urequests also. Here's what I currently use in my old version that is working.
import usocket import util class Response: def __init__(self, f): self.raw = f self.encoding = "utf-8" self._cached = None def close(self): if self.raw: self.raw.close() self.raw = None self._cached = None @property def content(self): if self._cached is None: try: self._cached = self.raw.read() finally: self.raw.close() self.raw = None return self._cached @property def text(self): return str(self.content, self.encoding) def json(self): import ujson return ujson.loads(self.content) def request(method, url, data=None, json=None, headers={}, stream=None): try: proto, dummy, host, path = url.split("/", 3) except ValueError: proto, dummy, host = url.split("/", 2) path = "" if proto == "http:": port = 80 elif proto == "https:": import ussl port = 443 else: raise ValueError("Unsupported protocol: " + proto) if ":" in host: host, port = host.split(":", 1) port = int(port) # ai = usocket.getaddrinfo(host, port, 0, usocket.SOCK_STREAM) # PATCH ai = usocket.getaddrinfo(host, port) ai = ai[0] s = usocket.socket(ai[0], ai[1], ai[2]) try: s.settimeout(20) s.connect(ai[-1]) if proto == "https:": s = ussl.wrap_socket(s, server_hostname=host) s.write(b"%s /%s HTTP/1.0\r\n" % (method, path)) if not "Host" in headers: s.write(b"Host: %s\r\n" % host) # Iterate over keys to avoid tuple alloc for k in headers: s.write(k) s.write(b": ") s.write(headers[k]) s.write(b"\r\n") if json is not None: assert data is None import ujson data = ujson.dumps(json) s.write(b"Content-Type: application/json\r\n") if data: s.write(b"Content-Length: %d\r\n" % len(data)) s.write(b"\r\n") if data: s.write(data) l = s.readline() #print(l) l = l.split(None, 2) status = int(l[1]) reason = "" if len(l) > 2: reason = l[2].rstrip() while True: l = s.readline() if not l or l == b"\r\n": break #print(l) if l.startswith(b"Transfer-Encoding:"): if b"chunked" in l: raise ValueError("Unsupported " + l) elif l.startswith(b"Location:") and not 200 <= status <= 299: raise NotImplementedError("Redirects not yet supported") # except OSError: # PATCH except: s.close() raise resp = Response(s) resp.status_code = status resp.reason = reason return resp def head(url, **kw): return request("HEAD", url, **kw) def get(url, **kw): return request("GET", url, **kw) def post(url, **kw): return request("POST", url, **kw) def put(url, **kw): return request("PUT", url, **kw) def patch(url, **kw): return request("PATCH", url, **kw) def delete(url, **kw): return request("DELETE", url, **kw) -
@kjm Thanks for the comment. The intention of wait_rx(n, timeout=10000) is to be a non-blocking call that waits for N chars to come into the uart or to timeout after 10,000 msecs by default. N should be less than the buffer size of the uart which is currently set to 512 bytes.
In looking at my current code, I see a bug. I should have had a sleep command in the while loop to make multi-tasking more efficient. Here's the corrected code. Thanks again!
##################################################################### # Wait For N Received Chars ##################################################################### def wait_rx(n, timeout=10000): # timeout in msecs t0 = ticks_ms() while True: if uart.any() >= n: break t1 = ticks_ms() tx = ticks_diff(t1, t0) if tx > timeout: return False sleep_ms(10) return True -
@kjm What would really be neat is if we could learn how to put the modem in a mode that allowed the urequests/usockets libraries to work using just AT commands!
I suspect it has to do with the AT+CGDCONT parameter named PDP_type. The manual shows the following values:
X.25 ITU-T/CCITT X.25 layer 3 (Obsolete)
IP Internet Protocol (IETF STD 5 [103])
IPV6 Internet Protocol, version 6 (see RFC 2460 [106])
IPV4V6 Virtual <PDP_type> introduced to handle dual IP stack UE capability. (See
3GPP TS 24.301 [83])
OSPIH Internet Hosted Octect Stream Protocol (Obsolete)
PPP Point to Point Protocol (IETF STD 51 [104])
Non-IP Transfer of Non-IP data to external packet data network (see 3GPP TS 23.401This is an area I'm unsure about. I would welcome comments from others about what would need to be done.
-
Here's updated code that is now running on two units overnight with good success. The code handles unsolicited responses gracefully. The code is non-intrusive to other multi-tasking threads. The code has not crashed (more testing to be done for sure). The code is easy to understand; everything is there to see and improve (no hidden commands being sent to the modem).
# MODEM.PY Module import globals as g from machine import UART import micropython import sys import util from util import log from utime import sleep_ms, ticks_ms, ticks_diff uart = None ##################################################################### # Send AT Command To Modem ##################################################################### def at(cmd, filterx=""): try: if uart.any() > 0: # get unsolicited modem response a = at_response() for s in a: at_process(s) log(" modem command: {}".format(cmd)) # send command to modem n = uart.write(cmd + '\r') if n != len(cmd) + 1: log("error sending command to modem: n = {}".format(n)) return [] a = at_response(filterx) return a except Exception as ex: sys.print_exception(ex) a = [] return a # return modem response lines in array ##################################################################### # Filter Modem Response ##################################################################### def at_filter(a, filterx): # a is passed as reference value if filterx == "": return # filter response n = len(filterx) i = len(a) while i > 0: i = i - 1 if a[i][0:n] != filterx: # remove lines that do not start with filter value at_process(a[i]) a.pop(i) ##################################################################### # Process Unsolicited Response ##################################################################### def at_process(s): if s == 'OK': pass else: log("unsolicited modem response: {}".format(s)) return ##################################################################### # Get AT Command Response ##################################################################### def at_response(filterx=""): try: t0 = ticks_ms() while True: if uart.any() > 0: # insure entire response has been received sleep_ms(50) break t1 = ticks_ms() tx = ticks_diff(t1, t0) if tx > 10000: log("timeout error - no modem response after 10 secs") return [] sleep_ms(100) b = uart.read() s = str(b, 'utf-8') a = s.split('\r\n') a = list(filter(None, a)) log(" modem response: {}".format(a)) at_filter(a, filterx) except Exception as ex: sys.print_exception(ex) a = [] return a # return modem response lines in array ##################################################################### # Attach Cellular Modem To Packet Domain Service ##################################################################### def attach(): try: log("attach modem to packet domain service") a = at('AT+CGATT=1','OK') if len(a) == 0: log("error attaching modem") except Exception as ex: sys.print_exception(ex) ##################################################################### # Check Registration ##################################################################### def check_registration(): # return 0-5, 99-unknown # response: ['+CEREG: 2,4', 'OK'] # response: ['+CEREG: 2,1,"2C29","0B54400F",7', 'OK'] # 2,0 - not registered, not searching # 2,1 - registered, home network # 2,2 - not registered, searching # 2,3 - registration denied # 2,4 - unknown (out of coverage) # 2,5 - registered, roaming try: while True: log("check registration") a = at('AT+CEREG?','+CEREG:') if len(a) == 0: break a = a[0].split(",") if (a[0] != "+CEREG: 2"): break reg = int(a[1]) if reg == 1: log("registered - home network") elif reg == 5: log("registered - roaming") else: log("not registered") return reg except Exception as ex: log("check registration exception") sys.print_exception(ex) # unknown registration log("error checking registration") return 99 ##################################################################### # Detach Cellular Modem From Packet Domain Service ##################################################################### def detach(): # modem does not get SMS messages while the modem is detached # old messages received while detached are only received when a new message is sent while attached try: log("detach modem from packet domain service") a = at('AT+CGATT=0','OK') if len(a) == 0: log("error detaching modem") except Exception as ex: sys.print_exception(ex) ##################################################################### # Disable LTE Modem ##################################################################### def disable_modem(): try: log("disable modem") a = at('AT+CFUN=0','OK') # minimum functionality if len(a) == 0: log("error disabling modem") except Exception as ex: sys.print_exception(ex) ##################################################################### # Enable Modem ##################################################################### def enable_modem(): try: log("enabling modem") a = at('AT+CFUN=1','OK') # full functionality if len(a) == 0: log("error enabling modem") except Exception as ex: sys.print_exception(ex) ##################################################################### # Get CFUN - Modem Level of Functionality ##################################################################### def get_cfun(): # 0 - minimum functionality # 1 - full functionality # 4 - radios disabled but access to SIM allowed # -1 - unknown # note: if modem does +SHUTDOWN/+SYSSTART, CFUN will be reset to 0 try: while True: a = at('AT+CFUN?','+CFUN:') # response: ['+CFUN: 1', 'OK'] if len(a) == 0: break cfun = int(a[0][7]) return cfun except Exception as ex: sys.print_exception(ex) return -1 ##################################################################### # Get SMS Mode ##################################################################### def get_sms_mode(): # return 0-PDU mode or unknown, 1-text mode try: log("get SMS mode") a = at('AT+CMGF?','+CMGF:') # response: ['+CMGF: 1', 'OK'] if len(a) == 0: return 0 # mode unknown mode = int(a[0][7]) return mode except Exception as ex: sys.print_exception(ex) return 0 # mode unknown ##################################################################### # HTTP Post ##################################################################### def http_post(uri, header, tx_data): # --> AT+SQNHTTPSND=<prof_id>,<command>,<resource>,<data_len>[,<post_param>[,<extra_header_line>]] # xx> POST <uri> HTTP/1.1 # xx> <header> # xx> Content-Length: <tx_size> # xx> Content-Type: application/octet-stream # xx> <blank line> # <-- '> ' # --> <tx_data bytes> # <-- OK # <-- +SQNHTTPRING: <prof_id,<http_status_code>,<content_type>,<rx_size> # --> AT+SQNHTTPRCV=1,0\r # <-- /r/n<<< # <-- <rx_data bytes> # <-- OK try: log("HTTP post") while True: # note: init_modem() sets configuration "AT+SQNHTTPCFG=..." at startup tx_size = len(tx_data) # log("writing message: {0}".format(tx_data)) log("tx_data length: {0}".format(tx_size)) # AT+SQNHTTPSND=<prof_id>,<command>,<resource>,<data_len>[,<post_param>[,<extra_header_line>]] # prof_id=1, command=0 (POST), <uri>, <tx_size>, post_parm="2" (application/octet-stream), header="Host: <host_name>" a = at('AT+SQNHTTPSND=1,0,"{0}",{1},"2","{2}"'.format(uri, tx_size, header),'> ') if len(a) == 0: break # error uart.write(tx_data) a = at_response('OK') if len(a) == 0: break # error # modem responds with "+SQNHTTPRING: <prof_id,<http_status_code>,<content_type>,<data_size>" a = at_response("+SQNHTTPRING:") if len(a) == 0: break # error a = a[0].split(",") http_status_code = int(a[1]) if http_status_code != 200: break # error rx_size = int(a[3]) log("rx_data length: {0}".format(rx_size)) # get response cmd = 'AT+SQNHTTPRCV=1,0\r' log(" modem command: {0}".format(cmd)) uart.write(cmd) r = wait_rx(5, 10000) # wait for b'\r\n<<<' if r == False: break # error b = uart.read(5) # log(" modem response: {0}".format(b)) if b[2:5] != b'<<<': break # error t0 = ticks_ms() b = b'' n = rx_size # no. bytes remaining to be read while True: if uart.any() > 0: log("attempt to read {0} bytes".format(n)) bx = uart.read(n) log("bytes read: {0}".format(len(bx))) # log(" modem response: {0}".format(bx)) b = b + bx if len(b) == rx_size: break # entire message received n = rx_size - len(b) t0 = ticks_ms() # reset timeout t1 = ticks_ms() tx = ticks_diff(t1, t0) if tx > 10000: log("timeout error receiving response") return b'' sleep_ms(400) log("total bytes read: {0}".format(len(b))) # log(" post response: {0}".format(b)) a = at_response('OK') if len(a) == 0: break # error # log(b) return b except Exception as ex: sys.print_exception(ex) # error return b'' ##################################################################### # Get SIM Card ICCID Number ##################################################################### def iccid(): # CFUN must be 1 or 4 try: log("get SIM card ICCID") a = at('AT+SQNCCID?','+SQNCCID:') # example response: ['+SQNCCID: "89014103271203065543",""', 'OK'] if len(a) == 0: return '' a = a[0].split(",") n = len(a[0]) r = a[0][11:n-1] except Exception as ex: sys.print_exception(ex) r = '' log("ICCID = {0}".format(r)) return r # ICCID as string ##################################################################### # Get Modem IMEI Number ##################################################################### def imei(): try: log("get modem IMEI number") a = at('AT+CGSN=1','+CGSN:') # example response: ['+CGSN: "354347094028575"', 'OK'] if len(a) == 0: return '' n = len(a[0]) r = a[0][8:n-1] except Exception as ex: sys.print_exception(ex) r = '' log("IMEI = {0}".format(r)) return r # IMEI as string ##################################################################### # Initialize Modem ##################################################################### def init_modem(): try: log("initializing modem") if uart == None: init_uart() disable_modem() # set profile #1 parameters for HTTP connection # prof_id=1 (profile), ip, port, auth_type=0 (None), username="", password="", # ssl_enabled=0, timeout, cid=1 (PDN Context Identifier) server = g.oIni.host_ip # note: host_name could also be used (auto DNS lookup) port = g.oIni.host_port timeout = 10 # secs to wait for response from server cmd = 'AT+SQNHTTPCFG=1,"{0}",{1},0,"","",0,{2},1'.format(server, port, timeout) a = at(cmd,'OK') if len(a) == 0: log("error initializing modem") set_sms_mode() enable_modem() except Exception as ex: sys.print_exception(ex) ##################################################################### # Initialize Uart Connected To LTE Modem ##################################################################### def init_uart(): global uart # pins=(TXD, RXD, RTS, CTS) # FiPy: pins=('P20', 'P18', 'P19', 'P17') # GPy: pins=('P5', 'P98', 'P7', 'P99') log("initializing UART") # rx_buffer_size: must be > 127, 200000 works, 250000 crashes, http_post seems to work with any size uart = UART(1, baudrate=921600, bits=8, parity=None, stop=1, timeout_chars=2, pins=('P5', 'P98', 'P7', 'P99'), rx_buffer_size=512) ##################################################################### # Is Cellular Modem Attached To Packet Domain Service ##################################################################### def isattached(): # returns: True/False try: a = at('AT+CGATT?','+CGATT:') # example response: ['+CGATT: 0', 'OK'] if len(a) == 0: return False if a[0][8] == '1': return True except Exception as ex: sys.print_exception(ex) return False ##################################################################### # Reset Modem ##################################################################### def reset_modem(): try: if uart == None: init_uart() log("reseting modem...") a = at('AT^RESET','+SHUTDOWN') # generates +SHUTDOWN ... +SYSSTART responses if len(a) == 0: log("error resetting modem - no +SHUTDOWN") a = at_response('+SYSSTART') if len(a) == 0: log("error resetting modem - no +SYSSTART") except Exception as ex: sys.print_exception(ex) ##################################################################### # Check Modem For Restart/Crash ##################################################################### def restarted(): # return True if modem has crashed / restarted try: # if CFUN != 1, modem has crashed / restarted cfun = get_cfun() if cfun != 1: return True # if SMS text mode is reset to zero, modem has crashed / restarted mode = get_sms_mode() if mode != 1: return True return False except Exception as ex: sys.print_exception(ex) return True ##################################################################### # Set APN ##################################################################### def set_apn(): try: log("set APN") # ICCID is only valid after registration r = iccid()[0:5] if r == "89014": apn = g.oIni.apn_89014 # Kore AT&T SIM elif r == "89012": apn = g.oIni.apn_89012 # Kore T-Mobile SIM elif r == "89445": apn = g.oIni.apn_89445 # Hologram SIM else: apn = g.oIni.apn_89014 # Kore AT&T SIM g.oIni.apn = apn log("APN: {0}".format(apn)) # try to reduce wear of non-volatile memory a = at('AT+CGDCONT?','+CGDCONT:') # response --> ['+CGDCONT: 1,"IP","10569.mcs",,,,0,0,0,0,0,0', ...] if len(a) > 0: a = a[0].split(",") if (a[0] == '+CGDCONT: 1') and (a[1] == '"IP"') and (a[2] == '"'+apn+'"'): return # apn needs setting cmd = 'AT+CGDCONT=1,"IP","{0}"'.format(apn) a = at(cmd,'OK') if len(a) == 0: log("error setting APN") except Exception as ex: sys.print_exception(ex) ##################################################################### # Set SMS Mode To Text ##################################################################### def set_sms_mode(): try: log("set SMS mode to text") a = at('AT+CMGF=1','OK') # set SMS mode to text if len(a) == 0: log("error setting SMS mode to text") except Exception as ex: sys.print_exception(ex) ##################################################################### # Get Signal Strength ##################################################################### def signal_strength(): # 0-31, 99-unknown try: while True: log("check signal strength") a = at('AT+CSQ','+CSQ:') # example response: ['+CSQ: 18,99', 'OK'] where 18 is signal strength if len(a) == 0: break n = len(a[0]) s = a[0][6:n-1] a = s.split(",") cell_signal = int(a[0]) g.oStatus.cell_signal = cell_signal log("signal strength = {0}".format(cell_signal)) return cell_signal except Exception as ex: sys.print_exception(ex) # unknown signal strength g.oStatus.cell_signal = 99 log("signal strength unknown") return 99 ##################################################################### # Delete Read, Sent, and Unsent SMS Messages ##################################################################### def sms_delete(): try: log("delete all READ, SENT, and UNSENT SMS messages") a = at('AT+CMGD=1,3','OK') if len(a) == 0: log("error trying to delete read, sent, and unsent SMS messages") except Exception as ex: sys.print_exception(ex) ##################################################################### # Delete All SMS Messages ##################################################################### def sms_delete_all(): # doesn't work until some time after attach() is called try: log("delete all SMS messages") a = at('AT+CMGD=1,4','OK') if len(a) == 0: log("error trying to delete all SMS messages") except Exception as ex: sys.print_exception(ex) ##################################################################### # Process SMS Messages ##################################################################### def sms_process(r): # example response: # ['+CMGL: 1,"REC READ","+1xxxxxxxxxx",,"19/01/07,09:16:53-20"', 'Test3', '+CMGL: 2,"REC READ","+1xxxxxxxxxx",,"19/01/07,11:12:30-20"', 'Test4', 'OK'] try: for m in r: if (m[0:5] == "+CMGL"): continue a = m.split("=") cmd = a[0].upper() except Exception as ex: sys.print_exception(ex) ##################################################################### # Read Unread SMS Messages ##################################################################### def sms_read(): # response example: ['+CMGL: 1,"REC UNREAD","+1xxxxxxxxxx",,"19/01/07,09:16:53-20"', 'Test3', 'OK'] log("read unread SMS messages") a = at('AT+CMGL="REC UNREAD"') return a ##################################################################### # Read All SMS Messages ##################################################################### def sms_read_all(): # response example: # ['+CMGL: 1,"REC READ","+1xxxxxxxxxx",,"19/01/07,09:16:53-20"', 'Test3', '+CMGL: 2,"REC READ","+1xxxxxxxxxx",,"19/01/07,11:12:30-20"', 'Test4', 'OK'] log("read all SMS messages") a = at('AT+CMGL="ALL"') return a ##################################################################### # Wait For N Received Chars ##################################################################### def wait_rx(n, timeout=10000): # timeout in msecs t0 = ticks_ms() while True: if uart.any() >= n: break t1 = ticks_ms() tx = ticks_diff(t1, t0) if tx > timeout: return False sleep_ms(10) return TrueHere's the code being used. The code shown is incomplete since it is application dependent. I'm somewhat disappointed in the necessary placement of the modem.apn_set() call. This may change in the future.
# COM.PY Module import globals as g import modem from poll import poll import pycom import sys import uos import util from util import log from utime import sleep_ms import wifi ##################################################################### # Communication Task ##################################################################### def com_task(): try: sleep_ms(500) log("com_task: started") wifi.deinit() modem.reset_modem() modem.init_modem() g.poll_host = True # triggers poll at startup except Exception as ex: sys.print_exception(ex) while True: try: if modem_ready(): if ready_to_poll(): modem.set_apn() transmit() sleep_ms(2000) except Exception as ex: sys.print_exception(ex) ##################################################################### # Modem Ready ##################################################################### def modem_ready(): try: log("** check for modem ready") # get modem signal strength g.oStatus.cell_signal = modem.signal_strength() # check modem registration r = modem.check_registration() if (r == 1) or (r == 5): # registered g.com_secs = 0 # handle SMS messages r = modem.sms_read() if len(r) > 1: modem.sms_process(r) modem.sms_delete() return True reset = False if g.com_secs > (10*60): # unable to attach in 10 mins, try resetting modem reset = True if modem.restarted(): # modem has crashed / restarted, try resetting modem reset = True if reset: modem.reset_modem() modem.init_modem() g.com_secs = 0 return False except Exception as ex: sys.print_exception(ex) return False -
I seemed to have hit a major problem using the AT+SQNHTTPSND modem command. Although the manual does not state this, the maximum size of the data posted is apparently 1500 bytes.
If anyone has ideas of how to increase this, please let me know. Unfortunately my app requires posting messages over 3000 bytes.
Disappointed...
-
I've now converted my code to use the Sequans modem socket commands in command mode (not on-line mode). I first tried to use the on-line mode, but ran into issues (no good way I could find to prevent overrunning tx buffer). This code resolves the 1500 byte limitation I ran into using the HTTP modem commands. There's still some cleanup to do to make the code more robust; however, it is working with large transmit and receive data blocks now.
During development while I had coding errors and the program would terminate abnormally, I did put the modem in a mode that I couldn't recover from without power cycling the GPy. The modem would not response to AT commands, Trying to do lte = LTE() would generate the following error (OSError: Couldn't connect to Modem (modem_state=disconnected)). Doing machine.reset() would not correct the problem.
Link to discussion of similar problem on Github
Hopefully whatever I did to cause the problem will not occur in debugged code. There needs to be a true hardware reset of the modem. It seems that a GPIO pin should be tied to a physical reset line going to the modem that we could pulse if needed. I'm also of the opinion that the watchdog timer in the GPy should have been a hardware circuit not dependent on software to work.
So far I have not seen any unexplained crashes while running the code.
This code has been added to my modem.py module.
##################################################################### # Socket Close ##################################################################### def socket_close(): # socket close # connID = 1 --> socket connection identifier a = at('AT+SQNSH=1', 'OK') if len(a) == 0: log("error closing connection") return ##################################################################### # Socket Connect ##################################################################### def socket_connect(ip, port, timeout_secs=10): # returns: True if connection successful log("connecting to ip - {0}, port - {1}".format(ip, port)) # make sure socket is closed socket_close() # socket configuration # connID = 1 --> socket connection identifier # cid = 1 --> PDP context identifier # pktSz = 1500 --> packet size # maxTo = 10 --> exchange timeout in secs # connTo = 600 --> connection timeout in msecs # txTo = 3 --> data sending timeout in hundreds of msecs # a = at('AT+SQNSCFG=1,1,1500,10,600,3', 'OK') a = at('AT+SQNSCFG=1,1,1500,{0},600,3'.format(timeout_secs), 'OK') if len(a) == 0: log("error connecting") return False # socket extended configuration # connID = 1 --> socket connection identifier # srMode = 1 --> SQNSRING URC data amount mode # recvDataMode = 1 --> data in hex # keepalive = 0 --> unused # listenAutoRsp = 0 --> deactivated # sendDataMode = 1 --> data in hex a = at('AT+SQNSCFGEXT=1,1,1,0,0,1', 'OK') if len(a) == 0: log("error connecting") return False # socket dial # connID = 1 --> socket connection identifier # txProt = 0 --> TCP protocol # rPort = <port> --> remote host port to contact # IPaddr = <ip> --> address of remote host # closureType = 0 --> local host closes immediately when remote host has closed # lPort = 0 --> UDP port (not used) # connMode = 1 --> command mode connection # acceptAnyRemote = 0 --> disabled a = at('AT+SQNSD=1,0,{0},"{1}",0,0,1,0'.format(port,ip), 'OK') if len(a) == 0: log("error connecting") return False log("** connection successful **") return True ##################################################################### # Socket Info ##################################################################### def socket_info(): # modem response: ['+SQNSI: 1,0,0,0,0', 'OK'] # connId: 1 # sent: total bytes sent # received: total bytes received # buff_in: total bytes not yet read (buffered) # ack_waiting: total bytes sent but not yet acknowledged a = at('AT+SQNSI=1') ##################################################################### # Socket Receive Data ##################################################################### def socket_rx(): # receives up to 1500 bytes maximum log("check for rx data") a = at('AT+SQNSRECV=1,1500') # no data ready --> modem response: ['+CME ERROR: operation not supported'] # data ready --> modem response: ['+SQNSRECV: 1,3', '010203', 'OK'] b = b'' for i in range(len(a)): if a[i].startswith('+SQNSRECV:'): # next line is the data b = unhexlify(a[i+1]) log("rx {0} bytes".format(len(b))) return b ##################################################################### # Socket Status ##################################################################### def socket_status(): # modem response: ['+SQNSS: 1,0', '+SQNSS: 2,0', '+SQNSS: 3,0', '+SQNSS: 4,0', '+SQNSS: 5,0', '+SQNSS: 6,0', 'OK'] # connID # state: # 0-socket closed # 1-socket with active data transfer connection # 2-socket suspended # 3-socket suspended with pending data # 4-socket listening # 5-socket with incoming connection # 6-socket in opening process a = at('AT+SQNSS?') return a ##################################################################### # Socket Transmit Data ##################################################################### def socket_tx(tx, n=0): # transmits up to 1500 bytes maximum # returns: True if successful if n == 0: n = len(tx) log("tx {0} bytes".format(n)) a = at('AT+SQNSSENDEXT=1,{0}'.format(n), '> ') hex = hexlify(tx) uart.write(hex) a = at_response('OK') if len(a) == 0: log("error transmitting data") return False return TrueThis code is in my http.py module.
import modem import sys import ubinascii import uos import ustruct import util from util import log from utime import sleep_ms, ticks_ms, ticks_diff BUFFER_SIZE = const(1500) CR = const(0x0d) LF = const(0x0a) MAX_HEADER_SIZE = const(128) MAX_TX_SIZE = const(1500) HTTP_TIMEOUT_MSECS = const(15000) # HTTP_TIMEOUT_MSECS = const(300000) # TESTING http_error = False # HTTP communication error: True/False http_size = 0 # HTTP POST message size rx_cnt = 0 # no. bytes in rx_data buffer rx_data = b'' # rx data buffer rx_pt = 0 # pointer into rx data buffer tx_cnt = 0 # no. bytes in tx_data buffer tx_data = bytearray(MAX_TX_SIZE) # tx data buffer tx_flag = False # controls tx_byte ##################################################################### # Close HTTP Socket ##################################################################### def close_http(): modem.socket_close() ##################################################################### # Open HTTP Socket ##################################################################### def open_http(host_ip, host_port): global http_error, rx_cnt, rx_data, rx_pt, tx_cnt http_error = False rx_cnt = 0 rx_data = b'' rx_pt = 0 tx_cnt = 0 r = modem.socket_connect(host_ip, host_port) if r == False: log("TCP socket connect error") http_error = True ##################################################################### # Receive BLOCK From Response ##################################################################### def rx_block(cnt): b = bytearray(cnt) i = 0 while i < cnt: b[i] = rx_byte() i = i + 1 return b ##################################################################### # Receive BYTE From Response ##################################################################### def rx_byte(): global http_error, http_size, rx_cnt, rx_data, rx_pt t0 = ticks_ms() while rx_cnt == 0: rx_data = modem.socket_rx() rx_cnt = len(rx_data) rx_pt = 0 if rx_cnt > 0: break t1 = ticks_ms() tx = ticks_diff(t1, t0) if (tx > HTTP_TIMEOUT_MSECS): http_error = True return 0 sleep_ms(200) b = rx_data[rx_pt] rx_cnt = rx_cnt - 1 rx_pt = rx_pt + 1 http_size = http_size - 1 # print("rx byte: {0}".format(hex(b))) return b ##################################################################### # Receive DWORD From Response ##################################################################### def rx_dword(): dw1 = rx_byte() dw2 = rx_byte() dw3 = rx_byte() dw4 = rx_byte() return (dw1 + (dw2<<8) + (dw3<<16) + (dw4<<24)) ##################################################################### # Receive One HTTP Header ##################################################################### def rx_http_header(): # returns header string global http_error n = 0 header = bytearray(MAX_HEADER_SIZE) s = "" while True: b = rx_byte() # print("byte = {0}".format(b)) if http_error: return "" if b == CR: continue if b == LF: break header[n] = b n = n + 1 if n >= MAX_HEADER_SIZE-1: http_error = True return "" if n > 0: # print("n = {0}, header = {1}".format(n,header)) s = header[0:n].decode('utf-8') return s ##################################################################### # Receive HTTP Headers # # HTTP/1.1 200 OK # <headers> # Content-Length: nnn # <headers> # <crlf> # # Updates: http_size, http_error ##################################################################### def rx_http_headers(): global http_error, http_size headers = [] # array of header lines http_size = 0 while True: s = rx_http_header() log("length = {0}, header = {1}".format(len(s), s)) if http_error: return headers if s == "": break headers.append(s) if len(headers) < 2: http_error = True return headers a = headers[0].split(' ') if (a[1] != "200") or (a[2]) != "OK": http_error = True return headers for s in headers: if s.startswith("Content-Length:"): a = s.split(' ') http_size = int(a[1]) return headers ##################################################################### # Receive String From Response ##################################################################### def rx_string(max_len): rx = bytearray(max_len) i = 0 while i < max_len: b = rx_byte() if b == 0: break rx[i] = b i = i + 1 if i == 0: return "" s = rx[0:i].decode('utf-8') return s ##################################################################### # Receive WORD From Response ##################################################################### def rx_word(): dw1 = rx_byte() dw2 = rx_byte() return (dw1 + (dw2<<8)) ##################################################################### # Transmit Block ##################################################################### def tx_block(b): for x in b: tx_byte(x) ##################################################################### # Transmit Byte ##################################################################### def tx_byte(b): global http_size, tx_cnt, tx_data, tx_flag if tx_flag == False: http_size = http_size + 1 # print("tx byte: {0}, http size: {1}".format(hex(b), http_size)) return # print("tx byte: {0}".format(hex(b))) tx_data[tx_cnt] = b tx_cnt = tx_cnt + 1 if tx_cnt >= MAX_TX_SIZE: tx_flush() ##################################################################### # Transmit DWORD ##################################################################### def tx_dword(dw): tx_word(dw & 0xffff) tx_word((dw >> 16) & 0xffff) ##################################################################### # Transmit Flush ##################################################################### def tx_flush(): global tx_cnt, tx_data if tx_cnt == 0: return r = modem.socket_tx(tx_data, tx_cnt) if not r: http_error = True return tx_cnt = 0 ##################################################################### # Transmit HTTP Headers ##################################################################### def tx_http_headers(http_post, host_name, host_port): # example: # POST <http_post> HTTP/1.1 # Host: <host_name>:<host_port> # Content-Length: <http_size> # Content-Type: application/octet-stream # <crlf> global http_error, http_size log("tx HTTP headers") if http_error: return s = "POST {0} HTTP/1.1\r\n".format(http_post) tx_string(s) if host_port != 80: s = "Host: {0}:{1}\r\n".format(host_name, host_port) else: s = "Host: {0}\r\n".format(host_name) tx_string(s) s = "Content-Length: {0}\r\n".format(http_size) tx_string(s) s = "Content-Type: application/octet-stream\r\n" tx_string(s) s = "\r\n" tx_string(s) ##################################################################### # Transmit String ##################################################################### def tx_string(s): b = s.encode('utf-8') for x in b: tx_byte(x) ##################################################################### # Transmit WORD ##################################################################### def tx_word(w): tx_byte(w & 0xff) tx_byte((w >> 8) & 0xff) -
We now have our program running completely independent of the LTE module. 26 units have been running for multiple days with no crashes. I now think we have found a reliable solution. The Sequans modem AT socket commands are used to do the HTTP posts. We reliably receive 90K files and transmit 1-5K of data with no issues. Pycom MicroPython V1.20.0.rc13 is the OS version we are using.
Currently we are using whatever version of the modem firmware (41065, 43818, and 47510 have been seen) that comes with the GPy. The version of the OS we are using has the ability to "downgrade" the modem firmware to 41065 if we decide it is needed. Some newer versions of the OS do not have this ability.
One note of importance is we did discover that in our case the default stack size was marginal when receiving large files. Yes, I did go back and check the old program and verified that the stack issue was not the cause of the original crash problems.
##################################################################### # Initialize Tasks ##################################################################### def init_tasks(): n = 8192 # 4096 is marginal _thread.stack_size(n) log("initializing tasks with stack size: {}".format(n)) _thread.start_new_thread(com.com_task, ()) _thread.start_new_thread(cycle.cycle_task, ()) -
@tlanier I really like the sound of what your doing here. I've been struggling with Pycom LTE module for months now. Really don't like the way it blocks other threads and randomly locks up for no apparent reason. Are you possibly in a position to share a complete "Hello LTE World" example. I'm sure I wouldn't be the only one around here who would be very interested. Hopefully someone from Pycom might also take an interest and finally admit they need to revise their code.
-
@ruok4t Sorry I didn't see your request earlier.
The software below is now working in production units. We ran about 20 units testing in house for two weeks with no crashes before beginning deployment with the new version. We are using MicroPython V1.20.0.rc13 with multiple versions of the modem firmware. It's a shame we had to resort to this, but it seems to work reliably.
I've had to break the post up into 3 parts because of limitations of this forum.
Part 1 of 3
Here's the modem.py module:
import globals as g import gpio from machine import UART import micropython import sys import util from ubinascii import hexlify, unhexlify from util import log, mem_info from utime import sleep_ms, ticks_ms, ticks_diff uart = None ##################################################################### # Send AT Command To Modem ##################################################################### def at(cmd, filterx=""): try: # util.mem_info("modem.at") if uart.any() > 0: # get unsolicited modem response a = at_response() for s in a: at_process(s) if gpio.G(): log(" modem command: {}".format(cmd)) # send command to modem n = uart.write(cmd + '\r') if n != len(cmd) + 1: if gpio.G(): log("error sending command to modem: n = {}".format(n)) return [] a = at_response(filterx) return a except Exception as ex: sys.print_exception(ex) a = [] return a # return modem response lines in array ##################################################################### # Filter Modem Response # ##################################################################### def at_filter(a, filterx): # a is passed as reference value # filter response if filterx == "": return n = len(filterx) i = len(a) while i > 0: i = i - 1 if not a[i].startswith(filterx): # remove lines that do not start with filter value at_process(a[i]) a.pop(i) ##################################################################### # Process Unsolicited Response ##################################################################### def at_process(s): if s == 'OK': return if s.startswith("ERROR"): return if s.startswith("+CME ERROR:"): return # process unsolicited response here if gpio.G(): log(" unsolicited modem response: {}".format(s)) ##################################################################### # Get AT Command Response ##################################################################### def at_response(filterx=""): try: t0 = ticks_ms() while True: if uart.any() > 0: # insure entire response has been received sleep_ms(50) break t1 = ticks_ms() tx = ticks_diff(t1, t0) if tx > 10000: log("timeout error - no modem response after 10 secs") return [] sleep_ms(100) b = uart.read() s = str(b, 'utf-8') a = s.split('\r\n') a = list(filter(None, a)) if gpio.G(): log(" modem response: {}".format(a)) at_filter(a, filterx) except Exception as ex: sys.print_exception(ex) a = [] return a # return modem response lines in array ##################################################################### # Check Registration ##################################################################### def check_registration(): # return 0-5, 99-unknown # response: ['+CEREG: 2,4', 'OK'] # response: ['+CEREG: 2,1,"2C29","0B54400F",7', 'OK'] # 2,0 - not registered, not searching # 2,1 - registered, home network # 2,2 - not registered, searching # 2,3 - registration denied # 2,4 - unknown (out of coverage) # 2,5 - registered, roaming try: while True: log("check registration") a = at('AT+CEREG?','+CEREG:') if len(a) == 0: break a = a[0].split(",") if (a[0] != "+CEREG: 2"): break reg = int(a[1]) if reg == 1: log("*** registered - home network ***") elif reg == 5: log("*** registered - roaming ***") else: log("*** NOT registered ***") return reg except Exception as ex: log("check registration exception") sys.print_exception(ex) # unknown registration log("error checking registration") return 99 ##################################################################### # Check SIM Status ##################################################################### def check_sim_status(): try: log("checking SIM card status") a = at('AT+CPIN?', '+CPIN: READY') if len(a) > 0: return True except Exception as ex: sys.print_exception(ex) log("error SIM card not ready") return False ##################################################################### # Disable LTE Modem ##################################################################### def disable_modem(): try: log("disable modem") a = at('AT+CFUN=0','OK') # minimum functionality if len(a) == 0: log("error disabling modem") except Exception as ex: sys.print_exception(ex) ##################################################################### # Enable Modem ##################################################################### def enable_modem(): try: log("enabling modem") a = at('AT+CFUN=1','OK') # full functionality if len(a) == 0: log("error enabling modem") except Exception as ex: sys.print_exception(ex) ##################################################################### # Get CFUN - Modem Level of Functionality ##################################################################### def get_cfun(): # 0 - minimum functionality # 1 - full functionality # 4 - radios disabled but access to SIM allowed # -1 - unknown # note: if modem does +SHUTDOWN/+SYSSTART, CFUN will be reset to 0 try: while True: a = at('AT+CFUN?','+CFUN:') # response: ['+CFUN: 1', 'OK'] if len(a) == 0: break cfun = int(a[0][7]) return cfun except Exception as ex: sys.print_exception(ex) return -1 ##################################################################### # Get SMS Mode ##################################################################### def get_sms_mode(): # return 0-PDU mode or unknown, 1-text mode try: log("get SMS mode") a = at('AT+CMGF?','+CMGF:') # response: ['+CMGF: 1', 'OK'] if len(a) == 0: return 0 # mode unknown mode = int(a[0][7]) return mode except Exception as ex: sys.print_exception(ex) return 0 # mode unknown ##################################################################### # Get SIM Card ICCID Number ##################################################################### def iccid(): # CFUN must be 1 or 4 # SIM status must be ready try: log("get SIM card ICCID") a = at('AT+SQNCCID?', '+SQNCCID:') # example response: ['+SQNCCID: "89014103271203065543",""', 'OK'] if len(a) == 0: return '' a = a[0].split(",") n = len(a[0]) r = a[0][11:n-1] except Exception as ex: sys.print_exception(ex) r = '' log("ICCID = {0}".format(r)) return r # ICCID as string ##################################################################### # Get Modem IMEI Number ##################################################################### def imei(): try: log("get modem IMEI number") a = at('AT+CGSN=1','+CGSN:') # example response: ['+CGSN: "354347094028575"', 'OK'] if len(a) == 0: return '' n = len(a[0]) r = a[0][8:n-1] except Exception as ex: sys.print_exception(ex) r = '' log("IMEI = {0}".format(r)) return r # IMEI as string ##################################################################### # Modem Firmware Info ##################################################################### def info(): log("get modem firmware version") a = at('ATI1') return a ##################################################################### # Initialize Modem ##################################################################### def init_modem(): try: log("initializing modem") if uart == None: init_uart() info() set_sms_mode() # enable +CME ERROR: <err> result code and use verbose <err> values at('AT+CMEE=2') enable_modem() log("wait for SIM card ready") n = 0 while True: log("checking SIM card status") a = at('AT+CPIN?', '+CPIN: READY') if len(a) > 0: break n = n + 1 if n > 20: util.reset(13, "No SIM card") sleep_ms(200) set_apn() except Exception as ex: sys.print_exception(ex) ##################################################################### # Initialize Uart Connected To LTE Modem ##################################################################### def init_uart(): global uart # pins=(TXD, RXD, RTS, CTS) # FiPy: pins=('P20', 'P18', 'P19', 'P17') # GPy: pins=('P5', 'P98', 'P7', 'P99') log("initializing UART") # rx_buffer_size: must be > 127, 200000 works, 250000 crashes uart = UART(1, baudrate=921600, bits=8, parity=None, stop=1, timeout_chars=2, pins=('P5', 'P98', 'P7', 'P99'), rx_buffer_size=512) ##################################################################### # Is Cellular Modem Attached To Packet Domain Service ##################################################################### def isattached(): # returns: True/False try: a = at('AT+CGATT?','+CGATT:') # example response: ['+CGATT: 0', 'OK'] if len(a) == 0: return False if a[0][8] == '1': return True except Exception as ex: sys.print_exception(ex) return False ##################################################################### # Reset Modem ##################################################################### def reset_modem(): try: if uart == None: init_uart() log("reseting modem...") a = at('AT^RESET', '+SHUTDOWN') # generates +SHUTDOWN ... +SYSSTART responses if len(a) == 0: log("error resetting modem - no +SHUTDOWN") a = at_response('+SYSSTART') if len(a) == 0: log("error resetting modem - no +SYSSTART") except Exception as ex: sys.print_exception(ex) ##################################################################### # Check Modem For Restart/Crash ##################################################################### def restarted(): # return True if modem has crashed / restarted try: # if CFUN != 1, modem has crashed / restarted cfun = get_cfun() if cfun != 1: return True # if SMS text mode is reset to zero, modem has crashed / restarted mode = get_sms_mode() if mode != 1: return True return False except Exception as ex: sys.print_exception(ex) return True ##################################################################### # Set APN ##################################################################### def set_apn(): try: log("set APN") # ICCID is only valid if SIM status is ready r = iccid()[0:5] if r == "89014": apn = g.oIni.apn_89014 # Kore AT&T SIM elif r == "89012": apn = g.oIni.apn_89012 # Kore T-Mobile SIM elif r == "89445": apn = g.oIni.apn_89445 # Hologram SIM else: apn = g.oIni.apn_89014 # Kore AT&T SIM g.oIni.apn = apn log("APN: {0}".format(apn)) # try to reduce wear of non-volatile memory a = at('AT+CGDCONT?', '+CGDCONT:') # response --> ['+CGDCONT: 1,"IP","10569.mcs",,,,0,0,0,0,0,0', ...] if len(a) > 0: a = a[0].split(",") if (a[0] == '+CGDCONT: 1') and (a[1] == '"IP"') and (a[2] == '"'+apn+'"'): return # apn needs setting cmd = 'AT+CGDCONT=1,"IP","{0}"'.format(apn) a = at(cmd, 'OK') if len(a) == 0: log("error setting APN") except Exception as ex: sys.print_exception(ex) ##################################################################### # Set SMS Mode To Text ##################################################################### def set_sms_mode(): try: log("set SMS mode to text") a = at('AT+CMGF=1', 'OK') # set SMS mode to text if len(a) == 0: log("error setting SMS mode to text") except Exception as ex: sys.print_exception(ex) ##################################################################### # Get Signal Strength ##################################################################### def signal_strength(): # 0-31, 99-unknown try: while True: log("check signal strength") a = at('AT+CSQ', '+CSQ:') # example response: ['+CSQ: 18,99', 'OK'] where 18 is signal strength if len(a) == 0: break n = len(a[0]) s = a[0][6:n-1] a = s.split(",") cell_signal = int(a[0]) g.oStatus.cell_signal = cell_signal log("signal strength = {0}".format(cell_signal)) return cell_signal except Exception as ex: sys.print_exception(ex) # unknown signal strength g.oStatus.cell_signal = 99 log("signal strength unknown") return 99 ##################################################################### # Delete Read, Sent, and Unsent SMS Messages ##################################################################### def sms_delete(): try: log("delete all READ, SENT, and UNSENT SMS messages") a = at('AT+CMGD=1,3', 'OK') if len(a) == 0: log("error trying to delete read, sent, and unsent SMS messages") except Exception as ex: sys.print_exception(ex) ##################################################################### # Delete All SMS Messages ##################################################################### def sms_delete_all(): # doesn't work until some time after attach() is called try: log("delete all SMS messages") a = at('AT+CMGD=1,4', 'OK') if len(a) == 0: log("error trying to delete all SMS messages") except Exception as ex: sys.print_exception(ex) ##################################################################### # Process SMS Messages ##################################################################### def sms_process(r): # example response: # ['+CMGL: 1,"REC READ","+1xxxxxxxxxx",,"19/01/07,09:16:53-20"', 'Test3', '+CMGL: 2,"REC READ","+1xxxxxxxxxx",,"19/01/07,11:12:30-20"', 'Test4', 'OK'] try: for m in r: if (m[0:5] == "+CMGL"): continue a = m.split("=") cmd = a[0].upper() # process message here except Exception as ex: sys.print_exception(ex) ##################################################################### # Read Unread SMS Messages ##################################################################### def sms_read(): # response example: ['+CMGL: 1,"REC UNREAD","+1xxxxxxxxxx",,"19/01/07,09:16:53-20"', 'Test3', 'OK'] log("read unread SMS messages") a = at('AT+CMGL="REC UNREAD"') return a ##################################################################### # Read All SMS Messages ##################################################################### def sms_read_all(): # response example: # ['+CMGL: 1,"REC READ","+1xxxxxxxxxx",,"19/01/07,09:16:53-20"', 'Test3', '+CMGL: 2,"REC READ","+1xxxxxxxxxx",,"19/01/07,11:12:30-20"', 'Test4', 'OK'] log("read all SMS messages") a = at('AT+CMGL="ALL"') return a ##################################################################### # Socket Close ##################################################################### def socket_close(): # socket close # connID = 1 --> socket connection identifier log("closing socket") a = at('AT+SQNSH=1', 'OK') if len(a) == 0: log("error closing connection") ##################################################################### # Socket Connect ##################################################################### def socket_connect(ip, port, timeout_secs=10): # returns: True if connection successful log("connecting to ip - {0}, port - {1}".format(ip, port)) # make sure socket is closed socket_close() # socket configuration # connID = 1 --> socket connection identifier # cid = 1 --> PDP context identifier # pktSz = 1500 --> packet size # maxTo = 10 --> exchange timeout in secs # connTo = 600 --> connection timeout in msecs # txTo = 3 --> data sending timeout in hundreds of msecs # a = at('AT+SQNSCFG=1,1,1500,10,600,3', 'OK') a = at('AT+SQNSCFG=1,1,1500,{0},600,3'.format(timeout_secs), 'OK') if len(a) == 0: log("error connecting") return False # socket extended configuration # connID = 1 --> socket connection identifier # srMode = 1 --> SQNSRING URC data amount mode # recvDataMode = 1 --> data in hex # keepalive = 0 --> unused # listenAutoRsp = 0 --> deactivated # sendDataMode = 1 --> data in hex a = at('AT+SQNSCFGEXT=1,1,1,0,0,1', 'OK') if len(a) == 0: log("error connecting") return False # socket dial # connID = 1 --> socket connection identifier # txProt = 0 --> TCP protocol # rPort = <port> --> remote host port to contact # IPaddr = <ip> --> address of remote host # closureType = 0 --> local host closes immediately when remote host has closed # lPort = 0 --> UDP port (not used) # connMode = 1 --> command mode connection # acceptAnyRemote = 0 --> disabled a = at('AT+SQNSD=1,0,{0},"{1}",0,0,1,0'.format(port,ip), 'OK') if len(a) == 0: log("error connecting") return False log("** connection successful **") return True ##################################################################### # Socket Info ##################################################################### def socket_info(): # modem response: ['+SQNSI: 1,0,0,0,0', 'OK'] # connId: 1 # sent: total bytes sent # received: total bytes received # buff_in: total bytes not yet read (buffered) # ack_waiting: total bytes sent but not yet acknowledged log("socket info") a = at('AT+SQNSI=1') ##################################################################### # Socket Receive Data ##################################################################### def socket_rx(): # receives up to 1500 bytes maximum log("check for rx data") a = at('AT+SQNSRECV=1,1500') # no data ready --> modem response: ['+CME ERROR: operation not supported'] # data ready --> modem response: ['+SQNSRECV: 1,3', '010203', 'OK'] b = b'' for i in range(len(a)): if a[i].startswith('+SQNSRECV:'): # next line is the data b = unhexlify(a[i+1]) log("rx {0} bytes".format(len(b))) return b ##################################################################### # Socket Status ##################################################################### def socket_status(): # modem response: ['+SQNSS: 1,0', '+SQNSS: 2,0', '+SQNSS: 3,0', '+SQNSS: 4,0', '+SQNSS: 5,0', '+SQNSS: 6,0', 'OK'] # connID # state: # 0-socket closed # 1-socket with active data transfer connection # 2-socket suspended # 3-socket suspended with pending data # 4-socket listening # 5-socket with incoming connection # 6-socket in opening process log("socket status") a = at('AT+SQNSS?') return a ##################################################################### # Socket Transmit Data ##################################################################### def socket_tx(tx, n=0): # transmits up to 1500 bytes maximum # returns: True if successful if n == 0: n = len(tx) log("tx {0} bytes".format(n)) a = at('AT+SQNSSENDEXT=1,{0}'.format(n), '> ') b = hexlify(tx) uart.write(b) a = at_response('OK') if len(a) == 0: log("error transmitting data") return False return True -
@tlanier
Part 2 of 3Here's our com.py module. It is more application dependent but it may give you some ideas of how to use the modem module.
import globals as g import modem from poll import poll import pycom import sys import uos import util from util import log from utime import sleep_ms import wifi _reset = True # modem reset ##################################################################### # Communication Task ##################################################################### def com_task(): global _reset try: sleep_ms(500) log("com_task: started") g.poll_host = True # triggers poll at startup # this opens the door for continuous resets to cause high data usage util.set_ppp_errors(0) wifi.deinit() except Exception as ex: sys.print_exception(ex) while True: try: _reset = False g.modem_state = g.MS_RESET_MODEM modem.reset_modem() g.modem_state = g.MS_INIT_MODEM modem.init_modem() while _reset == False: # util.mem_info("com_task") if modem_ready(): g.modem_state = g.MS_REGISTERED if ready_to_poll(): transmit() else: g.modem_state = g.MS_REGISTERING sleep_ms(2000) except Exception as ex: sys.print_exception(ex) ##################################################################### # Modem Ready ##################################################################### def modem_ready(): global _reset try: log("** check for modem ready") # get modem signal strength g.oStatus.cell_signal = modem.signal_strength() # check modem registration r = modem.check_registration() if (r == 1) or (r == 5): # registered g.com_secs = 0 # handle SMS messages r = modem.sms_read() if len(r) > 1: modem.sms_process(r) modem.sms_delete() return True if g.com_secs > (10*60): # unable to attach in 10 mins, try resetting modem _reset = True if modem.restarted(): # modem has crashed / restarted, try resetting modem _reset = True return False except Exception as ex: sys.print_exception(ex) return False ##################################################################### # Ready To Poll ##################################################################### def ready_to_poll(): # return: True/False try: log("--- time since last poll: {0} secs ---".format(g.oNvram.last_poll_secs)) if (g.ppp_error_secs > 0): log("*** poll throttling: {0} secs ***".format(g.ppp_error_secs)) return False if (g.oIni.intra_poll_secs > 0): # contact server every <oIni.intra_poll_secs> seconds if (g.oNvram.last_poll_secs > g.oIni.intra_poll_secs): return True if (g.oSetup.intra_cycle_count > 0): # contact server every <oSetup.intra_cycle_count> cycles if (g.oNvram.cycle_count >= g.oSetup.intra_cycle_count): return True # g.poll_host is set at startup and by SMS POLL_HOST message if (g.poll_host == True): return True except Exception as ex: sys.print_exception(ex) return False ##################################################################### # Transmit ##################################################################### def transmit(): try: util.lcd("Transmitting", "") g.modem_state = g.MS_CONNECTING r = poll() g.modem_state = g.MS_DISCONNECT if (r == False): util.lcd("Transmit failed", "") sleep_ms(2500) util.lcd("","") return util.lcd("Successful", "") sleep_ms(2500) util.lcd("","") g.poll_host = False g.oNvram.last_poll_secs = 0 g.poll_secs = 0 if (g.update_flag == True): # done after disconnect try: if util.file_exists("REMOTE.TAR"): log("remove old REMOTE.TAR") util.remove("REMOTE.TAR") log("rename TEMP to REMOTE.TAR") uos.rename("TEMP", "REMOTE.TAR") # create file to signal boot.py that update is available log("create UPDATE signal file") with open("UPDATE", "w") as f: f.write("UPDATE") log("** New REMOTE.TAR ready to update") except Exception as ex: sys.print_exception(ex) if (g.reboot_after_poll == True): log("SMS message changed INI setting, rebooting") util.reset(6, "INI changed") if (g.oHost.erase == True): log("Erase command received, rebooting") g.oNvram.erase() util.reset(7, "Erase NVRAM") if (g.oHost.reset == True): log("Reset command received, rebooting") util.reset(8, "Reset command") except Exception as ex: sys.print_exception(ex) -
@tlanier
Part 3 of 3Here's the http.py module:
Note that the modem socket commands have a 1500 byte limitation. The HTTP module overcomes that by breaking longer messages up into smaller blocks.
import modem import sys import ubinascii import uos import ustruct import util from util import log from utime import sleep_ms, ticks_ms, ticks_diff # note: HTTP errors raise exceptions BUFFER_SIZE = const(1500) CR = const(0x0d) LF = const(0x0a) MAX_HEADER_SIZE = const(128) MAX_TX_SIZE = const(1500) HTTP_TIMEOUT_MSECS = const(15000) # HTTP_TIMEOUT_MSECS = const(300000) # TESTING http_size = 0 # HTTP POST message size rx_cnt = 0 # no. bytes in rx_data buffer rx_data = b'' # rx data buffer rx_pt = 0 # pointer into rx data buffer tx_cnt = 0 # no. bytes in tx_data buffer tx_data = bytearray(MAX_TX_SIZE) # tx data buffer tx_flag = False # controls tx_byte ##################################################################### # Close HTTP Socket ##################################################################### def close_http(): modem.socket_close() ##################################################################### # Open HTTP Socket ##################################################################### def open_http(host_ip, host_port): global rx_cnt, rx_data, rx_pt, tx_cnt rx_cnt = 0 rx_data = b'' rx_pt = 0 tx_cnt = 0 r = modem.socket_connect(host_ip, host_port) if r == False: raise Exception("TCP socket connect error") ##################################################################### # Receive BLOCK From Response ##################################################################### def rx_block(cnt): b = bytearray(cnt) i = 0 while i < cnt: b[i] = rx_byte() i = i + 1 log("rx_block: {} bytes".format(len(b))) return b ##################################################################### # Receive BYTE From Response ##################################################################### def rx_byte(): global http_size, rx_cnt, rx_data, rx_pt t0 = ticks_ms() while rx_cnt == 0: rx_data = modem.socket_rx() rx_cnt = len(rx_data) rx_pt = 0 if rx_cnt > 0: break t1 = ticks_ms() tx = ticks_diff(t1, t0) if (tx > HTTP_TIMEOUT_MSECS): raise Exception("HTTP timeout in rx_byte") sleep_ms(200) b = rx_data[rx_pt] rx_cnt = rx_cnt - 1 rx_pt = rx_pt + 1 http_size = http_size - 1 # print("rx byte: {0}".format(hex(b))) return b ##################################################################### # Receive DWORD From Response ##################################################################### def rx_dword(): dw1 = rx_byte() dw2 = rx_byte() dw3 = rx_byte() dw4 = rx_byte() return (dw1 + (dw2<<8) + (dw3<<16) + (dw4<<24)) ##################################################################### # Receive One HTTP Header ##################################################################### def rx_http_header(): # returns header string n = 0 header = bytearray(MAX_HEADER_SIZE) s = "" while True: b = rx_byte() # print("byte = {0}".format(b)) if b == CR: continue if b == LF: break header[n] = b n = n + 1 if n >= MAX_HEADER_SIZE-1: raise Exception("Maximum header size error in rx_http_header") if n > 0: # older versions of MicroPython do not implement decode on bytearrays a = bytes(header[0:n]) s = a.decode('utf-8') log("header = {}".format(s)) return s ##################################################################### # Receive HTTP Headers # # HTTP/1.1 200 OK # <headers> # Content-Length: nnn # <headers> # <crlf> # # Updates: http_size ##################################################################### def rx_http_headers(): global http_size headers = [] # array of header lines http_size = 0 while True: s = rx_http_header() # log("length = {0}, header = {1}".format(len(s), s)) if s == "": break headers.append(s) if len(headers) < 2: raise Exception("Not enough headers received in rx_http_headers") a = headers[0].split(' ') if (a[1] != "200") or (a[2]) != "OK": raise Exception("Invalid first header line: {0}".format(headers[0])) for s in headers: if s.startswith("Content-Length:"): a = s.split(' ') http_size = int(a[1]) return headers ##################################################################### # Receive String From Response ##################################################################### def rx_string(max_len): rx = bytearray(max_len) i = 0 while i < max_len: b = rx_byte() if b == 0: break rx[i] = b i = i + 1 if i == 0: return "" # older versions of MicroPython do not implement decode on bytearrays a = bytes(rx[0:i]) s = a.decode('utf-8') return s ##################################################################### # Receive WORD From Response ##################################################################### def rx_word(): dw1 = rx_byte() dw2 = rx_byte() return (dw1 + (dw2<<8)) ##################################################################### # Transmit Block ##################################################################### def tx_block(b): for x in b: tx_byte(x) ##################################################################### # Transmit Byte ##################################################################### def tx_byte(b): global http_size, tx_cnt, tx_data, tx_flag if tx_flag == False: http_size = http_size + 1 # print("tx byte: {0}, http size: {1}".format(hex(b), http_size)) return # print("tx byte: {0}".format(hex(b))) tx_data[tx_cnt] = b tx_cnt = tx_cnt + 1 if tx_cnt >= MAX_TX_SIZE: tx_flush() ##################################################################### # Transmit DWORD ##################################################################### def tx_dword(dw): tx_word(dw & 0xffff) tx_word((dw >> 16) & 0xffff) ##################################################################### # Transmit Flush ##################################################################### def tx_flush(): global tx_cnt, tx_data if tx_cnt == 0: return r = modem.socket_tx(tx_data, tx_cnt) if not r: raise Exception("Error transmitting in tx_flush") tx_cnt = 0 ##################################################################### # Transmit HTTP Headers ##################################################################### def tx_http_headers(http_post, host_name, host_port): # example: # POST <http_post> HTTP/1.1 # Host: <host_name>:<host_port> # Content-Length: <http_size> # Content-Type: application/octet-stream # <crlf> global http_size log("tx HTTP headers") s = "POST {0} HTTP/1.1\r\n".format(http_post) tx_string(s) if host_port != 80: s = "Host: {0}:{1}\r\n".format(host_name, host_port) else: s = "Host: {0}\r\n".format(host_name) tx_string(s) s = "Content-Length: {0}\r\n".format(http_size) tx_string(s) s = "Content-Type: application/octet-stream\r\n" tx_string(s) s = "\r\n" tx_string(s) ##################################################################### # Transmit String ##################################################################### def tx_string(s): b = s.encode('utf-8') for x in b: tx_byte(x) ##################################################################### # Transmit WORD ##################################################################### def tx_word(w): tx_byte(w & 0xff) tx_byte((w >> 8) & 0xff) -
@tlanier Thanks, will download all parts and take a look.
-
@tlanier
Thank you for compiling all this information for the community. I really appreciate it. It could become handy for my project. -
@tlanier did you ever put this in a repo?
-
@Scott-Basgaard Sorry, I have not.
-
@tlanier Hey, I started looking into the three files you posted (http.py, com.py, modem.py) and it is looking very usefull! However, both com.py and modem.py have a line:
"import globals as g"
and I'm guessing it is a file you have, but I get an "ImportError: no module named 'globals'"Perhaps you could upload that file too? After a quick look throuh all "g.*" I could probably deduce most of it, but things like :
g.modem_state = g.MS_RESET_MODEM
would probably be usefull and save us some time.
Anyway, Thanks for helping the community at large with your efforts!!
Cheers,
Petteri -
@Petteri-Salo agree! Would be great @tlanier if you could share anything else here. Really struggling to get LTE working stable and any further insight for us others would be greatly appreciated. Digging into this more now and will let you know how far I get as well. Thanks for sharing!
-
@Petteri-Salo The g.modem_state global variable is used in a timer interrupt routine to control the blinking and color of the LED on the GPY. This gives the user a visual indication of what state the modem is in. This variable is not critical to using the supplied code.
Unfortunately the code I have published is just snippets of my production code. It was not designed to be a general purpose library (although I believe it comes close). Without publishing the entire application, there will be pieces that are missing.
Your message has motivated me to add a few random notes.
-
We had production, shipping GPy based systems that were experiencing reliability issues. Upgrades to newer versions of the OS seemed to make the problems worse. There was considerable evidence that the problems were caused by the LTE module. For months I waited hoping that Pycom would make the existing product more robust. Instead they seemed to be going in all directions introducing new products but not improving the fundamental building blocks of the GPy. I was especially concerned when we purchased a new batch of chips and they came by default with a version of the OS which would not run our application for more than a few minutes without crashing. In addition Pycom seemed to be moving in a direction requiring our code in the future to be dependent on a cloud based system (Pybytes). This seemed to be making everything even more complex. We couldn't even get the simple things to work reliably.
-
In principle our application was not complex. We just needed to transmit a few packets of data to a web site periodically over the LTE network reliably without crashing the system. It was also desired that the communication would not interfere with the real time task of sampling data.
-
After looking at the C/C++ source code to the LTE module, it seemed way too complex. I had no good way to make changes, improvements, or bug fixes (assuming I could find the problem). If we moved the modem code to Python, we could bypass the usage of the LTE module and use the TCP/IP stack included in the Sequans modem. Maybe by pushing the communication load to the Sequans modem, the MicroPython would be more reliable and responsive to real time tasks. Maybe by publishing the code more people could observe and help to improve the code. We desperately needed a reliable method of communication.
-
Since converting our code to controlling the cell modem directly in Python code, I can now say the random crashes have been solved. The real time task is more responsive.
-
I hope that someday the LTE module will be improved so that it can be used again. Having a TCP/IP stack that is usable by both WIFI and LTE is certainly an advantage. Can it be made reliable? Can the ESP32 chip smoothly multitask while using LTE methods?
-
I wish I knew of a method to marry our higher level Python calls for modem management with the TCP/IP stack of MicroPython. I think such a technique could make the LTE module code simpler.
-
I think the dream of most application developers is to have simple reliable software methods to connect/disconnect to the network. The fact that we are connected using the cell network, WIFI, or whatever should be mostly transparent.
-
We all want Pycom to be successful. They have developed a good product. We did what we did because their priorities were not ours at the time.
-
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login