OTA or not OTA that is the question...
-
After reading documentation and forum threads and done some trying I am not able to say if WiPy is (will be) able to manage full OTA for firmware update and for user program.
I am looking for same easy to use OTA functions I found with Particle but without dependencies to internet connection.
For OTA firmware update it seams to be not possible for now despite the fact that it was documented. I tried without success... Will it be possible one day or not ?
For OTA user program update it should be necessary to have a hardware reset (cf. If your board is running code at boot time, you might need to boot it in safe mode.). So no really OTA facility.
Could we have clear answers on these two points as they are main reasons to use or not Py devices.
Thanks
-
@robert-hh Thanks. That version worked perfectly and the simple OTA script from the earlier post worked great too!
-
@mfallavol I have a ftplib here that works for me. But I tested that only locally with numeric IP addresses. Version attached (still not possible to upload a zip file).
# -*- coding: utf-8 -*- """An FTP client class and some helper functions. Based on RFC 959: File Transfer Protocol (FTP), by J. Postel and J. Reynolds Example:: >>> from ftplib import FTP >>> ftp = FTP('ftp.python.org') # connect to host, default port >>> ftp.login() # default, i.e.: user anonymous, passwd anonymous@ '230 Guest login ok, access restrictions apply.' >>> ftp.dir() # list directory contents total 9 drwxr-xr-x 8 root wheel 1024 Jan 3 1994 . drwxr-xr-x 8 root wheel 1024 Jan 3 1994 .. drwxr-xr-x 2 root wheel 1024 Jan 3 1994 bin drwxr-xr-x 2 root wheel 1024 Jan 3 1994 etc d-wxrwxr-x 2 ftp wheel 1024 Sep 5 13:43 incoming drwxr-xr-x 2 root wheel 1024 Nov 17 1993 lib drwxr-xr-x 6 1094 wheel 1024 Sep 13 19:07 pub drwxr-xr-x 3 root wheel 1024 Jan 3 1994 usr -rw-r--r-- 1 root root 312 Aug 1 1994 welcome.msg '226 Transfer complete.' >>> ftp.quit() '221 Goodbye.' >>> """ # Changes and improvements suggested by Steve Majewski. # Modified by Jack to work on the mac. # Modified by Siebren to support docstrings and PASV. # Modified by Phil Schwartz to add storbinary and storlines callbacks. # Modified by Giampaolo Rodola' to add TLS support. # Modified, stripped down and cleaned up by Christopher Arndt for MicroPython import usocket as _socket import gc # Magic number from <socket.h> # Process data out of band MSG_OOB = 0x1 # Line terminators (we always output CRLF, but accept any of CRLF, CR, LF) CRLF = b'\r\n' # The standard FTP server control port FTP_PORT = const(21) # Range of possible client source ports for active transfers MIN_PORT = const(40001) MAX_PORT = const(40100) # The sizehint parameter passed to readline() calls MAXLINE = const(1024) BLOCKSIZE = const(1024) _GLOBAL_DEFAULT_TIMEOUT = object() # For compatibility with CPython version with SSL support _SSLSocket = None # Exception raised when an error or invalid response is received class Error(Exception): """Base FTP exception.""" pass def _resolve_addr(addr): if isinstance(addr, (bytes, bytearray)): return addr if not addr[0]: host = "127.0.0.1" if len(addr) == 2 else "::1" else: host = addr[0] return _socket.getaddrinfo(host, addr[1]) class socket: def __init__(self, *args, **kw): if args and isinstance(args[0], _socket.socket): self._sock = args[0] else: self._sock = _socket.socket(*args, **kw) def accept(self): s, addr = self._sock.accept() return self.__class__(s), addr def sendall(self, *args): return self._sock.send(*args) def __getattr__(self, name): return getattr(self._sock, name) def __enter__(self): return self def __exit__(self, *args): self._sock.close() # The main class itself class FTP: """An FTP client class. To create a connection, call the class using these arguments:: host, port, user, passwd, acct, timeout, source_address The host, user, passwd and acct arguments are all strings, while port is an integer. The default value for all is None, which means the following defaults will be used: host: localhost, port: 21, user: 'anonymous', passwd: anonymous@', acct: '' timeout must be numeric and also defaults to None, meaning that no timeout will be set on any ftp socket(s). If a timeout is passed, then this is now the default timeout for all ftp socket operations for this instance. If supplied, source_address must be a 2-tuple (host, port) for all sockets created by this instance to bind to as their source address before connecting. If you pass a host name or address to the constructor, the 'connect' method will be called directly with the host and port given. Otherwise use 'connect' later, optionally passing host and port arguments. If you also pass a non-empty value for user, the 'login' method will be called with user, passwd and acct given after calling 'connect'. To download a file, use ftp.retrlines('RETR ' + filename), or ftp.retrbinary() with slightly different arguments. To upload a file, use ftp.storbinary(), which has an open file as argument (see its definition below for details). The download/upload functions first issue appropriate TYPE and PORT or PASV commands. """ host = None port = FTP_PORT timeout = _GLOBAL_DEFAULT_TIMEOUT source_address = None sock = None file = None welcome = None passive = 1 encoding = "latin-1" def __init__(self, host=None, port=None, user=None, passwd=None, acct=None, timeout=_GLOBAL_DEFAULT_TIMEOUT, source_address=None): """Initialization method (called by class instantiation). See class docstring for supported arguments. """ # These two settings are not tied to the connection, so if they are # given, we override the defaults, regardless of whether an initial # host to conenct to has been given or not. if timeout is not None: self.timeout = timeout if source_address: self.source_address = source_address if host: self.connect(host, port) if user: self.login(user, passwd, acct) def __enter__(self): return self # Context management protocol: try to quit() if active def __exit__(self, *args): if self.sock is not None: try: self.quit() except (OSError, EOFError): pass finally: if self.sock is not None: self.close() def _create_connection(self, addr, timeout=None, source_address=None): sock = socket() addrinfos = _resolve_addr(addr) for af, _, _, _, addr in addrinfos: try: sock.connect(addr) except Exception as exc: print(exc) else: if timeout and timeout is not _GLOBAL_DEFAULT_TIMEOUT: sock.settimeout(timeout) try: sock.family = af except: pass return sock else: raise Error("Could not connect to %r" % (addr,)) def connect(self, host=None, port=None, timeout=None, source_address=None): """Connect to host. Arguments are: - host: hostname to connect to (string, default previous host) - port: port to connect to (integer, default previous port) - timeout: the timeout for *this connection's* socket - source_address: a 2-tuple (host, port) for *this connection's* socket to bind to as its source address before connecting. """ if host: self.host = host if port: self.port = port if timeout is None: timeout = self.timeout if not source_address: source_address = self.source_address self.sock = self._create_connection((self.host, self.port), timeout, source_address) self.af = self.sock.family self.file = self.sock.makefile('rb') self.welcome = self.getresp() return self.welcome # Internal: return one line from the server, stripping CRLF. # Raise EOFError if the connection is closed def getline(self): line = self.file.readline(MAXLINE + 1) if len(line) > MAXLINE: raise Error("got more than %d bytes" % self.maxline) if not line: raise EOFError return line.rstrip(CRLF).decode(self.encoding) # Internal: get a response from the server, which may possibly # consist of multiple lines. Return a single string with no # trailing CRLF. If the response consists of multiple lines, # these are separated by '\n' characters in the string def getmultiline(self): line = self.getline() if line[3:4] == '-': code = line[:3] while 1: nextline = self.getline() line = line + ('\n' + nextline) if nextline[:3] == code and \ nextline[3:4] != '-': break return line # Internal: get a response from the server. # Raise various errors if the response indicates an error def getresp(self): resp = self.getmultiline() self.lastresp = resp[:3] if resp[:1] in ('1', '2', '3'): return resp raise Error(resp) def voidresp(self): """Expect a response beginning with '2'.""" resp = self.getresp() if not resp.startswith('2'): raise Error(resp) return resp def abort(self): """Abort a file transfer. Uses out-of-band data. This does not follow the procedure from the RFC to send Telnet IP and Synch; that doesn't seem to work with the servers I've tried. Instead, just send the ABOR command as OOB data. """ line = b'ABOR' + CRLF self.sock.sendall(line) resp = self.getmultiline() if resp[:3] not in {'426', '225', '226'}: raise Error(resp) return resp def sendcmd(self, cmd): """Send a command and return the response.""" self.sock.sendall(cmd.encode(self.encoding) + CRLF) return self.getresp() def voidcmd(self, cmd): """Send a command and expect a response beginning with '2'.""" self.sock.sendall(cmd.encode(self.encoding) + CRLF) return self.voidresp() def sendport(self, host, port): """Send a PORT command with current host and given port number. """ hbytes = host.split('.') pbytes = [repr(port // 256), repr(port % 256)] bytes = hbytes + pbytes cmd = 'PORT ' + ','.join(bytes) return self.voidcmd(cmd) def sendeprt(self, host, port): """Send an EPRT command with current host and given port number.""" af = 0 if self.af == _socket.AF_INET: af = 1 if self.af == _socket.AF_INET6: af = 2 if af == 0: raise Error('unsupported address family') fields = ['', repr(af), host, repr(port), ''] cmd = 'EPRT ' + '|'.join(fields) return self.voidcmd(cmd) def makeport(self): """Create a new socket and send a PORT command for it.""" err = None sock = None if self.source_address and self.source_address[0]: host = self.source_address[0] else: # XXX: this will only work for connections to a server on the same # host! socket.getsocketname() would be needed find out the # correct socket address to report to the server host = "127.0.0.1" if self.af == _socket.AF_INET else "::1" for port in range(MIN_PORT, MAX_PORT): addrinfo = _socket.getaddrinfo(host, port) for af, socktype, proto, _, addr in addrinfo: if af == self.af and socktype == _socket.SOCK_STREAM: try: sock = socket(af, socktype, proto) sock.bind(addr) except OSError as _: err = _ if sock: sock.close() sock = None continue else: try: sock.family = af except: pass if isinstance(addr, tuple): host = addr[0] else: try: # XXX: socket.inet_ntop() is not supported on # all MicroPython ports! host = _socket.inet_ntop(af, addr[4:8]) except: pass break if sock: break if sock is None: if err is not None: raise err else: raise OSError("getaddrinfo returns an empty list") sock.listen(1) if self.af == _socket.AF_INET: self.sendport(host, port) else: self.sendeprt(host, port) if self.timeout is not _GLOBAL_DEFAULT_TIMEOUT: sock.settimeout(self.timeout) return sock def makepasv(self): if self.af == _socket.AF_INET: host, port = parse227(self.sendcmd('PASV')) else: port = parse229(self.sendcmd('EPSV')) try: host = self.sock.getpeername() except AttributeError: # XXX: getpeername() is not supported by usocket! host = self.host return host, port def ntransfercmd(self, cmd, rest=None): """Initiate a transfer over the data connection. If the transfer is active, send a port command and the transfer command, and accept the connection. If the server is passive, send a pasv command, connect to it, and start the transfer command. Either way, return the socket for the connection and the expected size of the transfer. The expected size may be None if it could not be determined. Optional `rest' argument can be a string that is sent as the argument to a REST command. This is essentially a server marker used to tell the server to skip over any data up to the given marker. """ size = None if self.passive: host, port = self.makepasv() conn = self._create_connection((host, port), self.timeout, self.source_address) try: if rest is not None: self.sendcmd("REST %s" % rest) resp = self.sendcmd(cmd) # Some servers apparently send a 200 reply to # a LIST or STOR command, before the 150 reply # (and way before the 226 reply). This seems to # be in violation of the protocol (which only allows # 1xx or error messages for LIST), so we just discard # this response. if resp[0] == '2': resp = self.getresp() if resp[0] != '1': raise Error(resp) except: conn.close() raise else: sock = self.makeport() try: if rest is not None: self.sendcmd("REST %s" % rest) resp = self.sendcmd(cmd) # See above. if resp[0] == '2': resp = self.getresp() if resp[0] != '1': raise Error(resp) conn, _ = sock.accept() if self.timeout is not _GLOBAL_DEFAULT_TIMEOUT: conn.settimeout(self.timeout) finally: sock.close() if resp.startswith('150'): # this is conditional in case we received a 125 size = parse150(resp) return conn, size def login(self, user='', passwd='', acct=''): """Login, default anonymous.""" if not user: user = 'anonymous' if user == 'anonymous' and not passwd: # If there is no anonymous ftp password specified # then we'll just use 'anonymous@' # We don't send any other thing because: # - We want to remain anonymous # - We want to stop SPAM # - We don't want to let ftp sites to discriminate by the user, # host or country. passwd = 'anonymous@' resp = self.sendcmd('USER ' + user) if resp.startswith('3'): resp = self.sendcmd('PASS ' + (passwd or '')) if resp.startswith('3'): resp = self.sendcmd('ACCT ' + (acct or '')) if resp[0] != '2': raise Error(resp) return resp def retrlines(self, cmd, callback=None): """Retrieve data in line mode. A new port is created for you. Args: cmd: A RETR, LIST, or NLST command. callback: An optional single parameter callable that is called for each line with the trailing CRLF stripped. [default: print] Returns: The response code. """ if callback is None: callback = print self.sendcmd('TYPE A') with self.ntransfercmd(cmd)[0] as conn: fp = conn.makefile('rb') while 1: line = fp.readline(MAXLINE + 1) if not line: break if len(line) > MAXLINE: raise Error("got more than %d bytes" % MAXLINE) line = line.rstrip(CRLF) callback(line.decode(self.encoding)) # shutdown ssl layer if _SSLSocket is not None and isinstance(conn, _SSLSocket): conn.unwrap() return self.voidresp() def retrbinary(self, cmd, callback, blocksize=BLOCKSIZE, rest=None): """Retrieve data in binary mode. A new port is created for you. Args: cmd: A RETR command. callback: A single parameter callable to be called on each block of data read. blocksize: The maximum number of bytes to read from the socket at one time. [default: BLOCKSIZE] rest: Passed to transfercmd(). [default: None] Returns: The response code. """ self.voidcmd('TYPE I') with self.ntransfercmd(cmd, rest)[0] as conn: while 1: data = conn.recv(blocksize) if not data: break callback(data) gc.collect() # shutdown ssl layer if _SSLSocket is not None and isinstance(conn, _SSLSocket): conn.unwrap() return self.voidresp() def storbinary(self, cmd, fp, blocksize=BLOCKSIZE, callback=None, rest=None): """Store a file in binary mode. A new port is created for you. Args: cmd: A STOR command. fp: A file-like object with a read(num_bytes) method. blocksize: The maximum data size to read from fp and send over the connection at once. [default: BLOCKSIZE] callback: An optional single parameter callable that is called on each block of data after it is sent. [default: None] rest: Passed to transfercmd(). [default: None] Returns: The response code. """ self.voidcmd('TYPE I') with self.ntransfercmd(cmd, rest)[0] as conn: while 1: buf = fp.read(blocksize) if not buf: break conn.sendall(buf) if callback: callback(buf) # shutdown ssl layer if _SSLSocket is not None and isinstance(conn, _SSLSocket): conn.unwrap() return self.voidresp() def dir(self, *args, **kw): """List a directory in long form. By default list current directory to stdout. Optional last argument is callback function; all non-empty arguments before it are concatenated to the LIST command. (This *should* only be used for a pathname.) """ func = kw.get('callback') self.retrlines(" ".join(['LIST'] + list(args)), func) def rename(self, fromname, toname): """Rename a file.""" resp = self.sendcmd('RNFR ' + fromname) if resp[0] != '3': raise Error(resp) return self.voidcmd('RNTO ' + toname) def delete(self, filename): """Delete a file.""" resp = self.sendcmd('DELE ' + filename) if resp[:3] in {'250', '200'}: return resp else: raise Error(resp) def cwd(self, dirname): """Change to a directory.""" if dirname == '..': try: return self.voidcmd('CDUP') except Error as msg: if msg.args[0][:3] != '500': raise elif dirname == '': dirname = '.' # does nothing, but could return error cmd = 'CWD ' + dirname return self.voidcmd(cmd) def size(self, filename): """Retrieve the size of a file.""" # The SIZE command is defined in RFC-3659 resp = self.sendcmd('SIZE ' + filename) if resp[:3] == '213': s = resp[3:].strip() return int(s) def mkd(self, dirname): """Make a directory, return its full pathname.""" resp = self.voidcmd('MKD ' + dirname) # fix around non-compliant implementations such as IIS shipped # with Windows server 2003 if not resp.startswith('257'): return '' return parse257(resp) def rmd(self, dirname): """Remove a directory.""" return self.voidcmd('RMD ' + dirname) def pwd(self): """Return current working directory.""" resp = self.voidcmd('PWD') # fix around non-compliant implementations such as IIS shipped # with Windows server 2003 if not resp.startswith('257'): return '' return parse257(resp) def quit(self): """Quit, and close the connection.""" resp = self.voidcmd('QUIT') self.close() return resp def close(self): """Close the connection without assuming anything about it.""" try: file = self.file self.file = None if file is not None: file.close() finally: sock = self.sock self.sock = None if sock is not None: sock.close() def _find_parentheses(s): left = s.find('(') if left < 0: raise ValueError("missing left delimiter") right = s.find(')', left + 1) if right < 0: # string should contain '(...)' raise ValueError("missing right delimiter") return left, right def parse150(resp): """Parse the '150' response for a RETR request. Returns the expected transfer size or None; size is not guaranteed to be present in the 150 message. """ try: left, right = _find_parentheses(resp) except ValueError: return None else: try: val, _ = resp[left+1:right].split(None, 1) return int(val) except (ValueError, TypeError) as exc: raise Error("Error parsing response '%s': %s" % (resp, exc)) def parse227(resp): """Parse the '227' response for a PASV request. Raises error_proto if it does not contain '(h1,h2,h3,h4,p1,p2)' Return ('host.addr.as.numbers', port#) tuple. """ if not resp.startswith('227'): raise Error("Unexpected response: %s" % resp) try: left, right = _find_parentheses(resp) numbers = tuple(int(i) for i in resp[left+1:right].split(',', 6)) host = '%i.%i.%i.%i' % numbers[:4] port = (numbers[4] << 8) + numbers[5] except Exception as exc: raise Error("Error parsing response '%s': %s" % (resp, exc)) return host, port def parse229(resp): """Parse the '229' response for an EPSV request. Raises error_proto if it does not contain '(|||port|)' Return port number as integer. """ if not resp.startswith('229'): raise Error("Unexpected response: %s" % resp) try: left, right = _find_parentheses(resp) if resp[left + 1] != resp[right - 1]: raise ValueError("separator mismatch") parts = resp[left + 1:right].split(resp[left+1]) if len(parts) != 5: raise ValueError("unexpected number of values") except ValueError as exc: raise Error("Error parsing response '%s': %s" % (resp, exc)) return int(parts[3]) def parse257(resp): """Parse the '257' response for a MKD or PWD request. This is a response to a MKD or PWD request: a directory name. Returns the directory name in the 257 reply. """ if resp[3:5] != ' "': # Not compliant to RFC 959, but UNIX ftpd does this return '' dirname = '' i = 5 n = len(resp) while i < n: c = resp[i] i = i+1 if c == '"': if i >= n or resp[i] != '"': break i = i+1 dirname = dirname + c return dirname
-
@robert-hh I know this is an old thread but does ftplib downloaded from github work for you with the current firmware (1.18.2.r6) or did you have to modify it? I'm having some issues with it. Firstly an error in resolve_addr() at the call to socket.getaddrinfo. I removed the family and now I'm getting an error trying to connect. Just wondering if someone else has already gone through this and can save me some time. Ultimately I want to see if I can get OTA working.
-- Mike
-
@seb and @robert-hh Thank. @seb I'm waiting for your post.
-
@mdnncl01 From what I understand of your question, you want to update both your application and the firmware. So you have to consider two aspects:
a) If your application is a Python source file or precompiled file, you can load that, store it into the file system and reboot.
b) If your application is embedded into the firmware, or if you want to update the firmware, you have to do a firmware update, and that's what is called OTA here. The mechanism itself is implemented and works. Below in the post you see two examples. (i) The first dowloads the new image from a ftp file server and flashes it during the download. That method does not need extra file space on the device. (ii) The second assumes, that the new image is downloaded to the file system and transfers it from the file into the flash. Since the firmware image is pretty large, the internal file system of the old devices cannot handle it. Therefore, in the example a SD card is used. The new pycom devices with 8M flash have sufficient space to store that file and a SD card is not required.
You migth have to implement your own method of downloading the files or images from your server. The given examples just demonstrate the use of the OTA system calls.
-
The firmware allows the python code to be updated over FTP or UART at the moment and the pycom module include some functions for updating the firmware OTA (https://docs.pycom.io/chapter/firmwareapi/pycom/pycom.html). I am currently working on an example project to demonstrate updating both firmware and python code over WiFi but this will take a couple weeks at least to complete. I will post a reply here once it has been completed.
-
Hi all guys, I'm a beginner in the Pycom and Python world. With my team we decided to start experimenting with these devices (WiPy),
but for us it is absolutely essential to update the OTA of the user application for a real use in the field. From all the discussions
and documentation this aspect is not very clear; It would also be nice to have a possible firmware upgrade of the module; this aspect
is also unclear. Is there currently a stable procedure to update at least the user application? I'm sorry for my english. Thanks
-
@robert-hh
I'm also trying. Occasionally I have memory allocation problems, nothing that can't be solved with the garbage collector.
Thanks for the tips
-
@paco88 Of course, several times. And I verified that the new image is then excuted, by embedding changing frozen bytecode.
The example below is simplified, in that I had to insert a gc.collect() into the do_write() function, epsecially with a larger buffer size.
I have sketched another example here, which uses a fixed preallocated buffer. And yes, I tested that variant too.# Firmware update by reading the image from the SD card # from pycom import ota_start, ota_write, ota_finish from os import mount from machine import SD BLOCKSIZE = const(4096) APPIMG = "/sd/appimg.bin" sd = SD() mount(sd, '/sd') with open(APPIMG, "rb") as f: buffer = bytearray(BLOCKSIZE) mv = memoryview(buffer) size=0 ota_start() while True: chunk = f.readinto(buffer) if chunk > 0: ota_write(mv[:chunk]) size += chunk print("\r%7d " % size, end="") else: break ota_finish()
-
@robert-hh
Have you performed a full OTA write?
-
@betterauto You do not have to store the file in the local files system, if there is no space. You can feed the data into OTA on the fly while fetching.
I made a few attempts with:- interrupting an update and then starting a new one.
- repeating the update w/o restart in between.
All was stable. Not a very exhaustive test, but at least a first check. Since the image itself carries a checksum & signature, which is checked at the call of ota_finish(), transmission errors should not harm.
-
So this code uploads a bin file from the local file system to the local ftp server? Looks like the other way round.
Edit: Nevermind, I see you are fetching the file from an ftp server to flash it. Beautiful. I can do the same but fetch the file with MQTT over TLS to the local file system and flash itself in chunks.
I hope to try it soon. Nice find.
-
@betterauto OK. So here is my very simple sample script, using ftplib:
# # OTA update using ftplib and the pycom.ota_* methods # from ftplib import * from pycom import * def do_write(data): ota_write(data) print(".", end="") def ota(): ftp = FTP("server_ip", 21, "user", "password") ota_start() ftp.retrbinary("RETR appimg.bin", do_write, 1024) ota_finish() ota()I added an intermediate call to do_write() to have an progress indication.
Obviously you can use ftplib to download other files as well.
-
@betterauto It ONLY updates the firmware. There is no documentation yet. I'll give it a try by using ftplib for downloading the file and let you know.
-
@robert-hh does that only update files or also the firmware? Where do I find more info on this?
-
@robert-hh By looking into the pycom module I found that an OTA interface for Python scripts already exists. There are three methods:
pycom.ota_start()
pycom.ota_write(data)
pycom.ota_finish()
So one can easily use them. Start the update by calling ota_start(). Then receive the image data in arbitrary chunks and call ota_write() with it. Once the data is transferred, call ota_finish().
-
This post is deleted!
-
What error did you get? Doesn't make sense it can't connect to itself, it's just another IP right?
-
@betterauto Actually I gave it a try. The ftp client is working, when I connect to a different device. But I could not get a connection to itself.
Unless there is another option, you could do that: If you have a cluster of devices, then one of them could act as an update server, which pulls the update from a remote hub and distributes that locally.
