OTA or not OTA that is the question...

[robert-hh]

@BetterAuto If you look at the flash communication, it's not as simply as just dumping a stream of data to the xxPy. Flashing consists of two parts, the stub loader and the actual image, and then the data is transferred in blocks of 16 kByte each, with a (simple) protocol.
Esptool.py by itself is most likely too big to be ported to an ESP8266, but it could be split in an preparation phase running on the PC, which creates the binary data stream to be sent to the ESP32, which is then transferred to the ESP3266, and a smaller loader, which just takes this prepared image and sends it over to the ESP32. The first part might already be implemented in esptool.py, there is an option called write_image, for the second the protocol can be extracted from esptool.py. That seems doable.
Besides that, you have to organize on how the image is retrieved by the ESP8266. That device has to be managed too, like switching it on only when needed, etc.
But still, I do not think it's worth the effort, since OTA updates are still on Pycom's to-do-list.

[BetterAuto]

@robert-hh said in OTA or not OTA that is the question...:

If you look at the flash communication, it's not as simply as just dumping a stream of data to the xxPy. Flashing consists of two parts, the stub loader and the actual image, and then the data is transferred in blocks of 16 kByte each, with a (simple) protocol.

Understood, but if that entire session of two parts is captured raw the sending device should not need much if any logic. If you capture the raw stream coming from the TX pin (maybe via an intermediary device) it should work fine. Thus the helper chip should not need to understand what it is sending, it should be able to just send exactly the same stream of bytes that the original, with esptool.py, had sent.

Besides that, you have to organize on how the image is retrieved by the ESP8266. That device has to be managed too, like switching it on only when needed, etc.

The device has an enable pin which must be held high to operate. You would have to put a small amount of logic to tell it when to start and what file to send but that should be all.

I do not think it's worth the effort, since OTA updates are still on Pycom's to-do-list.

It depends if their timeline is three weeks, three months, or three years. I'm making a product for market right now and can't wait three years.

Do you know their timeline?

[jmarcelino]

@BetterAuto
At the Eindhoven hackaton yesterday Daniel said they were still waiting for the bootloader section - which can't be updated via OTA - to stabilise before enabling over the air upgrades, he expected it to be possible in 3-4 weeks.

[robert-hh]

@BetterAuto @jmarcelino I understand your concept. B.T.W. since the EPS8266 has WiFI too, you can download the image directly to it, using a TCP/IP connection. Otherwise, any micro with sufficient flash would do, but then you have to implement a second transfer link between WiPy and ESP8266. But what I said, flashing the WiPy needs to run a protocol, of sending blocks, waiting for responses, etc. I do not know the protocol. I may also implement succeed/fail flags, which have to be handled by the loader accordingly. That's why I suggested to take that part out of esptool.py. Below is a picture of a small loading section. Green is the data sent to the LoPy at TX, yellow the response at RX! The response consists of an 8 Byte and a 4 Byte message. The zoomed window shows the 8 byte message (speed 921600 Baud).

P.S.: Daniel is known to be optimistic about the timelines. If he says 3-4 weeks, there's a good chance that we have it at the end of the year.

[jmarcelino]

@robert-hh
Understood :) My feeling, because of the Pybites platform, the FiPy etc is we'll have it within 3 months.

[BetterAuto]

My kids used to like to watch Dora the Explorer. On that show the cartoon character turns to the audience and asks questions and pretends to respond. Even though the show is prerecorded, the illusion of ask/response is sufficient to satisfy many toddlers.

What does that have to do with OTA?

The prerecorded flash session need not be actually interactive. Two identical devices would respond in two identical ways even if one of them is only listening to a prerecorded message. I still think my method could work.

But if the timeline is somewhere between three weeks and year's end then I'll wait.

[robert-hh]

@BetterAuto The protocol is a9 variable in timing and it b) may vary in response, if it allows for a "transmission failed, please repeat las message" response. For a) a worst case timin could be used, but b) cannot be prerecorded.

[BetterAuto]

I see, thanks. That's the kind of information I was looking for. Since we're so close to a solution I'll wait. I'm producing a product for sale and if a few of the first models cannot OTA it'll have to suffice.

I'm building OTA for the scripts in at least, which is where I think most potential vulnerabilities would lie anyway.

The solution for OTA for the scripts is I am downloading any updated scripts as something.py.new and will then reset the device. I will have code in boot.py to look for *.new, if found it will delete any existing something.py and move something.py.new to something.py. Then proceed with the rest of the bootup.

One could even extend this to back up existing files and provide a fallback option via GPIO but I've decided not to proceed with that.

[Innocenzo]

The stranger thing is that if I upload my appimg.bin using FileZilla FTP , the procedure complete without problems. But if I use a code like this:

import socket 
import uos
import machine
from network import WLAN
import utime
import gc
from machine import SD

wlan = WLAN(mode=WLAN.STA)
wlan.connect('TP-LINK_2.4GHz_E113DF', auth=(WLAN.WPA2,"***********"))
while not wlan.isconnected():
    machine.idle()

#sd = SD()
#uos.mount(sd, '/sd')

print("Mem_alloc: {}".format(gc.mem_alloc()))
print("Mem_free: {}".format(gc.mem_free()))
uos.chdir("sys")
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM, socket.IPPROTO_TCP)
addr = socket.getaddrinfo("**************", 80)[0][-1]
print(addr)
s.connect(addr)
print("connected")
s.write(b"GET /*************/appimg.bin\r\n")
s.write(b"\r\n")
#f = open("/sd/appimg.bin","wb")
f = open("appimg.bin","w")
while 1:
    req = s.recv(8192)
    if not req or req == b"\r\n" or req == b'':
        break
    f.write(req)
    gc.collect()
    
f.close()
print("DONE")

or like this:

import socket
import gc
import uos
import machine
from network import WLAN
import utime
import sys

wlan = WLAN(mode=WLAN.STA)
wlan.connect('TP-LINK_2.4GHz_E113DF', auth=(WLAN.WPA2,"******"))
while not wlan.isconnected():
    machine.idle()

def getPort(req):
    req = req.split('(')
    print(req)
    req = req[1].split(')')
    print(req)
    req = req[0].split(',')
    port = (int(req[4]) << 8) + int(req[5]) 
    return port

uos.chdir("sys")
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(socket.getaddrinfo("*********", 21)[0][-1])
s.sendall(b'USER *******\r\n')
req = str(s.recv(8192))
print(req)
s.sendall(b'PASS *********\r\n"')
req = str(s.recv(8192))
print(req)
s.sendall(b'PASV\r\n')
req = str(s.recv(8192))
print(req)
s.sendall(b'TYPE I\r\n')
req = str(s.recv(8192),'utf-8')
print(req)
port = getPort(req)
p = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
p.connect(socket.getaddrinfo("************", port)[0][-1])
print("Connected!")
f = open("appimg.bin",'wb')
s.sendall(b'RETR appimg.bin\r\n')
req = str(s.recv(8192))
print(req)
print("Start transfer")

while 1:
    req = p.recv(8192)
    if not req or req == b'':
        break
    try:    
        f.write(req)
    except:
        print("NO!")
    gc.collect()
    utime.sleep(0.1)
f.close()
print("DONE")

I have an exception on the f.write(req) and I can transfer only 497.664 bytes of the appimg.bin.

[robert-hh]

@Innocenzo As far as I understand the ftp.c and ftpupdater.c code, the file appimg.bin, when received by ftp, is not written to the file system, but treated separately and written to a dedicated buffer. That's why it does not show up in the file system. Ftp looks out for that special file name "/flash/sys/appimg.bin".
The file system size is 508 k and is way to small to hold the image file.

[BetterAuto]

@robert-hh But @Innocenzo said that the procedure completes without problems when he uploads using FileZilla FTP.

@Innocenzo perhaps do a capture of the data being passed during the working upload to see what may need to be added to your filesystem code.

Working back to my thoughts on a small daughter device, perhaps an ESP8266 can FTP the code over. I had ignored that method since I couldn't get it to work with FileZilla myself.

[BetterAuto]

@Innocenzo when you say the procedure completes without problems, are you saying it actually positively flashes to the newer version? Try going from a version behind to current using FileZilla.

[robert-hh]

@BetterAuto Using his own code, @Innocenzo could not transfer appimg.bin to /flash/sys. using the built-in ftp, he can transfer the file, because it is NOT stored in the file system, but in a separate area in flash, which is large enough to hold the image. The size is 1538 kByte, which changed recently from about 1000 kByte.
For Wipy1 ftp upload is the preferred method of updating. With WiPy 2 and Lopy you could update the application image using ftp, until the file size grew beyond the 1MByte size. I see the growth of the dedicated storage area for the image as an indication of work in progress for OTA. Trying to update my Lopy this way failed. I could upload appimg.bin, but the device did not start the, but only showed the "red RGB led" state.n I had to erase & reflash to make it working again.
Once ftp update works, it can be done directly to the final device. No daughter device needed in between, unless you want to add security, which is not given at the moment.

[Innocenzo]

@robert-hh Ok with firmware 1.7.5.b1 does not work! red RGB led also for me. With previous firmware version, using FileZilla FTP, I was able to flashing my custom firmware!

[robert-hh]

@Innocenzo When you say "previous", which version was it?

[Innocenzo]

@robert-hh I am sorry, but I do not remember precisely. Or the first release of 1.7 or the the latest 1.6 ( I had to change the file bootloader.h for IMG_SIZE field).

[jmarcelino]

@robert-hh
Back in April I flashed a 1.6.9 custom firmware OTA with a the simple partition size patch - change #define IMG_SIZE (1536 * 1024) in the bootloader.h. It was definitely running my OTA image as I changed the version string and added an extra function.

Haven't really tried it since then. I think the code is already all there but as it was suggested maybe the bootloader isn't stable, the partitioning is still in flux or maybe it'll change to be connected in some way to the Pybytes platform.

[robert-hh]

@jmarcelino At the moment it is defined as:
#define IMG_SIZE ((1024 + 512) * 1024)
So I'm not sure what#s going on. I tried to upload an self-compiled appimg.bin. I could try to extract that from the 'official' image and try again.

[BetterAuto]

If the issue is only one of storage space, perhaps a smallish external SPI flash. Cheap.
https://www.digikey.com/product-detail/en/winbond-electronics/W25Q64JVSFIQ/W25Q64JVSFIQ-ND/5803990

[jmarcelino]

@BetterAuto
There is enough storage space, all the kits have 4MB Flash. The partitioning setup allows for two 1536K application images (factory + OTA) and a 508K user area (what you see in as /flash)