OTA or not OTA that is the question...

[jmarcelino]

@robert-hh
Understood :) My feeling, because of the Pybites platform, the FiPy etc is we'll have it within 3 months.

[BetterAuto]

My kids used to like to watch Dora the Explorer. On that show the cartoon character turns to the audience and asks questions and pretends to respond. Even though the show is prerecorded, the illusion of ask/response is sufficient to satisfy many toddlers.

What does that have to do with OTA?

The prerecorded flash session need not be actually interactive. Two identical devices would respond in two identical ways even if one of them is only listening to a prerecorded message. I still think my method could work.

But if the timeline is somewhere between three weeks and year's end then I'll wait.

[robert-hh]

@BetterAuto The protocol is a9 variable in timing and it b) may vary in response, if it allows for a "transmission failed, please repeat las message" response. For a) a worst case timin could be used, but b) cannot be prerecorded.

[BetterAuto]

I see, thanks. That's the kind of information I was looking for. Since we're so close to a solution I'll wait. I'm producing a product for sale and if a few of the first models cannot OTA it'll have to suffice.

I'm building OTA for the scripts in at least, which is where I think most potential vulnerabilities would lie anyway.

The solution for OTA for the scripts is I am downloading any updated scripts as something.py.new and will then reset the device. I will have code in boot.py to look for *.new, if found it will delete any existing something.py and move something.py.new to something.py. Then proceed with the rest of the bootup.

One could even extend this to back up existing files and provide a fallback option via GPIO but I've decided not to proceed with that.

[Innocenzo]

The stranger thing is that if I upload my appimg.bin using FileZilla FTP , the procedure complete without problems. But if I use a code like this:

import socket 
import uos
import machine
from network import WLAN
import utime
import gc
from machine import SD

wlan = WLAN(mode=WLAN.STA)
wlan.connect('TP-LINK_2.4GHz_E113DF', auth=(WLAN.WPA2,"***********"))
while not wlan.isconnected():
    machine.idle()

#sd = SD()
#uos.mount(sd, '/sd')

print("Mem_alloc: {}".format(gc.mem_alloc()))
print("Mem_free: {}".format(gc.mem_free()))
uos.chdir("sys")
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM, socket.IPPROTO_TCP)
addr = socket.getaddrinfo("**************", 80)[0][-1]
print(addr)
s.connect(addr)
print("connected")
s.write(b"GET /*************/appimg.bin\r\n")
s.write(b"\r\n")
#f = open("/sd/appimg.bin","wb")
f = open("appimg.bin","w")
while 1:
    req = s.recv(8192)
    if not req or req == b"\r\n" or req == b'':
        break
    f.write(req)
    gc.collect()
    
f.close()
print("DONE")

or like this:

import socket
import gc
import uos
import machine
from network import WLAN
import utime
import sys

wlan = WLAN(mode=WLAN.STA)
wlan.connect('TP-LINK_2.4GHz_E113DF', auth=(WLAN.WPA2,"******"))
while not wlan.isconnected():
    machine.idle()

def getPort(req):
    req = req.split('(')
    print(req)
    req = req[1].split(')')
    print(req)
    req = req[0].split(',')
    port = (int(req[4]) << 8) + int(req[5]) 
    return port

uos.chdir("sys")
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(socket.getaddrinfo("*********", 21)[0][-1])
s.sendall(b'USER *******\r\n')
req = str(s.recv(8192))
print(req)
s.sendall(b'PASS *********\r\n"')
req = str(s.recv(8192))
print(req)
s.sendall(b'PASV\r\n')
req = str(s.recv(8192))
print(req)
s.sendall(b'TYPE I\r\n')
req = str(s.recv(8192),'utf-8')
print(req)
port = getPort(req)
p = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
p.connect(socket.getaddrinfo("************", port)[0][-1])
print("Connected!")
f = open("appimg.bin",'wb')
s.sendall(b'RETR appimg.bin\r\n')
req = str(s.recv(8192))
print(req)
print("Start transfer")

while 1:
    req = p.recv(8192)
    if not req or req == b'':
        break
    try:    
        f.write(req)
    except:
        print("NO!")
    gc.collect()
    utime.sleep(0.1)
f.close()
print("DONE")

I have an exception on the f.write(req) and I can transfer only 497.664 bytes of the appimg.bin.

[robert-hh]

@Innocenzo As far as I understand the ftp.c and ftpupdater.c code, the file appimg.bin, when received by ftp, is not written to the file system, but treated separately and written to a dedicated buffer. That's why it does not show up in the file system. Ftp looks out for that special file name "/flash/sys/appimg.bin".
The file system size is 508 k and is way to small to hold the image file.

[BetterAuto]

@robert-hh But @Innocenzo said that the procedure completes without problems when he uploads using FileZilla FTP.

@Innocenzo perhaps do a capture of the data being passed during the working upload to see what may need to be added to your filesystem code.

Working back to my thoughts on a small daughter device, perhaps an ESP8266 can FTP the code over. I had ignored that method since I couldn't get it to work with FileZilla myself.

[BetterAuto]

@Innocenzo when you say the procedure completes without problems, are you saying it actually positively flashes to the newer version? Try going from a version behind to current using FileZilla.

[robert-hh]

@BetterAuto Using his own code, @Innocenzo could not transfer appimg.bin to /flash/sys. using the built-in ftp, he can transfer the file, because it is NOT stored in the file system, but in a separate area in flash, which is large enough to hold the image. The size is 1538 kByte, which changed recently from about 1000 kByte.
For Wipy1 ftp upload is the preferred method of updating. With WiPy 2 and Lopy you could update the application image using ftp, until the file size grew beyond the 1MByte size. I see the growth of the dedicated storage area for the image as an indication of work in progress for OTA. Trying to update my Lopy this way failed. I could upload appimg.bin, but the device did not start the, but only showed the "red RGB led" state.n I had to erase & reflash to make it working again.
Once ftp update works, it can be done directly to the final device. No daughter device needed in between, unless you want to add security, which is not given at the moment.

[Innocenzo]

@robert-hh Ok with firmware 1.7.5.b1 does not work! red RGB led also for me. With previous firmware version, using FileZilla FTP, I was able to flashing my custom firmware!

[robert-hh]

@Innocenzo When you say "previous", which version was it?

[Innocenzo]

@robert-hh I am sorry, but I do not remember precisely. Or the first release of 1.7 or the the latest 1.6 ( I had to change the file bootloader.h for IMG_SIZE field).

[jmarcelino]

@robert-hh
Back in April I flashed a 1.6.9 custom firmware OTA with a the simple partition size patch - change #define IMG_SIZE (1536 * 1024) in the bootloader.h. It was definitely running my OTA image as I changed the version string and added an extra function.

Haven't really tried it since then. I think the code is already all there but as it was suggested maybe the bootloader isn't stable, the partitioning is still in flux or maybe it'll change to be connected in some way to the Pybytes platform.

[robert-hh]

@jmarcelino At the moment it is defined as:
#define IMG_SIZE ((1024 + 512) * 1024)
So I'm not sure what#s going on. I tried to upload an self-compiled appimg.bin. I could try to extract that from the 'official' image and try again.

[BetterAuto]

If the issue is only one of storage space, perhaps a smallish external SPI flash. Cheap.
https://www.digikey.com/product-detail/en/winbond-electronics/W25Q64JVSFIQ/W25Q64JVSFIQ-ND/5803990

[jmarcelino]

@BetterAuto
There is enough storage space, all the kits have 4MB Flash. The partitioning setup allows for two 1536K application images (factory + OTA) and a 508K user area (what you see in as /flash)

[robert-hh]

@BetterAuto At the moment it looks like only a somewhat small piece of software bug has to be changed. Maybe the same that causes the pre-build imagas and self-built images to be incompatible = using a different partitioning of the memory.

[BetterAuto]

I just found this post from back in April with a method from @jmarcelino which may be useful for us. I haven't tried it yet.
https://github.com/pycom/pycom-micropython-sigfox/issues/5

[robert-hh]

@BetterAuto The carnage @jmarcelino ist set by default now. You can do an OTA upload this way, but then the device either does not start any more or the uploaded image is ignored, depending on which state the device was before. There is a discrepancy between the partition as defined in lib/partitions.csv, compiled to partitions.bin, and the address definitions of bootloader.h. According to partitions.bin, the addresses where the image is stored with OTA is 0x10000 or 0x1a0000, according to bootloader.h it's 0x10000 or 0x190000. And some of the flags that are used to control the process are taken from the sixth partition called dummy, but the 4th partition is called ota_data, and may be the right one. The layout of memory and its use is not clear to me, and instead of me guessing, someone from pycom should fix that in the intended manner.
@daniel, what's your opinion?

[Paco88]

Any news about the OTA features?