How to solve the firmware update failures

[digsatman]

@daniel said in How to solve the firmware update failures:

python flash_debug.py /dev/ttyUSB0 write 0x0 2

Well I'm afraid that I stil can't update either of my LoPys using the updaters for Windows. OSX or Linux. All three return a "failure" message. One of them had indeed become "write only" so I rann the flash_debug.py utility and that sorted out the problem. Just to be doubly sure, I ran it on the other as well.

As the standalone utility does not give an info on its procgress, I tried using ftp (Filezilla on Windows 7 64) and it seems to upload in fits and starts, has disconnections and retries. Tried increasing timeouts and using the filezilla "throttle" all to no avail. My most recent attempt was to copy the new firmware onto my Android phone and try an ftp to the LoPy. This also failed to both LoPy's but seemed to consistently fail after around 480-486Kb.. Any comments/help/workarounds much appreciated!

[LoneTech]

It would appear both LoPys I have at home have a different problem updating. They can write to flash, and uploading appimg.bin takes 3.5s (readonly boards accept it in 1.7s), but they do not boot 0.9.1.b1; only 0.9.0.b2 (the one I've updated them with earlier, by default or safeboot latest) or 0.9.0b (factory image, also loaded with safeboot previous). They simply don't seem to accept the update.

[daniel]

I can release an updater tool that will re-flash everything, but the problem is that it will erase the LoRa MAC address. The way to avoid that is to read the MAC address as a first step, then do the flashing and program the MAC address back again, similar to the way it's done in the factory. This is the way it will be don in order to reprogram the bootloader once the big in the IDF that is locking the flash gets fixed.

We'll work on this solution and try to provide something ASAP.

[LoneTech]

I also took the liberty of backing my flash up with esptool. I found that it contains four copies of the firmware; 8785822 , f5444c7, a78d0fa and a78d0fa. So it has clearly been written to flash, it just doesn't boot on that. Of these locations the 4th has an oddball address. The booting image resides at 0x110000, the non-booting update at 0x210000, and the surprising copy of the non-booting version appears to sit at 0x385200.

However, the firmware updates are NOT intact. Only the booting one is an exact match, suggesting the copy at 0x210000 in fact shouldn't have passed an integrity check. 224439 bytes in they differ.

>>> fwbad[0x36cb0:0x36cd0].encode('hex')
'981982120ab80002c8fca1d7ff8a8d8080b4a0ab1080aa20a909b21c3a3bc80c'
>>> fw2[0x36cb0:0x36cd0].encode('hex')
'981982120ab80982c8fca1d7ff8a8d8080b4a0ab1080aa20a909b21c3a3bc80c'

Clearly we have a stray bit cleared, suggesting perhaps the flash block wasn't erased properly. I might be able to just rewrite the chunk with esptool. I did try overwriting it directly with esptool, but that didn't make it start; however, it did make the firmware update work. Looks like there's a bug with erasing during OTA update (not sure how rarely it triggers). Anyhow, after wiping the second image with esptool I have now updated the first board. Time for the other.

[daniel]

@LoneTech thanks for the super detailed analysis and for sharing your findings. If erasing the block with esptool fixes it, it's definitely related to the IDF bug with regards to the Status Register 2 being written with odd values. See this:

https://github.com/espressif/esp-idf/commit/f988dc9587e9d39f78250c403f3ef85d03d6cd30

Especially the new filed added:

components/spi_flash/spi_flash_rom_patch.c

We are preparing a new updated that includes this fix and uploads the bootloader as well.

We are aiming to release that on Friday after exhaustive testing.

[Xykon]

I also looked through the 4MB image I made of the LoPy flash using esptool.py --chip esp32 --port COM24 --baud 115200 read_flash 0 4194304 lopy_firmware.bin

It seems the "recovery" firmware is at 0x10000, the previous version is at 0x110000 and the main firmware at 0x210000.

After flashing with esptool.py --chip esp32 --port COM24 --baud 115200 write_flash 0x210000 lopy_0.9.1.b1_868.bin -fs 4MB -z the latest version 0.9.1.b1 is starting as default.

I also flashed 0x10000 the same way and now my recovery image (when I leave the safeboot jumper on until the blue heartbeat LED starts) is now also at 0.9.1.b1.

My boot.py file and MAC address don't seem to be affected by this.

[LoneTech]

@daniel I think that bug causes full on readonly mode (I could be wrong). The data present in flash indicates that it did write, but didn't erase properly. On the second board, I did try esptool erase then OTA update; while it marked the image as updated, it caused a memory access error on boot. So I used esptool to write the firmware, and after that it booted the new revision. So it would seem, for the secondary flash position to update correctly on these particular boards, I need to get both the OTA done and write it with esptool. I haven't analyzed the bugfix you refer to to see if it would solve this, but I am certainly willing to try; I have backups of my boards now and could recover. Right now I'm happy I got them to boot the most recent release. :)

[oneeyed]

@Xykon said in How to solve the firmware update failures:

esptool.py --chip esp32 --port COM24 --baud 115200 write_flash 0x210000 lopy_0.9.1.b1_868.bin -fs 4MB -z

Indeed it allowed be to program the firmware as well, when no other methods could. Thanks.

[daniel]

@daniel I think that bug causes full on readonly mode (I could be wrong). The data present in flash indicates that it did write, but didn't erase properly.

The bug causes random bits in the flash status register to be written with wrong values, which can lead to different behaviors, the most common is the entire flash to be write protected, but it also can write protect only some sections of the flash memory.

[Bashir]

@oneeyed said in How to solve the firmware update failures:

@Xykon said in How to solve the firmware update failures:

esptool.py --chip esp32 --port COM24 --baud 115200 write_flash 0x210000 lopy_0.9.1.b1_868.bin -fs 4MB -z

Indeed it allowed be to program the firmware as well, when no other methods could. Thanks.

Hi, I can confirm this method worked for me also. I flashed the default address and recovery areas.

Thank you

[Xykon]

@Bashir said in How to solve the firmware update failures:

Thank you

Glad I could help.

Would you mind up-voting my post above? This helps getting rid of certain restrictions on this forum.

[livius]

Hi,

i have problem with update my Wipy2.0.
I have unlocked successfully - all looks like on screen @Batilan

But i still ca not update my board. When i copy appimg.bin to /flash/sys and restart my WiPy
it does not start. If i turn on self mode it start with oldest firmware.

Two questions here:

1. how to upgrade to most recent version?
2. how to permanently revert this update? I see no way to revert firmware to any previous version PERMANENTLY. I can revert it temporary but not permanently.

I try update WiPy2.0 only because it crash on simple blink heart led in the loop. And i think first upgrade and test again.

[livius]

Hi,

i have problem with update my Wipy2.0.
I have unlocked successfully - all looks like on screen @Batilan

But i still ca not update my board. When i copy appimg.bin to /flash/sys and restart my WiPy
it does not start. If i turn on self mode it start with oldest firmware.

Two questions here:

1. how to upgrade to most recent version?
2. how to permanently revert this update? I see no way to revert firmware to any previous version PERMANENTLY. I can revert it temporary but not permanently.

I try update WiPy2.0 only because it crash on simple blink heart led in the loop. And i think first upgrade and test again.

[Ralph]

Hi, @livius

First, try using the update tool (go to Firmware Updates) to update your WiPy2.0 and see if it solves your problem. This tool wil automatically update to the latest version.

If you still have the crash at startup after doing this, please provide the code you are using that is making it crash, and we can help you fix it.

[jgmdavies]

@livius Did you try the Pycom updater, or where did you get the file appimg.bin?

Jim

[livius]

@jgmdavies
I have downloaded updated app and go to folder and rename file for wipy to appimg.bin
then i put it by ftp

[jgmdavies]

@livius Thanks!

[jgmdavies]

@livius Thanks, I hadn't realised that is an option. For anyone else using Windows, you can probably find the .bin files at:

C:\Program Files (x86)\Pycom\Pycom Firmware Update\Upgrader

[livius]

@Ralph
I have tried it
It can conect and send data to wipy
but it can not connect after rebot and say that update failed
It store image to Wipy because wipy can not start normally only in "self" mode and it revert temporarly to old version.

After that problems i have tried to put the same appimg.bin self by ftp with same result. It store bin file and of course it dissapear
but wipy can not start - only in self mode

[Xykon]

I don't have a WiPy2 but if someone can upload the 4MB flash dump I can take a look to confirm the memory regions for the firmware images. I'd be surprised if they are any different from the LoPy though.