How to solve the firmware update failures

[daniel]

Hello,

The upgrade problems are caused by a bug in Espressif's IDF that is causing the flash memory of the module to be write protected. Espressif is working to provide a permanent solution for this as soon as possible. In the meantime they have given us a temporal way to solve it. The procedure is the following:

Download this tool which needs to be executed as:

python flash_debug.py /dev/ttyUSB0 write 0x0 2

(/dev/ttyUSB0 will have to be replaced by the actual serial port on your system)

Running the tool requires a serial connection to the module, Python 2 and Pyserial installed on your computer. The easiest way is via the expansion board. If an expansion board is not available then any USB-to-serial converter will work. The connection must be made on pins P0 (WiPy/LoPy RXD) and P1 (WiPy/LoPy TXD). Before running the tool, connect a cable jumper between GND and P2 (G23 on the expansion board), then reset the module and run the command. The total steps are:

1. Connect the WiPy/LoPy via the serial port to your computer.
2. Connect a cable jumper between GND and P2 (G23 on the expansion board).
3. Reset the board.
4. Run the command outlined above (python flash_debug.py /dev/ttyUSB0 write 0x0 2) and remember to pass the actual serial port on your system.
5. Remove the cable jumper.
6. Power cycle the WiPy/LoPy.
7. Run the firmware update tool.

We apologize for this issue but is completely out of our control. This is happening with all ESP32 based boards. I expect a solution from Espressif within a few days. Once that's done we will release a special firmware upgrade that will also install a new bootloader.

For the firmware release next Tuesday we will integrate this unlock process in the firmware update to avoid any more manual steps.

[LoneTech]

Thank you! Providing this patch on a weekend is a step above and beyond. :)

This unlock procedure worked perfectly on the one board I've tested so far. The other two locked ones are at a different location, and our remaining seven updated earlier, one of which took two tries, which is not a huge deal.

The bootstrap cable used here means P2 must not be tied to ground in general, as it triggers the low level bootloader. That sort of thing should go in the documentation somewhere. Also, this being a workaround, don't apply the unlock if you don't need it.

[daniel]

@LoneTech thanks for the feedback. I've added an extra step to indicate that the cable jumper must be removed before running the firmware updater again.

Thank you! Providing this patch on a weekend is a step above and beyond. :)

Your are welcome! Our first priority right now is to give the best possible support and complete the firmware as soon as we can. Our tech team is working almost 24/7 with these goals in mind.

[Batilan]

Hi Daniel,

Thanks a lot indeed!

I'm trying to run the tool in windows but I get:

C:\Users\Corné\Downloads\flash_debug_tool\flash_debug_tool>python flash_debug.py COM3 write 0x0 2
Connecting...
Traceback (most recent call last):
 File "flash_debug.py", line 183, in <module>
esp_dev = esptool.ESP32ROM(port,115200)
File "C:\Users\CornÚ\Downloads\flash_debug_tool\flash_debug_tool\esptool.py", line 167, in __init__
self.connect()
File "C:\Users\CornÚ\Downloads\flash_debug_tool\flash_debug_tool\esptool.py", line 299, in connect
raise FatalError('Failed to connect to %s: %s' % (self.CHIP_NAME, last_exception))
esptool.FatalError: Failed to connect to ESP32: Invalid head of packet ('\x08')

This is different from choosing an invalid serial port: that results in:

C:\Users\Corné\Downloads\flash_debug_tool\flash_debug_tool>python flash_debug.py COM2 write 0x0 2
Traceback (most recent call last):
  File "flash_debug.py", line 183, in <module>
    esp_dev = esptool.ESP32ROM(port,115200)
  File "C:\Users\CornÚ\Downloads\flash_debug_tool\flash_debug_tool\esptool.py", line 159, in __init__
    self._port = serial.Serial(port)
  File "C:\Python273\lib\site-packages\serial\serialwin32.py", line 31, in __init__
    super(Serial, self).__init__(*args, **kwargs)
  File "C:\Python273\lib\site-packages\serial\serialutil.py", line 236, in __init__
    self.open()
  File "C:\Python273\lib\site-packages\serial\serialwin32.py", line 62, in open
    raise SerialException("could not open port {!r}: {!r}".format(self.portstr, ctypes.WinError()))
serial.serialutil.SerialException: could not open port 'COM2': WindowsError(2, 'Het systeem kan het opgegeven bestand niet vinden.')

I did close down PyMakr first.

Guess I'll switch to my Linux Box now :-)

[daniel]

@Batilan,

Did you place the cable jumper between GND and P2 and reset the board before running the script?

[Batilan]

@Batilan

Ok to myself: this happens when you forget the reset step (or push the right button on the wrong LoPy as I did :-S)
Discovered it immediately after my post, but had to wait 10 minutes because of "bad reputation" ;-)
That allowed Daniel to show off his great customer service skills :-)

Now it looks nicer:

C:\Users\Corné\Downloads\flash_debug_tool\flash_debug_tool>python flash_debug.py COM3 write 0x0 2
Connecting...
value: 0x0
length: 2 
write 0x60002008 0x00000000 0x00400000 
write 0x60002010 0x00000000 0x00000000 
write 0x60002000 0x08000000 0xffffffff
write 0x60002008 0x00000000 0x00400000
write 0x60002010 0x00000000 0x00000000
write 0x60002000 0x08000000 0xffffffff
write 0x60002008 0x00000000 0x00400000
write 0x60002010 0x00000000 0x00000000
write 0x60002000 0x08000000 0xffffffff
write 0x60002008 0x00400000 0x00400000
write 0x60002010 0x00000000 0x0000ffff
write 0x60002000 0x04000000 0xffffffff
write 0x60002008 0x00000000 0x00400000
write 0x60002010 0x00000000 0x00000000
write 0x60002000 0x08000000 0xffffffff
get status: 0x0
ESP DEV DISCONNECT...

Maybe it helps someone else :-)

[on4bhm]

running python 3.6.0a3 on Win10
I get this error:

C:\Users\guyro\Downloads\flash_debug_tool\flash_debug_tool>python flash_debug.py COM3 write 0x0 2
File "flash_debug.py", line 29
print"ESP DEV DISCONNECT..."
^
SyntaxError: invalid syntax

C:\Users\guyro\Downloads\flash_debug_tool\flash_debug_tool>python

what can I do?
even when I place a extra space between print" it will not work

kind regards

[Batilan]

Probably use Python version 2 (like 2.7)

[on4bhm]

I thought: the latest is always the greatest?

[on4bhm]

I removed my python3 and installed 2.7
now I get this:

C:\Users\guyro\Downloads\flash_debug_tool\flash_debug_tool>c:\Python27\python.exe flash_debug.py COM3 write 0x0 2
Traceback (most recent call last):
File "flash_debug.py", line 1, in <module>
import esptool
File "C:\Users\guyro\Downloads\flash_debug_tool\flash_debug_tool\esptool.py", line 25, in <module>
import serial
ImportError: No module named serial

C:\Users\guyro\Downloads\flash_debug_tool\flash_debug_tool>

what is going on?

[Batilan]

Install the PySerial package as explained also here: https://learn.adafruit.com/arduino-lesson-17-email-sending-movement-detector/installing-python-and-pyserial Disregard the version difference with latest (in this case latest is greatest it seems :-)

[littlebat]

Can you open a github issue for this at esp-idf repo so we can know what it is and when it gets fixed?

[Batilan]

Actually that isn't the best example for Windows, but just download the module from https://pypi.python.org/pypi/pyserial unpack it using a program that is also capable of unpacking .tgz files and perform the setup by typing '

python setup.py install

In the pyserial-3.2.1 directory

If this helps please give me an upvote so next time you don't have to wait 10 minutes (as I don't have 3 reputation points yet (just registered) and have to wait 10 minutes before I can post my next reply :-S)

[on4bhm]

The tool seemed to work...

but the upgrade is still not working.
I have a dos box that is constant pinging to 192.168.4.1
I have a ping reply.
I do the procedure of going into safeboot
I always get 'The upgrade failed.please check that you are connected to the LoPy wireless LAN and try again.
but I can ping him !!
my wired network is disconnected.... !!

[on4bhm]

I took my virgin LoPy. and it worked right from the first try.
my previous LoPy had been upgraded before, a few day's ago.

Just for your information....
Can someone please check this?

[digsatman]

@daniel said in How to solve the firmware update failures:

python flash_debug.py /dev/ttyUSB0 write 0x0 2

Well I'm afraid that I stil can't update either of my LoPys using the updaters for Windows. OSX or Linux. All three return a "failure" message. One of them had indeed become "write only" so I rann the flash_debug.py utility and that sorted out the problem. Just to be doubly sure, I ran it on the other as well.

As the standalone utility does not give an info on its procgress, I tried using ftp (Filezilla on Windows 7 64) and it seems to upload in fits and starts, has disconnections and retries. Tried increasing timeouts and using the filezilla "throttle" all to no avail. My most recent attempt was to copy the new firmware onto my Android phone and try an ftp to the LoPy. This also failed to both LoPy's but seemed to consistently fail after around 480-486Kb.. Any comments/help/workarounds much appreciated!

[LoneTech]

It would appear both LoPys I have at home have a different problem updating. They can write to flash, and uploading appimg.bin takes 3.5s (readonly boards accept it in 1.7s), but they do not boot 0.9.1.b1; only 0.9.0.b2 (the one I've updated them with earlier, by default or safeboot latest) or 0.9.0b (factory image, also loaded with safeboot previous). They simply don't seem to accept the update.

[daniel]

I can release an updater tool that will re-flash everything, but the problem is that it will erase the LoRa MAC address. The way to avoid that is to read the MAC address as a first step, then do the flashing and program the MAC address back again, similar to the way it's done in the factory. This is the way it will be don in order to reprogram the bootloader once the big in the IDF that is locking the flash gets fixed.

We'll work on this solution and try to provide something ASAP.

[LoneTech]

I also took the liberty of backing my flash up with esptool. I found that it contains four copies of the firmware; 8785822 , f5444c7, a78d0fa and a78d0fa. So it has clearly been written to flash, it just doesn't boot on that. Of these locations the 4th has an oddball address. The booting image resides at 0x110000, the non-booting update at 0x210000, and the surprising copy of the non-booting version appears to sit at 0x385200.

However, the firmware updates are NOT intact. Only the booting one is an exact match, suggesting the copy at 0x210000 in fact shouldn't have passed an integrity check. 224439 bytes in they differ.

>>> fwbad[0x36cb0:0x36cd0].encode('hex')
'981982120ab80002c8fca1d7ff8a8d8080b4a0ab1080aa20a909b21c3a3bc80c'
>>> fw2[0x36cb0:0x36cd0].encode('hex')
'981982120ab80982c8fca1d7ff8a8d8080b4a0ab1080aa20a909b21c3a3bc80c'

Clearly we have a stray bit cleared, suggesting perhaps the flash block wasn't erased properly. I might be able to just rewrite the chunk with esptool. I did try overwriting it directly with esptool, but that didn't make it start; however, it did make the firmware update work. Looks like there's a bug with erasing during OTA update (not sure how rarely it triggers). Anyhow, after wiping the second image with esptool I have now updated the first board. Time for the other.

[daniel]

@LoneTech thanks for the super detailed analysis and for sharing your findings. If erasing the block with esptool fixes it, it's definitely related to the IDF bug with regards to the Status Register 2 being written with odd values. See this:

https://github.com/espressif/esp-idf/commit/f988dc9587e9d39f78250c403f3ef85d03d6cd30

Especially the new filed added:

components/spi_flash/spi_flash_rom_patch.c

We are preparing a new updated that includes this fix and uploads the bootloader as well.

We are aiming to release that on Friday after exhaustive testing.