Firmware update elf vs tar

[robert-hh]

@jason_gladen The tar-gz file contain the binary image, which is not identical to the elf file. You can install the tar files with the pycom updater, but the most recent versions are not available for download, which I consider an omission. And it was promised to provides also the up-to-date tar files! Ping, @Paul-Thornton
Until then you could repack the tar file. The conversion of the .elf file into the .bin file is done with esptool.py. flashfreq=80Mhz, flashmode=dio. Once you have a bin file, you can grap an older tar file for your target, unpack it, replace the bin file, and re-pack it again. This new tar file can then be installed into your device.

[rcolistete]

It's here, Pycom Docs - Firmware Downgrade.

[robert-hh]

@rcolistete You're right. Sorry Paul.

[andreas]

Hi there,

yesterday, we tried to update to the newest 1.20.1.r1 firmware from the command line.

Rebundling

Following your suggestions (thanks again, @robert-hh!), we tried to use the tarball release from 1.20.0.rc13, convert the 1.20.1.r1 ELF file from the GitHub release page into a .bin file using esptool like:

esptool.py --chip esp32 elf2image --flash_freq=80m --flash_mode=dio --output FiPy-1.20.1r1/fipy.bin FiPy-1.20.1.r1-application.elf

and then wrap it into the tarball again.

Flashing

When trying to flash that guy, we firstly got the weird error message Invalid script command f (see https://forum.pycom.io/topic/3970/error-in-lopy-firmware-upgrade-when-using-local-file-for-both-cli-and-gui-linux) using the old Pycom Firmware Update tool.

After upgrading to the latest version of the Pycom Firmware Update tool, we are receiving this message:

$ pycom-fwtool-cli --verbose --port $MCU_PORT flash --tar FiPy-1.20.1r1.tar.gz

Running in PIC mode
Product ID: 152 HW Version: 7 FW Version: 0.0.11
Connecting....
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Auto-detected Flash size: 8MB
Error Loading script_8MB ... defaulting to legacy script2!
Compressed 4 bytes to 12...
 Writing o at 0x0000000B... (100 %)
Hash of data verified.
Wrote 4 bytes (12 compressed) at 0x0000000b in 0.1 seconds (0.4 kbit/s)...

Flash operation successful.

Obviously, it still doesn't work, the FiPy would not start.

Thoughts and aims

So, we assume we are lacking the appropriate bootloader.bin and partitions.bin files, maybe we also need updated variants of the script and script2 files located within the tarfile.

In this manner, we are humbly asking if someone of you could kindly provide us with these files from a current Dev build on their workstation to save us from running the build process on our own machines.

We would like to upload a ready made tarball https://packages.hiveeyes.org/hiveeyes/foss/pycom/ in order to streamline the installation for the people within our community. This might actually help others, too.

Thanks already and with kind regards,
Andreas.

[robert-hh]

@andreas I get this spurious Invalid script command f too. On tepeati attempts of the flash it succeeds. I did not look into the code yet, but it seems to be some inconsistent error handling.
Interestingly, the new firmware build package of Pycom generates the tarballs. The new images require to erase the flash beforehand.

[robert-hh]

@robert-hh I have two of these tarballs here: https://hidrive.ionos.com/share/eyz4etzbfv

[andreas]

Dear @robert-hh,

thanks for the quick response.

Flash success

I am able to confirm flashing these guys using the the new Pycom Firmware Update from the command line like

pycom-fwtool-cli --verbose --port $MCU_PORT flash --tar ~/Downloads/FiPy-1.20.1.r1.tar.gz

works excellent for me,

Pycom MiroPython 1.20.1.r1 [d4b4717] on 2019-10-12; FiPy with ESP32

so I uploaded them to [1]. Thanks a bunch!

Rebuild fail

However, even when trying to rebundle the tarfile using its original content, the flash process will insist on croaking like

Error Loading script_8MB ... defaulting to legacy script2!
Compressed 4 bytes to 12...

when using the new tarfile. I haven't come across Invalid script command f since updating the Pycom Firmware Updater to the most recent version for macOS.

As I am working on macOS, I even resorted to the GNU tools provided by Homebrew in order to rule out anything from BSD Tar oder Apple Gzip, so I tried both commands [2] and [3] to no avail at all. Bummer!

Conclusion

As I am really feeling a bit silly on that, I am now asking myself whether there might be any magic sauce required when cooking these tarballs and wanted to share my observations with the community. Saying that, it might be obvious for someone what I am doing wrong.

Nevertheless, thank you so much again for providing the tarballs!

With kind regards,
Andreas.

[1] https://packages.hiveeyes.org/hiveeyes/foss/pycom/
[2] tar -czf FiPy-1.20.1r1.tar.gz --directory FiPy-1.20.1r1 .
[3] gtar -c --directory FiPy-1.20.1r1 . | /usr/local/bin/gzip -5 > FiPy-1.20.1r1.tar.gz

[robert-hh]

@andreas The archives which you create contain a directory (named .) and inside that the files of the package, whereas the tarballs created by Pycom 8and me) are flat. No subdirectory. Maybe that's the reason.

[andreas]

Thanks for your thoughts @robert-hh. I believed I also created a flat tarball without any subdirectories and when looking at them, they look good to me. After failing when trying to make my first tarball, I found that tar -c --directory FiPy-1.20.1r1 . should be the appropriate syntax for doing so.

If you see that I might be going into the wrong direction here, may I humbly ask how you are creating these tarballs in the appropriate format/style?

Thanks in advance!

[robert-hh]

@andreas If change into that directory with the files and issue the command:

tar cvfz ../FiPy-1.20.1r1.tar.gz *

[andreas]

Dear @robert-hh,

that works like a charm, thank you very much!

We are able to confirm this procedure works for us:

Get and extract vanilla tarball

# Download confirmed tarball release.
wget https://packages.hiveeyes.org/hiveeyes/foss/pycom/FiPy-1.20.1.r1.tar.gz

# Extract into temporary build directory.
mkdir build
tar -xzf FiPy-1.20.1.r1.tar.gz --directory build

Use Pycom firmware release

# Download most recent ELF file as of 2019-10-17.
http --download --follow https://github.com/pycom/pycom-micropython-sigfox/releases/download/v1.20.1.r1/FiPy-1.20.1.r1-application.elf

# Convert to image file and put into place.
esptool.py --chip esp32 elf2image --output build/fipy.bin FiPy-1.20.1.r1-application.elf

Create new tarball

cd build
tar -czf ../FiPy-1.20.1.r1-rebundled.tar.gz *
cd -

Flash to device

pycom-fwtool-cli --verbose --port $MCU_PORT flash --tar FiPy-1.20.1.r1-rebundled.tar.gz

Finally!

Thanks again and with kind regards,
Andreas.

[andreas]

Just for the records: Instead of identifying as

@andreas said in Firmware update elf vs tar:

Pycom MicroPython 1.20.1.r1 [d4b4717] on 2019-10-12; FiPy with ESP32

the firmware identifies itself as

Pycom MicroPython 1.20.1.r1 [v1.11-3138a13] on 2019-10-08; FiPy with ESP32
Pybytes Version: 1.1.3

after flashing the rebundled tarball using the most recent FiPy-1.20.1.r1-application.elf file from the GitHub release page.

[andreas]

This post is deleted!