New beta firmware updater 1.13.1.b1

[Xykon]

A new beta firmware updater 1.13.1.b1 is now available (for Windows and MacOS. The Linux version should be available in the next 48h).

Windows download
MacOS download

This has the following improvements:

GUI interface:

• Option to delete the flash file system

• Option to flash stable or development firmware (see further details below)

• You can set the LoRa region by country or region (region only in offline mode)

• You can force update the LoRa region when flashing from file. This will not check the board or firmware type, as it is not harmful to set this on boards which do not support LoRa.

• If you disable the "High speed transfer" option in the Communication window, you are now be able to select from a number of different baud rates, including 115200bps.

Please note that currently when you choose development firmware, you will still receive the latest stable release, which is the most recent version available. This is because we currently have no development firmware published that is newer than the latest stable release.

CLI interface:

• The port and speed arguments can now be set by using environment variables ESPPORT and ESPSPEED.

• There is now a new copy command:

usage: pycom-fwtool-cli copy [-h] [-p PARTITION] [-f FILE] [-r] [-b]

optional arguments:
  -h, --help            show this help message and exit
  -p PARTITION, --partition PARTITION
                        The partition to read/write (all, fs, nvs, factory,
                        secureboot, bootloader, partitions, otadata, fs1,
                        ota_0, config)
  -f FILE, --file FILE  name of the binary file (default: <wmac>-<part>.bin)
  -r, --restore         restore partition from binary file
  -b, --backup          backup partition to binary file (default)

This will allow you to read/write the various partitions in the esp32 flash memory. Please see further below for more details.

The default is to backup the partition (read from esp32, write to PC). If no filename is specified, it will use <wireless-mac>-<partition-name>.bin It is not possible to use - for std-in

Furthermore the scripting engine used internally by the updater to flash the firmware files has been expanded. If you are interested in building your own firmware packages, please check out the next section.

[this.wiederkehr]

@Xykon
As mentioned by the esp-idf docs:

The OTA data partition is two flash sectors (0x2000 bytes)

Can you clarify why this is not the case for your otadata-partition?

Edit: Never Mind-> https://forum.pycom.io/topic/3161/solved-invalid-partition-table-of-otadata-size-can-corrupt-ota_0-partition/2 you are not using the esp-idf related functions to perform ota.

[Xykon]

New deployment options

If you have been looking into building your own firmware packages, you've probably come across the script file by now:

[
    ["e", "0x1000",      "0x40000"],
    ["e", "0x41000",     "0x40000"],
    ["e", "0x81000",     "0x40000"],
    ["e", "0xC1000",     "0x40000"],
    ["e", "0x101000",    "0x40000"],
    ["e", "0x141000",    "0x40000"],
    ["e", "0x181000",    "0x40000"],
    ["e", "0x1C1000",    "0x40000"],
    ["w", "0x1000", "bootloader.bin"],
    ["w", "0x8000", "partitions.bin"],
    ["w", "0x10000", "wipy.bin"]
]

This small json file tells the firmware updater where to write each of the 3 firmware files embedded in the package. While this lets you erase and write any memory region, it is very powerful but not very user friendly, as you need to know the offset for each flash location. So I decided to embed the information from the partition table into the updater so that users can use easy to identify names rather than hex offsets and byte lengths.

These are the partitions defined in the updater:

'secureboot' : ["0x0", "0x8000"]
'bootloader' : ["0x1000", "0x7000"]
'partitions' : ["0x8000", "0x1000"]
'nvs'        : ["0x9000", "0x7000"]
'factory'    : ["0x10000", "0x180000"]
'otadata'    : ["0x190000", "0x1000"]
'ota_0'      : ["0x1a0000", "0x180000"]
'fs'         : ["0x380000", "0x7F000"]
'config'     : ["0x3FF000", "0x1000"]
'fs1'        : ["0x400000", "0x400000"]
'all'        : ["0x0"]

For the last partition "all" the size is determined based on the flash size.

For additional convenience, the abbreviation ota can be used instead of otadata and cb can be used instead of config as this naming convention is also used in other commands.

Now you can also use these predefined partition definitions when writing the script file for your firmware package. As the size is known for each partition, you do not need to specify it when using a partition in the script file. If you want to force the erasure of the nvs partition, the otadata partition and the file system, your script file could look like this:

[["e", "nvs"],
["e", "ota"],
["e", "fs"]]

For the write ["w"] command, you can now also use the partition labels. Because the size of the partition is known in this case, the image to be written is either truncated or expanded and filled with 0xFF to fit into the available space. If you write the bootloader, partition table or firmware image, you no longer need to erase these regions first before writing to them.

A note of caution here that the updater will try its best to ensure the configuration partition (also known as config block) is not overwritten. This is because this partition contains configuration settings such as the lora mac, Sigfox credentials and other valuable details.

If you want to bypass these measures, such as when restoring a device backup, you can use the command "o" (as in overwrite) instead.

For example: You made a full backup of your device with the command pycom-fwtool-cli copy -p all (assuming you had ESPPORT set to the correct COM port), you will end up with a file like 240AC4027C18-all.bin

In order to "restore" the device via the firmware updater after packing it into an archive with tar -cvzf restore_240AC4027C18.tar.gz 240AC4027C18-all.bin script and flashing it with pycom-fwtool-cli flash -t restore_240AC4027C18.tar.gz

If your file script would have:
[["w", "all", "240AC4027C18-all.bin"]]
It would not overwrite the config partition and it would truncate the image if necessary to fit the available flash size.

If you want to make sure that everything is overwritten, use the following instead:

[["o", "all", "240AC4027C18-all.bin"]]

Beware though that if your binary is bigger than the available flash size, you will get an exception.

A similar thing is true for the file system partition. There are actually two different partitions, fs and fs1. In our 4MB devices, the file system is at offset 0x380000 and 508KB big. In our 8MB devices the file system starts at 0x400000 and has 4MB. The reason that there is not a single file system partition that is dependant on the flash size similar to the "all" partition is so that you can create firmware packages that include the fs for devices with different file system sizes, such as WiPy 2 & 3 or LoPy1 and L01.

If you have two file system images fs.bin and fs1.bin, for the two different file system regions, you can specify them in the script file as follows:

[["w", "fs", "fs.bin"],
 ["w", "fs1", "fs1.bin"]]

On a 4MB device, only the file system fs.bin is flashed, while on an 8MB device, both fs.bin and fs1.bin are flashed. If you target only 8MB devices such as the GPy, FiPy or LoPy4, you can omit the fs partition.

Finally, when flashing firmware images, the new partition table becomes even more beneficial. As the firmware binaries are smaller than their respective partitions, we first erase pretty much everything but the fs, nvs and config partitions. However, as the new "w" command automatically appends 0xFF to fill up the partition which is the same value that is written during erase, the erase is no longer needed. Additionally, the firmware updater is now able to write to either the factory or ota_0 partition, and if a firmware image is written to ota_0, the otadata partition is updated in order to activate this partition. If a firmware is only written into the factory partition, and the otadata partition has not been otherwise erased or written via the script, it is updated to boot from the factory partition. This allows you to distribute two firmwares in the same file, one written into the ota_0 partition and activated by default and the other when using the long safeboot option. We might consider adding a switch in future firmwares to toggle between factory and ota_0 firmware, which makes testing new features a lot less painful as you don't need to downgrade if things go wrong.

A final example putting together all of the above:

[["e", "nvs"],
 ["w", "bootloader", "bootloader.bin"],
 ["w", "partitions", "partitions.bin"],
 ["w", "factory", "wipy.bin"],
 ["w", "ota_0", "wipy-test.bin"],
 ["w", "fs", "fs.bin"],
 ["w", "fs1", "fs1.bin"]]

While I have tested all of these examples many times, please be aware that this is still a development / beta version and it is possible that there are some odd combinations that I haven't thought about that will throw an error or simply don't flash the device properly.

So please make sure to give this a good testing and report any issues you may encounter so we can fix them as quickly as possible in order to release this new updater as a stable version.