New Beta Release v1.20.3.b3

[peterp]

Dear Pycom community!

We have a new Beta release available: v1.20.3.b3.

This firmware contains major upgrades of the IDF and Micropython and a range of changes to Pycom specific features. The IDF and Micropython are foundational pieces of our firmware. The IDF is provided by Espressif, the makers of the ESP32. Compared to the latest v1.20.2 releases this new beta firmware brings the IDF from v3.3.1 to v4.1. The Micropython version is upgraded to version 1.13.

This is a Beta release which means, we do not advise you install this on production devices. Some reports we have received describe stability improvements, while others point to some problems which still need investigation. Some known issues are listed at the bottom.

We would love to hear your feedback! What works better for you? What has regressed compared to the regular v1.20.2 firmware?

Changes:

• IDF v4.1
• Micropython v1.13
• LoRaWAN 1.0.3 (Class A& C)
• Support for new regions (CN779, RU864 and KR920)
• BLE Mesh
• Reworked Coap with DTLS support
• ESP-NOW
• WiPy2, SiPy, and LoPy1 and are not supported anymore since they only have 4MB flash.
• WiPy3, LoPy4, FiPy, GPy, W01, L01, L04 and G01 are all supported
• improved wear levelling on LittleFS (thank you @robert-hh!)
• bugfixes

This firmware also contains improvements to the OTA (over the air) update functionality:

• the main firmware partition is larger than before
• there is a small 'factory' firmware partition, just enough for OTA
• support for compression and
• support for differential updates

The documentation and infrastructure (software server, installer, etc) for these OTA features will be released separately.

Known issues:

• Soft reset (Ctrl-D) causes core dumps
• No OTA support from Pybytes yet
• No ethernet support on Pygate yet
• Pygate on Gpy and Lopy4 seems to run out of memory. On Wipy there is just enough memory - For testing with wifi one could flash the Wipy firmware.
• Secure boot is not supported

You can

• install this update with the Firmware Updater,
• find the source code and detailed commit log in the git branch master_idf_v4.1,
• download the firmware packages (.tar.gz) from the docs and
• get the .elf files from the github release for debugging.
• Use toolchain version 2020r3.

[robert-hh]

@peterp Thanks for mentioning. Building failed. Compile works to far, but linking faild. Below are the first error messages of many:

gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/robert/Downloads/MicroPython/pycom-esp-idf/components/esp32/lib/libnet80211.a(ieee80211_api.o): in function `esp_wifi_deinit':
(.text.esp_wifi_deinit+0x60): multiple definition of `esp_wifi_deinit'; lib/libesp_wifi.a(wifi_init.o):/home/gezahusi/Work/Repositories/pycom-esp-idf/components/esp_wifi/src/wifi_init.c:111: first defined here
/home/robert/Downloads/xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/robert/Downloads/MicroPython/pycom-esp-idf/components/esp32/lib/libsmartconfig.a(sc_sniffer.o): in function `esp_smartconfig_stop':
(.text.esp_smartconfig_stop+0x0): multiple definition of `esp_smartconfig_stop'; lib/libesp_wifi.a(smartconfig.o):/home/gezahusi/Work/Repositories/pycom-esp-idf/components/esp_wifi/src/smartconfig.c:68: first defined here
/home/robert/Downloads/xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/robert/Downloads/MicroPython/pycom-esp-idf/components/esp32/lib/libsmartconfig.a(sc_sniffer.o): in function `esp_smartconfig_start':
(.text.esp_smartconfig_start+0x0): multiple definition of `esp_smartconfig_start'; lib/libesp_wifi.a(smartconfig.o):/home/gezahusi/Work/Repositories/pycom-esp-idf/components/esp_wifi/src/smartconfig.c:51: first defined here
/home/robert/Downloads/xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: lib/libbt.a(bt.o):(.literal.btdm_check_and_init_bb+0x4): undefined reference to `btdm_rf_bb_init_phase2'
/home/robert/Downloads/xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: lib/libbt.a(bt.o):(.literal.btdm_sleep_exit_phase3_wrapper+0x0): undefined reference to `btdm_controller_get_sleep_mode'
/home/robert/Downloads/xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: lib/libbt.a(bt.o):(.literal.coex_bt_wakeup_request+0x0): undefined reference to `btdm_power_state_active'
/home/robert/Downloads/xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: lib/libbt.a(bt.o):(.literal.coex_bt_wakeup_request+0x4): undefined reference to `btdm_wakeup_request'
/home/robert/Downloads/xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: lib/libbt.a(bt.o):(.literal.esp_vhci_host_send_packet+0x0): undefined reference to `btdm_wakeup_request_end'

[peterp]

@robert-hh said in New Beta Release v1.20.3.b3:

@peterp Thanks for mentioning. Building failed. Compile works to far, but linking faild. Below are the first error messages of many:

Mhm, Correct toolchain?
xtensa-esp32-elf-gcc --version

Correct idf version?
cd $IDF_PATH ; git rev-parse --short HEAD

[robert-hh]

@peterp There is a toolchain and idf version check in Makefile. They pass. And if you look at the short excerpt, you see xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64.

[peterp]

@robert-hh

git status ; git submodule update --init --recursive
in both folders please - that is, assuming you did get the code via git clone, yes?!

what make command you use to build? I can try a new clone ... but that's what Jenkins did anyway, well, I'm trying ...

[robert-hh]

@peterp Git status in the source directory:

Auf Branch master_idf_v4.1
Ihr Branch ist auf demselben Stand wie 'upstream/master_idf_v4.1'.

git status in pycom-esp-idf:

HEAD losgelöst bei d647204d1

I did not make a new close, but checked out in the existing directories. I can clone new copies.
I just call make, which calls a GNUmakefile first, which sets LoPy4 as default device and sets PATH and IDF_PATH.

[peterp]

it works for me. please do check the git submodule update --init --recursive

my test:

mkdir ~/build-check
cd ~/build-check

git clone --recursive -b idf_v4.1 https://github.com/pycom/pycom-esp-idf.git
cd pycom-esp-idf
git checkout d647204 && git submodule sync && git submodule update --init --recursive
cd ..

git clone --recursive -b master_idf_v4.1 https://github.com/pycom/pycom-micropython-sigfox.git
export IDF_PATH=~/build-check/pycom-esp-idf/
cd pycom-micropython-sigfox/
cd mpy-cross
make
cd ..

cd esp32
xtensa-esp32-elf-gcc --version
xtensa-esp32-elf-gcc (crosstool-NG esp-2020r3) 8.4.0
make BOARD=LOPY4

...
CC build/LOPY4/release/pins.c
AR build/LOPY4/release/application.a
CPP build/LOPY4/release/esp32_out.ld
LINK build/LOPY4/release/application.elf
xtensa-esp32-elf-size build/LOPY4/release/application.elf
   text	   data	    bss	    dec	    hex	filename
1727558	 283960	  87726	2099244	 20082c	build/LOPY4/release/application.elf
Building partitions from lib/partitions_4MB.csv...
Building partitions from lib/partitions_8MB_small_factory_fw.csv...
IMAGE build/LOPY4/release/lopy4.bin
esptool.py v2.9-dev

[DualMatic]

Hello @peterp

Regarding:

• Micropython v1.13

  • It's ULP implemented from original micropython? I want to do some tests or figure out ULP processor, since I'm waiting Pycom to support in the firmware.
    .

• LoRaWAN 1.0.3 (Class A& C)

  • What are the improvements from version 1.0.2. I'm looking if DeviceTimeReq, DeviceTimeAns is implemented on Pycom.

Thanks in advance and good looking release!! ;)

[robert-hh]

@peterp I did the git submodule updates & everything. Made a new clone of the micropython and esp-idf directories. No luck.

[DualMatic]

@robert-hh For me is compiling okay. I started under a fresh debian virtual machine BTW.

Just two warnings:

In file included from main.c:54:
/home/esp/pycom-esp-idf/components/esp_event/include/esp_event_loop.h:2:2: warning: #warning "esp_event_loop.h is deprecated, please include esp_event.h instead" [-Wcpp]
 #warning "esp_event_loop.h is deprecated, please include esp_event.h instead"
  ^~~~~~~
main.c: In function 'app_main':
main.c:140:5: warning: 'esp_event_loop_init' is deprecated [-Wdeprecated-declarations]
     ESP_ERROR_CHECK(esp_event_loop_init(app_sys_event_handler, NULL));
     ^~~~~~~~~~~~~~~
In file included from /home/esp/pycom-esp-idf/components/esp_event/include/esp_event.h:26,
                 from /home/esp/pycom-esp-idf/components/esp_event/include/esp_event_loop.h:3,
                 from main.c:54:
/home/esp/pycom-esp-idf/components/esp_event/include/esp_event_legacy.h:227:11: note: declared here
 esp_err_t esp_event_loop_init(system_event_cb_t cb, void *ctx) __attribute__ ((deprecated));

mods/modnetwork.c: In function 'network_select_nic':
mods/modnetwork.c:217:18: warning: assignment discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers]
         nic_type = mp_obj_get_type(nic_chosen);

I had to add github SSH key from my machine to github to avoid submodules access error, maybe this is your problem?

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:espressif/esp32-wifi-lib.git' into submodule path '/home/esp/pycom-esp-idf/components/esp_wifi/lib' failed
Failed to clone 'components/esp_wifi/lib'. Retry scheduled

BR

[robert-hh]

@DualMatic said in New Beta Release v1.20.3.b3:

I had to add github SSH key from my machine to github to avoid submodules access error

That could be the problem since all the missing links are related to wifi. How did you add the WIFi key?

But this lib exists. Here is the list of submodules.

 3b66e5b051381fb70de9c2791df70a06181c64e3 components/asio/asio (asio-1-12-0-200-g3b66e5b0)
 d037ec89546fad14b5c4d5456c2e23a71e554966 components/bootloader/subproject/components/micro-ecc/micro-ecc (v1.0)
 238fc047f2f63d99482aa51996f14ca134096310 components/bt/controller/lib (238fc04)
 fead24e5d5c0f4bd9b8c3d71cf72a87d75631399 components/bt/host/nimble/nimble (remotes/origin/master-254-gfead24e5)
 d2dd95cb8841d88d5a801e3ef9c328fd6200e7bd components/cbor/tinycbor (v0.5.2-17-gd2dd95c)
 98954eb30a2e728e172a6cd29430ae5bc999b585 components/coap/libcoap (v4.2.0-rc4-114-g98954eb)
 a17182e903a9e161a98930648eb6ff78019708ab components/esp_wifi/lib (a17182e)
 a66ee2da5ad059bb4d90e8a7317f907add1d8419 components/esptool_py/esptool (v2.8-23-ga66ee2d)
 968b8cc46dbee47b83318d5f31a8e7907199614b components/expat/expat (R_2_2_5)
 3c8935676a97c7c97bf006db8312875b4f292f6c components/json/cJSON (v1.7.12)
 70170c28c844a4786e75efc626e1aeebc93caebc components/libsodium/libsodium (1.0.12)
 50e4e0d1821a6e5deab0ead9f35ffb4e60625ea6 components/lwip/lwip (remotes/origin/master-194-g50e4e0d1)
 9ef92c551eb8d92677034c3ec8078a8076febf41 components/mbedtls/mbedtls (mbedtls-2.13.1-1358-g9ef92c551)
 6bc94add892437d0fd50f26bfabe78c646648c13 components/mqtt/esp-mqtt (ESP-MQTT_FOR_IDF_3.1-170-g6bc94ad)
 3bcc416e13cc790e2fb45fcfe9111d38609c5032 components/nghttp/nghttp2 (v1.24.0)
 dac1a65feac4ad72f612aab99f487056fbcf5c1a components/protobuf-c/protobuf-c (v1.3.0)
 f5e26c4e933189593a71c6b82cda381a7b21e41c components/spiffs/spiffs (0.2-221-gf5e26c4)
 7d2bf62b7e6afaf38153041a9d53c21aeeca9a25 components/unity/unity (v2.4.3-51-g7d2bf62)
 7e8e249990ec491ec15990cf95b6d871a66cf64a examples/build_system/cmake/import_lib/main/lib/tinyxml2 (2.0.2-659-g7e8e249)

[robert-hh]

@peterp Did the same steps with the same subdir: Same output besides the final linking step.
P.S.: I had quite a few build systems creations in the past

[DualMatic]

@robert-hh you must generate an SSH key in your machine:
Generating SSH key

Add the key to your SSH Agent
Add to SSH Agent

Once generated copy and paste it on your github settings:
Add SSH key to Github Account

Clone again the repository with --recursive option

Best regards, hope this is useful to you!

[robert-hh]

@DualMatic No Sir. No joy. The SSH key was installed. Would you mind to compare the list of submodule versions to your's?

[DualMatic]

@robert-hh said in New Beta Release v1.20.3.b3:

P.S.: I had quite a few build systems creations in the past

Yeah I know ;) (I'm novice user RG build enviroments) just I wanted to help you a little bit and return your past support my old username

@robert-hh said in New Beta Release v1.20.3.b3:

@DualMatic No Sir. No joy. The SSH key was installed. Would you mind to compare the list of submodule versions to your's?

/esp/pycom-micropython-sigfox# git submodule status
 d270f79aa16dd8fd4ae3b6c14544283dcb992e9c lib/asf4 (heads/master)
 43a6e6bd3bbc03dc501e16b89fba0ef042ed3ea0 lib/axtls (heads/master-38-g43a6e6b)
 35aaec4418ad78628a3b935885dd189d41ce779b lib/berkeley-db-1.xx (heads/master-7-g35aaec4)
 c8b9823f68c6af0fa52e2c4e009aba4dbf257232 lib/btstack (v1.2~13^2~33)
 e9de7e35f2339598b16cbb375f9992643ed81209 lib/libffi (v3.2.1-146-ge9de7e3)
 159e31b689577dbf69cf0683bbaffbd71fa5ee10 lib/lwip (STABLE-2_1_2_RELEASE)
 3f8d78411a26e833db18d9fbde0e2f0baeda87f0 lib/mbedtls (mbedtls-2.17.0)
 97ce3eacaaa79e8ed6cf71717149ced4f5328ee7 lib/mynewt-nimble (nimble_1_3_0_tag)
 7a4c9d946cf1801771fc180acdbf7b878f270093 lib/nrfx (v2.0.0)
 b618cb1d521cc9e133bdcd0fca154dee2d925dfe lib/nxp_driver (b618cb1)
 58fee7c92bd576814d3f2afd92fbc62990270ecc lib/stm32lib (F0-1.9.0+F4-1.16.0+F7-1.7.0+H7-1.6.0+L0-1.11.2+L4-1.8.1+WB-1.6.0-19-g58fee7c)
 a6b916ba85bef6aad50f1652532b02984dfe2484 lib/tinyusb (0.5.0-680-ga6b916ba)

/esp/pycom-esp-idf# git submodule status
 3b66e5b051381fb70de9c2791df70a06181c64e3 components/asio/asio (asio-1-12-0-200-g3b66e5b0)
 d037ec89546fad14b5c4d5456c2e23a71e554966 components/bootloader/subproject/components/micro-ecc/micro-ecc (v1.0)
 238fc047f2f63d99482aa51996f14ca134096310 components/bt/controller/lib (238fc04)
 fead24e5d5c0f4bd9b8c3d71cf72a87d75631399 components/bt/host/nimble/nimble (remotes/origin/master-254-gfead24e5)
 d2dd95cb8841d88d5a801e3ef9c328fd6200e7bd components/cbor/tinycbor (v0.5.2-17-gd2dd95c)
 98954eb30a2e728e172a6cd29430ae5bc999b585 components/coap/libcoap (v4.2.0-rc4-114-g98954eb)
 a17182e903a9e161a98930648eb6ff78019708ab components/esp_wifi/lib (a17182e)
 a66ee2da5ad059bb4d90e8a7317f907add1d8419 components/esptool_py/esptool (v2.8-23-ga66ee2d)
 968b8cc46dbee47b83318d5f31a8e7907199614b components/expat/expat (R_2_2_5)
 3c8935676a97c7c97bf006db8312875b4f292f6c components/json/cJSON (v1.7.12)
 70170c28c844a4786e75efc626e1aeebc93caebc components/libsodium/libsodium (1.0.12)
 50e4e0d1821a6e5deab0ead9f35ffb4e60625ea6 components/lwip/lwip (remotes/origin/master-194-g50e4e0d1)
 9ef92c551eb8d92677034c3ec8078a8076febf41 components/mbedtls/mbedtls (mbedtls-2.13.1-1358-g9ef92c551)
 6bc94add892437d0fd50f26bfabe78c646648c13 components/mqtt/esp-mqtt (ESP-MQTT_FOR_IDF_3.1-170-g6bc94ad)
 3bcc416e13cc790e2fb45fcfe9111d38609c5032 components/nghttp/nghttp2 (v1.24.0)
 dac1a65feac4ad72f612aab99f487056fbcf5c1a components/protobuf-c/protobuf-c (v1.3.0)
 f5e26c4e933189593a71c6b82cda381a7b21e41c components/spiffs/spiffs (0.2-221-gf5e26c4)
 7d2bf62b7e6afaf38153041a9d53c21aeeca9a25 components/unity/unity (v2.4.3-51-g7d2bf62)
 7e8e249990ec491ec15990cf95b6d871a66cf64a examples/build_system/cmake/import_lib/main/lib/tinyxml2 (2.0.2-659-g7e8e249)

PS. cannot reply on time because not sufficient reputation

[robert-hh]

@DualMatic said in New Beta Release v1.20.3.b3:

PS. cannot reply on time because not sufficient reputation

I added a few.

[robert-hh]

@peterp Building in a up-to-now rarely used mint linux virtual machine worked. So if have to tell now what is interfering with the other build process.
Edit: copying back the pycom-esp-idf directory from the virtual machine to my primary machine worked too. So: Why did the esp-idf not install properly?
Edit2: Found the culprit: two additional lib directories in components/bt and components/esp32. But HOW did they get into the fresh cloned esp-idf? They are in the linker search directories, hmmm.

[robert-hh]

@peterp said in New Beta Release v1.20.3.b3:

Soft reset (Ctrl-D) causes core dumps

Commenting out 'uart_deinit_all();' on line 381 in mptask.c lets the code survive soft reset.
So maybe the delay is to short for the 'MPY:soft reboot' message to get sent.
Yup: Increasing the delay at line 379 to 50000 does the trick.

[peterp]

@robert-hh said in New Beta Release v1.20.3.b3:

two additional lib directories in components/bt and components/esp32. But HOW did they get into the fresh cloned esp-idf? They are in the linker search directories, hmmm.

Maybe some git submodule stuff? I recall some trouble when doing a git checkout that is "too far". E.g., git clone idf3.3.1 and then git checkout idf4.1 would lead to some messed up git submodules and it was just easier to make a fresh git clone from a branch that is just a few commits away from the IDF_HASH without any major changes to submodules and then you can git checkout IDF_HASH and have all the submodules correct.

Did those two directories not show up with git status? I'm sure there is a " correct " way with git to do this checkout in any existing clone, but I couldn't figure it out and just settled on a new clone from a "close" branch as shown above.

[peterp]

@robert-hh said in New Beta Release v1.20.3.b3:

@peterp said in New Beta Release v1.20.3.b3:

Soft reset (Ctrl-D) causes core dumps

Commenting out 'uart_deinit_all();' on line 381 in mptask.c lets the code survive soft reset.
So maybe the delay is to short for the 'MPY:soft reboot' message to get sent.
Yup: Increasing the delay at line 379 to 50000 does the trick.

Awesome @robert-hh I'll check that today!