How to build Micropython from Source incuding esp-idf components

[this.wiederkehr]

Has anyone figured out how to build the current pycom micropython from source including build of the esp-idf components which right now are supplied as object code in the repo (pycom-micropython-sigfox/esp32/lib/)? That method should honour sdkconfig.h or any changes to it.

[campbellsinclair]

Refer to this thread for details on how Pycom builds the libraries:

https://forum.pycom.io/topic/2835/is-it-possible-to-recompile-pycom-s-esp-idf-with-different-settings

[tao.ye]

@husigeza awesome!!! I was trying to enable ULP on Pycom and almost gave up because pycom's lib was built without CONFIG_ULP_COPROC_ENABLED=y. This solution works without issue for me.

[loboris]

I've managed to build MicroPython in native esp-idf way, as esp-idf component. Pycom's MicroPython could be possible adapted to the same build system.
It works very well, no more manual sdkconfig.h tweaking is needed, MicroPython options can be configured from menuconfig, ESP32 can operate in unicore or dualcore mode, SPI Flash can operate in any mode and speed supported (QIO, QOUT, DIO, DOUT; 20/26/40/80 Mhz), many things works better because it is build as it should (from esp-idf's point of view).
psRAM is also suported (It is huge difference between MicroPython running with less than 100KB of free memory and running with 4MB of free memory.).

The project is available on GitHub.
Tested on ESP-WROVER-KIT with ESP-WROVER module with 4MB psRAM.

[this.wiederkehr]

@husigeza awsome!!! :-) Works without issues for me.

[husigeza]

@this.wiederkehr I have a working solution.
My sdkconfig file: https://drive.google.com/file/d/0BzYGHIbT62_UQzBDUkl3QkJMZFU/view?usp=sharing
Steps:

• Copy this sdkconfig and partitions.csv from pycom-micropython-sigfox/esp32/lib into pycom-esp-idf/examples/get-started/hello_world
• make menuconfig if you want to change something
• run "make all"
• Copy the necessary application libraries into pycom-micropython-sigfox/esp32/lib and the bootloader libraries to pycom-micropython-sigfox/esp32/bootloader/lib
• Replace the pycom-micropython-sigfox/esp32/sdkconfig.h with the generated sdkconfig.h (can be found in build/include/sdkconfig.h)
• Connect the board, connect the relevant pins, and "make erase" - this is needed because the bootloader will be recompiled with other parameters than the current one
• From pycom-micropython-sigfox/esp32 compile the application and the bootloader
• Flash the board

The linked sdkconfig seems okay, I tested the Wifi, Lora (in LoRaRAW), FTP and Telnet server functionalities.

To make your life easier, here is my bash script copying the relevant libraries (has to be executed from pycom-esp-idf/examples/get-started/hello_world):

ESP32_PATH=your_path_to_pycom-micropython-sigfox/esp32

cp -v ./build/include/sdkconfig.h $ESP32_PATH

cp -v ./build/bootloader/bootloader_support/libbootloader_support.a $ESP32_PATH/bootloader/lib/
cp -v ./build/bootloader/log/liblog.a $ESP32_PATH/bootloader/lib/
cp -v ./build/bootloader/micro-ecc/libmicro-ecc.a $ESP32_PATH/bootloader/lib/
cp -v ./build/bootloader/soc/libsoc.a $ESP32_PATH/bootloader/lib/
cp -v ./build/bootloader/spi_flash/libspi_flash.a $ESP32_PATH/bootloader/lib/

cp -v ./build/bt/libbt.a $ESP32_PATH/lib/
cp -v ./build/cxx/libcxx.a $ESP32_PATH/lib/
cp -v ./build/driver/libdriver.a $ESP32_PATH/lib/
cp -v ./build/esp32/libesp32.a $ESP32_PATH/lib/
cp -v ./build/expat/libexpat.a $ESP32_PATH/lib/
cp -v ./build/freertos/libfreertos.a $ESP32_PATH/lib/
cp -v ./build/jsmn/libjsmn.a $ESP32_PATH/lib/
cp -v ./build/json/libjson.a $ESP32_PATH/lib/
cp -v ./build/log/liblog.a $ESP32_PATH/lib/
cp -v ./build/lwip/liblwip.a $ESP32_PATH/lib/
cp -v ./build/mbedtls/libmbedtls.a $ESP32_PATH/lib/
cp -v ./build/micro-ecc/libmicro-ecc.a $ESP32_PATH/lib/
cp -v ./build/newlib/libnewlib.a $ESP32_PATH/lib/
cp -v ./build/nghttp/libnghttp.a $ESP32_PATH/lib/
cp -v ./build/nvs_flash/libnvs_flash.a $ESP32_PATH/lib/
cp -v ./build/openssl/libopenssl.a $ESP32_PATH/lib/
cp -v ./build/sdmmc/libsdmmc.a $ESP32_PATH/lib/
cp -v ./build/soc/libsoc.a $ESP32_PATH/lib/
cp -v ./build/spi_flash/libspi_flash.a $ESP32_PATH/lib/
cp -v ./build/tcpip_adapter/libtcpip_adapter.a $ESP32_PATH/lib/
cp -v ./build/vfs/libvfs.a $ESP32_PATH/lib/
cp -v ./build/wpa_supplicant/libwpa_supplicant.a $ESP32_PATH/lib/
cp -v ./build/xtensa-debug-module/libxtensa-debug-module.a $ESP32_PATH/lib/

[this.wiederkehr]

just to let anyone know, the above method does not work correctly. Its not building the libraries :-( got to do more work on it....

[this.wiederkehr]

@jmarcelino here is my approach which seems to work just fine. I'm using the hello_world example with an adjusted make file. ESP-IDF supports to add the components which you wanna build by a variable COMPONENTS build.
Copying sdkconfig back and forth is to make sure the one from the pycom repo is used.

adjusted make file placed in esp32/dummy/Makefile:

PROJECT_NAME := hello-world
COMPONENTS=core bt cxx driver esp32 expat jsmn json log lwip mbedtls mds micro-ecc newlib nghttp nvs_flash openssl sdmmc soc spi_flash tcpip_adapter vfs wp_supplicant xtensa-debug-module
include $(IDF_PATH)/make/project.mk

commands to execute in bash (except from my ci-script). to be executed from the root path of the pycom-micropython-sigfox repo:

  - ROOT_CI_PATH=$(pwd)
  - cp esp32/sdkconfig.h $IDF_PATH/examples/get-started/hello_world/sdkconfig
  - rm $IDF_PATH/examples/get-started/hello_world/Makefile
  - cp esp32/dummy/Makefile $IDF_PATH/examples/get-started/hello_world/Makefile
  - cd $IDF_PATH/examples/get-started/hello_world/
  - make all BATCH_BUILD=1 || true
  - find $IDF_PATH -name '*.a' -exec cp '{}' $ROOT_CI_PATH/esp32/lib/ \;
  - cd $ROOT_CI_PATH
  - cd mpy-cross
  - make all
  - cd ../esp32
  - cp sdkconfig.h sdkconfig
  - make BOARD=LOPY -j5 TARGET=boot
  - make BOARD=LOPY -j5 LORA_BAND=USE_BAND_868 TARGET=app

make all BATCH_BUILD=1 || true this command returns with an error therefore the ||true

[jmarcelino]

@this.wiederkehr

Yes, essentially I build one of the esp-idf examples (I picked a2dp example but later found this wasn't a great choice because it needs Bluetooth classic ) but using the sdkconfig.h from the MicroPython esp32 folder. This generates the component libs which I then replace in the MicroPython distro.

It took a lot of tweaks and I'm not 100% confident in it but seems to work in my testing. It would be better to have the original build process though. This is actually something I want to talk to with Pycom team at the hackaton this Saturday.