Wipy img max size

[serafimsaudade]

I build my custom firmware. The fw size have grow a lot. Now when I try to program my Wipy3 I'm getting the flowing msg:

Release package WiPy-1.20.1.r1.tar.gz created successfully.
checking size of image
1861824 bytes => Firmware image size exceeds avialable space on board!

But if I program my Lopy4 evrething runs OK.

Release package LoPy4-1.20.1.r1.tar.gz created successfully.
checking size of image
1944048 bytes => Size OK

I did some investigation, and check the size_check.sh code, and found this:

if [ "${BOARD}" != "LOPY4" -a "${BOARD}" != "GPY" -a "${BOARD}" != "FIPY" ] ; then
  IMG_MAX_SIZE=${IMG_MAX_SIZE_4MB}
 else
  IMG_MAX_SIZE=${IMG_MAX_SIZE_8MB}
fi

Why the wipy3 fw img max size is limited to 4MB?

[robert-hh]

@serafimsaudade Because WiPy comes in two flavours, WiPy2 with 4MB flash and WiPy3 with 8 MB flash. Since you are builing your own firmware, you have two options:
a) Change the build script
b) change the partition map in esp32/boards/WIPY

Option a) is easier. In any case you have to erase flash to ensure a clean new set-up.

[serafimsaudade]

@robert-hh
Ahh Ok, tks for the explanation.

[dutifulweave]

@andreas said in Pytest integration:

Dear @Summer,

pytest itself will probably never work on MicroPython. You might also want to follow the discussion at [1].

However, we have been working on building a test harness for our Terkin Datalogger based on pytest just a few days ago, see [2]. The main job was to apply appropriate mocking to the MicroPython ESP32 HALs and to make different modules from the MicroPython standard library compatible with CPython.

So, the test harness will not run on the device itself, but within a regular CPython. Let me know what you are aiming at and if I can support you in any way. In fact, I was planning to put all generic parts into a different package so it can be reused by others.

With kind regards,
Andreas.

[1] https://github.com/pfalcon/pycopy-lib/issues/49#issuecomment-603875987
[2] https://github.com/hiveeyes/terkin-happy wheels-datalogger/tree/master/test

Thank u for the good informations! It's working!

[rcolistete]

IMHO, WIPY should be divided in WIPY2 (with 4MB of flash) and WIPY3 (with 8MB of flash).

As the Pycom firmwares are larger than 1-2 years ago, some extra features (frozen modules, user C modules, double precision, etc) don't build for WIPY target.

[robert-hh]

@rcolistete There is just one BOARD type for for WiPy2 and WiPy3.
Frozen modules build for that board type. With WiPy2, only the available space is somewhat limited, and the file system is small. The BASE variant allows for more frozen modules, than the PYBYTES variant.

Anyhow: With the next 1.20.3 revision of the firmware the "old" devices like WiPy2 and LoPy will not be supported any more. Which is kind of sad, because these are more robust then the newer ones. I have a LoPy running as single channel Lora gateway since more than a year, and the only case it stopped unintentional was during a power outage.

[rcolistete]

@robert-hh said in Wipy img max size:

Anyhow: With the next 1.20.3 revision of the firmware the "old" devices like WiPy2 and LoPy will not be supported any more. Which is kind of sad, because these are more robust then the newer ones. I have a LoPy running as single channel Lora gateway since more than a year, and the only case it stopped unintentional was during a power outage.

LoPy v1, WiPy 2 and SiPy are more robust due to not using PSRAM ? Or there is another reason ?

If so, it would be useful (for some applications) to have FiPy/GPy/LoPy4/WiPy3 with optional firmwares without using PSRAM.

[robert-hh]

@rcolistete said in Wipy img max size:

LoPy v1, WiPy 2 and SiPy are more robust due to not using PSRAM ?

That's my guess. But the PSRAM boards got better. I have now a LoPy4 running since 37 days with a simple test loop code, in which it failed regularly at another forum member.

[rcolistete]

@robert-hh said in Wipy img max size:

@rcolistete There is just one BOARD type for for WiPy2 and WiPy3.
Frozen modules build for that board type. With WiPy2, only the available space is somewhat limited, and the file system is small. The BASE variant allows for more frozen modules, than the PYBYTES variant.

So I've finally understood, well, I think...

I've tested a WiPy 3 and even using the common firmware for WiPy (2 & 3) :

• the 'factory' partition has 1980 kB, like FiPy, LoPy4, GPy, as it is shown by ./pycom-fwtool-cli -v -p /dev/ttyACM0 flash -t <firmware.tar.gz>;
• the file system has approx. 4 MB (4064 kB), as shown by os.getfree('/flash').

So all the 8 MB of QSPI external flash is used.

Only issue is building a firmware, where the limitation for WiPy 2 ('factory' partition has 1720 kB) is used by 'esp32/tools/size_check.sh' for board=WIPY (which includes WiPy 2 & 3). One 'dirty' solution is simply edit this script, commenting line 58.

[rcolistete]

Some data about (firmware partition size - firmware size = free space), using firmware 1.20.2.rc10 daily (from August 4th 2020), on :

• WiPy 2 :
  • (1761280 - 1586304 = 174976) bytes = 171 kB free;
• WiPy 3 :
  • (2027520 - 1586304 = 441216) bytes = 431 kB free.

PyBytes adds aprox. 88 kB to the firmware (with single precision).
FP64 (double precision) adds aprox. 14 kB to the firmware (with single precision).
ulab in single precision adds approx. 66 kB to the firmware, while in double precision adds about 77 kB.

[kiltedpython]

So does this mean that if someone made a 8mb built image that it would work on the wipy3?

If that is the case then patching the build script for a board called "WIPY3" wouldn't be hard, in fact already did the stoof. Just wondering if a pull request would be worth it or not.

[robert-hh]

@kiltedpython With the next revision of the firmware and it's move to IDF 4.0 there will be the need to split the board definitions for WiPy2 and WiPy3, like is was done for LoPy and LoPy4. This new firmware will not fit any more for 4 MB flash devices. So the change will come anyhow.
Edit: Pycom may as well decide to cease the support for the 4MB flash boards.

[kiltedpython]

@robert-hh Thanks. That is good info.
Still wondering if changing the script to the below that the created image would still work or is there some partition mapping that should be considered. Tried to follow the build workflow to change all the things but has anyone tried it?

diff --git a/esp32/tools/makepkg.sh b/esp32/tools/makepkg.sh
index b7f041f4c..d5f4887ee 100755
--- a/esp32/tools/makepkg.sh
+++ b/esp32/tools/makepkg.sh
@@ -20,7 +20,7 @@ if ! [ $0 = "tools/makepkg.sh" ]; then echo >&2 "Need to run as tools/makepkg.sh
 if ! [ -d boards/$1 ]; then echo >&2 "Unknown board type!"; exit 1; fi
 if ! [ -d ${BUILD_DIR} ]; then echo >&2 "Build directory for $1 not found! Run make BOARD=$1 first!"; exit 1; fi
 if ! [ -r "pycom_version.h" ]; then echo >&2 "Cannot read pycom_version.h! Aborting."; exit 1; fi
-if [ "${BOARD}" = "GPY" -o  "${BOARD}" = "LOPY4" -o "${BOARD}" = "FIPY" ]; then
+if [ "${BOARD}" = "GPY" -o  "${BOARD}" = "LOPY4" -o "${BOARD}" = "FIPY" -o "${BOARD}" = "WIPY3" ]; then
     if ! [ -r "boards/$1/${SCRIPT_NAME_8MB}" ]; then echo >&2 "Cannot read boards/$1/${SCRIPT_NAME_8MB}! Aborting."; exit 1; fi
 elif [ "${BOARD}" = "LOPY" -o "${BOARD}" = "WIPY" ]; then
     if ! [ -r "boards/$1/${SCRIPT_NAME_4MB}" ]; then echo >&2 "Cannot read boards/$1/${SCRIPT_NAME_4MB}! Aborting."; exit 1; fi
diff --git a/esp32/tools/mpy-build-check.sh b/esp32/tools/mpy-build-check.sh
index 5556936d2..21d46516a 100644
--- a/esp32/tools/mpy-build-check.sh
+++ b/esp32/tools/mpy-build-check.sh
@@ -22,7 +22,7 @@ if ! [ $0 = "tools/mpy-build-check.sh" ]; then
 fi
 
 # Check board type
-if [ "${BOARD}" != "WIPY" -a "${BOARD}" != "SIPY" -a "${BOARD}" != "LOPY" -a "${BOARD}" != "LOPY4" -a "${BOARD}" != "GPY" -a "${BOARD}" != "FIPY" ] ; then
+if [ "${BOARD}" != "WIPY" -a "${BOARD}" != "WIPY3" -a "${BOARD}" != "SIPY" -a "${BOARD}" != "LOPY" -a "${BOARD}" != "LOPY4" -a "${BOARD}" != "GPY" -a "${BOARD}" != "FIPY" ] ; then
   echo "Invalid Board name for MPY build!" >&2
   exit 1
 fi
diff --git a/esp32/tools/size_check.sh b/esp32/tools/size_check.sh
index 6498f25e0..8895a4ff1 100644
--- a/esp32/tools/size_check.sh
+++ b/esp32/tools/size_check.sh
@@ -15,7 +15,7 @@ if ! [ $0 = "tools/size_check.sh" ]; then
 fi
 
 #Check Board Type
-if [ "${BOARD}" != "WIPY" -a "${BOARD}" != "SIPY" -a "${BOARD}" != "LOPY" -a "${BOARD}" != "LOPY4" -a "${BOARD}" != "GPY" -a "${BOARD}" != "FIPY" ] ; then
+if [ "${BOARD}" != "WIPY" --a "${BOARD}" != "WIPY3" a "${BOARD}" != "SIPY" -a "${BOARD}" != "LOPY" -a "${BOARD}" != "LOPY4" -a "${BOARD}" != "GPY" -a "${BOARD}" != "FIPY" ] ; then
   echo "Invalid Board name for MPY build!" >&2
   exit 1
 fi
@@ -58,4 +58,4 @@ if [ ${total_size} -gt ${IMG_MAX_SIZE} ] ; then
   exit 1
 else
   echo "${total_size} bytes => Size OK" >&2
-fi
\ No newline at end of file
+fi

Thanks.

[rcolistete]

@kiltedpython WIPY symbol is also used in other files : esp32/application.mk, esp32/mods/machspi.c, etc.

I think a PR separting WiPy (2) from WIPY3 is worth, even if it is not accepted by Pycom, a branch would be useful for other users.

[robert-hh]

@kiltedpython There is also size_check.sh, application.mk. You have to create board subdirectories and change the respective partition map files to get actually space for larger images. For WiPy3, you can use e.g. LoPy4 as template.

[Colateral]

This post is deleted!