Loading pyc Files

[ssmith]

Is there a way to load pyc files on the LoPy? We are running into 'out of memory errors' trying to import modules even though there seems to be plenty of free memory. This is making development very difficult and time consuming.

Thanks,
Steve

[robert-hh]

@ssmith MicroPython has its own cross-compiler, called mpy-cross, which takes a .py files and creates a .mpy file, which can be imported. You must build that cross-compiler yourself. I have some versions prepared for people who want to do a quick test :
https://github.com/robert-hh/Shared-Stuff
Look for the mpy-cross-pycom* files. Using these reduces the amount needed to start a script, but still the bytecode resides in RAM.
If you set up your own build environment for the firmware, you can also embed python scripts into the flash memory. Just put them into the subdirectory esp32/frozen. Then bytecode and static data do not occupies RAM anymore, just the dynamic data, it uses.
Follow the instructions in the repositories https://github.com/pycom/pycom-micropython-sigfox and https://github.com/pycom/pycom-esp-idf. There is a actual problem: the most recent extensa gcc comiler does not match the micropython repository. And older working version is here: https://dl.espressif.com/dl/xtensa-esp32-elf-osx-1.22.0-61-gab8375a-5.2.0.tar.gz (cudos to @jmarcelino)

[caseyrodgers]

@robert-hh Thanks for info ...

Should this work? (it does not):

def hello():
-> print('Hello World')
hello()

1. mpy-cross.exe test.py
2. FTP test.mpy to /flash on board
3. From REPL on board:

import test

ImportError: no module named 'test'

That does not seem to work (fw: 1.7.9b3).

Can you please provide a simple example that should work?

[robert-hh]

@caseyrodgers That feature was recently by default enabled. V1.9.0b1 supports it.
You example should work then.

[ssmith]

@robert-hh Thanks for the info but that creates another problem. We have units in the field running different FW versions from 1.7.3 on up. Any way to allow those units to load mpy files or is that an RMA?

[robert-hh]

@ssmith Sorry no, you need the new firmware. Maybe another advantage of the newer firmware is the option to do OTA updates with it.

[ssmith]

@robert-hh I tried this and am getting this error when running 1.9.0.b1

>>> import test
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid .mpy file
>>>

I compiled it using you mpy-cross.exe

test.py
def hello_world():
	print('Hello World from precompile')
hello_world()

When I import the .py file it ran.
Any ideas?

[robert-hh]

@ssmith did you use mpy-cross_pycom.exe?
mpy-cross.exe is for the micropython.org branch, and they are not compatible.

[ssmith]

@robert-hh I did try that but maybe something was out of sync. Just started over and it did work. Thanks! What about OTA? That's is something we're going to need. Can you point me to any info on using that?

[BetterAuto]

@ssmith said in Loading pyc Files:

What about OTA?

https://forum.pycom.io/topic/518/ota-or-not-ota-that-is-the-question

Also I just posted a tutorial for compiling a custom image for frozen bytecode. MUCH faster and no more out of memory errors.
https://forum.pycom.io/topic/1919/how-i-built-a-custom-firmware-for-frozen-bytecode

[robert-hh]

@ssmith In addtion to what @BetterAuto said, a few comments:
OTA (Over The Air) stands for firmware update by WiFi.
The initial method was simply storing the appimg.bin into /flash/sys with ftp. appimg.bin is created by the build process or extracted from the pycom file package. It requires at least version 1.8.0 to be loaded initially by USB.
The primitives used by ftp upload are available in the 'pycom' module. You'll find it in the docs at https://docs.pycom.io/chapter/firmwareapi/pycom/pycom.html. These allow to create your own method for downloading and flashing the image, e.g. using a protocol which ios more secure than non-encrpyted ftp.
At the moment, there is a little hiccup involved when you mix OTA and USB flashing.
OTA uses two memory areas for the firmware and toggles these. Let's call these A and B. On bootup, the bootloader checks a flag which area is active and starts this one. The USB flash always puts the image in area A and does NOT set the "active" flag accordingly. So, if the active area is B, and you do a firmware update by USB, the new firmware will NOT be excecuted. At the moment you have two options:

a) erase flash, which will also clear the file system (not nice)
b) do another OTA, which will set again A as the acitve area. Obviously you can use the new image.

There is another unrelated hiccup when alternating between firmware images downloaded from Pycom.io and self-built images. The partition maps seem incompatible, leading to either "Red LED frozen devices" or a corrrupted file system. In that case one has to erase flash and reload with the intended method.

Both hiccups will be fixed in one of the next versions.

[ssmith]

@robert-hh Thank you for all your help. I can now build the code and even managed to write it to flash. Unfortunately, now I think I hit the double buffered code bug. My device boots up but the LED is just solid red. I only get the boot messages on the console and then nothing. How can I erase the flash?

Thanks,
Steve

[robert-hh]

@ssmith To erase flash, just issue the command:
make erase
in the PC command window. Like flashing it requires the joint between GP3 and GND, and a Reset. After that, you can reflash the device with
make BOARD=LOPY flash

P.S.: This RED led state happens when you load a self-built image over the Pycom images. This IS a bug, and not fixed since several releases. But as long a you stick with either version, it will not hit you.

[ssmith]

@robert-hh That worked great. I'm now able to build the FW with our frozen bytecode. Next step is to figure out the OTA