/lib/ folder in /frozen/base

[techykermit]

Hi!

We're working on getting SecureBoot and Encryption working and we have come quite long so far - everything compiles nicely except one core part of our code - the libraries. The internal libraries we have are put in a directory relative to flash, /flash/lib. They are included in the build under ...esp32/frozen/Base. I'm wondering if this is the right way to put it though, as it doesn't seem to work. Running the compiled code on the device, we get "Module not found" errors.

See the REPL output when connected to an encrypted device:

>>> os.listdir("/flash/lib/")
[]
>>>

This makes me think that there could be some kind of conflict. The building of the firmware indicates that the lib/ folder at least is being built:

MPY frozen/base/lib/filename1.py
MPY frozen/base/lib/dirname1/filename2.py

I see there are multiple folders where you can put things, but they are not documented more thoroughly. For example, the Custom/ directory's README talks about "custom user scripts", but what does "custom user scripts" even mean? Therefore, I can't determine where I should put the lib/ folder or what I'm doing wrong.

Any ideas?

[techykermit]

I'm more involved with the techincal parts so I can fill in on some technical details for @Don-iot.

We have seen this behavior with a batch using 2 out of 10 boards. What happens is the following:

1. A building script downloads and compiles firmware for the devices. (SecureFirmware method is used, final image size is 2022224 bytes)
2. The firmware gets flashed onto devices.
3. On (in our tests) 8/10 units:
   The firmware gets flashed correctly and the device starts with no strange errors on first boot
4. On (in our tests) 2/10 units:
   The firmware gets flashed, but the device fails to read and write to files (we get ENOENT errors). It also fails to import modules (module name not found errors) that other devices can import just fine.

If you re-run the same script, with the exact same code being flashed enough times (in one case 4 flashes, in another case 8), a device that doesn't work (see 4.) boots up just fine with no errors at all.

This makes us think this is a PyCom problem, since the hardware side and setup has been consistent (Don wrote more about it above).
The flashing tool nor building tool reports no errors.

[Don-iot]

@robert-hh Working on the same project as @techykermit we have run into problems. We can encrypt devices but in the last batch we had 3 out of 10 devices that were not sucessful, using the same code in all devices but with different encryption keys. Some of these devices did start up after recurring uploads of the code. As we use the same code, it seems like the GPys are not happy, nor sucessful when doing this repeatedly across devices. What kind of work flow would you recommend to get a more stable solution? Currently we have developed our own system, and we are very careful using a good power supply, and a FTDI TTL adapter. The GPys are on our boards, soldered by professionals, on a CE-certified board. So when it comes to hardware and being consistent, I feel that we are in the right ballpark. Yet these 3-4-5 uploads before everything is working is just not going to scale. So any thoughts and ideas are welcome!

[techykermit]

@robert-hh Okay! Thanks a lot for that clarification and a fast answer.

[robert-hh]

@techykermit Actually there is no fix for that. The set of frozen files is not contained in a file-system-like structure. So you have to name the lib files in a appropriate manner.

[techykermit]

@robert-hh Hi, sorry for a little late response. I had to get hold of the test unit :)

Thanks for the command suggestion! Running it, I see that my libraries all show up, but they do show up in the list with lib/ in front of their import name. See below, where lib1, lib2 and lib3 are the internal libraries that have been frozen:

lib/lib1 select            uqueue
hashlib           lib/lib2 socket            ure
json              lib/lib3

All our previous code relies on not using lib in front of the library when importing it (for example, lib2 is imported with the line import lib2 instead of import lib.lib2 or from lib import lib2 in our source code. While the line import lib.lib2 works (so thanks a lot for pointing me to this function and figuring out that the libraries are there!), is there anything we can do to not have to change all our import statements to import from lib/ (is there anything to do so we can import the libraries from lib/ using a prefix like import lib2)?

[robert-hh]

@techykermit You can check with help("modules"), whether and under which name the frozen modules were included.

[techykermit]

@jcaron I try to load them as you can do "normally" (when not building your custom firmware):

for example we do:

import log

when we want to import the log module that is under lib.

This gives a ModuleNotFound error:

Traceback (most recent call last):
  File "main.py", line 1, in <module>
ModuleNotFoundError: No module named 'log'

[jcaron]

@techykermit frozen files are not present as actual files in the filesystem, so it’s normal you can’t list them. They should be available for import directly.

How do you import them and what error do you get?

[Don-iot]

Hi @techykermit this seems to be something that hopefully @kjm @robert-hh @jcaron @Gijs knows more about, let´s hope they can assist.