btree lib support

[InverseEffect]

I was looking for a very lightweight database-like (key-value pair) storage mechanism for micropython and stumbled across the btree micropython library but it is not included in Pycom's FW build. Is it easy to add, I cannot find single btree.py file which makes me think it is heavily dependent on C or something. I need a way to store records of RFID tags for example and then look it up as tag 'swipes' are detected. I cannot store this info in the cloud as the latency will be too high. Any ideas?
Also found that calling

help('modules')

returns:

object modules is of type str
  encode -- <function>
  find -- <function>
  rfind -- <function>
  index -- <function>
....
....

This help command is suppose to list all the installed python modules?

Thanks

[robert-hh]

@Fakstory I made a tarball and placed it here: https://github.com/robert-hh/Shared-Stuff.git
You should be able to load it with the Pycom update tools. I have no GPy here, so I cannot test, whether the firmware works.

[Fakstory]

@robert-hh I am looking exactly for something like that. The "firmware tarball" you're talking about, is it the firmware file needed to flash the pycom (gpy) device? using pycom-fwtools?
If so, I would really like a firmware tarball for the gpy (for testing)!
I'll consider building it later.

Thanks in advance!
Fak

[robert-hh]

@ASH247iT I have created a branch of the pycom repository which includes btree. It is here:
https://github.com/robert-hh/pycom-micropython-sigfox/tree/berkeley-db
I can also provide you a firmware tarball for testing, if you tell me the Pycom module you are using (WIPY, FIPY, ...).

[ASH247iT]

@robert-hh Many Thanks

[robert-hh]

@ASH247iT The espressif ESP32 Btree module will be different to the MicroPython Btree module. Even if you configure it into the image - which is the easier part- the Python API is missing, and access to the file system is also not given.
I tried to port in the MicroPython variant, but gave up when it requested the vfsposix module in addition.
So one option for you would be to use the MicroPython.org firmware for ESP32, if Btree is really essential for you. That should run on a WiPy3. Obviously this causes some inconvenience, like the GPIO numbers are not Pxx but the raw Chip numbers. And the Pycom infrastructure is missing, like PyBytes. Some modules are missing too, like ftp and telnet. FTP can be provided by a Python variant.

[ASH247iT]

@robert-hh Do you know if there is a step by step official document for doing this with the WiPy please? Thanks Adrian

[ASH247iT]

@robert-hh So not easy then ? Unless you know what you're doing !!
Looking on the ESP32 Expressif forum (fairly old post)
https://www.esp32.com/viewtopic.php?f=17&t=2446
It mentions something about a menuconfig - "Btree module is included, can be Enabled/Disabled via menuconfig." - is this applicable to a Pycom wipy or will this not work? (sorry if this is a stupid question - as I said Im a newbie)

[robert-hh]

@ASH247iT Btree is C-Code, not compiled Python code. mpconfigport.h is surely one place to change, but also application.mk, and others.
The Micropython.org branch has all the details, and that has to be replicated to the Python branch. Just setting MICROPY_PY_BTREE = 1 is not sufficient.

[ASH247iT]

@robert-hh Could it not be made available as a .mpy file so I could just drop it into a project or are there some simple instructions for compiling / flashing etc ?
is this the file you refer to above?
https://github.com/micropython/micropython/blob/master/ports/esp32/mpconfigport.h

[robert-hh]

@ASH247iT No change. The source files are present, but are not included in the build.

[ASH247iT]

@robert-hh I realise this is an old post - did you manage to prove BTree was available? If so could you give some more details how I might get it. Im a bit of a newbie to both Pycom and python ! Thanks

[InverseEffect]

@frida If jou are referring to Pycom's nvram key-value pair then no because it can only store a single 32bit int as the value. I want to save more information.

[InverseEffect]

@robert-hh thanks for trying though. I must still learn about make files and all that cross compiling stuff. I'm used to IDEs that hide all that complexity from me :)

[Frida]

Can nvram be used?

[InverseEffect]

@livius I don't necessarily need btree. Any memory efficient key-value pair will do. I thought about saving a dictionary to file via json but the issue is that depending on the number of elements, it might use up a lot of ram to read the whole file to json then to dictionary. I guess I can come up with a file scheme by saving certain ranges in seperate files.

[robert-hh]

@inverseeffect Including BTREE is not as easy as I assumed. All components are there, but the make files have to be aligned too.

[livius]

@inverseeffect
Do you really need btree? You say about key value pair. Dictionary is enough for that.
Or do you really need some benefits from btree?

[robert-hh]

@inverseeffect The code for the btree moddule is in the repository, but not enabled in the standard build. You can do that, but you have to set up the build environment for that, add to esp32/mpconfigport.h the line

#define MICROPY_PY_BTREE (1)

and then build your target. I must confirm that I did not try that, so I'm not sure if it build w/o errors. But there is a good chance that it does. I can try this night.