How to use "libpcre" on the Pycom MicroPython runtime?

[andreas]

Dear Pycom Team,

Introduction

First things first: Thanks for conceiving and building this wonderful platform. We are just getting started with MicroPython on the FiPy and had good experiences so far after resolving some minor obstacles we might report about later. For the records: We are running your build of Pycom MicroPython 1.20.0.rc7 as of 2019-02-08.

Problem

Now, after trying to incorporate the micropython-terkin library originally developed on MicroPython for Unix, we found that re.py errors out on the FiPy by

  File "dist-packages/urllib/parse.py", line 30, in <module>
  File "dist-packages/re.py", line 11, in <module>
AttributeError: 'NoneType' object has no attribute 'func'

after we installed the urllib.parse package using upip through micropython -m upip install -p dist-packages micropython-urllib.parse and then uploading the dist-packages folder with rshell like

mkdir /flash/dist-packages
rsync dist-packages /flash/dist-packages

Investigation

pcre.func is actually the first thing used after importing libpcre:

pcre = ffilib.open("libpcre")

#       pcre *pcre_compile(const char *pattern, int options,
#            const char **errptr, int *erroffset,
#            const unsigned char *tableptr);
pcre_compile = pcre.func("p", "pcre_compile", "sipps")

See also: https://github.com/micropython/micropython-lib/blob/v1.9.3/re-pcre/re.py#L6-L11

Question

So, we are now wondering whether the libpcre library is available on the Pycom MicroPython runtime at all? As we haven't been able to find a reference to it on the documentation pages and on the forum yet, we found it appropriate to just ask here.

Thanks again and keep up the spirit.

With kind regards,
Andreas.

[andreas]

Hi again,

we resolved this issue and it is easy to tackle if you know what has been happening to make things go south. So, let me explain.

When installing micropython-urllib.parse with upip, this obviously pulls in all package dependencies. One dependency, the package micropython-re-pcre brings in the re.py module along with ffilib.py as a utility module to be able to talk to libpcre.

However, all of this is not available on the Pycom platform, but Pycom itself ships a built-in re package, which would have worked right away when not being masked by the vanilla re.py shipped through micropython-re-pcre.

So, we manually remove re.py and ffilib.py from the dist-packages folder before uploading the firmware using rshell. Then, urllib will use the built-in re.py module and everything works like a charm.

Sorry for the noise and happy hacking,
Andreas.

---

P.S.: If you are interested in what we are building here, you are welcome to have a look at the Hiveeyes MPY data logger firmware. We'd love to hear about any feedback and suggestions you might have.

[andreas]

Hi there,

So, we conclude either micropython-urllib.urequest or micropython-urllib would be more appropriate than micropython-urllib.parse.

When looking at this again, we found the former just contains an urlopen function, while the latter is committed as a dummy module and contains no code at all.

So, our question is still standing and we are humbly asking again for any hints how to get any official implementation of urlsplit or urlencode into the MicroPython universe.

We are open to any option, either by somehow getting libpcre into the platform or by using an appropriate pure-Python module.

Thanks already,
Andreas.

[timh]

@martinnn micropython for unix is just what it says. You can build micropython for any linux distribution. And its usually an installable option via opkg on openwrt based images. Which is a good scenario for it - small cut down python ;-)

[Martinnn]

@andreas said in libpcre missing?:

MicroPython for Unix

Out of curiosity - what is micropython for unix? As I understand it, micropython is made for resource-limited microcontrollers and Unix is quite the opposite direction?
I assume you are aware of the pycom sources on github.

[andreas]

Looking at this from a different angle, we believe it would be more suitable to make the micropython-terkin library use another urllib library not based on the micropython-re-pcre module in order not to depend on OS-level bindings and reclaim pure-Python sanity ;].

So, we conclude either micropython-urllib.urequest or micropython-urllib would be more appropriate than micropython-urllib.parse. Bummer that we used the other one before not known about its system-level dependency to libpcre.