New firmware release 1.4.0.b1

[jasonriedy]

@jasonriedy Excuse me, connections and not GATTs.

[crankshaft]

@daniel -

Sorry if this appears to be a rant of complaints, but I was expecting much more, and are a little disappointed...

About a month ago, and after reading the specs for the wipy2 and ESP32 vs the ESP8266 which included 14ma with wifi enabled and so many features such as Dual Core, more ram, RTC, BLE, I2C, etc etc, I was so impressed with what was claimed that I purchased a handful of wipy2 and lopy devices from seeedstudio.

I initially got confused thinking that the features and specs were available now and not planned features, and that micropython and pycom's implementation of micropython were one and the same, and could not understand when libraries that were available in the micropython wiki page were not available in my wipy2, now I realise that it is only a much smaller subset of it, and that it is still very much work in progress.

Having now used it for a few weeks and built up some specific libraries, I have come to the conclusion that it is simply not ready for any real production environments yet, and is very much still in the mid beta stages.

The amount of available ram is just too small to do anything that includes anything more than one or 2 libraries, I managed to get the fantastic bosch BMP280 enviro sensor working, and cobbled together a mqtt client, however they cannot work together due to insufficient ram, and so I have to now find an alternative sensor that is easier to decode.

Given the specs, I would have expected that anything that ran on the ESP8266 would also run on the ESP32, and leave more available ram and run faster, but this is very much not the case, and libraries that ran on the older device won't run on the new one due to memory availability.

The power reduction in the latest firmware is really great, however it is still not close to the 14ma advertised and in my measurements swings between 60 and 120ma, probably with an average of 70 or 80 ma, although I have not measured it accurately.

I could further reduce power by using deep sleep, but to do that I need RTC interrupts which are not implemented yet, also I cannot sync the RTC clock as the socket.sendto() is not implemented yet and so although RTC is there, in the real world it is not really usable.

I know that pycomn have tasked espressif with freeing up the memory, and it would be great if this happens, but I struggle to understand how they can increase the available ram by 200% just by optimisation, without crippling the available features ?

Whilst you and your team are doing a really great job in adding new features and functions, it would be good if more effort could be made to concentrate on and make individual features more usable in the real world, and increase the available programmable memory back up to the 98K which I believe was the target, now it is only around 30K and it looks like this will only fall further as new features and functionality is added.

[daniel]

@crankshaft thanks for the direct feedback. We are very much aware that there's a lot to improve everywhere specially around sleep modes, but we can only do as much as the IDF permits. We will reach those current features as it's what the datasheet specifies, it simply needs more work on the firmware. Regarding the memory, we have good news around that. We have been able to release a good amount which has allowed us to increase the MicroPython heap size and also get TLS working again. All this will be available later this week.

Memory optimizations are on their way, and they won't come at the expense of less functionality.

Cheers,
Daniel

[mohpor]

@daniel

Hi,

just like @crankshaft, I have expressed my concerns on the state of things on firmware side before (hardware specs vs price is phenomenal, no doubt about it). A few weeks has passed and I regret expressing them the way I wrote them, in my defense, I was really frustrated by the fact that I couldn't do anything in the meantime, now, the firmware is getting closer to finish (very very far away at the moment, but big steps every week).
The reason I'm writing here again is that, as a veteran Software Developer, I can see the reason behind these frustrations: Issue Tracking, or the lack thereof. This forum is a great tool to discuss things, but when it comes to tracking or submitting an issue, it is not the right tool. If you provide a proper Issue Tracking service, be that your Github repo's Issues section or something as comprehensive as Jira, we would have a better understanding of what's missing, what is working, the progress, the plan ahead and many many more advantages that I'm sure you are aware of.
To see how important it is to provide a better Issue Tracking tool just read every Firmware update Announcement's comments! It's like a lot of issues being submitted to the post! This is not right @daniel.

Good news on Memory would make my week! I'm running into memory related issues left and right and I'm just hoping to optimize my code to get where I'm heading.

More power to you all,
Cheers,
M.

[sakis]

@mohpor

Hi,

We have an internal issue tracker but it's not exposed to the public. Public issues are available on GitHub, and they are updated (open/closed) as expected.

I agree that we should use GitHub more instead of the forum, especially for tagging and prioritising issues, but that's already happening (admittedly on a smaller scale than it should).

[mohpor]

@sakis
I'm well aware of the Github repo's existence but I guess the burden is on you guys to direct issues (and manage them to some extent) from here to there and make others engage the proper way. I will be submitting my issues there from now on.

Cheers,
M.

[livius]

@crankshaft said in New firmware release 1.4.0.b1:

You have right about overview but one point i must understand:

The amount of available ram is just too small to do anything that includes anything more than one or 2 libraries, I managed to get the fantastic bosch BMP280 enviro sensor working, and cobbled together a mqtt client, however they cannot work together due to insufficient ram, and so I have to now find an alternative sensor that is easier to decode.

i do not understand you corectly about this sentence
so I have to now find an alternative sensor that is easier to decode
do you mean that BMP280 is not usable to you because of RAM
I use BMP180 extensively (and difference between BMP280 is "only" accuracy) but i do not use "mqtt client,"
I connect to service(wirtten in Delphi) on computer and send there data throught socket. And back to Firebird database.
I suppose that your problem is about MQTT memory consumption.

I have connected to wipy many sensors at the same time e.g.
one wire temperature, i2c light, barometric, spi oled, pir, IR, few buttons, port expander
i have established connection between two wipy (i still wait for support AP+STA mode)
but now i simulate it by connecting one wipy to second and after packet of data arrive i turn my AP to sta and send data back to server throught tcp socket and back change to AP

[jmarcelino]

@crankshaft
I've been using umqtt.simple (https://github.com/micropython/micropython-lib/tree/master/umqtt.simple ) successfully with various sensors, umqtt only takes up about 7Kb. Admittedly I've not used it to do MQTT over SSL yet if that's what you're using, so it might have a larger footprint.

@mohpor
You touch on a good point and I'm also guilty of not following up quick informal reports here with proper Github issues. Everyone at Pycom is just so responsive it hasn't seemed necessary but you're right it would improve visibility and tracking.

That said following from the latest announcement I'd imagine in a couple of weeks most bug reports will really need to be filed with the main micropython repo and not on Pycom's.

[mohpor]

@jmarcelino
I'm sure it will get improved drastically if everyone follow the same repo. More power behind the tasks at hand.

Cheers,
M.

[jmarcelino]

@mohpor
Indeed it's excellent news having the the ESP32 - powered by Pycom - as an official member of the community, however the main repo is also a very noisy place. There are currently 300+ open reports there and from a quick glance most aren't labelled with the type of board despite being port specific, so if nothing there changes it could potentially be more confusing.

[mohpor]

@jmarcelino
I'm sure it'll worth the noise! I look at it this way: The more noise (activity) the more chance of getting stable! I am facing bugs everyday, imagine how long it would take for me to face all possible bugs. Now, when many others try what I'm trying, there is a good chance the time would be decimated massively! Not to mention use cases and code templates and known issues which will be documented for others to use!

The mess of not being labeled or assigned is a whole another issue though. I hope they'll get sorted out in time.

Cheers,
M.

[mohpor]

@crankshaft

The amount of available ram is just too small to do anything that includes anything more than one or 2 libraries, I managed to get the fantastic bosch BMP280 enviro sensor working, and cobbled together a mqtt client, however they cannot work together due to insufficient ram, and so I have to now find an alternative sensor that is easier to decode.

I was facing memory issues but I got to solve them. The key is gc.collect()! You can't over use it! I mean, I have gc.collect() all over the place! I'm not sure about it's implications on battery usage and performance, but that is the only way I can run my code now! Every loop iteration, every major memory altering event (especially around threads and interrupts). I have learnt it the hard way, it might help you with your application too.

A good pattern to use is try-finally in interrupt handlers and memory consuming functions:

try:
    # Do some memory consuming function
    for x in list:
       #do things
       gc.collect()
    wlan.deinit()
    gc.collect()
    ble.deinit()
    gc.collect()
finally:
    gc.collect() 

Hope it helps somebody.

Cheers,
M.

[robrush]

Hi folks,

• Still nothing about machine.sleep() or machine.deepsleep()?
• I also tried to measure the current i used by my LoPy with machine.idle() or not... I couldn't find any difference. Does someone have an idea of the current used by the CPU at 100% or close to 0%. The difference is probably too small to be seen.

[crankshaft]

@daniel - Thanks, and joining forces with Damien George sounds like a great move too :-)

[crankshaft]

@mohpor, @jmarcelino, @livius - Thanks for the suggestions regarding MQTT and BME280 and memory etc.

Compared to the Si7021 AND SHT21, the BME requires a lot more code in the driver library to extract the data, probably 3 times as much as for the S* devices, which are really simple and is what I am using now.

Other than getting data about the environment, I need to get data from other sensors, and really need to provide a simple HTTP server so that the device can initially be configured (an maintained) from a web browser and a few other things too.

My code needs to be very robust as once it is deployed (on the other side of the world) it needs to look after itself, and manage any and all errors, and with the limited code space right now I simply cannot do this.

I know that if I strip down to bare functionality, I could get the basics to work, but that's not what I am after, I need much more than I have now in order for this project to succeed.

If I were to use a 8266 device (nodeMCU or micropython) then I believe I can get it all in, and my expectations were that I would be able to do more on the ESP32 rather than less.

I don't really require any help with the programming side of things right now, just need more space to breathe :-)

[livius]

@crankshaft
you can compile py file it use then much less memory
but still this require custom firmware build with enabled option for compiled modules

look here:
https://forum.pycom.io/topic/486/v-1-8-7-micropython-released-support-importing-compiled-mpy-files-more-ram/2
and here - detile how to do this:
https://forum.pycom.io/topic/453/memoryerror-memory-allocation-failed-allocating-2048-bytes-lopy-1-2-0-b1/2

[tholden]

machine.idle is not working.

I am executing it as the first thing in the code, but it still continues execution

[robrush]

@tholden agree with you. See my comment below

[mohpor]

@sakis

Now I can see why everybody submit their issues here.

I have opened an issue on Github's repo, no one even acknowledges my issue there. People get way more attention from posting here.

[Colateral]

@livius Have you succeed to get data from BMP180 . I am always get "I2C bus error" when I am trying to instantiate bmp180.
Debuging, I2C bus scan displays 119 address available, but when I simply read the bmp180 chip id, with i2c.readfrom_mem(119, 0xD0, 2) I am getting "I2C bus error".

This is happening always on LoPy.