How do I diagnose MemoryError?
-
Need help diagnosing this. I have nothing in boot.py or main.py.
The err.py and data_store.py modules have been tested individually and did not throw any MemoryErrors. The err.py module captures error messages and stores them via data_store.py to my server if available, or on the local filesystem if the server is not available, for sending at the next opportunity. When testing err.py I necessarily had to also test data_store.py.
What should my next steps be?
>>> mem_info() stack: 736 out of 7168 GC: total: 67008, used: 7616, free: 59392 No. of 1-blocks: 23, 2-blocks: 6, max blk sz: 384, max free sz: 3702 >>> import test_motor Starting test_motor [SUCCESS] motor.voltage Traceback (most recent call last): File "<stdin>", line 1, in <module> File "test_motor.py", line 19, in <module> File "motor.py", line 51, in run File "err.py", line 17, in __init__ File "data_store.py", line 23, in __init__ MemoryError: >>> mem_info() MemoryError >>>
-
@robert-hh said in How do I diagnose MemoryError?:
b) follow the instructions in https://github.com/pycom/pycom-micropython-sigfox/README.md, section ESP32 (not Pyboard)
Correct URL
https://github.com/pycom/pycom-micropython-sigfox/blob/master/README.md
-
MOST helpful. How can I thank you beyond upvoting your replies?
-
@betterauto Import error happen at the import statement. so if File "umqtt/robust.py", line 7, in <module> is an import statement, the pre-compiling can help.
-
@robert-hh looking at the test_updates error I pasted earlier is that an error on import do you think? Not sure how to tell the difference.
-
@betterauto You can do so, but that only helps if you run into memory problems during import. If you use the compilers from my repository, be awary that the versions for micropython.org and pycom.io are different. Let me update the pycom.io version.
Edit: Updated. May not have been necessary.
-
Robert rather than compile into the flash can I use .mpy files instead? I read somewhere that WiPy runs everything out of RAM, even frozen bytecode. I found your mpy compiler:
https://forum.micropython.org/viewtopic.php?t=2299
-
@betterauto The adafruit link is for esp8266. For Pycom esp32, the method is the same, just another directory. And obviously you can put stable modules in frozen bytecode, and the ones in progress in RAM. The code looks the same.
-
Excellent, thanks. Will try to grok this when I'm more awake.
In the interim I also will trim down the modules to the barest minimum to test the functionality I'm after, then will port the code fixes into the fuller modules, freeze the code, and viola. That way I'm not taking a lot of extra time to recompile the image every time just to debug some functionality.
-
@betterauto It's not difficult, but uses a few steps, which may fail. But once you succeded, thigns get much more convenient.
You have to set-up your own build environment.
a) clone this repository https://github.com/pycom/pycom-micropython-sigfox
b) follow the instructions in https://github.com/pycom/pycom-micropython-sigfox/README.md, section ESP32 (not Pyboard)
c) Once you've cloned it, goto your_path_to_the_clone/pycom-micropython-sigfox/esp32 and follow the instructions in the esp32/README.md. That includes the cloning of another repository, https://github.com/pycom/pycom-esp-idf
d) once you have set-up everything, you'll end up using make for everything:- cleaning: make clean
- build bootloader: make TARGET=boot
- build for WIPY: make BOARD=WIPY
- build for LOPY, make BOARD=LOPY LORA_BAND = USE_BAND_868
- upload firmware: make flash (needs jumper G23->GND too, unless you use a non-Pycom board)
- erase board: make erase
The best environment for that is Linux. Some people use OS X too
Scripts to go into frozen bytecode have to be placed in the esp32/frozen directory. by default, this directory only contains _boot.py. Then run make and make flash.
The binaries built during that process are in esp32/build/xxPY/release/appimg.bin. That file can be upload by OTA too.
-
Cool thanks for the rapid reply. Do you have a link for a handy guide for doing this? Edit: This looks good:
https://learn.adafruit.com/micropython-basics-loading-modules/frozen-modules
-
@betterauto Frozen bytecode is it. gc.collect() won't change anything, if the majority of RAm is used by bytecode. Frozen bytecode moves that to flash.
-
Getting the same error in a different module but this one uses copious gc.collect(). I am at a loss what to do next. Frozen bytecode? Move to a different platform? Suggestions appreciated.
This module tests my update functionality. The updates module downloads either new system files or new data files. It uses the cloud module, which in turn uses the mqtt module, which in turn uses the Adafruit robust MQTT client. Too many modules?
I edited the Adafruit module to run gc.collect() very frequently, almost every other line.
You can see from running gc.collect() manually on the REPL that it doesn't free much. But why would running it there make any difference if I am also running it copiously within the code?
Frustrating.
>>> import test_updates Starting test_updates Traceback (most recent call last): File "<stdin>", line 1, in <module> File "test_updates.py", line 9, in <module> File "updates.py", line 7, in <module> File "cloud.py", line 4, in <module> File "mqtt.py", line 12, in <module> File "umqtt/robust.py", line 7, in <module> MemoryError: memory allocation failed, allocating 82 bytes >>> import gc >>> gc.mem_free() 18688 >>> gc.mem_alloc() 48432 >>> gc.collect() >>> gc.mem_free() 19808 >>> gc.mem_alloc() 47280 >>>
-
I do gc.collect() often but for some reason was not doing so inside this module. I'll try that. Good info on the other gc functions, will try those.
-
@betterauto You can use gc.mem_free() or gc.mem_alloc() to look at the change in allocation before after some activities. That's a roigh indication. It can still happen that even if mem_free() shows sufficient memory, you cannot allocate a large buffer because that free memory is fragmented. gc.collect() can be used to force garbage collection, which often helps.
-
How do I determine how much memory they use? What commands?
-
@betterauto Your code probably simply runs out of heap space.
And that happens in the creation of class instances. You should look if you create these instances once or repeatedly. If once, look how much memory they allocate and try to reduce that. If repeatedly, try to insert gc.collect() before creating the class instances.
Otherwise, you can save a lot or RAM by putting known working modules into frozen bytecode, at which point their code does not use RAM any more, at the penalty of a few percent slower execution.
-
Confirmed that the problem is in err.py and/or data_store.py. Commented it out and no more MemoryError. But how to troubleshoot those, I don't know. As mentioned, those don't error when tested directly.
