memory allocation failed - why?



  • Hi

    i do

    gc.collect()
    print('free', gc.mem_free())
    

    and i got value:

    free 38736
    

    now in the next line i have allocation of 15000 bytes:

    self.buf = bytearray(400 * 300 // 8)
    

    and i got an error:

    MemoryError: memory allocation failed, allocating 15000 bytes
    

    can someone tell me what can be a reason? As i have much more memory available 38736>15000....



  • @Gijs
    I also overlooked that you do not simply allocate new block
    but you add it to dict thing[i].
    Then even if gc collect something, then test itself was ok as reference exists.

    We are sometimes too fast when we work on "simple" things ;-)


  • Global Moderator

    @livius I should've indeed called gc.collect() before running the snippet, what you're seeing is the garbage collector freeing memory from a previous session.



  • @Gijs said in memory allocation failed - why?:

    . I dont think I'm fighting the garbage

    I take this conclusion when i look at your data

    156880
    36816
    159632 - here gc.collect something
    

    but this is most possible reasonmemory fragmentation - no longer a continuous block of 15000 bytes available - thanks to pointing it.


  • Global Moderator

    Ah yes, it appeared as a comment in the codeblock, which is why I disregarded it :), beginner mistake.

    The loop was mainly to fill up the memory to get to a place where we run out of memory. I dont think I'm fighting the garbage collect there, as I still have references to the allocated memory blocks. Once I dereference them, the gc should clean them up (If I understand that correctly). A more complete test would indeed first collect the garbage.

    I cannot get it to fail using your method either. The reason I can think of is memory fragmentation, where there is no longer a continuous block of 15000 bytes available, causing the allocation to fail, though Im not sure what you're doing with the other 2 megabytes of memory..



  • @Gijs said in memory allocation failed - why?:

    400*300 = 120000

    you miss // 8 as there is 400 * 300 // 8 and it is 15000

    And doing it with loop is not correct as you fight with automatic garbage collection.
    To test this in your setup you should run first gc.collect(), then print(gc.mem_free)
    and then allocate buf = bytearray(gc.mem_free//2)
    or if it not fail for you try something higher but after reseting a board buf = bytearray((gc.mem_free//3)*2)


  • Global Moderator

    Which firmware version are you using?

    I did some quick maths, and 400*300 = 120000, which is why the error MemoryError: memory allocation failed, allocating 15000 bytes is somewhat confusing. 120000 > 38736, which makes sense, as the allocation failed.

    Next I tried the following on 1.20.2.r4:

    import gc
    i = 0
    thing = {}
    while True:
      thing[i] = bytearray(400*300)
      i = i+1
      print(gc.mem_free())
    

    And get the following output:

    2077264
    1957248
    1837216
    1717200
    1597168
    1477152
    1357120
    1237104
    1117072
    997056
    877024
    757008
    636944
    516928
    396912
    276896
    156880
    36816
    159632
    39616
    Traceback (most recent call last):
      File "<stdin>", line 6, in <module>
    MemoryError: memory allocation failed, allocating 120000 bytes
    

    You can notice it does some collection near the end of cleaning up my previous test :)
    Also, running with a bytearray of 50*300 (15000 bytes), it only fails when (close to) no more space is available.

    Let me know


Log in to reply
 

Pycom on Twitter