memory allocation failed - why?

[livius]

Hi

i do

gc.collect()
print('free', gc.mem_free())

and i got value:

free 38736

now in the next line i have allocation of 15000 bytes:

self.buf = bytearray(400 * 300 // 8)

and i got an error:

MemoryError: memory allocation failed, allocating 15000 bytes

can someone tell me what can be a reason? As i have much more memory available 38736>15000....

[Gijs]

Which firmware version are you using?

I did some quick maths, and 400*300 = 120000, which is why the error MemoryError: memory allocation failed, allocating 15000 bytes is somewhat confusing. 120000 > 38736, which makes sense, as the allocation failed.

Next I tried the following on 1.20.2.r4:

import gc
i = 0
thing = {}
while True:
  thing[i] = bytearray(400*300)
  i = i+1
  print(gc.mem_free())

And get the following output:

2077264
1957248
1837216
1717200
1597168
1477152
1357120
1237104
1117072
997056
877024
757008
636944
516928
396912
276896
156880
36816
159632
39616
Traceback (most recent call last):
  File "<stdin>", line 6, in <module>
MemoryError: memory allocation failed, allocating 120000 bytes

You can notice it does some collection near the end of cleaning up my previous test :)
Also, running with a bytearray of 50*300 (15000 bytes), it only fails when (close to) no more space is available.

Let me know

[livius]

@Gijs said in memory allocation failed - why?:

400*300 = 120000

you miss // 8 as there is 400 * 300 // 8 and it is 15000

And doing it with loop is not correct as you fight with automatic garbage collection.
To test this in your setup you should run first gc.collect(), then print(gc.mem_free)
and then allocate buf = bytearray(gc.mem_free//2)
or if it not fail for you try something higher but after reseting a board buf = bytearray((gc.mem_free//3)*2)

[Gijs]

Ah yes, it appeared as a comment in the codeblock, which is why I disregarded it :), beginner mistake.

The loop was mainly to fill up the memory to get to a place where we run out of memory. I dont think I'm fighting the garbage collect there, as I still have references to the allocated memory blocks. Once I dereference them, the gc should clean them up (If I understand that correctly). A more complete test would indeed first collect the garbage.

I cannot get it to fail using your method either. The reason I can think of is memory fragmentation, where there is no longer a continuous block of 15000 bytes available, causing the allocation to fail, though Im not sure what you're doing with the other 2 megabytes of memory..

[livius]

@Gijs said in memory allocation failed - why?:

. I dont think I'm fighting the garbage

I take this conclusion when i look at your data

156880
36816
159632 - here gc.collect something

but this is most possible reasonmemory fragmentation - no longer a continuous block of 15000 bytes available - thanks to pointing it.

[Gijs]

@livius I should've indeed called gc.collect() before running the snippet, what you're seeing is the garbage collector freeing memory from a previous session.

[livius]

@Gijs
I also overlooked that you do not simply allocate new block
but you add it to dict thing[i].
Then even if gc collect something, then test itself was ok as reference exists.

We are sometimes too fast when we work on "simple" things ;-)