Minetest on Linux (glibc) is not able to give back freed MapBlocks to the OS

[RonnyBarthmann]

Minetest version

all Minetest versions till 5.8.0 tested

Irrlicht device

No response

Operating system and version

debian and all debain based

CPU model

all models

GPU model

not importan / also a problem on servers (headless)

Active renderer

not importan / also a problem on servers (headless)

Summary

https://forum.minetest.net/viewtopic.php?t=30509

minetest dont free memory because of fragmentation like on windows
so minetest builds up a huge amount of RAM and never free any of it

Steps to reproduce

start a minetest server and keep it running for days
or start a world and user "emergeblocks" to load a huge amount of blocks ... after it finishes it never goes down again

[sfan5]

As we already found out here and in the linked forum thread, this isn't a bug in Minetest itself, but glibc (and they don't seem to care either).
The next best thing we could do it work around it.

[RonnyBarthmann]

I thought the problem was that glibc doesn't trim automatically...
Can't you just call the trim function?
Maybe with a compile-if so that it is only done under Linux

[savilli]

We can't because the performance penalty of that solution is unclear. The execution time of the trim function depend on many factors, therefore it may work very slowly under certain conditions. Fixing one corner case by creating another one doesn't sound like a good idea.

[RonnyBarthmann]

but it wouldn't be a problem to offer something like that as an option under the advanced settings...
firstly, you could find out over time whether it has disadvantages and even if it does,
people can then decide for themselves whether the advantages outweigh the disadvantages

I think it would be worth a try

[sfan5]

We should take care not to call malloc_trim too often, but this is not a reason to avoid this entirely.
My idea: Very coarsely track how much memory has been freed, once that number crosses (e.g.) 128MiB call malloc_trim once, reset counter to zero.

[Desour]

Another solution would be to use a global synchronized_pool_resource for allocating mapblocks. Either it already allocates big enough memory chunks from the upstream allocator (i.e. bigger than glibc's M_MMAP_THRESHOLD mallopt), or if not, we can supply an upstream allocator that just uses mmap.