[Python-Dev] The untuned tunable parameter ARENA_SIZE (original) (raw)

Victor Stinner victor.stinner at gmail.com
Thu Jun 1 10:58:17 EDT 2017

It seems very complex and not portable at all to "free" a part of an arena. We already support freeing a whole arena using munmap(). I was a huge enhancement in our memory allocator. Change made in Python 2.5? I don't recall, ask Evan Jones: http://www.evanjones.ca/memoryallocator/ :-)

I'm not sure that it's worth it to increase the arena size and try to implement the MADV_DONTNEED / MADV_FREE thing.

Victor

2017-06-01 11:21 GMT+02:00 INADA Naoki <songofacandy at gmail.com>:

Thanks for detailed info.

But I don't think it's a big problem. Arenas are returned to system by chance. So other processes shouldn't relying to it. And I don't propose to stop returning arena to system. I just mean per pool (part of arena) MADVDONTNEED can reduce RSS. If we use very large arena, or stop returning arena to system, it can be problem. Regards, On Thu, Jun 1, 2017 at 6:05 PM, Siddhesh Poyarekar <siddhesh at gotplt.org> wrote: On Thursday 01 June 2017 01:53 PM, INADA Naoki wrote:

* On Linux, madvice(..., MADVDONTNEED) can be used.

madvise does not reduce the commit charge in the Linux kernel, so in high consumption scenarios (and where memory overcommit is disabled or throttled) you'll see programs dying with OOM despite the MADVDONTNEED. The way we solved it in glibc was to use mprotect to drop PROTREAD and PROTWRITE in blocks that we don't need when we detect that the system is not configured to overcommit (using /proc/sys/vm/overcommitmemory). You'll need to fix the protection again though if you want to reuse the block. Siddhesh

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

More information about the Python-Dev mailing list