[Python-Dev] The untuned tunable parameter ARENA_SIZE (original) (raw)
INADA Naoki songofacandy at gmail.com
Thu Jun 1 04:23:04 EDT 2017
- Previous message (by thread): [Python-Dev] The untuned tunable parameter ARENA_SIZE
- Next message (by thread): [Python-Dev] The untuned tunable parameter ARENA_SIZE
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hello.
AFAIK, allocating arena doesn't eat real (physical) memory.
- On Windows, VirtualAlloc is used for arena. Real memory page is assigned when the page is used first time.
- On Linux and some other *nix, anonymous mmap is used. Real page is assigned when first touch, like Windows.
Arena size is more important for freeing memory. Python returns memory to system when:
- When no block in pool is used, it returned to arena.
- When no pool is used, return the arena to system.
So only one memory block can disturb returning the whole arena.
Some VMs (e.g. mono) uses special APIs to return "real page" from allocated space.
- On Windows, VirtualFree() + VirtualAlloc() can be used to unassign pages.
- On Linux, madvice(..., MADV_DONTNEED) can be used.
- On other *nix, madvice(..., MADV_DONTNEED) + madvice(..., MADV_FREE) can be used.
See also:
https://github.com/corngood/mono/blob/ef186403b5e95a5c95c38f1f19d0c8d061f2ac37/mono/utils/mono-mmap.c#L204-L208 (Windows) https://github.com/corngood/mono/blob/ef186403b5e95a5c95c38f1f19d0c8d061f2ac37/mono/utils/mono-mmap.c#L410-L424 (Unix)
I think we can return not recently used free pools to system in same way. So more large arena size + more memory efficient can be achieved.
But I need more experiment.
Regards,
- Previous message (by thread): [Python-Dev] The untuned tunable parameter ARENA_SIZE
- Next message (by thread): [Python-Dev] The untuned tunable parameter ARENA_SIZE
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]