[Python-Dev] The untuned tunable parameter ARENA_SIZE (original) (raw)

Tim Peters tim.peters at gmail.com
Sun Jun 4 10:46:10 EDT 2017

[Tim]

A virtual address space span of a terabyte could hold 1M pools, so would "only" need a 1M/8 = 128KB bit vector. That's minor compared to a terabyte (one bit per megabyte).

[Antoine]

The virtual address space currently supported by x86-64 is 48 bits wide (spanning an address space of 2**48 bytes, that is 256 TB), so you would need a 2**(48-20) bits vector, i.e. 256 Mbits or 32 MB.

Note Intel has plans to extend the virtual address space to 2**57 bytes: https://www.phoronix.com/scan.php?page=newsitem&px=Intel-5-Level-Paging

Fill in the blanks ;-) There's only a need for the bit vector to cover the range of addresses actually returned by the system for arenas allocated so far (we're only trying to identify the memory obmalloc does control) . I didn't spell that out, but it was implicit in glosses like "(or if the bit address is out of the vector's domain)". That is, it's up to the bit vector implementation to intelligently represent what's almost always going to be a relatively tiny slice of a theoretically massive address space.

So my observation about a terabyte is to be read as applying to a case in which obmalloc has actually allocated arenas spanning a terabyte of address space.(2**14 arenas of 64MB each, and if the system is kind enough to keep them contiguous). Since that's about 100 times more address space than any Python program I've run actually needed, it's a bare minimum for a thought experiment.

More information about the Python-Dev mailing list