[Python-Dev] The untuned tunable parameter ARENA_SIZE (original) (raw)

Tim Peters tim.peters at gmail.com
Fri Jun 2 15:09:42 EDT 2017

[Tim]

While I would like to increase the pool size, it's fraught with danger.

[Antoine Pitrou <solipsis at pitrou.net>]

What would be the point of increasing the pool size? Apart from being able to allocate 4KB objects out of it, I mean.

Since 4KB+ objects are relatively uncommon (I mean we don't allocate hundreds of thousands of them per second), I don't think it's really worthwhile trying to have the small object allocator handle them.

I don't care about "large" objects here. It's educated intuition about speed: so long as pymalloc is working within a pool, it's blazing fast. When it has to muck with a new pool, much slower (compared to staying within a pool) code is required. When it has to muck with a new arena, slower still.

So the intuition is simple: the larger a pool, the more object operations it can handle staying within its best-case (fastest) code paths.

It would be nice to find a different way for pymalloc to figure out which addresses belong to it. The excruciating PyADDRESSINRANGE manages to do it in small constant (independent of the number of arenas and pools in use) time, which is its only virtue ;-)

So, to sum it up, it's excruciating but fast and works reliably. Why change it?

To enable using pools larger than the greatest common divisor of all OS's native pool sizes.

There's also that Py_ADDRESS_IN_RANGE is responsible for countless hours of head-scratching by poor developers trying to use magical memory debuggers - it's at best unusual for code to read up memory without caring a bit whether the memory has ever been stored to.

I should note that Py_ADDRESS_IN_RANGE is my code - this isn't a backhanded swipe at someone else. It's always been near the top of the "code stink" scale. So I thought I'd mention it in case someone has been sitting on a cleaner idea but has held back because they didn't want to offend me ;-)

More information about the Python-Dev mailing list