[Python-Dev] refleaks and caches (original) (raw)

Jeroen Ruigrok van der Werven asmodai at in-nomine.org
Mon Jan 28 11:55:21 CET 2008

-On [20080128 03:13], Christian Heimes (lists at cheimes.de) wrote:

Do the int/float free lists cause any trouble or can they eat lots of memory?

I hope I am interpreting it correctly, but it seems http://evanjones.ca/memoryallocator/ explanation on that still applies:

"The worst offenders are integers and floats. These two object types allocate their own blocks of memory of approximately 1kB, which are allocated with the system malloc(). These blocks are used as arrays of integers and float objects, which avoids waste from pymalloc rounding the object size up to the nearest multiple of 8. These objects are then linked on to a simple free list. When an object is needed, one is taken from the list or a new block is allocated. When an object is freed, it is returned to the free list.

This scheme is very simple and very fast, however, it exhibits a significant problem: the memory that is allocated to integers can never be used for anything else. That means if you write a program which goes and allocates 1000000 integers, then frees them and allocates 1000000 floats, Python will hold on to enough memory for 2000000 numerical objects. The solution is to apply a similar approach as was described above. Pools could be requested from pymalloc, so they are properly aligned. When freeing an integer or a float, the object would be put on a free list for its specific pool. When the pool was no longer needed, it could be returned to pymalloc. The challenge is that these types of objects are used frequently, so care is required to ensure good performance.

Dictionaries and lists use a different scheme. Python always keeps a maximum of 80 free lists and dictionaries, any extra are freed. This is not optimal because some applications would perform better with a larger list, while others need less. It is possible that self-tuning the list size could be more efficient."

-- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ We have met the enemy and they are ours...

More information about the Python-Dev mailing list