[Python-Dev] RE: Program very slow to finish (original) (raw)

Martin von Loewis loewis@informatik.hu-berlin.de
Tue, 6 Nov 2001 20:37:11 +0100 (MET)

Can any Python-Dev'er make time to dig into the advisability of making PyMalloc the default?

I was looking into making pymalloc the default a few weeks ago, when I noticed that allocations is pretty efficient on Linux. I studied glibc malloc a bit to see that it is indeed quite clever about allocation, even in the presence of MT locks (with per-arena locks, and recording a per-thread arena in a thread-local variable).

I believe the main difference in deallocation speed (which I didn't notice then) comes from the attempt to combine subsequent memory blocks in glibc; pymalloc doesn't attempt to do so. Also, there is a lock acquire/release cycle in glibc malloc, which may account for some overhead, too.

Looking at obmalloc.c, I see one aspect that I'd consider questionable: detection of pool-vs-system-malloc uses a heuristics, namely

offset = (off_t )p & POOL_SIZE_MASK;
pool = (poolp )((block *)p - offset);
if (pool->pooladdr != pool || pool->magic != (uint )POOL_MAGIC) {
    _SYSTEM_FREE(p);
    return;
}

So if the pool header has the pool magic and points to itself, it really is a pool header. That may result in a system malloc being recognized as a pool malloc, under rare circumstances. As a result, the system heap will be corrupted.

Another aspect that may be troubling is that obmalloc never returns memory to the system. It just detects free pools, and is willing to rearrange a free pool for use with a different object size. Of course, there is little chance that a complete arena will ever become empty, so this is probably not that bad.

Regards, Martin