[Python-Dev] Mixing memory management APIs (original) (raw)

Tim Peters tim.one@home.com
Wed, 30 Jan 2002 18:49:19 -0500

[NeilS, growing older but wiser, embraces the wisdom of giving up ]

[Michael Hudson]

And then change all the current uses of PyObjectDel to the new API?

It would mean changing all uses of all memory macros in the core to use new macros.

What would that buy us?

The possibility to move to Vladimir's malloc implementation without breaking any extension modules (none: no breakage at all). I want the Python core to use Vladimir's malloc/free, and until the fabled free-threading gets implemented, to use a version that exploits the GIL to eliminate malloc/free lock overhead too. We know for a fact that some major extension modules misused the existing memory API (via mismatching "get memory"->"free memory" pairs), and it's so Byzantine and ill-documented that this shouldn't be a surprise.

Beyond that, I don't believe we've ever said anything about thread safety wrt the existing memory API, simply because we relied on the platform malloc/free to provide thread safety even in the worst of cases. But if the Python core switches to a gimmick that relies on the GIL, then even extensions that use the current API properly (wrt correct matching pairs) may get into huge trouble if the underlying allocator stops doing its own layer of locking.

The intent with new macros would be to spell out the rules. Extensions that wanted to play along could switch, while extensions that ignored the issues would continue to work with the existing seven ways to spell "malloc" (and seven to spell "free").

... I don't what we have is so bad; a helpful tip is that if you're using the Free/FREE/Malloc/REALLOC/etc interfaces, stop. That gets rid of half the problem.

But only for extensions that are actively maintained by people who are keen to dig into how they've abused the current API. It's likely easier for the core to move to new macros than to fully debug even one large extension module that's been working so far by luck.

a-big-hammer-wouldn't-be-called-that-if-it-weren't-big-ly y'rs - tim