[Python-Dev] pymalloc killer (original) (raw)

Guido van Rossum guido@python.org
Fri, 29 Mar 2002 09:34:01 -0500

There may be hope for this, and another strong reason for pursuing it. [good stuff snipped]

A nasty subtlety: if PyMemNukeIt is called when the GIL isn't held, it's going to take a lot of thought to see whether this can be made safe when called simultaneously by multiple threads; e.g., the finger can change "mid-stream" then, and, worse, some thread that does hold the GIL may legitimately cause a new arena to be allocated midstream, mutating the address vector during the search. This may be intractable enough to kill the idea of mapping PyMemXXX to this.

I know of two places that calls PyMem_Malloc outside the GIL: PyOS_StdioReadline in Parser/myreadline.c and call_readline() in Modules/readline.c. The memory thus allocated is returned by PyOS_Readline() (which calls either function through a hook pointer), and both its callers (the tokenizer and raw_input()) free the result using PyMem_DEL or PyMem_FREE (these two seem to be used synonymically). The tokenizer code may also apply PyMem_RESIZE to it (it incorporated the input line in its buffer structure).

This would have to be fixed by changing the allocations to use malloc() (as they did up to 1.5.2 :-) and by changing the consumers to use free() and realloc(). (An alternative would be to let PyOS_Readline() copy the memory to PyMem_Malloc'ed memory.)

This is part of a "hook" API that allows 3rd parties to provide their own alternative to PyOS_Readline. This was put in at the request of some folks at LLNL who were providing their own GUI that fed into Python and who had some problems with sending it to stdin. I don't think anybody else has used this. There is not a single mention of PyOS_Readline in the entire set of Python documentation.

Given the alternatives:

  1. introduce new APIs PyMalloc_{New,Del}Object and tell all extension writers that they have to changes their extensions once again to use these brand new APIs if they want to benefit from pymalloc; or

  2. fix the issues with PyOS_Readline, make PyMem_{DEL,FREE,Del,Free} synonyms for Tim's clever PyMem_NukeIt, and continue to promote using PyObject_{New,Del} for use by extension writers;

I'm all for #2.

--Guido van Rossum (home page: http://www.python.org/~guido/)