[Python-Dev] pymalloc and overallocation (unicodeobject.c,2.139,2.140 checkin) (original) (raw)

Martin v. Loewis martin@v.loewis.de
27 Apr 2002 00:08:03 +0200

Tim Peters <tim.one@comcast.net> writes:

> Right. I confused this with the fact that PyMemRealloc won't return > the excess memory,

PyMemRealloc does whatever the system realloc does -- PyMemRealloc doesn't go thru pymalloc today (except in a PYMALLOCDEBUG build). Doesn't matter, though, since strings use the PyObject{Malloc, Free, Realloc} family today, and that does use pymalloc.

That's what I mean (I'm really confused about memory family APIs, ever since everything changed :-)

OTOH, there's no reason PyObjectRealloc has to hang on to all small-block memory on a shrinking realloc, and there's no reason pymalloc couldn't grow another realloc entry point specifying what the caller wants a shrinking realloc to do. These things are all easy to change, but I don't know what's truly desirable.

Neither do I. To establish whether releasing the extra memory is worth the effort would depend on knowledge whether the object will be long-living; neither pymalloc nor its caller is able to tell.

Note another subtlety: I expect you brought up PyMemRealloc because unicodeobject.c uses the PyMemXYZ family for managing the PyUnicodeObject.str member today.

No, because I assumed PyMem_Realloc was a synonym for PyObject_Realloc.

That means it normally never uses pymalloc at all, except to allocate fixed-size PyUnicodeObject structs (which use the PyObjectXYZ memory family). I don't know whether that's the best idea, but that's how it is today.

I do think that the Unicode data should be managed by pymalloc as well. Of course, DecodeUTF8 would then raise the same issue: decoding UTF-8 doesn't know how many characters you'll get, either. This currently does not try to be clever, but allocates enough memory for the worst case.

Regards, Martin