[Python-Dev] PEP-298: buffer interface widening too much? (original) (raw)

Todd Miller jmiller@stsci.edu
Sun, 15 Dec 2002 13:33:33 -0500

Martin v. L�wis wrote:

I've used this aspect of PyArgParseTuple, but forgot about it. I was thinking that you're right, the original object is "gone", we're stuck. But... it's still (in all probability) lying around in the arg tuple, so some sort of PyArgReleaseBuffers is possible.

That looks clumsy, Well, yes, especially in light of PEP-286. :)

and also too easy to forget unless done systematically, I really liked the idea of doing the buffer release behind the scenes, as suggested by PEP-286 and below.

in which case it will add a considerable performance cost. I think the But won't any systematic solution have a performance cost? Argtuples don't sound "free". I'm really playing devil's advocate here, because I like the concept of PEP-286.

locked buffer interface is much simpler in comparison. I disagree here. I didn't see PEP-298 address PyArg_ParseTuple at all, so there's no real basis for comparison on this point. I'll grant you that PyArg_ReleaseBuffers is probably a lame solution to a problem better solved by PEP-286; that said, because use of PyArg_ReleaseBuffers is explicit, there is no problem distributing the performance penalty to places where there is a corresponding benefit. I was not suggesting dropping the existing release call; merely adding a layer which could make sense out of PyArg_ParseTuple specs.

If Jack ever manages to design an alternative parse tuple interface, I assume it will arrange to automatically release the buffer when the function returns (or, more precisely, when the buffer is deallocated. In fact, my PEP 286 could address this issue: the locked buffers would be associated with the enhanced tuple, so that releasing the tuple would unlock the buffers. That sounds pretty cool.

This still has the potential for breaking code, since now the buffers are locked while the function runs (i.e. callbacks couldn't expand the array anymore); it is likely that such breakage would uncover a bug in the locking assumptions of the existing code.

I agree. But the means exist to work around such problem (release and re-aquire), and I would guess they are even rarer than the existing "buffer safety" issues.

Your attempt to fix your proposal indicates one of the strengths of the PEP process: most initial proposals have flaws, and, unless completely spelled out, readers will typically miss the flaws on initial reading. The weakness, of course, is that a PEP might be sitting around for a long time, and nobody would be reviewing it.

Incidentally, does PEP-298 solve this now?

Depends on what "this" is. Existing code will continue to work as before: No locking in ParseTuple happens, as a result, the arguments are unlocked when the function returns; this is desirable. Since the buffers aren't locked, relocation may happen. If such relocation is ever observed, people would have to replace the "s" parser with an "O" parser, and perform the locking themselves. I'm just saying that we don't need 2 extra "locking calls" to achieve this.

If the default behavior for the existing Python objects (array, mmap, ...) is a onetime warning when a lock is abused, where's the issue? And with the warning framework, the one time warning could be mutated into an exception or supressed by a variety of means.

My thinking is that all buffer pointers should be "logically" locked. Objects that don't need locks don't need to change. All extensions should be written and/or updated to handle objects which require locking. Extensions which aren't updated will trigger warnings which could be supressed or converted into exceptions.

For me, that's simpler than saying "correct extensions" lock the buffer, but "obsolete ones" may still segfault w/o warning.

Perhaps saying that "a warning is enough" is too limiting, but this is a fringe issue, so I think maybe it is.

So, yes, PEP-298 addresses this - although modification of existing code is necessary to make use of this feature (as is in your fixed proposal).

Well, this does sound bad, but I was assuming the core and library would be "fixed" at the first introduction of locking, and the only problem would be with non-conforming extensions. I now think that "lock violations" might be better handled using the warning framework, defaulting to a one time warning.

I think we should strive for a solution that keeps the function parameters locked while the function runs, and unlocks them automatically at function return time. I like this idea, but wonder if it won't inflict a performance penalty on all function calls.

This shows another flaw in the wording of PEP 298: It implies without saying that the expectation is that you should have as many release calls as you have lock calls. I was certainly assuming this was the case.

Regards, Martin

Regards, Todd