[Python-Dev] PEP 554 v3 (new interpreters module) (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Thu Oct 5 06:57:10 EDT 2017

On 5 October 2017 at 18:45, Eric Snow <ericsnowcurrently at gmail.com> wrote:

After we move to not sharing the GIL between interpreters:

Channel.send(obj): # in interp A incref(obj) if type(obj).tpshare == NULL: raise ValueError("not a shareable type") setowner(obj) # obj.owner or add an obj -> interp entry to global table ch.objects.append(obj) Channel.recv(): # in interp B orig = ch.objects.pop(0) obj = orig.tpshare() setshared(obj, orig) # add to a global table return obj

This would be hard to get to work reliably, because "orig.tp_share()" would be running in the receiving interpreter, but all the attributes of "orig" would have been allocated by the sending interpreter. It gets more reliable if it's Channel.send that calls tp_share() though, but moving the call to the sending side makes it clear that a tp_share protocol would still need to rely on a more primitive set of "shareable objects" that were the permitted return values from the tp_share call.

And that's the real pay-off that comes from defining this in terms of the memoryview protocol: Py_buffer structs aren't Python objects, so it's only a regular C struct that gets passed across the interpreter boundary (the reference to the original objects gets carried along passively as part of the CIV - it never gets used in the receiving interpreter).

bytes.tpshare(): obj = blankbytes(len(self)) obj.obsval = self.obsval # hand-wavy memory sharing return obj

This is effectively reinventing memoryview, while trying to pretend it's an ordinary bytes object. Don't reinvent memoryview :)

bytes.tpfree(): # under no-shared-GIL: # most of this could be pulled into a macro for re-use orig = lookupshared(self) if orig != NULL: current = releaseLIL() interp = lookupowner(orig) acquireLIL(interp) decref(orig) releaseLIL(interp) acquireLIL(current) # clear shared/owner tables # clear/release self.obsval free(self)

I don't think we should be touching the behaviour of core builtins solely to enable message passing to subinterpreters without a shared GIL.

The simplest possible variant of CIVs that I can think of would be able to avoid that outcome by being a memoryview subclass, since they just need to hold the extra reference to the original interpreter, and include some logic to swtich interpreters at the appropriate time.

That said, I think there's definitely a useful design question to ask in this area, not about bytes (which can be readily represented by a memoryview variant in the receiving interpreter), but about strings: they have a more complex internal layout than bytes objects, but as long as the receiving interpreter can make sure that the original string continues to exist, then you could usefully implement a "strview" type to avoid having to go through an encode/decode cycle just to pass a string to another subinterpreter.

That would provide a reasonable compelling argument that CIVs shouldn't be implemented as memoryview subclasses, but instead defined as containing a managed view of an object owned by a different interpreter.

That way, even if the initial implementation only supported CIVs that contained a memoryview instance, we'd have the freedom to define other kinds of views later (such as strview), while being able to reuse the same CIV machinery.

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20171005/2c041f5c/attachment.html>

More information about the Python-Dev mailing list