(original) (raw)
On 5 October 2017 at 18:45, Eric Snow <ericsnowcurrently@gmail.com> wrote:
I don't think we should be touching the behaviour of core builtins solely to enable message passing to subinterpreters without a shared GIL.
After we move to not sharing the GIL between interpreters:
Channel.send(obj): # in interp A
incref(obj)
if type(obj).tp\_share == NULL:
raise ValueError("not a shareable type")
set\_owner(obj) # obj.owner or add an obj -> interp entry to global table
ch.objects.append(obj)
Channel.recv(): # in interp B
orig = ch.objects.pop(0)
obj = orig.tp\_share()
set\_shared(obj, orig) # add to a global table
return obj
This would be hard to get to work reliably, because "orig.tp\_share()" would be running in the receiving interpreter, but all the attributes of "orig" would have been allocated by the sending interpreter. It gets more reliable if it's \*Channel.send\* that calls tp\_share() though, but moving the call to the sending side makes it clear that a tp\_share protocol would still need to rely on a more primitive set of "shareable objects" that were the permitted return values from the tp\_share call.
And that's the real pay-off that comes from defining this in terms of the memoryview protocol: Py\_buffer structs \*aren't\* Python objects, so it's only a regular C struct that gets passed across the interpreter boundary (the reference to the original objects gets carried along passively as part of the CIV - it never gets \*used\* in the receiving interpreter).
bytes.tp_share():
obj = blank_bytes(len(self))
obj.ob_sval = self.ob_sval # hand-wavy memory sharing
return obj
This is effectively reinventing memoryview, while trying to pretend it's an ordinary bytes object. Don't reinvent memoryview :)
bytes.tp_free(): # under no-shared-GIL:
# most of this could be pulled into a macro for re-use
orig = lookup_shared(self)
if orig != NULL:
current = release_LIL()
interp = lookup_owner(orig)
acquire_LIL(interp)
decref(orig)
release_LIL(interp)
acquire_LIL(current)
# clear shared/owner tables
# clear/release self.ob_sval
free(self)
The simplest possible variant of CIVs that I can think of would be able to avoid that outcome by being a memoryview subclass, since they just need to hold the extra reference to the original interpreter, and include some logic to swtich interpreters at the appropriate time.
That said, I think there's definitely a useful design question to ask in this area, not about bytes (which can be readily represented by a memoryview variant in the receiving interpreter), but about *strings*: they have a more complex internal layout than bytes objects, but as long as the receiving interpreter can make sure that the original string continues to exist, then you could usefully implement a "strview" type to avoid having to go through an encode/decode cycle just to pass a string to another subinterpreter.
That would provide a reasonable compelling argument that CIVs *shouldn't* be implemented as memoryview subclasses, but instead defined as *containing* a managed view of an object owned by a different interpreter.
That way, even if the initial implementation only supported CIVs that contained a memoryview instance, we'd have the freedom to define other kinds of views later (such as strview), while being able to reuse the same CIV machinery.
Cheers,
Nick.
--
Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia