[Python-Dev] PEP 574 -- Pickle protocol 5 with out-of-band data (original) (raw)

Nathaniel Smith njs at pobox.com
Thu Mar 29 07:25:13 EDT 2018

On Thu, Mar 29, 2018, 02:02 Chris Angelico <rosuav at gmail.com> wrote:

On Thu, Mar 29, 2018 at 7:56 PM, Paul Moore <p.f.moore at gmail.com> wrote: > On 29 March 2018 at 09:49, Chris Angelico <rosuav at gmail.com> wrote: >> On Thu, Mar 29, 2018 at 7:18 PM, Nathaniel Smith <njs at pobox.com> wrote: >>> Another example is the multiprocessing module: it's very safe to >>> assume that the parent and the child are using the same interpreter >>> :-). There's no fundamental reason you shouldn't be able to send >>> bytecode between them. >> >> You put a smiley on it, but is this actually guaranteed on all >> platforms? On Unix-like systems, presumably it's using fork() and thus >> will actually use the exact same binary, but what about on Windows, >> where a new process has to be spawned? Can you say "spawn me another >> of this exact binary blob", or do you have to identify it by a file >> name? >> >> It wouldn't be a problem for the nonportable mode to toss out an >> exception in weird cases like this, but it would be a problem if >> that causes a segfault or something. > > If you're embedding, you need multiprocessing.setexecutable() > ( https://docs.python.org/3.6/library/multiprocessing.html#multiprocessing.setexecutable ), > so in that case you definitely won't have the same binary...

Ah, and that also showed me that forking isn't mandatory on Unix either. So yeah, there's no assuming that they use the same binary.

Normally it spawns children using sys.executable, which I think on Windows in particular is guaranteed to be the same binary that started the main process, because the OS locks the file while it's executing. But yeah, I didn't think about the embedding case, and apparently there's also a little-known set of features for using multiprocessing between arbitrary python processes: https://docs.python.org/3/library/multiprocessing.html#multiprocessing-listeners-clients

I doubt it'll be a problem to pickle though as it'll use some form of versioning even in NONPORTABLE mode right?

I guess the (merged, but undocumented?) changes in https://bugs.python.org/issue28053 should make it possible to set the pickle version, and yeah, if we did add a NONPORTABLE mode then presumably it would have some kind of header saying which version of python it was created with, so version mismatches could give a sensible error message.

-n -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20180329/bde9b4c3/attachment.html>

More information about the Python-Dev mailing list