[Python-Dev] PEP 574 -- Pickle protocol 5 with out-of-band data (original) (raw)

Chris Jerdonek chris.jerdonek at gmail.com
Thu Mar 29 03:56:59 EDT 2018

On Wed, Mar 28, 2018 at 6:15 PM, Nathaniel Smith <njs at pobox.com> wrote:

On Wed, Mar 28, 2018 at 1:03 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:

28.03.18 21:39, Antoine Pitrou пише:

I'd like to submit this PEP for discussion. It is quite specialized and the main target audience of the proposed changes is users and authors of applications/libraries transferring large amounts of data (read: the scientific computing & data science ecosystems).

Currently I'm working on porting some features from cloudpickle to the stdlib. For these of them which can't or shouldn't be implemented in the general purpose library (like serializing local functions by serializing their code objects, because it is not portable) I want to add hooks that would allow to implement them in cloudpickle using official API. This would allow cloudpickle to utilize C implementation of the pickler and unpickler. There's obviously some tension here between pickle's use as a persistent storage format, and its use as a transient wire format. For the former, you definitely can't store code objects because there's no forwards- or backwards-compatibility guarantee for bytecode. But for the latter, transmitting bytecode is totally fine, because all you care about is whether it can be decoded once, right now, by some peer process whose python version you can control -- that's why cloudpickle exists.

Is it really true you'll always be able to control the Python version on the other side? Even if they're internal services, it seems like there could be times / reasons preventing you from upgrading the environment of all of your services at the same rate. Or did you mean to say "often" all you care about ...?

--Chris

Would it make sense to have a special pickle version that the transient wire format users could opt into, that only promises compatibility within a given 3.X release cycle? Like version=-2 or version=pickle.NONPORTABLE or something? (This is orthogonal to Antoine's PEP.) -n -- Nathaniel J. Smith -- https://vorpus.org

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com

More information about the Python-Dev mailing list