[Python-Dev] PEP 574 -- Pickle protocol 5 with out-of-band data (original) (raw)

Robert Collins robertc at robertcollins.net
Wed Mar 28 21:40:17 EDT 2018

One question..

On Thu., 29 Mar. 2018, 07:42 Antoine Pitrou, <solipsis at pitrou.net> wrote:

...

=======

Mutability ---------- PEP 3118 buffers [#pep-3118] can be readonly or writable. Some objects, such as Numpy arrays, need to be backed by a mutable buffer for full operation. Pickle consumers that use the buffercallback and buffers arguments will have to be careful to recreate mutable buffers. When doing I/O, this implies using buffer-passing API variants such as readinto (which are also often preferrable for performance). Data sharing ------------ If you pickle and then unpickle an object in the same process, passing out-of-band buffer views, then the unpickled object may be backed by the same buffer as the original pickled object. For example, it might be reasonable to implement reduction of a Numpy array as follows (crucial metadata such as shapes is omitted for simplicity):: class ndarray: def reduceex(self, protocol): if protocol == 5: return numpy.frombuffer, (PickleBuffer(self), self.dtype) # Legacy code for earlier protocols omitted Then simply passing the PickleBuffer around from dumps to loads will produce a new Numpy array sharing the same underlying memory as the original Numpy object (and, incidentally, keeping it alive)::

This seems incompatible with v4 semantics. There, a loads plus dumps combination is approximately a deep copy. This isn't. Sometimes. Sometimes it is.

Other than that way, I like it.

Rob

-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20180329/d021e9fb/attachment.html>

More information about the Python-Dev mailing list