[Python-Dev] Accepting PEP 3154 for 3.4? (original) (raw)

Antoine Pitrou solipsis at pitrou.net
Mon Nov 18 23:39:16 CET 2013

On Mon, 18 Nov 2013 16:25:07 -0600 Tim Peters <tim.peters at gmail.com> wrote:

[Antoine] > Are very small pickles that size-sensitive? I have the impression that > if 8 bytes vs. e.g. 15 bytes makes a difference for your application, > you'd be better off with a hand-made format. The difference between 8 and 15 is, e.g., nearly doubling the amount of network traffic (for apps that use pickles across processes or machines). A general approach has no way to guess how it will be used. For example, multiprocessing uses pickles extensively for inter-process communication of Python data. Some users try broadcasting giant arrays across processes, while others broadcast oceans of tiny integers (like indices into giant arrays inherited via fork()).

Well, sending oceans of tiny integers will also incur many system calls and additional synchronization costs, since sending data on a multiprocessing Queue has to acquire a semaphore. So it generally sounds like a bad idea, IMHO.

That said, I agree with:

Since pickle intends to be "the" Python serialization format, it really should try to be friendly for all plausible uses.

I simply don't think adding a fixed 8-byte overhead is actually annoying. It's less than the PyObject overhead in 64-bit mode...

Regards

Antoine.

More information about the Python-Dev mailing list