[Python-Dev] Accepting PEP 3154 for 3.4? (original) (raw)

Antoine Pitrou solipsis at pitrou.net
Mon Nov 18 23:31:34 CET 2013

On Mon, 18 Nov 2013 16🔞21 -0600 Tim Peters <tim.peters at gmail.com> wrote:

But it has a different kind of advantage: PREFETCH was optional. As Guido said, it's annoying to bloat the size of small pickles (which may, although individually small, occur in great numbers) by 8 bytes each. There's really no point to framing small chunks of data, right?

You can't know how much space the pickle will take until the pickling ends, though, which makes it difficult to decide whether you want to emit a PREFETCH opcode or not.

Which leads to another idea: after the PROTO opcode, there is, or is not, an optional PREFETCH opcde with an 8-byte argument. If the PREFETCH opcode exists, then it gives the number of bytes up to and including the pickle's STOP opcode. So there's exactly 0 or 1 PREFETCH opcodes per pickle.

Is there an advantage to spraying multiple 8-byte "frame counts" throughout a pickle stream?

Well, yes: much better memory usage for large pickles. Some people use pickles to store huge data, which was the motivation to add the 8-byte-size opcodes after all.

I'm not thinking of DOS at all, just general sanity as data objects get larger & larger. Pickles have almost no internal checks now. But I've seen my share of corrupted pickles!

IMO, any validation should use a dedicated CRC-like scheme, rather than relying on the fact that correct pickles are statistically unlikely :-)

(or you can implicitly rely on TCP or UDP checksums, or you disk subsystem's own sanity checks)

Regards

Antoine.

More information about the Python-Dev mailing list