(original) (raw)

Well, both fixed 8-byte framing and variable-size framing it introduces a new way of representing numbers in the stream, which means that everyone parsing and generating pickles must be able to support both styles. (But fixed is easier since the XXX8 opcodes use the same format.)

I'm thinking of how you correctly read a pickle from a non-buffering pipe with the minimum number of read() calls without ever reading beyond the end of a valid pickle. (That's a requirement, right?)

If you know it's protocol 4:

�� with fixed framing:

�� read 10 bytes, that's the magic word plus first frame; then you can start buffering

�� with variable framing:

�� read 3 bytes, then depending on the 3rd byte read some more to find the frame size; then you can start buffering

�� with mandatory frame opcode:

�� pretty much the same

�� with optional frame opcode:

�� pretty much the same (the 3rd byte must be a valid opcode, even if it isn't a frame opcode)

if you don't know the protocol number:

�� read the first byte, then read the second byte (or not if it's not explicitly versioned), then you know the protocol and can do the rest as above

On Tue, Nov 19, 2013 at 11:11 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:

On Tue, 19 Nov 2013 11:05:45 -0800

Guido van Rossum <guido@python.org> wrote:

> So using an opcode for framing is out? (Sorry, I've lost track of the
\> back-and-forth.)

It doesn't seem to bring anything, and it makes the overhead worse for
tiny pickles (since it will be two bytes at least, instead of one byte
with the current variable length encoding proposal).

If overhead doesn't matter, I'm fine with keeping a simple 8-bytes
frame size :-)

Regards

Antoine.

--
--Guido van Rossum (python.org/\~guido)