[Python-ideas] reducing multiprocessing.Queue contention (original) (raw)

Richard Oudkerk shibturn at gmail.com
Wed Jan 23 13:09:42 CET 2013

On 23/01/2013 11:16am, Charles-François Natali wrote:

Connection.send() and Connection.recv() have to serialize the data using pickle before writing them to the underlying file descriptor. While the locking is necessary to guarantee atomic read/write (well, it's not necessary if you're writing to a pipe less than PIPEBUF, and writes seem atomic on Windows), the locks don't have to be held while the data is serialized.

But you can only rely on the atomicity of writing less than PIPE_BUF bytes if you know that no other process is currently trying to send a message longer than PIPE_BUF. Otherwise the short message could be embedded in the long message (even if the process sending the long message is holding the lock).

-- Richard

More information about the Python-ideas mailing list