[Python-ideas] reducing multiprocessing.Queue contention (original) (raw)

Charles-François Natali cf.natali at gmail.com
Wed Jan 23 13:27:39 CET 2013

But you can only rely on the atomicity of writing less than PIPEBUF bytes if you know that no other process is currently trying to send a message longer than PIPEBUF. Otherwise the short message could be embedded in the long message (even if the process sending the long message is holding the lock).

Maybe I wasn't clear. I'm not suggesting to not hold the lock when sending less than PIPE_BUF, since it wouldn't work in the case you describe above. I'm suggesting to serialize the data prior to acquiring the writer lock, to reduce contention (and unserialize after releasing the reading lock).

(I only mentioned PIPE_BUF because I was sad to see that Windows supported atomic messages, and this comforted me a bit :-)

More information about the Python-ideas mailing list