[Python-Dev] http.client Nagle/delayed-ack optimization (original) (raw)

Ben Leslie benno at benno.id.au
Fri Dec 14 20:17:19 CET 2012

The http.client HTTPConnection._send_output method has an optimization for avoiding bad interactions between delayed-ack and the Nagle algorithm:

http://hg.python.org/cpython/file/f32f67d26035/Lib/http/client.py#l884

Unfortunately this interacts rather poorly if the case where the message_body is a bytes instance and is rather large.

If the message_body is bytes it is appended to the headers, which causes a copy of the data. When message_body is large this duplication of data can cause a significant spike in memory usage.

(In my particular case I was uploading a 200MB file to 30 hosts at the same leading to memory spikes over 6GB.

I've solved this by subclassing and removing the optimization, however I'd appreciate thoughts on how this could best be solved in the library itself.

Options I have thought of are:

1: Have some size threshold on the copy. A little bit too much magic. Unclear what the size threshold should be.

2: Provide an explicit argument to turn the optimization on/off. This is ugly as it would need to be attached up the call chain to the request method.

3: Provide a property on the HTTPConnection object which enables the optimization or not. Optionally configured as part of init.

4: Add a class level attribute (similar to auto_open, default_port, etc) which controls the optimization.

Be very interested to get some feedback so I can craft the appropriate patch.

Thanks,

Benno -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121215/6b7885ba/attachment.html>

More information about the Python-Dev mailing list