[Python-Dev] http.client Nagle/delayed-ack optimization (original) (raw)

Antoine Pitrou solipsis at pitrou.net
Fri Dec 14 20:27:00 CET 2012

On Sat, 15 Dec 2012 06:17:19 +1100 Ben Leslie <benno at benno.id.au> wrote:

The http.client HTTPConnection.sendoutput method has an optimization for avoiding bad interactions between delayed-ack and the Nagle algorithm:

http://hg.python.org/cpython/file/f32f67d26035/Lib/http/client.py#l884 Unfortunately this interacts rather poorly if the case where the messagebody is a bytes instance and is rather large. If the messagebody is bytes it is appended to the headers, which causes a copy of the data. When messagebody is large this duplication of data can cause a significant spike in memory usage. (In my particular case I was uploading a 200MB file to 30 hosts at the same leading to memory spikes over 6GB. I've solved this by subclassing and removing the optimization, however I'd appreciate thoughts on how this could best be solved in the library itself. Options I have thought of are: 1: Have some size threshold on the copy. A little bit too much magic. Unclear what the size threshold should be.

I think a hardcoded threshold is the right thing to do. It doesn't sound very useful to try doing a single send() call when you have a large chunk of data (say, more than 1 MB).

Regards

Antoine.

More information about the Python-Dev mailing list