Issue 7806: httplib.HTTPConnection.getresponse closes socket which destroys the response (original) (raw)

Issue7806

Created on 2010-01-29 15:01 by Robert.Buchholz, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (15)
msg98513 - (view)	Author: Robert Buchholz (Robert.Buchholz)	Date: 2010-01-29 15:01
Calling getresponse() on an httplib.HTTPConnection object returns a response object. Internally, the self.sock is handed over to the HTTPResponse object which transforms it into a file-like object. The response object is returned to the caller. If one calls response.read() later on, no or incomplete content will be returned because the underlying socket has been closed. The code path, simplified: class HTTPConnection: def getresponse(self): response = self.response_class(self.sock, ...) ... if response.will_close: # this effectively passes the connection to the response self.close() def close(self): if self.sock: self.sock.close() ... class HTTPResponse: def __init__(self, sock, debuglevel=0, strict=0, method=None): self.fp = sock.makefile('rb', 0) ...
msg98678 - (view)	Author: Sijin Joseph (sijinjoseph)	Date: 2010-02-01 19:42
Looking at the code in httplib it seems that response.will_close is set under the following circumstances, 1. HTTP version is 0.9 2. HTTP response header connection is set to close 3. Non-chunked content with a length of zero This suggests that the underlying socket closure is valid under the conditions and that a subsequent response.read() should not be returning any content. If you think this is still an issue, I'd suggest that you create a small example client/server script that Client: 1. Opens a HTTP connection 2. Continues to read data from it and dumps it to the console. Server: 1. Setup a script that does not close incoming HTTP connection requests and continues to keep sending data back to the client.
msg98695 - (view)	Author: Robert Buchholz (Robert.Buchholz)	Date: 2010-02-01 22:23
An example cannot be constructed using the standard python socket class. As you point out, the response.will_close attribute is set correctly: The client is supposed to close to connect after completion of the request (as does the server). However, when the HTTPConnection object calls close() on the socket, reading the request is not completed -- the request object merely created a file-like object representing the socket. The reason why this is not an issue is that the Python socket library does reference counting to determine when to close a socket object. When the HTTPConnection calls close(), its own socket object is destroyed and unreferenced. However, the file-like object in the HTTPResponse still has a copy of the socket. In alternative socket implementations (like Paramiko SOCKS-proxied sockets), this poses a problem: When the socket user calls close(), they actually close the socket -- as the original socket API documentation describes: close() Close the socket. It cannot be used after this call. makefile([mode[, bufsize]]) -> file object Return a regular file object corresponding to the socket. The mode and bufsize arguments are as for the built-in open() function. Consequently, I do not consider this to be a bug in Paramiko and reported it for the httplib. I can present example code using a paramiko tunneled socket (client and server) if you like.
msg99599 - (view)	Author: Jeremy Hylton (jhylton)	Date: 2010-02-19 22:57
I don't think the HTTPConnection class was designed to work with sockets that don't follow the Python socket API. If you want to use a different socket, you should create some wrapper that emulates the Python socket ref count behavior. Jeremy On Mon, Feb 1, 2010 at 5:23 PM, Robert Buchholz <report@bugs.python.org> wrote: > > Robert Buchholz <rbu@freitagsrunde.org> added the comment: > > An example cannot be constructed using the standard python socket class. As you point out, the response.will_close attribute is set correctly: The client is supposed to close to connect after completion of the request (as does the server). However, when the HTTPConnection object calls close() on the socket, reading the request is not completed -- the request object merely created a file-like object representing the socket. > > The reason why this is not an issue is that the Python socket library does reference counting to determine when to close a socket object. When the HTTPConnection calls close(), its own socket object is destroyed and unreferenced. However, the file-like object in the HTTPResponse still has a copy of the socket. > > In alternative socket implementations (like Paramiko SOCKS-proxied sockets), this poses a problem: When the socket user calls close(), they actually close the socket -- as the original socket API documentation describes: > >    close() >    Close the socket.  It cannot be used after this call. > >    makefile([mode[, bufsize]]) -> file object >    Return a regular file object corresponding to the socket.  The mode >    and bufsize arguments are as for the built-in open() function. > > Consequently, I do not consider this to be a bug in Paramiko and reported it for the httplib. I can present example code using a paramiko tunneled socket (client and server) if you like. > > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue7806> > _______________________________________ > _______________________________________________ > Python-bugs-list mailing list > Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/jeremy%40alum.mit.edu > >
msg99601 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2010-02-19 23:22
But a goal is for the standard library to work with Python implementations other than CPython, and the reference counting behavior described won't happen in non-reference counting implementations. So I don't think that requiring emulation of ref-counting behavior is valid.
msg99608 - (view)	Author: Jeremy Hylton (jhylton)	Date: 2010-02-20 04:29
On Fri, Feb 19, 2010 at 6:22 PM, R. David Murray <report@bugs.python.org> wrote: > > R. David Murray <rdmurray@bitdance.com> added the comment: > > But a goal is for the standard library to work with Python implementations other than CPython, and the reference counting behavior described won't happen in non-reference counting implementations.  So I don't think that requiring emulation of ref-counting behavior is valid. I don't think an implementation of Python sockets would be considered correct unless it supported this behavior. CPython goes to elaborate lengths to make it work across platforms. Jeremy > ---------- > nosy: +r.david.murray > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue7806> > _______________________________________ >
msg99610 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2010-02-20 05:05
But the docs (which presumably describe the API) say that the socket is unusable after the call to close, which argues that the paramiko sockets are following the documented API. Do the docs need to be corrected?
msg99680 - (view)	Author: Jeremy Hylton (jhylton)	Date: 2010-02-21 20:23
On Sat, Feb 20, 2010 at 12:06 AM, R. David Murray <report@bugs.python.org> wrote: > > R. David Murray <rdmurray@bitdance.com> added the comment: > > But the docs (which presumably describe the API) say that the socket is unusable after the call to close, which argues that the paramiko sockets are following the documented API.  Do the docs need to be corrected? I mean the documented socket API. Jeremy > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue7806> > _______________________________________ >
msg99681 - (view)	Author: Jeremy Hylton (jhylton)	Date: 2010-02-21 20:32
In particular, I mean this part of the socket API: socket.makefile([mode[, bufsize]]) Return a file object associated with the socket. (File objects are described in File Objects.) The file object references a dup()ped version of the socket file descriptor, so the file object and socket object may be closed or garbage-collected independently. The socket must be in blocking mode (it can not have a timeout). The optional mode and bufsize arguments are interpreted the same way as by the built-in file() function. The language may be a little vague, but it means that closing the file generated by makefile() should not close the underlying socket. Jeremy On Sun, Feb 21, 2010 at 3:23 PM, Jeremy Hylton <report@bugs.python.org> wrote: > > Jeremy Hylton <jeremy@alum.mit.edu> added the comment: > > On Sat, Feb 20, 2010 at 12:06 AM, R. David Murray > <report@bugs.python.org> wrote: >> >> R. David Murray <rdmurray@bitdance.com> added the comment: >> >> But the docs (which presumably describe the API) say that the socket is unusable after the call to close, which argues that the paramiko sockets are following the documented API.  Do the docs need to be corrected? > > I mean the documented socket API. > > Jeremy > >> ---------- >> >> _______________________________________ >> Python tracker <report@bugs.python.org> >> <http://bugs.python.org/issue7806> >> _______________________________________ >> > > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue7806> > _______________________________________ > _______________________________________________ > Python-bugs-list mailing list > Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/jeremy%40alum.mit.edu > >
msg99683 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2010-02-21 20:42
So do I. I'm saying that paramiko appears to be following the socket API as documented in the python docs (ie: that closing the socket means it is no longer usable).
msg99685 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2010-02-21 20:48
So HTTConnection is closing the thing returned by makefile and that is closing the socket, except that the socket library makes sure it doesn't actually close the socket until the dupped file handle is also closed? I guess I need to look at this more closely when I have time in order to fully understand it.
msg99695 - (view)	Author: Robert Buchholz (Robert.Buchholz)	Date: 2010-02-21 22:38
almost... HTTPConnection is calling close() on the socket object, but HTTPResponse still has an open file-like object from a previous makefile() call. That object still has an internal reference to the socket.
msg99696 - (view)	Author: Jeremy Hylton (jhylton)	Date: 2010-02-21 22:56
On Sun, Feb 21, 2010 at 5:38 PM, Robert Buchholz <report@bugs.python.org> wrote: > > Robert Buchholz <rbu@freitagsrunde.org> added the comment: > > almost... HTTPConnection is calling close() on the socket object, but HTTPResponse still has an open file-like object from a previous makefile() call. That object still has an internal reference to the socket. That's right. The makefile() method on sockets works that way, and the HTTP library depends on that behavior (and pretty much always has). Jeremy > > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue7806> > _______________________________________ >
msg99701 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2010-02-22 02:15
OK, then I think I understand Jeremy's point now: the paramiko socket is apparently not implementing makefile in a way that matches the documented API.
msg104014 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2010-04-23 14:15
Hearing no further argument to the contrary, I'm closing this as invalid.