Issue 10319: SocketServer.TCPServer truncates responses on close (in some situations) (original) (raw)

Behavior exists in at least Python 2.5 through 3.1.

The issue regards the socketserver's handling of the tcp sockets it works with. On object reaping, the sockets are properly closed (in some versions explicitly, in others implicitly). However, closing a socket with unread data will in some operating systems prevent sent data from being sent. Notably this occurs on Linux and Windows (I tested on Debian testing x86_64 and Windows Vista x86).

Extensive tcpdump sessions and a lot of head scratching finally clarified the issue. The OS would send an RST packet after just the first bit of data handed over to the socket implementation, in typical cases meaning hte first line of text python is writing would be sent to the client, but no more, causing invalid HTTP replies.

Here is a change to force the socket to be drained on close. It is of no matter that more data may arrive on the socket post-drain, because the socket implementation in the operating system should be happy to send off what was already queued by the python program at this point.

Testing bears this out, across many platforms here.

Here is one way to accomplish this in python 2.x, in 3.x the equivalent file is socketserver.py, but has the same issue. It lacks the explicit request.close() call, bgut this seems a matter of aesthetics.

Note that the use of select.select() is documented by Linux to be unsafe for this purpose in some circumstances, and may be on other platforms as well.

I sort of would rather have sockets have a socket.drain() function which is known to be safe and correct for this type of socket programming issue. If that would be more appropriate, please suggest.

select(2) Under Linux, select() may report a socket file descriptor as "ready for reading", while nevertheless a subsequent read blocks. This could for example happen when data has arrived but upon examination has wrong checksum and is discarded. There may be other circumstances in which a file descriptor is spuriously reported as ready. Thus it may be safer to use O_NONBLOCK on sockets that should not block.

--- SocketServer.py.orig 2010-11-04 21:36:23.000000000 -0700 +++ SocketServer.py 2010-11-04 21:38:19.000000000 -0700 @@ -445,6 +445,13 @@

 def close_request(self, request):
     """Called to clean up an individual request."""

```
   # drain socket
```
```
   request.setblocking(0)
```
```
   try:
```
```
       while True:
```
```
           request.recv(4096)
```
```
   except socket.error, e:
```
```
       pass
   request.close()
```

shutdown isn't the same as close. I'm not sure that's correct behavior in BaseHTTPServer depending upon whether it wants to keep the door open to weird pipeline behaviors.

Honestly, I'm having to read through tcp/ip illustrated because of this, so I'm not really the proper person to make the call. Perhaps the bug is, as you imply, in BaseHTTPServer, not TCPServer.

BaseHTTPServer documentation explains that "Usually, this module isn’t used directly, but is used as a basis for building functioning Web servers". Methods are usually subclassed with desirable behaviors customized.

If you are using SocketServer in your code directly and calling close_request (where you want all your requests to be drained on closing), I think, you should just use shutdown_request.

This bug can be closed.