[Python-bugs-list] [ python-Bugs-211710 ] socket.send() can do partial writes on some systems. (original) (raw)

noreply@sourceforge.net noreply@sourceforge.net
Sat, 22 Dec 2001 18:26:45 -0800

Bugs item #211710, was opened at 2000-08-11 13:25 You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=105470&aid=211710&group_id=5470

Category: Python Library Group: Platform-specific

Status: Closed Resolution: Fixed Priority: 5 Submitted By: scott cotton (scottc) Assigned to: Jeremy Hylton (jhylton) Summary: socket.send() can do partial writes on some systems.

Initial Comment: 0811 15:57 chronis:% uname -a FreeBSD chronis.pobox.com 4.0-STABLE FreeBSD 4.0-STABLE #0: Tue Mar 21 01:05:14 EST 2000 root@chronis.pobox.com:/usr/src/sys/compile/MYBSD i386 0811 15:57 chronis:%

0811 15:57 chronis:~% cat scratch/sendtstsrv.py
import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind("chronis.pobox.com", 8010)

while 1: s.listen(1) conn, addr = s.accept() print "connected from", addr while 1: data = conn.recv(1024) if not data: break print "done" conn.close()

0811 15:58 chronis:~% python scratch/sendtstsrv.py & [2] 76562

0811 15:58 chronis:~% cat scratch/sendtst.py #!/usr/bin/env python

s = "0" * 10000000

import socket sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect("chronis.pobox.com", 8010) sent = sock.send(s) if sent != len(s): print "sent %d/%d chars" % (sent, len(s)) else: print "sent all chars"

0811 15:58 chronis:% python !$ python scratch/sendtst.py connected from ('208.210.124.49', 1258) sent 17520/10000000 chars done 0811 15:58 chronis:%

NOTE: there is nothing in any man page for send() under linux (RH 6.1), Solaris (SunOS 5.5.1), OSF1 V4.0, or FreeBSD{3,4} that states that send() must not perform a partial write, though of those systems, it only seems to reproducibly do partial writes under FreeBSD4.0 Stable.

Additionally, W.Richard Stevens' Unix Network Programming Vol 1, second edition states that """ A read or write on a stream socket might input or output fewer bytes than requested, but this is not an error condition.""" (page 77). Later, Stevens says of send() and recv() "These two functions are similar to the standard read and write functions, but one additional argument is required". (page 354).

scott

Comment By: Anthony Baxter (anthonybaxter) Date: 2001-12-22 18:26

Message: Logged In: YES user_id=29957

Ah - should have gone back and closed this. The eventual design that was taken was to add a socket.sendall() call, and make the various libs call this. This was added to 2.2, and is also in 2.1.2-to-be.

Comment By: Anthony Baxter (anthonybaxter) Date: 2001-12-22 18:26

Message: Logged In: YES user_id=29957

Ah - should have gone back and closed this. The eventual design that was taken was to add a socket.sendall() call, and make the various libs call this. This was added to 2.2, and is also in 2.1.2-to-be.

Comment By: Mike Brown (mike_j_brown) Date: 2001-12-22 18:13

Message: Logged In: YES user_id=371366

I disagree with changing the behavior of socket.py. The API is clear and Python is not in the wrong for being a very thin wrapper over the underlying implementation. Failing to check the return value of send() is a common socket programming error in all languages. Just change the libs that make this mistake. There is probably a use case for seeing the returns from partial writes.

Comment By: Guido van Rossum (gvanrossum) Date: 2001-10-21 18:02

Message: Logged In: YES user_id=6380

Please go slowly here.

Comment By: Anthony Baxter (anthonybaxter) Date: 2001-10-21 00:46

Message: Logged In: YES user_id=29957

On second (or is this third?) thoughts, patching socket.py to make it wrap the _socketmodule send code is an alternative approach that may be safer.

Pity about the _fileobject, tho - it needs considerably more help.

Comment By: Anthony Baxter (anthonybaxter) Date: 2001-10-21 00:07

Message: Logged In: YES user_id=29957

The two approaches would seem to be:

patch the socketmodule to make sure send() actually sends all the bytes. or patch the offending std library calls.

The latter preserves the std unix semantics, but possibly violates the principle of Least Suprise.

For 2.1.2, I'm patching the std library.

Comment By: Guido van Rossum (gvanrossum) Date: 2001-10-19 05:20

Message: Logged In: YES user_id=6380

Reopened. You have a point. (I think we closed it becase we couldn't reproduce it. Our fault for not having access to FreeBSD. :-)

Comment By: Richard Jones (richard) Date: 2001-10-18 23:43

Message: Logged In: YES user_id=6405

This problem is only seen on FreeBSD, it seems. Not even OSX, the BSD derivative, has this problem. However, my FreeBSD (4.3-RELEASE) socket(2) man page states:

 If no messages space is available at the socket to

hold the message to be transmitted, then send() normally blocks, unless the socket has been placed in non-blocking I/O mode. The select(2) call may be used to de- termine when it is possible to send more data.

Linux (2.4.7) has a similar message:

When the message does not fit into the send buffer of the socket, send normally blocks, unless the socket has
been placed in non-blocking I/O mode. In non-blocking mode it would return EAGAIN in this case. The select(2) call
may be used to determine when it is possible to send more data.

So "normally" is being interpreted loosly by FreeBSD in this case, it seems.

A solution might be to have socket.send() try to send all the data. At a minimum, the standard library should be fixed so it works on FreeBSD.

From the Lib directory of 2.1.1: [richard@ike /usr/local/src/Python-2.1.1]% grep -r '.send(' Lib Lib/ftplib.py: self.sock.send(line) Lib/ftplib.py: self.sock.send(line, MSG_OOB) Lib/ftplib.py: conn.send(buf) Lib/ftplib.py: conn.send(buf) Lib/gopherlib.py: s.send(selector + CRLF) Lib/httplib.py: self.sock.send(str) Lib/httplib.py: self.send(str) Lib/httplib.py: self.send(str) Lib/httplib.py: self.send(str) Lib/httplib.py: self.send('\r\n') Lib/httplib.py: self.send(body) Lib/imaplib.py: self.sock.send('%s%s' % (data, CRLF)) Lib/imaplib.py: self.sock.send(literal) Lib/imaplib.py: self.sock.send(CRLF) Lib/nntplib.py: self.sock.send(line) Lib/poplib.py: self.sock.send('%s%s' % (line, CRLF)) Lib/smtplib.py: sendptr = sendptr + self.sock.send(str[sendptr:]) Lib/smtplib.py: self.send(str) Lib/smtplib.py: self.send(q) Lib/socket.py: self._sock.send(self._wbuf) Lib/telnetlib.py: self.sock.send(buffer) Lib/telnetlib.py: self.sock.send(IAC + WONT + opt) Lib/telnetlib.py: self.sock.send(IAC + DONT + opt) Lib/urllib.py: h.send(data) Lib/urllib.py: h.send(data) Lib/urllib2.py: h.send(data) Lib/webbrowser.py: s.send(action) Lib/test/test_asynchat.py: n = conn.send(buffer) Lib/test/test_socket.py: conn.send(data) Lib/test/test_socket.py: s.send(msg) Lib/test/test_socketserver.py: s.send(teststring)

Note that smtplib handles send()'s return, where nothing else does. Note also that the above grep does not include our patch.

Comment By: Richard Jones (richard) Date: 2001-10-18 23:04

Message: Logged In: YES user_id=6405

We have patched httplib.HTTPConnection.send() so it does the following:

sent = 0 while sent < len(str): sent = sent + self.sock.send(str[sent:])

Could this be the default behaviour in the socket module perhaps?

Comment By: Richard Jones (richard) Date: 2001-10-18 22:57

Message: Logged In: YES user_id=6405

Why is this bug closed? The standard library uses send() in a number of places - we've just run up against it in httplib. Again, on FreeBSD, we're seeing that send() is only sending a part of the data we want it to.

Comment By: Jeremy Hylton (jhylton) Date: 2000-09-15 11:50

Message: scott-- Is there a bug here? It seems clear that the send system call is not required to send all the bytes you asked it to send. It returns the number of bytes it did send. If you want to make sure all your data is sent, you can wrap send in a while loop that calls send until all the data is consumed.

Comment By: Jeremy Hylton (jhylton) Date: 2000-09-07 15:01

Message: Please do triage on this bug.

You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=105470&aid=211710&group_id=5470