Issue 33733: Add utilities to get/set pipe and socket buffer sizes? (original) (raw)

Issue33733

Created on 2018-06-01 13:18 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (3)
msg318411 - (view)	Author: STINNER Victor (vstinner) *	Date: 2018-06-01 13:18
Many Python unit tests require a blocking send into a pipe or a socket. support.PIPE_MAX_SIZE (4 MiB +1 B) has been added for pipes in bpo-17835: sending PIPE_MAX_SIZE into a pipe must block. On Linux, the maximum size of a pipe is /proc/sys/fs/pipe-max-size. Since Linux 2.6.35, it's now possible to get and set the size of a pipe using fcntl with F_GETPIPE_SZ and F_SETPIPE_SZ commands. For sockets, support.SOCK_MAX_SIZE (16 MiB) has been added in bpo-18643. It's possible to get/set the size of receive and send socket buffers using getsockopt() and setsockopt() with SO_RCVBUF and SO_SNDBUF commands. For pipes, I'm not sure that it's possible to get the size of a pipe buffer in a portable way. For example, F_GETPIPE_SZ was only introduced in Linux 2.6.35. Since the first user of these features are tests, maybe we can start with best-effort functions in test.support. Recently, I got issues with buffer sizes in test_multiprocessing_forkserver.test_ignore() (bpo-33532) and sendfile tests of test_asyncio (bpo-33353).
msg318428 - (view)	Author: Nathaniel Smith (njs) *	Date: 2018-06-01 15:54
Note that in my experience, socket systems treat the buffer sizes as more like... rough guidelines. Especially Windows and Linux. Which doesn't mean they're not useful to expose somehow, but you can't assume that just because you set the buffer to size X means that operations will start blocking after X bytes. When trio's tests need a clogged socket, we just keep sending in a loop until we observe a BlockingIOError, and then do the test. Pipe buffers are much more reasonably behaved IME.
msg320052 - (view)	Author: STINNER Victor (vstinner) *	Date: 2018-06-20 10:13
> Note that in my experience, socket systems treat the buffer sizes as more like... rough guidelines. Oh. Now I'm scared. I succeeded to fix bpo-33532 and bpo-33353, so I no longer need this issue. Since it seems hard to have a portable and *reliable* behaviour, I prefer to abandon this idea. Sorry!