Issue 27020: os.writev() does not accept generators (as buffers argument) (original) (raw)
Created on 2016-05-14 17:30 by socketpair, last changed 2022-04-11 14:58 by admin.
Messages (7)
Author: Марк Коренберг (socketpair) *
Date: 2016-05-14 17:30
Unlike socket.sendmsg(), os.writev() does not support generators.
Proof:
In [4]: os.writev(1, [b'aa', b'bb', b'\n']) aabb Out[4]: 5
In [5]: os.writev(1, (i for i in [b'aa', b'bb', b'\n'])) ... TypeError: writev() arg 2 must be a sequence
Author: Serhiy Storchaka (serhiy.storchaka) * 
Date: 2016-05-14 21:05
This makes sense. The purpose of writev() is to write all data in single atomic operation. In any case a sequence should be built.
This looks rather as intentional behavior than as a bug to me.
Author: Марк Коренберг (socketpair) *
Date: 2016-05-15 04:41
Well, I'm improving asyncio performance.
Instead of using bytearray as a big buffer and appending to it, I use deque of buffers. This eliminates memory copying.
When writing is possible, I call writev()/sendmsg().
Unfortunatelly, OS sets limit on count of buffers passed to writev()/sendmsg(). So I should slice deque to cut first N members. This is not possible too, deque()[:N] does not work.
So, I decide to use itertools.islice(deque(), 0, N) for that. This work great with sendmsg(). But fail with writev().
Also, writev guarantee atimicity with ther writes in kernel, we must not limit way of obtainig buffers before passing to kernel.
If Python have emulation of writev() on OS where it is not supported, it should be thrown out, because atomicity requirements will not be accomplished. And writes to non-seekable fd from another process may be intermixed. Say, for example, files opened in append-only mode.
Author: Марк Коренберг (socketpair) *
Date: 2016-05-15 04:42
Pull request to asyncio:
https://github.com/python/asyncio/pull/339
Author: Марк Коренберг (socketpair) *
Date: 2016-05-16 05:56
How sequence is iterated in writev():
How IOV is filled in semdmsg(): https://github.com/python/cpython/blob/fa5d369fc83f695d19eb57048efa7ec07ca3c1d7/Modules/socketmodule.c#L3817
Also, IMHO these parts of code shoud be merged.
Author: Марк Коренберг (socketpair) *
Date: 2016-05-16 06:00
Proof for sendmsg:
In [1]: import socket In [2]: (r, w) = socket.socketpair() In [3]: w.sendmsg((i for i in [b'1', b'2'])) Out[3]: 2
Author: Марк Коренберг (socketpair) *
Date: 2016-05-16 06:10
Moreover, sendmsg() flattens iterable to a list -- this is not very good for performance and memory.
Why not to fill IOV one-by-one, doubling it's size (via realloc) when need ?
History
Date
User
Action
Args
2022-04-11 14:58:31
admin
set
github: 71207
2016-05-16 06:10:27
socketpair
set
messages: +
2016-05-16 06:00:51
socketpair
set
messages: +
2016-05-16 05:56:30
socketpair
set
messages: +
2016-05-15 04:42:11
socketpair
set
messages: +
2016-05-15 04:41:28
socketpair
set
messages: +
2016-05-14 21:05:47
serhiy.storchaka
set
nosy: + serhiy.storchaka
messages: +
2016-05-14 17:30:43
socketpair
create