Sockets have a recv_into() method, io.IOBase has a readinto() method, but there is no os.read_into() function. It would avoid memory copies. It would benefit to the Python implementation FileIO (readall() and readinto() methods), see the issue #21859.
os.read_into() may be used by the following functions. subprocess.Popen._execute_child(): # Wait for exec to fail or succeed; possibly raising an # exception (limited in size) errpipe_data = bytearray() while True: part = os.read(errpipe_read, 50000) errpipe_data += part if not part or len(errpipe_data) > 50000: break subprocess.Popen.communicate(): self._fileobj2output = {} if self.stdout: self._fileobj2output[self.stdout] = [] ... data = os.read(key.fd, 32768) if not data: ... self._fileobj2output[key.fileobj].append(data) ... stdout = b''.join(...) multiprocessing.Connection._recv(): def _recv(self, size, read=_read): buf = io.BytesIO() handle = self._handle remaining = size while remaining > 0: chunk = read(handle, remaining) n = len(chunk) if n == 0: if remaining == size: raise EOFError else: raise OSError("got end of file during message") buf.write(chunk) remaining -= n return buf multiprocessing.read_unsigned(): def read_unsigned(fd): data = b'' length = UNSIGNED_STRUCT.size while len(data) < length: s = os.read(fd, length - len(data)) if not s: raise EOFError('unexpected EOF') data += s return UNSIGNED_STRUCT.unpack(data)[0] The problem is that some functions still require to return a bytes, not a bytearray or something else. Converting a bytearray to a bytes still require a memory copy...
Without more interested, I chose to defer this issue. Feel free to reopen it if you need it for more use cases, or if you are interested to implement it.
History
Date
User
Action
Args
2022-04-11 14:58:14
admin
set
github: 67942
2015-05-25 22:51:43
vstinner
set
status: open -> closedresolution: postponedmessages: +
