Issue 24550: maxint on 64 bit platforms breaks os.read (original) (raw)
Created on 2015-07-02 05:27 by Brian Mingus, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (5)
Author: Brian Mingus (Brian Mingus)
Date: 2015-07-02 05:27
The lower range for this bug may be anything greater than 32 bit maxint. Other modules such as multiprocessing are limited passing objects of size 32 bit maxint, even on 64 bit systems, likely due to this issue. I have demonstrated this by modifying multiprocessing/connection.py to use longs in send and recv, which it surfaces the following error (note that read is os.read).
Traceback (most recent call last): File "/usr/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap self.run() File "/usr/lib/python3.4/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.4/multiprocessing/pool.py", line 103, in worker initializer(*initargs) File "/usr/local/lib/python3.4/dist-packages/gensim-0.11.1_1-py3.4-linux-x86_64.egg/gensim/models/ldamulticore.py", line 266, in worker_e_step chunk_no, chunk, worker_lda = input_queue.get() File "/usr/lib/python3.4/multiprocessing/queues.py", line 96, in get res = self._recv_bytes() File "/usr/lib/python3.4/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength) File "/usr/lib/python3.4/multiprocessing/connection.py", line 420, in _recv_bytes return self._recv(size) File "/usr/lib/python3.4/multiprocessing/connection.py", line 383, in _recv chunk = read(handle, remaining) OverflowError: signed integer is greater than maximum
Which can be traced back to this cpython code:
https://github.com/python/cpython/blob/3.4/Modules/posixmodule.c#L8048-L8065
Author: Serhiy Storchaka (serhiy.storchaka) * 
Date: 2016-05-12 12:53
The 32-bit size limit in multiprocessing is . AFAIK read() and write() now support 64-bit size on 64-bit systems.
Author: Martin Panter (martin.panter) *
Date: 2016-05-13 02:31
Yes 3.5 should do 64-bit reads (if you have enough memory) thanks to revision 0c57aba6b1a3 (Argument Clinic conversion):
os.read(0, 232) Traceback (most recent call last): File "", line 1, in MemoryError os.read(0, 263) Traceback (most recent call last): File "", line 1, in OverflowError: Python int too large to convert to C ssize_t
I understand 3.4 is only open to security fixes, not changes like this. But 2.7 is affected:
os.read(0, 2**32) Traceback (most recent call last): File "", line 1, in OverflowError: signed integer is greater than maximum
I guess this could be considered a bug in 2.7 (not an enhancement), but I’m not really sure.
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2016-05-13 05:41
There is more specific issue for 32-bit reads in 2.7 (with patch): .
I think this issue can be closed as a duplicate of two other issues.
Author: Martin Panter (martin.panter) *
Date: 2016-05-22 01:24
Sorry in Python 3.5 the change was actually Issue 21932 (not Arg Clinic). But Victor said that change shouldn’t go into Python 2.
BTW Issue 21199 is about Python 2’s file.read() method, not os.read().
History
Date
User
Action
Args
2022-04-11 14:58:18
admin
set
github: 68738
2016-05-22 01:24:06
martin.panter
set
superseder: os.read() must use Py_ssize_t for the size parameter
resolution: duplicate -> wont fix
messages: +
2016-05-13 05:41:25
serhiy.storchaka
set
status: open -> closed
resolution: duplicate
messages: +
stage: resolved
2016-05-13 02:31:18
martin.panter
set
nosy: + martin.panter
messages: +
versions: + Python 2.7, - Python 3.4
2016-05-12 12:53:09
serhiy.storchaka
set
nosy: + vstinner, serhiy.storchaka
messages: +
2015-07-02 05:27:49
Brian Mingus
create