Issue 14794: slice.indices raises OverflowError (original) (raw)
Created on 2012-05-12 22:51 by Paul.Upchurch, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (30)
Author: Paul Upchurch (Paul.Upchurch)
Date: 2012-05-12 22:51
To reproduce the error:
Python 3.2.2 (default, Sep 5 2011, 22:09:30) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or "license" for more information.
slice(0,90000,None).indices(12600000000) Traceback (most recent call last): File "", line 1, in OverflowError: cannot fit 'int' into an index-sized integer
The correct behaviour is to return (0, 90000, 1).
slice(0,90000,None).indices(600000000) (0, 90000, 1)
This is related to http://bugs.python.org/issue1456470.
Author: Hynek Schlawack (hynek) * 
Date: 2012-05-12 22:57
This seems to have been fixed as of 3.2.3 (as shipped with Ubuntu Precise):
Python 3.2.3 (default, Apr 12 2012, 19:08:59) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information.
slice(0,90000,None).indices(12600000000) (0, 90000, 1)
Current tip works fine too:
Python 3.3.0a3+ (default:b32baa5b7626+, May 10 2012, 14:56:20) [GCC 4.6.3] on linux Type "help", "copyright", "credits" or "license" for more information.
slice(0,90000,None).indices(12600000000) (0, 90000, 1)
I'd close this bug unless I'm missing something?
Author: Paul Upchurch (Paul.Upchurch)
Date: 2012-05-12 23:13
Sorry. I didn't realize there was a 3.2.3 out. I'll close it as fixed.
Author: Éric Araujo (eric.araujo) *
Date: 2012-05-13 05:01
This seems to have been fixed as of 3.2.3 (as shipped with Ubuntu Precise)
Just a note: you can’t really trust the behavior of Python shipped by Debian or derivative systems because doko (the Debian Python maintainer) backports changes to released versions, which means that Debian’s 3.2.3 may not always behave as python.org’s 3.2.3.
Author: Paul Upchurch (Paul.Upchurch)
Date: 2012-05-13 17:31
That's true; it doesn't work with today's downloads from python.org. The version I tested was win32 but I don't think that should matter. Python has always supported large numbers on 32-bit OSs.
My observations:
[1] Debian Wheezy, python3.2, 3.2.3rc2-1: Fail
[2] Debian Wheezy, python2.7, 2.7.3rc2-2.1: Fail
[3] python.org, python3.3, 3.3.0a3: Fail
[4] python.org, python3.2, 3.2.3: Fail
I'll compile 64-bit linux from source and try that.
[1] Python 3.2.3rc2 (default, Mar 21 2012, 06:59:51) [GCC 4.6.3] on linux2 [2] Python 2.7.3rc2 (default, Apr 22 2012, 22:35:38) [GCC 4.6.3] on linux2 [3] Python 3.3.0a3 (default, May 1 2012, 16:25:20) [MSC v.1500 32 bit (Intel)] on win32 [4] Python 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] on win32
Author: Hynek Schlawack (hynek) *
Date: 2012-05-13 19:37
I did a little compiling party with official releases and all permutations of Linux, OS X x 3.2.2, 3.2.3 worked. Both ran on 64bit (Linux in a VirtualBox).
Python 3.2.2 (default, May 13 2012, 21:24:38) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information.
slice(0,90000,None).indices(12600000000) (0, 90000, 1)
Python 3.2.2 (default, May 13 2012, 21:33:57) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on darwin Type "help", "copyright", "credits" or "license" for more information.
slice(0,90000,None).indices(12600000000) (0, 90000, 1)
Can we narrow it down to 32bit hosts/OS?
Author: Paul Upchurch (Paul.Upchurch)
Date: 2012-05-13 23:53
The pre-built 64-bit Windows binaries from python.org works.
Python 3.3.0a3 (default, May 1 2012, 16:46:00) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
slice(0,90000,None).indices(12600000000) (0, 90000, 1)
Python 3.2.3 (default, Apr 11 2012, 07:12:16) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
slice(0,90000,None).indices(12600000000) (0, 90000, 1)
I think this issue is settled. There are several possible actions for people that find this discussion through a web search.
- Use Ubuntu 12.04 LTS
- Compile a fresh copy of 3.2 or 3.3 from python.org.
- Use a pre-built 3.2 or 3.3 64-bit Windows binary from python.org.
- Wait for your Linux distribution to catch up.
Hynek, Éric: Thanks for your help.
Author: Ned Deily (ned.deily) *
Date: 2012-05-14 05:05
The problem you described is definitely still an issue with 32-bit builds.
$ /usr/local/bin/python3.3 Python 3.3.0a3 (v3.3.0a3:0b53b70a40a0, May 1 2012, 11:39:35) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
import sys; sys.maxsize 9223372036854775807 slice(0,90000,None).indices(12600000000) (0, 90000, 1) $ /usr/local/bin/python3.3-32 Python 3.3.0a3 (v3.3.0a3:0b53b70a40a0, May 1 2012, 11:39:35) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. import sys; print(sys.maxsize) 2147483647 slice(0,90000,None).indices(12600000000) Traceback (most recent call last): File "", line 1, in OverflowError: cannot fit 'int' into an index-sized integer
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2012-11-02 09:07
This should be an issue on 64-bit too.
slice(0,1,None).indices(sys.maxsize+1)
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-02 17:27
For 2.7, I don't see any problem with raising OverflowError for a length that's > sys.maxsize, since it's hard to have sequences larger than that anyway.
For 3.x, I'd also see this behaviour as reasonable, and not a bug. If it's raising OverflowError for lengths smaller than sys.maxsize, that's a bug. Unless that's the case, I'd call this a feature request for 3.4, rather than a bug that needs fixing in all the maintenance branches.
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-02 17:32
If it's raising OverflowError for lengths smaller than sys.maxsize, that's a bug.
Ah, reading Ned's comment, it looks like that's exactly what it's doing.
Author: Paul Upchurch (Paul.Upchurch)
Date: 2012-11-02 18:23
For the concept of "reasonable", it should be noted that this behaviour will affect code that works with reasonably sized sequences despite the largeness of the parameter.
Consider an extremely large array. To work with such an array, one would typically break it into small segments. However, to simplify the code and reduce bugs it makes sense to use a consistent indexing method on each segment. The size of its parameter does not say anything about the size of a segment. Consider a class which implements virtual arrays.
def getitem(...): ... start,stop,step=slice.indices(start,stop,step).indices(12600000000) while True: if step>0 and start>=stop: break if step<0 and start<=stop: break p=pageid(start) make_page_resident(p) do work ... start=start+step
As you can see, slice.indices should not be limited to sys.maxsize. If Python can perform the arithmetic calculation sys.maxsize+1 then slice.indices(sys.maxsize+1) should also work. The usage of slice.indices is to ensure consistent behaviour of the slicing operator. Another workaround for this bug:
- write your own implementation of slice.indices
I consider this a workaround. The correct way to handle the index parameter to getitem and setitem is to use slice.indices. That way, if the semantics of slicing changes in future versions of Python, your class will behave consistently. It seems to me that this is the main reason why slice.indices exists at all: to prevent inconsistent behaviour when people implement getitem and setitem.
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2012-11-02 18:27
I think the issue is than slice constructor accepts integer out of Py_ssize_t range. And more, it accepts any objects, not only integers or None.
slice(3.4, 'a', {}) slice(3.4, 'a', {})
May be we should disallow creating of such doubtful slices and raise TypeError or OverflowError.
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-02 18:42
Paul: I think you make good arguments that this should be fixed for 3.4. I do however think that for versions earlier than 3.4 this 'fix' would be bordering on a new feature; it's also likely to require significant new code and tests, and so I'd be wary of introducing such a change in a maintenance release. I'd propose to fix this for 3.4 only.
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-02 18:49
I'll look at creating a patch for 3.4
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-03 19:17
Here's a patch.
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-03 19:28
I should note that this patch fixes/changes one other aspect of slice.indices: namely that it used to accept a negative length, and return essentially meaningless results in that case:
>>> slice(0, 10, 1).indices(-3)
(-3, -3, 1)With the patch, it now raises ValueError if given a negative length:
>>> slice(0, 10, 1).indices(-3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: length should not be negativeAuthor: Mark Dickinson (mark.dickinson) *
Date: 2012-11-03 21:42
Updated patch (only cosmetic fixes with respect to the first patch). Thanks Ezio Melotti for comments on #python-dev.
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-03 22:31
Updated for Serhiy's comments on Rietveld:
- fix some refleaks in error cases
- streamline the C code somewhat following Serhiy's suggestions.
Serhiy: you made a comment on the slice_indices function in test_slice.py: "Can we use Python implementation for builtin object?". I don't understand what you mean---can you elaborate?
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2012-11-04 09:55
I don't understand what you mean---can you elaborate?
The Python implementation of this method only 40 lines length, including blank lines, docstring and comments. The C implementation requires over 160 lines and less clear. Are there ways to use in Python interpreter core Python implementation for method of builtin class slice. As already used C implementations for some Python-implemented classes. May be add the file _builtins.py where Python version of some cumbersome methods will be implemented? Then execute the followed code on interpreter initialization:
from _builtins import slice_indices slice.indices = slice_indices del slice_indices
(or an analog in C).
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2012-11-04 10:04
The patch looks good to me. Now benchmarks and special casing for Py_ssize_t values needed.
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-04 10:19
Now benchmarks and special casing for Py_ssize_t values needed.
I thought about that, but I don't think it's worth it. I did some quick timings, and as expected the new version of slice.indices is somewhat slower than the original. But I think adding a special case for Py_ssize_t values would be premature optimization.
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2012-11-04 10:23
Look at compute_slice_indices() in Objects/rangeobject.c.
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-04 10:37
Hmm: one more thing that needs to be fixed before this can be committed---the error messages for maltyped start, stop and step are less informative than they used to be.
Before the patch:
>>> slice(0, 2.3, 4).indices(5)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: slice indices must be integers or None or have an __index__ methodAfter the patch:
>>> slice(0, 2.3, 4).indices(5)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'float' object cannot be interpreted as an integerI'll fix this.
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-04 11:00
New patch that fixes the error message for badly typed slice arguments.
Also tweaks a couple of other details:
replace Py_GE with Py_GT, Py_LE with Py_LT in the out-of-range comparisons, as suggested by Serhiy; this also makes it more closely match the Python reference implementation (since max(x, y) and min(x, y) both return x when x and y are equal)
make sure exception messages match between the Python reference implementation and the C version.
Look at compute_slice_indices() in Objects/rangeobject.c.
Will do. I'm not quite sure I even understand how that code's managing to work at the moment: I see the Py_ssize_t case, but I don't see the fallback code for the case when things are too large for a Py_ssize_t.
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2012-11-04 11:19
compute_slice_indices() and slice_indices() looks as partially duplicates. I think the similar code should be merged and reused.
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-04 11:31
Agreed.
Author: Roundup Robot (python-dev) 
Date: 2012-11-10 14:52
New changeset 9214f8440c44 by Mark Dickinson in branch 'default': Issue #14794: slice.indices no longer returns OverflowError for out-of-range start, stop, step or length. http://hg.python.org/cpython/rev/9214f8440c44
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-10 14:54
Committed the patch. I'll open a new issue for the refactoring. Many thanks to Serhiy Storchaka for the thorough reviews.
Now if only we could fix len, too... :-)
Python 3.4.0a0 (default:f02555353544, Nov 4 2012, 11:50:12) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information.
len(range(10**20)) Traceback (most recent call last): File "", line 1, in OverflowError: Python int too large to convert to C ssize_t [68571 refs]
Author: Mark Dickinson (mark.dickinson) *
Date: 2012-11-17 19:23
For the refactoring, see issue #16451
History
Date
User
Action
Args
2022-04-11 14:57:30
admin
set
github: 58999
2012-11-17 19:23:26
mark.dickinson
set
messages: +
2012-11-10 14:54:39
mark.dickinson
set
status: open -> closed
resolution: fixed
messages: +
2012-11-10 14:52:26
python-dev
set
nosy: + python-dev
messages: +
2012-11-04 11:31:14
mark.dickinson
set
messages: +
2012-11-04 11:19:53
serhiy.storchaka
set
messages: +
2012-11-04 11:00:46
mark.dickinson
set
files: + issue14794_v4.patch
messages: +
2012-11-04 10:37:02
mark.dickinson
set
messages: +
2012-11-04 10:23:48
serhiy.storchaka
set
messages: +
2012-11-04 10:19:24
mark.dickinson
set
messages: +
2012-11-04 10:04:39
serhiy.storchaka
set
messages: +
2012-11-04 09:55:13
serhiy.storchaka
set
messages: +
2012-11-03 22:31:19
mark.dickinson
set
files: + issue14794_v3.patch
messages: +
2012-11-03 21:42:08
mark.dickinson
set
files: + issue14794_v2.patch
messages: +
2012-11-03 19:28:20
mark.dickinson
set
messages: +
2012-11-03 19:17:33
mark.dickinson
set
files: + issue14794.patch
versions: - Python 2.7, Python 3.2, Python 3.3
messages: +
keywords: + patch
type: behavior -> enhancement
stage: needs patch -> patch review
2012-11-02 18:49:03
mark.dickinson
set
assignee: mark.dickinson
messages: +
2012-11-02 18:42:36
mark.dickinson
set
messages: +
2012-11-02 18:27:51
serhiy.storchaka
set
messages: +
2012-11-02 18:23:37
Paul.Upchurch
set
messages: +
2012-11-02 17:32:07
mark.dickinson
set
messages: +
2012-11-02 17:27:18
mark.dickinson
set
messages: +
2012-11-02 13:51:26
mark.dickinson
set
nosy: + mark.dickinson
2012-11-02 09:07:48
serhiy.storchaka
set
nosy: + serhiy.storchaka
messages: +
2012-11-02 07:04:30
hynek
set
versions: + Python 3.4
2012-05-14 05:05:58
ned.deily
set
status: closed -> open
versions: + Python 2.7, Python 3.3
nosy: + ned.deily
messages: +
resolution: out of date -> (no value)
stage: resolved -> needs patch
2012-05-13 23:53:39
Paul.Upchurch
set
messages: +
2012-05-13 19:37:36
hynek
set
messages: +
2012-05-13 17:31:32
Paul.Upchurch
set
messages: +
2012-05-13 05:01:08
eric.araujo
set
nosy: + eric.araujo
messages: +
resolution: fixed -> out of date
stage: resolved
2012-05-12 23:13:24
Paul.Upchurch
set
status: open -> closed
resolution: fixed
messages: +
2012-05-12 22:57:55
hynek
set
nosy: + hynek
messages: +
2012-05-12 22:51:37
Paul.Upchurch
create