| msg289136 - (view) |
Author: Oren Milman (Oren Milman) * |
Date: 2017-03-06 21:27 |
| ------------ current state ------------ import io class IntLike(): def __init__(self, num): self._num = num def __index__(self): return self._num __int__ = __index__ io.StringIO('blah blah').read(IntLike(2)) io.StringIO('blah blah').readline(IntLike(2)) io.StringIO('blah blah').truncate(IntLike(2)) io.BytesIO(b'blah blah').read(IntLike(2)) io.BytesIO(b'blah blah').readline(IntLike(2)) io.BytesIO(b'blah blah').truncate(IntLike(2)) The three StringIO methods are called without any error, but each of the three BytesIO methods raises a "TypeError: integer argument expected, got 'IntLike'". This is because the functions which implement the StringIO methods (in Modules/_io/stringio.c): - _io_StringIO_read_impl - _io_StringIO_readline_impl - _io_StringIO_truncate_impl use PyNumber_AsSsize_t, which might call nb_index. However, the functions which implement the BytesIO methods (in Modules/_io/bytesio.c): - _io_BytesIO_read_impl - _io_BytesIO_readline_impl - _io_BytesIO_truncate_impl use PyLong_AsSsize_t, which accepts only Python ints (or objects whose type is a subclass of int). ------------ proposed changes ------------ - change those BytesIO methods so that they would accept integer types (i.e. classes that define __index__), mainly by replacing PyLong_AsSsize_t with PyNumber_AsSsize_t - add tests to Lib/test/test_memoryio.py to verify that all six aforementioned methods accept integer types |
|
|
| msg289144 - (view) |
Author: Martin Panter (martin.panter) *  |
Date: 2017-03-06 23:10 |
| What is the use case? Unless changing the behaviour would be useful, I think the simplest solution would be to document that the methods should only be given instances of “int”, so that it is clear that other kinds of numbers are unsupported. |
|
|
| msg289148 - (view) |
Author: Oren Milman (Oren Milman) * |
Date: 2017-03-07 01:13 |
| I don't have a use case for that. (I noticed this behavior by chance, while working on some other issue.) However, IIUC, commit 4fa88fa0ba35e25ad9be66ebbdaba9aca553dc8b, by Benjamin Peterson, Antoine Pitrou and Amaury Forgeot d'Arc, includes patching the aforementioned StringIO functions, so that they would accept integer types. So I add them to the nosy list, as I guess that the use cases for accepting integer types in StringIO methods are also use cases for accepting integer types in BytesIO. And now that you mention the docs, according to them, both StringIO and BytesIO inherit these methods from BufferedIOBase or IOBase. Thus, the methods are already expected to behave the same, aren't they? |
|
|
| msg289151 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2017-03-07 06:18 |
| Other objects in the io module use special-purposed converter _PyIO_ConvertSsize_t() which checks PyNumber_Check() and calls PyNumber_AsSsize_t(). I think StringIO implementation can be changed to reuse _PyIO_ConvertSsize_t() for simplicity. After that BytesIO implementation can be changed to use the same converter just for uniformity. |
|
|
| msg289154 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2017-03-07 06:32 |
| Are there tests for accepting non-integers in other classes? If no then don't bother about tests. I suppose all this came from the implementation of the 'n' format unit in PyArg_Parse* functions. |
|
|
| msg289158 - (view) |
Author: Oren Milman (Oren Milman) * |
Date: 2017-03-07 10:43 |
| using _PyIO_ConvertSsize_t sounds great. with regard to tests for accepting integer types in other classes, there aren't a lot of them, so I guess it is not always tested. still, it is tested in (excluding tests of the actual feature of treating integer types as ints): test_bytes, test_cmath, test_int, test_math, test_range, test_re, test_slice, test_struct, test_unicode and test_weakref. |
|
|
| msg289160 - (view) |
Author: Oren Milman (Oren Milman) * |
Date: 2017-03-07 11:32 |
| also with regard to adding tests, consider the following scenario: in the future, someone writes a patch for these functions, which makes them not accept integer types anymore. assuming the tests don't exist, the writer of the patch doesn't realize the patch broke something, and so the patch is committed. years later, when the patch is finally a part of the stable release, it breaks a lot of code out there. lastly, ISTM adding these tests would be quite simple anyway. |
|
|
| msg289225 - (view) |
Author: Oren Milman (Oren Milman) * |
Date: 2017-03-08 10:05 |
| I wrote a patch, but I attach it here and not in a PR, as you haven't approved adding tests. (I would be happy to open a PR with or without the tests.) I ran the test module (on my Windows 10), and seems like the patch doesn't break anything. also, while running test_memoryio with my added tests, i noticed some places in Lib/_pyio.py which seemed like they should be changed. in particular, I changed 'var.__index__' to 'var = var.__index__()' in some places. I feel really uncomfortable about that change, as it undos a change committed by Florent Xicluna in b14930cd93e74cae3b7370262c6dcc7c28e0e712. Florent, what was the reason for that change? |
|
|
| msg289226 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2017-03-08 10:24 |
| LGTM. |
|
|
| msg289227 - (view) |
Author: Oren Milman (Oren Milman) * |
Date: 2017-03-08 10:31 |
| should I open a PR (even though we didn't give Florent enough time to respond)? |
|
|
| msg289228 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2017-03-08 10:37 |
| Open a PR. It will be not hard to make small changes after opening it. |
|
|
| msg289229 - (view) |
Author: Oren Milman (Oren Milman) * |
Date: 2017-03-08 10:40 |
| sure. In general, should drafts (like this one) be uploaded here? or is it always better to open a PR? |
|
|
| msg289230 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2017-03-08 10:58 |
| This for your preference. |
|
|
| msg289359 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2017-03-10 13:07 |
| I would accept changes to Modules/_io/ because I consider them as code cleanup (with little side effect). But I'm not sure about changes to Python implementation and tests. Could you create more narrow PR for Modules/_io/ changes and left other changes for the consideration of the io module maintainers? |
|
|
| msg289414 - (view) |
Author: Oren Milman (Oren Milman) * |
Date: 2017-03-10 22:33 |
| done |
|
|
| msg289416 - (view) |
Author: Oren Milman (Oren Milman) * |
Date: 2017-03-10 22:55 |
| thanks :) I would update the original PR soon. |
|
|
| msg289418 - (view) |
Author: Oren Milman (Oren Milman) * |
Date: 2017-03-10 23:05 |
| or maybe git is smart enough to realize what happened, and I don't have to update PR 560? |
|
|
| msg290235 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2017-03-24 22:28 |
| New changeset 740025478dcd0e9e4028507f32375c85f849fb07 by Serhiy Storchaka (orenmn) in branch 'master': bpo-29741: Clean up C implementations of BytesIO and StringIO. (#606) https://github.com/python/cpython/commit/740025478dcd0e9e4028507f32375c85f849fb07 |
|
|
| msg300796 - (view) |
Author: Steve Dower (steve.dower) * |
Date: 2017-08-24 18:33 |
| New changeset de50360ac2fec81dbf733f6c3c739b39a8822a39 by Steve Dower (Oren Milman) in branch 'master': bpo-29741: Update some methods in the _pyio module to also accept integer types. Patch by Oren Milman. (#560) https://github.com/python/cpython/commit/de50360ac2fec81dbf733f6c3c739b39a8822a39 |
|
|