Issue 29869: Underscores in numeric literals not supported in lib2to3. (original) (raw)

Created on 2017-03-22 00:12 by nevsan, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 752 merged nevsan,2017-03-22 00:13
PR 1109 merged Mariatta,2017-04-13 10:25
PR 1119 merged nevsan,2017-04-13 17:19
PR 1122 merged Mariatta,2017-04-13 23:17
PR 1125 merged Mariatta,2017-04-14 01:25
Messages (14)
msg289951 - (view) Author: Nevada Sanchez (nevsan) * Date: 2017-03-22 00:12
The following should work in Python 3.6 ``` from lib2to3.pgen2 import driver from lib2to3 import pytree from lib2to3 import pygram _GRAMMAR_FOR_PY3 = pygram.python_grammar_no_print_statement.copy() parser_driver = driver.Driver(_GRAMMAR_FOR_PY3, convert=pytree.convert) tree = parser_driver.parse_string('100_1\n', debug=False) ```
msg289982 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-22 11:56
Python uses more strong rules for underscores in numerical literals. Underscores are acceptable between digits and between the base prefix and digit. For example the regular expression for hexadecimals is r'0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?'. Underscores also are acceptable in the exponent of float literals: Exponent = r'[eE][-+]?\d+(?:_\d+)*'
msg289988 - (view) Author: Nevada Sanchez (nevsan) * Date: 2017-03-22 14:56
The existing regular expressions weren't actually strict enough as is, so I made them even more correct. In particular, we must have at least one digit following `0[xXbBoO]` and must be before any underscores. I have a small set of test cases to examine correctness of these regular expressions: https://gist.github.com/nevsan/7fc78dc61d309842406d67d6839b9861
msg290001 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2017-03-22 17:15
> In particular, we must have at least one digit following `0[xXbBoO]` and must be before any underscores. This is not true (but your test file does the right thing).
msg290007 - (view) Author: Nevada Sanchez (nevsan) * Date: 2017-03-22 18:41
Thanks, it seems I misspoke. Glad I tested it!
msg290008 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-22 18:57
I suggest to use my regular expression for haxadedecimals. Your regular expression starves from catastrophic backtracking. Compare two examples: re.match(r'0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?'+r'\b', '0x'+'0'*100+'z') re.match(r'0[xX](?:_?[\da-fA-F]+)+[lL]?'+r'\b', '0x'+'0'*100+'z')
msg290023 - (view) Author: Nevada Sanchez (nevsan) * Date: 2017-03-23 04:17
Good point. Updated.
msg291598 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-04-13 10:21
New changeset 97a40b7a5b2979fb17e1751c139fd4ba1ebd5276 by Mariatta (Nevada Sanchez) in branch '3.6': bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-752) https://github.com/python/cpython/commit/97a40b7a5b2979fb17e1751c139fd4ba1ebd5276
msg291600 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-04-13 11:03
New changeset 84c2d75489a84174d8993aea292828662e35a50f by Mariatta in branch '3.6': Revert "bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-752)" (GH-1109) https://github.com/python/cpython/commit/84c2d75489a84174d8993aea292828662e35a50f
msg291618 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-04-13 15:31
Nevada Sanchez, please create your PR against the master branch. Once that is merged, we will backport it to the 3.6 branch. Thanks :)
msg291623 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-04-13 17:32
New changeset a6e395dffadf8c5124903c01ad69fefa36b1a935 by Mariatta (Nevada Sanchez) in branch 'master': bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-1119) https://github.com/python/cpython/commit/a6e395dffadf8c5124903c01ad69fefa36b1a935
msg291634 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-04-13 23:54
New changeset 2cdf087d1fd48f7d0f95b5a0b31b9a624fa84751 by Mariatta in branch '3.6': [3.6] bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-1119) (GH-1122) https://github.com/python/cpython/commit/2cdf087d1fd48f7d0f95b5a0b31b9a624fa84751
msg291635 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-04-14 01:30
New changeset 947629916a5ecb1f6f6792e9b9234e084c5bf274 by Mariatta in branch 'master': bpo-29869: Add Nevada Sanchez to Misc/ACKS (GH-1125) https://github.com/python/cpython/commit/947629916a5ecb1f6f6792e9b9234e084c5bf274
msg291636 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-04-14 01:31
PR has been merged and backported to 3.6. I also added Nevada Sanchez to Misc/ACKS. Thanks all :)
History
Date User Action Args
2022-04-11 14:58:44 admin set github: 74055
2017-04-14 01:31:24 Mariatta set status: open -> closedstage: patch review -> resolved
2017-04-14 01:31:11 Mariatta set resolution: fixedmessages: +
2017-04-14 01:30:44 Mariatta set messages: +
2017-04-14 01:25:41 Mariatta set pull_requests: + <pull%5Frequest1260>
2017-04-13 23:54:51 Mariatta set messages: +
2017-04-13 23:17:39 Mariatta set pull_requests: + <pull%5Frequest1259>
2017-04-13 17:32:56 Mariatta set messages: +
2017-04-13 17:19:11 nevsan set pull_requests: + <pull%5Frequest1257>
2017-04-13 15:31:17 Mariatta set messages: +
2017-04-13 11:03:18 Mariatta set messages: +
2017-04-13 10:25:02 Mariatta set pull_requests: + <pull%5Frequest1250>
2017-04-13 10:21:07 Mariatta set nosy: + Mariattamessages: +
2017-03-23 04:17:38 nevsan set messages: +
2017-03-22 18:57:03 serhiy.storchaka set messages: +
2017-03-22 18:41:49 nevsan set messages: +
2017-03-22 17:15:17 georg.brandl set messages: +
2017-03-22 14:56:58 nevsan set messages: +
2017-03-22 11:56:56 serhiy.storchaka set nosy: + georg.brandl, serhiy.storchakamessages: +
2017-03-22 02:36:42 Mariatta set stage: patch reviewversions: + Python 3.7
2017-03-22 00:13:36 nevsan set pull_requests: + <pull%5Frequest668>
2017-03-22 00:12:25 nevsan create