Issue 18374: ast.parse gives wrong position (col_offset) for some BinOp-s (original) (raw)

Issue18374

Created on 2013-07-06 09:35 by Aivar.Annamaa, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
binop_bug.py Aivar.Annamaa,2013-07-06 09:35
18374-binop-col-offset-with-test.diff sam.kimbrel,2014-04-17 15:19 review
binop-offset.patch Carl.Friedrich.Bolz,2019-07-05 19:32
Pull Requests
URL Status Linked Edit
PR 14607 closed Carl.Friedrich.Bolz,2019-07-05 20:46
PR 14653 merged miss-islington,2019-07-08 21:18
PR 14659 merged Carl.Friedrich.Bolz,2019-07-09 09:15
PR 14672 merged miss-islington,2019-07-09 12:20
Messages (14)
msg192394 - (view) Author: Aivar Annamaa (Aivar.Annamaa) * Date: 2013-07-06 09:35
When parsing following program: 1+2-3 ast.parse gives col_offset=3 to outermost BinOp (as shown by the attached test script). Correct col_offset would be 0 for both BinOp-s
msg216696 - (view) Author: Sam Kimbrel (sam.kimbrel) * Date: 2014-04-17 15:19
Here's a patch that corrects col_offset for binops in both the ast module and in the compiler proper. I've incorporated Aivar's test into test_ast.py; if there are test suites for compile.c please let me know and I can add something there too.
msg347294 - (view) Author: Carl Friedrich Bolz-Tereick (Carl.Friedrich.Bolz) * Date: 2019-07-04 20:15
FWIW, in my opinion the col_offsets of the two nodes should be 1 and 3, respectively (the positions of the operators).
msg347313 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-07-05 08:46
I am not sure that changes in compile.c are needed now. Python/Python-ast.c is generated file. And perhaps it would be better to fix CST instead of AST.
msg347319 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-05 09:15
Pablo, Ivan: this old bug is able CST and AST, things that you know. You may want to have a look ;-) Note: This bug is discussed in https://twitter.com/cfbolz/status/1146866203975139328
msg347320 - (view) Author: Aivar Annamaa (Aivar.Annamaa) * Date: 2019-07-05 09:38
> in my opinion the col_offsets of the two nodes should be 1 and 3, respectively (the positions of the operators) This would not match the documentation: > one can get the source segment of a one-line expression node using source_line[node.col_offset : node.end_col_offset]
msg347324 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-07-05 09:51
> in my opinion the col_offsets of the two nodes should be 1 and 3, respectively (the positions of the operators) Such change was committed once, but it was quickly reverted because it was not well discussed and contradicts the documentation.
msg347325 - (view) Author: Carl Friedrich Bolz-Tereick (Carl.Friedrich.Bolz) * Date: 2019-07-05 10:02
OK, fair enough. That means right now there is no way to find the position of the operator using the ast module at the moment, correct?
msg347326 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-07-05 10:11
It is between the end position of the left operand and the start position of the right operand.
msg347370 - (view) Author: Carl Friedrich Bolz-Tereick (Carl.Friedrich.Bolz) * Date: 2019-07-05 19:32
I think the attached patch fixes the problem. Will create a pull request soon.
msg347509 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2019-07-08 21:17
New changeset 110a47c4f42cf4db88edc1876899fff8f05190fb by Ivan Levkivskyi (Carl Friedrich Bolz-Tereick) in branch 'master': bpo-18374: fix wrong col_offset of some ast.BinOp instances (GH-14607) https://github.com/python/cpython/commit/110a47c4f42cf4db88edc1876899fff8f05190fb
msg347510 - (view) Author: miss-islington (miss-islington) Date: 2019-07-08 21:41
New changeset c7be35c2abd598f02a633879133caec356593241 by Miss Islington (bot) in branch '3.8': bpo-18374: fix wrong col_offset of some ast.BinOp instances (GH-14607) https://github.com/python/cpython/commit/c7be35c2abd598f02a633879133caec356593241
msg347554 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2019-07-09 12:20
New changeset 430a9f44fe22f029ae8cfeecb46621d7e199414b by Ivan Levkivskyi (Carl Friedrich Bolz-Tereick) in branch 'master': bpo-18374: fix tests to check the correct thing about line numbers (GH-14659) https://github.com/python/cpython/commit/430a9f44fe22f029ae8cfeecb46621d7e199414b
msg347560 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2019-07-09 13:29
New changeset 68bd9c5691c4899d21cc7fe6cce7cd22b2f5ccb0 by Ivan Levkivskyi (Miss Islington (bot)) in branch '3.8': bpo-18374: fix tests to check the correct thing about line numbers (GH-14659) (GH-14672) https://github.com/python/cpython/commit/68bd9c5691c4899d21cc7fe6cce7cd22b2f5ccb0
History
Date User Action Args
2022-04-11 14:57:47 admin set github: 62574
2019-07-09 19:37:39 levkivskyi set status: open -> closedstage: patch review -> resolvedresolution: fixedversions: + Python 3.8, Python 3.9, - Python 3.5
2019-07-09 13:29:01 levkivskyi set messages: +
2019-07-09 12:39:53 vstinner set nosy: - vstinner
2019-07-09 12:20:13 miss-islington set pull_requests: + <pull%5Frequest14480>
2019-07-09 12:20:10 levkivskyi set messages: +
2019-07-09 09:15:38 Carl.Friedrich.Bolz set pull_requests: + <pull%5Frequest14466>
2019-07-08 21:41:37 miss-islington set nosy: + miss-islingtonmessages: +
2019-07-08 21🔞11 miss-islington set pull_requests: + <pull%5Frequest14460>
2019-07-08 21:17:59 levkivskyi set messages: +
2019-07-05 20:46:32 Carl.Friedrich.Bolz set stage: needs patch -> patch reviewpull_requests: + <pull%5Frequest14423>
2019-07-05 19:32:26 Carl.Friedrich.Bolz set files: + binop-offset.patchmessages: + versions: + Python 3.5, - Python 3.7, Python 3.8, Python 3.9
2019-07-05 10:11:20 serhiy.storchaka set messages: +
2019-07-05 10:02:03 Carl.Friedrich.Bolz set messages: +
2019-07-05 09:51:37 serhiy.storchaka set messages: +
2019-07-05 09:38:36 Aivar.Annamaa set messages: +
2019-07-05 09:15:30 vstinner set nosy: + vstinnermessages: +
2019-07-05 09:14:33 vstinner set nosy: + levkivskyi, pablogsal
2019-07-05 08:46:19 serhiy.storchaka set versions: + Python 3.7, Python 3.8, Python 3.9, - Python 3.5nosy: + serhiy.storchakamessages: + assignee: serhiy.storchakastage: patch review -> needs patch
2019-07-04 20:15:34 Carl.Friedrich.Bolz set nosy: + Carl.Friedrich.Bolzmessages: +
2014-08-22 10:40:14 Claudiu.Popa set stage: patch reviewtype: behaviorversions: + Python 3.5, - Python 3.2, Python 3.3
2014-04-17 15:19:52 sam.kimbrel set files: + 18374-binop-col-offset-with-test.diffnosy: + sam.kimbrelmessages: + keywords: + patch
2013-07-12 22🔞38 terry.reedy set nosy: + benjamin.peterson
2013-07-06 09:35:02 Aivar.Annamaa create