Issue 35766: Merge typed_ast back into CPython (original) (raw)

Created on 2019-01-18 00:33 by gvanrossum, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11645 closed gvanrossum,2019-01-22 02:26
PR 11645 closed gvanrossum,2019-01-22 02:26
PR 11645 closed gvanrossum,2019-01-22 02:26
PR 11714 merged gvanrossum,2019-01-31 17:05
PR 11714 merged gvanrossum,2019-01-31 17:05
PR 11714 merged gvanrossum,2019-01-31 17:05
PR 11766 merged gvanrossum,2019-02-06 00:36
PR 11766 merged gvanrossum,2019-02-06 00:36
PR 11766 merged gvanrossum,2019-02-06 00:36
PR 13984 merged gvanrossum,2019-06-11 18:54
PR 13987 merged miss-islington,2019-06-11 20:42
PR 13992 merged gvanrossum,2019-06-11 23:41
PR 13993 merged miss-islington,2019-06-12 00:23
PR 13994 merged vstinner,2019-06-12 00:34
PR 13995 closed miss-islington,2019-06-12 00:52
PR 14001 merged vstinner,2019-06-12 02:45
Messages (28)
msg333919 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-01-18 00:33
(This started at https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377. It's somewhat related to https://bugs.python.org/issue33337.) I now have a thorough understanding of what typed_ast does, and I think it would be straightforward to port it upstream. We’d need to define two new tokens to represent `# type: ignore` and `# type: `, and tokenizer code to recognize these. Then we need a new flag to be passed to the tokenizer (via the parser) that enables this behavior. We make a small number of changes to `Grammar` (inserting optional `TYPE_COMMENT` tokens and to `Python.asdl` (adding fields to a few node types to hold the optional type comment), and a fair number of changes to `ast.c` to extract the type comments. We have similar patches for 3.6 and 3.7, so it's a simple matter of porting those patches to 3.8. By default, `ast.parse()` should not return type comments, since this would reject some perfectly good Python code (with sonething looking like a type comment in a place where the grammar doesn’t allow it). But passing an new flag will cause the tokenizer to process type comments and the returned tree will contain them. I could produce a PR with this in a few days (having just gone over most of the process for porting typed_ast from 3.6 to 3.7). There’s one more feature I’d like to lobby for – a feature_version flag that modifies the grammar slightly so it resembles an older version of Python (going back to 3.4). This is used in mypy to decouple the Python version you’re running from the Python version for which you’re checking compatibility (useful when checking code that will be deployed on a system with a different Python version installed). I imagine this would be useful to other linters as well, and the implementation is mostly manipulating whether `async` and `await` are keywords. But if there’s pushback to this part I can live without it – the rest of the work is still useful. In the next few days I will produce a PR so people can see for themselves. In https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/17, Łukasz offered to merge my PR.
msg334068 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2019-01-19 21:15
Seems also related to https://bugs.python.org/issue24119 with python2 / python3.5 (hopefully) rapidly falling off in usage I would assume the specialized treatment of `# type: ...` comments would become less and less necessary Though I guess there's still `# type: ignore`, though if I recall correctly ignores are bubbled up to the module level and are applied on a line-by-line basis and wouldn't necessarily need specialized AST treatment (though a second parse over the token stream is a bit unfortunate)
msg334071 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-01-19 21:57
You’d be surprised how tenacious old versions are. Anywa, I am working on this on my copious spare time — you can follow my progress at https://github.com/gvanrossum/cpython/tree/ast-type-comments?files=1 .
msg334073 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2019-01-19 22:17
> You’d be surprised how tenacious old versions are. heh that's true, at my last job we finally got rid of python2.6 in 2016 :'( anyway -- I don't mean to discourage this, definitely seems valuable to the maintenance of mypy!
msg334371 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-01-25 22:26
The PR is ready for reviews now.
msg334500 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-01-29 03:10
Also the tests now all pass; for now I am happy with the solution I found for the indentation error (see https://github.com/python/cpython/pull/11645#issuecomment-456627216).
msg334624 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-01-31 11:40
New changeset dcfcd146f8e6fc5c2fc16a4c192a0c5f5ca8c53c by Łukasz Langa (Guido van Rossum) in branch 'master': bpo-35766: Merge typed_ast back into CPython (GH-11645) https://github.com/python/cpython/commit/dcfcd146f8e6fc5c2fc16a4c192a0c5f5ca8c53c
msg334714 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-02-01 19:37
New changeset 3a32e3bf880f0842ac73d18285f5a1caa902a2ca by Guido van Rossum in branch 'master': bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod (#11714) https://github.com/python/cpython/commit/3a32e3bf880f0842ac73d18285f5a1caa902a2ca
msg335234 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-02-11 16:10
New changeset 4b250fc1da9c893803cf724a4974450b5e10bd8a by Guido van Rossum in branch 'master': bpo-35766 follow-up: Add an error check to new_type_comment() (#11766) https://github.com/python/cpython/commit/4b250fc1da9c893803cf724a4974450b5e10bd8a
msg335276 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-02-12 00:11
I have some follow-up wishes but I'll create a new issue.
msg335278 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-02-12 00:23
See https://bugs.python.org/issue35975
msg345245 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-11 15:51
I don't see anything in What's New in Python 3.8?: https://docs.python.org/dev/whatsnew/3.8.html Would it be possible to document the change somewhere?
msg345248 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2019-06-11 16:06
> Would it be possible to document the change somewhere? Most of these things are in the docs for ast module. But indeed none of static typing related features have been added to What's New. I have an item on mypy todo list to add four typing PEPs plus new AST features (also adding `end_lineno` and `end_col_offset`) also there, but didn't have time last weekend.
msg345249 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2019-06-11 16:07
Typo: "mypy todo list" should be "my todo list" :-)
msg345256 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-06-11 17:54
I'll create a PR that updates the What's New docs with news about all of these.
msg345264 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-11 20:10
> ``feature_version=N`` allows specifying the minor version of an earlier Python 3 version. (For example, ``feature_version=4`` will treat ``async`` and ``await`` as non-reserved words.) What will happen with Python 4? Why not using "full" Python version like (3, 4) or 0x304?
msg345265 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-06-11 20:15
> What will happen with Python 4? Why not using "full" Python version like (3, 4) or 0x304? I don't think Python 4 is just around the corner, so I prefer to kick that problem down the road. I'm sure we can come up with a sufficiently backwards-compatible API. (I'd prefer not to go the hex route, because printing the value would show as 52, which is pretty meaningless.)
msg345267 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-06-11 20:42
New changeset 9b33ce48a7846dbdad32d4f8936b08e6b78a2faf by Guido van Rossum in branch 'master': bpo-35766: What's new in the ast and typing modules (#13984) https://github.com/python/cpython/commit/9b33ce48a7846dbdad32d4f8936b08e6b78a2faf
msg345270 - (view) Author: miss-islington (miss-islington) Date: 2019-06-11 21:13
New changeset 785688832de699270530a9418e99c5c4caedbffb by Miss Islington (bot) in branch '3.8': bpo-35766: What's new in the ast and typing modules (GH-13984) https://github.com/python/cpython/commit/785688832de699270530a9418e99c5c4caedbffb
msg345273 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-11 21:49
> I'm sure we can come up with a sufficiently backwards-compatible API. (I'd prefer not to go the hex route, because printing the value would show as 52, which is pretty meaningless.) I propose to replace 4 with (3, 4) to be more future-proof. This version format is already used by sys.version_info for example.
msg345285 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-06-11 23:09
But this is the format currently used by typed_ast. I think it would just cause a cascade of annoying failures for tools that switch between typed_ast and the 3.8 version of ast. Such as mypy.
msg345286 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-11 23:14
> But this is the format currently used by typed_ast. I think it would just cause a cascade of annoying failures for tools that switch between typed_ast and the 3.8 version of ast. Such as mypy. typed_ast is a 3rd party project. But here we are talking about Python stdlib. If a project moves from typed_ast to stdlib ast, IMHO it should be quite easy to replace 4 with (3, 4), especially if 4 raises an error, no? IMHO it's a good opportunity to fix it. I would prefer to not have to batch many backward incompatible changes into Python 4...
msg345292 - (view) Author: miss-islington (miss-islington) Date: 2019-06-12 00:23
New changeset 10b55c1643b512b3a2cae8ab89c53683a13ca43e by Miss Islington (bot) (Guido van Rossum) in branch 'master': bpo-35766: Change format for feature_version to (major, minor) (GH-13992) https://github.com/python/cpython/commit/10b55c1643b512b3a2cae8ab89c53683a13ca43e
msg345293 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-12 00:52
New changeset efdf6ca90f7702824e7aeee1ceca949e7c20288a by Victor Stinner in branch 'master': bpo-35766: compile(): rename feature_version parameter (GH-13994) https://github.com/python/cpython/commit/efdf6ca90f7702824e7aeee1ceca949e7c20288a
msg345294 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-12 00:55
New changeset 3ba21070c6ecab83c23cea41a92b42fa651c7ea2 by Victor Stinner (Miss Islington (bot)) in branch '3.8': bpo-35766: Change format for feature_version to (major, minor) (GH-13992) (GH-13993) https://github.com/python/cpython/commit/3ba21070c6ecab83c23cea41a92b42fa651c7ea2
msg345364 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-12 14:17
New changeset b2fd32b2f041402bb9fc8607f01566c055aafed9 by Victor Stinner in branch '3.8': bpo-35766: compile(): rename feature_version parameter (GH-13994) (GH-14001) https://github.com/python/cpython/commit/b2fd32b2f041402bb9fc8607f01566c055aafed9
msg345366 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-06-12 14:30
Thanks for taking care of this! -- --Guido (mobile)
msg345372 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-12 15:34
I created bpo-37253: "PyCompilerFlags got a new cf_feature_version field".
History
Date User Action Args
2022-04-11 14:59:10 admin set github: 79947
2019-06-12 15:34:26 vstinner set keywords:patch, patch, patchmessages: +
2019-06-12 14:30:21 gvanrossum set messages: +
2019-06-12 14:17:16 vstinner set messages: +
2019-06-12 02:45:39 vstinner set pull_requests: + <pull%5Frequest13864>
2019-06-12 00:55:31 vstinner set messages: +
2019-06-12 00:52:27 miss-islington set pull_requests: + <pull%5Frequest13858>
2019-06-12 00:52:24 vstinner set messages: +
2019-06-12 00:34:58 vstinner set pull_requests: + <pull%5Frequest13857>
2019-06-12 00:23:27 miss-islington set pull_requests: + <pull%5Frequest13856>
2019-06-12 00:23:17 miss-islington set messages: +
2019-06-11 23:41:52 gvanrossum set pull_requests: + <pull%5Frequest13855>
2019-06-11 23:14:26 vstinner set keywords:patch, patch, patchmessages: +
2019-06-11 23:09:14 gvanrossum set keywords:patch, patch, patchmessages: +
2019-06-11 21:49:33 vstinner set keywords:patch, patch, patchmessages: +
2019-06-11 21:13:35 miss-islington set nosy: + miss-islingtonmessages: +
2019-06-11 20:42:51 gvanrossum set messages: +
2019-06-11 20:42:47 miss-islington set pull_requests: + <pull%5Frequest13850>
2019-06-11 20:15:28 gvanrossum set keywords:patch, patch, patchmessages: +
2019-06-11 20:10:30 vstinner set keywords:patch, patch, patchmessages: +
2019-06-11 18:54:17 gvanrossum set pull_requests: + <pull%5Frequest13848>
2019-06-11 17:54:15 gvanrossum set keywords:patch, patch, patchmessages: +
2019-06-11 16:07:54 levkivskyi set keywords:patch, patch, patchmessages: +
2019-06-11 16:06:46 levkivskyi set keywords:patch, patch, patchmessages: +
2019-06-11 15:51:43 vstinner set keywords:patch, patch, patchnosy: + vstinnermessages: +
2019-02-12 00:23:54 gvanrossum set keywords:patch, patch, patchmessages: +
2019-02-12 00:11:24 gvanrossum set status: open -> closedmessages: + keywords:patch, patch, patchresolution: fixedstage: patch review -> resolved
2019-02-11 16:10:47 gvanrossum set messages: +
2019-02-06 00:36:39 gvanrossum set pull_requests: + <pull%5Frequest11725>
2019-02-06 00:36:28 gvanrossum set pull_requests: + <pull%5Frequest11724>
2019-02-06 00:36:15 gvanrossum set pull_requests: + <pull%5Frequest11723>
2019-02-01 19:37:48 gvanrossum set messages: +
2019-01-31 17:06:03 gvanrossum set pull_requests: + <pull%5Frequest11575>
2019-01-31 17:05:49 gvanrossum set pull_requests: + <pull%5Frequest11574>
2019-01-31 17:05:35 gvanrossum set pull_requests: + <pull%5Frequest11573>
2019-01-31 11:40:34 lukasz.langa set nosy: + lukasz.langamessages: +
2019-01-29 03:10:17 gvanrossum set keywords:patch, patch, patchmessages: +
2019-01-25 22:26:54 gvanrossum set keywords:patch, patch, patchmessages: +
2019-01-23 05:55:42 ethan smith set nosy: + ethan smith
2019-01-22 02:26:21 gvanrossum set keywords: + patchstage: needs patch -> patch reviewpull_requests: + <pull%5Frequest11429>
2019-01-22 02:26:14 gvanrossum set keywords: + patchstage: needs patch -> needs patchpull_requests: + <pull%5Frequest11428>
2019-01-22 02:26:07 gvanrossum set keywords: + patchstage: needs patch -> needs patchpull_requests: + <pull%5Frequest11427>
2019-01-21 06:06:21 gregory.p.smith set nosy: + gregory.p.smith
2019-01-19 22:17:50 Anthony Sottile set messages: +
2019-01-19 21:57:16 gvanrossum set messages: +
2019-01-19 21:15:42 Anthony Sottile set messages: +
2019-01-19 21:12:08 Anthony Sottile set nosy: + Anthony Sottile
2019-01-19 12:11:17 levkivskyi set nosy: + levkivskyi
2019-01-18 00:33:01 gvanrossum create