Issue 25580: async and await missing from token list (original) (raw)
Created on 2015-11-07 16:02 by SilentGhost, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (15)
Author: SilentGhost (SilentGhost) * 
Date: 2015-11-07 16:02
With introduction of async and await tokens in 3.5 the token documentation needs updating.
Author: Stéphane Wirtel (matrixise) * 
Date: 2015-11-07 16:06
With python 3.5, async is a token with the ASYNC type.
tokens = tokenize.generate_tokens(io.StringIO('async def foo').readline) pprint.pprint(list(tokens)) [TokenInfo(type=55 (ASYNC), string='async', start=(1, 0), end=(1, 5), line='async def foo'), TokenInfo(type=1 (NAME), string='def', start=(1, 6), end=(1, 9), line='async def foo'), TokenInfo(type=1 (NAME), string='foo', start=(1, 10), end=(1, 13), line='async def foo'), TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]
Author: Martin Panter (martin.panter) *
Date: 2015-11-08 11:29
I wonder if the new tokens need a “versionadded” notice. They were added in revision eeeb666a5365 for 3.5.
Author: SilentGhost (SilentGhost) *
Date: 2015-11-12 18:04
I've added versionchanged, since this is what's done in os, for similar lists.
Author: Martin Panter (martin.panter) *
Date: 2015-11-12 21:34
Do you have a new version of the patch?
Author: Yury Selivanov (yselivanov) *
Date: 2015-11-13 01:03
ASYNC/AWAIT tokens are temporary and will be removed in 3.7
Author: SilentGhost (SilentGhost) *
Date: 2015-11-13 07:10
Hm, not sure why the file didn't get uploaded, I clearly remember attaching it.
Yury, what is the implication for this issue? Are you suggesting that the note should state that these two are temporary? Or is this information available somewhere else, in What's new, for example?
Author: Yury Selivanov (yselivanov) *
Date: 2015-11-13 18:15
Are you suggesting that the note should state that these two are temporary?
Yes, that would be great.
Or is this information available somewhere else, in What's new, for example?
Not directly, I think. It was detailed to some extent in PEP 492. In what's new of 3.5 & 3.6 it's stated that async/await will be keywords in 3.7.
Author: SilentGhost (SilentGhost) *
Date: 2015-11-13 18:36
Here is the updated wording. I'm not familiar with other such cases in the stdlib, if any one knows of another similar case, I'd be glad to correct the language to match previous usage.
Author: Martin Panter (martin.panter) *
Date: 2015-11-13 22:58
Will they really be removed? What are the compatibility implications? Maybe it would be better to alias them to NAME, or keep them as unused values.
Author: Martin Panter (martin.panter) *
Date: 2015-11-13 23:04
My only background here is via the tokenize module, which seems to currently behave as described in <https://www.python.org/dev/peps/pep-0492/#transition-plan>: async and await are either NAME tokens (like ordinary identifiers and other reserved keywords) outside a coroutine definition, but they become special ASYNC and AWAIT tokens inside a coroutine.
Author: Yury Selivanov (yselivanov) *
Date: 2015-12-17 23:19
My only background here is via the tokenize module, which seems to currently behave as described in <https://www.python.org/dev/peps/pep-0492/#transition-plan>: async and await are either NAME tokens (like ordinary identifiers and other reserved keywords) outside a coroutine definition, but they become special ASYNC and AWAIT tokens inside a coroutine.
Exactly. And in 3.7 they will be NAME tokens in any context. That's why I'm reluctant to document them atm.
Author: Yury Selivanov (yselivanov) *
Date: 2015-12-17 23:26
Ah, I see that issue25580_2.diff does exactly what I proposed. I'm committing it.
Author: Roundup Robot (python-dev)
Date: 2015-12-17 23:27
New changeset aa79b2a5b2e1 by Yury Selivanov in branch '3.5': docs: Document ASYNC/AWAIT tokens (issue #25580) https://hg.python.org/cpython/rev/aa79b2a5b2e1
Author: Yury Selivanov (yselivanov) *
Date: 2015-12-17 23:27
Thanks for the patch!
History
Date
User
Action
Args
2022-04-11 14:58:23
admin
set
github: 69766
2015-12-17 23:27:28
yselivanov
set
status: open -> closed
type: enhancement
messages: +
resolution: fixed
stage: patch review -> resolved
2015-12-17 23:27:04
python-dev
set
nosy: + python-dev
messages: +
2015-12-17 23:26:11
yselivanov
set
keywords: + easy
messages: +
2015-12-17 23:19:03
yselivanov
set
messages: +
2015-11-13 23:04:25
martin.panter
set
messages: +
2015-11-13 22:58:22
martin.panter
set
messages: +
2015-11-13 18:36:49
SilentGhost
set
files: + issue25580_2.diff
messages: +
2015-11-13 18:15:35
yselivanov
set
messages: +
2015-11-13 07:10:18
SilentGhost
set
files: + issue25580.diff
messages: +
2015-11-13 01:03:04
yselivanov
set
messages: +
2015-11-12 21:34:49
martin.panter
set
messages: +
2015-11-12 18:04:06
SilentGhost
set
messages: +
2015-11-08 11:29:29
martin.panter
set
nosy: + yselivanov, martin.panter
messages: +
2015-11-07 16:06:22
matrixise
set
nosy: + matrixise
messages: +
2015-11-07 16:02:19
SilentGhost
create