| msg246663 - (view) |
Author: Stefan Krah (skrah) *  |
Date: 2015-07-12 19:29 |
| ELLIPSIS and RARROW are missing in EXACT_TOKEN_TYPES. Patch attached. |
|
|
| msg246692 - (view) |
Author: Meador Inge (meador.inge) * |
Date: 2015-07-13 15:31 |
| This looks good to me. A few questions. The patch is fixing an obvious bug, but there could be code that depends on the current broken behavior (the table has been incorrect since Python 3.3). How do we feel about such breaking changes? I am OK with this one. Will you add tests? |
|
|
| msg246693 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2015-07-13 17:04 |
| I think with the ASYNC/AWAIT changes any code using tokenize.py will have to be updated anyway in 3.5 and in 3.7 (when ASYNC/AWAIT will finally be real keywords). So this is probably an ideal time for the change. I'll add tests. |
|
|
| msg246694 - (view) |
Author: Meador Inge (meador.inge) * |
Date: 2015-07-13 17:11 |
| On Mon, Jul 13, 2015 at 12:04 PM, Stefan Krah <report@bugs.python.org> wrote: > I think with the ASYNC/AWAIT changes any code using tokenize.py > will have to be updated anyway in 3.5 and in 3.7 (when ASYNC/AWAIT > will finally be real keywords). Agreed. > So this is probably an ideal time for the change. I'll add tests. Thanks. |
|
|
| msg289568 - (view) |
Author: Vedran Čačić (veky) * |
Date: 2017-03-14 06:05 |
| We obviously missed this, at least for 3.5... any chance of having it for 3.7? |
|
|
| msg289610 - (view) |
Author: Jim Fasarakis-Hilliard (Jim Fasarakis-Hilliard) * |
Date: 2017-03-14 17:17 |
| Stefan, do you have time to make a PR for this? |
|
|
| msg289613 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2017-03-14 17:46 |
| Not right now, but if anyone makes a PR I can click the merge button. :) |
|
|
| msg289615 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2017-03-14 20:19 |
| Thanks for the PR! -- Merged. |
|
|
| msg318855 - (view) |
Author: Aaron Meurer (Aaron.Meurer) |
Date: 2018-06-06 17:33 |
| I would suggest adding this to the what's new document https://docs.python.org/3.7/whatsnew/3.7.html. The change affects user-facing code (the exact_type attribute of TokenInfo is OP for ... and -> tokens prior to this patch). I would also point out that this directly contradicts the documentation (from https://docs.python.org/3/library/tokenize.html, "To simplify token stream handling, all operator and delimiter tokens and Ellipsis are returned using the generic OP token type. The exact type can be determined by checking the exact_type property on the named tuple returned from tokenize.tokenize()."), so I don't see why it can't be backported. |
|
|
| msg318856 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2018-06-06 18:23 |
| I would not mind backporting this. Let's think about it for a while. |
|
|
| msg319087 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2018-06-08 17:35 |
| The 3.6.6rc is next Monday. 3.6.8 will be the last 3.6 maintenance release. |
|
|
| msg319103 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2018-06-08 20:08 |
| Yikes, I wasn't aware that we are that late in the 3.6 release cycle. Perhaps it would be unusual to change it now, even though it is a bug fix. What do others think? |
|
|
| msg319106 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2018-06-08 20:50 |
| > What do others think? +1 for adding a mention in the 3.7 What's New doc (anyone willing to produce a PR for (master Doc/whatsnew/3.7.rst ?) -1 for backporting to 3.6. Since 3.6 has been out there so long and is nearing the end of its maintenance life, it seems to me to not be a good idea to make a change like this that is more likely to break existing code than to fix something that a small number of 3.6 users may have been waiting for. |
|
|
| msg319107 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2018-06-08 20:55 |
| I agree, it would be strange to backport now. |
|
|
| msg375830 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2020-08-23 23:23 |
| Closing, this is out of date. EXACT_TOKEN_TYPES has been moved to token.py in 3.8 and is now automatically generated from Grammar/Tokens. 3.7 is too old to fix. |
|
|