| msg240322 - (view) |
Author: Eli Bendersky (eli.bendersky) *  |
Date: 2015-04-09 13:18 |
| lib2to3 tokenizes br'abc' as a single STRING token, but rb'abc' as two separate tokens (NAME "rb" and STRING 'abc') This is because pgen2/tokenize.py doesn't list rb'' as a viable prefix for a string, even though according to https://docs.python.org/3/reference/lexical_analysis.html it is |
|
|
| msg240444 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2015-04-10 18:23 |
| Should it? >>> rb'abc' File "", line 1 rb'abc' ^ SyntaxError: invalid syntax According to https://docs.python.org/2/reference/lexical_analysis.html#string-literals "rb" is not valid string prefix. |
|
|
| msg240445 - (view) |
Author: Eli Bendersky (eli.bendersky) * |
Date: 2015-04-10 18:44 |
| Serhiy, AFAIK lib2to3 has been repurposed to parse Python 3 as well - it has certainly gained quite a bit of support for that. rb'...' is valid in Python 3: $ python3 Python 3.4.0 (default, Apr 11 2014, 13:05:11) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> rb'abc' b'abc' >>> |
|
|
| msg256831 - (view) |
Author: Joseph Gordon (josephgordon) |
Date: 2015-12-22 08:02 |
| With the changes in the uploaded patch it looks like strings of the form rb'...' are now recognized as raw byte strings. |
|
|
| msg289343 - (view) |
Author: Łukasz Langa (lukasz.langa) * |
Date: 2017-03-10 08:29 |
| I don't think we'll fix it for 3.5 anymore but it's definitely worth doing so for 3.6.1. Ned, I have a patch, will upload momentarily. I'd like this to go in 3.6.1 since without it fixed tools depending on lib2to3 will crash seeing f-strings. This includes YAPF, a pretty widely used auto-formatter for Python code. The fix is relatively trivial. What do you think? |
|
|
| msg289415 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2017-03-10 22:44 |
| Sorry, the announced deadline for bugfixes for 3.6.1 has already passed; at this point, only showstopper problems with 3.6.1rc1 are on the table for the final. Trivial impact isn't a criterion for post-rc1 changes and there's a good reason for that: we ask everyone in the community to test a release candidate with the expectation that this is the final release. I don't think it's fair to add more unrelated changes and risk that it will break something and/or cause additional release candidates to be produced. The original problem has been open for nearly two years now and, while "f" string support adds to the importance of the tokenizer getting updated, this can wait 3 more months for 3.6.2 with a proper review cycle and with tests. |
|
|
| msg289423 - (view) |
Author: Łukasz Langa (lukasz.langa) * |
Date: 2017-03-11 00:08 |
| Alright, we'll merge this for 3.6.2 then! |
|
|
| msg291321 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2017-04-08 06:38 |
| Should this issue be closed now? |
|
|
| msg291572 - (view) |
Author: Łukasz Langa (lukasz.langa) * |
Date: 2017-04-12 20:58 |
| No Serhiy, I need a new patch with tests, etc. |
|
|
| msg294196 - (view) |
Author: Łukasz Langa (lukasz.langa) * |
Date: 2017-05-22 22:19 |
| New changeset 0c4aca54dcf0c54f299c78aa71fe8f48ff04f9d9 by Łukasz Langa in branch 'master': Make rb'' strings work in lib2to3 (#1724) https://github.com/python/cpython/commit/0c4aca54dcf0c54f299c78aa71fe8f48ff04f9d9 |
|
|
| msg294202 - (view) |
Author: Łukasz Langa (lukasz.langa) * |
Date: 2017-05-22 23:35 |
| New changeset 1398b1bc7d80df5bde17041e7ec0a3bdbf54b19e by Łukasz Langa in branch '3.6': [3.6] Make rb'' strings work in lib2to3 (GH-1724) (#1730) https://github.com/python/cpython/commit/1398b1bc7d80df5bde17041e7ec0a3bdbf54b19e |
|
|
| msg294203 - (view) |
Author: Łukasz Langa (lukasz.langa) * |
Date: 2017-05-22 23:35 |
| New changeset 1b9530c536664276ce866ae602ce04adce0810e1 by Łukasz Langa in branch 'master': bpo-23894: make lib2to3 recognize f-strings (#1733) https://github.com/python/cpython/commit/1b9530c536664276ce866ae602ce04adce0810e1 |
|
|
| msg294223 - (view) |
Author: Łukasz Langa (lukasz.langa) * |
Date: 2017-05-23 05:23 |
| New changeset e8412e684ed741df246e8430f4911b31b0d8be1f by Łukasz Langa in branch '3.6': [3.6] bpo-23894: make lib2to3 recognize f-strings (GH-1733) (#1737) https://github.com/python/cpython/commit/e8412e684ed741df246e8430f4911b31b0d8be1f |
|
|
| msg295836 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2017-06-13 02:00 |
| Misc/NEWS entries for these changes? |
|
|
| msg296227 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2017-06-17 02:30 |
| New changeset b7c59f0d05bde0d4dd2d7bffc1b8af72c25d3f71 by Ned Deily in branch '3.6': bpo-23894: add Misc/NEWS entry. (#2255) https://github.com/python/cpython/commit/b7c59f0d05bde0d4dd2d7bffc1b8af72c25d3f71 |
|
|
| msg296228 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2017-06-17 02:43 |
| New changeset ceb817dcfe8bf99725de033cce16a6f4cbced6ba by Ned Deily in branch 'master': bpo-23894: add Misc/NEWS entry. (#2256) https://github.com/python/cpython/commit/ceb817dcfe8bf99725de033cce16a6f4cbced6ba |
|
|