the Tokenize module has the following code: # An undocumented, backwards compatible, API for all the places in the standard # library that expect to be able to use tokenize with strings def generate_tokens(readline): return _tokenize(readline, None) So I'm going to assume it is Deprecated... (since 3.0 AFAICT). If it is expected from Python developers to not use it, may I suggest: 1) Marking it as deprecated in the docstring/ 2) Raise a deprecation warning. Indeed not everyone code by looking at the documentation and it is relatively easy to have your IDE/editor/REPL to complete it. Even tools that grab the source (IPython double question mark for example) will not show the comment above which make it kinda pointless.
There is related discussion in Issue 12486 about supporting unencoded text input. The current patch there actually already raises a warning and removes call sites from the Python library, though it does not add a doc string.
Thanks Thomas. I've been a long term user of tokenize.generate_tokens() and would be sad to see it go. The underlying _tokenize() functionality is used else within the module, so there the no benefit to removing the API.
I just stumbled upon this issue while triaging. With the related issue https://bugs.python.org/issue12486 closed making generate_tokens public can this issue be closed since the original issue is not valid? Thanks
Since generate_tokens was made public and documented with I am closing this issue as won't fix as part of triaging. Feel free to reopen this issue if needed. Thanks
History
Date
User
Action
Args
2022-04-11 14:58:38
admin
set
github: 72604
2018-09-24 15:15:28
xtreak
set
status: open -> closedresolution: wont fixmessages: + stage: resolved
