(original) (raw)

changeset: 74516:75baef657770 branch: 2.7 parent: 74514:a08e9e84f33f user: Meador Inge meadori@gmail.com date: Thu Jan 19 00:17:44 2012 -0600 files: Doc/library/tokenize.rst Misc/NEWS description: Issue #2134: Clarify token.OP handling rationale in tokenize documentation. diff -r a08e9e84f33f -r 75baef657770 Doc/library/tokenize.rst --- a/Doc/library/tokenize.rst Thu Jan 19 00:40:46 2012 +0200 +++ b/Doc/library/tokenize.rst Thu Jan 19 00:17:44 2012 -0600 @@ -15,6 +15,12 @@ well, making it useful for implementing "pretty-printers," including colorizers for on-screen displays. +To simplify token stream handling, all :ref:`operators` and :ref:`delimiters` +tokens are returned using the generic :data:`token.OP` token type. The exact +type can be determined by checking the token ``string`` field on the +:term:`named tuple` returned from :func:`tokenize.tokenize` for the character +sequence that identifies a specific operator token. + The primary entry point is a :term:`generator`: .. function:: generate_tokens(readline) diff -r a08e9e84f33f -r 75baef657770 Misc/NEWS --- a/Misc/NEWS Thu Jan 19 00:40:46 2012 +0200 +++ b/Misc/NEWS Thu Jan 19 00:17:44 2012 -0600 @@ -495,6 +495,9 @@ Documentation ------------- +- Issue #2134: The tokenize documentation has been clarified to explain why + all operator and delimiter tokens are treated as token.OP tokens. + - Issue #13513: Fix io.IOBase documentation to correctly link to the io.IOBase.readline method instead of the readline module. /meadori@gmail.com