(original) (raw)

changeset: 90285:d4f5a88b94b4 branch: 2.7 parent: 90269:ea677e26dbeb user: Terry Jan Reedy tjreedy@udel.edu date: Mon Apr 14 16:17:09 2014 -0400 files: Doc/library/tokenize.rst Misc/ACKS description: Closes #20956: 2.7 tokenize does not produce named tuples. Patch by Sam Kimbrel. diff -r ea677e26dbeb -r d4f5a88b94b4 Doc/library/tokenize.rst --- a/Doc/library/tokenize.rst Mon Apr 14 12:24:37 2014 -0400 +++ b/Doc/library/tokenize.rst Mon Apr 14 16:17:09 2014 -0400 @@ -17,9 +17,10 @@ To simplify token stream handling, all :ref:`operators` and :ref:`delimiters` tokens are returned using the generic :data:`token.OP` token type. The exact -type can be determined by checking the token ``string`` field on the -:term:`named tuple` returned from :func:`tokenize.tokenize` for the character -sequence that identifies a specific operator token. +type can be determined by checking the second field (containing the actual +token string matched) of the tuple returned from +:func:`tokenize.generate_tokens` for the character sequence that identifies a +specific operator token. The primary entry point is a :term:`generator`: diff -r ea677e26dbeb -r d4f5a88b94b4 Misc/ACKS --- a/Misc/ACKS Mon Apr 14 12:24:37 2014 -0400 +++ b/Misc/ACKS Mon Apr 14 16:17:09 2014 -0400 @@ -670,6 +670,7 @@ Jason Killen Jan Kim Taek Joo Kim +Sam Kimbrel W. Trevor King Paul Kippes Steve Kirsch /tjreedy@udel.edu