Although 3.x does yield a named-tuple, 2.x does not, despite the documentation: The exact type can be determined by checking the token string field on the named tuple returned from tokenize.tokenize() for the character sequence that identifies a specific operator token. (I'd much rather this functionality be back-ported from 3, as it's quite simple)
The 2.x documentation says generate_tokens returns a 5-tuple, not a namedtuple: http://docs.python.org/2/library/tokenize.html "The generator produces 5-tuples with these members: ..." The section you quote is talking about tokenize.tokenize, and I think that documentation should be fixed. Changing to namedtuple would be a new feature, which are not being added to 2.7.
