[Python-Dev] tokenize.py (generate_tokens yielding @ as OP) (original) (raw)
Guido van Rossum guido at python.org
Mon Aug 9 22:21:57 CEST 2004
- Previous message: [Python-Dev] tokenize.py (generate_tokens yielding @ as OP)
- Next message: [Python-Dev] Another test_compiler mystery
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
This is a question regarding the tokenize.generatetokens function.
As it is in CVS HEAD, generatetokens yields the '@' token as having type OP, not type AT. Is this how it should be?
Without looking at the code, I believe tokenize treats all operators as OP, for a rather liberal definition of "operator".
(A more serious problem that I just encountered in a different context: when you feed it input containing unsupported characters, the spaces before those unsupported characters are also returned as tokens. I think that's a bug.)
--Guido van Rossum (home page: http://www.python.org/~guido/)
- Previous message: [Python-Dev] tokenize.py (generate_tokens yielding @ as OP)
- Next message: [Python-Dev] Another test_compiler mystery
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]