[Python-Dev] Small tweak to tokenize.py? (original) (raw)
python at rcn.com python at rcn.com
Thu Nov 30 20:12:16 CET 2006
- Previous message: [Python-Dev] Small tweak to tokenize.py?
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
It would be trivial to add another yield to tokenize.py when the backslah is detected
+1
I think that it should probably yield a single NL pseudo-token whose value is a backslash followed by a newline; or perhaps it should yield the backslash as a comment token, or as a new token.
The first option is likely the most compatible with existing uses of tokenize. If a comment token were emitted, an existing colorizer or pretty-printer would markup the continuation as a comment (possibly not what the tool author intended). If a new token were created, it might break if-elif-else chains in tools that thought they knew the universe of possible token types.
Raymond
- Previous message: [Python-Dev] Small tweak to tokenize.py?
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]