The parser module generates an ast node number 54 for an entirely blank line. These lines are apparently ignored by the real parser, and don't have a real definition in include\token.h. Presumably because of that they don't have an entry in token.py. However, this broke an app where I hardcoded the number and recent language features have changed the mapping of numbers to token names. I think there should be a BLANK_LINE defined in token.py so that it can be referred to by name, even if it's not used in the real parser. While dealing with parsermodule generated ast-trees, "symbol.sym_name[node] or token.tok_name[node]" should always be true.
Logged In: YES user_id=699438 A few clarifications: On further review, the node number 54 comes from tokenize.py, not the parsermodule. I also meant to say that "symbol.sym_name.has_key(node) or token.tok_name.has_key(node)" should always be true.
As you found yourself, the patch is wrong. This token type is already exposed from tokenize as tokenize.NL; if you use tokenize, you should use its list of token types, not the one of token.py. Closing this patch as rejected.
