The token module appears to be designed to be used with import *. In fact it is used this way in the tokenize module. However it does not define __all__ and as a result, from token import * leaks symbol "main": >>> import tokenize >>> tokenize.main.__module__ 'token' Attached patch adds token.__all__ and "modernizes" generation of the tok_name dictionary. I also renamed main to _main because it is hard to imagine that any user code would ever want to use it.
Normally, I'd recommend leaving the "main" name alone, but in the case of modules that use it as an tool for module maintenance (symbol, token), I don't think it matters either way. Ok to commit.
