http://bugs.python.org/issue24622 made reminded me that a while back we added tests for the keyword module that includes a test that if you run it, you get the result that is checked in. The same thing could be done for the token.py module. And then we could add a cross-check test that tokenize.py has all the symbols defined as well.
This is my first real substantial testing change so I'd appreciate all feedback. The way I did the cross-check doesn't actually rely on any of the information from the test_keyword style regeneration test. I think this approach is a lot simpler, and will prevent mistakes like the one seen in http://bugs.python.org/issue24622
The cross check test itself doesn't depend on a regeneration, but it does depend on the information in token.h. Meanwhile the validity of *that* is checked by regeneration in your test_token tests. This is exactly what I had in mind :). So, I haven't done a full review, but the quick glance I took looked good.
I have no opinion on that issue. I opened this issue to fix a hole in our validation, but if that hole no longer exists after that (subsequently opened :) issue is resolved, then this isn't needed (sorry, Ammar, and thank you for your work!)
