[Python-Dev] Adding a token (original) (raw)

Greg Ewing greg.ewing at canterbury.ac.nz
Sun Aug 8 03:41:56 CEST 2010

I'm trying to add a '?' token to the parser, and weird things are happening.

I've added a #define to token.h, an entry to _PyParser_TokenNames in tokenizer.c and case for it in PyToken_OneChar(). But it's behaving as though the tokenizer is not recognising my token.

I put in some printfs to find out whether PyToken_OneChar() is recognising it. The results are confusing: while running pgen, PyToken_OneChar() is being called and recognising the new token correctly.

However, it doesn't seem to be getting called at all when parsing Python code. I don't see how this can happen, because pgen seems to use the same tokenizing code as Python itself.

Is there anything else I need to do? Does some file need to be manually re-made?

-- Greg

More information about the Python-Dev mailing list