The attached code snippet 'testcase.py' should produce an IndentationError, but does not. The code in tokenize.py is too trusting, and needs to add a check against bad indentation as it yields DEDENT tokens. I'm including a diff to tokenize.py that should at least raise an exception on bad indentation like this. Just in case, I'm including testcase.py here too: ------ import tokenize from StringIO import StringIO sampleBadText = """ def foo(): bar baz """ print list(tokenize.generate_tokens( StringIO(sampleBadText).readline))
Logged In: YES user_id=4771 Reopening this bug report: this might fix the problem at hand, but it breaks inspect.getsource() on cases where it used to work. See attached example.
Logged In: YES user_id=4771 Here is a proposed patch. It relaxes the dedent policy a bit. It assumes that the first line may already have some initial indentation, as is the case when tokenizing from the middle of a file (as inspect.getsource() does). It should also be back-ported to 2.4, given that the previous patch was. For 2.4, only the non-test part of the patch applies cleanly; I suggest to ignore the test part and just apply it, given that there are much more tests in 2.5 for inspect.getsource() anyway. The whole issue of inspect.getsource() being muddy anyway, I will go ahead and check this patch in unless someone spots a problem. For now the previously-applied patch makes parts of PyPy break with an uncaught IndentationError.
Logged In: YES user_id=149084 Tokenize Rev 39046 21Jun05 breaks tabnanny. tabnanny doesn't handle the IndentationError exception when tokenize detects a dedent. I patched up ScriptBinding.py in IDLE. The IndentationError probably should pass the same parms as TokenError and tabnanny should catch it.
