[Python-Dev] Lexical analysis and NEWLINE tokens (original) (raw)

Guido van Rossum guido at python.org
Thu Oct 6 17:30:45 CEST 2005

I think it is a relic from the distant past, when the lexer did generate NEWLINE for every blank line. I think the only case where you can still get a NEWLINE by itself is in interactive mode. This code is extremely convoluted and may be buggy in end cases; this could explain why you get a continuation prompt after entering a comment in interactive mode...

--Guido

On 10/6/05, Michael Hudson <mwh at python.net> wrote:

"Matthew F. Barnes" <mfb at lotusland.dyndns.org> writes:

> I posted this question to python-help, but I think I have a better chance > of getting the answer here. > > I'm looking for clarification on when NEWLINE tokens are generated during > lexical analysis of Python source code. In particular, I'm confused about > some of the top-level components in Python's grammar (fileinput, > interactiveinput, and evalinput). > > Section 2.1.7 of the reference manual states that blank lines (lines > consisting only of whitespace and possibly a comment) do not generate > NEWLINE tokens. This is supported by the definition of a suite, which > does not allow for standalone or consecutive NEWLINE tokens. > > suite ::= stmtlist NEWLINE | NEWLINE INDENT statement+ DEDENT I don't have the spare brain cells to think about your real problem (sorry) but something to be aware of is that the pseudo EBNF of the reference manual is purely descriptive -- it is not actually used in the parsing of Python code at all. Among other things this means it could well just be wrong :/ The real grammar is Grammar/Grammar in the source distribution. Cheers, mwh -- The Internet is full. Go away. -- http://www.disobey.com/devilshat/ds011101.htm

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

More information about the Python-Dev mailing list