[Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation (original) (raw)

Fabio Zadrozny fabiofz at gmail.com
Fri Nov 4 14:03:12 EDT 2016

On Fri, Nov 4, 2016 at 3:15 PM, Eric V. Smith <eric at trueblade.com> wrote:

On 11/4/2016 10:50 AM, Fabio Zadrozny wrote:

In what way do you think the implementation isn't ready for a final release?

Well, the cases listed in the docs​ (https://hg.python.org/cpython/file/default/Doc/reference/ lexicalanalysis.rst <https://hg.python.org/cpython/file/default/Doc/reference/_ _lexicalanalysis.rst>) don't work in the latest release (with SyntaxErrors) -- and the bug I created related to it: http://bugs.python.org/issue28597 <http://bugs.python.org/issue28597> was promptly closed as duplicate -- so, I assumed (maybe wrongly?) that the parsing still needs work. It's not the parsing that needs work, it's the documentation. Those examples used to work, but the parser was deliberately changed to not support them. There's a long discussion on python-ideas about it, starting at https://mail.python.org/pipermail/python-ideas/2016-August/041727.html

​Understood ;) ​

​It'd be nice if at least this description could be added to the PEP (as all other language implementations and IDEs will have to work the same way and will probably reference it) -- a grammar example, even if not used would be helpful (personally, I think hand-crafted parsers are always worse in the long run compared to having a proper grammar with a parser, although I understand that if you're not really used to it, it may be more work to set it up).

I've written a parser generator just to understand how they work, so I'm completely sympathetic to this. However, in this case, I don't think it would be any easier. I'm basically writing a tokenizer, not an expression parser. It's much simpler. The actual parsing is handled by PyParserASTFromString. And as I state below, you have to also consider the parser consumers. Also, I find it a bit troubling that ​​ PyParserASTFromString is used there and not just the node which would be related to an expression, although I understand it's probably an easier approach, although in the end you probably have to filter it and end up just accepting what's beneath the "test" from the grammar, no? (i.e.: that's what a lambda body accepts). Using PyParserASTFromString is the easiest possible way to do this. Given a string, it returns an AST node. What could be simpler?

​I think that for implementation purposes, given the python infrastructure, it's fine, but for specification purposes, probably incorrect... As I don't think f-strings should accept:

f"start {import sys; sys.version_info[0];} end" (i.e.: ​ PyParser_ASTFromString doesn't just return an expression, it accepts any valid Python code, even code which can't be used in an f-string).

​Well, I think all language implementations / IDEs (or at least those

which want to give syntax errors) will have to look inside f-strings.

While it's probably true that IDEs (and definitely language implementations) will want to parse f-strings, I think there are many more code scanners that are not language implementations or IDEs. And by being "just" regular strings with a new prefix, it's trivial to get any parser that doesn't care about the internal structure to at least see f-strings as normal strings. Also, you could still have a separate grammar saying how to look inside f-strings (this would make the lives of other implementors easier) even if it was a post-processing step as you're doing now. Yes. I've contemplated exposing the f-string scanner. That's the part that returns expressions (as strings) and literal strings. I realize that won't help 3.6.

Nice...

As a note, just for the record, my own interest on f-strings is knowing how exactly they are parsed for providing a preview of PyDev with syntax highlighting and preliminary support for f-strings (which at the very minimum besides syntax highlighting for the parts of f-strings should also show syntax errors inside them).

​Cheers,​

​Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20161104/28335572/attachment.html>

More information about the Python-Dev mailing list