[Python-Dev] Assign(expr* targets, expr value) (original) (raw)

anatoly techtonik techtonik at gmail.com
Fri Nov 22 15:01:52 CET 2013

On Fri, Nov 22, 2013 at 3:29 PM, Benjamin Peterson <benjamin at python.org> wrote:

2013/11/22 anatoly techtonik <techtonik at gmail.com>:

On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson <benjamin at python.org> wrote:

2013/11/15 anatoly techtonik <techtonik at gmail.com>:

On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson <benjamin at python.org> wrote:

2013/11/12 anatoly techtonik <techtonik at gmail.com>:

On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin at python.org> wrote:

2013/11/10 anatoly techtonik <techtonik at gmail.com>:

http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl

In Assign(expr* targets, expr value), why the first argument is a list? x = y = 42 Thanks. Speaking of this ASDL. expr* targets means that multiple entities of expr under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But expr definition contains things that can not be used as left side assignment targets: expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis -- the following expression can appear in assignment context | Attribute(expr value, identifier attr, exprcontext ctx) | Subscript(expr value, slice slice, exprcontext ctx) | Starred(expr value, exprcontext ctx) | Name(identifier id, exprcontext ctx) | List(expr* elts, exprcontext ctx) | Tuple(expr* elts, exprcontext ctx) If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve? Only valid expression targets are allowed during AST construction. See setexprcontext in ast.c. Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html There is no setexprcontext in my copy of CPython code, which seems to be some alpha of Python 3.4 It's actually called setcontext. Ok. So what is the process? SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST --> --> AST --> BYTECODE Is that right? I don't know what sentence stream is, but otherwise looks right.

I mean that when you have TOKENS, you need to validate that their order is valid and throw errors to console if it is not. Like every sentence should have certain word order to make sense, you won't be able to construct CST from tokens that are placed in wrong order. Right?

Or is CST itself a tree of tokens, which position is validated when the tree is transformed to AST?

FWIW, I've modified my astdump project to make it easy to explore Python AST and experiment with it. This dataset shows how to create coverage for AST transformations (visitors) https://bitbucket.org/techtonik/astdump/src/tip/dataset/?at=default

More information about the Python-Dev mailing list