[Python-Dev] Two laments about CPython's AST Nodes (original) (raw)

Frank Wierzbicki fwierzbicki at gmail.com
Thu Aug 20 15:20:49 CEST 2009

On Wed, Aug 19, 2009 at 5:00 PM, "Martin v. Löwis"<martin at v.loewis.de> wrote:

Now on to the complaints: Though I recently added support for this in Jython, I don't like that nodes can be defined without required attributes, for example:

node = ast.Assign() I think we disagree in two points in our evaluation of this behavior: a) ast.Assign is not a node of the CPython AST. The CPython AST is a set of C structures in Include/Python-ast.h. ast.Assign is merely a mirror structure of that. Ah -- that is different from Jython's current design (The nodes that are constructed by ast.Assign() in Jython actually are the exact nodes that are used in real parsing)

b) it is, IMO, not reasonable to require users who create AST trees out of nothing to have correct trees at all times. I.e. it must be possible to represent incorrect trees. That does seem reasonable. Luckily it was easy to implement for me :)

c) the AST is not part of the Python language or library. It may change at any time without notice, and Jython is not required to duplicate its behavior exactly. Sure, I'm really just talking about ast.py (though for Jython ATM they are the same thing). So that I understand better: when this call is made in Python:

x = compile("def foo():pass", "foo", "exec", _ast.PyCF_ONLY_AST)

Does x contain real AST nodes or does it contain mirror structures (feel free to just tell me: don't be lazy, go read the code). If it contains real nodes, this is where I have some implementation trouble. If a tree of real nodes is then manipulated so that you end up with a mix, I don't want to walk the entire thing over again to find the mirror objects (that might be incomplete) and replace them with real nodes. If this creates a tree of mirror nodes, then I may want to consider doing the same thing on the Jython side (it makes sense, now that I understand CPython better I realize that the cost I am incurring is probably due to having the real and mirror AST as the same beast).

[so that's three items - as there should be in any good list of two items :-] :)

What precisely is it that makes this difficult to implement. If you would follow CPython's implementation strategy (i.e. generate glue code out of ASDL), I feel that it should be straight-forward to provide exactly the same behavior in Jython. I do use the ASDL to generate this stuff, but again, the real and the mirror nodes are not separated ATM, and that is what makes it difficult.

BTW -- I am volunteering to attempt to implement these things in CPython if there is support :) I'm not sure I can support such a change. Giving the child nodes at creation time, optionally, would be fine with me. Requiring the tree to conform to the grammar at all times is unreasonable, IMO (you can't simultaneously create all nodes in the tree and glue them together, so you have to create the tree in steps - which means that it will be intermittently incorrect). That is quite reasonable, I'll withdraw gripe #1 -- in fact the reason I have already implemented this in Jython is that there is already real world use out there. I still need to understand gripe #2 a little better before I back down on that one.

-Frank

More information about the Python-Dev mailing list