[Python-Dev] AST Transformation Hooks for Domain Specific Languages (original) (raw)

Michael Foord fuzzyman at voidspace.org.uk
Fri Apr 8 15:57:54 CEST 2011

On 8 April 2011 12:29, Nick Coghlan <ncoghlan at gmail.com> wrote:

A few odds and ends from recent discussions finally clicked into something potentially interesting earlier this evening. Or possibly just something insane. I'm not quite decided on that point as yet (but leaning towards the latter).

The essence of the proposal is to allow arbitrary syntax within "standard python files". I don't think it stands much of a chance in core.

It would be an awesome tool for experimenting with new syntax and dsls though. :-)

Michael

Anyway, without further ado, I present:

AST Transformation Hooks for Domain Specific Languages ====================================================== Consider: # In some other module ast.registerdsl("dsl.sql", dsl.sql.TransformAST) # In a module using that DSL import dsl.sql def lookupaddress(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql: select address from people where name = {name} and dob = {dob}

Suppose that the standard AST for the latter looked something like: DSL(syntax="dsl.sql", name='lookupaddress', args=arguments( args=[arg(arg='name', annotation=<Normal AST for "dsl.sql.char">), arg(arg='dob', annotation=<Normal AST for "dsl.sql.date">)], vararg=None, varargannotation=None, kwonlyargs=[], kwarg=None, kwargannotation=None, defaults=[], kwdefaults=[]), body=[Expr(value=Str(s='select address\nfrom people\nwhere name = {name} and dob = {dob}'))], decoratorlist=[], returns=None) (For those not familiar with the AST, the above is actually just the existing Function node with a "syntax" attribute added) At compile time (note, not function definition time), the registered AST transformation hook would be invoked and would replace that DSL node with "standard" AST nodes. For example, depending on the design of the DSL and its support code, the above example might be equivalent to: @dsl.sql.escapeandvalidateargs def lookupaddress(name: dsl.sql.char, dob: dsl.sql.date): args = dict(name=name, dob=dob) query = "select address\nfrom people\nwhere name = {name} and dob = {dob}" return dsl.sql.cursor(query, args) As a simpler example, consider something like: def f() from allnonlocal: x += 1 y -= 2 That was translated at compile time into: def f(): nonlocal x, y x += 1 y -= 2 My first pass at a rough protocol for the AST transformers suggests they would only need two methods: getcookie() - Magic cookie to add to PYC files containing instances of the DSL (allows recompilation to be forced if the DSL is updated) transformAST(node) - a DSL() node is passed in, expected to return an AST containing no DSL nodes (SyntaxError if one is found) Attempts to use an unregistered DSL would trigger SyntaxError So there you are, that's the crazy idea. The stoning of the heretic may now commence :) Where this idea came from was the various discussions about "make statement" style constructs and a conversation I had with Eric Snow at Pycon about function definition time really being too late to do anything particularly interesting that couldn't already be handled better in other ways. Some tricks Dave Malcolm had done to support Python level manipulation of the AST during compilation also played a big part, as did Eugene Toder's efforts to add an AST optimisation step to the compilation process. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk

--

http://www.voidspace.org.uk/

May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20110408/4e6b3521/attachment.html>

More information about the Python-Dev mailing list