[Python-Dev] AST optimizer implemented in Python (original) (raw)

Victor Stinner victor.stinner at gmail.com
Sun Aug 12 22:49:57 CEST 2012

I started to implement an AST optimizer in Python. It's easy to create a new AST tree, so I'm surprised that I didn't find any existing project.

I done more research. I found the AST optimizer of PyPy, which implements basic optimizations: https://bitbucket.org/pypy/pypy/src/default/pypy/interpreter/astcompiler/optimize.py

So there is also something like an AST optimizer in Cython: https://github.com/cython/cython/blob/master/Cython/Compiler/Optimize.py https://github.com/cython/cython/blob/master/Cython/Compiler/ParseTreeTransforms.py https://github.com/cython/cython/blob/master/Cython/Compiler/Builtin.py https://github.com/cython/cython/blob/master/Cython/Compiler/Pipeline.py#L123

--

https://bitbucket.org/haypo/misc/src/tip/python/astoptimizer.py

I moved the script to a new dedicated project on Bitbucket: https://bitbucket.org/haypo/astoptimizer

Join the project if you want to help me to build a better optimizer!

It now works on Python 2.5-3.3.

There is BytecodeAssembler [1], but it seems to be specialized on bytecode. There are (at least?) 3 different issues to implement an AST optimizer, but in C, not in Python:

http://bugs.python.org/issue1346238 http://bugs.python.org/issue10399 http://bugs.python.org/issue11549

Oh, http://bugs.python.org/issue10399 includes an optimizer implemented in Python: Lib/optimizer.py. It inlines functions and create specialized versions of a function.

My proof-of-concept only implements very basic optimizations like 1+1 => 2 or "abcdef"[:3] => "abc", but it should easy to extend it to do more interesting optimization like function inlining.

It is also possible to call functions and methods at compile time, if there have no border effect (and don't depend on the environnement). For example, len("abc") is always 3. I added a lot of such functions, especially builtin functions.

I added options to enable more aggressive optimizations if we know that result will run on the same host than the compiler (os.name, sys.byteorder, ... are replaced by their value).

Victor

More information about the Python-Dev mailing list