| msg259680 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-02-05 17:34 |
| For using more efficient bytecode (either specialized 8-bit opcodes or 16-bit opcodes) we need to move some optimizations from bytecode level to AST level, since LOAD_CONST variants could have variable size. Now with the Constant node this should be easy. |
|
|
| msg259692 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-02-05 22:12 |
| *If* the PEP 511 is accepted, it will be super easy to start with an implementation in pure Python. But I think that we should benchmark the overhead of the Python API of PEP 511, because we need to convert all AST internal objects to Python objects, run the AST optimizer, and then again convert Python objects to internal AST objects. For a long running process, the time to compile a .py file doesn't matter. For a short script, it matters. At least, we need to compile the script itself. By the way, would it be insane to *not* optimize the script when running "python script.py"? > Now with the Constant node this should be easy. ast.Constant is *not* emited by the compiler to not break backward compatibility. I *know* that there is no stable API on AST, but I noticed some issues when working on my AST project. For example, pip doesn't work because an internal library uses AST and the code doesn't handle ast.Constant (it's probably super easy to fix it). I'm open to change the compiler to emit ast.Constant directly, but maybe only in "optimized mode" (ex: python -O?). ast.Constant for tuples and frozensets has a limitation: it doesn't store the location of items (line number, column offset). |
|
|
| msg259694 - (view) |
Author: Yury Selivanov (yselivanov) * |
Date: 2016-02-05 22:35 |
| > ast.Constant is *not* emited by the compiler to not break backward compatibility. I *know* that there is no stable API on AST, but I noticed some issues when working on my AST project. For example, pip doesn't work because an internal library uses AST and the code doesn't handle ast.Constant (it's probably super easy to fix it). Yeah, it's probably very easy to fix. And if pip stops working on 3.6-alpha it will be fixed. We don't give any guarantees on AST backwards compatibility. In fact, we even tell that the AST might change in every release in the docs "The abstract syntax itself might change with each Python release". > I'm open to change the compiler to emit ast.Constant directly, but maybe only in "optimized mode" (ex: python -O?). I'm -1 on using -O flag since some people would prefer to have optimizations *and* have their assert statements. In fact I'm -1 on using any kind of flag to trigger optimizations on or off. Those flags will only drive confusion. If the optimizations are any good, people will simply start using the flag all the time. |
|
|
| msg259696 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-02-05 22:40 |
| Yury Selivanov added the comment: > In fact I'm -1 on using any kind of flag to trigger optimizations on or off. Those flags will only drive confusion. If the optimizations are any good, people will simply start using the flag all the time. Yeah, Guido calls this a missed optimization opportunity. My only concern is: > ast.Constant for tuples and frozensets has a limitation: it doesn't store the location of items (line number, column offset). Is it a problem in practice? I didn't have time to investigate this. Since the PEP 511 is still under discussion, I prefer to focus on the PEP itself than this specific implementation detail ;-) |
|
|
| msg259699 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-02-05 23:01 |
| > we need to move some optimizations from bytecode level to AST level You may be aware that there is a long list of similar (old) issues proposing something similar: https://www.python.org/dev/peps/pep-0511/#usage-1-ast-optimizer My PEP 511 tries to propose a generic API to support all proposed optimizations. -- I didn't try yet to do something like importlib to compile an AST optimizer written in Python into a frozen module and then use it in Python. It would allow to implement an AST optimizer in pure Python to ease its maintenance *and* have it builtin into CPython, rather than having to play with AST in C (less fun). My plan for the PEP 511 is more to start to experiment multiple AST optimizers and bytecode optimizers outside CPython, wait until the code becomes more stable, and later discuss if one optimizer deserves to become part of Python stdlib. "Writing an optimizer or a preprocessor is out of the scope of this PEP." -- But maybe you would prefer to have a single and simple AST optimizer builtin Python without new additional API? I don't want to block your effort on experimentation :-) I just see a nice opportunity to sell you the PEP 511 ;-) |
|
|
| msg259795 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2016-02-07 18:10 |
| Issue1346238 has a patch. |
|
|