[Python-Dev] PEP 511 (code transformers) rejected (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Fri Nov 3 00:26:33 EDT 2017

On 2 November 2017 at 23:42, Victor Stinner <victor.stinner at gmail.com> wrote:

(Email resent, I first sent it to Nick privately by mistake.)

Oops, I didn't even notice that. Reposting my reply below.

2017-11-02 2:53 GMT+01:00 Nick Coghlan <ncoghlan at gmail.com>:

The piece that we're currently missing to make such workflows easier to manage is an equivalent of JavaScript's source maps (...) Code objects already have a instruction pointer => line number mapping table: the code.colnotab field. It's documented at: https://github.com/python/cpython/blob/master/Objects/lnotabnotes.txt This table is built from the line number information of the AST tree. The mapping table is optimized to be small. Before Python 3.5, line number had to be monotonic. Since Python 3.6, you "move" instructions at the AST level, and so have "non-monotonic" line numbers (ex: line 1, line 3, line 2, line 4).

Right, and linecache knows how to read that, However, it can only do so if the source files are on the running system with the bytecode files, and the source code we're interested in is the source code that was actually compiled by the interpreter.

Source code transformers fail that second precondition (since the interpreter only sees the post-transformation code), and this is one of the big reasons folks ended up writing actual single source 2/3 compatible code bases rather than running 2to3 as a source code transformer when building packages: with transformed source, conventional tracebacks quote the line from the transformed source code, not the line in the original pre-transformation source code.

However, if the code transformer were to emit a JavaScript style source map in addition to emitting the transformed code, then automated tooling could take a traceback that referenced lines in the transformed code, and work out the equivalent traceback for the pre-transformation code. (I believe Cython has something like that in order to provide it's HTML annotation mode, and PyPy's JIT can trace from machine code back to the related Python source lines, but we don't have anything that's independent of a particular toolchain the way source maps are)

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list