Issue 26452: Wrong line number attributed to comprehension expressions (original) (raw)

In a multi-line list comprehension (or dict or set comprehension), the code for the main expression of the comprehension is wrongly attributed to the last line of the comprehension, which might be several lines later.

This makes for quite baffling tracebacks when an exception occurs -- for example this program:

def f():
    return [j
            for i in range(3)
            if i]

f()

produces (with CPython from current default):

Traceback (most recent call last):
  File "foo.py", line 15, in <module>
    f()
  File "foo.py", line 3, in f
    for i in range(3)
  File "foo.py", line 4, in <listcomp>
    if i]
NameError: name 'j' is not defined

showing the line if i], which has nothing to do with the error and gives very little hint as to where the exception is being raised.

Disassembly confirms that the line numbers on the code object are wrong:

  2           0 BUILD_LIST               0
              3 LOAD_FAST                0 (.0)
        >>    6 FOR_ITER                18 (to 27)

  3           9 STORE_FAST               1 (i)

  4          12 LOAD_FAST                1 (i)
             15 POP_JUMP_IF_FALSE        6
             18 LOAD_GLOBAL              0 (j)
             21 LIST_APPEND              2
             24 JUMP_ABSOLUTE            6
        >>   27 RETURN_VALUE

The LOAD_GLOBAL instruction for j is attributed to line 4, when it should be line 2.

A similar issue affects multi-line function calls, which get attributed to a line in the last argument. This is less often so seriously confusing because the function called is right there as the next frame down on the stack, but it's much more common and it makes the traceback look a little off -- I've noticed this as a minor annoyance for years, before the more serious comprehension issue got my attention.

Historically, line numbers were constrained to be wrong in these ways because the line-number table co_lnotab on a code object required its line numbers to increase monotonically -- and the code for the main expression of a comprehension comes after all the for and if clauses, so it can't get a line number earlier than theirs. Victor Stinner's recent work in https://hg.python.org/cpython/rev/775b74e0e103 lifted that restriction in the co_lnotab data structure, so it's now just a matter of actually entering the correct line numbers there.

I have a draft patch to do this, attached here. It fixes the issue both for comprehensions and function calls, and includes tests. Things I'd still like to do before considering the patch ready:

There are a couple of bits of logic that I knock out that can probably be simplified further.
While I'm looking at this, there are several other forms of expression and statement that have or probably have similar issues, and I'll want to go and review them too to either fix or determine that they're fine. The ones I've thought of are included in the draft test file, either as actual tests (with their current answers) or TODO comments for me to investigate.

Comments very welcome on the issue and my draft patch, and meanwhile I'll continue with the further steps mentioned above.

Thanks to Benjamin Peterson for helping diagnose this issue with me when we ran into a confusing traceback that ran through a comprehension.

I think there have been some improvements merged with https://bugs.python.org/issue12458 . I am not sure if the patch covers more cases. I ran the tests attached in the patch and some of them cause RuntimeError in master.

bpo26452.py

def f(): return [j for i in range(3) if i]

f()

Master

./python.exe Python 3.8.0a0 (heads/master:c510c6b8b6, Sep 21 2018, 11:10:24) [Clang 7.0.2 (clang-700.1.81)] on darwin Type "help", "copyright", "credits" or "license" for more information.

./python.exe ../backups/bpo26452.py Traceback (most recent call last): File "../backups/bpo26452.py", line 6, in f() File "../backups/bpo26452.py", line 2, in f return [j File "../backups/bpo26452.py", line 2, in return [j NameError: name 'j' is not defined

Python 3.7

python3.7 ../backups/bpo26452.py Traceback (most recent call last): File "../backups/bpo26452.py", line 6, in f() File "../backups/bpo26452.py", line 3, in f for i in range(3) File "../backups/bpo26452.py", line 4, in if i] NameError: name 'j' is not defined

Tests

./python.exe ../backups/test26452.py ..F

FAIL: test_operators (main.TestPEP380Operation)

Traceback (most recent call last): File "../backups/test26452.py", line 21, in expect_line f() File "../backups/test26452.py", line 98, in f - RuntimeError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "../backups/test26452.py", line 100, in test_operators self.expect_line(f, 3) # want 2 File "../backups/test26452.py", line 26, in expect_line self.assertEqual(relative_line, expected_relative_line) AssertionError: 2 != 3

Ran 3 tests in 0.001s

FAILED (failures=1)

Thanks