| msg22737 - (view) |
Author: Kevin Quick (kquick) |
Date: 2004-10-17 21:55 |
| The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! |
|
|
| msg22738 - (view) |
Author: Raymond Hettinger (rhettinger) *  |
Date: 2004-10-17 23:04 |
| Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. |
|
|
| msg22739 - (view) |
Author: Kevin Quick (kquick) |
Date: 2004-10-18 04:02 |
| Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) |
|
|
| msg22740 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2004-10-18 04:34 |
| Logged In: YES user_id=80475 Closing as invalid -- this is a bug in the OP's script, not in Python itself. The error is obious when the code is simplified further by in-lining the function. Also, it helps to explicitly set the initial values of variables between the two sections: #!/usr/bin/env python num_elem = 100 key_src = range(num_elem+1) keynums1 = key_src[:] assert key_src[-1] == 100 for _ in xrange(num_elem): keynums1[key_src.pop()] keynums2 = [0] for _ in xrange(102): key_src[keynums2.pop()] I think your coding error was in assuming that keynum2 held a different value before the second half was run. If so, that could have been found by using pdb or by inserting print statements to reveal what is going on. |
|
|
| msg22741 - (view) |
Author: Kevin Quick (kquick) |
Date: 2004-10-18 04:48 |
| Logged In: YES user_id=6133 Reopened: Problem not addressed. Please *read* my previous comment. Yes, the script has a bug in it. That's not what's being reported here. The bug is in the backtrace produced by the Python interpreter. See below. Line number reported for lambda function raising exception is line 14 below. Actual line number for lambda generating exception is line 18. $ python lambda_bug.py Traceback (most recent call last): File "lambda_bug.py", line 23, in ? seqtest() File "lambda_bug.py", line 19, in seqtest seq_func1(nextkey2, num_elem+2) File "lambda_bug.py", line 5, in seq_func1 key_genfunc() File "lambda_bug.py", line 14, in nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()] IndexError: pop from empty list |
|
|
| msg22742 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2004-10-18 05:41 |
| Logged In: YES user_id=80475 Confirmed. Also, the behavior persists into Py2.4b1. The error disappears when the lambda is replaced by an equivalent one line def statement: def nextkey2(N=keynums2,K=key_src): return K[N.pop()] The opcodes are the same for both; however, the co_firstlineno attribute is incorrect for the lambda version. The pure python compiler module does not make the same error. |
|
|
| msg22743 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2004-10-18 06:42 |
| Logged In: YES user_id=80475 Under the hood, the compiler is recognizing that the function bodies are identical and it produces just one code object instead of two. Hence, the co_firstline number with always reflect the first time the code is defined. A simple demo shows the id being the same when the bodies are identical: src = """ f1 = lambda x=1: x f2 = lambda x=2: x """ from dis import dis c = compile(src, 'example', 'exec') dis(c) For the OP's code, the effect can be shown by differentiating either lambda body by adding zero to the pop index: nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()+0] I conclude that there is less to the bug than meets the eye and that it may even be considered a feature. Referring to Jeremy for final disposition. |
|
|
| msg22744 - (view) |
Author: Kevin Quick (kquick) |
Date: 2004-10-18 07:37 |
| Logged In: YES user_id=6133 Thanks for confirming this issue. My position is that while minor, this is still a bug and not a feature. It's good to detect identical lambda bodies and optimize for that case, but since argument binding occurs at declaration time and not execution time for lambdas, the bound arguments should properly be considered part of the body; the two lambda functions in the attached script should be disjoint. From the practical perspective, the python backtrace was misleading in my debug efforts. Internal optimizations should take second place to accuracy of displayed information. func_code.co_code (and possibly others) could still be the same, but func_code.co_firstlineno (& func_code.co_filename) should be uniquely set. Hmmm. In thinking more about this, it's not that argument binding occurs immediately, but moreso that defaults are supplied for arguments. If the backtrace/debugger points me to a lambda function with entirely different default args than the ones that are applicable to the problem, then that would be wrong, IMHO. |
|
|
| msg22745 - (view) |
Author: Michael Hudson (mwh) |
Date: 2004-10-18 21:27 |
| Logged In: YES user_id=6656 I *think* what's happening here is that co_firstlineno is not being considered when comparing code objects. Can you try the attached patch? There may well be other fields not being considered. |
|
|
| msg22746 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2004-10-18 22:29 |
| Logged In: YES user_id=80475 Michael, that is a reasonable solution. Go ahead and apply it along with a test case verifying the object id's are distinct after the change but not before. |
|
|
| msg22747 - (view) |
Author: Kevin Quick (kquick) |
Date: 2004-10-18 23:24 |
| Logged In: YES user_id=6133 Confirmed that the patch does indeed address the problem. Also verified that a lambda with the same code appearing at the same line in a different file is handled separately and distinctly. Thanks! |
|
|
| msg22748 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2004-10-24 00:11 |
| Logged In: YES user_id=80475 Fixed. See: Python/compile.c 2.331 |
|
|