Issue 1542308: Nested finally in generators don't follow PEP 342 (original) (raw)
Created on 2006-08-17 22:56 by bob.ippolito, last changed 2022-04-11 14:56 by admin.
Messages (8)
Author: Bob Ippolito (bob.ippolito) * 
Date: 2006-08-17 22:56
The close() and GC interaction of generators that use yield inside of finally blocks doesn't execute correctly when nested. See the attached example.
More information about the issue is in the Mozilla bug tracker (they found a similar bug in their implementation for JS 1.7): https://bugzilla.mozilla.org/show_bug.cgi?id=349012
Author: Guido van Rossum (gvanrossum) *
Date: 2006-08-22 20:47
Logged In: YES user_id=6380
I lost my patience halfway through reading the Mozilla bug tracker, but IMO this works as designed. The philosophical question is, would you rather see the outer finally clause reached in your example, or would you rather see the following generator terminated upon close? (If you "fix" the former, the latter ends up in an infinite loop when you attempt to close() or GC it.)
def gen(): while True: try: yield except: pass
Author: Bob Ippolito (bob.ippolito) *
Date: 2006-08-22 21:00
Logged In: YES user_id=139309
Uh no, that wouldn't reach an infinite loop because any attempt to yield during close will raise RuntimeError and terminate the loop.
The problem is that finally doesn't execute. Finally clauses always must execute. If they don't, then they're worthless.
The real strange issue is that if close is called, then GC makes a second attempt, and it does execute the outer finally clause. There are definitely bugs here.
Author: PJ Eby (pje) *
Date: 2006-08-22 22:23
Logged In: YES user_id=56214
Bob, the problem Guido is pointing out is that to run the finally clauses, we have to resume the generator with the RuntimeError thus generated. So it doesn't terminate the loop, because the RuntimeError is effectively raised at the point where the yield occurs. So, in order to run finally clauses sanely after a close() attempt, we would have to prevent such loops.
That doesn't mean Guido's example is valid as such; but I think it's possible to accidentally create quasi-indefinite loops using infinite iterators written prior to Python 2.5, if you had a try/except block that was expecting to catch something other than GeneratorExit or RuntimeError. So, the net effect would be an unintended hang, if we tried to support retrying until the generator can be closed.
Regarding the GC second attempt, this behavior is actually exactly as-specified by the PEP's pseudo-code explanation of how close() is handled. A RuntimeError raised from close() does not close the generator; it specifically indicates that the generator has not in fact closed.
At this point, after re-reading the PEP and looking at what the code is doing, it's clear that the behavior is "as specified", so the title of the bug is incorrect: the behavior is exactly as specified by PEP 342. So, the rest of my comments below are regarding whether PEP 342 should be changed.
And really, the question there is whether it's sane to attempt heroic measures in order to run finally clauses in a generator whose code is known to be buggy. The RuntimeError basically says, "this generator is broken", and the GC also tries a second time to close it. If two close attempts don't close it, it's a dead duck.
What other measures can we sanely add? We could try forcing the RuntimeError into the generator itself, but then we have to deal with the infinite loop possibility, and the oddities involved in getting to a language specification that can be implemented for Jython, IronPython, etc. On the whole, I think that we probably reached the right balance the first time around: a broken generator should not be allowed to handle the error of its own brokenness.
Author: Bob Ippolito (bob.ippolito) *
Date: 2006-08-22 22:30
Logged In: YES user_id=139309
It seems that the infinite loop would be the right thing to do, given that's what would happen in a similarly broken del (or anywhere else).
Author: PJ Eby (pje) *
Date: 2006-08-22 22:51
Logged In: YES user_id=56214
You'll need to propose a patch to PEP 342 to alter the specification, then, and get it past Python-Dev. Personally, I don't think that changing the spec is the right thing to do for 2.5.
But irrespective of those procedural matters, your del analogy is flawed on two points. First, we do not re-run a del method to handle an error that was raised by that del method! Second, if a generator contains an infinite, non-yielding loop, then of course it will loop forever. So del does not provide any actually useful guidance here.
Author: Antoine Pitrou (pitrou) *
Date: 2008-06-13 12:58
Is it useful to keep this bug open? Does it correspond to a real-world use case?
Author: Mark Lawrence (BreamoreBoy) *
Date: 2014-03-23 00:57
Just in case PEP 342 ever changes we might as well target the latest versions.
As an aside I followed the PEP 342 link to here http://legacy.python.org/dev/peps/pep-0342/ and clicked on the "last modified" field. I was taken here http://hg.python.org/peps/file/tip/pep-0342.txt. I was rather surprised to see this
view pep-0342.txt @ 5421:6ed8f836c072
PEP 466: finish incomplete sentence author Nick Coghlan <ncoghlan@gmail.com> date Sun, 23 Mar 2014 07:17:32 +1000 (3 hours ago) parents c25fa0913267
Could somebody take a look at this please.
History
Date
User
Action
Args
2022-04-11 14:56:19
admin
set
github: 43848
2014-03-23 00:57:03
BreamoreBoy
set
nosy: + BreamoreBoy
messages: +
2010-04-03 07:08:39
georg.brandl
set
status: open -> languishing
2008-06-13 12:58:32
pitrou
set
nosy: + pitrou
messages: +
2007-08-25 05:55:58
loewis
set
priority: release blocker -> normal
2006-08-17 22:56:50
bob.ippolito
create