[Python-Dev] context manager - generator interaction? (original) (raw)

Phillip J. Eby pje at telecommunity.com
Thu Apr 5 22:38:49 CEST 2007

At 03:54 PM 4/5/2007 -0400, Bob Sidebotham wrote:

The interaction shown below feels like a bug, or at least very much a trap for the unwary. It's some sort of interaction between a context manager and a generator, both of which can raise StopIteration. The code is excised from a real application, so it's a bit artificial looking. Nevertheless, it represents an entirely natural evolution of some code (whereby the call to a context manager was added deep within the code). Since it seems to work with open as the context manager, but not with my own null context manager defined with contextmanager(), it feels like this is probably a bug in contextmanager().

It actually appears to be a quirk in the "with" machinery itself, that only affects generator context managers because of their need to handle StopIteration specially. If you rewrite your context manager as a class with enter and exit, the problem will go away.

Checking what was happening via pdb, I found that the contextmanager's exit is being called with a "value" of None, rather than with an exception instance. This fouls up GeneratorContextManager.exit's special handling for StopIteration -- which was apparently never tested by any unit tests.

I can see a few different ways to work around the problem in GeneratorContextManager itself, the simplest of which is to add these two lines right before the self.gen.throw() call:

             if type is StopIteration and value is None:
                 value = type()

It seems to me that this is a bit of a kludge, but trying to make it so that exit always gets called with an instance is much more of a pain since it appears that the None a performance enhancement for simple errors like StopIteration, that avoids creating an instance.

More information about the Python-Dev mailing list