[Python-Dev] Rewrite @contextlib.contextmanager in C (original) (raw)

Giampaolo Rodola' g.rodola at gmail.com
Tue Aug 9 14:29:45 EDT 2016

On Mon, Aug 8, 2016 at 11:59 PM, Chris Angelico <rosuav at gmail.com> wrote:

On Tue, Aug 9, 2016 at 7:14 AM, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote: > Right, I think a fairer comparison would be to: > > class ctx2: > def enter(self): > self.it = iter(self) > return next(self.it) > > def exit(self, *args): > try: > next(self.it) > except StopIteration: > pass > > def iter(self): > yield > > With this change alone the slowdown diminishes to ~ 1.7x for me. The rest is > probably the extra overhead for being able to pass exceptions raised inside > the with block back into the generator and such.

I played around with a few other variants to see where the slowdown is. They all work out pretty much the same as the above; my two examples are both used the same way as contextlib.contextmanager is, but are restrictive on what you can do. import timeit import contextlib import functools class ctx1: def enter(self): pass def exit(self, *args): pass @contextlib.contextmanager def ctx2(): yield class SimplerContextManager: """Like contextlib.GeneratorContextManager but way simpler. * Doesn't reinstantiate itself - just reinvokes the generator * Doesn't allow yielded objects (returns self) * Lacks a lot of error checking. USE ONLY AS DIRECTED. """ def init(self, func): self.func = func functools.updatewrapper(self, func) def call(self, *a, **kw): self.gen = self.func(*a, **kw) return self def enter(self): next(self.gen) return self def exit(self, type, value, traceback): if type is None: try: next(self.gen) except StopIteration: return else: raise RuntimeError("generator didn't stop") try: self.gen.throw(type, value, traceback) except StopIteration: return True # Assume any instance of the same exception type is a proper reraise # This is way simpler than contextmanager normally does, and costs us # the ability to detect exception handlers that coincidentally raise # the same type of error (eg "except ZeroDivisionError: print(1/0)"). except type: return False # Check that it actually behaves correctly @SimplerContextManager def ctxdemo(): print("Before yield") try: yield 123 except ZeroDivisionError: print("Absorbing 1/0") return finally: print("Finalizing") print("After yield (no exception)") with ctxdemo() as val: print("1/0 =", 1/0) with ctxdemo() as val: print("1/1 =", 1/1) #with ctxdemo() as val: # print("1/q =", 1/q) @SimplerContextManager def ctx3(): yield class TooSimpleContextManager: """Now this time you've gone too far.""" def init(self, func): self.func = func def call(self): self.gen = self.func() return self def enter(self): next(self.gen) def exit(self, type, value, traceback): try: next(self.gen) except StopIteration: pass @TooSimpleContextManager def ctx4(): yield class ctx5: def enter(self): self.it = iter(self) return next(self.it) def exit(self, *args): try: next(self.it) except StopIteration: pass def iter(self): yield t1 = timeit.timeit("with ctx1(): pass", setup="from main import ctx1") print("%.3f secs" % t1) for i in range(2, 6): t2 = timeit.timeit("with ctx2(): pass", setup="from main import ctx%d as ctx2"%i) print("%.3f secs" % t2) print("slowdown: -%.2fx" % (t2 / t1))

My numbers are: 0.320 secs 1.354 secs slowdown: -4.23x 0.899 secs slowdown: -2.81x 0.831 secs slowdown: -2.60x 0.868 secs slowdown: -2.71x So compared to the tight custom-written context manager class, all the "pass it a generator function" varieties look pretty much the same. The existing contextmanager factory has several levels of indirection, and that's probably where the rest of the performance difference comes from, but there is some cost to the simplicity of the gen-func approach. My guess is that a C-implemented version could replace piles of error-handling code with simple pointer comparisons (hence my elimination of it), and may or may not be able to remove some of the indirection. I'd say it'd land about in the same range as the other examples here. Is that worth it? ChrisA

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/g.rodola% 40gmail.com

Thanks for all this useful info. And I agree, some sort of slowdown should be expected because it's the price you pay for the additional flexibility that @contextmanager offers over a "raw" ctx manager class, which will always be faster as "it does less".

-- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20160809/fa29dbbc/attachment.html>

More information about the Python-Dev mailing list