[Python-Dev] Coroutines and PEP 380 (original) (raw)

Matt Joiner anacrolix at gmail.com
Fri Jan 20 04:01:19 CET 2012

On Fri, Jan 20, 2012 at 8:41 AM, Greg <greg.ewing at canterbury.ac.nz> wrote:

Glyph wrote:

[Guido] mentions the point that coroutines that can implicitly switch out from under you have the same non-deterministic property as threads: you don't know where you're going to need a lock or lock-like construct to update any variables, so you need to think about concurrency more deeply than if you could explicitly always see a 'yield'. I'm not convinced that being able to see 'yield's will help all that much. In any system that makes substantial use of generator-based coroutines, you're going to see 'yield from's all over the place, from the lowest to the highest levels. But that doesn't mean you need a correspondingly large number of locks. You can't look at a 'yield' and conclude that you need a lock there or tell what needs to be locked. There's no substitute for deep thought where any kind of theading is involved, IMO. -- Greg

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com

I wasn't aware that Guido had brought this up, and I believe what he says to be true. Preemptive coroutines, are just a hack around the GIL, and reduce OS overheads. It's the explicit nature of the enhanced generators that is their greatest value.

FWIW, I wrote a Python 3 compatible equivalent to gevent (also greenlet based, and also very similar to Brett's et al coroutine proposal), which didn't really solve the concurrency problems I hoped. There were no guarantees whether functions would "switch out", so all the locking and threading issues simply reemerged, albeit with also needing to have all calls non-blocking, losing compatibility with any routine that didn't make use of nonblocking calls and/or expose it's "yield" in the correct way, but reducing GIL contention. Overall not worth it.

In short, implicit coroutines are just a GIL work around, that break compatibility for little gain.

Thanks Glyph for those links.

More information about the Python-Dev mailing list