[Python-Dev] Removing the GIL (Me, not you!) (original) (raw)

Jason Orendorff jason.orendorff at gmail.com
Thu Sep 13 19:29:23 CEST 2007

On 9/13/07, Justin Tulloss <tulloss2 at uiuc.edu> wrote:

1. Use message passing and transactions. [...] 2. Do it perl style. [...] 3. Come up with an elegant way of handling multiple python processes. [...] 4. Remove the GIL, use transactions for python objects, [...]

The SpiderMonkey JavaScript engine takes a very different approach, described here: http://developer.mozilla.org/en/docs/SpiderMonkey_Internals:_Thread_Safety

The SpiderMonkey C API threading model should sound familiar: C code can assume that simple operations, like dictionary lookups, are atomic and thread-safe. C code must explicitly JS_SuspendRequest() before doing blocking I/O or number-crunching (just like Py_BEGIN_ALLOW_THREADS). The main difference is that SpiderMonkey's "requests" are not mutually exclusive, the way the GIL is.

SpiderMonkey does fine-grained locking for mutable objects to avoid race conditions. The clever bit is that SpiderMonkey's per-object locking does not require a context switch or even an atomic instruction, in the usual case where an object is not shared among threads. (Programs that embed SpiderMonkey therefore run faster if they manage to ensure that threads share relatively few mutable objects. JavaScript doesn't have modules.)

Suppose Python went this route. There would still have to be a "stop-the-world" global lock, because the cycle collector won't work if other threads are going about changing pointers. (SpiderMonkey's GC does the same thing.) Retaining such a lock has another advantage: this change could be completely backward-compatible to extensions. Just use this global lock as the GIL when entering a non-thread-safe extension (all existing extensions would be considered non-thread-safe).

This means non-thread-safe extensions would be hoggish (but not much worse than they are already!). Making an existing extension thread-safe would require some thought, but it wouldn't be terribly hard. In the simplest cases, the extension writer could just add a flag to the type saying "ok, I'm thread-safe".

Refcounting is another major issue. SpiderMonkey uses GC instead. CPython would need to do atomic increfs/decrefs. (Deferred refcounting could mitigate the cost.)

The main drawback (aside from the amount of work) is the patent. SpiderMonkey's license grants a worldwide, royalty-free license, but not under the Python license. I think this could be wrangled, if the technical approach looks worthwhile.

-j

More information about the Python-Dev mailing list