[Python-Dev] Import lock considered mysterious (original) (raw)

P.J. Eby pje at telecommunity.com
Fri Jul 22 17:01:57 CEST 2011

At 02:48 PM 7/22/2011 +0200, Antoine Pitrou wrote:

See http://bugs.python.org/issue9260

There's a patch there but it needs additional sophistication to remove deadlocks when doing concurrent circular imports.

I don't think that approach can work, as PEP 302 loaders can currently assume the global import lock is being held when they run... and in general, there are too many global data structures in sys that need to be protected by code that uses such things.

A simpler solution to Greg's problem would be to have a timeout on attempts to acquire the import lock, and have it fail with a RuntimeError describing the problem. (Not an ImportError, mind you, which might get ignored and trigger a different code path.)

The timeout would need to be on the order of seconds to prevent false positives, and there'd need to be a way to change or remove the timeout in the event somebody really needs to. But it would eliminate the mysteriousness. A unique and Google-able error message would let someone find a clear explanation of what's going on, as well.

A second thing that could be helpful would be to issue a warning when a new thread is started (or waited on) while the import lock is held. This is already known to be a bad thing to do.

The tricky part is issuing the warning for the right caller level, but I suppose you could walk back up the call stack until you found some module-level code, and then fingered that line of code as the culprit.

Yes, that might do it: the code for starting or waiting on a thread, could check to see if the import lock is held by the current thread, and if so, walk up the stack to find a module frame (one where f_globals is f_locals and name in f_locals and sys.modules[name].dict is f_locals), and if one is found, issue a warning about not starting or waiting on threads in module-level code.

Between that and the timeout, the mysteriousness could be completely done away with, without throwing a monkey wrench into the current import mechanisms.

More information about the Python-Dev mailing list