[Python-Dev] Import lock considered mysterious (original) (raw)

Greg Ewing greg.ewing at canterbury.ac.nz
Fri Jul 22 11:29:23 CEST 2011

I recently encountered a very mysterious bug in which a background thread would inexplicably hang while attempting to make a connection using httplib.

Debugging as far as I could at the Python level led to the surprising conclusion that it was hanging while using the ascii codec to encode the host name.

I couldn't imagine why, until I resorted to breaking into it with gdb and found that it was trying to import "codecs.ascii", and blocked on acquiring the import lock.

The reason for that was that my main module was a stub that imported the real main module, which did all its work directly from the module code. So the whole program was effectively running inside an import statement and holding onto the import lock.

Once I realised what was happening, it was easy to fix, but I'm a bit disturbed about how difficult it was to track down the cause.

This whole episode seems to have resulted from a collision between two rather obscure things: that import statements involve locking things, and that some fairly innocuous looking calls, such as s.encode('ascii'), can trigger an import.

I'm wondering whether anything can be done to make problems like this less likely to occur, or at least to make the cause more readily apparent. One shouldn't really have to use gdb to track down bugs in Python code.

Is it really necessary to hold the import lock for so long? Presumably the import lock is there to protect access to things like sys.modules. Is that all? Could it be released after the module code is loaded and sys.modules has been updated, but before executing the module body?

-- Greg

More information about the Python-Dev mailing list