[Python-Dev] Dealing with import lock deadlock in Import Hooks (original) (raw)
Arnaud Fontaine arnaud.fontaine at nexedi.com
Mon Aug 12 09:39:36 CEST 2013
- Previous message: [Python-Dev] Reaping threads and subprocesses
- Next message: [Python-Dev] Dealing with import lock deadlock in Import Hooks
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
[I initially posted this email to python-list but didn't get any reply, probably because this is too related to python core, so I'm posting it again here, hope that's ok...]
Hello,
I'm currently working on implementing Import Hooks (PEP302) with Python 2.7 to be able to import modules whose code is in ZODB. However, I have stumbled upon a widely known issue about import deadlock[0][1] (note that this issue is not directly related to ZODB, but a more general question about dealing with import lock deadlock for Import Hooks), basically:
Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a package containing dynamic modules) handled by Import Hooks I implemented, so import lock is acquired before even running the hooks (Python/import.c:PyImport_ImportModuleLevel()). Then, these import hooks try to load objects from ZODB and a request is sent and handled by another thread (Thread 2) which itself tries to import another module. Of course, this causes a deadlock because the first thread still holds import lock.
I have thought about the following solutions:
Backport the patch applied in python 3.3 from issue 9260[0]. This would be the best option because it would mean that even when trying to import any module from package 'foo', other modules and packages can be imported, which would solve my issue. However, I'm not sure it could be released into python 2.7?
Within the hooks, protect the Import Hooks with a separate lock for the loader method. This would prevent any other thread to import any modules from 'foo' package but still allows to call the finder method (ignoring module fullname not starting with 'foo') along with other finder methods, so that other ZODB modules can be imported.
Then, in the loader method, until the module is actually inserted into sys.modules and then other load_module() PEP302 responsabilities being taken care of (such as exec the code), release the import lock so that Thread 2 can process requests and send objects back to Thread 1.
About the finder method, I think that the separate lock is enough and releasing the import lock until the end of the method should be enough.
However, even after trying to understand import.c, I'm not sure this is enough and that releasing import lock would not have nasty side-effects, any thoughts about that?
Fix the ZODB code to not avoid import but to me this seems like a dirty hack because it could happen again and I would prefer to fix this issue once and for all.
Any thoughts or suggestion welcome, thanks!
Regards, Arnaud Fontaine
- Previous message: [Python-Dev] Reaping threads and subprocesses
- Next message: [Python-Dev] Dealing with import lock deadlock in Import Hooks
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]