| msg305953 - (view) |
Author: Werner Smidt (Werner Smidt) |
Date: 2017-11-09 10:14 |
| Hi there I recently stumbled on an interesting behaviour. I won't call it an error, because I think it's a mistake I made. BACKGROUND: I want to spawn threads that handle pickled data. This works really well. However, I would like to execute the python script in question as a module, i.e. python -m mymodule. This is merely for aesthetic purposes. The attached script has two functions: 1. Pickle/unpickle an instance of a `namedtuple` 2. Pickle/unpickle a string Each of these functions are run in the main thread and then in subsequent spawned threads. If I run the script attached with "python testqueuepickle.py", it works fine. I get the data pickled/unpickled in the respective functions and nothing deadlocks and everything is printed to screen. If, however, I run it with the "-m" option (python -m testqueuepickle.py) , the program deadlocks at the pickling of the "namedtuple" instance. The pickling/unpickling of the string appears to be unaffected. Programming practices aside, what do you think could be the cause of this? |
|
|
| msg382182 - (view) |
Author: Irit Katriel (iritkatriel) *  |
Date: 2020-11-30 19:29 |
| I don't see the hang on Linux (3.7) or windows (3.10). I think this might be a python 2-only issue. There were problems in Python 2 with importer race conditions, and since you start threads at import time, that could be what was causing what you saw. Python 2.7 is no longer maintained, so unless the problems is relevant to Python 3 this issue can be closed. |
|
|
| msg382190 - (view) |
Author: Werner Smidt (Werner Smidt) |
Date: 2020-11-30 20:30 |
| The condition still stands. if I execute: python3 testqueuepickle3.py Everything is fine. If, however I execute: python3 -m testqueuepickle3.py It hangs. |
|
|
| msg382191 - (view) |
Author: Irit Katriel (iritkatriel) * |
Date: 2020-11-30 20:54 |
| Which system are seeing this on? |
|
|
| msg382229 - (view) |
Author: Sara Kelley (serakeri) |
Date: 2020-12-01 10:30 |
| >python3 -m testqueuepickle3.py >It hangs. I see this hang too. If you only specify the module name it does not hang. python3 -m testqueuepickle3 |
|
|
| msg382239 - (view) |
Author: Werner Smidt (Werner Smidt) |
Date: 2020-12-01 11:35 |
| Sorry for being lacking in providing some OS info. b Opensuse Tumbleweed, Linux kernel 5.8.10-1, Intel system I cannot explain why, but relating to Sara's answer, if you remove the .join() statements at the end, you get the following exception: /usr/bin/python3: Error while finding module specification for 'testqueuepickle3.py' (ModuleNotFoundError: __path__ attribute not found on 'testqueuepickle3' while trying to find 'testqueuepickle3.py') mynamedtuple(param1='INSIDE thread', param2='namedtuple') So I guess that it gets stuck after an exception is thrown? If I call it Sara's way, there is no exception thrown. Sorry, I know this is a very specific case, but I thank you both for taking the time to contribute :-) |
|
|
| msg382359 - (view) |
Author: Sara Kelley (serakeri) |
Date: 2020-12-03 00:07 |
| https://docs.python.org/3/using/cmdline.html#cmdoption-m The documentation says you must not give a file extension with the module option. Because the extension was given the thread is deadlocking in importlib._bootstrap when trying to acquire the module lock. The hang occurs in the script because join() is waiting for the deadlocked thread. |
|
|
| msg382370 - (view) |
Author: Sara Kelley (serakeri) |
Date: 2020-12-03 03:09 |
| After looking a little more I don't believe the extension is the problem, but it is an easier way to see the problem. When testqueuepickle3 is run as a script you don't see the error. When it is imported as a module, even from another script with valid syntax, it does deadlock. |
|
|
| msg382393 - (view) |
Author: Werner Smidt (Werner Smidt) |
Date: 2020-12-03 09:23 |
| Thanks for going to the trouble, Sara. Curiosity remains, but I'll mark this as closed. |
|
|