| msg283150 - (view) |
Author: Sean Murphy (Sean Murphy) |
Date: 2016-12-14 00:51 |
| Python fails to pass a Queue when calling Process with multiprocessing.set_start_method set to "spawn" or "forkserver". Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.5/multiprocessing/spawn.py", line 106, in spawn_main exitcode = _main(fd) File "/usr/lib/python3.5/multiprocessing/spawn.py", line 116, in _main self = pickle.load(from_parent) File "/usr/lib/python3.5/multiprocessing/synchronize.py", line 111, in __setstate__ self._semlock = _multiprocessing.SemLock._rebuild(*state) FileNotFoundError: [Errno 2] No such file or directory Here is a minimized example: ``` #!/usr/bin/env python3 import multiprocessing def check_child(q): print("Queue", q) if __name__ == '__main__': multiprocessing.set_start_method('spawn') # multiprocessing.set_start_method('fork') # multiprocessing.set_start_method('forkserver') q = multiprocessing.Queue(-1) print("q", q) proc = multiprocessing.Process(target=check_child, args=(q,)) proc.start() ``` Also, this fails when the Queue is implicitly passed to the child. ``` class Blerg(): def __init__(self): self.q = multiprocessing.Queue(-1) def print_queue(self): print("Queue", self.q) if __name__ == '__main__': multiprocessing.set_start_method('spawn') blerg = Blerg() blerg.print_queue() proc = multiprocessing.Process(target=blerg.print_queue) proc.start() ``` $ python3 --version Python 3.5.2 Windows (which defaults to "spawn" style multiprocessing) does not seem to have this issue (at least in 2.7.12). |
|
|
| msg283305 - (view) |
Author: kumaar.nd (kumaar.nd) |
Date: 2016-12-15 13:58 |
| hi, few comments on this: [when tested with 'spawn'] 1. the document(3x, 17.2.1.2) clearly mentions that 'spawn' doesnt inherit the file-handles (hence FileNotFoundError). 2. the document's examples has spawn() followed by join() where parent waits for the Child to finish and exits gracefully. The error "OSError: [Errno 9] Bad file descriptor" is seen for Child process. 3. Isnt exception-handling for null file descriptors sufficient (with corresponding Document updates) am not a core developer yet and still trying to learn and contribute. Thanks |
|
|
| msg298871 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2017-07-22 21:21 |
| Ok, this is really an error in your script. If you add `p.join()` to wait for the process to end, you'll see no error pops up. What really happens is that the parent process ends before the child is fully started, and the resource underlying the queue is cleaned up at this point. |
|
|
| msg298872 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-07-22 21:25 |
| More exactly, since the child process is not "daemonic", the parent process waits for it to end, but after collecting the queue's resources. |
|
|
| msg298873 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-07-22 21:26 |
| So, while it might be possible to improve on this, it would risk fragilizing other use cases, and besides, this is not a problem that is likely to happen in real usage. Therefore, I'd rather close this issue. |
|
|
| msg365280 - (view) |
Author: Mouse (mouse07410) |
Date: 2020-03-29 21:52 |
| On MacOS Catalina 10.15.4, I still see this problem occasionally even with p.join() added. See https://bugs.python.org/msg365251 and subsequent messages. Also, see https://bugs.python.org/issue40106. |
|
|