| msg301027 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2017-08-30 14:27 |
| It may happen that the forkserver process dies (for example if SIGINT is received because the user pressed Ctrl-C) while the parent process is still alive. In that case, if the parent tries to create a new Process instance, an exception is received. The exception looks like this: File "/xxx/lib/python3.5/multiprocessing/popen_forkserver.py", line 52, in _launch self.sentinel, w = forkserver.connect_to_new_process(self._fds) File "/xxx/lib/python3.5/multiprocessing/forkserver.py", line 66, in connect_to_new_process client.connect(self._forkserver_address) ConnectionRefusedError: [Errno 111] Connection refused |
|
|
| msg301854 - (view) |
Author: Davin Potts (davin) * |
Date: 2017-09-11 03:43 |
| I have two concerns with this: 1) The implicit restart of the forkserver process seems in conflict with the zen of making things explicit. 2) This would seem to make forkserver's behavior inconsistent with the behavior of things like the Manager which similarly creates its own process for managing resources but does not automatically restart that process if it should die or become unreachable. In the case of the Manager, I don't think we'd want it to automagically restart anything in these situations so it's not a simple matter of enhancing the Manager to adopt similar behavior. I do appreciate the use cases that would be addressed by having a convenient way to detect that a forkserver has died and then restart it. If the forkserver dies, I doubt we really want it to try to restart a potentially infinite number of times. Maybe a better path would be if we had a way to explicitly request that the Process trigger a restart of the forkserver, if necessary, but this setting/request defaults to False? |
|
|
| msg301868 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-09-11 09:18 |
| > 1) The implicit restart of the forkserver process seems in conflict with the zen of making things explicit. Well, the forkserver process really is an implementation detail. It already is started on-demand, so restarting it if needed sounded like a natural evolution. Another example: in a multiprocessing Pool, if a worker process crashes, another will be restarted transparently (same in concurrent.futures.ProcessPoolExecutor, incidentally). > 2) This would seem to make forkserver's behavior inconsistent with the behavior of things like the Manager which similarly creates its own process for managing resources but does not automatically restart that process if it should die or become unreachable. That's possible. I've never seen the Manager used in the wild. Though if some fundamental difference justifies the inconsistency, the inconsistency is not really a problem. |
|
|
| msg303458 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-10-01 12:15 |
| Looks like this will not make into 3.6.3. I'd like to sum up the discussion a bit: - I think at the very least least we must protect the forkserver against SIGINT, like the semaphore tracker already is - I think it's beneficial to also restart it on demand if necessary - I can't think of a use case where the user asks "no, please don't restart the forkserver on demand" as there is no way to start it manually anyway, and not restarting it breaks functionality Also, bpo-31310 is a similar issue for the semaphore tracker. |
|
|
| msg304468 - (view) |
Author: Matthew Rocklin (Matthew Rocklin) |
Date: 2017-10-16 12:21 |
| From a user's perspective I would definitely appreciate forkserver being protected from SIGINT. This bug affects me in practice. If the forkserver goes down I personally have no objection to it restarting automatically, though I appreciate that I have a narrow view of this topic. Davin, at your last comment it seemed like you had reservations about this going in. Did Antoine's recent comments resolve these concerns or no? Do you have any suggestions on what might be done to protect users from SIGINT crashing the forkserver? |
|
|
| msg305471 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-11-03 12:11 |
| Davin, I think I'm going to merge the PR for this. If you object it, it can still be reverted later. |
|
|
| msg305474 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-11-03 12:34 |
| New changeset fc6b348b12ad401cab0261b7b71a65c60a08c0a8 by Antoine Pitrou in branch 'master': bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary (#3246) https://github.com/python/cpython/commit/fc6b348b12ad401cab0261b7b71a65c60a08c0a8 |
|
|
| msg305477 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-11-03 12:59 |
| New changeset 019c99f325287741d1e0eefeef2b75c8e00b884f by Antoine Pitrou in branch '3.6': [3.6] bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary (GH-3246) (#4252) https://github.com/python/cpython/commit/019c99f325287741d1e0eefeef2b75c8e00b884f |
|
|