| msg71274 - (view) |
Author: Mark Dickinson (mark.dickinson) *  |
Date: 2008-08-17 13:42 |
| Here's a report from Ismail Donmez (cartman), extracted from the issue 3419 discussion in an effort to avoid putting multiple problems under one tracker issue. [cartman] With trunk when running test_multiprocessing in a tight loop I saw another problem: test_multiprocessing Process Process-61: Traceback (most recent call last): File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 229, in _bootstrap Process Process-60: Traceback (most recent call last): File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 229, in _bootstrap Process Process-62: Traceback (most recent call last): File "/Users/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 229, in _bootstrap util._run_after_forkers() File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 138, in _run_after_forkers util._run_after_forkers() File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 138, in _run_after_forkers util._run_after_forkers() File "/Users/cartman/Sources/py3k/Lib/multiprocessing/util.py", line 138, in _run_after_forkers items = list(_afterfork_registry.items()) items = list(_afterfork_registry.items()) File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items items = list(_afterfork_registry.items()) File "/Users/cartman/Sources/py3k/Lib/weakref.py", line 103, in items for key, wr in self.data.items(): RuntimeError: dictionary changed size during iteration for key, wr in self.data.items(): RuntimeError: dictionary changed size during iteration for key, wr in self.data.items(): RuntimeError: dictionary changed size during iteration |
|
|
| msg71343 - (view) |
Author: Ismail Donmez (donmez) * |
Date: 2008-08-18 15:52 |
| I don't know if this is reproducible in 2.6 but I can reproduce on py3k branch, and also thanks for creating the bug! |
|
|
| msg71346 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-08-18 16:12 |
| Mmmh, the problem with the list(X.items()) idiom is that it's thread-safe only if X.items() is implemented in C. Otherwise X can be mutated if there is a thread-switch while executing bytecode in X.items(). In weakref.py (line 103), by replacing: for key, wr in self.data.items(): with: for key, wr in list(self.data.items()): This particular error should disappear. But this doesn't say why the dictionary is mutated at all. Does multiprocessing (or at least that particular test) launch several threads in a given process? Otherwise there may be something fishy going on. |
|
|
| msg71366 - (view) |
Author: Ismail Donmez (donmez) * |
Date: 2008-08-18 19:37 |
| py3k branch gives another error now, when running test_multiprocessing in a tight loop: test test_multiprocessing failed -- Traceback (most recent call last): File "/Users/cartman/Sources/py3k/Lib/test/test_multiprocessing.py", line 1163, in test_remote queue = manager2.get_queue() File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", line 641, in temp authkey=self._authkey, exposed=exp File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", line 893, in AutoProxy incref=incref) File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", line 702, in __init__ self._incref() File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", line 749, in _incref dispatch(conn, None, 'incref', (self._id,)) File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", line 85, in dispatch raise convert_to_error(kind, result) multiprocessing.managers.RemoteError: ------------------------------------------------------------------------ --- Traceback (most recent call last): File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", line 187, in handle_request result = func(c, *args, **kwds) File "/Users/cartman/Sources/py3k/Lib/multiprocessing/managers.py", line 403, in incref self.id_to_refcount[ident] += 1 KeyError: '7389d0' |
|
|
| msg71367 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2008-08-18 19:40 |
| Ismail, that's the incref bug in issue 3419 |
|
|
| msg71368 - (view) |
Author: Ismail Donmez (donmez) * |
Date: 2008-08-18 19:42 |
| Ah cool, we might be at the end of multiprocessing problems then I guess :-) |
|
|
| msg71370 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-08-18 19:50 |
| Le lundi 18 août 2008 à 19:42 +0000, Ismail Donmez a écrit : > Ismail Donmez <ismail@namtrac.org> added the comment: > > Ah cool, we might be at the end of multiprocessing problems then I guess > :-) Well, not really, it should be diagnosed why the dictionary mutates in the first place. This dictionary is apparently processed in a child process just after a fork() occurred, so logically there shouldn't be several threads running. Unless processing the dictionary itself can create some new threads? |
|
|
| msg94411 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-10-24 12:55 |
| Actually, it's probably a duplicate of #7105. |
|
|