| msg102539 - (view) |
Author: Stefan Krah (skrah) *  |
Date: 2010-04-07 13:34 |
| On Windows/amd64, I get loads of pickling errors in test_multiprocessing. Type 1 error: Traceback (most recent call last): File "", line 1, in File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 347, in main self = load(from_parent) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 1378, in load return Unpickler(file).load() File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 858, in load dispatch[key](self) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 880, in load_eof raise EOFError EOFError Type 2 error: ====================================================================== ERROR: test_fork (__main__.WithManagerTestQueue) ---------------------------------------------------------------------- Traceback (most recent call last): File "..\..\Lib\test\test_multiprocessing.py", line 485, in test_fork p.start() File "C:\Users\stefan\svn\trunk\lib\multiprocessing\process.py", line 104, in start self._popen = Popen(self) File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 244, in __init__ dump(process_obj, to_child, HIGHEST_PROTOCOL) File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 167, in dump ForkingPickler(file, protocol).dump(obj) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 224, in dump self.save(obj) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce save(state) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 40, in dispatcher self.save_reduce(obj=obj, *rv) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 401, in save_reduce save(args) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 548, in save_tuple save(element) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce save(state) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce save(state) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 396, in save_reduce save(cls) File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 748, in save_global (obj, module, name)) PicklingError: Can't pickle <type 'cStringIO.StringO'>: it's not found as cStringIO.StringO |
|
|
| msg102542 - (view) |
Author: Tim Golden (tim.golden) * |
Date: 2010-04-07 14:48 |
| Seeing the same thing on 32-bit WinXP on x86 On 07/04/2010 14:34, Stefan Krah wrote: > > New submission from Stefan Krah<stefan-usenet@bytereef.org>: > > On Windows/amd64, I get loads of pickling errors in test_multiprocessing. > > Type 1 error: > > Traceback (most recent call last): > File "", line 1, in > File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 347, in main > self = load(from_parent) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 1378, in load > return Unpickler(file).load() > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 858, in load > dispatch[key](self) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 880, in load_eof > raise EOFError > EOFError > > > Type 2 error: > > ====================================================================== > ERROR: test_fork (__main__.WithManagerTestQueue) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "..\..\Lib\test\test_multiprocessing.py", line 485, in test_fork > p.start() > File "C:\Users\stefan\svn\trunk\lib\multiprocessing\process.py", line 104, in start > self._popen = Popen(self) > File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 244, in __init__ > dump(process_obj, to_child, HIGHEST_PROTOCOL) > File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 167, in dump > ForkingPickler(file, protocol).dump(obj) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 224, in dump > self.save(obj) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save > self.save_reduce(obj=obj, *rv) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce > save(state) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save > f(self, obj) # Call unbound method with explicit self > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict > self._batch_setitems(obj.iteritems()) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems > save(v) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save > f(self, obj) # Call unbound method with explicit self > File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 40, in dispatcher > self.save_reduce(obj=obj, *rv) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 401, in save_reduce > save(args) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save > f(self, obj) # Call unbound method with explicit self > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 548, in save_tuple > save(element) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save > self.save_reduce(obj=obj, *rv) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce > save(state) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save > f(self, obj) # Call unbound method with explicit self > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict > self._batch_setitems(obj.iteritems()) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems > save(v) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save > self.save_reduce(obj=obj, *rv) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce > save(state) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save > f(self, obj) # Call unbound method with explicit self > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict > self._batch_setitems(obj.iteritems()) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems > save(v) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save > self.save_reduce(obj=obj, *rv) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 396, in save_reduce > save(cls) > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save > f(self, obj) # Call unbound method with explicit self > File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 748, in save_global > (obj, module, name)) > PicklingError: Can't pickle<type 'cStringIO.StringO'>: it's not found as cStringIO.StringO > > ---------- > components: Library (Lib) > messages: 102539 > nosy: skrah > priority: high > severity: normal > stage: needs patch > status: open > title: test_multiprocessing: pickling failures > type: behavior > versions: Python 2.7 > > _______________________________________ > Python tracker<report@bugs.python.org> > <http://bugs.python.org/issue8333> > _______________________________________ > _______________________________________________ > Python-bugs-list mailing list > Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/mail%40timgolden.me.uk |
|
|
| msg102555 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2010-04-07 18:05 |
| When did this problem start? |
|
|
| msg102556 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2010-04-07 18:54 |
| It was introduced in r79623, and the attached patch makes the problem go away. I would be happy to apply it, but I perhaps you and Michael could analyze if there are unwanted dependencies between multiprocessing and unittest. |
|
|
| msg102561 - (view) |
Author: Michael Foord (michael.foord) * |
Date: 2010-04-07 19:17 |
| I expect the unittest tests will fail with that patch applied however... No problem with switching to just using StringIO in unittest though - preferably with a comment as to why. |
|
|
| msg102564 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2010-04-07 20:51 |
| Perhaps we could run the multiprocessing tests unbuffered? |
|
|
| msg102565 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2010-04-07 20:51 |
| BTW, this is also causing all Windows buildbots to fail. |
|
|
| msg102573 - (view) |
Author: Michael Foord (michael.foord) * |
Date: 2010-04-07 22:49 |
| TestResult creates its StringIO buffers even if they aren't use. I can create them on first use and use StringIO.StringIO rather than cStringIO. Will fix now. |
|
|
| msg102574 - (view) |
Author: Michael Foord (michael.foord) * |
Date: 2010-04-07 22:56 |
| Although merely switching to StringIO may not be enough. TestResult objects keep a reference to sys.stdout when they are instantiated and you can't pickle that either. It is preferable for the reference to be taken at instantiation time rather than stored as a module level reference to be compatible with test runners (like regrtest) that *themselves* replace sys.stdout... |
|
|
| msg102577 - (view) |
Author: Michael Foord (michael.foord) * |
Date: 2010-04-07 23:31 |
| Changing unittest.TestResult to use StringIO instead of cStringIO isn't sufficient. regrtest.py also needs to use StringIO... |
|
|
| msg102579 - (view) |
Author: Michael Foord (michael.foord) * |
Date: 2010-04-08 00:15 |
| I believe this issue to now be fixed on trunk. We can close this when the buildbots are passing again. |
|
|
| msg102581 - (view) |
Author: Michael Foord (michael.foord) * |
Date: 2010-04-08 00:44 |
| Green build on Windows XP, trunk. http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%20trunk/builds/282 |
|
|