| msg111827 - (view) |
Author: Nikolaus Rath (nikratio) * |
Date: 2010-07-28 16:03 |
| The attached test program calls apply_async with a function that will raise CalledProcessError. However, when result.get() is called, it raises a TypeError and the program hangs: $ ./bug.py ERROR:root:ops Traceback (most recent call last): File "./bug.py", line 19, in run_with dW1 = run_dcon() File "./bug.py", line 26, in run_dcon subprocess.check_call(['dcon'], stdout=fh, stderr=fh) File "/usr/lib/python2.6/subprocess.py", line 498, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['dcon']' returned non-zero exit status 127 Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results task = get() TypeError: ('__init__() takes exactly 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ()) |
|
|
| msg112118 - (view) |
Author: ysj.ray (ysj.ray) |
Date: 2010-07-31 07:48 |
| I got "OSError: [Errno 2] No such file or directory" instead of TypeError when running the "bug.py", did I miss something? |
|
|
| msg112150 - (view) |
Author: Jesse Noller (jnoller) *  |
Date: 2010-07-31 13:08 |
| @ray - you probably don't have the "dcon" binary on your path. bug.py calls a subprocess call. |
|
|
| msg112158 - (view) |
Author: Nikolaus Rath (nikratio) * |
Date: 2010-07-31 15:36 |
| @ray: Try it with the following dummy dcon program: $ cat dcon #!/bin/sh exit 127 (and change the path to dcon in bug.py accordingly). |
|
|
| msg112329 - (view) |
Author: ysj.ray (ysj.ray) |
Date: 2010-08-01 14:04 |
| This is because when an subprocess.CalledProcessError is raised, the CalledProcessError instance is picked through a socket and then read by "parent" process, but in fact CalledProcessError instances cannot be picked correctly. Because CalledProcessError extends PyExc_BaseException, which defines a __reduce__ method, that special method cause the pickle load to call the exception type's __init__ method with packed self.args as arguments. So if a subclass of "Exception" needs to behave correctly in pickling, it should make self.args meats its __init__ method's function signature. That is, ensure the calling to: self.__init__(*self.args) has no problem. But CalledProcessError doesn't meat this requirement. Here is a patch fixing the CalledProcessError.__init__ to call its base class's __init__ to store its arguments in self.args, which can fix this bug. |
|
|
| msg157433 - (view) |
Author: Daniel Blanchard (Daniel.Blanchard) |
Date: 2012-04-03 17:52 |
| I believe I'm still encountering this issue in 2.7: Exception in thread Thread-3: Traceback (most recent call last): File "/opt/python/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner self.run() File "/opt/python/2.7/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs) File "/opt/python/2.7/lib/python2.7/multiprocessing/pool.py", line 347, in _handle_results task = get() TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ()) |
|
|
| msg157444 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * |
Date: 2012-04-03 21:11 |
| Here is another patch, with test. |
|
|
| msg159733 - (view) |
Author: Daniel Blanchard (Daniel.Blanchard) |
Date: 2012-04-30 19:59 |
| The patch appears to fix the issue, so is there any chance of this actually getting accepted this time? It seems bizarre that such a simple bug that has been on the books for almost two years now can't get a patch accepted. |
|
|
| msg159752 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2012-05-01 11:57 |
| There are plenty of other "bad" exception classes apart from CalledProcessError, including TimeoutExpired in the same file. In fact I suspect this is true of the majority of the exception classes in the stdlib which override __init__. So I am not sure how much good it would do to fix just one example. Python 3.x's Pool wraps bad exception instances in a MaybeEncodingError class which at least lets you see a stringification of the original exception. I am not sure whether you would want to see a backport of this. Even though 2.7 is in bug fix mode, I think a backport would still be appropriate since it stops a pickling error from killing a worker process, causing a hang. |
|
|
| msg159799 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2012-05-02 15:39 |
| New changeset 26bbff4562a7 by Richard Oudkerk in branch '2.7': Issue #9400: Partial backport of fix for #9244 http://hg.python.org/cpython/rev/26bbff4562a7 |
|
|
| msg159804 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2012-05-02 15:49 |
| I have backported the fix for issue #9244 to 2.7. This should fix the hang and produce a traceback containing a representation of the original error. |
|
|
| msg176431 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2012-11-26 17:30 |
| I am reopening this issue because 26bbff4562a7 only dealt with objects which cannot be pickled. But CalledProcessError instances *can* be pickled: the problem is that the resulting data cannot be unpickled. Note that in Python 3.3 CalledProcessError can be pickled then unpickled because of the fix for #1692335. But there are other types which will still produce "corrupt" pickles. |
|
|
| msg274188 - (view) |
Author: Doug Coleman (Doug Coleman) |
Date: 2016-09-01 23:30 |
| Six years later and I'm still running into this exact bug with ``subprocess.CalledProcessError`` on python 2.7.12 when doing a ``multiprocessing.Pool.map`` and trying to catch errors from ``subprocess.check_output``. What's the reason it was never fixed and backported? |
|
|
| msg274630 - (view) |
Author: Stanislaw Pitucha (viraptor) |
Date: 2016-09-06 21:07 |
| Confirming this on python 2.7.12 on DragonflyBSD |
|
|
| msg274636 - (view) |
Author: Davin Potts (davin) * |
Date: 2016-09-06 21:34 |
| For clarity: * Confirmed, can repro on 2.7.12 on OSX. * Also confirmed that this can not be reproduced on the 3.5 or 3.6 branches. |
|
|
| msg274638 - (view) |
Author: Davin Potts (davin) * |
Date: 2016-09-06 21:54 |
| Reading through provides a sense of the concerns surrounding the patches they applied to the 3.x branches. Attempting to backport those patches to the 2.7 branch involves non-trivial risk, magnified by the now numerous differences between those branches. If we had a reasonable workaround to use in 2.7, that would simplify things. Example workaround that produces behavior in 2.7.12 very like what's observed when using 3.5 or 3.6: try: subprocess.check_call(['dcon'], stdout=fh, stderr=fh) except subprocess.CalledProcessError as cpe: raise Exception(str(cpe)) |
|
|
| msg275423 - (view) |
Author: Davin Potts (davin) * |
Date: 2016-09-09 20:19 |
| Richard already backported the fix for objects which can not be pickled but the second fix (applied to 3.3 in #1692335) would be problematic to similarly backport to 2.7. Because there are reasonable workarounds for this in 2.7, going ahead with marking this closed. |
|
|
| msg333232 - (view) |
Author: Marc Schlaich (schlamar) * |
Date: 2019-01-08 14:41 |
| Davin, why isn't just one of the simple one-line patches applied. This would be much more reasonable instead of closing this as won't fix. This is a really ugly bug and it took me hours to find the cause... |
|
|