Issue 17836: multiprocessing exceptions with useful traceback (original) (raw)

In pool.py, the worker function reads as follows:

http://svn.python.org/view/python/trunk/Lib/multiprocessing/pool.py?view=markup """ 68 job, i, func, args, kwds = task 69 try: 70 result = (True, func(*args, **kwds)) 71 except Exception, e: 72 result = (False, e) ... 488 if self._success: 489 return self._value 490 else: 491 raise self._value """

If an exception is raised in the function you sent to the pool, the exception you get has "raise self._value" as the last line; which is correct but useless for debugging.

mp_exception_bug.py reproduces this error. This is the output:

""" Traceback (most recent call last): File "mp_exception_bug.py", line 8, in print p.map(f, [1,2,3]) File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map return self.map_async(func, iterable, chunksize).get() File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get raise self._value NameError: global name 'y' is not defined """

As you can imagine, "NameError: global name 'y' is not defined" is not enough in complex projects.

If we apply some changes to the pool.py we could get something similar to this:

""" Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/pool.py", line 98, in worker result = (True, func(*args, **kwds)) File "/usr/lib/python2.7/multiprocessing/pool.py", line 67, in mapstar return map(args) File "mp_exception_bug.py", line 4, in f return xy NameError: global name 'y' is not defined

Traceback (most recent call last): File "mp_exception_bug.py", line 8, in print p.map(f, [1,2,3]) File "/usr/lib/python2.7/multiprocessing/pool.py", line 231, in map return self.map_async(func, iterable, chunksize).get() File "/usr/lib/python2.7/multiprocessing/pool.py", line 535, in get raise self._value[0] NameError: global name 'y' is not defined """

The patch is simple but ugly:

""" > import sys > import traceback 72c100,102 < result = (False, e)

        exc_info = sys.exc_info()
        tb_string = traceback.format_exc(exc_info[2])
        result = (False, (e, tb_string))

491c532,535 < raise self._value

        # Do something with the exception here, the simplest (but ugliest)
        # thing to do is to simply print it to the console
        print self._value[1]
        raise self._value[0]

"""

Note that I tried to replace the "raise self._value[0]" with a raise with three parameters, being the last one the traceback we get using "exc_info = sys.exc_info()" but sadly it is not possible to pickle tracebacks.

I understand that printing is not the best thing to do here, but I wanted to get this discussion started and come to a real solution.

Thanks