| msg134907 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2011-05-01 13:25 |
| Hello, I found this bit in my inbox, I forgot why I cared about it, but it raises an exception (at least on Windows): >>> import multiprocessing >>> p = multiprocessing.Process(target=bytes.maketrans, args=(b'abc', b'xyz')) >>> p.start() Traceback (most recent call last): File "C:\Python32\Lib\pickle.py", line 679, in save_global klass = getattr(mod, name) AttributeError: 'module' object has no attribute 'maketrans' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in File "C:\Python32\Lib\multiprocessing\process.py", line 130, in start self._popen = Popen(self) File "C:\Python32\Lib\multiprocessing\forking.py", line 267, in __init__ dump(process_obj, to_child, HIGHEST_PROTOCOL) File "C:\Python32\Lib\multiprocessing\forking.py", line 190, in dump ForkingPickler(file, protocol).dump(obj) File "C:\Python32\Lib\pickle.py", line 237, in dump self.save(obj) File "C:\Python32\Lib\pickle.py", line 344, in save self.save_reduce(obj=obj, *rv) File "C:\Python32\Lib\pickle.py", line 432, in save_reduce save(state) File "C:\Python32\Lib\pickle.py", line 299, in save f(self, obj) # Call unbound method with explicit self File "C:\Python32\Lib\pickle.py", line 623, in save_dict self._batch_setitems(obj.items()) File "C:\Python32\Lib\pickle.py", line 656, in _batch_setitems save(v) File "C:\Python32\Lib\pickle.py", line 299, in save f(self, obj) # Call unbound method with explicit self File "C:\Python32\Lib\pickle.py", line 683, in save_global (obj, module, name)) _pickle.PicklingError: Can't pickle : it's not found as __main__.maketrans If you do the same things with `threading.Thread`, it works, but for `multiprocessing.Process` it doesn't. Is this a general problem with pickling "unbound methods"? |
|
|
| msg135392 - (view) |
Author: Terry J. Reedy (terry.reedy) *  |
Date: 2011-05-07 00:33 |
| Running on winxp with IDLE, I get the second traceback, all the same after the first line. Given "target is the callable object to be invoked by the run() method.", I would have expected this to work too. Problem is not builtins: class C: def f(s): print( 'here') p = multiprocessing.Process(target=C.f, args=(C(),)) p.start() gives same error, ending in _pickle.PicklingError: Can't pickle <function f at 0x00FE5AE0>: it's not found as __main__.f |
|
|
| msg135460 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2011-05-07 13:05 |
| Do things like this in the REPL are known not to work. Things are not protected in the if __name__ block so we can import the script properly to run it. |
|
|
| msg136270 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2011-05-19 09:15 |
| Test attached. |
|
|
| msg136935 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2011-05-26 09:51 |
| Why is this still marked as "test needed"? |
|
|
| msg136936 - (view) |
Author: Ezio Melotti (ezio.melotti) * |
Date: 2011-05-26 09:54 |
| The test should be a diff against Lib/test/test_multiprocessing.py that adds a unit test. |
|
|
| msg137044 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2011-05-27 10:59 |
| Diff attached, is it good? I'm not very experienced with diffs, I usually work with pull requests. |
|
|
| msg137045 - (view) |
Author: Petri Lehtinen (petri.lehtinen) * |
Date: 2011-05-27 11:21 |
| Your patch is good in this case, as the person who applies the patch knows which file is affected. In the future, use diff -u original_file modified_file to get a unified diff. It's the "de facto" format for patches. |
|
|
| msg137046 - (view) |
Author: Petri Lehtinen (petri.lehtinen) * |
Date: 2011-05-27 11:28 |
| Actually, I only commented on the patch format and not on the actual contents of the patch, sorry :) Your test method missed the self parameter, and the test case needed to be added to the testcases_other list for the test to be actually run. I attached a corrected patch. And, now that I tested it, the test runs correctly on Linux, so maybe this is a Windows specific issue? |
|
|
| msg137061 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2011-05-27 15:26 |
| Thanks for the `-u` tip and the correction to the code, Petri. I removed my previous files since yours is the definite one. And yeah, it's a Windows issue. |
|
|
| msg137066 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2011-05-27 15:48 |
| Jesse, I do not understand your comment, including 'REPL' |
|
|
| msg137079 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2011-05-27 16:31 |
| REPL is the Python interactive prompt in this case (REPL is Read Eval Print Loop). So Jesse is saying that using multiprocessing from the REPL (at least on Windows) isn't supported. This is because on Windows multiprocessing needs to re-import the main program in order to start the function in a worker process. When using the REPL, there is no main program from which to import the function. The test case failing on windows may or may not be a related issue; I'm not familiar enough with mulitprocessing to say. |
|
|
| msg137090 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2011-05-27 17:57 |
| After looking at the doc chapter, I get that 'if __name__' block is needed on Windows. OK. batch mode with if __name__ block: testmp.py --------- print('Top of Module') class C: def f(s): print('Method C.f') if __name__ == '__main__': print('Start of main block') import multiprocessing p = multiprocessing.Process(target=C.f, args=(C(),)) p.start() p.join() command-prompt..\python32> python @misc2\testmp.py output (similar to that of OP): Top of Module Start of main block Traceback (most recent call last): File "C:\Programs\Python32\lib\pickle.py", line 679, in save_global klass = getattr(mod, name) AttributeError: 'module' object has no attribute 'f' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "@misc2\testmp.py", line 9, in p.start() File "C:\Programs\Python32\lib\multiprocessing\process.py", line 130, in start self._popen = Popen(self) File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 267, in __init__ dump(process_obj, to_child, HIGHEST_PROTOCOL) File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 190, in dump ForkingPickler(file, protocol).dump(obj) File "C:\Programs\Python32\lib\pickle.py", line 237, in dump self.save(obj) File "C:\Programs\Python32\lib\pickle.py", line 344, in save self.save_reduce(obj=obj, *rv) File "C:\Programs\Python32\lib\pickle.py", line 432, in save_reduce save(state) File "C:\Programs\Python32\lib\pickle.py", line 299, in save f(self, obj) # Call unbound method with explicit self File "C:\Programs\Python32\lib\pickle.py", line 623, in save_dict self._batch_setitems(obj.items()) File "C:\Programs\Python32\lib\pickle.py", line 656, in batch_setitems save(v) File "C:\Programs\Python32\lib\pickle.py", line 299, in save f(self, obj) # Call unbound method with explicit self File "C:\Programs\Python32\lib\pickle.py", line 683, in save_global (obj, module, name)) _pickle.PicklingError: Can't pickle <function f at 0x00B4B4B0>: it's not found as __main__.f C:\Programs\Python32>Top of Module Traceback (most recent call last): File "", line 1, in File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 370, in main self = load(from_parent) EOFError The essential problem seems to be the attempt to pickle the method attribute of the class as a class attribute of the module. |
|
|
| msg224242 - (view) |
Author: Mark Lawrence (BreamoreBoy) * |
Date: 2014-07-29 20:04 |
| This works perfectly on 64 bit Windows 8.1 for 3.4.1 and 3.5.0a0. |
|
|
| msg224243 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2014-07-29 20:07 |
| Confirmed here it's working in Python 3.4, I guess it was fixed sometime in the last few years. I guess the only thing we'd care about now is ensuring a test for this was added to the test suite, so there wouldn't be a regression. Can anyone confirm that? |
|
|
| msg224277 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2014-07-30 00:18 |
| Pickling of builtin functions and methods was indeed improved thanks to __qualname__ support. Closing. |
|
|