Issue 28422: multiprocessing Manager mutable type member access failure (original) (raw)

Accessing some Manager types (e.g. Lock) through a list, dict or Namespace proxy is not possible if both the mutable container (e.g. list) and contained type instance (e.g. Lock) have been created in the same process.

Symptoms:

In accessing process: multiprocessing.managers.RemoteError on access, e.g.: Unserializable message: ('#RETURN', <thread.lock object at 0x7fcf40d31370>)

.. and in underlying manager: _pickle.PicklingError: Can't pickle <class '_thread.lock'>: attribute lookup lock on _thread failed

The provided test script performs: 0) Initialise SyncManager (via multiprocessing.Manager())

  1. Create list proxy through manager
  2. Insert another proxied type into the list (e.g. Lock)
  3. Try to access type instance from (2) via container created in (1)

Note: When step (2) is run in a child process, everything work as expected. When all steps execute in the same process, one gets the aforementioned exception.

See also: https://mail.python.org/pipermail/python-list/2009-September/552988.html

If you un-comment the print_exc() call, you'll see that it still fails - for queue.Queue:

Manager failure (for Queue) Traceback (most recent call last): File "fish.py", line 74, in main add_type_in_own_process(mgr, type_name) File "fish.py", line 61, in add_type_in_own_process use_manager(obj, type_name, mgr=mgr) File "fish.py", line 35, in use_manager obj.append(getattr(mgr, type_name)()) File "", line 2, in append File "/usr/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod raise convert_to_error(kind, result) multiprocessing.managers.RemoteError:

Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/managers.py", line 243, in serve_client request = recv() File "/usr/lib/python3.8/multiprocessing/connection.py", line 251, in recv return _ForkingPickler.loads(buf.getbuffer()) File "/usr/lib/python3.8/multiprocessing/managers.py", line 959, in RebuildProxy return func(token, serializer, incref=incref, **kwds) TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned'