Issue 5162: multiprocessing cannot spawn child from a Windows service (original) (raw)

I think I've found a small bug with multiprocessing package on Windows. If you try to start a multiprocessing.Process from a Python- based Windows service, the child process will fail to run. When running the parent process as a regular Python program, everything works as expected. I've tracked the problem down to how main_path is prepared in multiprocessing.forking.get_preparation_data() (lines 370-377): def get_preparation_data(name): [...skipped a few lines...] if not WINEXE: main_path = getattr(sys.modules['main'], 'file', None) if not main_path and sys.argv[0] not in ('', '-c'): main_path = sys.argv[0] if main_path is not None: if not os.path.isabs(main_path) and \ process.ORIGINAL_DIR is not None: main_path = os.path.join(process.ORIGINAL_DIR, main_path) d['main_path'] = os.path.normpath(main_path) return d When the program is running as a Windows service, but is not packaged into a single executable, main_path will become the path to the service executable (typically, pythonservice.exe). When this data makes it to the child process, the prepare() function will treat main_path as a path to a python module, and will try to import it. This causes it to fail. My quick-and-dirty solution was to check in get_preparation_data() if main_path ends with '.exe', and if it does, to not pass it at all. This solves the problem in my case, but perhaps there's a better way to fix this? Here is my version of get_preparation_data(): def get_preparation_data(name): ''' Return info about parent needed by child to unpickle process object ''' from .util import _logger, _log_to_stderr d = dict( name=name, sys_path=sys.path, sys_argv=sys.argv, log_to_stderr=_log_to_stderr, orig_dir=process.ORIGINAL_DIR, authkey=process.current_process().authkey, ) if _logger is not None: d['log_level'] = _logger.getEffectiveLevel() if not WINEXE: main_path = getattr(sys.modules['main'], 'file', None) if not main_path and sys.argv[0] not in ('', '-c'): main_path = sys.argv[0] if main_path is not None: if not os.path.isabs(main_path) and \ process.ORIGINAL_DIR is not None: main_path = os.path.join(process.ORIGINAL_DIR, main_path) if not main_path.endswith('.exe'): d['main_path'] = os.path.normpath(main_path) return d

This issue is not fully fixed, there are some occasions where you can still run into it. One example is if you want to spawn a new multiprocessing.Process as sub process in a multiprocessing.Process:

pythonservice.exe

In this case you get:

WINSERVICE: False WINEXE: False _python_exe: C:\Python27\python.exe prep data: {'authkey': '...', 'sys_path': [...], 'name': 'test', 'orig_dir': '...', 'sys_argv': ['C:\Python27\lib\site-packages\win32\PythonService.exe'], 'main_path': 'C:\Python27\lib\site-packages\win32\PythonService.exe', 'log_to_stderr': False}