make altinstall for PGO is not parallel-safe · Issue #93584 · python/cpython (original) (raw)

Bug report

setup.py has a helper method add_multiarch_paths. The method creates a temporary file build/lib.{platform}/multiarch and unlinks it at the end of the function call. This is not parallel-safe.

PGO builds of Python use $(MAKE), so called recursive make. Recursive makes are considered harmful because the main make process has no understanding what the child make process is doing. With heavy parallel makes this can cause race conditions.

The combination of unsafe add_multiarch_paths(), recursive make and loooots of CPU cores can lead to build issues like #84461 (comment).

Possible workarounds

1. Rewrite our Makefile to not use $(MAKE)
2. Move CC -print-multiarch and dpkg-architecture ... -qDEB_HOST_MULTIARCH calls to configure.ac
3. Fix _bootsubprocess.check_output() and add_multiarch_paths() to use a more safe tmp file

Option (3) is the simplest approach. Even tmpfile = os.path.join(self.build_temp, f'multiarch-{os.getpid()}') would be good enough to avoid file conflicts in parallel builds.