Issue 30198: distutils build_ext: don't run newer_group() in parallel in multiple threads when using parallel (original) (raw)

Since Python 3.5, distutils is able to build extensions in parallel, nice enhancement! But setup.py of CPython is 2x slower in parallel mode when all modules are already built: (1 sec vs 500 ms).

Building extensions calls newer_group() which calls os.stat() 6,856 times. I wrote a cache for os.stat() but it has no impact on performance.

It seems like threads are fighting to death for the GIL in the os.stat() race...

Attached pull request calls newer_group() before spawning threads in parallel mode, so "setup.py build" takes the same time with and without parallel module, when all extensions are already built.

I didn't measure performance when all extensions must be built.