Issue 33171: multiprocessing won't utilize all of platform resources (original) (raw)

I'm using either multiprocessing package or concurrent.futures for some embarrassingly parallel application.

I performed a simple test: basically making n_jobs calls for a simple function - 'sum(list(range(n)))', with n large enough so that the operation is a few seconds long - where n_jobs > n_logical_cores.

Tried it on two platforms:

first platform: server with X4 Intel Xeon E5-4620 (8 physical, 16 logical), running a 64bit Windows Server 2012 R2 Standard.

second platform: server with X2 Intel Xeon Gold 6138 (20 physical, 40 logical), running a 64bit Windows Server 2016 Standard.

first platform reaches 100% utilization. second platform reaches 25% utilization.