Speedup GNU make build and compilation process (original) (raw)

Building/compiling libraries or projects frequently with GNU make takes long time. To speed up these builds you can use make -jN where N is the number of parallel builds. For example:

make -j4

will make 4 parallel builds.

Hardware considerations

Of course, you can’t do something like

make -j256

This is the best way to get your computer hangs !!! You must check your hardware CPU informations. How many processors (cores) you have (including multi-threading):

NB_CORES=$(grep -c '^processor' /proc/cpuinfo)

You can also use dmidecode :

dmidecode -t processor | grep "Core Count"
Core Count: 8
Core Count: 8

here you got two processors (2 times Core Count) with 8 cores. You can also uselspcu command:

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 23
Stepping:              7
CPU MHz:               1998.000
BogoMIPS:              4999.98
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              3072K
NUMA node0 CPU(s):     0-3

System considerations

Once you get your number of cores, we must keep a responsive system. We use -l options of make -l [load], –load-average[=load] {Specifies that no new jobs (commands) should be started if there are others jobs running and the load average is at least} load

To get a full saturation of the processors without hanging your system, export the following environment variable MAKEFLAGS:

NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
export MAKEFLAGS="-j$((NB_CORES+1)) -l${NB_CORES}"
make

Or:

NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
make -j$((NB_CORES+1)) -l${NB_CORES}

It means here that the load average does not exceed number of cores.

Persistent settings

To get these settings persistent, just add in your .bashrc or .bash_profile:

NB_CORES=$(grep -c '^processor' /proc/cpuinfo)
export MAKEFLAGS="-j$((NB_CORES+1)) -l${NB_CORES}"

At last a commandline example

I you dont want particular settings , just use commandline:

make -j9 -l8

You can usually get away with having twice as many jobs as you do CPUs if you have a decent amount of RAM. If your CPU supports hyper-threading then you can often get by with twice as many again. That’s 128 jobs if you have a 32 dual core machine with hyper-threading! (Thanks to Andrew Stormont for his contribution)