[R-sig-hpc] parallel and openblas (original) (raw)

Dirk Eddelbuettel edd at debian.org
Wed Apr 25 16:50:01 CEST 2012

[ Sylvestre: I am tossing you in the middle of a thread here. We may have a buglet in OpenBLAS where NO_AFFINITY=1 might be a good config value to add. ]

Steve,

Nice work, and mostly confirming against OpenBLAS, Atlas and a local (old) GotoBLAS2.

On 25 April 2012 at 10:24, Stephen Weston wrote: | I was able to confirm that when I built R using OpenBLAS on my | Linux machine, my CPU affinity was modified right at the | beginning of the R session: || $ grep Cpusallowed /proc/self/status | Cpusallowed: ffffffff,ffffffff | Cpusallowedlist: 0-63 | $ bin/R | > readLines('/proc/self/status')[32] | [1] "Cpusallowed:\t00000000,00000001"

What kernel is that? On 3.0.0-17 (Ubuntu 11.10, infrequently rebooted) I get

edd at max:$ grep Cpus_allowed /proc/self/status Cpus_allowed: ff Cpus_allowed_list: 0-7 edd at max:$

| I then confirmed that this causes problems for parallel packages | such as "parallel" by trying to use all six cores of my machine | using the "mclapply" function: || > library(parallel) | > cores <- detectCores()_ | _> mclapply(1:cores, function(i) repeat sqrt(3.14159), mc.cores=cores) || When I executed "top" from another window and pressed "1", it | showed that only one core was being used, and there were six R | sessions, each getting 17% of the CPU.

When I run these three commands as a single line for r (from the littler package)

edd at max:$ r -e 'library(parallel); cores <- detectCores(); print(cores); mclapply(1:cores, function(i) repeat sqrt(3.14159), mc.cores=cores)' [1] 8 ^C edd at max:$

I also get just one core covered. That is with

edd at max:$ COLUMNS=94 dpkg -l|grep "blas|atlas" | cut -c-78 ii gotoblas2-helper 0.1-12.local.1 GotoBLAS2 helper ii libblas-dev 1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, st ii libblas-test 1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, te ii libblas3gf 1.2.20110419-2ubu Basic Linear Algebra Reference impleme ii libopenblas-base 0.1alpha2.2-3 Optimized BLAS (linear algebra) librar ii libopenblas-dev 0.1alpha2.2-3 Optimized BLAS (linear algebra) librar edd at max:$

where OpenBLAS provides BLAS as default.

That was after I had removed Atlas which is still my default. So if I reiinstall Atlas (which "ranks higher" in the defaults and hence replaces OpenBLAS) everything is fine -- eight cores used.

edd at max:$ COLUMNS=94 dpkg -l|grep "blas|atlas" | cut -c-78 ii gotoblas2-helper 0.1-12.local.1 GotoBLAS2 helper ii libatlas-base-dev 3.8.4-3build1 Automatically Tuned Linear Algebra Sof ii libatlas-dev 3.8.4-3build1 Automatically Tuned Linear Algebra Sof ii libatlas3gf-base 3.8.4-3build1 Automatically Tuned Linear Algebra Sof ii libblas-dev 1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, st ii libblas-test 1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, te ii libblas3gf 1.2.20110419-2ubu Basic Linear Algebra Reference impleme ii libopenblas-base 0.1alpha2.2-3 Optimized BLAS (linear algebra) librar ii libopenblas-dev 0.1alpha2.2-3 Optimized BLAS (linear algebra) librar edd at max:$

| I also confirmed that "Cpusallowed" was being set to the same | value for each of the workers: || > mclapply(1:cores, function(i) readLines('/proc/self/status')[32], | mc.cores=cores) | [[1]] | [1] "Cpusallowed:\t00000000,00000001" || [[2]] | [1] "Cpusallowed:\t00000000,00000001" || [[3]] | [1] "Cpusallowed:\t00000000,00000001" || [[4]] | [1] "Cpusallowed:\t00000000,00000001" || [[5]] | [1] "Cpusallowed:\t00000000,00000001" || [[6]] | [1] "Cpusallowed:\t00000000,00000001" || That is definitely not what you want to see, and explains why | "mclapply" is only able to use one core. || When I rebuilt and reinstalled OpenBLAS after editing | Makefile.rule so that it contained the line: || NOAFFINITY = 1 || and then restarted R, the problem went away: || $ bin/R | > readLines('/proc/self/status')[32] | [1] "Cpusallowed:\tffffffff,ffffffff" || This time when I ran "mclapply", "top" confirmed that I was | using all six cores at about 100%. || I didn't try this experiment with the older GotoBLAS2, but I | believe the results would be the same.

I can confirm this. Using the packages

edd at max:$ COLUMNS=94 dpkg -l|grep "blas|atlas" | cut -c-78 ii gotoblas2 1.13-1 GotoBLAS2 ii gotoblas2-helper 0.1-12.local.1 GotoBLAS2 helper ii libblas-dev 1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, st ii libblas-test 1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, te ii libblas3gf 1.2.20110419-2ubu Basic Linear Algebra Reference impleme edd at max:$

where the GotoBLAS2 (locally built, using the gotoblas2-helper package) now provide BLAS, everything sticks to one core when running the mclapply.

I guess I'd need to fix gotoblas2-helper and rebuild the gotoblas2. Or stick with / hope for a corrected OpenBLAS build.

Dirk

| - Steve ||| On Tue, Apr 24, 2012 at 8:37 PM, Dirk Eddelbuettel <edd at debian.org> wrote: | > | > On 24 April 2012 at 15:45, Stephen Weston wrote: | > | There's an interesting discussion entitled "all processes run on | > | one CPU core" at: | > | | > |     https://github.com/ipython/ipython/issues/840 | > | | > | Someone was experiencing a very similar problem to the one that | > | Claudia described using GotoBLAS2 with IPython and NumPy. | > | Apparently it was fixed by recompiling GotoBLAS2 with the | > | "NOAFFINITY" parameter set to "1" in Makefile.rule, and then | > | rebuilding "NumPy". | > | | > | It seems pretty strange, but GotoBLAS2/OpenBLAS may be modifying | > | the affinity of the R process by calling schedsetaffinity() when | > | it is initialized, and that is causing the problems that Claudia | > | and Martin have seen. | > | | > | So perhaps the solution is to recompile GotoBLAS2/OpenBLAS with | > | NOAFFINITY=1, and then rebuild R with it. | > | > Good discussion, but one important nit: never a need to rebuild a R (provided | > you have external / dynamically linked BLAS). | > | > Just restart R. | > | > Dirk | > | > -- | > R/Finance 2012 Conference on May 11 and 12, 2012 at UIC in Chicago, IL | > See agenda, registration details and more at http://www.RinFinance.com

-- R/Finance 2012 Conference on May 11 and 12, 2012 at UIC in Chicago, IL See agenda, registration details and more at http://www.RinFinance.com

More information about the R-sig-hpc mailing list