| msg215620 - (view) |
Author: Ivan K (Ivan.K) |
Date: 2014-04-05 16:40 |
| Here is a code. https://gist.github.com/anonymous/9994217 When I run it in freeze and never end. If I run it without pool with loop or joblib - it works fine. Also there is a question on stackoverflow: http://stackoverflow.com/q/22881850/1888017 Python 2.7.6/2.7.3, cpu core i7 if it's important |
|
|
| msg215663 - (view) |
Author: Richard Oudkerk (sbt) *  |
Date: 2014-04-06 18:55 |
| I would guess that the problem is simply that LogisticRegression objects are not picklable. Does the problem still occur if you do not use freeze? |
|
|
| msg215664 - (view) |
Author: Ivan K (Ivan.K) |
Date: 2014-04-06 19:11 |
| Sorry, I'm not sure I describe it correct. Freeze that means = goes fozen, so stop progress. It's do no do anything, but computations still load single core of my cpu for 100% untill I do not kill the python process. But the same code work's fine if executed outside pool, or on the different platforms (like Mac for example), my friend run it successfully, but for me it not works. So it's look like pretty unpredictable issue with conflicting OS, CPU, and Python I'm afraid. |
|
|
| msg215665 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2014-04-06 19:12 |
| Ah, I misunderstood: you meant that it freezes/hangs, not that you used a freeze tool. |
|
|
| msg215666 - (view) |
Author: Ivan K (Ivan.K) |
Date: 2014-04-06 19:13 |
| Yes, I'm not using any tool. Code just not working. |
|
|
| msg215669 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2014-04-06 22:44 |
| Could you try pickling and unpickling the result of func(): import cPickle data = cPickle.dumps(func([1,2,3]), -1) print cPickle.loads(data) |
|
|
| msg215670 - (view) |
Author: Ivan K (Ivan.K) |
Date: 2014-04-06 22:55 |
| Sorry, could you specify what is 'func' ? |
|
|
| msg215671 - (view) |
Author: Ivan K (Ivan.K) |
Date: 2014-04-06 22:56 |
| Sorry, stupdi question. Forget that this is from my gist |
|
|
| msg215672 - (view) |
Author: Ivan K (Ivan.K) |
Date: 2014-04-06 22:59 |
| Yes, it work fine and output was LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, penalty=l2, random_state=None, tol=0.0001) |
|
|
| msg237455 - (view) |
Author: Davin Potts (davin) * |
Date: 2015-03-07 16:18 |
| I am unable to reproduce the described behavior using the script provided in that gist using Python 2.7.9 on OS X 10.10 with scikit-learn 0.15.2, scipy 0.14.0, numpy 1.9.0. For me, the sample code runs happily through to completion. Can the OP still reproduce the behavior? If not, it looks like scikit-learn and scipy have changed in the interim, addressing this behavior. |
|
|
| msg242789 - (view) |
Author: Davin Potts (davin) * |
Date: 2015-05-09 01:27 |
| It is no longer possible to reproduce the described issue. Changes to the relevant libraries since this issue was originally opened appear to have addressed the cause. Going ahead with closing this issue as there have been no further reports from the OP or others suggesting otherwise. |
|
|
| msg242802 - (view) |
Author: Ivan K (Ivan.K) |
Date: 2015-05-09 09:27 |
| Sorry, forget to answer. I think I know the reason. Issue I think placed inside ATLAS library. Issue not reproducable on Mac OS X because it's have a bit different thread model. The only single platform is Linux (not tested on Windows btw). So main guess right now that this code fail if scikit-learn are compiled against a singlethread ATLAS. Then paralleling code brokes. However there is very few information regarding how to build it with Multithreaded ATLAS so I'm still not 100% confident. |
|
|
| msg242832 - (view) |
Author: Davin Potts (davin) * |
Date: 2015-05-09 19:07 |
| @Ivan.K: Can you be more specific about which linux platform you are using? To reproduce, should I use Ubuntu, Fedora, OpenSUSE, ... and which version of that linux distro? |
|
|
| msg242853 - (view) |
Author: Ivan K (Ivan.K) |
Date: 2015-05-10 08:53 |
| This behaviour was reproduced on Ubuntu 12.04 and on Centos 7 (centos image running on vagrant) so I don't expect this is ddistro specific |
|
|
| msg251188 - (view) |
Author: Davin Potts (davin) * |
Date: 2015-09-21 02:55 |
| This appears to be highly dependent upon particular versions of operating systems and specific versions of various libraries, some built with certain options. It does not appear to be an issue in multiprocessing itself. @Ivan.K: Unless there is a clear example of how this issue can be reproduced with multiprocessing, I think the appropriate thing to do is close this issue as not a bug with multiprocessing. |
|
|
| msg255026 - (view) |
Author: Davin Potts (davin) * |
Date: 2015-11-20 22:43 |
| I'm going ahead with closing this issue as it originates outside of multiprocessing and Python itself as well as being highly dependent upon specific combinations of build options and versions of 3rd party libraries (such as ATLAS). |
|
|