| msg69777 - (view) |
Author: Barry A. Warsaw (barry) *  |
Date: 2008-07-16 12:57 |
| _multiprocessing.so has build problems on both OS X 10.5 and Ubuntu Linux 8.04. It's very strange though because there are no apparent errors in compilation, however when the build process tries to import the module, that fails and it gets moved to _multiprocessing_failed.so. Interestingly enough, a subsequent 'make' succeeds, as does just renaming the lib back to _multiprocessing.so. Sorry, I have no more clues than that, but see also but 3088. |
|
|
| msg69778 - (view) |
Author: Barry A. Warsaw (barry) * |
Date: 2008-07-16 12:57 |
| Er, make that bug 3088 |
|
|
| msg69781 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2008-07-16 13:13 |
| I'll try with a clean tree, but I've seen this before and I'm quite mystified. |
|
|
| msg69789 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2008-07-16 14:10 |
| Barry, I can't seem to repro this against trunk on both my Ubuntu and OS/X machines. If you get a chance, can you see if you can get the output from a make -d? |
|
|
| msg69800 - (view) |
Author: Andrii V. Mishkovskyi (mishok13) |
Date: 2008-07-16 16:36 |
| Attached the log of 'make -d' on clean checkout of py3k branch. This is on Ubuntu 8.04.1. |
|
|
| msg69801 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2008-07-16 16:37 |
| Andrii gave me make -d output with the failure: building '_multiprocessing' extension creating build/temp.linux-i686- 3.0/home/mishok/doc/python/tmp/py3k/Modules/_multiprocessing gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall - Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 - DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I. - I/home/mishok/doc/python/tmp/py3k/./Include -I. -IInclude -I./Include - I/usr/local/include -I/home/mishok/doc/python/tmp/py3k/Include - I/home/mishok/doc/python/tmp/py3k -c /home/mishok/doc/python/tmp/py3k/Modules/_multiprocessing/multiprocessin g.c -o build/temp.linux-i686- 3.0/home/mishok/doc/python/tmp/py3k/Modules/_multiprocessing/multiproces sing.o gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall - Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 - DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I. - I/home/mishok/doc/python/tmp/py3k/./Include -I. -IInclude -I./Include - I/usr/local/include -I/home/mishok/doc/python/tmp/py3k/Include - I/home/mishok/doc/python/tmp/py3k -c /home/mishok/doc/python/tmp/py3k/Modules/_multiprocessing/socket_connect ion.c -o build/temp.linux-i686- 3.0/home/mishok/doc/python/tmp/py3k/Modules/_multiprocessing/socket_conn ection.o gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall - Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 - DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I. - I/home/mishok/doc/python/tmp/py3k/./Include -I. -IInclude -I./Include - I/usr/local/include -I/home/mishok/doc/python/tmp/py3k/Include - I/home/mishok/doc/python/tmp/py3k -c /home/mishok/doc/python/tmp/py3k/Modules/_multiprocessing/semaphore.c -o build/temp.linux-i686- 3.0/home/mishok/doc/python/tmp/py3k/Modules/_multiprocessing/semaphore.o gcc -pthread -shared build/temp.linux-i686- 3.0/home/mishok/doc/python/tmp/py3k/Modules/_multiprocessing/multiproces sing.o build/temp.linux-i686- 3.0/home/mishok/doc/python/tmp/py3k/Modules/_multiprocessing/socket_conn ection.o build/temp.linux-i686- 3.0/home/mishok/doc/python/tmp/py3k/Modules/_multiprocessing/semaphore.o -L/usr/local/lib -o build/lib.linux-i686-3.0/_multiprocessing.so *** WARNING: renaming "_multiprocessing" since importing it failed: No module named _multiprocessing |
|
|
| msg69802 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2008-07-16 16:38 |
| Even with that output it's not clear what's happening during the compile step. Barry - is this on trunk and py3k? |
|
|
| msg69804 - (view) |
Author: Barry A. Warsaw (barry) * |
Date: 2008-07-16 17:14 |
| py3k. i think the thing to do is to try to figure out why the import is failing even though the compilation appears to succeed. it's the suppressed import error that's going to be the clue. |
|
|
| msg69833 - (view) |
Author: Barry A. Warsaw (barry) * |
Date: 2008-07-16 21:04 |
| Okay, I have more information, but still no diagnosis. I stuck a 'raise' in the setup.py so that when the ImportError occurs, it doesn't get swallowed. Instead it stops the build process in its tracks. The attached file contains the relevant output. |
|
|
| msg69869 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2008-07-17 04:47 |
| When a second 'make' fixes things, that usually means that there's a dependency on another extension that is built later in the normal build sequence. I vaguely we had this before and fixed it by reordering things. Grepping the sources of _multiprocessing for Import, I wonder if it could be _pickle? Or something that _pickle imports? (Since _pickle seems to be built before _multiprocessing.) |
|
|
| msg69870 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2008-07-17 04:51 |
| Here is the output of python -v during "import _multiprocessing". Maybe this gives someone a clue? >>> import _multiprocessing dlopen("/Users/guido/p3/build/lib.macosx-10.3-i386-3.0/_multiprocessing.so", 2); # /Users/guido/p3/Lib/pickle.pyc matches /Users/guido/p3/Lib/pickle.py import pickle # precompiled from /Users/guido/p3/Lib/pickle.pyc import marshal # builtin # /Users/guido/p3/Lib/struct.pyc matches /Users/guido/p3/Lib/struct.py import struct # precompiled from /Users/guido/p3/Lib/struct.pyc dlopen("/Users/guido/p3/build/lib.macosx-10.3-i386-3.0/_struct.so", 2); import _struct # dynamically loaded from /Users/guido/p3/build/lib.macosx-10.3-i386-3.0/_struct.so dlopen("/Users/guido/p3/build/lib.macosx-10.3-i386-3.0/binascii.so", 2); import binascii # dynamically loaded from /Users/guido/p3/build/lib.macosx-10.3-i386-3.0/binascii.so dlopen("/Users/guido/p3/build/lib.macosx-10.3-i386-3.0/_pickle.so", 2); import _pickle # dynamically loaded from /Users/guido/p3/build/lib.macosx-10.3-i386-3.0/_pickle.so import multiprocessing # directory /Users/guido/p3/Lib/multiprocessing # /Users/guido/p3/Lib/multiprocessing/__init__.pyc matches /Users/guido/p3/Lib/multiprocessing/__init__.py import multiprocessing # precompiled from /Users/guido/p3/Lib/multiprocessing/__init__.pyc # /Users/guido/p3/Lib/multiprocessing/process.pyc matches /Users/guido/p3/Lib/multiprocessing/process.py import multiprocessing.process # precompiled from /Users/guido/p3/Lib/multiprocessing/process.pyc dlopen("/Users/guido/p3/build/lib.macosx-10.3-i386-3.0/itertools.so", 2); import itertools # dynamically loaded from /Users/guido/p3/build/lib.macosx-10.3-i386-3.0/itertools.so import _multiprocessing # dynamically loaded from /Users/guido/p3/build/lib.macosx-10.3-i386-3.0/_multiprocessing.so import _multiprocessing # dynamically loaded from /Users/guido/p3/build/lib.macosx-10.3-i386-3.0/_multiprocessing.so >>> |
|
|
| msg69872 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2008-07-17 05:06 |
| The order thing was a red herring. However, I understand what's going on now. Somebody else can fix it hopefully. So what's going on: In the Python instance that runs setup.py, importing _multiprocessing fails. But if I start a new Python instance in exactly the same environment, it works. Why? Because at the *start* of running setup.py, the directory .../build/lib.macosx-10.3-i386-3.0 doesn't exist, and this is being cached in sys.path_importer_cache, which has a NullImporter instance for that key. Maybe the solution is to remove that cache entry (if it exists) in the code that creates that directory? I found this by (a) disabling the two except clauses in setup.py that catch exceptions from the attempt to import the module that was just built, and (b) adding a -i flag to the Python instance invoked by the Makefile to run setup.py. This gave me an interactive interpreter at the moment the "import _multiprocessing" failed. Poking around I could see that sys.path_importer_cache had a NullImporter instance for the directory from which _multiprocessing.so was to be imported. I could even delete that cache entry, and then the import would pass! Good luck fixing... |
|
|
| msg69882 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-07-17 15:41 |
| How about simply doing sys.path_importer_cache.clear() at the right point in setup.py? I don't think the performance loss would be overwhelming... |
|
|
| msg69886 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2008-07-17 16:26 |
| I've checked in a fix (r65063) that simply clear sys.path_importer_cache right before the attempted import of the freshly-built extension. This seems to work. |
|
|