| msg76848 - (view) |
Author: Brian Thorne (Thorney) |
Date: 2008-12-03 21:58 |
| I found this bug when using scipy's weave and having some inline C/C++ code. Weave uses distutils to compile the source in its own directory, python was installed under "program files". Unfortunately it doesn't seem to handle spaces in the path to either the output or the temporary source files. I checked that this was the only problem by copying the weave directory from site-packages and put it in another directory without spaces in the path - weave and distutils then worked fine. I then tried the g++ command seen in the stack trace at the command line and got the same error - so altered the command by correctly quoting the -c "path/to source/a.c" and the -o objects. This worked so set out changing it in distutils. The two files I altered were unixcompiler.py (surprising as I am under windows!) and cygwinccompiler.py This was clearly an older stable build so I checked out the python trunk r67510 on windows (32bit), made a patch, and tested with my weave dependant code. (I am not familiar enough with distutils to make a standalone test). The same method to cause a problem has the same effect in ubuntu - the build directory seems to default to users home, but as the source path has a space in it g++ gets confused. |
|
|
| msg77510 - (view) |
Author: Martin v. Löwis (loewis) *  |
Date: 2008-12-10 08:54 |
| The patch hasn't been accepted for the trunk, yet, so it is not a candidate for a 2.5 backport. |
|
|
| msg78632 - (view) |
Author: Tarek Ziadé (tarek) * |
Date: 2008-12-31 16:46 |
| Hi. I am not familiar with weave. Could you provide a small sample of code that raises this issue. This way, I will be able to write the standalone test we can integrate in distutils together with your fix. |
|
|
| msg79641 - (view) |
Author: Brian Thorne (Thorney) |
Date: 2009-01-12 01:20 |
| Ok - this bug has nothing to do with weave itself. I think any extension written in C or C++ that has a space in the filename will raise the error in both windows and gnu/linux. Here is a simple c extension with and without spaces. The one with spaces in the filename fails to build on standard python. Whereas the one without should build fine. Did for me on windows 2000 on both python 2.5 and 2.6 with and without spaces in their paths. |
|
|
| msg80066 - (view) |
Author: Tarek Ziadé (tarek) * |
Date: 2009-01-18 00:45 |
| I couldn't reproduce the problem yet, using your sample. I tried to compile on Mac OSX (Python 2.5 GCC) and Windows XP (Python 2.5 , MinGW 5.1.4 and SWIG 1.2.3) and it worked well. The output was right, (the .o file has a space, but the pyd/so files come out properly), and I could use the gcd() function without any problem. The call to the compiler quotes the c file correctly: ... c:\Program Files\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -Ic:\Python25\include -Ic:\Python25\PC -c "ex ample.c" -o "build\temp.win32-2.5\Release\ex ample.o" ... This is done automatically with distutils.spawn._nt_quote_args under Windows for instance. And this is basically what your patch is doing. When does it crash exactly for you ? Do you have a traceback ? |
|
|
| msg83748 - (view) |
Author: C. E. Ball (ceball) |
Date: 2009-03-18 13:57 |
| I also found this bug while using SciPy's weave on Windows XP to compile some inline c/c++ code. Thorney's distutils_compiler_quoting.patch fixed the problem for me. Here is the relevant part of the traceback (I hope): """ Found executable C:\Program Files\Topographica\python_topo\mingw\bin\g++.exe g++.exe: c:\docume~1\ceball\locals~1\temp\ceball\python25_intermediate\compiler_ 894ad5ed761bb51736c6d2b7872dc212\Release\program: No such file or directory g++.exe: files\topographica\python_topo\lib\site-packages\weave\scxx\weave_imp.o: No such file or directory """ Chris |
|
|
| msg114867 - (view) |
Author: Brian Thorne (Thorney) |
Date: 2010-08-25 01:04 |
| Comments on the SciPy-Dev mailing list: >>> While testing Scipy's weave on several different Windows installations, I came across some problems with spaces in paths that often prevent weave from working. I can see a change that could probably get weave working on most Windows installations, but it is a quick hack. Someone knowledgeable about distutils (and numpy.distutils?) might be able to help me fix this properly. Below I describe three common problems with weave on Windows, in the hope that this information helps others, or allows someone to suggest how to fix the spaces-in- paths problem properly. I think there are three common problems that stop weave from working on Windows. The first is not having a C compiler. Both Python(x,y) and EPD provide a C compiler that seems to work fine, which is great! The second problem is that if weave is installed to a location with a space in the path, linking fails. There is already a scipy bug report about this (http://projects.scipy.org/scipy/ticket/809). I've just commented on that report, saying the problem appears to be with distutils, and there is already a Python bug report about it (http://bugs.python.org/issue4508). Maybe someone could close this scipy bug, or link it to the Python one somehow? In any case, when using Python(x,y) or EPD, this bug will not show up if the default installation locations are accepted. So, that's also good news! The third problem is that if the Windows user name has a space in it (which in my experience is quite common), compilation fails. Weave uses the user name to create a path for its "intermediate" and "compiled" files. When the compilation command is issued, the path with the space in it is also not quoted. Presumably that is another error in distutils (or numpy.distutils)? Unfortunately I wasn't able to pinpoint what function is failing to quote strings properly, because I couldn't figure out the chain that leads to the compiler being called. However, I can avoid the problem by removing spaces from the user name in weave itself (catalog.py): def whoami(): """return a string identifying the user.""" return (os.environ.get("USER") or os.environ.get("USERNAME") or "unknown").replace(" ","") (where I have added .replace(" ","") to the existing code). <<< >>> As well as that, I have another problem on another computer (which I don't have access to right now) where I have a user name with a space in it, and that causes several problems (although I think these problems are the fault of distutils and nothing to do with scipy). I found a way to work around that though by changing the environment variable for USERNAME from "Dan Goodman" to "DanGoo~1". <<< >>> Unfortunately, it is still often necessary to patch weave to get it to work on Windows. If a Windows user name includes a space, Weave tries to use paths with spaces for its "intermediate" and "compiled" files. I have no idea if this is a problem with distutils (not quoting strings correctly), or if Weave itself should do something different, but I myself hack weave installations on Windows. <<< |
|
|
| msg114868 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2010-08-25 01:18 |
| I tried using a space in a directory rather than in the file name and it worked. Thorney, can you repost your example as plain text files (separate source files or a diff against an empty dir) or at worst in an archive format readable by free software? Thank you. (Don’t remember why I changed component, reverting.) |
|
|
| msg114870 - (view) |
Author: Brian Thorne (Thorney) |
Date: 2010-08-25 01:30 |
| Sorry, here is the example as a diff against an empty directory. (I'm 100% sure the Ubuntu archiving program that I just used is free software though...) |
|
|
| msg114872 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2010-08-25 01:47 |
| I can’t open the archive with unrar-free, and can’t apply the diff with patch (tried context mode and ed mode). Would you mind making it a unified diff this time? |
|
|
| msg114875 - (view) |
Author: Brian Thorne (Thorney) |
Date: 2010-08-25 02:01 |
| Hopefully this works! Otherwise would a tar.gz or a zip archive be ok? |
|
|
| msg114877 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2010-08-25 02:14 |
| Thanks, this time it worked. I had to remove the example.i source file in setup.py since I don’t have swig installed, but apart from that, compiling “ex ample.c” or “ex ample/ex ample.c” worked. Next steps: - Write something similar to your example (can’t just copy it since Python is not under GPL) as a unit test for distutils; I’ll do it if nobody does it shortly. - Confirm that the bug is present on windows and hunt for it (paths translation? argument quoting?) |
|
|
| msg147768 - (view) |
Author: (ipatrol) |
Date: 2011-11-16 14:02 |
| Ok, I found a similar problem with MMTK. I am currently altering my distutils package to add a function called nt_quote_dir that adds quotes to paths with spaces and then applies it to each path if the platform is win32. When I'm done I will submit a diff after confirming that it works. |
|
|
| msg147865 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2011-11-18 12:41 |
| > Ok, I found a similar problem with MMTK. I don’t know what that is. > I am currently altering my distutils package to add a function called nt_quote_dir > that adds quotes to paths with spaces and then applies it to each path if the platform > is win32. You shouldn’t have to do that; there’s already a similar function in distutils.spawn. > When I'm done I will submit a diff after confirming that it works. What is needed is a diff for distutils 2.7 or 3.2 adding a test with a file containing a space. I can’t do anything with diffs for other projects. |
|
|
| msg160151 - (view) |
Author: fishdude (fishdude) |
Date: 2012-05-07 15:27 |
| i have to report that this issue also happens on Linux and Mac as well. no matter wether python 2.6 or 2.7 is used. |
|
|
| msg160156 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2012-05-07 16:07 |
| Thanks for the confirmation. Note that the versions field in this bug tracker indicates the versions that will get fixed, not all versions with the bug; 2.6 only gets security fixes now. Are you interested in updating the patch with a test (see my previous messages)? |
|
|
| msg167484 - (view) |
Author: Brian Thorne (Thorney) |
Date: 2012-08-05 10:58 |
| I've put together a patch adding the test requested. There is no problem on my Ubuntu machine with python 3.3. There is a comment in the file saying Don't load the xx module more than once, I am unsure whether my patch (using a renamed c file) violates this? One extremely minor nitpick that I'll mention is inconsistency with spaces in the module name. One can create a python file "my file.py" and can import it with __import__("my file"). I couldn't do the same for a C extension. Building a C extension with distutils: module_name = 'ex ample' example_ext = Extension(module_name, [target_c]) dist = Distribution({'name': module_name, 'ext_modules': [example_ext]}) Doesn't work due to the import mechanism looks for a PyInit function with the new module name (including a space): ImportError: dynamic module does not define init function (PyInit_ex ample) I don't think this is worth making a ticket over. |
|
|
| msg212662 - (view) |
Author: Alexandru Gheorghe (drunax) |
Date: 2014-03-03 19:45 |
| I am having the same issue on Debian 7 x86-64 (3.2.0-4-amd64) Python 2.7.3 (default, Jan 2 2013, 13:56:14) // [GCC 4.7.2] on linux2 Not noticing this bug I've opened a duplicate, for which the interesting comment can be found here: http://bugs.python.org/issue20824#msg212548 Please let me know if you need any files and I will try hard to provide some in my free time (also unittest from what I've read in the comments posted here). |
|
|
| msg212872 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2014-03-07 10:37 |
| Thanks Brian, let’s try and get this fixed. > I've put together a patch adding the test requested. There is no problem on > my Ubuntu machine with python 3.3. Are you saying the test does not reproduce the bug discussed here? > There is a comment in the file saying Don't load the xx module more than > once, I am unsure whether my patch (using a renamed c file) violates this? Hm I’m not quite sure if it’s enough that the extensions use different file names, or if they should also have different names inside the code. Existing tests already create and import xx multiple times though… > One can create a python file "my file.py" and can import it with > __import__("my file"). I couldn't do the same for a C extension. One can’t do “import my file” though, so I would sweep this under the rug as an obscure corner case :‑) |
|
|
| msg323456 - (view) |
Author: Greg Toombs (Greg Toombs) |
Date: 2018-08-13 02:28 |
| Confirmed still broken in 3.6.3. I ran into this when running CFFI. It does very wrong things with the path to cl.exe and include files with path spaces. My workaround is to monkeypatch distutils.ccompiler.gen_preprocess_options such that include paths are formed as -I"path" and not -Ipath; and to monkeypatch distutils.spawn._nt_quote_args so that it leaves all of the arguments unquoted except the first arg (path to cl.exe), which I only got working by using tilde-mangled short names. |
|
|
| msg386345 - (view) |
Author: Steve Dower (steve.dower) * |
Date: 2021-02-03 18:19 |
| Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils. If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools |
|
|