| msg136483 - (view) |
Author: Tarek Ziadé (tarek) *  |
Date: 2011-05-21 22:32 |
| this test module looks for sysconfig.get_config_var('srcdir') which in turns uses the sys,executable path. multiprocess seems to change it in every process, leading to the errors. To reproduce: ./python Lib/test/regrtest.py -j2 -v test_packaging A workaround is to skip the test in case the file is not found, but we need to fix it because it boils down to sysconfig being broken in multiprocess |
|
|
| msg136484 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2011-05-21 22:34 |
| What's the difference with ? |
|
|
| msg136486 - (view) |
Author: Tarek Ziadé (tarek) * |
Date: 2011-05-21 22:40 |
| Oops. It's a duplicate. Keeping this one since the problem was narrowed to multiprocessing/sys,executable and sysconfig |
|
|
| msg136487 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2011-05-21 22:50 |
| > Oops. It's a duplicate. Keeping this one since the problem was > narrowed to multiprocessing/sys,executable and sysconfig I don't think it has anything to do with sys.executable. As the other issue says, it seems the test (or packaging/sysconfig itself) is simply unable to locate the root of the current checkout (which should be trivial using __file__). |
|
|
| msg136489 - (view) |
Author: Tarek Ziadé (tarek) * |
Date: 2011-05-21 22:59 |
| sysconfig is looking for the source dir when sysconfig.get_config_var('srcdir') is called. And this is done like this: if sys.executable: _PROJECT_BASE = os.path.dirname(_safe_realpath(sys.executable)) else: # sys.executable can be empty if argv[0] has been changed and Python is # unable to retrieve the real program name _PROJECT_BASE = _safe_realpath(os.getcwd()) Because sys.executable (argv[0] in fact) is not filled when you call multiprocess vvia the -j option. So yeah, this has to do with sys.executable |
|
|
| msg136493 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2011-05-21 23:24 |
| > Because sys.executable (argv[0] in fact) is not filled when you call > multiprocess vvia the -j option. sys.executable is set perfectly well when running regrtest in multiprocess mode, otherwise many other tests would fail. Try: $ grep "sys.executable" Lib/test/* Lib/test/regrtest.py:585: base_cmd = [sys.executable] + opt_args + ['-m', 'test.regrtest'] Lib/test/script_helper.py:20: cmd_line = [sys.executable] Lib/test/script_helper.py:60: cmd_line = [sys.executable, '-E'] Lib/test/test_base64.py:231: args = (sys.executable, '-m', 'base64') + args Lib/test/test_capi.py:42: p = subprocess.Popen([sys.executable, "-c", [etc.] Again, it seems sysconfig simply has a bug which you can reproduce on the command line: $ pwd /home/antoine/cpython/default/build $ ../python -c "import sys, sysconfig; print(sys.executable, sysconfig.get_config_var('srcdir'))" /home/antoine/cpython/default/build/../python /home/antoine/cpython/default/build |
|
|
| msg136494 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2011-05-21 23:46 |
| The test should be changed anyway to avoid the dependency on "srcdir", whose value has no meaning when the tests are not run from a simple in-source-directory build. The test could create its own temp copy of xxmodule.c. |
|
|
| msg136568 - (view) |
Author: Tarek Ziadé (tarek) * |
Date: 2011-05-22 20:15 |
| @ned: right. done, and fixes |
|
|
| msg138269 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2011-06-13 19:02 |
| Here are patches to install a copy of xxmodule.c in the distutils tests directory (for 3.3, 3.2, and 2.7) and a copy in packaging tests for 3.3. With them in place, test_build_ext/test_command_build_ext now executes when the tests are run from an installed Python where no source directory is available and each test case run copies the c file into its temp dir. Currently, those test are being silently skipped in the installed case or, worse, picking up the wrong version of xxmodule.c depending on what sysconfig.get_config_vars('srcdir') returns (this happened to me when I reused a source directory name in some builds). For 3.2, a patch is included to backport the Distutils part of the fix for . |
|
|
| msg138283 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2011-06-13 22:29 |
| Is it really necessary to copy this file? I haven't looked at the tests, but I'm wondering if they are using xxmodule.c only because it already existed. Could the test instead use a much simpler .c file that it creates on the fly? |
|
|
| msg138284 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2011-06-13 22:37 |
| I think one of the objectives of the test is to see that it works with the C API of that particular Python release as captured in its version of xxmodule.c. And, while it could be argued that both distutils and packaging don't need their own copies, I think it is cleaner to keep the two testing environment uncoupled. |
|
|
| msg138319 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2011-06-14 14:58 |
| Path LGTM. Also +1 on keeping distutils and packaging wholly separate, including in tests infrastructure. It’s just one file. |
|
|
| msg138609 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2011-06-18 21:45 |
| See also #10764. |
|
|
| msg139001 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2011-06-24 23:23 |
| Are the patches good to go? And would you like me to apply them? |
|
|
| msg139012 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2011-06-25 00:48 |
| I think they are ready, I gave a +1 (with a typo: path instead of patch :) two messages ago. |
|
|
| msg139349 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2011-06-28 07:54 |
| New changeset c8ffa3891d5e by Ned Deily in branch '2.7': Issue #12141: Install a copy of template C module file so that http://hg.python.org/cpython/rev/c8ffa3891d5e New changeset de226a510b52 by Ned Deily in branch '3.2': Issue #12141: Install a copy of template C module file so that http://hg.python.org/cpython/rev/de226a510b52 New changeset ef8e9e99de88 by Ned Deily in branch 'default': Issue #12141: Install copies of template C module file so that http://hg.python.org/cpython/rev/ef8e9e99de88 |
|
|
| msg139350 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2011-06-28 08:03 |
| Patches applied as described above for 3.3, 3.2.1, and 2.7.3. I'm setting the status of the issue to pending and, assuming there are no buildbot failures in the near future, I will close it unless anyone sees a reason not to. |
|
|
| msg140438 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2011-07-15 16:07 |
| BTW, doesn’t the change to Makefile.pre.in need to be ported to the MSI build system too? |
|
|
| msg142548 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2011-08-20 18:02 |
| New changeset 7d9fa30c5588 by Éric Araujo in branch '3.2': Refactor the copying of xxmodule.c in distutils tests (#12141). http://hg.python.org/cpython/rev/7d9fa30c5588 New changeset 900738175779 by Éric Araujo in branch 'default': Refactor the copying of xxmodule.c in packaging tests (#12141). http://hg.python.org/cpython/rev/900738175779 |
|
|
| msg143478 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2011-09-04 06:41 |
| New changeset 402c4bbedf9c by Éric Araujo in branch '3.2': Refactor the copying of xxmodule.c in distutils tests (#12141). http://hg.python.org/cpython/rev/402c4bbedf9c |
|
|