| msg275360 - (view) |
Author: Zachary Ware (zach.ware) *  |
Date: 2016-09-09 17:58 |
| As a follow-on to #28027, the attached patch removes the concept of platform-specific directories (Lib/plat-darwin, Lib/plat-x86_64-linux-gnu, etc). I believe this should be fine for cross-builds (at least as fine as the previous solution for _sysconfigdata), but I'd like someone to test it for me since I have no experience with cross-builds. |
|
|
| msg275363 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-09-09 18:01 |
| You might also remove Lib/_sysconfigdata.py from .gitignore. |
|
|
| msg275427 - (view) |
Author: Matthias Klose (doko) * |
Date: 2016-09-09 20:35 |
| not sure if I fully understand: - The patch encodes the multiarch triplet into the _sysconfigdata name and installs it into the standard lib dir. - It removes the PLATDIR macro - what happens to the constants modules which are currently found in the platdir? These vary across architectures and should not be install in the standard lib dir. |
|
|
| msg275429 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2016-09-09 20:36 |
| Those were removed in #28027, by BDFL decree. |
|
|
| msg275430 - (view) |
Author: Matthias Klose (doko) * |
Date: 2016-09-09 20:40 |
| this comes as a surprise. I can't remember that discussion and decision. |
|
|
| msg275433 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2016-09-09 20:47 |
| Sorry, it happened at the sprint at Instagram. Do you have a use for those modules that we need to reconsider? |
|
|
| msg275436 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-09-09 20:50 |
| > this comes as a surprise. I can't remember that discussion and decision. Removing these modules is not a new idea: I just posted a message to the issue #28027 to complete the history of this change. |
|
|
| msg275438 - (view) |
Author: Matthias Klose (doko) * |
Date: 2016-09-09 20:59 |
| The removal is not following the guidelines to first deprecate these for a release, and then to remove them. I know that at least the RTLD_* constants are used in a number of places, which will just stop working. I'm not against removing these in 3.7, however they should be deprecated in 3.6 first. There is no harm done with this approach. |
|
|
| msg275442 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-09-09 21:06 |
| Matthias Klose added the comment: > The removal is not following the guidelines to first deprecate these for a release, and then to remove them. I know that at least the RTLD_* constants are used in a number of places, which will just stop working. Please continue the discussion in the issue #28027, reopen it if you want. |
|
|
| msg275475 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2016-09-09 22:29 |
| Xavier: Could you please make sure this doesn't break anything for Android/cross-builds? |
|
|
| msg275477 - (view) |
Author: Xavier de Gaye (xdegaye) *  |
Date: 2016-09-09 22:35 |
| Sure, I was already planning to do it :) |
|
|
| msg275518 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-09-10 01:00 |
| New changeset 90396ec9a2f8 by Zachary Ware in branch 'default': Issue #28046: Remove platform-specific directories from sys.path https://hg.python.org/cpython/rev/90396ec9a2f8 |
|
|
| msg275519 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2016-09-10 01:01 |
| Xavier: I'm still interested in hearing whether this breaks anything for you, but I went ahead and pushed it. |
|
|
| msg275532 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-09-10 01:36 |
| New changeset 2bfe63a3eb5c by Zachary Ware in branch 'default': Issue #28046: Fix distutils https://hg.python.org/cpython/rev/2bfe63a3eb5c |
|
|
| msg275592 - (view) |
Author: (yan12125) * |
Date: 2016-09-10 07:36 |
| Seems after this changeset Python is broken: shell@ASUS_Z00E_2:/data/local/tmp/python3 $ python3.6 Failed to import the site module Traceback (most recent call last): File "/data/local/tmp/python3/lib/python3.6/site.py", line 549, in main() File "/data/local/tmp/python3/lib/python3.6/site.py", line 536, in main known_paths = addusersitepackages(known_paths) File "/data/local/tmp/python3/lib/python3.6/site.py", line 281, in addusersitepackages user_site = getusersitepackages() File "/data/local/tmp/python3/lib/python3.6/site.py", line 257, in getusersitepackages user_base = getuserbase() # this will also set USER_BASE File "/data/local/tmp/python3/lib/python3.6/site.py", line 247, in getuserbase USER_BASE = get_config_var('userbase') File "/data/local/tmp/python3/lib/python3.6/sysconfig.py", line 600, in get_config_var return get_config_vars().get(name) File "/data/local/tmp/python3/lib/python3.6/sysconfig.py", line 549, in get_config_vars _init_posix(_CONFIG_VARS) File "/data/local/tmp/python3/lib/python3.6/sysconfig.py", line 420, in _init_posix _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0) ModuleNotFoundError: No module named '_sysconfigdata_m_linux_aarch64-linux-android' sysconfig data is installed as lib-dynload/_sysconfigdata_m_linux_x86_64-linux-gnu.py. It should be at lib-dynload/_sysconfigdata_m_linux_aarch64-linux-android.py? PS: I'm building for Android AArch64 on Linux x86_64 |
|
|
| msg275599 - (view) |
Author: (yan12125) * |
Date: 2016-09-10 08:19 |
| Got it. When calling `python -m sysconfig --generate-posix-vars`, the host (Linux x86_64) Python is used, so _get_sysconfigdata_name() returns the name for Linux instead of for Android. Maybe _get_sysconfigdata_name() should check whether it's used for cross-compiling or not. |
|
|
| msg275635 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-09-10 13:51 |
| Confirming the problem reported by Chi Hsuan Yen. The attached patch fixes this. Another problem is that the shared libraries names of the extension modules are now suffixed with the wrong triplet, i.e. with the build system triplet instead of the target host triplet. So they cannot be imported. Not sure where this regression has been introduced. |
|
|
| msg275638 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2016-09-10 14:06 |
| Xavier, that change looks good to me, please commit it. |
|
|
| msg275642 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-09-10 14:20 |
| New changeset 1d71ce4531ac by Xavier de Gaye in branch 'default': Issue #28046: Fix get_sysconfigdata_name(). https://hg.python.org/cpython/rev/1d71ce4531ac |
|
|
| msg275643 - (view) |
Author: (yan12125) * |
Date: 2016-09-10 14:21 |
| With this patch no extensions in Modules/ can be built. Let me check whether there's a bug in my build script. |
|
|
| msg275650 - (view) |
Author: (yan12125) * |
Date: 2016-09-10 16:11 |
| Hmm things are quite complicated. Brief: the build is broken for out-of-source cross-compiling if the host Python is an in-source build. (Below builddirreferstothedirectorythatinvokesbuild_dir refers to the directory that invokes builddirreferstothedirectorythatinvokessource_dir/configure) In an out-of-source build, setup.py relies on sysconfig.get_config_var('srcdir') to get correct filenames. In _init_posix(), _get_sysconfigdata_name() still returns an incorrect name (for example, _sysconfigdata_m_linux_x86_64-linux-gnu, on Linux x86_64). There's no _sysconfigdata_m_linux_x86_64-linux-gnu.py in builddir/build/lib.linux−aarch64−3.6,sobuild_dir/build/lib.linux-aarch64-3.6, so builddir/build/lib.linux−aarch64−3.6,soPYTHON_FOR_BUILD imports the one from the host python, which has srcdir == '.' As a result, build_ext can't find source files: building 'xxlimited' extension /home/yen/Projects/python3-android/clang-bin/cc -fPIC -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -g -fvar-tracking-assignments -g -fvar-tracking-assignments -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -g -fvar-tracking-assignments -g -fvar-tracking-assignments -std=c99 -DPy_LIMITED_API=0x03050000 -I./Include -I. -IInclude -I/usr/include -I/usr/local/include -I/home/yen/Projects/python3-android/src/cpython/build-target/Include -I/home/yen/Projects/python3-android/src/cpython/build-target -c xxlimited.c -o build/temp.linux-aarch64-3.6/xxlimited.o clang: error: unknown argument: '-fvar-tracking-assignments' clang: error: unknown argument: '-fvar-tracking-assignments' clang: error: unknown argument: '-fvar-tracking-assignments' clang: error: unknown argument: '-fvar-tracking-assignments' clang: error: no such file or directory: 'xxlimited.c' clang: error: no input files Ironically, before 1d71ce4531ac, $build_dir/build/lib.linux-aarch64-3.6 has _sysconfigdata_m_linux_x86_64-linux-gnu.py, so building is OK. Again, I don't know to fix it :( [1] https://hg.python.org/cpython/file/1d71ce4531ac/setup.py#l218 |
|
|
| msg275664 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-09-10 18:12 |
| With the attached patch, the python test suite runs smoothly on android and without errors on linux. On android, there are two new failed test cases in test_sysconfig that I will look into later (in the frame of this issue, I guess ?), but otherwise about the same tests fail that were failing before. Here are some details about the fix: > Another problem is that the shared libraries names of the extension modules are now suffixed with the wrong triplet. This is because _init_posix() in Lib/distutils/sysconfig.py still imports the native python sysconfigdata module instead of the newly built one for the target host. After _init_posix() is fixed, another problem occurs: extension modules are now built by including /usr/include from the native system and this leads to some unresolved dlopen references at run time on android. The reason is that add_gcc_paths() in setup.py calls sysconfig.get_config_var('CC') and this returns the native compiler instead of the cross-compiler. So a solution is to set the _SYSCONFIGDATA_NAME environment variable in PYTHON_FOR_BUILD. |
|
|
| msg275667 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2016-09-10 18:23 |
| LGTM, thank you Xavier! |
|
|
| msg275677 - (view) |
Author: (yan12125) * |
Date: 2016-09-10 19:12 |
| Much thanks! It's now building fine and running fine with my configuration. The test suite is still running and I believe there won't be more surprise than failed tests. |
|
|
| msg275711 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-09-10 23:06 |
| One of the test_sysconfig failed test is caused by the changes made in issue 27917. I will fix it there. The other one is caused by a change in my android build setup that does not install anymore include/python3.6m/pyconfig.h, this is fixed now and the test does not fail anymore. Pending code reviews, I will commit the patch after changing the env var name from _SYSCONFIGDATA_NAME to _PYTHON_SYSCONFIGDATA_NAME. |
|
|
| msg275845 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-09-11 20:22 |
| New changeset 04b679dd11eb by Xavier de Gaye in branch 'default': Issue #28046: get_sysconfigdata_name() uses the _PYTHON_SYSCONFIGDATA_NAME https://hg.python.org/cpython/rev/04b679dd11eb |
|
|
| msg275846 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-09-11 20:25 |
| Thanks for your help in testing these changes Chi Hsuan Yen. |
|
|
| msg275850 - (view) |
Author: (yan12125) * |
Date: 2016-09-11 20:32 |
| No No, it's me who should say thank you. Thanks for all efforts on maintaining the Android port of CPython! |
|
|
| msg276245 - (view) |
Author: Matthias Klose (doko) * |
Date: 2016-09-13 10:54 |
| I don't like the _PYTHON_SYSCONFIGDATA_NAME hack. This should be based on the target, not on a name for just a particular file. Following up on that in #28125. |
|
|
| msg279508 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2016-10-26 19:05 |
| 'make install' fails to remove the sysconfigdata module from lib-dynload and prints now instead: rm: cannot remove '/path/to/install/lib/python3.7/lib-dynload/_sysconfigdata_m.py': No such file or directory The patch fixes this. It also removes a useless and now incorrect line that was meant to echo the previously executed command. |
|
|
| msg279592 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-10-28 09:41 |
| New changeset 86577b7100a4 by Xavier de Gaye in branch '3.6': Issue #28046: Fix the removal of the sysconfigdata module https://hg.python.org/cpython/rev/86577b7100a4 New changeset 28205c4cf20b by Xavier de Gaye in branch 'default': Issue #28046: Merge with 3.6. https://hg.python.org/cpython/rev/28205c4cf20b |
|
|
| msg303426 - (view) |
Author: Zachary Ware (zach.ware) * |
Date: 2017-09-30 20:19 |
| Are there any remaining outstanding issues here? |
|
|
| msg310539 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2018-01-24 00:27 |
| The cleanup is not complete, so I reopen the issue: https://github.com/python/cpython/pull/5289 |
|
|
| msg310605 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2018-01-24 16:02 |
| New changeset 5de15f1d5ff09085620f63f0597d0920d75be719 by Victor Stinner in branch 'master': bpo-28046: Remove MACHDEPPATH from Modules/Setup.dist (#5289) https://github.com/python/cpython/commit/5de15f1d5ff09085620f63f0597d0920d75be719 |
|
|
| msg310608 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2018-01-24 16:20 |
| Fixed. I close again the issue. I don't want to backport this change, it doesn't hurt Python 3.6. |
|
|