| msg157065 - (view) |
Author: Ross Lagerwall (rosslagerwall)  |
Date: 2012-03-29 16:35 |
| On an up to date Fedora 16: == CPython 3.3.0a1+ (default:ddddd528b2d2+, Mar 29 2012, 18:04:26) [GCC 4.6.3 20120306 (Red Hat 4.6.3-2)] == Linux-3.3.0-4.fc16.x86_64-x86_64-with-fedora-16-Verne little-endian == /home/ross/src/cpythondev/temp/build/test_python_21786 Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1) [1/1] test_distutils test_byte_compile (distutils.tests.test_install_lib.InstallLibTestCase) ... ok test_dont_write_bytecode (distutils.tests.test_install_lib.InstallLibTestCase) ... ok test_finalize_options (distutils.tests.test_install_lib.InstallLibTestCase) ... ok test_get_inputs (distutils.tests.test_install_lib.InstallLibTestCase) ... ok test_get_outputs (distutils.tests.test_install_lib.InstallLibTestCase) ... ok test_no_compiler (distutils.tests.test_msvc9compiler.msvc9compilerTestCase) ... skipped 'These tests are only for win32' test_reg_class (distutils.tests.test_msvc9compiler.msvc9compilerTestCase) ... skipped 'These tests are only for win32' test_remove_entire_manifest (distutils.tests.test_msvc9compiler.msvc9compilerTestCase) ... skipped 'These tests are only for win32' test_remove_visual_c_ref (distutils.tests.test_msvc9compiler.msvc9compilerTestCase) ... skipped 'These tests are only for win32' test_nt_quote_args (distutils.tests.test_spawn.SpawnTestCase) ... ok test_spawn (distutils.tests.test_spawn.SpawnTestCase) ... ok test_announce (distutils.tests.test_dist.DistributionTestCase) ... ok test_command_packages_cmdline (distutils.tests.test_dist.DistributionTestCase) ... ok test_command_packages_configfile (distutils.tests.test_dist.DistributionTestCase) ... ok test_command_packages_unspecified (distutils.tests.test_dist.DistributionTestCase) ... ok test_empty_options (distutils.tests.test_dist.DistributionTestCase) ... ok test_finalize_options (distutils.tests.test_dist.DistributionTestCase) ... ok test_get_command_packages (distutils.tests.test_dist.DistributionTestCase) ... ok test_classifier (distutils.tests.test_dist.MetadataTestCase) ... ok test_custom_pydistutils (distutils.tests.test_dist.MetadataTestCase) ... ok test_download_url (distutils.tests.test_dist.MetadataTestCase) ... ok test_fix_help_options (distutils.tests.test_dist.MetadataTestCase) ... ok test_long_description (distutils.tests.test_dist.MetadataTestCase) ... ok test_obsoletes (distutils.tests.test_dist.MetadataTestCase) ... ok test_obsoletes_illegal (distutils.tests.test_dist.MetadataTestCase) ... ok test_provides (distutils.tests.test_dist.MetadataTestCase) ... ok test_provides_illegal (distutils.tests.test_dist.MetadataTestCase) ... ok test_requires (distutils.tests.test_dist.MetadataTestCase) ... ok test_requires_illegal (distutils.tests.test_dist.MetadataTestCase) ... ok test_show_help (distutils.tests.test_dist.MetadataTestCase) ... ok test_simple_metadata (distutils.tests.test_dist.MetadataTestCase) ... ok test_simple_run (distutils.tests.test_install_headers.InstallHeadersTestCase) ... ok test_finalize_options (distutils.tests.test_upload.uploadTestCase) ... ok test_saved_password (distutils.tests.test_upload.uploadTestCase) ... ok test_server_empty_registration (distutils.tests.test_upload.uploadTestCase) ... ok test_server_registration (distutils.tests.test_upload.uploadTestCase) ... ok test_upload (distutils.tests.test_upload.uploadTestCase) ... ok test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase) ... ok test_check_extensions_list (distutils.tests.test_build_ext.BuildExtTestCase) ... ok test_compiler_option (distutils.tests.test_build_ext.BuildExtTestCase) ... ok test_deployment_target_default (distutils.tests.test_build_ext.BuildExtTestCase) ... skipped 'test only relevant for MacOSX' test_deployment_target_higher_ok (distutils.tests.test_build_ext.BuildExtTestCase) ... skipped 'test only relevant for MacOSX' test_deployment_target_too_low (distutils.tests.test_build_ext.BuildExtTestCase) ... skipped 'test only relevant for MacOSX' test_ext_fullpath (distutils.tests.test_build_ext.BuildExtTestCase) ... ok test_finalize_options (distutils.tests.test_build_ext.BuildExtTestCase) ... ok test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase) ... ok test_get_source_files (distutils.tests.test_build_ext.BuildExtTestCase) ... ok test_optional_extension (distutils.tests.test_build_ext.BuildExtTestCase) ... ok test_solaris_enable_shared (distutils.tests.test_build_ext.BuildExtTestCase) ... ok test_user_site (distutils.tests.test_build_ext.BuildExtTestCase) ... ok test_build (distutils.tests.test_build_scripts.BuildScriptsTestCase) ... ok test_default_settings (distutils.tests.test_build_scripts.BuildScriptsTestCase) ... ok test_version_int (distutils.tests.test_build_scripts.BuildScriptsTestCase) ... ok test_debug_mode (distutils.tests.test_core.CoreTestCase) ... ok test_run_setup_provides_file (distutils.tests.test_core.CoreTestCase) ... ok test_run_setup_uses_current_dir (distutils.tests.test_core.CoreTestCase) ... ok test_extension_init (distutils.tests.test_extension.ExtensionTestCase) ... ok test_read_setup_file (distutils.tests.test_extension.ExtensionTestCase) ... ok test_check_config_h (distutils.tests.test_cygwinccompiler.CygwinCCompilerTestCase) ... ok test_get_msvcr (distutils.tests.test_cygwinccompiler.CygwinCCompilerTestCase) ... ok test_get_versions (distutils.tests.test_cygwinccompiler.CygwinCCompilerTestCase) ... ok test_cmp (distutils.tests.test_version.VersionTestCase) ... ok test_cmp_strict (distutils.tests.test_version.VersionTestCase) ... ok test_prerelease (distutils.tests.test_version.VersionTestCase) ... ok test_byte_compile (distutils.tests.test_build_py.BuildPyTestCase) ... ok test_byte_compile_optimized (distutils.tests.test_build_py.BuildPyTestCase) ... ok test_dont_write_bytecode (distutils.tests.test_build_py.BuildPyTestCase) ... ok test_empty_package_dir (distutils.tests.test_build_py.BuildPyTestCase) ... ok test_package_data (distutils.tests.test_build_py.BuildPyTestCase) ... ok test_no_optimize_flag (distutils.tests.test_bdist_rpm.BuildRpmTestCase) ... error: Installed (but unpackaged) file(s) found: /usr/local/lib/python3.3/site-packages/foo.pyc /usr/local/lib/python3.3/site-packages/foo.pyo Installed (but unpackaged) file(s) found: /usr/local/lib/python3.3/site-packages/foo.pyc /usr/local/lib/python3.3/site-packages/foo.pyo ERROR test_quiet (distutils.tests.test_bdist_rpm.BuildRpmTestCase) ... error: Installed (but unpackaged) file(s) found: /usr/local/lib/python3.3/site-packages/foo.pyc /usr/local/lib/python3.3/site-packages/foo.pyo Installed (but unpackaged) file(s) found: /usr/local/lib/python3.3/site-packages/foo.pyc /usr/local/lib/python3.3/site-packages/foo.pyo ERROR test_clean (distutils.tests.test_config_cmd.ConfigTestCase) ... ok test_dump_file (distutils.tests.test_config_cmd.ConfigTestCase) ... ok test_finalize_options (distutils.tests.test_config_cmd.ConfigTestCase) ... ok test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase) ... ok test_class (distutils.tests.test_text_file.TextFileTestCase) ... ok test_move_file_verbosity (distutils.tests.test_file_util.FileUtilTestCase) ... ok test_check_archive_formats (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok test_compress_deprecated (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... skipped 'The compress program is required' test_make_archive (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok test_make_archive_cwd (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok test_make_tarball (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok test_make_tarball_extended (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok test_make_tarball_latin1 (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok test_make_zipfile (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok test_make_zipfile_no_zlib (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok test_tarfile_vs_tar (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... ok test_minimal (distutils.tests.test_bdist_msi.BDistMSITestCase) ... skipped 'these tests require Windows' test_build_libraries (distutils.tests.test_build_clib.BuildCLibTestCase) ... ok test_check_library_dist (distutils.tests.test_build_clib.BuildCLibTestCase) ... ok test_finalize_options (distutils.tests.test_build_clib.BuildCLibTestCase) ... ok test_get_source_files (distutils.tests.test_build_clib.BuildCLibTestCase) ... ok test_run (distutils.tests.test_build_clib.BuildCLibTestCase) ... ok test_add_defaults (distutils.tests.test_sdist.SDistTestCase) ... ok test_check_metadata_deprecated (distutils.tests.test_sdist.SDistTestCase) ... ok test_finalize_options (distutils.tests.test_sdist.SDistTestCase) ... ok test_get_file_list (distutils.tests.test_sdist.SDistTestCase) ... ok test_invalid_template_unknown_command (distutils.tests.test_sdist.SDistTestCase) ... ok test_invalid_template_wrong_arguments (distutils.tests.test_sdist.SDistTestCase) ... ok test_invalid_template_wrong_path (distutils.tests.test_sdist.SDistTestCase) ... skipped 'test relevant for Windows only' test_make_distribution (distutils.tests.test_sdist.SDistTestCase) ... ok test_manifest_comments (distutils.tests.test_sdist.SDistTestCase) ... ok test_manifest_marker (distutils.tests.test_sdist.SDistTestCase) ... ok test_manual_manifest (distutils.tests.test_sdist.SDistTestCase) ... ok test_metadata_check_option (distutils.tests.test_sdist.SDistTestCase) ... ok test_prune_file_list (distutils.tests.test_sdist.SDistTestCase) ... ok test_server_empty_registration (distutils.tests.test_sdist.SDistTestCase) ... ok test_server_registration (distutils.tests.test_sdist.SDistTestCase) ... ok test_show_formats (distutils.tests.test_sdist.SDistTestCase) ... ok test_customize_compiler (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok test_get_config_h_filename (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok test_get_config_vars (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok test_get_python_inc (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok test_get_python_lib (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok test_parse_makefile_base (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok test_parse_makefile_literal_dollar (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok test_sysconfig_module (distutils.tests.test_sysconfig.SysconfigTestCase) ... ok VersionPredicate (distutils.versionpredicate) Doctest: distutils.versionpredicate.VersionPredicate ... ok split_provision (distutils.versionpredicate) Doctest: distutils.versionpredicate.split_provision ... ok test_default_settings (distutils.tests.test_install_scripts.InstallScriptsTestCase) ... ok test_installation (distutils.tests.test_install_scripts.InstallScriptsTestCase) ... ok test_check_metadata_deprecated (distutils.tests.test_register.RegisterTestCase) ... ok test_create_pypirc (distutils.tests.test_register.RegisterTestCase) ... ok test_password_not_in_file (distutils.tests.test_register.RegisterTestCase) ... ok test_password_reset (distutils.tests.test_register.RegisterTestCase) ... ok test_registering (distutils.tests.test_register.RegisterTestCase) ... ok test_server_empty_registration (distutils.tests.test_register.RegisterTestCase) ... ok test_server_registration (distutils.tests.test_register.RegisterTestCase) ... ok test_strict (distutils.tests.test_register.RegisterTestCase) ... ok test_copy_tree_verbosity (distutils.tests.test_dir_util.DirUtilTestCase) ... ok test_create_tree_verbosity (distutils.tests.test_dir_util.DirUtilTestCase) ... ok test_ensure_relative (distutils.tests.test_dir_util.DirUtilTestCase) ... ok test_mkpath_remove_tree_verbosity (distutils.tests.test_dir_util.DirUtilTestCase) ... ok test_mkpath_with_custom_mode (distutils.tests.test_dir_util.DirUtilTestCase) ... ok test_formats (distutils.tests.test_bdist.BuildTestCase) ... ok test_skip_build (distutils.tests.test_bdist.BuildTestCase) ... ok test_simple_run (distutils.tests.test_install_data.InstallDataTestCase) ... ok test_simple_built (distutils.tests.test_bdist_dumb.BuildDumbTestCase) ... ok test_change_root (distutils.tests.test_util.UtilTestCase) ... ok test_check_environ (distutils.tests.test_util.UtilTestCase) ... ok test_convert_path (distutils.tests.test_util.UtilTestCase) ... ok test_dont_write_bytecode (distutils.tests.test_util.UtilTestCase) ... ok test_get_platform (distutils.tests.test_util.UtilTestCase) ... ok test_rfc822_escape (distutils.tests.test_util.UtilTestCase) ... ok test_split_quoted (distutils.tests.test_util.UtilTestCase) ... ok test_strtobool (distutils.tests.test_util.UtilTestCase) ... ok test_runtime_libdir_option (distutils.tests.test_unixccompiler.UnixCCompilerTestCase) ... ok test_debug_print (distutils.tests.test_filelist.FileListTestCase) ... ok test_exclude_pattern (distutils.tests.test_filelist.FileListTestCase) ... ok test_glob_to_re (distutils.tests.test_filelist.FileListTestCase) ... ok test_include_pattern (distutils.tests.test_filelist.FileListTestCase) ... ok test_process_template (distutils.tests.test_filelist.FileListTestCase) ... ok test_process_template_line (distutils.tests.test_filelist.FileListTestCase) ... ok test_remove_duplicates (distutils.tests.test_filelist.FileListTestCase) ... ok test_set_allfiles (distutils.tests.test_filelist.FileListTestCase) ... ok test_translate_pattern (distutils.tests.test_filelist.FileListTestCase) ... ok test_simple_run (distutils.tests.test_clean.cleanTestCase) ... ok test_finalize_options (distutils.tests.test_build.BuildTestCase) ... ok test_get_exe_bytes (distutils.tests.test_bdist_wininst.BuildWinInstTestCase) ... ok test_check_all (distutils.tests.test_check.CheckTestCase) ... ok test_check_document (distutils.tests.test_check.CheckTestCase) ... ok test_check_metadata (distutils.tests.test_check.CheckTestCase) ... ok test_check_restructuredtext (distutils.tests.test_check.CheckTestCase) ... ok test_debug_print (distutils.tests.test_cmd.CommandTestCase) ... ok test_dump_options (distutils.tests.test_cmd.CommandTestCase) ... ok test_ensure_dirname (distutils.tests.test_cmd.CommandTestCase) ... ok test_ensure_filename (distutils.tests.test_cmd.CommandTestCase) ... ok test_ensure_string (distutils.tests.test_cmd.CommandTestCase) ... ok test_ensure_string_list (distutils.tests.test_cmd.CommandTestCase) ... ok test_make_file (distutils.tests.test_cmd.CommandTestCase) ... ok test_non_ascii (distutils.tests.test_log.TestLog) ... ok test_server_empty_registration (distutils.tests.test_config.PyPIRCCommandTestCase) ... ok test_server_registration (distutils.tests.test_config.PyPIRCCommandTestCase) ... ok test_newer (distutils.tests.test_dep_util.DepUtilTestCase) ... ok test_newer_group (distutils.tests.test_dep_util.DepUtilTestCase) ... ok test_newer_pairwise (distutils.tests.test_dep_util.DepUtilTestCase) ... ok test_debug_mode (distutils.tests.test_install.InstallTestCase) ... ok test_finalize_options (distutils.tests.test_install.InstallTestCase) ... ok test_handle_extra_path (distutils.tests.test_install.InstallTestCase) ... ok test_home_installation_scheme (distutils.tests.test_install.InstallTestCase) ... ok test_record (distutils.tests.test_install.InstallTestCase) ... ok test_record_extensions (distutils.tests.test_install.InstallTestCase) ... ok test_user_site (distutils.tests.test_install.InstallTestCase) ... test test_distutils failed [?1034hok ====================================================================== ERROR: test_no_optimize_flag (distutils.tests.test_bdist_rpm.BuildRpmTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/ross/src/cpythondev/temp/Lib/distutils/tests/test_bdist_rpm.py", line 125, in test_no_optimize_flag cmd.run() File "/home/ross/src/cpythondev/temp/Lib/distutils/command/bdist_rpm.py", line 365, in run self.spawn(rpm_cmd) File "/home/ross/src/cpythondev/temp/Lib/distutils/cmd.py", line 366, in spawn spawn(cmd, search_path, dry_run=self.dry_run) File "/home/ross/src/cpythondev/temp/Lib/distutils/spawn.py", line 32, in spawn _spawn_posix(cmd, search_path, dry_run=dry_run) File "/home/ross/src/cpythondev/temp/Lib/distutils/spawn.py", line 163, in _spawn_posix % (cmd[0], exit_status)) distutils.errors.DistutilsExecError: command 'rpmbuild' failed with exit status 1 ====================================================================== ERROR: test_quiet (distutils.tests.test_bdist_rpm.BuildRpmTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/ross/src/cpythondev/temp/Lib/distutils/tests/test_bdist_rpm.py", line 81, in test_quiet cmd.run() File "/home/ross/src/cpythondev/temp/Lib/distutils/command/bdist_rpm.py", line 365, in run self.spawn(rpm_cmd) File "/home/ross/src/cpythondev/temp/Lib/distutils/cmd.py", line 366, in spawn spawn(cmd, search_path, dry_run=self.dry_run) File "/home/ross/src/cpythondev/temp/Lib/distutils/spawn.py", line 32, in spawn _spawn_posix(cmd, search_path, dry_run=dry_run) File "/home/ross/src/cpythondev/temp/Lib/distutils/spawn.py", line 163, in _spawn_posix % (cmd[0], exit_status)) distutils.errors.DistutilsExecError: command 'rpmbuild' failed with exit status 1 ---------------------------------------------------------------------- Ran 183 tests in 1.889s FAILED (errors=2, skipped=10) 1 test failed: test_distutils |
|
|
| msg157076 - (view) |
Author: Ross Lagerwall (rosslagerwall) |
Date: 2012-03-29 18:17 |
| The output of running rpmbuild from bash: """ $ rpmbuild RPM version 4.9.1.2 Copyright (C) 1998-2002 - Red Hat, Inc. This program may be freely redistributed under the terms of the GNU GPL Usage: rpmbuild [-v?] [-bp] [-bc] [-bi] [-bl] [-ba] [-bb] [-bs] [-tp] [-tc] [-ti] [-ta] [-tb] [-ts] [--rebuild] [--recompile] [--buildroot=DIRECTORY] [--clean] [--nobuild] [--nodeps] [--nodirtokens] [--rmsource] [--rmspec] [--short-circuit] [--target=CPU-VENDOR-OS] [-D|--define 'MACRO EXPR'] [-E |
--eval 'EXPR'] [--macros=FILE:...] [--nodigest] [--nosignature] [--rcfile=FILE:...] [-r |
--root ROOT] [--dbpath=DIRECTORY] [--querytags] [--showrc] [--quiet] [-v |
| msg157125 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2012-03-30 03:06 |
| Related to #1533164. Could you bisect the problem to see if it’s related to old commits or my changes for pycache directories? |
|
|
| msg157255 - (view) |
Author: Ross Lagerwall (rosslagerwall) |
Date: 2012-04-01 05:08 |
| The first bad revision is: changeset: 72818:27a36b05caed branch: 3.2 user: Éric Araujo <merwok@netwok.org> date: Sat Oct 08 00:34:13 2011 +0200 summary: Fix distutils byte-compilation to comply with PEP 3147 (#11254). Cheers |
|
|
| msg158923 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2012-04-21 16:48 |
| This failure is also affecting the new RHEL-6 buildbot I'm attempting to bring online. Failure on trunk: http://www.python.org/dev/buildbot/all/builders/x86%20RHEL%206%203.x/builds/12/steps/test/logs/stdio Failure on 3.2: http://www.python.org/dev/buildbot/all/builders/x86%20RHEL%206%203.2/builds/1/steps/test/logs/stdio Seems like it may be an old problem come back to life: https://www.redhat.com/archives/rhl-devel-list/2009-January/msg00389.html (The rh-devel-list link is interesting more for the problem description than it is for the suggested workaround) |
|
|
| msg158925 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2012-04-21 16:58 |
| I'm wondering if there may be a deeper problem here: how certain are we that bdist_rpm isn't using the system Python to handle the byte compilation step? It would explain why the files are still being generated in the old locations. And, in practical terms, who uses RPM to package Python software for any version of Python other than the system Python? |
|
|
| msg159249 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2012-04-25 04:02 |
| To answer my last question: plenty of people. Even within Fedora itself there are parallel python 2 and python 3 RPM stacks. |
|
|
| msg159250 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2012-04-25 04:21 |
| It occurs to me there's a way to check my theory: if we update the failing test to explicitly check the magic cookie in at least one of the precompiled pyc files (rather than just expecting the files' existence), then it should also start failing on the 2.7 RHEL 6 buildbot. |
|
|
| msg159251 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2012-04-25 04:36 |
| The most likely cause is that some code was updated to use pycache directories but other code that computes paths still uses the old way; this happened for bdist_wininst too. I’ll have a look at the code and reply to your questions tomorrow. I’ll also add links to another bug where I suggest to print the full command when distutils.spawn fails and DISTUTILS_DEBUG is set and another bug about rpm vs. rpmbuild. this-job-thing-really-takes-much-of-my-Python-time-ly yours |
|
|
| msg159313 - (view) |
Author: Éric Araujo (eric.araujo) * |
Date: 2012-04-25 16:25 |
| > I'm wondering if there may be a deeper problem here: how certain are we that bdist_rpm isn't using the system Python > to handle the byte compilation step? It would explain why the files are still being generated in the old locations. I’ve had a quick look at bdist_rpm and found nothing suspect (like filename + 'c'). You may have the right idea: There are --python and --fix-python options for this command that decide what executable will be used to run setup.py, so maybe the system Python is used. The default is 'python' instead of 'python3' though, so if that’s the cause I don’t understand how it worked at all. > It occurs to me there's a way to check my theory: if we update the failing test to explicitly check the magic cookie in at > least one of the precompiled pyc files (rather than just expecting the files' existence), then it should also start failing > on the 2.7 RHEL 6 buildbot. I’ve been looking for a way to inspect byte-compiled files for some time; please share any ideas on #13473. To help debug this, I could apply #11599. The rpm vs. rpmbuild bug is #11122. #5875 and #13307 are bugs similar to this one. |
|
|
| msg159350 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2012-04-26 01:47 |
| Antoine (added to nosy list) indicated he wasn't seeing the failure on a Mageia system, so that's another point in favour of a Fedora/RHEL specific problem. Also added Dave Malcolm as the Fedora Python package maintainer. Applying #11599 to get the full failing command line sounds worthwhile, but #11122 shouldn't be related, since the RHEL buildbot definitely has rpmbuild available. |
|
|
| msg159384 - (view) |
Author: Dave Malcolm (dmalcolm) |
Date: 2012-04-26 15:32 |
| As a post-processing step, rpmbuild will attempt to byte-compile any .py files it encounters, and the results must be listed in the %files manifest. [1] This is done by the script brp-python-bytecompile, which uses the compileall module. However, my guess is that it's not using the correct version of python when invoking "compileall", which would explain why it's using the pre-PEP3147 location for the .pyc/.pyo files. Can you run "file" on the .pyc files and confirm which version of Python they're bytecode for? My guess is that it's bytecompiled them with /usr/bin/python, rather than your local build of python. Some notes: In older versions of RPM, brp-python-bytecompile took a single optional argument: the python interpreter to use, defaulting to /usr/bin/python. I generalized this to support multiple defaults when adding Python 3 support to Fedora: see https://bugzilla.redhat.com/show_bug.cgi?id=531117 That patch could be generalized to support /usr/local/lib. [1] In Fedora we do this using "__os_install_post", which is defined in /usr/lib/rpm/redhat/macros (from the redhat-rpm-config package), which has the invocation of /usr/lib/rpm/brp-python-bytecompile So it could be possible to override the python interpreter to use by redefining __os_install_post |
|
|
| msg159396 - (view) |
Author: Dave Malcolm (dmalcolm) |
Date: 2012-04-26 16:26 |
| __os_install_post is defined within /usr/lib/rpm/redhat/macros and contains this fragment: /usr/lib/rpm/brp-python-bytecompile %{__python} %{?_python_bytecompile_errors_terminate_build} \ Hence it will use %{__python} as the default when byte-compiling. %__python has the default definition of /usr/bin/python, but this can be overridden, either in the spec file using: %global __python some_path_to_the_python_to_use or in the command-line invocation of rpmbuild: rpmbuild --define="__python some_path_to_the_python_to_use" You can use the --showrc option to rpmbuild to see all of the macro expansions it's using (lots of output). So if it is indeed using the wrong python executable to do the bytecompiling, the above ought to fix it. The simplest fix would probably be for bdist_rpm to add: --define="__python %s" % sys.executable to the rpmbuild invocation, I think. |
|
|
| msg159437 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2012-04-27 01:13 |
| I tried the simple fix a couple of different ways on the RHEL6 buildbot. First by changing line 315 of Lib/distutils/command/bdist_rpm.py" to be: rpm_cmd = ['rpmbuild', '--define', '__python %s' % sys.executable] And then a second time by adding the new "--define" after the "-ba" option. (changing the spawn command to show the full command on error indicating the new arguments *were* being passed in, though) Neither worked - I still got the same error. Since I'm seeing the same fault on my Fedora system, I'll start poking around there instead (not sure when I'll get to it though). |
|
|
| msg161216 - (view) |
Author: Ross Lagerwall (rosslagerwall) |
Date: 2012-05-20 16:30 |
| The attached patch fixed the test for me on Fedora 16. It was necessary for the `define` to be after the -ba switch. I don't know why this wouldn't work on RHEL6 then... |
|
|
| msg161772 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2012-05-28 12:36 |
| New changeset 3d61e27cc570 by Nick Coghlan in branch '3.2': Issue #14443: Tell rpmbuild to use the correct version of Python http://hg.python.org/cpython/rev/3d61e27cc570 |
|
|
| msg161773 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2012-05-28 12:42 |
| Applied to 3.2 and trunk (I forgot to mention the issue name in the merge commit message). I tweaked Ross's patch slightly to: 1. Use self.python rather than sys.executable 2. Also default self.python to "python3" if neither --python nor --fix-python are specified (I don't think we currently test that, and I don't see any way we *could* test it from an uninstalled source build) Setting to pending while I wait to see how the RHEL6 buildbot reacts. Since the patch definitely resolves the issue on Fedora, it's quite possible I just fumble-fingered the earlier manual testing. Alternatively, there may be more to this issue, since RHEL6 almost certainly has an older version of rpmbuild than F16. |
|
|
| msg161782 - (view) |
Author: Ross Lagerwall (rosslagerwall) |
Date: 2012-05-28 13:54 |
| Unfortunately, it seems like it's still failing on the RHEL 6 buildbot. |
|
|
| msg161787 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2012-05-28 14:15 |
| Dave, sending this one back in your direction. Is there a relevant difference between rpmbuild in RHEL6 and rpmbuild in current Fedora? |
|
|
| msg161922 - (view) |
Author: Dave Malcolm (dmalcolm) |
Date: 2012-05-30 00:38 |
| I'm able to reproduce this on a RHEL 6 box, and I did some investigating. The stray .pyc files are indeed reported by "file" as "python 2.6 byte-compiled" so yes, it's using /usr/bin/python to byte-compile them On RHEL 6, with redhat-rpm-config-9.0.3-33.el6.noarch, /usr/lib/rpm/redhat/macros defines os_install_post as: %__os_install_post \ /usr/lib/rpm/redhat/brp-compress \ %{!?__debug_package:/usr/lib/rpm/redhat/brp-strip %{__strip}} \ /usr/lib/rpm/redhat/brp-strip-static-archive %{__strip} \ /usr/lib/rpm/redhat/brp-strip-comment-note %{__strip} %{__objdump} \ /usr/lib/rpm/brp-python-bytecompile \ /usr/lib/rpm/redhat/brp-python-hardlink \ %{!?__jar_repack:/usr/lib/rpm/redhat/brp-java-repack-jars} \ %{nil} Note how in this definition, brp-python-bytecompile is passed in without any arguments (contrast with , which is from a later version of the macros). Hence it unconditionally (and erroneously) uses /usr/bin/python to byte-compile any .py files found in the package payload. The change to add %{__python} to the invocation of /usr/lib/rpm/brp-python-bytecompile appears to have been in redhat-rpm-config-9.1.0 for Fedora 13 (see https://bugzilla.redhat.com/show_bug.cgi?id=521141), whereas RHEL 6 has the earlier code. It may be possible to work around this by providing an overridden definition of __os_install_post in the specfile. We do this in the python26.spec for EPEL5; grep for "__os_install_post" within: http://pkgs.fedoraproject.org/gitweb/?p=python26.git;a=blob;f=python26.spec;h=6b490b9b71f42c26b7d4ec4031685fb3230c5602;hb=refs/heads/el5 |
|
|
| msg161925 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2012-05-30 02:43 |
| Now I'm wondering if the problem will also manifest itself on EL rebuilds like CentOS or Scientific Linux, since that will affect how specific we can get when it comes to implementing a workaround. That is, overriding __os_install_post should definitely work, but the question then becomes how we decide that overriding it is *necessary*, such that: - we don't bypass vendor hooks when they aren't broken (e.g. Fedora, hopefully a future version of RHEL) - we *do* bypass them when we know they don't work properly for Python 3 projects (e.g. RHEL6, probably EL6 rebuilds) I'm currently thinking of something like this: # At the top of the bdist_rpm command file try: # Support workaround for #14443 which affects some RPM based systems # such as RHEL6 (and probably derivatives) from rpm import expandMacro as _expand_rpm_macro except ImportError: _expand_rpm_macro = None # When generating the spec file if _expand_rpm_macro is not None: # We're on a system with a vendor installed rpm library # and at least RHEL6 tries to bytecompile Python modules # with the system Python instead of the running one problem = "brp-python-bytecompile \\" fixed = "brp-python-bytecompile %{__python} \\" vendor_hook = _expand_rpm_macro("%{__os_install_post}") fixed_hook = vendor_hook.replace(problem, fixed) if fixed_hook != vendor_hook: # Add fixed_hook to spec file template Thoughts? |
|
|
| msg163940 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2012-06-25 11:30 |
| Ping. I would like to have this test failure fixed for 3.3 rc. |
|
|
| msg164003 - (view) |
Author: Dave Malcolm (dmalcolm) |
Date: 2012-06-25 19:07 |
| The approach Nick suggests seems reasonable, but rpm.expandMacro isn't usable: there's no guarantee that the machine has Python bindings for rpm installed (especially not for the version of Python that we've just built). However, we can get at the macro by shelling out to the "rpm" command: subprocess.getoutput('rpm --eval %{__os_install_post}') I'm attaching a patch which does this; works on my RHEL 6.3 test box. |
|
|
| msg164006 - (view) |
Author: Dave Malcolm (dmalcolm) |
Date: 2012-06-25 19:19 |
| (with the patch, it also continues to work on a test Fedora 15 box) |
|
|
| msg164007 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2012-06-25 19:53 |
| New changeset a9a12ad553f0 by David Malcolm in branch 'default': Issue #14443: ensure that brp-python-bytecompile is invoked with the correct http://hg.python.org/cpython/rev/a9a12ad553f0 |
|
|
| msg164088 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2012-06-26 18:10 |
| New changeset 76f2b99f3568 by David Malcolm in branch '3.2': Issue #14443: ensure that brp-python-bytecompile is invoked with the correct http://hg.python.org/cpython/rev/76f2b99f3568 |
|
|
| msg164091 - (view) |
Author: Dave Malcolm (dmalcolm) |
Date: 2012-06-26 19:21 |
| Fixes committed to 3.2 and default: RHEL 6.3 is now green for both branches. Closing this bug out. |
|
|