Issue 20210: Support the disabled marker in Setup files (original) (raw)
Created on 2014-01-09 22:52 by thomas-petazzoni, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (34)
Author: Thomas Petazzoni (thomas-petazzoni)
Date: 2014-01-09 22:52
In the context of space-constrained embedded Linux systems, installing the entire set of Python modules and extensions is not necessarily desirable. For example, all the test modules, as well as certain extensions requiring third-parties libraries are often unnecessary, and uselessly consume precious storage on the embedded Linux system.
While we could certainly remove these undesired modules and extensions manually, it is much more convenient to have configuration options to selectively enable and disable them. Another very strong benefit of having configuration options is that we can actually disable the build of these unneeded modules and extensions, therefore saving a lot of build time, which is very nice when you're repeatedly cross-compiling an entire embedded Linux system.
The proposed set of patches add several --enable-/--disable- options to enable/disable certain Python modules and extensions. These patches have been part of Buildroot, an embedded Linux build system (used for example by Google, and many embedded processor vendors, as well as a huge number of embedded system makers) for a while, and are useful to all our users using Python on their embedded Linux systems. Instead of carrying them around, we would like to have them merged in upstream Python.
Of course, we are definitely open to discussion on the approach to take to implement this configurability, and I'm ready to rework the patches according to the comments received here.
Thanks!
Author: Antoine Pitrou (pitrou) * 
Date: 2014-03-24 21:50
I don't really like the idea of complicating our build tools even more. Can't you simply prune the install tree yourself?
Author: Thomas Petazzoni (thomas-petazzoni)
Date: 2014-03-24 22:09
No, because it's not simply about the size of the installed Python standard library: it's also about the number of dependencies to build before being able to build Python. For example, a normal Python installation requires OpenSSL, libncurses, and lots of other things. On many embedded systems, those are not needed.
Author: Antoine Pitrou (pitrou) *
Date: 2014-03-24 22:11
For example, a normal Python installation requires OpenSSL, libncurses, and lots of other things.
Not really. If some development libraries are not available, Python should still install fine without the corresponding modules.
Author: Éric Araujo (eric.araujo) *
Date: 2014-03-24 22:23
The main issue with the proposed changes is that it redefines what “the Python standard library” is. Right now, users can mostly expect modules listed in the official Python docs to be available in their installation, regardless of how they got their Python.
I say “mostly” because a distributor may exclude tests from a binay package, split some extensions modules like _tkinter and _lzma in other packages, etc. A big source of pain in the past was distributors splitting distutils, but I think they all stopped.
(A related issue is that some distributors backport many bug fixes and sometimes features, which is a pain when you think you’ve run your tests with “Python X.Y.Z” and it was actually “X.Y.Z+some-patches”.)
Author: Thomas Petazzoni (thomas-petazzoni)
Date: 2014-03-24 22:32
But this expectation is not true: if dependencies are not available, Python silently disables the build of certain modules. So this story of making the standard library always has all the modules does not really stand.
Author: Matthias Klose (doko) *
Date: 2014-03-25 08:14
But this expectation is not true: if dependencies are not available, Python silently disables the build of certain modules. So this story of making the standard library always has all the modules does not really stand.
I think this one is a valid concern. Did run into it myself, because a final package was built on another kernel and disabled some semaphore stuff by accident.
None of the extensions above are built as builtins by default, so you can always prune these after the build. What I think is needed is a mode which breaks the build when some extensions are not built. Whether this should be the default or not, ... but it would give you a chance for a deterministic build. And the build system already tells you at the end of the extension builds that some extensions didn't build which were expected to build.
Author: Kubilay Kocak (koobs) 
Date: 2014-03-25 08:42
These are a good step toward bringing the otherwise neglected Python build system back to the real world in terms of standard functionality, and will among other things, make life an absolute pleasure for downstreams and users alike.
"User-Serviceable" options are expected (in particular in autotools-based build systems), not a luxury, and have been missing from the start.
A complicated build system is not a function its feature-set or flexibility, but of the quality of its evolution.
There is also a distinction between the ability to customise the options of a build, and the defaults of those options. "Will no longer be a standard library" is a straw man.
These patches present only as a user-configurable extension to otherwise statically defined configurations that must be patched manually to modify. This is painful.
With my downstream (FreeBSD) porter & consumer-and-hacker-of-Python-build-mechanics hat on, I'd like to see these and more 'options' out-of-the-box.
Author: Antoine Pitrou (pitrou) *
Date: 2014-03-25 10:52
A complicated build system is not a function its feature-set or flexibility, but of the quality of its evolution.
Certainly, but that doesn't change the concrete issue: we have a complicated build system that these patches will make more complicated.
Author: Mark Lawrence (BreamoreBoy) *
Date: 2014-03-25 14:48
I certainly like the principle. Does this need a wider audience, python-dev maybe?
Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) *
Date: 2014-03-26 08:55
If it is planned to support BSD make, then partial rewrite of patches will be needed.
Example of syntax of GNU make: ifeq (something,something) … endif
Example of syntax of FreeBSD make: .if ${variable}==something … .endif
Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) *
Date: 2014-03-26 09:16
According to koobs, building of CPython with FreeBSD make works at least with -j1 (and sporadically fails with higher value).
Author: Kubilay Kocak (koobs)
Date: 2014-03-26 09:18
More precisely:
Python 3.3 fails at anything > -j1 (switching to gmake makes this go away) Python 3.4 has not failed up to -j8 (with bsd make)
Author: STINNER Victor (vstinner) *
Date: 2014-03-26 10:33
Antoine Pitrou wrote: "I don't really like the idea of complicating our build tools even more. Can't you simply prune the install tree yourself?"
In the embedded world, the (cross) compilation process is very complex and slow. Being able to disable features makes this task simpler.
Éric Araujo wrote: "The main issue with the proposed changes is that it redefines what “the Python standard library” is."
I disagree. It's a common practice that a vendor gives the user the choice to enable or disable some features. On Gentoo and FreeBSD, you can disable features like IPv6 or shared memory on some packages. I don't think that it's currently possible on Python, but I would not be surprised to be able to enable or disable some features.
Éric Araujo wrote: "Right now, users can mostly expect modules listed in the official Python docs to be available in their installation, regardless of how they got their Python."
How you get Python matters :-) Python documentations describes the vanilla flavor distributed at python.org. But the Python license allows to strip some features without changing the name of the Python.
--
0002-Add-an-option-to-disable-installation-of-test-module.patch is interested. I never understand why Python installs its test suite. Who use this test suite installed on the system? Maybe the packager of the module to test Python. Ok, but the test suite can then be removed.
I like the overall approach, by individual patches may be discussed. For example, 0008-Add-an-option-to-disable-expat.patch breaks Python XML modules. Are they still be installed? I mean the modules implemented in Python and relying on the expat Python module.
The changes should be be documented somewhere. In the Python documentation, or at least in the "devguide".
Author: Antoine Pitrou (pitrou) *
Date: 2014-03-26 10:39
Antoine Pitrou wrote: "I don't really like the idea of complicating our build tools even more. Can't you simply prune the install tree yourself?"
In the embedded world, the (cross) compilation process is very complex and slow. Being able to disable features makes this task simpler.
That's not really the point. The question is why we should have to maintain this ourselves. It is easy for interested people to maintain their own forks, especially when removing stuff.
For the record, we don't have a single cross-compiling buildbot: it isn't a supported setup.
0002-Add-an-option-to-disable-installation-of-test-module.patch is interested. I never understand why Python installs its test suite. Who use this test suite installed on the system? Maybe the packager of the module to test Python. Ok, but the test suite can then be removed.
How else do you want to test that your Python installation works, other than running the test suite?
Author: Martin v. Löwis (loewis) *
Date: 2014-03-26 11:31
I'm +1 on the general idea, but -1 on the implementation strategy used.
Instead of coming up with configure options for selected (apparently problematic) modules, I'd like to see a solution that covers all extension modules.
One approach could be to reserve the option prefix --enable-mod-XXX for this kind of configuration, allowing people to specify --disable-mod-zipimport (for example).
Another approach (closer to what we already have) would be to support a disabled marker in Modules/Setup (and Modules/Setup.local), so anybody wishing to disable modules could put
disabled zipimport _sre unicodedata
into Modules/Setup.local (rather than specifying it on the configure command line).
Author: Mark Lawrence (BreamoreBoy) *
Date: 2015-03-01 17:00
@Thomas do you intend following this up as people certainly seem interested?
Author: Kubilay Kocak (koobs)
Date: 2015-03-02 00:53
For what it's worth, we've been running with patch 0001 on FreeBSD's Python 3.4 port for 8 months with no issues [1].
It allows downstreams like us to easily customise what modules gets built, and offers substantial flexibility and benefits for various packaging scenarios.
This patch is the simplest approach that offers benefits without complicating the existing built infrastructure, or precluding a more 'elegant' solution if one is forthcoming in the future.
[1] https://svnweb.freebsd.org/ports/head/lang/python34/files/patch-issue20210
Author: Thomas Petazzoni (thomas-petazzoni)
Date: 2015-03-03 09:27
@Mark I would be happy to, but if you refer to the previous discussion about this bug report, the feedback was quite negative. And since I'm not really willing to do some clean up to finally get the patches rejected, I'd like to at least have 1/ an agreement on the principle itself, and 2/ some guidance as to what needs to be changed in the currently proposed patches to get them accepted.
Author: Krasimir (krasimir_vanev)
Date: 2015-04-27 20:12
@Thomas: Thank you for the patches! Adding more flexibility to the build system that allows for cross-compiling and building "embeddable/distributable" python is definitely something that needs to be done is my opinion. So I definitely find your work very valuable to me. I admit that I use all patches, related to feature configuration from http://git.buildroot.net/buildroot/tree/package/python Thank you and keep up the great work! I'm positive about the direction that a flexibility/configurability should be in place, especially when talking for such a mature project.
Author: Stefan Krah (skrah) *
Date: 2016-04-10 12:29
Normally I hate environment variables, but perhaps in this case PYTHON_DISABLE_MODULES="foo,bar,quux" would be sufficient?
Setup.py already has the "disabled_module_list" variable, it's just a matter of setting this variable somehow.
This is really something for specialists, not general users.
Author: Matthias Klose (doko) *
Date: 2016-06-04 23:33
fyi, this came up again at the 2016 Language Summit, the request here was to provide some minimal packages which are just enough to run a cloud image (the cloud-init package requires a somehow minimal python).
Author: Thomas Petazzoni (thomas-petazzoni)
Date: 2016-06-06 20:10
Hello,
On Sat, 04 Jun 2016 23:33:00 +0000, Matthias Klose wrote:
fyi, this came up again at the 2016 Language Summit, the request here was to provide some minimal packages which are just enough to run a cloud image (the cloud-init package requires a somehow minimal python).
Thanks for the feedback! Definitely glad to hear that some other folks are interested in reducing the footprint of the default Python installation. Since embedded people have been ignored, hopefully the cloud people will have more luck in this endeavor :)
Thanks!
Thomas
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
Author: STINNER Victor (vstinner) *
Date: 2016-06-06 20:50
Fedora 24 also uses a minimal "system python" to get small cloud image: https://fedoraproject.org/wiki/Changes/System_Python#Benefit_to_Fedora
The base package only contains the minimum subset of the stdlib, the full stdlib comes with a second package based on the first one.
Author: Marc-Andre Lemburg (lemburg) *
Date: 2016-06-06 22:16
Just found this ticket and apparently late to the game...
Some comments:
It is already possible to selectively build Python C extension modules in the stdlib via editing the Modules/Setup file and this is a lot better to maintain than a long list of configure options.
Martin's suggestion to add support for a disabled option would make this even easier. This would also complement the static and shared option we have for Modules/Setup to define modules to compile into the main executable rather than as shared module.
The only bit that's missing (if really needed), is to prevent setup.py from building modules not listed in Modules/Setup as shared libs. This could be had with a single configure option, e.g. --disable-setuppy-builds.
But all that said, I don't think, it's really necessary to cripple Python just to save some image space. As you can see with our eGenix PyRun runtime, it is possible to compress Python down to just a few MBs:
http://www.egenix.com/products/python/PyRun/
while still having most of the stdlib readily available.
Author: STINNER Victor (vstinner) *
Date: 2017-02-10 11:32
Antoine Pitrou: "That's not really the point. The question is why we should have to maintain this ourselves. It is easy for interested people to maintain their own forks, especially when removing stuff."
It's painful to have to maintain downstream patches. Supporting this feature would make Python usable on more platforms.
Marc-Andre Lemburg: "it is possible to compress Python down to just a few MBs: (...)"
Right, that "a few MBs" can be too big on small devices, and it has a price on runtime performance, at least on the boot duration.
Author: Antoine Pitrou (pitrou) *
Date: 2017-02-10 11:35
It's painful to have to maintain downstream patches. Supporting this feature would make Python usable on more platforms.
Who volunteers to support it? It's a PITA to maintain the configure script and setup.py, and adding options only makes it more painful. This kind of niche features needs some dedicated maintainer so that other core developers don't have to bother with it.
Author: Marc-Andre Lemburg (lemburg) *
Date: 2017-02-10 11:46
I'm still -1 on the approach taken by the OP, but +1 on Martin's approach of making such configurations more easily possible via Modules/Setup.
Author: STINNER Victor (vstinner) *
Date: 2017-02-10 11:59
Antoine Pitrou: " Who volunteers to support it? It's a PITA to maintain the configure script and setup.py, and adding options only makes it more painful. This kind of niche features needs some dedicated maintainer so that other core developers don't have to bother with it."
Oh, I forgot to put more context in my previous comment.
Xavier de Gaye opened the issue #27640 "add the '--disable-test-suite' option to configure". He is a core developer working on porting CPython to Android. Xavier is the requested dedicated maintainer ;-)
Author: Antoine Pitrou (pitrou) *
Date: 2017-02-10 13:45
Victor, thanks for clarifying! That eliminates my objection, if there's a tacit agreement that the functionality can be broken by mistake and it's the dedicated maintainer (Xavier or someone else)'s job to repair it :-)
Author: Matthias Klose (doko) *
Date: 2017-02-10 14:07
I don't see the point of having an option to do that. Will patches for third party projects be submitted as well, to not install the test suite? Or will pip gain such a feature? Note that removing the test suite completely, you'll make the test suites of some third party packages useless in some cases (I had a few bug reports for the Ubuntu packaging when I removed the tests unconditionally), but probably you don't care about these either.
Author: Xavier de Gaye (xdegaye) *
Date: 2017-02-16 11:28
The PR https://github.com/python/cpython/pull/132 implements the support of the disabled marker in a Setup file as suggested by Martin.
Author: Thomas Wouters (twouters) *
Date: 2017-05-22 19:34
This feature would also be nice to have for Python at Google: we want builds to be as reproducible as possible, avoid third-party libraries even when they are available on the build system, and avoid certain extension modules in any form. We currently delete them after building, but suppressing their builds makes much more sense.
Author: Xavier de Gaye (xdegaye) *
Date: 2017-05-27 16:25
New changeset c0364fc7c2693fb070917ee62aeb8d2551710821 by xdegaye in branch 'master': bpo-20210: Support the disabled marker in Setup files (GH-132) https://github.com/python/cpython/commit/c0364fc7c2693fb070917ee62aeb8d2551710821
History
Date
User
Action
Args
2022-04-11 14:57:56
admin
set
github: 64409
2017-10-21 14:13:11
xdegaye
set
pull_requests: - <pull%5Frequest2537>
2017-06-28 19:22:58
xdegaye
set
pull_requests: + <pull%5Frequest2537>
2017-05-27 16:28:37
xdegaye
set
status: open -> closed
title: Provide configure options to enable/disable Python modules and extensions -> Support the *disabled* marker in Setup files
resolution: fixed
stage: resolved
2017-05-27 16:25:05
xdegaye
set
messages: +
2017-05-22 19:34:25
twouters
set
nosy: + twouters
messages: +
2017-02-16 11:28:24
xdegaye
set
messages: +
versions: + Python 3.7, - Python 3.5
2017-02-16 11:21:52
xdegaye
set
pull_requests: + <pull%5Frequest92>
2017-02-10 14:07:25
doko
set
messages: +
2017-02-10 13:45:32
pitrou
set
messages: +
2017-02-10 11:59:32
vstinner
set
nosy: + xdegaye
messages: +
2017-02-10 11:46:42
lemburg
set
messages: +
2017-02-10 11:35:24
pitrou
set
messages: +
2017-02-10 11:32:33
vstinner
set
messages: +
2016-06-06 22:16:22
lemburg
set
nosy: + lemburg
messages: +
2016-06-06 20:50:52
vstinner
set
messages: +
2016-06-06 20:10:36
thomas-petazzoni
set
messages: +
2016-06-04 23:33:00
doko
set
messages: +
2016-04-10 15:45:06
BreamoreBoy
set
nosy: - BreamoreBoy
2016-04-10 12:29:33
skrah
set
nosy: + skrah
messages: +
2015-12-06 04:36:50
wdv4758h
set
nosy: + wdv4758h
2015-04-27 20:12:13
krasimir_vanev
set
nosy: + krasimir_vanev
messages: +
2015-03-03 09:27:19
thomas-petazzoni
set
messages: +
2015-03-02 00:53:58
koobs
set
messages: +
2015-03-01 17:00:36
BreamoreBoy
set
messages: +
2014-03-26 11:31:34
loewis
set
messages: +
2014-03-26 10:39:37
pitrou
set
nosy: + loewis
2014-03-26 10:39:12
pitrou
set
messages: +
2014-03-26 10:33:31
vstinner
set
messages: +
2014-03-26 09🔞19
koobs
set
messages: +
2014-03-26 09:16:42
Arfrever
set
messages: +
2014-03-26 08:55:24
Arfrever
set
nosy: + Arfrever
messages: +
2014-03-25 14:48:24
BreamoreBoy
set
nosy: + BreamoreBoy
messages: +
2014-03-25 10:52:21
pitrou
set
messages: +
2014-03-25 08:42:11
koobs
set
messages: +
2014-03-25 08:14:11
doko
set
messages: +
2014-03-24 22:32:01
thomas-petazzoni
set
messages: +
2014-03-24 22:23:46
eric.araujo
set
nosy: + eric.araujo
messages: +
2014-03-24 22:11:28
pitrou
set
messages: +
2014-03-24 22:09:03
thomas-petazzoni
set
messages: +
2014-03-24 21:50:31
pitrou
set
nosy: + pitrou
messages: +
2014-03-24 11:41:37
koobs
set
nosy: + koobs
2014-02-17 23:07:32
vstinner
set
2014-01-09 22:54:48
thomas-petazzoni
set
files: + 0012-Add-an-option-to-disable-IDLE.patch
2014-01-09 22:54:41
thomas-petazzoni
set
files: + 0011-Add-an-option-to-disable-unicodedata.patch
2014-01-09 22:54:35
thomas-petazzoni
set
files: + 0010-Add-an-option-to-disable-NIS.patch
2014-01-09 22:54:28
thomas-petazzoni
set
files: + 0009-Add-an-option-to-disable-CJK-codecs.patch
2014-01-09 22:54:17
thomas-petazzoni
set
files: + 0008-Add-an-option-to-disable-expat.patch
2014-01-09 22:54:04
thomas-petazzoni
set
files: + 0007-Add-an-option-to-disable-the-curses-module.patch
2014-01-09 22:53:55
thomas-petazzoni
set
files: + 0006-Add-an-option-to-disable-the-tk-module.patch
2014-01-09 22:53:47
thomas-petazzoni
set
files: + 0005-Add-option-to-disable-the-sqlite3-module.patch
2014-01-09 22:53:33
thomas-petazzoni
set
files: + 0004-Add-an-option-to-disable-lib2to3.patch
2014-01-09 22:53:25
thomas-petazzoni
set
files: + 0003-Add-an-option-to-disable-pydoc.patch
2014-01-09 22:53:16
thomas-petazzoni
set
files: + 0002-Add-an-option-to-disable-installation-of-test-module.patch
2014-01-09 22:53:04
thomas-petazzoni
set
files: + 0001-Add-infrastructure-to-disable-the-build-of-certain-e.patch
keywords: + patch
2014-01-09 22:52:52
thomas-petazzoni
set
type: enhancement
2014-01-09 22:52:42
thomas-petazzoni
create