gh-95299: Stop installing setuptools as a part of ensurepip and venv by pradyunsg · Pull Request #101039 · python/cpython (original) (raw)
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Conversation44 Commits21 Checks0 Files changed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})
This PR removes the bundled setuptools wheel from ensurepip, and stops installing setuptools in environments created by venv.
I based this PR off of my understanding of venv and ensurepip, and using the following search to locate all the relevant bits of code + ensuring that tests are happy locally.
In this PR, I've intentionally not touched the example in https://docs.python.org/3.12/library/venv.html#an-example-of-extending-envbuilder. That example is largely outdated and is likely better served being updated in a dedicated PR for it.
Hmm... @vstinner @encukou any opinions on what I should do with test_cppext (added in #91321) here?
That test has an implicit assumption that setuptools will be present in the virtual environment created by venv (added in #92639, as part of removing reliance on distutils).
Notably, some leading questions that affect what direction I could take for updating that test:
- Does this test need to use packaging tooling? (otherwise, I guess we can hand-craft building the extension and rely on that)
- If it should use setuptools, would it be OK for this test to require reaching out over the network to fetch setuptools via pip?
- Would it be OK to build the extension in pip-installable package, rather than using the deprecated approach of directly invoking setup.py?
Since it's cheap to revert commits/remove them from a PR, and nice to have a code change to discuss details over, I've gone ahead and pushed a change that assumes that the answer to the questions I raised above is yes, yes and yes. :)
Mentioning a GitHub UI thing that I find very useful (and non-initutive): you can click "force-pushed" on the event above to see what changed in the force pushes between the hashes that GitHub lists in the from ... to ... at the end of the event.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes seem reasonable to me, though I'm only the code owner for the venv and test_venv bits.
I guess we can hand-craft building the extension and rely on that
IMO, that would be best. Relatedly, I've been thinking how to handle distutils removal in the extension tutorial -- it should definitely mention setuptools, but a simple but working example command line, because setuptools is not always available.
Having that in the tests -- even in a way that only satisfies our CI on select platforms -- would help :)
CPython tests depending on setuptools isn't great in general, and neither is downloading code as part of the test run.
Anyway, if that sounds like more than you signed up for, go ahead with Setuptools.
I don't have time for a full PR review now, sorry! Whoever gets around to it, please test-with-buildbots before merging.
Anyway, if that sounds like more than you signed up for, go ahead with Setuptools.
I spent some time today looking into this after my work day, and I've not figured out the details in a portable manner. I'd prefer if this could be tackled in a follow up instead. :)
/cc @dstufft @ncoghlan as other experts on ensurepip.
I'm technically listed as an expert on ensurepip already and I'm ~sure that the changes I've made to it are correct; but it certainly won't hurt if y'all can review this. :)
🤖 New build scheduled with the buildbot fleet by @pradyunsg for commit 7457fd3 🤖
If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
I added a review comment that should fix the buildbot errors.
Should this also go in What's New, under Removed?
Should this also go in What's New, under Removed?
The inclusion of setuptools was always explicitly documented as an internal implementation detail, and as such, I don't think we have any obligation to report its removal. It could be mentioned in the same way that we might document the removal of an internal API - I did a quick check and the 3.11 What's New included "Removed the undocumented private float.__set_format__() method". So I guess there's a precedent.
Please add it as a convenience to users, even if there's technically no obligation. It's been very easy to end up depending on setuptools accidentally.
IMO, for removals/breaks, it's best to write What's New entries for the target audience of a poor soul who isn't too well-versed in Python but inherited some complex system with a “temporary” hack. Point them to a solution, don't apologize, but don't berate them either.
With that in mind, please mention pkg_resources, as that shows up in ImportErrors but the connection to setuptools may not be clear.
(cc @python/proofreaders -- hope the above is good general advice)
Thanks all! Holler on the issue, if there's any concerns! :)
carljm added a commit to carljm/cpython that referenced this pull request
webknjaz added a commit to webknjaz/cpython that referenced this pull request
This is a refactoring change that aims to simplify ensurepip.
Before it, this module had legacy infrastructure that made an
assumption that ensurepip would be provisioning more then just a
single wheel. That assumption is no longer true since [1][2][3].
In this change, the improvement is done around removing unnecessary
loops and supporting structures to change the assumptions to expect
only the bundled or replacement pip wheel.
webknjaz added a commit to webknjaz/cpython that referenced this pull request
This is a refactoring change that aims to simplify ensurepip.
Before it, this module had legacy infrastructure that made an
assumption that ensurepip would be provisioning more then just a
single wheel. That assumption is no longer true since [1][2][3].
In this change, the improvement is done around removing unnecessary
loops and supporting structures to change the assumptions to expect
only the bundled or replacement pip wheel.
webknjaz added a commit to webknjaz/cpython that referenced this pull request
This is a refactoring change that aims to simplify ensurepip.
Before it, this module had legacy infrastructure that made an
assumption that ensurepip would be provisioning more then just a
single wheel. That assumption is no longer true since [1][2][3].
In this change, the improvement is done around removing unnecessary
loops and supporting structures to change the assumptions to expect
only the bundled or replacement pip wheel.
webknjaz added a commit to webknjaz/cpython that referenced this pull request
This is a refactoring change that aims to simplify ensurepip.
Before it, this module had legacy infrastructure that made an
assumption that ensurepip would be provisioning more then just a
single wheel. That assumption is no longer true since [1][2][3].
In this change, the improvement is done around removing unnecessary
loops and supporting structures to change the assumptions to expect
only the bundled or replacement pip wheel.
webknjaz added a commit to webknjaz/cpython that referenced this pull request
This is a refactoring change that aims to simplify ensurepip.
Before it, this module had legacy infrastructure that made an
assumption that ensurepip would be provisioning more then just a
single wheel. That assumption is no longer true since [1][2][3].
In this change, the improvement is done around removing unnecessary
loops and supporting structures to change the assumptions to expect
only the bundled or replacement pip wheel.
edmorley added a commit to edmorley/get-pip that referenced this pull request
Currently get-pip.py installs not only pip, but also setuptools and
wheel by default, unless the --no-setuptools / --no-wheel (or
PIP_NO_SETUPTOOLS / PIP_NO_WHEEL env vars) are used.
This has historically been necessary, however, modern versions of pip
will now fallback to pyproject.toml (PEP 517: 1) based builds (which
will default to a setuptools backend, and thus automatically install
setuptools and wheel in the isolated build environment) if either
setuptools is not installed (as of pip 22.1: 2), or if wheel is not
installed (as of pip 23.1: 3).
In addition, as of Python 3.12, the stdlib's ensurepip and venv
modules no longer install setuptools, and only install pip (4).
As such, it is now time for get-pip.py to stop installing setuptools
and wheel by default on Python 3.12+, in order to:
- Act as another small step towards
pyproject.toml/ PEP 517 based builds eventually becoming the pip default. - Improve parity with the behaviour of
ensurepip/venvon Python 3.12+. - Allow
get-pip.pyto focus on its primary responsibility: bootstrapping Pip.
Closes pypa#200.
edmorley added a commit to edmorley/get-pip that referenced this pull request
Currently get-pip.py installs not only pip, but also setuptools and
wheel by default, unless the --no-setuptools / --no-wheel (or
PIP_NO_SETUPTOOLS / PIP_NO_WHEEL env vars) are used.
This has historically been necessary, however, modern versions of pip
will now fallback to pyproject.toml (PEP 517: 1) based builds (which
will default to a setuptools backend, and thus automatically install
setuptools and wheel in the isolated build environment) if either
setuptools is not installed (as of pip 22.1: 2), or if wheel is not
installed (as of pip 23.1: 3).
In addition, as of Python 3.12, the stdlib's ensurepip and venv
modules no longer install setuptools, and only install pip (4).
As such, it is now time for get-pip.py to stop installing setuptools
and wheel by default on Python 3.12+, in order to:
- Act as another small step towards
pyproject.toml/ PEP 517 based builds eventually becoming the pip default. - Improve parity with the behaviour of
ensurepip/venvon Python 3.12+. - Allow
get-pip.pyto focus on its primary responsibility: bootstrapping Pip.
Closes pypa#200.
edmorley added a commit to edmorley/get-pip that referenced this pull request
Currently get-pip.py installs not only pip, but also setuptools and
wheel by default, unless the --no-setuptools / --no-wheel (or
PIP_NO_SETUPTOOLS / PIP_NO_WHEEL env vars) are used.
This has historically been necessary, however, modern versions of pip
will now fallback to pyproject.toml (PEP 517: 1) based builds (which
will default to a setuptools backend, and thus automatically install
setuptools and wheel in the isolated build environment) if either
setuptools is not installed (as of pip 22.1: 2), or if wheel is not
installed (as of pip 23.1: 3).
In addition, as of Python 3.12, the stdlib's ensurepip and venv
modules no longer install setuptools, and only install pip (4).
As such, it is now time for get-pip.py to stop installing setuptools
and wheel by default on Python 3.12+, in order to:
- Act as another small step towards
pyproject.toml/ PEP 517 based builds eventually becoming the pip default. - Improve parity with the behaviour of
ensurepip/venvon Python 3.12+. - Allow
get-pip.pyto focus on its primary responsibility: bootstrapping Pip.
Closes pypa#200.
edmorley added a commit to edmorley/get-pip that referenced this pull request
Currently get-pip.py installs not only pip, but also setuptools and
wheel by default, unless the --no-setuptools / --no-wheel (or
PIP_NO_SETUPTOOLS / PIP_NO_WHEEL env vars) are used.
This has historically been necessary, however, modern versions of pip
will now fallback to pyproject.toml (PEP 517: 1) based builds (which
will default to a setuptools backend, and thus automatically install
setuptools and wheel in the isolated build environment) if either
setuptools is not installed (as of pip 22.1: 2), or if wheel is not
installed (as of pip 23.1: 3).
In addition, as of Python 3.12, the stdlib's ensurepip and venv
modules no longer install setuptools, and only install pip (4).
As such, it is now time for get-pip.py to stop installing setuptools
and wheel by default on Python 3.12+, in order to:
- Act as another small step towards
pyproject.toml/ PEP 517 based builds eventually becoming the pip default. - Improve parity with the behaviour of
ensurepip/venvon Python 3.12+. - Allow
get-pip.pyto focus on its primary responsibility: bootstrapping Pip.
Closes pypa#200.
edmorley added a commit to heroku/buildpacks-python that referenced this pull request
Currently the buildpack performs a system site-packages install of not only pip, but also setuptools and wheel. This has historically been necessary for pip to be able to build source distributions (sdists) for packages that don't ship with compatible wheels.
However:
- Thanks to PEP 518, packages can now (and many already do) specify an
explicit build backend using
[build-system]in theirpyproject.toml. The dependencies specified in that config (such as setuptools and wheel) will be installed by pip into an isolated and ephemeral build environment as part of the source distribution build process. Such packages therefore don't need/use globally installed setuptools/wheel versions. - As of pip v22.1, pip will now default to the isolated build environment
mode (along with a fallback legacy setuptools build backend), if the
setuptools package isn't installed globally. This means that packages
that haven't yet migrated to a PEP 518
pyproject.tomlbuild backend config can still build even if setuptools isn't installed globally.
There are a small number of rarely used packages in the wild that aren't compatible with build isolation mode, however, these typically require more build dependencies than just setuptools, which means they wouldn't have worked with this buildpack anyway.
As such, it's no longer necessary for us to install setuptools and wheel
globally. This matches the behaviour of the venv and ensurepip modules
in Python 3.12+, where setuptools and wheel installation has also been
removed. And it also matches the default behaviour of Poetry too, whose
install --sync command removes any implicitly installed packages in the
current environment (other than pip).
See: https://peps.python.org/pep-0518/ https://pip.pypa.io/en/stable/reference/build-system/ https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolation pypa/pip#10717 python/cpython#101039 pypa/get-pip#218 astral-sh/uv#2252
edmorley added a commit to heroku/buildpacks-python that referenced this pull request
Currently the buildpack performs a system site-packages install of not only pip, but also setuptools and wheel. This has historically been necessary for pip to be able to build source distributions (sdists) for packages that don't ship with compatible wheels.
However:
- Thanks to PEP 518, packages can now (and many already do) specify an
explicit build backend using
[build-system]in theirpyproject.toml. The dependencies specified in that config (such as setuptools and wheel) will be installed by pip into an isolated and ephemeral build environment as part of the source distribution build process. Such packages therefore don't need/use globally installed setuptools/wheel versions. - As of pip v22.1, pip will now default to the isolated build environment
mode (along with a fallback legacy setuptools build backend), if the
setuptools package isn't installed globally. This means that packages
that haven't yet migrated to a PEP 518
pyproject.tomlbuild backend config can still build even if setuptools isn't installed globally.
There are a small number of rarely used packages in the wild that aren't compatible with build isolation mode, however, these typically require more build dependencies than just setuptools, which means they wouldn't have worked with this buildpack anyway.
As such, it's no longer necessary for us to install setuptools and wheel
globally. This matches the behaviour of the venv and ensurepip modules
in Python 3.12+, where setuptools and wheel installation has also been
removed. And it also matches the default behaviour of Poetry too, whose
install --sync command removes any implicitly installed packages in the
current environment (other than pip).
See: https://peps.python.org/pep-0518/ https://pip.pypa.io/en/stable/reference/build-system/ https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolation pypa/pip#10717 python/cpython#101039 pypa/get-pip#218 astral-sh/uv#2252
edmorley added a commit to heroku/buildpacks-python that referenced this pull request
Currently the buildpack performs a system site-packages install of not only pip, but also setuptools and wheel. This has historically been necessary for pip to be able to build source distributions (sdists) for packages that don't ship with compatible wheels.
However:
- Thanks to PEP 518, packages can now (and many already do) specify an
explicit build backend using
[build-system]in theirpyproject.toml. The dependencies specified in that config (such as setuptools and wheel) will be installed by pip into an isolated and ephemeral build environment as part of the source distribution build process. Such packages therefore don't need/use globally installed setuptools/wheel versions. - As of pip v22.1, pip will now default to the isolated build environment
mode (along with a fallback legacy setuptools build backend), if the
setuptools package isn't installed globally. This means that packages
that haven't yet migrated to a PEP 518
pyproject.tomlbuild backend config can still build even if setuptools isn't installed globally.
There are a small number of rarely used packages in the wild that aren't compatible with build isolation mode, however, these typically require more build dependencies than just setuptools, which means they wouldn't have worked with this buildpack anyway.
As such, it's no longer necessary for us to install setuptools and wheel
globally. This matches the behaviour of the venv and ensurepip modules
in Python 3.12+, where setuptools and wheel installation has also been
removed. And it also matches the default behaviour of Poetry too, whose
install --sync command removes any implicitly installed packages in the
current environment (other than pip).
See: https://peps.python.org/pep-0518/ https://pip.pypa.io/en/stable/reference/build-system/ https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolation pypa/pip#10717 python/cpython#101039 pypa/get-pip#218 astral-sh/uv#2252
Tenzer added a commit to Tenzer/aws-lambda-python-runtime-interface-client that referenced this pull request
It is necessary for building the package when running the integration tests, but more recent Python versions no longer include it by default: python/cpython#101039.
This was referenced
Jan 1, 2025
Reviewers
