Tool recommendations - Python Packaging User Guide (original) (raw)

The Python packaging landscape consists of many different tools. For many tasks, the Python Packaging Authority(PyPA, the working group which encompasses many packaging tools and maintains this guide) purposefully does not make a blanket recommendation; for example, the reason there are many build backends is that the landscape was opened up in order to enable the development of new backends serving certain users’ needs better than the previously unique backend, setuptools. This guide does point to some tools that are widely recognized, and also makes some recommendations of tools that you should not use because they are deprecated or insecure.

Virtual environments¶

The standard tools to create and use virtual environments manually arevirtualenv (PyPA project) and venv (part of the Python standard library, though missing some features of virtualenv).

Installing packages¶

pip is the standard tool to install packages from PyPI. You may want to read pip’s recommendations forsecure installs. Pip is available by default in most Python installations through the standard library packageensurepip.

Alternatively, consider pipx for the specific use case of installing Python applications that are distributed through PyPI and run from the command line. Pipx is a wrapper around pip and venv that installs each application into a dedicated virtual environment. This avoids conflicts between the dependencies of different applications, and also with system-wide applications making use of the same Python interpreter (especially on Linux).

For scientific software specifically, consider conda or Spack.

Todo

Write a “pip vs. Conda” comparison, here or in a new discussion.

Do not use easy_install (part of Setuptools), which is deprecated in favor of pip (see pip vs easy_install for details). Likewise, donot use python setup.py install or python setup.py develop, which are also deprecated (see Is setup.py deprecated? for background andHow to modernize a setup.py based project? for migration advice).

Lock files¶

pip-tools and Pipenv are two recognized tools to create lock files, which contain the exact versions of all packages installed into an environment, for reproducibility purposes.

Build backends¶

Important

Please, remember: this document does not seek to steer the reader towards a particular tool, only to enumerate common tools. Different use cases often need specialized workflows.

Popular build backends for pure-Python packages include, in alphabetical order:

• Flit-core – developed with but separate from flit. A minimal and opinionated build backend. It does not support plugins.
• Hatchling – developed with but separate from hatch. Supports plugins.
• PDM-backend – developed with but separate from pdm. Supports plugins.
• Poetry-core – developed with but separate from poetry. Supports plugins.
  Unlike other backends on this list, Poetry-core does not support the standard[project] table (it uses a different format, in the [tool.poetry] table).
• Setuptools, which used to be the only build backend. Supports plugins.
  Caution
  If you use setuptools, please be aware that some features that predate standardisation efforts are now deprecated and only _temporarily kept_for compatibility.
  In particular, do not use direct python setup.py invocations. On the other hand, configuring setuptools with a setup.py file is still fully supported, although it is recommended to use the modern [project] table in pyproject.toml (or setup.cfg) whenever possible and keepsetup.py only if programmatic configuration is needed. SeeIs setup.py deprecated?.
  Other examples of deprecated features you should not use include thesetup_requires argument to setup() (use the [build-system] table in pyproject.toml instead), and the easy_install command (cf. pip vs easy_install).

Do not use distutils, which is deprecated, and has been removed from the standard library in Python 3.12, although it still remains available from setuptools.

For packages with extension modules, it is best to use a build system with dedicated support for the language the extension is written in, for example:

• Setuptools – natively supports C and C++ (with third-party plugins for Go and Rust),
• meson-python – C, C++, Fortran, Rust, and other languages supported by Meson,
• scikit-build-core – C, C++, Fortran, and other languages supported by CMake,
• Maturin – Rust, via Cargo.

Building distributions¶

The standard tool to build source distributions and wheels for uploading to PyPI is build. It will invoke whichever build backend you declared in pyproject.toml.

Do not use python setup.py sdist and python setup.py bdist_wheel for this task. All direct invocations of setup.py are deprecated.

If you have extension modules and want to distribute wheels for multiple platforms, use cibuildwheel as part of your CI setup to build distributable wheels.

Uploading to PyPI¶

For projects hosted on or published via supported CI/CD platforms, it is recommended to use the Trusted Publishing, which allows the package to be securely uploaded to PyPI from a CI/CD workflow without a manually configured API token.

As of November 2024, PyPI supports the following platforms as Trusted Publishing providers:

• GitHub Actions (on https://github.com)
• GitLab CI/CD (on https://gitlab.com)
• ActiveState
• Google Cloud

The other available method is to upload the package manually using twine.

Danger

Never use python setup.py upload for this task. In addition to beingdeprecated, it is insecure.

Workflow tools¶

These tools are environment managers that automatically manage virtual environments for a project. They also act as “task runners”, allowing you to define and invoke tasks such as running tests, compiling documentation, regenerating some files, etc. Some of them provide shortcuts for building distributions and uploading to PyPI, and some support lock files for applications. They often call the tools mentioned above under the hood. In alphabetical order:

• flit,
• hatch,
• nox,
• pdm,
• Pipenv,
• poetry,
• tox.