Compiled packaging (original) (raw)

Table of contents

Packaging Compiled Projects

There are a variety of ways to package compiled projects. In the past, the only way to do it was to use setuptools/distutils, which required using lots of fragile internals - distutils was intended primarily to compile CPython, and setuptools tried to stay away from changing the compile portions except as required. Now, however, we have several very nice options for compiled packages!

The most exciting developments have been new native build backends:

You should be familiar with packing a pure Python project - the metadata configuration is the same.

There are also classic setuptools plugins:

If you have a really complex build, the newer native build backends might not support your use case yet, but if that’s the case, ask - development is driven by community needs. The older, more fragile setuptools based plugins are still a bit more flexible if you really need that flexibility for a feature not yet implemented in the native backends.

pyproject.toml: build-system

PY001 Packages must have a pyproject.toml file PP001 that selects the backend:

[build-system]
requires = ["scikit-build-core"]
build-backend = "scikit_build_core.build"

[build-system]
requires = ["meson-python"]
build-backend = "mesonpy"

[build-system]
requires = ["maturin"]
build-backend = "maturin"

pyproject.toml: project table

The metadata is specified in a standards-based format:

[project]
name = "package"
description = "A great package."
readme = "README.md"
license = "BSD-3-Clause"
license-files = ["LICENSE"]
authors = [
  { name = "My Name", email = "me@email.com" },
]
maintainers = [
  { name = "My Organization", email = "myemail@email.com" },
]
requires-python = ">=3.9"

dependencies = [
  "typing_extensions",
]

classifiers = [
  "Development Status :: 4 - Beta",
  "Programming Language :: Python :: 3 :: Only",
  "Programming Language :: Python :: 3.9",
  "Programming Language :: Python :: 3.10",
  "Programming Language :: Python :: 3.11",
  "Programming Language :: Python :: 3.12",
  "Programming Language :: Python :: 3.13",
  "Topic :: Scientific/Engineering :: Physics",
]

[project.urls]
Homepage = "https://github.com/organization/package"
Documentation = "https://package.readthedocs.io/"
"Bug Tracker" = "https://github.com/organization/package/issues"
Discussions = "https://github.com/organization/package/discussions"
Changelog = "https://package.readthedocs.io/en/latest/changelog.html"

You can read more about each field, and all allowed fields, in packaging.python.org, Flit or Whey. Note that “Homepage” is special, and replaces the old url setting.

License

The license can be done one of two ways.

The modern way is to use the license field and an SPDX identifier expression. You can specify a list of files globs in license-files. Currently, hatchling>=1.26, flit-core>=1.11, pdm-backend>=2.4, setuptools>=77, and scikit-build-core>=0.12 support this. Only maturin, meson-python, and flit-core do not support this yet.

The classic convention uses one or more Trove Classifiers to specify the license. There also was a license.file field, required by meson-python, but other tools often did the wrong thing (such as load the entire file into the metadata’s free-form one line text field that was intended to describe deviations from the classifier license(s)).

classifiers = [
  "License :: OSI Approved :: BSD License",
]

You should not include the License :: classifiers if you use the license field PP007.

Sometimes you want to ship a package with optional dependencies. For example, you might have extra requirements that are only needed for running a CLI, or for plotting. Users must opt-in to get these dependencies by adding them to the package or wheel name when installing, like package[cli,mpl].

Here is an example of a simple extras:

[project.optional-dependencies]
cli = [
  "click",
]
mpl = [
  "matplotlib >=2.0",
]

Self dependencies can be used by using the name of the package, such as all = ["package[cli,mpl]"], (requires Pip 21.2+).

Command line

If you want to ship an “app” that a user can run from the command line, you need to add a script entry point. The form is:

[project.scripts]
cliapp = "package.__main__:main"

The format is command line app name as the key, and the value is the path to the function, followed by a colon, then the function to call. If you use __main__.py as the file, then python -m followed by the module will also work to call the app (__name__ will be "__main__" in that case).

Development dependencies

It is recommended to use dependency-groups instead of making requirement files. This allows you to specify dependencies that are only needed for development; unlike extras, they are not available when installing via PyPI, but they are available for local installation, and the dev group is even installed by default when using uv.

Here is an example:

[dependency-groups]
test = [
  "pytest >=6.0",
]
dev = [
  { include-group = "test" },
]

You can include one dependency group in another. Most tools allow you to install groups using --group, like pip (25.1+), uv pip, and the high level uv interface. You do not need to install the package, though usually you do (the high level uv interface does). Nox, Tox, and cibuildwheel all support groups too. The dependency-groups package provides tools to get the dependencies, too.

These tools all read the project table. They also have extra configuration options in tool.* settings.

Backend specific files

Example CMakeLists.txt file (using pybind11, so include pybind11 in build-system.requires too):

cmake_minimum_required(VERSION 3.15...3.26)
project(${SKBUILD_PROJECT_NAME} LANGUAGES CXX)

set(PYBIND11_FINDPYTHON ON)
find_package(pybind11 CONFIG REQUIRED)

pybind11_add_module(_core MODULE src/main.cpp)
install(TARGETS _core DESTINATION ${SKBUILD_PROJECT_NAME})

Example meson.build file (using pybind11, so include pybind11 in build-system.requires too):

project(
    'package',
    'cpp',
    version: '0.1.0',
    license: 'BSD',
    meson_version: '>= 1.1.0',
    default_options: [
        'cpp_std=c++11',
    ],
)

py = import('python').find_installation(pure: false)
pybind11_dep = dependency('pybind11')

py.extension_module('_core',
    'src/main.cpp',
    subdir: 'package',
    install: true,
    dependencies : [pybind11_dep],
)

install_subdir('src/package', install_dir: py.get_install_dir() / 'package', strip_directory: true)

Example Cargo.toml file:

[package]
name = "package"
version = "0.1.0"
edition = "2018"

[lib]
name = "_core"
# "cdylib" is necessary to produce a shared library for Python to import from.
crate-type = ["cdylib"]

[dependencies]
rand = "0.8.3"

[dependencies.pyo3]
version = "0.19.1"
# "extension-module" tells pyo3 we want to build an extension module (skips linking against libpython.so)
# "abi3-py39" tells pyo3 (and maturin) to build using the stable ABI with minimum Python version 3.9
features = ["extension-module", "abi3-py39"]

Example compiled file

Example src/main.cpp file:

#include <pybind11/pybind11.h>

int add(int i, int j) { return i + j; }

namespace py = pybind11;

PYBIND11_MODULE(_core, m) {
  m.doc() = R"pbdoc(
      Pybind11 example plugin
      -----------------------
      .. currentmodule:: python_example
      .. autosummary::
         :toctree: _generate
         add
         subtract
  )pbdoc";

  m.def("add", &add, R"pbdoc(
      Add two numbers
      Some other explanation about the add function.
  )pbdoc");

  m.def("subtract", [](int i, int j) { return i - j; }, R"pbdoc(
      Subtract two numbers
      Some other explanation about the subtract function.
  )pbdoc");
}

Example src/main.cpp file:

#include <pybind11/pybind11.h>

int add(int i, int j) { return i + j; }

namespace py = pybind11;

PYBIND11_MODULE(_core, m) {
  m.doc() = R"pbdoc(
      Pybind11 example plugin
      -----------------------
      .. currentmodule:: python_example
      .. autosummary::
         :toctree: _generate
         add
         subtract
  )pbdoc";

  m.def("add", &add, R"pbdoc(
      Add two numbers
      Some other explanation about the add function.
  )pbdoc");

  m.def("subtract", [](int i, int j) { return i - j; }, R"pbdoc(
      Subtract two numbers
      Some other explanation about the subtract function.
  )pbdoc");
}

use pyo3::prelude::*;

#[pyfunction]
fn add(x: i64, y: i64) -> i64 {
    x + y
}

#[pyfunction]
fn subtract(x: i64, y: i64) -> i64 {
    x - y
}

/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
fn _core(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(add, m)?)?;
    m.add_function(wrap_pyfunction!(subtract, m)?)?;
    m.add("__version__", env!("CARGO_PKG_VERSION"))?;

    Ok(())
}

Package structure

The recommendation (followed above) is to have source code in /src, and the Python package files in /src/<package>.

Versioning

Check the documentation for the tools above to see what forms of dynamic versioning the tool supports.

Including/excluding files in the SDist

Each tool uses a different mechanism to include or remove files from the SDist, though the defaults are reasonable.

Distributing

Unlike pure Python, you’ll need to build redistributable wheels for each platform and supported Python version if you want to avoid compilation on the user’s system. See the CI page on wheels for a suggested workflow.

Special considerations

NumPy

Modern versions of NumPy (1.25+) allow you to target older versions when building, which is highly recommended, and this will become required in NumPy 2.0. Now you add:

#define NPY_TARGET_VERSION NPY_1_22_API_VERSION

(Where that number is whatever version you support as a minimum) then make sure you build with NumPy 1.25+ (or 2.0+ when it comes out). Before 1.25, it was necessary to actually pin the oldest NumPy you supported (the oldest-supported-numpy package is the easiest method). If you support Python < 3.9, you’ll have to use the old method for those versions.

If using pybind11, you don’t need NumPy at build-time in the first place.