Building pyzmq — PyZMQ 27.0.0 documentation (original) (raw)

pyzmq publishes around a hundred wheels for each release, so hopefully very few folks need to build pyzmq from source.

pyzmq 26 has a whole new build system using CMake via scikit-build-core.

~all options can be specified via environment variables with the same name, in order to play nicely with pip.

Installing from source#

When compiling pyzmq, it is generally recommended that zeromq be installed separately, via homebrew, apt, yum, etc:

Debian-based

sudo apt-get install libzmq3-dev

Fedora-based

sudo yum install libzmq3-devel

homebrew

brew install zeromq

You can install pyzmq from source with pip by telling it --no-binary pyzmq:

python3 -m pip install pyzmq --no-binary pyzmq

or an editable install from a local checkout:

python3 -m pip install -e .

Building from source uses CMake via scikit-build-core. CMake >= 3.28 is required. scikit-build-core will attempt to download cmake if a satisfactory version is not found.

Examples#

First, some quick examples of influencing pyzmq’s build.

Build a wheel against already-installed libzmq:

export ZMQ_PREFIX=/usr/local python3 -m pip install pyzmq --no-binary pyzmq

Force building bundled libzmq with the draft API:

export ZMQ_PREFIX=bundled export ZMQ_BUILD_DRAFT=1 python3 -m pip install pyzmq --no-binary pyzmq

Finding libzmq#

First, pyzmq tries to find libzmq to link against it.

pyzmq will first try to search using standard CMake methods, followed by pkg-config.

You can pass through arguments to the build system via the CMAKE_ARGS environment variable. e.g.

CMAKE_ARGS="-DCMAKE_PREFIX_PATH=/path/to/something"

PKG_CONFIG_PATH="$PREFIX/lib/pkgconfig"

If pyzmq doesn’t find your libzmq via the default search, or you want to skip the search and tell pyzmq exactly where to look, set ZMQ_PREFIX (this skips cmake/pkgconfig entirely):

ZMQ_PREFIX=/path/to/zmq # should contain 'include', 'lib', etc.

You may also need to add:

export LDFLAGS="-Wl,-rpath,${ZMQ_PREFIX}/lib"

to make sure that your libzmq is found at runtime.

Disabling bundled build fallback#

You may want to keep the default search, which will import targets from CMake, pkg-config, etc., but make sure libzmq is found.

To do this, set PYZMQ_NO_BUNDLE. If you set only this, pyzmq will still search via standard means, but fail if libzmq is not found, rather than falling back on the bundled static library.

Building bundled libzmq#

If pyzmq doesn’t find a libzmq to link to, it will fall back on building libzmq itself. You can tell pyzmq to skip searching for libzmq and always build the bundled version with ZMQ_PREFIX=bundled.

When building a bundled libzmq, pyzmq downloads and builds libzmq and libsodium as static libraries. These static libraries are then linked to by the pyzmq extension and discarded.

Bundled libzmq is supported on a best-effort basis, and isn’t expected to work everywhere with zero configuration. If you have trouble building bundled libzmq, please do report it. But the best solution is usually to install libzmq yourself via the appropriate mechanism before building pyzmq.

Building bundled libsodium#

libsodium is built first, with configure most places:

./configure --enable-static --disable-shared --with-pic make make install

or msbuild on Windows:

msbuild /m /v:n /p:Configuration=StaticRelease /pPlatform=x64 builds/msvc/vs2022/libsodium.sln

You can add arguments to configure with a semicolon-separated list, by specifying PYZMQ_LIBSODIUM_CONFIGURE_ARGS variable:

PYZMQ_LIBSODIUM_CONFIGURE_ARGS="--without-pthread --enable-minimal"

or

CMAKE_ARGS="-DPYZMQ_LIBSODIUM_CONFIGURE_ARGS=--without-pthread;--enable-minimal"

and PYZMQ_LIBSODIUM_MSBUILD_ARGS on Windows:

PYZMQ_LIBSODIUM_MSBUILD_ARGS="/something /else"

or

CMAKE_ARGS="-DPYZMQ_LIBSODIUM_MSBUILD_ARGS=/something;/else"

Note

command-line arguments from environment variables are expected to be space-separated (-a -b), while CMake variables are expected to be CMake lists (semicolon-separated) (-a;-b).

Building bundled libzmq#

The libzmq-static static library target is imported via FetchContent, which means the libzmq CMake build is used on all platforms. This means that configuring the build of libzmq itself is done directly via CMAKE_ARGS, and all of libzmq’s cmake flags should be available. See libzmq’s install docs for more.

For example, to enable OpenPGM:

CMAKE_ARGS="-DWITH_OPENPGM=ON"

Specifying bundled versions#

You can specify which version of libsodium/libzmq to bundle with:

-DPYZMQ_LIBZMQ_VERSION=4.3.5 -DPYZMQ_LIBSODIUM_VERSION=1.0.20

or the specify the full URL to download (e.g. to test bundling an unreleased version):

-DPYZMQ_LIBZMQ_URL="https://github.com/zeromq/libzmq/releases/download/v4.3.5/zeromq-4.3.5.tar.gz" -DPYZMQ_LIBSODIUM_URL="https://download.libsodium.org/libsodium/releases/libsodium-1.0.20.tar.gz"

Warning

Only the default versions are supported and there is no guarantee that bundling versions will work, but you are welcome to try!

Windows notes#

I’m not at all confident in building things on Windows, but so far things work in CI. I’ve done my best to expose options to allow users to override things if they don’t work, but it’s not really meant to be customizable; it’s meant to allow you to workaround my mistakes without waiting for a release.

libsodium ships several solutions for msbuild, identified by /builds/msvc/vs{year}/libsodium.sln. pyzmq tries to guess which solution to use based on the MSVC_VERSION CMake variable, but you can skip the guess by specifying -D PYZMQ_LIBSODIUM_VS_VERSION=2022 to explicitly use the vs2022 solution.

Passing arguments#

pyzmq has a few CMake options to influence the build. All options are settable as environment variables, as well. Other than ZMQ_PREFIX and ZMQ_DRAFT_API which have been around forever, environment variables for building pyzmq have the prefix PYZMQ_.

The _ARGS variables that are meant to pass-through command-line strings accept standard command-line format from environment, or semicolon-separated lists when specified directly to cmake.

export ZMQ_PREFIX=bundled export PYZMQ_LIBZMQ_VERSION=4.3.4 export PYZMQ_LIBSODIUM_CONFIGURE_ARGS=--disable-pie --minimal

python3 -m build .

is equivalent to

export CMAKE_ARGS="-DZMQ_PREFIX=bundled -DPYZMQ_LIBZMQ_VERSION=4.3.4 -DPYZMQ_LIBSODIUM_CONFIGURE_ARGS=--disable-pie;--minimal" python3 -m build .

Most cmake options can be seen below:

cmake -LH output for pyzmq, which can be passed via CMAKE_ARGS. Most of these can also be specified via environment variables.

Path to a program.

CYTHON:FILEPATH=$PREFIX/bin/cython

semicolon-separated list of arguments to pass to ./configure for bundled libsodium

PYZMQ_LIBSODIUM_CONFIGURE_ARGS:STRING=

semicolon-separated list of arguments to pass to msbuild for bundled libsodium

PYZMQ_LIBSODIUM_MSBUILD_ARGS:STRING=

full URL to download bundled libsodium

PYZMQ_LIBSODIUM_URL:STRING=

libsodium version when bundling

PYZMQ_LIBSODIUM_VERSION:STRING=1.0.20

Visual studio solution version for bundled libsodium (default: detect from MSVC_VERSION)

PYZMQ_LIBSODIUM_VS_VERSION:STRING=

full URL to download bundled libzmq

PYZMQ_LIBZMQ_URL:STRING=

libzmq version when bundling

PYZMQ_LIBZMQ_VERSION:STRING=4.3.5

Prohibit building bundled libzmq. Useful for repackaging, to allow default search for libzmq and requiring it to succeed.

PYZMQ_NO_BUNDLE:BOOL=OFF

whether to build the libzmq draft API

ZMQ_DRAFT_API:BOOL=OFF

libzmq installation prefix or 'bundled'

ZMQ_PREFIX:STRING=auto

The directory containing a CMake configuration file for ZeroMQ.

ZeroMQ_DIR:PATH=$PREFIX/lib/cmake/ZeroMQ

cmake -LH output for libzmq, showing additional arguments that can be passed to CMAKE_ARGS when building bundled libzmq

Path to a program.

A2X_EXECUTABLE:FILEPATH=A2X_EXECUTABLE-NOTFOUND

Choose polling system for zmq_poll(er)_*. valid values are

poll or select [default=poll unless POLLER=select]

API_POLLER:STRING=

Whether or not to build the shared object

BUILD_SHARED:BOOL=ON

Whether or not to build the static archive

BUILD_STATIC:BOOL=ON

Whether or not to build the tests

BUILD_TESTS:BOOL=ON

Build with static analysis(make take very long)

ENABLE_ANALYSIS:BOOL=OFF

Build with address sanitizer

ENABLE_ASAN:BOOL=OFF

Run tests that require sudo and capsh (for cap_net_admin)

ENABLE_CAPSH:BOOL=OFF

Include Clang

ENABLE_CLANG:BOOL=ON

Enables cpack rules

ENABLE_CPACK:BOOL=ON

Enable CURVE security

ENABLE_CURVE:BOOL=OFF

Build and install draft classes and methods

ENABLE_DRAFTS:BOOL=ON

Enable/disable eventfd

ENABLE_EVENTFD:BOOL=OFF

Build using compiler intrinsics for atomic ops

ENABLE_INTRINSICS:BOOL=OFF

Automatically close libsodium randombytes. Not threadsafe without getrandom()

ENABLE_LIBSODIUM_RANDOMBYTES_CLOSE:BOOL=ON

Build with empty ZMQ_EXPORT macro, bypassing platform-based automated detection

ENABLE_NO_EXPORT:BOOL=OFF

Enable precompiled headers, if possible

ENABLE_PRECOMPILED:BOOL=ON

Use radix tree implementation to manage subscriptions

ENABLE_RADIX_TREE:BOOL=ON

Build with thread sanitizer

ENABLE_TSAN:BOOL=OFF

Build with undefined behavior sanitizer

ENABLE_UBSAN:BOOL=OFF

Enable WebSocket transport

ENABLE_WS:BOOL=ON

LIBZMQ_PEDANTIC:BOOL=ON

LIBZMQ_WERROR:BOOL=OFF

Choose polling system for I/O threads. valid values are

kqueue, epoll, devpoll, pollset, poll or select [default=autodetect]

POLLER:STRING=

Path to a library.

RT_LIBRARY:FILEPATH=RT_LIBRARY-NOTFOUND

Build html docs

WITH_DOCS:BOOL=ON

Use libbsd instead of builtin strlcpy

WITH_LIBBSD:BOOL=ON

Use libsodium

WITH_LIBSODIUM:BOOL=OFF

Use static libsodium library

WITH_LIBSODIUM_STATIC:BOOL=OFF

Enable militant assertions

WITH_MILITANT:BOOL=OFF

Build with support for NORM

WITH_NORM:BOOL=OFF

Use NSS instead of builtin sha1

WITH_NSS:BOOL=OFF

Build with support for OpenPGM

WITH_OPENPGM:BOOL=OFF

Build with perf-tools

WITH_PERF_TOOL:BOOL=ON

Use TLS for WSS support

WITH_TLS:BOOL=ON

Build with support for VMware VMCI socket

WITH_VMCI:BOOL=OFF

install path for ZeroMQConfig.cmake

ZEROMQ_CMAKECONFIG_INSTALL_DIR:STRING=lib/cmake/ZeroMQ

ZeroMQ library

ZEROMQ_LIBRARY:STRING=libzmq

Build as OS X framework

ZMQ_BUILD_FRAMEWORK:BOOL=OFF

Build the tests for ZeroMQ

ZMQ_BUILD_TESTS:BOOL=ON

Choose condition_variable_t implementation. Valid values are

stl11, win32api, pthreads, none [default=autodetect]

ZMQ_CV_IMPL:STRING=stl11

Output zmq library base name

ZMQ_OUTPUT_BASENAME:STRING=zmq

Cross-compiling pyzmq#

Cross-compiling Python extensions is tricky!

To cross-compile pyzmq, in general you need:

Python built for the ‘build’ machine
Python built for the ‘host’ machine (identical version)
cross-compiling toolchain (e.g. aarch64-linux-gnu-gcc)
Python setup to cross-compile (crossenv is the popular tool these days, and includes lots of info for cross-compiling for Python, but pyzmq makes no assumptions)

It is probably a good idea to build libzmq/libsodium separately and link them with ZMQ_PREFIX, as cross-compiling bundled libzmq is not guaranteed to work.

I don’t have a lot of experience cross-compiling, but we have two example Dockerfiles that appear to work to cross-compile pyzmq. These aren’t official or supported, but they appear to work and may be useful as reference to get you started.

Dockerfile for building x86_64 on aarch64

FROM ubuntu:22.04 RUN apt-get -y update
&& apt-get -y install curl unzip cmake ninja-build openssl xz-utils build-essential libz-dev libssl-dev

ENV BUILD_PREFIX=/opt/build ENV PATH=${BUILD_PREFIX}/bin:$PATH

ARG PYTHON_VERSION=3.11.8 WORKDIR /src RUN curl -L -o python.tar.xz https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tar.xz
&& tar -xf python.tar.xz
&& rm python.tar.xz
&& mv Python-* cpython

build our 'build' python

WORKDIR /src/cpython RUN ./configure --prefix=${BUILD_PREFIX} RUN make -j4 RUN make install

sanity check

RUN python3 -c 'import ssl'
&& python3 -m ensurepip
&& python3 -m pip install --upgrade pip

get our cross-compile toolchain

I'm on aarch64, so use x86_64 as host

ENV BUILD="aarch64-linux-gnu" ENV HOST="x86_64-linux-gnu" RUN HOST_PKG=$(echo $HOST | sed s@_@-@g)
&& apt-get -y install binutils-$HOST_PKG gcc-$HOST_PKG g++-$HOST_PKG ENV CC=$HOST-gcc
CXX=$HOST-g++

build our 'host' python

WORKDIR /src/cpython RUN make clean ENV HOST_PREFIX=/opt/host RUN ./configure
--prefix=${HOST_PREFIX}
--host=$HOST
--build=$BUILD
--with-build-python=$BUILD_PREFIX/bin/python3
--without-ensurepip
ac_cv_buggy_getaddrinfo=no
ac_cv_file__dev_ptmx=yes
ac_cv_file__dev_ptc=no RUN make -j4 RUN make install

WORKDIR /src

# (optional) cross-compile libsodium, libzmq

WORKDIR /src

ENV LIBSODIUM_VERSION=1.0.20

RUN curl -L -O "https://download.libsodium.org/libsodium/releases/libsodium-${LIBSODIUM_VERSION}.tar.gz" \

&& tar -xzf libsodium-${LIBSODIUM_VERSION}.tar.gz \

&& mv libsodium-stable libsodium \

&& rm libsodium*.tar.gz

WORKDIR /src/libsodium

RUN ./configure --prefix="${HOST_PREFIX}" --host=$HOST

RUN make -j4

RUN make install

# build libzmq

WORKDIR /src

ENV LIBZMQ_VERSION=4.3.5

RUN curl -L -O "https://github.com/zeromq/libzmq/releases/download/v${LIBZMQ_VERSION}/zeromq-${LIBZMQ_VERSION}.tar.gz" \

&& tar -xzf zeromq-${LIBZMQ_VERSION}.tar.gz \

&& mv zeromq-${LIBZMQ_VERSION} zeromq

WORKDIR /src/zeromq

RUN ./configure --prefix="$HOST_PREFIX" --host=$HOST --disable-perf --disable-Werror --without-docs --enable-curve --with-libsodium=$HOST_PREFIX --disable-drafts --disable-libsodium_randombytes_close

RUN make -j4

RUN make install

setup crossenv

WORKDIR /src ENV CROSS_PREFIX=/opt/cross RUN python3 -m pip install crossenv
&& python3 -m crossenv HOSTPREFIX/bin/python3{HOST_PREFIX}/bin/python3 HOSTPREFIX/bin/python3{CROSS_PREFIX} ENV PATH=${CROSS_PREFIX}/bin:$PATH

install build dependencies in crossenv

RUN . ${CROSS_PREFIX}/bin/activate
&& build-pip install build pyproject_metadata scikit-build-core pathspec cython

if pyzmq is bundling libsodium, tell it to cross-compile

not required if libzmq is already installed

ENV PYZMQ_LIBSODIUM_CONFIGURE_ARGS="--host $HOST" ARG PYZMQ_VERSION=26.0.0b2

build wheel of pyzmq

WORKDIR /src RUN python3 -m pip download --no-binary pyzmq --pre pyzmq==$PYZMQ_VERSION
&& tar -xzf pyzmq-.tar.gz
&& rm pyzmq-.tar.gz
&& . ${CROSS_PREFIX}/bin/activate
&& cross-python -m build --no-isolation --skip-dependency-check --wheel ./pyzmq*

there is now a pyzmq wheel in /src/pyzmq-$version/dist/pyzmq-$VERSION-cp311-cp311-linux_x86_64.whl

Dockerfile for building for android-aarch64 on x86_64

FROM ubuntu:22.04 RUN apt-get -y update
&& apt-get -y install curl unzip cmake ninja-build openssl xz-utils build-essential libz-dev libssl-dev

ENV BUILD_PREFIX=/opt/build ENV PATH=${BUILD_PREFIX}/bin:$PATH

build our 'build' python

WORKDIR /src/cpython RUN ./configure --prefix=${BUILD_PREFIX} RUN make -j4 RUN make install

sanity check

RUN python3 -c 'import ssl'
&& python3 -m ensurepip
&& python3 -m pip install --upgrade pip

get our cross-compile toolchain from NDK

WORKDIR /opt RUN curl -L -o ndk.zip https://dl.google.com/android/repository/android-ndk-r26c-linux.zip
&& unzip ndk.zip
&& rm ndk.zip
&& mv android-* ndk ENV BUILD="x86_64-linux-gnu" ENV HOST="aarch64-linux-android34" ENV PATH=/opt/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin:$PATH ENV CC=$HOST-clang
CXX=$HOST-clang++
READELF=llvm-readelf

build our 'host' python

(optional) cross-compile libsodium, libzmq

WORKDIR /src ENV LIBSODIUM_VERSION=1.0.20 RUN curl -L -O "https://download.libsodium.org/libsodium/releases/libsodium-${LIBSODIUM_VERSION}.tar.gz"
&& tar -xzf libsodium-${LIBSODIUM_VERSION}.tar.gz
&& mv libsodium-stable libsodium
&& rm libsodium*.tar.gz

WORKDIR /src/libsodium

need CFLAGS or libsodium >= 1.0.20 https://github.com/android/ndk/issues/1945

ENV CFLAGS="-march=armv8-a+crypto" RUN ./configure --prefix="${HOST_PREFIX}" --host=$HOST RUN make -j4 RUN make install

build libzmq

WORKDIR /src ENV LIBZMQ_VERSION=4.3.5 RUN curl -L -O "https://github.com/zeromq/libzmq/releases/download/v${LIBZMQ_VERSION}/zeromq-${LIBZMQ_VERSION}.tar.gz"
&& tar -xzf zeromq-${LIBZMQ_VERSION}.tar.gz
&& mv zeromq-${LIBZMQ_VERSION} zeromq WORKDIR /src/zeromq RUN ./configure --prefix="$HOST_PREFIX" --host=$HOST --disable-perf --disable-Werror --without-docs --enable-curve --with-libsodium=$HOST_PREFIX --disable-drafts --disable-libsodium_randombytes_close RUN make -j4 RUN make install

setup crossenv

ENV CROSS_PREFIX=/opt/cross RUN python3 -m pip install crossenv
&& python3 -m crossenv HOSTPREFIX/bin/python3{HOST_PREFIX}/bin/python3 HOSTPREFIX/bin/python3{CROSS_PREFIX} ENV PATH=${CROSS_PREFIX}/bin:$PATH

install build dependencies in crossenv

RUN . ${CROSS_PREFIX}/bin/activate
&& build-pip install build pyproject_metadata scikit-build-core pathspec cython

ENV ZMQ_PREFIX=${HOST_PREFIX}

if pyzmq is bundling libsodium, tell it to cross-compile

not required if libzmq is already installed

ENV PYZMQ_LIBSODIUM_CONFIGURE_ARGS="--host $HOST" ARG PYZMQ_VERSION=26.0.0b2

build wheel of pyzmq

there is now a pyzmq wheel in /src/pyzmq-$VERSION/dist/pyzmq-$VERSION-cp311-cp311-linux_aarch64.whl