Running Docker Builds — Apache Arrow v21.0.0.dev53 (original) (raw)

Most of our Linux based Continuous Integration tasks are decoupled from public CI services using Docker andDocker Compose. Keeping the CI configuration minimal makes local reproducibility possible.

Usage#

There are multiple ways to execute the Docker based builds. The recommended way is to use the Archery tool:

Examples#

List the available images:

Execute a build:

archery docker run conda-python

Archery calls the following docker compose commands:

docker compose pull --ignore-pull-failures conda-cpp docker compose pull --ignore-pull-failures conda-python docker compose build conda-cpp docker compose build conda-python docker compose run --rm conda-python

Show the Docker Compose commands instead of executing them:

archery docker --dry-run run conda-python

To disable the image pulling:

archery docker run --no-cache conda-python

Which translates to:

docker compose build --no-cache conda-cpp docker compose build --no-cache conda-python docker compose run --rm conda-python

To disable the cache only for the leaf image:

Useful to force building the development version of a dependency. In case of the example below the command builds theconda-cpp > conda-python > conda-python-pandas branch of the image tree where the leaf image is conda-python-pandas.

PANDAS=upstream_devel archery docker run --no-leaf-cache conda-python-pandas

Which translates to:

export PANDAS=upstream_devel docker compose pull --ignore-pull-failures conda-cpp docker compose pull --ignore-pull-failures conda-python docker compose build conda-cpp docker compose build conda-python docker compose build --no-cache conda-python-pandas docker compose run --rm conda-python-pandas

Note that it doesn’t pull the conda-python-pandas image and disable the cache when building it.

PANDAS is a build parameter, see the defaults in the .env file.

To entirely skip building the image:

The layer-caching mechanism of docker-compose can be less reliable than docker’s, depending on the version, the cache_from build entry, and the backend used (docker-py, docker-cli, docker-cli and buildkit). This can lead to different layer hashes - even when executing the same build command repeatedly - eventually causing cache misses full image rebuilds.

If the image has been already built but the cache doesn’t work properly, it can be useful to skip the build phases:

first run ensures that the image is built

archery docker run conda-python

if the second run tries the build the image again and none of the files

referenced in the relevant dockerfile have changed, then it indicates a

cache miss caused by the issue described above

archery docker run conda-python

since the image is properly built with the first command, there is no

need to rebuild it, so manually disable the pull and build phases to

spare the some time

archery docker run --no-pull --no-build conda-python

Pass environment variables to the container:

Most of the build scripts used within the containers can be configured through environment variables. Pass them using --env or -e CLI options - similar to the docker run and docker compose run interface.

archery docker run --env CMAKE_BUILD_TYPE=release ubuntu-cpp

For the available environment variables in the C++ builds see theci/scripts/cpp_build.sh script.

Run the image with custom command:

Custom docker commands may be passed as the second argument toarchery docker run.

The following example starts an interactive bash session in the container - useful for debugging the build interactively:

archery docker run ubuntu-cpp bash

Build the image with increased debugging output:

To enable additional logging output for debugging, pass the --debug flag to archery.

archery --debug docker run ubuntu-cpp

In addition to enabling DEBUG-level logging, this also translates to passing --progress=plain to docker(-compose) build command.

Docker Volume Caches#

Most of the compose container have specific directories mounted from the host to reuse ccache and maven artifacts. These docker volumes are placed in the .docker directory.

In order to clean up the cache simply delete one or more directories (or the whole .docker directory).

Development#

The Docker Compose configuration is tuned towards reusable development containers using hierarchical images. For example multiple language bindings are dependent on the C++ implementation, so instead of redefining the C++ environment multiple Dockerfiles, we can reuse the exact same base C++ image when building Glib, Ruby, R and Python bindings. This reduces duplication and streamlines maintenance, but makes the Docker Compose configuration more complicated.

Docker Build Parameters#

The build time parameters are pushed down to the dockerfiles to make the image building more flexible. These parameters are usually called as docker build args, but we pass these values as environment variables to docker-compose.yml. The build parameters are extensively used for:

• defining the docker registry used for caching
• platform architectures
• operation systems and versions
• defining various versions if dependencies

The default parameter values are stored in the top level .env file. For detailed examples see the docker-compose.yml.

Build Scripts#

The scripts maintained under ci/scripts directory should be kept parameterizable but reasonably minimal to clearly encapsulate the tasks it is responsible for. Like:

• cpp_build.sh: build the C++ implementation without running the tests.
• cpp_test.sh: execute the C++ tests.
• python_build.sh: build the Python bindings without running the tests.
• python_test.sh: execute the Python tests.
• docs_build.sh: build the Sphinx documentation.
• integration_dask.sh: execute the dask integration tests.
• integration_pandas.sh: execute the pandas integration tests.
• install_minio.sh: install minio server for multiple platforms.
• install_conda.sh: install miniconda for multiple platforms.
• install_gcs_testbench.sh: install the GCS testbench for multiple platforms.

The parametrization (like the C++ CMake options) is achieved via environment variables with useful defaults to keep the build configurations declarative.

A good example is cpp_build.sh build script which forwards environment variables as CMake options - so the same scripts can be invoked in various configurations without the necessity of changing it. For examples see how the environment variables are passed in the docker-compose.yml’s C++ images.

Adding New Images#

See the inline comments available in the docker-compose.yml file.