GitHub - freedomofpress/repro-build: A collection of helpers for reproducible images (original) (raw)

repro-build is a script that helps you build bit-for-bit reproducible containers. By "reproducible containers", we refer to container images which can be rebuilt at any time, anywhere, from the same Dockerfile and build environment, and be bit-for-bit equal to the original container image.

repro-build cannot assist you if your Dockerfile has sources of non-determinism in it. What it does though is help you with the second part of the equation, which is providing you with a build environment that is consistent across Operating Systems and container engines.

To demonstrate why reproducibly building a container image requires more than a "deterministic" Dockerfile, here's an example. Let's build the scratch image, arguably the most deterministic image possible, with Docker and Podman:

$ echo "FROM scratch" | docker build - [+] Building 0.0s (3/3) FINISHED docker:default => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 87B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => exporting to image 0.0s => => writing image sha256:3302e88f529a4acbc0bb93fe2e2c2da7fa5a4d70e348d54f5736b604b7293c46

$ echo "FROM scratch" | podman build - STEP 1/1: FROM scratch COMMIT --> bcccfc6e10db bcccfc6e10db600c78e86128f96c35d749e9c50aac2c7acd78874a4cbfaa51a0 $ podman images bcccfc6e10db --digests REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE sha256:251f716255f1732552091986ba7365fc195bae436a16d0d8e5a45e31adba97f0 bcccfc6e10db 2 minutes ago 1.06 kB

You can see that the image digests are different. That's not due to the contents of the image (there are none after all), but due to the different types of manifests and annotations that Podman and Docker use.

To make the digests exactly the same, you need to control various aspects of the environment. repro-build saves you time by doing just that. Not only that, but we have a nightly job which ensures that repro-build will continue to do so for future versions of Docker, Podman, and BuildKit.

How it works

In a nutshell, repro-build builds your container using a pinned version ofBuildKit, and its reproducibility features. If you use Docker, it creates a new buildx builder under the hood with a pinned BuildKit version. If you are using Podman, it runs BuildKit within a container. Then, it builds your container image, and stores it in tarball format. You can analyze the image tarball later on and ensure it has the digest you expect.

Features

Reproducible images in this repository

This repository automatically builds and publishes several reproducible images to GHCR. These images are updated daily and verified for reproducibility across different environments and BuildKit versions.

Distro Dockerfile GHCR Link
Debian Dockerfile.debian ghcr.io/freedomofpress/repro-build/debian

Usage

Build a container image locally

You can build a container image with:

$ ./repro-build build --sde 0 . 2025-02-24 09:17:48 - INFO - Build environment:

Build parameters:

Podman-only arguments:

Docker-only arguments:

2025-02-24 09:17:48 - DEBUG - Running: docker buildx create --name repro-build-6eb8a59ad67f3a251f19d5abdd82689923fe4f501a97a8fee73eeb935538a056 --driver-opt image=moby/buildkit:v0.19.0@sha256:14aa1b4dd92ea0a4cd03a54d0c6079046ea98cd0c0ae6176bdd7036ba370cbbe ERROR: existing instance for "repro-build-6eb8a59ad67f3a251f19d5abdd82689923fe4f501a97a8fee73eeb935538a056" but no append mode, specify the node name to make changes for existing instances 2025-02-24 09:17:48 - DEBUG - Running: docker buildx --builder repro-build-6eb8a59ad67f3a251f19d5abdd82689923fe4f501a97a8fee73eeb935538a056 build --build-arg SOURCE_DATE_EPOCH=0 --provenance false --output type=docker,dest=/Users/alex.p/repro-build/image.tar,rewrite-timestamp=true /Users/alex.p/repro-build [+] Building 81.6s (7/7) FINISHED docker-container:repro-build-6eb8a59ad67f3a251f19d5abdd82689923fe4f501a97a8fee73eeb935538a056 => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 522B 0.0s => [internal] load metadata for docker.io/library/debian:bookworm-20230904-slim 0.4s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load build context 0.0s => => transferring context: 5.49kB 0.0s => CACHED [stage-0 1/2] FROM docker.io/library/debian:bookworm-20230904-slim@sha256:050f00e86cc4d928b21de66096126fac52c2ea47885c232932b2e4c00f0c116d 0.0s => => resolve docker.io/library/debian:bookworm-20230904-slim@sha256:050f00e86cc4d928b21de66096126fac52c2ea47885c232932b2e4c00f0c116d 0.0s => [stage-0 2/2] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked --mount=type=bind,source=./repro-sources-list.sh,target=/usr/local/bin/repro-sources-list.sh 70.1s => exporting to docker image format 11.0s => => exporting layers 5.1s => => rewriting layers with source-date-epoch 0 (1970-01-01 00:00:00 +0000 UTC) 5.2s => => exporting manifest sha256:d2ed9626c60a7ea2b774b1e268ba74f1839de34808ed32ff99f9f7facde4de0b 0.0s => => exporting config sha256:b1fbf0683ddec2760c7cc4fada2cff4a28a6654958902ba42e6fc58295ead88e 0.0s => => sending tarball

Alternatively, if you have uv installed, you can run it directly with uvx:

$ uvx repro-build build --sde 0 .

build options

Option Description
--runtime Container runtime (docker or podman). Auto-detected if not provided.
--datetime ISO format datetime for image layers.
--source-date-epoch, --sde Unix timestamp for image layers.
--buildkit-image Custom BuildKit image to use.
--no-cache Build from scratch without using cache.
--rootless Run BuildKit in rootless mode (Podman only).
-f, --file Path to the Dockerfile.
-o, --output Path to save the output tarball (default: image.tar).
-t, --tag Tag the built image.
--build-arg Set build-time variables (can be used multiple times).
--annotation Add image annotations (can be used multiple times).
--platform Target platform(s) for the build.
--buildkit-args Extra arguments for BuildKit (Podman only).
--buildx-args Extra arguments for Docker Buildx (Docker only).
--dry Dry-run mode (prints commands instead of running them).

Build a container image on GitHub Actions

This repository provides two GitHub Actions to help you build and verify reproducible images.

Reproducible build action (freedomofpress/repro-build@1.0.0)

This action builds a container image reproducibly using Docker Buildx and the standard docker/build-push-action. It is a wrapper that handlesSOURCE_DATE_EPOCH validation and ensures the rewrite-timestamp=true output option is set.

Example Usage:

Inputs

Name Description Default
tags Tags for the image (comma-separated).
file Path to the Dockerfile.
context Build context.
platforms Platforms to build for (e.g., linux/amd64,linux/arm64).
buildkit_image BuildKit image to use. moby/buildkit:v0.19.0@...
source_date_epoch SOURCE_DATE_EPOCH value.
timestamp RFC 3339 timestamp to use as SOURCE_DATE_EPOCH.
push Whether to push to registry. false
outputs List of output destinations.
annotations List of annotations to set.
labels List of metadata for an image.
build-args List of build-time variables.
cache Whether to enable caching. true
cache-from List of external cache sources.
cache-to List of external cache destinations.
cache-map Mapping for buildkit-cache-dance.
secrets List of secrets to expose to the build.
secret-files List of secret files to expose to the build.
ssh List of SSH agent socket or keys.
target Sets the target stage to build.
no-cache Do not use cache. false
pull Always attempt to pull a newer version of the image. false
sbom Generate SBOM attestation. false
no-setup-buildx Whether to skip the Buildx setup step. false
Outputs

Name Description
imageid Image ID.
digest Image digest.
metadata Build metadata.

Reproduce and verify action (freedomofpress/repro-build/verify@1.0.0)

Rebuilds an image and verifies its digest against an expected value or a target image.

Example Usage:

Inputs

Name Description Default
expected_digest Expected image digest (e.g., sha256:...).
target_image Image to fetch digest from for verification.
file Path to the Dockerfile. Dockerfile
context Build context. .
platforms Target platform (e.g., linux/amd64). linux/amd64
buildkit_image BuildKit image to use. moby/buildkit:v0.19.0@...
runtime Container runtime (docker or podman). podman
source_date_epoch SOURCE_DATE_EPOCH value.
build-args Additional build arguments (comma-separated ARG=VALUE).
output Path to save the image tarball. /tmp/image.tar
tags Tags for the image.
annotations List of annotations to set.

Analyze a container image in .tar format

You can inspect the created tarball with:

$ ./repro-build analyze image.tar The OCI tarball contains an index and 1 manifest(s):

Image digest: sha256:d2ed9626c60a7ea2b774b1e268ba74f1839de34808ed32ff99f9f7facde4de0b

Index (index.json): Digest: sha256:e609199e7b564eba29ee3ccaa8509fed8c62a8ac91ee5caba46c9c0dc0ed6129 Media type: application/vnd.oci.image.index.v1+json Platform: - Contents: {"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","digest":"sha256:d2ed9626c60a7ea2b774b1e268ba74f1839de34808ed32ff99f9f7facde4de0b","size":703,"annotations":{"org.opencontainers.image.created":"1970-01-01T00:00:00Z"},"platform":{"architecture":"arm64","os":"linux"}}]}

Manifest 1 (blobs/sha256/d2ed9626c60a7ea2b774b1e268ba74f1839de34808ed32ff99f9f7facde4de0b): Digest: sha256:d2ed9626c60a7ea2b774b1e268ba74f1839de34808ed32ff99f9f7facde4de0b Media type: application/vnd.docker.distribution.manifest.v2+json Platform: linux/arm64 Contents: { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "digest": "sha256:b1fbf0683ddec2760c7cc4fada2cff4a28a6654958902ba42e6fc58295ead88e", "size": 1165 }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "digest": "sha256:155eab17d86c47443adc8cebe7fc62c847c03db8cfb1ca53aa6276564fff23ef", "size": 29157149 }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "digest": "sha256:7914d3c3eb039f [... 83 characters omitted. Pass --show-contents to print them in their entirety]

Tarball format

By default, repro-build uses thedocker exporter when creating an image tarball.

Pros and cons of docker exporter:

Pros and cons of oci exporter:

We feel it's more important to compare local digests with remote ones, as well as load the container image with docker load, so we prefer to use the docker load exporter.

Multi-platform images

If you are on macOS / Windows, the easiest way to build multi-platform images is via Docker, which has built-in BuildKit support. Any other option may require nested virtualization to work.

The docker exporter that repro-build uses under the hood does not support multi-platform images. You are advised to create a tarball per architecture, if you want to reproduce an image.

If you want to build and push an image, it's best to swap type=docker withtype=registry manually. You can try out a build with ./repro-build build --dry ..., and tweak the commands that would have ran.

If you want to build and push images with Podman, you may also need to mount the registry credentials in the BuildKit container.

Sources of non-determinism

Here are some lesser known sources of non-determinism that we have encountered while building images:

Other considerations

Read more

For a primer on what are "reproducible containers", and some sources to get started, we suggest reading the following:

License

Licensed under either of:

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you shall be dual licensed as above, without any additional terms or conditions.

Credits

Credits go to @AkihiroSuda who has provided the necessary scaffolding (seehttps://github.com/reproducible-containers) that this project is based on.