Why python in Debian Docker image is faster than official Docker's python image? (original) (raw)
Hi,
Debian’s python installed inside a docker container is faster than the official python from docker. How is it possible ?
# Dockerfile-debian
FROM debian:sid
RUN apt update && apt install -y python3.12
# Python code for benchmark : test.py
import sys
import timeit
def test_function():
numbers = list(range(1000))
[num ** 2 for num in numbers]
benchmark_result = timeit.timeit(test_function, number=int(sys.argv[1]) )
print(benchmark_result)
Results
$ docker build -f ./Dockerfile-debian -t python_debian .
$ docker run -v .:/app python_debian python3.12 /app/test.py
# 0.7484053150001273
$ docker run -v .:/app python:3.12 python3.12 test.py
# 1.1680918899983226
How does debian manage to have better performances than the official python docker image ?
I’ve tried to build python using the same build args than debian, but I’ve ended with an even slower binary.
Debian build args
'--enable-shared'
'--prefix=/usr'
'--libdir=/usr/lib/x86_64-linux-gnu'
'--enable-ipv6'
'--enable-loadable-sqlite-extensions'
'--with-dbmliborder=bdb:gdbm'
'--with-computed-gotos'
'--without-ensurepip'
'--with-system-expat'
'--with-dtrace'
'--with-ssl-default-suites=openssl'
'--with-wheel-pkg-dir=/usr/share/python-wheels/'
'MKDIR_P=/bin/mkdir -p'
'CC=x86_64-linux-gnu-gcc'
tiran (Christian Heimes) May 22, 2024, 3:55pm 2
Are you referring to https://hub.docker.com/_/python? These are “Docker Official Images”, not official Python images. The images are provided by Docker Community, not by the PSF or Python community.
barry-scott (Barry Scott) May 22, 2024, 4:07pm 3
A while ago debian started to build python without using a libpython.
They claimed that that change made starting python faster.
May be that is the difference?
Otherwise it could be differences in compiler optimisation options used.
Yes, I’m referring to the Python image from the Docker Hub. Sorry, I thought it was made by PSF or the Python community.
The “benchmark” I’ve used does not measure python boot time, only code execution. The difference is not here.
I’ve checked python build args using sysconfig.get_config_vars('CONFIG_ARGS')
Debian ones :
'--enable-shared'
'--prefix=/usr'
'--libdir=/usr/lib/x86_64-linux-gnu'
'--enable-ipv6'
'--enable-loadable-sqlite-extensions'
'--with-dbmliborder=bdb:gdbm'
'--with-computed-gotos'
'--without-ensurepip'
'--with-system-expat'
'--with-dtrace'
'--with-ssl-default-suites=openssl'
'--with-wheel-pkg-dir=/usr/share/python-wheels/'
'MKDIR_P=/bin/mkdir -p'
'CC=x86_64-linux-gnu-gcc'
Official docker’s python:
'--build=x86_64-linux-gnu'
'--enable-loadable-sqlite-extensions'
'--enable-optimizations'
'--enable-option-checking=fatal'
'--enable-shared'
'--with-lto'
'--with-system-expat'
'--without-ensurepip'
'build_alias=x86_64-linux-gnu'
Note that debian does not use --enable-optimizations
I’ve published the full benchmark I’ve made here : https://github.com/fabien-michel/compare-python
It compare python from different Linux distributions using docker and local Python.
Here are the results on my machine which run Arch Linux: (percent are diff with official Docker’s Python)
debian-on-docker 0.7246799930017005 -34.072282393951454%
ubuntu-on-docker 0.7611594549998699 -30.75356559169216%
arch-on-docker 0.9717710979984986 -11.593184377570633%
local 1.035610535998785 -5.785395448310301%
fedora-on-docker 1.035977895000542 -5.751974985814081%
official-docker 1.099203824000142 0.0%
barry-scott (Barry Scott) May 22, 2024, 8:26pm 7
What matters are the C compiler used, gcc I would expect, and the C compiler options.
Even if --enable-optimisations is not passed to python configure script that does mean there are no optimisations. Its typical to pass them in env vars like CFLAGS and LDFLAGS.
Do you know what they where when python was compiled?
fungi (fungi) May 22, 2024, 9:07pm 8
Do you know what they where when python was compiled?
This is the makefile used by Debian to build the Python 3.11 package
in their bookworm (current stable) release:
I assume that’s what the OP installed in their container anyway.
barry-scott (Barry Scott) May 22, 2024, 9:13pm 9
And this is the key line
DPKG_CFLAGS := <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>s</mi><mi>h</mi><mi>e</mi><mi>l</mi><mi>l</mi></mrow><annotation encoding="application/x-tex">(shell </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal">s</span><span class="mord mathnormal">h</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.01968em;">ll</span></span></span></span>(dpkg_buildflags) --get CFLAGS)
you will need to know what that evaluates as.
fabien-michel (Fabien MICHEL) May 22, 2024, 9:28pm 10
Ok thanks for your search. Python seems to store all config variables. It cam be retrieved using sysconfig.get_config_vars().
Here CFLAGS et LDFLAGS from debian’s python:
"CFLAGS": "-fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall",
"LDFLAGS": " -Wl,-z,relro -g -fwrapv -O2 ",
And CFLAGS and LDFLAGS from docker’s python:
"CFLAGS": "-fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall",
"LDFLAGS": "",
I’m really surprised there is so much difference in build options across distributions.
tiran (Christian Heimes) May 23, 2024, 4:59am 11
No, your answer wrong. --enable-optimization is used for PGO builds, which makes Python about 10% faster. PGO cannot be enabled by passing different CFLAGS and LDFLAGS alone. In PGO mode, make first compiles an instrumented build of Python, then runs some tests to create optimization profiles, and finally compiles Python a second time. You trade a slower build for a faster Python.
You can get the actual flags for Python interpreter core from sysconfig.get_config_vars("PY_CORE_CFLAGS") and sysconfig.get_config_vars("PY_CORE_LDFLAGS"). Fedora and RHEL use additional flags like -fno-semantic-interposition, which can increase performance by up to 30% with Python 3.8. AFAIK speedups are less impressive in 3.12.
barry-scott (Barry Scott) May 23, 2024, 7:20am 12
Thanks for the correction, I’d forgotten about the PGO stuff.