Docker In Docker Error in Nvidia Jetpack Environment (original) (raw)
We are using the Orin NX Devkit,
environment information is as follows
nvidia@ubuntu:~$ cat /etc/nv_tegra_release
# R36 (release), REVISION: 3.0, GCID: 36191598, BOARD: generic, EABI: aarch64, DATE: Mon May 6 17:34:21 UTC 2024
# KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
nvidia@ubuntu:~$ nvidia-smi
Wed Mar 26 14:23:09 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 540.3.0 Driver Version: N/A CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Orin (nvgpu) N/A | N/A N/A | N/A |
| N/A N/A N/A N/A / N/A | Not Supported | N/A N/A |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
We were running bitbucket pipeline in this environment and have been experiencing errors running the runner lately.
On a related note, there was a network related issue with the latest docker engine on the nvidia forum,
The following logs are from a later version that says the issue is fixed.
I am getting an error in nvidia jetpack environment, Orin NX Devkit when i try to do docker in docker,
I was hoping someone could give me a hint on how to fix it.
nvidia@ubuntu:~$ docker run --privileged --name dind -d docker
4b722935015bdd034d3ed7b0c60fd3b948513ac64370a9cf2d53076003a6f370
nvidia@ubuntu:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4b722935015b docker "dockerd-entrypoint.…" 2 seconds ago Up 1 second 2375-2376/tcp dind
nvidia@ubuntu:~$ docker logs -f dind
Certificate request self-signature ok
subject=CN=docker:dind server
/certs/server/cert.pem: OK
Certificate request self-signature ok
subject=CN=docker:dind client
/certs/client/cert.pem: OK
cat: can't open '/proc/net/arp_tables_names': No such file or directory
iptables v1.8.11 (nf_tables)
time="2025-03-26T05🔞28.844168004Z" level=info msg="Starting up"
time="2025-03-26T05🔞28.845285865Z" level=info msg="containerd not running, starting managed containerd"
time="2025-03-26T05🔞28.847866571Z" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=73
time="2025-03-26T05🔞28.862768261Z" level=info msg="starting containerd" revision=05044ec0a9a75232cad458027ca83437aae3f4da version=v1.7.27
time="2025-03-26T05🔞28.880147236Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2025-03-26T05🔞28.880214946Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2025-03-26T05🔞28.880642200Z" level=info msg="loading plugin \"io.containerd.warning.v1.deprecations\"..." type=io.containerd.warning.v1
time="2025-03-26T05🔞28.880667256Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." type=io.containerd.snapshotter.v1
time="2025-03-26T05🔞28.880770581Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." error="no scratch file generator: skip plugin" type=io.containerd.snapshotter.v1
time="2025-03-26T05🔞28.880789141Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2025-03-26T05🔞28.880807220Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." error="devmapper not configured: skip plugin" type=io.containerd.snapshotter.v1
time="2025-03-26T05🔞28.880818996Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2025-03-26T05🔞28.880906994Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2025-03-26T05🔞28.881206187Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2025-03-26T05🔞28.888216034Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"ip: can't find device 'aufs'\\nmodprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2025-03-26T05🔞28.888248482Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2025-03-26T05🔞28.888396190Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2025-03-26T05🔞28.888413502Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2025-03-26T05🔞28.888431453Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2025-03-26T05🔞28.888518587Z" level=info msg="metadata content store policy set" policy=shared
time="2025-03-26T05🔞28.891380279Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2025-03-26T05🔞28.891435413Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2025-03-26T05🔞28.891458261Z" level=info msg="loading plugin \"io.containerd.lease.v1.manager\"..." type=io.containerd.lease.v1
time="2025-03-26T05🔞28.891478804Z" level=info msg="loading plugin \"io.containerd.streaming.v1.manager\"..." type=io.containerd.streaming.v1
time="2025-03-26T05🔞28.891496980Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2025-03-26T05🔞28.891703663Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2025-03-26T05🔞28.891963785Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2025-03-26T05🔞28.892132516Z" level=info msg="loading plugin \"io.containerd.runtime.v2.shim\"..." type=io.containerd.runtime.v2
time="2025-03-26T05🔞28.892160964Z" level=info msg="loading plugin \"io.containerd.sandbox.store.v1.local\"..." type=io.containerd.sandbox.store.v1
time="2025-03-26T05🔞28.892188803Z" level=info msg="loading plugin \"io.containerd.sandbox.controller.v1.local\"..." type=io.containerd.sandbox.controller.v1
time="2025-03-26T05🔞28.892206691Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2025-03-26T05🔞28.892222402Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2025-03-26T05🔞28.892237058Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2025-03-26T05🔞28.892254658Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2025-03-26T05🔞28.892273249Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2025-03-26T05🔞28.892287809Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2025-03-26T05🔞28.892302304Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2025-03-26T05🔞28.892316800Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2025-03-26T05🔞28.892347327Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892365343Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892380927Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892394910Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892410462Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892425501Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892439517Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892454653Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892469628Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892490716Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892505436Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892520155Z" level=info msg="loading plugin \"io.containerd.grpc.v1.streaming\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892534907Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892552378Z" level=info msg="loading plugin \"io.containerd.transfer.v1.local\"..." type=io.containerd.transfer.v1
time="2025-03-26T05🔞28.892583738Z" level=info msg="loading plugin \"io.containerd.grpc.v1.transfer\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892601497Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892619993Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2025-03-26T05🔞28.892682455Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
time="2025-03-26T05🔞28.892703479Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.tracing.processor.v1
time="2025-03-26T05🔞28.892715350Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2025-03-26T05🔞28.892728918Z" level=info msg="skip loading plugin \"io.containerd.internal.v1.tracing\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.internal.v1
time="2025-03-26T05🔞28.892739958Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2025-03-26T05🔞28.892755957Z" level=info msg="loading plugin \"io.containerd.nri.v1.nri\"..." type=io.containerd.nri.v1
time="2025-03-26T05🔞28.892769493Z" level=info msg="NRI interface is disabled by configuration."
time="2025-03-26T05🔞28.893103757Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2025-03-26T05🔞28.893207307Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2025-03-26T05🔞28.893297992Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2025-03-26T05🔞28.893327272Z" level=info msg="containerd successfully booted in 0.031588s"
time="2025-03-26T05🔞28.901607617Z" level=info msg="OTEL tracing is not configured, using no-op tracer provider"
time="2025-03-26T05🔞28.911965896Z" level=info msg="Creating a containerd client" address=/var/run/docker/containerd/containerd.sock timeout=1m0s
time="2025-03-26T05🔞28.925590529Z" level=info msg="Loading containers: start."
time="2025-03-26T05🔞28.928485820Z" level=warning msg="Could not load necessary modules for IPSEC rules: protocol not supported"
time="2025-03-26T05🔞29.307269529Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2025-03-26T05🔞29.307507379Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2025-03-26T05🔞29.307558194Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to add jump rules to ipv4 NAT table: failed to append jump rules to nat-PREROUTING: (iptables failed: iptables --wait -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER: Warning: Extension addrtype revision 0 not supported, missing kernel module?
iptables v1.8.11 (nf_tables): CHAIN_ADD failed (No such file or directory): chain PREROUTING
(exit status 4))
nvidia@ubuntu:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
nvidia@ubuntu:~$ docker version
Client: Docker Engine - Community
Version: 28.0.4
API version: 1.48
Go version: go1.23.7
Git commit: b8034c0
Built: Tue Mar 25 15:07:33 2025
OS/Arch: linux/arm64
Context: default
Server: Docker Engine - Community
Engine:
Version: 28.0.4
API version: 1.48 (minimum version 1.24)
Go version: go1.23.7
Git commit: 6430e49
Built: Tue Mar 25 15:07:33 2025
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.26
GitCommit: 753481ec61c7c8955a23d6ff7bc8e4daed455734
nvidia:
Version: 1.2.5
GitCommit: v1.2.5-0-g59923ef
docker-init:
Version: 0.19.0
GitCommit: de40ad0
AastaLLL March 26, 2025, 9:07am 3
Hi,
cat: can't open '/proc/net/arp_tables_names': No such file or directory
The error is related to the network. Could you try exporting the host network to the container?
Thanks.
Although the cause is not clear,
I found a viable docker in docker version.
docker run --privileged --name dind -d docker:24.0.7-dind-alpine3.18
This error has nothing to do with running docker containers.
There was also an error related to the network module in the default jetpack environment,
However, even when running the machine with the compiled version with the network module related kernel option enabled
The symptoms were the same.
Currently, the error is occurring when running the docker in docker image, and it runs normally when running the version specified above.
AastaLLL March 31, 2025, 8:32am 5
Hi,
Do you want to use a rootless docker?
We test this command with sudo and the container can launch without error:
$ sudo docker run --privileged --name dind -d docker:24.0.7-dind-alpine3.18
31f9baeca2e2cfa2df6b8f271eeb376caf53134eca94a425b299891138978864
Thanks.
system Closed April 22, 2025, 12:48pm 7
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.