Problems with Docker version 28.0.1 on Jetson Orin NX (original) (raw)
February 28, 2025, 1:49pm 1
First i am aware of this thread: Error with “Nvidia Container Runtime with Docker Integration” on AGX Orin with JP6.2
However we ran into a problem that has not been solved by upgrading to Docker version 28.0.1.
We have a reverse proxy container that stops instantly with the following error message:
Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint reverse_proxy-1 (...): Unable to enable DIRECT ACCESS FILTERING - DROP rule: (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.18.0.4 --dport 80 ! -i br-51c7e31b90d4 -j DROP: iptables v1.8.7 (legacy): can't initialize iptables table `raw': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
Same container (native amd64 and emulated arm64) works on our amd64 pcs with the same docker version.
Doesnt work on our Jetsons with JP 6.1 (custom RT kernel, The ip_set etc. kernel modules were already enabled since we are using k8s).
Just happens with this one container other containers seem fine.
Downgrading Docker verion 27 solves the problem, however thats just a temporary solution ofc.
Has anbody encountered the same problem and has a solution?
ToeiRei February 28, 2025, 6:45pm 2
It all started with docker having a problem with the nvidia runtime and vice versa and now it seems like we need a new kernel.
Hi,
Do you have a custom kernel?
After the docker 28.0.1 release, it should be able to run on the JetPack 6.1/6.2 BSP without changes.
If it doesn’t work in your environment, could you share the following info with us?
$ sudo docker info
$ journalctl -xu docker.service
Thanks.
Hi,
I have been used the latest Jetson Linux BSP package 36.4.3, and also face the same problem,
my docker version is as follow:
$ docker --version
Docker version 28.0.1, build 068a01e
my docker info message is as follow:
$ sudo docker info
Client: Docker Engine - Community
Version: 28.0.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.21.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.33.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 2
Server Version: 28.0.1
Storage Driver: overlay2
Backing Filesystem: btrfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: journald
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2 nvidia
Default Runtime: runc
Init Binary: docker-init
containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc version: v1.2.4-0-g6c52b3f
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.148-tegra
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: aarch64
CPUs: 6
Total Memory: 7.443GiB
Name: tegra-ubuntu
ID: f98d7454-51e3-4b46-9d72-38580bf5c20e
Docker Root Dir: /opt/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Default Address Pools:
Base: 172.100.0.0/16, Size: 24
And the journalctl message from docker is as follow:
$ journalctl -xu docker.service
Feb 12 06:05:25 tegra-ubuntu dockerd[905]: time=“2025-02-12T06:05:25.854595681Z” level=error msg=“Handler for POST /v1.48/containers/e2ee74fea501e8620ddde770b9112cb5d89f17525fff53075f945bf722b1697f/start returned error: failed to set up container networking: driver failed programming external connectivity on endpoint 6c1e26a7-22c5-452d-9bd5-ca8ed7ac90cb-web1-1 (bb880b0c080de050631b7bf953b5dd49b0cba20c4abcd22f0ccc455efef4d163): Unable to enable DIRECT ACCESS FILTERING - DROP rule: (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.100.3.3 --dport 80 ! -i br-3ee37c5f8364 -j DROP: iptables v1.8.7 (legacy): can’t initialize iptables table `raw’: Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\n (exit status 3))” spanID=5a7d7e7132330817 traceID=53ea434e468c36e7988c272b10914852
bashaku March 4, 2025, 11:45pm 6
Docker 28.0.1 depends on the kernel module iptable_raw, but it doesn’t seem to be included in Jetson Linux.
$ cat /etc/nv_tegra_release
# R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan 8 01:49:37 UTC 2025
# KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
$ docker --version
Docker version 28.0.1, build 068a01e
$ docker run -dit -p 80:80 --rm --name alpine alpine:latest
44de6abaa53736b9ffa4eb5171b631475772b015aa9dc8f92005065c016685d8
docker: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint alpine (d373025c7fbf1f880fe55171c6673b2b8f036ee84160128712ffde9980c02266): Unable to enable DIRECT ACCESS FILTERING - DROP rule: (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.17.0.2 --dport 80 ! -i docker0 -j DROP: iptables v1.8.7 (legacy): can't initialize iptables table `raw': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3))
$ find /lib/modules/$(uname -r) -name '*iptable_raw*'
$ lsmod | grep -i iptable_raw
$ zgrep IP_NF_RAW /proc/config.gz
# CONFIG_IP_NF_RAW is not set
bashaku March 5, 2025, 4:56am 7
To resolve this, a kernel rebuild was required.
I added the following line to the end of defconfig
CONFIG_IP_NF_RAW=m
re-build kernel and destination for iptable_raw.ko is
/lib/modules/5.15.148-tegra/kernel/net/ipv4/netfilter/
$ docker --version
Docker version 28.0.1, build 068a01e
$ docker run -dit -p 80:80 --rm --name alpine alpine:latest
c2210a7f8d3be7e5326b2b3773dc0d92834f7c4db9c779ee0ee9910edf8df462
$ lsmod | grep -i iptable_raw
iptable_raw 16384 1
ip_tables 32768 3 iptable_filter,iptable_raw,iptable_nat
x_tables 45056 12 ip6table_filter,xt_conntrack,iptable_filter,ip6table_nat,xt_tcpudp,xt_addrtype,xt_nat,ip6_tables,iptable_raw,ip_tables,iptable_nat,xt_MASQUERADE
Hi,
Is the docker 28.0.1 working normally after the configure is enabled?
Thanks.
bashaku March 5, 2025, 7:40am 10
Hi, @AastaLLL
I enabled only CONFIG_IP_NF_RAW.
and deployed new kernal image file and iptable_raw.ko to my Jetson.
then, the docker 28.0.1 working normally.
Hi, @bashaku
Your solution works for me as well. Thanks!
john432 March 10, 2025, 10:45pm 13
I have this problem as well. I turned iptables off in /etc/docker/daemon.json and have been running my containers in host with JP62 which isn’t my ideal scenario. I really don’t want to manage custom kernels. @AastaLLL will iptable_raw be added to the Jetpack release?
In addition to knowing when this will be fixed in the current JetPack releases, are there detailed instructions anywhere on building the kernel? Normally this is done via the SDK Manager and I’ve never had to build it by hand, so instructions would be helpful.
AastaLLL March 12, 2025, 3:29am 15
Hi, all
We are discussing to enable CONFIG_IP_NF_RAW internally.
Will share more info with you later.
Please find below the build from the source tutorial.
You can also check the below comment for the detailed steps.
(please update the kernel config accordingly)
Thanks.
petlock March 12, 2025, 6:39am 16
Those of us who don’t rebuild linux kernels in their sleep might need a refresher on some of the steps. If I’m on an Orin Nano AGX what’s the path for the toolchain referenced in step 3???
I was following along until this step
export CROSS_COMPILE=<toolchain-path>/bin/aarch64-buildroot-linux-gnu-
This user crashes trying to resolve whatever toolchain-path actually is…
I have no clue what the toolchain path would be. I’m on my jetson orin nano agx if that helps.
Is it this? Jetson Linux Toolchain — Jetson Linux Developer Guide documentation
and then I’d use this link ? https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v3.0/toolchain/aarch64--glibc--stable-2022.08-1.tar.bz2
alex585 March 12, 2025, 1:18pm 17
That would be VERY nice…
Im currently getting stuck bilding new Kernel and now blocked due to the docker iptables issue…
I agree, having a fleet of these and not being able to do the upgrade without issues is very concerning. I will do the kernal update on our dev machine. However will have to hold back on all the others.
Kangalow March 13, 2025, 1:53am 19
@petlock @alex585 I’ve put up a precompiled version of the IP_NV_RAW module on Github. Could you test it and see if it works for you? It’s for JetPack 6.2: jetson-orin-kernel-builder/prebuilt/jetpack-6.2 at main · jetsonhacks/jetson-orin-kernel-builder · GitHub
You should know that the Docker folks have a pull request in to remove the requirement for the module that will probably happen pretty soon.
petlock March 13, 2025, 2:23am 20
I’ll try it tonight.
And while I have Nvidia’s attention I wish I had known I was forever locking my machine out of frigate by upgrading to the latest jetpack when I bought it. Hell of a realization after dropping two grand.
alex585 March 13, 2025, 11:19am 22
Hmm, i yesterday used your Scripts (jetson-orin-kernel-builder/scripts at main · jetsonhacks/jetson-orin-kernel-builder · GitHub) to compile a new Kernel and kernel Moduls :-D
Indeed, found them yesterday. Then i copied the new created Kernel with enabled IP_NV_RAW over /boot/Image, et voila my Docker is back in the Game.
Hopefully this setting will be standard in NV delivered Kernels…
Kangalow March 13, 2025, 8:04pm 23
I’m glad you found the scripts useful. My understanding from the Docker issues in their repository is someone was working on optimization and pulled in the IP tables to help. This broke a good number of devices, including Jetsons and many Windows machines. They tried to pull the IP tables back out in 28.01, but missed one. They release 28.0.2 today, which they say fixes the IP table problem.
The overall plan includes a different solution entirely for the original optimization issue which does not include the IP table modules. Coming to a future Docker release near you. Docker was well intentioned, but just happened to break a lot of things in the process.
So even if NVIDIA includes IP_NV_RAW in another release, that ship has already sailed.
We simply rolled back to 27.3 to resolve this issue.
apt-get install docker-ce-cli=5:27.3.0-1~ubuntu.22.04~jammy
apt-get install docker-ce=5:27.3.0-1~ubuntu.22.04~jammy