Unexpected lazy pull behaviour (original) (raw)

Hello,

I am observing strange and problematic lazy pulling behaviour for one of my php-based images. My expectation is that whenever a file is accessed on the container and that file has not yet been fetched from the registry, the snapshotter will locate the file in one of the layers, fetch it and return it to the caller. However, this is not the behaviour that I am observing with this specific image/container.

Problem

The ENTRYPOINT for this specific image is php artisan (the CLI tool of the Laravel framework), which proceeds to load the configuration of the application. When the estargz-formatted image is run on an instance that does not have the stargz-snapshotter installed, there is no problem and the container runs fine. However, when I run the same image on an instance with stargz-snapshotter installed and configured, the following error is returned from php artisan - Unable to load the 'app' configuration file. and this is not a one-time occurrence, i.e. the error persists even if the container is restarted.

The following piece of code in laravel produces the error - https://github.com/laravel/framework/blob/10.x/src/Illuminate/Foundation/Bootstrap/LoadConfiguration.php#L61-L95. As you can see, it constructs a $files array based on a wildcard *.php in the config directory.

All of the configuration files are located under /var/www/config and are *.php files.

When I exec into the running container, I can cat /var/www/config/app.php and the contents of the file are available and printed to STDOUT, but for some reason php artisan still cannot access the file. After some digging, I discovered that if I instead run touch /var/www/config/app.php, then php artisan starts succeeding, i.e. that particular file is now accessible by the php process, but the rest of the files under that same directory /var/www/config are still not visible to the php process.

Furthermore, if I run cp /var/www/config/app.php /var/www/config/whatever.php, then all of the files under /var/www/config become visible to the php process, which was previously not the case (I assume cp lists the whole directory under the hood and this is what triggers the fetching of all the files under this directory.).

It's worth stating that I have confirmed that the $app object and the $configPath are properly set when the error occurs, so this points me to an issue with lazy loading.

No errors are reported in the logs of containerd and containerd-stargz-grpc when the issue occurs.

Context

I usually build the image with the following command

docker buildx build -o type=registry,oci-mediatypes=true,compression=estargz,force-compression=true -t myreg/myimage:rev .

Docker Engine version - 25.0.5
Buildkit version - 0.18.2
Base image - php:8.1-fpm

I have also tried building with nerdctl and optimizing the image with ctr-remote image optimize but the issue persists.

The container is being run in a Kubernetes cluster with the following versions:

Main questions

Is it expected that there is a difference between the way cat, touch and cp trigger the lazy loading mechanism?
Why does cat not seem to trigger the lazy loading mechanism or at least not fully, as evident by the behaviour of the php process in the different scenarios explained above?
Is it expected that when files are listed through a wildcard (what Laravel does during the initialisation of the application), lazy loading is not triggered?

Thanks in advance!