Model Runner (original) (raw)
Docker Model Runner
Requires: Docker Engine or Docker Desktop (Windows) 4.41+ or Docker Desktop (MacOS) 4.40+
For: See Requirements section below
Docker Model Runner (DMR) makes it easy to manage, run, and deploy AI models using Docker. Designed for developers, Docker Model Runner streamlines the process of pulling, running, and serving large language models (LLMs) and other AI models directly from Docker Hub or any OCI-compliant registry.
With seamless integration into Docker Desktop and Docker Engine, you can serve models via OpenAI-compatible APIs, package GGUF files as OCI Artifacts, and interact with models from both the command line and graphical interface.
Whether you're building generative AI applications, experimenting with machine learning workflows, or integrating AI into your software development lifecycle, Docker Model Runner provides a consistent, secure, and efficient way to work with AI models locally.
- Pull and push models to and from Docker Hub
- Serve models on OpenAI-compatible APIs for easy integration with existing apps
- Package GGUF files as OCI Artifacts and publish them to any Container Registry
- Run and interact with AI models directly from the command line or from the Docker Desktop GUI
- Manage local models and display logs
- Display prompt and response details
- Conversational context support for multi-turn interactions
Docker Model Runner is supported on the following platforms:
Windows(amd64):
- NVIDIA GPUs
- NVIDIA drivers 576.57+
Windows(arm64):
OpenCL for Adreno
Qualcomm Adreno GPU (6xx series and later)
Some llama.cpp features might not be fully supported on the 6xx series.
Apple Silicon
Docker Engine only:
- Linux CPU & Linux NVIDIA
- NVIDIA drivers 575.57.08+
Models are pulled from Docker Hub the first time you use them and are stored locally. They load into memory only at runtime when a request is made, and unload when not in use to optimize resources. Because models can be large, the initial pull may take some time. After that, they're cached locally for faster access. You can interact with the model usingOpenAI-compatible APIs.
Using Testcontainers or Docker Compose?Testcontainers for Javaand Go, andDocker Compose now support Docker Model Runner.
docker model is not recognised
If you run a Docker Model Runner command and see:
It means Docker can't find the plugin because it's not in the expected CLI plugins directory.
To fix this, create a symlink so Docker can detect it:
Once linked, rerun the command.
No consistent digest support in Model CLI
The Docker Model CLI currently lacks consistent support for specifying models by image digest. As a temporary workaround, you should refer to models by name instead of digest.
Thanks for trying out Docker Model Runner. Give feedback or report any bugs you may find through the Give feedback link next to the Enable Docker Model Runner setting.