DiffusionGemma - How to Run Locally | Unsloth Documentation (original) (raw)

For the complete documentation index, see llms.txt. This page is also available as Markdown.

  1. Models

DiffusionGemma - How to Run Locally

DiffusionGemma 26B-A4B is Google DeepMind’s new open multimodal model, built on the Gemma 4 MoE architecture. With support for 256K context, 140+ languages, DiffusionGemma is designed for high-speed text generation across text, video and image inputs. DiffusionGemma can run locally on 18GB RAM, and fine-tuning is now supported via Unsloth.

Instead of standard token-by-token decoding, DiffusionGemma uses diffusion generation to produce outputs in parallel and gradually refine them into a final answer - similar to diffusion image models, but for text. Run the model via Unsloth Studio or llama.cpp. On a RTX 6000, DiffusionGemma can reach 2000+ tokens/s. GGUF: diffusiongemma-26B-A4B-it-GGUF

Run DiffusionGemmaFine-tune DiffusionGemma

Jun 12: You can now run DiffusionGemma via Unsloth Studio ✨ with 1.8x faster inferece!

DiffusionGemma is designed for users who need faster generation than standard models. It is suited to fast local inference, long-context doc analysis, image/video understanding, OCR and document parsing, code generation, tool use, agentic workflows, and low-latency small-batch inference.

Unlike standard Gemma 4 models, DiffusionGemma requires a diffusion-aware inference runtime. Autoregressive settings such as temperature, top_p, and top_k alone are not enough to reproduce the recommended behavior without the required diffusion sampler.

It's generally best to have at least 18GB RAM to run the model in 4-bit precision. GGUF: diffusiongemma-26B-A4B-it-GGUF

Table: DiffusionGemma Inference GGUF recommended hardware requirements (units = total memory: RAM + VRAM, or unified memory).

As a rule of thumb, your total available memory should at least exceed the size of the quantized model you download. If it does not you can still run using partial RAM / disk offload, but generation will be slower. You will also need more compute, depending on the context window you use.

Run DiffusionGemma Tutorials

It's best to use at least 4-bit precision so we'll use the 4-bit Q4_K_M quant which needs 18GB RAM. GGUF: diffusiongemma-26B-A4B-it-GGUF

🦥 Unsloth Studio Guide🦙 Llama.cpp Guide

DiffusionGemma can now be run and trained in Unsloth Studio, our new open-source web UI for local AI. Unsloth Studio lets you run models locally on MacOS, Windows, Linux and:

Ensure you use the latest v0.1.463-beta or 2026.6.6. Run in your terminal:

MacOS, Linux, WSL:

Windows PowerShell:

MacOS, Linux, WSL and Windows:

Then open http://127.0.0.1:8888 (or your specific URL) in your browser.

Search and download DiffusionGemma

On first launch you will need to create a password to secure your account and sign in again.

Then go to the Studio Chat tab and search for DiffusionGemma in the search bar and download your desired model and quant.

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our Unsloth Studio inference guide.

For this tutorial, we will be utilizing the Dynamic 4-bit Q4_K_M quant which needs 18GB RAM and llama.cpp for fast local inference, especially if you have a CPU.

Obtain the SPECIFIC llama.cpp PR on GitHub here. You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference. For Apple Mac / Metal devices, set -DGGML_CUDA=OFF then continue as usual - Metal support is on by default.

Download the model via (after installing pip install huggingface_hub). You can choose Q4_K_M or other quantized versions like Q8_0 . If downloads get stuck, see: Hugging Face Hub, XET debugging

Then run the below:

You will see:

And if you type a question like "Create a Flappy Bird Game", you will see steps:

Then afterwards you'll see the output:

You can continue conversing as well!

Change -n 2048 as the number of tokens you want to predict, so more will produce longer answers.

Live visualization of diffusion

To see diffusion actually live, use the below - specially enable --diffusion-visual:

You will again see:

And we get:

All parameters for llama.cpp using the branch:

You can now train and fine-tune DiffusionGemma directly with Unsloth. In our example, we demonstrate the impact of domain-specific training by fine-tuning the model on Sudoku. The base model initially performs poorly on Sudoku tasks, but after training on a targeted dataset, it learns how to actually solve sudoku and solves every example correctly.

You can use our Colab notebook (A100) to fine-tune DiffusionGemma with:

Unsloth Studio automatically sets the best inference settings for your model. Use below if necessary:

entropy_bounded_denoising

Adaptive stopping enabled

Adaptive Stopping Trigger Conditions

Adaptive stopping should trigger only when both conditions are met:

Highest-probability tokens stable for 2 consecutive steps

At each denoising step, the sampler should select the lowest-entropy tokens whose mutual information bound remains: entropy_bound = 0.1. Non-selected tokens should be fully renoised before the next denoising step.

DiffusionGemma supports Gemma 4-style thinking mode. To enable thinking, add the thinking token at the start of the system prompt:

When thinking is enabled, the model may emit an internal reasoning channel followed by the final answer:

To disable thinking, remove the <|think|> token from the system prompt. When thinking is disabled, the model may still emit an empty thought channel:

For multi-turn conversations, do not include previous hidden thoughts in the conversation history. Only include the final assistant response before the next user turn.

DiffusionGemma Best Practices

DiffusionGemma supports interleaved multimodal inputs, including text and images. Video can be processed as sequences of image frames.

For best results with multimodal prompts, place image or frame content before text instructions. Example:

For document parsing, OCR, chart understanding, UI understanding, or small text extraction, use a higher visual token budget.

Supported visual token budgets:

Fast classification, simple captioning

General image understanding

OCR, charts, UI screenshots

Dense documents, small text, detailed extraction

For video-style inputs, DiffusionGemma can process up to 60 seconds when sampled at 1 frame per second.

DiffusionGemma is not a normal next-token-only model. It generates a block of tokens, called a canvas, by repeatedly refining noisy token predictions. The generation process works roughly as follows:

  1. The encoder processes the prompt and builds a context cache.
  2. The decoder receives a 256-token generation canvas.
  3. The diffusion sampler iteratively denoises the canvas.
  4. Confident tokens are selected and preserved.
  5. Uncertain tokens are renoised and refined again.
  6. Once the canvas is complete, it is appended to the context.
  7. The model continues with the next canvas.

This block-autoregressive approach allows DiffusionGemma to generate many tokens in fewer forward passes than a standard autoregressive model.

DiffusionGemma is optimized for speed and multimodal reasoning, though standard Gemma 4 is stronger on conventional reasoning benchmarks.

MRCR v2 8 needle 128K average

Vision benchmarks:

OmniDocBench 1.5, lower is better

curl -fsSL https://unsloth.ai/install.sh | sh
irm https://unsloth.ai/install.ps1 | iex
unsloth studio -H 0.0.0.0 -p 8888
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
gh pr checkout 24423
# build with CUDA (drop -DGGML_CUDA=ON for a CPU-only build)
cmake -B build -DGGML_CUDA=ON
cmake --build build -j --config Release --target llama-diffusion-cli
cd ..
pip install -U "huggingface_hub[cli]"
hf download unsloth/diffusiongemma-26B-A4B-it-GGUF \
    --local-dir unsloth/diffusiongemma-26B-A4B-it-GGUF \
    --include "*Q8_0*" # Use "*Q4_K_M*" for a smaller 16 GB download
./build/bin/llama-diffusion-cli \
  -m unsloth/diffusiongemma-26B-A4B-it-GGUF/diffusiongemma-26B-A4B-it-Q8_0.gguf \
  -ngl 99 -cnv -n 2048
./build/bin/llama-diffusion-cli \
  -m unsloth/diffusiongemma-26B-A4B-it-GGUF/diffusiongemma-26B-A4B-it-Q8_0.gguf \
  -ngl 99 -cnv -n 2048 --diffusion-visual
<|channel>thought
[internal reasoning]
<channel|>
[final answer]
<|channel>thought
<channel|>
[final answer]
[image]
Describe the chart and summarize the key trend.