How to Run MTP Models: Multi-Token Prediction Guide | Unsloth Documentation (original) (raw)

For the complete documentation index, see llms.txt. This page is also available as Markdown.

Models

🪽How to Run MTP Models: Multi-Token Prediction Guide

MTP, or Multi-Token Prediction, speeds up inference by letting a model predict multiple upcoming tokens at once instead of generating one token per step. It enables faster inference without accuracy loss and is especially effective on GPUs. In this guide, you’ll learn how to use MTP models like Gemma 4 or Qwen3.6 on your local device.

MTP predicts multiple future tokens, which the main model verifies in parallel. This reduces generation forward passes, speeding output while preserving quality because only verified tokens are kept.

When running GGUFs, MTP can make generation ~1.4× to 2.2× faster. Dense models like Gemma-4-31B benefit most, reaching >1.4× speedup over the original. Gains are smaller on devices with lower memory bandwidth, such as older Macs. You can run MTP models directly in Unsloth Studio’s UI or llama.cpp.

MTP uses more memory than standard, so plan for ~2 GB additional RAM/VRAM headroom.

Gemma 4 MTP Qwen3.6 MTP

We found --spec-draft-n-max 2 is the best starting point however, do not assume **2** is optimal, as performance is hardware-dependent. Try any value from 1 through 6 and use whichever is fastest for your system. Unsloth Studio automatically sets the ideal MTP settings optimized for your specific hardware (Mac, CPU, GPU etc.) - you can still change it later.

Google DeepMind trained MTP separately from the original Gemma 4 models, including for QAT variants. Unlike Qwen, Google released specific MTP variants under the assistant name. For best results, we only upload 3 precision options: 8-bit and 16-bit (BF16, F16). For QAT - we applied the smart 4-bit recovery process like we did for Gemma 4 QAT quants, and so the MTP quants are also smart 4-bit derived.

We uploaded mtp- prefixed GGUFs to each repo, so you only need to use the regular original Gemma 4 GGUFs, no separate repo is needed. You can access Gemma MTP models here and they can now run in Unsloth. We benchmarked Gemma 4 QAT with MTP, and it runs 1.5x - 2.2x faster:

Table: MTP hardware requirements (units = total memory: RAM + VRAM, or unified memory)

Gemma 4 MTP is automatically enabled in Unsloth Studio. You only need to download the regular original Gemma 4 GGUFs. We updated the Gemma 4 GGUF files to include an additional MTP file inside a separate folder within the GGUF package, so there is no need to download a separate Gemma 4 assistant GGUF.

The only model that still requires a separate MTP GGUF is Qwen3.6.

To run the Gemma 4 MTP models, follow the steps either for Unsloth Studio or llama.cpp.

🦥 Run in Unsloth Studio 🦙 Run in llama.cpp

so the below just works (this uses the 8-bit one)

Qwen directly trained MTP inside of the Qwen3.6 and Qwen3.5 models. This enables Qwen3.6 27B MTP to reach 160 tokens/s and Qwen3.6 35B-A3B reach 240 tokens/s on an RTX 6000 GPU. GGUF uploads:

Table: MTP hardware requirements (units = total memory: RAM + VRAM, or unified memory)

Below are graphs of inference throughput for MTP vs. no MTP:

We also uploaded MTP GGUFs for the Qwen3.5 model family including: 0.8B, 2B, 4B, 9B, 27B, 35B-A3B, 122B-A10B and 397B-A17B. Llama.cpp is continually improving MTP performance, so expect it to get faster overtime!

To run the Qwen MTP models, follow the steps either for Unsloth Studio or llama.cpp.

🦥 Unsloth Studio MTP Guide

Unsloth Studio automatically sets the ideal MTP settings optimized for your specific hardware (Mac, CPU, GPU etc.) - you can still change it later.

Run in your terminal:

MacOS, Linux, WSL:

Windows PowerShell:

MacOS, Linux, WSL and Windows:

Then open http://127.0.0.1:8888 (or your specific URL) in your browser.

Search and download your desired model

On first launch you will need to create a password to secure your account and sign in again later. Then go to the Studio Chat tab and search for Qwen3.6 MTP or Gemma 4 in the search bar and download your desired model and quant.

Gemma 4 MTP is automatically enabled in Unsloth. You only need to download the regular original Gemma 4 GGUF. We updated the Gemma 4 GGUF files to include an additional MTP file inside a separate folder within the GGUF package, so there is no need to download a separate Gemma 4 assistant GGUF.

The only model that still requires a separate MTP GGUF is Qwen3.6.

Inference, MTP and speculative decoding settings should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit speculative decoding, the context length, chat template and other settings in the right side bar.

For more information, you can view our Unsloth Studio inference guide. Below, the 2-bit Qwen3.6 MTP GGUF made 10+ tool calls, searched 10 sites and executed Python code:

Install the latest version of llama.cpp on GitHub here. You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference. For Apple Mac / Metal devices, set -DGGML_CUDA=OFF then continue as usual - Metal support is on by default.

If you want to use llama.cpp directly to load models, you can do the below: (:Q4_K_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to ollama run . Use export LLAMA_CACHE="folder" to force llama.cpp to save to a specific location. The model has a maximum of 256K context length.

Follow one of the commands for the specific models:

Gemma 4 Qwen3.6

Don't forget to change the model name to your desired Gemma 4 model size like Gemma-4-26B-A4B etc. as the instructions below are for Gemma-4-12B. Notice we provided a mtp- prefixed GGUF, so the below -hf command should auto download and use MTP.

Thinking mode:

Non-thinking mode:

Don't forget to change the model name to your desired Qwen3.6 variant like Qwen3.6-35B-A3B or Qwen3.5 etc. as the instructions below are for Qwen3.6-27B:

Thinking mode (General tasks):

For precise coding tasks, change: temperature=0.6

Non-thinking mode (General tasks):

Manually downloading quants

If you want to manually download the quants and the MTP quants, you can also do that! Download the model via the code below (after installing pip install huggingface_hub hf_transfer). You can choose Q4_K_M or other quantized versions like UD-Q4_K_XL . We recommend using at least 2-bit dynamic quant UD-Q2_K_XL to balance size and accuracy. If downloads get stuck, see: Hugging Face Hub, XET debugging

Then run the model in conversation mode:

And you will see the below - ignore the error messages as well

To deploy Gemma-4 on llama-server, use:

Gemma 4 MTP
Qwen3.6 MTP
🦥 Unsloth Studio MTP Guide
🦙 Llama.cpp MTP Guide

llama-server \
    -hf unsloth/gemma-4-31B-it-GGUF \
    --spec-type draft-mtp \
    --spec-draft-n-max 4

curl -fsSL https://unsloth.ai/install.sh | sh

irm https://unsloth.ai/install.ps1 | iex

unsloth studio -H 127.0.0.1 -p 8888

apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp

export LLAMA_CACHE="unsloth/gemma-4-12b-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64  \
    --spec-type draft-mtp --spec-draft-n-max 2

export LLAMA_CACHE="unsloth/gemma-4-12b-it-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64  \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --chat-template-kwargs '{"enable_thinking":false}'

export LLAMA_CACHE="unsloth/Qwen3.6-27B-MTP-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --spec-type draft-mtp --spec-draft-n-max 2

export LLAMA_CACHE="unsloth/Qwen3.6-27B-MTP-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --presence-penalty 1.5 \
    --min-p 0.00 \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --chat-template-kwargs '{"enable_thinking":false}'

hf download unsloth/gemma-4-12B-it-qat-GGUF \
    --local-dir unsloth/gemma-4-12B-it-qat-GGUF \
    --include "*mmproj-F16*" \
    --include "mtp-*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit

hf download unsloth/Qwen3.6-27B-MTP-GGUF \
    --local-dir unsloth/Qwen3.6-27B-MTP-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit

./llama.cpp/llama-cli \
    --model unsloth/gemma-4-12B-it-qat-GGUF/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-4-12B-it-qat-GGUF/mmproj-F16.gguf \
    --model-draft unsloth/gemma-4-12B-it-qat-GGUF/mtp-gemma-4-12B-it.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64  \
    --spec-type draft-mtp --spec-draft-n-max 2

./llama.cpp/llama-cli \
    --model unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.6-27B-MTP-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.00 \
    --top-k 20 \
    --spec-type draft-mtp --spec-draft-n-max 2

./llama.cpp/llama-server \
    --model unsloth/gemma-4-12B-it-qat-GGUF/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-4-12B-it-qat-GGUF/mmproj-F16.gguf \
    --model-draft unsloth/gemma-4-12B-it-qat-GGUF/mtp-gemma-4-12B-it.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64 \
    --alias "unsloth/gemma-4-12b-it-qat-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'