NVIDIA Nemotron 3 Ultra - How To Run Locally | Unsloth Documentation (original) (raw)

For the complete documentation index, see llms.txt. This page is also available as Markdown.

  1. Models

🧩NVIDIA Nemotron 3 Ultra - How To Run Locally

Run Nemotron-3-Ultra-550B-A55B locally on your device!

NVIDIA Nemotron 3 Ultra is an open 550B parameter, 55B active frontier-reasoning model and is NVIDIA's largest model released so far. Nemotron-3-Ultra-550B-A55B is built for long-running autonomous agents and reasoning across coding, deep research workflows. It is the strongest Western open model, and adopts the new Open Model, Weights & Data License.

With up to 1M context, Nemotron 3 Ultra uses a Hybrid Transformer-Mamba MoE architecture and can preserve long agent state, logs, and plans across sustained sessions. GGUFs are atNemotron-3-Ultra-550B-A55B with dynamic 1bit taking 189GB of disk space. It's also pretrained using NVFP4. We als did GGUF KLD Benchmarks.

NVIDIA recommends these settings for inference:

550B total parameters / 55B active parameters

Hybrid Transformer-Mamba MoE with Latent MoE, Multi-Token Prediction (MTP currently not supported for GGUFs)

The chat template is like below:

<|im_start|>system\n<|im_end|>\n<|im_start|>user\nWhat is 1+1?<|im_end|>\n<|im_start|>assistant\n<think></think>2<|im_end|>\n<|im_start|>assistant\n<think>\n

The 3-bit versions of the model requires ~256GB RAM, 4-bit needs ~300GB and 8-bit requires 600GB. For these guides, we will be using 3-bit UD-IQ3_XXS which fits on a 256GB device and is a good balance between size and accuracy. Depending on your use-case you will need to use different settings. GGUF: Nemotron-3-Ultra-550B-A55B

Run in Unsloth StudioRun in llama.cpp

For this tutorial, we will be using Unsloth Studio, which is our UI for running and training LLMs. With Unsloth Studio, you can run models and input image and text locally on Mac, Windows, and Linux and:

MacOS, Linux, WSL:

Windows PowerShell:

Setup Unsloth Studio (one time)

Setup automatically installs Node.js (via nvm), builds the frontend, installs all Python dependencies, and builds llama.cpp with CUDA support.

WSL users: you will be prompted for your sudo password to install build dependencies (cmake, git, libcurl4-openssl-dev).

MacOS, Linux, WSL:

Windows Powershell:

Then open http://127.0.0.1:8888 in your browser.

Search and download Nemotron-3-Ultra

On first launch you will need to create a password to secure your account and sign in again later. Then go to the Studio Chat tab and search for Nemotron-3-Ultra in the search bar and download your desired model and quant.

Inference parameters should be auto-set when using Unsloth Studio, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our Unsloth Studio inference guide.

You can also use unsloth studio run to serve the model via llama-server like so:

Instructions to run in llama.cpp (note we will be using 4-bit to fit most devices):

Obtain the latest llama.cpp on GitHub here. You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference. For Apple Mac / Metal devices, set -DGGML_CUDA=OFF then continue as usual - Metal support is on by default.

Download the model via the code below (after installing pip install huggingface_hub). You can choose Q4_K_M or other quantized versions like UD-Q4_K_XL . We recommend using at least 2-bit dynamic quant UD-Q2_K_XL to balance size and accuracy. If downloads get stuck, see: Hugging Face Hub, XET debugging

Then run the model in conversation mode:

Llama-server serving & deployment

To deploy Nemotron-3-Ultra locally, use llama-server. In a new terminal, for example via tmux, deploy the model:

If you downloaded the model manually, use:

Then in a new terminal, after installing the OpenAI client with pip install openai:

And on 4 B200s, around 40 tokens / s is seen for generation!

We also did KLD analysis for our GGUF quants - on a log mean KLD scale, the model loses very little accuracy when quantized down to even 1bit due to our dynamic methodology where more important layers are left in higher precision and the rest in lower bits.

For a linear scale:

Nemotron 3 Ultra is NVIDIA's largest Nemotron 3 reasoning model and is positioned for leading accuracy on frontier reasoning, coding and agentic tasks while optimizing time to task completion through high throughput.

Ultra is especially suited for workloads where task success depends on sustained reasoning rather than short single-turn responses:

As shown in Figure 1 and Figure 2 Nemotron 3 Ultra leads on accuracy on agent productivity, instruction following, and long context tasks and provides leading throughout, saving 30% on costs compared to other leading open models.

Figure 1: Nemotron 3 Ultra leads among open models on agentic benchmarks for agent productivity, coding, and instruction following.

Image of a table showing Nemotron 3 Ultra leading among open models on agentic benchmarks for agent productivity, coding, and instruction following.

Figure 2: Nemotron 3 Ultra saves up to 30% in costs and leads on the cost efficiency frontier

Image showing Nemotron 3 Ultra saves up to 30% in costs and leads on the cost efficiency frontier

More benchmarks from NVIDIA:

Vals.ai Financial Agent 1.1

IMOAnswerBench (no tools)

IMOAnswerBench (with tools)

Apex-Shortlist (no tools)

Apex-Shortlist (with tools)

OmniScience Non-Hallucination

Chat & Instruction Following

MMLU-ProX (avg en/de/fr/es/it/ja/zh/hi/pt/ko)

curl -fsSL https://unsloth.ai/install.sh | sh
irm https://unsloth.ai/install.ps1 | iex
source unsloth_studio/bin/activate
unsloth studio -H 0.0.0.0 -p 8888
unsloth studio run --model unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF:UD-Q4_K_XL
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
pip install huggingface_hub
hf download unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF \
    --local-dir unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF \
    --include "*UD-IQ3_XXS*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
./llama.cpp/llama-cli \
    --model unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF/UD-IQ3_XXS/NVIDIA-Nemotron-3-Ultra-550B-A55B-UD-IQ3_XXS-00001-of-00006.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01
./llama.cpp/llama-server \
    -hf unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF:UD-IQ3_XXS \
    --alias "unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B" \
    --temp 1.0 \
    --top-p 0.95 \
    --port 8001
./llama.cpp/llama-server \
    --model unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF/UD-IQ3_XXS/NVIDIA-Nemotron-3-Ultra-550B-A55B-UD-IQ3_XXS-00001-of-00006.gguf \
    --alias "unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B" \
    --temp 1.0 \
    --top-p 0.95 \
    --port 8001
from openai import OpenAI
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B",
    messages = [
        {"role": "user", "content": "What is 2+2?"},
    ],
)
print(completion.choices[0].message.reasoning_content)
print(completion.choices[0].message.content)