Run Nemotron on a cloud GPU

The Nemotron 3.5 Lightning model runs on one 80 GB GPU: its BF16 checkpoint is 65.83 GB, and NVIDIA's card validates it on a single H100 at a 256K-token context. Nemotron 3 Super is 247.23 GB and takes 4x H100 or 2x B200. Both keep their KV cache tiny because most of their layers are Mamba-2, which holds a fixed state instead of a cache that grows with context.

Nemotron sizes: 22, 66, 80 and 247 GB

NVIDIA's Lightning card lists 30B parameters with 3B active, a context of up to 1M tokens and a release date of 11 August 2026. The Super card lists 120B with 12B active. Sizes below are the summed safetensors files, checked 1 October 2026, and the left-over figures are 92% of the VM's memory minus the checkpoint, calculated.

The Super card lists 8x H100 80 GB as its minimum GPU requirement, so treat any minimum as a starting point and read the engine's startup log. The TensorRT-LLM guide asks for an NVIDIA driver of version 575 or newer, so run nvidia-smi first. The vLLM recipe has the vLLM sizing.

Why 6 attention layers keep the cache at 6 KiB per token

Lightning has 52 layers, and its config.json counts 23 Mamba-2 layers, 23 mixture-of-experts layers and only 6 attention layers. Only attention layers grow a KV cache. With 2 KV heads of 128 dimensions in BF16, that is 6 KiB per token, so a 256K-token sequence needs about 1.5 GiB and a 1M-token sequence about 6 GiB (calculated). The Mamba-2 layers keep a fixed state instead. Each holds 64 heads of 64 by 128 state values, which across 23 layers is about 46 MiB per sequence in FP32, whatever the length (calculated, with the small convolution state left out). That fits the card validating one 80 GB H100 at 256K, which leaves about 8 GB after the weights. Super has 40 Mamba-2, 40 MoE and 8 attention layers, which is 8 KiB per token. The KV cache explainer covers the other layer types.

Serve Lightning on one 80 GB GPU with vLLM

This is the card's command for 1x H100, in the card's pinned image, wrapped in Docker with the port bound to loopback. The Mamba cache in FP16 is the memory saving the card calls out for a single H100:

sudo docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.27.1 \
  --model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --async-scheduling \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5

The vLLM recipe gives 0.27.1 as the minimum version. For the full 1M-token window the card uses 8x H100, GB200 or B200 and says a single H100 is memory-bound to about 256K. The card also lists an SGLang recipe for 1x H100 and 1x B200 in the lmsysorg/sglang:dev-nemotron3-5-lightning container, and SGLang's cookbook generates the tuned profiles. Reach the server through the tunnel in the vLLM guide.

NVFP4 and Ollama for smaller cards

NVIDIA's NVFP4 build of Lightning is 21.56 GB, and the card sends anyone who wants optimized inference there. For chat, Ollama lists nemotron-3.5-lightning:30b at 25 GB with a 1M context window, nemotron-3.5-lightning:30b-a3b-q8_0 at 35 GB and nemotron-3.5-lightning:30b-a3b-bf16 at 66 GB, checked 1 October 2026. A 48 GB card holds the first two. llama.cpp registers the NemotronH architecture, and ggml-org publishes GGUF files for Lightning.

Nemotron 3 Super on 4x H100

vLLM's recipe for Super gives this reference command for BF16 on 4x H100 with an FP8 KV cache, wrapped in Docker the same way. It needs vLLM 0.17.1 or newer:

sudo docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --served-model-name nemotron \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser nemotron_v3

On a B200, the card says the BF16 checkpoint fits on 2 GPUs, so set the tensor parallel size to 2. Ollama lists nemotron-3-super:120b at 87 GB with a 256K context window, which leaves about 60 GB on 2x 80 GB GPUs (calculated). The H100 page and the 96 GB and larger page cover the cards involved.

Licenses and what a stop deletes

Lightning uses the OpenMDW License Agreement 1.1. It permits dealing in the model without restriction, asks you to keep a copy of the agreement and the notices if you distribute it, and places no restrictions on outputs. Super uses the NVIDIA Nemotron Open Model License, which grants a royalty-free license to reproduce, modify, sublicense and distribute the model, requires recipients to get a copy of the license, and sets no revenue threshold in its text. These are summaries, so read the texts. Stopping a DeployGPU instance deletes its disk, so each launch downloads the 66 GB or 247 GB checkpoint again, and the stop guide covers what to copy off first.