Choosing a quantization format for deployment

The quantization format you can deploy depends on the GPU generation before it depends on the model. DeployGPU has four generations: Ampere in the A100 and RTX A6000, Ada in the L40, L40S and RTX 6000 Ada, Hopper in the H100 and H200, and Blackwell in the B200, B300 and RTX PRO 6000. INT4 weights run on all four, FP8 computes natively from Ada up, the 4-bit floating-point formats compute natively only on Blackwell, and GGUF is the format llama.cpp and Ollama read.

Which generation runs INT4, FP8 and FP4

The table combines vLLM's quantization support table with its FP8, INT4 and ModelOpt pages and NVIDIA's NVFP4 post, all checked 1 October 2026. The support table has columns for Ampere, Ada and Hopper but none for Blackwell, so the Blackwell column comes from the text of the other sources. In vLLM's names, W8A8 means 8-bit weights and 8-bit activations, and W4A16 means 4-bit weights with 16-bit activations. Weight-only means the weights are stored in the small format while the math still runs in 16 bits, so the card gets the memory saving and does no low-precision math.

The FP8 page says FP8 computation needs compute capability 8.9 or higher, which is Ada, Hopper and Blackwell, and that Ampere cards run FP8 models weight-only through Marlin kernels. The INT4 page lists all four generations. The ModelOpt page says that without a native FP4 kernel, vLLM falls back to weight-only W4A16 through Marlin and logs a warning, which may reduce throughput on compute-heavy work. NVIDIA's post lists MXFP4 and NVFP4 among the 4-bit formats that Blackwell supports.

The RTX A6000 and A100 therefore run an FP8 checkpoint for the memory saving only. A Hopper card does FP8 natively but takes a different code path from Blackwell for the MXFP4 weights of gpt-oss, which the gpt-oss page covers. The live table lists the main cards in generation order, Ampere first and Blackwell last.

One model in six formats: Llama 3.3 70B

Llama 3.3 70B has a published file in each of these formats, so it works as a yardstick. The sizes come from the Hugging Face repositories and from Ollama's tag list, checked 1 October 2026, and the GGUF rows are Ollama's tags, in a format llama.cpp reads too. The last column is 92% of the memory, vLLM's default limit, minus the file, calculated. It ignores activations, so what is left is the budget for the KV cache.

NVIDIA's FP8 card lists Blackwell, Hopper and Lovelace, its name for Ada, as supported hardware, and the NVFP4 card lists Blackwell only. Neither names Ampere. A 48 GB card holds the 43 GB Q4_K_M file with about 1 GB to spare (calculated), which leaves nothing for a cache, so keeping a 4-bit 70B fully on the GPU takes an 80 GB card. The Llama page has the full fit table for the family.

Quality is what you trade for the smaller files, and vendors publish their own numbers. NVIDIA's post reports 1% or less accuracy degradation against FP8 on DeepSeek-R1-0528 benchmarks after quantizing to NVFP4. Meta's Muse Glimmer card reports 0.2% and 1.0% average degradation for its two K-quant builds across 15 benchmarks. Both are vendor measurements on their own models, so test yours on your own prompts.

Bits per weight: MXFP4 4.25, NVFP4 4.5, Q4_K_M 4.89

The bits per weight set the file size, and block scales add a fraction of a bit to each value. The Microscaling paper describes MX formats as a per-block scaling factor combined with narrow floating-point and integer types for individual elements. MXFP4 uses one shared power-of-two scale per 32 values, per NVIDIA, and vLLM's docs give that scale as 8 bits, so the cost is 4 bits plus 0.25 bits of scale, or 4.25 bits (calculated). OpenAI's gpt-oss report gives the same 4.25 bits per parameter for its MXFP4 weights, checked 1 October 2026. NVIDIA says NVFP4 stores one 4-bit value plus one FP8 scale per 16 values, which is 4.5 bits per value and cuts the memory footprint by about 3.5 times against FP16 and 1.8 times against FP8.

llama.cpp's quantize README, checked 1 October 2026, measures its GGUF types on Llama 3.1 8B. Q4_K_M is 4.89 bits per weight, Q5_K_M 5.70, Q6_K 6.56, Q8_0 8.50, Q3_K_M 4.00 and Q2_K 3.16. Its size table for Llama 3.1 gives 43.1 GB for the 70B at Q4_K_M and 249.1 GB for the 405B.

AWQ and GPTQ are 4-bit integer methods for GPUs. vLLM's AutoAWQ page, checked 1 October 2026, says that library is deprecated and its functionality now lives in LLM Compressor, and its GPTQModel page says compatible checkpoints can use the Marlin and Machete kernels on Ampere and Hopper. GGUF is the file format of llama.cpp and Ollama. vLLM can read it, but its GGUF page calls that support highly experimental and under-optimized and says it needs a separately installed plugin. Ollama's import page says it does not quantize GGUF models during import, so quantize first with a GGUF tool such as llama.cpp's llama-quantize. The llama.cpp README's two steps are a conversion to a BF16 GGUF and then the quantize step:

python convert_hf_to_gguf.py --outfile gemma-4-E2B-it-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M

Run both from a llama.cpp checkout with its Python requirements installed and its tools built. The README warns that requantizing a file that is already quantized can severely reduce quality, so start from the 16-bit or 32-bit file, and it says an importance matrix file can minimize the loss. Models are fully loaded into memory, so the machine needs enough RAM and disk for the files involved. The llama.cpp server guide covers serving the result.

Pick FP8, AWQ, NVFP4 or GGUF by card and engine

My rule is to start from the card, then the engine, then the checkpoint that exists:

Quantize at load time with vLLM

vLLM's online quantization page, checked 1 October 2026, says it converts a BF16 or FP16 model's Linear and MoE weights to lower precision such as FP8 at load time, with no pre-quantized checkpoint and no calibration data. Its example is vllm serve meta-llama/Llama-3.1-8B --quantization fp8_per_tensor. In the container pattern of the vLLM guide, with an ungated model so no token is needed, it looks like this:

sudo docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.30.0 \
  --model Qwen/Qwen3-8B \
  --quantization fp8_per_tensor

The download is still the BF16 size, and on DeployGPU the disk is deleted when you stop, so a pre-quantized checkpoint also saves download time on every launch. The same page lists fp8_per_block, mxfp8 and mxfp4 among its other schemes, and says mxfp8 needs Blackwell for 8-bit weights and activations and uses 16-bit activations on other GPUs.

The KV cache is a separate choice from the weights. vLLM's quantized KV cache page says quantizing the KV cache to FP8 can significantly reduce its memory footprint, which lets you store more tokens, and the KV cache explainer shows the arithmetic. For fine-tuning, the 4-bit base of QLoRA is the same idea, covered in how much VRAM to fine-tune. The models hub covers other open models, and the GPU list is the place to pick a card.

My pick for trying every format in this guide is the RTX PRO 6000. It is Blackwell, so FP8 and the 4-bit floating-point formats run natively, and its 96 GB holds the 70B at FP8, Q8_0 or NVFP4 with room left, as the table above shows.