GGUF: what it is and how to run one on a GPU

The GGUF format stores a model's tensors, usually quantized, together with the metadata needed to load them in one binary file, and llama.cpp and Ollama read that file directly. This guide covers what is inside, how to read names such as Q4_K_M and Q8_0, how the file and the context cache add up on a GPU, how to run a GGUF with llama.cpp and with Ollama, how to convert your own, how GGUF compares with safetensors and what vLLM's documentation says about it. The format and tool facts were checked on 12 November 2026.

What a GGUF file contains

The format specification in the ggml repository describes GGUF as a binary format for storing models for inference with GGML and executors based on it, designed for fast loading and saving. It is meant to be a single file with everything needed to load the model. A file starts with the bytes GGUF, a version number that is 3 in the current spec, a tensor count and a count of metadata key-value pairs. Then come those pairs, a description of each tensor and the tensor data, padded to an alignment that is 32 bytes when the file does not say otherwise. Required metadata includes general.architecture, and general.quantization_version when any tensor is quantized.

Hugging Face's Hub documentation adds that, unlike tensor-only formats such as safetensors, GGUF encodes both the tensors and a standardized set of metadata, and the Hub has a viewer for that metadata on model pages. The metadata can carry the chat template: Ollama's Hub integration picks a template from the tokenizer.chat_template entry stored in the file. The spec's naming convention puts the size, version, encoding and shard into the name, as in Grok-100B-v1.0-Q4_0-00003-of-00009.gguf, shard 3 of 9. A mmproj- prefix marks a multimodal projector that loads beside a base model.

GGUF quantization names: Q4_K_M, Q8_0 and IQ4_XS

llama.cpp's quantize README measures its quantization types on Llama 3.1 8B:

The number is the nominal bit width. The K marks the k-quants scheme from the k-quants pull request, which stores weights in super-blocks, 8 blocks of 32 weights for Q4_K. The S, M and L suffixes name a mix of tensor types. In that pull request Q4_K_S uses Q4_K for all the tensors it lists, while Q4_K_M uses the 6-bit Q6_K for half of the attention value and feed-forward tensors and Q4_K for the rest, and every variant keeps the output tensor at 6 bits. The README sizes grow in that order, from 4.36 GiB for Q4_K_S to 4.58 GiB for Q4_K_M. The IQ names are i-quants, which Hugging Face's table describes as using an importance matrix, and llama-quantize takes one with --imatrix. Q8_0 and Q4_0 are older block types with 32 weights per block, F16, BF16 and F32 are not quantized, and MXFP4 is a 4-bit microscaling type. The quantization guide covers which format suits which GPU generation.

How much GPU memory a GGUF file needs

A GGUF fits when the file, the context cache and the working buffers fit in GPU memory. The site's rule is usable memory equal to 92% of the VM's GPU memory minus the file, and this table applies it to the ggml-org conversions of Qwen3.8-27B, with sizes read from the Hugging Face API on 12 November 2026:

The bits per weight come from the file size and the 26.9 billion parameters the Hub reads from the file. The Q4_K_M file works out to 5.77, against 4.89 in the README's measurement on a smaller model, so take the size from the repository's file list and not from the name. What is left is for the cache and buffers. The model's config.json lists 64 layers, of which 16 use full attention with 4 key-value heads of size 256. An F16 cache then costs 2 x 16 x 4 x 256 x 2 bytes, which is 64 KiB per token (calculated), or 2 GiB at 32,768 tokens and 16 GiB at 262,144. The other 48 layers use linear attention and add a fixed state that this figure leaves out. The KV cache explainer has the formula for other models.

On a 48 GB card the Q4_K_M leaves 24.8 GB, so the full 262,144-token cache of 17.2 GB fits with about 7.6 GB to spare, before buffers. The Q8_0 leaves 15.6 GB, which holds 131,072 tokens but not the full cache. The BF16 file needs an 80 GB card. llama.cpp's server README lists -ctk and -ctv for the cache type, with f16 as the default and q8_0 among the options, and Ollama's FAQ says a q8_0 cache uses approximately half the memory of f16.

Download one GGUF file

A GGUF repository is a menu, with one file per quantization, so download the item you will run. The Hugging Face CLI takes a repository, a file name and a target directory, and downloading models fast covers the token, the cache and the speed settings:

hf download ggml-org/Qwen3.8-27B-GGUF Qwen3.8-27B-Q4_K_M.gguf --local-dir ~/models

A model that is too large for one file comes in shards named like -00001-of-00002.gguf, and every shard has to be downloaded. llama.cpp can also fetch a file itself with -hf, as the llama.cpp server guide shows. The disk is deleted when the instance stops, so the download happens again on the next launch.

Run a GGUF on a GPU with llama.cpp

llama.cpp has custom CUDA kernels for NVIDIA GPUs, and its README lists CPU and GPU hybrid inference for models larger than the VRAM. The server needs an image or a build for your driver. The Docker documentation lists server-cuda for CUDA 12 and server-cuda13 for CUDA 13, so pick the one your driver allows, which is at or below the CUDA Version in nvidia-smi, as How to check your CUDA version shows. The command is the one from the llama.cpp guide with -m in place of -hf:

sudo docker run -d --name llama --gpus all \
  -p 127.0.0.1:8080:8080 \
  -v ~/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3.8-27B-Q4_K_M.gguf \
  --alias qwen3.8-27b -c 32768 --n-gpu-layers 99 \
  --host 0.0.0.0 --port 8080

The server README defines -m as the model path and --n-gpu-layers as the maximum number of layers stored in VRAM, either a number, auto or all, with auto as the default. A value of 99 keeps every layer on the GPU. The -c default of 0 loads the context from the model, so I set it myself. For this model that would otherwise mean a cache sized for 262,144 tokens, unless --fit shrinks it. The README's default port is 9931, so the --port flag keeps it at 8080. The 127.0.0.1 in the publish flag keeps the port private because DeployGPU does not firewall VMs, and an SSH tunnel reaches it, as the llama.cpp server guide explains. llama.cpp vs Ollama covers the trade-off between the two.

Run a local GGUF with Ollama

Ollama's import page says it does not quantize GGUF models during import, so quantize first. For a single file, save a Modelfile in the current directory and build the model from there:

FROM /path/to/Qwen3.8-27B-Q4_K_M.gguf

ollama create qwen38-27b
ollama run qwen38-27b

For a split model the page says to keep the original shard names and use a wildcard such as FROM /path/to/model-*.gguf. If you run the command with sudo docker exec, the path has to exist inside the container, so mount your models folder with -v when you start it. Ollama can also pull a GGUF straight from the Hub. The Hub's Ollama page uses ollama run hf.co/{username}/{repository}, defaults to Q4_K_M when the repository has it, and takes a quantization tag after a colon:

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

The Ollama server guide covers installing the service and reaching its API.

Convert and quantize your own model

The commands are the two steps in the quantize README, and the quantization guide shows them with a Hugging Face repository. With a model already on disk, give the directory and drop --remote:

python3 -m pip install -r requirements.txt
python3 convert_hf_to_gguf.py /path/to/model-directory --outfile model-bf16.gguf --outtype bf16
./build/bin/llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_M

Run them from a llama.cpp checkout with the tools built. The conversion script's --outtype accepts f32, f16, bf16, q8_0, tq1_0, tq2_0 and auto. The README warns that requantizing a quantized file can severely reduce quality, so start from the 16-bit file. Models are fully loaded into memory, and for Llama 3.1 70B the README lists 280.9 GB for the original and 43.1 GB at Q4_K_M. Check df -h before you start, because the disk is fixed on QuantaCloud, Massed Compute and Hyperstack and chosen at launch on Verda. The README also points to the GGUF-my-repo space on Hugging Face for building quants without any setup.

GGUF vs safetensors

The safetensors README describes the format as a simple way to store tensors safely, as opposed to pickle, that is still fast. The choice follows the engine you run. vLLM vs llama.cpp compares them for a rented GPU.

What vLLM says about GGUF

The vLLM v0.33.0 GGUF page calls GGUF support highly experimental and under-optimized, says it might be incompatible with other features and says GGUF is currently a way to reduce memory footprint. Support moved to a separate package, so install the plugin first:

uv pip install vllm-gguf-plugin
vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B

The repo_id:quant_type form loads from the Hub, and the page recommends the tokenizer of the base model because converting one from GGUF is time-consuming and unstable. Its hardware table marks GGUF as supported through Hopper and has no Blackwell column. The Dockerfile for v0.33.0 installs the gguf Python package into the vllm/vllm-openai image and does not mention the plugin, so check for it in a container. For a GGUF file I use llama.cpp or Ollama, and for vLLM I use safetensors checkpoints, with AWQ or FP8 for a smaller footprint, as the quantization guide and tuning vLLM show.