Tune vLLM for throughput and memory

The way to tune vLLM is to change one setting at a time and measure each change with the vLLM benchmark command, vllm bench serve, on the same prompts. The settings that move memory and throughput are few: how much of the GPU vLLM takes, how long a request may be, how many requests run at once, how the KV cache is stored, how many GPUs share the model, and whether prefix caching or speculative decoding fits your traffic. Everything here was checked on 12 November 2026 against vLLM 0.33.0, released on 2 November 2026, using its documentation and source at that tag. The vLLM guide covers install and first launch, and every flag below goes after the image name in its container command, with the tag pinned to vllm/vllm-openai:v0.33.0, which exists on Docker Hub and is built for CUDA 13.0, so the VM needs a driver of 580 or newer.

Read the numbers vLLM prints at startup

vLLM's parallelism documentation shows the two log lines that tell you what the settings bought you. The KV cache size is the number of tokens the cache can hold, and the concurrency line is how many requests of the maximum length fit:

INFO 07-23 13:56:04 [kv_cache_utils.py:775] GPU KV cache size: 643,232 tokens
INFO 07-23 13:56:04 [kv_cache_utils.py:779] Maximum concurrency for 40,960 tokens per request: 15.70x

sudo docker logs vllm 2>&1 | grep -i "kv cache"

vLLM's benchmark CLI guide gives the relation as max_concurrency = kv_cache_size / max_model_len, and 643,232 divided by 40,960 is the 15.70 above. Every setting in the next table moves one of those two numbers or the speed of a request. The KV cache explainer shows what a token costs for your model.

The settings that change memory and throughput

The defaults come from the engine arguments page for 0.33.0, except the two batch limits. That page lists no usable default for them and says the default shown in its text is mainly for convenience when testing, so I read the real ones from get_batch_defaults in vllm/engine/arg_utils.py at the tag. Run vllm serve --help=max-num-seqs to see a flag for your own release.

GPU memory utilization and the KV cache

--gpu-memory-utilization is the fraction of GPU memory vLLM may use for the model executor. The engine arguments page calls it a per-instance limit, so two servers on one GPU can each take 0.5. vLLM measures the weights, the activations and the CUDA graph memory at startup, and the KV cache gets what remains of that share. --kv-cache-memory-bytes sets the cache size in bytes instead and then ignores the fraction.

When the cache is too small for the traffic, vLLM preempts requests and recomputes them later, which its tuning page warns can hurt latency. It lists four fixes: raise --gpu-memory-utilization, lower --max-num-seqs or --max-num-batched-tokens, raise the tensor parallel size or raise the pipeline parallel size. When startup itself fails for memory, CUDA out of memory has the vLLM section, and CUDA_VISIBLE_DEVICES explains pinning a server to a card.

Context length, concurrency and the real defaults

A smaller --max-model-len lowers the cache one request can reserve, which is why I lower it first. The two batch limits depend on the card, and these are the 0.33.0 values for the API server and for the offline LLM class:

From the datasheet memory, calculated, the B200 is in the first row, the H100, H200 and RTX PRO 6000 Blackwell are in the second, and the RTX A6000, RTX 6000 Ada, L40, L40S and A100 are in the third, the A100 because the source excludes it by name. So a 48 GB card starts with a 2048-token budget. The tuning page says smaller values such as 2048 give better inter-token latency, higher values give a better time to first token, and it recommends more than 8192 for throughput, especially for smaller models on large GPUs. --performance-mode throughput favors aggregate tokens per second, and in the source it doubles both defaults when you have not set them yourself.

vLLM quantization for the KV cache and the weights

--kv-cache-dtype fp8 stores each cache entry in 1 byte instead of 2. The quantized KV cache page says it can significantly reduce the memory footprint, which lets you store more tokens, and that without calibration all scales are 1.0. It recommends calibrating with llm-compressor, and --kv-cache-dtype-skip-layers sliding_window leaves the layers that are more sensitive at their native type. Support differs by model and GPU, so test your pair.

For the weights, a pre-quantized checkpoint loads without a flag because vLLM reads the quantization_config in the model config. Online quantization converts a BF16 model at load time:

vllm serve meta-llama/Llama-3.1-8B --quantization fp8_per_tensor

The model name is the page's own and is a gated repository, so use your own. The download stays at the BF16 size. The quantization guide says which format each GPU generation runs.

Tensor parallel size on a 2, 4 or 8 GPU VM

Multi-GPU vLLM here means one VM: an offer has 1, 2, 4 or 8 GPUs, and there is no multi-node. vLLM's parallelism guide says that if the model fits on one GPU, distributed inference is probably unnecessary, and that if it is too large for one GPU but fits in one node, you use tensor parallelism, with tensor_parallel_size set to the GPU count. The tuning page adds that it shards the weights so each GPU has more memory left for cache, at the price of synchronization. Docker needs --ipc=host, or --shm-size, because tensor parallel inference uses shared memory.

--tensor-parallel-size 2

As an example, Qwen3.8-27B in BF16 is 55.56 GB of safetensors files. By the site's rule of 92% of the card minus the checkpoint, that leaves 18.0 GB on one 80 GB card and 32.8 GB across two 48 GB cards (calculated). Each process reads the whole model and takes its share, which the docs say lengthens the disk read in proportion to the tensor parallel size. Check the links with nvidia-smi topo -m, and the multi-GPU page explains what NVLink or PCIe means for performance. The parallelism guide says that on GPUs without NVLink, such as the L40S, pipeline parallelism can give higher throughput and lower communication overhead than tensor parallelism. For a small model, data parallelism replicates it, or you run one container per GPU. vLLM vs SGLang compares the flags of the two servers.

Prefix caching in vLLM

Automatic prefix caching keeps the KV cache of earlier requests so a new request with the same prefix skips computing it. The page names two workloads: repeated questions about one long document, and multi-round chat. It only shortens the prefill and does nothing for the decoding of a long answer. The page still tells you to set enable_prefix_caching=True, while the engine arguments list --enable-prefix-caching and --disable-prefix-caching and the 0.33.0 source enables it for generative models. That makes it on unless you switch it off. A benchmark can fool you here: repeating vllm bench serve against the same server can reuse cached prompts and inflate throughput.

Speculative decoding in vLLM

The speculative decoding page says it reduces inter-token latency under medium-to-low QPS, memory-bound workloads, and that real gains depend on the model, the traffic, the hardware and the sampling settings. Its qualitative table rates the methods:

N-gram is the one to try first, and its arguments are the page's own example:

--speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 4,
    "prompt_lookup_min": 2,
    "prompt_lookup_max": 5
  }'

The page says the standard rejection sampler is theoretically lossless up to hardware precision, and that the FLy method deliberately relaxes verification and is not distribution preserving.

Structured outputs and batch inference

Structured outputs are supported by default in the OpenAI-compatible server, with xgrammar or guidance as the backend and --structured-outputs-config.backend defaulting to auto. A request adds the constraint through extra_body, for example {"structured_outputs": {"choice": ["positive", "negative"]}}. For offline batch inference, the LLM class takes generate and chat, and vllm run-batch reads the OpenAI batch file format, which the batch example describes as a way to run batches and not the complete Batch REST API:

vllm run-batch -i batch.jsonl -o results.jsonl --model openai/gpt-oss-20b

Run the vLLM benchmark before and after each change

sudo docker exec vllm vllm bench serve --help
sudo docker exec vllm vllm bench serve \
  --model openai/gpt-oss-20b \
  --dataset-name random --random-input-len 1024 --random-output-len 128 \
  --num-prompts 200 --max-concurrency 32 --save-result

The vLLM CLI guide says the bench commands need the extras from pip install vllm[bench], so read the help output first and install them in the container if it complains. The flags above are documented in the benchmark CLI guide, and in the 0.33.0 source the client talks to 127.0.0.1:8000, uses the random dataset with 1024 input and 128 output tokens by default, and sends 1000 prompts unless --num-prompts says otherwise. The result lists request and token throughput, then mean, median and P99 for time to first token, time per output token and inter-token latency, which the guide defines at the client.

Use the load flags to match your traffic. --request-rate defaults to infinity and --max-concurrency to no limit, and the guide calls --request-rate=inf with a --max-concurrency limit the most common pattern for finding throughput. Between runs, vary --seed, restart the server, or use vllm bench sweep serve, which resets caches. Restarts recompile, and vLLM's Docker page says a named volume mounted at /root/.cache/vllm keeps the compile cache for the next container. The tuning page says a change to the model, the config, the relevant environment variables, the torch build or the GPU model invalidates that cache, so a changed flag may still recompile. The guide itself suggests GuideLLM for benchmarking production servers. If GPU utilization stays low during a run, vLLM's docs name CPU contention, and a deployment with N GPUs needs at least 2 + N physical cores, which the docs say is 2 x (2 + N) vCPUs when hyperthreading is on, so check nproc and monitor GPU usage.