Fix CUDA out-of-memory errors

The error below means a process asked the GPU for memory that was not free at that moment, so the first job is to find out who holds the rest before you change a single setting.

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate ... GPU 0 has a total capacity of ... of which ... is free. Process ... has ... memory in use. Of the allocated memory ... is allocated by PyTorch, and ... is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (...)
RuntimeError: CUDA error: out of memory
ValueError: Free memory on device ... on startup is less than desired GPU memory utilization (...). Decrease GPU memory utilization or reduce GPU memory used by other processes.

The first line is PyTorch's allocator, which training scripts, vLLM and ComfyUI all use. It prints as torch.OutOfMemoryError or torch.cuda.OutOfMemoryError, depending on the release. The second comes from the CUDA runtime. The third is vLLM refusing to start. ComfyUI words it as Allocation on device 0 would exceed allowed memory.

Check what holds the memory with nvidia-smi

My first command is nvidia-smi, because no setting helps if another process owns the card. The second lists each process using the GPU and its memory:

nvidia-smi
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv

The usual holders are a Jupyter kernel from an earlier cell (restart the kernel), a vLLM container you forgot (sudo docker stop vllm), a model still loaded in Ollama, or a second copy of your own script. On the Open WebUI template, sudo docker exec deploygpu-workload ollama ps shows loaded models, and Ollama keeps one in memory for 5 minutes after its last request by default.

PyTorch's CUDA notes add a trap: memory its caching allocator keeps for reuse still shows as used in nvidia-smi, and torch.cuda.empty_cache() only hands that cache to other processes.

Training: batch size, gradient checkpointing and bf16

For a training script I change four things in this order: batch size, gradient accumulation to keep the effective batch, gradient checkpointing, then mixed precision. Hugging Face's TrainingArguments reference names all four:

from transformers import TrainingArguments

args = TrainingArguments(
    output_dir="out",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    gradient_checkpointing=True,
    bf16=True,
)

Gradient checkpointing recomputes activations in the backward pass instead of storing them, which PyTorch's checkpointing notes describe as trading compute for memory. If the error says a lot of memory is reserved but unallocated, the allocator is fragmented, and PyTorch's notes offer an experimental fix. Their current documentation names the variable PYTORCH_ALLOC_CONF and keeps the old name as an alias (checked 1 October 2026):

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
python train.py

For fine-tuning, the memory page compares full, LoRA and QLoRA.

vLLM: lower --max-model-len before anything else

vLLM fails in two places. At startup it refuses to run when free memory is below its target, the third line above. After loading the weights it can stop because the KV cache cannot hold one full-length request, and that message names both fixes: raise --gpu-memory-utilization or lower --max-model-len. I lower the context first, because utilization cannot exceed what is free.

sudo docker rm -f vllm
sudo docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-openai:latest \
  --model openai/gpt-oss-20b \
  --max-model-len 16384 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.90

--max-num-seqs caps concurrent requests. --gpu-memory-utilization is a per-instance fraction, 0.92 by default in the current documentation and 0.9 in older releases, so two servers on one GPU need fractions that sum below 1. --enforce-eager turns off CUDA graphs. Sources: vLLM's engine arguments and memory conservation pages, checked 1 October 2026. The engine logs how much KV cache it found room for:

sudo docker logs vllm 2>&1 | grep -i "kv cache"

The KV cache explainer and the vLLM guide go further.

Ollama: a model that is too big runs on the CPU

When a model does not fit, Ollama can split it between GPU and CPU instead of raising an error, and ollama ps shows that as a value such as 48%/52% CPU/GPU. A larger context window is a quiet cause, because the KV cache grows with it. Ollama's FAQ gives the default as 4096 tokens and says parallel requests multiply the context. The CPU fallback page has the check.

ComfyUI: shrink the workflow, then try --reserve-vram

Lower the image size, frame count or batch size first. Then try the flags on ComfyUI's startup flags page, checked 1 October 2026: --reserve-vram 2 leaves 2 GB for other software, --disable-smart-memory pushes models out to system RAM, --cache-none re-runs nodes instead of caching results, and --novram aims for minimal VRAM use. The page also lists --lowvram, but says it has no effect when dynamic VRAM is on, and that is automatic on NVIDIA GPUs.

python main.py --reserve-vram 2 --cache-none

That suits a plain GPU VM where you start ComfyUI yourself. The ComfyUI template has no dashboard field for start arguments, but its base image reads extra ones from /workspace/runpod-slim/comfyui_args.txt inside the container, per the base image's README. Add the flags there, then restart the container:

sudo docker exec deploygpu-workload sh -c 'printf "%s\n" --reserve-vram 2 >> /workspace/runpod-slim/comfyui_args.txt'
sudo docker restart deploygpu-workload

When 80 GB or more is the answer

If the job only runs at settings you cannot live with, such as a batch of 1, the card is too small. Go up one memory class:

A VM can have up to 8 GPUs, but more GPUs help only when the software splits the model, as vLLM's --tensor-parallel-size and sharded training do. Plain data-parallel training keeps a full copy per GPU, a replica per process in PyTorch's DDP notes. The multi-GPU training guide has the options, and the GPUs page links the 48 GB, 80 GB and 96 GB and larger tiers.

When to contact support

Contact support@deploygpu.ai if nvidia-smi on a VM with nothing of yours running shows a lot of memory in use, or less total memory than the offer lists. Send the deployment ID and the nvidia-smi output.