Fix 'No space left on device' on a GPU VM

The error below means the filesystem is full, not the GPU or the network, and on a rented GPU it is almost always Docker images, model weights or a cache nobody pointed somewhere else.

No space left on device

Find out what is actually full

df -h shows free space by filesystem, and it is the first command I run, because the answer is not always the filesystem expected:

df -h

If df -h shows plenty of free space but the error still happens, check inodes instead of bytes. A directory with millions of small files, common in some dataset formats, can exhaust inodes while space looks fine:

df -i

Find what filled it

du totals usage by directory, which narrows down where to look next:

sudo du -xsh /* 2>/dev/null | sort -rh | head -20

For a directory worth exploring rather than just measuring, ncdu is an interactive version of the same idea, if it is installed:

sudo apt-get install -y ncdu
sudo ncdu -x /

Docker is the usual suspect

Docker's own images, stopped containers and build cache add up fast on a machine used for anything with a Dockerfile. Check the total first, then clear it:

sudo docker system df
sudo docker system prune -a --volumes

prune -a removes every image not in use by a running container, not only dangling ones, so confirm nothing needed is sitting only in a stopped container first.

Model and package caches most people forget

Hugging Face's own cache holds every model and dataset ever downloaded through its libraries, by default under the home directory, and it does not clean itself:

du -sh ~/.cache/huggingface

Point HF_HOME at a bigger disk or a volume before the next download if this is where the space went:

export HF_HOME=/mnt/data/huggingface

pip keeps its own download cache too, and it is safe to clear:

pip cache purge

Ollama's pulled models live under its own models directory, and a handful of large ones fills a disk quickly. Its OLLAMA_MODELS environment variable moves that directory the same way HF_HOME moves the Hugging Face cache, per Ollama's own documentation.

The disk is a fixed size chosen at launch

On QuantaCloud, Massed Compute and Hyperstack, the disk size is fixed to whatever the offer included. There is no resizing it after launch, so the only way to more room is to move data off, delete what is not needed, or launch again with a different offer. Verda is the exception: it allows choosing the disk size, up to 500 GB, before launch, so a job known to need more room than the usual offer includes should get it there from the start rather than fighting the cache afterward.

Keeping data off the instance disk entirely

A DeployGPU volume, available on Verda and Hyperstack, is separate from the instance disk and survives stopping the VM, which the instance disk never does. Attach one at launch, format and mount it, and point the caches above at it instead of the home directory:

lsblk
sudo mkfs.ext4 /dev/vdb
sudo mount /dev/vdb /mnt/data

Check lsblk every time rather than assuming the device name from a previous session, since it depends on the provider. Creating and attaching one is covered in the volumes documentation. A volume is billed hourly while it exists, attached or not, so it is a place for data returned to often, not a place to forget. The stop guide covers what does and does not survive between sessions in more depth.

When to contact support

Contact support if df -h shows the disk itself smaller than the offer described, which would be a provisioning problem rather than something filled by use. Everything else on this page is space used by you or your containers, which only you can clear. Model-heavy work fills disks fastest, so if you run ComfyUI or a vLLM server, start looking there.