Run llama.cpp server on a cloud GPU

The llama.cpp server puts a quantized GGUF model behind an OpenAI-compatible API with one container, one flag that pulls the model from Hugging Face and one SSH tunnel to reach it, which makes it a good fit when a single model and a small number of users are the whole job. This guide runs the official CUDA image on a plain GPU VM, keeps the port off the internet, and covers multi-GPU splitting and a source build for when you want one.

Pick a GPU and a quant that fit

GGUF is the file format llama.cpp loads, and the quant in a file's name, such as Q4_K_M or Q8_0, decides how big it is. The -hf flag takes user/model with an optional quant, and its documented default is Q4_K_M, or the first file in the repository if there is no Q4_K_M. The quantization guide explains the formats. These are real file sizes from the Hugging Face file listings, checked on 1 October 2026:

The two right-hand columns are calculated as the card's nominal memory minus the file size. They leave out the KV cache, compute buffers and the CUDA context, so they are an upper bound on what context can use. The cache grows with every token of context, and the KV cache explainer has the formula. The smallest card DeployGPU launches is 48 GB, so the first two rows are a comfortable start.

Check the driver, then run the CUDA image

Run nvidia-smi and read the CUDA version in the header. The Docker docs for llama.cpp publish a server-cuda image built with CUDA 12 and a server-cuda13 image built with CUDA 13, and say the GPU images are not tested by CI beyond being built. Take the one your header allows, and use sudo docker. The toolkit guide has the Docker GPU check and the tag rule.

sudo docker run -d --name llama --gpus all \
  -p 127.0.0.1:8080:8080 \
  -e LLAMA_CACHE=/models -v ~/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -hf ggml-org/gpt-oss-20b-GGUF -hff gpt-oss-20b-MXFP4.gguf \
  --alias gpt-oss-20b -c 32768 --n-gpu-layers 99 \
  --host 0.0.0.0 --port 8080

The flags come from the server README. -hf names the repository and -hff the exact file, which removes any doubt about which file gets picked. --alias sets the model name the API reports, -c sets the context size and --n-gpu-layers 99 keeps every layer on the GPU, as the README's own CUDA example does. Current builds also default to --fit on, which adjusts unset arguments to fit device memory, and I still set -c and --n-gpu-layers myself so memory use is predictable. The LLAMA_CACHE variable, which llama.cpp reads, puts the download in ~/models on the VM.

The server listens on 0.0.0.0 inside the container, as the project's Docker examples do. The 127.0.0.1 in the -p flag is what keeps it private, because DeployGPU does not firewall VMs and Hyperstack VMs are created with inbound traffic open. Watch the log. The health route answers 503 with a Loading model message until the file is loaded, then {"status": "ok"}:

sudo docker logs -f llama
curl http://127.0.0.1:8080/health

Reach the API through an SSH tunnel

Open a tunnel from your own machine, using the user, host and port from the deployment. The user is ubuntu on QuantaCloud and Hyperstack, Ubuntu with a capital U on Massed Compute and root on Verda. DeployGPU app links do not accept API keys and there is no llama.cpp template, so the tunnel is the route:

ssh -N -L 8080:127.0.0.1:8080 ubuntu@YOUR_VM_IP -p 22

While it runs, http://localhost:8080 opens the built-in web interface that the README says lives at the server's own address, and the OpenAI-compatible routes sit under /v1:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-oss-20b", "messages": [{"role": "user", "content": "Say hello in five words"}]}'

An OpenAI client works with base_url set to http://localhost:8080/v1, and the README's own examples use sk-no-key-required as the key. If anything else on the VM could reach the port, add --api-key with a random value. The port forwarding guide covers tunnels in more depth.

Use more than one GPU

With --gpus all, llama.cpp sees every card on the VM. --split-mode layer is the default and splits layers and the KV cache across the GPUs. The row mode splits weights by rows, and tensor is marked experimental in the README. --tensor-split 3,1 sets the share each GPU gets, so unequal cards can be used. As a calculation, a 2x 48 GB VM has 96 GB across its cards, which holds the 63.39 GB gpt-oss-120b file with about 33 GB left before cache and buffers. Use --gpus '"device=0,1"' to hand the container only some of the cards, and see the multi-GPU page for how the VM sizes differ.

Build from source with CUDA instead

The build documentation starts from an installed CUDA toolkit, so check nvcc --version first. The plain VM is documented as the provider's Ubuntu image with the driver and Docker, and the CUDA toolkit is not part of that description. The prebuilt Ubuntu CUDA archives for 12.8 and 13.4 on the releases page are the other route. To build:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
sudo apt-get install -y libssl-dev
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -t llama-server
./build/bin/llama-server -hf ggml-org/gpt-oss-20b-GGUF -hff gpt-oss-20b-MXFP4.gguf -c 32768 --n-gpu-layers 99

The libssl-dev line matters. The build docs say the project builds without SSL support if the OpenSSL development files are missing, and the source then refuses HTTPS downloads with HTTPS is not supported. Please rebuild with one of, which is what -hf needs. By default the build targets the GPU connected at that time, and the server listens on 127.0.0.1:8080 unless you pass --host. The newest README quick start also shows a llama serve launcher. This guide uses llama-server, the binary the server documentation and the Docker images use.

When something goes wrong

Stopping the deployment deletes the instance disk, including ~/models. The same download happens on the next launch, so note the file you used. The stop guide covers what to copy off first, and the self-hosting guide and the engine comparison help when llama.cpp is not the right server.