Deploy vLLM on a cloud GPU

The goal of this guide is an OpenAI-compatible endpoint that answers only to you: a laptop or an app server, an SSH tunnel, and a vLLM container inside a rented GPU VM that never listens on anything but localhost. DeployGPU has no vLLM template, so everything here runs on a plain GPU VM with SSH, and every command below comes from vLLM's own documentation and runs the official Docker image.

Choose a GPU

Pick a class before opening the marketplace, based on the model:

These are calculated from published checkpoints, not measured. The KV cache explainer covers what context length and concurrency add on top of the weights.

Launch a GPU VM

Open the marketplace, pick an offer in the class above, and choose Bare Metal under Template on the deploy page. It is the dashboard's name for a plain GPU VM rather than actual bare metal: the provider's Ubuntu image with NVIDIA drivers and Docker already installed, and nothing else running on top.

1. Check the cost summary and the SSH key on the deploy page, then deploy. The first hour is reserved from your balance.
2. Wait on the Deployments page until the status reads Running, usually 1 to 3 minutes for a plain VM.
3. Copy the exact ssh command shown on the deployment. The username differs by provider: ubuntu on QuantaCloud and Hyperstack, Ubuntu with a capital U on Massed Compute, root on Verda. The SSH guide has the full table.
4. If the first connection attempt is refused, wait a minute and try again. Some providers report Running slightly before SSH is ready to accept connections. The connection refused fix covers what to check if it does not clear on its own.

Confirm the driver is visible before pulling anything:

nvidia-smi

Start vLLM

Pull the official image and run it with the API port bound only to the VM's own loopback address, not to every interface:

sudo docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN=$HF_TOKEN \
  vllm/vllm-openai:latest \
  --model openai/gpt-oss-20b \
  --max-model-len 32768

--ipc=host gives the container access to the host's shared memory, which vLLM's own documentation says it needs for tensor parallel inference, and it is harmless on a single GPU too. The volume mount puts the Hugging Face cache on the VM's disk instead of the container's writable layer, which matters once a checkpoint runs into tens of gigabytes. HF_TOKEN only matters for gated repositories such as Llama, and it does no harm to set it for an ungated model like this one. Pin the image to a released version above vLLM's stated minimum for your model rather than latest once this VM is doing anything you depend on.

Watch the container's own log until the server reports it is ready, and read the line that states the KV cache size and maximum number of sequences it found room for on this specific GPU:

sudo docker logs -f vllm

Connect to it securely

From the machine that will call the API, a laptop or an app server, open a tunnel using the exact user, host and port from the deployment's SSH command:

ssh -N -L 8000:127.0.0.1:8000 ubuntu@YOUR_VM_IP -p 22

While that terminal stays open, the server answers on localhost:8000 on your machine:

curl http://localhost:8000/v1/models

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": "Say hello in five words"}],
)
print(response.choices[0].message.content)

On an app server rather than a laptop, run the same ssh command from a systemd unit or under autossh so the tunnel survives a dropped connection, rather than a terminal someone can close.

Size memory and context

--gpu-memory-utilization controls how much of the GPU vLLM reserves for the model executor, weights included, and it defaults to 0.92 on the current main branch, 0.9 in older releases, so check the version you pinned. --max-model-len bounds how much context a single request can reserve, and lowering it is usually the fastest way to fit more concurrent requests on a GPU that is short on room. My habit is to start it well below the model's stated maximum and raise it only once the KV cache math says there is room to spare. --kv-cache-dtype fp8 halves the cache at some cost to accuracy, where the model and GPU combination supports it. The KV cache explainer has the per-token math for sizing this properly instead of guessing.

If the container exits with an out-of-memory error, lower --max-model-len, lower --gpu-memory-utilization slightly, or move up a GPU class. The troubleshooting section is the next stop for anything this guide does not cover.

Scale across more than one GPU

On a VM with more than one GPU, --tensor-parallel-size splits the model across them:

sudo docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN=$HF_TOKEN \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3.6-35B-A3B \
  --tensor-parallel-size 2

Check whether the GPUs are connected by NVLink before relying on this for a latency-sensitive workload:

nvidia-smi topo -m

The multi-GPU page covers how the interconnect inside one VM affects tensor parallel performance.

Protect the endpoint if it ever leaves localhost

--api-key adds a bearer token requirement to the OpenAI-compatible routes:

sudo docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-openai:latest \
  --model openai/gpt-oss-20b \
  --api-key "$(openssl rand -hex 24)"

A client then sends Authorization: Bearer <the key> with every request. vLLM's own security documentation is worth reading before trusting this alone: the key only protects endpoints under the /v1, /v2, /inference and /cohere prefixes, a few operational endpoints such as /invocations stay open even with a key set, and the documentation's own recommendation is to put a reverse proxy in front of anything that needs to be reachable outside an SSH tunnel. On DeployGPU, the tunnel is the supported path regardless, since app links here are for a browser, not for code, and there is no platform-provided public endpoint for a model you run yourself.

Stop without losing the model

Stopping a DeployGPU instance deletes its disk, so a model pulled into the Hugging Face cache above disappears with it, and the next launch downloads it again. On Verda or Hyperstack, a volume can hold that cache between launches if you format and mount it the first time and just mount it after that, covered in the stop guide along with what else to copy off first. The volumes documentation covers creating and attaching one.