Run GPU containers on a cloud VM

The NVIDIA Container Toolkit is the piece that lets Docker hand a GPU to a container, and on a DeployGPU VM the quickest way to learn whether it is set up is to run one command and read what comes back. This guide covers that check, NVIDIA's install steps for when it fails, the --gpus flag, choosing a CUDA image your driver supports, a Compose file with a GPU, and the one networking rule that matters on a VM nobody firewalls for you.

Three commands tell you whether Docker can see the GPU

The plain GPU VM is the provider's Ubuntu image with NVIDIA drivers and Docker. Whether the toolkit is configured on top of that is not documented per provider, so check instead of assuming:

nvidia-smi
nvidia-ctk --version
sudo docker run --rm --gpus all ubuntu nvidia-smi

The first prints the driver and the GPUs. The second prints the toolkit version, and a command not found error means the toolkit is missing. The third is the example from Docker's GPU access documentation and NVIDIA's sample workload. If the container prints the same table as the host, Docker can see the GPU.

If it fails with could not select device driver "" with capabilities: [[gpu]], Docker has no NVIDIA driver registered. Docker's source registers one only when it finds a toolkit hook program on its path as the daemon starts, so a missing toolkit and a Docker daemon that was not restarted after the install look identical. App templates run a version of this check themselves: the startup script (public at api.deploygpu.ai/bootstrap.sh) starts a --gpus all container that runs nvidia-smi and gives up after 30 attempts, five seconds apart, so a template that reaches Running has already passed it.

Install the toolkit when the check fails

These are NVIDIA's steps for Ubuntu and Debian from the install guide, checked on 1 October 2026:

sudo apt-get update && sudo apt-get install -y --no-install-recommends \
   ca-certificates \
   curl \
   gnupg2

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
    nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

NVIDIA pins a version in that guide, 1.20.1-1 on 1 October 2026, so copy the current one from the page. The nvidia-ctk runtime configure step edits /etc/docker/daemon.json, and the restart is what makes Docker pick the toolkit up. Docker's live restore documentation says the daemon shuts down running containers when it terminates by default, so do this before you start your own containers.

The --gpus flag and choosing GPUs

--gpus all gives the container every GPU on the VM. On a 2, 4 or 8 GPU VM you can hand out some of them instead, by index or by UUID, which nvidia-smi -L lists. The comma form needs the odd quoting, because Docker's documentation notes the value contains a comma:

sudo docker run --rm --gpus device=0 ubuntu nvidia-smi
sudo docker run --rm --gpus '"device=0,2"' ubuntu nvidia-smi

That is how you run one container per GPU on a multi-GPU VM, for example one transcription worker on each card. The multi-GPU page covers what the VM sizes look like.

Pick a CUDA image tag your driver supports

Read the CUDA version in the top right of the nvidia-smi header. NVIDIA's nvidia-smi documentation calls it the CUDA UMD Version and defines it as the latest CUDA version the driver supports. Pick an image tag at or below it. Official CUDA images set a requirement on the CUDA version, and NVIDIA's Docker documentation says that when the driver is too old to run that version, the container does not start. The refusal reads like this, with the version the image wants after cuda>=:

requirement error: unsatisfied condition: cuda>=13.4, please update your driver to a newer version, or use an earlier cuda container

These tags exist on Docker Hub as of 1 October 2026:

Docker Hub describes three flavors: base has the CUDA runtime, runtime adds the math libraries and NCCL, and devel adds headers and build tools. There is no latest tag, so a bare sudo docker pull nvidia/cuda fails with a manifest unknown error. NVIDIA's compatibility guide gives the floors behind the rule: CUDA 13.x needs driver 580 or newer, and CUDA 12.x needs 525 or newer, with minor version compatibility inside each family. Matching the header is the conservative way to stay clear of all of it.

Docker Compose with a GPU

Compose reserves a GPU through a device entry. Docker's Compose GPU documentation says the capabilities field is required and that count and device_ids cannot be combined:

services:
  gpu-check:
    image: nvidia/cuda:12.2.2-base-ubuntu22.04
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Run it with sudo docker compose up. If sudo docker compose version fails, the Compose plugin is missing, and Docker's Compose install page covers adding it. For a server, add a ports entry that binds to loopback, as in the next section.

Publish ports on 127.0.0.1 only

DeployGPU does not firewall VMs, and Hyperstack VMs are created with inbound traffic open. Docker's port publishing documentation says publishing is insecure by default, because the port becomes reachable from the outside world, and that including 127.0.0.1 in the flag limits it to the Docker host. Its firewall documentation adds that traffic to published ports is diverted before it reaches the rules ufw manages. So ufw does not protect a published port, and the 127.0.0.1 binding is the control:

sudo docker run -d --name server --gpus all -p 127.0.0.1:8000:8000 YOUR_IMAGE
ssh -N -L 8000:127.0.0.1:8000 ubuntu@YOUR_VM_IP -p 22

In a Compose file the same line is - "127.0.0.1:8000:8000" under ports, and the Compose file reference warns that a port with no host IP binds to all interfaces. The user in the tunnel command is ubuntu on QuantaCloud and Hyperstack, Ubuntu with a capital U on Massed Compute and root on Verda, so copy the exact command from the deployment. The Docker page also says that releases older than 28.0.0 let hosts on the same layer 2 segment reach ports published to localhost, so check sudo docker version. The vLLM guide is a full example of a server run this way.

Errors you will meet

NVIDIA's page says a container can lose GPU access when systemctl daemon-reload runs on a system where systemd manages the container's cgroups, and lists setting "exec-opts": ["native.cgroupdriver=cgroupfs"] in /etc/docker/daemon.json as a workaround. Deleting the container on a template deployment deletes what is inside deploygpu-workload, so copy anything you need out with sudo docker cp first. The first five minutes guide has the rest of the checks to run on a new VM, and the GPU VM page covers the image and the login users.