Use CUDA_VISIBLE_DEVICES on multi-GPU instances

The CUDA_VISIBLE_DEVICES variable hides every GPU you do not list and renumbers the ones you keep from zero, and most of the surprises on a 2, 4 or 8 GPU VM follow from those two facts.

RuntimeError: CUDA error: invalid device ordinal
AssertionError: Invalid device id

Both appear when code addresses a GPU that is not visible, such as cuda:1 after CUDA_VISIBLE_DEVICES=1. NVIDIA's CUDA Programming Guide (checked 1 October 2026) says the variable controls which GPUs a CUDA application sees and in what order, and that the device count includes only the visible ones.

List the GPUs with nvidia-smi -L

A DeployGPU VM has 1, 2, 4 or 8 GPUs of one model. Print them with their UUIDs and bus IDs:

nvidia-smi -L
nvidia-smi --query-gpu=index,name,pci.bus_id,uuid --format=csv

Indices are fine for quick work. In anything you keep, use UUIDs, because the nvidia-smi documentation recommends UUID or PCI bus ID when you need consistency, since enumeration order is not guaranteed to be the same between reboots.

Pin a process with CUDA_VISIBLE_DEVICES=2,3

CUDA_VISIBLE_DEVICES=2,3 python3 -c "import torch; print(torch.cuda.device_count(), torch.cuda.get_device_name(0))"

That prints 2 and the card name, and inside the process the two GPUs are cuda:0 and cuda:1. NVIDIA's guide says CUDA_VISIBLE_DEVICES=2,1 hides device 0 and puts device 2 first. An empty value hides every GPU. An invalid entry cuts the list there, so 0,2,-1,1 shows only 0 and 2. A value copied from an 8 GPU VM, such as 6,7, therefore shows nothing on a 2 GPU VM, and torch.cuda.is_available() returns False.

Set the variable in the shell that starts the process, or at the top of the script before any CUDA call. An export stays in force for every later command in that shell, so check what you inherited:

echo "$CUDA_VISIBLE_DEVICES"

Make the order match with CUDA_DEVICE_ORDER

NVIDIA's guide says the default, FASTEST_FIRST, orders GPUs with a speed heuristic, while PCI_BUS_ID orders them by bus ID, ascending, and shows nvidia-smi --query-gpu=name,pci.bus_id for reading the IDs. That is why an index read from nvidia-smi can name a different GPU in your program when the two orders disagree. I set PCI_BUS_ID in anything that pins devices, because its order is defined by bus IDs you can print yourself:

export CUDA_DEVICE_ORDER=PCI_BUS_ID
CUDA_VISIBLE_DEVICES=1 python3 train.py

Run one job per GPU on a 4 GPU VM

Each process sees one GPU, so each one uses cuda:0:

for i in 0 1 2 3; do
  CUDA_VISIBLE_DEVICES=$i python3 evaluate.py --shard $i > "eval_$i.log" 2>&1 &
done
wait

Two jobs pinned to the same GPU share its memory, and that is the quickest way into CUDA out of memory.

Split an 8 GPU VM between two torchrun jobs

PyTorch's torchrun documentation says separate single-node jobs on one host need different ports, and that --rdzv-endpoint=localhost:0 lets torchrun pick a free one:

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --rdzv-backend=c10d --rdzv-endpoint=localhost:0 \
  --nnodes=1 --nproc-per-node=4 train_a.py &
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun --rdzv-backend=c10d --rdzv-endpoint=localhost:0 \
  --nnodes=1 --nproc-per-node=4 train_b.py &
wait

The multi-GPU training guide covers one job across all the GPUs, and the multi-GPU page covers the links between them.

Containers: use --gpus instead

Docker's --gpus flag hides the other GPUs at the container level, and the device list needs the quoted form from its documentation. Two vLLM servers on a 4 GPU VM, two GPUs each, both on localhost for an SSH tunnel:

sudo docker run -d --name vllm-a --gpus '"device=0,1"' --ipc=host -p 127.0.0.1:8000:8000 \
  vllm/vllm-openai:latest --model Qwen/Qwen3.8-27B --tensor-parallel-size 2
sudo docker run -d --name vllm-b --gpus '"device=2,3"' --ipc=host -p 127.0.0.1:8001:8000 \
  vllm/vllm-openai:latest --model openai/gpt-oss-120b --tensor-parallel-size 2

The app templates start their container with --gpus all and the dashboard has no field for environment variables. Inside one, set the variable per process, for example with sudo docker exec -e CUDA_VISIBLE_DEVICES=0 deploygpu-workload python3 job.py, and use a plain GPU VM for container-level pinning. Ollama reads the variable on a server you start yourself, and its GPU documentation says UUIDs are more reliable than numbers.

When to contact support

Contact support@deploygpu.ai if nvidia-smi -L on a fresh VM lists fewer GPUs than the offer says, for example 6 on an 8 GPU offer. Send the deployment ID and the output.