Multi-GPU training on a single cloud node

The command that starts multi-GPU training on one rented node is torchrun --standalone --nproc-per-node=gpu train.py. What changes with the model is what runs inside the script: DDP when a full copy of the model and its training state fits on one GPU, and FSDP or DeepSpeed ZeRO when it does not. Everything below runs inside a single DeployGPU VM, which has 1, 2, 4 or 8 GPUs. There is no multi-node and no InfiniBand, so a job that needs more than 8 GPUs goes through the quote form on the clusters page.

Size the node: 18 bytes per parameter before activations

Hugging Face's memory guide, checked 1 October 2026, breaks mixed-precision training with Adam into 6 bytes per parameter for the weights, 8 for the optimizer state and 4 for the gradients, which is 18 in all before activations and temporary tensors. The table is calculated from that rule with an even split across GPUs. It leaves out activations, temporary tensors and the layer that FSDP gathers at full size while it runs, so real use is higher.

DDP puts the whole state on every GPU, so adding GPUs does not make a bigger model fit. FSDP and ZeRO-3 shard it, so they do. LoRA and QLoRA train far fewer parameters, and how much VRAM to fine-tune covers them. These are the families with 2 or more GPUs per VM right now:

Check NVLink with nvidia-smi topo -m

nvidia-smi topo -m

The H100 SXM, A100 SXM, H200 SXM and B200 are NVLink through NVSwitch. The H100 PCIe and A100 PCIe take 2-way bridges and the H200 NVL takes 2-way or 4-way bridges. The RTX PRO 6000, L40, L40S and RTX 6000 Ada have no NVLink, so their GPUs exchange data over PCIe. Training works either way, and how much the link costs you depends on how often the GPUs exchange data, which FSDP does at every layer. Scaling depends on the model, the batch size and the link, so the figure to trust is your own: run 50 steps on one GPU and 50 on all of them, compare samples per second, and only then start the long run. The first five minutes guide explains how to read the matrix.

Launch with torchrun --standalone

torchrun --standalone --nproc-per-node=gpu --tee 3 train_ddp.py

The torchrun documentation, checked 1 October 2026, says --nproc-per-node accepts gpu for one process per GPU, cpu, or an integer, and that each process drives a single GPU. Every process gets LOCAL_RANK, RANK and WORLD_SIZE as environment variables. --tee 3 writes both streams of every worker to log files and echoes them with a prefix such as [default0]:, because by default the output of every rank interleaves on the console.

To use fewer GPUs than the VM has, pass an integer and limit what the processes can see with CUDA_VISIBLE_DEVICES=0,1, as CUDA_VISIBLE_DEVICES on multi-GPU instances explains. --max-restarts=3 restarts the workers when one fails, and the documentation leaves a lost node to the job manager. DeployGPU has no automatic failover, so a lost VM is not restarted for you. Start the command inside tmux, as keeping jobs running after you disconnect shows. My habit is to run a new script on 2 GPUs first and ask for the full node only when that works.

DDP: a full copy of the model on every GPU

This script follows the official DDP tutorial series. It falls back to CPU and gloo, so you can check it on your laptop with two processes before you pay for GPUs:

import os

import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, TensorDataset
from torch.utils.data.distributed import DistributedSampler


def main():
    use_cuda = torch.cuda.is_available()
    local_rank = int(os.environ["LOCAL_RANK"])
    if use_cuda:
        torch.cuda.set_device(local_rank)
    dist.init_process_group(backend="nccl" if use_cuda else "gloo")
    device = torch.device("cuda", local_rank) if use_cuda else torch.device("cpu")

    model = DDP(
        torch.nn.Linear(20, 1).to(device),
        device_ids=[local_rank] if use_cuda else None,
    )
    optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

    dataset = TensorDataset(torch.randn(2048, 20), torch.randn(2048, 1))
    sampler = DistributedSampler(dataset)
    loader = DataLoader(dataset, batch_size=32, sampler=sampler, pin_memory=use_cuda)

    for epoch in range(3):
        sampler.set_epoch(epoch)
        for x, y in loader:
            x, y = x.to(device), y.to(device)
            optimizer.zero_grad()
            loss = torch.nn.functional.mse_loss(model(x), y)
            loss.backward()
            optimizer.step()
        if dist.get_rank() == 0:
            print(f"epoch {epoch} loss {loss.item():.4f}", flush=True)
            torch.save(model.module.state_dict(), "checkpoint.pt")

    dist.destroy_process_group()


if __name__ == "__main__":
    main()

The sampler gives each GPU its own slice of the data, and set_epoch reshuffles it every epoch. The batch size is per GPU, so the effective batch is 32 times the number of GPUs here. Only rank 0 writes the checkpoint, and the DDP wrapper hides the real model behind model.module. Checkpoint and resume turns that save into a resumable run.

FSDP2: shard the weights, gradients and optimizer state

PyTorch's FSDP2 tutorial says FSDP1 is deprecated and that FSDP reduces memory by sharding parameters, gradients and optimizer states. You call fully_shard on each layer first and then on the root model, and build the optimizer afterwards:

from torch.distributed.fsdp import fully_shard

for layer in model.layers:
    fully_shard(layer)
fully_shard(model)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)

Everything else in the loop stays the same as the DDP script. For a Hugging Face model, shard each transformer block the way the tutorial shards model.layers. MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32) is the tutorial's typical setting for bf16 compute with fp32 gradient reduction. Saving needs care, because the state dict is sharded. The tutorial shows a full state dict through DTensor and the torch.distributed.checkpoint APIs, and the checkpoint guide uses them.

DeepSpeed ZeRO: the deepspeed launcher and a config file

DeepSpeed's getting started page says that without a hostfile the launcher queries the local machine for its GPUs, so one command uses the whole VM:

pip install deepspeed
deepspeed train.py --deepspeed --deepspeed_config ds_config.json

{
  "train_micro_batch_size_per_gpu": 4,
  "gradient_accumulation_steps": 4,
  "bf16": { "enabled": true },
  "zero_optimization": { "stage": 2, "contiguous_gradients": true, "overlap_comm": true }
}

The config reference says the effective batch is the micro batch times the accumulation steps times the number of GPUs, and that you may set any two of the three. The ZeRO tutorial says stage 1 partitions the optimizer states, stage 2 adds the gradients and stage 3 adds the model parameters. In your own script, pass the parsed arguments to deepspeed.initialize(args=...), which reads args.deepspeed_config and returns an engine, and the loop calls model_engine(batch), model_engine.backward(loss) and model_engine.step(). The same page says Hugging Face's Trainer takes --deepspeed ds_config.json directly.

Two traps. model_engine.save_checkpoint must run on every process, and DeepSpeed's documentation warns it hangs if only rank 0 calls it. And DeepSpeed compiles its C++ and CUDA ops just in time, which can need the CUDA toolkit on the VM, a part that plain PyTorch wheels do not need. Its installation notes require the toolkit's major version to match the CUDA version of PyTorch, and ds_report lists which ops your machine can build.

NCCL: two variables for diagnosing a hang

NCCL_DEBUG=INFO makes NCCL print debug information at startup, and NVIDIA's NCCL documentation lists it with VERSION, WARN and TRACE. NCCL_P2P_DISABLE=1 turns off direct GPU-to-GPU communication. Use the second as a diagnostic and not a fix: if a hang disappears with it set, the GPU interconnect path is the suspect, and communication will be slower while it is on. There is no InfiniBand on these VMs, so there is nothing to tune on that side.

Six things that go wrong