Checkpoint and resume training on cloud GPUs

The pattern that survives every way an instance can end is short: write a complete checkpoint to the local disk every few minutes, copy it off the machine as soon as it is complete, and start every run by looking for the newest checkpoint. On DeployGPU an instance ends in three ways, and none of them waits for your job, so the copy is the part that matters.

Three ways an instance ends, and what each leaves you

No redeploy follows any of them. DeployGPU has no failover, so you launch a new instance, which has a new IP address and an empty disk, and resume there. A volume on Verda or Hyperstack survives Stop, but it is deleted permanently, with no email, when your balance cannot pay for it, so treat a volume as a convenience and an off-machine copy as the real one. The dashboard shows Estimated time left on the home page, and it is the cheapest warning you get before a long run.

What a resumable checkpoint contains

PyTorch's tutorial on saving and loading, checked 1 October 2026, says a checkpoint for resuming training needs the optimizer's state_dict as well as the model's, plus items such as the epoch, and that such a checkpoint is often 2 to 3 times larger than the model alone. I also save the scheduler state and the RNG state, because a run that restarts its learning rate schedule is not the same run. By that rule, a model whose weights are 16 GB in bf16, which is 8 billion parameters at 2 bytes each, gives 32 to 48 GB per checkpoint. That is calculated and leaves out your data and environment, and two checkpoints take 64 to 96 GB of a disk that is fixed on QuantaCloud, Massed Compute and Hyperstack. Keep only the last two.

Save atomically and resume from the newest file

Write to a temporary name and rename it. The Python documentation says os.replace is an atomic rename when it succeeds, so a copy job that runs mid-save never sees half a file:

import glob
import os

import torch


def save_checkpoint(path, model, optimizer, scheduler, step):
    tmp = path + ".tmp"
    torch.save(
        {
            "step": step,
            "model": model.state_dict(),
            "optimizer": optimizer.state_dict(),
            "scheduler": scheduler.state_dict(),
            "torch_rng": torch.get_rng_state(),
        },
        tmp,
    )
    os.replace(tmp, path)


def load_checkpoint(path, model, optimizer, scheduler):
    ckpt = torch.load(path, map_location="cpu", weights_only=True)
    model.load_state_dict(ckpt["model"])
    optimizer.load_state_dict(ckpt["optimizer"])
    scheduler.load_state_dict(ckpt["scheduler"])
    torch.set_rng_state(ckpt["torch_rng"])
    return ckpt["step"]


def latest_checkpoint(directory):
    files = sorted(glob.glob(os.path.join(directory, "ckpt-*.pt")))
    return files[-1] if files else None

Every run starts by asking for the newest file, and the saved step is the next one to run:

os.makedirs("ckpt", exist_ok=True)
start = 0
latest = latest_checkpoint("ckpt")
if latest:
    start = load_checkpoint(latest, model, optimizer, scheduler)

for step in range(start, total_steps):
    ...  # forward, backward, optimizer.step(), scheduler.step()
    if (step + 1) % 500 == 0:
        save_checkpoint(f"ckpt/ckpt-{step + 1:08d}.pt", model, optimizer, scheduler, step + 1)

Choose the interval by what you can afford to lose. If a checkpoint lands every 30 minutes, a lost machine costs up to 30 minutes of GPU time plus the relaunch. Time one save first, because a save that takes minutes is a cost at every interval.

Copy every checkpoint off the machine

Pulling from your own machine keeps credentials off the VM, which is shared infrastructure, and the loop skips the temporary files:

while true; do
  rsync -a --exclude '*.tmp' -e "ssh -p 22" ubuntu@YOUR_VM_IP:~/ckpt/ ./ckpt/
  sleep 600
done

To push from the VM instead, hf upload your-name/run-1 ./ckpt . --private --exclude "*.tmp" --every 10 commits the folder every 10 minutes, per the Hugging Face CLI documentation, and needs a token with write access to that one repository. Run it in tmux, as keeping jobs running after you disconnect explains. The transfer guide covers rclone, and the volumes documentation covers creating a volume.

Resume on a new instance in four steps

1. Launch a new instance. Check the SSH user for its provider, because the IP address and the host key are new.
2. Rebuild the environment from the commands you kept, as the PyTorch install guide shows.
3. Copy the newest checkpoint back with rsync, rclone or hf download.
4. Start the same command. The script finds the checkpoint and carries on.

A full state dict saved from one rank loads on any number of GPUs, but the effective batch size changes with the count, so adjust the per-GPU batch or the gradient accumulation to keep it constant. Stock changes by the minute, so the replacement may well have a different number of GPUs.

Hugging Face Trainer: save_steps, save_total_limit and the Hub

from transformers import Trainer, TrainingArguments

args = TrainingArguments(
    output_dir="out",
    save_strategy="steps",
    save_steps=500,
    save_total_limit=2,
    push_to_hub=True,
    hub_model_id="your-name/run-1",
    hub_strategy="checkpoint",
    hub_private_repo=True,
)
trainer = Trainer(model=model, args=args, train_dataset=train_ds)
trainer.train(resume_from_checkpoint=resume)

The Trainer documentation says save_steps defaults to 500, save_total_limit deletes older checkpoints in output_dir, and hub_strategy="checkpoint" pushes each save and also the latest checkpoint to a last-checkpoint subfolder for easy resuming. Passing True as resume_from_checkpoint loads the last checkpoint in output_dir. On a fresh VM, that folder is empty, so download the subfolder first and pass its path:

hf download your-name/run-1 --include "last-checkpoint/*" --local-dir out

Then set resume to "out/last-checkpoint", or to None when nothing has been saved yet.

FSDP and DeepSpeed: sharded checkpoints

With FSDP, no rank holds the whole model, so save with torch.distributed.checkpoint. PyTorch's documentation says it saves from many ranks in parallel and handles load-time resharding, so a checkpoint written under one cluster topology loads under another:

import torch.distributed.checkpoint as dcp
from torch.distributed.checkpoint.state_dict import get_state_dict, set_state_dict

model_sd, optim_sd = get_state_dict(model, optimizer)
dcp.save({"model": model_sd, "optim": optim_sd, "step": step}, checkpoint_id="ckpt/step-500")

# On resume, build the model and optimizer first, then load into their state dicts
model_sd, optim_sd = get_state_dict(model, optimizer)
state = {"model": model_sd, "optim": optim_sd, "step": 0}
dcp.load(state, checkpoint_id="ckpt/step-500")
set_state_dict(model, optimizer, model_state_dict=state["model"], optim_state_dict=state["optim"])

Every rank calls both functions. In DeepSpeed, model_engine.save_checkpoint also has to run on every rank, as the multi-GPU guide explains.

Five ways a resume goes wrong