Test a rented GPU before you trust it

The way to run a GPU stress test on Linux is to do three checks in order: read what the driver reports, run NVIDIA's DCGM diagnostics as a health check, and load every GPU with gpu-burn while you watch temperature and errors. This page is about Linux GPU servers, not the Windows gaming tools that a search for the same words also returns, which are a different job. Every check runs on paid time, so I go from the cheapest test to the longest and stop at the first failure. The NVIDIA, DCGM and gpu-burn facts below were checked on 12 November 2026.

Check that the hardware matches the offer

nvidia-smi --query-gpu=index,name,memory.total,driver_version --format=csv
nvidia-smi topo -m
nvidia-smi nvlink -s

Compare the rows with the offer you launched: the GPU name, the count of 1, 2, 4 or 8, the memory and the driver. NVIDIA's nvidia-smi manual says the reported memory can be lower than the nominal size, because ECC takes several percent for parity bits and the driver may reserve a little, so expect a number somewhat under the datasheet. The first five minutes guide explains the topology matrix and which cards have NVLink at all. On a card that does, nvlink -s prints the status of each link: the bandwidth when it is active, Inactive when the link is present but not active, and Sleep when it sleeps. A wrong model or a missing GPU is a reason to stop now, not to test further.

Run a GPU health check with dcgmi diag

NVIDIA DCGM, the Data Center GPU Manager, has documentation, last updated on 17 June 2026, that describes dcgmi diag as a system-level tool to assess readiness before a workload is deployed, and says it is not a comprehensive hardware diagnostic. DCGM supports bare metal and virtualized setups with full passthrough only. DeployGPU does not document how each provider attaches the GPU, so if the service does not start in your VM, move on to gpu-burn. The feature overview lists all diagnostic levels for Tesla products and level 1 only for Titan, Quadro and GeForce, and it does not name the RTX A6000 or RTX 6000 Ada. NVIDIA's requirements are at least 16 GB of host memory and at least as many CPU cores as GPUs. The packages come from NVIDIA's CUDA apt repository, so add its keyring first, with ubuntu2204 or ubuntu2604 on those releases:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
CUDA_VERSION=$(nvidia-smi -q | sed -E -n 's/CUDA Version[ :]+([0-9]+)[.].*/\1/p')
sudo apt-get install --yes --install-recommends datacenter-gpu-manager-4-cuda${CUDA_VERSION}
sudo systemctl --now enable nvidia-dcgm
dcgmi discovery -l

The package name carries the major CUDA version from the nvidia-smi header, and dcgmi discovery -l should list every GPU on the VM. NVIDIA warns that DCGM is tested with its datacenter drivers and that other drivers could mean missing functionality. Adding a repository makes its driver packages visible to apt, so read apt list --upgradeable before any later upgrade, as the NVIDIA drivers guide explains. These are the levels, with the times NVIDIA measured on Hopper systems:

NVIDIA gives no figures for 1 or 2 GPUs, and its feature overview quotes rougher numbers of about 2 minutes for level 2 and about 15 minutes for level 3. Level 1 is cheap and already fails when pages are pending retirement, when a row remap is pending or has failed, or when another graphics process is using the GPU. So I run level 1, then level 2, and go to level 3 only if something looked wrong or the job is long enough to justify it, because NVIDIA's own goals list levels 3 and 4 as post-mortem tools. Put a limit on the run with -t, which takes seconds and has no limit by default, and save the result with -j for JSON:

dcgmi diag -r 1
dcgmi diag -r 2 -t 900 -j > dcgm-level2.json

The level 2 GPU memory test writes test patterns to 75 percent of the memory by default and reads them back, and a mismatch or an uncorrectable double-bit ECC error fails it. The level 3 diagnostic test runs large matrix multiplies, for five minutes by NVIDIA's description and 180 seconds by the test_duration default in its parameter table, while it watches for Xids, temperature violations and uncorrectable memory errors, and it checks the data it writes. The nccl_tests plugin is skipped unless the DCGM_NCCL_TESTS_BIN_PATH variable points at NCCL test binaries. The details go to /var/log/nvidia-dcgm/nvvs.log. Run long levels in tmux, as keeping jobs running shows.

Run a GPU stress test on Linux with gpu-burn

The gpu-burn README calls it a multi-GPU CUDA stress test, and gives a Docker route that builds inside a CUDA image. I add sudo to the Docker commands, because the default user is not in the docker group on every provider:

git clone https://github.com/wilicc/gpu-burn
cd gpu-burn
sudo docker build -t gpu-burn .
sudo docker run --rm --gpus all gpu-burn
sudo docker run --rm --gpus all gpu-burn ./gpu_burn -m 50% 3600

By default the container burns for 60 seconds, and the README's second run command shows how to pass options: the last argument is the run time in seconds, so 3600 is an hour of paid time. The README's usage lists -m X for X MB of memory, -m N% for a share of the available memory, -d for doubles, -tc to try Tensor Cores, -l to list the GPUs and -i N to test only GPU N. The man page and the source say the default is 90 percent of the memory. If --gpus all fails, the Container Toolkit guide covers the fix.

The Dockerfile defaults to CUDA 11.8.0 and compute capability 7.5, and the README says build arguments change both:

sudo docker build --build-arg CUDA_VERSION=13.0.0 --build-arg COMPUTE=75 --build-arg IMAGE_DISTRO=ubi8 -t gpu-burn .

That line is the README's example. Set COMPUTE to the card's compute capability from NVIDIA's CUDA GPU list, without the dot, and choose a CUDA_VERSION tag from Docker Hub that is at or below the CUDA Version in the nvidia-smi header and new enough for the card, since NVIDIA's Blackwell guide says nvcc can generate native code for compute capability 10.0 from CUDA 12.8:

The README does not describe the output, so I read the source. The progress line shows for each GPU the iterations and Gflop/s, an error count that gets a (WARNING!) note when it is not zero or (DIED!) when the process died, and the temperature in C. The run ends with Tested N GPUs: and one line per GPU, OK or FAULTY. FAULTY means the repeated multiplications on that GPU disagreed with each other, and it does not say why. If every GPU dies, the program prints No clients are alive! Aborting. The Gflop/s figures are a way to compare the GPUs of one VM with each other. I publish no benchmark numbers here, because they depend on the card and the clocks, and DCGM can flag a GPU that is far from the mean of the others with diagnostic.gflops_tolerance_pcnt at level 3.

Repeat a matrix multiply in PyTorch

NVIDIA's cuBLAS documentation says its routines give the same bit-wise results at every run on GPUs with the same architecture and number of SMs, as long as a single CUDA stream is active. So the same multiplication repeated on one GPU should return identical numbers, and a difference means the GPU gave two answers to one question:

import torch

for i in range(torch.cuda.device_count()):
    device = torch.device("cuda", i)
    a = torch.randn(8192, 8192, device=device)
    b = torch.randn(8192, 8192, device=device)
    first = a @ b
    mismatches = 0
    for _ in range(200):
        if not torch.equal(first, a @ b):
            mismatches += 1
    torch.cuda.synchronize(device)
    print(i, torch.cuda.get_device_name(device), "mismatches:", mismatches)

PyTorch documents torch.equal as true when two tensors have the same size and elements. A pass proves little, because the test touches a small part of the memory and a few kernels, so I use it as a quick extra beside the two tools above. The PyTorch install guide covers the install.

Watch temperature, power and clocks under load

Start a second pane with the logging command from the monitoring guide, run the burn, and read this while it runs and after it ends:

nvidia-smi -q -d TEMPERATURE,POWER,CLOCK
nvidia-smi -q

The plain -q output includes Clocks Event Reasons, which the manual describes as the factors reducing the frequency of clocks. If all of them read Not Active, NVIDIA says the clocks run as high as possible. These are four of its definitions:

The last two point at the machine and not at your job, so I send them to support with the output.

What failures look like: Xids, ECC and row remapping

sudo dmesg | grep "NVRM: Xid"
nvidia-smi -q -d ECC,PAGE_RETIREMENT,ROW_REMAPPER

NVIDIA's Xid documentation, last updated on 11 November 2026, says Xid messages are driver error reports printed to the kernel log, that they can point at hardware, at NVIDIA software or at your application, and that the value alone often cannot settle which. Its catalog describes the ones I look for:

Xid 154 appears beside other Xids and names the recovery action, from None to GPU Reset Required and Node Reboot Required. The manual says single-bit ECC errors are corrected by the hardware and double-bit errors are not. Volatile counters count since the driver loaded and aggregate counters are a lifetime count, so run the ECC query before the test and after it, and look for a number that grows. On Ampere and newer GPUs a failing memory row is remapped to a spare, a pending remap needs a GPU reset, and the Remapping Failure Occurred flag records a failed remap. Older GPUs retire pages instead. A field that reads N/A is not reported in your environment, and that is different from clean.

What to do on DeployGPU when the GPU is faulty

Do these in order, because stopping deletes the disk. First save the evidence: the gpu-burn output, which you can pipe through tee gpu-burn.log, plus dcgm-level2.json, sudo dmesg and nvidia-smi -q written to files. NVIDIA's sudo nvidia-bug-report.sh collects the kernel logs into nvidia-bug-report.log.gz, and NVIDIA says to allow up to an hour in rare cases. Then copy off what you need, as the transfer guide shows. Then stop the instance from the dashboard or the API, which settles the unused seconds of the current hour to the second, and launch another one, as the stop guide lists. Finally send the files through the contact page with the provider, the deployment ID and the index of the GPU that failed.

I make no promise here about the time you already used. Do not reboot to clear an Xid. NVIDIA's catalog says a GPU reset or a node reboot clears some of them, but a reboot is a risk on DeployGPU: copy off first, never run shutdown inside the VM, and on Verda B200, B300 and RTX PRO 6000 machines do not reboot from inside at all. What QuantaCloud and Massed Compute do during a reboot has not been measured, and the drivers guide has the details. If nvidia-smi itself stops answering, NVIDIA-SMI has failed is the next page. The renting guide covers choosing the next machine.