Fix the NVML driver/library version mismatch on a GPU VM

The error means the NVIDIA driver files on the VM's disk and the driver loaded in its kernel are different versions, and on a rented VM the usual cause is a package upgrade that replaced the files while the old kernel module kept running.

Failed to initialize NVML: Driver/library version mismatch

Confirm that the two versions differ

NVIDIA's driver guide says /proc/driver/nvidia/version shows the version of the driver that is loaded. Compare it with the library and the packages on disk:

cat /proc/driver/nvidia/version
readlink -f "$(ldconfig -p | awk '/libnvidia-ml.so.1/ {print $NF; exit}')"
dpkg -l | grep -i nvidia

The first line is the running kernel module. The second prints the NVML library file, which NVIDIA's README lists with the driver version at the end of its name, and the third lists installed package versions. If the first differs from the others, you have found it.

A package upgrade replaced the driver while the old module kept running

Ubuntu installs unattended-upgrades by default, and it applies security updates once a day without being asked, according to Ubuntu's automatic updates documentation. NVIDIA's Ubuntu steps say a normal package update upgrades the driver, and its README says the new driver is loaded after a reboot, or when the old kernel modules are unloaded and loaded again. grep -i nvidia /var/log/unattended-upgrades/*.log shows whether an upgrade touched an NVIDIA package. To reload without a reboot, stop everything that uses the GPU first:

sudo docker ps -q | xargs -r sudo docker stop
sudo systemctl stop nvidia-persistenced
sudo fuser -v /dev/nvidia*
lsmod | grep nvidia
sudo rmmod nvidia_uvm
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia
sudo modprobe nvidia
nvidia-smi

fuser -v lists any process still holding the devices, and nothing should be listed before rmmod runs. Run rmmod once for each module that lsmod shows, dependents first and nvidia last, and skip the ones it does not show, because the rmmod manual says it exits at once when a module is not found. Read each result, because a failed removal means something still holds that module. If the VM runs the NVIDIA Fabric Manager service, NVIDIA's Fabric Manager guide lists sudo systemctl stop nvidia-fabricmanager as the way to stop it. If nvidia-smi prints its table, start nvidia-persistenced and your containers again. If a holder will not let go, use the reboot below or relaunch.

Elsewhere, sudo reboot loads the new module. For the Verda machines named above, shut down, wait until /power reports stopped, then start. Full compute billing continues while the VM is off, and SSH needs a few more minutes after the start, so a refused connection at first is normal, as the connection refused page explains. The API reference has the details:

API=https://api.deploygpu.ai/api/v1/deployments/DEPLOYMENT_ID
curl -s -X POST "$API/shutdown" -H "X-API-Key: $DEPLOYGPU_API_KEY"
until curl -s "$API/power" -H "X-API-Key: $DEPLOYGPU_API_KEY" | grep -Eq '"power_state" *: *"stopped"'; do sleep 10; done
curl -s -X POST "$API/start" -H "X-API-Key: $DEPLOYGPU_API_KEY"

If nothing on the VM is worth keeping, the cleanest fix is to Stop it and launch a new one, which starts from a matching image and refunds the unused seconds. The new machine gets a new address, so the host key page may apply.

To stop it recurring, hold the driver packages, which Ubuntu's apt-mark manual says prevents them from being automatically upgraded:

dpkg -l | awk '/^ii/ && /nvidia/ {print $2}' | sudo xargs apt-mark hold

The other route is Unattended-Upgrade::Package-Blacklist in /etc/apt/apt.conf.d/50unattended-upgrades. Its entries are regular expressions matched from the start of the package name, and Ubuntu warns that blocking a package can hold back updates that depend on it.

CUDA driver version is insufficient for CUDA runtime version

This is a different cause. NVIDIA's runtime API reference says the installed driver is older than the CUDA runtime library, and PyTorch reports it as The NVIDIA driver on your system is too old. A container, toolkit or wheel is newer than the driver. NVIDIA's compatibility guide puts the minimum at driver 525 for CUDA 12.x and 580 for CUDA 13.x, and the CUDA release notes list 570.26 or newer for CUDA 12.8, the build PyTorch's 2.7 notes added for Blackwell. Pick an image tag at or below the CUDA version in the nvidia-smi header, as the toolkit guide explains, or a PyTorch build from the install guide.

Two errors that look similar

If nvidia-smi says it could not communicate with the driver, no module is loaded, which NVML reports as driver not loaded. lsmod | grep nvidia prints nothing then. NVIDIA's FAQ says DKMS sometimes fails to rebuild the module after a kernel update, and the fix is to run it by hand and reboot, with the caveats above. Inside a container, Failed to initialize NVML: Unknown Error means it lost GPU access, and the toolkit guide covers that.

When to contact support

Email support@deploygpu.ai if the loaded and installed versions match and nvidia-smi still fails, if a fresh deployment shows no GPU, or if a Verda machine does not return after the power cycle. Send the nvidia-smi output, the three version commands, the provider and the deployment ID. The contact page is the other way in, and the first five minutes guide lists the checks for a new VM.