Fix 'NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver'

The kernel log explains why nvidia-smi says it could not communicate with the NVIDIA driver or says no devices were found, so I read it before I change anything. On a rented VM there are five causes, and I check them in this order: a kernel update that left the module behind, a GPU that has fallen off the bus, a driver that does not fit the GPU, Secure Boot, and nouveau. The NVIDIA, Ubuntu and kernel facts below were checked on 12 November 2026.

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

No devices were found

Both messages come from reports in NVIDIA's own open-gpu-kernel-modules tracker, issue 398 for the first and issue 862 for the second. A third message, Failed to initialize NVML: Driver/library version mismatch, means the driver files and the loaded module differ, and the NVML mismatch page covers it.

Read the kernel log when nvidia-smi says no devices were found

NVIDIA's README says that when the kernel module fails to load you should check the output of dmesg and try modprobe nvidia. NVIDIA's Xid page says Linux logs Xid messages to the kernel log buffer, which distributions usually pass on to the journal, and tells you to search for NVRM: Xid.

uname -r
cat /proc/driver/nvidia/version
lsmod | grep -E 'nvidia|nouveau'
sudo dmesg | grep -i -E 'nvrm|xid|nvidia|nouveau'
sudo journalctl -k | grep -i -E 'nvrm|xid|nvidia'

/proc/driver/nvidia/version shows the driver version only while the driver is loaded, and the journalctl manual says -k shows only kernel messages from the current boot. The table maps what you find to a cause.

A kernel update left the NVIDIA module behind

A new kernel needs a module built or packaged for it. NVIDIA's FAQ says that when kernel components are updated without a reboot, DKMS sometimes fails to rebuild the NVIDIA modules for the new kernel, and that running DKMS by hand and then rebooting is usually enough. The modprobe manual says it looks in the module directory of the running kernel, so a module built for another kernel is invisible. Ubuntu's driver documentation describes precompiled modules and DKMS as two routes that exclude each other, so check which one the VM uses:

uname -r
dkms status
dpkg -l | grep -E 'linux-modules-nvidia|nvidia-dkms'

For DKMS, NVIDIA's Ubuntu steps install the headers of the running kernel first, and the README says dkms autoinstall triggers the rebuild by hand:

sudo apt install linux-headers-$(uname -r)
sudo dkms autoinstall
sudo modprobe nvidia
nvidia-smi

For the precompiled route, Ubuntu says to install the module package for the running kernel when it is missing, with your branch in place of NNN:

sudo apt install linux-modules-nvidia-NNN-server-$(uname -r)

If modprobe prints nothing and nvidia-smi prints its table, nothing more is needed. If the way out is a reboot, as in NVIDIA's FAQ, read the box below first.

The GPU has fallen off the bus

Xid is NVIDIA's name for an error report from its driver, and the introduction says it can point to a hardware, driver or application problem. A report in NVIDIA's tracker shows what Xid 79 looks like in the log:

NVRM: Xid (PCI:0000:2b:00): 79, GPU has fallen off the bus.

NVIDIA's Xid catalog says Xid 79 is logged when the driver tries to reach the GPU over PCI Express and finds it not accessible, often because of hardware failures on the link, and sometimes because of failing GPU hardware or other driver issues. These are the catalog's entries for it and its neighbours:

The bare metal in that action is the provider's host, and you cannot restart it from the guest. NVIDIA's analysis table says the same for suspected hardware problems: contact the hardware vendor. So on a rented VM I do four things in order. First, save the evidence while the VM exists:

nvidia-smi -L
lspci | grep -i nvidia
sudo dmesg > ~/dmesg.txt
sudo nvidia-bug-report.sh

The nvidia-smi manual says -L lists each GPU with its UUID, so compare that count with the offer. The Xid page says to run nvidia-bug-report.sh as root and that it writes nvidia-bug-report.log.gz. Second, copy off your data with the transfer guide, because stopping deletes the disk. Third, stop the instance, which returns the unused seconds of the current hour to your balance. Fourth, email support with what you saved. Do not keep paying for a VM with a missing GPU: billing continues in hourly blocks while it runs, and there is no auto-stop timer.

The driver does not fit the GPU

NVIDIA's open kernel modules README says the proprietary modules support Turing, Ampere, Ada and Hopper, and that Blackwell and later are supported only by the open modules. A kernel log in issue 862 shows the message that results, requires use of the NVIDIA open kernel modules, and the README's list of load failures also names a driver that supports none of the GPUs present. This cause is most likely after you installed or replaced the driver yourself, for example by following an older tutorial.

modinfo nvidia | grep license
cat /proc/driver/nvidia/version
lspci -nn | grep -i nvidia

The README says the open flavor reports Dual MIT/GPL and NVIDIA UNIX Open Kernel Module, and the proprietary flavor reports NVIDIA and NVIDIA UNIX x86_64 Kernel Module. The lspci manual says -nn prints the PCI IDs, which the README's appendix of supported products lists. The fix is the open modules, and the drivers guide has the packages and the reboot rules.

Secure Boot rejects the module

The kernel documentation says that with CONFIG_MODULE_SIG_FORCE or module.sig_enforce=1 the kernel loads only validly signed modules for which it has a key. NVIDIA's README lists an unsigned module on a UEFI Secure Boot system as one way the load fails, and Ubuntu says its DKMS modules are not signed with Canonical's key while its precompiled packages work with Secure Boot. Issue 398 shows the messages:

modprobe: ERROR: could not insert 'nvidia': Key was rejected by service
Loading of unsigned module is rejected

mokutil --sb-state shows the Secure Boot state, per its manual. The lighter fix is Ubuntu's precompiled packages, which the drivers guide explains. The heavier one is to sign the module with a key the kernel trusts, which the README describes.

Nouveau is holding the GPU

Ubuntu's documentation lists No devices were found under troubleshooting and says it may occur when the open source nouveau driver is loaded and conflicts with the NVIDIA driver. lsmod | grep nouveau shows it, and Ubuntu's fix is to block it and rebuild the initramfs, then reboot, which the box above governs:

echo "blacklist nouveau" | sudo tee /etc/modprobe.d/disable-nouveau.conf
echo "options nouveau modeset=0" | sudo tee -a /etc/modprobe.d/disable-nouveau.conf
sudo rmmod nouveau || true
sudo update-initramfs -u

Containers show it differently

A container on a host with no loaded driver fails before nvidia-smi runs, with nvidia-container-cli: initialization error: nvml error: driver not loaded, as in a report in NVIDIA's tracker. Fix the host with the sections above, and the container toolkit guide applies after that. If the command itself is missing, read the command not found page.

When to contact support

Email support@deploygpu.ai for any Xid 79, for a GPU count lower than the offer, and for a fresh VM that shows either message before you changed anything. Send the saved lines, the nvidia-smi -L and lspci output, the bug report, the provider and the deployment ID. The contact page is the other way in, and the stress test guide covers testing a rented GPU before you trust it.