The quickest way to check GPU usage on Linux is nvidia-smi, which NVIDIA documents as its System Management Interface for monitoring and managing its GPUs. One snapshot is rarely enough on a rented GPU, because the meter runs while you watch. This page covers how to read the table, the options that repeat, filter and log it, watch, nvtop and gpustat as alternatives, how to find the process that holds the memory, and what low utilization usually means. The NVIDIA and tool facts were checked on 12 November 2026.
Check GPU usage on Linux: read the nvidia-smi table
nvidia-smi
nvidia-smi -L
The first command prints one snapshot of every GPU and its processes, and the second lists each GPU with its UUID, which is the quickest way to count the cards on a 2, 4 or 8 GPU VM. The first five minutes guide shows what to expect on a fresh machine. These are the fields I read, with the definitions from NVIDIA's nvidia-smi manual:
| Field | What NVIDIA says it is | What I read it for |
|---|---|---|
| FB memory used and total | On-board frame buffer memory. The driver may reserve a small amount for internal use, even without active work | Whether the card is empty before a job, and how close the job is to the limit |
| GPU utilization | Percent of time over the past sample period during which one or more kernels was executing, with a period of 1 second to 1/6 second depending on the product | Whether the GPU is kept busy |
| Memory utilization | Percent of time over the past sample period during which global (device) memory was being read or written | A second view of load, which is not the same as memory used |
| Performance state | P0 for maximum performance down to P12 for minimum | A card parked in a low state while a job runs |
| Power draw | The last measured power draw for the entire board, in watts | How hard the card is working |
| Processes | PID, type, name and GPU memory, with C for compute, G for graphics, M for MPS and O for other | Who holds the memory |
The default display uses shorter labels such as Memory-Usage, GPU-Util, Perf and Pwr:Usage/Cap, as in the sample in NVIDIA's vGPU monitoring guide. If the command is missing or fails, nvidia-smi: command not found and NVIDIA-SMI has failed are the pages to read.
The nvidia-smi commands I use
| Command | What it does |
|---|---|
nvidia-smi -l 2 | Repeats the snapshot every 2 seconds until Ctrl+C. Without a number the interval is 5 seconds |
nvidia-smi -lms 500 | The same loop in milliseconds |
nvidia-smi -i 0 | Shows only GPU 0, picked by index, UUID or PCI bus ID |
nvidia-smi -q -d MEMORY,UTILIZATION | The long query, limited to the memory and utilization sections |
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total --format=csv | Chosen fields as CSV. The manual marks csv as mandatory, and noheader and nounits are optional |
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv | One row per compute process |
nvidia-smi pmon -s um | Per-process SM utilization and memory use, once a second |
nvidia-smi dmon -s pum -o T | One line per GPU per second with power, temperature, utilization and memory use, prefixed with the time |
nvidia-smi topo -m | The connection matrix between GPUs, covered in the first five minutes guide |
For --query-gpu and --query-compute-apps the manual says to call --help-query-gpu or --help-query-compute-apps for the property names, so I print that list on the VM before I trust a field name. The manual also says dmon and pmon are supported under bare metal 64-bit Linux and print a dash for a metric the device does not report. If a column is all dashes on your VM, use the --query-gpu loop instead. pmon takes a -d interval between 1 and 10 seconds.
watch nvidia-smi or nvidia-smi -l
watch -n 1 nvidia-smi
watch -n 1 'nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv'
nvidia-smi -l 1
The watch manual says it runs a command repeatedly and shows the output full screen, every 2 seconds by default, with -n setting the interval in seconds and -d highlighting what changed between updates. NVIDIA's own IGX Orin guide uses watch -n 0.25 'nvidia-smi --query-gpu=utilization.gpu --format=csv' to watch a GPU. The difference from -l is where the loop lives. watch starts a new nvidia-smi each interval and redraws the screen, while nvidia-smi -l is one process that sleeps between queries, as the manual describes. I use watch -d -n 1 nvidia-smi for a quick look and -l when the output goes to a file.
nvtop and gpustat for a live view
sudo apt install nvtop
nvtop
The nvtop README describes it as an htop-like task monitor for GPUs that handles multiple GPUs, which makes it the Linux GPU monitor to open when you want charts. It gives sudo apt install nvtop for Ubuntu 20.04 and newer, with an AppImage as another route. Press F2 for the setup window, where you choose the charts, and F12 to save them. Two messages from its troubleshooting list matter over SSH. ncurses: cannot initialize terminal type ($TERM="unknown") means $TERM needs a real terminal type such as xterm-256color, and NO GPU to monitor. means libnvml.so from the NVIDIA driver is missing. I keep it in a tmux pane under the job, which keeping jobs running after you disconnect shows how to split.
pip install gpustat
gpustat -cpu -i 2
The gpustat README shows pip install gpustat, so run it inside a virtual environment, or use uv tool install gpustat, which the uv documentation describes as installing a tool into its own environment. It prints one line per GPU with the index, name, temperature, utilization and memory, and -c, -p and -u add the command, PID and owner of each process. -i 2 repeats every 2 seconds and --json prints machine-readable output. PyPI's latest release is 1.1.1 from 22 August 2023 and the README on GitHub already mentions 1.2, so run gpustat --help before you rely on a flag. Both tools read NVIDIA's NVML library, the same source as nvidia-smi.
Find the process that holds GPU memory
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv
ps -o pid,user,etime,args -p PID
sudo docker ps
sudo docker top CONTAINER
sudo lsof /dev/nvidia*
The first command gives PIDs and memory. The ps manual lets -p select a PID and -o choose the columns, so the second shows who started it and how long ago. A model server in Docker looks like any other process, so sudo docker ps lists the containers and sudo docker top lists what one of them runs. On a template deployment the container is deploygpu-workload. NVIDIA's manual adds one case: when every process using the GPU has exited but a forked child still holds an open device handle, nvidia-smi prints Unidentified processes found instead of entries, and the manual says to use lsof on Linux to identify processes using NVIDIA devices.
The usual holders are a Jupyter kernel from an earlier cell, a model server you forgot, and an Ollama model that stays loaded for 5 minutes after its last request. CUDA out of memory has the fixes, and CUDA_VISIBLE_DEVICES explains how to point a job at one card of several.
Log GPU usage to a file during a training run
nvidia-smi --query-gpu=timestamp,index,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu,power.draw --format=csv -l 5 -f gpu-usage.csv
NVIDIA's manual describes -f as redirecting query output to a file, which it overwrites, and its examples pair -l 10 with -f out.log to record an interval indefinitely. One run writes one file, so name it per run. Start it in a second tmux pane before the job, or with nohup as the keeping jobs running guide shows, and press Ctrl+C when the job ends. Use -l 5 for a long run and -lms 500 to catch short spikes. The CSV has one row per GPU per sample, so an 8 GPU VM writes 8 rows every 5 seconds. Stopping the instance deletes the disk, so copy the file off first, as the stop guide lists, with scp ubuntu@YOUR_VM_IP:gpu-usage.csv . from your own machine and the user from your deployment.
What low GPU utilization usually means
NVIDIA defines GPU utilization as the share of time with at least one kernel running, so 100 does not mean every core is busy, and a low number means the GPU waited. These are the causes I check in order:
| What you see | Likely cause | What to check |
|---|---|---|
| Near 0 percent and almost no memory used while the job runs | The job is not on the GPU: a CPU build of PyTorch, or a hidden device | torch.cuda.is_available(), then Torch not compiled with CUDA enabled and echo "$CUDA_VISIBLE_DEVICES" |
| Utilization jumps between 0 and 100 | The GPU waits for data | PyTorch's tuning guide says DataLoader defaults to num_workers=0, which loads data in the main process, and pin_memory=False. Raise the workers, and compare with nproc |
| One GPU busy and the rest idle on a 4 or 8 GPU VM | The job uses one device | Multi-GPU training on one node |
| Memory high and utilization low on a model server | The server holds its weights while idle | vLLM reserves its --gpu-memory-utilization share at startup, and Ollama keeps a model loaded for 5 minutes. See Ollama on a GPU server and tuning vLLM |
| Memory used near the total | The next allocation fails | CUDA out of memory |
| Utilization high but the job is slow | Clocks are reduced | nvidia-smi -q includes Clocks Event Reasons, which the manual describes as the factors reducing the frequency of clocks, and the GPU stress test guide checks temperature and power |
For vLLM, the v0.33.0 optimization guide adds a CPU rule: a deployment with N GPUs needs at least 2 + N physical cores, and if GPU utilization is lower than expected, CPU contention may be the bottleneck.
Frequently asked questions
How do I check GPU usage on Linux?
Run nvidia-smi for a snapshot, nvidia-smi -l 2 to repeat it every 2 seconds, or nvtop for an interactive monitor. Read GPU utilization and memory used, and list the processes with nvidia-smi --query-compute-apps.
What is the difference between watch nvidia-smi and nvidia-smi -l?
watch starts a new nvidia-smi every interval and redraws the screen. nvidia-smi -l is a single process that sleeps between queries, which suits writing output to a file.
How do I log GPU usage to a file?
Run nvidia-smi with --query-gpu and a list of fields, --format=csv, -l with an interval in seconds, and -f with a file name. Start it before the job and stop it with Ctrl+C when the job ends.
Why does nvidia-smi show memory in use when nothing is running?
NVIDIA says the driver may reserve a small amount of memory for internal use, even without active work. Another process may also hold memory, so list compute processes, and remember that an idle model server keeps its weights loaded.
Should I use nvtop or gpustat?
nvtop is a full-screen monitor with charts and per-process rows. gpustat prints one line per GPU and can emit JSON for scripts. Both read NVIDIA's NVML library, the same source nvidia-smi uses.
What does GPU-Util mean in nvidia-smi?
NVIDIA defines it as the percent of time over the past sample period during which one or more kernels was executing on the GPU. It is a measure of busy time, not of how many cores were in use.