Fix Ollama falling back to the CPU

The only reliable check for Ollama running on the CPU is the PROCESSOR column of ollama ps, because slow answers have other causes, and on a rented GPU the first suspect is a model that does not fully fit.

NAME        ID            SIZE    PROCESSOR   UNTIL
llama3:70b  bcfb190ca3a7  42 GB   100% GPU    4 minutes from now

That sample comes from Ollama's FAQ, checked 1 October 2026. PROCESSOR reads 100% GPU when the model sits entirely on the GPU, 100% CPU when it sits in system memory, and a split such as 48%/52% CPU/GPU when it is on both. ollama ps lists only loaded models, so send a short prompt first. On the Open WebUI template, Ollama runs inside the deploygpu-workload container:

sudo docker exec deploygpu-workload ollama ps

Run nvidia-smi on the VM and in the container

If nvidia-smi fails on the VM, Ollama is not the problem. A driver error such as Driver/library version mismatch has its own page, NVML driver/library version mismatch. On the template, run it inside the container too:

nvidia-smi
sudo docker exec deploygpu-workload nvidia-smi
watch -n 1 nvidia-smi

The last command refreshes every second while a reply generates, so you can see the GPU working. The template container is started with --gpus all, and DeployGPU's startup script only starts it after Docker can see the GPU. A missing GPU inside it is unlikely, so the next section is where I would look first.

A split in ollama ps means the model does not fit

When a model does not fit, Ollama splits it between GPU and CPU instead of failing, which works but is much slower. On a VM with several GPUs, the FAQ says Ollama loads a model on one GPU if it fits there and spreads it across all of them if not. Compare the memory in use with the total:

nvidia-smi --query-gpu=memory.used,memory.total --format=csv

A model needs at least the download size Ollama's library lists for its tag, plus room for the KV cache, which grows with context. The fixes are a smaller tag, a lower quantization, a shorter context, or a bigger GPU. The Open WebUI + Ollama template page lists which GPU fits which tags, and the KV cache explainer covers context.

Check ollama ps for a second model or a large context

Ollama keeps a model loaded for 5 minutes by default and can hold several at once, so a model you stopped chatting with may still hold GPU memory. ollama ps lists them, and ollama stop unloads one:

sudo docker exec deploygpu-workload ollama stop MODEL_NAME

The FAQ gives the default context window as 4096 tokens, and parallel requests multiply the context by their number. A context length set in Open WebUI's Chat Controls is the one it sends to Ollama, according to its chat parameters documentation (checked 1 October 2026), so a high value there can push a model off the GPU. On a plain VM you can set OLLAMA_CONTEXT_LENGTH on the server. The template container's environment is fixed by DeployGPU, so there you lower it in Open WebUI.

Your own Ollama container needs --gpus=all

On a plain GPU VM, the first suspects are a missing --gpus flag and a missing NVIDIA Container Toolkit. Ollama's troubleshooting guide checks the runtime with this command, and if it does not print the GPU table, Ollama cannot see the GPU either:

sudo docker run --rm --gpus all ubuntu nvidia-smi

Ollama's Docker instructions use --gpus=all. I bind the port to localhost, because DeployGPU does not firewall VMs and anything bound to 0.0.0.0 is on the internet:

sudo docker run -d --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama

The GPU containers guide covers the toolkit setup. Ollama also honors CUDA_VISIBLE_DEVICES, and its GPU documentation says an invalid value such as -1 disables the GPU, so on a VM with 2 or more GPUs check what your shell exports:

echo "$CUDA_VISIBLE_DEVICES"

The CUDA_VISIBLE_DEVICES page explains the numbering.

Read docker logs for GPU errors

The Ollama server log records what it found. On the template, the server runs inside the same container, so sudo docker logs deploygpu-workload shows it. For your own container use sudo docker logs ollama, and for a native install journalctl -u ollama --no-pager --follow --pager-end. Setting OLLAMA_DEBUG=1 adds detail on a server you start yourself. Ollama's troubleshooting guide lists GPU initialization error codes 3, 46, 100 and 999, with steps such as updating the driver and checking sudo dmesg | grep -i nvidia.

When to contact support

Contact support@deploygpu.ai if nvidia-smi on a fresh VM shows no GPU or fewer GPUs than the offer lists. On the template, write in if nvidia-smi lists the GPU inside the container but a small model still shows 100% CPU. Send the deployment ID and the output of the commands above.