Run Ollama on a cloud GPU server

The way to run an Ollama server on a rented GPU is the Linux install script, which sets up a systemd service that listens on 127.0.0.1:11434, plus an SSH tunnel from your own machine to reach it. This guide covers that route, the Docker and Compose route, the environment variables that matter on a server, the native and OpenAI-compatible APIs, several GPUs and updating. Ollama's documentation and source were checked on 12 November 2026 against release 0.40.16, published on 9 November 2026.

Pick this guide or the Open WebUI template

The Open WebUI + Ollama template is a chat app in a browser. Ollama runs inside the same container and its API port is not exposed, and the private app link does not accept API keys, so code cannot call it. This guide is for the other case: a script, an OpenAI client or a coding agent such as OpenCode with a self-hosted model that needs the Ollama API. For many users at once, vLLM vs Ollama explains where vLLM fits better, llama.cpp vs Ollama covers the engine underneath, and Ollama Cloud vs self-hosted covers the hosted alternative.

Install Ollama on Ubuntu with the install script

nvidia-smi
sudo apt-get install -y zstd
curl -fsSL https://ollama.com/install.sh | sh
ollama -v
sudo systemctl status ollama

Ollama's hardware support page says it supports NVIDIA GPUs with compute capability 5.0 or higher and driver version 550 or newer, so read the driver from nvidia-smi first. Its table names the RTX A6000, L40, A100, H100, H200 and RTX PRO 6000 Blackwell, and it has no row for the B200. The script itself asks for zstd when it is missing, which is why the second line is there. The Linux documentation gives the install line and ollama -v as the check, and the script does the rest. It creates an ollama user, writes /etc/systemd/system/ollama.service with ollama serve as its command, starts it, and ends with a message that the API is available at 127.0.0.1:11434. When nvidia-smi is present it reports an NVIDIA GPU and leaves the driver alone.

That is the answer to ollama serve on a server: the service already runs it. The quickstart says to start the server by hand only if it is not already running, and two servers cannot share one port. Test the install, and read the PROCESSOR column to see whether the model sits on the GPU:

ollama pull gpt-oss:20b
ollama run gpt-oss:20b "Say hello in one sentence."
ollama ps

The Ollama library lists gpt-oss:20b at 14GB. If ollama ps shows CPU, Ollama not using the GPU has the checks. Logs are in journalctl -u ollama --no-pager --follow --pager-end.

Set Ollama server environment variables under systemd

The FAQ gives the method for a systemd service: run systemctl edit ollama.service, add Environment lines under [Service], then reload systemd and restart.

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_CONTEXT_LENGTH=32768"

sudo systemctl daemon-reload
sudo systemctl restart ollama
sudo journalctl -u ollama --no-pager | grep "server config"

The server logs the values it read when it starts, so the last line shows whether a setting took effect. These are the variables that matter on a server, with defaults from the FAQ and from envconfig/config.go in the repository:

Context length, parallel requests and keep-alive

The default context is the first setting to check on a big card. The context length page says the default is 4k tokens below 24 GiB of VRAM, 32k from 24 to 48 GiB and 256k from 48 GiB up. The server source (server/routes.go) adds up the VRAM of all GPUs, cuts at 23 and 47 GiB, and logs the result as vram-based default context. A 48 GB card sits at the edge of the top tier, so read that log line instead of guessing. The FAQ still says the default is 4096, which matches neither that page nor the code, so I set OLLAMA_CONTEXT_LENGTH myself and read the allocated value from the CONTEXT column of ollama ps. The same page recommends at least 64000 tokens for agents and coding tools.

Parallel requests multiply the context. The FAQ's own example is a 2K context with 4 parallel requests, which becomes 8K and needs more memory. By the same arithmetic, OLLAMA_NUM_PARALLEL=4 with OLLAMA_CONTEXT_LENGTH=32768 allocates cache for 131,072 tokens (calculated), and the KV cache explainer shows what a token costs for your model. OLLAMA_KV_CACHE_TYPE=q8_0 uses about half the memory of f16, according to the FAQ. Per request, the API takes num_ctx under options, and ollama run takes /set parameter num_ctx.

Keep-alive decides how long a model holds GPU memory after you stop asking. With the default 5 minutes, ollama ps still lists the model and nvidia-smi still shows its memory used. ollama stop MODEL unloads one immediately, and a keep_alive of -1 in a request keeps it loaded.

Reach the Ollama API through an SSH tunnel

ssh -N -L 11434:127.0.0.1:11434 ubuntu@YOUR_VM_IP -p 22

Use the user, host and port from the SSH command on your deployment. The user is ubuntu on QuantaCloud and Hyperstack, Ubuntu with a capital U on Massed Compute and root on Verda. If your own machine already runs Ollama, publish another local port with -L 11435:127.0.0.1:11434, and the port forwarding guide covers keepalives. With the tunnel open, the commands from Ollama's quickstart work on localhost. The native API lives under /api, and the API introduction gives http://localhost:11434/api as its local base URL:

curl http://localhost:11434/api/tags
curl http://localhost:11434/api/ps
curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-oss:20b", "messages": [{"role": "user", "content": "Say hello in one sentence."}], "stream": false}'

/api/tags lists the models on disk and /api/ps the ones loaded. The OpenAI-compatible API sits under /v1, and the compatibility page says Ollama supports a subset of the OpenAI API, and that the client needs an API key value that Ollama ignores:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
reply = client.chat.completions.create(
    model="gpt-oss:20b",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(reply.choices[0].message.content)

Run Ollama in Docker and Docker Compose with a GPU

sudo docker run -d --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama:0.40.16
sudo docker exec ollama ollama pull gpt-oss:20b
sudo docker logs ollama

This is the command from Ollama's Docker documentation with three changes: sudo, a pinned tag that exists on Docker Hub, and 127.0.0.1: in front of the port. Docker's Compose reference says that without a host IP, Docker binds to all interfaces and bypasses host firewall rules, so the prefix is the control. The GPU flag needs the NVIDIA Container Toolkit, and the toolkit guide has the check and the install. Ollama's troubleshooting page says that if the container starts on the GPU and later falls back to CPU, setting "exec-opts": ["native.cgroupdriver=cgroupfs"] in /etc/docker/daemon.json is the documented fix.

The same container as a Compose file, using the GPU reservation syntax from Docker's GPU documentation:

services:
  ollama:
    image: ollama/ollama:0.40.16
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=30m
      - OLLAMA_CONTEXT_LENGTH=32768
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama:

Start it with sudo docker compose up -d, pull with sudo docker compose exec ollama ollama pull gpt-oss:20b, and read the log with sudo docker compose logs ollama. Docker says capabilities is required, and count and device_ids cannot be combined. The environment lines in the container are the same variables as in the table above.

Use multiple GPUs with Ollama

The FAQ describes the placement rule. When Ollama loads a model, it compares the VRAM the model needs with what is free. If the model fits on any single GPU, it loads there, which the FAQ says usually performs best because less data crosses the PCI bus. If it does not fit on one GPU, the model is spread across all the GPUs. OLLAMA_SCHED_SPREAD always spreads, and OLLAMA_MAX_LOADED_MODELS defaults to 3 times the number of GPUs, so a model that fits on one card leaves the others free for more models. To limit Ollama to some GPUs, set CUDA_VISIBLE_DEVICES. The GPU page says numeric IDs can change order and UUIDs are more reliable, and nvidia-smi -L prints them:

nvidia-smi -L

[Service]
Environment="CUDA_VISIBLE_DEVICES=GPU-REPLACE-WITH-A-UUID"

An invalid value such as -1 forces CPU, so check what your shell exports, and see CUDA_VISIBLE_DEVICES. The multi-GPU page shows the VM sizes. For many concurrent users spread over several GPUs, a server built for tensor parallelism is the better fit, as the comparison pages explain.

Keep models, update Ollama and know what stopping deletes

Stopping an instance deletes its disk, and the service keeps models on that disk, so every launch pulls them again. On Verda and Hyperstack a volume can hold them if you format and mount it first, as the stop guide shows. Then point Ollama at it and give the ollama user the directory, as the FAQ says:

sudo mkdir -p /mnt/data/ollama
sudo chown -R ollama:ollama /mnt/data/ollama

[Service]
Environment="OLLAMA_MODELS=/mnt/data/ollama"

To update on Linux, run the install script again. The Linux documentation also shows how to pin a release with OLLAMA_VERSION, and the script restarts the service at the end, so loaded models are unloaded:

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.40.16 sh

In Docker, change the image tag and run sudo docker compose up -d, or pull the new tag and recreate the container, and the models stay in the named volume. Do not run shutdown inside the VM when you are done. Stop it from the dashboard or the API, after you copy off anything you need.