Run Gemma 4 on a cloud GPU

The four Gemma 4 sizes all carry the same Apache-2.0 license, and the one to rent depends only on which of them fits the GPU class in front of you. E4B and 12B fit a 48 GB card with room to spare. The 26B-A4B mixture-of-experts model and the 31B dense model both need an 80 GB GPU in their native BF16 format.

Google's own memory table

Google publishes a weights-only memory table for Gemma 4, and it is worth citing directly because it is one of the few vendor tables in this space that says exactly what it does and does not cover. In BF16, SFP8 and Q4_0, the official table lists E4B at 17.9, 8.9 and 4.5 GB, 12B at 26.7, 13.4 and 6.7 GB, 26B-A4B at 57.7, 28.8 and 14.4 GB, and 31B at 69.9, 34.9 and 17.5 GB. Google's own disclaimer explains that the table accounts only for loading the static model weights, and excludes the additional VRAM needed for supporting software or the context window. That is the same distinction this page makes everywhere else: weights are the floor, and the KV cache is what you add on top.

The actual checkpoints

Summing the published safetensors files gives slightly different numbers from Google's rounded table, because a real checkpoint includes more than a clean parameter count times two bytes. E4B comes to 16.0 GB, 12B to 23.9 GB, 26B-A4B to 51.6 GB, and 31B to 62.5 GB, all in BF16. Both sets of numbers are weights only, and both point at the same GPU classes: E4B and 12B on a 48 GB card, 26B-A4B and 31B on an 80 GB card. My starting point for a general assistant is 12B on the 48 GB card, and I only move up once a real workload shows it falling short.

Sliding windows, global layers and K equals V

Most Gemma 4 layers use a sliding attention window rather than full attention: 50 of 60 layers in the 31B model, 25 of 30 in 26B-A4B, and 40 of 48 in 12B. A sliding layer's cache stops growing once the window fills, so only the small number of global layers cost memory that scales with the whole conversation. The 31B model's 10 global layers cost 80 KiB per token, calculated from their 4 KV heads and 512-dimension head size, which comes to 2.5 GiB for a 32,000-token conversation. 26B-A4B's 5 global layers cost 20 KiB per token, and 12B's 8 global layers cost 16 KiB per token. The sliding layers themselves add a fixed amount that never grows past it: about 800 MiB for the 31B model, 200 MiB for 26B-A4B and 320 MiB for 12B, however long the chat runs.

One more detail is specific to this family: Gemma 4's global layers are configured with the key and value projections equal to each other, so an engine that recognizes this only needs to store one of them instead of both, halving the global-layer figures above. Not every serving engine has a code path for that yet, so treat the numbers as an upper bound rather than a promise.

The mixture-of-experts size: 26B-A4B

26B-A4B activates 4 billion parameters per token, and the model card says it runs almost as fast as a 4B-parameter model. Memory does not follow that shortcut. Every expert has to sit in GPU memory whether a given token routes to it or not, so 26B-A4B needs close to what the 31B dense model needs, 57.7 GB against 69.9 GB by Google's own table, not the few gigabytes a real 4B model would take.

Serve it with vLLM or Ollama

vLLM's minimum version differs by size: 0.19.1 for 31B and E4B, 0.23.0 for 12B, which ships as a distinct unified architecture, and 0.25.0 for 26B-A4B. The serve command is the same shape across all four:

sudo docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN=$HF_TOKEN \
  vllm/vllm-openai:latest \
  --model google/gemma-4-31B-it \
  --max-model-len 65536

Binding to 127.0.0.1 and reaching it over an SSH tunnel matters here as much as anywhere else, since DeployGPU does not firewall VMs. The vLLM guide has the tunnel command and the tensor parallel flags for splitting a model across more than one GPU.

For chat, Ollama's library lists gemma4:e4b at 9.6 GB, gemma4:12b at 7.6 GB, gemma4:26b at 19 GB and gemma4:31b at 20 GB, checked 28 September 2026. Context is 262,144 tokens for 12B, 26B-A4B and 31B, and 131,072 for E4B.

Stopping and the next run

Stopping a DeployGPU instance deletes its disk, so a pulled Ollama tag or a downloaded Hugging Face checkpoint has to come back down again on the next launch. The 62.5 GB BF16 checkpoint for 31B takes longer over a slow connection than the 20 GB Ollama tag for the same model, worth factoring in before picking a format for a session you plan to repeat. The guide to stopping without losing work covers what to copy off first, and pricing covers what the hour itself costs.

License

All four sizes are released under Apache-2.0, so commercial use, fine-tuning and redistribution need no separate agreement. Axolotl ships an example configuration for the family, and Unsloth supports it directly.