The honest answer to "should I self-host an LLM?" is: only if you need something a hosted API will not give you. The usual reasons are prompts that must not leave a machine you control, a specific open model or your own fine-tune, heavy steady use where paying by the hour beats paying by the token, or simply not wanting a provider to change the model under you. If none of those apply, an API is less work. If one does, this guide covers the four decisions in order: the model, the GPU, the serving stack and the security, and then what it costs.
1. Pick the model first
Everything else follows from the model, because the model decides how much GPU memory you need. Open-weight models come with a license on their model card, so read it before you build on one. gpt-oss, the current Qwen 27B models and Gemma 4 are all published under Apache 2.0 at the time of writing.
| Model | Format | Weights | Smallest GPU that fits it on DeployGPU |
|---|---|---|---|
| gpt-oss-20b | MXFP4 | 13.8 GB | A 48 GB card, with lots of room for context |
| Qwen3.8-27B | 4-bit (Ollama) | 18 GB | A 48 GB card |
| Gemma 4 31B | 4-bit | about 17.5 GB | A 48 GB card |
| Qwen3.8-27B | BF16 | 55.6 GB | An 80 GB A100 or H100 |
| gpt-oss-120b | MXFP4 | 65.2 GB | An 80 GB A100 or H100, with little room for context |
| Llama 3.3 70B | BF16 | 141.1 GB | 2x H200, or 4x 80 GB GPUs |
The sizes come from the published checkpoints and the Ollama library, and the fit column is calculated rather than measured. The models section has more models and the arithmetic behind the table.
2. Size the GPU for weights plus context
The weights are only the first part of the bill. Every conversation in flight also keeps a KV cache in GPU memory, and it grows with the length of the context. The serving engine needs working memory on top of that.
A worked example, calculated from the published config: Qwen3.8-27B in BF16 needs about 64 KB of KV cache per token, so one 32,000-token conversation holds about 2 GB. On an 80 GB GPU, vLLM's default memory limit of 92% leaves about 18 GB after the 55.6 GB of weights. That is room for roughly 8 full-length conversations at once before the engine's own working memory, so fewer in practice. Switch to Qwen's official FP8 checkpoint and the weights roughly halve, which is why the vLLM recipe lists 38 GB as its minimum and a 48 GB card becomes an option.
The KV cache explainer has the formula and more examples. The short rule I use: pick the card where the weights take no more than about two thirds of the memory, unless you know you only need short contexts for one user.
3. Pick the serving stack by what you need
| What you need | Stack | How to run it on DeployGPU |
|---|---|---|
| A private ChatGPT-style chat in your browser | Open WebUI with Ollama | The Open WebUI + Ollama template, on a private link only your account can open |
| An OpenAI-compatible API for your own apps | vLLM, or SGLang | A GPU VM with Docker, reached through an SSH tunnel: deploy vLLM |
| One person running a quantized GGUF model | llama.cpp or Ollama | A GPU VM, installed yourself over SSH |
The template is the quickest start and needs no command line at all. The API route takes more setup, and it is the one to choose when code, not a person, sends the prompts. The trade-offs between engines are in vLLM, Ollama, llama.cpp or SGLang.
Opens the deploy page on the cheapest 1x RTX A6000 offer at 2026-10-16 02:34 UTC: QuantaCloud, $0.48/hr. You can pick another GPU there before paying.
4. Keep the endpoint private
This is the step people skip. DeployGPU does not put a firewall in front of your VM, and Hyperstack VMs are created with inbound traffic open, so a server started on 0.0.0.0 is on the internet with no password. The pattern I recommend has two parts. First, bind the server to 127.0.0.1 on the VM. Second, reach it from your machine through an SSH tunnel:
ssh -N -L 8000:127.0.0.1:8000 ubuntu@YOUR_VM_IP -p 22
Your code then talks to http://localhost:8000 as if the model ran on your laptop. If other machines must reach the API directly, put an API key on it (vLLM has an --api-key option) and a firewall rule in front of it, and treat it like any public service.
The private links that DeployGPU templates get are a different thing: they open only for your DeployGPU account, in a browser, and cannot be called from code or shared with a teammate. They suit the chat template, not an API.
5. Work out what it costs
A rented GPU costs the same per hour whether it generates tokens or sits idle, so the number that matters is cost per token at your real load:
cost per million tokens = hourly rate / (tokens per second x 3600) x 1,000,000
With an example rate of $2.00 per hour, a server producing 1,000 tokens per second across all its users costs about $0.56 per million tokens. The same GPU serving one user at 50 tokens per second costs about $11.11 per million. Those are example numbers, not measurements: measure your own throughput under your real traffic before comparing it with a per-token API price.
That arithmetic is also the case for renting before buying. An hour on the GPU you are considering tells you what your model really needs, for a few dollars, before you commit to hardware. For a plan-based alternative, Ollama Cloud or your own GPU compares the two directly. The live rates for every GPU are on the GPU pages.
6. Plan for the restart
Stopping a DeployGPU instance deletes its disk, model files included. Most model weights download again in minutes from Hugging Face or the Ollama library, so for occasional use the cheapest habit is to stop the GPU when you are done and download again next time. If you restart daily with very large weights, a volume on Verda or Hyperstack keeps them between sessions, but it is billed every hour it exists and is deleted permanently if your balance cannot pay for it. The stop guide covers both.
What self-hosting here does not give you
A self-hosted model on DeployGPU runs on one VM of up to 8 GPUs. There is no autoscaling, no managed endpoint, no load balancer and no uptime SLA, and if the provider loses the machine, you launch a new one. That is fine for personal use, internal tools, batch jobs and evaluation. For a customer-facing product with traffic spikes, plan for more than one machine and your own failover, or use a hosted API for the spiky part.
Frequently asked questions
What GPU do I need to self-host an LLM?
Enough memory for the weights plus the KV cache for your context length and number of users. A 48 GB card runs most 20B to 31B models at 4 or 8 bits, and an 80 GB card runs gpt-oss-120b or 27B to 31B models in BF16.
Is self-hosting an LLM cheaper than an API?
Only when the GPU is busy. A rented GPU costs the same per hour whether it works or idles, so divide the hourly rate by the tokens you really generate per hour and compare that with the API price.
Can I expose my self-hosted model as an OpenAI-compatible API on DeployGPU?
Yes, by running vLLM or SGLang yourself on a GPU VM. Keep it bound to localhost and reach it through an SSH tunnel, or protect it with an API key and a firewall if it must be public.
Do I lose my model when I stop the instance?
Yes, the disk is deleted. Download the weights again next time, or keep them on a Verda or Hyperstack volume, which is billed hourly while it exists.
Are my prompts private when I self-host on a rented GPU?
They stay on a VM you rent instead of going to a model provider. The cloud provider still operates the hardware, so treat it like any cloud server rather than a machine you own.