Use Claude Code with a model you host yourself

The Claude Code local model setups that vLLM, SGLang and Ollama document all work the same way: you set ANTHROPIC_BASE_URL to a server that speaks the Anthropic Messages API, give Claude Code a placeholder token and name your model in a few more variables. Anthropic does not support this. Its gateway page says it does not support routing Claude Code to non-Claude models through any gateway, so the working recipe comes from the server projects, whose documentation I checked on 12 November 2026. On DeployGPU the server runs on a rented GPU VM and your own machine reaches it through an SSH tunnel.

What Claude Code needs from a server

Claude Code's gateway compatibility guide says it treats whatever sits behind ANTHROPIC_BASE_URL as an Anthropic-format endpoint. It posts to /v1/messages, reads the reply as a stream of events and treats a stream that ends before the final message_delta as a dropped connection. The /v1/messages/count_tokens route is optional: without it Claude Code falls back to a character-based estimate, so /context shows approximate counts. Beyond the API, the model has to call tools, because that is how Claude Code reads and edits files and runs commands, and vLLM's page says it needs a model with strong tool calling. The context has to be large. For a model name it does not recognize, Claude Code assumes 200,000 tokens, and its auto-compact window cannot be set below 100,000, so I treat 100,000 tokens as the smallest server context worth running.

The sources, all checked 12 November 2026, are vLLM's Claude Code page and tool calling page, SGLang's Anthropic API page, the llama.cpp server README and Ollama's Anthropic compatibility page. The llama.cpp README says it makes no strong claims of compatibility with the Anthropic spec and that in its experience the route suffices for many apps. Ollama lists tool_choice, metadata, prompt caching and the batches API as not supported.

Pick the GPU before the VM

A coding agent needs the weights plus a KV cache for a long context. The table is calculated with the rule I use across the site, 92% of the card's memory minus the checkpoint, and it leaves out activations and CUDA graphs, so the GPU KV cache size line in vLLM's startup log is the real number. The model is Qwen3-Coder-30B-A3B, Apache-2.0 on its Hugging Face cards, with 48 layers and 4 KV heads, which makes its BF16 KV cache 96 KiB per token (2 x 48 x 4 x 128 x 2 bytes, from its config.json). Checkpoint sizes are the summed safetensors files of the BF16 and FP8 repositories, read on 12 November 2026.

One 131,072-token session needs 12.9 GB of cache (calculated). The 48 GB row therefore holds one session with nothing to spare, which activations will not allow, and the 80 GB FP8 row holds about three. Claude Code sends the main conversation, subagent turns, compaction and side requests such as session titles to the same server, so I start from the 80 GB FP8 row. Larger coding models need multi-GPU VMs, and the open coding models page lists what to host and the GPU each one needs.

Set up a Claude Code local model on a rented GPU

Launch a plain GPU VM with an 80 GB GPU, the dashboard's Bare Metal template, and run nvidia-smi before you pull anything.

vLLM's release notes for v0.33.0 say the CUDA 12.9 images are no longer published, so the latest image is built for CUDA 13.0, which vLLM's GPU installation page says needs an R580 or newer driver. If the nvidia-smi header shows CUDA 12, pin an older tag such as v0.32.0-cu129, or read the section on older drivers on that page. The container follows the loopback pattern of the vLLM guide, with the flags Claude Code adds:

sudo docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 \
  --served-model-name coder \
  --max-model-len 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

--served-model-name coder matters because vLLM's Claude Code page says model names with a slash in them, such as the Hugging Face name, do not work with Claude Code. --enable-auto-tool-choice and --tool-call-parser switch tool calls on, and vLLM's tool calling page lists qwen3_xml for the Qwen3-Coder models. The argument reference describes --max-model-len as prompt and output together, so 131072 is the window I give Claude Code below. sudo docker logs -f vllm shows when the server is ready and prints the KV cache size.

From your own machine, open the tunnel with the user, host and port from the SSH command on your deployment, then check the Anthropic route. The curl line is adapted from the verification request on Claude Code's connect page, pointed at the tunnel and at the served name:

ssh -N -L 8000:127.0.0.1:8000 ubuntu@YOUR_VM_IP -p 22

curl -sS http://localhost:8000/v1/messages \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model": "coder", "max_tokens": 64, "messages": [{"role": "user", "content": "Say hello in five words"}]}'

A JSON reply with a content array means the route works. The port forwarding guide has the SSH user for each provider and the keepalive options that stop an idle tunnel from dropping in the middle of a session.

Environment variables for Claude Code

Set these in the shell that starts Claude Code, then run it from your project folder:

export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_AUTH_TOKEN=dummy
export ANTHROPIC_DEFAULT_OPUS_MODEL=coder
export ANTHROPIC_DEFAULT_SONNET_MODEL=coder
export ANTHROPIC_DEFAULT_HAIKU_MODEL=coder
export CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
claude

The variable names and behavior come from Claude Code's variable reference and model configuration pages. The /status command inside Claude Code should show an Anthropic base URL line with your address and an Auth token line naming ANTHROPIC_AUTH_TOKEN. A line naming a claude.ai login means the token variable never reached the session. To make the setup permanent, use the env block of ~/.claude/settings.json, which wins over the shell when both set a variable. Keep it out of a project's .claude/settings.json, because that file is committed and shared.

Claude Code with Ollama

Ollama's Claude Code page offers ollama launch claude. With the server on a rented VM I use its manual setup instead, because the Claude Code on my own machine has to reach the VM. The Open WebUI and Ollama template does not expose Ollama's API, so this variant also runs on a plain GPU VM, and the Ollama guide covers the install in detail. Ollama's context page says coding tools should get at least 64,000 tokens and sets the default by VRAM, 4k under 24 GiB, 32k from 24 to 48 GiB and 256k from 48 GiB. I set it explicitly to match the vLLM example:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl edit ollama

In the editor that opens, add these two lines, then restart the service and pull the model:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=131072"

sudo systemctl daemon-reload
sudo systemctl restart ollama
ollama pull qwen3-coder

The library page lists qwen3-coder:30b as the default tag, 19 GB with a 256K context window. A 131,072-token cache at the default f16 type adds the same 12.9 GB as above, about 32 GB in total before compute buffers (calculated), so a 48 GB card holds it. After the first request, the CONTEXT column of ollama ps shows what the server really allocated. Open a tunnel to port 11434, as in the vLLM section, and use Ollama's own placeholders:

export ANTHROPIC_AUTH_TOKEN=ollama
export ANTHROPIC_API_KEY=""
export ANTHROPIC_BASE_URL=http://localhost:11434
export CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072
claude --model qwen3-coder

Ollama's FAQ gives OLLAMA_NUM_PARALLEL a default of 1, so subagent and background requests wait for the main conversation. Raising it multiplies the memory the context needs.

LiteLLM and Claude Code Router

Because all four servers above serve /v1/messages themselves, I do not put a second program in front of them. A translation layer earns its place when a server speaks only the OpenAI API, or when one endpoint should front several models. LiteLLM's Claude Code page says Claude Code talks to the Anthropic Messages API and that LiteLLM serves that format at /v1/messages. Its tutorial for non-Anthropic models says it translates each request to the target provider's format and the reply back. Its vLLM page routes to a vLLM server with the hosted_vllm/ prefix:

model_list:
  - model_name: coder
    litellm_params:
      model: hosted_vllm/coder
      api_base: http://127.0.0.1:8000/v1

Install it with uv tool install 'litellm[proxy]' and start it with litellm --config config.yaml. The tutorial then sets ANTHROPIC_BASE_URL to the proxy, port 4000 in its examples, and ANTHROPIC_AUTH_TOKEN to the master key you configured. The proxy runs on your machine and reaches vLLM through the same tunnel. The api_base ends in /v1 because LiteLLM's OpenAI-compatible page says to add that postfix when a request returns Not Found.

Claude Code Router is a separate project, not Anthropic's. Its README, checked on 12 November 2026, calls it a local model gateway and control plane for coding agents, lists Claude Code and OpenCode among the agents it connects, and says you add a provider as a built-in preset or a custom endpoint. The CLI needs Node.js 22 or newer, installs with npm install -g @musistudio/claude-code-router and starts its interface with ccr ui. The gateway listens on http://127.0.0.1:3456 by default, the repository is MIT licensed and the latest release, v3.1.3, is from 9 October 2026. The project's own documentation is the place for the endpoint fields.

DeepSeek, GLM, Kimi and MiniMax

Nothing above depends on Qwen, so Claude Code with other models works the same way as long as your server can parse their tool calls. vLLM's tool calling page lists a parser per family, for example deepseek_v3 and deepseek_v31, glm45 and glm47, kimi_k2 and openai for gpt-oss, and for the newer releases the flags come from each model's own card or recipe. The model pages link those and give the GPU each model needs: DeepSeek, GLM, MiniMax-M3 and Kimi K3. This page covers weights you serve yourself. Pointing Claude Code at a vendor's hosted API is a different setup, and that vendor's documentation is the source for it.

What works worse or not at all

In a self-hosted Claude Code setup like this one, the agent still runs on your own machine and only the model runs on the VM. A rented GPU is not an offline setup, because your prompts travel over the tunnel to a machine on the internet. What the documentation does cover is Claude Code's own background calls. CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 turns them off, and the connect page names one call it leaves: the WebFetch domain safety check still contacts api.anthropic.com, and skipWebFetchPreflight: true in settings turns that off.