Deploy SGLang on a cloud GPU

The SGLang Docker image is one command on a GPU VM, and three things decide whether that command works: the driver on the VM, the shared memory you give the container and the port you publish. DeployGPU has no SGLang template, so everything here runs on a plain GPU VM over SSH. The SGLang flags come from the documentation pages named in each section, checked on 1 October 2026, and the Docker lines add sudo, a container name and a port bound to loopback.

Check that the driver supports the image

SGLang's installation page lists the Docker requirements: Linux, a supported NVIDIA GPU, a CUDA 13-compatible driver, Docker Engine and the NVIDIA Container Toolkit. Its images ship a CUDA 13 environment. The CUDA 12 images, tagged -cu12 or -cu129, are no longer published, and lmsysorg/sglang:v0.5.19-cu129 is the last CUDA 12 tag. NVIDIA's compatibility guide puts the floor for CUDA 13.x at driver 580.

So run nvidia-smi before you pull anything. If the CUDA version in its header is 13 or higher, use the current image. If it shows CUDA 12, the last tag built for CUDA 12 is v0.5.19-cu129, an older release, and the other option is a VM image with a newer driver. The toolkit guide has the Docker GPU check and the rule for reading that header. Docker Hub shows latest and v0.5.20 both updated on 18 September 2026, at about 15 GB compressed, and v0.5.19-cu129 at about 18.6 GB. The page says latest contains the newest stable release and that reproducible deployments should use a specific tag, so I pin one.

Run the server on loopback

This is the Docker example from SGLang's examples page, changed to a pinned tag, with the model cache and shared memory flags from the installation page. The small Qwen model is what that page uses to try the workflow:

sudo docker run -d --name sglang --gpus all --shm-size 32g --ipc=host \
  -p 127.0.0.1:30000:30000 \
  -v ~/hf-cache:/root/.cache/huggingface \
  lmsysorg/sglang:v0.5.20 \
  sglang serve --model-path Qwen/Qwen3-0.6B --host 0.0.0.0 --port 30000

The server arguments page says Docker runs need shared memory for communication between processes and points at --shm-size. Inside the container the server listens on 0.0.0.0, as the examples page does, and the 127.0.0.1 in the -p flag keeps it private, because DeployGPU does not firewall VMs and Hyperstack VMs are created with inbound traffic open. For a gated model, add --env HF_TOKEN=$HF_TOKEN before the image name, with your token exported in the shell. The installation page describes the same flag. Watch the log until it prints The server is fired up and ready to roll!:

sudo docker logs -f sglang

The documentation also shows python3 -m sglang.launch_server with the same --model-path, --host and --port flags, and I use sglang serve because the quickstart does.

Reach it through an SSH tunnel

DeployGPU app links do not accept API keys, so a client reaches the server through a tunnel. Run this on your own machine with the user, host and port from the deployment. The user is ubuntu on QuantaCloud and Hyperstack, Ubuntu with a capital U on Massed Compute and root on Verda:

ssh -N -L 30000:127.0.0.1:30000 ubuntu@YOUR_VM_IP -p 22

The quickstart verifies the server with one request, and the answer is in choices[0].message.content:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'

An OpenAI client works with base_url set to http://localhost:30000/v1 and api_key="EMPTY", the values the examples page uses. If anything else on the VM could reach the port, the server arguments page lists --api-key. The port forwarding guide covers tunnels in detail.

Size the GPU from the weights

SGLang gives a fraction of GPU memory to the weights and the KV cache pool together. The server arguments page says --mem-fraction-static sets it, that a smaller value helps with out-of-memory errors, and that unset it is computed as GPU memory minus reserved memory, divided by GPU memory. Start from the weights. These checkpoint sizes come from the model pages for Qwen and gpt-oss, and the table subtracts them from each card's memory (calculated, before the KV cache, activations and CUDA graphs):

The KV cache explainer shows how context and concurrency eat what is left. The examples page serves Qwen3.8-27B-FP8 on one H200, and for anything bigger than one card the multi-GPU page and the DeepSeek page cover the node sizes.

Use several GPUs with --tp-size

The server arguments page documents --tp-size, also written --tensor-parallel-size, with a default of 1. On a VM with 2, 4 or 8 GPUs, set it to the number of GPUs the container should share, and keep --gpus all in the Docker command:

sudo docker run -d --name sglang --gpus all --shm-size 32g --ipc=host \
  -p 127.0.0.1:30000:30000 \
  -v ~/hf-cache:/root/.cache/huggingface \
  lmsysorg/sglang:v0.5.20 \
  sglang serve --model-path YOUR_MODEL --tp-size 2 --host 0.0.0.0 --port 30000

The same page says that if you see the error peer access is not supported between these two devices, you add --enable-p2p-check. It also documents --dp for data parallelism, which it says is better for throughput if there is enough memory, and the multi-GPU page explains how the cards in one VM talk to each other.

Where tuned launch profiles come from

Model-specific flags, such as parsers, KV cache types and memory settings for hybrid models, change with each SGLang release. SGLang's cookbook has a page per model, but those pages build the command in the browser from the choices you make, so the page source does not contain it. Open the page for your model, pick your GPU, copy the command it prints, and keep the Docker flags from this guide around it: --gpus all, --shm-size 32g, --ipc=host and the loopback publish. The examples page shows one such profile, Qwen3.8-27B-FP8 on an H200, and I leave its flags there because they will not stay current in a guide.

When something goes wrong

Stopping the deployment deletes the instance disk, so the 15 GB image and the weights in ~/hf-cache are pulled again on the next launch. Budget that time, and see the stop guide for what to copy off first. If a simpler server is enough, the vLLM guide and the engine comparison cover the alternatives.