Run Muse Glimmer 30B on a cloud GPU

The Muse Glimmer 30B model fits one 80 GB GPU in BF16, at 59.55 GB with about 14 GB left, and fits any 48 GB card in Meta's official 4-bit GGUF, at 16.76 GB. Meta built it for 24 GB and 32 GB consumer hardware, so DeployGPU's smallest class is already roomy for the GGUF. The BF16 weights are the reference precision, and a single 80 GB GPU or an RTX PRO 6000 runs them with vLLM.

Muse Glimmer sizes: 17, 34 and 60 GB

Meta's model card, checked 1 October 2026, describes a dense 30B-class model with a perception encoder for images, released in August 2026 under Apache 2.0. It has 52 layers, a context of 131,072 tokens and 32 query heads against 2 KV heads. Meta says it compresses the language model to about 4 bits, under 20 GB, and targets 24 GB and 32 GB of VRAM. Sizes below are the summed files on Hugging Face, and the left-over figures are 92% of the VM's memory minus the file size, calculated.

The image encoder is a separate 1.4 GB file in the GGUF repo, and a 1.63 GB DFlash drafter for speculative decoding is optional. There is no official BF16 GGUF, so use the safetensors repo for full precision.

Why 13 KiB per token is all the cache needs

Muse Glimmer repeats three sliding-window layers of 2,048 tokens and then one global layer, which makes 39 sliding layers and 13 global ones. Only the 13 global layers grow with context. With 2 KV heads of 128 dimensions in BF16, they cost 13 KiB per token, and the 39 sliding layers add a fixed 78 MiB per sequence (both calculated from the config). A full 131,072-token session is about 1.62 GiB plus those 78 MiB, so the 14 GB left on an 80 GB card is, by this arithmetic, room for several full-length sessions before activations. The KV cache explainer covers the method for sliding-window layers.

Serve the BF16 model with vLLM

vLLM's supported models page lists the architecture, and the recipe gives 0.28.0 as the minimum version with the vllm/vllm-openai:v0.28.0 image. The recipe lists GB300, DGX Spark, RTX 5090, AMD and Intel cards as verified hardware and does not list an H100 or A100, so check the startup log on an 80 GB card. This is its command adapted to one card, with a tensor parallel size of 1 and the Hugging Face ID in place of its local model path, bound to loopback:

sudo docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.28.0 \
  --model meta-models/Muse-Glimmer-30B \
  --served-model-name muse-glimmer \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --generation-config auto

The model needs its own tool-call and reasoning parsers, because it emits XML-style tool calls and channel-scoped reasoning. Use the card's sampling of temperature 1.0, top_p 0.95 and top_k 64, and do not run it greedy. Reasoning strength is a Reasoning strength: high line in the system prompt, with low, medium, high and xhigh as the levels. The recipe warns that a tight max_tokens can end before the final channel closes and return empty content. Reach the server through the tunnel in the vLLM guide.

Run the official GGUF with llama.cpp

The GGUF card says llama.cpp support merged on 10 August 2026 and needs build b10353 or newer, which older builds refuse to load. Download the text model and the image encoder as the card shows:

pip install huggingface_hub

hf download meta-models/Muse-Glimmer-30B-GGUF \
    --local-dir ~/models/Muse-Glimmer-30B-GGUF \
    --include "Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf" \
    --include "mmproj-Muse-Glimmer-30B-Q4_K_M.gguf"

Then run the card's llama-server command in llama.cpp's own Docker image. That page lists a CUDA 12 and a CUDA 13 server image, so pick the one your driver supports, which nvidia-smi shows. The card's example binds 127.0.0.1. Inside the container the host has to be 0.0.0.0, and the published port stays on loopback. I also set -np 1, because the card explains that -c is split across slots, so its example of -np 4 gives each request a quarter of the context:

sudo docker run -d --name llama --gpus all \
  -p 127.0.0.1:8080:8080 \
  -v ~/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  --mmproj /models/Muse-Glimmer-30B-GGUF/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  -a muse-glimmer-30B \
  -ngl 99 -c 131072 -np 1 \
  --jinja \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --host 0.0.0.0 --port 8080

--jinja is not optional, according to the card, because the chat template is embedded in the file. The llama.cpp server guide covers the rest of this setup, and the Llama page covers Meta's older Llama models.

Ollama, SGLang and the license

Ollama lists muse-glimmer at 18 GB by default with a 128K context window and image input, muse-glimmer:30b-q8_0 at 31 GB and muse-glimmer:30b-bf16 at 57 GB, checked 1 October 2026. A 48 GB card holds the first two. Meta's card also shows SGLang, run from the lmsysorg/sglang:latest image.

All the files are Apache-2.0, so commercial use and fine-tuning need no separate agreement. Stopping a DeployGPU instance deletes its disk, so each launch downloads the weights again, and the stop guide covers what to copy off first. The 48 GB page and the 80 GB page compare the cards.