Practical guides

Cost guardrails for GPU automation

Guardrails that keep automated GPU launches from running up a bill: runtime limits, stop-on-exit traps, a sweeper job, balance checks and key hygiene.

The honest answer to how you stop idle GPU instances automatically on DeployGPU is that you build it yourself. As of 1 October 2026 the platform has no idle detection, no auto-stop timer and no maximum runtime, so a forgotten GPU bills until you stop it or your balance runs out. Five guardrails cover the gap, in the order I would add them. Each one catches what the one before it misses.

Guardrail 1: stop in an EXIT trap

The cheapest guardrail fires when your own script ends. The batch jobs guide does it in Python with finally. In a shell script, a trap on EXIT runs when the script ends, including after a failed command under set -e, and it does nothing if the create call returned no id:

Code example
#!/usr/bin/env bash
set -euo pipefail
API=https://api.deploygpu.ai/api/v1
DEPLOYMENT_ID=""

stop_instance() {
  [ -n "$DEPLOYMENT_ID" ] || return 0
  sleep 3  # a brand-new id can answer 404 for about a second
  curl -s --retry 3 --max-time 30 -X POST "$API/deployments/$DEPLOYMENT_ID/stop" \
    -H "X-API-Key: $DEPLOYGPU_API_KEY" > /dev/null
}
trap stop_instance EXIT

DEPLOYMENT_ID=$(curl -s -X POST "$API/deployments" \
  -H "X-API-Key: $DEPLOYGPU_API_KEY" -H "Content-Type: application/json" \
  -d "{\"provider\": \"quantacloud\", \"offer_id\": \"$OFFER_ID\", \"disk_gb\": 100}" \
  | python3 -c 'import json, sys; print(json.load(sys.stdin)["id"])')

# ... wait for active, run the job over SSH, copy the results out ...

The create call has no --retry, because a retried create can start a second GPU. The stop call does, because stopping twice is harmless: the second answer is a 400 saying the deployment is already stopping. A trap cannot catch SIGKILL or a crashed machine, so it needs a backstop.

Guardrail 2: sweep for anything older than a cap

A GPU left running over a weekend bills 48 hours of its hourly rate, and an 8 GPU VM bills all 8 GPUs. The sweeper below is the backstop. It pages through your deployments, 100 at a time, history included, keeps the active ones and stops any older than MAX_HOURS, counted from created_at because paid time starts when provisioning starts. The stop guide has a shorter version.

Code example
"""Stop DeployGPU instances that ran too long or sat idle, and report how long the balance lasts.

Not yet run against the real API. DRY_RUN defaults to on: it only prints what it would stop.
"""
import json
import os
import subprocess
import sys
import time
from datetime import datetime, timezone

import requests

API = "https://api.deploygpu.ai/api/v1"
HEADERS = {"X-API-Key": os.environ["DEPLOYGPU_API_KEY"]}
DRY_RUN = os.environ.get("DRY_RUN", "1") != "0"
SSH_KEY_FILE = os.environ.get("DEPLOYGPU_SSH_KEY_FILE")  # optional: enables the idle check
STATE_FILE = os.path.expanduser("~/.deploygpu-guard.json")

MAX_HOURS = 8      # stop anything older than this, counted from creation
IDLE_MINUTES = 30  # stop after this long below IDLE_PERCENT GPU utilization
IDLE_PERCENT = 5
LOW_HOURS = 3      # exit with status 2 when the balance covers fewer hours than this
FAILED = []


def active_deployments():
    page, found = 0, []
    while True:
        resp = requests.get(f"{API}/deployments", headers=HEADERS, params={"page": page, "size": 100}, timeout=30)
        resp.raise_for_status()
        body = resp.json()
        found += [d for d in body["data"] if d["status"] == "active"]
        page += 1
        if not body["data"] or page * body["size"] >= body["total_elements"]:
            return found


def age_hours(d):
    created = datetime.strptime(d["created_at"][:19], "%Y-%m-%dT%H:%M:%S").replace(tzinfo=timezone.utc)
    return (datetime.now(timezone.utc) - created).total_seconds() / 3600


def gpu_utilization(conn):
    """Highest GPU utilization in percent, or None when the VM cannot be asked."""
    cmd = ["ssh", "-i", SSH_KEY_FILE, "-p", str(conn["ssh_port"]), "-o", "BatchMode=yes",
           "-o", "ConnectTimeout=10", "-o", "StrictHostKeyChecking=accept-new",
           f"{conn['ssh_user']}@{conn['public_ip']}",
           "nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits"]
    try:
        out = subprocess.run(cmd, capture_output=True, text=True, timeout=30, check=True).stdout
        return max(int(line) for line in out.split())
    except (subprocess.SubprocessError, ValueError):
        return None


def stop(d, reason):
    print(f"{'WOULD STOP' if DRY_RUN else 'STOPPING'} {d['id']}: {reason}")
    if DRY_RUN:
        return
    try:
        requests.post(f"{API}/deployments/{d['id']}/stop", headers=HEADERS, timeout=60).raise_for_status()
    except requests.RequestException as exc:
        print(f"Could not stop {d['id']}: {exc}", file=sys.stderr)
        FAILED.append(d["id"])


def main():
    active = active_deployments()
    balance = float(requests.get(f"{API}/account/balance", headers=HEADERS, timeout=30).json()["balance"])
    rate = sum(float(d["hourly_rate"]) for d in active)

    try:
        with open(STATE_FILE) as f:
            idle_since = json.load(f)
    except (OSError, ValueError):
        idle_since = {}
    now = time.time()
    for d in active:
        if age_hours(d) > MAX_HOURS:
            stop(d, f"older than {MAX_HOURS} hours")
        elif SSH_KEY_FILE and d.get("connection"):
            busy = gpu_utilization(d["connection"])
            if busy is None:
                continue  # cannot tell, so leave it running
            if busy >= IDLE_PERCENT:
                idle_since.pop(d["id"], None)
            elif now - idle_since.setdefault(d["id"], now) > IDLE_MINUTES * 60:
                stop(d, f"GPU under {IDLE_PERCENT}% for {IDLE_MINUTES} minutes")
    live = {d["id"] for d in active}
    with open(STATE_FILE, "w") as f:
        json.dump({k: v for k, v in idle_since.items() if k in live}, f)

    if rate:
        hours_left = balance / rate
        print(f"Balance {balance:.2f} USD, running {rate:.2f} USD per hour, about {hours_left:.1f} hours left")
        if hours_left < LOW_HOURS:
            sys.exit(2)
    if FAILED:
        sys.exit(1)


if __name__ == "__main__":
    main()

It stops every active deployment on the account, and the API has no tags to tell yours from a teammate's. Run it on an account that automation owns, or set the cap above your longest legitimate run. Stopping deletes the instance and its disk, so work under a cap should checkpoint, as the checkpoint guide shows. Run it by hand until the dry run prints what you expect. Then schedule it from a machine you control, reading DEPLOYGPU_API_KEY and, for the idle check below, DEPLOYGPU_SSH_KEY_FILE from an env file that only you can read:

Code example
*/10 * * * * . "$HOME/.deploygpu-guard.env" && DRY_RUN=0 python3 "$HOME/guard.py" >> "$HOME/guard.log" 2>&1

Guardrail 3: an idle check that knows its limits

When DEPLOYGPU_SSH_KEY_FILE is set, the sweeper logs in to each active VM and asks nvidia-smi for GPU utilization with --query-gpu=utilization.gpu. It remembers when a VM first looked idle, resets the clock whenever utilization reaches 5%, and stops the VM after 30 minutes. If a VM cannot be reached it does nothing, because a failed login is not evidence of idleness.

Utilization is a weak signal. A model server waiting for requests reads 0%, and so can a job that is downloading data, so set IDLE_MINUTES above your longest quiet phase. Samples come 10 minutes apart, which also means a job that only works between samples looks idle. For interactive sessions I would rely on the runtime cap alone.

Guardrail 4: let the balance be the hard cap

Keys have no scopes, so the balance behind the account is the only limit on what a key can spend. A separate account for automation, funded with what you can afford to lose, turns the balance into a hard cap. When it cannot cover the next hour, an instance switches to per-minute billing and keeps running until the balance reaches zero, then it is stopped and its disk is deleted. The billing documentation lists the three emails: about 2 hours of runtime left, per-minute billing starting, and the stop. They go to the account address, not to your code.

Auto top-up changes the cap. With it on, DeployGPU charges your saved card when the balance falls below your threshold, so a runaway job is limited by the card instead. My rule for an account that automation owns is to leave it off and size the balance to a known number of runs, because losing a run to an empty balance costs less than an open-ended bill. For a job you cannot repeat or a volume you cannot recreate, the billing documentation advises the opposite. The sweeper also prints how long the balance lasts: the balance divided by the combined hourly rate of your active instances, the same estimate as the dashboard's estimated time left, which leaves out storage, transfer and future top-ups. Below LOW_HOURS it exits with status 2, which gives cron or your monitoring something to alert on. The pricing page has the rest of the billing rules.

Guardrail 5: keep the key off the VM

A key copied onto a GPU VM can be used by anyone with access to that VM, including to launch more GPUs. Run your launch scripts and the sweeper from machines you control, with the key in a secret store. Create one key per system on the API keys page and name it for what uses it. To rotate, create the new key, switch the system over, then revoke the old one, which the API keys documentation says takes effect immediately.

Four mechanisms and which of them stop billing

MechanismStops billingWhy
POST /api/v1/deployments/{id}/stop or DELETEYesTerminates the VM and deletes its disk
shutdown or poweroff inside the VMNot as a stopOn QuantaCloud and Massed Compute a VM reported as powered off is treated as lost at the next 5-minute health check and deleted with its disk, with no email. On Verda and Hyperstack it keeps billing
Verda or Hyperstack POST /api/v1/deployments/{id}/shutdownNoThe deployment stays active and full compute billing continues
Closing a terminal or killing your scriptNoNothing tells DeployGPU to stop

The deployments reference and the API page cover the endpoints.

Frequently asked questions

Does DeployGPU stop idle GPU instances automatically?

No. There is no idle detection, auto-stop timer or maximum runtime. You stop instances yourself, or with your own scripts through the API.

What happens when my balance runs out while an instance is running?

When the balance cannot cover the next hour, the instance switches to per-minute billing and keeps running until the balance reaches zero. Then it is stopped and its disk is deleted.

Can I limit what an API key is allowed to spend?

Not on the key itself. Keys have no scopes or expiry, so the balance behind the account is the limit. A separate account with a small balance works for automation.

Is powering a VM off a way to stop the billing?

No. Stopping through the API is. On Verda and Hyperstack the shutdown endpoint keeps full billing, and on QuantaCloud and Massed Compute a VM that looks powered off is treated as lost and deleted with its disk.

How do I know how long my balance will last?

Divide the balance by the combined hourly rate of your running instances, which is what the dashboard's estimated time left shows. It leaves out storage, transfer and future top-ups.