The safest way to keep a job running after SSH drops is to start it inside tmux, and the quickest is nohup with the output sent to a file. Both survive a closed laptop or a dead Wi-Fi connection. Neither one stops the meter: closing SSH does not stop the instance, and a finished job does not stop it either.
Why a job dies when the SSH connection ends
A command you start in a plain SSH shell is a child of that shell. When the connection goes away the shell receives a hangup signal and passes it on, so the job ends with it. The SSH guide has a short tmux example, and this page is the longer version: four ways to detach a job, how to find it again, and what none of them can protect. Every manual and documentation page cited below was checked on 1 October 2026.
A keepalive stops a router from dropping an idle session, and it is one option on the command line:
ssh -o ServerAliveInterval=30 ubuntu@YOUR_VM_IP
The ssh_config manual defines ServerAliveInterval as a timeout in seconds after which ssh asks the server for a response, with a default of 0 that never asks. It keeps an idle session open. It does not save a job when the connection really drops, which is the case tmux and nohup exist for.
tmux: start the job in a named session
sudo apt-get install -y tmux # only if tmux is missing
tmux new -s train
python -u train.py 2>&1 | tee -a train.log
Press Ctrl-b, then d, to detach. The shell returns and the job keeps running. I use tmux for anything I want to watch, and nohup when the log is all I need. After you reconnect:
tmux ls
tmux attach -t train
The tmux wiki describes detaching as leaving the session and everything in it running in the background. Ctrl-b then the left square bracket enters copy mode, which lets you scroll back through the output, and Ctrl-b then a double quote splits the pane in two so nvtop can sit under the job.
Two details matter on a paid GPU. First, keep the log. If you hand tmux a command, as in tmux new -d -s train 'python -u train.py', the tmux manual says the window closes when the command completes, and the scrollback goes with it, so the tee -a train.log above is your record. Second, python -u keeps Python from buffering its output, per the Python documentation, so the screen and the log stay current.
nohup: start the job and walk away
nohup python -u train.py > train.log 2>&1 &
echo $!
tail -f train.log
The nohup manual says nohup runs a command while ignoring hangup signals. If standard input is a terminal it is redirected from an unreadable file, and if standard output is a terminal the output is appended to nohup.out. Redirecting to your own file as above avoids the surprise file. A job that waits for keystrokes cannot run this way, and that is the moment to use tmux. tail -f follows the log, and Ctrl-c stops the tail without touching the job.
systemd: restart the job when it crashes
Make the job a service when it should come back by itself, for example a model server, or a training script that resumes from its last checkpoint. The systemd.service manual says Restart=on-failure restarts the service when the process exits with a non-zero code or is terminated by a signal. Use the login user of your provider in User=:
| Provider | SSH user | Port |
|---|---|---|
| QuantaCloud | ubuntu | 22 |
| Massed Compute | Ubuntu, with a capital U | 22 |
| Hyperstack | ubuntu | 22 |
| Verda | root | 22 |
Write the unit with absolute paths. pwd and which python inside your virtual environment give you the values:
[Unit]
Description=Training run
[Service]
User=ubuntu
WorkingDirectory=/home/ubuntu/project
ExecStart=/home/ubuntu/venv/bin/python -u train.py --resume
Restart=on-failure
RestartSec=30
Save it as /etc/systemd/system/train.service, then:
sudo systemctl daemon-reload
sudo systemctl start train
sudo journalctl -u train -f
A job that crashes at startup, for example with an out-of-memory error on a GPU that is too small, will restart every 30 seconds until you stop it, and each restart reloads the model. Watch the first restart in the journal before you walk away. For a one-off command without a file, systemd-run --unit=train creates the same kind of service, per the systemd-run manual.
A container is a fourth option. sudo docker run -d --name train --gpus all IMAGE runs detached from your shell, and sudo docker logs -f train follows it, which is how the vLLM guide starts its server.
Find the job again after you reconnect
| Started with | Find it | Watch it | Stop it |
|---|---|---|---|
| tmux | tmux ls | tmux attach -t train | tmux kill-session -t train |
| nohup | pgrep -af train.py | tail -f train.log | pkill -f train.py |
| systemd | systemctl status train | sudo journalctl -u train -f | sudo systemctl stop train |
| Docker | sudo docker ps | sudo docker logs -f train | sudo docker stop train |
The process table in nvidia-smi is the quickest check that the GPU is still doing the work, and the first five minutes guide covers reading it. If SSH will not reconnect at all, Connection refused lists the causes, and one of them is that the VM is gone.
Make the job tell you when it ends
DeployGPU has no idle detection, and it sends no email or webhook when your job finishes, so a finished job leaves an idle GPU that keeps billing. Put a notification at the end of the command, in a script that you start with nohup or inside tmux:
python -u train.py > train.log 2>&1
status=$?
curl -fsS -d "train.py exited with status $status on $(hostname)" "$NOTIFY_URL"
NOTIFY_URL is any endpoint that accepts a POST. The ntfy publishing documentation shows the simplest form, curl -d "message" ntfy.sh/mytopic, and says a topic is essentially a password, so pick a name nobody can guess and keep secrets out of the message.
If you launch through the API, the job can stop its own instance instead. The batch jobs guide and the cost guardrails guide cover it, including the catch that DeployGPU API keys have no scopes, so a key copied onto a VM can do anything your account can do. Create a key for the purpose on the API keys page and delete it when you are done.
The job survives the disconnect, not the instance
When the balance cannot cover the next hour, the instance keeps running on per-minute billing until the balance reaches zero and is then stopped. Emails go out when about 2 hours of runtime are left, when per-minute billing starts and when the instance is stopped. When a provider loses a machine, the 5-minute health check ends the deployment and refunds the unused time, but it sends no email and the deployment disappears from the dashboard list. Save checkpoints off the machine as the job runs, as checkpoint and resume shows.
A reboot is not a fix to try in the middle of a run. It ends every tmux session and nohup job, and Verda's B200, B300 and RTX PRO 6000 machines do not come back from a reboot started inside the VM at all. If a fix needs a reboot, copy off what you cannot lose first.
Frequently asked questions
Does closing my SSH session stop my job?
Not if the job runs in tmux, under nohup, in a detached container or as a systemd service. A job started in a plain shell receives a hangup when the connection ends and stops with it.
Does closing my SSH session stop billing?
No. The instance keeps running and billing until you stop it or your balance runs out. DeployGPU has no idle detection and no auto-stop timer.
Should I use tmux or nohup?
Use tmux when you want to reattach and watch the job or type into it. Use nohup when you only need it to run and write a log. Use a systemd service when it must restart after a crash.
Will my job survive if the instance is stopped?
No. Stopping deletes the instance and its disk, and the same happens when the balance reaches zero or the provider loses the machine. Keep checkpoints somewhere other than the instance disk.
How do I know when my job has finished?
DeployGPU does not tell you. End your command with a notification such as a curl call to a service you control, or check nvidia-smi and your log from time to time.