Reconnecting To Background Processes With Gnu Screen After Ssh Session Disconnect
When working on remote servers over SSH, processes and tasks started in the SSH session will terminate when the network connection drops or the client disconnects. However, in many cases we want long-running processes, such as training machine learning models, downloading large files, or running computational workflows, to continue executing in the background after we logout. This allows us to disconnect without losing work and reconnect later to check status and results.
The GNU Screen utility enables this persistent background execution by creating resumable terminal sessions that can detach and reattach as needed. Understanding Screen best practices empowers users to reliably start processes on remote systems that continue running indefinitely, survive disconnects, and allow later reentry to check results and manage jobs.
Why Processes Die When SSH Sessions Disconnect
On most UNIX-based systems, the shell session and processes spawned within it are tied to the stdin, stdout and stderr streams of the controlling terminal. When the SSH client disconnects, the terminal disappears and the kernel sends SIGHUP signals to foreground processes tied to that terminal, causing them to terminate. Background commands running in subshells will appear stopped in ps listings as the controlling terminal has exited. Ultimately orphaned background processes will realize the terminal exited and will also terminate.
Introducing GNU Screen for Session Persistence
GNU Screen enables persisting shells and processes through disconnections by virtualizing terminals that remain active whether user input/output is attached or not. Processes running within a Screen shell will survive an SSH disconnect and continue executing because their controlling terminal persists as a virtual session in the background. Screen allows the user to later reattach, re-enter the Shell to inspect output, check running jobs, and manage the virtual session.
Key Capabilities
- Virtual terminals remain active during detachments
- Shells and subprocess keep running after client SSH disconnect
- Later reattachment to the virtual terminal to reenter shell
- Inspect output, check running jobs, manage virtual session
- Supports multiple concurrent virtual terminals
Attaching and Detaching Screen Sessions
To introduce persistency with Screen, we first launch a new Screen session which opens a virtual shell. While attached, we start jobs and processes as normal from within the Screen shell. When ready to disconnect our client, we detach from the Screen session causing it to persist in the background with all running processes. Later we reattach to the existing Screen session to return to the persistent virtual shell and check running jobs.
Launching a Detached Screen
Use the -d option to start Screen detached from the terminal:
screen -d -m
This creates a background virtual terminal and shell while staying on the original SSH terminal.
Attaching to an Existing Screen
To connect to the detached session above:
screen -r
This attaches input/output to the persistent virtual terminal, letting you interact with the shell and processes running within Screen.
Detaching from Attached Screen
To detach from an attached Screen press Ctrl+A then D.
This key sequence sends the detach signal to Screen, causing it to persist in the background while returning you to the underlying SSH shell outside of Screen.
Reconnecting to Detached Screen Sessions
Even if the SSH connection drops or the client disconnects, the detached Screen persists on the server with running processes intact. When SSH reconnects to the host, we can reattach to the disconnected Screen. If only one detached session exists, attach using:
screen -r
If multiple Screens are detached in the background, list with -ls to view session IDs:
screen -ls
Then attach to a specific session by id:
screen -r [id]
This locates the persistent session and reconnects our terminal input and output.
Checking Running Screen Sessions
When reconnecting to a server, we may be unsure if Screen sessions are running detached. Query detached and attached Screens using:
screen -ls
Information includes session id, detached/attached state, session name, and creation date/time. We can inspect sessions and choose which to reattach if multiple exist.
Killing Stale Detached Screen Sessions
Over time, abandoned Screens with old processes may accumulate on servers after users disconnect without formally exiting sessions. List detached Screens with -ls, then kill a session by id:
screen -ls screen -X -S [id] quit
The -X sends the terminate signal to the target Screen session, killing the shell and child processes safely. This keeps servers tidy by removing obsolete detached sessions.
Example Code for Creating and Reconnecting to Screen
Here is an example sequence demonstrating a common Screen workflow:
# Connect to server over SSH ssh user@host # Start new detached Screen screen -d -m # Attach to Screen session screen -r # Run process from within Screen shell long_process # Detach Persistent Screen Ctrl+A D # Disconnect SSH completely exit # Later reconnect to server over SSH ssh user@host # Reattach persistent Screen screen -r # Check running process inside Screen ps -ef # Kill Screen when finished exit screen -X -S [id] quit
