Debugging Background Processes After Ssh Session Disconnect
Using Screen to Detach Processes
The screen command allows users to detach interactive terminal sessions and reattach them later. This allows processes running within a screen session to continue executing even after disconnecting the SSH session that started it.
To start a screen session, simply run the screen
command. You will be presented with a new shell prompt inside the screen. To detach the screen session – leaving any processes running inside it – use the key sequence CTRL+A then D. This will drop you back to your original shell session, with the screen continuing to run in the background.
To reattach to a detached screen, use the screen -r
command. This will open the previously detached screen with all running processes intact.
Key capabilities provided by screen for backgrounding processes:
- Start new screen session with
screen
- Detach current screen with
CTRL+A
thenD
- Re-attach detached screen with
screen -r
- Multiple screens can be started with names using
screen -S name
- List existing screens with
screen -ls
- Works for long-running interactive processes like text editors, mutt, irc clients etc.
Nohup for Backgrounding Processes
The nohup command allows processes to continue executing even after the terminal or SSH session that launched it is closed. It works by ignoring HUP signals that would normally terminate processes when a session ends.
To use nohup, simply prefix the command you want to run with nohup
like:
nohup myprocess &
This will run myprocess
in the background immune to hangups.
Since standard output and error will still be connected to the launched terminal, its better to redirect stdout/stderr to a file:
nohup myprocess &> mylog.out &
Now myprocess
will run in the background with all output going to mylog.out
even after disconnect.
Key capabilities provided by nohup:
- Launch background process immune to hangups with
nohup
- Redirect stdout and stderr to file with
&> file
- Simple and lightweight backgrounding of non-interactive processes
Disowning Jobs in Current Shell
The disown
bash builtin can be used to remove a running process from the shell’s job control. This prevents the process from receiving SIGHUP signals when the parent shell exits.
To disown a running job, use:
disown -h %1
Where %1 is the jobspec of the process to disown.
jobs
command lists running jobs and their jobspecs.
Some key points about disown:
- Only works on jobs launched from the current shell
- Use
jobs
to view running jobs and jobspecs disown -h
prevents process from receiving SIGHUP
Example disconnecting from an SSH session with a disowned process:
someprocess &
# Start background processjobs
# Check jobspec numberdisown -h %1
# Disown background processexit
# Disconnect SSH session
Systemd User Services for Persistence
Systemd user services can be created to start processes automatically when a user logs in. This provides a simple way to keep processes persistent across sessions.
To create a user service:
- Create service file
~/.config/systemd/user/myprocess.service
- Edit file to have ExecStart, Description and other directives
- Reload systemd config:
systemctl --user daemon-reload
- Enable service:
systemctl --user enable myprocess
An example file with essential directives:
[Unit] Description=My Custom Process [Service] ExecStart=/usr/bin/python /home/user/myscript.py Restart=always [Install] WantedBy=default.target
Now myscript.py
will be started on each login session due to WantedBy directive.
Automation with tmux
tmux is a terminal multiplexer similar to screen, allowing customized persistent sessions and detach capabilities.
Some ways tmux helps with process continuity:
- Start new tmux session with
tmux
- Detach session with CTRL+B then D
- Reattach with
tmux attach
- List sessions with
tmux ls
- Automate session creation with tmuxinator tool
Example workflow:
tmux
start main sessionvim file
edit files within tmux- CTRL+B D detach tmux
- ssh logout, reconnect
tmux attach
rejoin session
This keeps vim editing session up across disconnects.
Best Practices
- For interactive programs like vim/mutt/etc use screen or tmux
- For one-off scripts or batch jobs use nohup or disown
- For startup persistence use systemd user services
Additional tips:
- Check log files for background job outputs
- Use descriptive names for screen/tmux sessions
- Enable automatic reconnection with ssh ControlPersist
Following these best practices will lead to smooth sailing even when disconnecting from difficult to terminate processes.