Debugging Background Processes After Ssh Session Disconnect

Using Screen to Detach Processes

The screen command allows users to detach interactive terminal sessions and reattach them later. This allows processes running within a screen session to continue executing even after disconnecting the SSH session that started it.

To start a screen session, simply run the screen command. You will be presented with a new shell prompt inside the screen. To detach the screen session – leaving any processes running inside it – use the key sequence CTRL+A then D. This will drop you back to your original shell session, with the screen continuing to run in the background.

To reattach to a detached screen, use the screen -r command. This will open the previously detached screen with all running processes intact.

Table of Contents

Key capabilities provided by screen for backgrounding processes:

Start new screen session with screen
Detach current screen with CTRL+A then D
Re-attach detached screen with screen -r
Multiple screens can be started with names using screen -S name
List existing screens with screen -ls
Works for long-running interactive processes like text editors, mutt, irc clients etc.

Nohup for Backgrounding Processes

The nohup command allows processes to continue executing even after the terminal or SSH session that launched it is closed. It works by ignoring HUP signals that would normally terminate processes when a session ends.

To use nohup, simply prefix the command you want to run with nohup like:

nohup myprocess &

This will run myprocess in the background immune to hangups.

Since standard output and error will still be connected to the launched terminal, its better to redirect stdout/stderr to a file:

nohup myprocess &> mylog.out &

Now myprocess will run in the background with all output going to mylog.out even after disconnect.

Key capabilities provided by nohup:

Launch background process immune to hangups with nohup
Redirect stdout and stderr to file with &> file
Simple and lightweight backgrounding of non-interactive processes

Disowning Jobs in Current Shell

The disown bash builtin can be used to remove a running process from the shell’s job control. This prevents the process from receiving SIGHUP signals when the parent shell exits.

To disown a running job, use:

disown -h %1

Where %1 is the jobspec of the process to disown.

jobs command lists running jobs and their jobspecs.

Some key points about disown:

Only works on jobs launched from the current shell
Use jobs to view running jobs and jobspecs
disown -h prevents process from receiving SIGHUP

Example disconnecting from an SSH session with a disowned process:

someprocess & # Start background process
jobs # Check jobspec number
disown -h %1 # Disown background process
exit # Disconnect SSH session

Systemd User Services for Persistence

Systemd user services can be created to start processes automatically when a user logs in. This provides a simple way to keep processes persistent across sessions.

To create a user service:

Create service file ~/.config/systemd/user/myprocess.service
Edit file to have ExecStart, Description and other directives
Reload systemd config: systemctl --user daemon-reload
Enable service: systemctl --user enable myprocess

An example file with essential directives:

[Unit]
Description=My Custom Process

[Service]
ExecStart=/usr/bin/python /home/user/myscript.py
Restart=always

[Install]  
WantedBy=default.target

Now myscript.py will be started on each login session due to WantedBy directive.

Automation with tmux

tmux is a terminal multiplexer similar to screen, allowing customized persistent sessions and detach capabilities.

Some ways tmux helps with process continuity:

Start new tmux session with tmux
Detach session with CTRL+B then D
Reattach with tmux attach
List sessions with tmux ls
Automate session creation with tmuxinator tool

Example workflow:

tmux start main session
vim file edit files within tmux
CTRL+B D detach tmux
ssh logout, reconnect
tmux attach rejoin session

This keeps vim editing session up across disconnects.

Best Practices

For interactive programs like vim/mutt/etc use screen or tmux
For one-off scripts or batch jobs use nohup or disown
For startup persistence use systemd user services

Additional tips:

Check log files for background job outputs
Use descriptive names for screen/tmux sessions
Enable automatic reconnection with ssh ControlPersist

Following these best practices will lead to smooth sailing even when disconnecting from difficult to terminate processes.