Job Control Techniques For Suspended Ssh Processes

Keeping SSH Sessions Active After Disconnection

The SSH client can be configured to keep sessions open with the ClientAliveInterval setting. This will send keepalive packets periodically to prevent the server from dropping the connection due to inactivity. Using nohup and appending & will allow processes to continue running even after disconnecting from an SSH session. Screen and tmux are alternative terminal multiplexers that can detach sessions while keeping processes active.

Using nohup and & to keep processes running after SSH disconnects

The nohup command will ignore hangup signals sent to processes when the SSH client disconnects. Appending & will run the process in the background. Combined, nohup and & allow users to start long-running processes over SSH that will continue independently of client connectivity.

Configuring SSH to keep sessions open with ClientAliveInterval

The ClientAliveInterval setting in SSH config files controls how often the client will send keepalive messages to the server. Setting this to a reasonable interval will stop the server from terminating SSH sessions due to inactivity while still allowing disconnection of unused connections. This provides persistence for long-running SSH connections without relying on external tools.

Using screen and tmux for session persistence

Screen and tmux are terminal multiplexers that enable multiple console sessions over a single SSH connection. Detaching from a multiplexed session will keep all running programs active and allow later reattachment even from a different client location. This approach avoids directly configuring SSH keepalive packets but provides similar session persistence.

Reattaching to background processes with fg and bg

The fg and bg commands move processes between foreground and background execution on the remote server. Background processes started with bg will persist independently of the SSH client. The fg command brings the most recently backgrounded process back to the foreground, allowing interactive usage after reconnecting to the server over SSH.

Automating Reconnections with AutoSSH

AutoSSH is a tool that monitors SSH connections and automatically restarts them when disrupted. It can be configured for key-based authentication to simplify reconnection. Running commands with AutoSSH will execute them persistently across any SSH disconnects.

How AutoSSH reconnects SSH sessions automatically

AutoSSH works by establishing a monitoring SSH process that keeps pinging the server. If the main connection goes down for any reason, the monitor process detects this and restarts the connection. This automation keeps SSH sessions persistent in cases where manual recovery would be difficult.

Configuring key-based authentication for AutoSSH

Using SSH keys allows AutoSSH to reconnect without entering passwords. The public key gets installed on the remote server, while the private key gets referenced in the AutoSSH client configuration. With keys set up, AutoSSH can keep SSH sessions alive without any user intervention.

Running commands and processes persistently with AutoSSH

The AutoSSH client includes options to launch commands and processes on connection. Even if the underlying SSH session temporarily goes down, AutoSSH will restart it and rerun anything defined to execute on connect. This keeps essential programs running continuously through inevitable network disruptions.

Managing Remote Processes with Disown

The disown builtin command in bash can detach processes from their parent shell session. This allows programs to continue running remotely even after terminating an SSH connection. Disowning processes is conceptually similar but functionally different from nohup.

How disown detaches processes from terminal sessions

By default, processes end when their parent shell exits. Disown removes a process from the shell’s job control mechanism, allowing it to persist independently. The process will no longer react to terminal signals and will run detached in the background after disowning.

Differences between nohup and disown

While both nohup and disown allow persisting processes past SSH disconnects, nohup also ignores hangup signals. Disowned processes will still react to hangups. Additionally, disown affects running jobs while nohup must be used when first launching programs. The commands can be combined for maximum process persistence.

Disowning running processes without interrupting them

Jobs can be disowned in an atomic way by using & to background the process first before disassociating it from the shell with disown. This avoids any terminal interruptions which could destabilize databases, long-running computations or other sensitive applications.

Debugging Suspended SSH Sessions

Diagnosing issues with disconnected SSH processes relies on commands like ps to identify the affected programs. Attempts to forcibly kill orphaned processes could disrupt active child jobs. Redirecting input and output streams is necessary to capture information for debugging.

Identifying disconnected processes with ps

The ps command shows status details about currently running processes, including those no longer associated with terminals after SSH disconnects. grepping the ps output for sshd child processes will display information like PID needed to further inspect and manage the jobs.

Issues killing orphaned SSH child processes

Care must taken when attempting to kill processes disconnected from SSH sessions, as they may have active child processes still running important jobs. Forcibly terminating the top-level parent process could unintentionally destabilize other essential applications.

Checking process output with redirecting stdin, stdout, stderr

To troubleshoot issues with suspended processes, input and output streams can be redirected to files that will persist independently of SSH connectivity. This log data provides diagnostics to analyze errors even if no terminal is attached.

Recovering Unintentionally Suspended Processes

Techniques like using terminal multiplexers or configuring background notifications allow users to easily reattach sessions with suspended jobs. When recovering job control is impossible, care must be taken on process restart to avoid losing data.

Techniques for reattaching suspended screen and tmux sessions

Screen and tmux simplify recovering disconnected jobs with features listing all detached sessions. Attaching to an existing session restores access to its processes where they left off after SSH disconnects. This avoids losing the context of long-running jobs.

Recovering suspended jobs with fg, bg and kill commands

The fg and bg commands bring suspended jobs back into the terminal foreground and background respectively. In severe cases where processes are entirely orphaned, the kill signal can provide a structured restart.

Avoiding loss of data from disconnected processes

Redirecting a processes’ output to a file before forcing a restart provides a recovery mechanism for any buffered data. Carefully terminating the tree of orphaned child processes can also maximize recovering of lost application state data.