Persist sessions across server restarts via per-session holders

Make dosh terminal sessions survive a dosh-server restart
(crash/upgrade/systemctl restart): a reattaching client lands on the
SAME shell with screen state intact, instead of a fresh one.

Design: when persist_sessions is on (new config, default true) each
terminal session's shell runs in a small detached per-session holder
process (the dosh-server binary re-exec'd as `dosh-server hold`). The
holder double-forks + setsid into its own session, opens the PTY, spawns
the shell as ITS child, and serves the PTY master fd over a per-session
Unix socket via SCM_RIGHTS. The server adopts that fd to build a PtyHandle
that DETACHES (does not kill the shell) on drop, so a server exit leaves
holder + shell alive. On startup the server scans the runtime dir under
sessions_dir/run, reconnects to each live holder, receives the master fd
again, restores the persisted vt100 screen, and rebuilds the Session.

Screen/scrollback (server-memory only) is mirrored to disk atomically:
byte-throttled on the output hot path plus a 2s periodic flush, and
restored on re-adoption so reattach repaints. Truly-abandoned persistent
sessions are still reaped per the existing grace logic by asking the
holder to shut down. Anything in the persistent path failing degrades
gracefully to the original in-process shell.

PtyHandle gains an Owned (kill-on-drop, unchanged default) vs Adopted
(detach-on-drop) backing; resize on an adopted master uses TIOCSWINSZ.

No wire-format change, so protocol::VERSION stays 3. New deps: libc
(direct). New config: persist_sessions (default true); existing
integration tests pin it false to keep exercising the non-persistent
path unchanged.

Adds tests/integration_smoke.rs::session_survives_server_restart_same_
shell_and_screen: attaches, sets `cd /tmp; export MARK=...`, paints a
screen marker, KILLs the server, restarts it on the same config, reattaches
and asserts same shell pid + MARK + PWD + restored screen. Plus persist.rs
unit tests (name round-trip, screen save/load, dead-holder scan cleanup,
SCM_RIGHTS fd round-trip). cargo fmt --check and cargo test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

src/pty.rs

@@ -1,15 +1,36 @@
 use anyhow::{Context, Result};
 use portable_pty::{Child, CommandBuilder, MasterPty, NativePtySystem, PtySize, PtySystem};
 use std::io::{Read, Write};
+use std::os::unix::io::{AsRawFd, FromRawFd, RawFd};
 use std::path::Path;
 use std::sync::{Arc, Mutex};
 use std::thread;
 use tokio::sync::mpsc;
 
+/// Backing for a PTY master held by the server.
+///
+/// `Owned` means this process spawned the shell as a child and is responsible
+/// for it: dropping the handle kills the shell. This is the original,
+/// non-persistent model and stays the default.
+///
+/// `Adopted` means the shell lives in a separate holder process and this handle
+/// only borrows the master fd (received over a Unix socket via SCM_RIGHTS).
+/// Dropping it must NOT kill the shell — it just closes our copy of the fd and
+/// stops the reader thread, leaving the holder + shell alive so a server restart
+/// can re-adopt them.
+enum Backing {
+    Owned {
+        child: Mutex<Box<dyn Child + Send + Sync>>,
+        _master: Box<dyn MasterPty + Send>,
+    },
+    Adopted {
+        master: Mutex<std::fs::File>,
+    },
+}
+
 pub struct PtyHandle {
     writer: Arc<Mutex<Box<dyn Write + Send>>>,
-    child: Mutex<Box<dyn Child + Send + Sync>>,
-    _master: Box<dyn MasterPty + Send>,
+    backing: Backing,
 }
 
 impl PtyHandle {
@@ -21,23 +42,40 @@ impl PtyHandle {
     }
 
     pub fn resize(&self, cols: u16, rows: u16) -> Result<()> {
-        self._master.resize(PtySize {
-            rows,
-            cols,
-            pixel_width: 0,
-            pixel_height: 0,
-        })?;
+        match &self.backing {
+            Backing::Owned { _master, .. } => {
+                _master.resize(PtySize {
+                    rows,
+                    cols,
+                    pixel_width: 0,
+                    pixel_height: 0,
+                })?;
+            }
+            Backing::Adopted { master } => {
+                let file = master.lock().expect("pty master poisoned");
+                resize_fd(file.as_raw_fd(), cols, rows)?;
+            }
+        }
         Ok(())
     }
 
+    /// True for a handle backed by a separate holder process. Such a handle must
+    /// be detached (not killed) when the server lets go of a session, so the
+    /// shell survives a server restart.
+    pub fn is_persistent(&self) -> bool {
+        matches!(self.backing, Backing::Adopted { .. })
+    }
+
     /// Terminate the shell process backing this PTY and reap it.
     ///
-    /// Without this the child shell outlives the session: dropping the master
-    /// alone is not guaranteed to take the process down, and the `Child` handle
-    /// used to be discarded at spawn time, which leaked one shell per
-    /// abandoned session.
+    /// Only meaningful for an `Owned` backing (the server spawned the shell as a
+    /// child). For an `Adopted` backing the shell belongs to the holder process,
+    /// so this is a no-op here; reaping a persistent session is done by asking
+    /// the holder to shut down (see `persist::request_shutdown`).
     pub fn kill(&self) {
-        if let Ok(mut child) = self.child.lock() {
+        if let Backing::Owned { child, .. } = &self.backing
+            && let Ok(mut child) = child.lock()
+        {
             let _ = child.kill();
             let _ = child.wait();
         }
@@ -46,10 +84,29 @@ impl PtyHandle {
 
 impl Drop for PtyHandle {
     fn drop(&mut self) {
-        self.kill();
+        // Adopted handles must NOT kill the shell: it lives in the holder so it
+        // can outlive this server. Dropping just closes our fd / stops the
+        // reader. Owned handles keep the original kill-on-drop behavior.
+        if let Backing::Owned { .. } = &self.backing {
+            self.kill();
+        }
     }
 }
 
+fn resize_fd(fd: RawFd, cols: u16, rows: u16) -> Result<()> {
+    let winsize = libc::winsize {
+        ws_row: rows,
+        ws_col: cols,
+        ws_xpixel: 0,
+        ws_ypixel: 0,
+    };
+    let rc = unsafe { libc::ioctl(fd, libc::TIOCSWINSZ, &winsize) };
+    if rc != 0 {
+        return Err(std::io::Error::last_os_error()).context("TIOCSWINSZ");
+    }
+    Ok(())
+}
+
 #[derive(Debug)]
 pub struct PtyOutput {
     pub session: String,
@@ -74,6 +131,27 @@ pub fn spawn_pty_session(
             pixel_height: 0,
         })
         .context("open pty")?;
+    let cmd = build_shell_command(shell, env);
+    let child = pair.slave.spawn_command(cmd).context("spawn shell")?;
+    drop(pair.slave);
+
+    let writer = pair.master.take_writer().context("take pty writer")?;
+    let reader = pair.master.try_clone_reader().context("clone pty reader")?;
+    spawn_reader_thread(session, reader, tx)?;
+
+    Ok(PtyHandle {
+        writer: Arc::new(Mutex::new(writer)),
+        backing: Backing::Owned {
+            child: Mutex::new(child),
+            _master: pair.master,
+        },
+    })
+}
+
+/// Build the [`CommandBuilder`] for a dosh shell, identically for the in-process
+/// `spawn_pty_session` and the out-of-process holder, so a persistent session's
+/// environment matches a non-persistent one.
+pub fn build_shell_command(shell: &str, env: &[(String, String)]) -> CommandBuilder {
     let mut cmd = CommandBuilder::new(shell);
     cmd.env("TERM", "xterm-256color");
     cmd.env("COLORTERM", "truecolor");
@@ -88,11 +166,39 @@ pub fn spawn_pty_session(
     } else if let Some(parent) = Path::new(shell).parent() {
         cmd.env("PWD", parent.as_os_str());
     }
-    let child = pair.slave.spawn_command(cmd).context("spawn shell")?;
-    drop(pair.slave);
+    cmd
+}
 
-    let writer = pair.master.take_writer().context("take pty writer")?;
-    let mut reader = pair.master.try_clone_reader().context("clone pty reader")?;
+/// Build a [`PtyHandle`] from a master fd received from a holder process.
+///
+/// `master_fd` is an fd this handle takes ownership of (it is wrapped in a
+/// `File` and closed on drop). The shell is NOT a child of this process; it
+/// belongs to the holder, so dropping this handle leaves it running.
+pub fn adopt_pty_from_fd(
+    session: String,
+    master_fd: RawFd,
+    tx: mpsc::UnboundedSender<PtyOutput>,
+) -> Result<PtyHandle> {
+    // Take ownership of the fd. A clone gives us an independent reader so the
+    // reader thread and the writer/resize side hold separate `File`s and don't
+    // get closed out from under each other.
+    let master = unsafe { std::fs::File::from_raw_fd(master_fd) };
+    let reader_file = master.try_clone().context("clone master for reader")?;
+    let writer_file = master.try_clone().context("clone master for writer")?;
+    spawn_reader_thread(session, Box::new(reader_file), tx)?;
+    Ok(PtyHandle {
+        writer: Arc::new(Mutex::new(Box::new(writer_file))),
+        backing: Backing::Adopted {
+            master: Mutex::new(master),
+        },
+    })
+}
+
+fn spawn_reader_thread(
+    session: String,
+    mut reader: Box<dyn Read + Send>,
+    tx: mpsc::UnboundedSender<PtyOutput>,
+) -> Result<()> {
     let reader_session = session.clone();
     thread::Builder::new()
         .name(format!("dosh-pty-{session}"))
@@ -127,10 +233,5 @@ pub fn spawn_pty_session(
             }
         })
         .context("spawn pty reader")?;
-
-    Ok(PtyHandle {
-        writer: Arc::new(Mutex::new(writer)),
-        child: Mutex::new(child),
-        _master: pair.master,
-    })
+    Ok(())
 }