Persist sessions across server restarts via per-session holders

Make dosh terminal sessions survive a dosh-server restart
(crash/upgrade/systemctl restart): a reattaching client lands on the
SAME shell with screen state intact, instead of a fresh one.

Design: when persist_sessions is on (new config, default true) each
terminal session's shell runs in a small detached per-session holder
process (the dosh-server binary re-exec'd as `dosh-server hold`). The
holder double-forks + setsid into its own session, opens the PTY, spawns
the shell as ITS child, and serves the PTY master fd over a per-session
Unix socket via SCM_RIGHTS. The server adopts that fd to build a PtyHandle
that DETACHES (does not kill the shell) on drop, so a server exit leaves
holder + shell alive. On startup the server scans the runtime dir under
sessions_dir/run, reconnects to each live holder, receives the master fd
again, restores the persisted vt100 screen, and rebuilds the Session.

Screen/scrollback (server-memory only) is mirrored to disk atomically:
byte-throttled on the output hot path plus a 2s periodic flush, and
restored on re-adoption so reattach repaints. Truly-abandoned persistent
sessions are still reaped per the existing grace logic by asking the
holder to shut down. Anything in the persistent path failing degrades
gracefully to the original in-process shell.

PtyHandle gains an Owned (kill-on-drop, unchanged default) vs Adopted
(detach-on-drop) backing; resize on an adopted master uses TIOCSWINSZ.

No wire-format change, so protocol::VERSION stays 3. New deps: libc
(direct). New config: persist_sessions (default true); existing
integration tests pin it false to keep exercising the non-persistent
path unchanged.

Adds tests/integration_smoke.rs::session_survives_server_restart_same_
shell_and_screen: attaches, sets `cd /tmp; export MARK=...`, paints a
screen marker, KILLs the server, restarts it on the same config, reattaches
and asserts same shell pid + MARK + PWD + restored screen. Plus persist.rs
unit tests (name round-trip, screen save/load, dead-holder scan cleanup,
SCM_RIGHTS fd round-trip). cargo fmt --check and cargo test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

tests/hostile_network.rs

@@ -67,6 +67,7 @@ sessions_dir = "{sessions}"
 secret_path = "{secret}"
 host_key = "{host_key}"
 authorized_keys = ["{authorized_keys}"]
+persist_sessions = false
 "#,
             sessions = dir.path().join("sessions").display(),
             secret = dir.path().join("secret").display(),

tests/integration_smoke.rs

@@ -83,6 +83,7 @@ sessions_dir = "{sessions}"
 secret_path = "{secret}"
 host_key = "{host_key}"
 authorized_keys = ["{authorized_keys}"]
+persist_sessions = false
 "#,
             sessions = dir.path().join("sessions").display(),
             secret = dir.path().join("secret").display(),
@@ -1800,3 +1801,185 @@ fn native_agent_forwarding_requires_client_opt_in() {
         "agent proxy dir {agent_dir:?} must not exist without client -A opt-in"
     );
 }
+
+// ---------------------------------------------------------------------------
+// Session persistence across a server restart.
+// ---------------------------------------------------------------------------
+
+/// Write a server config with `persist_sessions = true` so each session's shell
+/// runs in a detached holder that survives a server restart.
+fn write_persistent_server_config(dir: &tempfile::TempDir, port: u16) -> std::path::PathBuf {
+    let config = write_server_config(dir, port);
+    let mut raw = fs::read_to_string(&config).unwrap();
+    // The base writer hardcodes `persist_sessions = false`; flip it on.
+    raw = raw.replace("persist_sessions = false", "persist_sessions = true");
+    fs::write(&config, raw).unwrap();
+    config
+}
+
+/// Mirror the server's hex encoding of a session name to its runtime dir.
+fn session_runtime_dir(sessions_dir: &Path, session: &str) -> PathBuf {
+    let mut hex = String::new();
+    for b in session.as_bytes() {
+        hex.push_str(&format!("{b:02x}"));
+    }
+    sessions_dir.join("run").join(hex)
+}
+
+/// Read a holder's recorded shell pid from its meta file, if present.
+fn holder_shell_pid(sessions_dir: &Path, session: &str) -> Option<i32> {
+    let meta = session_runtime_dir(sessions_dir, session).join("meta");
+    let body = fs::read_to_string(meta).ok()?;
+    body.lines().nth(1)?.parse().ok()
+}
+
+/// Tear down a persistent session's holder + shell so a test never leaks them
+/// (the TempDir drop would otherwise leave detached processes running).
+fn kill_holder(sessions_dir: &Path, session: &str) {
+    if let Some(pid) = holder_shell_pid(sessions_dir, session) {
+        let _ = Command::new("kill").arg("-9").arg(pid.to_string()).status();
+    }
+    // Best-effort: also kill any matching holder process for this runtime dir.
+    let dir = session_runtime_dir(sessions_dir, session);
+    let _ = Command::new("pkill")
+        .arg("-9")
+        .arg("-f")
+        .arg(format!("dosh-server hold --runtime-dir {}", dir.display()))
+        .status();
+}
+
+/// Send one encrypted Input line and give the shell a moment to act.
+fn type_line(socket: &UdpSocket, port: u16, ok: &AttachOk, key: &[u8; 32], seq: u64, line: &str) {
+    let input = Input {
+        bytes: line.as_bytes().to_vec(),
+    };
+    send_encrypted(
+        socket,
+        port,
+        PacketKind::Input,
+        ok.client_id,
+        seq,
+        0,
+        key,
+        &protocol::to_body(&input).unwrap(),
+    );
+    thread::sleep(Duration::from_millis(150));
+}
+
+/// Proof of survival: a session's shell + state + screen outlive a full
+/// `dosh-server` process restart.
+///
+/// We attach, set durable shell state (`cd /tmp`, `export MARK=...`) and paint a
+/// recognizable marker on the screen, wait for the screen to be mirrored to disk,
+/// then KILL the server process (simulating a crash/upgrade). We restart the
+/// server on the same config/sockets, reattach, and assert we land on the SAME
+/// shell — the exported variable and working directory are still there, and the
+/// restored attach snapshot still shows the pre-restart screen — not a fresh
+/// shell.
+#[test]
+fn session_survives_server_restart_same_shell_and_screen() {
+    let dir = tempfile::tempdir().unwrap();
+    let sessions_dir = dir.path().join("sessions");
+    let port = free_udp_port();
+    let config = write_persistent_server_config(&dir, port);
+
+    let mut server = start_server(&dir, &config);
+
+    // First attach: establish durable shell state and a screen marker.
+    let (socket, bootstrap, ok) = direct_attach(&config, port, "read-write");
+    let key = bootstrap.session_key;
+    type_line(&socket, port, &ok, &key, 2, "cd /tmp\n");
+    type_line(
+        &socket,
+        port,
+        &ok,
+        &key,
+        3,
+        "export MARK=zebra_persist_42\n",
+    );
+    // Paint a unique, stable marker into the screen (printf, no trailing newline
+    // scroll surprises) so the restored snapshot is easy to recognize.
+    type_line(
+        &socket,
+        port,
+        &ok,
+        &key,
+        4,
+        "printf 'PRE_RESTART_SCREEN_MARKER\\n'\n",
+    );
+    // Drain output so the parser/screen state and disk mirror catch up.
+    let pre = collect_frame_text(&socket, &key, 1500);
+    assert!(
+        pre.contains("PRE_RESTART_SCREEN_MARKER"),
+        "expected to see the screen marker before restart, got {pre:?}"
+    );
+    // The periodic flush mirrors the screen every ~2s; wait long enough that the
+    // pre-restart screen is guaranteed on disk before we crash the server.
+    thread::sleep(Duration::from_secs(3));
+
+    let shell_pid_before = holder_shell_pid(&sessions_dir, "default");
+    assert!(
+        shell_pid_before.is_some(),
+        "a holder shell pid should be recorded for a persistent session"
+    );
+
+    // CRASH: kill the server process outright (its in-memory state is lost; only
+    // the holder + shell + on-disk screen remain).
+    let _ = server.kill();
+    let _ = server.wait();
+    thread::sleep(Duration::from_millis(300));
+
+    // The holder + shell must still be alive after the server died.
+    let pid = shell_pid_before.unwrap();
+    let alive = unsafe { libc::kill(pid, 0) } == 0;
+    assert!(
+        alive,
+        "shell pid {pid} must survive the server crash (holder keeps it alive)"
+    );
+
+    // RESTART on the same config/sockets; the server re-adopts the holder.
+    let mut server = start_server(&dir, &config);
+
+    // Reattach. A fresh bootstrap attach lands on the EXISTING re-adopted
+    // session (same name), not a new shell.
+    let (socket2, bootstrap2, ok2) = direct_attach(&config, port, "read-write");
+    let key2 = bootstrap2.session_key;
+
+    // 1) Screen survived: the restored attach snapshot still shows the marker.
+    let snapshot = String::from_utf8_lossy(&ok2.snapshot).to_string();
+
+    // 2) Shell identity + state survived: ask the SAME shell for its state.
+    type_line(
+        &socket2,
+        port,
+        &ok2,
+        &key2,
+        2,
+        "printf 'MARK=%s PWD=%s\\n' \"$MARK\" \"$PWD\"\n",
+    );
+    let post = collect_frame_text(&socket2, &key2, 2000);
+
+    let shell_pid_after = holder_shell_pid(&sessions_dir, "default");
+
+    // Clean up BEFORE asserting so a failed assert never leaks the holder.
+    let _ = server.kill();
+    let _ = server.wait();
+    kill_holder(&sessions_dir, "default");
+
+    assert_eq!(
+        shell_pid_after, shell_pid_before,
+        "re-adopted session must be backed by the same shell pid, not a fresh one"
+    );
+    assert!(
+        post.contains("MARK=zebra_persist_42"),
+        "exported variable must survive the restart (same shell), got {post:?}"
+    );
+    assert!(
+        post.contains("PWD=/tmp"),
+        "working directory must survive the restart (same shell), got {post:?}"
+    );
+    assert!(
+        snapshot.contains("PRE_RESTART_SCREEN_MARKER"),
+        "restored attach snapshot must repaint the pre-restart screen, got {snapshot:?}"
+    );
+}