dosh/docs/BENCHMARKS.md

# Dosh Benchmarks

This document explains how to run the Dosh benchmarks, what each metric means,
and how to read the results honestly. Dosh's headline claim is *insane speed on
repeat attach/reconnect* — these benchmarks exist to make that claim measurable
and reproducible.

> **Read first:** `docs/PUBLIC_READINESS.md`. The defensible public claim is fast
> attach/reconnect, not full Mosh feature parity, and **not** cold startup. Cite
> `dosh_cached_attach_ms` for the core speed claim; cite cold metrics only to show
> the fallback path stays competitive with ordinary SSH.

## TL;DR

```bash
make bench-local          # safe, self-contained matrix on a throwaway server
make bench-local-json     # same, machine-readable JSON with raw samples
make bench-docker-ssh     # containerized SSH-vs-Dosh gate used by CI
make bench-docker-mosh    # same, with Mosh installed for a three-way comparison
make bench-report         # Markdown report wrapper around dosh-bench
```

`make bench-local` never touches a running production server: it builds release
binaries, starts its own `dosh-server` bound to `127.0.0.1` on a random free UDP
port inside a temporary `HOME`, runs the matrix, prints raw samples plus summary
statistics, and tears everything down on exit. It will refuse to bind UDP port
`50000` (the default production port).

## The benchmark binary: `dosh-bench`

`dosh-bench` spawns the real `dosh-client` once per iteration and times the whole
process from launch to terminal-ready-and-detached. That wall-clock cost is the
apples-to-apples comparison against `ssh host true`: it is the startup price each
tool pays before any useful remote work begins.

Every run prints, per metric, a summary line (count, min, median, p95, mean, max
in milliseconds) **and** a raw-samples line. Pass `--json` for one machine-readable
object per run (with the full `samples_ms` array) so published numbers can always
include raw data, as required by `docs/PUBLIC_READINESS.md`. Pass `--markdown` for a
publishable Markdown table, and `--output path/to/report.md` to write it to disk.

### Path matrix

`dosh-bench` can benchmark these Dosh attach paths. Without an explicit path flag
it keeps its legacy single-path behavior (driven by `--local-auth` / `--warm-cache`
/ `--no-cache`) so existing CI scripts keep working.

| Flag | Metric | What it measures |
| --- | --- | --- |
| `--cold-native` | `dosh_cold_native_ms` | Native cold auth: full `--auth native --no-cache` handshake to first frame and detach. The cold fallback when no cache exists. |
| `--cached-ticket` | `dosh_cached_attach_ms` | Cached attach-ticket fast path. Warms the cache once, then measures repeat attaches using cached UDP credentials/tickets. **This is the core speed claim.** |
| `--resume` | `dosh_resume_ms` | UDP resume of a session whose endpoint changes. See the caveat below — only meaningful when a live session is kept open out-of-band. |
| `--local-auth` | `dosh_local_attach_ms` | Self-contained local bootstrap; no SSH, no native handshake. Useful as a lower-bound sanity check on loopback. |
| *(default, no flag)* | `dosh_attach_ms` | Cold SSH bootstrap plus UDP attach (legacy single-path mode). |

Existing baseline/comparison metrics are unchanged:

| Metric | Meaning |
| --- | --- |
| `ssh_true_ms` | `ssh host true` with the same key/options. |
| `mosh_start_true_ms` | `mosh host -- true` bootstrap, run `true`, exit (timed inside a PTY). |

### Assertions / gates

| Flag | Gate |
| --- | --- |
| `--assert-ssh-plus-ms N` | The primary Dosh metric must be ≤ `ssh_true_ms` mean + `N` ms. |
| `--assert-mosh-minus-ms N` | The primary Dosh metric must be at least `N` ms faster than `mosh_start_true_ms`. |
| `--assert-dosh-max-ms N` | The primary Dosh metric mean must be ≤ `N` ms. |

The "primary Dosh metric" for assertions is, in priority order:
`dosh_cached_attach_ms` → `dosh_resume_ms` → `dosh_cold_native_ms` →
`dosh_attach_ms` → `dosh_local_attach_ms`.

## What each benchmark does

### `make bench-local` (`scripts/bench-local.sh`)

Self-contained, no SSH, no Docker. It:

1. Builds release binaries (`DOSH_BENCH_PROFILE=debug` for the debug profile).
2. Picks a free UDP port (never `50000`) and a temp `HOME`.
3. Generates a throwaway client identity (`ssh-keygen`), authorizes it on the
   server, and writes a throwaway server + client config (native auth,
   trust-on-first-use, localhost UDP).
4. Starts `dosh-server serve` bound to `127.0.0.1` on that port.
5. Runs `dosh-bench --cold-native --cached-ticket`, then `--local-auth --no-cache`.
6. Sanity-checks that native cold auth actually trusted the host (so a silent
   config-parse fallback can't make the benchmark measure the wrong path).
7. Kills the server and removes the temp `HOME` on exit.

Tunables: `scripts/bench-local.sh [ITERATIONS]` (default 20), `DOSH_BENCH_JSON=1`,
`DOSH_BENCH_PROFILE=release|debug`.

### `make bench-docker-ssh` / `make bench-docker-mosh` (`scripts/ci-docker-ssh-bench.sh`)

Builds one Ubuntu image with OpenSSH, `dosh-server`, `dosh-auth` (and Mosh for the
`-mosh` target), then runs `ssh_true_ms`, cold + ControlMaster SSH-bootstrap
`dosh_attach_ms`, one-time `dosh trust` followed by native-cold
`dosh_cold_native_ms`, `dosh_cached_attach_ms`, and optionally
`mosh_start_true_ms` against the same container, key, and loopback network path.
This is the gate CI enforces; it asserts cold SSH-bootstrap Dosh stays within
500 ms of SSH, native-cold Dosh beats SSH mean by default, and cached attach stays
under a small budget. See `README.md` "Develop" for the exact invocations.

## Methodology and caveats

- **Same machine, same network path.** `bench-local` runs everything on loopback,
  so it isolates Dosh's own process/handshake/render overhead and removes network
  RTT from the comparison. Real-world cached attach is approximately *network RTT
  plus the local overhead these numbers measure*. Publish the machine, OS, CPU,
  network path, and sample count alongside any numbers.
- **Wall-clock of the whole client process.** Each sample includes process spawn,
  attach, first-frame render, and detach. That is deliberately the same thing
  `ssh host true` pays, but it means a few ms is fixed process-startup cost, not
  protocol cost.
- **Do not cite cold metrics as the headline.** `dosh_cold_native_ms` and
  `dosh_attach_ms` still pay first-authentication cost. They exist to show the
  fallback path stays competitive with ordinary SSH. Cite `dosh_cached_attach_ms`
  for repeat attach/reconnect, per `docs/PUBLIC_READINESS.md`.
- **UDP resume is the roaming path, not cold reconnect.** Resume reconnects a
  client that is *still attached* when its network endpoint changes (sleep/Wi-Fi
  switch). The `--attach-only` benchmark model detaches after every iteration,
  which removes the client on the server, so a fresh-process resume has nothing
  live to resume and is not meaningful in `bench-local`. The cold fresh-process
  reconnect fast path is the attach ticket (`dosh_cached_attach_ms`). Roaming
  resume correctness is covered by the integration test
  `resume_updates_udp_endpoint_for_roaming` in `tests/integration_smoke.rs`.
  `dosh-bench --resume` remains available for scenarios that keep a live session
  open out-of-band (for example remote soak tests).
- **These are not identical workloads.** SSH/Mosh/Dosh do different work at
  startup. The comparison is still useful because it measures the startup tax each
  tool charges before useful remote work begins.

## Sample results (loopback, self-contained)

Captured with `make bench-local` (`scripts/bench-local.sh 20`), all times in
milliseconds, 20 samples per metric.

- Machine: Intel Core i5-9500 @ 3.00 GHz, 6 cores
- OS: Ubuntu 24.04.4 LTS, Linux 6.8.0-124-generic, x86_64
- Toolchain: rustc 1.96.0, release profile
- Network path: loopback (`127.0.0.1`), throwaway server on a free UDP port
- Workload: `dosh-client --attach-only` to first frame and detach

| Metric | n | min | median | p95 | mean | max |
| --- | --- | --- | --- | --- | --- | --- |
| `dosh_cold_native_ms` | 20 | 8.32 | 8.77 | 9.34 | 8.82 | 9.52 |
| `dosh_cached_attach_ms` | 20 | 2.83 | 3.23 | 3.75 | 3.24 | 3.77 |
| `dosh_local_attach_ms` | 20 | 2.84 | 3.21 | 3.96 | 3.28 | 3.98 |

Raw samples (ms):

```
dosh_cold_native_ms:
8.82, 8.60, 8.69, 8.71, 9.33, 8.82, 8.90, 8.64, 8.39, 8.75, 8.79, 8.70, 8.64,
9.52, 8.72, 8.32, 9.06, 8.93, 8.96, 9.04

dosh_cached_attach_ms:
3.04, 3.06, 2.98, 2.91, 2.96, 2.95, 3.05, 3.66, 3.37, 2.88, 2.83, 3.16, 3.49,
3.30, 3.77, 3.32, 3.49, 3.75, 3.56, 3.33

dosh_local_attach_ms:
3.08, 3.96, 3.36, 3.30, 2.88, 2.99, 3.96, 3.98, 3.25, 2.84, 3.29, 3.59, 3.65,
3.47, 3.13, 2.92, 2.87, 2.91, 2.98, 3.18
```

Reading these numbers: cold native auth is ~9 ms because it pays the native
handshake; cached attach is ~3 ms because it reuses cached credentials and skips
the handshake entirely. On loopback there is no network RTT, so this is the local
overhead floor. Over a real link, expect cached attach ≈ this floor + one network
round trip — which is exactly the "near network RTT" target in the spec. There is
no SSH/Mosh baseline in this loopback table because those paths need an SSH server;
use `make bench-docker-ssh` / `make bench-docker-mosh` for the head-to-head.

## Sample results (Docker SSH/Mosh comparison)

Captured with `make bench-docker-mosh`, all times in milliseconds. This is the
same-container comparison gate: one generated key, one OpenSSH server, one
`dosh-server`, loopback-published TCP/UDP ports.

| Metric | n | min | median | p95 | mean | max |
| --- | --- | --- | --- | --- | --- | --- |
| `ssh_true_ms` (cold SSH baseline for native-cold gate) | 5 | 214.53 | 216.84 | 220.55 | 217.60 | 220.91 |
| `dosh_cold_native_ms` | 5 | 8.62 | 8.96 | 9.78 | 9.15 | 9.88 |
| `dosh_cached_attach_ms` | 10 | 8.41 | 8.83 | 9.52 | 8.90 | 9.53 |
| `mosh_start_true_ms` | 3 | 528.03 | 530.87 | 539.23 | 533.02 | 540.16 |

Raw samples (ms):

```
ssh_true_ms (native-cold gate):
216.84, 216.61, 214.53, 220.91, 219.09

dosh_cold_native_ms:
9.39, 8.88, 8.96, 8.62, 9.88

dosh_cached_attach_ms:
8.44, 8.41, 8.80, 8.48, 9.53, 9.51, 8.85, 8.69, 9.33, 8.96

mosh_start_true_ms:
540.16, 528.03, 530.87
```