dosh/docs/BENCHMARKS.md

# Dosh Benchmarks

This document explains how to run the Dosh benchmarks, what each metric means,
and how to read the results honestly. Dosh's headline claim is *insane speed on
repeat attach/reconnect* — these benchmarks exist to make that claim measurable
and reproducible.

> **Read first:** `docs/PUBLIC_READINESS.md`. The defensible public claim is fast
> attach/reconnect, not full Mosh feature parity, and **not** cold startup. Cite
> `dosh_cached_attach_ms` for the core speed claim; cite cold metrics only to show
> the fallback path stays competitive with ordinary SSH.

## TL;DR

```bash
make bench-local          # safe, self-contained matrix on a throwaway server
make bench-local-json     # same, machine-readable JSON with raw samples
make bench-docker-ssh     # containerized SSH-vs-Dosh gate used by CI
make bench-docker-mosh    # same, with Mosh installed for a three-way comparison
```

`make bench-local` never touches a running production server: it builds release
binaries, starts its own `dosh-server` bound to `127.0.0.1` on a random free UDP
port inside a temporary `HOME`, runs the matrix, prints raw samples plus summary
statistics, and tears everything down on exit. It will refuse to bind UDP port
`50000` (the default production port).

## The benchmark binary: `dosh-bench`

`dosh-bench` spawns the real `dosh-client` once per iteration and times the whole
process from launch to terminal-ready-and-detached. That wall-clock cost is the
apples-to-apples comparison against `ssh host true`: it is the startup price each
tool pays before any useful remote work begins.

Every run prints, per metric, a summary line (count, min, median, p95, mean, max
in milliseconds) **and** a raw-samples line. Pass `--json` for one machine-readable
object per run (with the full `samples_ms` array) so published numbers can always
include raw data, as required by `docs/PUBLIC_READINESS.md`.

### Path matrix

`dosh-bench` can benchmark these Dosh attach paths. Without an explicit path flag
it keeps its legacy single-path behavior (driven by `--local-auth` / `--warm-cache`
/ `--no-cache`) so existing CI scripts keep working.

| Flag | Metric | What it measures |
| --- | --- | --- |
| `--cold-native` | `dosh_cold_native_ms` | Native cold auth: full `--auth native --no-cache` handshake to first frame and detach. The cold fallback when no cache exists. |
| `--cached-ticket` | `dosh_cached_attach_ms` | Cached attach-ticket fast path. Warms the cache once, then measures repeat attaches using cached UDP credentials/tickets. **This is the core speed claim.** |
| `--resume` | `dosh_resume_ms` | UDP resume of a session whose endpoint changes. See the caveat below — only meaningful when a live session is kept open out-of-band. |
| `--local-auth` | `dosh_local_attach_ms` | Self-contained local bootstrap; no SSH, no native handshake. Useful as a lower-bound sanity check on loopback. |
| *(default, no flag)* | `dosh_attach_ms` | Cold SSH bootstrap plus UDP attach (legacy single-path mode). |

Existing baseline/comparison metrics are unchanged:

| Metric | Meaning |
| --- | --- |
| `ssh_true_ms` | `ssh host true` with the same key/options. |
| `mosh_start_true_ms` | `mosh host -- true` bootstrap, run `true`, exit (timed inside a PTY). |

### Assertions / gates

| Flag | Gate |
| --- | --- |
| `--assert-ssh-plus-ms N` | The primary Dosh metric must be ≤ `ssh_true_ms` mean + `N` ms. |
| `--assert-mosh-minus-ms N` | The primary Dosh metric must be at least `N` ms faster than `mosh_start_true_ms`. |
| `--assert-dosh-max-ms N` | The primary Dosh metric mean must be ≤ `N` ms. |

The "primary Dosh metric" for assertions is, in priority order:
`dosh_cached_attach_ms` → `dosh_resume_ms` → `dosh_cold_native_ms` →
`dosh_attach_ms` → `dosh_local_attach_ms`.

## What each benchmark does

### `make bench-local` (`scripts/bench-local.sh`)

Self-contained, no SSH, no Docker. It:

1. Builds release binaries (`DOSH_BENCH_PROFILE=debug` for the debug profile).
2. Picks a free UDP port (never `50000`) and a temp `HOME`.
3. Generates a throwaway client identity (`ssh-keygen`), authorizes it on the
   server, and writes a throwaway server + client config (native auth,
   trust-on-first-use, localhost UDP).
4. Starts `dosh-server serve` bound to `127.0.0.1` on that port.
5. Runs `dosh-bench --cold-native --cached-ticket`, then `--local-auth --no-cache`.
6. Sanity-checks that native cold auth actually trusted the host (so a silent
   config-parse fallback can't make the benchmark measure the wrong path).
7. Kills the server and removes the temp `HOME` on exit.

Tunables: `scripts/bench-local.sh [ITERATIONS]` (default 20), `DOSH_BENCH_JSON=1`,
`DOSH_BENCH_PROFILE=release|debug`.

### `make bench-docker-ssh` / `make bench-docker-mosh` (`scripts/ci-docker-ssh-bench.sh`)

Builds one Ubuntu image with OpenSSH, `dosh-server`, `dosh-auth` (and Mosh for the
`-mosh` target), then runs `ssh_true_ms`, cold + ControlMaster `dosh_attach_ms`,
`dosh_cached_attach_ms`, and optionally `mosh_start_true_ms` against the same
container, key, and loopback network path. This is the gate CI enforces; it asserts
cold Dosh stays within 500 ms of SSH and that cached attach stays under a small
budget. See `README.md` "Develop" for the exact invocations.

## Methodology and caveats

- **Same machine, same network path.** `bench-local` runs everything on loopback,
  so it isolates Dosh's own process/handshake/render overhead and removes network
  RTT from the comparison. Real-world cached attach is approximately *network RTT
  plus the local overhead these numbers measure*. Publish the machine, OS, CPU,
  network path, and sample count alongside any numbers.
- **Wall-clock of the whole client process.** Each sample includes process spawn,
  attach, first-frame render, and detach. That is deliberately the same thing
  `ssh host true` pays, but it means a few ms is fixed process-startup cost, not
  protocol cost.
- **Do not cite cold metrics as the headline.** `dosh_cold_native_ms` and
  `dosh_attach_ms` still pay first-authentication cost. They exist to show the
  fallback path stays competitive with ordinary SSH. Cite `dosh_cached_attach_ms`
  for repeat attach/reconnect, per `docs/PUBLIC_READINESS.md`.
- **UDP resume is the roaming path, not cold reconnect.** Resume reconnects a
  client that is *still attached* when its network endpoint changes (sleep/Wi-Fi
  switch). The `--attach-only` benchmark model detaches after every iteration,
  which removes the client on the server, so a fresh-process resume has nothing
  live to resume and is not meaningful in `bench-local`. The cold fresh-process
  reconnect fast path is the attach ticket (`dosh_cached_attach_ms`). Roaming
  resume correctness is covered by the integration test
  `resume_updates_udp_endpoint_for_roaming` in `tests/integration_smoke.rs`.
  `dosh-bench --resume` remains available for scenarios that keep a live session
  open out-of-band (for example remote soak tests).
- **These are not identical workloads.** SSH/Mosh/Dosh do different work at
  startup. The comparison is still useful because it measures the startup tax each
  tool charges before useful remote work begins.

## Sample results (loopback, self-contained)

Captured with `make bench-local` (`scripts/bench-local.sh 30`), all times in
milliseconds, 30 samples per metric.

- Machine: Intel Core i5-9500 @ 3.00 GHz, 6 cores
- OS: Ubuntu 24.04.4 LTS, Linux 6.8.0-124-generic, x86_64
- Toolchain: rustc 1.96.0, release profile
- Network path: loopback (`127.0.0.1`), throwaway server on a free UDP port
- Workload: `dosh-client --attach-only` to first frame and detach

| Metric | n | min | median | p95 | mean | max |
| --- | --- | --- | --- | --- | --- | --- |
| `dosh_cold_native_ms` | 30 | 8.10 | 9.01 | 10.40 | 9.18 | 10.82 |
| `dosh_cached_attach_ms` | 30 | 2.73 | 2.96 | 3.25 | 2.96 | 3.30 |
| `dosh_local_attach_ms` | 30 | 2.66 | 2.78 | 3.24 | 2.87 | 3.33 |

Raw samples (ms):

```
dosh_cold_native_ms:
8.32, 8.23, 10.31, 10.37, 8.83, 8.81, 9.20, 9.04, 10.14, 8.98, 10.82, 9.87,
10.22, 10.42, 9.76, 9.09, 9.19, 8.94, 8.69, 8.67, 8.74, 9.33, 9.25, 8.92, 8.39,
8.55, 8.38, 9.81, 8.17, 8.10

dosh_cached_attach_ms:
2.99, 2.99, 2.93, 2.87, 2.95, 2.80, 2.80, 2.90, 3.10, 2.73, 3.19, 2.80, 2.87,
2.82, 2.89, 3.03, 3.06, 3.21, 2.97, 2.83, 3.00, 2.82, 3.04, 2.97, 3.03, 3.24,
3.30, 2.73, 2.79, 3.25

dosh_local_attach_ms:
3.05, 3.00, 2.84, 3.04, 3.33, 2.89, 2.76, 2.74, 2.73, 3.32, 3.14, 2.75, 2.83,
2.88, 2.99, 2.77, 2.75, 2.79, 2.69, 2.78, 2.75, 2.77, 2.75, 2.77, 3.03, 2.75,
2.94, 2.74, 2.82, 2.66
```

Reading these numbers: cold native auth is ~9 ms because it pays the native
handshake; cached attach is ~3 ms because it reuses cached credentials and skips
the handshake entirely. On loopback there is no network RTT, so this is the local
overhead floor. Over a real link, expect cached attach ≈ this floor + one network
round trip — which is exactly the "near network RTT" target in the spec. There is
no SSH/Mosh baseline in this loopback table because those paths need an SSH server;
use `make bench-docker-ssh` / `make bench-docker-mosh` for the head-to-head.