dosh/docs/BENCHMARKS.md

Dosh Benchmarks

This document explains how to run the Dosh benchmarks, what each metric means, and how to read the results honestly. Dosh's headline claim is insane speed on repeat attach/reconnect — these benchmarks exist to make that claim measurable and reproducible.

Read first: docs/PUBLIC_READINESS.md. The defensible public claim is fast attach/reconnect, not full Mosh feature parity, and not cold startup. Cite dosh_cached_attach_ms for the core speed claim; cite cold metrics only to show the fallback path stays competitive with ordinary SSH.

TL;DR

make bench-local          # safe, self-contained matrix on a throwaway server
make bench-local-json     # same, machine-readable JSON with raw samples
make bench-docker-ssh     # containerized SSH-vs-Dosh gate used by CI
make bench-docker-mosh    # same, with Mosh installed for a three-way comparison

make bench-local never touches a running production server: it builds release binaries, starts its own dosh-server bound to 127.0.0.1 on a random free UDP port inside a temporary HOME, runs the matrix, prints raw samples plus summary statistics, and tears everything down on exit. It will refuse to bind UDP port 50000 (the default production port).

The benchmark binary: dosh-bench

dosh-bench spawns the real dosh-client once per iteration and times the whole process from launch to terminal-ready-and-detached. That wall-clock cost is the apples-to-apples comparison against ssh host true: it is the startup price each tool pays before any useful remote work begins.

Every run prints, per metric, a summary line (count, min, median, p95, mean, max in milliseconds) and a raw-samples line. Pass --json for one machine-readable object per run (with the full samples_ms array) so published numbers can always include raw data, as required by docs/PUBLIC_READINESS.md.

Path matrix

dosh-bench can benchmark these Dosh attach paths. Without an explicit path flag it keeps its legacy single-path behavior (driven by --local-auth / --warm-cache / --no-cache) so existing CI scripts keep working.

Flag	Metric	What it measures
--cold-native	dosh_cold_native_ms	Native cold auth: full --auth native --no-cache handshake to first frame and detach. The cold fallback when no cache exists.
--cached-ticket	dosh_cached_attach_ms	Cached attach-ticket fast path. Warms the cache once, then measures repeat attaches using cached UDP credentials/tickets. This is the core speed claim.
--resume	dosh_resume_ms	UDP resume of a session whose endpoint changes. See the caveat below — only meaningful when a live session is kept open out-of-band.
--local-auth	dosh_local_attach_ms	Self-contained local bootstrap; no SSH, no native handshake. Useful as a lower-bound sanity check on loopback.
(default, no flag)	dosh_attach_ms	Cold SSH bootstrap plus UDP attach (legacy single-path mode).

Existing baseline/comparison metrics are unchanged:

Metric	Meaning
ssh_true_ms	ssh host true with the same key/options.
mosh_start_true_ms	mosh host -- true bootstrap, run true, exit (timed inside a PTY).

Assertions / gates

Flag	Gate
--assert-ssh-plus-ms N	The primary Dosh metric must be ≤ ssh_true_ms mean + N ms.
--assert-mosh-minus-ms N	The primary Dosh metric must be at least N ms faster than mosh_start_true_ms.
--assert-dosh-max-ms N	The primary Dosh metric mean must be ≤ N ms.

The "primary Dosh metric" for assertions is, in priority order: dosh_cached_attach_ms → dosh_resume_ms → dosh_cold_native_ms → dosh_attach_ms → dosh_local_attach_ms.

What each benchmark does

make bench-local (scripts/bench-local.sh)

Self-contained, no SSH, no Docker. It:

1. Builds release binaries (DOSH_BENCH_PROFILE=debug for the debug profile).
2. Picks a free UDP port (never 50000) and a temp HOME.
3. Generates a throwaway client identity (ssh-keygen), authorizes it on the server, and writes a throwaway server + client config (native auth, trust-on-first-use, localhost UDP).
4. Starts dosh-server serve bound to 127.0.0.1 on that port.
5. Runs dosh-bench --cold-native --cached-ticket, then --local-auth --no-cache.
6. Sanity-checks that native cold auth actually trusted the host (so a silent config-parse fallback can't make the benchmark measure the wrong path).
7. Kills the server and removes the temp HOME on exit.

Tunables: scripts/bench-local.sh [ITERATIONS] (default 20), DOSH_BENCH_JSON=1, DOSH_BENCH_PROFILE=release|debug.

make bench-docker-ssh / make bench-docker-mosh (scripts/ci-docker-ssh-bench.sh)

Builds one Ubuntu image with OpenSSH, dosh-server, dosh-auth (and Mosh for the -mosh target), then runs ssh_true_ms, cold + ControlMaster SSH-bootstrap dosh_attach_ms, one-time dosh trust followed by native-cold dosh_cold_native_ms, dosh_cached_attach_ms, and optionally mosh_start_true_ms against the same container, key, and loopback network path. This is the gate CI enforces; it asserts cold SSH-bootstrap Dosh stays within 500 ms of SSH, native-cold Dosh beats SSH mean by default, and cached attach stays under a small budget. See README.md "Develop" for the exact invocations.

Methodology and caveats

• Same machine, same network path. bench-local runs everything on loopback, so it isolates Dosh's own process/handshake/render overhead and removes network RTT from the comparison. Real-world cached attach is approximately network RTT plus the local overhead these numbers measure. Publish the machine, OS, CPU, network path, and sample count alongside any numbers.
• Wall-clock of the whole client process. Each sample includes process spawn, attach, first-frame render, and detach. That is deliberately the same thing ssh host true pays, but it means a few ms is fixed process-startup cost, not protocol cost.
• Do not cite cold metrics as the headline. dosh_cold_native_ms and dosh_attach_ms still pay first-authentication cost. They exist to show the fallback path stays competitive with ordinary SSH. Cite dosh_cached_attach_ms for repeat attach/reconnect, per docs/PUBLIC_READINESS.md.
• UDP resume is the roaming path, not cold reconnect. Resume reconnects a client that is still attached when its network endpoint changes (sleep/Wi-Fi switch). The --attach-only benchmark model detaches after every iteration, which removes the client on the server, so a fresh-process resume has nothing live to resume and is not meaningful in bench-local. The cold fresh-process reconnect fast path is the attach ticket (dosh_cached_attach_ms). Roaming resume correctness is covered by the integration test resume_updates_udp_endpoint_for_roaming in tests/integration_smoke.rs. dosh-bench --resume remains available for scenarios that keep a live session open out-of-band (for example remote soak tests).
• These are not identical workloads. SSH/Mosh/Dosh do different work at startup. The comparison is still useful because it measures the startup tax each tool charges before useful remote work begins.

Sample results (loopback, self-contained)

Captured with make bench-local (scripts/bench-local.sh 20), all times in milliseconds, 20 samples per metric.

• Machine: Intel Core i5-9500 @ 3.00 GHz, 6 cores
• OS: Ubuntu 24.04.4 LTS, Linux 6.8.0-124-generic, x86_64
• Toolchain: rustc 1.96.0, release profile
• Network path: loopback (127.0.0.1), throwaway server on a free UDP port
• Workload: dosh-client --attach-only to first frame and detach

Metric	n	min	median	p95	mean	max
dosh_cold_native_ms	20	8.32	8.77	9.34	8.82	9.52
dosh_cached_attach_ms	20	2.83	3.23	3.75	3.24	3.77
dosh_local_attach_ms	20	2.84	3.21	3.96	3.28	3.98

Raw samples (ms):

dosh_cold_native_ms:
8.82, 8.60, 8.69, 8.71, 9.33, 8.82, 8.90, 8.64, 8.39, 8.75, 8.79, 8.70, 8.64,
9.52, 8.72, 8.32, 9.06, 8.93, 8.96, 9.04

dosh_cached_attach_ms:
3.04, 3.06, 2.98, 2.91, 2.96, 2.95, 3.05, 3.66, 3.37, 2.88, 2.83, 3.16, 3.49,
3.30, 3.77, 3.32, 3.49, 3.75, 3.56, 3.33

dosh_local_attach_ms:
3.08, 3.96, 3.36, 3.30, 2.88, 2.99, 3.96, 3.98, 3.25, 2.84, 3.29, 3.59, 3.65,
3.47, 3.13, 2.92, 2.87, 2.91, 2.98, 3.18

Reading these numbers: cold native auth is ~9 ms because it pays the native handshake; cached attach is ~3 ms because it reuses cached credentials and skips the handshake entirely. On loopback there is no network RTT, so this is the local overhead floor. Over a real link, expect cached attach ≈ this floor + one network round trip — which is exactly the "near network RTT" target in the spec. There is no SSH/Mosh baseline in this loopback table because those paths need an SSH server; use make bench-docker-ssh / make bench-docker-mosh for the head-to-head.

Sample results (Docker SSH/Mosh comparison)

Captured with make bench-docker-mosh, all times in milliseconds. This is the same-container comparison gate: one generated key, one OpenSSH server, one dosh-server, loopback-published TCP/UDP ports.

Metric	n	min	median	p95	mean	max
ssh_true_ms (cold SSH baseline for native-cold gate)	5	214.53	216.84	220.55	217.60	220.91
dosh_cold_native_ms	5	8.62	8.96	9.78	9.15	9.88
dosh_cached_attach_ms	10	8.41	8.83	9.52	8.90	9.53
mosh_start_true_ms	3	528.03	530.87	539.23	533.02	540.16

Raw samples (ms):

ssh_true_ms (native-cold gate):
216.84, 216.61, 214.53, 220.91, 219.09

dosh_cold_native_ms:
9.39, 8.88, 8.96, 8.62, 9.88

dosh_cached_attach_ms:
8.44, 8.41, 8.80, 8.48, 9.53, 9.51, 8.85, 8.69, 9.33, 8.96

mosh_start_true_ms:
540.16, 528.03, 530.87