Add runner update channels for canary field testing on Bencher Cloud

[epompeii]

What

Adds runner update channels so runner changes can be field-tested on Bencher Cloud before a versioned release, mirroring how the API server is dogfooded via cloud branch deploys.

• New runner up --update-channel <stable|canary> flag (env BENCHER_UPDATE_CHANNEL, default stable, conflicts with --no-auto-update)
• The stable channel keeps today's behavior: exact version match against the server, updates from versioned GitHub Releases
• The canary channel converges by content: the runner reports the SHA-256 of its own binary in Ready metadata, and the server compares it against the published checksum of a rolling canary prerelease (TTL-cached per architecture), sending the existing Update directive on mismatch

How

• Protocol: JsonRunnerMetadata gains optional channel and checksum fields; absent means stable, so old runners and old servers interoperate unchanged and the stable wire format is byte-identical
• Server: new RunnerUpdate context (URL construction + canary checksum cache with injectable Clock), channel-aware handle_ready with an injectable checksum fetcher for unit testing, and a new runners.update_base_url config knob. Checksum fetch failures now log, count (runner.self_update.check.failed; updates count as runner.self_update.sent), and skip the update offer instead of erroring the channel into a reconnect loop (this also fixes the stable path when GitHub is unreachable)
• CI: the runner build now runs on cloud pushes (path-filtered, build-only skips the KVM scenario tests already run on devel) and is versioned canary; a new publish_runner_canary job clobbers the assets on a rolling canary prerelease, binaries before checksums so a partial upload never advertises a missing binary. The tag is canary rather than cloud because a tag named like the branch is ambiguous in git ref resolution and would break git checkout cloud in rebase_cloud.yml
• Ops: cargo ops deploy|start --update-channel and an optional update_channel field in runners.json write BENCHER_UPDATE_CHANNEL into the systemd drop-in
• Docs: runner up reference, Runner Protocol reference, and the Self-Hosted Runners explanation updated in all 9 languages; changelog entries added under Pending v0.6.9

Self-update remains idle-only on both sides (server sends Update only from wait_for_ready; the runner handles it only in AwaitingJob), so a canary rollout never interrupts a running benchmark.

Field testing flow

1. Merge a runner change to devel, reset cloud to devel, and push
2. Cloud CI publishes the new canary assets while the API deploys as usual
3. At their next idle poll, canary channel runners detect the checksum mismatch, self-update, and reconnect
4. Rollback is re-uploading previous assets with --clobber or pushing a revert to cloud

[github-actions]

🤖 Claude Code Review

PR: #927
Base: devel
Head: claude/runner-canary-channel
Commit: db4f747d04325d6cc4045f967db06942d4cff449

---

I've now reviewed the full diff. I was unable to run cargo builds (sandbox blocked them), so compilation claims below are based on careful reading; the PR includes extensive tests and the type-generation artifacts (openapi.json, bencher.ts) are regenerated.

PR Review: Runner update channels (canary) for field testing

Overall this is a high-quality, well-tested change. It adds a canary update channel so Bencher Cloud runners self-update to the rolling cloud-branch build (converging by binary checksum) while stable runners keep converging by version. Strong types, deterministic clock-injected tests, backward-compatible wire format, and OTel metrics are all handled correctly.

Strengths

• Backward-compatible wire format. channel/checksum are Option with skip_serializing_if, and build_runner_metadata deliberately emits stable runners as channel: None, so a stable runner's Ready message is byte-identical to a pre-channel runner. The ready_stable_metadata_wire_format_is_legacy and ready_legacy_metadata_deserializes tests lock this in. Good.
• Testability refactor. handle_ready was made generic over a fetch_checksum fn, enabling the FakeFetcher unit tests to cover match/mismatch/fetch-failure/caching without network. The extraction of handle_idle_terminal is behavior-preserving.
• Graceful degradation. Checksum-fetch failures skip the update (warn + RunnerSelfUpdateCheckFailed counter) rather than erroring the channel. Verified by both unit tests and integration tests using an unroutable 127.0.0.1:1 base URL.
• Correct convergence loop. Server sends checksum: published; runner (self_update) verifies the download against it, installs byte-identical bytes, then self_checksum() recomputes and matches. Rollback (re-publishing an older build) also converges. No update loop.
• CI ordering. Publishing binaries before .sha256 files (so a checksum never advertises a missing binary) is correct, and publish_runner_canary is gated on runner.result == 'success' so an unchanged runner doesn't churn the canary fleet.
• Standards compliance. Strong validated UpdateChannel type with Diesel-free TryFrom/FromStr; thiserror error enums wrapping original errors; #[expect(...)] with reasons; Clock injection for deterministic TTL tests; camino/destructuring-on-consume patterns; OTel adverse-event counters with dimensional update.channel attribute. No emdashes.

Minor observations (non-blocking)

1. conflicts_with + env var edge case (services/runner/src/parser/up.rs:60). update_channel has both default_value_t and conflicts_with = "no_auto_update". Clap treats an env-sourced value (BENCHER_UPDATE_CHANNEL) as user-supplied, so a self-hosted operator who exports BENCHER_UPDATE_CHANNEL globally and then runs --no-auto-update will hit a conflict error. The default value itself does not trigger this (confirmed by the tests), so it only bites the env-var case. Low severity, but worth a doc note or a test.

2. No single-flight on canary checksum fetch (runner_update.rs:20). Concurrent Ready polls after TTL expiry can each re-fetch. This is explicitly documented as acceptable given tiny responses and a small fleet. Fine as-is.

3. Cache sharing assumption. The RunnerUpdate TTL cache (tokio::sync::Mutex<HashMap<...>>) only works if ApiContext is a single shared instance (Arc) rather than cloned per connection. This matches the existing heartbeat_tasks/shutdown fields, so it's almost certainly correct, but it's an implicit invariant worth being aware of.

One thing to double-check before merge

• tasks/runner_ops/src/task/download.rs: let artifact_version = if branch == "cloud" { "canary" } else { &branch }; relies on if/else LUB coercion unifying &'static str and &String to &str. This should compile (deref coercion applies in LUB), but since I couldn't run cargo check here, please confirm CI's build is green on this line.

No correctness bugs found. Nice work on the test coverage and the legacy-compat wire format.

---

_{Model: claude-opus-4-8}

[github-actions]

Bencher Report

Branch	claude/runner-canary-channel
Testbed	intel-v1

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
Adapter::Json	📈 view plot 🚷 view threshold	4.63 µs (+0.05%) Baseline: 4.62 µs	4.90 µs (94.34%)
Adapter::Magic (JSON)	📈 view plot 🚷 view threshold	4.49 µs (-0.21%) Baseline: 4.50 µs	4.72 µs (95.07%)
Adapter::Magic (Rust)	📈 view plot 🚷 view threshold	26.44 µs (+3.52%) Baseline: 25.54 µs	26.64 µs (99.25%)
Adapter::Rust	📈 view plot 🚷 view threshold	3.52 µs (+0.93%) Baseline: 3.49 µs	3.60 µs (97.72%)
Adapter::RustBench	📈 view plot 🚷 view threshold	3.52 µs (+0.97%) Baseline: 3.48 µs	3.59 µs (97.89%)

🐰 View full continuous benchmarking report in Bencher