Flaky: dns_lane_bounds_tcp_http_and_drains_all_work times out on DNS server shutdown (all-features only)

[jonaswre]

Recording a single observed failure so a second occurrence has something to build on.

Observed failure

On PR #22's CI, CI Test Suite / Tests (ubuntu-latest, stable, all) and the equivalent Windows job:

FAIL [ 20.810s] krikos-bench canary::workloads::tests::dns_lane_bounds_tcp_http_and_drains_all_work
thread '...' panicked at krikos/bench/src/canary/workloads.rs:3070:14:
bounded DNS lane: Custom { kind: Other, error: "DNS server shutdown timed out" }

It passed on rerun with no code change. macOS-arm-latest, stable, all had also failed on the first run and passed on the rerun.

Evidence it is not PR #22's doing

• That PR's diff touches four files — krikos/src/endpoint/tests.rs, protocols/krikos-blobs/src/store/fs.rs, protocols/krikos-docs/tests/sync.rs, scripts/determinism-boundaries.txt. None is in krikos/bench.
• The failure is a DNS canary workload shutdown, unrelated to document-sync event matchers or a blobs round-trip.
• Ran locally 8 times with --all-features on that branch: 0 failures.
• The same job is 6/6 success on main over recent runs (note: reruns overwrite a job's conclusion, so treat that as an upper bound, not proof).

A controlled branch-vs-main A/B was started but not completed, so this is "does not reproduce on the branch", not "proven identical to main".

Only surfaces under --all-features

krikos-bench is in CARGO_MAKE_WORKSPACE_SKIP_MEMBERS, and this test appears only in the all feature matrix entries. A default-feature run will not exercise it — worth knowing before concluding "it passes locally".

Pattern worth noting

This is the third distinct test in this repo observed failing by a deadline expiring rather than an assertion failing:

Test	Deadline
test_download_policies (krikos-docs)	120s — root-caused and fixed in #20: a double-counted download made an equality condition permanently unsatisfiable
test_two_devices_roundtrip_network_change_only_a (krikos)	20s — see #21, not reproducible
dns_lane_bounds_tcp_http_and_drains_all_work (krikos-bench)	20.8s — this issue

#20 turned out to be a genuine logic error rather than a timing assumption, so the pattern is not automatically "the timeouts are too tight". But three tests failing this way suggests it is worth asking, when one of these fails, whether the test asserts a bound the system actually guarantees.

Picking this up later

1. Wait for a second occurrence. The nightly flaky sweep is now capable of reporting real flakes (fix(ci): make the daily flaky sweep able to detect a flake #19 fixed it being red by construction), though note it runs the all feature set, so this test is in scope for it.
2. To diagnose rather than guess, determine whether the DNS server is slow to shut down (raise or justify the bound) or failing to shut down at all (a real defect in shutdown handling). Those need opposite fixes; capturing the shutdown path with RUST_LOG=trace on a failing run distinguishes them.
3. If a controlled comparison is wanted, run it in a separate git worktree — mutating the live checkout to compare trees is how this investigation nearly lost work.

🤖 Filed by Claude Code