fix(#760): pin the health clock in test fixtures so a handler's HTTP status cannot depend on machine load - #778
fix(#760): pin the health clock in test fixtures so a handler's HTTP status cannot depend on machine load#778djhenry wants to merge 8 commits into
Conversation
…status cannot depend on machine load `require_live_session` refuses a write command when `snapshot_age_ms >= 5000`, and `snapshot_age_ms` was `NetHealth::last_tick.elapsed()` — wall time between the fixture's construction and the handler being served. On a loaded box that crossed 5s and `combat::tests::cast_empty_gem_is_409_and_queues_nothing` got a 503 instead of its 409, from the first line of `post_cast`, before the gem was ever looked at. Re-derived before fixing: a probe that slept 5.1s between `empty_state()` and the same request returned `503 ... has not ticked in 5100ms`. The issue's attribution is correct. Introduce `eqoxide_ipc::HealthClock` — a newtype over a PRIVATE `Option<Instant>` whose only non-wall constructor, `frozen_at`, is behind `#[cfg(any(test, feature = "test-fixtures"))]`. `NetHealth` carries one (`WALL` by default) and `HttpState::health()` derives every age from it. A release build therefore cannot represent a frozen health clock, which matters in the other direction too: a pinned clock would report a dead net thread as live forever (#343). `testkit::empty_state()` now pins its clock at its own stamps, so its ages are structurally 0 however long a test takes — the bad state is unrepresentable rather than merely unlikely. The seven `observe` tests whose subject IS the clock opt back in via a documented `testkit::empty_state_wall_clock()`. Mutation-checked: reverting either half turns three tests red, including a deterministic end-to-end repro of the reported 503. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQVEpaaKeXsZcW9VT2roeV
Independent acceptance review — REQUEST CHANGESReviewed at head BLOCKINGB1 — The fix re-arms #760's exact failure mode on a test it did not migrate
Before this PR the same age was Measured, using the PR's own repro technique (a 5.1s sleep injected immediately after On this PR's head: Control — identical injection on this PR's merge-base ( Same edit, same bound, opposite outcome. The regression is this PR's. The mechanism, spelled out: 15s − 5.1s = 9.9s, under the 10s timeout, so no wedge verdict is reached; I swept every other site: this is the only one. All other Why the doc did not prevent it. Suggested fix — either is fine, and both are small:
NON-BLOCKINGN1 — "A release build cannot represent a frozen health clock" — I could not refute it. Measured both directions.This is the load-bearing universal, so I did not read the manifests and stop. Method: insert
Supporting manifest audit, which the build result confirms rather than replaces: every N2 — the eighth-caller sharp edge: loud, except for one degenerate writingI wrote the eighth caller rather than judging the docs. Two shapes, both on the default
So a new clock-subject test written the strict way fails loudly and immediately, with a message N3 — M1 and M2 reproduced exactly, and they fail different setsReverted by copy-aside / copy-back,
The identical M1's end-to-end failure carries the issue's text verbatim: Does M2 reproduce #760, or an adjacent defect? Neither, strictly — and the PR is right not to N4 — the double-lock self-deadlock claim is TRUE, with one correction that makes our log standard stronger, not weakerI reintroduced the two- … and then nothing, until I killed it. Zero Two corrections to how this should be generalised:
On exit codes: my run reported 124 because I imposed the timeout myself, so I did not independently N5 — smaller notes
The five figures — my own
|
| commit | test attrs | delta | reported workspace passes |
|---|---|---|---|
b0246bc (this PR's merge-base) |
1668 | — | — |
daab1ef (#765) |
1673 | +5 | — |
44003f7 (current main, #772) |
1673 | +0 (docs only) | 1633 |
b404926 (#778) |
1676 | +8 over its base | 1636 |
| #774 head | 1678 | +5 over daab1ef |
1638 |
Every row has the same offset of exactly 40 between static attributes and reported passes (the
ignored tests plus attributes not compiled in this configuration), and it is constant across all
three independently-reported runs. So:
- fix(#760): pin the health clock in test fixtures so a handler's HTTP status cannot depend on machine load #778's +8 = 4 new
eqoxide-httplib tests (2 inlib.rs, 2 incombat.rs; 244→248, observed) +
4 neweqoxide-ipchealth_clock_tests. - 1636 = 1628 (its base) + 8.
- It reads lower than fix(#766): retire
nav_localwith the goal — a zone change no longer publishes the previous zone's fine-tier verdict #774's 1638 only because it sits two commits earlier and misses fix(#732): retirenav_goalwhen the goal is retired — one writer for every route toidle#765's 5
tests while adding 8. Nothing is missing. - After rebasing/merging onto
44003f7, expect 1633 + 8 = 1641. Please confirm that number on
the re-run; anything else means the merge dropped or duplicated something.
Merge order
I checked semantically, not just for filename collisions.
- fix(#766): retire
nav_localwith the goal — a zone change no longer publishes the previous zone's fine-tier verdict #774 and fix(#760): pin the health clock in test fixtures so a handler's HTTP status cannot depend on machine load #778 both editcrates/eqoxide-ipc/src/lib.rs, but in disjoint regions: fix(#760): pin the health clock in test fixtures so a handler's HTTP status cannot depend on machine load #778 adds
HealthClock/ aNetHealthfield / aDefaultarm / a test module; fix(#766): retirenav_localwith the goal — a zone change no longer publishes the previous zone's fine-tier verdict #774 edits
NavStatus::retire_to_idle. The exhaustive-destructure hazard does not apply — fix(#766): retirenav_localwith the goal — a zone change no longer publishes the previous zone's fine-tier verdict #774's
let NavStatus { … } = selfdestructuresNavStatus, and fix(#760): pin the health clock in test fixtures so a handler's HTTP status cannot depend on machine load #778 adds a field toNetHealth. No
E0027net is stretched across the two. Neither PR adds an exhaustiveNetHealth { … }literal. - Both also edit
crates/eqoxide-http/src/observe.rs, again disjointly (fix(#760): pin the health clock in test fixtures so a handler's HTTP status cannot depend on machine load #778's are one-line fixture
swaps around the clock tests plus a doc line; fix(#766): retirenav_localwith the goal — a zone change no longer publishes the previous zone's fine-tier verdict #774 inserts a new test several hundred lines below). - The one real cross-effect runs fix(#760): pin the health clock in test fixtures so a handler's HTTP status cannot depend on machine load #778 → fix(#766): retire
nav_localwith the goal — a zone change no longer publishes the previous zone's fine-tier verdict #774: after fix(#760): pin the health clock in test fixtures so a handler's HTTP status cannot depend on machine load #778,empty_state()hands out a frozen
clock, so any neweqoxide-httptest inherits it. I read fix(#766): retirenav_localwith the goal — a zone change no longer publishes the previous zone's fine-tier verdict #774's new
debug_publishes_no_nav_local_once_the_goal_is_retired_to_idle_766— it touches only
nav.nav_stateand asserts nothing time-derived, so it is unaffected either way. No hazard. - docs(#770): delete slot.rs's false backstop claim, name the canonical spelling in guild.rs's assert #775 touches
guild.rs/slot.rs(an assert message and a doc deletion). No interaction with
testkit, the fixture, or the guard.
Recommended order: #775 → #774 → #778.
Reasons: #775 is trivial and independent, so it should not wait behind anything. #774 is
behaviour-carrying and its file overlap with #778 is textual-only. #778 goes last because it is
the one that changes the fixture every eqoxide-http test is built on — landing it last means its
final workspace run validates it against the final observe.rs, including #774's new test, rather
than against a tree that no longer exists. #778 also needs a rebase onto main regardless, since it
is two commits behind; do that as part of the round-2 fix for B1 and report the 1641.
Verdict: REQUEST CHANGES, on B1 alone. Everything else I attacked held up, and the two claims I
most expected to break — the release-build universal and the M1/M2 mutation evidence — are accurate
as written; the release universal is now measured rather than reasoned. The one disclosed gap
("I did not audit downstream consumers") turned out to be empty. The sharp edge the author flagged
is real and it has already cut once, in the tree, on the one shape the doc's rule does not name.
Please fix that test, widen the rule to cover threshold-shaped ages, rebase onto main, and send it
back to me.
…he clock that reads it Review found the fix re-armed #760's failure mode one level down: `debug_reports_world_unresponsive_when_a_probe_goes_unanswered_while_the_link_acks` stayed on the now-frozen `empty_state()` while stamping `first_unanswered_probe_sent = Some(ago(15))` — a wall-clock stamp read back against a clock pinned at fixture construction. Effective age `15s − (time since empty_state())` against a 10s bound: a 5s margin machine load eats, in the same direction as the bug being fixed. Before this PR the age GREW, so it was unconditionally safe; the PR made a safe test fragile. Structural fix rather than moving that one test back to the wall clock: add `HealthClock::ago`, the inverse of `age_of`, so a stamp is always derived from the clock that will read it. It is correct for BOTH clocks — on a wall clock it is exactly `Instant::now() - N` — so the flagged form has no case where it is the right one. All nine net-health stamp sites in `observe.rs` move to it; the seven wall-clock tests keep `empty_state_wall_clock()` because their subject really is that an age moves. The prose rule did not prevent this (it named only "an age that must MOVE", not "an age that must EXCEED a bound"), so the rule is now executable: a source-scanning guard test bans a bare `ago(` on any net-health stamp assignment across observe/lib/combat/testkit. The doc rule is widened to name both shapes and to point at the guard. Also: narrow the `HealthClock` unrepresentability claim to what is actually true — the field is private to the crate, so the guarantee is against every OTHER crate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQVEpaaKeXsZcW9VT2roeV
…of`, and the cross-clock drift it removes Asserts the property the B1 fix rests on in both directions: on one clock `ago` and `age_of` are exact inverses (pinned and wall alike, so a test never has to choose a spelling), and a wall-clock stamp read back against a pinned clock ages SHORT — the drift that ate a threshold test's margin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQVEpaaKeXsZcW9VT2roeV
…quality that is true The first draft asserted `age_of(ago(0)) == ZERO` on the wall clock; it failed at `left: 100ns` because `now` advances between the two calls. That is this PR's own defect class — a wall-clock-dependent assertion — and the suite caught it. Monotonicity gives the claim that needs no tolerance window: the age can only read LONGER than asked, never shorter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQVEpaaKeXsZcW9VT2roeV
Round 2 — B1 fixed, back to youHead: B1 was real, and I re-derived the sweep before fixing itYour finding stands: the round-1 fix made I enumerated it myself rather than taking your list: all 24 What changed
B1 mutation — measured, both armsInjected a 5.1s
The guard catches the same reverted state statically in 0.01s with no sleep, naming What I deliberately did NOT change
Corrections I owed you
Two framings of yours I think are wrong, with evidence
Five figures on this head (full
|
Independent acceptance review, round 2 — REQUEST CHANGESHead B1 is genuinely fixed and the fix is load-bearing — I reproduced the reverted arm myself. The two blocking items below are both false claims in tracked files about the new guard, of the class this project keeps paying for. Neither is a code defect; both are one-paragraph edits. The guard: it is real, and its coverage is 1 of 6 shapesFirst, the failure mode you asked me to hunt — green because it matched nothing — is not present. The guard matches:
So it is a live guard that catches the shape the actual bug took. Now the attack. Unit of matching = one physical line, and within that line: first
Two of these are not hypothetical. P2 is the shape rustfmt produces, and it already occurs on P4 is the sharpest one. Tenth field: ⛔ Blocking B2 — three tracked claims overstate this
Fix I would accept: state what the guard matches (single-line ⛔ Blocking B3 — "exact" survives in the docs after the test was corrected to
|
…n receiver-aware, and stop the docs overstating it Review round 2 (B2/B3), both findings about tracked prose rather than code. B2 — the guard matched ONE physical line: first `=` splits it, lhs must name a stamp field, rhs must contain `ago(` and not `.ago(`. Review injected seven shapes and it caught 1 of 6 violations. The misses included the shape rustfmt itself writes (an assignment wrapped so the `=` and the `ago(` land on different lines), which already occurs on NetHealth stamps in eqoxide-net today, and `wall.ago(30)` — exempt because the `.ago(` exemption never looked at the receiver, though a WALL receiver stamped into a pinned fixture IS #760. The unit of matching is now a comment-stripped STATEMENT (`/* */` and `//` stripped, then cut at `;`, `{`, `}`), and the exemption is receiver-aware: only `c.ago(` and `….clock.ago(` are exempt, so a bare `ago(` and `WALL.ago(` are both flagged. Also flags `now() -` and `checked_sub`, the past-dating forms review's widened sweep found that a bare `ago(` grep cannot see. The remaining blind spot — aliasing through a local — is named in the test's own doc, because a text scan cannot close it. testkit.rs no longer says the ban "is enforced by" the guard; it says partially guarded and points at the measured blind spots. B3 — the r2 commit corrected the test to `>=` but left "exactly `secs` old" / "exact for both kinds of clock" / "exact inverse … alike" in the docs two lines above it. On the wall clock it is not exact, which is why the assertion is an inequality. Exactness claims deleted rather than restated. Non-blocking: eqoxide-ipc's `test-fixtures` doc now mentions the health-clock injection points it exposes, not just `Roster::insert_for_test`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQVEpaaKeXsZcW9VT2roeV
#755 adds asset-sync login observability to observe.rs; the round-3 guard matches statements rather than physical lines, so its false-positive surface against that new code has to be measured on the merged tree, not reasoned about.
Round 3 — B2 and B3 fixed, and the guard widened rather than only documentedHead: B3 — exactness claims deleted, not restatedYou were right that a replacement tends to re-instantiate the finding, so I deleted rather than reworded. B2 — what the guard claims is now what it does, and what it does is widerThe two overstatements are gone: I did widen it, because your P2 is not hypothetical —
I also took your non-blocking methodological finding into the guard itself, since grep the concept, not the spelling applies to the guard as much as to my sweep: Twelve cells, both directions, two compiles
Violating arm: P3 stays uncaught and is now stated as such in the test's own doc and in the body — a text scan cannot follow a value through a binding; that needs dataflow. I am not claiming otherwise, and I did not go hunting for a stronger word. Base moved again — and that is why I re-merged
Merge, not rebase, for the same two reasons as round 2 (force-push blocked; two posted reviews whose line references a rewrite would invalidate). Conflict-free both times. Five figures on this head
Non-blocking item taken
What I did NOT change
|
|
REQUEST CHANGES Round 3, head B2, B3 and the But the round-3 matcher brought a new failure mode with it, and it is worse than a missed shape. On this head the guard is blind to roughly 88% of the source it claims to scan, including 100% of C1 (blocking) — the comment stripper is string-blind, and it is already blinding the guard on this head
That is not hypothetical here. It is armed by the first line of two of the four scanned files: The route glob Effective coverage on
914 of 7323 lines. Note what the one visible window in Decisive measurement. I injected the same canonical violation three times — same shape, same field, same file set: h.last_packet = std::time::Instant::now() - std::time::Duration::from_secs(30);at with the offender list, complete and verbatim: Two of three identical violations invisible. The control proves the shape is detectable, so this is not a shape miss — it is a scanner stop. All three files restored from copy-asides, This also confounds the twelve-cell table. P1/P2/P4/P5/P6 are reported at None of this is covered by the three stated bounds. Those are all shape bounds (aliasing, file set, Cheapest sufficient remedy. I am not asking for a Rust lexer. One line makes the silent stop loud, with no possible false positive, because no real source file ends inside a block comment: assert!(!in_block, "{file}: the scanner ended inside a block comment — a `/*` inside a string or \
doc comment blinded the rest of this file, so its result is not trustworthy");On this head that assert fires for three of the four files ( C2 (blocking) —
|
…s, and stop it flagging the correct clock
Review round 3 found the source-scanning guard added in round 3 was blind to
87% of its own corpus: `statements()` stripped block comments with a bare
`find("/*")` BEFORE truncating at `//`, so a `/*` inside a line or doc comment
latched the block state permanently. Route globs in `//!` docs on line 1 of
two of the four scanned files had it armed already. Round 3's twelve-cell
shape table was therefore self-graded -- every probe happened to sit in the
one window that was still visible.
C1: replace the line-at-a-time stripper with a character-level `strip_to_code`
(line comments, nesting block comments, strings with escapes, raw strings with
`#` counts, char-literal vs lifetime), and add two PERMANENT controls rather
than ad-hoc probes:
* `assert!(ended_clean, ...)` -- a real source file never ends inside a
comment or a string, so a latched scanner now fails loudly instead of
returning a short clean list.
* `GUARD_REACH_SENTINEL_*` at the end of each scanned file, asserted visible
in the stripped output. The test binds all four by value, so deleting one
is a compile error rather than a silent weakening.
C2: `now() -` / `.checked_sub` were matched by a bare `contains()` with no
receiver test, so `c.now().checked_sub(d)` -- the CORRECT clock -- was flagged,
as was the body of `HealthClock::ago` itself. Route `now()` through the same
`receiver_is_the_reading_clock` test as `ago(`. Delete the falsified sentence
"there is no case where the flagged form is the right one" rather than reword
it; the docs now say a flagged line is not automatically a wrong line.
Also record, in the guard's own doc so the next person inherits it, the rule
that produced this finding: a probe of a source scanner must be placed at
several depths in EVERY scanned file, with a positive control proving the
shape is detectable and a reach control proving the scanner arrives.
N1: narrow the `testkit.rs` sentence that read as exhaustive, and document the
brace/`match` cut misses -- measured 0/12 both ways against a 12/12 positive
control, not reasoned.
N2: the widening justification named "three" rustfmt-wrapped sites in
eqoxide-net. There are seven, none reachable by this guard, and all would be
false positives if they were. Restated.
N3: `empty_state_with_net_health` silently un-pins the health clock; say so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQVEpaaKeXsZcW9VT2roeV
Round 4 — C1 and C2 fixed. Round 3's twelve-cell table is void.New head Both blocking findings were real, and both were code defects in the guard, not prose. The C1 finding is the one that matters: I published a table grading an instrument that was not reading the code it graded. I have not tried to salvage any of it. C1 — the scanner latched, and reported cleanConfirmed. Fixed structurally, not by reordering the two branches: I also corrected my own first draft of that code comment, which claimed " Beyond what was asked — the two controls are permanent, not ad-hoc:
Not tier 1 — a scanner that reaches every line is still a scanner — but the specific bad state, "the instrument silently stopped early", is no longer representable without a red test.
|
| arm | result |
|---|---|
head 5d9166b, controls in place, clean tree |
test result: ok. 1 passed; 0 failed … finished in **0.06s**, EXIT=0 |
| legacy line stripper restored, controls kept | RED — panicked at crates/eqoxide-http/src/observe.rs:3563:13:observe.rs: the scanner ended stranded inside a comment or string literal, so it stopped seeing code partway through and every 'clean' result below is worthless (#760/C1)test result: FAILED. 0 passed; 1 failed |
The RED arm is the load-bearing one: it proves the corpus latches today, so the control is live rather than vacuous. (0.06s vs the round-3 guard's 0.01s is itself consistent with reading ~7,300 lines instead of 914.)
Run E — the silent blindness, demonstrated
Legacy stripper and both controls removed, with 12 identical violations injected at three depths in all four files:
testkit.rs:35: h.last_packet = std::time::Instant::now() - std::time::Duration::from_secs(30)
testkit.rs:109: h.last_packet = std::time::Instant::now() - std::time::Duration::from_secs(30)
testkit.rs:139: h.last_packet = std::time::Instant::now() - std::time::Duration::from_secs(30)
3 of 12. Nine invisible — and had the three testkit.rs cells not existed, the test would have said ok.
Re-measured from scratch, with a reach control at every cell
Probes are now injected programmatically at three depths in each of the four scanned files (top / mid / immediately before the sentinel), and every hit is reported with file and line, so the line number is the reach evidence.
Reach control (run A) — 12/12:
| file | depth 1 | depth 2 | depth 3 |
|---|---|---|---|
observe.rs |
:1864 ✅ |
:3731 ✅ |
:4801 ✅ |
lib.rs |
:1071 ✅ |
:1210 ✅ |
:1308 ✅ |
combat.rs |
:362 ✅ |
:589 ✅ |
:856 ✅ |
testkit.rs |
:35 ✅ |
:109 ✅ |
:139 ✅ |
All three non-testkit rows would have been 0/3 under the round-3 scanner — measured in run E, not reasoned.
Shape table (deep shapes at observe.rs:4841–4852, i.e. in the region the old scanner could not see at all):
| # | shape | violating (run A) | corrected (run C) |
|---|---|---|---|
| P1 | canonical inline h.last_packet = ago(30); |
RED :4841 |
GREEN |
| P2 | rustfmt-wrapped assignment | RED :4842 |
GREEN |
| P3 | aliased via a local | not caught — documented blind spot | GREEN |
| P4 | h.last_tick = wall.ago(30) |
RED :4847 |
GREEN |
| P5 | struct-literal NetHealth { last_datagram: ago(30), .. } |
RED :4848 |
GREEN |
| P6 | earlier = on the same line |
RED :4849 |
GREEN |
| P7 | true-negative control Some(c.ago(3)) |
correctly not flagged | not flagged |
| P8 | Instant::now() - Duration::from_secs(60) |
RED :4851 |
GREEN |
| P9 | Instant::now().checked_sub(…) wrapped |
RED :4852 |
GREEN |
| P10 | C2 control c.now().checked_sub(…) |
correctly not flagged | not flagged |
Run A: FAILED. 0 passed; 1 failed (exit 101), 19 offenders — the 12 reach probes plus exactly 7 deep shapes, nothing else.
Run C (every form corrected, at every depth): ok. 1 passed; 0 failed … 0.04s (exit 0), zero offenders.
C2 — the matcher flagged the correct clock
Confirmed, including that it banned the body of HealthClock::ago (crates/eqoxide-ipc/src/lib.rs:436-440). now() now goes through the same receiver_is_the_reading_clock test as ago( before the -/.checked_sub lookahead. P10 is the regression cell, placed at the deep end of observe.rs so the reach control covers it.
The falsified sentence — "there is no case where the flagged form is the right one" — is deleted, not reworded, from the guard doc, from eqoxide-ipc, and from the PR body. The doc now states the opposite and names self.now().checked_sub(…) as a correct-but-flagged form.
The probe rule is now in the guard's doc
Written into no_past_dated_net_health_stamp_…'s doc under "How to probe this guard, and why it is written down", including the record that round 3's table was void and why — so the next person inherits the failure mode, not just the rule.
Non-blocking, all taken
N1 — measured, not asserted. The exhaustive-sounding testkit.rs sentence is narrowed. Both brace shapes injected at all twelve locations alongside a positive control in the same function:
| injected at 12 locations | flagged |
|---|---|
h.last_packet = Instant::now() - …from_secs(32); (positive control) |
12 / 12 |
h.last_tick = { Instant::now() - …from_secs(30) }; |
0 / 12 |
h.last_datagram = match 1 { _ => Instant::now() - …from_secs(31) }; |
0 / 12 |
The control firing at every location is what makes the zeros mean missed rather than not reached. The //-in-a-string miss is closed by the new lexer (literal contents are dropped).
N2 — restated, and my count was wrong. Round 3 justified the now() widening by pointing at "three" rustfmt-wrapped NetHealth sites in eqoxide-net. Re-enumerated by sweeping that crate for all eight stamp fields against all three past-dating spellings: there are seven — gameplay.rs:1569, 1831, 1835, 1839 and transport.rs:1885, 1914, 2679. And none of them justifies the widening on its own terms: they are all in eqoxide-net, which this guard cannot scan and never will (include_str! reaches only eqoxide-http's own files), and if it could, all seven would be false positives — each builds its NetHealth via Default (checked at every one of the seven), so it past-dates from the wall clock and reads back on the wall clock, which is exactly the property the guard requires. What survives is the methodological result — grep the concept, not the field name — plus a smaller, real benefit: the spelling is now caught inside the four scanned files, where today it does not occur. The guard's doc says this in those words.
N3 — documented. empty_state_with_net_health hands the caller's Arc through and callers build theirs with NetHealth::default(), silently opting out of the tier-1 fixture guarantee. It now says so, and names crates/eqoxide-net/src/transport.rs:2679 as a live caller that is correct precisely because the clock is the wall clock.
#755 clearance, re-derived under a scanner that reaches
Agreed that the previous green did not carry. Re-derived: the ago(300) sites at observe.rs:2109, :2364, :2500 are all inside the old blind region and all now read (bracketed by the observe.rs reach column at :1864 and :3731). None is an offender, and structurally so rather than incidentally: they stamp tick_stamped/asset_sync_slot and name no STAMP_FIELDS entry, so they are outside the guard's subject matter entirely.
Five figures — cargo test --workspace --locked --no-fail-fast
- one
Finishedline:`test` profile … in 25m 08s(attests the build only) - 50
test result:lines matching 50 headers; allok; zeroFAILED, zeroerror, zeropanicked at - non-canonical 0; all-zero (empty targets) 15
1673 passed + 0 failed + 45 ignored + 0 measured = 1718- headers sum to 1718 ✅;
EXIT=0(recorded, not relied upon)
Unchanged from round 3, as expected — round 4 added no tests. Caveat carried: measured on this branch merged with 644ba2c, not a48760b.
A measurement error I made and caught here: my first pass reported 16 all-zero summaries, because grep "0 passed; …" matches inside 10 passed; …. Correct figure 15. Same defect class as everything else in this round.
The workspace run predates the N1/N2/N3 doc edits. Those are comments, which the scanner strips — but rather than rest on that argument, the final tree was re-run: -p eqoxide-http --locked --no-fail-fast, Finished in 3m 21s, test result: ok. **259 passed; 0 failed**, EXIT=0.
Files restored from copy-aside after every mutation, md5sum-verified, zero probe residue; worktree and the shared checkout both clean.
Closes #760.
The mechanism, re-derived (not taken on trust)
The issue's attribution is correct, and I confirmed it before changing anything.
A throwaway probe test slept 5.1s between
testkit::empty_state()and the exactPOST /cast {"gem":7}request thatcombat::tests::cast_empty_gem_is_409_and_queues_nothingissues, and printed the response:post_castcallsrequire_live_sessionas its first statement (crates/eqoxide-http/src/combat.rs:214), that guard trips onh.snapshot_age_ms >= SESSION_STALE_TICK_MS(5_000,crates/eqoxide-http/src/lib.rs:108), andsnapshot_age_mswaslast_tick.elapsed()— wall time between the fixture's construction and the handler being served. On a loaded box that exceeds 5s and the handler answers 503 before it ever looks at the gem. The probe test has been deleted; it existed only to establish the mechanism.Design chosen: inject the clock, as a type
The issue preferred an injectable clock over raising the threshold, and I agree — a bigger number moves the flake, it does not remove it. Implemented as a newtype rather than a flag:
NetHealthgains aclock: HealthClockfield (WALLinDefault), and every age inHttpState::health()is nowh.clock.age_of(stamp)instead ofstamp.elapsed().Why a type with a private field and a gated constructor: a frozen health clock is itself an honesty violation if it can ever reach production — it would pin
snapshot_age_msat 0 and report a dead net thread as live forever, which is the #343 lie this projection exists to prevent. With the field private andfrozen_atbehind#[cfg(any(test, feature = "test-fixtures"))], no other crate can represent a frozen clock. (Precision added in round 2: the field is private toeqoxide-ipc, so this is a crate boundary, not a cfg boundary — production code inside that crate's ownsrc/could still construct one. The guarantee is against every other crate, and it is now measured — see Confirmed by review.) Production behaviour is unchanged:HealthClock::WALL.now()isInstant::now().require_live_sessionitself is untouched — the bound is still 5s, it just now reads an age a test can pin.Round-2 fix: B1 — the same failure mode, one level down
Review caught that
observe::tests::debug_reports_world_unresponsive_when_a_probe_goes_unanswered_while_the_link_acksstayed on the now-frozenempty_state()while stampingfirst_unanswered_probe_sent = Some(ago(15))— a wall-clock stamp read back against a clock pinned at fixture construction. Effective age15s − (time since empty_state()), against a 10s bound: a 5-second margin machine load eats, in the same direction as the bug this PR exists to remove. Before this PR the age grew (15s + elapsed), so it was unconditionally safe. The PR made a safe test fragile. That is a real regression and the review was right to block on it.I re-derived the sweep rather than accepting it. Method: this change only altered how
HttpState::health()computes ages, so the affected surface is exactly tests on a frozen fixture that past-date one of the eightNetHealthstamp fields the projection ages. I enumerated all 24ago()call sites in the workspace plus every other past-dating form (checked_sub,Instant::now() - …) ineqoxide-http. Nine are net-health writes, all inobserve.rs; seven were already migrated toempty_state_wall_clock(); two remained on the frozen fixture. Of those two,debug_defaults_world_responsive_true_before_the_first_probestampsago(20)under a 40s bound — a shrinking age stays on the safe side, so it was harmless. The other is B1. The five remainingago()sites inobserve.rsfeedasset_sync'stick_stamped, which is aged by its own publisher and never touchesHealthClock. Same conclusion as the reviewer, reached independently: exactly one fragile site.Fixed structurally, not by moving that one test back to the wall clock.
HealthClock::agois the inverse ofage_of, so a stamp is always derived from the clock that will read it. All nine net-health stamp sites now use it (let c = h.clock; h.last_probe_sent = Some(c.ago(15));). The seven wall-clock tests keepempty_state_wall_clock(), because their subject really is that an age moves.The rule is now executable, not prose. The old rule named only "an age that must move" — which is exactly why the threshold shape slipped through. Rather than widen the sentence and hope,
observe::tests::no_past_dated_net_health_stamp_is_taken_from_a_clock_other_than_the_one_that_reads_itscansobserve.rs/lib.rs/combat.rs/testkit.rsand fails when a statement both names a net-health stamp field and past-dates it from a clock other than the one that reads it back.B1 mutation check, both directions, measured (5.1s sleep injected immediately after
empty_state()in that test, nothing else changed — the reviewer's probe):c.ago(15))test result: ok. 1 passed; 0 failed; … finished in 5.11s(exit 0)ago(15))test result: FAILED. 0 passed; 1 failed; … finished in 5.22s(exit 101) —left: Bool(true) right: Bool(false), the reviewer's exact failureVerification tier
Tier 1 for the fixture.
testkit::empty_state()buildsNetHealth::frozen_at(Instant::now(), 0, 0, 0)— clock pinned to the same instant as the stamps — so its projected ages are structurally0, not merely small. The test cannot observe wall time at all.For B1's defect class: a guard, and I will not oversell it as tier 1. The wrong spelling is not merely discouraged — it fails a test that reads the source, in hundredths of a second, without needing the race to occur. But it is not literally unrepresentable:
ago()still exists for the asset-sync stamps that legitimately need it. Making it impossible would mean minting net-health stamps through a distinct type onlyHealthClockcan produce; that is a larger change than this PR should carry, and I did not do it.Round 4 — the instrument was blind, and the table that graded it was self-graded
C1 (blocking) — the scanner stopped reading, and reported clean
Review found the guard's
statements()stripped block comments with a barefind("/*")before truncating at//, and had no string-literal awareness. So a/*occurring inside a line or doc comment latched the block-comment state permanently. This was not hypothetical: it was already armed on the round-3 head by route globs in//!docs atobserve.rs:1,combat.rs:1and bylib.rs:661.Reviewer's measured coverage on head
eeaa41d— their measurement, attributed, not mine:observe.rslib.rscombat.rstestkit.rsRound 3's twelve-cell table is therefore VOID. Every probe in it sat at
observe.rs:3465–3483— inside the only windowobserve.rsstill had, opened by this test's own doc comment and closed by its ownmatch rest.find("/*"). The instrument was measured exactly where it happened to work. That table has been replaced below by one re-measured from scratch; it is recorded as void in Corrections rather than quietly swapped out, because the failure mode is the point.Fixed structurally, not by patching the ordering.
statements()is fed by a new character-levelstrip_to_codewith an explicit state machine: line comments, nesting block comments, string literals with escapes, raw strings with#counts, and char-literals disambiguated from lifetimes. Literal contents are dropped, so a//inside a string can no longer truncate a line either.I also recorded a correction to my own first draft of that code comment, which claimed "
//is tested BEFORE/*: that ordering is the whole C1 fix." With a character-level scanner the branch order is irrelevant —//and/*differ in their second character and can never both match. The actual fix is that theLinestate skips to end-of-line, so a/*inside a comment is never read as an opener at all. The comment now says that.And the bad state is now made loud rather than merely fixed — two permanent controls, not ad-hoc probes:
assert!(ended_clean, …)—strip_to_codereturns whether it finished inCode/Linestate. A real source file never ends inside a comment or a string, so a latched scanner now fails the test instead of returning a clean list.pub(crate) const GUARD_REACH_SENTINEL_<FILE>: u8 = 0;, and the guard asserts it can see that text in the stripped output of each file. Deleting a sentinel is a compile error, not a silent weakening: the test binds all four by value in a_sentinels_must_existtuple.This is not tier 1 — a scanner that reaches every line is still a scanner — but the specific bad state ("the instrument silently stopped early") is no longer representable without a red test.
The
assert!(ended_clean)mutation check, both directionsTo mutate I copied each file aside, edited, measured, restored from the copy, and
md5sum-verified. Nogit checkout/git restore/git stashwas used at any point.test result: ok. 1 passed; 0 failed … finished in **0.06s**(EXIT=0)panicked at crates/eqoxide-http/src/observe.rs:3563:13:observe.rs: the scanner ended stranded inside a comment or string literal, so it stopped seeing code partway through and every 'clean' result below is worthless (#760/C1)—test result: FAILED. 0 passed; 1 failedThe RED arm is the important one: it proves the corpus latches today, so the control is live rather than vacuous. The 0.06s vs the round-3 guard's 0.01s is itself consistent with actually reading ~7,300 lines instead of 914.
Run E — what the old scanner did silently, demonstrated
Legacy stripper restored and both controls deleted, with 12 identical violations injected at three depths in all four files:
3 of 12 reported. 9 invisible, and the test would have said
okhad the three intestkit.rsnot existed. That is exactly the reviewer's finding, reproduced independently.Re-measured, with a reach control at every cell
Round 3's cells were placed by hand inside this test's body. Round 4 injects them programmatically at three depths in each of the four scanned files — near the top, mid-file, and immediately before the sentinel — and every shape is reported with its file and line, so the line number is the reach evidence.
Reach control (run A), 12/12 — a canonical violation planted at each depth, all flagged:
observe.rs:1864✅:3731✅:4801✅lib.rs:1071✅:1210✅:1308✅combat.rs:362✅:589✅:856✅testkit.rs:35✅:109✅:139✅Under the round-3 scanner, all three non-
testkitrows would have been 0/3 — not reasoned, measured directly in run E below.Shape table, re-run from scratch. Every cell is now backed by the reach control above (the deep shapes sit at
observe.rs:4841–4852, in the region the old scanner could not see at all):h.last_packet = ago(30);observe.rs:4841:4842h.last_tick = wall.ago(30):4847NetHealth { last_datagram: ago(30), .. }:4848=on the same line:4849Some(c.ago(3))Instant::now() - Duration::from_secs(60):4851Instant::now().checked_sub(…)(wrapped):4852c.now().checked_sub(…)— banned shape, correct clockRun A:
test result: FAILED. 0 passed; 1 failed(exit 101), 19 offenders — the 12 reach probes plus the 7 deep shapes, and nothing else.Run C (all forms corrected, at every depth):
test result: ok. 1 passed; 0 failed … finished in 0.04s(exit 0), zero offenders.C2 (blocking) — the matcher flagged the correct clock
now() -andchecked_subwere matched by a barecontains()with no receiver test, soh.last_datagram = c.now().checked_sub(d).unwrap()— the correct clock — was flagged, and so was the body ofHealthClock::agoitself. Fixed:now()occurrences now go through the samereceiver_is_the_reading_clocktest asago(, and only then check whether the following token is-or.checked_sub. Cell P10 above is the regression test, and it is placed at the deep end ofobserve.rsso it is covered by the reach control.The falsified sentence — "there is no case where the flagged form is the right one" — is deleted, not reworded, in both the PR body (see the round-4 correction block above) and
crates/eqoxide-ipc/src/lib.rs. The guard's doc now states the opposite explicitly and namesself.now().checked_sub(…)as a correct-but-flagged form.N1 — the brace-cut misses, measured rather than asserted
The
testkit.rssentence that read as exhaustive has been narrowed, and the two gaps behind it were measured with the controls this PR now demands. Both spellings were injected at all twelve locations alongside a plain positive control in the same function:h.last_packet = Instant::now() - …from_secs(32);(positive control)h.last_tick = { Instant::now() - …from_secs(30) };h.last_datagram = match 1 { _ => Instant::now() - …from_secs(31) };The controls firing at every location is what makes the two zeros mean "missed" rather than "not reached". Cause: the statement cut at
{/}separates the field from the call. Now documented as a limitation in the guard's own doc.N2 — the widening justification, restated honestly
Round 3 justified widening the matcher to
now() -/checked_subby pointing at "three" rustfmt-wrappedNetHealthstamp sites ineqoxide-net. Two corrections:eqoxide-netfor all eight stamp fields against all three past-dating spellings:crates/eqoxide-net/src/gameplay.rs:1569, 1831, 1835, 1839andcrates/eqoxide-net/src/transport.rs:1885, 1914, 2679. My published "three" was wrong.eqoxide-net, which this guard cannot scan and never will (include_str!reaches onlyeqoxide-http's own files) — and if it could, all seven would be false positives: every one builds itsNetHealthviaDefault(checked at each of the seven), so it past-dates from the wall clock and reads back on the wall clock, which is precisely the property the guard requires.What survives is the methodological result — grep the concept, not the field name — and a real, if smaller, benefit: the spelling is now caught inside the four scanned files, where today it does not occur. The guard's doc says this in those words.
N3 —
empty_state_with_net_healthun-pins the clock, and now says soThat helper hands the caller's
Arcstraight through, and callers build theirs withNetHealth::default()— i.e. the wall clock — so it silently opts out of this PR's tier-1 fixture guarantee. It was undocumented. It now carries a paragraph saying so, and namingcrates/eqoxide-net/src/transport.rs:2679as a live caller that past-dates through this path and is correct precisely because the clock is the wall clock.#755 clearance, re-derived under a scanner that reaches
The previous green does not carry — it was produced by the blind instrument. Re-derived: #755's three
ago(300)sites are atobserve.rs:2109,:2364and:2500, all of which are inside the region the old scanner could not see and all of which the new scanner now reads (proven by theobserve.rsreach column above bracketing them at:1864and:3731). None is an offender, and the reason is structural rather than incidental: they stamptick_stamped/asset_sync_slot, so they name noSTAMP_FIELDSentry at all and are outside the guard's subject matter. Clean-p eqoxide-httpon the final tree: 259 passed, 0 failed.What was measured
require_live_sessionto always 503): 106 of 248eqoxide-httplib tests failed, spanningcombat,group,interact,inventory,merchant,move_api,pet,quests,trainer, plus 8live_session_guard_tests. Notably nochat,guildorlifecycletests appeared despite those modules calling the guard.NOT exhaustive: cargo stopped at the first failing binary, so crates ordered after
eqoxide-httpnever ran under the mutation. The app-crate integration binariestests/http_observe_apply.rsandtests/net_thread_death_reaches_http.rsdid run and passed, so they do not depend on the guard.empty_state()toNetHealth::default():FAILED. 245 passed; 3 failed, the end-to-end test failingleft: 503 right: 409with the issue's body text verbatim.snapshot_age_mstoh.last_tick.elapsed():FAILED. 245 passed; 3 failed.The identical totals are a coincidence and mean nothing on their own — the two mutations fail different sets. M1 trips
handler_fixture_pins_its_health_clock; M2 tripshealth_ages_are_measured_against_the_injected_clock; two tests are shared. The two halves of the fix are therefore independently pinned, and it is the differing sets — not the equal counts — that establish it.ended_cleanmutation, runs A/C/E and the N1 brace table, above.net_health.lock()calls inside oneassert_eq!.std::sync::Mutexis not reentrant and the left-hand temporary guard lives to the end of the statement, so on the passing path it hung the wholeeqoxide-httptest binary. Fixed by taking the lock once into a tuple.Corrections to earlier claims in this body
running Nheader survives; only thetest result:summary is lost. So header-count vs summary-count does detect it — the two are not lost together, as I had implied. Also,Finishedwas present on a run that never completed: it attests the build, not the tests.eqoxide-netis wrong — there are seven, and none of them is reachable by this guard. See N2.//-before-/*branch ordering was the fix. It is not; branch order is irrelevant in a character-level scanner. Rewritten before commit.Confirmed by review — recorded as the reviewer's measurement, not mine
The disclosed gap "I did not audit downstream consumers for accidental non-dev enablement of
eqoxide-ipc/test-fixtures" was closed by the reviewer, who measured it in both directions: with a#[cfg(feature = "test-fixtures")] compile_error!planted ineqoxide-ipc,cargo build --release --locked --bin eqoxidefinished clean in 22m 17s with the probe unfired, whilecargo test --no-run -p eqoxide-httpwith the same probe errored as expected (proving the probe live, not a no-op). Everyfeatures = ["test-fixtures"]entry workspace-wide is under[dev-dependencies];cargo tree --edges no-devshows zerotest-fixturesedges and the with-dev tree shows six. This is the reviewer's measurement, attributed, not my own. The C1 coverage table is likewise the reviewer's measurement.What was NOT verified
matcharm escapes it — measured in N1, 0/12 both ways; (c) it binds only the four files it names, so an offending stamp in another file is invisible — andeqoxide-net, where the motivating shapes actually live, is one of those files it can never reach; (d)STAMP_FIELDSis hand-maintained with no cross-check against the struct, so a ninthInstantfield would be unguarded; (e) it is a spelling rule for receivers, so a correct stamp under a binding named other thancis a false positive. It is a partial guard and the tracked docs say so in those words.strip_to_codeis a hand-written lexer, notrustc's. The two controls catch the failure mode that actually bit (silently stopping early) and would catch any unbalanced-state bug, but a lexer that mis-classifies a construct without ending stranded would still pass. I did not fuzz it.Branch is on current
mainBrought up to
644ba2cby merge, not rebase — force-pushing is blocked in this environment, and rewriting a branch that already carries posted reviews would invalidate their line references. All merges were conflict-free.mainhas since advanced toa48760b.git merge-treeof this head againsta48760bis conflict-free, so I did not merge again — doing so would have invalidated the workspace figures below for no behavioural gain. Carry that as a caveat on the figures: they were measured on this branch merged with644ba2c, not witha48760b.The five figures — my own
cargo test --workspace --locked --no-fail-faston the round-4 headFinished— exactly one such line:Finished `test` profile [unoptimized + debuginfo] target(s) in 25m 08s. (It attests the build only — see Corrections Neriak Third Gate (neriakc) navmesh pathing stall #2.)test result:lines — 50, matching 50 headers (Running <binary>+Doc-tests). All 50 sayok; zero sayFAILED; zeroerrorlines; zeropanicked at.1673 passed + 0 failed + 45 ignored + 0 measured = 1718.EXIT=0(recorded, not relied upon).Unchanged from round 3, as expected: round 4 added no tests — it rewrote one test's body and appended four
constitems.A measurement error I made and caught while computing these: my first pass reported 16 all-zero summaries.
grep "0 passed; 0 failed; …"matches inside10 passed; 0 failed; …. The correct figure is 15. Recorded because it is the same defect class as everything else in this round: an instrument that quietly measures something other than what you asked it.Two runs launched after the workspace job cover the tree the workspace run did not:
-p eqoxide-http --locked --no-fail-faston the final tree, including the N1/N2/N3 doc edits:Finishedin 3m 21s,test result: ok. 259 passed; 0 failed,EXIT=0.The only changes made after the workspace run were doc comments; the scanner strips comments, so they cannot alter its result — but rather than rest on that reasoning, the final tree was re-run.
eqoxide-httplib 259;eqoxide-ipclib 59.Reconciliation:
mainat644ba2cmeasures 1663 (measured by #755's reviewer on that tree, not by me). This branch adds 10 test attributes — 5 ineqoxide-http(combat.rs+2,lib.rs+2,observe.rs+1) and 5 ineqoxide-ipc— so 1663 + 10 = 1673. Nothing dropped or duplicated.One more defect the suite caught, worth recording because it is this PR's own defect class. My first draft of the
ago/age_ofinverse test assertedage_of(ago(0)) == ZEROon the wall clock. It failed:left: 100ns, right: 0ns—nowadvances between the two calls. A wall-clock-dependent assertion, written by the author of a PR about wall-clock-dependent assertions, caught by the workspace run rather than by re-reading. Restated as the inequality that is actually true (>= 15s; a monotonic clock cannot make a stamp younger than it was minted), which needs no tolerance window.