fix(ci): narrow trace-drift fingerprint to firmware-observable state#491
Open
blue-az wants to merge 1 commit into
Open
fix(ci): narrow trace-drift fingerprint to firmware-observable state#491blue-az wants to merge 1 commit into
blue-az wants to merge 1 commit into
Conversation
The nightly Trace Drift Assertions gate (Advanced Validation) has been red on `arm_uart_ok` because the fingerprint hashes the full end-of-run CPU register snapshot plus cycles/instructions. Those are internal model state: any legitimate refinement of the CPU or cycle-accounting model shifts them even when the firmware behaves identically. The fixture still prints "OK" and the assertion still passes — only the fingerprint moved. That makes the gate a re-baseline treadmill rather than a regression signal. Narrow the fingerprint to the firmware-visible contract: status, stop_reason, steps_executed, assertions, and uart output. A real behavioural regression — wrong UART output, wrong termination reason, a failed assertion, or a changed step count — still trips the gate; benign CPU-model evolution no longer does. All four baselines recomputed under the narrowed payload and verified: the new script reproduces each committed baseline from this branch's own nightly artifacts (trace-drift-assert + digital-twin-verification), which are deterministic (the digital-twin _a/_b double-run is byte-identical). Tradeoff: this intentionally lowers sensitivity to pure CPU-register/cycle changes. If the gate should force acknowledgement of any CPU-state change, keep the wide fingerprint and re-baseline instead; this takes the position that the gate should test firmware-visible behaviour, not model internals. Signed-off-by: blue-az <efehn2000@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The nightly Trace Drift Assertions gate (Advanced Validation) has been red on
arm_uart_ok. Root cause is not a regression:scripts/trace_drift_assert.shhashes the full end-of-run CPU register snapshot pluscycles/instructionsinto each case's fingerprint. Those are internal model state — any legitimate refinement of the CPU or cycle-accounting model shifts them even when the firmware behaves identically. The fixture still printsOKand the assertion still passes; only the fingerprint moved. That turns the gate into a re-baseline treadmill rather than a regression signal.(Confirmed deterministic, not flaky: the nightly's own digital-twin
_a/_bdouble-run of each case is byte-identical.)Change
Narrow the fingerprint to the firmware-observable contract:
status,stop_reason,steps_executed,assertions,uart.A real behavioural regression — wrong UART output, wrong termination reason, a failed assertion, or a changed step count — still trips the gate. Benign CPU-model evolution no longer does.
All four baselines recomputed under the narrowed payload.
Verification
The modified script's exact payload was run against this repo's own latest nightly artifacts (
trace-drift-assert/+digital-twin-verification/, run 28765480985) and reproduces all four committed baselines byte-for-byte. Since those runs are deterministic, CI regenerates identicalresult.json/uart.logand matches.Note:
core-ci(fmt/clippy/build/test) is unaffected — this only touches a CI shell script and four baseline hash files, no Rust. The trace-drift job itself runs in the nightly, not on PRs.Tradeoff (maintainer's call)
This intentionally lowers sensitivity to pure CPU-register/cycle changes. If you'd rather the gate force acknowledgement of any CPU-state change, keep the wide fingerprint and re-baseline instead. This PR takes the position that the gate should test firmware-visible behaviour, not model internals.
🤖 Generated with Claude Code