Skip to content

Make the exploratory subset design record platform-independent and close the plan's executability gaps - #62

Merged
ahrazzle merged 4 commits into
mainfrom
protocols/pilot-001-exploratory-plan-followup
Nov 6, 2026
Merged

ahrazzle merged 4 commits into
mainfrom
protocols/pilot-001-exploratory-plan-followup

Conversation

@zero-aska

@zero-aska zero-aska commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

The exploratory subset design record previously depended on floating-point summation and libm, and the plan left several execution choices unspecified. This change makes the offline design arithmetic reproducible and fixes the contract before any study-data execution.

  • Binomial-tail sign tests use exact rational arithmetic. Endpoints round to the nearest binary64 value, including ties-to-even at exact halfway roots. The Mann–Whitney table uses bounded memory; critical decision thresholds remain unchanged.
  • Regression coverage checks committed bytes, exact-root rounding, public-API halfway cases in both directions, and input bounds. The design artifact and implementation hash are regenerated.
  • The protocol specifies discovery-key preprocessing, frozen-side membership equality, local workbook bytes with no network fetch, bootstrap ordering/runtime, metric interfaces and arithmetic, module hashes, output paths and the execution record.
  • Before any threshold or metric, distinct Decimal scores must remain distinct after conversion to binary64. A collision stops the future run; ordinary equal-score ties remain allowed. This preserves the reused DeLong helper’s ranking and prevents a precision-induced degenerate interval.
  • Stop reports record the stage reached and state that no accepted estimate is reported, including stops after intermediate calculations. Original authorship and Codex/Astra contributions are recorded in CONTRIBUTORS.md.

Validation performed on the final files: python3 tools/validate.py; all 361 Python unit tests; python3 tools/check_implementation_hashes.py; and the design --check on Python 3.12.14 and 3.13.5. Artificial precision-guard probes reject collisions, accept equal Decimals and preserve pair ordering. A separate Astra agent independently reviews the published head; the required GitHub workflows must pass before merge.

No study data, private patient files, owner-host access or real-data pipeline execution is involved. The Tier A implementation remains prospective; these tests establish arithmetic and contract checks, not model performance or clinical validity. The pre-existing R-assertion count discrepancy in the synthetic-validation prose is outside these changes.

AI disclosure: original work by Claude (Anthropic) through Claude Code under human direction; follow-up fixes by OpenAI Codex with GPT-6 Astra agents, with a separate independent Astra reviewer. Internal review is not external scientific peer review.

claude and others added 4 commits November 6, 2026 09:51
…ty gaps

The committed Clopper-Pearson endpoints were bisection roots evaluated in
binary64, so their bytes depended on CPython 3.12's compensated float
summation and on libm: --check failed on 3.11 and would differ on the
macOS host the plan names. Sign tests now run in exact rational arithmetic
and each endpoint is the correctly rounded root, so the record no longer
depends on the interpreter or platform (72 of 114 endpoints change in the
last bits; the 13/13 bounds now equal the capacity record exactly). The
Mann-Whitney table is built row by row with bounded memory, and the
binomial bounds are limited to what the arithmetic supports. Three tests
cover the rendered bytes, correct rounding against exact tails, and the
edge cases; the CI pin moves to 320.

The plan now fixes what an executor would otherwise have to invent: the
discovery-key preprocessing behind the 21, the frozen-side membership
equality, exact patient-key comparison, local workbook bytes with no
network request, Decimal scores with a binary64 threshold, the ordering
the bootstrap draws address, the DeLong call's positional lists and label
strings, reimplemented AP and Brier, every imported module's hash, fixed
artifact and note paths with required contents, wording for a stopped
run, and a before/after split of the execution record.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BuwmdHVixi4BxQmXNM3F6d
The ledger said 259 cases while CI pinned 310 on main and 320 here; the
verified statement now names the count the pinned run checks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BuwmdHVixi4BxQmXNM3F6d
…xploratory-plan-followup

# Conflicts:
#	.github/workflows/verify.yml
…gaps

Round exact midpoint roots to even with six public-API regression cases. Stop the future subset run before any metric when Decimal-to-binary64 conversion collapses distinct scores, and report stopped stages accurately. Refresh design provenance and the 361-test count.

Prepared with OpenAI Codex and GPT-6 Astra under human direction. No study data accessed.

@ahrazzle ahrazzle left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent Codex agent review of head 8957653. I did not author or edit these changes. GitHub's commit tree matches the clean local tree reviewed and tested (5d7d44d4f54b56a1c758e73eff3f7ccdbf2aa10d).

No material findings remain in the reviewed changes. Exact halfway roots now round ties to even, with public-API regressions for both neighbouring parity cases. The protocol requires rejecting Decimal-to-binary64 collisions across all 61 scores before any threshold or metric, permits genuine Decimal ties, and preserves the score comparisons used by DeLong when the guard passes. Stopped-run wording now records the stage and distinguishes intermediate calculations from accepted estimates.

Checks I ran on CPython 3.12.14: all 361 Python unit tests passed; tools/validate.py, the design artifact --check, and the tracked study-source check passed; implementation hashes were current or covered by the existing two explicit stale-hash allowances; git diff --check was clean. Additional independent artificial checks covered 25 exact midpoint roots, including subnormal boundaries, and executed the protocol's literal collision guard on artificial 61-score fixtures to verify collision rejection, repeated-tie acceptance, and preserved pair ordering.

This is internal software/protocol review, not external scientific peer review. No study files, private records, owner-host preflight, or study-data execution were accessed or performed. The Tier A executor and real-data gates remain future work. Final-head CI must pass before merge.

Copy link
Copy Markdown
Contributor Author

Thanks for the Hermes review. On the three non-blocking notes, nothing changes in this PR:

  • CLASS_LIMIT = 60: the bound is documented in the tool next to the constants ("They cap memory and time, not a tested general-purpose operating range"). The previous 200 could not be served: the cached table grows with (m·n)², and (200, 200) extrapolates to tens of gigabytes. The design serves the documented counts (at most 29), and the test suite covers the bound.
  • count() / positive_count() default: both helpers are module-private to this tool; every caller passes its bound explicitly so the two limits (class counts versus binomial trials) cannot be confused. No external caller exists in the repository.
  • Exact equality with the capacity record: intentional, and the test already carries the comment ("there computed as 0.05 ** (1 / 13) and 0.025 ** (1 / 13) … Correctly rounded roots agree to the last bit"). If a future libm ever rounds those powers differently, the capacity record is the side that moves, which is the point of the comparison.

The test_pilot001_first_run failure you saw is the macOS tmutil preflight case; it passes on the Linux CI runners and is unrelated to this change.


Generated by Claude Code

@ahrazzle
ahrazzle merged commit 49c27ea into main Nov 6, 2026
4 checks passed
@ahrazzle
ahrazzle deleted the protocols/pilot-001-exploratory-plan-followup branch November 6, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants