Skip to content

D79 addition: skeleton sanity check — stats-informed flash judge on the parser path - #167

Merged
fazpu merged 3 commits into
mainfrom
design/skeleton-sanity-check
Jul 28, 2026
Merged

D79 addition: skeleton sanity check — stats-informed flash judge on the parser path#167
fazpu merged 3 commits into
mainfrom
design/skeleton-sanity-check

Conversation

@fazpu

@fazpu fazpu commented Jul 28, 2026

Copy link
Copy Markdown
Member

Design-only PR extending merged D79 (§4.1 of the E0 design + a dated amendment on D79 in decisions.md), owner-directed after the complicated-template-PDF discussion.

What it adds

The skeleton sanity check — a third demotion trigger on the parser path, between parse and roles, catching "headings present but the tree is template junk" (running headers as headings, TOC pages, scrambled reading order), which heading density alone misses:

  1. Seven deterministic red-flag stats with exact formulas (duplicate-title ratio, level jumps, numbering inversions + coverage, tiny-section ratio, oversized-leaf ratio, heading density, title character-class shapes) — always computed, always persisted, no decision thresholds in v1 (thresholds must be earned from data, not invented).
  2. One bounded flash-class check call (own D70 seat) reading the stats block + ordered (depth, title, size) lines — never section content; >200 headings head+tail sampled. Output: closed micro-schema (coherent | incoherent + closed reason enum), no free text, structurally unable to propose structure — a judge, never a proposer.
  3. Routing: incoherent → one-way demotion to the LLM fallback route; the fallback's output is not re-checked. Fail-open: a broken guard never takes down the cheap correct path.
  4. Auditable: verdicts + stats persist with the skeleton generation; the Claimify-stage losses are silent: 78% of kept candidates never became claims and no ledger row says why #161 per-document loss signal (omission/rejection spikes) audits the checker's false negatives from data already collected.
  5. Scope fence: judges the tree, not the text under it — intra-section reading-order scrambles are a conversion-layer (D38) track, filed separately.

Plus an owner optional requirement: evaluate a maintained CommonMark tokenizer (markdown-it-py first candidate) under the skeleton parser before hand-rolling.

Review focus

Consistency with merged D79 (triggers, fail-open vs degrade rules, seat/provenance model), the stats' formula soundness, the §2.4 micro-schema discipline, and whether the fail-open choice is right.

fazpu and others added 2 commits July 28, 2026 16:40
… parser path

Owner-directed after the complicated-template-PDF discussion: heading
density can be healthy while the parsed tree is template junk, so the
parser path gains a third demotion trigger between parse and roles. Seven
deterministic red-flag stats (exact formulas, no v1 thresholds) are
computed, persisted, and handed to one bounded flash-class check call
reading titles and numbers only; the call is a judge with a closed
micro-schema and cannot propose structure. incoherent demotes to the LLM
fallback route, one-way; the check fails open. Verdicts + stats persist so
false negatives are auditable against the #161 per-document loss signal,
and any stat-based gating must be earned from that data. Scope fence:
judges the tree, not the text under it — reading-order scrambles remain a
conversion-layer (D38) track.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKENhTLJg1HqhbdwCmmkbc
…arser

Owner-directed optional requirement: evaluate markdown-it-py (maintained,
CommonMark-compliant, source line maps) before hand-rolling heading
tokenization; the tree-building and degrade rules stay ours either way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKENhTLJg1HqhbdwCmmkbc
@github-actions

Copy link
Copy Markdown

Coverage report

This PR does not seem to contain any modification to coverable code.

…EQUEST-CHANGES)

All findings applied: normative stat schema (raw heading levels from
blockizer tokens — tree depth would make jumps identically zero; direct-body
sizes; same-scheme-run numbering with scheme-switch counting; sibling
duplicate ratio beside the global one; every floor a named versioned
constant); single-closed-enum verdict per section 2.4's own lesson (paired
verdict+reasons allowed contradictory states); hard prompt budget with
deterministic anomaly-exemplar sampling (head+tail alone hides mid-document
template corruption); terminal check over the fallback tree degrading to
synthetic root, no cycles; check_outcome separates provider_error /
invalid_response / not_run_short from verdicts so fail-open is never
bookkept as coherence, with the rationale corrected to non-authority-guard
economics; fifth skeleton_check generation with D52 append-only per-document
records, route tags, and D53 recorded N/A; number-161 demoted to a
supplementary correlative with labeled-sample audit as primary; exact state
machine including gate/eligibility order; the markdown-it-py note replaced —
the repo's blockizer already runs it under D57's single-tokenizer rule, so
the requirement resolves to reuse plus heading metadata under a
blockizer_version bump, never a second parse; v1 phrasing dropped; cost
paragraph updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKENhTLJg1HqhbdwCmmkbc
@fazpu

fazpu commented Jul 28, 2026

Copy link
Copy Markdown
Member Author

Revision e972e96 applies both reviews (Grok-4.5: 4 MAJOR / 8 MINOR / 2 NIT; Codex gpt-5.6-sol xhigh: 7 MAJOR / 3 MINOR — heavy overlap). Highlights: normative stat schema with raw blockizer heading levels (tree depth would make level jumps identically zero — Codex), direct-body sizes, same-scheme-run numbering, sibling-duplicate ratio, every floor a named versioned constant; single-closed-enum verdict (coherent | incoherent_<primary-defect>) per §2.4's own paired-fields lesson; hard prompt budget with anomaly-exemplar sampling (head+tail alone hides mid-document corruption — the motivating case); terminal check over the fallback tree degrading to synthetic root, no cycles; check_outcome separates outages from verdicts so fail-open is never bookkept as coherence, with the rationale corrected to non-authority-guard economics; fifth skeleton_check generation with D52 append-only records, route tags, D53 recorded N/A; #161 demoted to supplementary correlative with a labeled skeleton-quality sample as the primary audit; and the markdown-it-py note replaced — the repo's canonical blockizer already runs markdown-it-py under D57's single-tokenizer rule, so the owner's no-reinvention requirement resolves to reuse + heading metadata under a blockizer_version bump, never a second parse (Codex's catch).

@fazpu
fazpu merged commit 69891b6 into main Jul 28, 2026
6 checks passed
@fazpu
fazpu deleted the design/skeleton-sanity-check branch July 28, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant