Cross-artifact scanners and side-by-side audit view (v0.6.1 + v0.6.3 first slice)#5
Merged
Merged
Conversation
… (v0.6.1) Gold answers can now be checked against extraction artifacts on disk. --files-root <dir> resolves <dir>/<sample_id>/ per record; every non-empty *.txt / *.md file there (except ground_truth.* and index.md) is treated as one extraction tool's text output. - text_layer_recall (consensus-aware): gold words found by no tool are high-severity typo/hallucination candidates; for full-page gold, words found by every tool but missing from the gold are medium-severity omissions. LaTeX math spans are excluded from the existence check (they render as glyphs) but still count as gold coverage; ligatures are NFKC-normalised and hyphenated linebreaks re-joined. - numeric_provenance: every number in the gold (comma/$-normalised) must appear in at least one tool output; misses are high-severity transcription-error candidates, anchored to their gold line. Validated against a real 14-sample benchmark corpus with a 4-tool extraction cache: zero findings on the clean corpus; seeded typos (one digit, one word) are caught by both scanners. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLg5cbBcfNyyrRGJVPfsA6
Viewer support for auditing markdown ground truth against its source page: - Sample detail renders multi-line answers as real markdown (react-markdown + remark-gfm, Bootstrap-styled tables) with an "Audit view" button - AuditView: full-screen three-pane overlay — page image | rendered gold | raw source with line numbers; finding line anchors are highlighted in the raw pane (file lines mapped through the sample's frontmatter offset); a source selector switches between the gold markdown and each cached extraction tool's output - View server: local annotation datasets reload for sample detail; scan_summary.json now records --files-root so the server can resolve per-sample artifacts, serving page.png as a data URL plus tool text outputs and the markdown body line offset - API-level tests for the local-source flow; frontend bundle rebuilt Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLg5cbBcfNyyrRGJVPfsA6
Records the known benchmark-vocabulary leakage in the cross-artifact scanners and audit view (skip-field name, task_type magic string, loader schema fallback, artifact layout), the proposed record-adapter contract with reserved dunder fields, and longer-term directions: eval packages shipping their own adapters, a declarative audit UI spec, and inferring the contract from HuggingFace dataset Features. Explicitly deferred until the features have seen real use — and gated before the first PyPI release that includes v0.6, when conventions become API commitments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLg5cbBcfNyyrRGJVPfsA6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two roadmap phases for markdown ground-truth auditing (PLAN.md v0.6.1 / v0.6.3): gold answers can now be checked against extraction artifacts on disk, and the explorer gains a side-by-side audit view for triaging what the scanners find.
v0.6.1 — Cross-artifact scanners
--files-root <dir>resolves<dir>/<sample_id>/per record; every non-empty*.txt/*.mdfile there (exceptground_truth.*andindex.md) is one extraction tool's text output.text_layer_recall(consensus-aware): gold words found by no tool are high-severity typo/hallucination candidates, anchored to their gold line; for full-page gold (task_typecontains "page"), words found by every tool but missing from the gold are medium-severity omissions. LaTeX math spans are excluded from the existence check (they render as glyphs) but still count as gold coverage; ligatures are NFKC-normalised and hyphenated linebreaks re-joined.numeric_provenance: every number in the gold (comma/$-normalised) must appear in at least one tool output; misses are high-severity transcription-error candidates with line anchors.Validated against a real 14-sample benchmark corpus with a 4-tool extraction cache: zero findings on the clean corpus (after fixing two false-positive classes found during validation), and seeded errors — one digit changed, one word misspelled — are caught by both scanners.
v0.6.3 — Side-by-side audit view (first slice)
lineanchors highlighted in the raw pane (file lines mapped through the sample's frontmatterline_offset) and a selector to switch the source pane between the gold markdown and each tool's cached outputscan_summary.jsonrecords--files-rootso the server can resolve per-sample artifacts (page.pngas data URL, tool text outputs, line offset)Still open from v0.6.3 (tracked in PLAN.md): word-level diff, edit-in-place write-back, audit provenance.
Validation
22 new tests (cross-artifact scanner unit tests + API-level tests for the local-source view flow); full suite 184 passing, ruff/mypy/pre-commit clean, frontend
tsc/eslint green, bundle rebuilt. The UI was verified at the API level end-to-end; visual verification was not possible in this sandbox (no launchable browser) — the Playwright e2e suite covers it where browsers exist.🤖 Generated with Claude Code
https://claude.ai/code/session_01LLg5cbBcfNyyrRGJVPfsA6
Generated by Claude Code