Repository navigation
Single-source and justify tunable constants - #21
Merged
Merged
Conversation
Duplicated literals collapsed to one definition, and the numbers that were
asserted without a reason now carry one.
De-duplication (no behavior change):
- The ZIP readers hardcoded the fixed record sizes 46/30/22 across 15 sites
while LOCAL_HEADER_BYTES / CENTRAL_HEADER_BYTES / EOCD_BYTES already existed
and were used only by the writer. Moved the declarations up beside the
signature constants and substituted them throughout both walks.
- PDF and XLSX both join blocks with '\n\n' and both charged its width as a
bare 2. Extracted BLOCK_SEPARATOR so the separator and the length the cap
accounting charges for it cannot disagree.
- XML_SLICE_CHARS and DOCX_CHUNK_BYTES were the same 16 KB serving the same
purpose under two names; merged into STREAM_SLICE_UNITS.
- GLUED_ATTRIBUTION_RE repeated its {0,240} scan budget four times and its verb
list twice inside one 600-character literal. Assembled from named parts
instead; the generated source is byte-identical to the literal it replaces.
Named and justified:
- CAP_STORAGE_SLACK for the docx `maxOutputChars + 2`. Verified load-bearing:
at +0 the central trim cannot see the overshoot, skips its surrogate check,
and emits a lone half as U+FFFD (tests/docx-streaming surrogate test fails).
- MAX_SHARED_STRING_INDEX for the bare 10_000_000, now derived from
MAX_UNCOMPRESSED_BYTES over the 16-byte minimum <si> record.
- TAG_CAPTURE_LIMIT, MAX_PDF_PAGES, SNIFF_BYTES, SNIFF_TEXT_RATIO,
DETECT_SAMPLE_BYTES and DETECT_MIN_CONFIDENCE gained the missing rationale.
SNIFF_BYTES in particular is not only a budget: isWellFormedUtf16 splits
"truncated sample" from "genuinely unpaired" at exactly that boundary.
- body.ts had three lookahead windows, one named and two bare, at two different
values with no stated reason for the difference. Now ATTRIBUTION_WRAP_LINES,
DIVIDER_LOOKAHEAD_LINES and HEADER_BLOCK_LINES, each explaining what bounds it.
Still undocumented and flagged for follow-up: the dataset-tuned thresholds in
body.ts (the 400-char signature cap, the 0.5 quoted-block ratio, the >=2-line
signature body, and the HTML inter-tag windows 200/250/80/25/40/60).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sidharth0612
approved these changes
Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Collapses duplicated numeric literals to single definitions and adds the missing rationale to constants whose values were asserted without one. The de-duplication is behavior-preserving: the ZIP readers now use the
CENTRAL_HEADER_BYTES/LOCAL_HEADER_BYTES/EOCD_BYTESconstants that already existed but were used only by the writer (15 sites), PDF and XLSX share oneBLOCK_SEPARATORso the separator and the width the cap accounting charges for it cannot disagree,XML_SLICE_CHARS/DOCX_CHUNK_BYTESmerge intoSTREAM_SLICE_UNITS, andGLUED_ATTRIBUTION_REis assembled from named parts rather than repeating its{0,240}scan budget four times inside one 600-character literal — the generated source is byte-identical to the literal it replaces. The docxmaxOutputChars + 2becameCAP_STORAGE_SLACKafter probing confirmed it is load-bearing: at+0the central trim cannot see the overshoot, skips its surrogate check, and emits a lone half as U+FFFD.MAX_SHARED_STRING_INDEXis now derived fromMAX_UNCOMPRESSED_BYTESover the 16-byte minimum<si>record, whileMAX_PDF_PAGES,TAG_CAPTURE_LIMIT, the sniff/detect constants and body.ts's three lookahead windows (previously one named and two bare, at two different values with no stated reason) gained names and explanations — notably thatSNIFF_BYTESis not purely a budget, sinceisWellFormedUtf16splits "truncated sample" from "genuinely unpaired" at exactly that boundary. The dataset-tuned body.ts thresholds (the 400-char signature cap, the 0.5 quoted-block ratio, the >=2-line signature body, and the HTML inter-tag windows) are deliberately left undocumented pending the corpus measurements needed to justify them honestly, and are flagged as follow-up; 529 tests, typecheck and build all pass.🤖 Generated with Claude Code
Open workspace in Conductor