Repository navigation
GO-7383 Make schema and validation prose caller-facing - #12
Merged
Merged
Conversation
The JSON Schema descriptions and the validation messages are what an API caller and an LLM authoring a document actually read. They had grown into essays that cited internal spec paragraphs, quoted corpus statistics, and named Go symbols the caller has no access to — none of which a reader outside this repository can act on, and all of which they pay for in tokens on every schema fetch. Rewrite 223 descriptions across the six published schemas: 93,920 -> 35,757 characters, 61% smaller. Rationale that is worth keeping moves into Go comments, which readers of this repository can see and callers do not pay for. Also drop readerFor() from the codec's document-kind surface: its verdicts named UnmarshalPropertyDictionary and friends, so a caller who sent the wrong shape was told to call a Go function they cannot reach. Two guards keep it that way, because compression is where a true statement becomes false: - format/v2/schema/prose_test.go bounds every description (200 chars, 600 at a schema root, 800 for named grammar slots) and rejects section marks, SPEC references, constraint codes, .md filenames, corpus evidence and em dashes. - internal/prose/userfacing_test.go applies the same rules to Go string literals via go/ast, so a message cannot reintroduce what a description gave up. Comments are exempt by design. Five tests asserted on the old prose and encode the new contract instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The JSON Schema
descriptionvalues and the validation messages are what an API caller — and an LLM authoring a document — actually read. They had grown into essays citing internal spec paragraphs (§6.2), corpus statistics ("1,760 corpus type documents"), constraint codes (C13), and Go symbols the caller cannot reach.None of it is actionable outside this repository, and callers pay for all of it in tokens on every schema fetch.
223 descriptions rewritten across the six published schemas: 93,920 → 35,757 characters (61% smaller).
Rationale worth keeping moved into Go comments — visible to readers of this repo, free for callers.
Also
readerFor()is gone from the codec's document-kind surface. Its verdicts namedUnmarshalPropertyDictionaryand friends, so a caller who sent the wrong shape was told to call a Go function they have no access to.Guards
Compression is where a true statement quietly becomes false, so two tests hold the line:
format/v2/schema/prose_test.go— bounds every description (200 chars; 600 at a schema root; 800 for named grammar slots, keyed by<file>::<pointer>) and rejects section marks,SPECreferences, constraint codes,.mdfilenames, corpus evidence and em dashes.internal/prose/userfacing_test.go— the same rules over Go string literals viago/ast, so a validation message cannot reintroduce what a description gave up. Comments are exempt by design; moving a citation out of a served string and into the comment above it is the intended fix.Five existing tests asserted on the old prose (one required the literal text
§6.2); they encode the new contract now.Verification
go build ./...,go test ./...andgofmt -lare clean.The compression was reviewed by three independent adversarial passes over before/after pairs, which caught ~20 statements that the rewrite had made false — e.g. a description promising raw numbers on two enum-only slots, and a dataview rule inverted for type documents. Those were corrected before this commit.
Note for the reviewer
anytype-heartdepends on this. Its matching branch currently pinsreplace github.com/anyproto/any-block => ../any-block_prose, so this needs to merge first and the pin be swapped for a real version before that side can land.