Homogenize the QVR grammar surface and parse through the vendored grammar#47
Merged
Conversation
Grammar: signed and scientific number literals; optional option blocks on morphism and contraction declarations; brace-delimited constructor options; single |- turnstile for rules and deductions; bundle and decoder headers use ':'; composition level moves into the option block; top-level 'define' binds morphism expressions; plural-name object/morphism/lexicon declarations; parenthesized variable patterns for sample and observe; keyword-led encoder op rules; unused compose operators removed. The grammar is now conflict-free. AST: MorphismDecl/ObjectDecl carry names tuples; LexiconEntry carries words; DefineDecl replaces the top-level let node; ObserveStep shares the SampleStep pattern shape; constructor kwargs carry typed values.
Parsing now fails loudly on any ERROR or zero-width vertex anywhere in the tree, with the innermost damaged span's line, col, and snippet. Registry-level failures re-raise as ParseError; a program body without a return step names the missing step instead of leaking an internal message. Walkers cover plural names, define bindings with where blocks, brace constructor options with typed keyword values, signed and scientific option numbers, shared sample/observe patterns, option- sourced composition levels, and multi-word lexicon entries.
module_to_source covers all statement, expression, let-arithmetic, step, and structural node kinds; compiler-only IR without a surface form raises a typed EmitError. The round-trip suite asserts a canonical fixed point over every repository .qvr file and fenced doc block.
Morphism and define tables expand plural declarations per name; reverse composition expands right-to-left (the operator was previously ignored when flattening chains); step arguments cross the transpile boundary as the typed DrawArg union; ResolvedDist is a didactic model.
Fifty corpus cases cover every declaration, step, expression, and comment form; editor indent and highlight rules, the pygments lexer, and the package quick-start snippet match the shipped grammar.
Adds the adjacent-pair migrator for the homogenized surface (turnstile, bundle and decoder headers, composition level option, define bindings, parenthesized patterns, brace constructor options, keyword-led encoder op rules) with span-preserving edits, registers it on the migration chain, and rebuilds the grammar VCS store and parser snapshots. All repository .qvr sources and fenced doc blocks are migrated; the regression fixture drops a stray colon its old tolerant parse had swallowed silently.
Every declaration and step kind validates its option keys against an explicit set, with did-you-mean suggestions at the offending entry's own position and a braces hint when a constructor key lands on a declaration. A morphism without role= is a kernel; every other role is explicit. Plural-name morphism and object declarations expand to independent parameters; lexicon entries expand per word. Top-level define bindings compile as the old value bindings did; observe takes a single variable with a clear arity error; the algebra-tagged composition operators are gone from expression dispatch in favor of change_base and composition declarations.
Per-name definition targets and document symbols for plural declarations; define where-blocks nest as child symbols; diagnostics extract the parser's line and col contract exactly; formatting calls the complete emitter directly. The compiler composes << as the reversed pipeline of >>. Five example sources drop their inferable role options.
Every role=kernel option across the example and benchmark corpus is gone (kernel is the default); textually identical consecutive declarations merge into plural-name families; markdown twins stay in sync with their sources. All fifty corpus files parse, compile, and round-trip through the emitter.
…rface Where-bound names no longer leak into the module namespace after the outer binding compiles. New end-to-end coverage: a term-autoencoder example exercising the signature, encoder, decoder, and loss flow with keyword-led op rules and doc comments; tests for file-loaded and multi-word lexicons, curry and trace methods, doc-comment attachment, and from_data tensors flowing through inference.
The parser() and chart_fold() walkers read their argument keywords from anonymous-token field constraints; previously every argument was dropped and no surface form of either expression could compile. A schema-bundled categorial chart parser joins the example gallery with end-to-end tests covering pattern schemas, bundle splicing, forward scoring with gradients, and the unknown-rule diagnostic.
A ternary contraction joins predicate and argument embeddings through a third-order interaction kernel, exercising operadic wiring that no chain of binary composition expresses. Tests cover einsum-spec inference, forward equivalence with the reference contraction, gradient flow, and the unknown-rule and did-you-mean diagnostics.
A parametric program template instantiated at two spreads exercises typed program parameters, labeled return tuples, score steps, and export selection, with tests asserting the bound parameter value changes the forward result and the score term moves log_joint by its exact analytic value. A Program without an exported morphism now raises a typed error naming its templates and the instantiation call instead of an attribute error swallowed by the module getattr.
The formula compiler builds ObjectDecl and ObserveStep values directly; didactic validation rejected the removed singular fields, which broke every fit() call and the analysis-pipelines tutorial.
…suite Embedded QVR snippets across the suite move to the current grammar; AST assertions read plural names, define bindings, and observe vars; cut-operator dispatch tests become declared-algebra composition tests with a positive reversed-composition case; the local-grammar skip gates are gone and CI builds the in-tree grammar unconditionally. A new diagnostics suite pins the rejected-input catalogue, message quality, and line accuracy.
… REPL Plate graphs, chain shapes, scope graphs, and the REPL's declaration lookup, statement replacement, and step renderers all consume the plural-name and vars-tuple AST shapes; observe sites no longer vanish from scope graphs or render with placeholder names.
…ded rules The FIFO agenda merges contributions pushed for an already-pending item through the semiring's plus, so contractive cycles reach their tolerance fixed point with at most one queue entry per item per wavefront instead of geometrically many. Bounded rule weights carry a joint sub-stochastic cap of one over the deduction's bounded-rule count, keeping interlocking cycles contractive for every parameter value.
…n prose Rewrites the composition-operator sections of the typing and expressions references to the two shipped operators, redirecting cross-algebra composition to change_base; converts composition-level, bundle, decoder, object, rule-turnstile, and top-level binding references throughout the guides, tutorials, and remaining example prose to the current surface.
Kernel is the default role, so the example, guide, and tutorial prose no longer writes it explicitly; the grammar-summary EBNF in the DSL overview and grammar reference now shows define bindings, brace constructor options, the level option, the single turnstile, and the colon-headed bundle and decoder forms.
…d bracket indices The object-slash walker read its direction from the grammar's direction field instead of scanning source bytes, so a comment carrying a backslash between the operator and its right operand no longer flips the residuation direction. A bracket index that is not a bare identifier now raises at parse time instead of silently dropping to an empty index tuple.
A role-less morphism defaults to a kernel; a deterministic '~ <expression>' initializer on one now raises with the fixing roles instead of silently discarding the initializer and building a random kernel.
String inner text is stored verbatim by the parser, so the emitter wraps it without re-escaping, keeping the round-trip a fixed point for strings that contain backslashes or quotes. An index draw argument with no indices raises rather than emitting the unparseable 'name[]'.
The per-file migrate loop catches MigrationError and exits 2 with a message instead of an uncaught traceback, and the per-decl parse gate raises MigrationError so callers can handle it uniformly.
…rride The QVR grammar is vendored in panproto-grammars-all, so parsing goes through the standard panproto registry. The QVR_USE_LOCAL_GRAMMAR override and its env gate are gone; the grammar-build helpers the Pygments lexer needs move to _grammar_build.py.
…butor guide The README requirements and feature bullets, the CONTRIBUTING DSL pipeline and distribution-family sections, and the architecture module map now describe the current tree-sitter-via-panproto parser, didactic AST models, and the 0.58.0 dependency floor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Homogenizes the QVR grammar surface so malformed input can no longer be silently reinterpreted, sands down the ergonomic sharp edges the grammar audit surfaced, and moves parsing onto the grammar now vendored in
panproto-grammars-all.Motivation
A deep audit of the grammar and compiler found that the surface was homogenized but its boundaries were sharp: tree-sitter error recovery plus lenient AST walkers would rewrite malformed input into a different valid model (a dropped minus sign, a vanished
samplestep) instead of rejecting it; misplaced or misspelled options were silently ignored; roughly half the language had no end-to-end coverage; and the vendored grammar lagged head-of-tree, forcing a dev-only local-grammar override. This PR fixes all of that.Refs panproto/panproto#213 (grammar revendored), panproto/panproto#214 (walker now surfaces MISSING anonymous tokens).
Changes
grammars/qvr/(grammar): optional option blocks with a kernel-default role; brace-delimited constructor options (Real 1 {low=-1.0, high=1.0}), which makes the grammar conflict-free; signed and scientific number literals; a single|-turnstile;bundle/decoderheaders use:; composition level moves into the option block;definebinds top-level morphism expressions; plural-nameobject/morphism/lexicon declarations; parenthesizedsample/observetuples; keyword-led encoderoprules; the nine algebra-tagged compose operators removed.dsl/parser/(hardening): parsing rejects anyERRORor missing node anywhere in the tree with the offending token's line and column; walkers cover the new node shapes; slash direction is read from the grammar field; bracket indices must be bare identifiers.dsl/compiler/: closed per-declaration option-key sets with did-you-mean suggestions; kernel-default role with a clear error on a deterministic initializer; plural-name expansion; the cut operators removed from dispatch.dsl/emit.py: a complete source emitter (every AST node) with a parse -> emit -> parse canonical-fixed-point suite; strings emit verbatim.cli/migrations/: av0.14.0 -> v0.15.0hop migrates every repository.qvrfile and fenced doc block; the CLI reports an unmigratable source and exits non-zero.stochastic/: the FIFO agenda merges pending contributions via the semiring's plus (cyclic deductions converge in linearly many pops);boundedrules carry a joint sub-stochastic cap.qvrgrammar vendored inpanproto-grammars-all; the floor moves topanproto>=0.58.0, and theQVR_USE_LOCAL_GRAMMARdev override is gone.API impact
.qvrsurface changes across the items above.qvr migrate --from v0.14.0 --to v0.15.0 <paths>rewrites sources automatically with byte-preserving span edits; every repository source and fenced doc block is already migrated. The compose operators>=>,*>,~>,||>,?>,&&>,+>,$>,%>have no rewrite and must be expressed via.change_base(...)or acompositiondeclaration.Tests
pytest -xpasses (2560 passed, via the standard vendored-grammar path).pyright src/quiversis clean (0 errors; the remaining warnings are the established didactic / mixin-MRO baseline).ruff checkandruff format --checkpass.Documentation
docs/updated (grammar reference, guides, tutorials, examples, semantics, getting-started).docs/developer/changelog.mdhas an entry for this change (0.15.0).docs/semantics/updated (composition operators, typing, expressions, composition rules).Checklist