Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
f7c0ea8
feat(dsl): homogenized grammar surface with deterministic parses
aaronstevenwhite Jul 2, 2026
286c23c
feat(dsl): reject malformed parse trees; walk the homogenized surface
aaronstevenwhite Jul 2, 2026
d005b62
feat(dsl): emit every AST node; add parse-emit-parse round-trip suite
aaronstevenwhite Jul 2, 2026
0e2f6ce
feat(transpile): plural morphism names, define bindings, typed draw args
aaronstevenwhite Jul 2, 2026
b63f624
test(dsl): rewrite tree-sitter corpus for the current surface
aaronstevenwhite Jul 2, 2026
8e66603
feat(dsl): grammar migration hop and corpus migration
aaronstevenwhite Jul 2, 2026
2b46514
feat(dsl): closed option-key sets, kernel default role, plural expansion
aaronstevenwhite Jul 2, 2026
f6be7b6
feat(lsp): plural-name symbols, define nesting, grammar-driven tokens
aaronstevenwhite Jul 2, 2026
4b8d5ac
refactor(dsl): drop inferable role options; use plural-name families
aaronstevenwhite Jul 2, 2026
331149a
feat(dsl): scope define where-blocks; cover structural and lexicon su…
aaronstevenwhite Jul 2, 2026
d2d74cb
test(dsl): assert where-bound defines stay scoped
aaronstevenwhite Jul 2, 2026
d20e5b5
feat(dsl): read parser and chart_fold keyword args; chart-parser example
aaronstevenwhite Jul 2, 2026
50a8da9
feat(dsl): bilinear tensor-contraction example with end-to-end tests
aaronstevenwhite Jul 2, 2026
f633ae1
docs(examples): add gallery nav and index entries for the new examples
aaronstevenwhite Jul 2, 2026
14477b0
feat(dsl): parametric partial-pooling example; typed template errors
aaronstevenwhite Jul 2, 2026
28a7324
fix(formulas): construct AST nodes with plural names and vars fields
aaronstevenwhite Jul 2, 2026
ad02dec
fix(formulas): read plural declaration names and observe vars in the …
aaronstevenwhite Jul 2, 2026
b06a531
feat(dsl): release 0.15.0
aaronstevenwhite Jul 2, 2026
fa8960e
test(dsl): migrate suite to the homogenized surface; add diagnostics …
aaronstevenwhite Jul 2, 2026
7eb5894
fix(analysis): read plural names and observe vars across analysis and…
aaronstevenwhite Jul 2, 2026
fa0626e
fix(stochastic): merge pending agenda contributions; jointly cap boun…
aaronstevenwhite Jul 3, 2026
4860b29
docs: link math concepts and align bounded-rule semantics with the ru…
aaronstevenwhite Jul 3, 2026
4ec3d01
docs: retire the eleven-operator surface and old declaration syntax i…
aaronstevenwhite Jul 3, 2026
6b68bf9
docs(changelog): sync mirrored changelog with the release entry
aaronstevenwhite Jul 3, 2026
8520b52
docs: drop redundant role=kernel and finish old-surface prose sweep
aaronstevenwhite Jul 3, 2026
6ac22fe
chore: lock quivers 0.15.0
aaronstevenwhite Jul 3, 2026
2d54628
fix(dsl): read slash direction from the grammar field; reject compoun…
aaronstevenwhite Jul 3, 2026
02ab4f7
fix(dsl): reject a deterministic initializer on a kernel morphism
aaronstevenwhite Jul 3, 2026
c796883
fix(dsl): emit strings verbatim; reject index draw args with no indices
aaronstevenwhite Jul 3, 2026
7db8f6e
fix(cli): surface migration errors per file as a clean exit code
aaronstevenwhite Jul 3, 2026
0956efd
test(dsl): pin the review-driven front-end regressions
aaronstevenwhite Jul 3, 2026
5fd0151
refactor(dsl): parse through the vendored grammar; drop the local ove…
aaronstevenwhite Jul 14, 2026
9dd1125
build: require panproto 0.58.0; document the vendored grammar in 0.15.0
aaronstevenwhite Jul 14, 2026
e168fcf
docs: describe the vendored-grammar pipeline in the README and contri…
aaronstevenwhite Jul 14, 2026
82b7917
style(dsl): apply ruff format to the release surface
aaronstevenwhite Jul 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,49 @@ All notable changes to the quivers library are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).

## [0.15.0] - 2026-07-02

### Changed

#### Grammar surface

- **Optional option blocks and a default role.** `morphism` and `contraction` declarations no longer require an option block, and a morphism without `role=` is a kernel. The other roles (`latent`, `observed`, `embed`, `discretize`, `let`) remain explicit. `morphism f : A -> B` is now a complete declaration.
- **Brace-delimited constructor options.** Continuous-space constructors take keyword options in braces: `Real 1 {low=-1.0, high=1.0}`. A trailing `[...]` always belongs to the enclosing declaration, so the constructor-versus-declaration attachment of an option block is decided by syntax rather than by parse-order luck, and the grammar is now conflict-free.
- **Signed and scientific numeric literals.** Option values, option-list items, option-call arguments, and constructor keyword values accept signed numbers; floats admit trailing-dot (`1.`), leading-dot (`.5`), and exponent (`1e-3`, `2.5e-3`) forms.
- **One turnstile.** Top-level `rule` declarations state their conclusion with `|-` (or `⊢`), the same marker deduction rules use.
- **Homogenized declaration headers.** `bundle NAME : [...]` and `decoder NAME : SIG` use the same `:` connective as every other declaration; the composition level moves into the option block (`composition NAME [level=algebra]`).
- **`define` binds morphism expressions.** The top-level value binding is `define NAME = EXPR`, with `where` blocks of nested defines scoped to the binding; the program-step `let` binds tensor arithmetic. The two binding forms no longer share a keyword.
- **Plural-name declarations.** `object A, B : V`, `morphism f, g : A -> B`, and lexicon entries `"a", "an" : CAT = LF` declare one item per name with shared shape and independent parameters, matching the existing `category` list form.
- **Parenthesized variable patterns.** `sample (a, b) <- f` and `return (a, b)` share one tuple shape; `observe` accepts the same pattern and reports a clear arity error for tuples, which its runtime does not support.
- **Keyword-led encoder rules.** Encoder constructor rewrites carry a leading `op` (`op App(fun, arg) |-> ...`), so an operator named `dim` or `init` cannot shadow the sibling entry keywords.
- **Composition operators.** Sequential composition is `>>` and `<<` (the same pipeline written right to left), transformation composition is `>>>`, and the tensor product is `@`. The algebra-tagged operator family (`>=>`, `*>`, `~>`, `||>`, `?>`, `&&>`, `+>`, `$>`, `%>`) is gone; algebra-tagged composition is expressed with `.change_base(...)` or a `composition` declaration.
- **Migration.** `qvr migrate --from v0.14.0 --to v0.15.0` rewrites sources across all of the above with byte-preserving span edits; the hop is registered on the migration chain with full coverage of the removed rules, and every repository `.qvr` file and fenced doc block is migrated. When a source cannot be migrated (a removed compose operator has no rewrite), the CLI reports the location and exits non-zero rather than raising.

#### Packaging

- **The QVR grammar ships vendored.** Parsing goes through the `qvr` grammar bundled in `panproto-grammars-all`; the floor moves to `panproto>=0.58.0` and `panproto-grammars-all>=0.58.0`, which vendor the current surface and surface tree-sitter's inserted (MISSING) tokens to the walker.

#### Diagnostics

- **Malformed input is rejected, never reinterpreted.** Parsing fails loudly on any damaged span anywhere in the tree, with the innermost offending token's line, column, and source snippet. Inputs that previously parsed to a silently different model, such as `[low=-1.0]` dropping its sign, `[scale=.5]` reading as `5.0`, a mis-bracketed `sample` step vanishing from the program, or a declaration truncated by an unbalanced `{` or `[`, now raise `ParseError` at the exact position.
- **Closed option-key sets with suggestions.** Every declaration and step kind validates its option keys; an unknown key raises at the entry's own position with a did-you-mean suggestion and the valid set, and a constructor key such as `low` on a declaration adds the hint to attach it with braces on the codomain.
- **Named failure for a missing `return`.** A program body without a return step reports that directly instead of leaking an internal registry message, and a `Program` holding only parametric templates raises a typed error naming the template and its instantiation call when `.domain` is touched.

### Added

- **A complete source emitter.** [`module_to_source`](https://FACTSlab.github.io/quivers/api/dsl/emit) covers every AST node kind, and a round-trip suite asserts that parsing, emitting, and re-parsing reaches a byte-identical canonical fixed point over every repository `.qvr` file and fenced doc block. LSP formatting uses it directly.
- **Four gallery examples with end-to-end tests.** A schema-bundled categorial chart parser (`schema`, `bundle`, `parser(...)`, `chart_fold(...)`), a bilinear tensor contraction (`contraction` with operadic three-way wiring), a term autoencoder (`signature`, `encoder`, `decoder`, `loss`), and parametric partial pooling (typed program parameters, labeled return tuples, `score` steps, `export` selection). Tests also cover file-loaded and plural-word lexicons, `define ... where`, doc comments, `.curry_left`/`.curry_right`/`.trace`, and `from_data` tensors flowing through inference.
- **A diagnostics regression suite** pinning the rejected-input catalogue and message quality, including line and column accuracy.

- **Editor and highlighting support.** The tree-sitter corpus (fifty cases), the VS Code and Zed extensions, and the Pygments lexer that renders `qvr` blocks in the docs all track the current surface.

### Fixed

- **`parser(...)` and `chart_fold(...)` keyword arguments.** The walkers read argument keywords from anonymous-token field constraints; every argument was previously dropped, so no surface form of either expression could compile.
- **`<<` composition** compiles as the reversed pipeline of `>>`, and reverse chains expand right to left in the transpiler.
- **Cyclic deductions converge in linearly many agenda pops.** The FIFO agenda merges contributions pushed for an item that is already pending via the semiring's plus, so a contractive cycle reaches its `tolerance` fixed point with at most one queue entry per item per wavefront; previously every contribution was enqueued separately and the pending-entry count grew geometrically with derivation depth.
- **`bounded` rule weights carry a joint sub-stochastic cap.** Each bounded rule's per-firing factor is strictly below one over the count of the deduction's bounded rules, so the total mass an item can push through them stays below 1 and interlocking cycles (e.g. an introduction / elimination pair over nested constructors) stay contractive for every parameter value; the previous per-rule cap of 1 left such systems free to diverge under fitting.

## [0.14.1] - 2026-07-01

### Fixed
Expand Down
124 changes: 62 additions & 62 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This guide covers setting up a development environment, understanding the projec

### Prerequisites

- Python 3.12 or later
- Python 3.14 or later
- pip or conda
- git

Expand Down Expand Up @@ -51,9 +51,9 @@ quivers/
├── src/quivers/ # Main package
│ ├── __init__.py
│ ├── categorical/ # Categorical algebra
│ ├── continuous/ # Continuous distributions (30+ families)
│ ├── continuous/ # Continuous distributions (40+ families)
│ ├── core/ # Core types and utilities
│ ├── dsl/ # QVR DSL (lexer, parser, compiler)
│ ├── dsl/ # QVR DSL (grammar walker, compiler, emitter)
│ ├── enriched/ # Enriched categories
│ ├── inference/ # Variational inference
│ ├── monadic/ # Monadic programs (draw, observe, return)
Expand All @@ -67,7 +67,7 @@ quivers/

### Type Hints

Include type hints in all function signatures. Use modern Python 3.12+ syntax:
Include type hints in all function signatures. Use modern Python 3.14+ syntax:

- Use `dict[K, V]` not `Dict[K, V]`
- Use `list[T]` not `List[T]`
Expand Down Expand Up @@ -129,7 +129,7 @@ Avoid stating the obvious. Comments should explain "why," not "what."

### Python Version and Modern Features

Maintain compatibility with Python 3.12 and later. Use modern features:
Maintain compatibility with Python 3.14 and later. Use modern features:

- Type union syntax: `X | None` instead of `Union[X, None]`
- Positional-only parameters: `def func(a, /, b)`
Expand All @@ -138,56 +138,55 @@ Maintain compatibility with Python 3.12 and later. Use modern features:

The QVR DSL processes `.qvr` files through these stages:

### 1. Tokenization (tokens.py)
### 1. Grammar and parsing (`grammars/qvr/`, `dsl/parser/`)

The lexer breaks source text into tokens. Each token carries type, value, line, and column:
There is no hand-written lexer or recursive-descent parser. The
grammar is a [tree-sitter](https://tree-sitter.github.io/) grammar
(`grammars/qvr/grammar.js`), compiled to a parser that ships vendored
in `panproto-grammars-all` and is served through panproto's
`AstParserRegistry`. Parsing a `.qvr` source yields a panproto schema
(the parse tree as vertices, edges, and field constraints), and the
walkers in `src/quivers/dsl/parser/` turn that schema into AST nodes.
The walker rejects any `ERROR` or missing node, so a malformed source
fails loudly with a line and column rather than parsing to a silently
different tree.

```python
class TokenType(Enum):
ALGEBRA = auto()
OBJECT = auto()
PROGRAM = auto()
DRAW = auto()
OBSERVE = auto()
...
```

Keywords like `program`, `draw`, `observe`, `return` map to specific token types. Operators (`->`, `>>`, `@`, `~`) and punctuation are also tokenized.

### 2. Parsing (parser.py)

The recursive descent parser transforms the token stream into an Abstract Syntax Tree (AST). The grammar is documented in the module docstring:

- **Statements**: algebra, object, morphism, space, continuous, stochastic, discretize, embed, program, let, output declarations
- **Programs**: blocks with draw/observe steps and return statements
- **Expressions**: identity, composition (>>), tensor product (@), marginalization
- **Types**: products (*), coproducts (+)
Editing the grammar means editing `grammar.js`, regenerating with
`tree-sitter generate`, and re-vendoring through `panproto-grammars-all`;
the `grammars/qvr/vcs/` panproto store and the `qvr migrate` chain
carry `.qvr` sources across grammar releases.

### 3. AST Nodes (ast_nodes.py)
### 2. AST nodes (`dsl/ast_nodes/`)

Each syntax construct maps to a dataclass:
Each grammar production maps to a [didactic](https://github.com/panproto/didactic)
model, not a dataclass. Statement variants live under a `dx.TaggedUnion`
keyed on `kind`; leaf records are `dx.Model` subclasses:

```python
@dataclass
class ProgramDecl(Statement):
name: str
params: tuple[str, ...] | None
domain: TypeExpr
codomain: TypeExpr
draws: tuple[DrawStep | LetStep, ...]
return_vars: tuple[str, ...]
return_labels: tuple[str, ...] | None
params: tuple[str, ...] | None = None
type_params: tuple[ProgramParam, ...] | None = None
domain: ObjectExpr
codomain: ObjectExpr
options: tuple[OptionEntry, ...] = ()
draws: tuple[ProgramStep, ...] = ()
return_vars: tuple[str, ...] = ()
return_labels: tuple[str, ...] | None = None
docs: tuple[str, ...] = ()
line: int = 0
col: int = 0
kind: Literal["program_decl"] = "program_decl"
```

### 4. Compilation (compiler.py)
### 3. Compilation (`dsl/compiler/`)

The compiler walks the AST and builds executable programs:

- Builds up a scope mapping variable names to values
- Executes draw steps by sampling from morphism distributions
- Executes observe steps by conditioning
- Processes let steps to bind computed expressions
- Returns final values according to the return statement
- Resolves object and morphism references against the module's declarations.
- Reads each declaration's option block against a closed key set, reporting an unknown key with a did-you-mean suggestion.
- Expands surface program steps (`sample`, `observe`, `marginalize`, `let`, `score`) into the internal bind IR.
- Emits a [`Program`][quivers.program.Program] whose morphism is a Kleisli arrow in the (discrete or continuous) Giry monad.

## Adding a New Distribution Family

Expand All @@ -197,31 +196,32 @@ To add a new continuous distribution family:

Create a new class in `src/quivers/continuous/families.py` or a new module:

Distribution families subclass `torch.distributions.Distribution`
(often alongside the measure-algebra combinators in
`quivers.continuous.measure`), so they compose with PyTorch's
sampling and scoring machinery:

```python
from dataclasses import dataclass
import torch
from torch import Tensor
from torch.distributions.distribution import Distribution

@dataclass(frozen=True)
class MyDistribution:
"""My custom probability distribution.

Parameters
----------
param1 : float
First parameter.
param2 : float
Second parameter.
"""
param1: float
param2: float
class MyDistribution(Distribution):
"""My custom probability distribution."""

def __init__(self, param1: Tensor, param2: Tensor) -> None:
self.param1 = param1
self.param2 = param2
super().__init__(batch_shape=param1.shape)

def sample(self, size: int) -> torch.Tensor:
def sample(self, sample_shape: torch.Size = torch.Size()) -> Tensor:
"""Draw samples from this distribution."""
pass
...

def log_prob(self, value: torch.Tensor) -> torch.Tensor:
"""Compute log probability density."""
pass
def log_prob(self, value: Tensor) -> Tensor:
"""Compute the log probability of ``value``."""
...
```

### 2. Register in the DSL
Expand All @@ -234,12 +234,12 @@ Create test cases in `tests/continuous/`:

```python
def test_mydistribution_sample_shape():
dist = MyDistribution(param1=1.0, param2=2.0)
dist = MyDistribution(param1=torch.tensor(1.0), param2=torch.tensor(2.0))
samples = dist.sample(1000)
assert samples.shape == (1000,)

def test_mydistribution_log_prob():
dist = MyDistribution(param1=1.0, param2=2.0)
dist = MyDistribution(param1=torch.tensor(1.0), param2=torch.tensor(2.0))
value = torch.tensor([0.5])
log_prob = dist.log_prob(value)
assert log_prob.shape == ()
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@

Quivers is a functional probabilistic programming language for PyTorch. The surface will look familiar if you have used Pyro, NumPyro, Stan, or PyMC. But it has a few distinguishing features:

- **Programs are first-class composable typed values.** A program has a domain, codomain, algebra, and effect signature (`! Sample, Score, Marginal, Pure`), checked at compile time. Programs compose with `>>`, parallel-compose with `@`, change base across algebras with `change_base`, and marginalize discrete latents with `marginalize z : K <- ... in { ... }`.
- **Shared substrate for inference, deduction, and structural compression.** A CKY parser in a `deduction { atoms ... rule ... }` block, a transformer-as-encoder over a `signature { ... }` block, and a Bayesian regression all compile to the same underlying semantics, with the same composition operators, and can therefore compose with each other.
- **Programs are first-class composable typed values.** A program has a domain, codomain, algebra, and effect signature (`[effects=[Sample, Score, Marginal]]`), checked at compile time. Programs compose with `>>`, parallel-compose with `@`, change base across algebras with `change_base`, and marginalize discrete latents with a scoped `marginalize z : K <- ...` block.
- **Shared substrate for inference, deduction, and structural compression.** A CKY parser in a `deduction` block (its `atoms`, `rule`, and `lexicon` entries), a transformer-as-encoder over a `signature` block, and a Bayesian regression all compile to the same underlying semantics, with the same composition operators, and can thus compose with each other.
- **Algebra-parametric semantics.** Programs can be parameterized by eleven built-in or user-defined algebras. Homomorphisms between algebras are values you can transport models along, with the laws checked at compile time.

It also has some features you are used to from other PPLs:
Expand Down Expand Up @@ -79,7 +79,7 @@ The full walkthrough is in the [tutorial](https://FACTSlab.github.io/quivers/tut
## Documentation

- [**Tutorial**](https://FACTSlab.github.io/quivers/tutorials/): the QVR DSL tutorial walks probabilistic-programming users from linear regression to inference-algorithm choice with PyMC, NumPyro, and Stan equivalents shown side-by-side, while the Python API tutorial covers the typed categorical surface.
- [**Examples gallery**](https://FACTSlab.github.io/quivers/examples/): 36 end-to-end models covering regression, latent-variable, state-space, language models, seq2seq, and formal grammars.
- [**Examples gallery**](https://FACTSlab.github.io/quivers/examples/): 41 end-to-end models covering regression, latent-variable, state-space, language models, seq2seq, and formal grammars.
- [**Conceptual guides**](https://FACTSlab.github.io/quivers/guides/): feature-area deep dives.
- [**API reference**](https://FACTSlab.github.io/quivers/api/): the typed Python surface.
- [**Denotational semantics**](https://FACTSlab.github.io/quivers/semantics/): the meaning of every well-typed program in a $\mathcal{V}$-enriched symmetric monoidal closed category.
Expand All @@ -98,7 +98,7 @@ cd quivers
pip install -e ".[dev]"
```

Requirements: Python 3.14+, PyTorch 2.0+, didactic 0.7.1+, panproto 0.47.3+, panproto-grammars-all 0.47.3+.
Requirements: Python 3.14+, PyTorch 2.0+, didactic 0.7.1+, panproto 0.58.0+, panproto-grammars-all 0.58.0+.

Optional extras:

Expand Down
Loading
Loading