From 6f550c3313ed6dc0be07644d0519ad422a637351 Mon Sep 17 00:00:00 2001 From: Dylan Thomas Date: Mon, 6 Apr 2026 22:09:26 -0600 Subject: [PATCH 1/6] docs: requirements and plan for named tables and array type (#118) Add brainstorm requirements document and implementation plan for the named tables and array type feature. Tables become computation sources via directive-based naming, with element-wise arithmetic and aggregate functions operating on column arrays. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...06-named-tables-and-arrays-requirements.md | 103 +++ ...6-003-feat-named-tables-and-arrays-plan.md | 596 ++++++++++++++++++ 2 files changed, 699 insertions(+) create mode 100644 docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md create mode 100644 docs/plans/2026-04-06-003-feat-named-tables-and-arrays-plan.md diff --git a/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md b/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md new file mode 100644 index 00000000..06578350 --- /dev/null +++ b/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md @@ -0,0 +1,103 @@ +--- +date: 2026-04-06 +topic: named-tables-and-arrays +--- + +# Named Tables and Array Type + +## Problem Frame + +CalcMark excels at linear calculations but can't work with tabular data natively. Users building living documents — consulting proposals, budget plans, project cost models — must duplicate data between markdown tables (for presentation) and calc blocks (for computation). There's no way to say "sum this column" or "multiply these two columns together." + +The canonical use cases: +- **Consulting SOW**: A table of roles, rates, and headcount. CalcMark computes line totals and grand totals from the table data, with per-row computed values displayed in the table and totals referenced elsewhere via interpolation. +- **Project budget**: Regional spend by quarter in a table. CalcMark sums columns and rows, renders totals via interpolation. +- **Published living documents**: A git-tracked `.cm` file rendered to HTML (like evidence.dev), where changing one input (e.g., an exchange rate) recomputes everything downstream. + +## Requirements + +**Named Tables** + +- R1. A markdown table becomes a named data source when preceded by a directive comment: ``. The directive declares both the table name and the column names used in calc block references. The markdown header row is display-only. +- R2. Names in the directive undergo simple normalization: lowercased, unicode whitespace replaced with underscores. The resulting names must be valid CalcMark identifiers. No complex normalization (stripping emoji, transliterating accented chars, etc.) — if the normalized name isn't a valid identifier, it's an error. +- R3. The number of column names in the directive must match the number of columns in the markdown table. A mismatch produces a diagnostic. +- R4. Column values are parsed as CalcMark literal types (currencies, quantities, numbers, etc.) using existing parsing rules. Table data cells must contain literal values only — no interpolation tags or expressions. +- R5. Named tables are registered in the environment where they appear in the document. Calc blocks below can reference them; calc blocks above cannot (preserves top-to-bottom execution). +- R5a. Referencing a nonexistent table or column produces a diagnostic error (e.g., "column 'revenue' not found in table 'sales'"). Same treatment as referencing an undefined variable. +- R5b. Tables without a directive are inert markdown — no name, no column extraction, no impact on the environment. This preserves backward compatibility. + +**Array Type** + +- R6. A new `Array` type joins the CalcMark type system. An array is an ordered collection of values. All elements in a column-derived array must be the same CalcMark type (e.g., all Currency, all Number). Columns with mixed types produce a diagnostic at table registration time. +- R7. Table columns are arrays. `sales.q1` evaluates to the array of values in the `q1` column of the `sales` table. This requires a new dot-access syntax (`identifier.identifier`) in the lexer, parser, and AST — distinct from the existing `@globals.field` directive syntax. + +**Element-wise Arithmetic** + +- R8. Binary operations (`+`, `-`, `*`, `/`) between two arrays of the same length produce a new array (element-wise). Mismatched lengths produce a type error diagnostic referencing both operands. E.g., `rates.rate * rates.hc` → array of per-role costs. +- R9. Binary operations between an array and a scalar apply the scalar to every element. E.g., `rates.rate * 1.1` → array of increased rates. + +**Aggregate Functions** + +- R10. `sum()`, `avg()`, `min()`, `max()`, `count()` accept a single array argument and return a scalar. Note: `min()`, `max()`, and `count()` are entirely new functions that do not exist in the codebase today. +- R11. These functions also retain their existing variadic scalar behavior: `sum(a, b, c)` still works. Mixed array+scalar arguments are a type error — `sum(array)` uses the aggregate path; `sum(scalar, scalar, ...)` requires 2+ arguments (existing behavior). This subsumes the existing `sum()` brainstorm. +- R12. Aggregates compose with element-wise arithmetic: `sum(rates.rate * rates.hc)` computes element-wise product then sums. + +**Display and Output** + +- R13. Scalar computed results are displayable via existing `{{interpolation}}` syntax. +- R14. Arrays display as comma-separated values in square brackets when shown directly (e.g., in JSON output or when assigned and displayed). +- R15. When `{{array_var}}` appears in a table cell and the variable is an Array type, each data row receives the corresponding array element (row 0 gets element 0, etc.). Array length must match the number of data rows; a mismatch produces a diagnostic. + +## Success Criteria + +- A consulting SOW with a rates table can compute total cost via `sum(rates.rate * rates.hc)` without duplicating any data between the table and calc blocks. +- Per-row computed values (e.g., `line_cost = rates.rate * rates.hc`) can be displayed in the table via `{{line_cost}}` array interpolation, with each row showing its corresponding value. +- Computed totals can be referenced elsewhere in the document via `{{total_cost}}` interpolation. +- Changing one value in a table (e.g., a rate) recomputes all downstream results correctly. +- Existing CalcMark documents with markdown tables continue to work unchanged (tables without names are inert). +- `sum()` works on both arrays and variadic scalars. + +## Scope Boundaries + +- **No array literals** — arrays come from tables only. Literal syntax (`[1, 2, 3]`) is a future addition if needed. +- **No indexing or row access** — `arr[0]`, `sales.row("East")`, filtering, slicing are all out of scope. +- **No cell-level formulas** — table cells contain literal data only, not expressions. Computation happens in calc blocks. +- **No CSV/file loading** — data must be inline in the document. File loading (`table sales from "data.csv"`) is a natural future extension but out of scope. +- **No dependency graph execution** — execution remains top-to-bottom. Tables are data declarations, not bidirectional spreadsheet cells. +- **No DuckDB or external query engine** — arrays are native CalcMark values, evaluated by the existing interpreter. + +## Key Decisions + +- **Directive-based naming**: Table and column names are declared explicitly via ``. Heading text and header row text are display-only. This eliminates normalization ambiguity — you know exactly what identifier to use in calc blocks because you wrote it. Renaming a display heading or header never breaks references; changing the directive name does, and produces diagnostics on broken references. +- **First-class array type**: Arrays are real values in the type system, not opaque references. This enables element-wise arithmetic and composability with aggregates, which is essential for the consulting SOW case. +- **Strict column typing**: All values in a column must be the same CalcMark type. Mixed-type columns produce a diagnostic at registration. This avoids the "homogeneous-ish" ambiguity and keeps the type system predictable. +- **Tables only (no array literals)**: Keeps the feature grounded in the "document that computes" identity. Arrays serve tables, not general-purpose programming. +- **Top-to-bottom preserved**: No execution model change. Tables register data where they appear; calc blocks reference it below. Interpolation handles display (including existing forward-reference support for scalar results). +- **Element-wise arithmetic in v1**: Without it, users must pre-compute line totals in the table itself, defeating the purpose. `sum(rates.rate * rates.hc)` is the killer expression. +- **Array interpolation in table cells**: When `{{array_var}}` appears in a table column, each row receives the corresponding element. This enables the per-row computed column use case without introducing cell-level formulas or indexing. +- **Broken reference diagnostics**: Renaming headings, columns, or tables that breaks downstream references produces diagnostics, consistent with how undefined variables are handled today. + +## Dependencies / Assumptions + +- The existing `sum()` brainstorm (`docs/brainstorms/2026-03-09-sum-function-brainstorm.md`) is subsumed by this work. `sum()` should be implemented as part of this feature with both variadic scalar and array support. +- Interpolation (`{{var}}`) already supports forward references for scalar values. Array interpolation in table cells (R15) is a new behavior that extends the existing interpolation system. +- Column value parsing reuses the existing lexer/parser for individual CalcMark literal values. +- Table data cells contain literals only — no `{{var}}` tags in data cells (avoids chicken-and-egg with forward references during table registration). +- The evaluator must be modified to inspect TextBlocks for named table syntax during the evaluation pass, extracting column data and registering it in the environment. This is a change to the current block-type separation where TextBlocks are only processed for interpolation after evaluation. + +## Outstanding Questions + +### Deferred to Planning + +- [Affects R1][Technical] How should collisions be handled when two directives declare the same table name? +- [Affects R5][Technical] Should tables in the same calc block section (between `---` separators) be accessible within that section, or only in subsequent sections? +- [Affects R5][Technical] What happens when a table name collides with an existing variable name — error, shadow, or undefined? +- [Affects R7][Technical] Dot-access syntax requires a new AST node (e.g., `MemberAccess`) distinct from `DirectiveRef`. How should this interact with the existing `@globals.field` syntax? +- [Affects R8][Technical] What happens when element-wise operations produce incompatible types (e.g., `currency_array / currency_array` → plain numbers)? Follow existing CalcMark type arithmetic rules. +- [Affects R6][Needs research] What is the right representation for the Array type — `[]types.Type` or a dedicated struct with metadata (source table, column name)? +- [Affects R5][Needs research] Should table registration happen during document parsing (new block type) or during evaluation? This affects the spec/ vs impl/ dependency boundary. +- [Affects R15][Technical] How does array interpolation in table cells interact with the existing interpolation pass? The interpolation system needs to be table-structure-aware to map array elements to rows. + +## Next Steps + +-> `/ce:plan` for structured implementation planning diff --git a/docs/plans/2026-04-06-003-feat-named-tables-and-arrays-plan.md b/docs/plans/2026-04-06-003-feat-named-tables-and-arrays-plan.md new file mode 100644 index 00000000..c7d0ef31 --- /dev/null +++ b/docs/plans/2026-04-06-003-feat-named-tables-and-arrays-plan.md @@ -0,0 +1,596 @@ +--- +title: "feat: Named Tables and Array Type" +type: feat +status: active +date: 2026-04-06 +origin: docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md +--- + +# feat: Named Tables and Array Type + +## Overview + +Add named markdown tables and a first-class Array type to CalcMark, enabling tabular data to serve as a computation source. Users declare table and column names via an HTML comment directive (``), and calc blocks reference columns as arrays via dot-access syntax (`rates.rate`). Element-wise arithmetic and aggregate functions operate on arrays, with results displayable via interpolation — including per-row array interpolation in table cells. + +## Problem Frame + +CalcMark users building living documents (consulting SOWs, budget plans, project cost models) must duplicate data between markdown tables and calc blocks. There is no way to say "sum this column" or "multiply these two columns." This feature makes the markdown table the single source of truth for both display and computation. (see origin: `docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md`) + +## Requirements Trace + +- R1. Directive-based table naming: `` +- R2. Simple normalization of directive names (lowercase, whitespace → underscore) +- R3. Column count in directive must match table column count +- R4. Column values parsed as CalcMark literal types +- R5. Tables registered in environment top-to-bottom; R5a broken-ref diagnostics; R5b tables without directives are inert +- R6. New Array type — ordered, same-type elements, diagnostic on mixed types +- R7. Dot-access syntax: `sales.q1` → column array +- R8. Element-wise arithmetic (same-length arrays); length mismatch → diagnostic +- R9. Array-scalar broadcasting +- R10. Aggregate functions: `sum()`, `avg()`, `min()`, `max()`, `count()` on arrays +- R11. Retain variadic scalar behavior; mixed array+scalar args → error +- R12. Aggregates compose with element-wise: `sum(rates.rate * rates.hc)` +- R13. Scalar interpolation unchanged +- R14. Array display as `[val, val, ...]` +- R15. Array interpolation in table cells: each row gets corresponding element + +## Scope Boundaries + +- No array literals — arrays come from tables only +- No indexing or row access (`arr[0]`, filtering, slicing) +- No cell-level formulas — table data cells are literals only +- No CSV/file loading +- No dependency graph — top-to-bottom preserved +- No DuckDB or external query engine +- No NL syntax for new functions in v1 (functional syntax only for min/max/count) + +## Context & Research + +### Relevant Code and Patterns + +- **Type addition checklist**: `docs/solutions/logic-errors/adding-new-type-fraction-cross-layer-checklist.md` — 9-layer process. Classifier is the most commonly missed layer. +- **Expression form addition**: `docs/solutions/language-features/directive-as-value-cross-layer-learnings.md` — 12-layer cookbook including interpolation, autocomplete, LSP. +- **Function addition pattern**: `spec/features/registry.go` (metadata), `spec/types/param_types.go` (FunctionSpecs), `impl/interpreter/functions.go` (BuiltinFunctions + eval). Follow unified registry pattern from `docs/solutions/code-organization/unified-feature-registry-three-to-one.md`. +- **Operator dispatch**: `impl/interpreter/operators.go` — normalization blocks at top of `evalBinaryOperation()`, then type-assertion dispatch. See `docs/solutions/feature-gaps/rate-type-arithmetic-widening.md` for the pattern. +- **Existing dot-access**: `@globals.field` uses `DirectiveRef` AST node. DOT token only emitted in `@globals` context (`spec/lexer/lexer.go:~1115`). +- **Interpolation**: `impl/document/interpolation.go` — regex `\{\{\s*(@?\w+(?:\.\w+)?)\s*\}\}` already matches dotted names but resolves via flat `env[ref]` lookup. +- **Evaluator block loop**: `impl/document/evaluator.go:129-140` — TextBlocks only get `checkTextBlockForLikelyCalculations`. Table extraction inserts here. +- **Environment**: `impl/interpreter/environment.go` — flat `map[string]types.Type`. Table stored as a new Type value. + +### Institutional Learnings + +- Always set `Range` on new AST nodes (from NL function Range learning) +- Classifier (`spec/classifier/classifier.go` AND `spec/document/detector.go`) is the #1 missed layer +- Defense-in-depth: validation in both semantic checker and interpreter +- `InterpolatedSource()` not `Source()` for display +- New keyword categories must be registered in `IsReservedKeywordToken()` and diagnostic pipeline +- Test full type dispatch matrix for new type combinations + +## Key Technical Decisions + +- **Table as a Type**: A new `*Table` type in `spec/types/` implements the `Type` interface and is stored in the environment as `vars["rates"] = &Table{...}`. `MemberAccess` on a Table returns the column's Array. This leverages the existing environment without adding a parallel storage mechanism. +- **Array as a Type**: A new `*Array` type in `spec/types/` wraps `[]types.Type` with element-type metadata. Implements `Type` interface. Stored in environment when assigned to a variable. +- **MemberAccess AST node**: New node `MemberAccess{Object Node, Field string}` in `spec/ast/nodes.go`. Distinct from `DirectiveRef`. Parsed when `IDENTIFIER DOT IDENTIFIER` is encountered (not preceded by `@`). +- **DOT token generalization**: Extend the lexer to emit DOT tokens after any IDENTIFIER when followed by `.` and another identifier-start character. The existing `@globals` DOT path continues to work via `DirectiveRef` parsing; the new MemberAccess path handles non-`@` cases. +- **Table extraction in evaluator**: The evaluator's block processing loop gains a new step for TextBlocks: scan source lines for `` directives, parse the subsequent markdown table, extract column data, and register in the environment. This happens during evaluation (not document parsing) so it participates in the top-to-bottom ordering. +- **Array interpolation**: The interpolation system gains table-structure awareness. When `{{var}}` resolves to an Array and the tag is inside a markdown table row, the interpolation pass maps array elements to rows by index. +- **Aggregate function dispatch**: `sum()` and `avg()` gain an array path alongside their existing variadic scalar path. Dispatch on first argument type: if Array, use aggregate path; if scalar, use variadic path. `min()`, `max()`, `count()` are new functions with array-only support initially. + +## Open Questions + +### Resolved During Planning + +- **Where does table extraction happen?** In the evaluator (`impl/document/evaluator.go`), during the block processing loop. TextBlocks are scanned for directive comments. This keeps table registration in the evaluation pass (top-to-bottom ordering) without modifying the document parser or creating a new block type. +- **How is MemberAccess distinguished from DirectiveRef?** By the `@` prefix. `@globals.field` → DirectiveRef (existing path). `sales.q1` → MemberAccess (new path). The parser dispatches based on whether the expression starts with `AT_SIGN`. +- **How does the interpolation regex handle dotted names for arrays?** The regex already matches `\w+\.\w+`. The resolution logic changes: if `ref` contains a `.`, split into `table.column`, look up the table in env, get the column Array. For array-in-table-cell display, detect markdown table row context and index by row position. +- **Should Table implement Type?** Yes. It makes the environment's `map[string]types.Type` work without modification. `Table.String()` returns a summary like `table(3 rows, 4 cols)`. + +### Deferred to Implementation + +- Exact identifier validation rules for directive names (which invalid characters produce diagnostics) +- How table name collisions with existing variables are reported (likely same as variable redefinition: diagnostic error) +- Whether `NormalizeForDisplay` needs Array-specific behavior or delegates to per-element formatting +- Interaction between `@scale` directive and table cell values + +## High-Level Technical Design + +> *This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.* + +``` +Document source: + + + | Role | Rate | HC | + |--------|---------|-----| + | Senior | $250/hr | 3 | + | Junior | $150/hr | 5 | + + --- + + line_costs = rates.rate * rates.hc + total = sum(line_costs) + +Evaluation flow: + + 1. Evaluator encounters TextBlock containing directive + table + 2. Parse directive: name="rates", columns=["role","rate","hc"] + 3. Parse markdown table rows, lex each cell as CalcMark literal + 4. Build column arrays: + role → Array[String?] (non-numeric, accessible but errors on arithmetic) + rate → Array[Rate: $250/hr, $150/hr] + hc → Array[Number: 3, 5] + 5. Create Table{columns: map[string]*Array{...}} + 6. Register: env["rates"] = &Table{...} + + 7. Evaluator reaches CalcBlock + 8. Parse: line_costs = rates.rate * rates.hc + 9. Evaluate MemberAccess "rates.rate" → Array[$250/hr, $150/hr] + 10. Evaluate MemberAccess "rates.hc" → Array[3, 5] + 11. Element-wise multiply: Array[$750/hr, $750/hr] + 12. Assign: env["line_costs"] = Array[$750/hr, $750/hr] + + 13. Parse: total = sum(line_costs) + 14. Evaluate sum(Array) → aggregate → $1,500/hr + 15. Assign: env["total"] = Rate($1,500/hr) + +Interpolation pass (post-evaluation): + - {{total}} in prose → "$1,500/hr" (scalar, existing behavior) + - {{line_costs}} in table cell row 0 → "$750/hr" (array element 0) + - {{line_costs}} in table cell row 1 → "$750/hr" (array element 1) +``` + +## Implementation Units + +### Phase 1: Type Foundation + +- [ ] **Unit 1: Array and Table Types** + +**Goal:** Introduce `Array` and `Table` types implementing the `Type` interface. + +**Requirements:** R6, R14 + +**Dependencies:** None — foundational types that everything else builds on. + +**Files:** +- Create: `spec/types/array.go` +- Create: `spec/types/table.go` +- Test: `spec/types/array_test.go` +- Test: `spec/types/table_test.go` + +**Approach:** +- `Array` struct: `Elements []types.Type`, `ElementType string` (for display/diagnostics). Constructor validates all elements are compatible types; returns error on mixed types. `String()` returns `[val1, val2, ...]` using element `String()`. +- `Table` struct: `Name string`, `Columns map[string]*Array`, `ColumnOrder []string` (preserves declaration order), `RowCount int`. `String()` returns summary. `Column(name string) (*Array, bool)` accessor. +- Both implement `Type` interface. + +**Patterns to follow:** +- `spec/types/number.go`, `spec/types/quantity.go` for struct + constructor + String() pattern +- `spec/types/param_types.go` for ArgType constants if needed + +**Test scenarios:** +- Happy path: Create Array of Numbers, verify String() output is `[1, 2, 3]` +- Happy path: Create Array of Currencies, verify String() preserves currency symbols +- Happy path: Create Table with multiple columns, verify Column() accessor returns correct arrays +- Edge case: Create Array with zero elements — should succeed (empty table column) +- Edge case: Create Table with zero rows — should succeed (header-only table) +- Error path: Create Array with mixed types (Number + Currency) — constructor returns error +- Error path: Table.Column("nonexistent") — returns nil, false + +**Verification:** +- `go test ./spec/types/...` passes +- Array and Table values can be stored in and retrieved from the Environment + +--- + +### Phase 2: Syntax (Lexer, AST, Parser) + +- [ ] **Unit 2: MemberAccess AST Node** + +**Goal:** Add a `MemberAccess` AST node for `table.column` dot-access expressions. + +**Requirements:** R7 + +**Dependencies:** None + +**Files:** +- Modify: `spec/ast/nodes.go` (add MemberAccess node and ContainsScaleRef case) + +**Approach:** +- `MemberAccess{Object Node, Field string, Range *Range}` — Object is typically an `Identifier`, Field is the column name string. +- Add case to `ContainsScaleRef()` that recurses into `Object`. +- `String()` returns `Object.Field`. + +**Patterns to follow:** +- `DirectiveRef` in `spec/ast/nodes.go` for a dot-access node pattern +- Always set `Range` from token position (learning: NL function Range) + +**Test scenarios:** +- Happy path: MemberAccess node has correct String() representation +- Happy path: ContainsScaleRef correctly recurses into MemberAccess.Object +- Happy path: GetRange() returns the range set at construction + +**Verification:** +- `go test ./spec/ast/...` passes +- Node can be constructed, printed, and inspected + +--- + +- [ ] **Unit 3: Lexer DOT Token Generalization** + +**Goal:** Emit DOT tokens after regular identifiers (not just `@globals`), enabling `sales.q1` to tokenize as `IDENTIFIER DOT IDENTIFIER`. + +**Requirements:** R7 + +**Dependencies:** Unit 2 (AST node exists) + +**Files:** +- Modify: `spec/lexer/lexer.go` (DOT emission logic) +- Modify: `spec/lexer/token.go` (if new token types needed for min/max/count) +- Test: `spec/lexer/lexer_test.go` + +**Approach:** +- Current DOT emission is guarded by `dirToken.Value == "globals" && l.currentChar() == '.'`. Generalize: after emitting an IDENTIFIER token, if the next char is `.` and the char after that is an identifier-start character (letter or underscore), emit DOT and continue to the next IDENTIFIER. +- Must not break: decimal numbers (`1.5`), existing `@globals.field` path, range notation if any. +- Add `FUNC_MIN`, `FUNC_MAX`, `FUNC_COUNT` tokens and their `ReservedKeywords` entries. + +**Patterns to follow:** +- Existing DOT handling in `spec/lexer/lexer.go:~1115` +- `FUNC_SUM`, `FUNC_AVG` token pattern for new function tokens + +**Test scenarios:** +- Happy path: `sales.q1` tokenizes as `IDENTIFIER("sales") DOT IDENTIFIER("q1")` +- Happy path: `@globals.tax_rate` still tokenizes correctly (regression) +- Happy path: `1.5` still tokenizes as a number, not `NUMBER DOT NUMBER` +- Happy path: `min`, `max`, `count` tokenize as their function tokens +- Edge case: `a.b.c` tokenizes as `IDENTIFIER DOT IDENTIFIER DOT IDENTIFIER` (parser will reject nested access) +- Edge case: `a.123` does NOT emit DOT (digit after dot → not identifier start) +- Error path: reserved keywords like `sum.x` — `sum` tokenizes as FUNC_SUM, then `.x` is an error (function tokens don't get dot-access) + +**Verification:** +- `go test ./spec/lexer/...` passes +- Existing lexer tests pass (no regressions in number parsing) + +--- + +- [ ] **Unit 4: Parser — MemberAccess and New Function Tokens** + +**Goal:** Parse `identifier.field` into MemberAccess nodes. Parse `min()`, `max()`, `count()` function calls. + +**Requirements:** R7, R10 + +**Dependencies:** Unit 2 (AST node), Unit 3 (lexer tokens) + +**Files:** +- Modify: `spec/parser/primary.go` (`parsePrimary` — identifier case, function call case) +- Test: `spec/parser/parser_test.go` +- Create: `testdata/spec/valid/features/named_tables.cm` +- Create: `testdata/spec/invalid/features/named_tables.cm` + +**Approach:** +- In `parsePrimary()`, after matching an `IDENTIFIER` token: peek for `DOT`. If DOT follows, consume it, expect another `IDENTIFIER`, build `MemberAccess{Object: &Identifier{Name: first}, Field: second}`. Set Range from first token. +- Reject nested dot access (`a.b.c`) with a clear diagnostic: "nested dot access is not supported." +- Add `FUNC_MIN`, `FUNC_MAX`, `FUNC_COUNT` to the `p.match(...)` call in the function call section of `parsePrimary()`. +- Add arg count validation for new functions in `parseFunctionCall()`. +- **P0 fix**: Relax parser-level arg count validation for `sum()` and `avg()` from `< 2` to `< 1`. The parser currently rejects `sum(array)` at parse time (`spec/parser/primary.go:532`). With arrays, these functions accept either a single array arg or 2+ scalar args — the interpreter (Unit 8) validates based on argument type, not the parser based on count. (NL syntax for min/max/count is deferred to v2; this unit covers functional syntax only.) + +**Patterns to follow:** +- DirectiveRef parsing in `parsePrimary()` for the `AT_SIGN → IDENTIFIER → DOT → IDENTIFIER` pattern +- `parseFunctionCall()` for arg count validation + +**Test scenarios:** +- Happy path: `x = sales.q1` parses as Assignment with MemberAccess RHS +- Happy path: `sum(sales.q1)` parses as FunctionCall with MemberAccess argument +- Happy path: `sum(a.b * c.d)` parses with element-wise expression inside function call +- Happy path: `min(x, y)`, `max(x, y)`, `count(arr)` parse correctly +- Error path: `a.b.c` produces "nested dot access is not supported" diagnostic +- Error path: `min()` with no args produces argument count error +- Integration: Parsing still works for `@globals.field` (DirectiveRef regression) + +**Verification:** +- `go test ./spec/parser/...` passes +- Golden test files parse correctly + +--- + +### Phase 3: Document Pipeline + +- [ ] **Unit 5: Table Directive Parsing and Table Extraction** + +**Goal:** Detect `` directives in TextBlocks, parse the subsequent markdown table, extract column data as Arrays, and register as a Table in the environment. + +**Requirements:** R1, R2, R3, R4, R5, R5a, R5b, R6 + +**Dependencies:** Unit 1 (Array and Table types) + +**Files:** +- Create: `impl/document/table_extraction.go` +- Create: `impl/document/table_extraction_test.go` +- Modify: `impl/document/evaluator.go` (block processing loop, ~line 136) + +**Approach:** +- New function `extractNamedTables(tb *document.TextBlock, env *Environment) ([]*types.Table, []document.Diagnostic)`: + - Scan TextBlock source lines for regex matching ``. + - Parse directive: extract table name + column names. Apply normalization (lowercase, whitespace → underscore). Validate names are valid identifiers. + - Find the next markdown table in subsequent lines (detect pipe-delimited rows, skip separator row). + - Validate column count matches directive. + - For each data row, split by `|`, trim whitespace, lex each cell value using the existing lexer/parser to produce a `types.Type`. + - Build column Arrays (validate same-type within each column). + - Construct Table, register in environment via `env.Set(name, table)`. +- In the evaluator's block loop, add a TextBlock case that calls `extractNamedTables` before `checkTextBlockForLikelyCalculations`. Tables are registered in the environment immediately, making them available to subsequent CalcBlocks. +- **All three evaluator entry points must extract tables**: `Evaluate()` (~line 136), `EvaluateBlock()` (two-pass, ~line 224), and `EvaluateAffectedBlocks()` (~line 391). Currently all three only process CalcBlocks for TextBlocks. Without this, TUI incremental evaluation will hold stale table data when a TextBlock is edited. +- Diagnostics for: invalid name, column count mismatch, unparseable cell value, mixed-type column, duplicate table name. + +**Patterns to follow:** +- `checkTextBlockForLikelyCalculations` for the pattern of inspecting TextBlock source lines during evaluation +- `impl/document/interpolation.go` for line-by-line TextBlock processing + +**Test scenarios:** +- Happy path: Directive + table with 3 numeric columns → 3 Arrays registered +- Happy path: Directive with Currency column → Array of Currency values +- Happy path: Table with mixed column types (text + numeric) — each column gets its own type +- Happy path: Multiple tables in different TextBlocks → both registered +- Edge case: Directive with whitespace in names → normalized correctly (`Rate Per Hour` → `rate_per_hour`) +- Edge case: Table with empty cells → diagnostic on that cell +- Edge case: Directive with no subsequent table → diagnostic +- Error path: Column count mismatch (directive says 3, table has 4) → diagnostic +- Error path: Mixed types within a single column ($100 and 50%) → diagnostic +- Error path: Duplicate table name → diagnostic +- Error path: Invalid identifier after normalization → diagnostic +- Integration: TextBlock without directive is untouched (backward compatibility) + +**Verification:** +- `go test ./impl/document/...` passes +- A `.cm` file with a directive+table evaluates without errors and populates the environment + +--- + +### Phase 4: Semantic Checker and Interpreter + +- [ ] **Unit 6: Semantic Checker — MemberAccess and Array Awareness** + +**Goal:** Add MemberAccess to the semantic checker's node dispatch. Validate that the object part is a known variable (when possible). Track Array variables for function call validation. + +**Requirements:** R5a, R7 + +**Dependencies:** Unit 2 (AST node), Unit 4 (parser produces MemberAccess) + +**Files:** +- Modify: `spec/semantic/checker.go` (`checkNode` switch) +- Test: `spec/semantic/checker_test.go` + +**Approach:** +- Add `*ast.MemberAccess` case to `checkNode()`. Mark `node.Object` (if Identifier) as a referenced variable. The semantic checker cannot fully validate column names at check time (tables are registered during evaluation, not parsing), so the primary check is variable-exists for the table name. +- Extend function call validation for `min`, `max`, `count`: accept 1+ arguments. +- Add `FunctionSpec` entries in `spec/types/param_types.go` for min, max, count. + +**Patterns to follow:** +- `DirectiveRef` case in `checkNode()` for dot-access validation +- Existing `FunctionSpecs` entries for sum, avg + +**Test scenarios:** +- Happy path: `sales.q1` does not produce an "undefined variable" warning when `sales` is known +- Happy path: `min(x)`, `max(x, y)`, `count(arr)` pass validation +- Error path: `unknown_table.col` produces undefined variable diagnostic for `unknown_table` +- Integration: Existing semantic checks still pass (regression) + +**Verification:** +- `go test ./spec/semantic/...` passes + +--- + +- [ ] **Unit 7: Interpreter — MemberAccess, Element-wise Ops, Array Assignment** + +**Goal:** Evaluate MemberAccess nodes (table.column → Array), element-wise binary operations on Arrays, and Array-scalar broadcasting. + +**Requirements:** R7, R8, R9, R12 + +**Dependencies:** Unit 1 (types), Unit 2 (AST), Unit 5 (tables in env) + +**Files:** +- Modify: `impl/interpreter/interpreter.go` (`evalNode` switch) +- Modify: `impl/interpreter/operators.go` (`evalBinaryOperation`) +- Create: `impl/interpreter/array_ops.go` +- Test: `impl/interpreter/array_ops_test.go` +- Create: `testdata/eval/success/features/named_tables.cm` +- Create: `testdata/eval/errors/features/named_tables.cm` + +**Approach:** +- `evalNode`: add `*ast.MemberAccess` case → `evalMemberAccess()`. Look up Object in env, assert `*types.Table`, call `table.Column(field)`. Return the Array. Produce diagnostic if table or column not found. +- `evalBinaryOperation`: add Array normalization block at the top (before existing normalization). If both operands are Arrays: validate same length, zip-apply the operation element-wise (recursing into `evalBinaryOperation` for each pair), return new Array. If one operand is Array and other is scalar: broadcast scalar, apply element-wise. +- New file `array_ops.go` for `evalArrayBinaryOp` and `evalArrayScalarOp` helpers. + +**Patterns to follow:** +- `evalBinaryOperation` normalization blocks at top (from rate-widening learning) +- Existing `evalDirectiveRef` for the pattern of resolving a dot-access expression + +**Test scenarios:** +- Happy path: `rates.rate` evaluates to Array of Rate values +- Happy path: `rates.rate * rates.hc` → element-wise Array result +- Happy path: `rates.rate * 1.1` → broadcast scalar to each element +- Happy path: `line_costs = rates.rate * rates.hc` assigns Array to variable +- Happy path: Chained: `sum(rates.rate * rates.hc)` evaluates correctly end-to-end +- Edge case: Element-wise on single-element arrays → single-element result +- Edge case: Array * Array where element types differ but are coercible (Number * Currency) +- Error path: `rates.nonexistent` → diagnostic "column 'nonexistent' not found in table 'rates'" +- Error path: `nonexistent.col` → diagnostic "undefined variable 'nonexistent'" +- Error path: `scalar_var.field` → diagnostic "dot access on non-table value" +- Error path: Arrays of different lengths → diagnostic with both lengths + +**Verification:** +- `go test ./impl/interpreter/...` passes +- Golden test files in `testdata/eval/` pass + +--- + +- [ ] **Unit 8: Aggregate Functions — sum, avg, min, max, count** + +**Goal:** Implement array-accepting aggregate functions. Extend existing `sum()` and `avg()` with array dispatch. Add new `min()`, `max()`, `count()` functions. + +**Requirements:** R10, R11, R12 + +**Dependencies:** Unit 1 (Array type), Unit 4 (parser support for min/max/count tokens), Unit 7 (element-wise ops for R12) + +**Files:** +- Modify: `impl/interpreter/functions.go` (sum, avg eval functions + new min/max/count) +- Modify: `spec/types/param_types.go` (FunctionSpecs for min, max, count) +- Modify: `spec/features/registry.go` (feature entries for min, max, count + update sum) +- Test: `impl/interpreter/functions_test.go` + +**Approach:** +- `evalSumFunc`: check first arg type. If `*types.Array`, iterate elements and accumulate (reuse existing addition logic). If scalar, use existing variadic path (min 2 args). +- `evalAvgFunc`: same dispatch pattern — Array path divides sum by count. +- New `evalMinFunc`, `evalMaxFunc`: accept single Array arg, iterate with comparison. Use existing `evalComparison` for element comparison. +- New `evalCountFunc`: accept single Array arg, return `types.NewNumber(len(elements))`. +- All aggregate functions: if Array is empty, return diagnostic. +- Register new functions in `BuiltinFunctions` and `functionEvalMap`. +- Add Feature entries in registry with Category `CategoryMath`. + +**Patterns to follow:** +- Existing `evalAvgFunc` structure for sum/avg extension +- `BuiltinFunctions` registration pattern in `functions.go` +- Feature registration in `spec/features/registry.go` (`getFunctions()`) + +**Test scenarios:** +- Happy path: `sum(array_of_numbers)` → correct sum +- Happy path: `sum(array_of_currencies)` → correct currency sum with auto-scaling +- Happy path: `avg(array_of_numbers)` → correct average +- Happy path: `min(array_of_numbers)` → smallest element +- Happy path: `max(array_of_currencies)` → largest currency +- Happy path: `count(array)` → Number equal to element count +- Happy path: `sum(a, b, c)` still works (variadic scalar, regression) +- Happy path: `sum(rates.rate * rates.hc)` → aggregate over element-wise result +- Edge case: Single-element array in sum/avg/min/max → returns that element +- Error path: `sum(array, scalar)` → mixed args diagnostic +- Error path: `sum(single_scalar)` → still requires 2+ args (existing behavior) +- Error path: `min()` with no args → argument error +- Error path: Aggregate on empty array → diagnostic + +**Verification:** +- `go test ./impl/interpreter/...` passes +- Feature registry includes min, max, count entries + +--- + +### Phase 5: Display and Interpolation + +- [ ] **Unit 9: Display Formatting and Array Interpolation** + +**Goal:** Format Array values for display output. Extend interpolation to map Array elements to markdown table rows. + +**Requirements:** R13, R14, R15 + +**Dependencies:** Unit 1 (Array type), Unit 5 (table extraction), Unit 7 (MemberAccess) + +**Files:** +- Modify: `format/display/formatter.go` (`Format` method — Array case) +- Modify: `format/json_formatter.go` (`populateResult` type switch — Array/Table cases) +- Modify: `impl/document/interpolation.go` (array-in-table-cell logic) +- Test: `format/display/formatter_test.go` +- Test: `format/json_formatter_test.go` +- Test: `impl/document/interpolation_test.go` + +**Approach:** +- **Display**: Add `*types.Array` case to `Formatter.Format()`. Iterate elements, format each with existing per-type formatter, join with `, `, wrap in `[...]`. +- **Scalar interpolation** (`{{total}}`): No change needed — env lookup returns scalar, existing path formats it. +- **Dotted interpolation** (`{{rates.rate}}`): Extend `interpolateLine` resolution: if `ref` contains `.`, split into table+column, look up table in env, get column Array. If scalar context (not in a table row), format as `[...]`. If in a markdown table row, use row-index mapping. +- **Array-in-table interpolation** (`{{line_costs}}` in table rows): The interpolation pass must detect when a `{{var}}` resolving to an Array is inside a pipe-delimited markdown table row. When detected, determine the data row index (skip header and separator rows), and substitute the corresponding Array element. Length mismatch → leave tag unresolved and add diagnostic. +- The interpolation regex already matches dotted names — no regex change needed. + +**Patterns to follow:** +- Existing `interpolateLine` for per-line resolution pattern +- STX/ETX sentinel wrapping for HTML output + +**Test scenarios:** +- Happy path: Array formats as `[$250/hr, $150/hr]` in non-table context +- Happy path: `{{total}}` (scalar) in prose → formatted value (regression) +- Happy path: `{{line_costs}}` in table row 0 → first element; row 1 → second element +- Happy path: `{{rates.rate}}` dotted access in interpolation resolves correctly +- Edge case: `{{array_var}}` outside a table → formats as `[val, val]` +- Edge case: Table with `{{var}}` where var is scalar → same value in every row (existing behavior) +- Error path: Array length doesn't match table row count → tag left unresolved, diagnostic +- Happy path: JSON output (`--format json`) serializes Array as a JSON array of typed results +- Integration: Existing interpolation tests pass unchanged + +**Verification:** +- `go test ./format/display/...` passes +- `go test ./impl/document/...` passes +- End-to-end: a `.cm` file with table + calc block + interpolation produces correct output + +--- + +### Phase 6: Integration + +- [ ] **Unit 10: Classifier, Feature Registry, and Golden Tests** + +**Goal:** Ensure the classifier recognizes table-referencing expressions, update feature registry, add comprehensive golden tests, and update documentation. + +**Requirements:** All — integration verification + +**Dependencies:** All previous units + +**Files:** +- Modify: `spec/classifier/classifier.go` (recognize MemberAccess patterns) +- Modify: `spec/document/detector.go` (if needed for table-referencing lines) +- Modify: `spec/features/registry.go` (Array type feature, named tables feature) +- Create: `testdata/eval/success/features/named_tables.cm` (comprehensive golden test) +- Create: `testdata/eval/errors/features/named_tables_errors.cm` (error golden test) +- Create: `testdata/spec/valid/features/named_tables.cm` (parse-only golden test) + +**Execution note:** Run `task test` after each sub-change. The classifier is the most commonly missed layer (institutional learning). + +**Approach:** +- **Classifier**: Lines containing `identifier.identifier` patterns (table column references) should be classified as calculation lines, not prose. Add pattern recognition for dot-access expressions. +- **Feature registry**: Add entries for "Named Tables" (keyword category) and "Array" (type category). Update sum() feature entry to mention array support. +- **Golden tests**: Comprehensive `.cm` files covering the consulting SOW use case end-to-end, including directive, table, calc block, interpolation, and error cases. + +**Patterns to follow:** +- Existing classifier patterns in `spec/classifier/classifier.go` +- Feature entries in `spec/features/registry.go` +- Golden test format: `var = expression` followed by `# Expected: value` + +**Test scenarios:** +- Happy path: Full consulting SOW example evaluates correctly end-to-end +- Happy path: `rates.rate` line classified as calculation, not prose +- Happy path: Feature registry includes named tables, array, min, max, count +- Error path: All error golden tests produce expected diagnostics +- Integration: `task test` passes with all existing and new tests +- Integration: `task quality` passes + +**Verification:** +- `task test` passes (full suite) +- `task quality` passes +- Golden test files validate correct behavior + +## System-Wide Impact + +- **Interaction graph:** Table extraction inserts into the evaluator's block processing loop. Interpolation gains array awareness. The semantic checker gains MemberAccess support. The operator dispatch gains Array normalization. These are all additive — no existing paths are modified in ways that change their behavior for non-array values. +- **Error propagation:** Diagnostics from table extraction (parse errors, type mismatches) flow through the existing `document.Diagnostic` pipeline. MemberAccess errors in the interpreter use the same error recovery pattern as other eval errors. +- **State lifecycle risks:** Table data is extracted from TextBlock source on each evaluation pass. No caching across passes — consistent with how CalcBlocks are re-evaluated. In the TUI's `EvaluateAffectedBlocks`, editing a TextBlock with a table directive should trigger re-extraction and re-evaluation of dependent CalcBlocks. +- **API surface parity:** JSON output (`--format json`) needs to handle Array values. The LSP should provide completions for table names and column names (deferred — not in v1 scope but should not be blocked by the design). +- **Integration coverage:** The full pipeline (directive → extraction → env registration → MemberAccess → element-wise ops → aggregation → interpolation) must be tested end-to-end, not just per-unit. +- **Unchanged invariants:** Existing CalcMark documents with markdown tables but no directives are completely unaffected. The `Type` interface is unchanged. The `Environment` API is unchanged. All existing functions retain their current behavior for scalar arguments. + +## Risks & Dependencies + +| Risk | Mitigation | +|------|------------| +| DOT token generalization breaks number parsing (e.g., `1.5`) | Lexer only emits DOT after IDENTIFIER tokens, never after NUMBER. Extensive regression tests on decimal literals. | +| Table extraction performance for large documents with many TextBlocks | Extraction is O(lines) per TextBlock, only triggered when directive pattern is found. Early-exit regex check before full parsing. | +| Array type creates blast radius across operator dispatch | Array normalization is a single block at the top of `evalBinaryOperation`, before all existing type dispatch. Existing scalar paths are untouched. | +| TUI incremental evaluation may not re-extract tables on TextBlock edit | Ensure `EvaluateAffectedBlocks` marks table-containing TextBlocks as dirty when their source changes. Add catwalk test for this scenario. | +| Interpolation regex matches `table.column` but existing resolution treats it as flat key | Resolution logic is extended: if `ref` contains a `.`, split into table+column and resolve via Table.Column(). Since CalcMark identifiers cannot contain dots, no backward compatibility risk exists for dotted names. | + +## Documentation / Operational Notes + +- Update `site/content/` documentation with named tables feature guide +- Add a "Consulting SOW" worked example to the documentation +- Update `ARCHITECTURE.md` if the 8-layer checklist needs a "table extraction" note +- The `sum()` function documentation should mention array support + +## Sources & References + +- **Origin document:** [docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md](docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md) +- **Type addition checklist:** `docs/solutions/logic-errors/adding-new-type-fraction-cross-layer-checklist.md` +- **Expression form cookbook:** `docs/solutions/language-features/directive-as-value-cross-layer-learnings.md` +- **Unified registry pattern:** `docs/solutions/code-organization/unified-feature-registry-three-to-one.md` +- **Operator dispatch pattern:** `docs/solutions/feature-gaps/rate-type-arithmetic-widening.md` +- Related brainstorm: `docs/brainstorms/2026-03-09-sum-function-brainstorm.md` (subsumed) From 9806fe32c08d0f12a568bfc24668570eba63a31f Mon Sep 17 00:00:00 2001 From: Dylan Thomas Date: Mon, 6 Apr 2026 22:42:44 -0600 Subject: [PATCH 2/6] docs: add future considerations for computed rows and NL syntax (#118) Note the insight that computed table rows can be auto-injected during the pre-processing pass without special user markup, and that NL syntax for array functions is deferred due to expression ambiguity. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../2026-04-06-named-tables-and-arrays-requirements.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md b/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md index 06578350..5051fc5b 100644 --- a/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md +++ b/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md @@ -98,6 +98,12 @@ The canonical use cases: - [Affects R5][Needs research] Should table registration happen during document parsing (new block type) or during evaluation? This affects the spec/ vs impl/ dependency boundary. - [Affects R15][Technical] How does array interpolation in table cells interact with the existing interpolation pass? The interpolation system needs to be table-structure-aware to map array elements to rows. +## Future Considerations + +**Computed rows and columns**: v1 requires users to define calculations in calc blocks and use `{{interpolation}}` to display results. A future version could automatically bolt computed values onto tables — if a calc block clearly references a table (e.g., `sum(data.b)`), CalcMark could inject a summary row into the rendered output without the user adding any extra markup. The pre-processing pipeline already transforms TextBlocks before markdown-to-HTML rendering (interpolation does this today), so the architecture supports it. The key principle: avoid inventing CalcMark-specific magic syntax. If the computation is obvious from context, just do it. + +**NL syntax for array functions**: `sum of rates.rate` reads naturally, but `sum of rates.rate * rates.hc` is ambiguous. NL syntax for array-accepting functions is deferred until the interaction with element-wise expressions is better understood. + ## Next Steps -> `/ce:plan` for structured implementation planning From 4293ade37ee61561e1190487d5af1dfa1dadea84 Mon Sep 17 00:00:00 2001 From: Dylan Thomas Date: Mon, 6 Apr 2026 22:55:25 -0600 Subject: [PATCH 3/6] docs: golden test files and cross-table lookup future consideration (#118) Add executable spec golden files for named tables feature: - 4 success files (SOW, types, multi-table, edge cases) - 1 error file (11 diagnostic cases) - 2 parse-level files (valid/invalid syntax) Add cross-table lookup/countif to future considerations in requirements doc to ensure v1 design doesn't block the extension path. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...06-named-tables-and-arrays-requirements.md | 2 + .../errors/features/named_tables_errors.cm | 124 ++++++++++++++++++ .../eval/success/features/named_tables.cm | 19 +++ .../features/named_tables_edge_cases.cm | 83 ++++++++++++ .../success/features/named_tables_multi.cm | 39 ++++++ .../success/features/named_tables_types.cm | 63 +++++++++ .../spec/invalid/features/named_tables.cm | 6 + testdata/spec/valid/features/named_tables.cm | 26 ++++ 8 files changed, 362 insertions(+) create mode 100644 testdata/eval/errors/features/named_tables_errors.cm create mode 100644 testdata/eval/success/features/named_tables.cm create mode 100644 testdata/eval/success/features/named_tables_edge_cases.cm create mode 100644 testdata/eval/success/features/named_tables_multi.cm create mode 100644 testdata/eval/success/features/named_tables_types.cm create mode 100644 testdata/spec/invalid/features/named_tables.cm create mode 100644 testdata/spec/valid/features/named_tables.cm diff --git a/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md b/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md index 5051fc5b..0b59eb10 100644 --- a/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md +++ b/docs/brainstorms/2026-04-06-named-tables-and-arrays-requirements.md @@ -102,6 +102,8 @@ The canonical use cases: **Computed rows and columns**: v1 requires users to define calculations in calc blocks and use `{{interpolation}}` to display results. A future version could automatically bolt computed values onto tables — if a calc block clearly references a table (e.g., `sum(data.b)`), CalcMark could inject a summary row into the rendered output without the user adding any extra markup. The pre-processing pipeline already transforms TextBlocks before markdown-to-HTML rendering (interpolation does this today), so the architecture supports it. The key principle: avoid inventing CalcMark-specific magic syntax. If the computation is obvious from context, just do it. +**Cross-table lookups**: The common pattern of a rates table + staff roster requires looking up values across tables. Two functions would cover this without recreating SQL: `lookup(team.level, rates.level, rates.hourly)` (match a column, return corresponding values — like VLOOKUP) and `countif(team.level, "Senior")` (count matching elements). v1's workaround is putting derived values directly in the table. The v1 Array type and element-wise design should not block adding these functions later — they take arrays in and return arrays/scalars out, same as `sum()`. + **NL syntax for array functions**: `sum of rates.rate` reads naturally, but `sum of rates.rate * rates.hc` is ambiguous. NL syntax for array-accepting functions is deferred until the interaction with element-wise expressions is better understood. ## Next Steps diff --git a/testdata/eval/errors/features/named_tables_errors.cm b/testdata/eval/errors/features/named_tables_errors.cm new file mode 100644 index 00000000..0f44a68b --- /dev/null +++ b/testdata/eval/errors/features/named_tables_errors.cm @@ -0,0 +1,124 @@ +# Named Tables: Error Cases +# Each section demonstrates an error that should produce a diagnostic. + +## Undefined table reference + +x = missing_table.col + +--- + +## Undefined column reference + + +| A | B | +|----|-----| +| 10 | 20 | + +--- + +x = data.nonexistent + +--- + +## Column count mismatch (directive says 2, table has 3) + + +| A | B | C | +|----|----|----| +| 1 | 2 | 3 | + +--- + +## Mixed types in a column + + +| Value | +|-------| +| $100 | +| 50% | + +--- + +## Element-wise length mismatch + + +| A | +|---| +| 1 | +| 2 | + +--- + + +| B | +|---| +| 1 | +| 2 | +| 3 | + +--- + +bad = short.a + long.b + +--- + +## Duplicate table name + + +| X | +|---| +| 1 | + +--- + + +| Y | +|---| +| 2 | + +--- + +## Empty cell in table + + +| A | B | +|----|-----| +| 10 | | +| 20 | 30 | + +--- + +## Dot access on non-table variable + +scalar = 42 + +--- + +x = scalar.field + +--- + +## Mixed array and scalar in sum + + +| V | +|---| +| 1 | +| 2 | + +--- + +bad_sum = sum(nums.v, 10) + +--- + +## Nested dot access + + +| A | +|---| +| 1 | + +--- + +bad = t.a.something diff --git a/testdata/eval/success/features/named_tables.cm b/testdata/eval/success/features/named_tables.cm new file mode 100644 index 00000000..5a06cb78 --- /dev/null +++ b/testdata/eval/success/features/named_tables.cm @@ -0,0 +1,19 @@ +# Named Tables: Consulting SOW +# The canonical use case — a table of roles, rates, and headcount +# that computes line costs and grand total without data duplication. + + +| Role | Hourly Rate | Headcount | +|----------|-------------|-----------| +| Senior | $250/hr | 3 | +| Mid | $175/hr | 5 | +| Junior | $120/hr | 4 | + +--- + +line_costs = team.rate * team.hc +total_cost = sum(line_costs) +avg_rate = avg(team.rate) +team_size = count(team.hc) +highest_rate = max(team.rate) +lowest_rate = min(team.rate) diff --git a/testdata/eval/success/features/named_tables_edge_cases.cm b/testdata/eval/success/features/named_tables_edge_cases.cm new file mode 100644 index 00000000..c20f6ccd --- /dev/null +++ b/testdata/eval/success/features/named_tables_edge_cases.cm @@ -0,0 +1,83 @@ +# Named Tables: Edge Cases +# Valid but unusual inputs that should work correctly. + +## Single row table + + +| Value | +|-------| +| 42 | + +--- + +solo_sum = sum(solo.val) + +--- + +## Single column table + + +| ID | +|----| +| 1 | +| 2 | +| 3 | + +--- + +id_count = count(ids.id) + +--- + +## Alignment colons in separator (standard GFM) + + +| Name | Amount | +|:-------|--------:| +| Alpha | $100 | +| Beta | $200 | + +--- + +aligned_total = sum(aligned.amount) + +--- + +## Whitespace in cells (extra padding) + + +| A | B | +|-------|-------| +| 10 | 20 | +| 30 | 40 | + +--- + +padded_sum = sum(padded.a) + sum(padded.b) + +--- + +## Directive name normalization + + +| Hourly | Daily | +|--------|-------| +| $50 | $400 | +| $75 | $600 | + +--- + +my_rates_total = sum(my_rates.hourly) + +--- + +## Element-wise with single element arrays + + +| X | Y | +|---|---| +| 5 | 3 | + +--- + +product = one.x * one.y diff --git a/testdata/eval/success/features/named_tables_multi.cm b/testdata/eval/success/features/named_tables_multi.cm new file mode 100644 index 00000000..59004182 --- /dev/null +++ b/testdata/eval/success/features/named_tables_multi.cm @@ -0,0 +1,39 @@ +# Named Tables: Multiple Tables and Cross-References +# Two tables in one document with calculations that reference both. + + +| Item | Unit Cost | Qty | +|----------|-----------|-----| +| Server | $4500 | 12 | +| Switch | $800 | 6 | +| Storage | $12000 | 4 | + +--- + +hw_line = hardware.unit_cost * hardware.qty +hw_total = sum(hw_line) + +--- + + +| Role | Monthly Rate | Duration | +|------------|--------------|----------| +| Architect | $18000 | 6 | +| Engineer | $14000 | 12 | + +--- + +labor_line = labor.monthly * labor.months +labor_total = sum(labor_line) +project_total = hw_total + labor_total + +--- + +## Table without directive is inert + +Regular markdown tables are untouched — no directive means no extraction. + +| This | Table | +|------|-------| +| is | inert | +| just | prose | diff --git a/testdata/eval/success/features/named_tables_types.cm b/testdata/eval/success/features/named_tables_types.cm new file mode 100644 index 00000000..21d81322 --- /dev/null +++ b/testdata/eval/success/features/named_tables_types.cm @@ -0,0 +1,63 @@ +# Named Tables: Column Type Variety +# Verifies that different CalcMark types parse correctly from table cells. + +## Numbers + + +| Student | Grade | +|---------|-------| +| Alice | 92 | +| Bob | 87 | +| Carol | 95 | + +--- + +class_avg = avg(scores.grade) +top_score = max(scores.grade) + +--- + +## Currencies + + +| Department | Q1 Spend | Q2 Spend | +|------------|----------|----------| +| Eng | $450K | $520K | +| Sales | $200K | $180K | +| Ops | $150K | $160K | + +--- + +q1_total = sum(budget.q1) +q2_total = sum(budget.q2) +total_spend = q1_total + q2_total + +--- + +## Quantities + + +| Item | Weight | Count | +|---------|--------|-------| +| Widget | 2.5 kg | 100 | +| Gizmo | 1.2 kg | 250 | + +--- + +total_weight = sum(inventory.weight * inventory.count) + +--- + +## Scalar broadcasting + + +| Product | Base Price | +|---------|------------| +| Basic | $50 | +| Pro | $120 | +| Ent | $350 | + +--- + +with_tax = prices.base_price * 1.08 +discounted = prices.base_price * 0.85 diff --git a/testdata/spec/invalid/features/named_tables.cm b/testdata/spec/invalid/features/named_tables.cm new file mode 100644 index 00000000..6b812427 --- /dev/null +++ b/testdata/spec/invalid/features/named_tables.cm @@ -0,0 +1,6 @@ +# Named Tables: Parse Errors +# These expressions should fail at parse time. + +## Nested dot access (only one level allowed) + +x = a.b.c diff --git a/testdata/spec/valid/features/named_tables.cm b/testdata/spec/valid/features/named_tables.cm new file mode 100644 index 00000000..c9b543a5 --- /dev/null +++ b/testdata/spec/valid/features/named_tables.cm @@ -0,0 +1,26 @@ +# Named Tables: Parse Validation +# These expressions should parse successfully (syntax is valid). + +## Dot access expressions + +x = sales.revenue +y = data.col_name +z = my_table.field + +## Dot access in expressions + +result = a.x + b.y +product = table.col1 * table.col2 +scaled = data.values * 2 + +## Dot access in function calls + +total = sum(data.values) +average = avg(scores.grade) +smallest = min(data.col) +largest = max(data.col) +n = count(data.col) + +## Chained operations + +total = sum(rates.hourly * rates.hc) From c73e5b71607e556ff464d52edaca61ce15e701ed Mon Sep 17 00:00:00 2001 From: Dylan Thomas Date: Tue, 7 Apr 2026 15:47:59 -0600 Subject: [PATCH 4/6] docs: CalcMark Notebooks product requirements (#118) Consumer-oriented web notebook product: social login, private by default, share readonly rendered views, clone into own workspace. Single Go binary deployment with SQLite + Litestream for backups. Jupyter-like block editor with context-aware documentation sidebar. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...6-04-07-calcmark-notebooks-requirements.md | 120 ++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md diff --git a/docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md b/docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md new file mode 100644 index 00000000..0ee3873c --- /dev/null +++ b/docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md @@ -0,0 +1,120 @@ +--- +date: 2026-04-07 +topic: calcmark-notebooks +--- + +# CalcMark Notebooks + +## Problem Frame + +CalcMark is a powerful calculation language with rich diagnostics, autosuggest, and documentation — but it's locked behind a CLI. The language can be broad and complex because the editor experience (LSP, inline help, contextual docs) makes it approachable. What's missing is a way to create, edit, and share CalcMark documents without installing anything. + +The unique wedge: **plain-text source of truth + single-binary self-hosting + calculations that read like documents**. Spreadsheets are opaque. Notebooks (Jupyter) store JSON blobs. Docs tools (Notion) can't compute. CalcMark documents are readable text files where prose and calculations coexist — and the web product makes that accessible to anyone with a browser. + +## Requirements + +**Editor Experience** + +- R1. Jupyter-like block UI: blocks show beautifully rendered output by default. Click a block to reveal and edit the CalcMark source. Frontmatter editable via click-to-reveal. +- R2. The editor must produce and consume valid `.cm` files. The file format is the product — no proprietary models. Everything is derived from the source text. +- R3. Context-aware documentation sidebar: calcmark.org docs appear alongside the editor, contextual to what the user is typing. The same diagnostics, autosuggest, and inline help from the LSP are available in the web editor. +- R4. Beautiful markdown rendering — tables, headings, prose, interpolated values all render with high visual quality. +- R5. Single-user editing for MVP. + +**Storage and State** + +- R6. Each document is stored as its `.cm` source text plus metadata (owner, timestamps). The source text is the single source of truth. +- R7. SQLite as the storage backend. One file, simple backups (Litestream to S3/GCS, or plain rsync/cp of the database file). No storage abstraction layer in MVP — clean Go package boundary is sufficient to enable future backend changes if needed. +- R8. "Save" persists the current source text. Backup must be as simple as rsync or streaming the SQLite WAL. + +**Authentication and Sharing** + +- R9. Social login only (Google, GitHub OAuth). No custom user/password system. +- R10. Documents are private by default (owner only). +- R11. Share rendered output: a readonly link renders the document beautifully with all calculations resolved. No editing, no account required to view. +- R12. "Open your own copy": any viewer of a shared document can clone it into their own authenticated workspace as a new private document. + +**URL Patterns** + +- R13. Clean, predictable URLs: + - `/{id}` — rendered view (readonly, shareable) + - `/{id}/edit` — editor (authenticated owner only) + - `/new` — blank document (authenticated) + - `/{id}/clone` — fork into own copy (authenticated) + - `/{id}/raw` — plain `.cm` source text + +**Deployment** + +- R14. Single Go binary that serves the web UI, evaluates CalcMark, and stores documents in SQLite. Zero external infrastructure required. Deploy to a VPS, Docker, or fly.io. +- R15. API-first: the web UI communicates with the backend via HTTP. Clean package separation in Go enables future extraction if needed, but no formal API versioning in MVP. + +**Evaluation** + +- R16. Documents are evaluated server-side using the existing CalcMark Go library. +- R17. Evaluation must be fast enough for interactive editing. Target: sub-100ms for typical documents (same as TUI performance). + +## Success Criteria + +- A user can sign in with Google/GitHub, create a CalcMark document in their browser, see live computed results, and share a readonly link — without installing anything. +- The shared link renders the document beautifully. A viewer can "open their own copy" to fork it. +- A self-hosted instance runs as a single binary, stores data in SQLite, and requires zero external services beyond an OAuth provider. +- Every document is a valid `.cm` file that works identically in the CLI. +- Backups are as simple as rsync, cp, or Litestream streaming to a bucket. + +## Scope Boundaries + +- **No multiplayer editing** — single-user editing only. +- **No team/org visibility** — documents are private (owner) or public (shared link). Team features come later if there's demand. +- **No PDF export** — HTML rendering is the primary output. +- **No mobile-optimized editor** — responsive readonly view is fine, desktop editor. +- **No storage abstraction** — SQLite directly. Refactor when a second backend is concretely needed. +- **No custom auth** — social login only (OAuth). +- **No git integration in MVP** — future consideration. + +## Key Decisions + +- **Consumer-oriented, not enterprise**: Social login, private by default, share via link. No team management, no RBAC, no SSO. The simplest model that works for individual knowledge workers and consultants. +- **`.cm` file as source of truth**: The web app is a lens on plain text. CLI compatibility, git-friendliness, and portability preserved. Tradeoff: the editor must parse and produce valid `.cm`. +- **Server-side evaluation**: CalcMark runs on the server (Go), not in the browser. Simplifies the frontend. Tradeoff: keystrokes need a round-trip (mitigated by debouncing). WASM is a future option if latency becomes an issue. +- **Context-aware documentation with a flywheel**: The editor surfaces CalcMark docs, diagnostics, and autosuggest inline — the same quality of help the LSP provides in IDEs. This is what makes a complex language approachable for non-developers. The docs improve continuously: gaps found in the editor → new pages added to calcmark.org with Hugo frontmatter tags → the sidebar search gets smarter. The product and documentation improve together. The editor can also suggest example documents to explore. +- **SQLite + Litestream**: One database file. Backup = stream WAL changes to S3 or just rsync the file. No Postgres, no managed DB, no ops complexity. Turso (SQLite-compatible with vector extensions) is a potential future backend for semantic doc search. +- **Jupyter-like block UI**: Click to edit source, otherwise see rendered output. Side-by-side alignment is nearly impossible in web. The block pattern is more natural for the target audience. + +## Dependencies / Assumptions + +- The CalcMark Go library is the evaluation backend. No engine changes needed. +- The web frontend tech stack is deferred to planning. +- The named tables feature should land before the web product (tabular data is a primary notebook use case). +- OAuth providers (Google, GitHub) are the only auth dependency. + +## Outstanding Questions + +### Deferred to Planning + +- [Affects R1][Needs research] What web editor framework best supports the Jupyter-like block editing pattern with CalcMark syntax highlighting? (CodeMirror 6, Monaco, ProseMirror, custom) +- [Affects R3][Needs research] How to surface contextual CalcMark docs in the sidebar — embed the Hugo site content, use a search index, or build a semantic search with vector embeddings (Turso)? +- [Affects R14][Technical] How to embed the web frontend assets in the single Go binary (Go embed, or similar). The product must be a single binary deployment — no separate frontend build artifact. +- [Affects R16][Technical] Evaluation API: WebSocket for streaming updates vs HTTP request/response with debouncing. +- [Affects R9][Technical] OAuth implementation — use an existing Go library (e.g., goth, oauth2) or minimal custom implementation? + +## Future Considerations + +**Multiplayer editing**: Real-time collaboration via CRDTs (Yjs) over block source text. The Jupyter-like block model helps (blocks can be independently locked). MVP's single-user model should not architecturally block this. + +**Semantic doc search**: Turso's vector extension could power "search CalcMark docs by meaning" in the sidebar. User types a question, gets the most relevant documentation section. Requires embedding calcmark.org content into a vector index. + +**Git integration**: "Save" = "git commit" for users who want version control. Local git repo on the server or GitHub API integration. + +**PDF export**: Server-side (headless Chrome) or separate service. Keep the core binary lean. + +**Hugo/static site integration**: Hugo preprocessor for `.cm` files or embedded `cm` code blocks. Separate from notebooks but shares the library. + +**GitHub code block rendering**: If GitHub supports custom code block renderers, CalcMark's engine + formatters would be the integration surface. Already embeddable via `calcmark.Eval()`. + +**Template library**: Pre-built `.cm` templates (consulting SOW, project budget, capacity planning). Users clone templates to get started. + +**Computed table rows**: Auto-inject summary rows into rendered output without special markup (from the named-tables brainstorm). + +## Next Steps + +-> `/ce:plan` for structured implementation planning From 03dcd48ae5d0d7c6b6483f053f20017c635e976d Mon Sep 17 00:00:00 2001 From: Dylan Thomas Date: Tue, 7 Apr 2026 16:10:58 -0600 Subject: [PATCH 5/6] docs: CalcMark Notebooks implementation plan (#118) 10-unit plan across 4 phases for the web notebook product: - Separate repo consuming go-calcmark as a library - Go HTTP server + embedded SvelteKit frontend + SQLite - OAuth (Google/GitHub), Jupyter-like block editor - Litestream for continuous backup - Export APIs (single doc + bulk zip) Co-Authored-By: Claude Opus 4.6 (1M context) --- ...-04-07-001-feat-calcmark-notebooks-plan.md | 565 ++++++++++++++++++ 1 file changed, 565 insertions(+) create mode 100644 docs/plans/2026-04-07-001-feat-calcmark-notebooks-plan.md diff --git a/docs/plans/2026-04-07-001-feat-calcmark-notebooks-plan.md b/docs/plans/2026-04-07-001-feat-calcmark-notebooks-plan.md new file mode 100644 index 00000000..9ad24d9a --- /dev/null +++ b/docs/plans/2026-04-07-001-feat-calcmark-notebooks-plan.md @@ -0,0 +1,565 @@ +--- +title: "feat: CalcMark Notebooks — Web App MVP" +type: feat +status: active +date: 2026-04-07 +origin: docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md +--- + +# feat: CalcMark Notebooks — Web App MVP + +## Overview + +A web application for creating, editing, saving, and sharing CalcMark documents. Single Go binary with embedded Svelte frontend, SQLite storage (with Litestream for continuous backup), and social login (Google/GitHub OAuth). Jupyter-like block editor where blocks show rendered output by default and reveal source on click. + +**This is a separate project/repository** (e.g., `CalcMark/calcmark-web`) that consumes `github.com/CalcMark/go-calcmark` as a Go library dependency. The `go-calcmark` repo stays clean: language spec, interpreter, formatters, CLI/TUI. The web product has its own binary, release cycle, and deployment model. + +## Problem Frame + +CalcMark is a powerful calculation language locked behind a CLI. The web product makes it accessible to anyone with a browser — create a document, see live results, share a readonly link. Files are always valid `.cm`, backups are a simple file copy. (see origin: `docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md`) + +## Requirements Trace + +- R1. Jupyter-like block UI (rendered by default, click to edit source) +- R2. Files are valid `.cm` — format is the product +- R3. Context-aware documentation sidebar +- R4. Beautiful markdown rendering +- R5. Single-user editing +- R6. Document = `.cm` source + metadata, source is truth +- R7. SQLite storage, backup = rsync/cp +- R8. Save persists source text +- R9. Social login (Google, GitHub OAuth) +- R10. Private by default +- R11. Share rendered output via readonly link +- R12. Clone: "open your own copy" +- R13. Clean URL patterns +- R14. Single Go binary deployment +- R15. API-first (HTTP between frontend and backend) +- R16. Server-side evaluation +- R17. Sub-100ms evaluation target +- Export: single doc raw download, bulk zip export + +## Scope Boundaries + +- No multiplayer editing +- No team/org visibility — private or public (shared link) only +- No PDF export +- No mobile-optimized editor +- No storage abstraction — SQLite directly (modernc.org/sqlite, pure Go) +- No Turso/vector search in MVP — add when semantic doc search is needed +- No custom CalcMark Lezer grammar in MVP — use CodeMirror's plain text mode with CalcMark evaluation for feedback. Custom grammar is a future enhancement. +- No git integration + +## Context & Research + +### Relevant Code and Patterns + +- **Existing HTTP server**: `cmd/calcmark/cmd/watch.go` — stdlib `net/http`, WebSocket for live reload, security headers (CSP, X-Frame-Options, nosniff). Uses `calcmark.Convert()` directly. +- **Public API**: `calcmark.Convert(input, opts)` → rendered HTML string. `calcmark.Eval(input)` → Result with values + diagnostics. `calcmark.NewSession()` → stateful evaluation across calls. +- **Go embed pattern**: `format/html_formatter.go` embeds CSS/templates via `//go:embed`. Same pattern for frontend assets. +- **HTML templates**: `format/templates/default.gohtml` (standalone page), `format/templates/preview.gohtml` (fragment). `format.StyleCSS()` provides shared CalcMark CSS. +- **GoReleaser**: `.goreleaser.yaml` — single binary, CGO_ENABLED=0, cross-platform. Must remain CGO-free (hence modernc.org/sqlite over go-libsql). + +### External References + +- **SvelteKit + Go embed**: Build SvelteKit with `adapter-static`, embed `build/` directory via `//go:embed all:build`, serve with `http.FileServer`. Use `all:` prefix for underscore files. +- **OAuth**: `golang.org/x/oauth2` (minimal) or `github.com/go-pkgz/auth/v2` (batteries-included with JWT sessions, XSRF, multi-provider). Recommend go-pkgz/auth for faster MVP. +- **CodeMirror 6 + Svelte**: `svelte-codemirror-editor` package (v2.1.0, Svelte 5 compatible). Exclude from Vite optimizeDeps. +- **SQLite pure Go**: `modernc.org/sqlite` — no CGO, registers as `"sqlite"` with `database/sql`. + +## Key Technical Decisions + +- **modernc.org/sqlite over Turso for MVP**: Pure Go, no CGO, preserves CGO_ENABLED=0 cross-compilation. Turso (with vector extensions for doc search) is a future upgrade path — the database/sql interface makes swapping straightforward. +- **go-pkgz/auth for OAuth**: Handles JWT sessions, cookies, XSRF, multiple providers with `AddProvider()`. Avoids writing session management from scratch. MIT licensed. +- **SvelteKit with adapter-static**: Compiles to static HTML/JS/CSS, embedded in Go binary via `//go:embed`. Single binary contains everything. During dev, run SvelteKit dev server separately. +- **stdlib net/http (no router library)**: The existing watch server uses stdlib. The API surface is small enough (10-15 endpoints) that a router library adds dependency without value. Use `http.NewServeMux` with Go 1.22+ pattern matching. +- **CalcMark evaluation via Convert()**: The existing `calcmark.Convert(input, Options{Format: "html"})` produces full rendered HTML. The web backend calls this on every save/evaluation request. No new evaluation infrastructure needed. +- **Document ID generation**: Short random alphanumeric IDs (8 chars, like `a3kf92x1`). URL-safe, collision-resistant at small scale, human-shareable. + +## Open Questions + +### Resolved During Planning + +- **SQLite or Turso?** SQLite (modernc.org/sqlite) for MVP. Pure Go, no CGO, simple backups. Turso later for vector search. +- **OAuth library?** go-pkgz/auth/v2 — handles sessions, JWT, XSRF, multi-provider out of the box. +- **Frontend framework?** SvelteKit with adapter-static, embedded in Go binary. +- **How to embed frontend?** `//go:embed all:build` in a `web/frontend/embed.go` file. `go generate` runs `npm run build`. +- **Licensing?** All dependencies MIT or permissive. Safe for commercial use. + +### Deferred to Implementation + +- Exact SQLite schema (migrations, indexes) — discover during Unit 2 +- CodeMirror block-level editing UX details — prototype during Unit 6 +- Documentation sidebar content source and search — future feature, not MVP +- WebSocket vs HTTP polling for live evaluation updates — start with HTTP, add WebSocket if latency is a problem + +## High-Level Technical Design + +> *This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.* + +``` +Architecture: + + Browser (SvelteKit SPA) + ↕ HTTP/JSON + Go HTTP Server (net/http) + ├── /auth/* → go-pkgz/auth (OAuth, sessions) + ├── /api/docs → Document CRUD + ├── /api/docs/:id/eval → CalcMark evaluation + ├── /api/export → Bulk zip download + ├── /d/:id → Rendered readonly view (server-rendered HTML) + ├── /d/:id/raw → Raw .cm source + └── /* → Embedded SvelteKit static files + ↕ + SQLite (modernc.org/sqlite) + └── documents table (id, owner_id, source, visibility, created_at, updated_at) + └── users table (id, provider, provider_id, name, email, avatar_url, created_at) + +Request Flow (evaluation): + 1. User edits block in CodeMirror → Svelte sends source text to /api/docs/:id/eval + 2. Go handler calls calcmark.Convert(source, Options{Format: "html"}) + 3. Returns {html: "...", diagnostics: [...], values: [...]} + 4. Svelte renders HTML in the block, shows diagnostics inline + +Request Flow (sharing): + 1. Owner clicks "Share" → POST /api/docs/:id/share → sets visibility=public + 2. Returns shareable URL: /d/:id + 3. Viewer hits /d/:id → server calls calcmark.Convert(), returns full HTML page + 4. "Open your own copy" button → POST /api/docs/:id/clone (requires auth) +``` + +## Implementation Units + +### Phase 1: Backend Foundation + +- [ ] **Unit 1: Go HTTP Server Scaffold + SQLite Schema** + +**Goal:** Minimal Go HTTP server that serves a "Hello CalcMark" page, connects to SQLite, and runs migrations. + +**Requirements:** R7, R14, R15 + +**Dependencies:** None — foundation for everything. + +**Files:** +- Create: `web/server.go` (HTTP server, mux, middleware) +- Create: `web/db.go` (SQLite connection, migrations) +- Create: `web/db_test.go` +- Create: `web/server_test.go` + +**Approach:** +- `http.NewServeMux` with Go 1.22+ pattern matching (e.g., `GET /api/docs/{id}`). +- SQLite via `modernc.org/sqlite` with `database/sql`. Schema: `users` (id, provider, provider_id, name, email, avatar_url, created_at) and `documents` (id TEXT PRIMARY KEY, owner_id, source TEXT, visibility TEXT DEFAULT 'private', title TEXT, created_at, updated_at). +- Document IDs are 8-char random alphanumeric strings generated at creation time. +- Migrations: simple version-numbered SQL files embedded via `//go:embed`. Run on startup. +- Security middleware: CSP headers, X-Frame-Options, nosniff (follow `watch.go` pattern). + +**Patterns to follow:** +- `cmd/calcmark/cmd/watch.go` for HTTP server structure, security headers +- `format/html_formatter.go` for `//go:embed` pattern + +**Test scenarios:** +- Happy path: Server starts, connects to SQLite, runs migrations, responds to health check +- Happy path: Schema creates users and documents tables with correct columns +- Edge case: Server starts with empty/new database — migrations run cleanly +- Error path: Invalid database path — server fails with clear error message +- Integration: Migrations are idempotent — running twice produces no errors + +**Verification:** +- `go test ./web/...` passes +- Server starts on a port and responds to HTTP requests + +--- + +- [ ] **Unit 2: Document CRUD API** + +**Goal:** REST API for creating, reading, updating, listing, and deleting documents. + +**Requirements:** R6, R8, R13, R15 + +**Dependencies:** Unit 1 (server + database) + +**Files:** +- Create: `web/handlers_docs.go` +- Create: `web/handlers_docs_test.go` + +**Approach:** +- `POST /api/docs` — create new document (generates ID, stores source text, returns doc metadata) +- `GET /api/docs` — list user's documents (requires auth, returns metadata array) +- `GET /api/docs/{id}` — get single document (source + metadata) +- `PUT /api/docs/{id}` — update source text (owner only) +- `DELETE /api/docs/{id}` — delete document (owner only) +- `GET /api/docs/{id}/raw` — raw `.cm` source text (plain text content type) +- All endpoints return JSON except `/raw` which returns `text/plain`. +- Owner validation on write operations. Public documents readable by anyone. + +**Patterns to follow:** +- Standard Go HTTP handler patterns with `http.NewServeMux` + +**Test scenarios:** +- Happy path: Create document → returns ID and metadata +- Happy path: List documents → returns only user's documents +- Happy path: Get document by ID → returns source and metadata +- Happy path: Update document → source text changes, updated_at advances +- Happy path: Get raw → returns plain .cm text +- Edge case: Create document with empty source → succeeds (blank document) +- Error path: Get nonexistent document → 404 +- Error path: Update document owned by another user → 403 +- Error path: Delete document owned by another user → 403 +- Integration: Create → Get → Update → Get → Delete → Get returns 404 + +**Verification:** +- `go test ./web/...` passes +- Full CRUD lifecycle works via HTTP client tests + +--- + +- [ ] **Unit 3: OAuth Authentication** + +**Goal:** Social login with Google and GitHub. Session management via JWT cookies. + +**Requirements:** R9, R10 + +**Dependencies:** Unit 1 (server + database) + +**Files:** +- Create: `web/auth.go` (provider config, user upsert) +- Create: `web/auth_test.go` +- Modify: `web/server.go` (mount auth routes, add auth middleware) + +**Approach:** +- Use `github.com/go-pkgz/auth/v2` for OAuth flow, JWT sessions, XSRF protection. +- `service.AddProvider("github", clientID, clientSecret)` + `service.AddProvider("google", clientID, clientSecret)`. +- On successful auth, upsert user in SQLite (provider + provider_id as unique key). Store name, email, avatar_url. +- Auth middleware on `/api/*` routes. Readonly routes (`/d/{id}` rendered view) do not require auth. +- Configuration via environment variables: `CALCMARK_GITHUB_CLIENT_ID`, `CALCMARK_GITHUB_CLIENT_SECRET`, `CALCMARK_GOOGLE_CLIENT_ID`, `CALCMARK_GOOGLE_CLIENT_SECRET`, `CALCMARK_JWT_SECRET`. + +**Patterns to follow:** +- go-pkgz/auth documentation for handler mounting and middleware usage + +**Test scenarios:** +- Happy path: OAuth callback creates new user in database +- Happy path: Returning user — OAuth callback finds existing user, updates name/avatar +- Happy path: Authenticated request to /api/docs succeeds with user context +- Error path: Unauthenticated request to /api/docs → 401 +- Error path: Missing OAuth env vars → server fails to start with clear error +- Edge case: User logs in with GitHub, then later with Google (same email) — treated as separate accounts (provider + provider_id is the key) + +**Verification:** +- `go test ./web/...` passes +- OAuth flow works end-to-end in a browser (manual verification) + +--- + +### Phase 2: Evaluation + Rendering + +- [ ] **Unit 4: CalcMark Evaluation API** + +**Goal:** API endpoint that accepts `.cm` source text, evaluates it, and returns rendered HTML + diagnostics. + +**Requirements:** R16, R17 + +**Dependencies:** Unit 2 (document CRUD for context) + +**Files:** +- Create: `web/handlers_eval.go` +- Create: `web/handlers_eval_test.go` + +**Approach:** +- `POST /api/docs/{id}/eval` — accepts `{source: "..."}`, calls `calcmark.Convert(source, Options{Format: "html"})`, returns `{html: "...", diagnostics: [...]}`. +- Also supports `POST /api/eval` (anonymous, no document context) for quick evaluation without saving. +- Diagnostics extracted from the evaluation error (CalcMark returns partial results + errors). +- Response includes both the full rendered HTML and structured diagnostics (line, message, severity) for inline editor display. + +**Patterns to follow:** +- `calcmark.Convert()` in `convert.go` — the existing public API +- `cmd/calcmark/cmd/watch.go` — calls Convert() on file change + +**Test scenarios:** +- Happy path: Valid .cm source → returns rendered HTML with computed values +- Happy path: Source with diagnostics → returns HTML + diagnostics array +- Happy path: Evaluation completes within 100ms for a typical document (10 blocks) +- Edge case: Empty source → returns empty HTML, no diagnostics +- Error path: Completely invalid source → returns partial HTML + error diagnostics +- Integration: Update document source via PUT, then eval → results reflect updated source + +**Verification:** +- `go test ./web/...` passes +- A consulting SOW `.cm` document evaluates and renders correctly via the API + +--- + +- [ ] **Unit 5: Rendered Readonly View (Sharing)** + +**Goal:** Server-rendered HTML page for shared documents. Beautiful output at `GET /d/{id}`. + +**Requirements:** R4, R11, R13 + +**Dependencies:** Unit 2 (document CRUD), Unit 4 (evaluation) + +**Files:** +- Create: `web/handlers_share.go` +- Create: `web/handlers_share_test.go` +- Create: `web/templates/view.gohtml` (rendered document page template) + +**Approach:** +- `GET /d/{id}` — fetch document, evaluate with `calcmark.Convert()`, render into a full HTML page using a Go template. +- The page template includes CalcMark's CSS (`format.StyleCSS()`), the rendered HTML content, a header with document title, and a "Open your own copy" button (links to `/{id}/clone`). +- Public documents: no auth required. Private documents: 404 (don't leak existence). +- `POST /api/docs/{id}/share` — toggle visibility to "public", returns shareable URL. +- `POST /api/docs/{id}/clone` — requires auth. Creates a new private document with the same source text, owned by the authenticated user. + +**Patterns to follow:** +- `format/templates/default.gohtml` for HTML page structure +- `cmd/calcmark/cmd/watch.go` for injecting StyleCSS into page templates + +**Test scenarios:** +- Happy path: Public document at /d/{id} → returns beautiful rendered HTML page +- Happy path: Clone → creates new document with same source, different ID, owned by cloner +- Happy path: Share toggle → document becomes public, URL is returned +- Error path: Private document at /d/{id} → 404 +- Error path: Nonexistent document → 404 +- Error path: Clone without auth → 401 +- Edge case: Clone a document with evaluation errors → clone succeeds (source is copied, not evaluated) + +**Verification:** +- `go test ./web/...` passes +- A shared document URL renders a beautiful standalone page in a browser + +--- + +### Phase 3: Frontend (Svelte) + +- [ ] **Unit 6: SvelteKit Scaffold + Go Embed** + +**Goal:** SvelteKit project with adapter-static, embedded in the Go binary, served as the SPA. + +**Requirements:** R14 + +**Dependencies:** Unit 1 (HTTP server to serve the files) + +**Files:** +- Create: `web/frontend/` (SvelteKit project: package.json, svelte.config.js, vite.config.js, src/) +- Create: `web/frontend/embed.go` (`//go:embed all:build`) +- Modify: `web/server.go` (serve embedded SvelteKit files as fallback) + +**Approach:** +- `npx sv create web/frontend` (SvelteKit skeleton with TypeScript). +- Configure `adapter-static` with `fallback: 'index.html'` for SPA routing. +- `embed.go` with `//go:generate npm --prefix . run build` and `//go:embed all:build`. +- Go server: API routes first (`/api/*`, `/auth/*`, `/d/*`), then fallback to SvelteKit static files for everything else. +- Development mode: `CALCMARK_DEV=1` env var proxies frontend requests to SvelteKit dev server (port 5173) instead of serving embedded files. +- Vite config: exclude `svelte-codemirror-editor` and `@codemirror/*` from optimizeDeps. + +**Patterns to follow:** +- `format/html_formatter.go` for `//go:embed` pattern +- SvelteKit + Go embed pattern from external research + +**Test scenarios:** +- Happy path: `go generate ./web/frontend/...` + `go build` produces a binary that serves the SvelteKit app +- Happy path: SPA routing works — `/new`, `/{id}/edit` all resolve to index.html with client-side routing +- Edge case: API routes take priority over SPA fallback — `/api/docs` hits the API, not the SPA +- Integration: Binary starts, serves both API and frontend from a single port + +**Verification:** +- Binary serves the SvelteKit app at `/` +- Navigation between pages works without full page reloads + +--- + +- [ ] **Unit 7: Document List + Management Page** + +**Goal:** Svelte page showing the user's documents with create/delete/share actions. + +**Requirements:** R13 + +**Dependencies:** Unit 2 (CRUD API), Unit 3 (auth), Unit 6 (SvelteKit scaffold) + +**Files:** +- Create: `web/frontend/src/routes/+page.svelte` (document list — home page) +- Create: `web/frontend/src/routes/+layout.svelte` (app shell, nav, auth state) +- Create: `web/frontend/src/lib/api.ts` (API client helper) + +**Approach:** +- Home page (`/`) shows the authenticated user's documents as cards (title, last updated, visibility badge). +- "New Document" button → `POST /api/docs` → redirect to `/{id}/edit`. +- Delete, Share toggle actions on each card. +- Auth state in layout: if not logged in, show login buttons (Google, GitHub). If logged in, show user avatar + name. +- `api.ts` helper: wraps `fetch()` with auth headers, base URL handling, JSON parsing. + +**Patterns to follow:** +- Standard SvelteKit page/layout patterns + +**Test scenarios:** +- Happy path: Authenticated user sees their documents listed +- Happy path: "New Document" creates a doc and navigates to editor +- Happy path: Delete removes document from list +- Happy path: Share toggle changes visibility badge +- Edge case: User with zero documents sees empty state with "Create your first document" prompt +- Error path: API error → user-friendly error message displayed + +**Verification:** +- User can create, see, and manage documents from the browser + +--- + +- [ ] **Unit 8: Block Editor (CodeMirror 6)** + +**Goal:** Jupyter-like block editing experience. Blocks show rendered HTML by default; click to reveal CodeMirror source editor. + +**Requirements:** R1, R2, R4, R5, R16 + +**Dependencies:** Unit 4 (evaluation API), Unit 6 (SvelteKit), Unit 7 (navigation) + +**Files:** +- Create: `web/frontend/src/routes/[id]/edit/+page.svelte` (editor page) +- Create: `web/frontend/src/lib/components/Block.svelte` (single block: rendered/editing states) +- Create: `web/frontend/src/lib/components/Editor.svelte` (document editor orchestrator) +- Create: `web/frontend/src/lib/components/FrontmatterPanel.svelte` (frontmatter click-to-edit) + +**Approach:** +- Parse `.cm` source into blocks (split on `---` separators, same as CalcMark's document model). +- Each block is a `` component with two states: **rendered** (shows HTML from evaluation) and **editing** (shows CodeMirror with source text). +- Click a rendered block → switch to editing. Click outside or press Escape → save block, re-evaluate, switch to rendered. +- On block save: reassemble full `.cm` source from all blocks, send to `/api/docs/{id}/eval`, update all rendered blocks with new HTML. +- CodeMirror instance per block using `svelte-codemirror-editor`. Plain text mode for MVP (no custom Lezer grammar). +- Debounce evaluation: 300ms after last keystroke (same pattern as LSP debounce). +- Frontmatter panel: click the document header area to reveal/edit YAML frontmatter. +- Auto-save: `PUT /api/docs/{id}` on evaluation success (debounced, not on every keystroke). + +**Patterns to follow:** +- Jupyter Notebook's cell editing model +- `svelte-codemirror-editor` component API + +**Test scenarios:** +- Happy path: Load document → all blocks render as beautiful HTML +- Happy path: Click block → CodeMirror appears with source text +- Happy path: Edit block, click away → block re-evaluates, shows updated rendered output +- Happy path: Add new block (type `---` at end) → new empty block appears +- Happy path: Frontmatter editable via click-to-reveal panel +- Edge case: Document with single block (no separators) → one editable block +- Edge case: Document with only TextBlocks (no calculations) → renders as beautiful markdown +- Error path: Evaluation returns diagnostics → shown inline in the block +- Integration: Edit → auto-save → refresh page → edits persisted + +**Verification:** +- User can open a document, click blocks to edit, see live evaluation results, and save + +--- + +### Phase 4: Export + Polish + +- [ ] **Unit 9: Export APIs** + +**Goal:** Download individual documents as `.cm` files and bulk export all documents as a zip. + +**Requirements:** Export requirement from brainstorm + +**Dependencies:** Unit 2 (CRUD API), Unit 3 (auth) + +**Files:** +- Create: `web/handlers_export.go` +- Create: `web/handlers_export_test.go` + +**Approach:** +- `GET /api/docs/{id}/raw` already exists from Unit 2 (returns `text/plain` `.cm` source). +- `GET /api/export` — authenticated. Queries all user's documents, creates a zip archive in memory, streams it as `application/zip` with `Content-Disposition: attachment; filename="calcmark-export.zip"`. Each document is `{title_or_id}.cm` in the zip. +- `GET /api/docs/{id}/export` — single document download with `Content-Disposition` header for browser download prompt. + +**Patterns to follow:** +- Go `archive/zip` for in-memory zip creation + +**Test scenarios:** +- Happy path: Export single document → downloads as `.cm` file +- Happy path: Export all → downloads zip containing all user's documents +- Edge case: User with zero documents → empty zip (or 204 No Content) +- Edge case: Two documents with same title → filenames deduplicated (append ID) +- Error path: Unauthenticated bulk export → 401 + +**Verification:** +- Downloaded `.cm` files are valid and work in `cm eval` +- Zip contains all expected documents + +--- + +- [ ] **Unit 10: Binary Entry Point + Deployment** + +**Goal:** Main binary entry point for the web product. Dockerfile with Litestream for continuous backup. + +**Requirements:** R14, R7, R8 + +**Dependencies:** All previous units + +**Files:** +- Create: `main.go` (entry point — starts HTTP server) +- Create: `Dockerfile` (multi-stage: build frontend + Go binary, runtime with Litestream) +- Create: `docker-compose.yml` (binary + Caddy + Litestream example) +- Create: `litestream.yml` (Litestream config for S3/GCS WAL streaming) +- Create: `README.md` (deployment guide) + +**Approach:** +- Binary is the web product's own binary (separate repo from go-calcmark), not a `cm` subcommand. +- Flags: `--port` (default 8080), `--data-dir` (default `./data/`), `--dev` (proxy to SvelteKit dev server). +- SQLite database at `{data-dir}/calcmark.db`. +- OAuth config from environment variables. +- Graceful shutdown on SIGINT/SIGTERM. +- Startup banner: prints URL, data directory, auth status. +- No OAuth env vars → public-only mode (no auth, all docs public). +- **Continuous backup**: Litestream runs alongside the binary (in Docker, as a wrapper process). Streams SQLite WAL changes to S3/GCS/any S3-compatible bucket. Restore = `litestream restore`. Backup is continuous and automatic. +- **Simple backup alternative**: `cp data/calcmark.db backup/` or `rsync` — SQLite is a single file. + +**Patterns to follow:** +- Litestream deployment patterns (litestream.io) +- `cmd/calcmark/cmd/watch.go` for CLI flag patterns + +**Test scenarios:** +- Happy path: Binary starts, prints banner with URL and data directory +- Happy path: `--port 8080` binds to custom port +- Happy path: No OAuth vars → starts in public-only mode with warning +- Edge case: Data directory doesn't exist → created automatically +- Error path: Port already in use → clear error message +- Integration: Docker build produces a single image with embedded frontend + +**Verification:** +- Binary starts the full web application +- Litestream config streams changes to configured bucket +- `cp` or `rsync` of the SQLite file produces a valid backup + +## System-Wide Impact + +- **Interaction graph:** The web server calls `calcmark.Convert()` and `calcmark.Eval()` — the existing public API. No changes to the CalcMark engine. The web layer is purely additive. +- **Error propagation:** CalcMark evaluation errors become JSON diagnostic arrays in the API response. The Svelte frontend renders them inline. No new error types needed. +- **State lifecycle risks:** SQLite WAL mode for concurrent reads. Only one writer at a time (single-user editing). Auto-save debouncing prevents rapid writes. +- **API surface parity:** The web app uses the same `calcmark.Convert()` that the CLI uses. Documents created in the web app work identically in `cm eval` and `cm edit`. +- **Binary size:** Adding the web frontend (SvelteKit static build + SQLite driver) will increase the binary. Estimate: +5-10MB. Acceptable for a single-binary deployment. +- **Unchanged invariants:** All existing CLI commands, TUI, LSP, and the CalcMark engine are completely unaffected. The web product is a separate repository and binary — it consumes `go-calcmark` as a library dependency. Zero changes to `go-calcmark` are required. +- **Backup strategy:** Litestream for continuous WAL streaming (S3/GCS). Simple alternative: `cp`/`rsync` of the SQLite file. Both are documented deployment patterns. + +## Risks & Dependencies + +| Risk | Mitigation | +|------|------------| +| SvelteKit + Go embed build complexity | `go generate` automates npm build. CI/CD runs `go generate` before `go build`. Dev mode proxies to SvelteKit dev server. | +| Binary size increase from embedded frontend | SvelteKit adapter-static produces small bundles. Monitor size. If too large, consider gzip compression of embedded assets. | +| CodeMirror block editing UX is complex | Start with a simple two-state component (rendered/editing). Iterate on UX during implementation. The block model is well-understood from Jupyter. | +| modernc.org/sqlite performance for concurrent reads | SQLite WAL mode handles concurrent reads well. Single-user editing means minimal write contention. Benchmark during implementation. | +| OAuth requires HTTPS in production | Caddy handles TLS automatically. Dev mode uses HTTP on localhost (OAuth providers allow localhost callbacks). | +| go-pkgz/auth may not fit perfectly | Library is well-maintained and MIT licensed. If it doesn't fit, fall back to `golang.org/x/oauth2` with manual session management. | + +## Documentation / Operational Notes + +- Add `cm serve` to CLI help and documentation +- Document deployment: single binary + Caddy + Litestream pattern +- Document environment variables for OAuth configuration +- Add a "Getting Started" page for the web product on calcmark.org + +## Sources & References + +- **Origin document:** [docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md](docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md) +- **Existing watch server:** `cmd/calcmark/cmd/watch.go` +- **CalcMark public API:** `convert.go`, `eval.go`, `session.go` +- **GoReleaser config:** `.goreleaser.yaml` +- **SQLite pure Go:** `modernc.org/sqlite` +- **OAuth library:** `github.com/go-pkgz/auth/v2` +- **SvelteKit adapter-static:** `@sveltejs/adapter-static` +- **CodeMirror Svelte wrapper:** `svelte-codemirror-editor` +- Related issue: CalcMark/go-calcmark#118 From a20683ee22f1c609a8bd8440fca6857d6dce8c9b Mon Sep 17 00:00:00 2001 From: Dylan Thomas Date: Tue, 7 Apr 2026 16:18:53 -0600 Subject: [PATCH 6/6] docs: add semantic editor intelligence and cmw deployment model (#118) Add R18-R21 covering the killer editor features: interpolation autosuggest, calc-table preview, visual table editor, and fluid prose-calculation boundary. Add cmw binary naming, cmw backup command, and separate project requirement. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../2026-04-07-calcmark-notebooks-requirements.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md b/docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md index 0ee3873c..53c3956c 100644 --- a/docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md +++ b/docs/brainstorms/2026-04-07-calcmark-notebooks-requirements.md @@ -21,6 +21,13 @@ The unique wedge: **plain-text source of truth + single-binary self-hosting + ca - R4. Beautiful markdown rendering — tables, headings, prose, interpolated values all render with high visual quality. - R5. Single-user editing for MVP. +**Semantic Editor Intelligence (the killer feature)** + +- R18. **Interpolation autosuggest**: typing `{{ ` in any markdown block triggers a dropdown of all defined CalcMark variables from anywhere in the document. Suggestions include variable name, current value, and a natural language description (e.g., "sum of hc column in people table" or "total project cost: $1.2M"). +- R19. **Calc-table preview**: when a user clicks a CalcMark block that references a named table, the table renders inline as a mini-spreadsheet showing computed column values overlaid on the source data. The user sees inputs and outputs together without switching context. +- R20. **Visual table editor**: markdown tables can be edited visually — drag to reorder columns, click cells to edit values, add/remove rows. The editor produces valid markdown pipe table syntax. When a table has a CalcMark directive, editing a cell triggers re-evaluation of dependent calculations. +- R21. **Fluid prose-calculation boundary**: the editor understands the document semantically. Editing flows naturally between prose, calculations, and data without hard modal switches. CalcMark diagnostics, autosuggest, and documentation appear contextually wherever the cursor is — in a calc block, in an interpolation tag, or in a table cell. + **Storage and State** - R6. Each document is stored as its `.cm` source text plus metadata (owner, timestamps). The source text is the single source of truth. @@ -45,8 +52,10 @@ The unique wedge: **plain-text source of truth + single-binary self-hosting + ca **Deployment** -- R14. Single Go binary that serves the web UI, evaluates CalcMark, and stores documents in SQLite. Zero external infrastructure required. Deploy to a VPS, Docker, or fly.io. +- R14. Single Go binary (`cmw`) that serves the web UI, evaluates CalcMark, and stores documents in SQLite. Zero external infrastructure required. Deploy to a VPS, Docker, or fly.io. Works identically locally: `cmw start --port 3141` and start working. - R15. API-first: the web UI communicates with the backend via HTTP. Clean package separation in Go enables future extraction if needed, but no formal API versioning in MVP. +- R15a. `cmw backup` command: copies the SQLite database to a destination (local path, S3, etc.). For continuous backup in production, Litestream streams WAL changes. For local use, `cmw backup ./my-backup/` is enough. +- R15b. Separate project and binary (`CalcMark/calcmark-web`). Consumes `go-calcmark` as a library. Own release cycle, own deployment. Named tables is NOT a prerequisite — the web product ships with current CalcMark capabilities and gains table support when it lands in the library. **Evaluation**