diff --git a/docs/checkpoints.md b/docs/checkpoints.md new file mode 100644 index 0000000..ce5da6f --- /dev/null +++ b/docs/checkpoints.md @@ -0,0 +1,173 @@ +# Checkpoints + +Automatically seal a GROUP or LIMIT result into an anonymous CTE so subsequent operations compose on top of it instead of corrupting it. + +## Why + +GROUP and LIMIT produce a final, bounded result set — further joins, or further filtering/ordering/selecting, appended after them produce malformed SQL if just tacked onto the same query (a GROUP BY inside a subquery that is then joined, a LIMIT applied before the join filter, or — for select:/where:/order: — the query builder falling back to treating the pipeline as an ordinary ungrouped select, since it decides how to build the SQL from the *last* operation's type, not from whether a GROUP happened earlier). Checkpoints solve this by detecting when such an operation follows a GROUP or LIMIT and implicitly wrapping the preceding query in a CTE, then continuing on top of that CTE. + +Without checkpoints, `company | l: 10 | employee` would attempt to build a single query with LIMIT and a JOIN, which contradicts the intent (get 10 companies, then navigate to their employees). Similarly, `company | group: name | o: name desc` would resolve ORDER BY against the pre-group table instead of the grouped result. + +## Terminology + +"Checkpoint" and "seal" name two different things, not two names for one thing: + +- A **checkpoint** is the *pending state* — the noun. After a GROUP or LIMIT, the pipeline has a checkpoint pending (`:pending-checkpoint`) until something resolves it. +- **Sealing** is the *action* that resolves a pending checkpoint — the verb. `seal-as-cte` is the one function that does it: snapshot the state, seed its references, and inject it as a CTE. + +So a checkpoint is *pending*, and gets *sealed* into a CTE — the same relationship as a transaction being pending and then committed. The doc and public feature name is "checkpoints"; "seal" only ever refers to the specific act of materializing one into a CTE. + +## Syntax + +No syntax change is required. Checkpoints fire automatically when a TABLE operation, or a select:/where:/order: (complete or partial), follows a GROUP or LIMIT: + +``` + | (group:|limit:) ... | [| more-ops...] +``` + +Optionally assign the auto-generated CTE a name by placing `|= name` between the checkpoint op and the table: + +``` + | (group:|limit:) ... |= |
[| more-ops...] +``` + +## Examples + +### LIMIT checkpoint (auto-named) + +``` +x.company | l: 10 | employee +``` + +Pine seals `x.company | l: 10` as an anonymous CTE `__pine_0__` and joins `employee` to it: + +```sql +WITH "__pine_0__" AS ( + SELECT "c_0".* FROM "x"."company" AS "c_0" LIMIT 10 +) +SELECT "e_1".id AS "__e_1__id", "e_1".* +FROM "__pine_0__" AS "__pine_0__" +JOIN "employee" AS "e_1" ON "__pine_0__"."id" = "e_1"."company_id" +LIMIT 250 +``` + +### LIMIT checkpoint (user-named via `|=`) + +``` +x.company | l: 10 |= pg | employee +``` + +Same result but the CTE is named `pg`: + +```sql +WITH "pg" AS ( + SELECT "c_0".* FROM "x"."company" AS "c_0" LIMIT 10 +) +SELECT "e_1".id AS "__e_1__id", "e_1".* +FROM "pg" AS "pg" +JOIN "employee" AS "e_1" ON "pg"."id" = "e_1"."company_id" +LIMIT 250 +``` + +### GROUP checkpoint + +``` +x.company | group: id => count | employee +``` + +The GROUP result is sealed as a CTE and employee is joined to it: + +```sql +WITH "__pine_0__" AS ( + SELECT "c_0"."id", COUNT(1) AS "count" + FROM "x"."company" AS "c_0" + GROUP BY "c_0"."id" +) +SELECT "e_1".id AS "__e_1__id", "e_1".* +FROM "__pine_0__" AS "__pine_0__" +JOIN "employee" AS "e_1" ON "__pine_0__"."id" = "e_1"."company_id" +LIMIT 250 +``` + +### ORDER checkpoint + +``` +company as c | employee .company_id | group: c.name | o: name desc +``` + +`o:` is not a table op, but it still fires the pending GROUP checkpoint — otherwise `build-query` would dispatch on `:order` (the last op) instead of `:group`, and silently build an ungrouped query with `ORDER BY` resolved against the stale pre-group table alias. Sealing first makes this identical to the sealed cross-expression form (`group: c.name |= x` in one expression, `x | o: name desc` in the next): + +```sql +WITH "__pine_0__" AS ( + SELECT "c"."name", COUNT(1) AS "count" + FROM "company" AS "c" + JOIN "employee" AS "e_1" ON "c"."id" = "e_1"."company_id" + GROUP BY "c"."name" +) +SELECT "__pine_0__".* FROM "__pine_0__" AS "__pine_0__" ORDER BY "__pine_0__"."name" DESC LIMIT 250 +``` + +The same applies to `select:`/`where:`, and to their `-partial` forms (e.g. `o: name desc,` with a dangling trailing comma) — a partial with a value already carries whatever was fully typed before the comma, so it seals exactly like the complete form. + +### No checkpoint when an action op follows + +``` +company | limit: 100 | count: +``` + +`count:` (like `delete:`/`update:`) builds its own wrapper query generically regardless of what came before it, so it doesn't need the checkpoint sealed first — no checkpoint fires, and the pipeline remains a single query with COUNT wrapping it normally: + +```sql +WITH x AS ( SELECT "c_0".* FROM "company" AS "c_0" LIMIT 100 ) +SELECT COUNT(*) FROM x +``` + +## How it works + +- **Detection**: after processing a GROUP or LIMIT op, `handle-ops` sets `:pending-checkpoint {:needs-assign true}` on the state. +- **Fire on table, checkpoint, or checkpoint-consuming op**: at the start of each `handle-ops` iteration, `flush-checkpoint` checks whether a checkpoint is pending and whether the incoming op is a TABLE op, another checkpoint op (GROUP or LIMIT), or a checkpoint-*consuming* op (`select`/`select-partial`/`where`/`where-partial`/`order`/`order-partial`). If so, the current state is snapshotted into `:pending-assignments` under an auto-generated name (`__pine_0__`, `__pine_1__`, ...), the references map is seeded for that name (same FK propagation used by variables), and the state is reset. The CTE name is then injected as the first table so subsequent ops compose on top of it. A LIMIT following a GROUP therefore fires the GROUP checkpoint, then applies the limit to the outer query. +- **User-named CTE**: if an `|= name` op appears between the checkpoint op and the following op, `flush-checkpoint` records the name and waits. `assign/handle` stores the snapshot under `name` as normal. When the next firing op arrives, `seal-as-cte` activates it. +- **Hold for non-triggering ops**: if the op after GROUP/LIMIT is something that has its own query-building path — `count:`, `delete:`, `update:` — the checkpoint stays pending and that op is processed without firing the checkpoint. +- **SQL generation**: because the CTE table has an `:ast` entry in `:aliases`, `collect-ctes` automatically picks it up and emits the `WITH` clause. No changes to `eval.clj` were needed. + +## Constraints + +- Checkpoint op types (what *creates* a pending checkpoint) are GROUP and LIMIT. +- Checkpoint-consuming op types (what *fires*/seals an already-pending checkpoint, alongside TABLE) are `select`/`select-partial`/`where`/`where-partial`/`order`/`order-partial`. `count:`/`delete:`/`update:` deliberately do not fire — see above. +- Only one table-level composition step is supported per checkpoint; chaining `l: 10 | employee | document` creates one CTE for `l: 10` and then navigates normally through employee to document. +- Auto-generated CTE names (`__pine_0__`, etc.) are numbered per expression and are not exposed to the user. +- Checkpoint CTEs do not receive Pine's auto-id columns; those are suppressed for all CTE-backed tables. + +See also: [variables.md](variables.md) for cross-expression named CTEs. + +--- + +## Implementation + +### State fields (`ast/main.clj`) + +| Field | Default | Purpose | +|---|---|---| +| `:auto-cte-count` | `0` | Counter for generating `__pine_0__`, `__pine_1__`, etc. | +| `:pending-checkpoint` | `nil` | `{:needs-assign true}` after a checkpoint op; `{:name n :needs-table true}` after an explicit `|=` | + +### Core functions (`ast/main.clj`) + +- `checkpoint-op-types` — set `#{:group :limit}`; ops that *create* a pending checkpoint +- `checkpoint-consuming-op-types` — set `#{:select :select-partial :where :where-partial :order :order-partial}`; ops that *fire* (seal) an already-pending checkpoint, alongside a TABLE op or another checkpoint op. Partial and complete forms are treated identically — a partial with a value has already fully typed whatever precedes a dangling trailing comma. +- `reset-for-cte` — clears all query-building fields (tables, columns, joins, where, order, group, limit) while preserving references, variables, and pending-assignments +- `seal-as-cte` — the single shared action: stores the snapshot under `cname` in `:pending-assignments`, seeds FK references for `cname` (same three passes used by cross-expression variables in `pre-handle`), resets query-building state, then injects `cname` as the first table via `handle-op`; `table/handle` resolves it via `:pending-assignments` +- `flush-checkpoint` — state-machine dispatch called at the start of each `handle-ops` reduce iteration; calls `seal-as-cte` when a TABLE op, another checkpoint op, or a checkpoint-consuming op follows the pending checkpoint, using either a freshly-snapshotted state (auto-named) or the snapshot already stored by `assign/handle` (user-named) + +### `table/handle` (`ast/table.clj`) + +Updated to check `:pending-assignments` alongside `:variables` when looking up the CTE for a table name: + +```clojure +var-ast (or (get-in state [:variables table]) + (get-in state [:pending-assignments table])) +``` + +### `add-auto-id-columns` (`ast/select.clj`) + +Changed the predicate from checking `variables` membership to checking `(:ast (get aliases %))`. Any alias that carries an `:ast` entry is a CTE-backed table and should not receive an auto-id column. diff --git a/docs/expressions.md b/docs/expressions.md new file mode 100644 index 0000000..6bf69ed --- /dev/null +++ b/docs/expressions.md @@ -0,0 +1,122 @@ +# Expressions + +A unit of Pine input — one pipe chain — and how multiple expressions compose. + +## Why + +Pine is a REPL-style tool. A session naturally builds up state: filter a set, name it, join through it, +aggregate. A single pipe chain can do a lot, but some queries need intermediate steps that are cleaner as +separate named expressions than as deeply nested subqueries. Multiple expressions, evaluated in order, +give you that incremental composition while keeping each step readable on its own. + +## Syntax + +A single expression is one pipe chain: + +``` +company | where: active = true | employee | s: name +``` + +Multiple expressions are separated by blank lines: + +``` +company | where: active = true |= active_companies + +active_companies | employee | s: name +``` + +The API receives them as an array of strings — one string per expression, blank lines already stripped. + +## Operations + +Each pipe segment maps to a typed operation. Operations are applied left to right to build the query state: + +| Pipe segment | What it does | +|------------------|-----------------------------------------------------------------------------| +| `table` | Registers the table, resolves FK joins, tracks current context | +| `select` / `s:` | Records which columns to include | +| `where` / `w:` | Records filter conditions and parameters | +| `group` / `g:` | Records group-by columns and aggregate functions | +| `order` / `o:` | Records sort columns and direction | +| `limit` / `l:` | Records the row limit | +| `count:` | Switches operation mode to COUNT | +| `update!` / `u!` | Records column assignments for UPDATE | +| `delete!` | Marks the operation as DELETE | +| `= name` | Snapshots the current state into `:pending-assignments`; pipeline continues | + +See [pipeline.md](pipeline.md) for how these operations are processed internally. + +## How evaluation works + +Expressions are evaluated left to right. Only the **last expression** produces output (query or result +rows). Earlier expressions exist solely to define variables that the last expression can use. + +``` +expr₁ → variables₁ +expr₂(variables₁) → variables₁₂ +expr₃(variables₁₂) → SQL / rows ← this is what's returned +``` + +Each expression passes through the full parse → generate → eval pipeline independently. Variables +accumulate: every `|= name` op in an expression adds a named snapshot to the variable map, which is +merged into the map passed to the next expression. + +If any expression fails — parse error or generate error — evaluation stops immediately and the error is +returned. The last expression is never run. + +## Constraints + +- Only the last expression's SQL is built or executed. Earlier expressions are only evaluated to collect + variables — they do not produce query output. +- A variable used but not defined in a preceding expression causes a table-not-found error. +- Expressions are stateless per request. No state carries over between API calls. +- Circular variable references are not supported. + +--- + +## Implementation + +### API entry points (`api.clj`) + +`api-build` and `api-eval` both receive an `expressions` array. Both split it the same way: + +- **context expressions** — all but the last, passed to `evaluate-expressions` +- **last expression** — processed separately with the accumulated variable map + +`api-build` returns the query string and AST (used by the UI to show hints and the SQL preview). +`api-eval` runs the query and returns rows. + +### Variable threading (`evaluate-expressions`, `api.clj`) + +```clojure +(reduce (fn [{:keys [variables]} expression] + (let [{:keys [result error]} (generate-state expression nil connection-id variables)] + (if error + (reduced {:error error}) ;; short-circuit on first error + {:variables (merge variables (:pending-assignments result)) + :last-state result}))) + {:variables {} :last-state nil} + expressions) +``` + +Each expression's `:pending-assignments` map (keyed by variable name, valued by state snapshots from +`|=` ops) is merged into the running `variables` map. `reduced` short-circuits the reduce on error so +later expressions are never evaluated. + +### Per-expression pipeline + +Each call to `generate-state` runs: + +1. `parser/parse` — tokenises the expression into a typed operation list. +2. `ast/generate` — folds operations into a state map, seeding the DB schema and any in-scope variables + at the start. `|=` ops produce `:pending-assignments` entries in the returned state. +3. The caller merges `:pending-assignments` into `variables` for the next expression. + +The last expression goes through the same `generate-state` call and then into `build-query` or +`run-query` depending on the endpoint. + +### Hints for the last expression + +`api-build` calls `generate-state` on the last expression **twice**: once with the full expression +(for the AST and query), and once with the trimmed expression (for the `query` field in the response). +Both calls receive the same `variables` map so hints reflect variables defined by earlier expressions. diff --git a/docs/joins.md b/docs/joins.md new file mode 100644 index 0000000..e60ffa0 --- /dev/null +++ b/docs/joins.md @@ -0,0 +1,168 @@ +# Joins + +Pipe two table names together and Pine figures out the JOIN condition automatically. + +## Why + +SQL joins require you to spell out `ON table_a.col = table_b.col` every time, even when the relationship is +already encoded in the schema as a foreign key. Pine reads the FK graph once at startup and resolves the join +condition for you. When there is no FK (e.g. a multi-tenant column added by convention), Pine falls back to +heuristic detection based on column naming. + +## Syntax + +``` +table_a | table_b +table_a | table_b .hint_col +table_a | table_b .left_col = .right_col +table_a | table_b :parent +table_a | table_b :left +``` + +- **No modifier** — Pine picks the join direction automatically. +- **`.hint_col`** — disambiguate when two tables share more than one FK. +- **`.col1 = .col2`** — override both sides explicitly; bypasses the reference map entirely. +- **`:parent`** — force the join to treat `table_b` as the parent (i.e. the table `table_a` refers to, + not the table that refers to `table_a`). +- **`:child`** — inverse of `:parent`; explicit but rarely needed since it is the default. +- **`:left` / `:right`** — emit `LEFT JOIN` / `RIGHT JOIN`. + +## Examples + +### Basic (FK-resolved) + +``` +company | employee +``` + +```sql +SELECT "c_0".id AS "__c_0__id", "e_1".* +FROM "company" AS "c_0" +JOIN "employee" AS "e_1" ON "c_0"."id" = "e_1"."company_id" +LIMIT 250 +``` + +### Disambiguation hint + +`document` has two FKs to `employee` (`employee_id` and `created_by`). Without a hint Pine picks the first +one alphabetically. + +``` +employee | document .created_by +``` + +```sql +JOIN "document" AS "d_1" ON "e_0"."id" = "d_1"."created_by" +``` + +### Forcing parent direction + +``` +employee | company :parent +``` + +Forces Pine to treat `company` as the parent even though `:has` would also match. Equivalent to the default +here, but necessary when the automatic direction would be wrong. + +### Left join + +``` +company | employee :left +``` + +```sql +LEFT JOIN "employee" AS "e_1" ON "c_0"."id" = "e_1"."company_id" +``` + +### Explicit columns + +``` +employee | document .company_id = .id +``` + +Bypasses the reference map. `company_id` is on `document` (right table); `id` is on `employee` (left table). + +## How it works + +1. At startup, Pine queries the database for all foreign keys and scans all column names. It builds a + **references map** used for every subsequent join resolution. +2. When the parser sees `table_a | table_b`, it emits two consecutive `:table` operations. +3. `table/handle` in `ast/table.clj` calls `update-joins`, which calls `join-tables`, which calls `join-helper`. +4. `join-helper` looks up the resolved join vector from the references map and records it on the AST + state's `:joins` vector. +5. `eval/build-join-clause` turns each entry in `:joins` into a SQL `JOIN … ON …` fragment. + +## Constraints + +- Circular joins are not detected — the query will compile but the SQL may be nonsensical. +- Heuristic joins are only inferred when no FK already covers the same pair. +- Self-referential heuristic joins are suppressed. Real self-referential FKs (e.g. + `employee.reports_to → employee.id`) are supported. + +--- + +## Implementation + +### Reference map structure (`db/postgres.clj`) + +`index-references` builds the map in three passes: + +1. **`index-foreign-keys`** — queries `pg_constraint` and indexes every FK in both directions: + - `refs[:table f-table :referred-by table :via col]` — child direction ("who points at me") + - `refs[:table table :refers-to f-table :via col]` — parent direction ("who I point at") + + Each entry is a list of **join vectors**: + ``` + [f-schema f-table f-col :referred-by schema table col :foreign-key] + [schema table col :refers-to f-schema f-table f-col :foreign-key] + ``` + +2. **`index-columns`** — adds column metadata to each table entry. Needed before the next pass. + +3. **`index-heuristic-relations`** — scans every column looking for `_id` / `Id` suffixes: + - `extract-table-from-column` strips the suffix: `company_id` → `company`, `tenantId` → `tenant`. + - `normalize-plural` generates candidate names: `company` → `#{"company" "companies"}`. + - Matches against all known tables via `build-table-lookup`. + - Skips if the candidate table has no `id` column, if the FK already exists, or if it would be a + self-referential heuristic. + - Adds the same two-direction index entries as FK detection, tagged `:heuristic` instead of `:foreign-key`. + +The same map structure is used for both FK and heuristic entries; the only difference is the tag in position 7 +of the join vector. Callers can inspect it if they want to surface confidence level. + +### Join direction resolution (`ast/table.clj`) + +`join-tables` tries two strategies in order: + +1. **`:has`** — `join-helper` looks up `refs[:table t1 :referred-by t2]`. This succeeds when `t2` has a + FK (or heuristic) pointing at `t1`. Returns `[a1 col :has a2 f-col]`. +2. **`:of`** — arguments swapped: `join-helper` looks up `refs[:table t2 :referred-by t1]`. + Returns `[a2 f-col :of a1 col]`. + +`:has` is tried first unless the `:parent` modifier is set, in which case only `:of` is attempted. + +When `.hint_col` is present, `join-column` is set. `join-helper` uses it to select a specific key from the +`via` map instead of taking `first`. + +When `.col1 = .col2` is present, `join-left-column` and `join-right-column` are set. `update-joins` bypasses +`join-tables` entirely and records the explicit pair directly. + +### Grammar (`pine.bnf`) + +``` +TABLE := table table-mods +table-mods := ( table-mod)* +table-mod := <":"> ("parent"|"child"|"left"|"right") | as-alias | hint-columns +hint-columns := hint-column | explicit-columns +explicit-columns := hint-column <"="> hint-column +``` + +### SQL generation (`eval.clj`) + +`build-join-clause` maps each `[from-alias to-alias relation join-type]` entry in `:joins` to: + +```sql +[LEFT|RIGHT] JOIN "schema"."table" AS "alias" ON "a1"."col1" = "a2"."col2" +``` + +The `ON` columns are positions 1 and 4 of the relation vector: `[a1 col _ a2 f-col]`. diff --git a/docs/pipeline.md b/docs/pipeline.md new file mode 100644 index 0000000..dd9472f --- /dev/null +++ b/docs/pipeline.md @@ -0,0 +1,162 @@ +# Evaluation Pipeline + +How a Pine expression goes from text to SQL results. + +## Why + +Pine is a pipe-based DSL that compiles to SQL. Understanding the pipeline helps when debugging unexpected +query output, tracing where hints come from, or extending the language with new operations. + +## Overview + +Every expression passes through three stages: + +``` +text → parse → generate → eval +``` + +**Parse** turns the text into a list of typed operations. +**Generate** folds those operations into a state map, using the DB schema to resolve joins and compute hints. +**Eval** translates the state map into SQL, and optionally runs it against the database. + +## Stages + +### 1. Parse + +The expression is tokenised against the Pine grammar and turned into a flat list of typed operations — +one per pipe segment. + +Input: +``` +user as u | document name="passport" | s: u.email, id | l: 10 +``` + +Output: +``` +[ table(user as u), table(document name="passport"), select(u.email, id), limit(10) ] +``` + +Each operation has a type and a value. `|= name` produces an `:assign` operation in the list like any +other op, so the pipeline can continue after it. + +### 2. Generate + +Each operation is applied left to right to a state map, using the appropriate handler. Before the first +operation runs, the DB schema is loaded so that join paths and column lists are available. For the full +list of operations and what each one does, see [expressions.md](expressions.md). + +After all operations are applied, a **post-processing** step runs: autocomplete hints are computed, +hidden auto-id columns are appended for row tracking, and a prettified form of the expression is attached. + +The result is a single **state map** that fully describes the query. + +### 3. Eval + +The state map is handed to the eval layer, which has two paths: + +**Build** — translates the state into a SQL string and parameter list. No database call. Used to show +the query in the UI as the user types. + +**Run** — builds the SQL then executes it against the database, returning rows. + +## How it works + +- The pipeline is stateless per request — each call to the API runs the full parse → generate → eval + cycle from scratch. +- Hints are computed inside `generate`, using a second pass over the expression truncated at the cursor + position. This gives hints that reflect what the user has typed so far, not the full completed expression. +- When multiple expressions are present, they are evaluated left to right, each seeing variables defined + by earlier ones. See [expressions.md](expressions.md). + +## Constraints + +- The pipeline is synchronous and single-pass — there is no incremental or lazy evaluation. +- All schema information is resolved at generate time. A table or column that doesn't exist in the DB + schema will fail at that stage, not at SQL execution time. + +--- + +## Implementation + +### Parse (`parser/parse`, `pine.bnf`) + +Instaparse runs the BNF grammar over the input string. The raw parse tree is normalised into a vector of +`{:type :value }` maps. `|= name` produces `{:type :assign :value "name"}` in the list +like any other operation. + +### Generate (`ast/generate`, `ast/main.clj`) + +Full signature: + +```clojure +(generate parse-tree connection-id expression cursor variables) +``` + +Each parameter fills a distinct role that cannot be inferred from the others: + +| Parameter | Source | Used for | +|-----------------|----------------------------------------|--------------------------------------------------------------| +| `parse-tree` | `(parser/parse expression)` | `handle-ops` — the typed operation list | +| `connection-id` | caller / DB config | `db/init-references` — loading the schema | +| `expression` | raw input string | `add-prettify` (ranges), `truncate-at-cursor` (cursor hints) | +| `cursor` | UI (user's caret position) | Building `truncated-state` for cursor-aware hints | +| `variables` | caller, accumulated across expressions | `pre-handle` — seeding FK references for CTEs | + +`parse-tree` comes from `parser/parse`, but `expression` is still required separately because +`add-prettify` and `truncate-at-cursor` operate on raw text. `cursor` and `variables` are external +context that is not derivable from the expression itself — `cursor` comes from the UI, and `variables` +accumulates as the caller evaluates earlier expressions in a multi-expression session. + +`generate` orchestrates three sub-steps: + +**`pre-handle`** — seeds the initial state with the DB schema via `db/init-references` (cached per +connection). If variables are in scope, `seed-variable-references` merges their underlying FK references +so join resolution works through CTEs. + +**`handle-ops`** — calls `reduce` over the operation list, dispatching each operation to its handler +module (`ast/table`, `ast/select`, `ast/where`, `ast/group`, `ast/order`, `ast/limit`, `ast/count`, +`ast/delete-action`, `ast/update-action`, `ast/assign`). The operation index (`i`) is threaded through +so later stages know the order columns were introduced. `:assign` ops do not update `:operation` so +`build-query` dispatch always sees the last SQL-producing operation. + +**`post-handle`** — runs after all operations: +- `ast/hints/handle` computes autocomplete hints using the truncated-at-cursor state. +- `ast/select/add-auto-id-columns` appends hidden `id` columns for each real table. + See [result-updates.md](result-updates.md). +- `add-prettify` attaches a formatted expression and per-operation character ranges for cursor highlighting. + +The final state map: + +```clojure +{ :tables [ {:table "user" :alias "u"} + {:table "document" :alias "d_1"} ] + :aliases { "u" {:table "user"} "d_1" {:table "document"} } + :current "d_1" + :context "u" + :columns [ {:alias "u" :column "email"} {:alias "d_1" :column "id"} ] + :joins [ ["u" "d_1" nil] ] + :where [] + :limit 10 + :operation {:type :select} + :hints {:table [...] :select [...] :where [...]} } +``` + +### Eval (`eval/build-query`, `eval/run-query`) + +`build-query` dispatches on operation type: + +| Type | Builder | +|--------------------------------------|------------------------| +| `:select` (default) | `build-select-query` | +| `:count` | `build-count-query` | +| `:group` | `build-group-query` | +| `:update-action` / `:update-partial` | `build-update-queries` | +| `:delete-action` | `build-delete-query` | + +`build-select-query` checks for variables in scope and, if present, prepends CTE clauses via `collect-ctes`. +`run-query` calls `build-query` then executes against the connection pool via `db/run-query`. + +### Empty-expression guard + +`build-query` checks whether the current table name (looked up via the aliases map) is an empty string. +This handles blank input — returns `{:query "" :params nil}` rather than generating a malformed SELECT. diff --git a/docs/result-updates.md b/docs/result-updates.md new file mode 100644 index 0000000..66801ab --- /dev/null +++ b/docs/result-updates.md @@ -0,0 +1,171 @@ +# Updating Results from Join Queries + +Editing cells in multi-table result sets. + +## Why + +A Pine expression like `user | document` returns a result set that mixes columns from two tables. You +might want to edit a document's `name` field directly in that result. The challenge is that a naive +`UPDATE document SET name = ?` doesn't know which specific document row to target — it needs the `id` +of the document for the row you edited, not just any document. + +Pine solves this with two mechanisms working together: + +- **Auto-id columns**: for each table in the query, Pine silently appends a hidden column + `alias.id AS "__alias__id"`. These are never displayed but are always present in the result, so the + UI can look up the exact row ID for any cell. +- **`update!`**: generates separate UPDATE statements per table, each scoped to the correct row via a + subquery that inherits the full join condition. + +Together they make any join result editable, not just readable. + +## Syntax + +``` + | update! = + | u! = +``` + +Multiple assignments, optionally targeting different tables: + +``` +user as u | document | u! u.name = 'Alice', title = 'Passport' +``` + +## Examples + +### Single-table update + +``` +company | where: id = 42 | u! name = 'Acme Corp' +``` + +```sql +UPDATE "company" SET "name" = ? WHERE id IN ( + SELECT "c_0"."id" FROM "company" AS "c_0" WHERE "c_0"."id" = 42 +) +``` + +### Multi-table update + +``` +user as u | document | u! u.name = 'Alice', title = 'Passport' +``` + +Pine generates one UPDATE per table: + +```sql +UPDATE "user" SET "name" = ? WHERE id IN ( + SELECT "u"."id" FROM "user" AS "u" JOIN "document" AS "d_0" ON ... +) + +UPDATE "document" SET "title" = ? WHERE id IN ( + SELECT "d_0"."id" FROM "user" AS "u" JOIN "document" AS "d_0" ON ... +) +``` + +Each subquery targets only the relevant table's `id`, scoped by the full join condition. + +### Auto-id columns in the result + +``` +user as u | document +``` + +The generated SQL includes hidden auto-id columns alongside the visible data: + +```sql +SELECT "u".id AS "__u__id", + "d_0".id AS "__d_0__id", + "d_0".* +FROM "user" AS "u" +JOIN "document" AS "d_0" ON "u"."id" = "d_0"."user_id" +LIMIT 250 +``` + +`__u__id` and `__d_0__id` are hidden in the grid but available when a cell is edited. + +### Inline cell edit + +When you edit a cell directly in the result grid, the UI constructs an `update!` expression automatically: + +1. The edited cell belongs to column index N. +2. `colIndexToAliasLookup[N]` → table alias (e.g. `"d_0"`). +3. `aliasToIdLookup["d_0"]` → index of the `__d_0__id` column in the row. +4. The ID value is read from that column in the current row. +5. Pine evaluates: ` | where: d_0.id = | u! = ` + +## How it works + +- **Auto-id columns** are appended automatically for every real table that has an `id` column. They are + marked `hidden: true` (not shown in the grid) and `auto-id: true` (so internal logic can identify + them). Variable/CTE tables are excluded — they don't have a stable `id`. +- **Column qualification**: unqualified columns (e.g. `name`) default to the last table in the expression. + Qualified columns (`u.name`) target the specified alias. +- **One UPDATE per table**: assignments are grouped by target alias. Each group becomes an independent + UPDATE with a subquery to identify the rows. +- **Subquery for row targeting**: Pine uses `WHERE id IN (SELECT id FROM ... JOIN ...)` rather than a + direct `WHERE id = ?`. This ensures the update respects the full join condition. + +## Constraints + +- Only tables with an `id` column in the DB schema get an auto-id. Tables without `id` cannot be + targeted by `update!`. +- If the user explicitly selects `id` (e.g. `s: id, name`), the auto-id column is still added with its + `__alias__id` alias to avoid ambiguity. +- Auto-id columns are not added inside CTE bodies — `SELECT *` already includes `id`, and a second + `id` would cause an ambiguous-column error. +- `update!` on a variable/CTE table is not supported — the CTE has no physical table to update. + +--- + +## Implementation + +### Auto-id columns (`ast/select.clj`) + +`add-auto-id-columns` runs in `post-handle` after all operations are applied. For each table alias +whose underlying table has an `id` column (checked via `has-id-column?`), it appends: + +```clojure +{ :column "id" + :alias alias + :column-alias "__alias__id" + :hidden true + :auto-id true + :operation-index N } +``` + +Variable tables are skipped: if the table name for an alias is a key in `state :variables`, no auto-id +is added. + +### CTE body deduplication (`eval.clj`) + +`build-cte-body` generates the inner SQL for a variable's CTE. Before calling `build-bare-select`, it +checks whether user-specified columns already cover the current table. If so, the auto-id column is +dropped to avoid `SELECT id, ..., id` duplication. + +### Frontend column metadata (`default.plugin.tsx`) + +After a query runs, the UI builds two lookup maps from the response column metadata: + +- `colIndexToAliasLookup` — column position → table alias +- `aliasToIdLookup` — table alias → position of its auto-id column in the row + +### Frontend cell editing (`Result.tsx`) + +`processRowUpdate` is called by MUI DataGrid when a cell is committed. It uses the two lookup maps to +find the row ID, builds an update expression via `createUpdateExpression`, and evaluates it in a +virtual session before refreshing the main session. + +### SQL generation (`eval/build-update-queries`) + +Assignments are grouped by target alias. For each group, `build-single-update-query` produces the UPDATE +with a subquery built by temporarily replacing the state's column list with +`[{:column "id" :alias update-alias}]` and calling `build-select-query` — so it inherits the full JOIN +and WHERE conditions from the original expression. + +### Parsing (`pine.bnf`, `parser.clj`) + +`update!` / `u!` are parsed as `:update-action`. Each assignment carries +`{:column {:alias ... :column ...} :value {...}}`. Partial typing (`u! col`) is parsed as +`:update-partial` for hint generation. diff --git a/docs/variables.md b/docs/variables.md new file mode 100644 index 0000000..9be1342 --- /dev/null +++ b/docs/variables.md @@ -0,0 +1,215 @@ +# Variables + +Name and reuse intermediate query results across expressions. + +## Why + +Complex queries often need to build up results in steps — filter a set of rows, then join that filtered +set to another table, then aggregate. Without variables, the only way to do this is with nested subqueries, +which are hard to read and hard to iterate on in a REPL-style tool. Variables let you give each step a +name, treat it as a table, and compose them incrementally. + +## Syntax + +``` + |= [| more-ops...] +``` + +Place `|= name` anywhere in a pipe chain to snapshot the query state at that point under `name`. +The pipeline continues after the assignment — operations after `|=` refine the current expression's +output while the snapshot remains frozen. Use `name` as a table in any later expression. + +## Examples + +### Basic + +``` +company | where: active = true |= active_companies + +active_companies | employee +``` + +`active_companies` becomes a CTE. The second expression joins through it. +`|=` can appear at the end (as here) or mid-pipeline: + +```sql +WITH "active_companies" AS ( + SELECT "c_0".* FROM "company" AS "c_0" WHERE "c_0"."active" = true +) +SELECT "e_1".id AS "__e_1__id", "e_1".* +FROM "active_companies" AS "ac_0" +JOIN "employee" AS "e_1" ON "ac_0"."id" = "e_1"."company_id" +LIMIT 250 +``` + +### Mid-pipeline assign + +``` +company |= all_companies | where: active = true + +all_companies +``` + +`all_companies` is the full unfiltered company snapshot — the snapshot was taken before `where:`. +The current expression still returns only active companies. + +### Chained + +Expressions are separated by blank lines and evaluated in order. Each one can build on the last. + +``` +company | where: active = true |= active_companies + +active_companies | l: 10 |= small_active + +small_active +``` + +This produces two CTEs and selects from the final one. + +### Grouped + +``` +tenant as t | public.company .tenantId | g: t.title |= x + +x +``` + +`x` exposes only the columns the CTE actually produces — here `title` and `count` — so hints for +`x | s:` show just those two columns. + +Neither `tenant` nor `company` survives as a join source, though: grouping by `t.title` doesn't keep +`tenant`'s `id`, so `x | company` (or any other table) shows no join hints at all — see +[get-source-tables](#join-resolution-through-variables) below. Grouping by `t.id, t.title` instead would +keep `tenant` joinable. + +## How it works + +- **Assignment** (`|= name`): snapshots the pipeline state at that point. The snapshot becomes the CTE body + when `name` is used in a later expression. Operations after `|=` continue refining the current expression's + output — they do not affect the snapshot. +- **Usage**: when `name` appears as a table in a later expression, Pine substitutes the CTE. Joins resolve + in both directions — `variable | table`, `table | variable`, and `variable | variable` — using the FK + schema of the underlying real tables. +- **Scoping**: variables accumulate left-to-right. Each expression sees all variables defined before it. + Mid-pipeline assignments within the current expression are NOT visible to that same expression. +- **Column hints**: when a variable has explicit output columns (from `s:`, `g:`, etc.), hints reflect + those columns, not the full schema of the source table. Variables using `*` (no explicit selection) + inherit the source table's columns. +- **No LIMIT in CTE body**: limits only apply to the outer query. +- **No auto-id columns**: Pine's hidden `id` tracking columns are not added for variable tables. + See [result-updates.md](result-updates.md). + +## Constraints + +- Variable names must be valid Pine identifiers (letters, digits, underscores). +- A variable used but not defined in a preceding expression causes a table-not-found error. +- Circular references are not supported. + +See also: [checkpoints.md](checkpoints.md) for same-expression anonymous CTEs after GROUP/LIMIT. + +--- + +## Implementation + +### Grammar and parsing + +`|= name` is parsed as a regular pipe operation of type `:assign` in `pine.bnf`. The ASSIGN rule sits +inside OPERATION alongside TABLE, SELECT, WHERE, etc., so it participates in the normal pipeline. +`parser/parse` returns `{:result [ops...]}` with the assign op included in the list. + +### State threading + +The API receives an array of expressions. Each expression is parsed and evaluated in order: + +1. `parser/parse(expr)` returns `{:result [ops...]}`. The operation list may include `:assign` ops. +2. `ast/generate(ops, ..., variables)` runs the full pipeline. When a `:assign` op is encountered, + `assign/handle` snapshots the current state into `:pending-assignments` under the variable name. +3. After `ast/generate` returns, the caller merges `:pending-assignments` from the returned state into + the running `variables` map. +4. The updated `variables` map is passed to `ast/generate` for every subsequent expression. + +This is done purely in memory within the request — there is no persistence layer. + +### CTE generation (`eval.clj`) + +When `build-select-query` runs on a state that has variables, it calls `collect-ctes` to walk the variable +map and build `WITH name AS ( ... )` clauses. Each CTE body is generated by `build-cte-body`, which calls +`build-bare-select` (no LIMIT). WHERE params from inside CTEs are accumulated and merged into the outer +query's params list. + +### Join resolution through variables + +All three passes below build on one question, answered by the shared helper `get-source-tables`: +**which real tables can this variable actually be joined to?** A table counts as a source only if its +own `id` column is explicitly present among the CTE's output columns. Pine never adds one on the user's +behalf, for any operation — if `id` wasn't selected, that table isn't joinable through this variable. + +**Why a raw table join doesn't need this, but a sealed variable does**: `t | c` (no variable involved) +compiles to `... JOIN "c" ON "t"."id" = "c"."tenantId"` — a `JOIN ... ON` clause can reference any column +of a real table in `FROM`/`JOIN`, regardless of what's in `SELECT`. A CTE is different in kind, not degree: +once state is sealed into `WITH "x" AS ( SELECT ... )`, the outer query can no longer see `x`'s underlying +real tables at all — only whatever `x`'s own `:columns` produced. So a table stays a valid join source +*through a variable* only if the user explicitly selected its `id`. + +- **No explicit columns** (`*`, e.g. a bare `where:`/`limit:` with no `s:`/`g:`): the CTE implicitly + selects everything, which always includes `id`. The variable's `:current` table is the sole source. +- **Explicit columns** (`s:`, or `group:`'s grouped columns — both populate `:columns` the same way): + a table is a source only if `id` is literally among them. `s: name` or `group: name` alone therefore + resolves to **zero** sources (nothing joinable, no join hints should appear); `s: id, name` or + `group: id, name` resolves to one; `s: t.id, c.id` or `group: t.id, c.id` can resolve to more than + one — the same multi-table join support a plain, variable-free pipeline already has when it references + more than one table. + +With that settled, the three passes propagate joins for whichever source tables `get-source-tables` +returned, inside `pre-handle` (`ast/main.clj`): + +**Pass 1 — `seed-variable-references`**: For each variable V, for each of its source tables S, copies S's +FK reference entry into the references map under V's name. This enables `V | table` and `table | V` when +the join helper can find `table[:referred-by][V]` — but only if that entry already exists, which it +doesn't yet after pass 1 alone. A variable with zero source tables (see above) has nothing to seed and +stays unjoinable to anything, correctly. + +**Pass 2 — `patch-variable-relations`** (runs `patch-direction` once per direction): Builds a reverse +index from the references map — `source-table -> entities that already relate to it` — once per direction, +instead of scanning every entity per variable. For each variable V, for each source S, looks up S in that +index to find every entity T where `T[:referred-by][S]` exists, and registers `T[:referred-by][V]` with the +same relation data. The index is kept in sync with entries added mid-pass, so a variable-of-variable (V2 +wrapping V1, where V1 is itself a variable processed earlier in the same pass) still picks up V1's freshly +patched relations. This propagates the relationship bidirectionally: + +- `T | V` and `V | T` (real table ↔ variable) +- `V | W` and `W | V` (variable ↔ variable) + +**Pass 3 — `patch-same-source-variable-joins`**: Groups variables by source table first, so only variables +that actually share a source table are ever paired. For each ordered pair of distinct variables (V1, V2) +within a group, and sharing a common source table with an `id` column, registers a synthetic `id = id` +join at `refs[:table V1 :referred-by V2]`. This makes `V1 | V2` resolve even when the source table has no +self-referential FK. Only adds entries where no join path already exists — existing FK-based propagation +(e.g. two employee-wrapping variables joined via `reports_to`) is not overridden. + +### Column hints for variables + +`seed-variable-references` also overrides the column list for the variable entry in the references map: + +- **Explicit columns** (`s:`, `g:`, etc.): the column list is built from the user-selected columns. The + name used is `column-alias` if set, otherwise `column`. For a GROUP-sourced CTE this naturally includes + the aggregate — group.clj folds it directly into `:columns`, so it's read here like any other column + rather than appended separately. (`=> count` is optional in the *grammar*, but parser.clj defaults + `:functions` to `["count"]` whenever it's omitted — there's no way to write a truly aggregate-less + GROUP, so this case always has a `count` entry in practice.) +- **No explicit columns** (`*`): the source table's full column list is inherited. + +### Alias disambiguation + +Each table gets an alias derived from its name initials (`active_companies` → `ac`). Because `make-alias ""` +(empty input) also returns `"x"` as a fallback, the empty-expression guard in `build-query` checks the +actual table name in the aliases map rather than the alias string, to avoid false matches on a variable +named `x`. + +### Duplicate column guard in CTE body + +When the user selects explicit columns that include `id` (e.g. `s: id, name`), the CTE body must not +also emit the auto-id column, since `SELECT id, id` is ambiguous. `build-cte-body` detects whether +user-specified columns already cover the current table and drops the auto-id column from the CTE body +if so. diff --git a/src/pine/api.clj b/src/pine/api.clj index 7ed8218..2c57256 100644 --- a/src/pine/api.clj +++ b/src/pine/api.clj @@ -1,4 +1,11 @@ (ns pine.api + "HTTP API layer: routes, request parsing, and multi-expression evaluation. + + Why multi-expression evaluation exists: Pine expressions are designed to be + composed across blank-line-separated blocks. Each block can assign its result + to a variable (|= name) which subsequent blocks use as a CTE. This file + threads variables between expressions so each one sees what earlier ones + produced. The last expression's SQL is the one actually returned or executed." (:require [cheshire.generate :refer [add-encoder encode-str]] [clojure.string :as str] @@ -28,15 +35,36 @@ (defn- generate-state ([expression] - (generate-state expression nil nil)) + (generate-state expression nil nil {})) ([expression cursor] - (generate-state expression cursor nil)) + (generate-state expression cursor nil {})) ([expression cursor connection-id] + (generate-state expression cursor connection-id {})) + ([expression cursor connection-id variables] (let [{:keys [result error]} (->> expression parser/parse) conn-id (or connection-id @db/connection-id)] - (if result {:result (ast/generate result conn-id expression cursor)} - {:error-type "parse" - :error error})))) + (if result + {:result (ast/generate result conn-id expression cursor variables)} + {:error-type "parse" + :error error})))) + +(defn- evaluate-expressions + "Evaluate a sequence of pine expressions, threading variables from |= assignments + into subsequent expressions. Returns {:last-state :error }. + + Why pending-assignments: |= is now a mid-pipeline op that snapshots state at the + point of assignment. An expression can assign multiple variables before its last op + determines the final SQL. All assignments from one expression become available as + CTE variables to the next." + [expressions connection-id] + (reduce (fn [{:keys [variables]} expression] + (let [{:keys [result error]} (generate-state expression nil connection-id variables)] + (if error + (reduced {:error error}) + {:variables (merge variables (:pending-assignments result)) + :last-state result}))) + {:variables {} :last-state nil} + expressions)) (defn- trim-pipes [s] (-> s @@ -44,22 +72,64 @@ (str/replace #"^\|\s*|\s*\|$" "") (str/trim))) +(defn- prune-table + "Prune a table entry down to what the frontend's Table type (client.ts) uses. + Critical: a variable-backed table entry carries a full :ast (the variable's own + var-ast) for the query builder's CTE generation — left unpruned, it recursively + re-embeds that variable's entire state (and, transitively, everything it wraps in + turn) inside every table list that references it." + [table] + (select-keys table [:schema :table :alias])) + +(defn- prune-var-ast + "Prune a variable/pending-assignment snapshot down to what the frontend actually + uses (VariableAst in client.ts). Critical: a raw snapshot still carries :variables + and :references from pre-handle/post-handle — left unpruned, each additional + chained |= block would re-embed every earlier block's full snapshot inside the new + one, growing the response payload superlinearly instead of linearly. Its own + :tables/:selected-tables entries need the same per-table pruning (prune-table) for + a variable-of-variable chain, or the same recursive embedding reappears one level + down." + [var-ast] + (-> (select-keys var-ast [:tables :selected-tables :joins :columns]) + (update :tables #(mapv prune-table %)) + (update :selected-tables #(mapv prune-table %)))) + +(defn- prune-ast + "Prune a generated state down to the :ast value returned to the frontend." + [state] + (-> (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges :assign]) + (update :selected-tables #(mapv prune-table %)) + (assoc :variables + (into {} (for [[k v] (:variables state)] [k (prune-var-ast v)]))) + (assoc :pending-assignments + (into {} (for [[k v] (:pending-assignments state)] [k (prune-var-ast v)]))))) + (defn api-build - ([expression] - (api-build expression nil nil)) - ([expression cursor] - (api-build expression cursor nil)) - ([expression cursor connection-id] + ([expressions] + (api-build expressions nil nil)) + ([expressions cursor] + (api-build expressions cursor nil)) + ([expressions cursor connection-id] (let [conn-id (or connection-id @db/connection-id) connection-name (connections/get-connection-name conn-id)] (try - (let [result (generate-state expression cursor conn-id) - {state :result error :error} result] - (if error result - {:connection-id connection-name - :version version - :query (-> expression trim-pipes (generate-state nil conn-id) :result eval/build-query eval/formatted-query) - :ast (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges])})) + (let [exprs (if (string? expressions) [expressions] expressions) + context-exprs (butlast exprs) + last-expr (last exprs)] + (if (str/blank? last-expr) + {:connection-id connection-name} + (let [{:keys [variables error]} (evaluate-expressions context-exprs conn-id)] + (if error + {:connection-id connection-name :error error} + (let [result (generate-state last-expr cursor conn-id variables) + {state :result build-error :error} result] + (if build-error + {:connection-id connection-name :error build-error} + {:connection-id connection-name + :version version + :query (-> last-expr trim-pipes (generate-state nil conn-id variables) :result eval/build-query eval/formatted-query) + :ast (prune-ast state)})))))) (catch Exception e {:connection-id connection-name :error (.getMessage e)}))))) @@ -82,28 +152,38 @@ [alias]))))) (defn api-eval - ([expression] - (api-eval expression nil)) - ([expression connection-id] + ([expressions] + (api-eval expressions nil)) + ([expressions connection-id] (let [conn-id (or connection-id @db/connection-id) connection-name (connections/get-connection-name conn-id)] (try - (let [result (generate-state expression nil conn-id) - {state :result error :error} result] - (if error result - (let [rows (eval/run-query state) - op-type (get-in state [:operation :type]) - ;; For action results we control the format; columns come from header row - columns (if (contains? #{:update-action :delete-action} op-type) - (get-columns rows) - (get-columns state rows))] - {:connection-id connection-name - :version version - ;; :time (db/run-query (state :connection-id) {:query "SELECT NOW() as now, NOW() AT TIME ZONE 'UTC' AS utc;"}) - ;; :server_time (str (java.time.Instant/now)) - :result rows - :columns columns}))) - + (let [exprs (if (string? expressions) [expressions] expressions) + context-exprs (butlast exprs) + last-expr (last exprs) + trimmed (trim-pipes (or last-expr ""))] + (if (str/blank? trimmed) + {:connection-id connection-name} + (let [{:keys [variables error]} (evaluate-expressions context-exprs conn-id)] + (if error + {:connection-id connection-name :error error} + (let [{last-state :result build-error :error} (generate-state trimmed nil conn-id variables)] + (if build-error + {:connection-id connection-name :error build-error} + (let [query (-> last-state eval/build-query eval/formatted-query)] + (try + (let [rows (eval/run-query last-state) + op-type (get-in last-state [:operation :type]) + columns (if (contains? #{:update-action :delete-action} op-type) + (get-columns rows) + (get-columns last-state rows))] + {:connection-id connection-name + :version version + :result rows + :columns columns}) + (catch Exception e {:connection-id connection-name + :error (.getMessage e) + :query query}))))))))) (catch Exception e {:connection-id connection-name :error (.getMessage e)}))))) @@ -190,11 +270,13 @@ ;; query building and evaluation (POST "/api/v1/build" {params :params} - (let [{:keys [expression cursor connection-id]} params] - (->> (api-build expression cursor connection-id) response))) + (let [{:keys [expressions expression cursor connection-id]} params + exprs (or expressions (when expression [expression]))] + (->> (api-build exprs cursor connection-id) response))) (POST "/api/v1/eval" {params :params} - (let [{:keys [expression connection-id]} params] - (->> (api-eval (trim-pipes expression) connection-id) response))) + (let [{:keys [expressions expression connection-id]} params + exprs (or expressions (when expression [expression]))] + (->> (api-eval exprs connection-id) response))) ;; raw SQL execution (POST "/api/v1/sql" {params :params} @@ -206,7 +288,7 @@ ;; pine-mode.el (POST "/api/v1/build-with-params" {params :params} (let [{:keys [expression connection-id]} params] - (->> (api-build (trim-pipes expression) nil connection-id) :query response))) + (->> (api-build [(trim-pipes expression)] nil connection-id) :query response))) ;; default case (route/not-found "Not Found")) (def app diff --git a/src/pine/ast/assign.clj b/src/pine/ast/assign.clj new file mode 100644 index 0000000..97c5c1e --- /dev/null +++ b/src/pine/ast/assign.clj @@ -0,0 +1,23 @@ +(ns pine.ast.assign + "Records a named checkpoint in the pipeline so the current expression-state + can be referenced as a CTE in later expressions. + + Why assignment is a pipe op (not a terminal): keeping '|= name' as a regular + operation preserves composability. 'company |= x | w: active = true' labels + company as 'x' (unfiltered) while the pipeline continues to filter. Subsequent + expressions that use 'x' get the unfiltered company CTE; the current expression + still produces the filtered SELECT. + + Why a snapshot: storing a dissoc'd copy of state (not the live state) prevents + the pending-assignments map from accumulating nested copies across multiple + assignments in a single expression.") + +(defn handle + "Snapshot the current state under varname in :pending-assignments. + The snapshot excludes :pending-assignments itself to prevent nesting. + Also registers varname as a local alias for the current SQL alias so + subsequent ops in the same expression can write e.g. x.id to mean c_0.id." + [state varname] + (-> state + (assoc-in [:pending-assignments varname] (dissoc state :pending-assignments)) + (assoc :assign varname))) diff --git a/src/pine/ast/group.clj b/src/pine/ast/group.clj index e779424..358fb2e 100644 --- a/src/pine/ast/group.clj +++ b/src/pine/ast/group.clj @@ -5,6 +5,8 @@ (defn handle [state value] (let [i (state :index) current (state :current) + ;; A live alias (e.g. re-bound via `as`) always wins over a stale |= snapshot + resolve-alias #(if (contains? (:aliases state) %) % (or (get-in state [:pending-assignments % :current]) %)) ;; Get existing columns from state (e.g., from previous select with date extraction) existing-columns (state :columns) ;; Filter out auto-id columns from existing columns @@ -34,15 +36,18 @@ ;; Use the existing column (preserves original alias like "t") (take 1 matching) ;; No match, use the group column with default alias - [(assoc g-col :alias (or (:alias g-col) current))]))) + [(assoc g-col :alias (resolve-alias (or (:alias g-col) current)))]))) raw-group-columns) ;; No existing selected columns, use group columns with default alias - (map #(assoc % :alias (or (:alias %) current)) raw-group-columns)) + (map #(assoc % :alias (resolve-alias (or (:alias %) current))) raw-group-columns)) ;; SELECT clause: merged columns + aggregate functions - select-columns (concat merged-columns fn-columns)] + ;; Must be a vector: :columns is later grown with (update :columns into ...) + ;; elsewhere (e.g. select.clj), and `into`/`conj` prepend onto a plain seq + ;; (concat's return type) instead of appending, silently reversing order. + select-columns (vec (concat merged-columns fn-columns))] (-> state (assoc :columns select-columns) ;; GROUP BY uses all the merged columns - (assoc :group merged-columns)))) + (assoc :group (vec merged-columns))))) diff --git a/src/pine/ast/hints.clj b/src/pine/ast/hints.clj index 248376e..de89b92 100644 --- a/src/pine/ast/hints.clj +++ b/src/pine/ast/hints.clj @@ -14,12 +14,15 @@ ;; --------------------------------------------------------------------------- (defn create-hint-from-table [state tables] - (let [refs (-> state :references :table)] + (let [refs (-> state :references :table) + variables (-> state :variables)] (mapcat identity (for [table tables :let [schemas (->> table refs :in keys)]] - (for [schema schemas] - {:schema schema :table table}))))) + (if (contains? variables table) + [{:schema nil :table table}] + (for [schema schemas] + {:schema schema :table table})))))) (defn table-hints "No context - get all the tables matching the token" @@ -41,10 +44,11 @@ ;; ["z" "document" "created_by" :refers-to "y" "employee" "id"] (defn- create-hint-from-relation-array [table via-details] (map (fn [vd] - (let [direction (nth vd 3) - parent? (= direction :refers-to) - schema (nth vd 4) - column (nth vd (if parent? 2 6)) + (let [direction (nth vd 3) + parent? (= direction :refers-to) + schema (nth vd 4) + variable? (= (last vd) :variable-join) + column (when-not variable? (nth vd (if parent? 2 6))) heuristic? (= (last vd) :heuristic)] {:schema schema :table table @@ -64,14 +68,17 @@ (defn relation-hints [state token] (let [from-alias (state :context) from-table (-> state :aliases (get from-alias) :table) - parents (-> state :references :table (get from-table) :refers-to) - children (-> state :references :table (get from-table) :referred-by) + variables (-> state :variables) + parents (-> state :references :table (get from-table) :refers-to) + children (-> state :references :table (get from-table) :referred-by) suggestions (filter-relations token (concat parents children))] - (-> suggestions - create-hint-from-relations - ;; TODO: instead of doing a distinct, we can do a reduce and keep track - ;; of duplicates - distinct))) + (->> suggestions + create-hint-from-relations + (map (fn [h] + (if (contains? variables (:table h)) + (assoc h :schema nil) + h))) + distinct))) (defn generate-table-hints [state] (let [{token :table parent :parent} (-> state :tables reverse first) @@ -126,20 +133,22 @@ ;; - Order partial: simple exclude-already-selected logic works with generic generate-column-hints ;; - Where partial: complex state-dependent filtering based on partial completion: ;; * `where:` → show all columns - ;; * `w: i` → filter to columns matching "i" + ;; * `w: i` → filter to columns matching "i" + ;; * `w: e.i` → filter to columns matching "i" from alias "e" ;; * `w: id =` → show all columns (ready for next condition) (let [operation (state :operation) where-data (get operation :value {}) partial-condition (:partial-condition where-data) - current-alias (state :current)] + current-alias (state :current) + lookup-alias (or (and partial-condition (:alias partial-condition)) current-alias)] (if (nil? partial-condition) ;; No partial condition yet, show all columns (generate-all-column-hints state current-alias) ;; Has partial condition, filter based on it (if (and (:column partial-condition) (not (contains? partial-condition :operator))) - ;; Just a column specified, filter hints for that column + ;; Just a column specified — use the condition's alias if present, else current (find-relevant-columns - (generate-all-column-hints state current-alias) + (generate-all-column-hints state lookup-alias) partial-condition) ;; Column + operator, show all columns for next condition (generate-all-column-hints state current-alias))))) @@ -148,7 +157,8 @@ (let [assignments (get-in state [:update :assignments] []) partial-column (get-in state [:update :partial-column]) assigned-columns (map (fn [a] {:column (get-in a [:column :column])}) assignments) - hints (-> (generate-all-column-hints state (state :current)) + lookup-alias (or (and partial-column (:alias partial-column)) (state :current)) + hints (-> (generate-all-column-hints state lookup-alias) (exclude-columns assigned-columns))] (if partial-column (find-relevant-columns hints partial-column) @@ -156,18 +166,23 @@ (defn generate-column-hints [state columns] ;; Generic column hints logic that works for most operations: - ;; - Complete operations (select, order, where): filter hints to match current column being typed - ;; - Partial operations (select-partial, order-partial): exclude already-selected columns - ;; - Note: where-partial needs custom logic (see generate-where-hints) due to complex state-dependent filtering + ;; - Complete operations (select, order): filter hints to match the column being typed, using its alias + ;; - Partial operations (select-partial, order-partial): exclude already-selected columns, + ;; defaulting to the current context table so the next slot shows relevant columns + ;; - Note: where-partial needs custom logic (see generate-where-hints) (let [column (some-> columns reverse first) - ;; Use operation-index to determine table context for hints - ;; If column was added in a later operation than current context, use that column's alias - a (if (and (seq column) - (> (column :operation-index) (state :current-index))) - (column :alias) - (state :current)) - hints (generate-all-column-hints state a) - type (-> state :operation :type)] + type (-> state :operation :type) + ;; For partial ops, always use the current context table — the user hasn't started typing + ;; the next column yet, so the last completed column's alias is irrelevant. + ;; For complete ops, use the typed column's alias when it was added in the current scope. + a (if (or (= type :select-partial) (= type :order-partial)) + (or (get-in state [:operation :partial-alias :alias]) (state :current)) + (if (and (seq column) + (column :operation-index) + (> (column :operation-index) (state :current-index))) + (column :alias) + (state :current))) + hints (generate-all-column-hints state a)] (cond ;; If the type is :select, :order, or :where and columns exist, filter hints using the columns (and (or (= type :select) (= type :order) (= type :where)) column) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 3e44933..dbc30a6 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -1,6 +1,19 @@ (ns pine.ast.main + "Folds a parsed operation list into a state map that fully describes the query. + + Why a state map: rather than building SQL strings incrementally, each operation + updates a data structure (tables, columns, joins, where clauses, etc.). This + separates intent (what the query means) from rendering (how to express it in SQL), + which makes hints, prettification, and multiple output formats possible without + re-parsing. + + Why variables are seeded before handle-ops: joins are resolved using a references + map built from the DB schema. Variables (CTEs from earlier expressions) need to + appear in that map — with the same FK relationships as their source tables — so + join resolution treats them like real tables." (:require [clojure.string :as str] + [pine.ast.assign :as assign] [pine.ast.count :as pine-count] [pine.ast.delete-action :as delete-action] [pine.ast.from :as from] @@ -21,6 +34,9 @@ ;; - connection :connection-id nil :references {} + :variables {} ;; {"varname" }, populated from |= assignments in prior expressions + :assign nil ;; variable name from the last |= op in this expression + :pending-assignments {} ;; {"varname" } accumulated by |= ops in this expression :expression nil ;; Expression string for cursor-aware hints :cursor nil ;; Cursor position {:line N :character M} (zero-indexed) @@ -64,15 +80,284 @@ ;; POST ;; --- ;; - hints - :hints {:table [] :select [] :order [] :where [] :update []}}) + :hints {:table [] :select [] :order [] :where [] :update []} -(defn pre-handle [state connection-id ops-count expression cursor] - (-> state - (assoc :references (db/init-references connection-id)) - (assoc :connection-id connection-id) - (assoc :pending-count ops-count) - (assoc :expression expression) - (assoc :cursor cursor))) + ;; --- + ;; Checkpoint + ;; --- + ;; Tracks auto-generated CTE names (__pine_0__, __pine_1__, ...) + :auto-cte-count 0 + ;; Set after a checkpoint op (group/limit) to signal the next table op + ;; should become a CTE instead of a direct join. + ;; nil | {:needs-assign true} | {:name "n" :needs-table true} + :pending-checkpoint nil}) + +(defn- variable-output-columns + "Return the column list a variable's CTE actually exposes, for hint generation. + Returns nil when the CTE selects *, meaning the source table's columns apply. + + For a GROUP-sourced CTE, :columns already includes the aggregate entry — + group.clj folds it in directly (parser.clj defaults :functions to [\"count\"] + even when `=> count` is omitted; there's no way to write a truly aggregate- + less GROUP) — so it's read here like any other column." + [var-ast] + (let [user-cols (remove :auto-id (:columns var-ast))] + (when (seq user-cols) + (->> user-cols + (map #(or (:column-alias %) (:column %))) + distinct + (mapv #(hash-map :column %)))))) + +(defn- get-source-tables + "Return the tables that are valid join sources for a variable's CTE. + + Once a table's data is sealed into a CTE, the outer query can only see what + that CTE's own :columns actually selected — nothing else about the underlying + table is reachable. A table is therefore only a join source if its own id + column is among those columns. Pine never adds one on its own: if the user + didn't select it, it isn't there, and that table simply isn't joinable + through this variable. + + No explicit columns at all means the CTE selects '*', which includes every + column of the source table, id included — the :current table is then the + sole source. With explicit columns, each one is checked for its own id column + independently, so this can return zero tables (`group: name` alone has no id + anywhere), one, or more than one (`s: t.id, c.id` makes both t and c valid + sources) — the same multi-table join support a plain, variable-free pipeline + already has when it references more than one table. + + Used by both seeding and bidirectional patching." + [var-ast] + (let [columns (:columns var-ast) + aliases (:aliases var-ast) + explicit (remove :auto-id columns)] + (or (if (empty? explicit) + (when-let [current-alias (:current var-ast)] + [(get-in aliases [current-alias :table])]) + (->> columns + (filter #(= "id" (:column %))) + (map :alias) + (map #(get-in aliases [% :table])) + (remove nil?) + distinct)) + []))) + +(defn- seed-variable-references + "Copy reference entries from the real source tables of a variable into the + local references map under the variable name, so join resolution treats the + variable identically to a real table. Column hints are overridden with the + CTE's actual output columns when they can be determined." + [refs var-ast varname] + (let [source-tables (get-source-tables var-ast) + seeded (reduce (fn [r source-table] + (let [source-refs (get-in r [:table source-table])] + (if source-refs + (update-in r [:table varname] merge source-refs) + r))) + refs + source-tables)] + (if-let [output-cols (variable-output-columns var-ast)] + (-> seeded + (assoc-in [:table varname :columns] output-cols) + (assoc-in [:table varname :column-set] (set output-cols))) + seeded))) + +(defn- build-direction-index + "Reverse index: source-table -> {entity-name -> existing-relation}, for one + direction. Lets patch-direction look up 'who already relates to S' directly + instead of scanning every table in the schema per variable." + [refs direction] + (reduce-kv (fn [idx entity-name entity-refs] + (reduce-kv (fn [idx target existing] + (assoc-in idx [target entity-name] existing)) + idx + (get entity-refs direction))) + {} + (get refs :table {}))) + +(defn- patch-direction + "For each variable V wrapping source S, copy T[direction][S] → T[direction][V] + for every entity T that already relates to S. Used by patch-variable-relations. + + Uses a reverse index (source-table -> referring entities) built once instead + of scanning every schema table per variable, which turns this from O(v · T) + into O(T + v · k) where k is the actual fan-in for each source table. The + index is kept in sync with entries this pass adds, since a later variable's + source table can itself be an earlier variable processed in this same pass + (variable-of-variable composition)." + [refs variables direction] + (:refs + (reduce + (fn [{:keys [refs index]} [varname var-ast]] + (reduce + (fn [{:keys [refs index]} source-table] + (reduce + (fn [{:keys [refs index]} [entity-name existing]] + (let [refs* (update-in refs [:table entity-name direction varname] merge existing) + merged (get-in refs* [:table entity-name direction varname])] + {:refs refs* :index (assoc-in index [varname entity-name] merged)})) + {:refs refs :index index} + (get index source-table))) + {:refs refs :index index} + (get-source-tables var-ast))) + {:refs refs :index (build-direction-index refs direction)} + variables))) + +(defn- patch-variable-relations + "Bidirectional pass: for each variable V wrapping source S, find every + entity T where T already knows about S via :referred-by or :refers-to, + and register V there too. + + This enables: + - 'T | V' and 'V | T' (real table ↔ variable) + - 'V | W' and 'W | V' (variable ↔ variable) + + Must run after all variables have been seeded." + [refs variables] + (-> refs + (patch-direction variables :referred-by) + (patch-direction variables :refers-to))) + +(defn- patch-same-source-variable-joins + "For each ordered pair of distinct variables (v1, v2) that share a source table + with an 'id' column, register a synthetic id=id join at refs[:table v1 :referred-by v2]. + Only adds entries where no join path already exists, so self-referential FK propagation + (e.g. employee/reports_to) is preserved. + + Pairs are generated only within groups of variables that actually share a source + table (via an inverted source-table -> variables index), instead of scanning every + ordered pair among all variables. This turns the common case (variables spread + across distinct source tables) from O(v²) into roughly O(v), while the worst case + (all variables sharing one source) stays O(g²) for that group — unavoidable since + the output itself is a g² set of pairwise joins." + [refs variables] + (let [var-sources (->> variables + (map (fn [[vname var-ast]] + [vname (set (get-source-tables var-ast))])) + (into {})) + by-source (reduce-kv (fn [idx vname sources] + (reduce (fn [idx source] (update idx source (fnil conj []) vname)) + idx + sources)) + {} + var-sources) + pairs (distinct (for [[_ vnames] by-source + v1 vnames v2 vnames + :when (not= v1 v2)] + [v1 v2]))] + (reduce (fn [r [v1 v2]] + (let [shared-sources (filter (var-sources v1) (var-sources v2)) + has-id? (fn [tbl] (some #(= "id" (:column %)) (get-in r [:table tbl :columns])))] + (if (and (some has-id? shared-sources) + (not (get-in r [:table v1 :referred-by v2]))) + (update-in r [:table v1 :referred-by v2 :via "id"] + (fnil conj []) + [nil v1 "id" :referred-by nil v2 "id" :variable-join]) + r))) + refs + pairs))) + +(defn pre-handle [state connection-id ops-count expression cursor variables] + (let [refs (db/init-references connection-id) + seeded-refs (reduce (fn [r [varname var-ast]] + (seed-variable-references r var-ast varname)) + refs + variables) + aug-refs (-> seeded-refs + (patch-variable-relations variables) + (patch-same-source-variable-joins variables))] + (-> state + (assoc :references aug-refs) + (assoc :connection-id connection-id) + (assoc :pending-count ops-count) + (assoc :expression expression) + (assoc :cursor cursor) + (assoc :variables variables)))) + +;; --------------------------------------------------------------------------- +;; Checkpoint helpers +;; --------------------------------------------------------------------------- + +(declare handle-op) + +(def ^:private checkpoint-op-types #{:group :limit}) + +;; Ops that consume/query the checkpoint's result rather than composing another +;; table join onto it. Fired on regardless of partial-vs-complete: an -partial op +;; (e.g. order-partial from a dangling trailing comma) already carries whatever was +;; fully typed before the comma, so it needs the same sealed scope a complete op +;; would. count/delete/update are deliberately excluded — they build their own +;; wrapper query generically (see build-count-query) and don't need the checkpoint's +;; group-shaped state separated into a CTE first. +(def ^:private checkpoint-consuming-op-types + #{:select :select-partial :where :where-partial :order :order-partial}) + +(defn- reset-for-cte [state] + (assoc state + :tables [] :columns [] :limit nil :joins [] + :where [] :order [] :group [] :update nil + :current nil :context nil :current-index 0 + :table-count 0 :operation {:type nil :value nil})) + +(defn- seal-as-cte + "Store snapshot under cname, seed its join references, reset the query-building + state, then inject cname as the first table so subsequent ops compose on top of it." + [state cname snapshot] + (let [new-refs (-> (:references state) + (seed-variable-references snapshot cname) + (patch-variable-relations {cname snapshot}) + (patch-same-source-variable-joins {cname snapshot}))] + (-> state + (assoc :references new-refs) + (assoc-in [:pending-assignments cname] snapshot) + reset-for-cte + (handle-op {:type :table :value {:table cname}})))) + +(defn- flush-checkpoint + "State-machine step: called at the start of each handle-ops iteration. + Converts a pending checkpoint into a CTE when the right op type is seen. + + Fires when the incoming op is a table (join composition), another checkpoint + op (e.g. LIMIT after GROUP), or an op that queries the checkpoint's result + (select/where/order, complete or partial). Does not fire for count/delete/update + since those have their own query-building paths that do not need CTE separation." + [state op] + (let [checkpoint (:pending-checkpoint state) + op-type (:type op) + fire? (or (= op-type :table) + (contains? checkpoint-op-types op-type) + (contains? checkpoint-consuming-op-types op-type))] + (cond + (nil? checkpoint) + state + + ;; Explicit assign after a checkpoint op: record the user name, wait + (and (:needs-assign checkpoint) (= op-type :assign)) + (assoc state :pending-checkpoint {:name (:value op) :needs-table true}) + + ;; Auto-named: fire when a table or another checkpoint op arrives + (and (:needs-assign checkpoint) fire?) + (let [n (:auto-cte-count state) + cname (str "__pine_" n "__") + snapshot (dissoc state :pending-assignments)] + (-> state + (update :auto-cte-count inc) + (assoc :pending-checkpoint nil) + (seal-as-cte cname snapshot))) + + ;; Waiting for assign, non-triggering op (count, where, etc.) — hold + (:needs-assign checkpoint) + state + + ;; User-named: fire when a table or another checkpoint op arrives + (and (:needs-table checkpoint) fire?) + (let [cname (:name checkpoint) + snapshot (get-in state [:pending-assignments cname])] + (-> state + (assoc :pending-checkpoint nil) + (seal-as-cte cname snapshot))) + + :else state))) (defn handle-op [state {:keys [type value]}] (case type @@ -90,19 +375,24 @@ :delete-action (delete-action/handle state value) :update-action (update-action/handle state value) :update-partial (update-action/handle state value) + :assign (assign/handle state value) ;; No operations :no-op state (update state :errors conj [type "Unknown operation type in parse tree"]))) (defn handle-ops [state ops] (reduce (fn [s [i o]] - (-> s - (assoc :index i) - (handle-op o) ; Pass the index and operation - (update :pending-count dec) - (assoc :operation o))) + ;; flush-checkpoint runs before handle-op; it may reset state and inject + ;; a synthetic CTE table — the :index must be set first so the injected + ;; table gets the correct operation index. + (let [s (-> s (assoc :index i) (flush-checkpoint o))] + (cond-> (-> s + (handle-op o) + (update :pending-count dec)) + (not= (:type o) :assign) (assoc :operation o) + (checkpoint-op-types (:type o)) (assoc :pending-checkpoint {:needs-assign true})))) state - (map-indexed vector ops))) ; Pair each operation with its index + (map-indexed vector ops))) (declare generate) @@ -121,9 +411,9 @@ (str/join "\n" (concat lines-before [truncated-current]))))))) (defn- generate-truncated-state - "Generate state for truncated expression at cursor position. + "Generate state for truncated expression at cursor position. Keep references for hint generation." - [expression cursor connection-id] + [expression cursor connection-id variables] (let [truncated-expr (truncate-at-cursor expression cursor) {:keys [result error]} (parser/parse truncated-expr)] (if (or error (nil? result)) @@ -132,7 +422,7 @@ ;; Successfully parsed, build state without going through post-handle ;; to preserve references for hint generation (-> state - (pre-handle connection-id (count result) nil nil) + (pre-handle connection-id (count result) nil nil variables) (handle-ops result))))) (defn- offset->position @@ -192,14 +482,16 @@ (defn generate ([parse-tree] - (generate parse-tree @db/connection-id nil nil)) + (generate parse-tree @db/connection-id nil nil {})) ([parse-tree connection-id] - (generate parse-tree connection-id nil nil)) + (generate parse-tree connection-id nil nil {})) ([parse-tree connection-id expression cursor] + (generate parse-tree connection-id expression cursor {})) + ([parse-tree connection-id expression cursor variables] (let [full-state (-> state - (pre-handle connection-id (count parse-tree) expression cursor) + (pre-handle connection-id (count parse-tree) expression cursor variables) (handle-ops parse-tree)) truncated-state (when (and cursor expression) - (generate-truncated-state expression cursor connection-id))] + (generate-truncated-state expression cursor connection-id variables))] (post-handle full-state truncated-state)))) diff --git a/src/pine/ast/order.clj b/src/pine/ast/order.clj index 4e2da56..9911e78 100644 --- a/src/pine/ast/order.clj +++ b/src/pine/ast/order.clj @@ -1,10 +1,12 @@ (ns pine.ast.order) (defn handle [state value] - (let [i (state :index) - current (state :current) + (let [i (state :index) + current (state :current) + ;; A live alias (e.g. re-bound via `as`) always wins over a stale |= snapshot + resolve-alias #(if (contains? (:aliases state) %) % (or (get-in state [:pending-assignments % :current]) %)) columns (map #(-> %1 - (assoc :alias (or (:alias %1) current)) + (assoc :alias (resolve-alias (or (:alias %1) current))) (assoc :operation-index i)) value)] (-> state diff --git a/src/pine/ast/select.clj b/src/pine/ast/select.clj index abf7c77..d8bc045 100644 --- a/src/pine/ast/select.clj +++ b/src/pine/ast/select.clj @@ -4,9 +4,11 @@ (let [i (state :index) current (state :current) ;; Process each column and handle date functions + ;; A live alias (e.g. re-bound via `as`) always wins over a stale |= snapshot + resolve-alias #(if (contains? (:aliases state) %) % (or (get-in state [:pending-assignments % :current]) %)) columns (mapcat (fn [col] (let [col-with-defaults (-> col - (assoc :alias (or (:alias col) current)) + (assoc :alias (resolve-alias (or (:alias col) current))) (assoc :operation-index i))] (if-let [col-fn (:column-function col)] ;; Column function: apply function to column @@ -56,8 +58,11 @@ next-operation-index (inc (:index state)) references (:references state) aliases (:aliases state) - ;; Only create auto-ID columns for tables that actually have an 'id' column - valid-aliases (filter #(has-id-column? references aliases %) table-aliases) + variables (:variables state) + ;; Only create auto-ID columns for real tables (not variables/CTEs) that have an 'id' column + valid-aliases (filter #(and (not (:ast (get aliases %))) + (has-id-column? references aliases %)) + table-aliases) auto-id-columns (map-indexed #(create-auto-id-column %2 (+ next-operation-index %1)) valid-aliases)] (update state :columns into auto-id-columns)) state)) diff --git a/src/pine/ast/table.clj b/src/pine/ast/table.clj index 7decb7e..fa2eeaa 100644 --- a/src/pine/ast/table.clj +++ b/src/pine/ast/table.clj @@ -56,8 +56,7 @@ initials (map #(subs % 0 1) words)] (apply str initials))) -;; todo: spec for the :value for a :table -(defn handle [state value] +(defn- handle-as-table [state value] (let [index (state :index) {:keys [table alias schema parent join-column join-left-column join-right-column join]} value a (or alias (str (make-alias table) "_" (state :table-count))) @@ -66,18 +65,37 @@ :join-right-column join-right-column :join join :index index}] (-> state - ;; pre (assoc :context (state :current)) (assoc :current a) (assoc :current-index index) - ;; (update :tables conj current) (update :aliases assoc a current) (update-joins current) - ;; post - ;; - ;; TODO: These are metadata - mayebe I should move them to a different - ;; ns e.g. if the operation is table, then I update the following there - ;; (update :table-count inc)))) +(defn- handle-as-variable [state value var-ast] + (let [index (state :index) + {:keys [table alias join-column join-left-column join-right-column join]} value + a (or alias table) + current {:schema nil :table table :ast var-ast :alias a + :join-column join-column :join-left-column join-left-column + :join-right-column join-right-column :join join + :index index}] + (-> state + (assoc :context (state :current)) + (assoc :current a) + (assoc :current-index index) + (update :tables conj current) + (update :aliases assoc a current) + (update-joins current) + (update :table-count inc)))) + +;; todo: spec for the :value for a :table +(defn handle [state value] + (let [{:keys [table]} value + var-ast (or (get-in state [:variables table]) + (get-in state [:pending-assignments table]))] + (if var-ast + (handle-as-variable state value var-ast) + (handle-as-table state value)))) + diff --git a/src/pine/ast/where.clj b/src/pine/ast/where.clj index b75fa47..d4acae2 100644 --- a/src/pine/ast/where.clj +++ b/src/pine/ast/where.clj @@ -17,8 +17,10 @@ (defn handle [state [column operator value]] (let [a (state :current) + ;; A live alias (e.g. re-bound via `as`) always wins over a stale |= snapshot + resolve-alias #(if (contains? (:aliases state) %) % (or (get-in state [:pending-assignments % :current]) %)) [alias col cast] (:value column) - alias (or alias a) + alias (resolve-alias (or alias a)) converted-value (if (and (not= (:type value) :symbol) (not= (:type value) :column)) (convert-condition-value value alias col state) value)] @@ -27,11 +29,12 @@ (defn handle-partial [state {:keys [complete-conditions partial-condition]}] ;; For WHERE-PARTIAL, we only store the complete conditions in :where ;; The partial condition is used for hints, not for query generation - (let [a (state :current)] + (let [a (state :current) + resolve-alias #(if (contains? (:aliases state) %) % (or (get-in state [:pending-assignments % :current]) %))] (reduce (fn [s condition] (let [[column operator value] (:value condition) [alias col cast] (:value column) - alias (or alias a) + alias (resolve-alias (or alias a)) converted-value (if (and (not= (:type value) :symbol) (not= (:type value) :column)) (convert-condition-value value alias col s) value)] diff --git a/src/pine/db/connections.clj b/src/pine/db/connections.clj index 4241525..43c4ab0 100644 --- a/src/pine/db/connections.clj +++ b/src/pine/db/connections.clj @@ -26,6 +26,17 @@ (def pools "Database connection pools" (atom {})) +(def test-connection-id + "Sentinel connection id that bypasses real connection pools and live schema + lookups in favor of fixtures. Defined here — the db namespace nothing else + in pine.db depends on — so every place that needs to recognize it (schema + lookup in postgres.clj, connection-name lookup below) checks the one + predicate instead of each hardcoding :test separately." + :test) + +(defn test-connection? [id] + (= id test-connection-id)) + (defn get-connection-pool [id] (let [pool-or-fn (@pools id)] (if pool-or-fn @@ -49,7 +60,9 @@ (jdbc-url->label (.getJdbcUrl pool))) (defn get-connection-name [id] - (-> id get-connection-pool make-connection-id)) + (if (test-connection? id) + "test" + (-> id get-connection-pool make-connection-id))) (defn list-connections [] (mapv (fn [[id _]] diff --git a/src/pine/db/postgres.clj b/src/pine/db/postgres.clj index ddaaafa..4d92c7d 100644 --- a/src/pine/db/postgres.clj +++ b/src/pine/db/postgres.clj @@ -225,7 +225,7 @@ FROM information_schema.columns"] (defn get-indexed-references [id] (let [references (cond - (= id :test) fixtures/references + (connections/test-connection? id) fixtures/references :else (get-references-helper id))] (index-references references))) diff --git a/src/pine/eval.clj b/src/pine/eval.clj index 8285cb5..aae5a7f 100644 --- a/src/pine/eval.clj +++ b/src/pine/eval.clj @@ -90,7 +90,7 @@ (q alias column))) group))))) -(defn build-select-query [state] +(defn- build-bare-select [state] (let [{:keys [tables _columns limit where aliases]} state from (let [{a :alias} (first tables) {table :table schema :schema} (get aliases a)] @@ -123,6 +123,57 @@ {:query query :params params})) +(defn- build-cte-body + "Generate the inner SQL for a variable's AST used as a CTE. + When the current table has no explicit user columns, .* is added and already + includes id — so the auto-id column is dropped to avoid duplicate id names. + When explicit columns are present (no .*), the auto-id is kept but its alias + is stripped so id is accessible for join conditions. + Returns {:query ... :params ...}." + [ast] + (let [current-alias (:current ast) + user-columns (remove :auto-id (:columns ast)) + has-explicit? (some #(= (:alias %) current-alias) user-columns) + columns (keep (fn [col] + (if (and (:auto-id col) (= (:alias col) current-alias)) + (when has-explicit? (dissoc col :column-alias)) + col)) + (:columns ast))] + (build-bare-select (assoc ast :columns columns)))) + +(defn- collect-ctes + "Recursively collect [name query params] triples from variable tables in + topological order (deepest dependencies first). Deduplicates by name." + [tables aliases] + (->> tables + (mapcat (fn [{:keys [alias]}] + (let [entry (get aliases alias)] + (when-let [ast (:ast entry)] + (let [var-name (:table entry) + nested-ctes (collect-ctes (:tables ast) (:aliases ast)) + {:keys [query params]} (build-cte-body ast)] + (conj nested-ctes [var-name query params])))))) + (reduce (fn [[seen acc] [name _ _ :as cte]] + (if (contains? seen name) + [seen acc] + [(conj seen name) (conj acc cte)])) + [#{} []]) + second)) + +(defn build-select-query [state] + (let [ctes (collect-ctes (:tables state) (:aliases state)) + result (build-bare-select state) + cte-params (mapcat #(nth % 2 nil) ctes) + cte-prefix (when (seq ctes) + (str "WITH " + (s/join ", " (map (fn [[name body _]] + (str (q name) " AS ( " body " )")) + ctes)) + " "))] + (-> result + (update :query #(str cte-prefix %)) + (update :params #(seq (concat cte-params %)))))) + (defn build-count-query [state] (let [{:keys [query params]} (build-select-query state)] {:query (str "WITH x AS ( " query " ) SELECT COUNT(*) FROM x") @@ -259,7 +310,9 @@ (defn build-query [state] (let [{:keys [type]} (state :operation)] (cond - (= (-> state :current) "x_0") {:query "" :params nil} + (let [cur (-> state :current)] + (or (nil? cur) + (= "" (get-in state [:aliases cur :table])))) {:query "" :params nil} (= type :delete-action) (build-delete-query state) (= type :update-action) {:queries (build-update-queries state)} (= type :update-partial) {:queries (build-update-queries state)} diff --git a/src/pine/parser.clj b/src/pine/parser.clj index bc76da8..a8d9883 100644 --- a/src/pine/parser.clj +++ b/src/pine/parser.clj @@ -1,6 +1,14 @@ (ns pine.parser - "The parser is responsible for generating a parse tree from the bnf and - normalize the output which is used for the input for generating the ast" + "Turns raw Pine text into a normalized operation list. + + Why this layer exists: the BNF grammar produces a raw Instaparse tree + that's verbose and shape-inconsistent across operation types. Normalizing + here gives the rest of the system a single, predictable contract: + a vector of {:type :value } maps, one per pipe segment. + + Assignment (|= name) is now a regular pipe operation of type :assign. + It appears in the operation list like any other op, so the pipeline + can continue after it: 'company |= x | w: active = true' is valid." (:require [clojure.core.match :refer [match]] [clojure.string :as s] @@ -79,6 +87,12 @@ [:aliased-column [:column [:alias [:symbol a]] [:symbol c] [:column-function fn]] [:alias [:symbol ca]]] {:alias a :column c :column-function fn :column-alias ca} :else (throw (ex-info "Unknown COLUMN operation" {:_ column})))) +(defn- parse-partial-alias [partial-alias] + (match partial-alias + [:partial-alias [:alias [:symbol a]]] + {:alias a :column ""} + :else (throw (ex-info "Unknown partial alias" {:_ partial-alias})))) + (defn normalize-select [payload type] (match payload [:aliased-columns & columns] {:type type :value (mapv -normalize-column columns)} @@ -87,10 +101,22 @@ (defmethod -normalize-op :SELECT [[_ payload]] (normalize-select payload :select)) -(defmethod -normalize-op :SELECT-PARTIAL [[_ payload]] - (if (empty? payload) - {:type :select-partial, :value []} - (normalize-select payload :select-partial))) +(defmethod -normalize-op :SELECT-PARTIAL [operation] + (let [children (vec (rest operation)) + partial-alias-node (first (filter #(= (first %) :partial-alias) children)) + aliased-columns-node (first (filter #(= (first %) :aliased-columns) children))] + (cond + (nil? aliased-columns-node) + (if partial-alias-node + {:type :select-partial :value [] :partial-alias (parse-partial-alias partial-alias-node)} + {:type :select-partial :value []}) + + partial-alias-node + (assoc (normalize-select aliased-columns-node :select-partial) + :partial-alias (parse-partial-alias partial-alias-node)) + + :else + (normalize-select aliased-columns-node :select-partial)))) ;; ------ ;; ORDER @@ -117,10 +143,22 @@ (defmethod -normalize-op :ORDER [[_ payload]] (-normalize-order payload :order)) -(defmethod -normalize-op :ORDER-PARTIAL [[_ payload]] - (if (empty? payload) - {:type :order-partial, :value []} - (-normalize-order payload :order-partial))) +(defmethod -normalize-op :ORDER-PARTIAL [operation] + (let [children (vec (rest operation)) + partial-alias-node (first (filter #(= (first %) :partial-alias) children)) + order-columns-node (first (filter #(= (first %) :order-columns) children))] + (cond + (nil? order-columns-node) + (if partial-alias-node + {:type :order-partial :value [] :partial-alias (parse-partial-alias partial-alias-node)} + {:type :order-partial :value []}) + + partial-alias-node + (assoc (-normalize-order order-columns-node :order-partial) + :partial-alias (parse-partial-alias partial-alias-node)) + + :else + (-normalize-order order-columns-node :order-partial)))) ;; ----- ;; WHERE @@ -296,6 +334,10 @@ (defn- parse-partial-condition [partial-condition] (match partial-condition + ;; Alias-dot only (e.g. "e.") + [:partial-condition [:partial-alias [:alias [:symbol a]]]] + {:alias a :column ""} + ;; Just a column [:partial-condition [:column [:symbol column]]] {:column column} @@ -429,6 +471,10 @@ [] {:type :update-partial :value {:assignments [] :partial-column nil}} + [[:partial-update-column [:partial-alias [:alias [:symbol a]]]]] + {:type :update-partial :value {:assignments [] + :partial-column {:alias a :column ""}}} + [[:partial-update-column column-pattern]] {:type :update-partial :value {:assignments [] :partial-column (extract-column-info column-pattern)}} @@ -437,12 +483,23 @@ {:type :update-partial :value {:assignments (mapv parse-update-assignment assignments) :partial-column nil}} + [[:update-assignments & assignments] [:partial-update-column [:partial-alias [:alias [:symbol a]]]]] + {:type :update-partial :value {:assignments (mapv parse-update-assignment assignments) + :partial-column {:alias a :column ""}}} + [[:update-assignments & assignments] [:partial-update-column column-pattern]] {:type :update-partial :value {:assignments (mapv parse-update-assignment assignments) :partial-column (extract-column-info column-pattern)}} :else (throw (ex-info "Unknown UPDATE-PARTIAL operation" {:_ operation})))) +;; ----- +;; ASSIGN +;; ----- + +(defmethod -normalize-op :ASSIGN [[_ [_ varname]]] + {:type :assign :value varname}) + ;; ----- ;; NO-OP ;; ----- @@ -458,8 +515,9 @@ grammar (slurp file)] (insta/parser grammar))) -(defn- normalize-ops [[_ & ops]] - (mapv (fn [[_ op]] (-normalize-op op)) ops)) +(defn- normalize-ops [[_ & nodes]] + (mapv (fn [[_ op]] (-normalize-op op)) + (filter #(= (first %) :OPERATION) nodes))) (defn parse "Parse an expression and return the normalized operations or failure as a string" @@ -488,8 +546,9 @@ {:error (with-out-str (println (insta/get-failure result)))} (let [operations (rest result) op-infos (mapv (fn [op] - (let [[start end] (insta/span op)] - {:expression (s/trim (subs expression start end)) + (let [[start end] (insta/span op) + expr (s/trim (subs expression start end))] + {:expression expr :start start :end end})) operations)] diff --git a/src/pine/pine.bnf b/src/pine/pine.bnf index ef6e597..381fe53 100644 --- a/src/pine/pine.bnf +++ b/src/pine/pine.bnf @@ -1,11 +1,12 @@ OPERATIONS := OPERATION (<"|"> OPERATION )* +ASSIGN := <"="> symbol -OPERATION := SELECT-PARTIAL | SELECT | TABLE | WHERE-PARTIAL | WHERE | LIMIT| FROM | ORDER-PARTIAL | ORDER | COUNT | DELETE-ACTION | DELETE | UPDATE-PARTIAL | UPDATE-ACTION | GROUP +OPERATION := SELECT-PARTIAL | SELECT | WHERE-PARTIAL | WHERE | LIMIT | FROM | ORDER-PARTIAL | ORDER | COUNT | DELETE-ACTION | DELETE | UPDATE-PARTIAL | UPDATE-ACTION | GROUP | ASSIGN | TABLE TABLE := table table-mods -SELECT-PARTIAL := aliased-columns? <",">? +SELECT-PARTIAL := aliased-columns <","> partial-alias | partial-alias | aliased-columns? <",">? SELECT := aliased-columns WHERE-PARTIAL := conditions <","> partial-condition | partial-condition | @@ -13,7 +14,7 @@ WHERE := conditions | conditions LIMIT := number | number FROM := alias -ORDER-PARTIAL := order-columns? <",">? +ORDER-PARTIAL := order-columns <","> partial-alias | partial-alias | order-columns? <",">? ORDER := order-columns DELETE-ACTION := hint-column @@ -45,12 +46,13 @@ aggregate-function := "count" | "string_agg" | "sum" | "avg" | "min" | "max" update-assignments := update-assignment (<","> update-assignment)* update-assignment := column <"="> (column|string|number|boolean|null|date) -partial-update-column := column +partial-update-column := partial-alias | column conditions := condition (<","> condition)* condition := column (condition-default|condition-in) ( cast)? -partial-condition := column operator | column +partial-condition := partial-alias | column operator | column +partial-alias := alias <"."> := operator (column|string|number|boolean|null|date) := (in | not-in) <"("> strings* <")"> cast := <"::"> ("text"|"uuid") diff --git a/test/pine/api_test.clj b/test/pine/api_test.clj new file mode 100644 index 0000000..3c6c488 --- /dev/null +++ b/test/pine/api_test.clj @@ -0,0 +1,67 @@ +(ns pine.api-test + (:require [clojure.test :refer [deftest is testing]] + [pine.api :as api])) + +(defn- assert-clean-table [table] + (is (= #{:schema :table :alias} (set (keys table))) + "table entries must not carry :ast (a variable's own full state snapshot)")) + +;; These call the real public entry point (api/api-build), not the private +;; prune-ast helper directly. Testing prune-ast in isolation only proves the +;; helper itself behaves correctly — it proves nothing about whether api-build +;; still routes its result through prune-ast at all, so a future edit that +;; forgets to call it, passes the wrong state, or leaks a new field through +;; some other path would go uncaught. Calling api-build with connection-id +;; :test is what makes this possible: :test is a shared sentinel +;; (pine.db.connections/test-connection-id) that both the schema lookup +;; (postgres.clj) and the connection-name lookup (connections.clj) recognize, +;; so api-build's connections/get-connection-name call — which normally +;; requires a real registered connection pool — succeeds without one. +(deftest test-api-build-ast + (let [single-block (:ast (api/api-build + ["tenant as t | company .tenantId | group: t.title |= x"] + nil :test)) + chained-blocks (:ast (api/api-build + ["tenant as t | company .tenantId | group: t.title |= x" + "x | s: count, | o: count desc |= y" + "y | s: count, |= z" + "z | "] + nil :test))] + + (testing "ast.variables entries are pruned like pending-assignments, not raw snapshots" + ;; A raw variable snapshot carries :variables and :references from + ;; pre-handle/post-handle. Left unpruned, each chained |= re-embeds every + ;; earlier variable's own full snapshot inside the new one, growing the + ;; response payload superlinearly with the number of chained expressions + ;; instead of linearly. + (is (= #{"x" "y" "z"} (set (keys (:variables chained-blocks))))) + (doseq [[name var-ast] (:variables chained-blocks)] + (testing (str "variable " name) + (is (= #{:tables :selected-tables :joins :columns} (set (keys var-ast))) + "should only carry the fields VariableAst (client.ts) actually uses") + (is (not (contains? var-ast :variables)) + "must not recursively embed earlier variables' own snapshots") + (is (not (contains? var-ast :references)) + "must not carry the full schema references map")))) + + (testing "table entries (top-level and nested inside variables) never carry :ast" + ;; A variable-backed table entry carries a full :ast (the variable's own + ;; var-ast) for the query builder's CTE generation. Left in place, that + ;; recursively re-embeds the variable's entire state — and everything IT + ;; wraps in turn — inside every table list that references it, one level + ;; down from the :variables map itself. + (doseq [table (:selected-tables chained-blocks)] + (assert-clean-table table)) + (doseq [[_name var-ast] (:variables chained-blocks) + table (concat (:tables var-ast) (:selected-tables var-ast))] + (assert-clean-table table))) + + (testing "an earlier variable's own entry is unaffected by how many blocks chain after it" + ;; The actual bug wasn't about absolute size (which is arbitrary and brittle to + ;; pin to a byte count) — it was that x's entry kept growing every time another + ;; block chained onto it. Pruning removes the machinery (:variables/:references/ + ;; :ast) that let that happen, so x's pruned entry here should be byte-for-byte + ;; identical whether it's standing alone or three more blocks have chained onto + ;; it since. + (is (= (get-in single-block [:pending-assignments "x"]) + (get-in chained-blocks [:variables "x"])))))) diff --git a/test/pine/ast_test.clj b/test/pine/ast_test.clj index 756bc38..1499492 100644 --- a/test/pine/ast_test.clj +++ b/test/pine/ast_test.clj @@ -296,6 +296,16 @@ ;; (generate :update "customer | update! name = 'John'"))) ) + (testing "Generate ast for `assign`" + ;; The snapshot captures the current table alias at the point of |= + (is (= "c_0" + (get-in (generate :pending-assignments "company |= x") + ["x" :current]))) + ;; Mid-pipeline: snapshot is taken at |=, not at the end of the expression + (is (= "c_0" + (get-in (generate :pending-assignments "company |= x | employee") + ["x" :current])))) + (testing "Schema-based type conversion in WHERE operations" ;; Test that JSONB column gets proper type conversion in WHERE clause (let [where-result (generate :where "customer | w: data = '{\"key\": \"value\"}'")] diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 8090920..163c0e9 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -14,6 +14,20 @@ (ast/generate :test) eval/build-query)) +(defn- generate-expressions + "Evaluate a list of pine expressions sequentially, threading variables. + Returns the SQL of the last expression." + [expressions] + (let [{:keys [last-state]} + (reduce (fn [{:keys [variables]} expr] + (let [{:keys [result]} (parser/parse expr) + state (ast/generate result :test nil nil variables)] + {:variables (merge variables (:pending-assignments state)) + :last-state state})) + {:variables {} :last-state nil} + expressions)] + (eval/build-query last-state))) + (deftest test-build-query (testing "qualify table" @@ -70,7 +84,7 @@ :params (map dt/number ["1"])} (generate "company | where: id != 1"))) (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"id\" IS NULL LIMIT 250", - :params []} + :params nil} (generate "company | where: id is null")))) (testing "Condition : !=" @@ -85,24 +99,24 @@ (testing "Condition : columns" (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"name\" = \"country\" LIMIT 250", - :params (map dt/string [])} + :params nil} (generate "company | where: name = country"))) (is (= {:query "SELECT \"c\".id AS \"__c__id\", \"c\".* FROM \"company\" AS \"c\" WHERE \"c\".\"name\" != \"c\".\"country\" LIMIT 250", - :params (map dt/string [])} + :params nil} (generate "company as c | name != c.country"))) (is (= {:query "SELECT \"c\".id AS \"__c__id\", \"c\".* FROM \"company\" AS \"c\" WHERE \"c\".\"name\" != \"c\".\"country\" LIMIT 250", - :params (map dt/string [])} + :params nil} (generate "company as c | c.name != c.country")))) (testing "Condition : NULL" (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"deleted_at\" IS NULL LIMIT 250", - :params []} + :params nil} (generate "company | where: deleted_at is null"))) (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"deleted_at\" IS NOT NULL LIMIT 250", - :params []} + :params nil} (generate "company | where: deleted_at is not null"))) (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"deleted_at\" IS NULL LIMIT 250", - :params []} + :params nil} (generate "company | where: deleted_at = null")))) (testing "Condition with cast" @@ -136,7 +150,11 @@ (generate "company | employee :right"))) (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"e_1\".id AS \"__e_1__id\", \"d_2\".id AS \"__d_2__id\", \"d_2\".* FROM \"x\".\"company\" AS \"c_0\" LEFT JOIN \"y\".\"employee\" AS \"e_1\" ON \"c_0\".\"id\" = \"e_1\".\"company_id\" RIGHT JOIN \"z\".\"document\" AS \"d_2\" ON \"e_1\".\"id\" = \"d_2\".\"employee_id\" LIMIT 250", :params nil} - (generate "x.company | y.employee :left | z.document :right")))) + (generate "x.company | y.employee :left | z.document :right"))) + ;; Self-join: :parent flips join direction (e_0 has the FK, not e_1) + (is (true? (clojure.string/includes? + (:query (generate "employee | employee :parent")) + "ON \"e_0\".\"reports_to\" = \"e_1\".\"id\"")))) (testing "Joins with explicit columns" ;; Basic explicit join (tables a, b, c don't exist in schema so no auto-id columns) @@ -302,6 +320,10 @@ :query "UPDATE \"company\" SET \"active\" = true WHERE id IN ( SELECT \"c_0\".\"id\" FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"id\" = ? )" :params (list (dt/number "1"))}]} (generate "company | where: id = 1 | update! active = true"))) + (is (= {:queries [{:table "company" + :query "UPDATE \"company\" SET \"deleted_at\" = NULL WHERE id IN ( SELECT \"c_0\".\"id\" FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"id\" = ? )" + :params (list (dt/number "1"))}]} + (generate "company | where: id = 1 | update! deleted_at = null"))) ;; Test JSONB type conversion (is (= {:queries [{:table "customer" @@ -379,4 +401,217 @@ (generate "company /* get by name */ | where: name = 'Acme' -- exact match"))) (is (= {:query "SELECT \"c_0\".\"id\", \"e_1\".\"name\", \"c_0\".id AS \"__c_0__id\", \"e_1\".id AS \"__e_1__id\" FROM \"company\" AS \"c_0\" JOIN \"employee\" AS \"e_1\" ON \"c_0\".\"id\" = \"e_1\".\"company_id\" LIMIT 250", :params nil} - (generate "-- companies and employees\ncompany | s: id /* company id */ | employee | s: name -- employee name")))) \ No newline at end of file + (generate "-- companies and employees\ncompany | s: id /* company id */ | employee | s: name -- employee name")))) + +(deftest test-variables + (testing "Single expression with |= produces normal SQL (assign is metadata)" + (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" LIMIT 250" + :params nil} + (generate-expressions ["company |= active_companies"])))) + + (testing "Variable used as table generates CTE" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"active_companies\".* FROM \"active_companies\" AS \"active_companies\" LIMIT 250" + :params nil} + (generate-expressions ["company |= active_companies" + "active_companies"])))) + + (testing "Variable with WHERE filter generates filtered CTE" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"name\" = ? ) SELECT \"active_companies\".* FROM \"active_companies\" AS \"active_companies\" LIMIT 250" + :params (map dt/string ["Acme"])} + (generate-expressions ["company | where: name = 'Acme' |= active_companies" + "active_companies"])))) + + (testing "Join through a variable resolves correctly" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"e_1\".id AS \"__e_1__id\", \"e_1\".* FROM \"active_companies\" AS \"active_companies\" JOIN \"employee\" AS \"e_1\" ON \"active_companies\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + :params nil} + (generate-expressions ["company |= active_companies" + "active_companies | employee"])))) + + (testing "Composed variables (variable of variable) generates flat CTEs" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"small_active\" AS ( SELECT \"active_companies\".* FROM \"active_companies\" AS \"active_companies\" LIMIT 10 ) SELECT \"small_active\".* FROM \"small_active\" AS \"small_active\" LIMIT 250" + :params nil} + (generate-expressions ["company |= active_companies" + "active_companies | l: 10 |= small_active" + "small_active"])))) + + (testing "Reverse join: child table navigates to variable wrapping its parent" + ;; employee.company_id -> company.id + ;; mytest = company (parent); employee | mytest should join employee to the CTE + (is (= {:query "WITH \"mytest\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"e_0\".id AS \"__e_0__id\", \"mytest\".* FROM \"employee\" AS \"e_0\" JOIN \"mytest\" AS \"mytest\" ON \"e_0\".\"company_id\" = \"mytest\".\"id\" LIMIT 250" + :params nil} + (generate-expressions ["company |= mytest" + "employee | mytest"])))) + + (testing "Reverse join: parent table navigates to variable wrapping its child" + ;; employee.company_id -> company.id + ;; mytest = employee (child); company | mytest should join company to the CTE + (is (= {:query "WITH \"mytest\" AS ( SELECT \"e_0\".* FROM \"employee\" AS \"e_0\" ) SELECT \"c_0\".id AS \"__c_0__id\", \"mytest\".* FROM \"company\" AS \"c_0\" JOIN \"mytest\" AS \"mytest\" ON \"c_0\".\"id\" = \"mytest\".\"company_id\" LIMIT 250" + :params nil} + (generate-expressions ["employee |= mytest" + "company | mytest"])))) + + (testing "Mid-pipeline assign: expression continues after |=" + ;; The assign snapshots the state at that point; subsequent ops still apply + (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" LIMIT 10" + :params nil} + (generate-expressions ["company |= x | l: 10"]))) + ;; x is the unfiltered snapshot (no limit) — CTE body has no LIMIT + (is (= {:query "WITH \"x\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"x\".* FROM \"x\" AS \"x\" LIMIT 250" + :params nil} + (generate-expressions ["company |= x | l: 10" + "x"])))) + + (testing "Column reference via variable name resolves to the real SQL alias" + ;; Within-expression: |= c still routes through pending-assignments → real table alias c_0 + (is (= {:query "SELECT \"e_1\".\"id\", \"c_0\".\"id\", \"c_0\".id AS \"__c_0__id\", \"e_1\".id AS \"__e_1__id\" FROM \"company\" AS \"c_0\" JOIN \"employee\" AS \"e_1\" ON \"c_0\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + :params nil} + (generate-expressions ["company |= c | employee | s: id, c.id"]))) + (is (true? (clojure.string/includes? + (:query (generate-expressions ["company |= c | employee | w: c.id = 1"])) + "WHERE \"c_0\".\"id\" = ?"))) + (is (true? (clojure.string/includes? + (:query (generate-expressions ["company |= c | employee | o: c.id"])) + "ORDER BY \"c_0\".\"id\""))) + + ;; Cross-expression: CTE alias = variable name, so x.id resolves to "x"."id" + (is (true? (clojure.string/includes? + (:query (generate-expressions ["company |= x" + "x | w: x.id = 1"])) + "WHERE \"x\".\"id\" = ?"))) + (is (true? (clojure.string/includes? + (:query (generate-expressions ["company |= x" + "x | o: x.id"])) + "ORDER BY \"x\".\"id\"")))) + + (testing "Re-aliasing a variable name via 'as' takes precedence over the |= snapshot" + ;; x is bound to company via |=, then re-aliased to employee via `as x`. + ;; The live alias should win: x.id must mean employee.id (the most recent, + ;; explicit binding), not the stale company snapshot captured at the |= point. + (is (true? (clojure.string/includes? + (:query (generate-expressions ["company |= x | employee as x | w: x.id = 1"])) + "WHERE \"x\".\"id\" = ?"))) + (is (false? (clojure.string/includes? + (:query (generate-expressions ["company |= x | employee as x | w: x.id = 1"])) + "WHERE \"c_0\".\"id\" = ?")))) + + (testing "Same-source variables: two variables wrapping the same table join on id" + (is (= {:query "WITH \"c1\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"c2\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"c2\".* FROM \"c1\" AS \"c1\" JOIN \"c2\" AS \"c2\" ON \"c1\".\"id\" = \"c2\".\"id\" LIMIT 250" + :params nil} + (generate-expressions ["company |= c1" + "company |= c2" + "c1 | c2"]))) + (is (= {:query "WITH \"c2\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"c1\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"c1\".* FROM \"c2\" AS \"c2\" JOIN \"c1\" AS \"c1\" ON \"c2\".\"id\" = \"c1\".\"id\" LIMIT 250" + :params nil} + (generate-expressions ["company |= c1" + "company |= c2" + "c2 | c1"])))) + + (testing "A table is a join source through a variable only if its id is present" + ;; `s: name` alone has no id anywhere in x's snapshot — Pine doesn't add + ;; one, so company is not a valid join source. + (is (clojure.string/includes? + (:query (generate-expressions ["company as c | s: name |= x" "x | employee"])) + "ON \"\" = \"\"")) + + ;; `s: id, name` includes it explicitly, so the join resolves. + (is (clojure.string/includes? + (:query (generate-expressions ["company as c | s: id, name |= x" "x | employee"])) + "\"x\".\"id\" = \"e_1\".\"company_id\"")) + + ;; Same for GROUP: grouping by a non-id column has no id anywhere, so + ;; company is not a valid join source. + (is (clojure.string/includes? + (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= x" "x | employee"])) + "ON \"\" = \"\"")))) + +(deftest test-checkpoints + (testing "LIMIT checkpoint: auto-CTE when a table op follows limit" + (is (= {:query "WITH \"__pine_0__\" AS ( SELECT \"c_0\".* FROM \"x\".\"company\" AS \"c_0\" LIMIT 10 ) SELECT \"e_1\".id AS \"__e_1__id\", \"e_1\".* FROM \"__pine_0__\" AS \"__pine_0__\" JOIN \"employee\" AS \"e_1\" ON \"__pine_0__\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + :params nil} + (generate "x.company | l: 10 | employee")))) + + (testing "LIMIT checkpoint with explicit user-named CTE via |=" + (is (= {:query "WITH \"pg\" AS ( SELECT \"c_0\".* FROM \"x\".\"company\" AS \"c_0\" LIMIT 10 ) SELECT \"e_1\".id AS \"__e_1__id\", \"e_1\".* FROM \"pg\" AS \"pg\" JOIN \"employee\" AS \"e_1\" ON \"pg\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + :params nil} + (generate "x.company | l: 10 |= pg | employee")))) + + (testing "GROUP checkpoint: auto-CTE when a table op follows group" + (is (= {:query "WITH \"__pine_0__\" AS ( SELECT \"c_0\".\"id\", COUNT(1) AS \"count\" FROM \"x\".\"company\" AS \"c_0\" GROUP BY \"c_0\".\"id\" ) SELECT \"e_1\".id AS \"__e_1__id\", \"e_1\".* FROM \"__pine_0__\" AS \"__pine_0__\" JOIN \"employee\" AS \"e_1\" ON \"__pine_0__\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + :params nil} + (generate "x.company | group: id => count | employee")))) + + (testing "Checkpoint does not fire for non-table ops after limit" + ;; LIMIT followed by COUNT: checkpoint holds, COUNT builds its own wrapper CTE + (is (= {:query "WITH x AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" LIMIT 100 ) SELECT COUNT(*) FROM x" + :params nil} + (generate "company | limit: 100 | count:")))) + + (testing "Standalone GROUP without following table still uses build-group-query" + ;; Existing behaviour must not regress: the GROUP dispatch path is unaffected + (is (clojure.string/includes? + (:query (generate "x.company | group: id => count")) + "GROUP BY"))) + + (testing "GROUP followed by LIMIT seals group as CTE, applies limit to outer" + ;; All three forms should produce equivalent SQL (modulo CTE name) + (let [expected-cte-body "SELECT \"c_0\".\"id\", COUNT(1) AS \"count\" FROM \"x\".\"company\" AS \"c_0\" GROUP BY \"c_0\".\"id\"" + ;; Auto-named + q1 (:query (generate "x.company | group: id => count | l: 1")) + ;; User-named via |= + q2 (:query (generate "x.company | group: id => count |= grp | l: 1"))] + ;; Both contain the same GROUP CTE body + (is (clojure.string/includes? q1 expected-cte-body)) + (is (clojure.string/includes? q2 expected-cte-body)) + ;; Both apply LIMIT 1 on the outer query + (is (clojure.string/ends-with? q1 "LIMIT 1")) + (is (clojure.string/ends-with? q2 "LIMIT 1")) + ;; Auto-named CTE uses generated name; user-named uses "grp" + (is (clojure.string/includes? q1 "\"__pine_0__\"")) + (is (clojure.string/includes? q2 "\"grp\"")))) + + (testing "GROUP + LIMIT cross-expression baseline matches same-expression form" + ;; x | l: 1 in a separate expression should produce the same structure + (let [q-cross (:query (generate-expressions ["x.company | group: id => count |= grp" + "grp | l: 1"])) + q-same (:query (generate "x.company | group: id => count |= grp | l: 1"))] + (is (= q-cross q-same)))) + + (testing "GROUP followed by a complete select: matches the sealed cross-expression form" + ;; select:/where:/order: after GROUP were not checkpoint-firing ops (unlike + ;; table/group/limit), so the group was never sealed into a CTE — build-query + ;; then dispatched on the trailing op's type instead of :group, silently using + ;; the plain-select builder against a group-shaped state (wrong SQL: raw table + ;; columns leaking in, stale pre-group alias in WHERE/ORDER BY). The correctly + ;; sealed cross-expression form is the oracle: same GROUP, split so the second + ;; expression references the CTE by name, should equal the inline form exactly. + (let [q-cross (:query (generate-expressions ["x.company | group: id => count |= grp" + "grp | s: id"])) + q-same (:query (generate "x.company | group: id => count |= grp | s: id"))] + (is (= q-cross q-same)))) + + (testing "GROUP followed by a complete where: matches the sealed cross-expression form" + (let [q-cross (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= grp" + "grp | w: name = 'Acme'"])) + q-same (:query (generate "company as c | employee .company_id | group: c.name |= grp | w: name = 'Acme'"))] + (is (= q-cross q-same)))) + + (testing "GROUP followed by a complete order: matches the sealed cross-expression form" + (let [q-cross (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= grp" + "grp | o: name desc"])) + q-same (:query (generate "company as c | employee .company_id | group: c.name |= grp | o: name desc"))] + (is (= q-cross q-same)))) + + (testing "GROUP followed by an order-partial (trailing comma) matches the sealed cross-expression form" + ;; order-partial with a non-empty value (e.g. a dangling trailing comma after a + ;; fully-typed column) is not meaningfully different from a complete order: here — + ;; the already-typed column should seal and resolve exactly the same way. + (let [q-cross (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= grp" + "grp | o: name desc,"])) + q-same (:query (generate "company as c | employee .company_id | group: c.name |= grp | o: name desc,"))] + (is (= q-cross q-same)))) + + (testing "GROUP followed by a complete select: auto-named CTE also seals correctly (no |=)" + ;; Without an explicit |=, the checkpoint still seals — just with a generated + ;; name instead of a user-given one. + (is (= "WITH \"__pine_0__\" AS ( SELECT \"c_0\".\"id\", COUNT(1) AS \"count\" FROM \"x\".\"company\" AS \"c_0\" GROUP BY \"c_0\".\"id\" ) SELECT \"__pine_0__\".\"id\" FROM \"__pine_0__\" AS \"__pine_0__\" LIMIT 250" + (:query (generate "x.company | group: id => count | s: id")))))) \ No newline at end of file diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index 7900c77..abbd177 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -15,6 +15,19 @@ (ast/generate :test expression cursor) :hints))) +(defn- gen-with-variables + "Evaluate expressions sequentially, threading variables. Returns :hints from the last expression." + [expressions] + (let [{:keys [last-hints]} + (reduce (fn [{:keys [variables]} expr] + (let [{:keys [result]} (parser/parse expr) + state (ast/generate result :test expr nil variables)] + {:variables (merge variables (:pending-assignments state)) + :last-hints (:hints state)})) + {:variables {} :last-hints nil} + expressions)] + last-hints)) + (deftest test-hints (testing "Generate hints" (is (= [{:schema "x", :table "company" @@ -95,16 +108,40 @@ (is (= ["id" "company_id" "reports_to"] (->> "company | s: id | employee | s: " gen :select (map :column)))) (is (= ["company_id" "reports_to"] (->> "company | s: id | employee | s: id, " gen :select (map :column)))) - ;; The following doesn't get parsed at the moment - ;; We need to update the pine.bnf to support the syntax - ;; - ;; (is (= ["reports_to" "company_id" "id"] (->> "employee as e | company | s: id, e." gen :select (map :column)))) - ) + ;; Cross-table: after selecting from company, next slot defaults to current context (employee) + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | s: c.id," gen :select (map :column)))) + + ;; Alias-dot partial: s: e. should show all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | s: e." gen :select (map :column)))) + + ;; Alias-dot after completed column: s: id, e. should show all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "employee as e | company | s: id, e." gen :select (map :column)))) + + ;; Alias-dot excludes already-selected columns for that alias: s: e.id, e. omits id + (is (= ["company_id" "reports_to"] + (->> "x.company as c | y.employee as e | s: e.id, e." gen :select (map :column))))) + + (testing "Generate `select-partial` hints after an un-sealed GROUP checkpoint" + ;; group: c.name hasn't been sealed into a CTE yet (no table op follows before + ;; s:), but the columns available for hints should be the group's own output + ;; (name, count), not the underlying employee table's raw schema. + (is (= ["name" "count"] + (->> "company as c | employee .company_id | group: c.name | s: " gen :select (map :column)))) + ;; Same, with an explicit |= assign before the un-sealed s: — still same-expression, + ;; still not sealed into a CTE until a table op follows. + (is (= ["name" "count"] + (->> "company as c | employee .company_id | group: c.name |= x | s: " gen :select (map :column))))) (testing "Generate `order-partial` hints" (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | o:" gen :order))) (is (= [{:column "created_at" :alias "c_0"}] (-> "company | o: id," gen :order))) - (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | s: id | o:" gen :order)))) + (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | s: id | o:" gen :order))) + ;; Alias-dot partial: o: e. should show all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | o: e." gen :order (map :column))))) (testing "Generate `where-partial` hints" (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | where:" gen :where))) @@ -118,6 +155,19 @@ (is (= [] (-> "company | w: xyz" gen :where))) (is (= ["id" "company_id"] (->> "y.employee | w: id" gen :where (map :column)))) + ;; Explicit alias: w: e.col should use alias "e" for column lookup, not the current context + (is (= [{:column "company_id" :alias "e"}] + (-> "y.employee as e | x.company as c | w: e.company_id" gen :where))) + (is (= [{:column "id" :alias "c"}] + (-> "y.employee as e | x.company as c | w: c.id" gen :where))) + ;; Alias-dot partial: w: e. should show all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | w: e." gen :where (map :column)))) + + ;; Complete condition then alias-dot: w: id = 1, e. shows all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | w: id = 1, e." gen :where (map :column)))) + ;; How to auto-complete the right hand side? Values or other columns? ;; Right now it shows the same hints as the left hand side ;; (is (= [{:column "id" :alias "c_0"}] (-> "company | w: id =" gen :where))) @@ -127,10 +177,20 @@ (testing "Generate `update-partial` hints" (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | u!" gen :update))) + ;; Explicit alias in update: u! e.col should use alias "e" for column lookup, not current context + (is (= [{:column "company_id" :alias "e"}] + (-> "y.employee as e | x.company as c | u! e.company_id" gen :update))) (is (= [{:column "created_at" :alias "c_0"}] (-> "company | u! id = '1'," gen :update))) (is (= [{:column "id" :alias "c_0"}] - (-> "company | u! i" gen :update)))) + (-> "company | u! i" gen :update))) + ;; Alias-dot partial: u! e. should show all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | u! e." gen :update (map :column)))) + + ;; Prior assignment then alias-dot: u! id = '1', e. excludes already-assigned id + (is (= ["company_id" "reports_to"] + (->> "x.company as c | y.employee as e | u! id = '1', e." gen :update (map :column))))) (testing "Generate hints with cursor position" ;; Basic cursor truncation test - cursor at "company | s: " should show select hints for company @@ -163,4 +223,52 @@ (is (= ["id" "company_id" "reports_to"] (->> (gen "company | s: id | employee | s: " {:line 0 :character 100}) :select - (map :column)))))) + (map :column))))) + + (testing "Variable relation hints" + ;; mytest = employee (child); company | my should suggest mytest + (is (= [{:schema nil :table "mytest" :column "company_id" :parent false :heuristic false + :pine "mytest .company_id"}] + (-> (gen-with-variables ["employee |= mytest" "company | my"]) + :table))) + + ;; mytest = company (parent); employee | my should suggest mytest + (is (= [{:schema nil :table "mytest" :column "company_id" :parent true :heuristic false + :pine "mytest .company_id :parent"}] + (-> (gen-with-variables ["company |= mytest" "employee | my"]) + :table))) + + ;; same-source variables: var_x and var_y both wrap company; + ;; when typing "var_x | var", var_y should appear (partial match, unambiguous token) + (is (= [{:schema nil :table "var_y" :column nil :parent false :heuristic false + :pine "var_y"}] + (-> (gen-with-variables ["customer |= var_x" "customer |= var_y" "var_x | var"]) + :table)))) + + (testing "A table is only a valid join source through a variable if its own id survives" + ;; x explicitly selects only name — Pine doesn't add an id on its own, so + ;; company's id is nowhere in x's actual CTE output. employee must NOT be + ;; suggested. + (is (= [] + (-> (gen-with-variables ["company as c | s: name |= x" "x | emp"]) + :table))) + + ;; Same table, but id is explicitly selected this time — now it's present, + ;; and employee is correctly suggested. + (is (= ["employee"] + (->> (gen-with-variables ["company as c | s: id, name |= x" "x | emp"]) + :table + (map :table)))) + + ;; Same rule applies to GROUP: grouping by a non-id column loses company's + ;; id just the same — employee must NOT be suggested. + (is (= [] + (-> (gen-with-variables ["company as c | employee .company_id | group: c.name |= x" "x | doc"]) + :table))) + + ;; ...but grouping by id (alongside other columns) preserves it, same as s: id, name. + (is (= ["document"] + (->> (gen-with-variables ["company as c | employee .company_id | group: c.id, c.name |= x" "x | doc"]) + :table + (map :table)))))) + diff --git a/test/pine/parser_test.clj b/test/pine/parser_test.clj index 753a5bd..004df06 100644 --- a/test/pine/parser_test.clj +++ b/test/pine/parser_test.clj @@ -1,6 +1,6 @@ (ns pine.parser-test (:require [clojure.test :refer [deftest is testing]] - [pine.parser :refer [parse-or-fail prettify]] + [pine.parser :refer [parse parse-or-fail prettify]] [pine.data-types :as dt])) (defn- p [e] @@ -99,10 +99,11 @@ (is (= [{:type :select-partial, :value []}] (p "s: "))) (is (= [{:type :select-partial, :value [{:column "id"}]}] (p "s: id,"))) (is (= [{:type :select-partial, :value [{:column "id"}]}] (-> "company | s: id," p rest))) - ;; Not supported yet - ;; (is (= [{:type :select-partial, :value [{:alias "c" :column ""}]}] (-> "company as c | s: c." p rest))) - ;; (is (= [{:type :select-partial, :value [{:alias "c" :column ""}]}] (-> "company as c | s: id, c." p rest))) - ) + ;; Alias-dot partial + (is (= [{:type :select-partial :value [] :partial-alias {:alias "e" :column ""}}] + (-> "company | s: e." p rest))) + (is (= [{:type :select-partial :value [{:column "id"}] :partial-alias {:alias "e" :column ""}}] + (-> "company | s: id, e." p rest)))) (testing "Parse `select` expressions" (is (= [{:type :select, :value [{:column "name"}]}] (p "select: name"))) @@ -162,12 +163,20 @@ (is (= [{:type :where-partial, :value {:complete-conditions [] :partial-condition {:column "id"}}}] (p "w: id"))) (is (= [{:type :where-partial, :value {:complete-conditions [] :partial-condition {:alias "u", :column "id"}}}] (p "w: u.id"))) (is (= [{:type :where-partial, :value {:complete-conditions [] :partial-condition {:column "id", :operator :equals}}}] (p "w: id ="))) - (is (= [{:type :where-partial, :value {:complete-conditions [] :partial-condition {:column "id", :operator :like}}}] (p "w: id like")))) + (is (= [{:type :where-partial, :value {:complete-conditions [] :partial-condition {:column "id", :operator :like}}}] (p "w: id like"))) + ;; Alias-dot partial + (is (= [{:type :where-partial :value {:complete-conditions [] :partial-condition {:alias "e" :column ""}}}] + (-> "company | w: e." p rest)))) (testing "Parse `order-partial` expressions" (is (= [{:type :order-partial, :value []}] (p "order:"))) (is (= [{:type :order-partial, :value []}] (p "o: "))) - (is (= [{:type :order-partial, :value [{:column "id", :direction "DESC"}]}] (p "o: id,")))) + (is (= [{:type :order-partial, :value [{:column "id", :direction "DESC"}]}] (p "o: id,"))) + ;; Alias-dot partial + (is (= [{:type :order-partial :value [] :partial-alias {:alias "e" :column ""}}] + (-> "company | o: e." p rest))) + (is (= [{:type :order-partial :value [{:column "id" :direction "DESC"}] :partial-alias {:alias "e" :column ""}}] + (-> "company | o: id, e." p rest)))) (testing "Parse `order` expressions" (is (= [{:type :order, :value [{:column "name" :direction "DESC"}]}] (p "order: name"))) @@ -224,7 +233,15 @@ (is (= [{:type :update-partial :value {:assignments [{:column {:alias nil :column "id"} :value (dt/string "1")}] :partial-column {:alias nil :column "n"}}}] - (p "u! id = '1', n")))) + (p "u! id = '1', n"))) + ;; Alias-dot partial + (is (= {:type :update-partial :value {:assignments [] + :partial-column {:alias "e" :column ""}}} + (-> "employee as e | company | u! e." p last))) + (is (= {:type :update-partial :value {:assignments [{:column {:alias nil :column "id"} + :value (dt/string "1")}] + :partial-column {:alias "e" :column ""}}} + (-> "employee as e | company | u! id = '1', e." p last)))) (testing "Parse No Operation expressions" (is (= [{:value {:table "company"}, :type :table} {:type :delete, :value nil}] (p "company | d:")))) @@ -370,5 +387,50 @@ (prettify "company | where: id = 1 | ")))) (testing "Incomplete expressions return an error" - (is (contains? (prettify "company | w: name = 'test |") :error)))) + (is (contains? (prettify "company | w: name = 'test |") :error))) + + (testing "Assignment is formatted without double pipe" + (is (= {:result "company\n | = active_companies" + :operations [{:expression "company" :start 0 :end 7} + {:expression "= active_companies" :start 9 :end 27}]} + (prettify "company |= active_companies"))) + (is (= {:result "company\n | = x" + :operations [{:expression "company" :start 0 :end 7} + {:expression "= x" :start 10 :end 13}]} + (prettify "company | = x"))) + (is (= {:result "company\n | where: active = true\n | = active_companies" + :operations [{:expression "company" :start 0 :end 7} + {:expression "where: active = true" :start 10 :end 30} + {:expression "= active_companies" :start 32 :end 50}]} + (prettify "company | where: active = true |= active_companies"))) + ;; |= is mid-pipeline — operations after it are valid + (is (= {:result "company\n | = x\n | " + :operations [{:expression "company" :start 0 :end 7} + {:expression "= x" :start 9 :end 12} + {:expression "" :start 14 :end 14}]} + (prettify "company |= x | "))))) + +(deftest test-assign + (testing "|= appears in :result as an assign op" + (is (= [{:type :table :value {:table "company"}} + {:type :assign :value "active_companies"}] + (:result (parse "company |= active_companies")))) + (is (= [{:type :table :value {:table "company"}} + {:type :assign :value "active_companies"}] + (:result (parse "company | = active_companies")))) + (is (= [{:type :table :value {:table "company"}}] + (:result (parse "company"))))) + + (testing ":assign is no longer a top-level field in parse result" + (is (nil? (:assign (parse "company |= active_companies")))) + (is (nil? (:assign (parse "company"))))) + + (testing "|= is mid-pipeline: operations continue after it" + (is (= [{:type :table :value {:table "company"}} + {:type :assign :value "x"} + {:type :table :value {:table "employee"}}] + (:result (parse "company |= x | employee")))) + (is (= 4 (count (:result (parse "company | where: id = 1 | s: name |= my_var"))))) + (is (= {:type :assign :value "my_var"} + (last (:result (parse "company | where: id = 1 | s: name |= my_var")))))))