From 42aab568b066f588fddfc577aa14d90923c3323d Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 00:49:48 +0200 Subject: [PATCH 01/60] feat: introduce pine variables (|= syntax) [WIP] From 893effe1e415ebf1234fe3315dad891f090384c2 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 00:49:48 +0200 Subject: [PATCH 02/60] feat: variable store and multi-expression evaluation [WIP] From ab6cf17743a36f4491d30f210f0bb638142b6bff Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 12:06:54 +0200 Subject: [PATCH 03/60] feat: variables, multi-expression eval, and CTE generation [WIP] - Add |= syntax to assign expressions to named variables - Variables compile to CTEs; later expressions reference them as tables - Multi-expression API: expressions[] array, last is active, preceding are context - Fix build-query empty-expression guard (was hardcoded "x_0" alias, now checks actual table name) - Skip auto-id columns for variable/CTE tables (only real tables get them) - Add docs/variables.md explaining the feature Co-Authored-By: Claude Sonnet 4.6 --- docs/variables.md | 59 +++++++++++++++++++ src/pine/api.clj | 119 ++++++++++++++++++++++++-------------- src/pine/ast/main.clj | 64 +++++++++++++++----- src/pine/ast/select.clj | 7 ++- src/pine/ast/table.clj | 35 ++++++++--- src/pine/eval.clj | 57 +++++++++++++++++- src/pine/parser.clj | 15 ++++- src/pine/pine.bnf | 3 +- test/pine/eval_test.clj | 61 ++++++++++++++++--- test/pine/parser_test.clj | 21 ++++++- 10 files changed, 356 insertions(+), 85 deletions(-) create mode 100644 docs/variables.md diff --git a/docs/variables.md b/docs/variables.md new file mode 100644 index 0000000..a46d825 --- /dev/null +++ b/docs/variables.md @@ -0,0 +1,59 @@ +# Variables + +Variables let you name and reuse intermediate query results across expressions. They compile to CTEs (`WITH` clauses) in the generated SQL. + +## Syntax + +``` + |= +``` + +Append `|= name` at the end of any pipe chain to assign the result to a variable. + +## Example + +``` +company | where: active = true |= active_companies + +active_companies | employee +``` + +The first expression defines `active_companies`. The second expression uses it as a table — Pine expands it into a CTE: + +```sql +WITH "active_companies" AS ( + SELECT "c_0".* FROM "company" AS "c_0" WHERE "c_0"."active" = true +) +SELECT "e_1".id AS "__e_1__id", "e_1".* +FROM "active_companies" AS "ac_0" +JOIN "employee" AS "e_1" ON "ac_0"."id" = "e_1"."company_id" +LIMIT 250 +``` + +## Multi-expression input + +Expressions are separated by blank lines. Each expression is evaluated in order; earlier expressions provide context (as CTEs) for later ones. Only the last expression is executed and returned. + +``` +company | where: active = true |= active_companies + +active_companies | l: 10 |= small_active + +small_active +``` + +This produces two CTEs (`active_companies`, `small_active`) and selects from the last. + +## How it works + +- **Assignment** (`|= name`): the expression is stored as a variable. Its SQL becomes the body of a CTE named `name`. +- **Usage**: when `name` appears as a table in a later expression, Pine substitutes the CTE. Joins through variable tables resolve using the schema of the underlying real table(s). +- **Scoping**: variables accumulate left-to-right. Each expression sees all variables defined before it. +- **Auto-id columns**: Pine normally adds a hidden `id` column per table for internal row tracking. Variable tables (CTEs) are excluded — only real database tables get auto-id columns. +- **CTE body**: the CTE body is generated without a `LIMIT` (limits only apply to the outer query). + +## Constraints + +- Variable names must be valid Pine identifiers (letters, digits, underscores). +- A variable used but not defined in a preceding expression will cause a parse/table-not-found error. +- Circular references are not supported. diff --git a/src/pine/api.clj b/src/pine/api.clj index 7ed8218..c1f45db 100644 --- a/src/pine/api.clj +++ b/src/pine/api.clj @@ -28,15 +28,34 @@ (defn- generate-state ([expression] - (generate-state expression nil nil)) + (generate-state expression nil nil {})) ([expression cursor] - (generate-state expression cursor nil)) + (generate-state expression cursor nil {})) ([expression cursor connection-id] - (let [{:keys [result error]} (->> expression parser/parse) + (generate-state expression cursor connection-id {})) + ([expression cursor connection-id variables] + (let [{:keys [result error assign]} (->> expression parser/parse) conn-id (or connection-id @db/connection-id)] - (if result {:result (ast/generate result conn-id expression cursor)} - {:error-type "parse" - :error error})))) + (if result + {:result (ast/generate result conn-id expression cursor variables assign)} + {:error-type "parse" + :error error})))) + +(defn- evaluate-expressions + "Evaluate a sequence of pine expressions, threading variables from |= assignments + into subsequent expressions. Returns {:last-state :error }." + [expressions connection-id] + (reduce (fn [{:keys [variables]} expression] + (let [{:keys [result error]} (generate-state expression nil connection-id variables)] + (if error + (reduced {:error error}) + (let [assign (:assign result)] + {:variables (if assign + (assoc variables assign result) + variables) + :last-state result})))) + {:variables {} :last-state nil} + expressions)) (defn- trim-pipes [s] (-> s @@ -44,22 +63,30 @@ (str/replace #"^\|\s*|\s*\|$" "") (str/trim))) + (defn api-build - ([expression] - (api-build expression nil nil)) - ([expression cursor] - (api-build expression cursor nil)) - ([expression cursor connection-id] + ([expressions] + (api-build expressions nil nil)) + ([expressions cursor] + (api-build expressions cursor nil)) + ([expressions cursor connection-id] (let [conn-id (or connection-id @db/connection-id) connection-name (connections/get-connection-name conn-id)] (try - (let [result (generate-state expression cursor conn-id) - {state :result error :error} result] - (if error result - {:connection-id connection-name - :version version - :query (-> expression trim-pipes (generate-state nil conn-id) :result eval/build-query eval/formatted-query) - :ast (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges])})) + (let [exprs (if (string? expressions) [expressions] expressions) + context-exprs (butlast exprs) + last-expr (last exprs) + {:keys [variables error]} (evaluate-expressions context-exprs conn-id)] + (if error + {:connection-id connection-name :error error} + (let [result (generate-state last-expr cursor conn-id variables) + {state :result build-error :error} result] + (if build-error + {:connection-id connection-name :error build-error} + {:connection-id connection-name + :version version + :query (-> last-expr trim-pipes (generate-state nil conn-id variables) :result eval/build-query eval/formatted-query) + :ast (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges :variables :assign])})))) (catch Exception e {:connection-id connection-name :error (.getMessage e)}))))) @@ -82,28 +109,30 @@ [alias]))))) (defn api-eval - ([expression] - (api-eval expression nil)) - ([expression connection-id] + ([expressions] + (api-eval expressions nil)) + ([expressions connection-id] (let [conn-id (or connection-id @db/connection-id) connection-name (connections/get-connection-name conn-id)] (try - (let [result (generate-state expression nil conn-id) - {state :result error :error} result] - (if error result - (let [rows (eval/run-query state) - op-type (get-in state [:operation :type]) - ;; For action results we control the format; columns come from header row - columns (if (contains? #{:update-action :delete-action} op-type) - (get-columns rows) - (get-columns state rows))] - {:connection-id connection-name - :version version - ;; :time (db/run-query (state :connection-id) {:query "SELECT NOW() as now, NOW() AT TIME ZONE 'UTC' AS utc;"}) - ;; :server_time (str (java.time.Instant/now)) - :result rows - :columns columns}))) - + (let [exprs (if (string? expressions) [expressions] expressions) + {:keys [last-state error]} (evaluate-expressions exprs conn-id)] + (if error + {:connection-id connection-name :error error} + (let [query (-> last-state eval/build-query eval/formatted-query)] + (try + (let [rows (eval/run-query last-state) + op-type (get-in last-state [:operation :type]) + columns (if (contains? #{:update-action :delete-action} op-type) + (get-columns rows) + (get-columns last-state rows))] + {:connection-id connection-name + :version version + :result rows + :columns columns}) + (catch Exception e {:connection-id connection-name + :error (.getMessage e) + :query query}))))) (catch Exception e {:connection-id connection-name :error (.getMessage e)}))))) @@ -190,11 +219,17 @@ ;; query building and evaluation (POST "/api/v1/build" {params :params} - (let [{:keys [expression cursor connection-id]} params] - (->> (api-build expression cursor connection-id) response))) + (let [{:keys [expressions expression cursor connection-id]} params + exprs (or expressions (when expression [expression]))] + (->> (api-build exprs cursor connection-id) response))) (POST "/api/v1/eval" {params :params} - (let [{:keys [expression connection-id]} params] - (->> (api-eval (trim-pipes expression) connection-id) response))) + (let [{:keys [expressions expression connection-id]} params + exprs (or expressions + (when expression + (->> (split-expressions expression) + (map :text) + (map trim-pipes))))] + (->> (api-eval exprs connection-id) response))) ;; raw SQL execution (POST "/api/v1/sql" {params :params} @@ -206,7 +241,7 @@ ;; pine-mode.el (POST "/api/v1/build-with-params" {params :params} (let [{:keys [expression connection-id]} params] - (->> (api-build (trim-pipes expression) nil connection-id) :query response))) + (->> (api-build [(trim-pipes expression)] nil connection-id) :query response))) ;; default case (route/not-found "Not Found")) (def app diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 3e44933..a130202 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -21,6 +21,8 @@ ;; - connection :connection-id nil :references {} + :variables {} ;; {"varname" }, populated from |= assignments + :assign nil ;; variable name from |=, set after handle-ops :expression nil ;; Expression string for cursor-aware hints :cursor nil ;; Cursor position {:line N :character M} (zero-indexed) @@ -66,13 +68,42 @@ ;; - hints :hints {:table [] :select [] :order [] :where [] :update []}}) -(defn pre-handle [state connection-id ops-count expression cursor] - (-> state - (assoc :references (db/init-references connection-id)) - (assoc :connection-id connection-id) - (assoc :pending-count ops-count) - (assoc :expression expression) - (assoc :cursor cursor))) +(defn- seed-variable-references + "Copy reference entries from the real source tables of a variable into the + local references map under the variable name, so join resolution treats the + variable identically to a real table." + [refs var-ast varname] + (let [columns (:columns var-ast) + aliases (:aliases var-ast) + explicit (remove :auto-id columns) + source-tables (if (empty? explicit) + (when-let [current-alias (:current var-ast)] + [(get-in aliases [current-alias :table])]) + (->> explicit + (map #(get-in aliases [(:alias %) :table])) + (remove nil?) + distinct))] + (reduce (fn [r source-table] + (let [source-refs (get-in r [:table source-table])] + (if source-refs + (update-in r [:table varname] merge source-refs) + r))) + refs + source-tables))) + +(defn pre-handle [state connection-id ops-count expression cursor variables] + (let [refs (db/init-references connection-id) + aug-refs (reduce (fn [r [varname var-ast]] + (seed-variable-references r var-ast varname)) + refs + variables)] + (-> state + (assoc :references aug-refs) + (assoc :connection-id connection-id) + (assoc :pending-count ops-count) + (assoc :expression expression) + (assoc :cursor cursor) + (assoc :variables variables)))) (defn handle-op [state {:keys [type value]}] (case type @@ -121,9 +152,9 @@ (str/join "\n" (concat lines-before [truncated-current]))))))) (defn- generate-truncated-state - "Generate state for truncated expression at cursor position. + "Generate state for truncated expression at cursor position. Keep references for hint generation." - [expression cursor connection-id] + [expression cursor connection-id variables] (let [truncated-expr (truncate-at-cursor expression cursor) {:keys [result error]} (parser/parse truncated-expr)] (if (or error (nil? result)) @@ -132,7 +163,7 @@ ;; Successfully parsed, build state without going through post-handle ;; to preserve references for hint generation (-> state - (pre-handle connection-id (count result) nil nil) + (pre-handle connection-id (count result) nil nil variables) (handle-ops result))))) (defn- offset->position @@ -192,14 +223,17 @@ (defn generate ([parse-tree] - (generate parse-tree @db/connection-id nil nil)) + (generate parse-tree @db/connection-id nil nil {} nil)) ([parse-tree connection-id] - (generate parse-tree connection-id nil nil)) + (generate parse-tree connection-id nil nil {} nil)) ([parse-tree connection-id expression cursor] + (generate parse-tree connection-id expression cursor {} nil)) + ([parse-tree connection-id expression cursor variables assign] (let [full-state (-> state - (pre-handle connection-id (count parse-tree) expression cursor) - (handle-ops parse-tree)) + (pre-handle connection-id (count parse-tree) expression cursor variables) + (handle-ops parse-tree) + (assoc :assign assign)) truncated-state (when (and cursor expression) - (generate-truncated-state expression cursor connection-id))] + (generate-truncated-state expression cursor connection-id variables))] (post-handle full-state truncated-state)))) diff --git a/src/pine/ast/select.clj b/src/pine/ast/select.clj index abf7c77..d3d1709 100644 --- a/src/pine/ast/select.clj +++ b/src/pine/ast/select.clj @@ -56,8 +56,11 @@ next-operation-index (inc (:index state)) references (:references state) aliases (:aliases state) - ;; Only create auto-ID columns for tables that actually have an 'id' column - valid-aliases (filter #(has-id-column? references aliases %) table-aliases) + variables (:variables state) + ;; Only create auto-ID columns for real tables (not variables/CTEs) that have an 'id' column + valid-aliases (filter #(and (not (contains? variables (get-in aliases [% :table]))) + (has-id-column? references aliases %)) + table-aliases) auto-id-columns (map-indexed #(create-auto-id-column %2 (+ next-operation-index %1)) valid-aliases)] (update state :columns into auto-id-columns)) state)) diff --git a/src/pine/ast/table.clj b/src/pine/ast/table.clj index 7decb7e..a9ca524 100644 --- a/src/pine/ast/table.clj +++ b/src/pine/ast/table.clj @@ -56,8 +56,7 @@ initials (map #(subs % 0 1) words)] (apply str initials))) -;; todo: spec for the :value for a :table -(defn handle [state value] +(defn- handle-as-table [state value] (let [index (state :index) {:keys [table alias schema parent join-column join-left-column join-right-column join]} value a (or alias (str (make-alias table) "_" (state :table-count))) @@ -66,18 +65,36 @@ :join-right-column join-right-column :join join :index index}] (-> state - ;; pre (assoc :context (state :current)) (assoc :current a) (assoc :current-index index) - ;; (update :tables conj current) (update :aliases assoc a current) (update-joins current) - ;; post - ;; - ;; TODO: These are metadata - mayebe I should move them to a different - ;; ns e.g. if the operation is table, then I update the following there - ;; (update :table-count inc)))) +(defn- handle-as-variable [state value var-ast] + (let [index (state :index) + {:keys [table alias join-column join-left-column join-right-column join]} value + a (or alias (str (make-alias table) "_" (state :table-count))) + current {:schema nil :table table :ast var-ast :alias a + :join-column join-column :join-left-column join-left-column + :join-right-column join-right-column :join join + :index index}] + (-> state + (assoc :context (state :current)) + (assoc :current a) + (assoc :current-index index) + (update :tables conj current) + (update :aliases assoc a current) + (update-joins current) + (update :table-count inc)))) + +;; todo: spec for the :value for a :table +(defn handle [state value] + (let [{:keys [table]} value + var-ast (get-in state [:variables table])] + (if var-ast + (handle-as-variable state value var-ast) + (handle-as-table state value)))) + diff --git a/src/pine/eval.clj b/src/pine/eval.clj index 8285cb5..aae5a7f 100644 --- a/src/pine/eval.clj +++ b/src/pine/eval.clj @@ -90,7 +90,7 @@ (q alias column))) group))))) -(defn build-select-query [state] +(defn- build-bare-select [state] (let [{:keys [tables _columns limit where aliases]} state from (let [{a :alias} (first tables) {table :table schema :schema} (get aliases a)] @@ -123,6 +123,57 @@ {:query query :params params})) +(defn- build-cte-body + "Generate the inner SQL for a variable's AST used as a CTE. + When the current table has no explicit user columns, .* is added and already + includes id — so the auto-id column is dropped to avoid duplicate id names. + When explicit columns are present (no .*), the auto-id is kept but its alias + is stripped so id is accessible for join conditions. + Returns {:query ... :params ...}." + [ast] + (let [current-alias (:current ast) + user-columns (remove :auto-id (:columns ast)) + has-explicit? (some #(= (:alias %) current-alias) user-columns) + columns (keep (fn [col] + (if (and (:auto-id col) (= (:alias col) current-alias)) + (when has-explicit? (dissoc col :column-alias)) + col)) + (:columns ast))] + (build-bare-select (assoc ast :columns columns)))) + +(defn- collect-ctes + "Recursively collect [name query params] triples from variable tables in + topological order (deepest dependencies first). Deduplicates by name." + [tables aliases] + (->> tables + (mapcat (fn [{:keys [alias]}] + (let [entry (get aliases alias)] + (when-let [ast (:ast entry)] + (let [var-name (:table entry) + nested-ctes (collect-ctes (:tables ast) (:aliases ast)) + {:keys [query params]} (build-cte-body ast)] + (conj nested-ctes [var-name query params])))))) + (reduce (fn [[seen acc] [name _ _ :as cte]] + (if (contains? seen name) + [seen acc] + [(conj seen name) (conj acc cte)])) + [#{} []]) + second)) + +(defn build-select-query [state] + (let [ctes (collect-ctes (:tables state) (:aliases state)) + result (build-bare-select state) + cte-params (mapcat #(nth % 2 nil) ctes) + cte-prefix (when (seq ctes) + (str "WITH " + (s/join ", " (map (fn [[name body _]] + (str (q name) " AS ( " body " )")) + ctes)) + " "))] + (-> result + (update :query #(str cte-prefix %)) + (update :params #(seq (concat cte-params %)))))) + (defn build-count-query [state] (let [{:keys [query params]} (build-select-query state)] {:query (str "WITH x AS ( " query " ) SELECT COUNT(*) FROM x") @@ -259,7 +310,9 @@ (defn build-query [state] (let [{:keys [type]} (state :operation)] (cond - (= (-> state :current) "x_0") {:query "" :params nil} + (let [cur (-> state :current)] + (or (nil? cur) + (= "" (get-in state [:aliases cur :table])))) {:query "" :params nil} (= type :delete-action) (build-delete-query state) (= type :update-action) {:queries (build-update-queries state)} (= type :update-partial) {:queries (build-update-queries state)} diff --git a/src/pine/parser.clj b/src/pine/parser.clj index bc76da8..2c6deea 100644 --- a/src/pine/parser.clj +++ b/src/pine/parser.clj @@ -458,8 +458,16 @@ grammar (slurp file)] (insta/parser grammar))) -(defn- normalize-ops [[_ & ops]] - (mapv (fn [[_ op]] (-normalize-op op)) ops)) +(defmethod -normalize-op :ASSIGN [[_ [_ varname]]] + {:type :assign :value varname}) + +(defn- normalize-ops [[_ & nodes]] + (mapv (fn [[_ op]] (-normalize-op op)) + (filter #(= (first %) :OPERATION) nodes))) + +(defn- extract-assign [[_ & nodes]] + (when-let [node (first (filter #(= (first %) :ASSIGN) nodes))] + (let [[_ [_ varname]] node] varname))) (defn parse "Parse an expression and return the normalized operations or failure as a string" @@ -470,7 +478,8 @@ (let [failure (insta/get-failure result) error (with-out-str (println (insta/get-failure result)))] {:error error :failure failure}) - {:result (normalize-ops result)}))) + {:result (normalize-ops result) + :assign (extract-assign result)}))) (defn parse-or-fail [expression] (-> expression parser normalize-ops)) diff --git a/src/pine/pine.bnf b/src/pine/pine.bnf index ef6e597..2e1810f 100644 --- a/src/pine/pine.bnf +++ b/src/pine/pine.bnf @@ -1,4 +1,5 @@ -OPERATIONS := OPERATION (<"|"> OPERATION )* +OPERATIONS := OPERATION (<"|"> OPERATION )* ASSIGN? +ASSIGN := <"|"> <"="> symbol OPERATION := SELECT-PARTIAL | SELECT | TABLE | WHERE-PARTIAL | WHERE | LIMIT| FROM | ORDER-PARTIAL | ORDER | COUNT | DELETE-ACTION | DELETE | UPDATE-PARTIAL | UPDATE-ACTION | GROUP diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 8090920..48f1caf 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -14,6 +14,20 @@ (ast/generate :test) eval/build-query)) +(defn- generate-expressions + "Evaluate a list of pine expressions sequentially, threading variables. + Returns the SQL of the last expression." + [expressions] + (let [{:keys [last-state]} + (reduce (fn [{:keys [variables]} expr] + (let [{:keys [result assign]} (parser/parse expr) + state (ast/generate result :test nil nil variables assign)] + {:variables (if assign (assoc variables assign state) variables) + :last-state state})) + {:variables {} :last-state nil} + expressions)] + (eval/build-query last-state))) + (deftest test-build-query (testing "qualify table" @@ -70,7 +84,7 @@ :params (map dt/number ["1"])} (generate "company | where: id != 1"))) (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"id\" IS NULL LIMIT 250", - :params []} + :params nil} (generate "company | where: id is null")))) (testing "Condition : !=" @@ -85,24 +99,24 @@ (testing "Condition : columns" (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"name\" = \"country\" LIMIT 250", - :params (map dt/string [])} + :params nil} (generate "company | where: name = country"))) (is (= {:query "SELECT \"c\".id AS \"__c__id\", \"c\".* FROM \"company\" AS \"c\" WHERE \"c\".\"name\" != \"c\".\"country\" LIMIT 250", - :params (map dt/string [])} + :params nil} (generate "company as c | name != c.country"))) (is (= {:query "SELECT \"c\".id AS \"__c__id\", \"c\".* FROM \"company\" AS \"c\" WHERE \"c\".\"name\" != \"c\".\"country\" LIMIT 250", - :params (map dt/string [])} + :params nil} (generate "company as c | c.name != c.country")))) (testing "Condition : NULL" (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"deleted_at\" IS NULL LIMIT 250", - :params []} + :params nil} (generate "company | where: deleted_at is null"))) (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"deleted_at\" IS NOT NULL LIMIT 250", - :params []} + :params nil} (generate "company | where: deleted_at is not null"))) (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"deleted_at\" IS NULL LIMIT 250", - :params []} + :params nil} (generate "company | where: deleted_at = null")))) (testing "Condition with cast" @@ -379,4 +393,35 @@ (generate "company /* get by name */ | where: name = 'Acme' -- exact match"))) (is (= {:query "SELECT \"c_0\".\"id\", \"e_1\".\"name\", \"c_0\".id AS \"__c_0__id\", \"e_1\".id AS \"__e_1__id\" FROM \"company\" AS \"c_0\" JOIN \"employee\" AS \"e_1\" ON \"c_0\".\"id\" = \"e_1\".\"company_id\" LIMIT 250", :params nil} - (generate "-- companies and employees\ncompany | s: id /* company id */ | employee | s: name -- employee name")))) \ No newline at end of file + (generate "-- companies and employees\ncompany | s: id /* company id */ | employee | s: name -- employee name")))) + +(deftest test-variables + (testing "Single expression with |= produces normal SQL (assign is metadata)" + (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" LIMIT 250" + :params nil} + (generate-expressions ["company |= active_companies"])))) + + (testing "Variable used as table generates CTE" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"ac_0\".* FROM \"active_companies\" AS \"ac_0\" LIMIT 250" + :params nil} + (generate-expressions ["company |= active_companies" + "active_companies"])))) + + (testing "Variable with WHERE filter generates filtered CTE" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"name\" = ? ) SELECT \"ac_0\".* FROM \"active_companies\" AS \"ac_0\" LIMIT 250" + :params (map dt/string ["Acme"])} + (generate-expressions ["company | where: name = 'Acme' |= active_companies" + "active_companies"])))) + + (testing "Join through a variable resolves correctly" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"e_1\".id AS \"__e_1__id\", \"e_1\".* FROM \"active_companies\" AS \"ac_0\" JOIN \"employee\" AS \"e_1\" ON \"ac_0\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + :params nil} + (generate-expressions ["company |= active_companies" + "active_companies | employee"])))) + + (testing "Composed variables (variable of variable) generates flat CTEs" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"small_active\" AS ( SELECT \"ac_0\".* FROM \"active_companies\" AS \"ac_0\" LIMIT 10 ) SELECT \"sa_0\".* FROM \"small_active\" AS \"sa_0\" LIMIT 250" + :params nil} + (generate-expressions ["company |= active_companies" + "active_companies | l: 10 |= small_active" + "small_active"]))))) \ No newline at end of file diff --git a/test/pine/parser_test.clj b/test/pine/parser_test.clj index 753a5bd..e337a41 100644 --- a/test/pine/parser_test.clj +++ b/test/pine/parser_test.clj @@ -1,6 +1,6 @@ (ns pine.parser-test (:require [clojure.test :refer [deftest is testing]] - [pine.parser :refer [parse-or-fail prettify]] + [pine.parser :refer [parse parse-or-fail prettify]] [pine.data-types :as dt])) (defn- p [e] @@ -361,14 +361,29 @@ (testing "Trailing pipe is preserved" (is (= {:result "company\n | " :operations [{:expression "company" :start 0 :end 7} - {:expression "" :start 9 :end 9}]} + {:expression "" :start 10 :end 10}]} (prettify "company | "))) (is (= {:result "company\n | where: id = 1\n | " :operations [{:expression "company" :start 0 :end 7} {:expression "where: id = 1" :start 10 :end 23} - {:expression "" :start 25 :end 25}]} + {:expression "" :start 26 :end 26}]} (prettify "company | where: id = 1 | ")))) (testing "Incomplete expressions return an error" (is (contains? (prettify "company | w: name = 'test |") :error)))) +(deftest test-assign + (testing "|= sets :assign in parse result" + (is (= "active_companies" (:assign (parse "company |= active_companies")))) + (is (= "active_companies" (:assign (parse "company | = active_companies")))) + (is (nil? (:assign (parse "company"))))) + + (testing ":result ops do not include the assign node" + (is (= 1 (count (:result (parse "company |= active_companies"))))) + (is (= [{:type :table :value {:table "company"}}] + (:result (parse "company |= active_companies"))))) + + (testing "|= works after a full expression" + (is (= "my_var" (:assign (parse "company | where: id = 1 | s: name |= my_var")))) + (is (= 3 (count (:result (parse "company | where: id = 1 | s: name |= my_var"))))))) + From 2bdd0fab517e0b1d99edfaafe01e50a879b22471 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 12:15:56 +0200 Subject: [PATCH 04/60] fix: handle empty/nil expression gracefully; expand variables docs - Guard api-build and api-eval against blank/nil last expression to avoid Instaparse NPE ("text is null") when the input is empty - Add technical implementation section to docs/variables.md Co-Authored-By: Claude Sonnet 4.6 --- docs/variables.md | 32 +++++++++++++++++++++ src/pine/api.clj | 72 +++++++++++++++++++++++------------------------ 2 files changed, 68 insertions(+), 36 deletions(-) diff --git a/docs/variables.md b/docs/variables.md index a46d825..29d241f 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -57,3 +57,35 @@ This produces two CTEs (`active_companies`, `small_active`) and selects from the - Variable names must be valid Pine identifiers (letters, digits, underscores). - A variable used but not defined in a preceding expression will cause a parse/table-not-found error. - Circular references are not supported. + +--- + +## Technical implementation + +### Grammar and parsing + +`|= name` is parsed as an `assign` operation in `pine.bnf`. The parser (`parser.clj`) extracts the assigned name separately from the main operation list and returns it as `{:assign "name" :result [...ops...]}`. This keeps the assign out of the normal operation pipeline so it doesn't interfere with SQL generation. + +### State threading + +The API receives an array of expressions. Each expression is parsed and evaluated in order. The resulting AST state for each assigned expression is stored in a `variables` map keyed by name and threaded into every subsequent `ast/generate` call. This is done purely in memory within the request — there is no persistence layer. + +### CTE generation (`eval.clj`) + +When `build-select-query` runs on a state that has variables, it calls `collect-ctes` to walk the variable map and build `WITH name AS ( ... )` clauses. Each CTE body is generated by `build-cte-body`, which calls `build-bare-select` (no LIMIT). Variable params (e.g. WHERE values) are accumulated alongside the CTE SQL and merged into the outer query's params list. + +### Join resolution through variables + +When a variable is used as a table, its real source tables need to be known so that joins can be resolved (e.g. joining `active_companies` to `employee` requires knowing `active_companies` wraps `company`). The `seed-variable-references` function in `ast/main.clj` copies the schema reference entries from the underlying real tables into the references map under the variable name at build time. + +### Alias disambiguation + +Each table gets an alias derived from its name initials (e.g. `active_companies` → `ac`). Because `make-alias ""` (empty input) also returns `"x"` as a fallback, the empty-expression guard in `build-query` checks the actual table name in the aliases map rather than the alias string, to avoid false matches on a variable named `x`. + +### Auto-id exclusion for CTEs + +Pine normally appends a hidden `id` column per table for row-identity tracking on the frontend. CTEs don't have a stable `id` in the same sense, and including one would cause ambiguous-column errors when the CTE body already selects `*`. So `add-auto-id-columns` (`ast/select.clj`) skips any table whose name appears as a key in the `variables` map. + +### Duplicate column guard in CTE body + +When the user selects explicit columns that include `id` (e.g. `s: id, name`), the CTE body must not also add the auto-id column, since `SELECT id, id` is ambiguous. `build-cte-body` detects whether user-specified columns already cover the current table and drops the auto-id column from the CTE body if so. diff --git a/src/pine/api.clj b/src/pine/api.clj index c1f45db..baffba7 100644 --- a/src/pine/api.clj +++ b/src/pine/api.clj @@ -73,20 +73,22 @@ (let [conn-id (or connection-id @db/connection-id) connection-name (connections/get-connection-name conn-id)] (try - (let [exprs (if (string? expressions) [expressions] expressions) + (let [exprs (if (string? expressions) [expressions] expressions) context-exprs (butlast exprs) - last-expr (last exprs) - {:keys [variables error]} (evaluate-expressions context-exprs conn-id)] - (if error - {:connection-id connection-name :error error} - (let [result (generate-state last-expr cursor conn-id variables) - {state :result build-error :error} result] - (if build-error - {:connection-id connection-name :error build-error} - {:connection-id connection-name - :version version - :query (-> last-expr trim-pipes (generate-state nil conn-id variables) :result eval/build-query eval/formatted-query) - :ast (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges :variables :assign])})))) + last-expr (last exprs)] + (if (str/blank? last-expr) + {:connection-id connection-name} + (let [{:keys [variables error]} (evaluate-expressions context-exprs conn-id)] + (if error + {:connection-id connection-name :error error} + (let [result (generate-state last-expr cursor conn-id variables) + {state :result build-error :error} result] + (if build-error + {:connection-id connection-name :error build-error} + {:connection-id connection-name + :version version + :query (-> last-expr trim-pipes (generate-state nil conn-id variables) :result eval/build-query eval/formatted-query) + :ast (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges :variables :assign])})))))) (catch Exception e {:connection-id connection-name :error (.getMessage e)}))))) @@ -115,24 +117,26 @@ (let [conn-id (or connection-id @db/connection-id) connection-name (connections/get-connection-name conn-id)] (try - (let [exprs (if (string? expressions) [expressions] expressions) - {:keys [last-state error]} (evaluate-expressions exprs conn-id)] - (if error - {:connection-id connection-name :error error} - (let [query (-> last-state eval/build-query eval/formatted-query)] - (try - (let [rows (eval/run-query last-state) - op-type (get-in last-state [:operation :type]) - columns (if (contains? #{:update-action :delete-action} op-type) - (get-columns rows) - (get-columns last-state rows))] - {:connection-id connection-name - :version version - :result rows - :columns columns}) - (catch Exception e {:connection-id connection-name - :error (.getMessage e) - :query query}))))) + (let [exprs (if (string? expressions) [expressions] expressions)] + (if (str/blank? (last exprs)) + {:connection-id connection-name} + (let [{:keys [last-state error]} (evaluate-expressions exprs conn-id)] + (if error + {:connection-id connection-name :error error} + (let [query (-> last-state eval/build-query eval/formatted-query)] + (try + (let [rows (eval/run-query last-state) + op-type (get-in last-state [:operation :type]) + columns (if (contains? #{:update-action :delete-action} op-type) + (get-columns rows) + (get-columns last-state rows))] + {:connection-id connection-name + :version version + :result rows + :columns columns}) + (catch Exception e {:connection-id connection-name + :error (.getMessage e) + :query query}))))))) (catch Exception e {:connection-id connection-name :error (.getMessage e)}))))) @@ -224,11 +228,7 @@ (->> (api-build exprs cursor connection-id) response))) (POST "/api/v1/eval" {params :params} (let [{:keys [expressions expression connection-id]} params - exprs (or expressions - (when expression - (->> (split-expressions expression) - (map :text) - (map trim-pipes))))] + exprs (or expressions (when expression [expression]))] (->> (api-eval exprs connection-id) response))) ;; raw SQL execution From 6c1d1715a4bac2db0421d2f7f47cc524efa05a20 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 16:36:57 +0200 Subject: [PATCH 05/60] fix: variable column hints reflect CTE output, not source table columns When a variable has explicit user columns (select, group, etc.), override the column hint list with the actual CTE output columns rather than all columns of the underlying source tables. For group operations, also add a synthetic "count" column. Variables using * (no explicit columns) fall back to the source table's columns as before. Co-Authored-By: Claude Sonnet 4.6 --- src/pine/ast/main.clj | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index a130202..49f7378 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -68,10 +68,22 @@ ;; - hints :hints {:table [] :select [] :order [] :where [] :update []}}) +(defn- variable-output-columns + "Return the column list a variable's CTE actually exposes, for hint generation. + Returns nil when the CTE selects *, meaning the source table's columns apply." + [var-ast] + (let [user-cols (remove :auto-id (:columns var-ast)) + op-type (-> var-ast :operation :type)] + (when (seq user-cols) + (let [names (distinct (map #(or (:column-alias %) (:column %)) user-cols)) + with-count (if (= op-type :group) (concat names ["count"]) names)] + (mapv #(hash-map :column %) with-count))))) + (defn- seed-variable-references "Copy reference entries from the real source tables of a variable into the local references map under the variable name, so join resolution treats the - variable identically to a real table." + variable identically to a real table. Column hints are overridden with the + CTE's actual output columns when they can be determined." [refs var-ast varname] (let [columns (:columns var-ast) aliases (:aliases var-ast) @@ -82,14 +94,19 @@ (->> explicit (map #(get-in aliases [(:alias %) :table])) (remove nil?) - distinct))] - (reduce (fn [r source-table] - (let [source-refs (get-in r [:table source-table])] - (if source-refs - (update-in r [:table varname] merge source-refs) - r))) - refs - source-tables))) + distinct)) + seeded (reduce (fn [r source-table] + (let [source-refs (get-in r [:table source-table])] + (if source-refs + (update-in r [:table varname] merge source-refs) + r))) + refs + source-tables)] + (if-let [output-cols (variable-output-columns var-ast)] + (-> seeded + (assoc-in [:table varname :columns] output-cols) + (assoc-in [:table varname :column-set] (set output-cols))) + seeded))) (defn pre-handle [state connection-id ops-count expression cursor variables] (let [refs (db/init-references connection-id) From 43ce7ab185505bc75b075ec83c8433723895c783 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 16:41:06 +0200 Subject: [PATCH 06/60] docs: add variable column hints section to variables.md Co-Authored-By: Claude Sonnet 4.6 --- docs/variables.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/variables.md b/docs/variables.md index 29d241f..7f153ff 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -78,6 +78,15 @@ When `build-select-query` runs on a state that has variables, it calls `collect- When a variable is used as a table, its real source tables need to be known so that joins can be resolved (e.g. joining `active_companies` to `employee` requires knowing `active_companies` wraps `company`). The `seed-variable-references` function in `ast/main.clj` copies the schema reference entries from the underlying real tables into the references map under the variable name at build time. +### Column hints for variables + +Column hints (for `select:`, `where:`, etc.) are derived from the references map. For variables, `seed-variable-references` overrides the column list with the CTE's actual output columns rather than leaving the source table's full column list in place: + +- **Explicit columns** (`s:`, `g:`, etc.): the hint list is built from the user-selected columns. The column name used is the `column-alias` if one was set, otherwise the `column` name. For group operations, a synthetic `count` column is appended. +- **No explicit columns** (`*`): the source table's columns are used as-is — the variable inherits the full schema of its underlying table. + +This means typing `x | s:` after `company | g: name |= x` shows `name` and `count`, not all columns of `company`. + ### Alias disambiguation Each table gets an alias derived from its name initials (e.g. `active_companies` → `ac`). Because `make-alias ""` (empty input) also returns `"x"` as a fallback, the empty-expression guard in `build-query` checks the actual table name in the aliases map rather than the alias string, to avoid false matches on a variable named `x`. From b9f7e370c3ab49596b3e5e87e1e7cafd3c513d73 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 16:43:28 +0200 Subject: [PATCH 07/60] =?UTF-8?q?docs:=20add=20pipeline.md=20=E2=80=94=20e?= =?UTF-8?q?xpression=20evaluation=20overview?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 4.6 --- docs/pipeline.md | 147 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 147 insertions(+) create mode 100644 docs/pipeline.md diff --git a/docs/pipeline.md b/docs/pipeline.md new file mode 100644 index 0000000..e7653b9 --- /dev/null +++ b/docs/pipeline.md @@ -0,0 +1,147 @@ +# Evaluation Pipeline + +How a Pine expression goes from text to SQL results. + +``` +user as u | document name="passport" | s: u.email, id | l: 10 +``` + +--- + +## 1. Parse (`parser/parse`) + +The expression is tokenised and parsed against the BNF grammar (`pine.bnf`). +The output is a flat vector of typed operations: + +``` +[ {:type :table, :value {:table "user" :alias "u"}} + {:type :table, :value {:table "document" ...}} + {:type :select, :value [{:column "email" :alias "u"} {:column "id"}]} + {:type :limit, :value 10} ] +``` + +Each operation maps 1-to-1 with a pipe segment. Assignment (`|= name`) is +extracted separately and returned as `:assign` alongside the operation list. + +--- + +## 2. Generate (`ast/generate`) + +The parse result is fed through `ast/generate`, which: + +1. **`pre-handle`** — loads DB schema via `db/init-references` (cached per + connection). If variables are in scope, their FK references are merged in + via `seed-variable-references` so join resolution works. + +2. **`handle-ops`** — folds each operation over an accumulator state, calling + the appropriate handler: + + | Operation | Handler | What it adds to state | + |---|---|---| + | `:table` | `ast/table` | `tables`, `aliases`, `joins`, `context` | + | `:select` | `ast/select` | `columns` | + | `:limit` | `ast/limit` | `limit` | + | `:where` | `ast/where` | `where` | + | `:group` | `ast/group` | `columns`, `group` | + | `:order` | `ast/order` | `order` | + | `:count` | `ast/count` | `operation` | + | `:delete-action` | `ast/delete-action` | `operation` | + | `:update-action` | `ast/update-action` | `update` | + +3. **`post-handle`** — runs after all operations: + - Generates column and table hints (`ast/hints`) + - Appends hidden auto-id columns for row tracking (`ast/select/add-auto-id-columns`) + - Attaches prettified expression and cursor ranges + +The result is a **state map**: + +```clojure +{ :tables [ {:table "user" :alias "u"} {:table "document" :alias "d_1"} ] + :aliases { "u" {:table "user"} "d_1" {:table "document"} } + :context "d_1" + :current "d_1" + :columns [ {:alias "u" :column "email"} {:alias "d_1" :column "id"} ] + :limit 10 + :joins [ ... ] + :where [] + :hints { :table [...] :select [...] } + ... } +``` + +--- + +## 3. Eval (`eval/build-query` and `eval/run-query`) + +The state map is passed to the eval layer, which has two independent paths: + +### Build only (`eval/build-query`) + +Produces `{:query "SELECT ..." :params [...]}` without hitting the database. +Used by the `/api/v1/build` endpoint to return the SQL and AST to the UI. + +The query builder dispatches on operation type: + +| Operation type | Builder | +|---|---| +| default (select) | `build-select-query` | +| `:count` | `build-count-query` | +| `:group` | `build-group-query` | +| `:update-action` | `build-update-queries` | +| `:delete-action` | `build-delete-query` | + +For expressions with variables in scope, `build-select-query` prepends CTE +clauses via `collect-ctes` before the main `SELECT`. + +### Run (`eval/run-query`) + +Calls `build-query` internally, then executes the SQL against the database via +`db/run-query`. Used by the `/api/v1/eval` endpoint. + +--- + +## Full flow (single expression) + +``` +Pine text + │ + ▼ +parser/parse ── BNF grammar → [ ops... ] + │ + ▼ +ast/generate + ├── db/init-references ── loads schema from DB (cached) + ├── handle-ops ── folds ops → state map + └── post-handle ── hints, auto-id columns, prettify + │ + ▼ +state map { tables, aliases, columns, joins, where, limit, ... } + │ + ├──► eval/build-query ── state → { :query "..." :params [...] } + │ + └──► eval/run-query ── build-query + db/run-query → rows +``` + +--- + +## Multi-expression flow (variables) + +When multiple expressions are separated by blank lines, the API evaluates them +left-to-right. Each assigned expression (`|= name`) has its state stored in a +`variables` map. Subsequent expressions receive that map and treat each entry +as a CTE. + +``` +expr 1: company | where: active = true |= active_co + │ + ▼ ast/generate → state-1 (stored as variables["active_co"]) + │ +expr 2: active_co | employee + │ + ▼ ast/generate (variables = {"active_co": state-1}) + └── pre-handle: seeds active_co FK refs from company's schema + └── handle-ops: resolves join active_co → employee + │ + ▼ eval/build-query + └── collect-ctes: WITH "active_co" AS ( SELECT ... FROM company ) + └── main SELECT with JOIN +``` From 812c7514eedd2eeba063c65636fc3d73d429815e Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 16:46:29 +0200 Subject: [PATCH 08/60] =?UTF-8?q?docs:=20rewrite=20pipeline.md=20=E2=80=94?= =?UTF-8?q?=20human-readable=20first,=20technical=20details=20second?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 4.6 --- docs/pipeline.md | 191 +++++++++++++++++++++-------------------------- 1 file changed, 87 insertions(+), 104 deletions(-) diff --git a/docs/pipeline.md b/docs/pipeline.md index e7653b9..8968e25 100644 --- a/docs/pipeline.md +++ b/docs/pipeline.md @@ -1,6 +1,8 @@ # Evaluation Pipeline -How a Pine expression goes from text to SQL results. +A Pine expression goes through three stages before results come back: **parse**, **generate**, and **eval**. + +## Example expression ``` user as u | document name="passport" | s: u.email, id | l: 10 @@ -8,140 +10,121 @@ user as u | document name="passport" | s: u.email, id | l: 10 --- -## 1. Parse (`parser/parse`) +## Stage 1 — Parse -The expression is tokenised and parsed against the BNF grammar (`pine.bnf`). -The output is a flat vector of typed operations: +The text is tokenised against the Pine grammar and turned into a flat list of typed operations — one per pipe segment: ``` -[ {:type :table, :value {:table "user" :alias "u"}} - {:type :table, :value {:table "document" ...}} - {:type :select, :value [{:column "email" :alias "u"} {:column "id"}]} - {:type :limit, :value 10} ] +[ table(user as u), table(document name="passport"), select(u.email, id), limit(10) ] ``` -Each operation maps 1-to-1 with a pipe segment. Assignment (`|= name`) is -extracted separately and returned as `:assign` alongside the operation list. +This list is the input to the next stage. If the expression contains `|= name`, the assigned name is extracted separately and carried alongside the operation list. --- -## 2. Generate (`ast/generate`) +## Stage 2 — Generate -The parse result is fed through `ast/generate`, which: +Each operation is folded over a state map, left to right. Before the first operation runs, the DB schema is loaded so that join paths and column lists are available. Each operation handler adds its contribution to the state: -1. **`pre-handle`** — loads DB schema via `db/init-references` (cached per - connection). If variables are in scope, their FK references are merged in - via `seed-variable-references` so join resolution works. +| Pipe segment | What it contributes | +|---|---| +| `table` | Registers the table, resolves joins, tracks current context | +| `select` / `s:` | Records which columns to include | +| `where` / `w:` | Records filter conditions | +| `group` / `g:` | Records group-by columns and aggregate functions | +| `order` / `o:` | Records sort columns and direction | +| `limit` / `l:` | Records the row limit | +| `count:` | Switches operation to COUNT mode | +| `update!` | Records assignments for an UPDATE | +| `delete!` | Marks as a DELETE operation | + +After all operations are processed, the stage produces a **state map** — a single structure that fully describes the query: tables, joins, column list, filters, limit, and so on. Hints for autocomplete are also computed here, using the DB schema and the current cursor position. -2. **`handle-ops`** — folds each operation over an accumulator state, calling - the appropriate handler: +--- - | Operation | Handler | What it adds to state | - |---|---|---| - | `:table` | `ast/table` | `tables`, `aliases`, `joins`, `context` | - | `:select` | `ast/select` | `columns` | - | `:limit` | `ast/limit` | `limit` | - | `:where` | `ast/where` | `where` | - | `:group` | `ast/group` | `columns`, `group` | - | `:order` | `ast/order` | `order` | - | `:count` | `ast/count` | `operation` | - | `:delete-action` | `ast/delete-action` | `operation` | - | `:update-action` | `ast/update-action` | `update` | +## Stage 3 — Eval -3. **`post-handle`** — runs after all operations: - - Generates column and table hints (`ast/hints`) - - Appends hidden auto-id columns for row tracking (`ast/select/add-auto-id-columns`) - - Attaches prettified expression and cursor ranges +The state map is handed to the eval layer, which has two independent paths: -The result is a **state map**: +**Build** — translates the state map into a SQL string and a parameter list. No database call. Used by the UI to display the query as the user types. -```clojure -{ :tables [ {:table "user" :alias "u"} {:table "document" :alias "d_1"} ] - :aliases { "u" {:table "user"} "d_1" {:table "document"} } - :context "d_1" - :current "d_1" - :columns [ {:alias "u" :column "email"} {:alias "d_1" :column "id"} ] - :limit 10 - :joins [ ... ] - :where [] - :hints { :table [...] :select [...] } - ... } -``` +**Run** — builds the SQL and then executes it against the database, returning rows. --- -## 3. Eval (`eval/build-query` and `eval/run-query`) +## Multi-expression flow (variables) -The state map is passed to the eval layer, which has two independent paths: +When the input contains multiple expressions separated by blank lines, they are evaluated left to right. An expression ending in `|= name` stores its state as a named variable. All subsequent expressions can use that name as a table — Pine injects it as a CTE in the generated SQL. -### Build only (`eval/build-query`) +``` +company | where: active = true |= active_co ← defines active_co -Produces `{:query "SELECT ..." :params [...]}` without hitting the database. -Used by the `/api/v1/build` endpoint to return the SQL and AST to the UI. +active_co | employee ← uses active_co as a CTE +``` -The query builder dispatches on operation type: +Each expression in the sequence sees all variables defined before it. -| Operation type | Builder | -|---|---| -| default (select) | `build-select-query` | -| `:count` | `build-count-query` | -| `:group` | `build-group-query` | -| `:update-action` | `build-update-queries` | -| `:delete-action` | `build-delete-query` | +--- -For expressions with variables in scope, `build-select-query` prepends CTE -clauses via `collect-ctes` before the main `SELECT`. +## Technical implementation -### Run (`eval/run-query`) +### Parse (`parser/parse`, `pine.bnf`) -Calls `build-query` internally, then executes the SQL against the database via -`db/run-query`. Used by the `/api/v1/eval` endpoint. +Instaparse runs the BNF grammar over the input string. The raw parse tree is normalized into a vector of `{:type :value }` maps. `|= name` is handled separately — it is extracted as `:assign` and stripped from the operation list so downstream code is unaware of it. ---- +### Generate (`ast/generate`, `ast/main.clj`) -## Full flow (single expression) +`generate` orchestrates three sub-steps: -``` -Pine text - │ - ▼ -parser/parse ── BNF grammar → [ ops... ] - │ - ▼ -ast/generate - ├── db/init-references ── loads schema from DB (cached) - ├── handle-ops ── folds ops → state map - └── post-handle ── hints, auto-id columns, prettify - │ - ▼ -state map { tables, aliases, columns, joins, where, limit, ... } - │ - ├──► eval/build-query ── state → { :query "..." :params [...] } - │ - └──► eval/run-query ── build-query + db/run-query → rows -``` +1. **`pre-handle`** — seeds the state with the DB schema via `db/init-references` (cached per connection). If variables are in scope, `seed-variable-references` merges their underlying FK references into the schema map so join resolution works through CTEs. ---- +2. **`handle-ops`** — calls `reduce` over the operation list, dispatching each operation to its handler module (`ast/table`, `ast/select`, `ast/where`, `ast/group`, `ast/order`, `ast/limit`, `ast/count`, `ast/delete-action`, `ast/update-action`). Each handler returns a modified state. The operation index (`i`) is threaded through so later stages (hints, auto-id columns) know the order in which columns were introduced. -## Multi-expression flow (variables) +3. **`post-handle`** — runs after all operations: + - `ast/hints/handle` computes autocomplete hints (column hints for `select:`, `where:`, etc.; table hints for join suggestions) using the truncated-at-cursor state so hints reflect what the user has typed so far, not the full expression. + - `ast/select/add-auto-id-columns` appends hidden `id` columns for each real (non-variable) table that has one, used by the UI for row identity tracking. + - `add-prettify` attaches a formatted version of the expression and per-operation character ranges for cursor-based highlighting. -When multiple expressions are separated by blank lines, the API evaluates them -left-to-right. Each assigned expression (`|= name`) has its state stored in a -`variables` map. Subsequent expressions receive that map and treat each entry -as a CTE. +The final state map shape: +```clojure +{ :tables [ {:table "user" :alias "u" :schema nil} + {:table "document" :alias "d_1" :schema nil} ] + :aliases { "u" {:table "user"} + "d_1" {:table "document"} } + :current "d_1" ; alias of the last table in the chain + :context "u" ; alias of the table before the last + :columns [ {:alias "u" :column "email"} + {:alias "d_1" :column "id"} ] + :joins [ ["u" "d_1" nil] ] + :where [] + :limit 10 + :operation {:type :select} + :hints {:table [...] :select [...] :where [...]} + :prettified "user as u\n | document name=\"passport\"\n | s: u.email, id\n | l: 10" + :assign nil } ``` -expr 1: company | where: active = true |= active_co - │ - ▼ ast/generate → state-1 (stored as variables["active_co"]) - │ -expr 2: active_co | employee - │ - ▼ ast/generate (variables = {"active_co": state-1}) - └── pre-handle: seeds active_co FK refs from company's schema - └── handle-ops: resolves join active_co → employee - │ - ▼ eval/build-query - └── collect-ctes: WITH "active_co" AS ( SELECT ... FROM company ) - └── main SELECT with JOIN -``` + +### Eval (`eval/build-query`, `eval/run-query`) + +`build-query` dispatches on `(-> state :operation :type)`: + +| Type | Builder | +|---|---| +| `:select` (default) | `build-select-query` | +| `:count` | `build-count-query` | +| `:group` | `build-group-query` | +| `:update-action` / `:update-partial` | `build-update-queries` | +| `:delete-action` | `build-delete-query` | + +`build-select-query` checks whether the state has variables in scope and, if so, prepends CTE clauses produced by `collect-ctes`. Each CTE body is built by `build-cte-body` using `build-bare-select` (no LIMIT). WHERE params from inside CTEs are collected and merged with the outer query's params. + +`run-query` calls `build-query` then hands the result to `db/run-query`, which executes it against the connection pool and returns rows as a vector of maps. + +### Truncated-state and cursor hints + +To generate accurate autocomplete hints, `generate` builds a second, truncated version of the state by re-parsing the expression cut off at the cursor position. This truncated state is passed to `hints/handle` so the hint context reflects what the user has typed so far — not the full completed expression. + +### Empty-expression guard + +`build-query` checks whether the current table name (looked up from the aliases map) is an empty string. This handles the case where the input is blank — no parse output, no table — and returns `{:query "" :params nil}` rather than attempting to generate a SELECT with a missing table. From 81aaa77fe4a24bc453f09f12e3d36c99b2defa91 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 16:53:36 +0200 Subject: [PATCH 09/60] docs: standardise format and add auto-id and result-updates docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All docs now follow: why → syntax/overview → examples → how it works → constraints → implementation. Add auto-id.md (hidden id columns for row tracking) and result-updates.md (editing cells in join query results). Co-Authored-By: Claude Sonnet 4.6 --- docs/auto-id.md | 74 +++++++++++++++++++++++ docs/pipeline.md | 131 +++++++++++++++++++++-------------------- docs/result-updates.md | 112 +++++++++++++++++++++++++++++++++++ docs/variables.md | 63 ++++++++++++-------- 4 files changed, 289 insertions(+), 91 deletions(-) create mode 100644 docs/auto-id.md create mode 100644 docs/result-updates.md diff --git a/docs/auto-id.md b/docs/auto-id.md new file mode 100644 index 0000000..f6bd1c3 --- /dev/null +++ b/docs/auto-id.md @@ -0,0 +1,74 @@ +# Auto-ID Columns + +Hidden columns Pine adds to track the primary key of each table in a result set. + +## Why + +When a Pine expression joins multiple tables, each row in the result contains data from more than one table. If you want to update or delete a cell in that row, Pine needs to know two things: which table that column belongs to, and what that table's row ID is. Without this, a join result is read-only — you can see the data but can't target a specific row for mutation. + +Auto-id columns carry this information. For each table in the query, Pine appends a hidden column `table_alias.id AS "__alias__id"`. The UI reads these to build the correct `WHERE id = ?` clause when a cell is edited. + +## Example + +``` +user as u | document +``` + +The result includes two visible tables and two hidden auto-id columns: + +```sql +SELECT "u".id AS "__u__id", + "d_0".id AS "__d_0__id", + "d_0".* +FROM "user" AS "u" +JOIN "document" AS "d_0" ON "u"."id" = "d_0"."user_id" +LIMIT 250 +``` + +`__u__id` tracks the `user` row, `__d_0__id` tracks the `document` row. Both are hidden in the UI by default but available to the update logic. + +## How it works + +- For every table in the query that has an `id` column, Pine appends a hidden auto-id column: `alias.id AS "__alias__id"`. +- The column is marked `hidden: true` so the UI doesn't display it in the grid. +- The column is marked `auto-id: true` so internal logic can identify and handle it separately from user-selected columns. +- The UI maintains a `colIndexToAliasLookup` and `aliasToIdLookup` map. When a cell is edited, the UI finds the table alias for that column, looks up the corresponding auto-id column index, reads the ID value from that row, and constructs the update expression. +- Variable tables (CTEs) are excluded — they don't have a stable `id` and their underlying table's auto-id is already tracked separately. See [variables.md](variables.md). + +## Constraints + +- Only tables with an `id` column in the DB schema get an auto-id. Tables without `id` are skipped. +- If the user explicitly selects `id` (e.g. `s: id, name`), the auto-id column is still added but with its `__alias__id` alias to avoid ambiguity. +- Auto-id columns are not included in CTE bodies — adding them when `SELECT *` is already in the CTE would create a duplicate `id` column, causing a SQL error. + +--- + +## Implementation + +### Adding auto-id columns (`ast/select.clj`) + +`add-auto-id-columns` runs in `post-handle` after all operations are applied. It iterates over all table aliases in the state and, for each alias whose underlying table has an `id` column (checked via `has-id-column?` against the references map), appends a column entry: + +```clojure +{ :column "id" + :alias alias + :column-alias "__alias__id" + :hidden true + :auto-id true + :operation-index N } +``` + +Variable tables are excluded: if the table name for an alias is a key in `state :variables`, no auto-id is added. + +### CTE body deduplication (`eval.clj`) + +`build-cte-body` generates the inner SQL for a variable's CTE using `build-bare-select`. Before doing so, it checks whether the user's explicit columns already include a column for the current table. If they do, the auto-id column (which would generate `alias.id`) is dropped to avoid `SELECT id, ..., id` duplication. + +### Frontend column metadata (`default.plugin.tsx`, `Result.tsx`) + +After a query runs, the UI builds two lookup maps from the column metadata: + +- `colIndexToAliasLookup` — maps column position → table alias +- `aliasToIdLookup` — maps table alias → position of its auto-id column + +When a cell is edited, `Result.tsx` uses these to find the right ID value in the current row and constructs an `update!` expression that Pine evaluates against the database. See [result-updates.md](result-updates.md). diff --git a/docs/pipeline.md b/docs/pipeline.md index 8968e25..6e928f6 100644 --- a/docs/pipeline.md +++ b/docs/pipeline.md @@ -1,113 +1,120 @@ # Evaluation Pipeline -A Pine expression goes through three stages before results come back: **parse**, **generate**, and **eval**. +How a Pine expression goes from text to SQL results. -## Example expression +## Why + +Pine is a pipe-based DSL that compiles to SQL. Understanding the pipeline helps when debugging unexpected query output, tracing where hints come from, or extending the language with new operations. + +## Overview + +Every expression passes through three stages: ``` -user as u | document name="passport" | s: u.email, id | l: 10 +text → parse → generate → eval ``` ---- +**Parse** turns the text into a list of typed operations. +**Generate** folds those operations into a state map, using the DB schema to resolve joins and compute hints. +**Eval** translates the state map into SQL, and optionally runs it against the database. + +## Stages -## Stage 1 — Parse +### 1. Parse -The text is tokenised against the Pine grammar and turned into a flat list of typed operations — one per pipe segment: +The expression is tokenised against the Pine grammar and turned into a flat list of typed operations — one per pipe segment. +Input: +``` +user as u | document name="passport" | s: u.email, id | l: 10 +``` + +Output: ``` [ table(user as u), table(document name="passport"), select(u.email, id), limit(10) ] ``` -This list is the input to the next stage. If the expression contains `|= name`, the assigned name is extracted separately and carried alongside the operation list. +Each operation has a type and a value. If the expression ends with `|= name`, that name is extracted separately and carried alongside the list. ---- +### 2. Generate -## Stage 2 — Generate +Each operation is applied left to right to a state map, using the appropriate handler. Before the first operation runs, the DB schema is loaded so that join paths and column lists are available. -Each operation is folded over a state map, left to right. Before the first operation runs, the DB schema is loaded so that join paths and column lists are available. Each operation handler adds its contribution to the state: - -| Pipe segment | What it contributes | +| Pipe segment | What it adds to state | |---|---| -| `table` | Registers the table, resolves joins, tracks current context | +| `table` | Registers the table, resolves FK joins, tracks current context | | `select` / `s:` | Records which columns to include | -| `where` / `w:` | Records filter conditions | +| `where` / `w:` | Records filter conditions and parameters | | `group` / `g:` | Records group-by columns and aggregate functions | | `order` / `o:` | Records sort columns and direction | | `limit` / `l:` | Records the row limit | -| `count:` | Switches operation to COUNT mode | -| `update!` | Records assignments for an UPDATE | -| `delete!` | Marks as a DELETE operation | - -After all operations are processed, the stage produces a **state map** — a single structure that fully describes the query: tables, joins, column list, filters, limit, and so on. Hints for autocomplete are also computed here, using the DB schema and the current cursor position. +| `count:` | Switches operation mode to COUNT | +| `update!` / `u!` | Records column assignments for UPDATE | +| `delete!` | Marks the operation as DELETE | ---- +After all operations are applied, a **post-processing** step runs: autocomplete hints are computed, hidden auto-id columns are appended for row tracking, and a prettified form of the expression is attached. -## Stage 3 — Eval +The result is a single **state map** that fully describes the query. -The state map is handed to the eval layer, which has two independent paths: +### 3. Eval -**Build** — translates the state map into a SQL string and a parameter list. No database call. Used by the UI to display the query as the user types. +The state map is handed to the eval layer, which has two paths: -**Run** — builds the SQL and then executes it against the database, returning rows. - ---- +**Build** — translates the state into a SQL string and parameter list. No database call. Used to show the query in the UI as the user types. -## Multi-expression flow (variables) +**Run** — builds the SQL then executes it against the database, returning rows. -When the input contains multiple expressions separated by blank lines, they are evaluated left to right. An expression ending in `|= name` stores its state as a named variable. All subsequent expressions can use that name as a table — Pine injects it as a CTE in the generated SQL. +## How it works -``` -company | where: active = true |= active_co ← defines active_co +- The pipeline is stateless per request — each call to the API runs the full parse → generate → eval cycle from scratch. +- Hints are computed inside `generate`, using a second pass over the expression truncated at the cursor position. This gives hints that reflect what the user has typed so far, not the full completed expression. +- When multiple expressions are present (separated by blank lines), they are evaluated left to right. Each assigned expression (`|= name`) stores its state as a variable, threaded into subsequent calls. See [variables.md](variables.md). -active_co | employee ← uses active_co as a CTE -``` +## Constraints -Each expression in the sequence sees all variables defined before it. +- The pipeline is synchronous and single-pass — there is no incremental or lazy evaluation. +- All schema information is resolved at generate time. A table or column that doesn't exist in the DB schema will fail at that stage, not at SQL execution time. --- -## Technical implementation +## Implementation ### Parse (`parser/parse`, `pine.bnf`) -Instaparse runs the BNF grammar over the input string. The raw parse tree is normalized into a vector of `{:type :value }` maps. `|= name` is handled separately — it is extracted as `:assign` and stripped from the operation list so downstream code is unaware of it. +Instaparse runs the BNF grammar over the input string. The raw parse tree is normalised into a vector of `{:type :value }` maps. `|= name` is extracted as `:assign` and stripped from the operation list so the rest of the pipeline is unaware of it. ### Generate (`ast/generate`, `ast/main.clj`) `generate` orchestrates three sub-steps: -1. **`pre-handle`** — seeds the state with the DB schema via `db/init-references` (cached per connection). If variables are in scope, `seed-variable-references` merges their underlying FK references into the schema map so join resolution works through CTEs. +**`pre-handle`** — seeds the initial state with the DB schema via `db/init-references` (cached per connection). If variables are in scope, `seed-variable-references` merges their underlying FK references so join resolution works through CTEs. -2. **`handle-ops`** — calls `reduce` over the operation list, dispatching each operation to its handler module (`ast/table`, `ast/select`, `ast/where`, `ast/group`, `ast/order`, `ast/limit`, `ast/count`, `ast/delete-action`, `ast/update-action`). Each handler returns a modified state. The operation index (`i`) is threaded through so later stages (hints, auto-id columns) know the order in which columns were introduced. +**`handle-ops`** — calls `reduce` over the operation list, dispatching each operation to its handler module (`ast/table`, `ast/select`, `ast/where`, `ast/group`, `ast/order`, `ast/limit`, `ast/count`, `ast/delete-action`, `ast/update-action`). The operation index (`i`) is threaded through so later stages know the order columns were introduced. -3. **`post-handle`** — runs after all operations: - - `ast/hints/handle` computes autocomplete hints (column hints for `select:`, `where:`, etc.; table hints for join suggestions) using the truncated-at-cursor state so hints reflect what the user has typed so far, not the full expression. - - `ast/select/add-auto-id-columns` appends hidden `id` columns for each real (non-variable) table that has one, used by the UI for row identity tracking. - - `add-prettify` attaches a formatted version of the expression and per-operation character ranges for cursor-based highlighting. +**`post-handle`** — runs after all operations: +- `ast/hints/handle` computes autocomplete hints using the truncated-at-cursor state. +- `ast/select/add-auto-id-columns` appends hidden `id` columns for each real table. See [auto-id.md](auto-id.md). +- `add-prettify` attaches a formatted expression and per-operation character ranges for cursor highlighting. -The final state map shape: +The final state map: ```clojure -{ :tables [ {:table "user" :alias "u" :schema nil} - {:table "document" :alias "d_1" :schema nil} ] - :aliases { "u" {:table "user"} - "d_1" {:table "document"} } - :current "d_1" ; alias of the last table in the chain - :context "u" ; alias of the table before the last - :columns [ {:alias "u" :column "email"} - {:alias "d_1" :column "id"} ] - :joins [ ["u" "d_1" nil] ] - :where [] - :limit 10 +{ :tables [ {:table "user" :alias "u"} + {:table "document" :alias "d_1"} ] + :aliases { "u" {:table "user"} "d_1" {:table "document"} } + :current "d_1" + :context "u" + :columns [ {:alias "u" :column "email"} {:alias "d_1" :column "id"} ] + :joins [ ["u" "d_1" nil] ] + :where [] + :limit 10 :operation {:type :select} - :hints {:table [...] :select [...] :where [...]} - :prettified "user as u\n | document name=\"passport\"\n | s: u.email, id\n | l: 10" - :assign nil } + :hints {:table [...] :select [...] :where [...]} } ``` ### Eval (`eval/build-query`, `eval/run-query`) -`build-query` dispatches on `(-> state :operation :type)`: +`build-query` dispatches on operation type: | Type | Builder | |---|---| @@ -117,14 +124,8 @@ The final state map shape: | `:update-action` / `:update-partial` | `build-update-queries` | | `:delete-action` | `build-delete-query` | -`build-select-query` checks whether the state has variables in scope and, if so, prepends CTE clauses produced by `collect-ctes`. Each CTE body is built by `build-cte-body` using `build-bare-select` (no LIMIT). WHERE params from inside CTEs are collected and merged with the outer query's params. - -`run-query` calls `build-query` then hands the result to `db/run-query`, which executes it against the connection pool and returns rows as a vector of maps. - -### Truncated-state and cursor hints - -To generate accurate autocomplete hints, `generate` builds a second, truncated version of the state by re-parsing the expression cut off at the cursor position. This truncated state is passed to `hints/handle` so the hint context reflects what the user has typed so far — not the full completed expression. +`build-select-query` checks for variables in scope and, if present, prepends CTE clauses via `collect-ctes`. `run-query` calls `build-query` then executes against the connection pool via `db/run-query`. ### Empty-expression guard -`build-query` checks whether the current table name (looked up from the aliases map) is an empty string. This handles the case where the input is blank — no parse output, no table — and returns `{:query "" :params nil}` rather than attempting to generate a SELECT with a missing table. +`build-query` checks whether the current table name (looked up via the aliases map) is an empty string. This handles blank input — returns `{:query "" :params nil}` rather than generating a malformed SELECT. diff --git a/docs/result-updates.md b/docs/result-updates.md new file mode 100644 index 0000000..5d121bc --- /dev/null +++ b/docs/result-updates.md @@ -0,0 +1,112 @@ +# Updating Results from Join Queries + +Editing cells in multi-table result sets. + +## Why + +A Pine expression like `user | document` returns a result set that mixes columns from two tables. You might want to edit a document's `name` field directly in that result. The challenge is that a naive `UPDATE document SET name = ?` doesn't know which specific document row to target — it needs the `id` of the document for the row you edited. + +Pine solves this with two mechanisms working together: auto-id columns that track each table's primary key in every result row, and `update!` which generates separate UPDATE statements per table. This means you can edit any column in a join result and Pine will route the update to the correct table automatically. + +## Syntax + +``` + | update! = + | u! = +``` + +Multiple assignments, optionally across different tables: + +``` +user as u | document | u! u.name = 'Alice', title = 'Passport' +``` + +## Examples + +### Single-table update + +``` +company | where: id = 42 | u! name = 'Acme Corp' +``` + +```sql +UPDATE "company" SET "name" = ? WHERE id IN ( + SELECT "c_0"."id" FROM "company" AS "c_0" WHERE "c_0"."id" = 42 +) +``` + +### Multi-table update + +``` +user as u | document | u! u.name = 'Alice', title = 'Passport' +``` + +Pine generates one UPDATE per table: + +```sql +UPDATE "user" SET "name" = ? WHERE id IN ( + SELECT "u"."id" FROM "user" AS "u" JOIN "document" AS "d_0" ON ... +) + +UPDATE "document" SET "title" = ? WHERE id IN ( + SELECT "d_0"."id" FROM "user" AS "u" JOIN "document" AS "d_0" ON ... +) +``` + +Each subquery targets only the relevant table's `id`, scoped by the full join condition. + +### Inline cell edit + +When you edit a cell directly in the result grid, the UI constructs an `update!` expression automatically: + +1. The edited cell belongs to column index N. +2. `colIndexToAliasLookup[N]` → table alias (e.g. `"d_0"`). +3. `aliasToIdLookup["d_0"]` → index of `__d_0__id` in the row. +4. The ID value is read from the current row at that index. +5. Pine evaluates: ` | where: d_0.id = | u! = ` + +## How it works + +- **Column qualification**: unqualified columns (e.g. `name`) default to the last table in the expression. Qualified columns (`u.name`) target the specified alias. +- **One UPDATE per table**: assignments are grouped by target alias. Each group becomes an independent UPDATE with a subquery to identify the rows. +- **Subquery for row targeting**: rather than a direct `WHERE id = ?`, Pine uses `WHERE id IN (SELECT id FROM ... JOIN ...)`. This ensures the update respects the full join condition, not just a naked ID lookup. +- **Auto-id columns enable inline editing**: hidden `__alias__id` columns in every result row carry the primary key for each table. The UI uses them to route cell edits to the right UPDATE. See [auto-id.md](auto-id.md). + +## Constraints + +- The target tables must have an `id` column. Tables without `id` cannot be targeted by `update!`. +- Only one value per column per `update!` expression. +- `update!` on a variable/CTE table is not supported — the CTE has no physical table to update. + +--- + +## Implementation + +### Parsing (`pine.bnf`, `parser.clj`) + +`update!` and `u!` are parsed as `:update-action` operations. Each assignment in the value list carries a `{:column {:alias ... :column ...} :value {...}}` map. Partial typing (`u! col`) is parsed as `:update-partial` for hint generation. + +### AST (`ast/update-action.clj`) + +`handle` records the assignments in `state :update`. Column aliases default to the current table alias when not explicitly qualified. + +### SQL generation (`eval/build-update-queries`) + +Assignments are grouped by their target alias. For each group, `build-single-update-query` produces: + +```sql +UPDATE "schema"."table" SET "col" = ? +WHERE id IN ( SELECT "alias"."id" FROM ... ) +``` + +The subquery is built by temporarily replacing the state's column list with just `[{:column "id" :alias update-alias}]` and calling `build-select-query`, so it inherits the full JOIN and WHERE conditions from the original expression. + +### Frontend cell editing (`Result.tsx`) + +`processRowUpdate` is called by MUI DataGrid when a cell is committed. It: + +1. Finds the changed field (column index). +2. Looks up the table alias via `colIndexToAliasLookup`. +3. Looks up the auto-id column index via `aliasToIdLookup`. +4. Reads the row ID from the current row data. +5. Builds an update expression via `createUpdateExpression` and evaluates it in a virtual session, then refreshes the main session. diff --git a/docs/variables.md b/docs/variables.md index 7f153ff..2aa5b8c 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -1,6 +1,10 @@ # Variables -Variables let you name and reuse intermediate query results across expressions. They compile to CTEs (`WITH` clauses) in the generated SQL. +Name and reuse intermediate query results across expressions. + +## Why + +Complex queries often need to build up results in steps — filter a set of rows, then join that filtered set to another table, then aggregate. Without variables, the only way to do this is with nested subqueries, which are hard to read and hard to iterate on in a REPL-style tool. Variables let you give each step a name, treat it as a table, and compose them incrementally. ## Syntax @@ -8,9 +12,11 @@ Variables let you name and reuse intermediate query results across expressions. |= ``` -Append `|= name` at the end of any pipe chain to assign the result to a variable. +Append `|= name` at the end of any pipe chain to assign the result to a variable. Use the name as a table in any later expression. + +## Examples -## Example +### Basic ``` company | where: active = true |= active_companies @@ -18,7 +24,7 @@ company | where: active = true |= active_companies active_companies | employee ``` -The first expression defines `active_companies`. The second expression uses it as a table — Pine expands it into a CTE: +`active_companies` becomes a CTE. The second expression joins through it: ```sql WITH "active_companies" AS ( @@ -30,9 +36,9 @@ JOIN "employee" AS "e_1" ON "ac_0"."id" = "e_1"."company_id" LIMIT 250 ``` -## Multi-expression input +### Chained -Expressions are separated by blank lines. Each expression is evaluated in order; earlier expressions provide context (as CTEs) for later ones. Only the last expression is executed and returned. +Expressions are separated by blank lines and evaluated in order. Each one can build on the last. ``` company | where: active = true |= active_companies @@ -42,25 +48,36 @@ active_companies | l: 10 |= small_active small_active ``` -This produces two CTEs (`active_companies`, `small_active`) and selects from the last. +This produces two CTEs and selects from the final one. + +### Grouped + +``` +tenant as t | public.company .tenantId | g: t.title |= x + +x +``` + +`x` exposes only the columns the CTE actually produces — here `title` and `count` — so hints for `x | s:` show just those two columns. ## How it works -- **Assignment** (`|= name`): the expression is stored as a variable. Its SQL becomes the body of a CTE named `name`. -- **Usage**: when `name` appears as a table in a later expression, Pine substitutes the CTE. Joins through variable tables resolve using the schema of the underlying real table(s). +- **Assignment** (`|= name`): the expression is stored as a variable. Its SQL becomes the body of a CTE. +- **Usage**: when `name` appears as a table in a later expression, Pine substitutes the CTE. Joins through variable tables resolve using the schema of the underlying real tables. - **Scoping**: variables accumulate left-to-right. Each expression sees all variables defined before it. -- **Auto-id columns**: Pine normally adds a hidden `id` column per table for internal row tracking. Variable tables (CTEs) are excluded — only real database tables get auto-id columns. -- **CTE body**: the CTE body is generated without a `LIMIT` (limits only apply to the outer query). +- **Column hints**: when a variable has explicit output columns (from `s:`, `g:`, etc.), hints reflect those columns, not the full schema of the source table. Variables using `*` (no explicit selection) inherit the source table's columns. +- **No LIMIT in CTE body**: limits only apply to the outer query. +- **No auto-id columns**: Pine's hidden `id` tracking columns are not added for variable tables. See [auto-id.md](auto-id.md). ## Constraints - Variable names must be valid Pine identifiers (letters, digits, underscores). -- A variable used but not defined in a preceding expression will cause a parse/table-not-found error. +- A variable used but not defined in a preceding expression causes a table-not-found error. - Circular references are not supported. --- -## Technical implementation +## Implementation ### Grammar and parsing @@ -72,29 +89,23 @@ The API receives an array of expressions. Each expression is parsed and evaluate ### CTE generation (`eval.clj`) -When `build-select-query` runs on a state that has variables, it calls `collect-ctes` to walk the variable map and build `WITH name AS ( ... )` clauses. Each CTE body is generated by `build-cte-body`, which calls `build-bare-select` (no LIMIT). Variable params (e.g. WHERE values) are accumulated alongside the CTE SQL and merged into the outer query's params list. +When `build-select-query` runs on a state that has variables, it calls `collect-ctes` to walk the variable map and build `WITH name AS ( ... )` clauses. Each CTE body is generated by `build-cte-body`, which calls `build-bare-select` (no LIMIT). WHERE params from inside CTEs are accumulated and merged into the outer query's params list. ### Join resolution through variables -When a variable is used as a table, its real source tables need to be known so that joins can be resolved (e.g. joining `active_companies` to `employee` requires knowing `active_companies` wraps `company`). The `seed-variable-references` function in `ast/main.clj` copies the schema reference entries from the underlying real tables into the references map under the variable name at build time. +When a variable is used as a table, its real source tables need to be known so that joins can be resolved (e.g. joining `active_companies` to `employee` requires knowing `active_companies` wraps `company`). `seed-variable-references` in `ast/main.clj` copies the FK reference entries from the underlying real tables into the references map under the variable name at build time. ### Column hints for variables -Column hints (for `select:`, `where:`, etc.) are derived from the references map. For variables, `seed-variable-references` overrides the column list with the CTE's actual output columns rather than leaving the source table's full column list in place: +`seed-variable-references` also overrides the column list for the variable entry in the references map: -- **Explicit columns** (`s:`, `g:`, etc.): the hint list is built from the user-selected columns. The column name used is the `column-alias` if one was set, otherwise the `column` name. For group operations, a synthetic `count` column is appended. -- **No explicit columns** (`*`): the source table's columns are used as-is — the variable inherits the full schema of its underlying table. - -This means typing `x | s:` after `company | g: name |= x` shows `name` and `count`, not all columns of `company`. +- **Explicit columns** (`s:`, `g:`, etc.): the column list is built from the user-selected columns. The name used is `column-alias` if set, otherwise `column`. Group operations append a synthetic `count` entry. +- **No explicit columns** (`*`): the source table's full column list is inherited. ### Alias disambiguation -Each table gets an alias derived from its name initials (e.g. `active_companies` → `ac`). Because `make-alias ""` (empty input) also returns `"x"` as a fallback, the empty-expression guard in `build-query` checks the actual table name in the aliases map rather than the alias string, to avoid false matches on a variable named `x`. - -### Auto-id exclusion for CTEs - -Pine normally appends a hidden `id` column per table for row-identity tracking on the frontend. CTEs don't have a stable `id` in the same sense, and including one would cause ambiguous-column errors when the CTE body already selects `*`. So `add-auto-id-columns` (`ast/select.clj`) skips any table whose name appears as a key in the `variables` map. +Each table gets an alias derived from its name initials (`active_companies` → `ac`). Because `make-alias ""` (empty input) also returns `"x"` as a fallback, the empty-expression guard in `build-query` checks the actual table name in the aliases map rather than the alias string, to avoid false matches on a variable named `x`. ### Duplicate column guard in CTE body -When the user selects explicit columns that include `id` (e.g. `s: id, name`), the CTE body must not also add the auto-id column, since `SELECT id, id` is ambiguous. `build-cte-body` detects whether user-specified columns already cover the current table and drops the auto-id column from the CTE body if so. +When the user selects explicit columns that include `id` (e.g. `s: id, name`), the CTE body must not also emit the auto-id column, since `SELECT id, id` is ambiguous. `build-cte-body` detects whether user-specified columns already cover the current table and drops the auto-id column from the CTE body if so. From 9b62bf086f922c9c20a23913122dfd8bca19c4f0 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 17:03:32 +0200 Subject: [PATCH 10/60] docs: merge auto-id.md into result-updates.md; update cross-references auto-id is an implementation detail of result-updates, not a standalone concept. Co-Authored-By: Claude Sonnet 4.6 --- docs/auto-id.md | 74 ------------------------------------ docs/pipeline.md | 2 +- docs/result-updates.md | 86 +++++++++++++++++++++++++++++------------- docs/variables.md | 2 +- 4 files changed, 62 insertions(+), 102 deletions(-) delete mode 100644 docs/auto-id.md diff --git a/docs/auto-id.md b/docs/auto-id.md deleted file mode 100644 index f6bd1c3..0000000 --- a/docs/auto-id.md +++ /dev/null @@ -1,74 +0,0 @@ -# Auto-ID Columns - -Hidden columns Pine adds to track the primary key of each table in a result set. - -## Why - -When a Pine expression joins multiple tables, each row in the result contains data from more than one table. If you want to update or delete a cell in that row, Pine needs to know two things: which table that column belongs to, and what that table's row ID is. Without this, a join result is read-only — you can see the data but can't target a specific row for mutation. - -Auto-id columns carry this information. For each table in the query, Pine appends a hidden column `table_alias.id AS "__alias__id"`. The UI reads these to build the correct `WHERE id = ?` clause when a cell is edited. - -## Example - -``` -user as u | document -``` - -The result includes two visible tables and two hidden auto-id columns: - -```sql -SELECT "u".id AS "__u__id", - "d_0".id AS "__d_0__id", - "d_0".* -FROM "user" AS "u" -JOIN "document" AS "d_0" ON "u"."id" = "d_0"."user_id" -LIMIT 250 -``` - -`__u__id` tracks the `user` row, `__d_0__id` tracks the `document` row. Both are hidden in the UI by default but available to the update logic. - -## How it works - -- For every table in the query that has an `id` column, Pine appends a hidden auto-id column: `alias.id AS "__alias__id"`. -- The column is marked `hidden: true` so the UI doesn't display it in the grid. -- The column is marked `auto-id: true` so internal logic can identify and handle it separately from user-selected columns. -- The UI maintains a `colIndexToAliasLookup` and `aliasToIdLookup` map. When a cell is edited, the UI finds the table alias for that column, looks up the corresponding auto-id column index, reads the ID value from that row, and constructs the update expression. -- Variable tables (CTEs) are excluded — they don't have a stable `id` and their underlying table's auto-id is already tracked separately. See [variables.md](variables.md). - -## Constraints - -- Only tables with an `id` column in the DB schema get an auto-id. Tables without `id` are skipped. -- If the user explicitly selects `id` (e.g. `s: id, name`), the auto-id column is still added but with its `__alias__id` alias to avoid ambiguity. -- Auto-id columns are not included in CTE bodies — adding them when `SELECT *` is already in the CTE would create a duplicate `id` column, causing a SQL error. - ---- - -## Implementation - -### Adding auto-id columns (`ast/select.clj`) - -`add-auto-id-columns` runs in `post-handle` after all operations are applied. It iterates over all table aliases in the state and, for each alias whose underlying table has an `id` column (checked via `has-id-column?` against the references map), appends a column entry: - -```clojure -{ :column "id" - :alias alias - :column-alias "__alias__id" - :hidden true - :auto-id true - :operation-index N } -``` - -Variable tables are excluded: if the table name for an alias is a key in `state :variables`, no auto-id is added. - -### CTE body deduplication (`eval.clj`) - -`build-cte-body` generates the inner SQL for a variable's CTE using `build-bare-select`. Before doing so, it checks whether the user's explicit columns already include a column for the current table. If they do, the auto-id column (which would generate `alias.id`) is dropped to avoid `SELECT id, ..., id` duplication. - -### Frontend column metadata (`default.plugin.tsx`, `Result.tsx`) - -After a query runs, the UI builds two lookup maps from the column metadata: - -- `colIndexToAliasLookup` — maps column position → table alias -- `aliasToIdLookup` — maps table alias → position of its auto-id column - -When a cell is edited, `Result.tsx` uses these to find the right ID value in the current row and constructs an `update!` expression that Pine evaluates against the database. See [result-updates.md](result-updates.md). diff --git a/docs/pipeline.md b/docs/pipeline.md index 6e928f6..6615d92 100644 --- a/docs/pipeline.md +++ b/docs/pipeline.md @@ -93,7 +93,7 @@ Instaparse runs the BNF grammar over the input string. The raw parse tree is nor **`post-handle`** — runs after all operations: - `ast/hints/handle` computes autocomplete hints using the truncated-at-cursor state. -- `ast/select/add-auto-id-columns` appends hidden `id` columns for each real table. See [auto-id.md](auto-id.md). +- `ast/select/add-auto-id-columns` appends hidden `id` columns for each real table. See [result-updates.md](result-updates.md). - `add-prettify` attaches a formatted expression and per-operation character ranges for cursor highlighting. The final state map: diff --git a/docs/result-updates.md b/docs/result-updates.md index 5d121bc..38042dd 100644 --- a/docs/result-updates.md +++ b/docs/result-updates.md @@ -4,9 +4,14 @@ Editing cells in multi-table result sets. ## Why -A Pine expression like `user | document` returns a result set that mixes columns from two tables. You might want to edit a document's `name` field directly in that result. The challenge is that a naive `UPDATE document SET name = ?` doesn't know which specific document row to target — it needs the `id` of the document for the row you edited. +A Pine expression like `user | document` returns a result set that mixes columns from two tables. You might want to edit a document's `name` field directly in that result. The challenge is that a naive `UPDATE document SET name = ?` doesn't know which specific document row to target — it needs the `id` of the document for the row you edited, not just any document. -Pine solves this with two mechanisms working together: auto-id columns that track each table's primary key in every result row, and `update!` which generates separate UPDATE statements per table. This means you can edit any column in a join result and Pine will route the update to the correct table automatically. +Pine solves this with two mechanisms working together: + +- **Auto-id columns**: for each table in the query, Pine silently appends a hidden column `alias.id AS "__alias__id"`. These are never displayed but are always present in the result, so the UI can look up the exact row ID for any cell. +- **`update!`**: generates separate UPDATE statements per table, each scoped to the correct row via a subquery that inherits the full join condition. + +Together they make any join result editable, not just readable. ## Syntax @@ -15,7 +20,7 @@ Pine solves this with two mechanisms working together: auto-id columns that trac | u! = ``` -Multiple assignments, optionally across different tables: +Multiple assignments, optionally targeting different tables: ``` user as u | document | u! u.name = 'Alice', title = 'Passport' @@ -55,58 +60,87 @@ UPDATE "document" SET "title" = ? WHERE id IN ( Each subquery targets only the relevant table's `id`, scoped by the full join condition. +### Auto-id columns in the result + +``` +user as u | document +``` + +The generated SQL includes hidden auto-id columns alongside the visible data: + +```sql +SELECT "u".id AS "__u__id", + "d_0".id AS "__d_0__id", + "d_0".* +FROM "user" AS "u" +JOIN "document" AS "d_0" ON "u"."id" = "d_0"."user_id" +LIMIT 250 +``` + +`__u__id` and `__d_0__id` are hidden in the grid but available when a cell is edited. + ### Inline cell edit When you edit a cell directly in the result grid, the UI constructs an `update!` expression automatically: 1. The edited cell belongs to column index N. 2. `colIndexToAliasLookup[N]` → table alias (e.g. `"d_0"`). -3. `aliasToIdLookup["d_0"]` → index of `__d_0__id` in the row. -4. The ID value is read from the current row at that index. +3. `aliasToIdLookup["d_0"]` → index of the `__d_0__id` column in the row. +4. The ID value is read from that column in the current row. 5. Pine evaluates: ` | where: d_0.id = | u! = ` ## How it works +- **Auto-id columns** are appended automatically for every real table that has an `id` column. They are marked `hidden: true` (not shown in the grid) and `auto-id: true` (so internal logic can identify them). Variable/CTE tables are excluded — they don't have a stable `id`. - **Column qualification**: unqualified columns (e.g. `name`) default to the last table in the expression. Qualified columns (`u.name`) target the specified alias. - **One UPDATE per table**: assignments are grouped by target alias. Each group becomes an independent UPDATE with a subquery to identify the rows. -- **Subquery for row targeting**: rather than a direct `WHERE id = ?`, Pine uses `WHERE id IN (SELECT id FROM ... JOIN ...)`. This ensures the update respects the full join condition, not just a naked ID lookup. -- **Auto-id columns enable inline editing**: hidden `__alias__id` columns in every result row carry the primary key for each table. The UI uses them to route cell edits to the right UPDATE. See [auto-id.md](auto-id.md). +- **Subquery for row targeting**: Pine uses `WHERE id IN (SELECT id FROM ... JOIN ...)` rather than a direct `WHERE id = ?`. This ensures the update respects the full join condition. ## Constraints -- The target tables must have an `id` column. Tables without `id` cannot be targeted by `update!`. -- Only one value per column per `update!` expression. +- Only tables with an `id` column in the DB schema get an auto-id. Tables without `id` cannot be targeted by `update!`. +- If the user explicitly selects `id` (e.g. `s: id, name`), the auto-id column is still added with its `__alias__id` alias to avoid ambiguity. +- Auto-id columns are not added inside CTE bodies — `SELECT *` already includes `id`, and a second `id` would cause an ambiguous-column error. - `update!` on a variable/CTE table is not supported — the CTE has no physical table to update. --- ## Implementation -### Parsing (`pine.bnf`, `parser.clj`) +### Auto-id columns (`ast/select.clj`) -`update!` and `u!` are parsed as `:update-action` operations. Each assignment in the value list carries a `{:column {:alias ... :column ...} :value {...}}` map. Partial typing (`u! col`) is parsed as `:update-partial` for hint generation. +`add-auto-id-columns` runs in `post-handle` after all operations are applied. For each table alias whose underlying table has an `id` column (checked via `has-id-column?`), it appends: -### AST (`ast/update-action.clj`) +```clojure +{ :column "id" + :alias alias + :column-alias "__alias__id" + :hidden true + :auto-id true + :operation-index N } +``` -`handle` records the assignments in `state :update`. Column aliases default to the current table alias when not explicitly qualified. +Variable tables are skipped: if the table name for an alias is a key in `state :variables`, no auto-id is added. -### SQL generation (`eval/build-update-queries`) +### CTE body deduplication (`eval.clj`) -Assignments are grouped by their target alias. For each group, `build-single-update-query` produces: +`build-cte-body` generates the inner SQL for a variable's CTE. Before calling `build-bare-select`, it checks whether user-specified columns already cover the current table. If so, the auto-id column is dropped to avoid `SELECT id, ..., id` duplication. -```sql -UPDATE "schema"."table" SET "col" = ? -WHERE id IN ( SELECT "alias"."id" FROM ... ) -``` +### Frontend column metadata (`default.plugin.tsx`) + +After a query runs, the UI builds two lookup maps from the response column metadata: -The subquery is built by temporarily replacing the state's column list with just `[{:column "id" :alias update-alias}]` and calling `build-select-query`, so it inherits the full JOIN and WHERE conditions from the original expression. +- `colIndexToAliasLookup` — column position → table alias +- `aliasToIdLookup` — table alias → position of its auto-id column in the row ### Frontend cell editing (`Result.tsx`) -`processRowUpdate` is called by MUI DataGrid when a cell is committed. It: +`processRowUpdate` is called by MUI DataGrid when a cell is committed. It uses the two lookup maps to find the row ID, builds an update expression via `createUpdateExpression`, and evaluates it in a virtual session before refreshing the main session. + +### SQL generation (`eval/build-update-queries`) + +Assignments are grouped by target alias. For each group, `build-single-update-query` produces the UPDATE with a subquery built by temporarily replacing the state's column list with `[{:column "id" :alias update-alias}]` and calling `build-select-query` — so it inherits the full JOIN and WHERE conditions from the original expression. + +### Parsing (`pine.bnf`, `parser.clj`) -1. Finds the changed field (column index). -2. Looks up the table alias via `colIndexToAliasLookup`. -3. Looks up the auto-id column index via `aliasToIdLookup`. -4. Reads the row ID from the current row data. -5. Builds an update expression via `createUpdateExpression` and evaluates it in a virtual session, then refreshes the main session. +`update!` / `u!` are parsed as `:update-action`. Each assignment carries `{:column {:alias ... :column ...} :value {...}}`. Partial typing (`u! col`) is parsed as `:update-partial` for hint generation. diff --git a/docs/variables.md b/docs/variables.md index 2aa5b8c..242ed58 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -67,7 +67,7 @@ x - **Scoping**: variables accumulate left-to-right. Each expression sees all variables defined before it. - **Column hints**: when a variable has explicit output columns (from `s:`, `g:`, etc.), hints reflect those columns, not the full schema of the source table. Variables using `*` (no explicit selection) inherit the source table's columns. - **No LIMIT in CTE body**: limits only apply to the outer query. -- **No auto-id columns**: Pine's hidden `id` tracking columns are not added for variable tables. See [auto-id.md](auto-id.md). +- **No auto-id columns**: Pine's hidden `id` tracking columns are not added for variable tables. See [result-updates.md](result-updates.md). ## Constraints From fd0a2143991ecd7889bdde35df28d200af240714 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 17:14:13 +0200 Subject: [PATCH 11/60] fix: variable names appear without schema prefix in table hints Variables don't belong to a DB schema. seed-variable-references copies the source table's :in map, causing variables to display as e.g. "public.active_co" in hints. Now create-hint-from-table checks against state :variables and emits {:schema nil :table name} for any variable, giving a clean unqualified hint. Co-Authored-By: Claude Sonnet 4.6 --- src/pine/ast/hints.clj | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/pine/ast/hints.clj b/src/pine/ast/hints.clj index 248376e..92822ef 100644 --- a/src/pine/ast/hints.clj +++ b/src/pine/ast/hints.clj @@ -14,12 +14,15 @@ ;; --------------------------------------------------------------------------- (defn create-hint-from-table [state tables] - (let [refs (-> state :references :table)] + (let [refs (-> state :references :table) + variables (-> state :variables)] (mapcat identity (for [table tables :let [schemas (->> table refs :in keys)]] - (for [schema schemas] - {:schema schema :table table}))))) + (if (contains? variables table) + [{:schema nil :table table}] + (for [schema schemas] + {:schema schema :table table})))))) (defn table-hints "No context - get all the tables matching the token" From b2c5309abae5805685cf5866c463fce259cfb785 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 10 May 2026 22:47:09 +0200 Subject: [PATCH 12/60] fix: prettify ASSIGN node to avoid double pipe; add tests The ASSIGN span includes the leading "|" from the grammar rule. When prettify joins operations with "\n | ", this produced "| | =name". Strip the leading pipe from ASSIGN expressions before joining. Also add parser tests covering assignment prettification: - bare assignment (|= name) - spaced assignment (| = name) - assignment after filter (where: ... |= name) - trailing pipe after assignment is a parse error (assignment must be last) Co-Authored-By: Claude Sonnet 4.6 --- src/pine/parser.clj | 11 +++++++++-- test/pine/parser_test.clj | 19 ++++++++++++++++++- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/src/pine/parser.clj b/src/pine/parser.clj index 2c6deea..bdc8f2f 100644 --- a/src/pine/parser.clj +++ b/src/pine/parser.clj @@ -497,8 +497,15 @@ {:error (with-out-str (println (insta/get-failure result)))} (let [operations (rest result) op-infos (mapv (fn [op] - (let [[start end] (insta/span op)] - {:expression (s/trim (subs expression start end)) + (let [[start end] (insta/span op) + raw (s/trim (subs expression start end)) + ;; ASSIGN spans include the leading "|" from the + ;; grammar rule. Strip it so the join doesn't + ;; produce a double pipe ("| | =name"). + expr (if (= (first op) :ASSIGN) + (s/trim (s/replace-first raw #"^\|" "")) + raw)] + {:expression expr :start start :end end})) operations)] diff --git a/test/pine/parser_test.clj b/test/pine/parser_test.clj index e337a41..0fcf9f9 100644 --- a/test/pine/parser_test.clj +++ b/test/pine/parser_test.clj @@ -370,7 +370,24 @@ (prettify "company | where: id = 1 | ")))) (testing "Incomplete expressions return an error" - (is (contains? (prettify "company | w: name = 'test |") :error)))) + (is (contains? (prettify "company | w: name = 'test |") :error))) + + (testing "Assignment is formatted without double pipe" + (is (= {:result "company\n | = active_companies" + :operations [{:expression "company" :start 0 :end 7} + {:expression "= active_companies" :start 8 :end 27}]} + (prettify "company |= active_companies"))) + (is (= {:result "company\n | = x" + :operations [{:expression "company" :start 0 :end 7} + {:expression "= x" :start 8 :end 13}]} + (prettify "company | = x"))) + (is (= {:result "company\n | where: active = true\n | = active_companies" + :operations [{:expression "company" :start 0 :end 7} + {:expression "where: active = true" :start 10 :end 30} + {:expression "= active_companies" :start 31 :end 50}]} + (prettify "company | where: active = true |= active_companies"))) + ;; trailing pipe after assignment is a parse error — assignment must be last + (is (contains? (prettify "company |= x | ") :error)))) (deftest test-assign (testing "|= sets :assign in parse result" From a9a357d7d157c05c7528320ec1f12aa2a59f238a Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 11 May 2026 15:41:44 +0200 Subject: [PATCH 13/60] fix: trim trailing pipe in api-eval before building query api-eval was passing the raw last expression (e.g. "tenant |") directly to generate-state/run-query, producing an empty query and a PostgreSQL "No results returned" error. Now mirrors api-build: splits context/last, trims pipes on the last expression, and guards on blank after trimming. Co-Authored-By: Claude Sonnet 4.6 --- src/pine/api.clj | 41 +++++++++++++++++++++++------------------ 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/src/pine/api.clj b/src/pine/api.clj index baffba7..0763711 100644 --- a/src/pine/api.clj +++ b/src/pine/api.clj @@ -63,7 +63,6 @@ (str/replace #"^\|\s*|\s*\|$" "") (str/trim))) - (defn api-build ([expressions] (api-build expressions nil nil)) @@ -117,26 +116,32 @@ (let [conn-id (or connection-id @db/connection-id) connection-name (connections/get-connection-name conn-id)] (try - (let [exprs (if (string? expressions) [expressions] expressions)] - (if (str/blank? (last exprs)) + (let [exprs (if (string? expressions) [expressions] expressions) + context-exprs (butlast exprs) + last-expr (last exprs) + trimmed (trim-pipes (or last-expr ""))] + (if (str/blank? trimmed) {:connection-id connection-name} - (let [{:keys [last-state error]} (evaluate-expressions exprs conn-id)] + (let [{:keys [variables error]} (evaluate-expressions context-exprs conn-id)] (if error {:connection-id connection-name :error error} - (let [query (-> last-state eval/build-query eval/formatted-query)] - (try - (let [rows (eval/run-query last-state) - op-type (get-in last-state [:operation :type]) - columns (if (contains? #{:update-action :delete-action} op-type) - (get-columns rows) - (get-columns last-state rows))] - {:connection-id connection-name - :version version - :result rows - :columns columns}) - (catch Exception e {:connection-id connection-name - :error (.getMessage e) - :query query}))))))) + (let [{last-state :result build-error :error} (generate-state trimmed nil conn-id variables)] + (if build-error + {:connection-id connection-name :error build-error} + (let [query (-> last-state eval/build-query eval/formatted-query)] + (try + (let [rows (eval/run-query last-state) + op-type (get-in last-state [:operation :type]) + columns (if (contains? #{:update-action :delete-action} op-type) + (get-columns rows) + (get-columns last-state rows))] + {:connection-id connection-name + :version version + :result rows + :columns columns}) + (catch Exception e {:connection-id connection-name + :error (.getMessage e) + :query query}))))))))) (catch Exception e {:connection-id connection-name :error (.getMessage e)}))))) From 1c02ed008e83e33b688b9e9a878ee3f4b719dccd Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 11 May 2026 15:49:07 +0200 Subject: [PATCH 14/60] feat: bidirectional reference patching for variables Previously seed-variable-references only gave the variable its own entry in the refs map (inheriting the source table's relationships). This meant 'real-table | variable' and 'variable-x | variable-y' joins could never be resolved because the other side had no knowledge of the variable. New patch-variable-relations pass runs after all variables are seeded: for every entity T (real table or variable) where T[:referred-by][S] exists (S is the variable's source table), T[:referred-by][variable] is also registered with the same via-details. This enables: - T | V and V | T (real table <-> variable) - V | W and W | V (variable <-> variable, when W already inherited S from its own source table via the seeding pass) Co-Authored-By: Claude Sonnet 4.6 --- src/pine/ast/main.clj | 65 +++++++++++++++++++++++++++++++++---------- 1 file changed, 50 insertions(+), 15 deletions(-) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 49f7378..f82cc87 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -79,22 +79,29 @@ with-count (if (= op-type :group) (concat names ["count"]) names)] (mapv #(hash-map :column %) with-count))))) +(defn- get-source-tables + "Return the tables whose columns are exposed by a variable's CTE. + Used by both seeding and bidirectional patching." + [var-ast] + (let [columns (:columns var-ast) + aliases (:aliases var-ast) + explicit (remove :auto-id columns)] + (or (if (empty? explicit) + (when-let [current-alias (:current var-ast)] + [(get-in aliases [current-alias :table])]) + (->> explicit + (map #(get-in aliases [(:alias %) :table])) + (remove nil?) + distinct)) + []))) + (defn- seed-variable-references "Copy reference entries from the real source tables of a variable into the local references map under the variable name, so join resolution treats the variable identically to a real table. Column hints are overridden with the CTE's actual output columns when they can be determined." [refs var-ast varname] - (let [columns (:columns var-ast) - aliases (:aliases var-ast) - explicit (remove :auto-id columns) - source-tables (if (empty? explicit) - (when-let [current-alias (:current var-ast)] - [(get-in aliases [current-alias :table])]) - (->> explicit - (map #(get-in aliases [(:alias %) :table])) - (remove nil?) - distinct)) + (let [source-tables (get-source-tables var-ast) seeded (reduce (fn [r source-table] (let [source-refs (get-in r [:table source-table])] (if source-refs @@ -108,12 +115,40 @@ (assoc-in [:table varname :column-set] (set output-cols))) seeded))) +(defn- patch-variable-relations + "Bidirectional pass: for each variable V wrapping source S, find every + entity (real table or other variable) T where T already knows that S + refers to it — i.e. T[:referred-by][S] exists — and register V there too. + + This enables: + - 'T | V' and 'V | T' (real table ↔ variable) + - 'V | W' and 'W | V' (variable ↔ variable) + + Must run after all variables have been seeded so that variable entries + already carry the inherited :referred-by data from their source tables." + [refs variables] + (reduce (fn [r [varname var-ast]] + (let [source-tables (get-source-tables var-ast)] + (reduce (fn [r source-table] + (reduce (fn [r entity-name] + (let [existing (get-in r [:table entity-name :referred-by source-table])] + (if existing + (update-in r [:table entity-name :referred-by varname] merge existing) + r))) + r + (keys (get r :table {})))) + r + source-tables))) + refs + variables)) + (defn pre-handle [state connection-id ops-count expression cursor variables] - (let [refs (db/init-references connection-id) - aug-refs (reduce (fn [r [varname var-ast]] - (seed-variable-references r var-ast varname)) - refs - variables)] + (let [refs (db/init-references connection-id) + seeded-refs (reduce (fn [r [varname var-ast]] + (seed-variable-references r var-ast varname)) + refs + variables) + aug-refs (patch-variable-relations seeded-refs variables)] (-> state (assoc :references aug-refs) (assoc :connection-id connection-id) From 06cbcc64bcf2e70082c0b4588e5502792fcc92ed Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Thu, 14 May 2026 01:47:55 +0200 Subject: [PATCH 15/60] fix: patch :refers-to direction for variables; nil schema in relation hints - Refactor patch-variable-relations to cover both :referred-by and :refers-to directions via a patch-direction helper, so joins like `employee | mytest` (where mytest wraps the parent table) resolve correctly. - Fix relation-hints to nil the schema for variable entries, preventing hints like "y.mytest .col" when "mytest .col" is correct. - Add tests for both join directions and both hint directions. Co-Authored-By: Claude Sonnet 4.6 --- src/pine/ast/hints.clj | 17 ++++++++++------- src/pine/ast/main.clj | 35 +++++++++++++++++++++-------------- test/pine/eval_test.clj | 18 +++++++++++++++++- test/pine/hints_test.clj | 28 +++++++++++++++++++++++++++- 4 files changed, 75 insertions(+), 23 deletions(-) diff --git a/src/pine/ast/hints.clj b/src/pine/ast/hints.clj index 92822ef..023ae5d 100644 --- a/src/pine/ast/hints.clj +++ b/src/pine/ast/hints.clj @@ -67,14 +67,17 @@ (defn relation-hints [state token] (let [from-alias (state :context) from-table (-> state :aliases (get from-alias) :table) - parents (-> state :references :table (get from-table) :refers-to) - children (-> state :references :table (get from-table) :referred-by) + variables (-> state :variables) + parents (-> state :references :table (get from-table) :refers-to) + children (-> state :references :table (get from-table) :referred-by) suggestions (filter-relations token (concat parents children))] - (-> suggestions - create-hint-from-relations - ;; TODO: instead of doing a distinct, we can do a reduce and keep track - ;; of duplicates - distinct))) + (->> suggestions + create-hint-from-relations + (map (fn [h] + (if (contains? variables (:table h)) + (assoc h :schema nil) + h))) + distinct))) (defn generate-table-hints [state] (let [{token :table parent :parent} (-> state :tables reverse first) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index f82cc87..f7057b4 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -115,25 +115,17 @@ (assoc-in [:table varname :column-set] (set output-cols))) seeded))) -(defn- patch-variable-relations - "Bidirectional pass: for each variable V wrapping source S, find every - entity (real table or other variable) T where T already knows that S - refers to it — i.e. T[:referred-by][S] exists — and register V there too. - - This enables: - - 'T | V' and 'V | T' (real table ↔ variable) - - 'V | W' and 'W | V' (variable ↔ variable) - - Must run after all variables have been seeded so that variable entries - already carry the inherited :referred-by data from their source tables." - [refs variables] +(defn- patch-direction + "For each variable V wrapping source S, copy T[direction][S] → T[direction][V] + for every entity T in the references map. Used by patch-variable-relations." + [refs variables direction] (reduce (fn [r [varname var-ast]] (let [source-tables (get-source-tables var-ast)] (reduce (fn [r source-table] (reduce (fn [r entity-name] - (let [existing (get-in r [:table entity-name :referred-by source-table])] + (let [existing (get-in r [:table entity-name direction source-table])] (if existing - (update-in r [:table entity-name :referred-by varname] merge existing) + (update-in r [:table entity-name direction varname] merge existing) r))) r (keys (get r :table {})))) @@ -142,6 +134,21 @@ refs variables)) +(defn- patch-variable-relations + "Bidirectional pass: for each variable V wrapping source S, find every + entity T where T already knows about S via :referred-by or :refers-to, + and register V there too. + + This enables: + - 'T | V' and 'V | T' (real table ↔ variable) + - 'V | W' and 'W | V' (variable ↔ variable) + + Must run after all variables have been seeded." + [refs variables] + (-> refs + (patch-direction variables :referred-by) + (patch-direction variables :refers-to))) + (defn pre-handle [state connection-id ops-count expression cursor variables] (let [refs (db/init-references connection-id) seeded-refs (reduce (fn [r [varname var-ast]] diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 48f1caf..34d3919 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -424,4 +424,20 @@ :params nil} (generate-expressions ["company |= active_companies" "active_companies | l: 10 |= small_active" - "small_active"]))))) \ No newline at end of file + "small_active"])))) + + (testing "Reverse join: child table navigates to variable wrapping its parent" + ;; employee.company_id -> company.id + ;; mytest = company (parent); employee | mytest should join employee to the CTE + (is (= {:query "WITH \"mytest\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"e_0\".id AS \"__e_0__id\", \"m_1\".* FROM \"employee\" AS \"e_0\" JOIN \"mytest\" AS \"m_1\" ON \"e_0\".\"company_id\" = \"m_1\".\"id\" LIMIT 250" + :params nil} + (generate-expressions ["company |= mytest" + "employee | mytest"])))) + + (testing "Reverse join: parent table navigates to variable wrapping its child" + ;; employee.company_id -> company.id + ;; mytest = employee (child); company | mytest should join company to the CTE + (is (= {:query "WITH \"mytest\" AS ( SELECT \"e_0\".* FROM \"employee\" AS \"e_0\" ) SELECT \"c_0\".id AS \"__c_0__id\", \"m_1\".* FROM \"company\" AS \"c_0\" JOIN \"mytest\" AS \"m_1\" ON \"c_0\".\"id\" = \"m_1\".\"company_id\" LIMIT 250" + :params nil} + (generate-expressions ["employee |= mytest" + "company | mytest"]))))) \ No newline at end of file diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index 7900c77..21991d8 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -15,6 +15,19 @@ (ast/generate :test expression cursor) :hints))) +(defn- gen-with-variables + "Evaluate expressions sequentially, threading variables. Returns :hints from the last expression." + [expressions] + (let [{:keys [last-hints]} + (reduce (fn [{:keys [variables]} expr] + (let [{:keys [result assign]} (parser/parse expr) + state (ast/generate result :test expr nil variables assign)] + {:variables (if assign (assoc variables assign state) variables) + :last-hints (:hints state)})) + {:variables {} :last-hints nil} + expressions)] + last-hints)) + (deftest test-hints (testing "Generate hints" (is (= [{:schema "x", :table "company" @@ -163,4 +176,17 @@ (is (= ["id" "company_id" "reports_to"] (->> (gen "company | s: id | employee | s: " {:line 0 :character 100}) :select - (map :column)))))) + (map :column))))) + + (testing "Variable relation hints" + ;; mytest = employee (child); company | my should suggest mytest + (is (= [{:schema nil :table "mytest" :column "company_id" :parent false :heuristic false + :pine "mytest .company_id"}] + (-> (gen-with-variables ["employee |= mytest" "company | my"]) + :table))) + + ;; mytest = company (parent); employee | my should suggest mytest + (is (= [{:schema nil :table "mytest" :column "company_id" :parent true :heuristic false + :pine "mytest .company_id :parent"}] + (-> (gen-with-variables ["company |= mytest" "employee | my"]) + :table))))) From 6d3e18b0f21634341af8d1dce32fbd9b7419b2fb Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Thu, 14 May 2026 01:47:58 +0200 Subject: [PATCH 16/60] docs: add joins.md; reflow all docs to 120-char line width Co-Authored-By: Claude Sonnet 4.6 --- docs/joins.md | 168 +++++++++++++++++++++++++++++++++++++++++ docs/pipeline.md | 53 +++++++++---- docs/result-updates.md | 57 ++++++++++---- docs/variables.md | 66 ++++++++++++---- 4 files changed, 299 insertions(+), 45 deletions(-) create mode 100644 docs/joins.md diff --git a/docs/joins.md b/docs/joins.md new file mode 100644 index 0000000..e60ffa0 --- /dev/null +++ b/docs/joins.md @@ -0,0 +1,168 @@ +# Joins + +Pipe two table names together and Pine figures out the JOIN condition automatically. + +## Why + +SQL joins require you to spell out `ON table_a.col = table_b.col` every time, even when the relationship is +already encoded in the schema as a foreign key. Pine reads the FK graph once at startup and resolves the join +condition for you. When there is no FK (e.g. a multi-tenant column added by convention), Pine falls back to +heuristic detection based on column naming. + +## Syntax + +``` +table_a | table_b +table_a | table_b .hint_col +table_a | table_b .left_col = .right_col +table_a | table_b :parent +table_a | table_b :left +``` + +- **No modifier** — Pine picks the join direction automatically. +- **`.hint_col`** — disambiguate when two tables share more than one FK. +- **`.col1 = .col2`** — override both sides explicitly; bypasses the reference map entirely. +- **`:parent`** — force the join to treat `table_b` as the parent (i.e. the table `table_a` refers to, + not the table that refers to `table_a`). +- **`:child`** — inverse of `:parent`; explicit but rarely needed since it is the default. +- **`:left` / `:right`** — emit `LEFT JOIN` / `RIGHT JOIN`. + +## Examples + +### Basic (FK-resolved) + +``` +company | employee +``` + +```sql +SELECT "c_0".id AS "__c_0__id", "e_1".* +FROM "company" AS "c_0" +JOIN "employee" AS "e_1" ON "c_0"."id" = "e_1"."company_id" +LIMIT 250 +``` + +### Disambiguation hint + +`document` has two FKs to `employee` (`employee_id` and `created_by`). Without a hint Pine picks the first +one alphabetically. + +``` +employee | document .created_by +``` + +```sql +JOIN "document" AS "d_1" ON "e_0"."id" = "d_1"."created_by" +``` + +### Forcing parent direction + +``` +employee | company :parent +``` + +Forces Pine to treat `company` as the parent even though `:has` would also match. Equivalent to the default +here, but necessary when the automatic direction would be wrong. + +### Left join + +``` +company | employee :left +``` + +```sql +LEFT JOIN "employee" AS "e_1" ON "c_0"."id" = "e_1"."company_id" +``` + +### Explicit columns + +``` +employee | document .company_id = .id +``` + +Bypasses the reference map. `company_id` is on `document` (right table); `id` is on `employee` (left table). + +## How it works + +1. At startup, Pine queries the database for all foreign keys and scans all column names. It builds a + **references map** used for every subsequent join resolution. +2. When the parser sees `table_a | table_b`, it emits two consecutive `:table` operations. +3. `table/handle` in `ast/table.clj` calls `update-joins`, which calls `join-tables`, which calls `join-helper`. +4. `join-helper` looks up the resolved join vector from the references map and records it on the AST + state's `:joins` vector. +5. `eval/build-join-clause` turns each entry in `:joins` into a SQL `JOIN … ON …` fragment. + +## Constraints + +- Circular joins are not detected — the query will compile but the SQL may be nonsensical. +- Heuristic joins are only inferred when no FK already covers the same pair. +- Self-referential heuristic joins are suppressed. Real self-referential FKs (e.g. + `employee.reports_to → employee.id`) are supported. + +--- + +## Implementation + +### Reference map structure (`db/postgres.clj`) + +`index-references` builds the map in three passes: + +1. **`index-foreign-keys`** — queries `pg_constraint` and indexes every FK in both directions: + - `refs[:table f-table :referred-by table :via col]` — child direction ("who points at me") + - `refs[:table table :refers-to f-table :via col]` — parent direction ("who I point at") + + Each entry is a list of **join vectors**: + ``` + [f-schema f-table f-col :referred-by schema table col :foreign-key] + [schema table col :refers-to f-schema f-table f-col :foreign-key] + ``` + +2. **`index-columns`** — adds column metadata to each table entry. Needed before the next pass. + +3. **`index-heuristic-relations`** — scans every column looking for `_id` / `Id` suffixes: + - `extract-table-from-column` strips the suffix: `company_id` → `company`, `tenantId` → `tenant`. + - `normalize-plural` generates candidate names: `company` → `#{"company" "companies"}`. + - Matches against all known tables via `build-table-lookup`. + - Skips if the candidate table has no `id` column, if the FK already exists, or if it would be a + self-referential heuristic. + - Adds the same two-direction index entries as FK detection, tagged `:heuristic` instead of `:foreign-key`. + +The same map structure is used for both FK and heuristic entries; the only difference is the tag in position 7 +of the join vector. Callers can inspect it if they want to surface confidence level. + +### Join direction resolution (`ast/table.clj`) + +`join-tables` tries two strategies in order: + +1. **`:has`** — `join-helper` looks up `refs[:table t1 :referred-by t2]`. This succeeds when `t2` has a + FK (or heuristic) pointing at `t1`. Returns `[a1 col :has a2 f-col]`. +2. **`:of`** — arguments swapped: `join-helper` looks up `refs[:table t2 :referred-by t1]`. + Returns `[a2 f-col :of a1 col]`. + +`:has` is tried first unless the `:parent` modifier is set, in which case only `:of` is attempted. + +When `.hint_col` is present, `join-column` is set. `join-helper` uses it to select a specific key from the +`via` map instead of taking `first`. + +When `.col1 = .col2` is present, `join-left-column` and `join-right-column` are set. `update-joins` bypasses +`join-tables` entirely and records the explicit pair directly. + +### Grammar (`pine.bnf`) + +``` +TABLE := table table-mods +table-mods := ( table-mod)* +table-mod := <":"> ("parent"|"child"|"left"|"right") | as-alias | hint-columns +hint-columns := hint-column | explicit-columns +explicit-columns := hint-column <"="> hint-column +``` + +### SQL generation (`eval.clj`) + +`build-join-clause` maps each `[from-alias to-alias relation join-type]` entry in `:joins` to: + +```sql +[LEFT|RIGHT] JOIN "schema"."table" AS "alias" ON "a1"."col1" = "a2"."col2" +``` + +The `ON` columns are positions 1 and 4 of the relation vector: `[a1 col _ a2 f-col]`. diff --git a/docs/pipeline.md b/docs/pipeline.md index 6615d92..9cc18d2 100644 --- a/docs/pipeline.md +++ b/docs/pipeline.md @@ -4,7 +4,8 @@ How a Pine expression goes from text to SQL results. ## Why -Pine is a pipe-based DSL that compiles to SQL. Understanding the pipeline helps when debugging unexpected query output, tracing where hints come from, or extending the language with new operations. +Pine is a pipe-based DSL that compiles to SQL. Understanding the pipeline helps when debugging unexpected +query output, tracing where hints come from, or extending the language with new operations. ## Overview @@ -22,7 +23,8 @@ text → parse → generate → eval ### 1. Parse -The expression is tokenised against the Pine grammar and turned into a flat list of typed operations — one per pipe segment. +The expression is tokenised against the Pine grammar and turned into a flat list of typed operations — +one per pipe segment. Input: ``` @@ -34,11 +36,13 @@ Output: [ table(user as u), table(document name="passport"), select(u.email, id), limit(10) ] ``` -Each operation has a type and a value. If the expression ends with `|= name`, that name is extracted separately and carried alongside the list. +Each operation has a type and a value. If the expression ends with `|= name`, that name is extracted +separately and carried alongside the list. ### 2. Generate -Each operation is applied left to right to a state map, using the appropriate handler. Before the first operation runs, the DB schema is loaded so that join paths and column lists are available. +Each operation is applied left to right to a state map, using the appropriate handler. Before the first +operation runs, the DB schema is loaded so that join paths and column lists are available. | Pipe segment | What it adds to state | |---|---| @@ -52,7 +56,8 @@ Each operation is applied left to right to a state map, using the appropriate ha | `update!` / `u!` | Records column assignments for UPDATE | | `delete!` | Marks the operation as DELETE | -After all operations are applied, a **post-processing** step runs: autocomplete hints are computed, hidden auto-id columns are appended for row tracking, and a prettified form of the expression is attached. +After all operations are applied, a **post-processing** step runs: autocomplete hints are computed, +hidden auto-id columns are appended for row tracking, and a prettified form of the expression is attached. The result is a single **state map** that fully describes the query. @@ -60,20 +65,26 @@ The result is a single **state map** that fully describes the query. The state map is handed to the eval layer, which has two paths: -**Build** — translates the state into a SQL string and parameter list. No database call. Used to show the query in the UI as the user types. +**Build** — translates the state into a SQL string and parameter list. No database call. Used to show +the query in the UI as the user types. **Run** — builds the SQL then executes it against the database, returning rows. ## How it works -- The pipeline is stateless per request — each call to the API runs the full parse → generate → eval cycle from scratch. -- Hints are computed inside `generate`, using a second pass over the expression truncated at the cursor position. This gives hints that reflect what the user has typed so far, not the full completed expression. -- When multiple expressions are present (separated by blank lines), they are evaluated left to right. Each assigned expression (`|= name`) stores its state as a variable, threaded into subsequent calls. See [variables.md](variables.md). +- The pipeline is stateless per request — each call to the API runs the full parse → generate → eval + cycle from scratch. +- Hints are computed inside `generate`, using a second pass over the expression truncated at the cursor + position. This gives hints that reflect what the user has typed so far, not the full completed expression. +- When multiple expressions are present (separated by blank lines), they are evaluated left to right. + Each assigned expression (`|= name`) stores its state as a variable, threaded into subsequent calls. + See [variables.md](variables.md). ## Constraints - The pipeline is synchronous and single-pass — there is no incremental or lazy evaluation. -- All schema information is resolved at generate time. A table or column that doesn't exist in the DB schema will fail at that stage, not at SQL execution time. +- All schema information is resolved at generate time. A table or column that doesn't exist in the DB + schema will fail at that stage, not at SQL execution time. --- @@ -81,19 +92,27 @@ The state map is handed to the eval layer, which has two paths: ### Parse (`parser/parse`, `pine.bnf`) -Instaparse runs the BNF grammar over the input string. The raw parse tree is normalised into a vector of `{:type :value }` maps. `|= name` is extracted as `:assign` and stripped from the operation list so the rest of the pipeline is unaware of it. +Instaparse runs the BNF grammar over the input string. The raw parse tree is normalised into a vector of +`{:type :value }` maps. `|= name` is extracted as `:assign` and stripped from the +operation list so the rest of the pipeline is unaware of it. ### Generate (`ast/generate`, `ast/main.clj`) `generate` orchestrates three sub-steps: -**`pre-handle`** — seeds the initial state with the DB schema via `db/init-references` (cached per connection). If variables are in scope, `seed-variable-references` merges their underlying FK references so join resolution works through CTEs. +**`pre-handle`** — seeds the initial state with the DB schema via `db/init-references` (cached per +connection). If variables are in scope, `seed-variable-references` merges their underlying FK references +so join resolution works through CTEs. -**`handle-ops`** — calls `reduce` over the operation list, dispatching each operation to its handler module (`ast/table`, `ast/select`, `ast/where`, `ast/group`, `ast/order`, `ast/limit`, `ast/count`, `ast/delete-action`, `ast/update-action`). The operation index (`i`) is threaded through so later stages know the order columns were introduced. +**`handle-ops`** — calls `reduce` over the operation list, dispatching each operation to its handler +module (`ast/table`, `ast/select`, `ast/where`, `ast/group`, `ast/order`, `ast/limit`, `ast/count`, +`ast/delete-action`, `ast/update-action`). The operation index (`i`) is threaded through so later +stages know the order columns were introduced. **`post-handle`** — runs after all operations: - `ast/hints/handle` computes autocomplete hints using the truncated-at-cursor state. -- `ast/select/add-auto-id-columns` appends hidden `id` columns for each real table. See [result-updates.md](result-updates.md). +- `ast/select/add-auto-id-columns` appends hidden `id` columns for each real table. + See [result-updates.md](result-updates.md). - `add-prettify` attaches a formatted expression and per-operation character ranges for cursor highlighting. The final state map: @@ -124,8 +143,10 @@ The final state map: | `:update-action` / `:update-partial` | `build-update-queries` | | `:delete-action` | `build-delete-query` | -`build-select-query` checks for variables in scope and, if present, prepends CTE clauses via `collect-ctes`. `run-query` calls `build-query` then executes against the connection pool via `db/run-query`. +`build-select-query` checks for variables in scope and, if present, prepends CTE clauses via `collect-ctes`. +`run-query` calls `build-query` then executes against the connection pool via `db/run-query`. ### Empty-expression guard -`build-query` checks whether the current table name (looked up via the aliases map) is an empty string. This handles blank input — returns `{:query "" :params nil}` rather than generating a malformed SELECT. +`build-query` checks whether the current table name (looked up via the aliases map) is an empty string. +This handles blank input — returns `{:query "" :params nil}` rather than generating a malformed SELECT. diff --git a/docs/result-updates.md b/docs/result-updates.md index 38042dd..66801ab 100644 --- a/docs/result-updates.md +++ b/docs/result-updates.md @@ -4,12 +4,18 @@ Editing cells in multi-table result sets. ## Why -A Pine expression like `user | document` returns a result set that mixes columns from two tables. You might want to edit a document's `name` field directly in that result. The challenge is that a naive `UPDATE document SET name = ?` doesn't know which specific document row to target — it needs the `id` of the document for the row you edited, not just any document. +A Pine expression like `user | document` returns a result set that mixes columns from two tables. You +might want to edit a document's `name` field directly in that result. The challenge is that a naive +`UPDATE document SET name = ?` doesn't know which specific document row to target — it needs the `id` +of the document for the row you edited, not just any document. Pine solves this with two mechanisms working together: -- **Auto-id columns**: for each table in the query, Pine silently appends a hidden column `alias.id AS "__alias__id"`. These are never displayed but are always present in the result, so the UI can look up the exact row ID for any cell. -- **`update!`**: generates separate UPDATE statements per table, each scoped to the correct row via a subquery that inherits the full join condition. +- **Auto-id columns**: for each table in the query, Pine silently appends a hidden column + `alias.id AS "__alias__id"`. These are never displayed but are always present in the result, so the + UI can look up the exact row ID for any cell. +- **`update!`**: generates separate UPDATE statements per table, each scoped to the correct row via a + subquery that inherits the full join condition. Together they make any join result editable, not just readable. @@ -91,16 +97,24 @@ When you edit a cell directly in the result grid, the UI constructs an `update!` ## How it works -- **Auto-id columns** are appended automatically for every real table that has an `id` column. They are marked `hidden: true` (not shown in the grid) and `auto-id: true` (so internal logic can identify them). Variable/CTE tables are excluded — they don't have a stable `id`. -- **Column qualification**: unqualified columns (e.g. `name`) default to the last table in the expression. Qualified columns (`u.name`) target the specified alias. -- **One UPDATE per table**: assignments are grouped by target alias. Each group becomes an independent UPDATE with a subquery to identify the rows. -- **Subquery for row targeting**: Pine uses `WHERE id IN (SELECT id FROM ... JOIN ...)` rather than a direct `WHERE id = ?`. This ensures the update respects the full join condition. +- **Auto-id columns** are appended automatically for every real table that has an `id` column. They are + marked `hidden: true` (not shown in the grid) and `auto-id: true` (so internal logic can identify + them). Variable/CTE tables are excluded — they don't have a stable `id`. +- **Column qualification**: unqualified columns (e.g. `name`) default to the last table in the expression. + Qualified columns (`u.name`) target the specified alias. +- **One UPDATE per table**: assignments are grouped by target alias. Each group becomes an independent + UPDATE with a subquery to identify the rows. +- **Subquery for row targeting**: Pine uses `WHERE id IN (SELECT id FROM ... JOIN ...)` rather than a + direct `WHERE id = ?`. This ensures the update respects the full join condition. ## Constraints -- Only tables with an `id` column in the DB schema get an auto-id. Tables without `id` cannot be targeted by `update!`. -- If the user explicitly selects `id` (e.g. `s: id, name`), the auto-id column is still added with its `__alias__id` alias to avoid ambiguity. -- Auto-id columns are not added inside CTE bodies — `SELECT *` already includes `id`, and a second `id` would cause an ambiguous-column error. +- Only tables with an `id` column in the DB schema get an auto-id. Tables without `id` cannot be + targeted by `update!`. +- If the user explicitly selects `id` (e.g. `s: id, name`), the auto-id column is still added with its + `__alias__id` alias to avoid ambiguity. +- Auto-id columns are not added inside CTE bodies — `SELECT *` already includes `id`, and a second + `id` would cause an ambiguous-column error. - `update!` on a variable/CTE table is not supported — the CTE has no physical table to update. --- @@ -109,7 +123,8 @@ When you edit a cell directly in the result grid, the UI constructs an `update!` ### Auto-id columns (`ast/select.clj`) -`add-auto-id-columns` runs in `post-handle` after all operations are applied. For each table alias whose underlying table has an `id` column (checked via `has-id-column?`), it appends: +`add-auto-id-columns` runs in `post-handle` after all operations are applied. For each table alias +whose underlying table has an `id` column (checked via `has-id-column?`), it appends: ```clojure { :column "id" @@ -120,11 +135,14 @@ When you edit a cell directly in the result grid, the UI constructs an `update!` :operation-index N } ``` -Variable tables are skipped: if the table name for an alias is a key in `state :variables`, no auto-id is added. +Variable tables are skipped: if the table name for an alias is a key in `state :variables`, no auto-id +is added. ### CTE body deduplication (`eval.clj`) -`build-cte-body` generates the inner SQL for a variable's CTE. Before calling `build-bare-select`, it checks whether user-specified columns already cover the current table. If so, the auto-id column is dropped to avoid `SELECT id, ..., id` duplication. +`build-cte-body` generates the inner SQL for a variable's CTE. Before calling `build-bare-select`, it +checks whether user-specified columns already cover the current table. If so, the auto-id column is +dropped to avoid `SELECT id, ..., id` duplication. ### Frontend column metadata (`default.plugin.tsx`) @@ -135,12 +153,19 @@ After a query runs, the UI builds two lookup maps from the response column metad ### Frontend cell editing (`Result.tsx`) -`processRowUpdate` is called by MUI DataGrid when a cell is committed. It uses the two lookup maps to find the row ID, builds an update expression via `createUpdateExpression`, and evaluates it in a virtual session before refreshing the main session. +`processRowUpdate` is called by MUI DataGrid when a cell is committed. It uses the two lookup maps to +find the row ID, builds an update expression via `createUpdateExpression`, and evaluates it in a +virtual session before refreshing the main session. ### SQL generation (`eval/build-update-queries`) -Assignments are grouped by target alias. For each group, `build-single-update-query` produces the UPDATE with a subquery built by temporarily replacing the state's column list with `[{:column "id" :alias update-alias}]` and calling `build-select-query` — so it inherits the full JOIN and WHERE conditions from the original expression. +Assignments are grouped by target alias. For each group, `build-single-update-query` produces the UPDATE +with a subquery built by temporarily replacing the state's column list with +`[{:column "id" :alias update-alias}]` and calling `build-select-query` — so it inherits the full JOIN +and WHERE conditions from the original expression. ### Parsing (`pine.bnf`, `parser.clj`) -`update!` / `u!` are parsed as `:update-action`. Each assignment carries `{:column {:alias ... :column ...} :value {...}}`. Partial typing (`u! col`) is parsed as `:update-partial` for hint generation. +`update!` / `u!` are parsed as `:update-action`. Each assignment carries +`{:column {:alias ... :column ...} :value {...}}`. Partial typing (`u! col`) is parsed as +`:update-partial` for hint generation. diff --git a/docs/variables.md b/docs/variables.md index 242ed58..4bf290a 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -4,7 +4,10 @@ Name and reuse intermediate query results across expressions. ## Why -Complex queries often need to build up results in steps — filter a set of rows, then join that filtered set to another table, then aggregate. Without variables, the only way to do this is with nested subqueries, which are hard to read and hard to iterate on in a REPL-style tool. Variables let you give each step a name, treat it as a table, and compose them incrementally. +Complex queries often need to build up results in steps — filter a set of rows, then join that filtered +set to another table, then aggregate. Without variables, the only way to do this is with nested subqueries, +which are hard to read and hard to iterate on in a REPL-style tool. Variables let you give each step a +name, treat it as a table, and compose them incrementally. ## Syntax @@ -12,7 +15,8 @@ Complex queries often need to build up results in steps — filter a set of rows |= ``` -Append `|= name` at the end of any pipe chain to assign the result to a variable. Use the name as a table in any later expression. +Append `|= name` at the end of any pipe chain to assign the result to a variable. Use the name as a +table in any later expression. ## Examples @@ -58,16 +62,22 @@ tenant as t | public.company .tenantId | g: t.title |= x x ``` -`x` exposes only the columns the CTE actually produces — here `title` and `count` — so hints for `x | s:` show just those two columns. +`x` exposes only the columns the CTE actually produces — here `title` and `count` — so hints for +`x | s:` show just those two columns. ## How it works - **Assignment** (`|= name`): the expression is stored as a variable. Its SQL becomes the body of a CTE. -- **Usage**: when `name` appears as a table in a later expression, Pine substitutes the CTE. Joins through variable tables resolve using the schema of the underlying real tables. +- **Usage**: when `name` appears as a table in a later expression, Pine substitutes the CTE. Joins resolve + in both directions — `variable | table`, `table | variable`, and `variable | variable` — using the FK + schema of the underlying real tables. - **Scoping**: variables accumulate left-to-right. Each expression sees all variables defined before it. -- **Column hints**: when a variable has explicit output columns (from `s:`, `g:`, etc.), hints reflect those columns, not the full schema of the source table. Variables using `*` (no explicit selection) inherit the source table's columns. +- **Column hints**: when a variable has explicit output columns (from `s:`, `g:`, etc.), hints reflect + those columns, not the full schema of the source table. Variables using `*` (no explicit selection) + inherit the source table's columns. - **No LIMIT in CTE body**: limits only apply to the outer query. -- **No auto-id columns**: Pine's hidden `id` tracking columns are not added for variable tables. See [result-updates.md](result-updates.md). +- **No auto-id columns**: Pine's hidden `id` tracking columns are not added for variable tables. + See [result-updates.md](result-updates.md). ## Constraints @@ -81,31 +91,61 @@ x ### Grammar and parsing -`|= name` is parsed as an `assign` operation in `pine.bnf`. The parser (`parser.clj`) extracts the assigned name separately from the main operation list and returns it as `{:assign "name" :result [...ops...]}`. This keeps the assign out of the normal operation pipeline so it doesn't interfere with SQL generation. +`|= name` is parsed as an `assign` operation in `pine.bnf`. The parser (`parser.clj`) extracts the assigned +name separately from the main operation list and returns it as `{:assign "name" :result [...ops...]}`. This +keeps the assign out of the normal operation pipeline so it doesn't interfere with SQL generation. ### State threading -The API receives an array of expressions. Each expression is parsed and evaluated in order. The resulting AST state for each assigned expression is stored in a `variables` map keyed by name and threaded into every subsequent `ast/generate` call. This is done purely in memory within the request — there is no persistence layer. +The API receives an array of expressions. Each expression is parsed and evaluated in order. The resulting +AST state for each assigned expression is stored in a `variables` map keyed by name and threaded into every +subsequent `ast/generate` call. This is done purely in memory within the request — there is no persistence +layer. ### CTE generation (`eval.clj`) -When `build-select-query` runs on a state that has variables, it calls `collect-ctes` to walk the variable map and build `WITH name AS ( ... )` clauses. Each CTE body is generated by `build-cte-body`, which calls `build-bare-select` (no LIMIT). WHERE params from inside CTEs are accumulated and merged into the outer query's params list. +When `build-select-query` runs on a state that has variables, it calls `collect-ctes` to walk the variable +map and build `WITH name AS ( ... )` clauses. Each CTE body is generated by `build-cte-body`, which calls +`build-bare-select` (no LIMIT). WHERE params from inside CTEs are accumulated and merged into the outer +query's params list. ### Join resolution through variables -When a variable is used as a table, its real source tables need to be known so that joins can be resolved (e.g. joining `active_companies` to `employee` requires knowing `active_companies` wraps `company`). `seed-variable-references` in `ast/main.clj` copies the FK reference entries from the underlying real tables into the references map under the variable name at build time. +Reference seeding happens in two passes inside `pre-handle` (`ast/main.clj`): + +**Pass 1 — `seed-variable-references`**: For each variable V wrapping source table S, copies S's FK +reference entry into the references map under V's name. This enables `V | table` and `table | V` when +the join helper can find `table[:referred-by][V]` — but only if that entry already exists, which it +doesn't yet after pass 1 alone. + +**Pass 2 — `patch-variable-relations`**: Iterates every entity (real table or already-seeded variable) +in the references map. For each entity T where `T[:referred-by][S]` exists, registers `T[:referred-by][V]` +with the same relation data. This propagates the relationship bidirectionally: + +- `T | V` and `V | T` (real table ↔ variable) +- `V | W` and `W | V` (variable ↔ variable) + +The shared helper `get-source-tables` extracts which real tables a variable wraps — used by both passes to +avoid duplicating that logic. ### Column hints for variables `seed-variable-references` also overrides the column list for the variable entry in the references map: -- **Explicit columns** (`s:`, `g:`, etc.): the column list is built from the user-selected columns. The name used is `column-alias` if set, otherwise `column`. Group operations append a synthetic `count` entry. +- **Explicit columns** (`s:`, `g:`, etc.): the column list is built from the user-selected columns. The + name used is `column-alias` if set, otherwise `column`. Group operations append a synthetic `count` entry. - **No explicit columns** (`*`): the source table's full column list is inherited. ### Alias disambiguation -Each table gets an alias derived from its name initials (`active_companies` → `ac`). Because `make-alias ""` (empty input) also returns `"x"` as a fallback, the empty-expression guard in `build-query` checks the actual table name in the aliases map rather than the alias string, to avoid false matches on a variable named `x`. +Each table gets an alias derived from its name initials (`active_companies` → `ac`). Because `make-alias ""` +(empty input) also returns `"x"` as a fallback, the empty-expression guard in `build-query` checks the +actual table name in the aliases map rather than the alias string, to avoid false matches on a variable +named `x`. ### Duplicate column guard in CTE body -When the user selects explicit columns that include `id` (e.g. `s: id, name`), the CTE body must not also emit the auto-id column, since `SELECT id, id` is ambiguous. `build-cte-body` detects whether user-specified columns already cover the current table and drops the auto-id column from the CTE body if so. +When the user selects explicit columns that include `id` (e.g. `s: id, name`), the CTE body must not +also emit the auto-id column, since `SELECT id, id` is ambiguous. `build-cte-body` detects whether +user-specified columns already cover the current table and drops the auto-id column from the CTE body +if so. From 787196f5071d8a965d2c1e631cc2b70e43939097 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Thu, 14 May 2026 02:11:13 +0200 Subject: [PATCH 17/60] fix: join same-source variables via synthetic id=id path Two variables wrapping the same source table (e.g. company |= c1 and company |= c2) had no join path between them because company has no self-referential FK. Added patch-same-source-variable-joins which registers a synthetic :variable-join entry at refs[:table v1 :referred-by v2] for each pair of variables sharing a source table with an id column, gated on the absence of an existing join path so self-referential FK propagation (employee/reports_to) is not disturbed. Co-Authored-By: Claude Sonnet 4.6 --- src/pine/ast/main.clj | 27 ++++++++++++++++++++++++++- test/pine/eval_test.clj | 14 +++++++++++++- 2 files changed, 39 insertions(+), 2 deletions(-) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index f7057b4..803afee 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -149,13 +149,38 @@ (patch-direction variables :referred-by) (patch-direction variables :refers-to))) +(defn- patch-same-source-variable-joins + "For each ordered pair of distinct variables (v1, v2) that share a source table + with an 'id' column, register a synthetic id=id join at refs[:table v1 :referred-by v2]. + Only adds entries where no join path already exists, so self-referential FK propagation + (e.g. employee/reports_to) is preserved." + [refs variables] + (let [var-sources (->> variables + (map (fn [[vname var-ast]] + [vname (set (get-source-tables var-ast))])) + (into {})) + vnames (vec (keys var-sources))] + (reduce (fn [r [v1 v2]] + (let [shared-sources (filter (var-sources v1) (var-sources v2)) + has-id? (fn [tbl] (some #(= "id" (:column %)) (get-in r [:table tbl :columns])))] + (if (and (some has-id? shared-sources) + (not (get-in r [:table v1 :referred-by v2]))) + (update-in r [:table v1 :referred-by v2 :via "id"] + (fnil conj []) + [nil v1 "id" :referred-by nil v2 "id" :variable-join]) + r))) + refs + (for [v1 vnames v2 vnames :when (not= v1 v2)] [v1 v2])))) + (defn pre-handle [state connection-id ops-count expression cursor variables] (let [refs (db/init-references connection-id) seeded-refs (reduce (fn [r [varname var-ast]] (seed-variable-references r var-ast varname)) refs variables) - aug-refs (patch-variable-relations seeded-refs variables)] + aug-refs (-> seeded-refs + (patch-variable-relations variables) + (patch-same-source-variable-joins variables))] (-> state (assoc :references aug-refs) (assoc :connection-id connection-id) diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 34d3919..8122ec0 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -440,4 +440,16 @@ (is (= {:query "WITH \"mytest\" AS ( SELECT \"e_0\".* FROM \"employee\" AS \"e_0\" ) SELECT \"c_0\".id AS \"__c_0__id\", \"m_1\".* FROM \"company\" AS \"c_0\" JOIN \"mytest\" AS \"m_1\" ON \"c_0\".\"id\" = \"m_1\".\"company_id\" LIMIT 250" :params nil} (generate-expressions ["employee |= mytest" - "company | mytest"]))))) \ No newline at end of file + "company | mytest"])))) + + (testing "Same-source variables: two variables wrapping the same table join on id" + (is (= {:query "WITH \"c1\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"c2\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"c_1\".* FROM \"c1\" AS \"c_0\" JOIN \"c2\" AS \"c_1\" ON \"c_0\".\"id\" = \"c_1\".\"id\" LIMIT 250" + :params nil} + (generate-expressions ["company |= c1" + "company |= c2" + "c1 | c2"]))) + (is (= {:query "WITH \"c2\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"c1\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"c_1\".* FROM \"c2\" AS \"c_0\" JOIN \"c1\" AS \"c_1\" ON \"c_0\".\"id\" = \"c_1\".\"id\" LIMIT 250" + :params nil} + (generate-expressions ["company |= c1" + "company |= c2" + "c2 | c1"]))))) \ No newline at end of file From 9f27c05ea9a0874d21bdb42dce05a9bbd8226215 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Thu, 14 May 2026 12:29:12 +0200 Subject: [PATCH 18/60] fix: omit column qualifier in hints for same-source variable joins synthetic :variable-join entries always join on id=id so there is no disambiguation need. Suppress the column from the generated hint so the pine expression reads "c2" instead of "c2 .id". Co-Authored-By: Claude Sonnet 4.6 --- src/pine/ast/hints.clj | 9 +++++---- test/pine/hints_test.clj | 6 ++++++ 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/src/pine/ast/hints.clj b/src/pine/ast/hints.clj index 023ae5d..1b150d4 100644 --- a/src/pine/ast/hints.clj +++ b/src/pine/ast/hints.clj @@ -44,10 +44,11 @@ ;; ["z" "document" "created_by" :refers-to "y" "employee" "id"] (defn- create-hint-from-relation-array [table via-details] (map (fn [vd] - (let [direction (nth vd 3) - parent? (= direction :refers-to) - schema (nth vd 4) - column (nth vd (if parent? 2 6)) + (let [direction (nth vd 3) + parent? (= direction :refers-to) + schema (nth vd 4) + variable? (= (last vd) :variable-join) + column (when-not variable? (nth vd (if parent? 2 6))) heuristic? (= (last vd) :heuristic)] {:schema schema :table table diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index 21991d8..5cefe25 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -189,4 +189,10 @@ (is (= [{:schema nil :table "mytest" :column "company_id" :parent true :heuristic false :pine "mytest .company_id :parent"}] (-> (gen-with-variables ["company |= mytest" "employee | my"]) + :table))) + + ;; same-source variables: c1 and c2 both wrap company; c1 | c2 should suggest c2 + (is (= [{:schema nil :table "c2" :column nil :parent false :heuristic false + :pine "c2"}] + (-> (gen-with-variables ["company |= c1" "company |= c2" "c1 | c2"]) :table))))) From b7fd762a138bbf4d9a7b0ccf6c9d8fa5ac6e2c33 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Thu, 14 May 2026 13:59:15 +0200 Subject: [PATCH 19/60] test: fix same-source variable hint test to use partial token MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous test used "c1 | c2" where "c2" exactly matches the target variable — trivially correct. Changed to "var_x | var" with var_x and var_y as names so the test exercises partial typing (token "var" filters to var_y without needing the full name), closer to actual UI behaviour. Co-Authored-By: Claude Sonnet 4.6 --- src/pine/parser.clj | 11 ++++++++--- test/pine/hints_test.clj | 10 ++++++---- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/src/pine/parser.clj b/src/pine/parser.clj index bdc8f2f..eb6bf1d 100644 --- a/src/pine/parser.clj +++ b/src/pine/parser.clj @@ -443,6 +443,14 @@ :else (throw (ex-info "Unknown UPDATE-PARTIAL operation" {:_ operation})))) + +;; ----- +;; ASSIGN +;; ----- + +(defmethod -normalize-op :ASSIGN [[_ [_ varname]]] + {:type :assign :value varname}) + ;; ----- ;; NO-OP ;; ----- @@ -458,9 +466,6 @@ grammar (slurp file)] (insta/parser grammar))) -(defmethod -normalize-op :ASSIGN [[_ [_ varname]]] - {:type :assign :value varname}) - (defn- normalize-ops [[_ & nodes]] (mapv (fn [[_ op]] (-normalize-op op)) (filter #(= (first %) :OPERATION) nodes))) diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index 5cefe25..17fc264 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -191,8 +191,10 @@ (-> (gen-with-variables ["company |= mytest" "employee | my"]) :table))) - ;; same-source variables: c1 and c2 both wrap company; c1 | c2 should suggest c2 - (is (= [{:schema nil :table "c2" :column nil :parent false :heuristic false - :pine "c2"}] - (-> (gen-with-variables ["company |= c1" "company |= c2" "c1 | c2"]) + ;; same-source variables: var_x and var_y both wrap company; + ;; when typing "var_x | var", var_y should appear (partial match, unambiguous token) + (is (= [{:schema nil :table "var_y" :column nil :parent false :heuristic false + :pine "var_y"}] + (-> (gen-with-variables ["company |= var_x" "company |= var_y" "var_x | var"]) :table))))) + From 5d3c9af3a80c130a7d87edd05bcfef284cd31fcc Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Thu, 14 May 2026 14:42:02 +0200 Subject: [PATCH 20/60] docs: clarify generate signature; document Pass 3 in variables.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pipeline.md: add parameter table for ast/generate explaining why both parse-tree and expression are needed, and what cursor/variables/assign do. variables.md: clarify state threading loop (parse → generate → store); add Pass 3 (patch-same-source-variable-joins) to join resolution section. Co-Authored-By: Claude Sonnet 4.6 --- docs/pipeline.md | 22 ++++++++++++++++++++++ docs/variables.md | 24 ++++++++++++++++++------ 2 files changed, 40 insertions(+), 6 deletions(-) diff --git a/docs/pipeline.md b/docs/pipeline.md index 9cc18d2..7daa53c 100644 --- a/docs/pipeline.md +++ b/docs/pipeline.md @@ -98,6 +98,28 @@ operation list so the rest of the pipeline is unaware of it. ### Generate (`ast/generate`, `ast/main.clj`) +Full signature: + +```clojure +(generate parse-tree connection-id expression cursor variables assign) +``` + +Each parameter fills a distinct role that cannot be inferred from the others: + +| Parameter | Source | Used for | +|---|---|---| +| `parse-tree` | `(parser/parse expression)` | `handle-ops` — the typed operation list | +| `connection-id` | caller / DB config | `db/init-references` — loading the schema | +| `expression` | raw input string | `add-prettify` (ranges), `truncate-at-cursor` (cursor hints) | +| `cursor` | UI (user's caret position) | Building `truncated-state` for cursor-aware hints | +| `variables` | caller, accumulated across expressions | `pre-handle` — seeding FK references for CTEs | +| `assign` | `(parser/parse expression)` | Stored on the returned state; not processed as an op | + +`parse-tree` and `assign` both come from `parser/parse`, but `expression` is still required separately +because `add-prettify` and `truncate-at-cursor` operate on raw text. `cursor` and `variables` are +external context that is not derivable from the expression itself — `cursor` comes from the UI, and +`variables` accumulates as the caller evaluates earlier expressions in a multi-expression session. + `generate` orchestrates three sub-steps: **`pre-handle`** — seeds the initial state with the DB schema via `db/init-references` (cached per diff --git a/docs/variables.md b/docs/variables.md index 4bf290a..8c60744 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -97,10 +97,16 @@ keeps the assign out of the normal operation pipeline so it doesn't interfere wi ### State threading -The API receives an array of expressions. Each expression is parsed and evaluated in order. The resulting -AST state for each assigned expression is stored in a `variables` map keyed by name and threaded into every -subsequent `ast/generate` call. This is done purely in memory within the request — there is no persistence -layer. +The API receives an array of expressions. Each expression is parsed and evaluated in order: + +1. `parser/parse(expr)` returns `{:result [ops...] :assign "name"}`. The `assign` field is the variable + name from `|= name` — stripped from the op list so it doesn't enter the pipeline. +2. `ast/generate(ops, ..., variables, assign)` runs the full pipeline and stores `assign` on the returned + state as metadata. +3. If `assign` is set, the caller stores the returned state in the `variables` map under that name. +4. The updated `variables` map is passed to `ast/generate` for every subsequent expression. + +This is done purely in memory within the request — there is no persistence layer. ### CTE generation (`eval.clj`) @@ -125,8 +131,14 @@ with the same relation data. This propagates the relationship bidirectionally: - `T | V` and `V | T` (real table ↔ variable) - `V | W` and `W | V` (variable ↔ variable) -The shared helper `get-source-tables` extracts which real tables a variable wraps — used by both passes to -avoid duplicating that logic. +**Pass 3 — `patch-same-source-variable-joins`**: For each ordered pair of distinct variables (V1, V2) +that share a common source table with an `id` column, registers a synthetic `id = id` join at +`refs[:table V1 :referred-by V2]`. This makes `V1 | V2` resolve even when the source table has no +self-referential FK. Only adds entries where no join path already exists — existing FK-based propagation +(e.g. two employee-wrapping variables joined via `reports_to`) is not overridden. + +The shared helper `get-source-tables` extracts which real tables a variable wraps — used by all three +passes to avoid duplicating that logic. ### Column hints for variables From 2b290cec52336d916c7adfe2c69a167d37bbde6b Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Thu, 14 May 2026 14:57:54 +0200 Subject: [PATCH 21/60] fix: fmt --- src/pine/parser.clj | 1 - 1 file changed, 1 deletion(-) diff --git a/src/pine/parser.clj b/src/pine/parser.clj index eb6bf1d..ed17fec 100644 --- a/src/pine/parser.clj +++ b/src/pine/parser.clj @@ -443,7 +443,6 @@ :else (throw (ex-info "Unknown UPDATE-PARTIAL operation" {:_ operation})))) - ;; ----- ;; ASSIGN ;; ----- From 3c20dab3c19885bcf78525f75242d0b79d86061b Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sat, 16 May 2026 13:41:34 +0200 Subject: [PATCH 22/60] feat: make |= mid-pipeline and allow variable names as column qualifiers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - |= is now a regular pipe operation (not a terminal), enabling 'company |= x | w: active = true' — the snapshot is taken at the assignment point while the pipeline continues - Variable names assigned via |= can be used as column qualifiers in subsequent ops within the same expression (e.g. 's: x.id', 'w: x.id = 1') by resolving through :pending-assignments rather than a separate map - Adds docs/expressions.md covering multi-expression evaluation, operation table, variable threading, and error short-circuiting - Updates docs/pipeline.md and docs/variables.md accordingly - Adds what/why ns docstrings to api.clj, ast/main.clj, ast/assign.clj, parser.clj Co-Authored-By: Claude Sonnet 4.6 --- docs/expressions.md | 122 ++++++++++++++++++++++++++++++++++++++ docs/pipeline.md | 70 +++++++++------------- docs/variables.md | 42 +++++++++---- src/pine/api.clj | 25 +++++--- src/pine/ast/assign.clj | 23 +++++++ src/pine/ast/group.clj | 5 +- src/pine/ast/main.clj | 45 +++++++++----- src/pine/ast/order.clj | 7 ++- src/pine/ast/select.clj | 3 +- src/pine/ast/where.clj | 8 ++- src/pine/parser.clj | 27 ++++----- src/pine/pine.bnf | 6 +- test/pine/eval_test.clj | 28 ++++++++- test/pine/hints_test.clj | 6 +- test/pine/parser_test.clj | 51 ++++++++++------ 15 files changed, 340 insertions(+), 128 deletions(-) create mode 100644 docs/expressions.md create mode 100644 src/pine/ast/assign.clj diff --git a/docs/expressions.md b/docs/expressions.md new file mode 100644 index 0000000..6bf69ed --- /dev/null +++ b/docs/expressions.md @@ -0,0 +1,122 @@ +# Expressions + +A unit of Pine input — one pipe chain — and how multiple expressions compose. + +## Why + +Pine is a REPL-style tool. A session naturally builds up state: filter a set, name it, join through it, +aggregate. A single pipe chain can do a lot, but some queries need intermediate steps that are cleaner as +separate named expressions than as deeply nested subqueries. Multiple expressions, evaluated in order, +give you that incremental composition while keeping each step readable on its own. + +## Syntax + +A single expression is one pipe chain: + +``` +company | where: active = true | employee | s: name +``` + +Multiple expressions are separated by blank lines: + +``` +company | where: active = true |= active_companies + +active_companies | employee | s: name +``` + +The API receives them as an array of strings — one string per expression, blank lines already stripped. + +## Operations + +Each pipe segment maps to a typed operation. Operations are applied left to right to build the query state: + +| Pipe segment | What it does | +|------------------|-----------------------------------------------------------------------------| +| `table` | Registers the table, resolves FK joins, tracks current context | +| `select` / `s:` | Records which columns to include | +| `where` / `w:` | Records filter conditions and parameters | +| `group` / `g:` | Records group-by columns and aggregate functions | +| `order` / `o:` | Records sort columns and direction | +| `limit` / `l:` | Records the row limit | +| `count:` | Switches operation mode to COUNT | +| `update!` / `u!` | Records column assignments for UPDATE | +| `delete!` | Marks the operation as DELETE | +| `= name` | Snapshots the current state into `:pending-assignments`; pipeline continues | + +See [pipeline.md](pipeline.md) for how these operations are processed internally. + +## How evaluation works + +Expressions are evaluated left to right. Only the **last expression** produces output (query or result +rows). Earlier expressions exist solely to define variables that the last expression can use. + +``` +expr₁ → variables₁ +expr₂(variables₁) → variables₁₂ +expr₃(variables₁₂) → SQL / rows ← this is what's returned +``` + +Each expression passes through the full parse → generate → eval pipeline independently. Variables +accumulate: every `|= name` op in an expression adds a named snapshot to the variable map, which is +merged into the map passed to the next expression. + +If any expression fails — parse error or generate error — evaluation stops immediately and the error is +returned. The last expression is never run. + +## Constraints + +- Only the last expression's SQL is built or executed. Earlier expressions are only evaluated to collect + variables — they do not produce query output. +- A variable used but not defined in a preceding expression causes a table-not-found error. +- Expressions are stateless per request. No state carries over between API calls. +- Circular variable references are not supported. + +--- + +## Implementation + +### API entry points (`api.clj`) + +`api-build` and `api-eval` both receive an `expressions` array. Both split it the same way: + +- **context expressions** — all but the last, passed to `evaluate-expressions` +- **last expression** — processed separately with the accumulated variable map + +`api-build` returns the query string and AST (used by the UI to show hints and the SQL preview). +`api-eval` runs the query and returns rows. + +### Variable threading (`evaluate-expressions`, `api.clj`) + +```clojure +(reduce (fn [{:keys [variables]} expression] + (let [{:keys [result error]} (generate-state expression nil connection-id variables)] + (if error + (reduced {:error error}) ;; short-circuit on first error + {:variables (merge variables (:pending-assignments result)) + :last-state result}))) + {:variables {} :last-state nil} + expressions) +``` + +Each expression's `:pending-assignments` map (keyed by variable name, valued by state snapshots from +`|=` ops) is merged into the running `variables` map. `reduced` short-circuits the reduce on error so +later expressions are never evaluated. + +### Per-expression pipeline + +Each call to `generate-state` runs: + +1. `parser/parse` — tokenises the expression into a typed operation list. +2. `ast/generate` — folds operations into a state map, seeding the DB schema and any in-scope variables + at the start. `|=` ops produce `:pending-assignments` entries in the returned state. +3. The caller merges `:pending-assignments` into `variables` for the next expression. + +The last expression goes through the same `generate-state` call and then into `build-query` or +`run-query` depending on the endpoint. + +### Hints for the last expression + +`api-build` calls `generate-state` on the last expression **twice**: once with the full expression +(for the AST and query), and once with the trimmed expression (for the `query` field in the response). +Both calls receive the same `variables` map so hints reflect variables defined by earlier expressions. diff --git a/docs/pipeline.md b/docs/pipeline.md index 7daa53c..dd9472f 100644 --- a/docs/pipeline.md +++ b/docs/pipeline.md @@ -36,25 +36,14 @@ Output: [ table(user as u), table(document name="passport"), select(u.email, id), limit(10) ] ``` -Each operation has a type and a value. If the expression ends with `|= name`, that name is extracted -separately and carried alongside the list. +Each operation has a type and a value. `|= name` produces an `:assign` operation in the list like any +other op, so the pipeline can continue after it. ### 2. Generate Each operation is applied left to right to a state map, using the appropriate handler. Before the first -operation runs, the DB schema is loaded so that join paths and column lists are available. - -| Pipe segment | What it adds to state | -|---|---| -| `table` | Registers the table, resolves FK joins, tracks current context | -| `select` / `s:` | Records which columns to include | -| `where` / `w:` | Records filter conditions and parameters | -| `group` / `g:` | Records group-by columns and aggregate functions | -| `order` / `o:` | Records sort columns and direction | -| `limit` / `l:` | Records the row limit | -| `count:` | Switches operation mode to COUNT | -| `update!` / `u!` | Records column assignments for UPDATE | -| `delete!` | Marks the operation as DELETE | +operation runs, the DB schema is loaded so that join paths and column lists are available. For the full +list of operations and what each one does, see [expressions.md](expressions.md). After all operations are applied, a **post-processing** step runs: autocomplete hints are computed, hidden auto-id columns are appended for row tracking, and a prettified form of the expression is attached. @@ -76,9 +65,8 @@ the query in the UI as the user types. cycle from scratch. - Hints are computed inside `generate`, using a second pass over the expression truncated at the cursor position. This gives hints that reflect what the user has typed so far, not the full completed expression. -- When multiple expressions are present (separated by blank lines), they are evaluated left to right. - Each assigned expression (`|= name`) stores its state as a variable, threaded into subsequent calls. - See [variables.md](variables.md). +- When multiple expressions are present, they are evaluated left to right, each seeing variables defined + by earlier ones. See [expressions.md](expressions.md). ## Constraints @@ -93,32 +81,31 @@ the query in the UI as the user types. ### Parse (`parser/parse`, `pine.bnf`) Instaparse runs the BNF grammar over the input string. The raw parse tree is normalised into a vector of -`{:type :value }` maps. `|= name` is extracted as `:assign` and stripped from the -operation list so the rest of the pipeline is unaware of it. +`{:type :value }` maps. `|= name` produces `{:type :assign :value "name"}` in the list +like any other operation. ### Generate (`ast/generate`, `ast/main.clj`) Full signature: ```clojure -(generate parse-tree connection-id expression cursor variables assign) +(generate parse-tree connection-id expression cursor variables) ``` Each parameter fills a distinct role that cannot be inferred from the others: -| Parameter | Source | Used for | -|---|---|---| -| `parse-tree` | `(parser/parse expression)` | `handle-ops` — the typed operation list | -| `connection-id` | caller / DB config | `db/init-references` — loading the schema | -| `expression` | raw input string | `add-prettify` (ranges), `truncate-at-cursor` (cursor hints) | -| `cursor` | UI (user's caret position) | Building `truncated-state` for cursor-aware hints | -| `variables` | caller, accumulated across expressions | `pre-handle` — seeding FK references for CTEs | -| `assign` | `(parser/parse expression)` | Stored on the returned state; not processed as an op | +| Parameter | Source | Used for | +|-----------------|----------------------------------------|--------------------------------------------------------------| +| `parse-tree` | `(parser/parse expression)` | `handle-ops` — the typed operation list | +| `connection-id` | caller / DB config | `db/init-references` — loading the schema | +| `expression` | raw input string | `add-prettify` (ranges), `truncate-at-cursor` (cursor hints) | +| `cursor` | UI (user's caret position) | Building `truncated-state` for cursor-aware hints | +| `variables` | caller, accumulated across expressions | `pre-handle` — seeding FK references for CTEs | -`parse-tree` and `assign` both come from `parser/parse`, but `expression` is still required separately -because `add-prettify` and `truncate-at-cursor` operate on raw text. `cursor` and `variables` are -external context that is not derivable from the expression itself — `cursor` comes from the UI, and -`variables` accumulates as the caller evaluates earlier expressions in a multi-expression session. +`parse-tree` comes from `parser/parse`, but `expression` is still required separately because +`add-prettify` and `truncate-at-cursor` operate on raw text. `cursor` and `variables` are external +context that is not derivable from the expression itself — `cursor` comes from the UI, and `variables` +accumulates as the caller evaluates earlier expressions in a multi-expression session. `generate` orchestrates three sub-steps: @@ -128,8 +115,9 @@ so join resolution works through CTEs. **`handle-ops`** — calls `reduce` over the operation list, dispatching each operation to its handler module (`ast/table`, `ast/select`, `ast/where`, `ast/group`, `ast/order`, `ast/limit`, `ast/count`, -`ast/delete-action`, `ast/update-action`). The operation index (`i`) is threaded through so later -stages know the order columns were introduced. +`ast/delete-action`, `ast/update-action`, `ast/assign`). The operation index (`i`) is threaded through +so later stages know the order columns were introduced. `:assign` ops do not update `:operation` so +`build-query` dispatch always sees the last SQL-producing operation. **`post-handle`** — runs after all operations: - `ast/hints/handle` computes autocomplete hints using the truncated-at-cursor state. @@ -157,13 +145,13 @@ The final state map: `build-query` dispatches on operation type: -| Type | Builder | -|---|---| -| `:select` (default) | `build-select-query` | -| `:count` | `build-count-query` | -| `:group` | `build-group-query` | +| Type | Builder | +|--------------------------------------|------------------------| +| `:select` (default) | `build-select-query` | +| `:count` | `build-count-query` | +| `:group` | `build-group-query` | | `:update-action` / `:update-partial` | `build-update-queries` | -| `:delete-action` | `build-delete-query` | +| `:delete-action` | `build-delete-query` | `build-select-query` checks for variables in scope and, if present, prepends CTE clauses via `collect-ctes`. `run-query` calls `build-query` then executes against the connection pool via `db/run-query`. diff --git a/docs/variables.md b/docs/variables.md index 8c60744..8564a7c 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -12,11 +12,12 @@ name, treat it as a table, and compose them incrementally. ## Syntax ``` - |= + |= [| more-ops...] ``` -Append `|= name` at the end of any pipe chain to assign the result to a variable. Use the name as a -table in any later expression. +Place `|= name` anywhere in a pipe chain to snapshot the query state at that point under `name`. +The pipeline continues after the assignment — operations after `|=` refine the current expression's +output while the snapshot remains frozen. Use `name` as a table in any later expression. ## Examples @@ -28,7 +29,8 @@ company | where: active = true |= active_companies active_companies | employee ``` -`active_companies` becomes a CTE. The second expression joins through it: +`active_companies` becomes a CTE. The second expression joins through it. +`|=` can appear at the end (as here) or mid-pipeline: ```sql WITH "active_companies" AS ( @@ -40,6 +42,17 @@ JOIN "employee" AS "e_1" ON "ac_0"."id" = "e_1"."company_id" LIMIT 250 ``` +### Mid-pipeline assign + +``` +company |= all_companies | where: active = true + +all_companies +``` + +`all_companies` is the full unfiltered company snapshot — the snapshot was taken before `where:`. +The current expression still returns only active companies. + ### Chained Expressions are separated by blank lines and evaluated in order. Each one can build on the last. @@ -67,11 +80,14 @@ x ## How it works -- **Assignment** (`|= name`): the expression is stored as a variable. Its SQL becomes the body of a CTE. +- **Assignment** (`|= name`): snapshots the pipeline state at that point. The snapshot becomes the CTE body + when `name` is used in a later expression. Operations after `|=` continue refining the current expression's + output — they do not affect the snapshot. - **Usage**: when `name` appears as a table in a later expression, Pine substitutes the CTE. Joins resolve in both directions — `variable | table`, `table | variable`, and `variable | variable` — using the FK schema of the underlying real tables. - **Scoping**: variables accumulate left-to-right. Each expression sees all variables defined before it. + Mid-pipeline assignments within the current expression are NOT visible to that same expression. - **Column hints**: when a variable has explicit output columns (from `s:`, `g:`, etc.), hints reflect those columns, not the full schema of the source table. Variables using `*` (no explicit selection) inherit the source table's columns. @@ -91,19 +107,19 @@ x ### Grammar and parsing -`|= name` is parsed as an `assign` operation in `pine.bnf`. The parser (`parser.clj`) extracts the assigned -name separately from the main operation list and returns it as `{:assign "name" :result [...ops...]}`. This -keeps the assign out of the normal operation pipeline so it doesn't interfere with SQL generation. +`|= name` is parsed as a regular pipe operation of type `:assign` in `pine.bnf`. The ASSIGN rule sits +inside OPERATION alongside TABLE, SELECT, WHERE, etc., so it participates in the normal pipeline. +`parser/parse` returns `{:result [ops...]}` with the assign op included in the list. ### State threading The API receives an array of expressions. Each expression is parsed and evaluated in order: -1. `parser/parse(expr)` returns `{:result [ops...] :assign "name"}`. The `assign` field is the variable - name from `|= name` — stripped from the op list so it doesn't enter the pipeline. -2. `ast/generate(ops, ..., variables, assign)` runs the full pipeline and stores `assign` on the returned - state as metadata. -3. If `assign` is set, the caller stores the returned state in the `variables` map under that name. +1. `parser/parse(expr)` returns `{:result [ops...]}`. The operation list may include `:assign` ops. +2. `ast/generate(ops, ..., variables)` runs the full pipeline. When a `:assign` op is encountered, + `assign/handle` snapshots the current state into `:pending-assignments` under the variable name. +3. After `ast/generate` returns, the caller merges `:pending-assignments` from the returned state into + the running `variables` map. 4. The updated `variables` map is passed to `ast/generate` for every subsequent expression. This is done purely in memory within the request — there is no persistence layer. diff --git a/src/pine/api.clj b/src/pine/api.clj index 0763711..02fbd06 100644 --- a/src/pine/api.clj +++ b/src/pine/api.clj @@ -1,4 +1,11 @@ (ns pine.api + "HTTP API layer: routes, request parsing, and multi-expression evaluation. + + Why multi-expression evaluation exists: Pine expressions are designed to be + composed across blank-line-separated blocks. Each block can assign its result + to a variable (|= name) which subsequent blocks use as a CTE. This file + threads variables between expressions so each one sees what earlier ones + produced. The last expression's SQL is the one actually returned or executed." (:require [cheshire.generate :refer [add-encoder encode-str]] [clojure.string :as str] @@ -34,26 +41,28 @@ ([expression cursor connection-id] (generate-state expression cursor connection-id {})) ([expression cursor connection-id variables] - (let [{:keys [result error assign]} (->> expression parser/parse) + (let [{:keys [result error]} (->> expression parser/parse) conn-id (or connection-id @db/connection-id)] (if result - {:result (ast/generate result conn-id expression cursor variables assign)} + {:result (ast/generate result conn-id expression cursor variables)} {:error-type "parse" :error error})))) (defn- evaluate-expressions "Evaluate a sequence of pine expressions, threading variables from |= assignments - into subsequent expressions. Returns {:last-state :error }." + into subsequent expressions. Returns {:last-state :error }. + + Why pending-assignments: |= is now a mid-pipeline op that snapshots state at the + point of assignment. An expression can assign multiple variables before its last op + determines the final SQL. All assignments from one expression become available as + CTE variables to the next." [expressions connection-id] (reduce (fn [{:keys [variables]} expression] (let [{:keys [result error]} (generate-state expression nil connection-id variables)] (if error (reduced {:error error}) - (let [assign (:assign result)] - {:variables (if assign - (assoc variables assign result) - variables) - :last-state result})))) + {:variables (merge variables (:pending-assignments result)) + :last-state result}))) {:variables {} :last-state nil} expressions)) diff --git a/src/pine/ast/assign.clj b/src/pine/ast/assign.clj new file mode 100644 index 0000000..97c5c1e --- /dev/null +++ b/src/pine/ast/assign.clj @@ -0,0 +1,23 @@ +(ns pine.ast.assign + "Records a named checkpoint in the pipeline so the current expression-state + can be referenced as a CTE in later expressions. + + Why assignment is a pipe op (not a terminal): keeping '|= name' as a regular + operation preserves composability. 'company |= x | w: active = true' labels + company as 'x' (unfiltered) while the pipeline continues to filter. Subsequent + expressions that use 'x' get the unfiltered company CTE; the current expression + still produces the filtered SELECT. + + Why a snapshot: storing a dissoc'd copy of state (not the live state) prevents + the pending-assignments map from accumulating nested copies across multiple + assignments in a single expression.") + +(defn handle + "Snapshot the current state under varname in :pending-assignments. + The snapshot excludes :pending-assignments itself to prevent nesting. + Also registers varname as a local alias for the current SQL alias so + subsequent ops in the same expression can write e.g. x.id to mean c_0.id." + [state varname] + (-> state + (assoc-in [:pending-assignments varname] (dissoc state :pending-assignments)) + (assoc :assign varname))) diff --git a/src/pine/ast/group.clj b/src/pine/ast/group.clj index e779424..89bbc5d 100644 --- a/src/pine/ast/group.clj +++ b/src/pine/ast/group.clj @@ -5,6 +5,7 @@ (defn handle [state value] (let [i (state :index) current (state :current) + resolve-alias #(or (get-in state [:pending-assignments % :current]) %) ;; Get existing columns from state (e.g., from previous select with date extraction) existing-columns (state :columns) ;; Filter out auto-id columns from existing columns @@ -34,10 +35,10 @@ ;; Use the existing column (preserves original alias like "t") (take 1 matching) ;; No match, use the group column with default alias - [(assoc g-col :alias (or (:alias g-col) current))]))) + [(assoc g-col :alias (resolve-alias (or (:alias g-col) current)))]))) raw-group-columns) ;; No existing selected columns, use group columns with default alias - (map #(assoc % :alias (or (:alias %) current)) raw-group-columns)) + (map #(assoc % :alias (resolve-alias (or (:alias %) current))) raw-group-columns)) ;; SELECT clause: merged columns + aggregate functions select-columns (concat merged-columns fn-columns)] diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 803afee..3559a30 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -1,6 +1,19 @@ (ns pine.ast.main + "Folds a parsed operation list into a state map that fully describes the query. + + Why a state map: rather than building SQL strings incrementally, each operation + updates a data structure (tables, columns, joins, where clauses, etc.). This + separates intent (what the query means) from rendering (how to express it in SQL), + which makes hints, prettification, and multiple output formats possible without + re-parsing. + + Why variables are seeded before handle-ops: joins are resolved using a references + map built from the DB schema. Variables (CTEs from earlier expressions) need to + appear in that map — with the same FK relationships as their source tables — so + join resolution treats them like real tables." (:require [clojure.string :as str] + [pine.ast.assign :as assign] [pine.ast.count :as pine-count] [pine.ast.delete-action :as delete-action] [pine.ast.from :as from] @@ -21,8 +34,9 @@ ;; - connection :connection-id nil :references {} - :variables {} ;; {"varname" }, populated from |= assignments - :assign nil ;; variable name from |=, set after handle-ops + :variables {} ;; {"varname" }, populated from |= assignments in prior expressions + :assign nil ;; variable name from the last |= op in this expression + :pending-assignments {} ;; {"varname" } accumulated by |= ops in this expression :expression nil ;; Expression string for cursor-aware hints :cursor nil ;; Cursor position {:line N :character M} (zero-indexed) @@ -205,19 +219,23 @@ :delete-action (delete-action/handle state value) :update-action (update-action/handle state value) :update-partial (update-action/handle state value) + :assign (assign/handle state value) ;; No operations :no-op state (update state :errors conj [type "Unknown operation type in parse tree"]))) (defn handle-ops [state ops] (reduce (fn [s [i o]] - (-> s - (assoc :index i) - (handle-op o) ; Pass the index and operation - (update :pending-count dec) - (assoc :operation o))) + (cond-> (-> s + (assoc :index i) + (handle-op o) + (update :pending-count dec)) + ;; :assign does not change the query being built — skip :operation + ;; so build-query dispatch and post-handle selected-tables logic + ;; see the last real SQL-producing operation, not the assign op. + (not= (:type o) :assign) (assoc :operation o))) state - (map-indexed vector ops))) ; Pair each operation with its index + (map-indexed vector ops))) (declare generate) @@ -307,16 +325,15 @@ (defn generate ([parse-tree] - (generate parse-tree @db/connection-id nil nil {} nil)) + (generate parse-tree @db/connection-id nil nil {})) ([parse-tree connection-id] - (generate parse-tree connection-id nil nil {} nil)) + (generate parse-tree connection-id nil nil {})) ([parse-tree connection-id expression cursor] - (generate parse-tree connection-id expression cursor {} nil)) - ([parse-tree connection-id expression cursor variables assign] + (generate parse-tree connection-id expression cursor {})) + ([parse-tree connection-id expression cursor variables] (let [full-state (-> state (pre-handle connection-id (count parse-tree) expression cursor variables) - (handle-ops parse-tree) - (assoc :assign assign)) + (handle-ops parse-tree)) truncated-state (when (and cursor expression) (generate-truncated-state expression cursor connection-id variables))] (post-handle full-state truncated-state)))) diff --git a/src/pine/ast/order.clj b/src/pine/ast/order.clj index 4e2da56..148c244 100644 --- a/src/pine/ast/order.clj +++ b/src/pine/ast/order.clj @@ -1,10 +1,11 @@ (ns pine.ast.order) (defn handle [state value] - (let [i (state :index) - current (state :current) + (let [i (state :index) + current (state :current) + resolve-alias #(or (get-in state [:pending-assignments % :current]) %) columns (map #(-> %1 - (assoc :alias (or (:alias %1) current)) + (assoc :alias (resolve-alias (or (:alias %1) current))) (assoc :operation-index i)) value)] (-> state diff --git a/src/pine/ast/select.clj b/src/pine/ast/select.clj index d3d1709..0a5a37a 100644 --- a/src/pine/ast/select.clj +++ b/src/pine/ast/select.clj @@ -4,9 +4,10 @@ (let [i (state :index) current (state :current) ;; Process each column and handle date functions + resolve-alias #(or (get-in state [:pending-assignments % :current]) %) columns (mapcat (fn [col] (let [col-with-defaults (-> col - (assoc :alias (or (:alias col) current)) + (assoc :alias (resolve-alias (or (:alias col) current))) (assoc :operation-index i))] (if-let [col-fn (:column-function col)] ;; Column function: apply function to column diff --git a/src/pine/ast/where.clj b/src/pine/ast/where.clj index b75fa47..3916d3d 100644 --- a/src/pine/ast/where.clj +++ b/src/pine/ast/where.clj @@ -17,8 +17,9 @@ (defn handle [state [column operator value]] (let [a (state :current) + resolve-alias #(or (get-in state [:pending-assignments % :current]) %) [alias col cast] (:value column) - alias (or alias a) + alias (resolve-alias (or alias a)) converted-value (if (and (not= (:type value) :symbol) (not= (:type value) :column)) (convert-condition-value value alias col state) value)] @@ -27,11 +28,12 @@ (defn handle-partial [state {:keys [complete-conditions partial-condition]}] ;; For WHERE-PARTIAL, we only store the complete conditions in :where ;; The partial condition is used for hints, not for query generation - (let [a (state :current)] + (let [a (state :current) + resolve-alias #(or (get-in state [:pending-assignments % :current]) %)] (reduce (fn [s condition] (let [[column operator value] (:value condition) [alias col cast] (:value column) - alias (or alias a) + alias (resolve-alias (or alias a)) converted-value (if (and (not= (:type value) :symbol) (not= (:type value) :column)) (convert-condition-value value alias col s) value)] diff --git a/src/pine/parser.clj b/src/pine/parser.clj index ed17fec..d2c1405 100644 --- a/src/pine/parser.clj +++ b/src/pine/parser.clj @@ -1,6 +1,14 @@ (ns pine.parser - "The parser is responsible for generating a parse tree from the bnf and - normalize the output which is used for the input for generating the ast" + "Turns raw Pine text into a normalized operation list. + + Why this layer exists: the BNF grammar produces a raw Instaparse tree + that's verbose and shape-inconsistent across operation types. Normalizing + here gives the rest of the system a single, predictable contract: + a vector of {:type :value } maps, one per pipe segment. + + Assignment (|= name) is now a regular pipe operation of type :assign. + It appears in the operation list like any other op, so the pipeline + can continue after it: 'company |= x | w: active = true' is valid." (:require [clojure.core.match :refer [match]] [clojure.string :as s] @@ -469,10 +477,6 @@ (mapv (fn [[_ op]] (-normalize-op op)) (filter #(= (first %) :OPERATION) nodes))) -(defn- extract-assign [[_ & nodes]] - (when-let [node (first (filter #(= (first %) :ASSIGN) nodes))] - (let [[_ [_ varname]] node] varname))) - (defn parse "Parse an expression and return the normalized operations or failure as a string" [expression] @@ -482,8 +486,7 @@ (let [failure (insta/get-failure result) error (with-out-str (println (insta/get-failure result)))] {:error error :failure failure}) - {:result (normalize-ops result) - :assign (extract-assign result)}))) + {:result (normalize-ops result)}))) (defn parse-or-fail [expression] (-> expression parser normalize-ops)) @@ -502,13 +505,7 @@ (let [operations (rest result) op-infos (mapv (fn [op] (let [[start end] (insta/span op) - raw (s/trim (subs expression start end)) - ;; ASSIGN spans include the leading "|" from the - ;; grammar rule. Strip it so the join doesn't - ;; produce a double pipe ("| | =name"). - expr (if (= (first op) :ASSIGN) - (s/trim (s/replace-first raw #"^\|" "")) - raw)] + expr (s/trim (subs expression start end))] {:expression expr :start start :end end})) diff --git a/src/pine/pine.bnf b/src/pine/pine.bnf index 2e1810f..26c325c 100644 --- a/src/pine/pine.bnf +++ b/src/pine/pine.bnf @@ -1,7 +1,7 @@ -OPERATIONS := OPERATION (<"|"> OPERATION )* ASSIGN? -ASSIGN := <"|"> <"="> symbol +OPERATIONS := OPERATION (<"|"> OPERATION )* +ASSIGN := <"="> symbol -OPERATION := SELECT-PARTIAL | SELECT | TABLE | WHERE-PARTIAL | WHERE | LIMIT| FROM | ORDER-PARTIAL | ORDER | COUNT | DELETE-ACTION | DELETE | UPDATE-PARTIAL | UPDATE-ACTION | GROUP +OPERATION := SELECT-PARTIAL | SELECT | WHERE-PARTIAL | WHERE | LIMIT | FROM | ORDER-PARTIAL | ORDER | COUNT | DELETE-ACTION | DELETE | UPDATE-PARTIAL | UPDATE-ACTION | GROUP | ASSIGN | TABLE TABLE := table table-mods diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 8122ec0..80856bd 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -20,9 +20,9 @@ [expressions] (let [{:keys [last-state]} (reduce (fn [{:keys [variables]} expr] - (let [{:keys [result assign]} (parser/parse expr) - state (ast/generate result :test nil nil variables assign)] - {:variables (if assign (assoc variables assign state) variables) + (let [{:keys [result]} (parser/parse expr) + state (ast/generate result :test nil nil variables)] + {:variables (merge variables (:pending-assignments state)) :last-state state})) {:variables {} :last-state nil} expressions)] @@ -442,6 +442,28 @@ (generate-expressions ["employee |= mytest" "company | mytest"])))) + (testing "Mid-pipeline assign: expression continues after |=" + ;; The assign snapshots the state at that point; subsequent ops still apply + (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"c_0\".* FROM \"company\" AS \"c_0\" LIMIT 10" + :params nil} + (generate-expressions ["company |= x | l: 10"]))) + ;; x is the unfiltered snapshot (no limit) — CTE body has no LIMIT + (is (= {:query "WITH \"x\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"x_0\".* FROM \"x\" AS \"x_0\" LIMIT 250" + :params nil} + (generate-expressions ["company |= x | l: 10" + "x"])))) + + (testing "Column reference via variable name resolves to the real SQL alias" + (is (= {:query "SELECT \"e_1\".\"id\", \"c_0\".\"id\", \"c_0\".id AS \"__c_0__id\", \"e_1\".id AS \"__e_1__id\" FROM \"company\" AS \"c_0\" JOIN \"employee\" AS \"e_1\" ON \"c_0\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + :params nil} + (generate-expressions ["company |= c | employee | s: id, c.id"]))) + (is (true? (clojure.string/includes? + (:query (generate-expressions ["company |= c | employee | w: c.id = 1"])) + "WHERE \"c_0\".\"id\" = ?"))) + (is (true? (clojure.string/includes? + (:query (generate-expressions ["company |= c | employee | o: c.id"])) + "ORDER BY \"c_0\".\"id\"")))) + (testing "Same-source variables: two variables wrapping the same table join on id" (is (= {:query "WITH \"c1\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"c2\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"c_1\".* FROM \"c1\" AS \"c_0\" JOIN \"c2\" AS \"c_1\" ON \"c_0\".\"id\" = \"c_1\".\"id\" LIMIT 250" :params nil} diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index 17fc264..8b7f9da 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -20,9 +20,9 @@ [expressions] (let [{:keys [last-hints]} (reduce (fn [{:keys [variables]} expr] - (let [{:keys [result assign]} (parser/parse expr) - state (ast/generate result :test expr nil variables assign)] - {:variables (if assign (assoc variables assign state) variables) + (let [{:keys [result]} (parser/parse expr) + state (ast/generate result :test expr nil variables)] + {:variables (merge variables (:pending-assignments state)) :last-hints (:hints state)})) {:variables {} :last-hints nil} expressions)] diff --git a/test/pine/parser_test.clj b/test/pine/parser_test.clj index 0fcf9f9..8adaf39 100644 --- a/test/pine/parser_test.clj +++ b/test/pine/parser_test.clj @@ -361,12 +361,12 @@ (testing "Trailing pipe is preserved" (is (= {:result "company\n | " :operations [{:expression "company" :start 0 :end 7} - {:expression "" :start 10 :end 10}]} + {:expression "" :start 9 :end 9}]} (prettify "company | "))) (is (= {:result "company\n | where: id = 1\n | " :operations [{:expression "company" :start 0 :end 7} {:expression "where: id = 1" :start 10 :end 23} - {:expression "" :start 26 :end 26}]} + {:expression "" :start 25 :end 25}]} (prettify "company | where: id = 1 | ")))) (testing "Incomplete expressions return an error" @@ -375,32 +375,45 @@ (testing "Assignment is formatted without double pipe" (is (= {:result "company\n | = active_companies" :operations [{:expression "company" :start 0 :end 7} - {:expression "= active_companies" :start 8 :end 27}]} + {:expression "= active_companies" :start 9 :end 27}]} (prettify "company |= active_companies"))) (is (= {:result "company\n | = x" :operations [{:expression "company" :start 0 :end 7} - {:expression "= x" :start 8 :end 13}]} + {:expression "= x" :start 10 :end 13}]} (prettify "company | = x"))) (is (= {:result "company\n | where: active = true\n | = active_companies" :operations [{:expression "company" :start 0 :end 7} {:expression "where: active = true" :start 10 :end 30} - {:expression "= active_companies" :start 31 :end 50}]} + {:expression "= active_companies" :start 32 :end 50}]} (prettify "company | where: active = true |= active_companies"))) - ;; trailing pipe after assignment is a parse error — assignment must be last - (is (contains? (prettify "company |= x | ") :error)))) + ;; |= is mid-pipeline — operations after it are valid + (is (= {:result "company\n | = x\n | " + :operations [{:expression "company" :start 0 :end 7} + {:expression "= x" :start 9 :end 12} + {:expression "" :start 14 :end 14}]} + (prettify "company |= x | "))))) (deftest test-assign - (testing "|= sets :assign in parse result" - (is (= "active_companies" (:assign (parse "company |= active_companies")))) - (is (= "active_companies" (:assign (parse "company | = active_companies")))) - (is (nil? (:assign (parse "company"))))) - - (testing ":result ops do not include the assign node" - (is (= 1 (count (:result (parse "company |= active_companies"))))) + (testing "|= appears in :result as an assign op" + (is (= [{:type :table :value {:table "company"}} + {:type :assign :value "active_companies"}] + (:result (parse "company |= active_companies")))) + (is (= [{:type :table :value {:table "company"}} + {:type :assign :value "active_companies"}] + (:result (parse "company | = active_companies")))) (is (= [{:type :table :value {:table "company"}}] - (:result (parse "company |= active_companies"))))) - - (testing "|= works after a full expression" - (is (= "my_var" (:assign (parse "company | where: id = 1 | s: name |= my_var")))) - (is (= 3 (count (:result (parse "company | where: id = 1 | s: name |= my_var"))))))) + (:result (parse "company"))))) + + (testing ":assign is no longer a top-level field in parse result" + (is (nil? (:assign (parse "company |= active_companies")))) + (is (nil? (:assign (parse "company"))))) + + (testing "|= is mid-pipeline: operations continue after it" + (is (= [{:type :table :value {:table "company"}} + {:type :assign :value "x"} + {:type :table :value {:table "employee"}}] + (:result (parse "company |= x | employee")))) + (is (= 4 (count (:result (parse "company | where: id = 1 | s: name |= my_var"))))) + (is (= {:type :assign :value "my_var"} + (last (:result (parse "company | where: id = 1 | s: name |= my_var"))))))) From bcbdb9b3a3890125604f325e573e97e737c8e358 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sat, 16 May 2026 23:12:38 +0200 Subject: [PATCH 23/60] test: add coverage for cross-expression column refs, update null, parent join, and assign snapshot Co-Authored-By: Claude Sonnet 4.6 --- src/pine/ast/table.clj | 2 +- test/pine/ast_test.clj | 10 ++++++++++ test/pine/eval_test.clj | 41 ++++++++++++++++++++++++++++++----------- 3 files changed, 41 insertions(+), 12 deletions(-) diff --git a/src/pine/ast/table.clj b/src/pine/ast/table.clj index a9ca524..7c050ad 100644 --- a/src/pine/ast/table.clj +++ b/src/pine/ast/table.clj @@ -76,7 +76,7 @@ (defn- handle-as-variable [state value var-ast] (let [index (state :index) {:keys [table alias join-column join-left-column join-right-column join]} value - a (or alias (str (make-alias table) "_" (state :table-count))) + a (or alias table) current {:schema nil :table table :ast var-ast :alias a :join-column join-column :join-left-column join-left-column :join-right-column join-right-column :join join diff --git a/test/pine/ast_test.clj b/test/pine/ast_test.clj index 756bc38..1499492 100644 --- a/test/pine/ast_test.clj +++ b/test/pine/ast_test.clj @@ -296,6 +296,16 @@ ;; (generate :update "customer | update! name = 'John'"))) ) + (testing "Generate ast for `assign`" + ;; The snapshot captures the current table alias at the point of |= + (is (= "c_0" + (get-in (generate :pending-assignments "company |= x") + ["x" :current]))) + ;; Mid-pipeline: snapshot is taken at |=, not at the end of the expression + (is (= "c_0" + (get-in (generate :pending-assignments "company |= x | employee") + ["x" :current])))) + (testing "Schema-based type conversion in WHERE operations" ;; Test that JSONB column gets proper type conversion in WHERE clause (let [where-result (generate :where "customer | w: data = '{\"key\": \"value\"}'")] diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 80856bd..54acc62 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -150,7 +150,11 @@ (generate "company | employee :right"))) (is (= {:query "SELECT \"c_0\".id AS \"__c_0__id\", \"e_1\".id AS \"__e_1__id\", \"d_2\".id AS \"__d_2__id\", \"d_2\".* FROM \"x\".\"company\" AS \"c_0\" LEFT JOIN \"y\".\"employee\" AS \"e_1\" ON \"c_0\".\"id\" = \"e_1\".\"company_id\" RIGHT JOIN \"z\".\"document\" AS \"d_2\" ON \"e_1\".\"id\" = \"d_2\".\"employee_id\" LIMIT 250", :params nil} - (generate "x.company | y.employee :left | z.document :right")))) + (generate "x.company | y.employee :left | z.document :right"))) + ;; Self-join: :parent flips join direction (e_0 has the FK, not e_1) + (is (true? (clojure.string/includes? + (:query (generate "employee | employee :parent")) + "ON \"e_0\".\"reports_to\" = \"e_1\".\"id\"")))) (testing "Joins with explicit columns" ;; Basic explicit join (tables a, b, c don't exist in schema so no auto-id columns) @@ -316,6 +320,10 @@ :query "UPDATE \"company\" SET \"active\" = true WHERE id IN ( SELECT \"c_0\".\"id\" FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"id\" = ? )" :params (list (dt/number "1"))}]} (generate "company | where: id = 1 | update! active = true"))) + (is (= {:queries [{:table "company" + :query "UPDATE \"company\" SET \"deleted_at\" = NULL WHERE id IN ( SELECT \"c_0\".\"id\" FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"id\" = ? )" + :params (list (dt/number "1"))}]} + (generate "company | where: id = 1 | update! deleted_at = null"))) ;; Test JSONB type conversion (is (= {:queries [{:table "customer" @@ -402,25 +410,25 @@ (generate-expressions ["company |= active_companies"])))) (testing "Variable used as table generates CTE" - (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"ac_0\".* FROM \"active_companies\" AS \"ac_0\" LIMIT 250" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"active_companies\".* FROM \"active_companies\" AS \"active_companies\" LIMIT 250" :params nil} (generate-expressions ["company |= active_companies" "active_companies"])))) (testing "Variable with WHERE filter generates filtered CTE" - (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"name\" = ? ) SELECT \"ac_0\".* FROM \"active_companies\" AS \"ac_0\" LIMIT 250" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" WHERE \"c_0\".\"name\" = ? ) SELECT \"active_companies\".* FROM \"active_companies\" AS \"active_companies\" LIMIT 250" :params (map dt/string ["Acme"])} (generate-expressions ["company | where: name = 'Acme' |= active_companies" "active_companies"])))) (testing "Join through a variable resolves correctly" - (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"e_1\".id AS \"__e_1__id\", \"e_1\".* FROM \"active_companies\" AS \"ac_0\" JOIN \"employee\" AS \"e_1\" ON \"ac_0\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"e_1\".id AS \"__e_1__id\", \"e_1\".* FROM \"active_companies\" AS \"active_companies\" JOIN \"employee\" AS \"e_1\" ON \"active_companies\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" :params nil} (generate-expressions ["company |= active_companies" "active_companies | employee"])))) (testing "Composed variables (variable of variable) generates flat CTEs" - (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"small_active\" AS ( SELECT \"ac_0\".* FROM \"active_companies\" AS \"ac_0\" LIMIT 10 ) SELECT \"sa_0\".* FROM \"small_active\" AS \"sa_0\" LIMIT 250" + (is (= {:query "WITH \"active_companies\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"small_active\" AS ( SELECT \"active_companies\".* FROM \"active_companies\" AS \"active_companies\" LIMIT 10 ) SELECT \"small_active\".* FROM \"small_active\" AS \"small_active\" LIMIT 250" :params nil} (generate-expressions ["company |= active_companies" "active_companies | l: 10 |= small_active" @@ -429,7 +437,7 @@ (testing "Reverse join: child table navigates to variable wrapping its parent" ;; employee.company_id -> company.id ;; mytest = company (parent); employee | mytest should join employee to the CTE - (is (= {:query "WITH \"mytest\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"e_0\".id AS \"__e_0__id\", \"m_1\".* FROM \"employee\" AS \"e_0\" JOIN \"mytest\" AS \"m_1\" ON \"e_0\".\"company_id\" = \"m_1\".\"id\" LIMIT 250" + (is (= {:query "WITH \"mytest\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"e_0\".id AS \"__e_0__id\", \"mytest\".* FROM \"employee\" AS \"e_0\" JOIN \"mytest\" AS \"mytest\" ON \"e_0\".\"company_id\" = \"mytest\".\"id\" LIMIT 250" :params nil} (generate-expressions ["company |= mytest" "employee | mytest"])))) @@ -437,7 +445,7 @@ (testing "Reverse join: parent table navigates to variable wrapping its child" ;; employee.company_id -> company.id ;; mytest = employee (child); company | mytest should join company to the CTE - (is (= {:query "WITH \"mytest\" AS ( SELECT \"e_0\".* FROM \"employee\" AS \"e_0\" ) SELECT \"c_0\".id AS \"__c_0__id\", \"m_1\".* FROM \"company\" AS \"c_0\" JOIN \"mytest\" AS \"m_1\" ON \"c_0\".\"id\" = \"m_1\".\"company_id\" LIMIT 250" + (is (= {:query "WITH \"mytest\" AS ( SELECT \"e_0\".* FROM \"employee\" AS \"e_0\" ) SELECT \"c_0\".id AS \"__c_0__id\", \"mytest\".* FROM \"company\" AS \"c_0\" JOIN \"mytest\" AS \"mytest\" ON \"c_0\".\"id\" = \"mytest\".\"company_id\" LIMIT 250" :params nil} (generate-expressions ["employee |= mytest" "company | mytest"])))) @@ -448,12 +456,13 @@ :params nil} (generate-expressions ["company |= x | l: 10"]))) ;; x is the unfiltered snapshot (no limit) — CTE body has no LIMIT - (is (= {:query "WITH \"x\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"x_0\".* FROM \"x\" AS \"x_0\" LIMIT 250" + (is (= {:query "WITH \"x\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"x\".* FROM \"x\" AS \"x\" LIMIT 250" :params nil} (generate-expressions ["company |= x | l: 10" "x"])))) (testing "Column reference via variable name resolves to the real SQL alias" + ;; Within-expression: |= c still routes through pending-assignments → real table alias c_0 (is (= {:query "SELECT \"e_1\".\"id\", \"c_0\".\"id\", \"c_0\".id AS \"__c_0__id\", \"e_1\".id AS \"__e_1__id\" FROM \"company\" AS \"c_0\" JOIN \"employee\" AS \"e_1\" ON \"c_0\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" :params nil} (generate-expressions ["company |= c | employee | s: id, c.id"]))) @@ -462,15 +471,25 @@ "WHERE \"c_0\".\"id\" = ?"))) (is (true? (clojure.string/includes? (:query (generate-expressions ["company |= c | employee | o: c.id"])) - "ORDER BY \"c_0\".\"id\"")))) + "ORDER BY \"c_0\".\"id\""))) + + ;; Cross-expression: CTE alias = variable name, so x.id resolves to "x"."id" + (is (true? (clojure.string/includes? + (:query (generate-expressions ["company |= x" + "x | w: x.id = 1"])) + "WHERE \"x\".\"id\" = ?"))) + (is (true? (clojure.string/includes? + (:query (generate-expressions ["company |= x" + "x | o: x.id"])) + "ORDER BY \"x\".\"id\"")))) (testing "Same-source variables: two variables wrapping the same table join on id" - (is (= {:query "WITH \"c1\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"c2\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"c_1\".* FROM \"c1\" AS \"c_0\" JOIN \"c2\" AS \"c_1\" ON \"c_0\".\"id\" = \"c_1\".\"id\" LIMIT 250" + (is (= {:query "WITH \"c1\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"c2\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"c2\".* FROM \"c1\" AS \"c1\" JOIN \"c2\" AS \"c2\" ON \"c1\".\"id\" = \"c2\".\"id\" LIMIT 250" :params nil} (generate-expressions ["company |= c1" "company |= c2" "c1 | c2"]))) - (is (= {:query "WITH \"c2\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"c1\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"c_1\".* FROM \"c2\" AS \"c_0\" JOIN \"c1\" AS \"c_1\" ON \"c_0\".\"id\" = \"c_1\".\"id\" LIMIT 250" + (is (= {:query "WITH \"c2\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"c1\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"c1\".* FROM \"c2\" AS \"c2\" JOIN \"c1\" AS \"c1\" ON \"c2\".\"id\" = \"c1\".\"id\" LIMIT 250" :params nil} (generate-expressions ["company |= c1" "company |= c2" From cb6137da122369ec347d0cb01ad8bb249ea1fef0 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sat, 16 May 2026 23:26:53 +0200 Subject: [PATCH 24/60] fix: use explicit alias for column lookup in where, update, and select-partial hints When a partial condition/column had an explicit alias (e.g. w: e.company_id), hints were incorrectly looking up columns from the current context table instead of the aliased table. Also, after a comma in select-partial, hints now default to the current context table instead of the last completed column's table. Co-Authored-By: Claude Sonnet 4.6 --- src/pine/ast/hints.clj | 39 +++++++++++++++++++++++---------------- test/pine/hints_test.clj | 13 +++++++++++++ 2 files changed, 36 insertions(+), 16 deletions(-) diff --git a/src/pine/ast/hints.clj b/src/pine/ast/hints.clj index 1b150d4..66cd18e 100644 --- a/src/pine/ast/hints.clj +++ b/src/pine/ast/hints.clj @@ -133,20 +133,22 @@ ;; - Order partial: simple exclude-already-selected logic works with generic generate-column-hints ;; - Where partial: complex state-dependent filtering based on partial completion: ;; * `where:` → show all columns - ;; * `w: i` → filter to columns matching "i" + ;; * `w: i` → filter to columns matching "i" + ;; * `w: e.i` → filter to columns matching "i" from alias "e" ;; * `w: id =` → show all columns (ready for next condition) (let [operation (state :operation) where-data (get operation :value {}) partial-condition (:partial-condition where-data) - current-alias (state :current)] + current-alias (state :current) + lookup-alias (or (and partial-condition (:alias partial-condition)) current-alias)] (if (nil? partial-condition) ;; No partial condition yet, show all columns (generate-all-column-hints state current-alias) ;; Has partial condition, filter based on it (if (and (:column partial-condition) (not (contains? partial-condition :operator))) - ;; Just a column specified, filter hints for that column + ;; Just a column specified — use the condition's alias if present, else current (find-relevant-columns - (generate-all-column-hints state current-alias) + (generate-all-column-hints state lookup-alias) partial-condition) ;; Column + operator, show all columns for next condition (generate-all-column-hints state current-alias))))) @@ -155,7 +157,8 @@ (let [assignments (get-in state [:update :assignments] []) partial-column (get-in state [:update :partial-column]) assigned-columns (map (fn [a] {:column (get-in a [:column :column])}) assignments) - hints (-> (generate-all-column-hints state (state :current)) + lookup-alias (or (and partial-column (:alias partial-column)) (state :current)) + hints (-> (generate-all-column-hints state lookup-alias) (exclude-columns assigned-columns))] (if partial-column (find-relevant-columns hints partial-column) @@ -163,18 +166,22 @@ (defn generate-column-hints [state columns] ;; Generic column hints logic that works for most operations: - ;; - Complete operations (select, order, where): filter hints to match current column being typed - ;; - Partial operations (select-partial, order-partial): exclude already-selected columns - ;; - Note: where-partial needs custom logic (see generate-where-hints) due to complex state-dependent filtering + ;; - Complete operations (select, order): filter hints to match the column being typed, using its alias + ;; - Partial operations (select-partial, order-partial): exclude already-selected columns, + ;; defaulting to the current context table so the next slot shows relevant columns + ;; - Note: where-partial needs custom logic (see generate-where-hints) (let [column (some-> columns reverse first) - ;; Use operation-index to determine table context for hints - ;; If column was added in a later operation than current context, use that column's alias - a (if (and (seq column) - (> (column :operation-index) (state :current-index))) - (column :alias) - (state :current)) - hints (generate-all-column-hints state a) - type (-> state :operation :type)] + type (-> state :operation :type) + ;; For partial ops, always use the current context table — the user hasn't started typing + ;; the next column yet, so the last completed column's alias is irrelevant. + ;; For complete ops, use the typed column's alias when it was added in the current scope. + a (if (or (= type :select-partial) (= type :order-partial)) + (state :current) + (if (and (seq column) + (> (column :operation-index) (state :current-index))) + (column :alias) + (state :current))) + hints (generate-all-column-hints state a)] (cond ;; If the type is :select, :order, or :where and columns exist, filter hints using the columns (and (or (= type :select) (= type :order) (= type :where)) column) diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index 8b7f9da..444a084 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -108,6 +108,10 @@ (is (= ["id" "company_id" "reports_to"] (->> "company | s: id | employee | s: " gen :select (map :column)))) (is (= ["company_id" "reports_to"] (->> "company | s: id | employee | s: id, " gen :select (map :column)))) + ;; Cross-table: after selecting from company, next slot defaults to current context (employee) + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | s: c.id," gen :select (map :column)))) + ;; The following doesn't get parsed at the moment ;; We need to update the pine.bnf to support the syntax ;; @@ -131,6 +135,12 @@ (is (= [] (-> "company | w: xyz" gen :where))) (is (= ["id" "company_id"] (->> "y.employee | w: id" gen :where (map :column)))) + ;; Explicit alias: w: e.col should use alias "e" for column lookup, not the current context + (is (= [{:column "company_id" :alias "e"}] + (-> "y.employee as e | x.company as c | w: e.company_id" gen :where))) + (is (= [{:column "id" :alias "c"}] + (-> "y.employee as e | x.company as c | w: c.id" gen :where))) + ;; How to auto-complete the right hand side? Values or other columns? ;; Right now it shows the same hints as the left hand side ;; (is (= [{:column "id" :alias "c_0"}] (-> "company | w: id =" gen :where))) @@ -140,6 +150,9 @@ (testing "Generate `update-partial` hints" (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | u!" gen :update))) + ;; Explicit alias in update: u! e.col should use alias "e" for column lookup, not current context + (is (= [{:column "company_id" :alias "e"}] + (-> "y.employee as e | x.company as c | u! e.company_id" gen :update))) (is (= [{:column "created_at" :alias "c_0"}] (-> "company | u! id = '1'," gen :update))) (is (= [{:column "id" :alias "c_0"}] From 8749d7699ea444066d57eaeecc64befa00d5fcbc Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 17 May 2026 08:06:49 +0200 Subject: [PATCH 25/60] feat: support alias-dot partial syntax in all partial operations Add `alias <".">` (e.g., `e.`) as a valid partial form in s:, w:, o:, and u! operations so the LSP can generate column hints for a specific alias even when no column name has been typed yet. - Grammar: new `partial-alias` rule; SELECT-PARTIAL, ORDER-PARTIAL, WHERE-PARTIAL (via partial-condition), and UPDATE-PARTIAL now accept it alongside their existing forms, including the "columns, alias." composite case (e.g. `s: id, e.`) - Parser: `parse-partial-alias` helper; SELECT/ORDER-PARTIAL handlers rewritten to detect the partial-alias child node and attach it as `:partial-alias` on the op map; UPDATE-PARTIAL gets two new match arms; WHERE-PARTIAL encodes it inside `partial-condition` - Hints: `generate-column-hints` reads `:partial-alias` from the operation for select/order-partial before falling back to current Co-Authored-By: Claude Sonnet 4.6 --- src/pine/ast/hints.clj | 2 +- src/pine/parser.clj | 58 +++++++++++++++++++++++++++++++++------ src/pine/pine.bnf | 9 +++--- test/pine/hints_test.clj | 25 ++++++++++++----- test/pine/parser_test.clj | 31 ++++++++++++++++----- 5 files changed, 98 insertions(+), 27 deletions(-) diff --git a/src/pine/ast/hints.clj b/src/pine/ast/hints.clj index 66cd18e..0661159 100644 --- a/src/pine/ast/hints.clj +++ b/src/pine/ast/hints.clj @@ -176,7 +176,7 @@ ;; the next column yet, so the last completed column's alias is irrelevant. ;; For complete ops, use the typed column's alias when it was added in the current scope. a (if (or (= type :select-partial) (= type :order-partial)) - (state :current) + (or (get-in state [:operation :partial-alias :alias]) (state :current)) (if (and (seq column) (> (column :operation-index) (state :current-index))) (column :alias) diff --git a/src/pine/parser.clj b/src/pine/parser.clj index d2c1405..a8d9883 100644 --- a/src/pine/parser.clj +++ b/src/pine/parser.clj @@ -87,6 +87,12 @@ [:aliased-column [:column [:alias [:symbol a]] [:symbol c] [:column-function fn]] [:alias [:symbol ca]]] {:alias a :column c :column-function fn :column-alias ca} :else (throw (ex-info "Unknown COLUMN operation" {:_ column})))) +(defn- parse-partial-alias [partial-alias] + (match partial-alias + [:partial-alias [:alias [:symbol a]]] + {:alias a :column ""} + :else (throw (ex-info "Unknown partial alias" {:_ partial-alias})))) + (defn normalize-select [payload type] (match payload [:aliased-columns & columns] {:type type :value (mapv -normalize-column columns)} @@ -95,10 +101,22 @@ (defmethod -normalize-op :SELECT [[_ payload]] (normalize-select payload :select)) -(defmethod -normalize-op :SELECT-PARTIAL [[_ payload]] - (if (empty? payload) - {:type :select-partial, :value []} - (normalize-select payload :select-partial))) +(defmethod -normalize-op :SELECT-PARTIAL [operation] + (let [children (vec (rest operation)) + partial-alias-node (first (filter #(= (first %) :partial-alias) children)) + aliased-columns-node (first (filter #(= (first %) :aliased-columns) children))] + (cond + (nil? aliased-columns-node) + (if partial-alias-node + {:type :select-partial :value [] :partial-alias (parse-partial-alias partial-alias-node)} + {:type :select-partial :value []}) + + partial-alias-node + (assoc (normalize-select aliased-columns-node :select-partial) + :partial-alias (parse-partial-alias partial-alias-node)) + + :else + (normalize-select aliased-columns-node :select-partial)))) ;; ------ ;; ORDER @@ -125,10 +143,22 @@ (defmethod -normalize-op :ORDER [[_ payload]] (-normalize-order payload :order)) -(defmethod -normalize-op :ORDER-PARTIAL [[_ payload]] - (if (empty? payload) - {:type :order-partial, :value []} - (-normalize-order payload :order-partial))) +(defmethod -normalize-op :ORDER-PARTIAL [operation] + (let [children (vec (rest operation)) + partial-alias-node (first (filter #(= (first %) :partial-alias) children)) + order-columns-node (first (filter #(= (first %) :order-columns) children))] + (cond + (nil? order-columns-node) + (if partial-alias-node + {:type :order-partial :value [] :partial-alias (parse-partial-alias partial-alias-node)} + {:type :order-partial :value []}) + + partial-alias-node + (assoc (-normalize-order order-columns-node :order-partial) + :partial-alias (parse-partial-alias partial-alias-node)) + + :else + (-normalize-order order-columns-node :order-partial)))) ;; ----- ;; WHERE @@ -304,6 +334,10 @@ (defn- parse-partial-condition [partial-condition] (match partial-condition + ;; Alias-dot only (e.g. "e.") + [:partial-condition [:partial-alias [:alias [:symbol a]]]] + {:alias a :column ""} + ;; Just a column [:partial-condition [:column [:symbol column]]] {:column column} @@ -437,6 +471,10 @@ [] {:type :update-partial :value {:assignments [] :partial-column nil}} + [[:partial-update-column [:partial-alias [:alias [:symbol a]]]]] + {:type :update-partial :value {:assignments [] + :partial-column {:alias a :column ""}}} + [[:partial-update-column column-pattern]] {:type :update-partial :value {:assignments [] :partial-column (extract-column-info column-pattern)}} @@ -445,6 +483,10 @@ {:type :update-partial :value {:assignments (mapv parse-update-assignment assignments) :partial-column nil}} + [[:update-assignments & assignments] [:partial-update-column [:partial-alias [:alias [:symbol a]]]]] + {:type :update-partial :value {:assignments (mapv parse-update-assignment assignments) + :partial-column {:alias a :column ""}}} + [[:update-assignments & assignments] [:partial-update-column column-pattern]] {:type :update-partial :value {:assignments (mapv parse-update-assignment assignments) :partial-column (extract-column-info column-pattern)}} diff --git a/src/pine/pine.bnf b/src/pine/pine.bnf index 26c325c..381fe53 100644 --- a/src/pine/pine.bnf +++ b/src/pine/pine.bnf @@ -6,7 +6,7 @@ OPERATION := SELECT-PARTIAL | SELECT | WHERE-PARTIAL | WHERE | LIMIT | FROM | TABLE := table table-mods -SELECT-PARTIAL := aliased-columns? <",">? +SELECT-PARTIAL := aliased-columns <","> partial-alias | partial-alias | aliased-columns? <",">? SELECT := aliased-columns WHERE-PARTIAL := conditions <","> partial-condition | partial-condition | @@ -14,7 +14,7 @@ WHERE := conditions | conditions LIMIT := number | number FROM := alias -ORDER-PARTIAL := order-columns? <",">? +ORDER-PARTIAL := order-columns <","> partial-alias | partial-alias | order-columns? <",">? ORDER := order-columns DELETE-ACTION := hint-column @@ -46,12 +46,13 @@ aggregate-function := "count" | "string_agg" | "sum" | "avg" | "min" | "max" update-assignments := update-assignment (<","> update-assignment)* update-assignment := column <"="> (column|string|number|boolean|null|date) -partial-update-column := column +partial-update-column := partial-alias | column conditions := condition (<","> condition)* condition := column (condition-default|condition-in) ( cast)? -partial-condition := column operator | column +partial-condition := partial-alias | column operator | column +partial-alias := alias <"."> := operator (column|string|number|boolean|null|date) := (in | not-in) <"("> strings* <")"> cast := <"::"> ("text"|"uuid") diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index 444a084..c674b10 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -112,16 +112,21 @@ (is (= ["id" "company_id" "reports_to"] (->> "x.company as c | y.employee as e | s: c.id," gen :select (map :column)))) - ;; The following doesn't get parsed at the moment - ;; We need to update the pine.bnf to support the syntax - ;; - ;; (is (= ["reports_to" "company_id" "id"] (->> "employee as e | company | s: id, e." gen :select (map :column)))) - ) + ;; Alias-dot partial: s: e. should show all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | s: e." gen :select (map :column)))) + + ;; Alias-dot after completed column: s: id, e. should show all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "employee as e | company | s: id, e." gen :select (map :column))))) (testing "Generate `order-partial` hints" (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | o:" gen :order))) (is (= [{:column "created_at" :alias "c_0"}] (-> "company | o: id," gen :order))) - (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | s: id | o:" gen :order)))) + (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | s: id | o:" gen :order))) + ;; Alias-dot partial: o: e. should show all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | o: e." gen :order (map :column))))) (testing "Generate `where-partial` hints" (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | where:" gen :where))) @@ -140,6 +145,9 @@ (-> "y.employee as e | x.company as c | w: e.company_id" gen :where))) (is (= [{:column "id" :alias "c"}] (-> "y.employee as e | x.company as c | w: c.id" gen :where))) + ;; Alias-dot partial: w: e. should show all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | w: e." gen :where (map :column)))) ;; How to auto-complete the right hand side? Values or other columns? ;; Right now it shows the same hints as the left hand side @@ -156,7 +164,10 @@ (is (= [{:column "created_at" :alias "c_0"}] (-> "company | u! id = '1'," gen :update))) (is (= [{:column "id" :alias "c_0"}] - (-> "company | u! i" gen :update)))) + (-> "company | u! i" gen :update))) + ;; Alias-dot partial: u! e. should show all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | u! e." gen :update (map :column))))) (testing "Generate hints with cursor position" ;; Basic cursor truncation test - cursor at "company | s: " should show select hints for company diff --git a/test/pine/parser_test.clj b/test/pine/parser_test.clj index 8adaf39..004df06 100644 --- a/test/pine/parser_test.clj +++ b/test/pine/parser_test.clj @@ -99,10 +99,11 @@ (is (= [{:type :select-partial, :value []}] (p "s: "))) (is (= [{:type :select-partial, :value [{:column "id"}]}] (p "s: id,"))) (is (= [{:type :select-partial, :value [{:column "id"}]}] (-> "company | s: id," p rest))) - ;; Not supported yet - ;; (is (= [{:type :select-partial, :value [{:alias "c" :column ""}]}] (-> "company as c | s: c." p rest))) - ;; (is (= [{:type :select-partial, :value [{:alias "c" :column ""}]}] (-> "company as c | s: id, c." p rest))) - ) + ;; Alias-dot partial + (is (= [{:type :select-partial :value [] :partial-alias {:alias "e" :column ""}}] + (-> "company | s: e." p rest))) + (is (= [{:type :select-partial :value [{:column "id"}] :partial-alias {:alias "e" :column ""}}] + (-> "company | s: id, e." p rest)))) (testing "Parse `select` expressions" (is (= [{:type :select, :value [{:column "name"}]}] (p "select: name"))) @@ -162,12 +163,20 @@ (is (= [{:type :where-partial, :value {:complete-conditions [] :partial-condition {:column "id"}}}] (p "w: id"))) (is (= [{:type :where-partial, :value {:complete-conditions [] :partial-condition {:alias "u", :column "id"}}}] (p "w: u.id"))) (is (= [{:type :where-partial, :value {:complete-conditions [] :partial-condition {:column "id", :operator :equals}}}] (p "w: id ="))) - (is (= [{:type :where-partial, :value {:complete-conditions [] :partial-condition {:column "id", :operator :like}}}] (p "w: id like")))) + (is (= [{:type :where-partial, :value {:complete-conditions [] :partial-condition {:column "id", :operator :like}}}] (p "w: id like"))) + ;; Alias-dot partial + (is (= [{:type :where-partial :value {:complete-conditions [] :partial-condition {:alias "e" :column ""}}}] + (-> "company | w: e." p rest)))) (testing "Parse `order-partial` expressions" (is (= [{:type :order-partial, :value []}] (p "order:"))) (is (= [{:type :order-partial, :value []}] (p "o: "))) - (is (= [{:type :order-partial, :value [{:column "id", :direction "DESC"}]}] (p "o: id,")))) + (is (= [{:type :order-partial, :value [{:column "id", :direction "DESC"}]}] (p "o: id,"))) + ;; Alias-dot partial + (is (= [{:type :order-partial :value [] :partial-alias {:alias "e" :column ""}}] + (-> "company | o: e." p rest))) + (is (= [{:type :order-partial :value [{:column "id" :direction "DESC"}] :partial-alias {:alias "e" :column ""}}] + (-> "company | o: id, e." p rest)))) (testing "Parse `order` expressions" (is (= [{:type :order, :value [{:column "name" :direction "DESC"}]}] (p "order: name"))) @@ -224,7 +233,15 @@ (is (= [{:type :update-partial :value {:assignments [{:column {:alias nil :column "id"} :value (dt/string "1")}] :partial-column {:alias nil :column "n"}}}] - (p "u! id = '1', n")))) + (p "u! id = '1', n"))) + ;; Alias-dot partial + (is (= {:type :update-partial :value {:assignments [] + :partial-column {:alias "e" :column ""}}} + (-> "employee as e | company | u! e." p last))) + (is (= {:type :update-partial :value {:assignments [{:column {:alias nil :column "id"} + :value (dt/string "1")}] + :partial-column {:alias "e" :column ""}}} + (-> "employee as e | company | u! id = '1', e." p last)))) (testing "Parse No Operation expressions" (is (= [{:value {:table "company"}, :type :table} {:type :delete, :value nil}] (p "company | d:")))) From 411d4d235a31010e503bf7d2df22ad79ae4992d9 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Sun, 17 May 2026 11:28:23 +0200 Subject: [PATCH 26/60] test: add interaction coverage for alias-dot partial hints MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cover three gaps where the partial-alias path crosses existing logic: - s: e.id, e. → exclude already-selected alias column - w: id = 1, e. → complete condition then alias-dot shows right table - u! id = '1', e. → prior assignment excluded from alias-dot results Co-Authored-By: Claude Sonnet 4.6 --- test/pine/hints_test.clj | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index c674b10..b0343f7 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -118,7 +118,11 @@ ;; Alias-dot after completed column: s: id, e. should show all columns for alias e (is (= ["id" "company_id" "reports_to"] - (->> "employee as e | company | s: id, e." gen :select (map :column))))) + (->> "employee as e | company | s: id, e." gen :select (map :column)))) + + ;; Alias-dot excludes already-selected columns for that alias: s: e.id, e. omits id + (is (= ["company_id" "reports_to"] + (->> "x.company as c | y.employee as e | s: e.id, e." gen :select (map :column))))) (testing "Generate `order-partial` hints" (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | o:" gen :order))) @@ -149,6 +153,10 @@ (is (= ["id" "company_id" "reports_to"] (->> "x.company as c | y.employee as e | w: e." gen :where (map :column)))) + ;; Complete condition then alias-dot: w: id = 1, e. shows all columns for alias e + (is (= ["id" "company_id" "reports_to"] + (->> "x.company as c | y.employee as e | w: id = 1, e." gen :where (map :column)))) + ;; How to auto-complete the right hand side? Values or other columns? ;; Right now it shows the same hints as the left hand side ;; (is (= [{:column "id" :alias "c_0"}] (-> "company | w: id =" gen :where))) @@ -167,7 +175,11 @@ (-> "company | u! i" gen :update))) ;; Alias-dot partial: u! e. should show all columns for alias e (is (= ["id" "company_id" "reports_to"] - (->> "x.company as c | y.employee as e | u! e." gen :update (map :column))))) + (->> "x.company as c | y.employee as e | u! e." gen :update (map :column)))) + + ;; Prior assignment then alias-dot: u! id = '1', e. excludes already-assigned id + (is (= ["company_id" "reports_to"] + (->> "x.company as c | y.employee as e | u! id = '1', e." gen :update (map :column))))) (testing "Generate hints with cursor position" ;; Basic cursor truncation test - cursor at "company | s: " should show select hints for company From fcc737c94c6ff11db32577b3f57cab2082df91d7 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 18 May 2026 01:03:50 +0200 Subject: [PATCH 27/60] feat: auto-seal group/limit results as CTEs when a table op follows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GROUP and LIMIT produce bounded result sets that cannot be directly joined — further table ops after them produce malformed SQL. This change detects when a table operation follows a GROUP or LIMIT op within the same expression and automatically wraps the preceding query in an anonymous CTE (__pine_0__, etc.), then continues the pipeline on top of it. An optional |= name between the checkpoint op and the following table lets the user name the CTE explicitly instead of using the auto-name. Introduces: checkpoint-op-types, reset-for-cte, seal-as-cte, flush-checkpoint in ast/main.clj; pending-assignments lookup in table/handle; :ast-based CTE guard in add-auto-id-columns. Co-Authored-By: Claude Sonnet 4.6 --- docs/checkpoints.md | 142 ++++++++++++++++++++++++++++++++++++++++ docs/variables.md | 2 + src/pine/ast/main.clj | 95 ++++++++++++++++++++++++--- src/pine/ast/select.clj | 2 +- src/pine/ast/table.clj | 3 +- test/pine/eval_test.clj | 30 ++++++++- 6 files changed, 262 insertions(+), 12 deletions(-) create mode 100644 docs/checkpoints.md diff --git a/docs/checkpoints.md b/docs/checkpoints.md new file mode 100644 index 0000000..e364200 --- /dev/null +++ b/docs/checkpoints.md @@ -0,0 +1,142 @@ +# Checkpoints + +Automatically seal a GROUP or LIMIT result into an anonymous CTE so subsequent table operations compose on top of it. + +## Why + +GROUP and LIMIT produce a final, bounded result set — further joins appended after them produce malformed SQL (a GROUP BY inside a subquery that is then joined, or a LIMIT applied before the join filter). Checkpoints solve this by detecting when a table operation follows a GROUP or LIMIT and implicitly wrapping the preceding query in a CTE, then continuing the pipeline on top of that CTE. + +Without checkpoints, `company | l: 10 | employee` would attempt to build a single query with LIMIT and a JOIN, which contradicts the intent (get 10 companies, then navigate to their employees). + +## Syntax + +No syntax change is required. Checkpoints fire automatically when a TABLE operation follows a GROUP or LIMIT: + +``` + | (group:|limit:) ... | [| more-ops...] +``` + +Optionally assign the auto-generated CTE a name by placing `|= name` between the checkpoint op and the table: + +``` + | (group:|limit:) ... |= |
[| more-ops...] +``` + +## Examples + +### LIMIT checkpoint (auto-named) + +``` +x.company | l: 10 | employee +``` + +Pine seals `x.company | l: 10` as an anonymous CTE `__pine_0__` and joins `employee` to it: + +```sql +WITH "__pine_0__" AS ( + SELECT "c_0".* FROM "x"."company" AS "c_0" LIMIT 10 +) +SELECT "e_1".id AS "__e_1__id", "e_1".* +FROM "__pine_0__" AS "__pine_0__" +JOIN "employee" AS "e_1" ON "__pine_0__"."id" = "e_1"."company_id" +LIMIT 250 +``` + +### LIMIT checkpoint (user-named via `|=`) + +``` +x.company | l: 10 |= pg | employee +``` + +Same result but the CTE is named `pg`: + +```sql +WITH "pg" AS ( + SELECT "c_0".* FROM "x"."company" AS "c_0" LIMIT 10 +) +SELECT "e_1".id AS "__e_1__id", "e_1".* +FROM "pg" AS "pg" +JOIN "employee" AS "e_1" ON "pg"."id" = "e_1"."company_id" +LIMIT 250 +``` + +### GROUP checkpoint + +``` +x.company | group: id => count | employee +``` + +The GROUP result is sealed as a CTE and employee is joined to it: + +```sql +WITH "__pine_0__" AS ( + SELECT "c_0"."id", COUNT(1) AS "count" + FROM "x"."company" AS "c_0" + GROUP BY "c_0"."id" +) +SELECT "e_1".id AS "__e_1__id", "e_1".* +FROM "__pine_0__" AS "__pine_0__" +JOIN "employee" AS "e_1" ON "__pine_0__"."id" = "e_1"."company_id" +LIMIT 250 +``` + +### No checkpoint when non-table op follows + +``` +company | limit: 100 | count: +``` + +`count:` is not a table op — no checkpoint fires. The pipeline remains a single query and COUNT wraps it normally: + +```sql +WITH x AS ( SELECT "c_0".* FROM "company" AS "c_0" LIMIT 100 ) +SELECT COUNT(*) FROM x +``` + +## How it works + +- **Detection**: after processing a GROUP or LIMIT op, `handle-ops` sets `:pending-checkpoint {:needs-assign true}` on the state. +- **Fire on table**: at the start of each `handle-ops` iteration, `flush-checkpoint` checks whether a checkpoint is pending and whether the incoming op is a TABLE op. If both are true, the current state is snapshotted into `:pending-assignments` under an auto-generated name (`__pine_0__`, `__pine_1__`, ...), the references map is seeded for that name (same FK propagation used by variables), and the state is reset. The CTE name is then injected as the first table so subsequent ops join to it. +- **User-named CTE**: if an `|= name` op appears between the checkpoint op and the table op, `flush-checkpoint` records the name and waits for the table. `assign/handle` stores the snapshot under `name` as normal. When the table op arrives, `activate-checkpoint-cte` seeds references for the named CTE and resets state. +- **Hold for non-table ops**: if the op after GROUP/LIMIT is something other than a table or assign (e.g. `count:`, `where:`), the checkpoint stays pending and the op is processed normally without firing. +- **SQL generation**: because the CTE table has an `:ast` entry in `:aliases`, `collect-ctes` automatically picks it up and emits the `WITH` clause. No changes to `eval.clj` were needed. + +## Constraints + +- Checkpoint op types are GROUP and LIMIT. ORDER does not create a checkpoint. +- Only one table-level composition step is supported per checkpoint; chaining `l: 10 | employee | document` creates one CTE for `l: 10` and then navigates normally through employee to document. +- Auto-generated CTE names (`__pine_0__`, etc.) are numbered per expression and are not exposed to the user. +- Checkpoint CTEs do not receive Pine's auto-id columns; those are suppressed for all CTE-backed tables. + +See also: [variables.md](variables.md) for cross-expression named CTEs. + +--- + +## Implementation + +### State fields (`ast/main.clj`) + +| Field | Default | Purpose | +|---|---|---| +| `:auto-cte-count` | `0` | Counter for generating `__pine_0__`, `__pine_1__`, etc. | +| `:pending-checkpoint` | `nil` | `{:needs-assign true}` after a checkpoint op; `{:name n :needs-table true}` after an explicit `|=` | + +### Core functions (`ast/main.clj`) + +- `checkpoint-op-types` — set `#{:group :limit}` +- `reset-for-cte` — clears all query-building fields (tables, columns, joins, where, order, group, limit) while preserving references, variables, and pending-assignments +- `seal-as-cte` — the single shared action: stores the snapshot under `cname` in `:pending-assignments`, seeds FK references for `cname` (same three passes used by cross-expression variables in `pre-handle`), resets query-building state, then injects `cname` as the first table via `handle-op`; `table/handle` resolves it via `:pending-assignments` +- `flush-checkpoint` — state-machine dispatch called at the start of each `handle-ops` reduce iteration; calls `seal-as-cte` when a TABLE op follows the checkpoint, using either a freshly-snapshotted state (auto-named) or the snapshot already stored by `assign/handle` (user-named) + +### `table/handle` (`ast/table.clj`) + +Updated to check `:pending-assignments` alongside `:variables` when looking up the CTE for a table name: + +```clojure +var-ast (or (get-in state [:variables table]) + (get-in state [:pending-assignments table])) +``` + +### `add-auto-id-columns` (`ast/select.clj`) + +Changed the predicate from checking `variables` membership to checking `(:ast (get aliases %))`. Any alias that carries an `:ast` entry is a CTE-backed table and should not receive an auto-id column. diff --git a/docs/variables.md b/docs/variables.md index 8564a7c..724fa32 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -101,6 +101,8 @@ x - A variable used but not defined in a preceding expression causes a table-not-found error. - Circular references are not supported. +See also: [checkpoints.md](checkpoints.md) for same-expression anonymous CTEs after GROUP/LIMIT. + --- ## Implementation diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 3559a30..aa65c1a 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -80,7 +80,17 @@ ;; POST ;; --- ;; - hints - :hints {:table [] :select [] :order [] :where [] :update []}}) + :hints {:table [] :select [] :order [] :where [] :update []} + + ;; --- + ;; Checkpoint + ;; --- + ;; Tracks auto-generated CTE names (__pine_0__, __pine_1__, ...) + :auto-cte-count 0 + ;; Set after a checkpoint op (group/limit) to signal the next table op + ;; should become a CTE instead of a direct join. + ;; nil | {:needs-assign true} | {:name "n" :needs-table true} + :pending-checkpoint nil}) (defn- variable-output-columns "Return the column list a variable's CTE actually exposes, for hint generation. @@ -203,6 +213,72 @@ (assoc :cursor cursor) (assoc :variables variables)))) +;; --------------------------------------------------------------------------- +;; Checkpoint helpers +;; --------------------------------------------------------------------------- + +(declare handle-op) + +(def ^:private checkpoint-op-types #{:group :limit}) + +(defn- reset-for-cte [state] + (assoc state + :tables [] :columns [] :limit nil :joins [] + :where [] :order [] :group [] :update nil + :current nil :context nil :current-index 0 + :table-count 0 :operation {:type nil :value nil})) + +(defn- seal-as-cte + "Store snapshot under cname, seed its join references, reset the query-building + state, then inject cname as the first table so subsequent ops compose on top of it." + [state cname snapshot] + (let [new-refs (-> (:references state) + (seed-variable-references snapshot cname) + (patch-variable-relations {cname snapshot}) + (patch-same-source-variable-joins {cname snapshot}))] + (-> state + (assoc :references new-refs) + (assoc-in [:pending-assignments cname] snapshot) + reset-for-cte + (handle-op {:type :table :value {:table cname}})))) + +(defn- flush-checkpoint + "State-machine step: called at the start of each handle-ops iteration. + Converts a pending checkpoint into a CTE when the right op type is seen." + [state op] + (let [checkpoint (:pending-checkpoint state)] + (cond + (nil? checkpoint) + state + + ;; Explicit assign after a checkpoint op: record the user name, wait for table + (and (:needs-assign checkpoint) (= (:type op) :assign)) + (assoc state :pending-checkpoint {:name (:value op) :needs-table true}) + + ;; Table without explicit assign: auto-generate a CTE name + (and (:needs-assign checkpoint) (= (:type op) :table)) + (let [n (:auto-cte-count state) + cname (str "__pine_" n "__") + snapshot (dissoc state :pending-assignments)] + (-> state + (update :auto-cte-count inc) + (assoc :pending-checkpoint nil) + (seal-as-cte cname snapshot))) + + ;; Waiting for assign but got some other op (e.g. where, limit) — hold + (:needs-assign checkpoint) + state + + ;; User explicitly named the CTE; snapshot already stored by assign/handle + (and (:needs-table checkpoint) (= (:type op) :table)) + (let [cname (:name checkpoint) + snapshot (get-in state [:pending-assignments cname])] + (-> state + (assoc :pending-checkpoint nil) + (seal-as-cte cname snapshot))) + + :else state))) + (defn handle-op [state {:keys [type value]}] (case type :select (select/handle state value) @@ -226,14 +302,15 @@ (defn handle-ops [state ops] (reduce (fn [s [i o]] - (cond-> (-> s - (assoc :index i) - (handle-op o) - (update :pending-count dec)) - ;; :assign does not change the query being built — skip :operation - ;; so build-query dispatch and post-handle selected-tables logic - ;; see the last real SQL-producing operation, not the assign op. - (not= (:type o) :assign) (assoc :operation o))) + ;; flush-checkpoint runs before handle-op; it may reset state and inject + ;; a synthetic CTE table — the :index must be set first so the injected + ;; table gets the correct operation index. + (let [s (-> s (assoc :index i) (flush-checkpoint o))] + (cond-> (-> s + (handle-op o) + (update :pending-count dec)) + (not= (:type o) :assign) (assoc :operation o) + (checkpoint-op-types (:type o)) (assoc :pending-checkpoint {:needs-assign true})))) state (map-indexed vector ops))) diff --git a/src/pine/ast/select.clj b/src/pine/ast/select.clj index 0a5a37a..e438e89 100644 --- a/src/pine/ast/select.clj +++ b/src/pine/ast/select.clj @@ -59,7 +59,7 @@ aliases (:aliases state) variables (:variables state) ;; Only create auto-ID columns for real tables (not variables/CTEs) that have an 'id' column - valid-aliases (filter #(and (not (contains? variables (get-in aliases [% :table]))) + valid-aliases (filter #(and (not (:ast (get aliases %))) (has-id-column? references aliases %)) table-aliases) auto-id-columns (map-indexed #(create-auto-id-column %2 (+ next-operation-index %1)) valid-aliases)] diff --git a/src/pine/ast/table.clj b/src/pine/ast/table.clj index 7c050ad..fa2eeaa 100644 --- a/src/pine/ast/table.clj +++ b/src/pine/ast/table.clj @@ -93,7 +93,8 @@ ;; todo: spec for the :value for a :table (defn handle [state value] (let [{:keys [table]} value - var-ast (get-in state [:variables table])] + var-ast (or (get-in state [:variables table]) + (get-in state [:pending-assignments table]))] (if var-ast (handle-as-variable state value var-ast) (handle-as-table state value)))) diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 54acc62..b21e6bf 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -493,4 +493,32 @@ :params nil} (generate-expressions ["company |= c1" "company |= c2" - "c2 | c1"]))))) \ No newline at end of file + "c2 | c1"]))))) + +(deftest test-checkpoints + (testing "LIMIT checkpoint: auto-CTE when a table op follows limit" + (is (= {:query "WITH \"__pine_0__\" AS ( SELECT \"c_0\".* FROM \"x\".\"company\" AS \"c_0\" LIMIT 10 ) SELECT \"e_1\".id AS \"__e_1__id\", \"e_1\".* FROM \"__pine_0__\" AS \"__pine_0__\" JOIN \"employee\" AS \"e_1\" ON \"__pine_0__\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + :params nil} + (generate "x.company | l: 10 | employee")))) + + (testing "LIMIT checkpoint with explicit user-named CTE via |=" + (is (= {:query "WITH \"pg\" AS ( SELECT \"c_0\".* FROM \"x\".\"company\" AS \"c_0\" LIMIT 10 ) SELECT \"e_1\".id AS \"__e_1__id\", \"e_1\".* FROM \"pg\" AS \"pg\" JOIN \"employee\" AS \"e_1\" ON \"pg\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + :params nil} + (generate "x.company | l: 10 |= pg | employee")))) + + (testing "GROUP checkpoint: auto-CTE when a table op follows group" + (is (= {:query "WITH \"__pine_0__\" AS ( SELECT \"c_0\".\"id\", COUNT(1) AS \"count\" FROM \"x\".\"company\" AS \"c_0\" GROUP BY \"c_0\".\"id\" ) SELECT \"e_1\".id AS \"__e_1__id\", \"e_1\".* FROM \"__pine_0__\" AS \"__pine_0__\" JOIN \"employee\" AS \"e_1\" ON \"__pine_0__\".\"id\" = \"e_1\".\"company_id\" LIMIT 250" + :params nil} + (generate "x.company | group: id => count | employee")))) + + (testing "Checkpoint does not fire for non-table ops after limit" + ;; LIMIT followed by COUNT: checkpoint holds, COUNT builds its own wrapper CTE + (is (= {:query "WITH x AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" LIMIT 100 ) SELECT COUNT(*) FROM x" + :params nil} + (generate "company | limit: 100 | count:")))) + + (testing "Standalone GROUP without following table still uses build-group-query" + ;; Existing behaviour must not regress: the GROUP dispatch path is unaffected + (is (clojure.string/includes? + (:query (generate "x.company | group: id => count")) + "GROUP BY")))) \ No newline at end of file From 616ea5d6058dcacdbfecbc1d8f0b0a096d845c0d Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 18 May 2026 02:27:42 +0200 Subject: [PATCH 28/60] fix: fire checkpoint when another checkpoint op follows (GROUP | LIMIT) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When LIMIT followed GROUP, flush-checkpoint held instead of firing — then handle-ops overwrote the GROUP checkpoint with a LIMIT checkpoint, silently dropping the CTE wrapping. Same bug applied to |= x | l: 1. Fix: flush-checkpoint now fires when the incoming op is in checkpoint-op-types (GROUP or LIMIT), not only when it's a :table op. count: is unaffected since it is not a checkpoint op type. Adds tests for GROUP | LIMIT (auto-named and user-named) and a cross-expression baseline that verifies all three forms produce equivalent SQL. Co-Authored-By: Claude Sonnet 4.6 --- docs/checkpoints.md | 6 +++--- src/pine/ast/main.clj | 29 ++++++++++++++++++----------- test/pine/eval_test.clj | 26 +++++++++++++++++++++++++- 3 files changed, 46 insertions(+), 15 deletions(-) diff --git a/docs/checkpoints.md b/docs/checkpoints.md index e364200..da5d0cc 100644 --- a/docs/checkpoints.md +++ b/docs/checkpoints.md @@ -96,9 +96,9 @@ SELECT COUNT(*) FROM x ## How it works - **Detection**: after processing a GROUP or LIMIT op, `handle-ops` sets `:pending-checkpoint {:needs-assign true}` on the state. -- **Fire on table**: at the start of each `handle-ops` iteration, `flush-checkpoint` checks whether a checkpoint is pending and whether the incoming op is a TABLE op. If both are true, the current state is snapshotted into `:pending-assignments` under an auto-generated name (`__pine_0__`, `__pine_1__`, ...), the references map is seeded for that name (same FK propagation used by variables), and the state is reset. The CTE name is then injected as the first table so subsequent ops join to it. -- **User-named CTE**: if an `|= name` op appears between the checkpoint op and the table op, `flush-checkpoint` records the name and waits for the table. `assign/handle` stores the snapshot under `name` as normal. When the table op arrives, `activate-checkpoint-cte` seeds references for the named CTE and resets state. -- **Hold for non-table ops**: if the op after GROUP/LIMIT is something other than a table or assign (e.g. `count:`, `where:`), the checkpoint stays pending and the op is processed normally without firing. +- **Fire on table or checkpoint op**: at the start of each `handle-ops` iteration, `flush-checkpoint` checks whether a checkpoint is pending and whether the incoming op is a TABLE op or another checkpoint op (GROUP or LIMIT). If so, the current state is snapshotted into `:pending-assignments` under an auto-generated name (`__pine_0__`, `__pine_1__`, ...), the references map is seeded for that name (same FK propagation used by variables), and the state is reset. The CTE name is then injected as the first table so subsequent ops compose on top of it. A LIMIT following a GROUP therefore fires the GROUP checkpoint, then applies the limit to the outer query. +- **User-named CTE**: if an `|= name` op appears between the checkpoint op and the following op, `flush-checkpoint` records the name and waits. `assign/handle` stores the snapshot under `name` as normal. When the next table or checkpoint op arrives, `seal-as-cte` activates it. +- **Hold for non-triggering ops**: if the op after GROUP/LIMIT is something that has its own query-building path — `count:`, `delete:`, etc. — the checkpoint stays pending and that op is processed without firing the checkpoint. - **SQL generation**: because the CTE table has an `:ast` entry in `:aliases`, `collect-ctes` automatically picks it up and emits the `WITH` clause. No changes to `eval.clj` were needed. ## Constraints diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index aa65c1a..0a39f76 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -244,33 +244,40 @@ (defn- flush-checkpoint "State-machine step: called at the start of each handle-ops iteration. - Converts a pending checkpoint into a CTE when the right op type is seen." + Converts a pending checkpoint into a CTE when the right op type is seen. + + Fires when the incoming op is a table (join composition) or another checkpoint + op (e.g. LIMIT after GROUP). Does not fire for count/delete/update since those + have their own query-building paths that do not need CTE separation." [state op] - (let [checkpoint (:pending-checkpoint state)] + (let [checkpoint (:pending-checkpoint state) + op-type (:type op) + fire? (or (= op-type :table) + (contains? checkpoint-op-types op-type))] (cond (nil? checkpoint) state - ;; Explicit assign after a checkpoint op: record the user name, wait for table - (and (:needs-assign checkpoint) (= (:type op) :assign)) + ;; Explicit assign after a checkpoint op: record the user name, wait + (and (:needs-assign checkpoint) (= op-type :assign)) (assoc state :pending-checkpoint {:name (:value op) :needs-table true}) - ;; Table without explicit assign: auto-generate a CTE name - (and (:needs-assign checkpoint) (= (:type op) :table)) - (let [n (:auto-cte-count state) - cname (str "__pine_" n "__") + ;; Auto-named: fire when a table or another checkpoint op arrives + (and (:needs-assign checkpoint) fire?) + (let [n (:auto-cte-count state) + cname (str "__pine_" n "__") snapshot (dissoc state :pending-assignments)] (-> state (update :auto-cte-count inc) (assoc :pending-checkpoint nil) (seal-as-cte cname snapshot))) - ;; Waiting for assign but got some other op (e.g. where, limit) — hold + ;; Waiting for assign, non-triggering op (count, where, etc.) — hold (:needs-assign checkpoint) state - ;; User explicitly named the CTE; snapshot already stored by assign/handle - (and (:needs-table checkpoint) (= (:type op) :table)) + ;; User-named: fire when a table or another checkpoint op arrives + (and (:needs-table checkpoint) fire?) (let [cname (:name checkpoint) snapshot (get-in state [:pending-assignments cname])] (-> state diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index b21e6bf..69833c2 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -521,4 +521,28 @@ ;; Existing behaviour must not regress: the GROUP dispatch path is unaffected (is (clojure.string/includes? (:query (generate "x.company | group: id => count")) - "GROUP BY")))) \ No newline at end of file + "GROUP BY"))) + + (testing "GROUP followed by LIMIT seals group as CTE, applies limit to outer" + ;; All three forms should produce equivalent SQL (modulo CTE name) + (let [expected-cte-body "SELECT \"c_0\".\"id\", COUNT(1) AS \"count\" FROM \"x\".\"company\" AS \"c_0\" GROUP BY \"c_0\".\"id\"" + ;; Auto-named + q1 (:query (generate "x.company | group: id => count | l: 1")) + ;; User-named via |= + q2 (:query (generate "x.company | group: id => count |= grp | l: 1"))] + ;; Both contain the same GROUP CTE body + (is (clojure.string/includes? q1 expected-cte-body)) + (is (clojure.string/includes? q2 expected-cte-body)) + ;; Both apply LIMIT 1 on the outer query + (is (clojure.string/ends-with? q1 "LIMIT 1")) + (is (clojure.string/ends-with? q2 "LIMIT 1")) + ;; Auto-named CTE uses generated name; user-named uses "grp" + (is (clojure.string/includes? q1 "\"__pine_0__\"")) + (is (clojure.string/includes? q2 "\"grp\"")))) + + (testing "GROUP + LIMIT cross-expression baseline matches same-expression form" + ;; x | l: 1 in a separate expression should produce the same structure + (let [q-cross (:query (generate-expressions ["x.company | group: id => count |= grp" + "grp | l: 1"])) + q-same (:query (generate "x.company | group: id => count |= grp | l: 1"))] + (is (= q-cross q-same))))) \ No newline at end of file From a8042f2207b03ff509fd6f07a1408ce06cab3a0f Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 18 May 2026 09:38:15 +0200 Subject: [PATCH 29/60] feat: expose pending-assignments in build API response Projects the current expression's checkpoint CTEs (sealed GROUP/LIMIT results) into the API ast payload so the frontend can render them as container nodes. Co-Authored-By: Claude Sonnet 4.6 --- src/pine/api.clj | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/pine/api.clj b/src/pine/api.clj index 02fbd06..f47ad5e 100644 --- a/src/pine/api.clj +++ b/src/pine/api.clj @@ -96,7 +96,10 @@ {:connection-id connection-name :version version :query (-> last-expr trim-pipes (generate-state nil conn-id variables) :result eval/build-query eval/formatted-query) - :ast (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges :variables :assign])})))))) + :ast (-> (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges :variables :assign]) + (assoc :pending-assignments + (into {} (for [[k v] (:pending-assignments state)] + [k (select-keys v [:tables :selected-tables :joins :columns])]))))})))))) (catch Exception e {:connection-id connection-name :error (.getMessage e)}))))) From d3c8ee8a592ad5afe20e6bda049ace319b28a8dc Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 20 Jul 2026 15:51:01 +0200 Subject: [PATCH 30/60] test: use customer instead of company in same-source variable-join test Pre-existing uncommitted fixture edit; company works identically here in isolation, but customer (no FK relations in test fixtures) is the more minimal, isolated choice for a test that's specifically about same-source variable joins rather than real FK-derived ones. --- test/pine/hints_test.clj | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index b0343f7..f0d72f6 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -231,6 +231,6 @@ ;; when typing "var_x | var", var_y should appear (partial match, unambiguous token) (is (= [{:schema nil :table "var_y" :column nil :parent false :heuristic false :pine "var_y"}] - (-> (gen-with-variables ["company |= var_x" "company |= var_y" "var_x | var"]) + (-> (gen-with-variables ["customer |= var_x" "customer |= var_y" "var_x | var"]) :table))))) From 5331fd81fa54148302a6b53b48b6f0518603896d Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 20 Jul 2026 15:52:52 +0200 Subject: [PATCH 31/60] =?UTF-8?q?perf:=20replace=20O(v=C2=B7T)=20schema=20?= =?UTF-8?q?scan=20with=20reverse=20index=20in=20patch-direction?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit patch-direction scanned every table in the schema (T) for every variable (v) to find entities that already relate to that variable's source table. With a reverse index built once (source-table -> referring entities), each variable only visits entities that actually have a relation to patch, cutting this from O(v · T) to O(T + v · k) where k is real fan-in per source table. The index is threaded alongside refs and updated on every write so entries added mid-pass stay visible — this matters for variable-of-variable composition, where a later variable's source table is an earlier variable processed in the same pass. --- src/pine/ast/main.clj | 52 ++++++++++++++++++++++++++++++------------- 1 file changed, 37 insertions(+), 15 deletions(-) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 0a39f76..582276f 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -139,24 +139,46 @@ (assoc-in [:table varname :column-set] (set output-cols))) seeded))) +(defn- build-direction-index + "Reverse index: source-table -> {entity-name -> existing-relation}, for one + direction. Lets patch-direction look up 'who already relates to S' directly + instead of scanning every table in the schema per variable." + [refs direction] + (reduce-kv (fn [idx entity-name entity-refs] + (reduce-kv (fn [idx target existing] + (assoc-in idx [target entity-name] existing)) + idx + (get entity-refs direction))) + {} + (get refs :table {}))) + (defn- patch-direction "For each variable V wrapping source S, copy T[direction][S] → T[direction][V] - for every entity T in the references map. Used by patch-variable-relations." + for every entity T that already relates to S. Used by patch-variable-relations. + + Uses a reverse index (source-table -> referring entities) built once instead + of scanning every schema table per variable, which turns this from O(v · T) + into O(T + v · k) where k is the actual fan-in for each source table. The + index is kept in sync with entries this pass adds, since a later variable's + source table can itself be an earlier variable processed in this same pass + (variable-of-variable composition)." [refs variables direction] - (reduce (fn [r [varname var-ast]] - (let [source-tables (get-source-tables var-ast)] - (reduce (fn [r source-table] - (reduce (fn [r entity-name] - (let [existing (get-in r [:table entity-name direction source-table])] - (if existing - (update-in r [:table entity-name direction varname] merge existing) - r))) - r - (keys (get r :table {})))) - r - source-tables))) - refs - variables)) + (:refs + (reduce + (fn [{:keys [refs index]} [varname var-ast]] + (reduce + (fn [{:keys [refs index]} source-table] + (reduce + (fn [{:keys [refs index]} [entity-name existing]] + (let [refs* (update-in refs [:table entity-name direction varname] merge existing) + merged (get-in refs* [:table entity-name direction varname])] + {:refs refs* :index (assoc-in index [varname entity-name] merged)})) + {:refs refs :index index} + (get index source-table))) + {:refs refs :index index} + (get-source-tables var-ast))) + {:refs refs :index (build-direction-index refs direction)} + variables))) (defn- patch-variable-relations "Bidirectional pass: for each variable V wrapping source S, find every From 904fbe0a762149dce892523f0125ca8b30246b03 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 20 Jul 2026 15:53:23 +0200 Subject: [PATCH 32/60] perf: group by source table before pairing in patch-same-source-variable-joins MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previously scanned every ordered pair among all variables regardless of whether they shared a source table, making the has-id?/shared-sources check run v² times even when no two variables were related. Grouping variables by source table first means only variables that could plausibly match are ever paired, degrading to v² only in the (rare) case where a whole group shares one source — which is the actual output size in that case, not avoidable overhead. --- src/pine/ast/main.clj | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 582276f..6d13789 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -199,13 +199,29 @@ "For each ordered pair of distinct variables (v1, v2) that share a source table with an 'id' column, register a synthetic id=id join at refs[:table v1 :referred-by v2]. Only adds entries where no join path already exists, so self-referential FK propagation - (e.g. employee/reports_to) is preserved." + (e.g. employee/reports_to) is preserved. + + Pairs are generated only within groups of variables that actually share a source + table (via an inverted source-table -> variables index), instead of scanning every + ordered pair among all variables. This turns the common case (variables spread + across distinct source tables) from O(v²) into roughly O(v), while the worst case + (all variables sharing one source) stays O(g²) for that group — unavoidable since + the output itself is a g² set of pairwise joins." [refs variables] (let [var-sources (->> variables (map (fn [[vname var-ast]] [vname (set (get-source-tables var-ast))])) (into {})) - vnames (vec (keys var-sources))] + by-source (reduce-kv (fn [idx vname sources] + (reduce (fn [idx source] (update idx source (fnil conj []) vname)) + idx + sources)) + {} + var-sources) + pairs (distinct (for [[_ vnames] by-source + v1 vnames v2 vnames + :when (not= v1 v2)] + [v1 v2]))] (reduce (fn [r [v1 v2]] (let [shared-sources (filter (var-sources v1) (var-sources v2)) has-id? (fn [tbl] (some #(= "id" (:column %)) (get-in r [:table tbl :columns])))] @@ -216,7 +232,7 @@ [nil v1 "id" :referred-by nil v2 "id" :variable-join]) r))) refs - (for [v1 vnames v2 vnames :when (not= v1 v2)] [v1 v2])))) + pairs))) (defn pre-handle [state connection-id ops-count expression cursor variables] (let [refs (db/init-references connection-id) From 8b31fa939de7c0a17510d120147c8b0bfaa51d19 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 20 Jul 2026 15:54:00 +0200 Subject: [PATCH 33/60] test: add regression test for alias re-binding after |= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit resolve-alias unconditionally prefers a |= variable's snapshot alias over a same-named alias assigned later via 'as' in the same expression. Repro: 'company |= x | employee as x | w: x.id = 1' filters on company's id instead of employee's — silently wrong SQL, no error. Currently failing; fix follows. --- test/pine/eval_test.clj | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 69833c2..a03af18 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -483,6 +483,17 @@ "x | o: x.id"])) "ORDER BY \"x\".\"id\"")))) + (testing "Re-aliasing a variable name via 'as' takes precedence over the |= snapshot" + ;; x is bound to company via |=, then re-aliased to employee via `as x`. + ;; The live alias should win: x.id must mean employee.id (the most recent, + ;; explicit binding), not the stale company snapshot captured at the |= point. + (is (true? (clojure.string/includes? + (:query (generate-expressions ["company |= x | employee as x | w: x.id = 1"])) + "WHERE \"x\".\"id\" = ?"))) + (is (false? (clojure.string/includes? + (:query (generate-expressions ["company |= x | employee as x | w: x.id = 1"])) + "WHERE \"c_0\".\"id\" = ?")))) + (testing "Same-source variables: two variables wrapping the same table join on id" (is (= {:query "WITH \"c1\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ), \"c2\" AS ( SELECT \"c_0\".* FROM \"company\" AS \"c_0\" ) SELECT \"c2\".* FROM \"c1\" AS \"c1\" JOIN \"c2\" AS \"c2\" ON \"c1\".\"id\" = \"c2\".\"id\" LIMIT 250" :params nil} From 4b36778668a82f6ae991df37982ea85d3de99015 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 20 Jul 2026 15:54:42 +0200 Subject: [PATCH 34/60] fix: prefer live alias over stale |= snapshot when resolving column aliases MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit resolve-alias in where/select/order/group always translated a token through :pending-assignments first, even when that token had since been re-bound to a different table via an explicit `as` in the same expression. Since `as` registers a live entry in :aliases, checking that first makes the most recent, explicit binding win — matching what the user actually wrote. Fixes the regression test added in the previous commit. --- src/pine/ast/group.clj | 3 ++- src/pine/ast/order.clj | 3 ++- src/pine/ast/select.clj | 3 ++- src/pine/ast/where.clj | 5 +++-- 4 files changed, 9 insertions(+), 5 deletions(-) diff --git a/src/pine/ast/group.clj b/src/pine/ast/group.clj index 89bbc5d..73f2526 100644 --- a/src/pine/ast/group.clj +++ b/src/pine/ast/group.clj @@ -5,7 +5,8 @@ (defn handle [state value] (let [i (state :index) current (state :current) - resolve-alias #(or (get-in state [:pending-assignments % :current]) %) + ;; A live alias (e.g. re-bound via `as`) always wins over a stale |= snapshot + resolve-alias #(if (contains? (:aliases state) %) % (or (get-in state [:pending-assignments % :current]) %)) ;; Get existing columns from state (e.g., from previous select with date extraction) existing-columns (state :columns) ;; Filter out auto-id columns from existing columns diff --git a/src/pine/ast/order.clj b/src/pine/ast/order.clj index 148c244..9911e78 100644 --- a/src/pine/ast/order.clj +++ b/src/pine/ast/order.clj @@ -3,7 +3,8 @@ (defn handle [state value] (let [i (state :index) current (state :current) - resolve-alias #(or (get-in state [:pending-assignments % :current]) %) + ;; A live alias (e.g. re-bound via `as`) always wins over a stale |= snapshot + resolve-alias #(if (contains? (:aliases state) %) % (or (get-in state [:pending-assignments % :current]) %)) columns (map #(-> %1 (assoc :alias (resolve-alias (or (:alias %1) current))) (assoc :operation-index i)) diff --git a/src/pine/ast/select.clj b/src/pine/ast/select.clj index e438e89..d8bc045 100644 --- a/src/pine/ast/select.clj +++ b/src/pine/ast/select.clj @@ -4,7 +4,8 @@ (let [i (state :index) current (state :current) ;; Process each column and handle date functions - resolve-alias #(or (get-in state [:pending-assignments % :current]) %) + ;; A live alias (e.g. re-bound via `as`) always wins over a stale |= snapshot + resolve-alias #(if (contains? (:aliases state) %) % (or (get-in state [:pending-assignments % :current]) %)) columns (mapcat (fn [col] (let [col-with-defaults (-> col (assoc :alias (resolve-alias (or (:alias col) current))) diff --git a/src/pine/ast/where.clj b/src/pine/ast/where.clj index 3916d3d..d4acae2 100644 --- a/src/pine/ast/where.clj +++ b/src/pine/ast/where.clj @@ -17,7 +17,8 @@ (defn handle [state [column operator value]] (let [a (state :current) - resolve-alias #(or (get-in state [:pending-assignments % :current]) %) + ;; A live alias (e.g. re-bound via `as`) always wins over a stale |= snapshot + resolve-alias #(if (contains? (:aliases state) %) % (or (get-in state [:pending-assignments % :current]) %)) [alias col cast] (:value column) alias (resolve-alias (or alias a)) converted-value (if (and (not= (:type value) :symbol) (not= (:type value) :column)) @@ -29,7 +30,7 @@ ;; For WHERE-PARTIAL, we only store the complete conditions in :where ;; The partial condition is used for hints, not for query generation (let [a (state :current) - resolve-alias #(or (get-in state [:pending-assignments % :current]) %)] + resolve-alias #(if (contains? (:aliases state) %) % (or (get-in state [:pending-assignments % :current]) %))] (reduce (fn [s condition] (let [[column operator value] (:value condition) [alias col cast] (:value column) From dd080cb6cc8fd346b16450d92977230941020762 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 20 Jul 2026 15:55:36 +0200 Subject: [PATCH 35/60] test: add regression test for GROUP followed by complete select: crashing x.company | group: id => count | s: id throws an NPE in hints/generate-column-hints: the group's COUNT(1) aggregate column has no :operation-index, and (> nil (state :current-index)) blows up. select: isn't a checkpoint-firing op, so the group's columns are still live when hints run. Currently erroring; fix follows. --- test/pine/eval_test.clj | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index a03af18..2e46cc2 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -556,4 +556,13 @@ (let [q-cross (:query (generate-expressions ["x.company | group: id => count |= grp" "grp | l: 1"])) q-same (:query (generate "x.company | group: id => count |= grp | l: 1"))] - (is (= q-cross q-same))))) \ No newline at end of file + (is (= q-cross q-same)))) + + (testing "GROUP followed by a complete select: does not crash generating hints" + ;; select: is not a checkpoint-firing op (unlike table/group/limit), so the + ;; group's aggregate columns (e.g. COUNT(1), which has no :operation-index) + ;; are still in state when hints are generated — this used to throw an NPE + ;; in hints/generate-column-hints comparing a nil :operation-index with >. + (is (clojure.string/includes? + (:query (generate "x.company | group: id => count | s: id")) + "GROUP BY")))) \ No newline at end of file From 85cd0da652b884fef7bf666638664d60c05badb5 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Mon, 20 Jul 2026 15:58:21 +0200 Subject: [PATCH 36/60] fix: stop GROUP from producing a plain seq that reverses later column order MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit group.clj built :columns via (concat merged-columns fn-columns), a lazy seq. Later ops that grow :columns via (update :columns into ...) (e.g. select.clj) call conj under the hood, which prepends onto a plain seq instead of appending onto a vector — silently reversing column order so the GROUP's aggregate column was no longer last. hints/generate-column-hints assumes the last :columns entry is the one just typed and reads its :operation-index, which the aggregate column doesn't have, throwing an NPE the moment a non-checkpoint op (e.g. select:) follows GROUP. Fixes it at the source (:columns and :group are now real vectors), plus a defensive nil-guard in hints.clj so a column missing :operation-index for any other reason falls back to :current instead of crashing. Fixes the regression test added in the previous commit. --- src/pine/ast/group.clj | 7 +++++-- src/pine/ast/hints.clj | 1 + 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/src/pine/ast/group.clj b/src/pine/ast/group.clj index 73f2526..358fb2e 100644 --- a/src/pine/ast/group.clj +++ b/src/pine/ast/group.clj @@ -42,9 +42,12 @@ (map #(assoc % :alias (resolve-alias (or (:alias %) current))) raw-group-columns)) ;; SELECT clause: merged columns + aggregate functions - select-columns (concat merged-columns fn-columns)] + ;; Must be a vector: :columns is later grown with (update :columns into ...) + ;; elsewhere (e.g. select.clj), and `into`/`conj` prepend onto a plain seq + ;; (concat's return type) instead of appending, silently reversing order. + select-columns (vec (concat merged-columns fn-columns))] (-> state (assoc :columns select-columns) ;; GROUP BY uses all the merged columns - (assoc :group merged-columns)))) + (assoc :group (vec merged-columns))))) diff --git a/src/pine/ast/hints.clj b/src/pine/ast/hints.clj index 0661159..de89b92 100644 --- a/src/pine/ast/hints.clj +++ b/src/pine/ast/hints.clj @@ -178,6 +178,7 @@ a (if (or (= type :select-partial) (= type :order-partial)) (or (get-in state [:operation :partial-alias :alias]) (state :current)) (if (and (seq column) + (column :operation-index) (> (column :operation-index) (state :current-index))) (column :alias) (state :current))) From 2bff55f392a1072e7fbfd83142c3dbd17232a4dc Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 09:22:37 +0200 Subject: [PATCH 37/60] docs: update variables.md for the indexed patch-direction/patch-same-source-variable-joins passes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pass 2 previously described the O(v·T) "iterate every entity" scan; Pass 3 previously described an unconditional all-pairs scan. Both were replaced with indexed approaches (reverse index for Pass 2, source-table grouping for Pass 3) — this updates the doc to describe the actual mechanism instead of the superseded one. Also fixes "two passes" -> "three passes" in the section intro, which was already inconsistent with the doc describing three. --- docs/variables.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/variables.md b/docs/variables.md index 724fa32..eb63bb8 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -135,23 +135,28 @@ query's params list. ### Join resolution through variables -Reference seeding happens in two passes inside `pre-handle` (`ast/main.clj`): +Reference seeding happens in three passes inside `pre-handle` (`ast/main.clj`): **Pass 1 — `seed-variable-references`**: For each variable V wrapping source table S, copies S's FK reference entry into the references map under V's name. This enables `V | table` and `table | V` when the join helper can find `table[:referred-by][V]` — but only if that entry already exists, which it doesn't yet after pass 1 alone. -**Pass 2 — `patch-variable-relations`**: Iterates every entity (real table or already-seeded variable) -in the references map. For each entity T where `T[:referred-by][S]` exists, registers `T[:referred-by][V]` -with the same relation data. This propagates the relationship bidirectionally: +**Pass 2 — `patch-variable-relations`** (runs `patch-direction` once per direction): Builds a reverse +index from the references map — `source-table -> entities that already relate to it` — once per direction, +instead of scanning every entity per variable. For each variable V wrapping source S, looks up S in that +index to find every entity T where `T[:referred-by][S]` exists, and registers `T[:referred-by][V]` with the +same relation data. The index is kept in sync with entries added mid-pass, so a variable-of-variable (V2 +wrapping V1, where V1 is itself a variable processed earlier in the same pass) still picks up V1's freshly +patched relations. This propagates the relationship bidirectionally: - `T | V` and `V | T` (real table ↔ variable) - `V | W` and `W | V` (variable ↔ variable) -**Pass 3 — `patch-same-source-variable-joins`**: For each ordered pair of distinct variables (V1, V2) -that share a common source table with an `id` column, registers a synthetic `id = id` join at -`refs[:table V1 :referred-by V2]`. This makes `V1 | V2` resolve even when the source table has no +**Pass 3 — `patch-same-source-variable-joins`**: Groups variables by source table first, so only variables +that actually share a source table are ever paired. For each ordered pair of distinct variables (V1, V2) +within a group, and sharing a common source table with an `id` column, registers a synthetic `id = id` +join at `refs[:table V1 :referred-by V2]`. This makes `V1 | V2` resolve even when the source table has no self-referential FK. Only adds entries where no join path already exists — existing FK-based propagation (e.g. two employee-wrapping variables joined via `reports_to`) is not overridden. From dde3dbe78ee0e94d210130bd700c07f978a643fb Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 09:29:07 +0200 Subject: [PATCH 38/60] test: add regression test for select hints after an un-sealed GROUP group: c.name | s: (no table op in between, so the checkpoint hasn't sealed into a CTE yet) shows hints for the underlying employee table's raw schema instead of the group's own output columns (name, count). :current still points at the pre-group table alias since GROUP never seals/reassigns it, and hint generation falls back to a real schema lookup for that alias. Repros with and without an explicit |= before the trailing s:. Currently failing; fix follows. --- test/pine/hints_test.clj | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index f0d72f6..7c566e1 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -124,6 +124,17 @@ (is (= ["company_id" "reports_to"] (->> "x.company as c | y.employee as e | s: e.id, e." gen :select (map :column))))) + (testing "Generate `select-partial` hints after an un-sealed GROUP checkpoint" + ;; group: c.name hasn't been sealed into a CTE yet (no table op follows before + ;; s:), but the columns available for hints should be the group's own output + ;; (name, count), not the underlying employee table's raw schema. + (is (= ["name" "count"] + (->> "company as c | employee .company_id | group: c.name | s: " gen :select (map :column)))) + ;; Same, with an explicit |= assign before the un-sealed s: — still same-expression, + ;; still not sealed into a CTE until a table op follows. + (is (= ["name" "count"] + (->> "company as c | employee .company_id | group: c.name |= x | s: " gen :select (map :column))))) + (testing "Generate `order-partial` hints" (is (= [{:column "id" :alias "c_0"} {:column "created_at" :alias "c_0"}] (-> "company | o:" gen :order))) (is (= [{:column "created_at" :alias "c_0"}] (-> "company | o: id," gen :order))) From 88dae690dc7360a30f5b287a7e77b36029c4bba9 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 09:29:43 +0200 Subject: [PATCH 39/60] fix: use the group's own columns for hints in an un-sealed checkpoint scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GROUP doesn't change :current or seal into a CTE until a table op follows (flush-checkpoint), so hint generation for anything typed right after an un-sealed group: (e.g. a trailing s:) fell back to generate-all-column-hints' normal schema lookup for :current — which still names the pre-group real table, showing its full raw column list instead of the group's own grouped columns + aggregate. pending-group-columns short-circuits that lookup when we're in exactly this scope (:pending-checkpoint set, :group non-empty, hinting for :current) and builds hints straight from the state's own :columns instead — mirroring how seed-variable-references already derives a variable's output columns once a checkpoint IS sealed. Applies uniformly to select/select-partial/order/ order-partial/where, since they all route through generate-all-column-hints. Fixes the regression test added in the previous commit. --- src/pine/ast/hints.clj | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/src/pine/ast/hints.clj b/src/pine/ast/hints.clj index de89b92..504c7a8 100644 --- a/src/pine/ast/hints.clj +++ b/src/pine/ast/hints.clj @@ -96,18 +96,36 @@ add-pine-expression (fn [h] (assoc h :pine (generate-expression h)))] (map add-pine-expression table-hints))) +(defn- pending-group-columns + "When a GROUP checkpoint hasn't been sealed into a CTE yet (no table op has + followed it), :current still points at the pre-group real table alias — but the + columns actually available at this scope are the group's own output (grouped + columns + aggregate), not that table's full schema. Returns nil (deferring to the + normal schema lookup) unless we're exactly in that pending, unsealed scope for the + current alias." + [state alias] + (when (and (:pending-checkpoint state) + (seq (:group state)) + (= alias (:current state))) + (->> (:columns state) + (remove :auto-id) + (map #(or (:column-alias %) (:column %))) + distinct + (map (fn [c] {:column c :alias alias}))))) + (defn generate-all-column-hints ([state] (generate-all-column-hints state (state :current))) ;; Overload for default `a` ([state a] - (let [aliases (state :aliases) - {table :table schema :schema} (->> a (get aliases)) - columns (if schema - (get-in state [:references :schema schema :table table :columns]) - (get-in state [:references :table table :columns]))] - (for [column columns] - (-> column - (select-keys [:column]) - (assoc :alias a)))))) + (or (pending-group-columns state a) + (let [aliases (state :aliases) + {table :table schema :schema} (->> a (get aliases)) + columns (if schema + (get-in state [:references :schema schema :table table :columns]) + (get-in state [:references :table table :columns]))] + (for [column columns] + (-> column + (select-keys [:column]) + (assoc :alias a))))))) (defn find-relevant-columns [hints column] (if column From c2f1c2a52dd97b79fa294edcea15a88a298b7e16 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 10:12:21 +0200 Subject: [PATCH 40/60] test: strengthen and expand GROUP+trailing-op tests to check full query equality The prior GROUP+select test only asserted the query includes "GROUP BY", which passes even on malformed SQL (verified: it currently emits duplicated id columns and unrelated auto-id pollution). select:/where:/order: after an un-sealed GROUP checkpoint don't seal it into a CTE, so build-query dispatches on the trailing op's literal type instead of :group, silently falling through to the plain-select builder against a group-shaped state: - raw non-aggregated table columns and .* leak into SELECT - WHERE/ORDER BY resolve against the stale pre-group table alias, not the group's own scope The correctly-sealed cross-expression form (splitting the same pipeline so the second expression references the GROUP by name) is used as the oracle: it must produce byte-identical SQL to the inline form, including for an order-partial with a trailing comma, which carries the same parsed value as its complete counterpart and should seal identically. All four currently fail. Fix follows. --- test/pine/eval_test.clj | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 2e46cc2..98c1c2d 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -558,11 +558,36 @@ q-same (:query (generate "x.company | group: id => count |= grp | l: 1"))] (is (= q-cross q-same)))) - (testing "GROUP followed by a complete select: does not crash generating hints" - ;; select: is not a checkpoint-firing op (unlike table/group/limit), so the - ;; group's aggregate columns (e.g. COUNT(1), which has no :operation-index) - ;; are still in state when hints are generated — this used to throw an NPE - ;; in hints/generate-column-hints comparing a nil :operation-index with >. - (is (clojure.string/includes? - (:query (generate "x.company | group: id => count | s: id")) - "GROUP BY")))) \ No newline at end of file + (testing "GROUP followed by a complete select: matches the sealed cross-expression form" + ;; select:/where:/order: after GROUP were not checkpoint-firing ops (unlike + ;; table/group/limit), so the group was never sealed into a CTE — build-query + ;; then dispatched on the trailing op's type instead of :group, silently using + ;; the plain-select builder against a group-shaped state (wrong SQL: raw table + ;; columns leaking in, stale pre-group alias in WHERE/ORDER BY). The correctly + ;; sealed cross-expression form is the oracle: same GROUP, split so the second + ;; expression references the CTE by name, should equal the inline form exactly. + (let [q-cross (:query (generate-expressions ["x.company | group: id => count |= grp" + "grp | s: id"])) + q-same (:query (generate "x.company | group: id => count | s: id"))] + (is (= q-cross q-same)))) + + (testing "GROUP followed by a complete where: matches the sealed cross-expression form" + (let [q-cross (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= grp" + "grp | w: name = 'Acme'"])) + q-same (:query (generate "company as c | employee .company_id | group: c.name | w: name = 'Acme'"))] + (is (= q-cross q-same)))) + + (testing "GROUP followed by a complete order: matches the sealed cross-expression form" + (let [q-cross (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= grp" + "grp | o: name desc"])) + q-same (:query (generate "company as c | employee .company_id | group: c.name | o: name desc"))] + (is (= q-cross q-same)))) + + (testing "GROUP followed by an order-partial (trailing comma) matches the sealed cross-expression form" + ;; order-partial with a non-empty value (e.g. a dangling trailing comma after a + ;; fully-typed column) is not meaningfully different from a complete order: here — + ;; the already-typed column should seal and resolve exactly the same way. + (let [q-cross (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= grp" + "grp | o: name desc,"])) + q-same (:query (generate "company as c | employee .company_id | group: c.name | o: name desc,"))] + (is (= q-cross q-same))))) \ No newline at end of file From 77ce67966e77d611a2ec1a3405a6e9aafbd9750e Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 10:15:20 +0200 Subject: [PATCH 41/60] fix: seal GROUP checkpoint on complete or partial select/where/order too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit flush-checkpoint only fired (sealed a pending GROUP/LIMIT into a CTE) for a following table op or another checkpoint op. Any select:/where:/order: typed directly after an un-sealed GROUP — complete or partial, e.g. an order-partial from a dangling trailing comma — left the checkpoint pending, so build-query dispatched on that trailing op's literal type instead of :group and silently fell through to the plain-select builder against a group-shaped state: raw non-aggregated columns and `.*` leaked into SELECT, and WHERE/ORDER BY resolved against the stale pre-group table alias instead of the group's own scope. checkpoint-consuming-op-types now fires on select/select-partial/where/ where-partial/order/order-partial uniformly — partial vs complete makes no difference here, since a partial with a value already carries whatever was fully typed before the trailing comma. count/delete/update stay excluded: they build their own wrapper query generically and were never broken. This also fixes the root duplication behind the previous commit's hints fix: variable-output-columns and the hints-only pending-group-columns special case were two independent implementations of "what columns does this GROUP actually expose", and only one of them was correct. With checkpoint sealing now covering these ops, every hint request goes through the same sealed-CTE path the working cross-expression case always used — so pending-group-columns is removed as redundant, and the one remaining implementation (variable-output-columns) is fixed: it was unconditionally appending "count" whenever op-type was :group, even though group.clj already folds the aggregate into :columns when one exists (`=> count` is optional in the grammar), so it was double-counting when an aggregate was given and fabricating one when it wasn't. Also aligns two of the new regression tests to include an explicit |= so the inline and cross-expression forms share a CTE name and compare byte-for-byte, and adds a fourth case (no |=) confirming auto-named sealing works too. --- src/pine/ast/hints.clj | 36 +++++++++--------------------------- src/pine/ast/main.clj | 38 ++++++++++++++++++++++++++++---------- test/pine/eval_test.clj | 16 +++++++++++----- 3 files changed, 48 insertions(+), 42 deletions(-) diff --git a/src/pine/ast/hints.clj b/src/pine/ast/hints.clj index 504c7a8..de89b92 100644 --- a/src/pine/ast/hints.clj +++ b/src/pine/ast/hints.clj @@ -96,36 +96,18 @@ add-pine-expression (fn [h] (assoc h :pine (generate-expression h)))] (map add-pine-expression table-hints))) -(defn- pending-group-columns - "When a GROUP checkpoint hasn't been sealed into a CTE yet (no table op has - followed it), :current still points at the pre-group real table alias — but the - columns actually available at this scope are the group's own output (grouped - columns + aggregate), not that table's full schema. Returns nil (deferring to the - normal schema lookup) unless we're exactly in that pending, unsealed scope for the - current alias." - [state alias] - (when (and (:pending-checkpoint state) - (seq (:group state)) - (= alias (:current state))) - (->> (:columns state) - (remove :auto-id) - (map #(or (:column-alias %) (:column %))) - distinct - (map (fn [c] {:column c :alias alias}))))) - (defn generate-all-column-hints ([state] (generate-all-column-hints state (state :current))) ;; Overload for default `a` ([state a] - (or (pending-group-columns state a) - (let [aliases (state :aliases) - {table :table schema :schema} (->> a (get aliases)) - columns (if schema - (get-in state [:references :schema schema :table table :columns]) - (get-in state [:references :table table :columns]))] - (for [column columns] - (-> column - (select-keys [:column]) - (assoc :alias a))))))) + (let [aliases (state :aliases) + {table :table schema :schema} (->> a (get aliases)) + columns (if schema + (get-in state [:references :schema schema :table table :columns]) + (get-in state [:references :table table :columns]))] + (for [column columns] + (-> column + (select-keys [:column]) + (assoc :alias a)))))) (defn find-relevant-columns [hints column] (if column diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 6d13789..0e1783e 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -94,14 +94,20 @@ (defn- variable-output-columns "Return the column list a variable's CTE actually exposes, for hint generation. - Returns nil when the CTE selects *, meaning the source table's columns apply." + Returns nil when the CTE selects *, meaning the source table's columns apply. + + For a GROUP-sourced CTE, :columns already includes the aggregate entry (e.g. + count) whenever the GROUP actually specified one — group.clj folds it in + directly — so it's read here like any other column, never added separately. + (`=> count` is optional in the grammar; a bare `group: col` has no aggregate + at all, and fabricating one would be just as wrong as double-counting it.)" [var-ast] - (let [user-cols (remove :auto-id (:columns var-ast)) - op-type (-> var-ast :operation :type)] + (let [user-cols (remove :auto-id (:columns var-ast))] (when (seq user-cols) - (let [names (distinct (map #(or (:column-alias %) (:column %)) user-cols)) - with-count (if (= op-type :group) (concat names ["count"]) names)] - (mapv #(hash-map :column %) with-count))))) + (->> user-cols + (map #(or (:column-alias %) (:column %))) + distinct + (mapv #(hash-map :column %)))))) (defn- get-source-tables "Return the tables whose columns are exposed by a variable's CTE. @@ -259,6 +265,16 @@ (def ^:private checkpoint-op-types #{:group :limit}) +;; Ops that consume/query the checkpoint's result rather than composing another +;; table join onto it. Fired on regardless of partial-vs-complete: an -partial op +;; (e.g. order-partial from a dangling trailing comma) already carries whatever was +;; fully typed before the comma, so it needs the same sealed scope a complete op +;; would. count/delete/update are deliberately excluded — they build their own +;; wrapper query generically (see build-count-query) and don't need the checkpoint's +;; group-shaped state separated into a CTE first. +(def ^:private checkpoint-consuming-op-types + #{:select :select-partial :where :where-partial :order :order-partial}) + (defn- reset-for-cte [state] (assoc state :tables [] :columns [] :limit nil :joins [] @@ -284,14 +300,16 @@ "State-machine step: called at the start of each handle-ops iteration. Converts a pending checkpoint into a CTE when the right op type is seen. - Fires when the incoming op is a table (join composition) or another checkpoint - op (e.g. LIMIT after GROUP). Does not fire for count/delete/update since those - have their own query-building paths that do not need CTE separation." + Fires when the incoming op is a table (join composition), another checkpoint + op (e.g. LIMIT after GROUP), or an op that queries the checkpoint's result + (select/where/order, complete or partial). Does not fire for count/delete/update + since those have their own query-building paths that do not need CTE separation." [state op] (let [checkpoint (:pending-checkpoint state) op-type (:type op) fire? (or (= op-type :table) - (contains? checkpoint-op-types op-type))] + (contains? checkpoint-op-types op-type) + (contains? checkpoint-consuming-op-types op-type))] (cond (nil? checkpoint) state diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 98c1c2d..ea0618a 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -568,19 +568,19 @@ ;; expression references the CTE by name, should equal the inline form exactly. (let [q-cross (:query (generate-expressions ["x.company | group: id => count |= grp" "grp | s: id"])) - q-same (:query (generate "x.company | group: id => count | s: id"))] + q-same (:query (generate "x.company | group: id => count |= grp | s: id"))] (is (= q-cross q-same)))) (testing "GROUP followed by a complete where: matches the sealed cross-expression form" (let [q-cross (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= grp" "grp | w: name = 'Acme'"])) - q-same (:query (generate "company as c | employee .company_id | group: c.name | w: name = 'Acme'"))] + q-same (:query (generate "company as c | employee .company_id | group: c.name |= grp | w: name = 'Acme'"))] (is (= q-cross q-same)))) (testing "GROUP followed by a complete order: matches the sealed cross-expression form" (let [q-cross (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= grp" "grp | o: name desc"])) - q-same (:query (generate "company as c | employee .company_id | group: c.name | o: name desc"))] + q-same (:query (generate "company as c | employee .company_id | group: c.name |= grp | o: name desc"))] (is (= q-cross q-same)))) (testing "GROUP followed by an order-partial (trailing comma) matches the sealed cross-expression form" @@ -589,5 +589,11 @@ ;; the already-typed column should seal and resolve exactly the same way. (let [q-cross (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= grp" "grp | o: name desc,"])) - q-same (:query (generate "company as c | employee .company_id | group: c.name | o: name desc,"))] - (is (= q-cross q-same))))) \ No newline at end of file + q-same (:query (generate "company as c | employee .company_id | group: c.name |= grp | o: name desc,"))] + (is (= q-cross q-same)))) + + (testing "GROUP followed by a complete select: auto-named CTE also seals correctly (no |=)" + ;; Without an explicit |=, the checkpoint still seals — just with a generated + ;; name instead of a user-given one. + (is (= "WITH \"__pine_0__\" AS ( SELECT \"c_0\".\"id\", COUNT(1) AS \"count\" FROM \"x\".\"company\" AS \"c_0\" GROUP BY \"c_0\".\"id\" ) SELECT \"__pine_0__\".\"id\" FROM \"__pine_0__\" AS \"__pine_0__\" LIMIT 250" + (:query (generate "x.company | group: id => count | s: id")))))) \ No newline at end of file From 813e854a13ec0aaf37d2d6e3535e8acd1e296ddb Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 10:16:41 +0200 Subject: [PATCH 42/60] docs: update checkpoints.md for select/where/order checkpoint-firing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previously described firing as table-op-or-checkpoint-op only, and stated "ORDER does not create a checkpoint" in a way that read as ORDER never seals — no longer accurate now that select:/where:/order: (complete or partial) also fire a pending checkpoint. Adds an ORDER checkpoint example mirroring the existing GROUP/LIMIT ones, clarifies the count:/delete:/ update: exclusion is about not needing sealing (not about being a non-table op), and documents the new checkpoint-consuming-op-types set alongside checkpoint-op-types. --- docs/checkpoints.md | 46 +++++++++++++++++++++++++++++++++------------ 1 file changed, 34 insertions(+), 12 deletions(-) diff --git a/docs/checkpoints.md b/docs/checkpoints.md index da5d0cc..950bff3 100644 --- a/docs/checkpoints.md +++ b/docs/checkpoints.md @@ -1,16 +1,16 @@ # Checkpoints -Automatically seal a GROUP or LIMIT result into an anonymous CTE so subsequent table operations compose on top of it. +Automatically seal a GROUP or LIMIT result into an anonymous CTE so subsequent operations compose on top of it instead of corrupting it. ## Why -GROUP and LIMIT produce a final, bounded result set — further joins appended after them produce malformed SQL (a GROUP BY inside a subquery that is then joined, or a LIMIT applied before the join filter). Checkpoints solve this by detecting when a table operation follows a GROUP or LIMIT and implicitly wrapping the preceding query in a CTE, then continuing the pipeline on top of that CTE. +GROUP and LIMIT produce a final, bounded result set — further joins, or further filtering/ordering/selecting, appended after them produce malformed SQL if just tacked onto the same query (a GROUP BY inside a subquery that is then joined, a LIMIT applied before the join filter, or — for select:/where:/order: — the query builder falling back to treating the pipeline as an ordinary ungrouped select, since it decides how to build the SQL from the *last* operation's type, not from whether a GROUP happened earlier). Checkpoints solve this by detecting when such an operation follows a GROUP or LIMIT and implicitly wrapping the preceding query in a CTE, then continuing on top of that CTE. -Without checkpoints, `company | l: 10 | employee` would attempt to build a single query with LIMIT and a JOIN, which contradicts the intent (get 10 companies, then navigate to their employees). +Without checkpoints, `company | l: 10 | employee` would attempt to build a single query with LIMIT and a JOIN, which contradicts the intent (get 10 companies, then navigate to their employees). Similarly, `company | group: name | o: name desc` would resolve ORDER BY against the pre-group table instead of the grouped result. ## Syntax -No syntax change is required. Checkpoints fire automatically when a TABLE operation follows a GROUP or LIMIT: +No syntax change is required. Checkpoints fire automatically when a TABLE operation, or a select:/where:/order: (complete or partial), follows a GROUP or LIMIT: ``` | (group:|limit:) ... |
[| more-ops...] @@ -80,13 +80,33 @@ JOIN "employee" AS "e_1" ON "__pine_0__"."id" = "e_1"."company_id" LIMIT 250 ``` -### No checkpoint when non-table op follows +### ORDER checkpoint + +``` +company as c | employee .company_id | group: c.name | o: name desc +``` + +`o:` is not a table op, but it still fires the pending GROUP checkpoint — otherwise `build-query` would dispatch on `:order` (the last op) instead of `:group`, and silently build an ungrouped query with `ORDER BY` resolved against the stale pre-group table alias. Sealing first makes this identical to the sealed cross-expression form (`group: c.name |= x` in one expression, `x | o: name desc` in the next): + +```sql +WITH "__pine_0__" AS ( + SELECT "c"."name", COUNT(1) AS "count" + FROM "company" AS "c" + JOIN "employee" AS "e_1" ON "c"."id" = "e_1"."company_id" + GROUP BY "c"."name" +) +SELECT "__pine_0__".* FROM "__pine_0__" AS "__pine_0__" ORDER BY "__pine_0__"."name" DESC LIMIT 250 +``` + +The same applies to `select:`/`where:`, and to their `-partial` forms (e.g. `o: name desc,` with a dangling trailing comma) — a partial with a value already carries whatever was fully typed before the comma, so it seals exactly like the complete form. + +### No checkpoint when an action op follows ``` company | limit: 100 | count: ``` -`count:` is not a table op — no checkpoint fires. The pipeline remains a single query and COUNT wraps it normally: +`count:` (like `delete:`/`update:`) builds its own wrapper query generically regardless of what came before it, so it doesn't need the checkpoint sealed first — no checkpoint fires, and the pipeline remains a single query with COUNT wrapping it normally: ```sql WITH x AS ( SELECT "c_0".* FROM "company" AS "c_0" LIMIT 100 ) @@ -96,14 +116,15 @@ SELECT COUNT(*) FROM x ## How it works - **Detection**: after processing a GROUP or LIMIT op, `handle-ops` sets `:pending-checkpoint {:needs-assign true}` on the state. -- **Fire on table or checkpoint op**: at the start of each `handle-ops` iteration, `flush-checkpoint` checks whether a checkpoint is pending and whether the incoming op is a TABLE op or another checkpoint op (GROUP or LIMIT). If so, the current state is snapshotted into `:pending-assignments` under an auto-generated name (`__pine_0__`, `__pine_1__`, ...), the references map is seeded for that name (same FK propagation used by variables), and the state is reset. The CTE name is then injected as the first table so subsequent ops compose on top of it. A LIMIT following a GROUP therefore fires the GROUP checkpoint, then applies the limit to the outer query. -- **User-named CTE**: if an `|= name` op appears between the checkpoint op and the following op, `flush-checkpoint` records the name and waits. `assign/handle` stores the snapshot under `name` as normal. When the next table or checkpoint op arrives, `seal-as-cte` activates it. -- **Hold for non-triggering ops**: if the op after GROUP/LIMIT is something that has its own query-building path — `count:`, `delete:`, etc. — the checkpoint stays pending and that op is processed without firing the checkpoint. +- **Fire on table, checkpoint, or checkpoint-consuming op**: at the start of each `handle-ops` iteration, `flush-checkpoint` checks whether a checkpoint is pending and whether the incoming op is a TABLE op, another checkpoint op (GROUP or LIMIT), or a checkpoint-*consuming* op (`select`/`select-partial`/`where`/`where-partial`/`order`/`order-partial`). If so, the current state is snapshotted into `:pending-assignments` under an auto-generated name (`__pine_0__`, `__pine_1__`, ...), the references map is seeded for that name (same FK propagation used by variables), and the state is reset. The CTE name is then injected as the first table so subsequent ops compose on top of it. A LIMIT following a GROUP therefore fires the GROUP checkpoint, then applies the limit to the outer query. +- **User-named CTE**: if an `|= name` op appears between the checkpoint op and the following op, `flush-checkpoint` records the name and waits. `assign/handle` stores the snapshot under `name` as normal. When the next firing op arrives, `seal-as-cte` activates it. +- **Hold for non-triggering ops**: if the op after GROUP/LIMIT is something that has its own query-building path — `count:`, `delete:`, `update:` — the checkpoint stays pending and that op is processed without firing the checkpoint. - **SQL generation**: because the CTE table has an `:ast` entry in `:aliases`, `collect-ctes` automatically picks it up and emits the `WITH` clause. No changes to `eval.clj` were needed. ## Constraints -- Checkpoint op types are GROUP and LIMIT. ORDER does not create a checkpoint. +- Checkpoint op types (what *creates* a pending checkpoint) are GROUP and LIMIT. +- Checkpoint-consuming op types (what *fires*/seals an already-pending checkpoint, alongside TABLE) are `select`/`select-partial`/`where`/`where-partial`/`order`/`order-partial`. `count:`/`delete:`/`update:` deliberately do not fire — see above. - Only one table-level composition step is supported per checkpoint; chaining `l: 10 | employee | document` creates one CTE for `l: 10` and then navigates normally through employee to document. - Auto-generated CTE names (`__pine_0__`, etc.) are numbered per expression and are not exposed to the user. - Checkpoint CTEs do not receive Pine's auto-id columns; those are suppressed for all CTE-backed tables. @@ -123,10 +144,11 @@ See also: [variables.md](variables.md) for cross-expression named CTEs. ### Core functions (`ast/main.clj`) -- `checkpoint-op-types` — set `#{:group :limit}` +- `checkpoint-op-types` — set `#{:group :limit}`; ops that *create* a pending checkpoint +- `checkpoint-consuming-op-types` — set `#{:select :select-partial :where :where-partial :order :order-partial}`; ops that *fire* (seal) an already-pending checkpoint, alongside a TABLE op or another checkpoint op. Partial and complete forms are treated identically — a partial with a value has already fully typed whatever precedes a dangling trailing comma. - `reset-for-cte` — clears all query-building fields (tables, columns, joins, where, order, group, limit) while preserving references, variables, and pending-assignments - `seal-as-cte` — the single shared action: stores the snapshot under `cname` in `:pending-assignments`, seeds FK references for `cname` (same three passes used by cross-expression variables in `pre-handle`), resets query-building state, then injects `cname` as the first table via `handle-op`; `table/handle` resolves it via `:pending-assignments` -- `flush-checkpoint` — state-machine dispatch called at the start of each `handle-ops` reduce iteration; calls `seal-as-cte` when a TABLE op follows the checkpoint, using either a freshly-snapshotted state (auto-named) or the snapshot already stored by `assign/handle` (user-named) +- `flush-checkpoint` — state-machine dispatch called at the start of each `handle-ops` reduce iteration; calls `seal-as-cte` when a TABLE op, another checkpoint op, or a checkpoint-consuming op follows the pending checkpoint, using either a freshly-snapshotted state (auto-named) or the snapshot already stored by `assign/handle` (user-named) ### `table/handle` (`ast/table.clj`) From d56a91250de365a9f0b11faed60ccd76d0be342c Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 10:16:59 +0200 Subject: [PATCH 43/60] docs: fix stale "Group operations append a synthetic count entry" claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit variable-output-columns no longer appends count separately — it double- counted when a GROUP had an aggregate and fabricated one when it didn't, since :columns already includes the aggregate whenever group.clj folded one in. Describes the corrected (read-only) behavior instead. --- docs/variables.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/variables.md b/docs/variables.md index eb63bb8..8db17de 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -168,7 +168,10 @@ passes to avoid duplicating that logic. `seed-variable-references` also overrides the column list for the variable entry in the references map: - **Explicit columns** (`s:`, `g:`, etc.): the column list is built from the user-selected columns. The - name used is `column-alias` if set, otherwise `column`. Group operations append a synthetic `count` entry. + name used is `column-alias` if set, otherwise `column`. For a GROUP-sourced CTE this naturally includes + the aggregate (e.g. `count`) whenever the GROUP specified one — group.clj folds it directly into + `:columns`, so it's read here like any other column rather than appended separately. (`=> count` is + optional in the grammar; a bare `group: col` with no aggregate correctly yields no `count` entry.) - **No explicit columns** (`*`): the source table's full column list is inherited. ### Alias disambiguation From 9fe6e7d8c30f03fa55fc76b517196c80e22dd5ee Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 10:28:38 +0200 Subject: [PATCH 44/60] docs: add Terminology section explaining checkpoint (noun) vs seal (verb) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit No code or docs anywhere actually stated the relationship between the two terms, which read as arbitrary/redundant to a fresh reader. Checkpoint is the pending state; sealing is the action that resolves it into a CTE — the same relationship as a pending transaction and its commit. No renames: seal-as-cte is private with zero external callers and reads clearly once the relationship is explained. --- docs/checkpoints.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/checkpoints.md b/docs/checkpoints.md index 950bff3..ce5da6f 100644 --- a/docs/checkpoints.md +++ b/docs/checkpoints.md @@ -8,6 +8,15 @@ GROUP and LIMIT produce a final, bounded result set — further joins, or furthe Without checkpoints, `company | l: 10 | employee` would attempt to build a single query with LIMIT and a JOIN, which contradicts the intent (get 10 companies, then navigate to their employees). Similarly, `company | group: name | o: name desc` would resolve ORDER BY against the pre-group table instead of the grouped result. +## Terminology + +"Checkpoint" and "seal" name two different things, not two names for one thing: + +- A **checkpoint** is the *pending state* — the noun. After a GROUP or LIMIT, the pipeline has a checkpoint pending (`:pending-checkpoint`) until something resolves it. +- **Sealing** is the *action* that resolves a pending checkpoint — the verb. `seal-as-cte` is the one function that does it: snapshot the state, seed its references, and inject it as a CTE. + +So a checkpoint is *pending*, and gets *sealed* into a CTE — the same relationship as a transaction being pending and then committed. The doc and public feature name is "checkpoints"; "seal" only ever refers to the specific act of materializing one into a CTE. + ## Syntax No syntax change is required. Checkpoints fire automatically when a TABLE operation, or a select:/where:/order: (complete or partial), follows a GROUP or LIMIT: From 79b7a06597bc7f4498aa70d89c39c937173ed4e7 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 10:59:37 +0200 Subject: [PATCH 45/60] fix: stop ast.variables/selected-tables from recursively re-embedding prior blocks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause of the frontend sluggishness that scaled with the number of chained expression blocks (not fixed by the earlier CodeMirror memoization, which addressed a real but separate per-keystroke cost). api-build's :ast response trimmed :pending-assignments entries down to {:tables :selected-tables :joins :columns} (matching VariableAst in client.ts), but sent :variables untouched. A raw variable snapshot still carries the full :variables and :references it was built from (pre-handle seeds them, post-handle only strips :references off the *top-level* state, never off snapshots captured mid-pipeline by assign/handle). So variable N's snapshot embedded variable N-1's entire untrimmed snapshot inside its own :variables key, which embedded N-2's, and so on — every additional chained |= block re-embedded the full history of every earlier block on top of itself, growing the response payload superlinearly instead of linearly. Measured on a 4-block chain (tenant/company grouped, chained through three |= assignments): the :ast payload went 843 -> 935 -> 978 -> 935 bytes after the fix, versus 1,271 -> 26,939 -> 112,750 -> 284,287 bytes before it (a 224x blowup by the 4th block). That JSON has to be parsed and made MobX-observable on every debounced keystroke, which is what actually scaled with expression count. There was a second, independent leak at the same root cause: any variable-backed table entry (in :tables/:selected-tables, added by table.clj's handle-as-variable for the query builder's CTE generation) carries a full :ast — the wrapped variable's own var-ast. Trimming only the :variables map wasn't enough; each table entry needed the same trim (select-table), both at the top level and recursively inside each trimmed variable's own :tables/:selected-tables, or a variable-of-variable chain reintroduced the exact same recursive embedding one level down. select-var-ast and select-table are the two shared trims, replacing the inline select-keys logic that only handled :pending-assignments. The :query field is unaffected — it's built from the original untrimmed state, which still needs the full :ast-carrying table entries for eval.clj's CTE generation. New test/pine/api_test.clj (api.clj had no test file before this) verifies variable/table entries never carry :variables/:references/:ast, and pins the payload size for a 4-block chain under 5KB as a regression guard. --- src/pine/api.clj | 38 +++++++++++++++++++++--- test/pine/api_test.clj | 65 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+), 4 deletions(-) create mode 100644 test/pine/api_test.clj diff --git a/src/pine/api.clj b/src/pine/api.clj index f47ad5e..e2dc4ab 100644 --- a/src/pine/api.clj +++ b/src/pine/api.clj @@ -72,6 +72,39 @@ (str/replace #"^\|\s*|\s*\|$" "") (str/trim))) +(defn- select-table + "Trim a table entry down to what the frontend's Table type (client.ts) uses. + Critical: a variable-backed table entry carries a full :ast (the variable's own + var-ast) for the query builder's CTE generation — leaving it in would recursively + re-embed that variable's entire state (and, transitively, everything it wraps in + turn) inside every table list that references it." + [table] + (select-keys table [:schema :table :alias])) + +(defn- select-var-ast + "Trim a variable/pending-assignment snapshot down to what the frontend actually + uses (VariableAst in client.ts). Critical: a raw snapshot still carries :variables + and :references from pre-handle/post-handle, so leaving those in would recursively + re-embed every earlier variable's own full (untrimmed) snapshot inside this one — + each additional chained |= block would then re-embed all prior blocks again on top + of that, growing the response payload superlinearly instead of linearly. Its own + :tables/:selected-tables entries need the same per-table trim (select-table) for a + variable-of-variable chain, or the same recursive embedding reappears one level down." + [var-ast] + (-> (select-keys var-ast [:tables :selected-tables :joins :columns]) + (update :tables #(mapv select-table %)) + (update :selected-tables #(mapv select-table %)))) + +(defn- build-ast + "Build the :ast value returned to the frontend from a generated state." + [state] + (-> (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges :assign]) + (update :selected-tables #(mapv select-table %)) + (assoc :variables + (into {} (for [[k v] (:variables state)] [k (select-var-ast v)]))) + (assoc :pending-assignments + (into {} (for [[k v] (:pending-assignments state)] [k (select-var-ast v)]))))) + (defn api-build ([expressions] (api-build expressions nil nil)) @@ -96,10 +129,7 @@ {:connection-id connection-name :version version :query (-> last-expr trim-pipes (generate-state nil conn-id variables) :result eval/build-query eval/formatted-query) - :ast (-> (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges :variables :assign]) - (assoc :pending-assignments - (into {} (for [[k v] (:pending-assignments state)] - [k (select-keys v [:tables :selected-tables :joins :columns])]))))})))))) + :ast (build-ast state)})))))) (catch Exception e {:connection-id connection-name :error (.getMessage e)}))))) diff --git a/test/pine/api_test.clj b/test/pine/api_test.clj new file mode 100644 index 0000000..7aacf3a --- /dev/null +++ b/test/pine/api_test.clj @@ -0,0 +1,65 @@ +(ns pine.api-test + (:require [cheshire.core :as json] + [clojure.test :refer [deftest is testing]] + [pine.api :as api] + [pine.ast.main :as ast] + [pine.parser :as parser])) + +(defn- generate-with-variables + "Evaluate a sequence of expressions sequentially, threading variables between + them the same way api-build's evaluate-expressions does. Returns the final state." + [expressions] + (:last-state + (reduce (fn [{:keys [variables]} expr] + (let [{:keys [result]} (parser/parse expr) + state (ast/generate result :test nil nil variables)] + {:variables (merge variables (:pending-assignments state)) + :last-state state})) + {:variables {} :last-state nil} + expressions))) + +(defn- assert-clean-table [table] + (is (= #{:schema :table :alias} (set (keys table))) + "table entries must not carry :ast (a variable's own full state snapshot)")) + +(deftest test-build-ast + (let [state (generate-with-variables + ["tenant as t | company .tenantId | group: t.title |= x" + "x | s: count, | o: count desc |= y" + "y | s: count, |= z" + "z | "]) + built-ast (#'api/build-ast state)] + + (testing "ast.variables entries are trimmed like pending-assignments, not raw snapshots" + ;; A raw variable snapshot carries :variables and :references from + ;; pre-handle/post-handle. Left untrimmed, each chained |= re-embeds every + ;; earlier variable's own full snapshot inside the new one, growing the + ;; response payload superlinearly with the number of chained expressions + ;; instead of linearly. + (is (= #{"x" "y" "z"} (set (keys (:variables built-ast))))) + (doseq [[name var-ast] (:variables built-ast)] + (testing (str "variable " name) + (is (= #{:tables :selected-tables :joins :columns} (set (keys var-ast))) + "should only carry the fields VariableAst (client.ts) actually uses") + (is (not (contains? var-ast :variables)) + "must not recursively embed earlier variables' own snapshots") + (is (not (contains? var-ast :references)) + "must not carry the full schema references map")))) + + (testing "table entries (top-level and nested inside variables) never carry :ast" + ;; A variable-backed table entry carries a full :ast (the variable's own + ;; var-ast) for the query builder's CTE generation. Left in place, that + ;; recursively re-embeds the variable's entire state — and everything IT + ;; wraps in turn — inside every table list that references it, one level + ;; down from the :variables map itself. + (doseq [table (:selected-tables built-ast)] + (assert-clean-table table)) + (doseq [[_name var-ast] (:variables built-ast) + table (concat (:tables var-ast) (:selected-tables var-ast))] + (assert-clean-table table))) + + (testing "payload stays flat as more expressions chain, not superlinear" + ;; Regression guard: before the fix this was ~285KB for this 4-expression + ;; chain (each block re-embedding every prior block's full state); it + ;; should stay in the same ballpark regardless of chain length. + (is (< (count (json/generate-string built-ast)) 5000))))) From 1b33843a2cafac0c7676f0623c0a34846d062c1f Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 11:09:41 +0200 Subject: [PATCH 46/60] fmt: fix indentation in patch-same-source-variable-joins Left un-reformatted by an earlier commit; clj -M:fmt fix catches it now. --- src/pine/ast/main.clj | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 0e1783e..0152e22 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -219,11 +219,11 @@ [vname (set (get-source-tables var-ast))])) (into {})) by-source (reduce-kv (fn [idx vname sources] - (reduce (fn [idx source] (update idx source (fnil conj []) vname)) - idx - sources)) - {} - var-sources) + (reduce (fn [idx source] (update idx source (fnil conj []) vname)) + idx + sources)) + {} + var-sources) pairs (distinct (for [[_ vnames] by-source v1 vnames v2 vnames :when (not= v1 v2)] From e40885bd199f5510741522851d8fea4e1318b3ea Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 11:09:48 +0200 Subject: [PATCH 47/60] refactor: rename select-* to prune-* to name what they actually do MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit select-table/select-var-ast/build-ast read as plain projection, but the whole reason they exist is to cut away recursive, self-referential growth (a variable's embedded :variables/:references, a table's embedded :ast) — prune-table/prune-var-ast/prune-ast names that intent directly. Also replaces the brittle "<5000 bytes" regression guard in api_test.clj with a structural invariant: an earlier variable's own pruned entry must be byte-for-byte identical whether it's standing alone or more blocks have chained onto it since. That's the actual property the bug violated (entries kept growing as more blocks chained on) — a byte ceiling was an arbitrary proxy for it that could drift or break for unrelated reasons. --- src/pine/api.clj | 40 ++++++++++++++++++------------------ test/pine/api_test.clj | 46 ++++++++++++++++++++++++------------------ 2 files changed, 46 insertions(+), 40 deletions(-) diff --git a/src/pine/api.clj b/src/pine/api.clj index e2dc4ab..2c57256 100644 --- a/src/pine/api.clj +++ b/src/pine/api.clj @@ -72,38 +72,38 @@ (str/replace #"^\|\s*|\s*\|$" "") (str/trim))) -(defn- select-table - "Trim a table entry down to what the frontend's Table type (client.ts) uses. +(defn- prune-table + "Prune a table entry down to what the frontend's Table type (client.ts) uses. Critical: a variable-backed table entry carries a full :ast (the variable's own - var-ast) for the query builder's CTE generation — leaving it in would recursively - re-embed that variable's entire state (and, transitively, everything it wraps in + var-ast) for the query builder's CTE generation — left unpruned, it recursively + re-embeds that variable's entire state (and, transitively, everything it wraps in turn) inside every table list that references it." [table] (select-keys table [:schema :table :alias])) -(defn- select-var-ast - "Trim a variable/pending-assignment snapshot down to what the frontend actually +(defn- prune-var-ast + "Prune a variable/pending-assignment snapshot down to what the frontend actually uses (VariableAst in client.ts). Critical: a raw snapshot still carries :variables - and :references from pre-handle/post-handle, so leaving those in would recursively - re-embed every earlier variable's own full (untrimmed) snapshot inside this one — - each additional chained |= block would then re-embed all prior blocks again on top - of that, growing the response payload superlinearly instead of linearly. Its own - :tables/:selected-tables entries need the same per-table trim (select-table) for a - variable-of-variable chain, or the same recursive embedding reappears one level down." + and :references from pre-handle/post-handle — left unpruned, each additional + chained |= block would re-embed every earlier block's full snapshot inside the new + one, growing the response payload superlinearly instead of linearly. Its own + :tables/:selected-tables entries need the same per-table pruning (prune-table) for + a variable-of-variable chain, or the same recursive embedding reappears one level + down." [var-ast] (-> (select-keys var-ast [:tables :selected-tables :joins :columns]) - (update :tables #(mapv select-table %)) - (update :selected-tables #(mapv select-table %)))) + (update :tables #(mapv prune-table %)) + (update :selected-tables #(mapv prune-table %)))) -(defn- build-ast - "Build the :ast value returned to the frontend from a generated state." +(defn- prune-ast + "Prune a generated state down to the :ast value returned to the frontend." [state] (-> (select-keys state [:hints :selected-tables :joins :context :current :operation :columns :order :where :prettified :ranges :assign]) - (update :selected-tables #(mapv select-table %)) + (update :selected-tables #(mapv prune-table %)) (assoc :variables - (into {} (for [[k v] (:variables state)] [k (select-var-ast v)]))) + (into {} (for [[k v] (:variables state)] [k (prune-var-ast v)]))) (assoc :pending-assignments - (into {} (for [[k v] (:pending-assignments state)] [k (select-var-ast v)]))))) + (into {} (for [[k v] (:pending-assignments state)] [k (prune-var-ast v)]))))) (defn api-build ([expressions] @@ -129,7 +129,7 @@ {:connection-id connection-name :version version :query (-> last-expr trim-pipes (generate-state nil conn-id variables) :result eval/build-query eval/formatted-query) - :ast (build-ast state)})))))) + :ast (prune-ast state)})))))) (catch Exception e {:connection-id connection-name :error (.getMessage e)}))))) diff --git a/test/pine/api_test.clj b/test/pine/api_test.clj index 7aacf3a..7699a81 100644 --- a/test/pine/api_test.clj +++ b/test/pine/api_test.clj @@ -1,6 +1,5 @@ (ns pine.api-test - (:require [cheshire.core :as json] - [clojure.test :refer [deftest is testing]] + (:require [clojure.test :refer [deftest is testing]] [pine.api :as api] [pine.ast.main :as ast] [pine.parser :as parser])) @@ -22,22 +21,25 @@ (is (= #{:schema :table :alias} (set (keys table))) "table entries must not carry :ast (a variable's own full state snapshot)")) -(deftest test-build-ast - (let [state (generate-with-variables - ["tenant as t | company .tenantId | group: t.title |= x" - "x | s: count, | o: count desc |= y" - "y | s: count, |= z" - "z | "]) - built-ast (#'api/build-ast state)] +(deftest test-prune-ast + (let [single-block (generate-with-variables + ["tenant as t | company .tenantId | group: t.title |= x"]) + chained-blocks (generate-with-variables + ["tenant as t | company .tenantId | group: t.title |= x" + "x | s: count, | o: count desc |= y" + "y | s: count, |= z" + "z | "]) + single-ast (#'api/prune-ast single-block) + chained-ast (#'api/prune-ast chained-blocks)] - (testing "ast.variables entries are trimmed like pending-assignments, not raw snapshots" + (testing "ast.variables entries are pruned like pending-assignments, not raw snapshots" ;; A raw variable snapshot carries :variables and :references from - ;; pre-handle/post-handle. Left untrimmed, each chained |= re-embeds every + ;; pre-handle/post-handle. Left unpruned, each chained |= re-embeds every ;; earlier variable's own full snapshot inside the new one, growing the ;; response payload superlinearly with the number of chained expressions ;; instead of linearly. - (is (= #{"x" "y" "z"} (set (keys (:variables built-ast))))) - (doseq [[name var-ast] (:variables built-ast)] + (is (= #{"x" "y" "z"} (set (keys (:variables chained-ast))))) + (doseq [[name var-ast] (:variables chained-ast)] (testing (str "variable " name) (is (= #{:tables :selected-tables :joins :columns} (set (keys var-ast))) "should only carry the fields VariableAst (client.ts) actually uses") @@ -52,14 +54,18 @@ ;; recursively re-embeds the variable's entire state — and everything IT ;; wraps in turn — inside every table list that references it, one level ;; down from the :variables map itself. - (doseq [table (:selected-tables built-ast)] + (doseq [table (:selected-tables chained-ast)] (assert-clean-table table)) - (doseq [[_name var-ast] (:variables built-ast) + (doseq [[_name var-ast] (:variables chained-ast) table (concat (:tables var-ast) (:selected-tables var-ast))] (assert-clean-table table))) - (testing "payload stays flat as more expressions chain, not superlinear" - ;; Regression guard: before the fix this was ~285KB for this 4-expression - ;; chain (each block re-embedding every prior block's full state); it - ;; should stay in the same ballpark regardless of chain length. - (is (< (count (json/generate-string built-ast)) 5000))))) + (testing "an earlier variable's own entry is unaffected by how many blocks chain after it" + ;; The actual bug wasn't about absolute size (which is arbitrary and brittle to + ;; pin to a byte count) — it was that x's entry kept growing every time another + ;; block chained onto it. Pruning removes the machinery (:variables/:references/ + ;; :ast) that let that happen, so x's pruned entry here should be byte-for-byte + ;; identical whether it's standing alone or three more blocks have chained onto + ;; it since. + (is (= (get-in single-ast [:pending-assignments "x"]) + (get-in chained-ast [:variables "x"])))))) From e8ac6873e22338f6ae3bc6ce6e5fff6a1d0a956c Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 15:56:55 +0200 Subject: [PATCH 48/60] test: exercise api-build directly instead of the private prune-ast helper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prior test called #'api/prune-ast directly, which only proves the helper itself behaves correctly — it proves nothing about whether api-build still routes its result through prune-ast at all, or would catch a future edit that forgets to call it, passes the wrong state, or leaks a new field through some other path. None of that wiring was under test. api-build couldn't be called directly in tests before this: it calls connections/get-connection-name, which requires a real registered connection pool and throws "Connection not found" for :test. :test was already a recognized bypass for schema lookups (postgres.clj, used throughout the existing test suite) but not for the connection-name lookup. Rather than hardcode a second (= id :test) check in connections.clj, test-connection-id/test-connection? are now defined once in connections.clj (the lowest-level db namespace, with no dependents in this graph) and postgres.clj's schema lookup calls the same predicate instead of its own copy. api-build and api-eval can now be called directly with connection-id :test, so the test exercises the real public entry point end to end. --- src/pine/db/connections.clj | 15 +++++++++- src/pine/db/postgres.clj | 2 +- test/pine/api_test.clj | 60 +++++++++++++++++-------------------- 3 files changed, 43 insertions(+), 34 deletions(-) diff --git a/src/pine/db/connections.clj b/src/pine/db/connections.clj index 4241525..43c4ab0 100644 --- a/src/pine/db/connections.clj +++ b/src/pine/db/connections.clj @@ -26,6 +26,17 @@ (def pools "Database connection pools" (atom {})) +(def test-connection-id + "Sentinel connection id that bypasses real connection pools and live schema + lookups in favor of fixtures. Defined here — the db namespace nothing else + in pine.db depends on — so every place that needs to recognize it (schema + lookup in postgres.clj, connection-name lookup below) checks the one + predicate instead of each hardcoding :test separately." + :test) + +(defn test-connection? [id] + (= id test-connection-id)) + (defn get-connection-pool [id] (let [pool-or-fn (@pools id)] (if pool-or-fn @@ -49,7 +60,9 @@ (jdbc-url->label (.getJdbcUrl pool))) (defn get-connection-name [id] - (-> id get-connection-pool make-connection-id)) + (if (test-connection? id) + "test" + (-> id get-connection-pool make-connection-id))) (defn list-connections [] (mapv (fn [[id _]] diff --git a/src/pine/db/postgres.clj b/src/pine/db/postgres.clj index ddaaafa..4d92c7d 100644 --- a/src/pine/db/postgres.clj +++ b/src/pine/db/postgres.clj @@ -225,7 +225,7 @@ FROM information_schema.columns"] (defn get-indexed-references [id] (let [references (cond - (= id :test) fixtures/references + (connections/test-connection? id) fixtures/references :else (get-references-helper id))] (index-references references))) diff --git a/test/pine/api_test.clj b/test/pine/api_test.clj index 7699a81..3c6c488 100644 --- a/test/pine/api_test.clj +++ b/test/pine/api_test.clj @@ -1,36 +1,32 @@ (ns pine.api-test (:require [clojure.test :refer [deftest is testing]] - [pine.api :as api] - [pine.ast.main :as ast] - [pine.parser :as parser])) - -(defn- generate-with-variables - "Evaluate a sequence of expressions sequentially, threading variables between - them the same way api-build's evaluate-expressions does. Returns the final state." - [expressions] - (:last-state - (reduce (fn [{:keys [variables]} expr] - (let [{:keys [result]} (parser/parse expr) - state (ast/generate result :test nil nil variables)] - {:variables (merge variables (:pending-assignments state)) - :last-state state})) - {:variables {} :last-state nil} - expressions))) + [pine.api :as api])) (defn- assert-clean-table [table] (is (= #{:schema :table :alias} (set (keys table))) "table entries must not carry :ast (a variable's own full state snapshot)")) -(deftest test-prune-ast - (let [single-block (generate-with-variables - ["tenant as t | company .tenantId | group: t.title |= x"]) - chained-blocks (generate-with-variables - ["tenant as t | company .tenantId | group: t.title |= x" - "x | s: count, | o: count desc |= y" - "y | s: count, |= z" - "z | "]) - single-ast (#'api/prune-ast single-block) - chained-ast (#'api/prune-ast chained-blocks)] +;; These call the real public entry point (api/api-build), not the private +;; prune-ast helper directly. Testing prune-ast in isolation only proves the +;; helper itself behaves correctly — it proves nothing about whether api-build +;; still routes its result through prune-ast at all, so a future edit that +;; forgets to call it, passes the wrong state, or leaks a new field through +;; some other path would go uncaught. Calling api-build with connection-id +;; :test is what makes this possible: :test is a shared sentinel +;; (pine.db.connections/test-connection-id) that both the schema lookup +;; (postgres.clj) and the connection-name lookup (connections.clj) recognize, +;; so api-build's connections/get-connection-name call — which normally +;; requires a real registered connection pool — succeeds without one. +(deftest test-api-build-ast + (let [single-block (:ast (api/api-build + ["tenant as t | company .tenantId | group: t.title |= x"] + nil :test)) + chained-blocks (:ast (api/api-build + ["tenant as t | company .tenantId | group: t.title |= x" + "x | s: count, | o: count desc |= y" + "y | s: count, |= z" + "z | "] + nil :test))] (testing "ast.variables entries are pruned like pending-assignments, not raw snapshots" ;; A raw variable snapshot carries :variables and :references from @@ -38,8 +34,8 @@ ;; earlier variable's own full snapshot inside the new one, growing the ;; response payload superlinearly with the number of chained expressions ;; instead of linearly. - (is (= #{"x" "y" "z"} (set (keys (:variables chained-ast))))) - (doseq [[name var-ast] (:variables chained-ast)] + (is (= #{"x" "y" "z"} (set (keys (:variables chained-blocks))))) + (doseq [[name var-ast] (:variables chained-blocks)] (testing (str "variable " name) (is (= #{:tables :selected-tables :joins :columns} (set (keys var-ast))) "should only carry the fields VariableAst (client.ts) actually uses") @@ -54,9 +50,9 @@ ;; recursively re-embeds the variable's entire state — and everything IT ;; wraps in turn — inside every table list that references it, one level ;; down from the :variables map itself. - (doseq [table (:selected-tables chained-ast)] + (doseq [table (:selected-tables chained-blocks)] (assert-clean-table table)) - (doseq [[_name var-ast] (:variables chained-ast) + (doseq [[_name var-ast] (:variables chained-blocks) table (concat (:tables var-ast) (:selected-tables var-ast))] (assert-clean-table table))) @@ -67,5 +63,5 @@ ;; :ast) that let that happen, so x's pruned entry here should be byte-for-byte ;; identical whether it's standing alone or three more blocks have chained onto ;; it since. - (is (= (get-in single-ast [:pending-assignments "x"]) - (get-in chained-ast [:variables "x"])))))) + (is (= (get-in single-block [:pending-assignments "x"]) + (get-in chained-blocks [:variables "x"])))))) From 8ab23460c357b296933f73e83ddc0c0a3a8bb45e Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 16:09:19 +0200 Subject: [PATCH 49/60] docs: correct claim that a bare group: col has no aggregate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit parser.clj defaults :functions to ["count"] whenever => is omitted — there's no syntax for a truly aggregate-less GROUP. Found while tracing a variable's join behavior: the code change (reading :columns as-is, never appending count separately) was already correct either way, but the docstring/doc claimed a case that doesn't actually occur. --- docs/variables.md | 7 ++++--- src/pine/ast/main.clj | 10 +++++----- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/variables.md b/docs/variables.md index 8db17de..d4f9068 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -169,9 +169,10 @@ passes to avoid duplicating that logic. - **Explicit columns** (`s:`, `g:`, etc.): the column list is built from the user-selected columns. The name used is `column-alias` if set, otherwise `column`. For a GROUP-sourced CTE this naturally includes - the aggregate (e.g. `count`) whenever the GROUP specified one — group.clj folds it directly into - `:columns`, so it's read here like any other column rather than appended separately. (`=> count` is - optional in the grammar; a bare `group: col` with no aggregate correctly yields no `count` entry.) + the aggregate — group.clj folds it directly into `:columns`, so it's read here like any other column + rather than appended separately. (`=> count` is optional in the *grammar*, but parser.clj defaults + `:functions` to `["count"]` whenever it's omitted — there's no way to write a truly aggregate-less + GROUP, so this case always has a `count` entry in practice.) - **No explicit columns** (`*`): the source table's full column list is inherited. ### Alias disambiguation diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 0152e22..2adffcd 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -96,11 +96,11 @@ "Return the column list a variable's CTE actually exposes, for hint generation. Returns nil when the CTE selects *, meaning the source table's columns apply. - For a GROUP-sourced CTE, :columns already includes the aggregate entry (e.g. - count) whenever the GROUP actually specified one — group.clj folds it in - directly — so it's read here like any other column, never added separately. - (`=> count` is optional in the grammar; a bare `group: col` has no aggregate - at all, and fabricating one would be just as wrong as double-counting it.)" + For a GROUP-sourced CTE, :columns already includes the aggregate entry — + group.clj folds it in directly (parser.clj defaults :functions to [\"count\"] + even when `=> count` is omitted; there's no way to write a truly aggregate- + less GROUP) — so it's read here like any other column, never added + separately. Appending it again on top, as this used to do, double-counted." [var-ast] (let [user-cols (remove :auto-id (:columns var-ast))] (when (seq user-cols) From eda5c1dd95aad5b2e1e8563ce4a12c2f3ab78024 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 16:21:10 +0200 Subject: [PATCH 50/60] test: a table is only a valid join source through a variable if its id survives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit get-source-tables currently treats any table referenced by an explicit column as a join source, regardless of whether that table's own id column is among the explicit columns. Since a variable's CTE never gets an auto-id added (post-handle, which adds it, runs after handle-ops — after any |= snapshot is already taken), a variable with explicit columns that don't include a table's id has no way to actually join back to it. Joining inherits that table's FK relations anyway, producing a query that references e.g. "x"."id" — a column that doesn't exist in the CTE. Confirmed this isn't GROUP-specific: `s: name` (no group involved) hits the exact same bug as `group: name`. `s: id, name` / `group: id, name` are fine since id survives. Two of the four cases here currently fail (the two missing id); fix follows. --- test/pine/hints_test.clj | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index 7c566e1..ab537f0 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -243,5 +243,31 @@ (is (= [{:schema nil :table "var_y" :column nil :parent false :heuristic false :pine "var_y"}] (-> (gen-with-variables ["customer |= var_x" "customer |= var_y" "var_x | var"]) - :table))))) + :table)))) + + (testing "A table is only a valid join source through a variable if its own id survives" + ;; x explicitly selects only name — company's id is not in x's own CTE output, so a + ;; join inherited from company's FK relations would reference a column ("x"."id") + ;; that doesn't exist. employee must NOT be suggested. + (is (= [] + (-> (gen-with-variables ["company as c | s: name |= x" "x | emp"]) + :table))) + + ;; Same table, but id IS explicitly selected — company's id survives in x's output, + ;; so the join is valid and employee should be suggested. + (is (= ["employee"] + (->> (gen-with-variables ["company as c | s: id, name |= x" "x | emp"]) + :table + (map :table)))) + + ;; GROUP is not special-cased: grouping by a non-id column loses id the same way. + (is (= [] + (-> (gen-with-variables ["company as c | employee .company_id | group: c.name |= x" "x | doc"]) + :table))) + + ;; ...but grouping by id (alongside other columns) preserves it, same as s: id, name. + (is (= ["document"] + (->> (gen-with-variables ["company as c | employee .company_id | group: c.id, c.name |= x" "x | doc"]) + :table + (map :table)))))) From fbd2a52a822dd492d3c9cd875e9fc6231f25f7f0 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 16:22:59 +0200 Subject: [PATCH 51/60] fix: a table is only a join source through a variable if its id survives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit get-source-tables previously treated any table referenced by an explicit column as a valid join source, regardless of whether that table's own id column was among the explicit columns. A variable's CTE never gets an auto-id column added (post-handle, which adds it, runs after handle-ops — after any |= snapshot is already taken), so joining in a relation inherited from a table whose id isn't actually in the CTE's output produces a query referencing a column that doesn't exist (e.g. "x"."id"). This wasn't GROUP-specific, despite first looking that way: `s: name` hits the identical bug without any GROUP involved, since explicit column selection (via s: or group:) is the actual trigger, not the operation type. No explicit columns (implicit '*') was always safe and is unaffected. Now filters explicit columns down to ones literally named "id" before mapping to tables, matching the id-based join convention used everywhere else in this file (patch-same-source-variable-joins's has-id? check). Can return zero sources (nothing joinable, correctly disabling hints/joins entirely), one, or several — e.g. `s: t.id, c.id` makes both valid, mirroring the multi-table joins a plain non-variable pipeline already supports. Fixes the two previously-failing cases in the last commit's regression test. --- src/pine/ast/main.clj | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 2adffcd..a59c8aa 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -110,7 +110,25 @@ (mapv #(hash-map :column %)))))) (defn- get-source-tables - "Return the tables whose columns are exposed by a variable's CTE. + "Return the tables that are valid join sources for a variable's CTE. + + A variable's CTE never gets an auto-id column added — post-handle (which adds + it) runs after handle-ops, after any |= snapshot has already been taken — so a + table is only actually joinable through this variable if ITS OWN id column is + among the CTE's explicit output columns. Otherwise, inheriting joins from that + table's FK relations would reference a column (e.g. \"x\".\"id\") that doesn't + exist in the CTE at all. + + No explicit columns at all means the CTE implicitly selects '*' (see + variable-output-columns), which always includes id — the :current table is + then the sole, always-safe source. + + With explicit columns, this can return zero tables (nothing joinable — e.g. + `s: name` or `group: name` alone, neither of which keeps any table's id), + one, or more than one (e.g. `s: t.id, c.id` makes both t and c valid sources, + the same multi-table join support a plain multi-table pipeline already has + without any variable involved). + Used by both seeding and bidirectional patching." [var-ast] (let [columns (:columns var-ast) @@ -120,7 +138,9 @@ (when-let [current-alias (:current var-ast)] [(get-in aliases [current-alias :table])]) (->> explicit - (map #(get-in aliases [(:alias %) :table])) + (filter #(= "id" (:column %))) + (map :alias) + (map #(get-in aliases [% :table])) (remove nil?) distinct)) []))) From 1be1bc7d581340d1fc153610e55078729efaada7 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 16:24:27 +0200 Subject: [PATCH 52/60] docs: document the id-survival rule for variable join sources MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updates "Join resolution through variables" to lead with what get-source-tables actually decides (a table is a join source only if its id survives among the variable's exposed columns, zero/one/many tables can qualify) before describing the three passes that build on it — they were previously described as if any referenced table were automatically a source. Also annotates the existing "Grouped" example, which is exactly this scenario: grouping by title alone leaves neither tenant nor company joinable through x. --- docs/variables.md | 46 +++++++++++++++++++++++++++++++++++++--------- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/docs/variables.md b/docs/variables.md index d4f9068..3d7a74c 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -78,6 +78,11 @@ x `x` exposes only the columns the CTE actually produces — here `title` and `count` — so hints for `x | s:` show just those two columns. +Neither `tenant` nor `company` survives as a join source, though: grouping by `t.title` doesn't keep +`tenant`'s `id`, so `x | company` (or any other table) shows no join hints at all — see +[get-source-tables](#join-resolution-through-variables) below. Grouping by `t.id, t.title` instead would +keep `tenant` joinable. + ## How it works - **Assignment** (`|= name`): snapshots the pipeline state at that point. The snapshot becomes the CTE body @@ -135,16 +140,42 @@ query's params list. ### Join resolution through variables -Reference seeding happens in three passes inside `pre-handle` (`ast/main.clj`): - -**Pass 1 — `seed-variable-references`**: For each variable V wrapping source table S, copies S's FK -reference entry into the references map under V's name. This enables `V | table` and `table | V` when +All three passes below build on one question, answered by the shared helper `get-source-tables`: +**which real tables can this variable actually be joined to?** A table only counts as a source if its +own `id` column is present among the variable's exposed columns — not merely referenced somewhere in the +pipeline that produced it. + +This matters because a variable's CTE never gets Pine's auto-id column added. `post-handle` (which adds +it for ordinary tables) runs after `handle-ops` — after any `|=` snapshot has already been taken — so an +explicit column list that doesn't happen to include a table's `id` leaves that table with no way back +into a join. Seeding or patching a join from that table's FK relations anyway would generate SQL that +references a column the CTE doesn't have (e.g. `"x"."id"`), silently, since the query builder isn't aware +that particular table's identity didn't survive. + +`get-source-tables`' rule, concretely: + +- **No explicit columns** (`*`, e.g. a bare `where:`/`limit:` with no `s:`/`g:`): the CTE implicitly + selects everything, which always includes `id`. The variable's `:current` table is the sole, + always-safe source. +- **Explicit columns** (from `s:`, or from `group:`'s grouped columns — GROUP is not special-cased; both + populate `:columns` the same way): only tables whose own `id` column is literally among those explicit + columns count as sources. This can resolve to **zero** tables (`s: name` or `group: name` alone — + nothing is joinable, and no join hints should appear), **one**, or **more than one** (`s: t.id, c.id` + makes both `t` and `c` valid sources) — the same multi-table join support a plain, variable-free + pipeline already has when it references more than one table. + +With that settled, the three passes propagate joins for whichever source tables `get-source-tables` +returned, inside `pre-handle` (`ast/main.clj`): + +**Pass 1 — `seed-variable-references`**: For each variable V, for each of its source tables S, copies S's +FK reference entry into the references map under V's name. This enables `V | table` and `table | V` when the join helper can find `table[:referred-by][V]` — but only if that entry already exists, which it -doesn't yet after pass 1 alone. +doesn't yet after pass 1 alone. A variable with zero source tables (see above) has nothing to seed and +stays unjoinable to anything, correctly. **Pass 2 — `patch-variable-relations`** (runs `patch-direction` once per direction): Builds a reverse index from the references map — `source-table -> entities that already relate to it` — once per direction, -instead of scanning every entity per variable. For each variable V wrapping source S, looks up S in that +instead of scanning every entity per variable. For each variable V, for each source S, looks up S in that index to find every entity T where `T[:referred-by][S]` exists, and registers `T[:referred-by][V]` with the same relation data. The index is kept in sync with entries added mid-pass, so a variable-of-variable (V2 wrapping V1, where V1 is itself a variable processed earlier in the same pass) still picks up V1's freshly @@ -160,9 +191,6 @@ join at `refs[:table V1 :referred-by V2]`. This makes `V1 | V2` resolve even whe self-referential FK. Only adds entries where no join path already exists — existing FK-based propagation (e.g. two employee-wrapping variables joined via `reports_to`) is not overridden. -The shared helper `get-source-tables` extracts which real tables a variable wraps — used by all three -passes to avoid duplicating that logic. - ### Column hints for variables `seed-variable-references` also overrides the column list for the variable entry in the references map: From 362781304818f5a4c0f3a7958c28b70af0a4e6b4 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 19:36:33 +0200 Subject: [PATCH 53/60] fix: add auto-id to variable/checkpoint snapshots instead of requiring explicit id MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit made get-source-tables require a table's id to be explicitly selected before treating it as a join source — correct for GROUP, but overly conservative for everything else: a plain `s: name |= x` doesn't need explicit id in an ordinary (non-variable) pipeline, because post-handle silently adds a hidden auto-id column regardless of what was explicitly selected. That's the actual reason "you don't need explicit id to join" is true in general — and it wasn't happening for variable/checkpoint snapshots at all, since assign/handle and flush-checkpoint's auto-named branch take their snapshot mid-fold, before post-handle ever runs. Fixes the root cause instead: assign/handle and flush-checkpoint's auto-named branch now run select/add-auto-id-columns on the snapshot the same way post-handle would, guarded by the same should-add-auto-ids? check that already excludes :group (along with :count/:delete-action/ :update-action). So: - Non-GROUP snapshots always end up with an id for every real table in :tables, restoring the original permissive behavior for plain s:/where/ table variables — now backed by an id that's actually in the CTE, not an assumption. - GROUP snapshots still only get an id if the user explicitly grouped by it — there's no way to silently add an unaggregated column to a GROUP BY without changing what's grouped by, which is why this is the one operation that genuinely needs the explicit-id check. get-source-tables no longer needs to special-case operation type at all: it just filters :columns (including auto-id entries now) for "id" instead of explicit-only columns. GROUP naturally has no auto-id entries to find; everything else naturally does. Updates the first hints_test.clj case from the previous commit: `s: name` now correctly shows employee as joinable again (auto-id fixes it for real), matching what a non-variable pipeline already does. The GROUP-without-id case stays correctly disabled. --- src/pine/ast/assign.clj | 23 ++++++++++++++++--- src/pine/ast/main.clj | 49 +++++++++++++++++++++++++--------------- test/pine/hints_test.clj | 21 +++++++++-------- 3 files changed, 63 insertions(+), 30 deletions(-) diff --git a/src/pine/ast/assign.clj b/src/pine/ast/assign.clj index 97c5c1e..a3eec5a 100644 --- a/src/pine/ast/assign.clj +++ b/src/pine/ast/assign.clj @@ -10,14 +10,31 @@ Why a snapshot: storing a dissoc'd copy of state (not the live state) prevents the pending-assignments map from accumulating nested copies across multiple - assignments in a single expression.") + assignments in a single expression. + + Why add-auto-id-columns here: post-handle normally adds auto-id columns after + handle-ops finishes, but a |= snapshot is taken mid-fold, before post-handle + ever runs — so without this, no snapshot would ever carry an id column, even + for an ordinary select/where/table with no explicit :s: at all. Applying it + to the snapshot directly (guarded the same way post-handle's is, via + should-add-auto-ids? — excluded for :group/:count/:delete-action/:update-action) + makes a variable's join eligibility match what an equivalent non-variable + pipeline already gets for free. GROUP is excluded because there's no way to + silently add an unaggregated id column to a GROUP BY query — see + get-source-tables in ast/main.clj for how that case is handled instead." + (:require + [pine.ast.select :as select])) (defn handle "Snapshot the current state under varname in :pending-assignments. - The snapshot excludes :pending-assignments itself to prevent nesting. + The snapshot excludes :pending-assignments itself to prevent nesting, and + gets auto-id columns added the same way an ordinary (non-variable) pipeline + would via post-handle — see the namespace docstring for why that doesn't + already happen by the time a snapshot is taken. Also registers varname as a local alias for the current SQL alias so subsequent ops in the same expression can write e.g. x.id to mean c_0.id." [state varname] (-> state - (assoc-in [:pending-assignments varname] (dissoc state :pending-assignments)) + (assoc-in [:pending-assignments varname] + (-> state (dissoc :pending-assignments) select/add-auto-id-columns)) (assoc :assign varname))) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index a59c8aa..a37a47a 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -110,24 +110,34 @@ (mapv #(hash-map :column %)))))) (defn- get-source-tables - "Return the tables that are valid join sources for a variable's CTE. - - A variable's CTE never gets an auto-id column added — post-handle (which adds - it) runs after handle-ops, after any |= snapshot has already been taken — so a - table is only actually joinable through this variable if ITS OWN id column is - among the CTE's explicit output columns. Otherwise, inheriting joins from that - table's FK relations would reference a column (e.g. \"x\".\"id\") that doesn't - exist in the CTE at all. - - No explicit columns at all means the CTE implicitly selects '*' (see - variable-output-columns), which always includes id — the :current table is + "Return the tables that are valid join sources for a variable's CTE — the + tables whose own id column is present among the CTE's actual output columns + (:columns, auto-id included). + + assign/handle and flush-checkpoint's auto-named branch add auto-id columns + to a snapshot the same way an ordinary (non-variable) pipeline gets them via + post-handle, for every operation type except :group (should-add-auto-ids? in + select.clj) — GROUP can't silently add an unaggregated id column to a + GROUP BY without changing what the aggregation groups by. In practice: + + - Non-GROUP CTEs always end up with an auto-id for every real table in + :tables, so every table referenced by an explicit column stays a valid + source — same as before this existed, now backed by an id that's + actually present in the CTE. + - GROUP CTEs only get an id for a table if the user explicitly grouped by + that table's id. Grouping by anything else (e.g. `group: name`) means + that table's id doesn't survive the aggregation, and it correctly stops + being a valid source — joining to it would otherwise reference a column + (e.g. \"x\".\"id\") the CTE doesn't have. + + No explicit columns at all means the CTE implicitly selects '*', which + always includes id regardless of operation type — the :current table is then the sole, always-safe source. - With explicit columns, this can return zero tables (nothing joinable — e.g. - `s: name` or `group: name` alone, neither of which keeps any table's id), - one, or more than one (e.g. `s: t.id, c.id` makes both t and c valid sources, - the same multi-table join support a plain multi-table pipeline already has - without any variable involved). + Can return zero tables (e.g. `group: name` alone), one, or more than one + (e.g. `s: t.id, c.id`, or any non-GROUP CTE spanning multiple tables) — the + same multi-table join support a plain, variable-free pipeline already has + when it references more than one table. Used by both seeding and bidirectional patching." [var-ast] @@ -137,7 +147,7 @@ (or (if (empty? explicit) (when-let [current-alias (:current var-ast)] [(get-in aliases [current-alias :table])]) - (->> explicit + (->> columns (filter #(= "id" (:column %))) (map :alias) (map #(get-in aliases [% :table])) @@ -339,10 +349,13 @@ (assoc state :pending-checkpoint {:name (:value op) :needs-table true}) ;; Auto-named: fire when a table or another checkpoint op arrives + ;; add-auto-id-columns mirrors what assign/handle does for a |= snapshot: this + ;; snapshot is also taken mid-fold, before post-handle would otherwise add it. + ;; Excluded for :group via should-add-auto-ids? — see get-source-tables. (and (:needs-assign checkpoint) fire?) (let [n (:auto-cte-count state) cname (str "__pine_" n "__") - snapshot (dissoc state :pending-assignments)] + snapshot (-> state (dissoc :pending-assignments) select/add-auto-id-columns)] (-> state (update :auto-cte-count inc) (assoc :pending-checkpoint nil) diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index ab537f0..7ba79e5 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -246,21 +246,24 @@ :table)))) (testing "A table is only a valid join source through a variable if its own id survives" - ;; x explicitly selects only name — company's id is not in x's own CTE output, so a - ;; join inherited from company's FK relations would reference a column ("x"."id") - ;; that doesn't exist. employee must NOT be suggested. - (is (= [] - (-> (gen-with-variables ["company as c | s: name |= x" "x | emp"]) - :table))) + ;; x explicitly selects only name, but a plain (non-GROUP) select gets an auto-id + ;; column added to the snapshot the same way an ordinary pipeline would (see + ;; assign/handle) — company's id is present in x's actual CTE output either way, + ;; so employee is correctly still suggested. + (is (= ["employee"] + (->> (gen-with-variables ["company as c | s: name |= x" "x | emp"]) + :table + (map :table)))) - ;; Same table, but id IS explicitly selected — company's id survives in x's output, - ;; so the join is valid and employee should be suggested. + ;; Same table, id also explicitly selected — same result, redundantly safe. (is (= ["employee"] (->> (gen-with-variables ["company as c | s: id, name |= x" "x | emp"]) :table (map :table)))) - ;; GROUP is not special-cased: grouping by a non-id column loses id the same way. + ;; GROUP is the one operation that can't get an auto-id: an unaggregated id column + ;; can't be silently added to a GROUP BY without changing what's grouped by. Grouping + ;; by a non-id column genuinely loses company's id — employee must NOT be suggested. (is (= [] (-> (gen-with-variables ["company as c | employee .company_id | group: c.name |= x" "x | doc"]) :table))) From 6451d92d563de8b84ae802c2f662a37280a2daad Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 19:37:23 +0200 Subject: [PATCH 54/60] docs: update join-source rule for the auto-id fix, not just the GROUP gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previous doc update described get-source-tables as uniformly requiring explicit id for any table with explicit columns — accurate right after that commit, but superseded by the follow-up fix: auto-id is now added to non-GROUP snapshots the same way an ordinary pipeline gets it via post-handle, so explicit id is only actually required for GROUP. Rewrites the section to explain why that asymmetry exists (post-handle's should-add-auto-ids? guard, which was already excluding :group) instead of presenting a single flat rule. --- docs/variables.md | 42 +++++++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/docs/variables.md b/docs/variables.md index 3d7a74c..4c677ab 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -142,27 +142,31 @@ query's params list. All three passes below build on one question, answered by the shared helper `get-source-tables`: **which real tables can this variable actually be joined to?** A table only counts as a source if its -own `id` column is present among the variable's exposed columns — not merely referenced somewhere in the -pipeline that produced it. - -This matters because a variable's CTE never gets Pine's auto-id column added. `post-handle` (which adds -it for ordinary tables) runs after `handle-ops` — after any `|=` snapshot has already been taken — so an -explicit column list that doesn't happen to include a table's `id` leaves that table with no way back -into a join. Seeding or patching a join from that table's FK relations anyway would generate SQL that -references a column the CTE doesn't have (e.g. `"x"."id"`), silently, since the query builder isn't aware -that particular table's identity didn't survive. - -`get-source-tables`' rule, concretely: +own `id` column is present among the CTE's *actual* output columns — auto-id included, not just what was +explicitly typed. + +That distinction matters because of when a variable's snapshot is taken. `post-handle` normally adds a +hidden auto-id column for every real table in the pipeline, regardless of what was explicitly selected — +that's the reason an ordinary, variable-free `s: name | employee` doesn't need explicit `id` to join +correctly. But a `|=` or checkpoint snapshot is taken mid-`handle-ops`, before `post-handle` ever runs, so +without deliberately re-adding it, no snapshot would ever carry an id column at all. `assign/handle` and +`flush-checkpoint`'s auto-named branch both now call `select/add-auto-id-columns` on the snapshot +themselves — guarded by the same `should-add-auto-ids?` check `post-handle` already uses, which excludes +`:group` (along with `:count`/`:delete-action`/`:update-action`). That one guard is what makes +`get-source-tables` behave differently depending on how the CTE's columns came to be: - **No explicit columns** (`*`, e.g. a bare `where:`/`limit:` with no `s:`/`g:`): the CTE implicitly - selects everything, which always includes `id`. The variable's `:current` table is the sole, - always-safe source. -- **Explicit columns** (from `s:`, or from `group:`'s grouped columns — GROUP is not special-cased; both - populate `:columns` the same way): only tables whose own `id` column is literally among those explicit - columns count as sources. This can resolve to **zero** tables (`s: name` or `group: name` alone — - nothing is joinable, and no join hints should appear), **one**, or **more than one** (`s: t.id, c.id` - makes both `t` and `c` valid sources) — the same multi-table join support a plain, variable-free - pipeline already has when it references more than one table. + selects everything, which always includes `id` regardless of operation type. The variable's `:current` + table is the sole, always-safe source. +- **Non-`GROUP` explicit columns** (`s:`): auto-id gets added for every real table in `:tables`, so every + table referenced by an explicit column stays a valid source — the same result as before auto-id was + reintroduced here, just backed by an id that's actually present now instead of assumed. +- **`GROUP`'s grouped columns**: no auto-id is ever added — an unaggregated id column can't be silently + added to a `GROUP BY` without changing what's grouped by. A table is a source here *only* if the user + explicitly grouped by that table's `id`. `group: name` alone therefore resolves to **zero** sources + (nothing joinable, no join hints should appear); `group: id, name` resolves to one; `group: t.id, c.id` + can resolve to more than one — the same multi-table join support a plain, variable-free pipeline already + has when it references more than one table. With that settled, the three passes propagate joins for whichever source tables `get-source-tables` returned, inside `pre-handle` (`ast/main.clj`): From e3d6ac676bce6dc42880d8d81e71ad502291888d Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 19:49:22 +0200 Subject: [PATCH 55/60] refactor: decouple join-key preservation from update-tracking auto-id MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous fix reused select/add-auto-id-columns to make sealed CTEs joinable — it happened to produce correct SQL, but for the wrong reason: auto-id columns exist so the UI can identify which row to update (see result-updates.md), an unrelated concern. Coupling the two meant a future change to update-tracking (e.g. making it opt-out) could silently break variable/checkpoint joins. The actual reason a raw table join never needs explicit id in SELECT: a JOIN ... ON clause can reference any column of a real table in FROM/JOIN regardless of what's selected — SQL doesn't restrict that based on the SELECT list. A CTE is different in kind, not degree: once state is sealed into one, the outer query can only see whatever that CTE's own :columns selected. So a table stays a valid join source through a sealed variable only if its id is actually there — not because of anything to do with auto-id specifically. assign.clj now owns this as its own mechanism: preserve-join-keys adds a column marked :hidden and :join-key (deliberately NOT :auto-id) for every real table when a snapshot's explicit columns are non-empty and it isn't a GROUP — GROUP still can't get one unless the user explicitly grouped by it, since an unaggregated id can't be patched into a GROUP BY after the fact. assign/snapshot (dissoc + preserve-join-keys) is the one shared entry point for sealing state into a CTE, used identically by a |= assignment (assign/handle) and a checkpoint's auto-named seal (flush-checkpoint in main.clj) — the two places this actually happens. get-source-tables and variable-output-columns now filter on :hidden (shared by both mechanisms, for "don't show this in hints") rather than :auto-id specifically, so they no longer care which mechanism produced a column, only whether it's meant to be internal. select/has-id-column? made public — a purpose-agnostic schema lookup now shared between the two concerns, without sharing anything else. Behavior is unchanged from the previous commit; this is a decoupling refactor, not a bug fix, confirmed by the full suite passing unchanged plus one new test asserting the join-key column is never marked :auto-id. --- src/pine/ast/assign.clj | 88 +++++++++++++++++++++++++++++++++-------- src/pine/ast/main.clj | 38 ++++++++++-------- src/pine/ast/select.clj | 7 +++- test/pine/eval_test.clj | 34 +++++++++++++++- 4 files changed, 131 insertions(+), 36 deletions(-) diff --git a/src/pine/ast/assign.clj b/src/pine/ast/assign.clj index a3eec5a..4d0eef2 100644 --- a/src/pine/ast/assign.clj +++ b/src/pine/ast/assign.clj @@ -12,29 +12,83 @@ the pending-assignments map from accumulating nested copies across multiple assignments in a single expression. - Why add-auto-id-columns here: post-handle normally adds auto-id columns after - handle-ops finishes, but a |= snapshot is taken mid-fold, before post-handle - ever runs — so without this, no snapshot would ever carry an id column, even - for an ordinary select/where/table with no explicit :s: at all. Applying it - to the snapshot directly (guarded the same way post-handle's is, via - should-add-auto-ids? — excluded for :group/:count/:delete-action/:update-action) - makes a variable's join eligibility match what an equivalent non-variable - pipeline already gets for free. GROUP is excluded because there's no way to - silently add an unaggregated id column to a GROUP BY query — see - get-source-tables in ast/main.clj for how that case is handled instead." + Why join-key preservation here, and why it's NOT select/add-auto-id-columns: + once a snapshot is sealed into a CTE, the outer query can no longer see its + underlying real tables directly — only whatever the CTE's own :columns select. + A raw table join never needs an id column in SELECT (a JOIN ... ON clause can + reference any column of a real table regardless of what's selected), but a CTE + is different in kind: its column list IS its entire visible schema. So a table + is only actually joinable through a sealed variable if its own id survived into + that snapshot's :columns. + + This is unrelated to select/add-auto-id-columns, which exists purely so the UI + can identify which row to update (see result-updates.md) — a different concern + that happens to also add an id column, for different reasons, at a different + time (post-handle, on the final returned state, not on a mid-fold snapshot). + Coupling the two would mean a future change to update-tracking (e.g. making it + opt-out) could silently break variable joins for a completely unrelated reason. + preserve-join-keys is deliberately its own mechanism: its columns are marked + :hidden (excluded from column hints, same as auto-id) but NOT :auto-id, so nothing + downstream can conflate 'has a join key' with 'is tracked for updates'. + + GROUP is excluded (see preserve-join-keys): there's no way to silently add an + unaggregated id column to a GROUP BY without changing what's grouped by. See + get-source-tables in ast/main.clj, which is what actually enforces that a + GROUP-sourced variable needs its id explicitly grouped by to be joinable." (:require [pine.ast.select :as select])) +(defn- create-join-key-column + "A hidden column that exists purely so a sealed CTE can still be joined via id. + Deliberately not shaped like select/create-auto-id-column's output (no + :column-alias, not marked :auto-id) — this is a different mechanism for a + different purpose; see the namespace docstring." + [alias operation-index] + {:column "id" :alias alias :hidden true :join-key true :operation-index operation-index}) + +(defn- has-explicit-id? + [columns alias] + (some #(and (= alias (:alias %)) (= "id" (:column %))) columns)) + +(defn preserve-join-keys + "Ensure a state that's about to be sealed into a CTE keeps enough of an id + column, per real table, to remain joinable once sealing hides its underlying + tables from the outer query. No-op when :columns is empty (an implicit '*' + already includes id, regardless of operation type) or when this is a GROUP + (aggregation means an id can only survive if the user explicitly grouped by + it — silently adding one here would change what's being grouped by)." + [state] + (let [explicit (remove :hidden (:columns state))] + (if (or (seq (:group state)) (empty? explicit)) + state + (let [table-aliases (map :alias (:tables state)) + next-operation-index (inc (:index state)) + references (:references state) + aliases (:aliases state) + valid-aliases (filter #(and (not (:ast (get aliases %))) + (not (has-explicit-id? (:columns state) %)) + (select/has-id-column? references aliases %)) + table-aliases) + join-key-columns (map-indexed #(create-join-key-column %2 (+ next-operation-index %1)) + valid-aliases)] + (update state :columns into join-key-columns))))) + +(defn snapshot + "Prepare state to be stored as a CTE-backing snapshot — used both for a |= + assignment (below) and for a checkpoint's auto-named seal (flush-checkpoint + in ast/main.clj), the two places a piece of pipeline state gets frozen for + later reference as a CTE. Drops :pending-assignments (preventing nested + accumulation across multiple assignments in one expression) and preserves + join keys (see preserve-join-keys)." + [state] + (-> state (dissoc :pending-assignments) preserve-join-keys)) + (defn handle - "Snapshot the current state under varname in :pending-assignments. - The snapshot excludes :pending-assignments itself to prevent nesting, and - gets auto-id columns added the same way an ordinary (non-variable) pipeline - would via post-handle — see the namespace docstring for why that doesn't - already happen by the time a snapshot is taken. + "Snapshot the current state under varname in :pending-assignments (see + snapshot above for what that entails). Also registers varname as a local alias for the current SQL alias so subsequent ops in the same expression can write e.g. x.id to mean c_0.id." [state varname] (-> state - (assoc-in [:pending-assignments varname] - (-> state (dissoc :pending-assignments) select/add-auto-id-columns)) + (assoc-in [:pending-assignments varname] (snapshot state)) (assoc :assign varname))) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index a37a47a..0da6912 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -96,13 +96,17 @@ "Return the column list a variable's CTE actually exposes, for hint generation. Returns nil when the CTE selects *, meaning the source table's columns apply. + Excludes :hidden columns — both select/create-auto-id-column's update-tracking + entries and assign/preserve-join-keys' join-key entries are internal, never + meant to appear as a selectable column in hints. + For a GROUP-sourced CTE, :columns already includes the aggregate entry — group.clj folds it in directly (parser.clj defaults :functions to [\"count\"] even when `=> count` is omitted; there's no way to write a truly aggregate- less GROUP) — so it's read here like any other column, never added separately. Appending it again on top, as this used to do, double-counted." [var-ast] - (let [user-cols (remove :auto-id (:columns var-ast))] + (let [user-cols (remove :hidden (:columns var-ast))] (when (seq user-cols) (->> user-cols (map #(or (:column-alias %) (:column %))) @@ -112,18 +116,20 @@ (defn- get-source-tables "Return the tables that are valid join sources for a variable's CTE — the tables whose own id column is present among the CTE's actual output columns - (:columns, auto-id included). + (:columns, join-key entries included). - assign/handle and flush-checkpoint's auto-named branch add auto-id columns - to a snapshot the same way an ordinary (non-variable) pipeline gets them via - post-handle, for every operation type except :group (should-add-auto-ids? in - select.clj) — GROUP can't silently add an unaggregated id column to a - GROUP BY without changing what the aggregation groups by. In practice: + assign/preserve-join-keys adds those join-key columns to a snapshot whenever + it's sealed into a CTE (via assign/snapshot, used by both a |= assignment and + a checkpoint's auto-named seal), for every case except :group — GROUP can't + silently add an unaggregated id column to a GROUP BY without changing what + the aggregation groups by. See assign.clj's namespace docstring for why this + is deliberately a separate mechanism from select/add-auto-id-columns (which + exists for update-row identification, an unrelated concern). In practice: - - Non-GROUP CTEs always end up with an auto-id for every real table in + - Non-GROUP CTEs always end up with a join key for every real table in :tables, so every table referenced by an explicit column stays a valid - source — same as before this existed, now backed by an id that's - actually present in the CTE. + source — same as before either mechanism existed, now backed by an id + that's actually present in the CTE. - GROUP CTEs only get an id for a table if the user explicitly grouped by that table's id. Grouping by anything else (e.g. `group: name`) means that table's id doesn't survive the aggregation, and it correctly stops @@ -143,7 +149,7 @@ [var-ast] (let [columns (:columns var-ast) aliases (:aliases var-ast) - explicit (remove :auto-id columns)] + explicit (remove :hidden columns)] (or (if (empty? explicit) (when-let [current-alias (:current var-ast)] [(get-in aliases [current-alias :table])]) @@ -348,14 +354,14 @@ (and (:needs-assign checkpoint) (= op-type :assign)) (assoc state :pending-checkpoint {:name (:value op) :needs-table true}) - ;; Auto-named: fire when a table or another checkpoint op arrives - ;; add-auto-id-columns mirrors what assign/handle does for a |= snapshot: this - ;; snapshot is also taken mid-fold, before post-handle would otherwise add it. - ;; Excluded for :group via should-add-auto-ids? — see get-source-tables. + ;; Auto-named: fire when a table or another checkpoint op arrives. + ;; assign/snapshot mirrors what |= does for its own snapshot: this is the + ;; other place a piece of state gets sealed into a CTE, so it needs the + ;; same join-key preservation (see assign.clj's namespace docstring). (and (:needs-assign checkpoint) fire?) (let [n (:auto-cte-count state) cname (str "__pine_" n "__") - snapshot (-> state (dissoc :pending-assignments) select/add-auto-id-columns)] + snapshot (assign/snapshot state)] (-> state (update :auto-cte-count inc) (assoc :pending-checkpoint nil) diff --git a/src/pine/ast/select.clj b/src/pine/ast/select.clj index d8bc045..380ee2e 100644 --- a/src/pine/ast/select.clj +++ b/src/pine/ast/select.clj @@ -23,8 +23,11 @@ (-> state (update :columns into columns)))) -(defn- has-id-column? - "Check if a table has an 'id' column by looking up the table info in references" +(defn has-id-column? + "Check if a table has an 'id' column by looking up the table info in references. + Public: shared with assign.clj's join-key preservation, a different concern (join + resolution through a sealed CTE, not update-row identification) that happens to + need the same schema lookup." [references aliases alias] (when-let [{:keys [table schema]} (get aliases alias)] (let [columns (if schema diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index ea0618a..9fe0286 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -504,7 +504,39 @@ :params nil} (generate-expressions ["company |= c1" "company |= c2" - "c2 | c1"]))))) + "c2 | c1"])))) + + (testing "Join-key preservation is a distinct mechanism from update-tracking auto-id" + ;; A sealed CTE (variable or checkpoint) needs its own id column to remain + ;; joinable, since the outer query can no longer see its underlying real + ;; tables once sealed — a raw table join never needs this (a JOIN ... ON + ;; clause sees the full real table regardless of what's selected). This is + ;; unrelated to select/create-auto-id-column, which exists so the UI can + ;; identify which row to update (see result-updates.md), so the column + ;; preserving the join key must never be marked :auto-id. + (let [state (-> "company as c | s: name |= x" + parser/parse :result + (ast/generate :test)) + join-key (->> (get-in state [:pending-assignments "x" :columns]) + (filter #(= "id" (:column %))) + first)] + (is (some? join-key)) + (is (true? (:hidden join-key))) + (is (true? (:join-key join-key))) + (is (not (:auto-id join-key)) + "must not be marked :auto-id — that's a different mechanism (update-row identification)")) + + ;; End to end: joining through a plain (non-GROUP) explicit select still works, + ;; now backed by an actual id in the CTE rather than an assumption. + (is (clojure.string/includes? + (:query (generate-expressions ["company as c | s: name |= x" "x | employee"])) + "\"x\".\"id\" = \"e_1\".\"company_id\"")) + + ;; GROUP still can't get a join key patched in silently — grouping by a + ;; non-id column genuinely has no id to preserve. + (is (clojure.string/includes? + (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= x" "x | employee"])) + "ON \"\" = \"\"")))) (deftest test-checkpoints (testing "LIMIT checkpoint: auto-CTE when a table op follows limit" From f184c99dd4c30265f190ec9e035554cb5ff0ea6a Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 19:50:11 +0200 Subject: [PATCH 56/60] docs: describe join-key preservation as its own mechanism, not auto-id reuse Updates "Join resolution through variables" for the decoupling refactor: attributes join-key survival to assign/preserve-join-keys and assign/snapshot instead of select/add-auto-id-columns, and leads with the actual SQL reason a sealed CTE needs this while a raw table join doesn't (JOIN ... ON sees the real table regardless of SELECT; a CTE's :columns is its entire visible schema once sealed) rather than a post-handle timing detail. --- docs/variables.md | 41 ++++++++++++++++++++++++----------------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/docs/variables.md b/docs/variables.md index 4c677ab..3ecce66 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -142,27 +142,34 @@ query's params list. All three passes below build on one question, answered by the shared helper `get-source-tables`: **which real tables can this variable actually be joined to?** A table only counts as a source if its -own `id` column is present among the CTE's *actual* output columns — auto-id included, not just what was -explicitly typed. - -That distinction matters because of when a variable's snapshot is taken. `post-handle` normally adds a -hidden auto-id column for every real table in the pipeline, regardless of what was explicitly selected — -that's the reason an ordinary, variable-free `s: name | employee` doesn't need explicit `id` to join -correctly. But a `|=` or checkpoint snapshot is taken mid-`handle-ops`, before `post-handle` ever runs, so -without deliberately re-adding it, no snapshot would ever carry an id column at all. `assign/handle` and -`flush-checkpoint`'s auto-named branch both now call `select/add-auto-id-columns` on the snapshot -themselves — guarded by the same `should-add-auto-ids?` check `post-handle` already uses, which excludes -`:group` (along with `:count`/`:delete-action`/`:update-action`). That one guard is what makes -`get-source-tables` behave differently depending on how the CTE's columns came to be: +own `id` column is present among the CTE's *actual* output columns — including hidden, internally-added +ones, not just what was explicitly typed. + +**Why a raw table join doesn't need this, but a sealed variable does**: `t | c` (no variable involved) +compiles to `... JOIN "c" ON "t"."id" = "c"."tenantId"` — a `JOIN ... ON` clause can reference any column +of a real table in `FROM`/`JOIN`, regardless of what's in `SELECT`. A CTE is different in kind, not degree: +once state is sealed into `WITH "x" AS ( SELECT ... )`, the outer query can no longer see `x`'s underlying +real tables at all — only whatever `x`'s own `:columns` produced. So a table stays a valid join source +*through a variable* only if its id actually survived into that snapshot. + +This has nothing to do with `select/add-auto-id-columns` (which exists purely so the UI can identify which +row to update — see [result-updates.md](result-updates.md) — an unrelated concern that happens to also add +an id column, for different reasons, at a different time). It's handled by its own mechanism instead, +`assign/preserve-join-keys`, called from `assign/snapshot` — the one shared entry point for sealing state +into a CTE, used identically by a `|=` assignment (`assign/handle`) and a checkpoint's auto-named seal +(`flush-checkpoint` in `ast/main.clj`, the other place this happens). Its columns are marked `:hidden` +(excluded from column hints, same treatment as auto-id columns) but deliberately never `:auto-id`, so nothing +downstream can conflate "has a join key" with "is tracked for row updates." `preserve-join-keys` is a no-op +for `:group` — an unaggregated id column can't be silently added to a `GROUP BY` without changing what's +grouped by — which is what makes `get-source-tables` behave differently depending on how the CTE's columns +came to be: - **No explicit columns** (`*`, e.g. a bare `where:`/`limit:` with no `s:`/`g:`): the CTE implicitly selects everything, which always includes `id` regardless of operation type. The variable's `:current` table is the sole, always-safe source. -- **Non-`GROUP` explicit columns** (`s:`): auto-id gets added for every real table in `:tables`, so every - table referenced by an explicit column stays a valid source — the same result as before auto-id was - reintroduced here, just backed by an id that's actually present now instead of assumed. -- **`GROUP`'s grouped columns**: no auto-id is ever added — an unaggregated id column can't be silently - added to a `GROUP BY` without changing what's grouped by. A table is a source here *only* if the user +- **Non-`GROUP` explicit columns** (`s:`): `preserve-join-keys` adds a join-key column for every real table + in `:tables`, so every table referenced by an explicit column stays a valid source. +- **`GROUP`'s grouped columns**: no join-key is ever added — a table is a source here *only* if the user explicitly grouped by that table's `id`. `group: name` alone therefore resolves to **zero** sources (nothing joinable, no join hints should appear); `group: id, name` resolves to one; `group: t.id, c.id` can resolve to more than one — the same multi-table join support a plain, variable-free pipeline already From db9cb8dabf7cb825f9246669967197031d995988 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 20:25:46 +0200 Subject: [PATCH 57/60] fix: reuse add-auto-id-columns directly for variable join sources MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Simplifies the previous commit's decoupled join-key mechanism away. The actual rule is simpler: a table is a join source through a sealed variable if its id is present in the CTE's :columns, however it got there — Pine never adds one on its own for that purpose. get-source-tables only ever checks; it doesn't need to know or care which mechanism produced the id. For that check to find anything on a non-GROUP variable, the id needs to already be in :columns by the time the snapshot is taken — the same auto-id columns an ordinary query's post-handle would add work fine here, since the only real requirement is that they run before the snapshot is read for join resolution, not that they come from a separate mechanism. assign/handle and flush-checkpoint's auto-named branch now call select/add-auto-id-columns directly when building the snapshot, in that order, for exactly that reason. GROUP still ends up with no id unless the user explicitly grouped by one — not through any special-casing in get-source-tables, but because should-add-auto-ids? (select.clj) already excludes :group from getting an auto-id column, for its own reasons. get-source-tables doesn't need to know that either; it stays a plain, mechanism-agnostic check. Net removal: assign/preserve-join-keys, create-join-key-column, has-explicit-id?, the :join-key marker, and select/has-id-column?'s public export are all gone — nothing outside select.clj needs that lookup now. --- src/pine/ast/assign.clj | 82 ++++++----------------------------------- src/pine/ast/main.clj | 68 +++++++++++++--------------------- src/pine/ast/select.clj | 7 +--- test/pine/eval_test.clj | 35 ++++++------------ 4 files changed, 49 insertions(+), 143 deletions(-) diff --git a/src/pine/ast/assign.clj b/src/pine/ast/assign.clj index 4d0eef2..2a6cbc0 100644 --- a/src/pine/ast/assign.clj +++ b/src/pine/ast/assign.clj @@ -12,83 +12,23 @@ the pending-assignments map from accumulating nested copies across multiple assignments in a single expression. - Why join-key preservation here, and why it's NOT select/add-auto-id-columns: - once a snapshot is sealed into a CTE, the outer query can no longer see its - underlying real tables directly — only whatever the CTE's own :columns select. - A raw table join never needs an id column in SELECT (a JOIN ... ON clause can - reference any column of a real table regardless of what's selected), but a CTE - is different in kind: its column list IS its entire visible schema. So a table - is only actually joinable through a sealed variable if its own id survived into - that snapshot's :columns. - - This is unrelated to select/add-auto-id-columns, which exists purely so the UI - can identify which row to update (see result-updates.md) — a different concern - that happens to also add an id column, for different reasons, at a different - time (post-handle, on the final returned state, not on a mid-fold snapshot). - Coupling the two would mean a future change to update-tracking (e.g. making it - opt-out) could silently break variable joins for a completely unrelated reason. - preserve-join-keys is deliberately its own mechanism: its columns are marked - :hidden (excluded from column hints, same as auto-id) but NOT :auto-id, so nothing - downstream can conflate 'has a join key' with 'is tracked for updates'. - - GROUP is excluded (see preserve-join-keys): there's no way to silently add an - unaggregated id column to a GROUP BY without changing what's grouped by. See - get-source-tables in ast/main.clj, which is what actually enforces that a - GROUP-sourced variable needs its id explicitly grouped by to be joinable." + Ordering dependency: the snapshot must have any auto-id columns added before + it's stored, not after. get-source-tables (ast/main.clj) decides which tables + a variable can join to by checking whether a table's own id column is present + in the snapshot's :columns — it doesn't add one itself, it only ever looks. So + whatever ends up in :columns by the time this snapshot is taken is final." (:require [pine.ast.select :as select])) -(defn- create-join-key-column - "A hidden column that exists purely so a sealed CTE can still be joined via id. - Deliberately not shaped like select/create-auto-id-column's output (no - :column-alias, not marked :auto-id) — this is a different mechanism for a - different purpose; see the namespace docstring." - [alias operation-index] - {:column "id" :alias alias :hidden true :join-key true :operation-index operation-index}) - -(defn- has-explicit-id? - [columns alias] - (some #(and (= alias (:alias %)) (= "id" (:column %))) columns)) - -(defn preserve-join-keys - "Ensure a state that's about to be sealed into a CTE keeps enough of an id - column, per real table, to remain joinable once sealing hides its underlying - tables from the outer query. No-op when :columns is empty (an implicit '*' - already includes id, regardless of operation type) or when this is a GROUP - (aggregation means an id can only survive if the user explicitly grouped by - it — silently adding one here would change what's being grouped by)." - [state] - (let [explicit (remove :hidden (:columns state))] - (if (or (seq (:group state)) (empty? explicit)) - state - (let [table-aliases (map :alias (:tables state)) - next-operation-index (inc (:index state)) - references (:references state) - aliases (:aliases state) - valid-aliases (filter #(and (not (:ast (get aliases %))) - (not (has-explicit-id? (:columns state) %)) - (select/has-id-column? references aliases %)) - table-aliases) - join-key-columns (map-indexed #(create-join-key-column %2 (+ next-operation-index %1)) - valid-aliases)] - (update state :columns into join-key-columns))))) - -(defn snapshot - "Prepare state to be stored as a CTE-backing snapshot — used both for a |= - assignment (below) and for a checkpoint's auto-named seal (flush-checkpoint - in ast/main.clj), the two places a piece of pipeline state gets frozen for - later reference as a CTE. Drops :pending-assignments (preventing nested - accumulation across multiple assignments in one expression) and preserves - join keys (see preserve-join-keys)." - [state] - (-> state (dissoc :pending-assignments) preserve-join-keys)) - (defn handle - "Snapshot the current state under varname in :pending-assignments (see - snapshot above for what that entails). + "Snapshot the current state under varname in :pending-assignments, with + auto-id columns added (select/add-auto-id-columns) the same way they'd be + added to any other query's final result. The snapshot excludes + :pending-assignments itself to prevent nesting. Also registers varname as a local alias for the current SQL alias so subsequent ops in the same expression can write e.g. x.id to mean c_0.id." [state varname] (-> state - (assoc-in [:pending-assignments varname] (snapshot state)) + (assoc-in [:pending-assignments varname] + (-> state (dissoc :pending-assignments) select/add-auto-id-columns)) (assoc :assign varname))) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 0da6912..3775345 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -96,17 +96,12 @@ "Return the column list a variable's CTE actually exposes, for hint generation. Returns nil when the CTE selects *, meaning the source table's columns apply. - Excludes :hidden columns — both select/create-auto-id-column's update-tracking - entries and assign/preserve-join-keys' join-key entries are internal, never - meant to appear as a selectable column in hints. - For a GROUP-sourced CTE, :columns already includes the aggregate entry — group.clj folds it in directly (parser.clj defaults :functions to [\"count\"] even when `=> count` is omitted; there's no way to write a truly aggregate- - less GROUP) — so it's read here like any other column, never added - separately. Appending it again on top, as this used to do, double-counted." + less GROUP) — so it's read here like any other column." [var-ast] - (let [user-cols (remove :hidden (:columns var-ast))] + (let [user-cols (remove :auto-id (:columns var-ast))] (when (seq user-cols) (->> user-cols (map #(or (:column-alias %) (:column %))) @@ -114,42 +109,29 @@ (mapv #(hash-map :column %)))))) (defn- get-source-tables - "Return the tables that are valid join sources for a variable's CTE — the - tables whose own id column is present among the CTE's actual output columns - (:columns, join-key entries included). - - assign/preserve-join-keys adds those join-key columns to a snapshot whenever - it's sealed into a CTE (via assign/snapshot, used by both a |= assignment and - a checkpoint's auto-named seal), for every case except :group — GROUP can't - silently add an unaggregated id column to a GROUP BY without changing what - the aggregation groups by. See assign.clj's namespace docstring for why this - is deliberately a separate mechanism from select/add-auto-id-columns (which - exists for update-row identification, an unrelated concern). In practice: - - - Non-GROUP CTEs always end up with a join key for every real table in - :tables, so every table referenced by an explicit column stays a valid - source — same as before either mechanism existed, now backed by an id - that's actually present in the CTE. - - GROUP CTEs only get an id for a table if the user explicitly grouped by - that table's id. Grouping by anything else (e.g. `group: name`) means - that table's id doesn't survive the aggregation, and it correctly stops - being a valid source — joining to it would otherwise reference a column - (e.g. \"x\".\"id\") the CTE doesn't have. - - No explicit columns at all means the CTE implicitly selects '*', which - always includes id regardless of operation type — the :current table is - then the sole, always-safe source. - - Can return zero tables (e.g. `group: name` alone), one, or more than one - (e.g. `s: t.id, c.id`, or any non-GROUP CTE spanning multiple tables) — the - same multi-table join support a plain, variable-free pipeline already has - when it references more than one table. + "Return the tables that are valid join sources for a variable's CTE. + + Once a table's data is sealed into a CTE, the outer query can only see what + that CTE's own :columns actually selected — nothing else about the underlying + table is reachable. A table is therefore only a join source if its own id + column is among those columns. Pine never adds one on its own: an id is + either already there (the user selected it, or it arrived some other way — + e.g. auto-id, for tables where that runs) or it isn't, in which case that + table simply isn't joinable through this variable. + + No explicit columns at all means the CTE selects '*', which includes every + column of the source table, id included — the :current table is then the + sole source. With explicit columns, each one is checked for its own id column + independently, so this can return zero tables (`group: name` alone has no id + anywhere), one, or more than one (`s: t.id, c.id` makes both t and c valid + sources) — the same multi-table join support a plain, variable-free pipeline + already has when it references more than one table. Used by both seeding and bidirectional patching." [var-ast] (let [columns (:columns var-ast) aliases (:aliases var-ast) - explicit (remove :hidden columns)] + explicit (remove :auto-id columns)] (or (if (empty? explicit) (when-let [current-alias (:current var-ast)] [(get-in aliases [current-alias :table])]) @@ -354,14 +336,14 @@ (and (:needs-assign checkpoint) (= op-type :assign)) (assoc state :pending-checkpoint {:name (:value op) :needs-table true}) - ;; Auto-named: fire when a table or another checkpoint op arrives. - ;; assign/snapshot mirrors what |= does for its own snapshot: this is the - ;; other place a piece of state gets sealed into a CTE, so it needs the - ;; same join-key preservation (see assign.clj's namespace docstring). + ;; Auto-named: fire when a table or another checkpoint op arrives. The + ;; snapshot needs auto-id added before seal-as-cte below, since that's what + ;; decides (via get-source-tables) which tables it can join to — see + ;; assign.clj's namespace docstring for why the ordering matters. (and (:needs-assign checkpoint) fire?) (let [n (:auto-cte-count state) cname (str "__pine_" n "__") - snapshot (assign/snapshot state)] + snapshot (-> state (dissoc :pending-assignments) select/add-auto-id-columns)] (-> state (update :auto-cte-count inc) (assoc :pending-checkpoint nil) diff --git a/src/pine/ast/select.clj b/src/pine/ast/select.clj index 380ee2e..d8bc045 100644 --- a/src/pine/ast/select.clj +++ b/src/pine/ast/select.clj @@ -23,11 +23,8 @@ (-> state (update :columns into columns)))) -(defn has-id-column? - "Check if a table has an 'id' column by looking up the table info in references. - Public: shared with assign.clj's join-key preservation, a different concern (join - resolution through a sealed CTE, not update-row identification) that happens to - need the same schema lookup." +(defn- has-id-column? + "Check if a table has an 'id' column by looking up the table info in references" [references aliases alias] (when-let [{:keys [table schema]} (get aliases alias)] (let [columns (if schema diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 9fe0286..93845e0 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -506,34 +506,21 @@ "company |= c2" "c2 | c1"])))) - (testing "Join-key preservation is a distinct mechanism from update-tracking auto-id" - ;; A sealed CTE (variable or checkpoint) needs its own id column to remain - ;; joinable, since the outer query can no longer see its underlying real - ;; tables once sealed — a raw table join never needs this (a JOIN ... ON - ;; clause sees the full real table regardless of what's selected). This is - ;; unrelated to select/create-auto-id-column, which exists so the UI can - ;; identify which row to update (see result-updates.md), so the column - ;; preserving the join key must never be marked :auto-id. - (let [state (-> "company as c | s: name |= x" - parser/parse :result - (ast/generate :test)) - join-key (->> (get-in state [:pending-assignments "x" :columns]) - (filter #(= "id" (:column %))) - first)] - (is (some? join-key)) - (is (true? (:hidden join-key))) - (is (true? (:join-key join-key))) - (is (not (:auto-id join-key)) - "must not be marked :auto-id — that's a different mechanism (update-row identification)")) - - ;; End to end: joining through a plain (non-GROUP) explicit select still works, - ;; now backed by an actual id in the CTE rather than an assumption. + (testing "A table is a join source through a variable only if its id is present" + ;; company's id is present in x's snapshot here via auto-id, even though + ;; the user only wrote `s: name` — so the join resolves. (is (clojure.string/includes? (:query (generate-expressions ["company as c | s: name |= x" "x | employee"])) "\"x\".\"id\" = \"e_1\".\"company_id\"")) - ;; GROUP still can't get a join key patched in silently — grouping by a - ;; non-id column genuinely has no id to preserve. + ;; Same result when id is explicit instead of auto-added — get-source-tables + ;; doesn't care which mechanism put it there, only that it's present. + (is (clojure.string/includes? + (:query (generate-expressions ["company as c | s: id, name |= x" "x | employee"])) + "\"x\".\"id\" = \"e_1\".\"company_id\"")) + + ;; GROUP never gets an id added automatically — grouping by a non-id column + ;; genuinely has no id anywhere, so company is not a valid join source. (is (clojure.string/includes? (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= x" "x | employee"])) "ON \"\" = \"\"")))) From 410e0d90067bc8e4af80523a4274982bdd3192cb Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Tue, 21 Jul 2026 20:26:37 +0200 Subject: [PATCH 58/60] docs: describe join-source resolution as reusing add-auto-id-columns directly Updates "Join resolution through variables" for the simplified design: get-source-tables is a plain, mechanism-agnostic check (is a table's id present in :columns, however it got there), and select/add-auto-id-columns is called directly at snapshot time rather than through a separate mechanism. GROUP ends up different only because should-add-auto-ids? already excludes it, not because get-source-tables special-cases it. --- docs/variables.md | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/docs/variables.md b/docs/variables.md index 3ecce66..def30d6 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -141,39 +141,34 @@ query's params list. ### Join resolution through variables All three passes below build on one question, answered by the shared helper `get-source-tables`: -**which real tables can this variable actually be joined to?** A table only counts as a source if its -own `id` column is present among the CTE's *actual* output columns — including hidden, internally-added -ones, not just what was explicitly typed. +**which real tables can this variable actually be joined to?** A table counts as a source only if its +own `id` column is present among the CTE's *actual* output columns — however it got there. Pine never +adds one for this purpose; `get-source-tables` only ever looks. **Why a raw table join doesn't need this, but a sealed variable does**: `t | c` (no variable involved) compiles to `... JOIN "c" ON "t"."id" = "c"."tenantId"` — a `JOIN ... ON` clause can reference any column of a real table in `FROM`/`JOIN`, regardless of what's in `SELECT`. A CTE is different in kind, not degree: once state is sealed into `WITH "x" AS ( SELECT ... )`, the outer query can no longer see `x`'s underlying real tables at all — only whatever `x`'s own `:columns` produced. So a table stays a valid join source -*through a variable* only if its id actually survived into that snapshot. - -This has nothing to do with `select/add-auto-id-columns` (which exists purely so the UI can identify which -row to update — see [result-updates.md](result-updates.md) — an unrelated concern that happens to also add -an id column, for different reasons, at a different time). It's handled by its own mechanism instead, -`assign/preserve-join-keys`, called from `assign/snapshot` — the one shared entry point for sealing state -into a CTE, used identically by a `|=` assignment (`assign/handle`) and a checkpoint's auto-named seal -(`flush-checkpoint` in `ast/main.clj`, the other place this happens). Its columns are marked `:hidden` -(excluded from column hints, same treatment as auto-id columns) but deliberately never `:auto-id`, so nothing -downstream can conflate "has a join key" with "is tracked for row updates." `preserve-join-keys` is a no-op -for `:group` — an unaggregated id column can't be silently added to a `GROUP BY` without changing what's -grouped by — which is what makes `get-source-tables` behave differently depending on how the CTE's columns -came to be: +*through a variable* only if its id actually survived into that snapshot — either because the user +selected it explicitly, or because it arrived some other way (see below). + +That "some other way" is `select/add-auto-id-columns` — the same hidden-id mechanism an ordinary query's +final result already gets, added here to the snapshot at the point it's taken (`assign/handle`, and +`flush-checkpoint`'s auto-named branch for a checkpoint seal) rather than left to run later. It excludes +`:group` (`should-add-auto-ids?` in `select.clj`) for its own reason — an unaggregated id column can't be +added to a `GROUP BY` without changing what's grouped by — and that's what makes a `GROUP`-sourced CTE +behave differently here, without `get-source-tables` itself needing to know anything about operation types: - **No explicit columns** (`*`, e.g. a bare `where:`/`limit:` with no `s:`/`g:`): the CTE implicitly - selects everything, which always includes `id` regardless of operation type. The variable's `:current` - table is the sole, always-safe source. -- **Non-`GROUP` explicit columns** (`s:`): `preserve-join-keys` adds a join-key column for every real table - in `:tables`, so every table referenced by an explicit column stays a valid source. -- **`GROUP`'s grouped columns**: no join-key is ever added — a table is a source here *only* if the user - explicitly grouped by that table's `id`. `group: name` alone therefore resolves to **zero** sources - (nothing joinable, no join hints should appear); `group: id, name` resolves to one; `group: t.id, c.id` - can resolve to more than one — the same multi-table join support a plain, variable-free pipeline already - has when it references more than one table. + selects everything, which always includes `id`. The variable's `:current` table is the sole source. +- **Non-`GROUP` explicit columns** (`s:`): every real table in `:tables` ends up with an id, so every + table referenced by an explicit column stays a valid source. +- **`GROUP`'s grouped columns**: a table is a source here only if the user explicitly grouped by that + table's `id`. `group: name` alone therefore resolves to **zero** sources (nothing joinable, no join + hints should appear); `group: id, name` resolves to one; `group: t.id, c.id` can resolve to more than + one — the same multi-table join support a plain, variable-free pipeline already has when it references + more than one table. With that settled, the three passes propagate joins for whichever source tables `get-source-tables` returned, inside `pre-handle` (`ast/main.clj`): From e80cb6a2e75a84276224bca83551e39e716e4d3d Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Wed, 22 Jul 2026 00:43:57 +0200 Subject: [PATCH 59/60] fix: stop adding auto-id when sealing a variable/checkpoint into a CTE get-source-tables only ever checks whether a table's id is present in a CTE's :columns; it never needed anything to actively put one there. The add-auto-id-columns call at snapshot time was Pine still doing that on the user's behalf for non-GROUP cases. Simpler and now the actual rule: if the user explicitly selected a table's id, that table is a valid join source through the variable; if not, it isn't. No exception for non-GROUP operations. assign/handle and flush-checkpoint's auto-named branch are back to a plain dissoc'd snapshot. Updates the two test cases this flips: `s: name |= x` no longer shows the sibling table as joinable (no id, no join); `s: id, name |= x` still does. --- src/pine/ast/assign.clj | 19 ++++--------------- src/pine/ast/main.clj | 14 +++++--------- test/pine/eval_test.clj | 13 ++++++------- test/pine/hints_test.clj | 22 ++++++++++------------ 4 files changed, 25 insertions(+), 43 deletions(-) diff --git a/src/pine/ast/assign.clj b/src/pine/ast/assign.clj index 2a6cbc0..97c5c1e 100644 --- a/src/pine/ast/assign.clj +++ b/src/pine/ast/assign.clj @@ -10,25 +10,14 @@ Why a snapshot: storing a dissoc'd copy of state (not the live state) prevents the pending-assignments map from accumulating nested copies across multiple - assignments in a single expression. - - Ordering dependency: the snapshot must have any auto-id columns added before - it's stored, not after. get-source-tables (ast/main.clj) decides which tables - a variable can join to by checking whether a table's own id column is present - in the snapshot's :columns — it doesn't add one itself, it only ever looks. So - whatever ends up in :columns by the time this snapshot is taken is final." - (:require - [pine.ast.select :as select])) + assignments in a single expression.") (defn handle - "Snapshot the current state under varname in :pending-assignments, with - auto-id columns added (select/add-auto-id-columns) the same way they'd be - added to any other query's final result. The snapshot excludes - :pending-assignments itself to prevent nesting. + "Snapshot the current state under varname in :pending-assignments. + The snapshot excludes :pending-assignments itself to prevent nesting. Also registers varname as a local alias for the current SQL alias so subsequent ops in the same expression can write e.g. x.id to mean c_0.id." [state varname] (-> state - (assoc-in [:pending-assignments varname] - (-> state (dissoc :pending-assignments) select/add-auto-id-columns)) + (assoc-in [:pending-assignments varname] (dissoc state :pending-assignments)) (assoc :assign varname))) diff --git a/src/pine/ast/main.clj b/src/pine/ast/main.clj index 3775345..dbc30a6 100644 --- a/src/pine/ast/main.clj +++ b/src/pine/ast/main.clj @@ -114,10 +114,9 @@ Once a table's data is sealed into a CTE, the outer query can only see what that CTE's own :columns actually selected — nothing else about the underlying table is reachable. A table is therefore only a join source if its own id - column is among those columns. Pine never adds one on its own: an id is - either already there (the user selected it, or it arrived some other way — - e.g. auto-id, for tables where that runs) or it isn't, in which case that - table simply isn't joinable through this variable. + column is among those columns. Pine never adds one on its own: if the user + didn't select it, it isn't there, and that table simply isn't joinable + through this variable. No explicit columns at all means the CTE selects '*', which includes every column of the source table, id included — the :current table is then the @@ -336,14 +335,11 @@ (and (:needs-assign checkpoint) (= op-type :assign)) (assoc state :pending-checkpoint {:name (:value op) :needs-table true}) - ;; Auto-named: fire when a table or another checkpoint op arrives. The - ;; snapshot needs auto-id added before seal-as-cte below, since that's what - ;; decides (via get-source-tables) which tables it can join to — see - ;; assign.clj's namespace docstring for why the ordering matters. + ;; Auto-named: fire when a table or another checkpoint op arrives (and (:needs-assign checkpoint) fire?) (let [n (:auto-cte-count state) cname (str "__pine_" n "__") - snapshot (-> state (dissoc :pending-assignments) select/add-auto-id-columns)] + snapshot (dissoc state :pending-assignments)] (-> state (update :auto-cte-count inc) (assoc :pending-checkpoint nil) diff --git a/test/pine/eval_test.clj b/test/pine/eval_test.clj index 93845e0..163c0e9 100644 --- a/test/pine/eval_test.clj +++ b/test/pine/eval_test.clj @@ -507,20 +507,19 @@ "c2 | c1"])))) (testing "A table is a join source through a variable only if its id is present" - ;; company's id is present in x's snapshot here via auto-id, even though - ;; the user only wrote `s: name` — so the join resolves. + ;; `s: name` alone has no id anywhere in x's snapshot — Pine doesn't add + ;; one, so company is not a valid join source. (is (clojure.string/includes? (:query (generate-expressions ["company as c | s: name |= x" "x | employee"])) - "\"x\".\"id\" = \"e_1\".\"company_id\"")) + "ON \"\" = \"\"")) - ;; Same result when id is explicit instead of auto-added — get-source-tables - ;; doesn't care which mechanism put it there, only that it's present. + ;; `s: id, name` includes it explicitly, so the join resolves. (is (clojure.string/includes? (:query (generate-expressions ["company as c | s: id, name |= x" "x | employee"])) "\"x\".\"id\" = \"e_1\".\"company_id\"")) - ;; GROUP never gets an id added automatically — grouping by a non-id column - ;; genuinely has no id anywhere, so company is not a valid join source. + ;; Same for GROUP: grouping by a non-id column has no id anywhere, so + ;; company is not a valid join source. (is (clojure.string/includes? (:query (generate-expressions ["company as c | employee .company_id | group: c.name |= x" "x | employee"])) "ON \"\" = \"\"")))) diff --git a/test/pine/hints_test.clj b/test/pine/hints_test.clj index 7ba79e5..abbd177 100644 --- a/test/pine/hints_test.clj +++ b/test/pine/hints_test.clj @@ -246,24 +246,22 @@ :table)))) (testing "A table is only a valid join source through a variable if its own id survives" - ;; x explicitly selects only name, but a plain (non-GROUP) select gets an auto-id - ;; column added to the snapshot the same way an ordinary pipeline would (see - ;; assign/handle) — company's id is present in x's actual CTE output either way, - ;; so employee is correctly still suggested. - (is (= ["employee"] - (->> (gen-with-variables ["company as c | s: name |= x" "x | emp"]) - :table - (map :table)))) + ;; x explicitly selects only name — Pine doesn't add an id on its own, so + ;; company's id is nowhere in x's actual CTE output. employee must NOT be + ;; suggested. + (is (= [] + (-> (gen-with-variables ["company as c | s: name |= x" "x | emp"]) + :table))) - ;; Same table, id also explicitly selected — same result, redundantly safe. + ;; Same table, but id is explicitly selected this time — now it's present, + ;; and employee is correctly suggested. (is (= ["employee"] (->> (gen-with-variables ["company as c | s: id, name |= x" "x | emp"]) :table (map :table)))) - ;; GROUP is the one operation that can't get an auto-id: an unaggregated id column - ;; can't be silently added to a GROUP BY without changing what's grouped by. Grouping - ;; by a non-id column genuinely loses company's id — employee must NOT be suggested. + ;; Same rule applies to GROUP: grouping by a non-id column loses company's + ;; id just the same — employee must NOT be suggested. (is (= [] (-> (gen-with-variables ["company as c | employee .company_id | group: c.name |= x" "x | doc"]) :table))) From ac88cdad14f456a94c6a6dabf1fce37708cf29d4 Mon Sep 17 00:00:00 2001 From: Ahmad Nazir Raja Date: Wed, 22 Jul 2026 00:45:03 +0200 Subject: [PATCH 60/60] docs: describe join-source resolution as purely explicit, no auto-id Updates "Join resolution through variables" for the final simplification: a table is a join source only if the user explicitly selected its id, with no exception for non-GROUP operations. Pine adds nothing on its own for this purpose, for any operation type. --- docs/variables.md | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/docs/variables.md b/docs/variables.md index def30d6..9be1342 100644 --- a/docs/variables.md +++ b/docs/variables.md @@ -142,31 +142,22 @@ query's params list. All three passes below build on one question, answered by the shared helper `get-source-tables`: **which real tables can this variable actually be joined to?** A table counts as a source only if its -own `id` column is present among the CTE's *actual* output columns — however it got there. Pine never -adds one for this purpose; `get-source-tables` only ever looks. +own `id` column is explicitly present among the CTE's output columns. Pine never adds one on the user's +behalf, for any operation — if `id` wasn't selected, that table isn't joinable through this variable. **Why a raw table join doesn't need this, but a sealed variable does**: `t | c` (no variable involved) compiles to `... JOIN "c" ON "t"."id" = "c"."tenantId"` — a `JOIN ... ON` clause can reference any column of a real table in `FROM`/`JOIN`, regardless of what's in `SELECT`. A CTE is different in kind, not degree: once state is sealed into `WITH "x" AS ( SELECT ... )`, the outer query can no longer see `x`'s underlying real tables at all — only whatever `x`'s own `:columns` produced. So a table stays a valid join source -*through a variable* only if its id actually survived into that snapshot — either because the user -selected it explicitly, or because it arrived some other way (see below). - -That "some other way" is `select/add-auto-id-columns` — the same hidden-id mechanism an ordinary query's -final result already gets, added here to the snapshot at the point it's taken (`assign/handle`, and -`flush-checkpoint`'s auto-named branch for a checkpoint seal) rather than left to run later. It excludes -`:group` (`should-add-auto-ids?` in `select.clj`) for its own reason — an unaggregated id column can't be -added to a `GROUP BY` without changing what's grouped by — and that's what makes a `GROUP`-sourced CTE -behave differently here, without `get-source-tables` itself needing to know anything about operation types: +*through a variable* only if the user explicitly selected its `id`. - **No explicit columns** (`*`, e.g. a bare `where:`/`limit:` with no `s:`/`g:`): the CTE implicitly selects everything, which always includes `id`. The variable's `:current` table is the sole source. -- **Non-`GROUP` explicit columns** (`s:`): every real table in `:tables` ends up with an id, so every - table referenced by an explicit column stays a valid source. -- **`GROUP`'s grouped columns**: a table is a source here only if the user explicitly grouped by that - table's `id`. `group: name` alone therefore resolves to **zero** sources (nothing joinable, no join - hints should appear); `group: id, name` resolves to one; `group: t.id, c.id` can resolve to more than +- **Explicit columns** (`s:`, or `group:`'s grouped columns — both populate `:columns` the same way): + a table is a source only if `id` is literally among them. `s: name` or `group: name` alone therefore + resolves to **zero** sources (nothing joinable, no join hints should appear); `s: id, name` or + `group: id, name` resolves to one; `s: t.id, c.id` or `group: t.id, c.id` can resolve to more than one — the same multi-table join support a plain, variable-free pipeline already has when it references more than one table.