nextsteps.md

SQE — Next Steps

WANTED: object-level grant admin. Make a principal admin OF A TABLE, and let them update the grants on it without engine-wide admin.

The Snowflake / WITH GRANT OPTION shape: authority scoped to the object, not to a role that can grant anywhere. SQE has the opt-in grant_authority = "ranger-delegate" for this today, and it works by handing the decision to Ranger: the plugin grant endpoint authorizes the request's grantor field against delegateAdmin per resource AND per access type, so a grantor holding delegate admin for table-data-read is still refused when the request names table-data-write.

That mechanism does not survive Ranger 2.9.0. The per-resource grantor check exists ONLY on /service/plugins/services/{grant,revoke}/*, which Ranger declares security="none" and 2.9.0 stops serving unless ranger.admin.allow.unauthenticated.access is enabled. Measured: with the write and read paths moved to authenticated endpoints, 41 of 42 access-control cases pass on 2.9.0, and the single failure is a_delegated_owner_grants_on_their_own_table_without_an_admin_role, HTTP 400 "Unauthenticated access not allowed". Turning that property on would restore the feature by keeping an unauthenticated write endpoint open, which is the wrong trade for a feature about tightening authority.

So the choice is: keep Ranger as the authority and stay on 2.8.x, or move the delegateAdmin evaluation INTO SQE (read the policies, decide whether the caller holds delegate admin on that resource and those access types, then write through the authenticated policy API). Only the second works on 2.9.0+, and it means SQE owns a security decision Ranger used to own, including the measured fact that delegateAdmin does NOT cascade upward, so a delegated grantor needs the already-held-traversal skip.

ANSWERED, and the fork closed: Ranger has an authenticated twin of the grant endpoints. /service/plugins/secure/services/{grant,revoke}/{service} is covered by the catch-all isAuthenticated() rule rather than security="none", runs Ranger's own server-side merge, and STILL authorizes the named grantor per resource and per access type. So neither horn of the fork was necessary: Ranger keeps the authority, SQE does not reimplement delegateAdmin, and delegate mode works on 2.9.0.

Measured on 2.9.0 with admin REST credentials and a non-admin grantor: no credentials 401; grantor holding delegateAdmin for the access type 200; grantor holding none 403; grantor holding it for a DIFFERENT access type 403. Confirmed present on 2.8.0 as well, so one transport covers both. make test-access-control is 42 of 42 on 2.9.0 AND on 2.8.0, including a_delegated_owner_grants_on_their_own_table_without_an_admin_role.

Two traps found while probing, both now in the code comments. The denial discriminator is the HTTP STATUS: the body carries "statusCode":0 on success and on denial alike, so a body-based check would read a refused grant as a successful one. And grantorGroups is taken only from the request for a privileged caller, so group-based delegateAdmin would be ignored; that is not load-bearing here because this deployment materialises Keycloak groups as Ranger ROLES and role membership IS resolved server-side (verified: grantor delegated through role analyst, authorized with the field absent). A deployment delegating through real Ranger groups needs session groups threaded onto GrantStatement.

What remains for "grant admin on a table": GRANT ... WITH GRANT OPTION already maps to delegateAdmin: true on the object's policy item, so the mechanism is there. The open pieces are the already-held-traversal skip (delegateAdmin does not cascade upward, so a table-level delegate cannot write the catalog/namespace traversal policies a grant plan emits) and, if non-transferable ownership is wanted, gating the delegateAdmin flag in the coordinator, since Ranger lets a delegate pass grant-option onward for types they hold.

The 2.9.0 hold in renovate.json STAYS, for a different reason: Spark. SQE is 42 of 42 on 2.9.0, but Kyuubi's bundled Ranger plugin reads policies from the unauthenticated /service/plugins/policies/download/{service}, which 2.9.0 refuses, so the parity demo drops to 5 passed 2 failed. The plugin picks that URL from isKerberosEnabled(ugi) = !forceNonKerberos && UGI.isSecurityEnabled() && ugi.hasKerberosCredentials(), and forceNonKerberos can only turn secure mode OFF, so no configuration reaches the /secure/ twin without real Kerberos. The basic-auth credentials in ranger-spark-security.xml are sent on every call and buy nothing, because only the path matters.

FIXED 2026-08-16, issue #395: [catalog] require_vended_credentials (default false) stops Iceberg FileIO receiving the shared [storage] access/secret keys and sets s3.disable-config-load / s3.disable-ec2-metadata so env, profile, IRSA, and IMDSv2 cannot substitute. production_mode refuses to start unless every REST catalog has the flag on. Env: SQE_CATALOG__REQUIRE_VENDED_CREDENTIALS. Helm values-production.yaml sets it. Distributed ScanTask still ships the static key; that remains the s3vending follow-up.

Waiting for a newer Kyuubi does not help, and that is the trap worth recording: kyuubi-spark-authz-shaded_2.12 1.11.1 and 1.12.0 bundle byte-identical Ranger classes (RangerAdminRESTClient sha256 684c0eda..., both stamped 2025-02-14), because Kyuubi's pom pins ranger.version 2.6.0. The version to watch is ranger.version inside kyuubi-spark-authz's pom, not Kyuubi's own release number.

Two ways out were considered and neither was taken. Enabling ranger.admin.allow.unauthenticated.download.access (a property distinct from ...unauthenticated.access, and defaulting to it) would reopen anonymous READ of the whole policy set while keeping grant/revoke authenticated: narrower than 2.8.0, still a real concession. A rewriting proxy in front of Ranger, mapping /service/{x}/download/ to /service/{x}/secure/download/ and passing the plugin's existing basic auth through, relaxes nothing but puts a shim into a reference quickstart and would weaken the parity claim, since SQE needs no such shim. Decision 2026-08-14: stay on 2.8.0, which is measured at 43 of 43.

DOCUMENTED 2026-08-16, issue #412: Partitioned CTAS + sort-on-write still cannot spill. SortMemoryRule fails a single sort that cannot reserve merge headroom; it does not yet budget N partition merges. Workaround: omit ORDER BY when PARTITIONED BY already clusters. Engine-level bounded/spillable partition writers remain open.

DOCUMENTED 2026-08-16, issue #396: REVOKE SELECT is not a read gate. table-properties-read unlocks LOAD_TABLE; INSERT keeps that type, so a surviving writer still reads. Direction 3 for the release: document the implication graph, tell operators to use REVOKE ALL PRIVILEGES + CHECK ACCESS. A profile rewrite cannot split SELECT from INSERT without breaking writer LOAD_TABLE. See grant-revoke.md "Closing a gate" and limitations.md "Grant model gaps".

IN PROGRESS 2026-08-16, issue #426: Denied INSERT was already compare_write_denied in both engines. The missing cell is ADD COLUMN on a masked table through Spark as well as SQE. Probes added to scripts/access-control-parity-demo.sh; Spark ADD COLUMN is non-fatal so an uncalibrated probe cannot abort later sections. Re-calibrate on the Ranger + Spark stack.

FIXED 2026-08-16, issue #406 (first slice): legacy ShuffleReceiver::send_batch waits when resident bytes would exceed 64 MiB. Waiters enable() before the cap check so notify_waiters cannot be lost. DoExchange still prefers SpillablePartitionBuffer when spill is on. Full per-partition disk spill on the mpsc path remains open.

FIXED 2026-08-17, issue #407 (second slice): encoded Flight frames are charged, not just measured. The first slice bounded fetch and decode. What it left open is the copy nobody owned: AccountedEncodeStream releases a batch's Arrow permit as soon as the encoder is done with it, and the encoded IPC bytes tonic hands to h2 were written to flight_inflight_bytes with a bare set(n). A gauge, per frame, not even a sum, and nothing gated on it. One stream never notices. Fifty concurrent DoGet streams hold fifty encoded frames outside the DataFusion pool, which is the term that exhausted the 4 GB worker pool on SF10 TPC-DS inventory queries.

accounted_frame_stream charges each frame against a ByteBudget and holds the charge until the transport polls for the next one, the same "being polled again proves the last item was consumed" idiom AccountedEncodeStream uses for batches. Release happens before the next acquire, deliberately: holding two charges at once would deadlock a budget sized for a single frame. Backpressure then propagates the whole way back. Waiting on a frame permit stops polling the encoder, which stops draining the accounted batch stream, which stops the scan queue.

worker.memory.flight_budget already existed and was wired to nothing. It parsed, it resolved, and configured_need_bytes counted it in the startup headroom check, so every worker has been reserving RAM for a budget no code drew from. resolve_memory_budgets dropped it on the floor along with the accounting granularity. Both are forwarded now, and do_get charges unconditionally: an unconfigured worker gets a pool-derived default (a tenth of the pool, matching the config default) rather than an Option branch that silently reverts to the ungated path. While reading those defaults, four of the [worker.memory] doc comments turned out to disagree with the code they document (flight said 12.5% and gives 10%, scan said 25% and gives 20%, operator said 50% and gives 40%, shuffle said 12.5% and gives 20%). Corrected, because they are the numbers an operator sizes a worker from.

Two design calls worth recording. The budget takes no DataFusion pool reservation, because encoded IPC frames are not pool allocations and charging them there would fail queries under operator pressure; the capacity is already reserved at boot. And a frame wider than the whole budget ships UNCHARGED instead of failing: ByteBudget::acquire returns ItemTooLarge rather than hanging, and turning a query that works today into an error would be a regression. That second call is the one a test cannot catch by passing, so it was mutation-checked by making the case fatal and confirming the test goes red.

One thing found while reading the plan: the Phase 1 checklist item covering this work was already ticked. It says "retain the Arrow permit while encoding and charge encoded FlightData until gRPC releases/sends it", and only the first half had shipped. A half-done item reads exactly like a done one, which is why the audit had to find it a month later.

Still open on #407: adaptive decode concurrency from outbound queue depth (wait_queue_below is the coarse version and stays), and the #[ignore]d slow_consumer_caps_bytes_when_client_pauses gate, which needs a counting object store to assert GETs actually freeze. The DoExchange response path ships frames too and is not charged; that belongs with #406.

FIXED 2026-08-16, issue #407 (first slice): worker scan waits to acquire the fetch charge before decoding batches, and does not open the next parquet file until the outbound queue is under half the scan budget. Watermark wait is Notify-driven and exits if the Flight consumer drops. Full Flight-frame release + adaptive decode concurrency remain open.

FIXED 2026-08-16, issue #408 (first slice): query.max_concurrent_sorted_writes (default 2) rejects extra CTAS/INSERT with a top-level ORDER BY so several greedy-pool sorted writes cannot starve ExternalSorterMerge. Detection is from the parsed statement; the error is ResourceExhausted. Adaptive pool / DF#17334 remain open.

SPIKED 2026-08-16, issue #415: DF 54 PhysicalDynamicFilterNode serializes HashTableLookupExpr as lit(true). Membership cannot ride proto to workers. InList below runtime_filter_inlist_max_values already converts. Join-order (part on the probe side) is the remaining distributed SSB gap.

DOCUMENTED 2026-08-16, issue #405: Service level is restartable, not HA. One coordinator. Restart kills in-flight queries and live sessions. PDB minAvailable: 1 only blocks eviction. CREATE SECRET, ATTACH, query tracker, and session restore (tokens omitted) are process-local. Do not set coordinator.replicas above 1.

DOCUMENTED 2026-08-16, issue #411: HashJoin cannot spill (DF#17267). JoinStrategyRule rewrites to SMJ only on an exact over-threshold build estimate. Unknown Iceberg stats keep HashJoin. Rewriting on Unknown doubled TPC-DS at SF1.

CLOSED 2026-08-17, issue #387: the two dind integration jobs are gone; the suites are local gates now. integration-test and distributed-smoke both needed a privileged docker:dind sidecar the shared runners answer no route to host on, so neither ever executed a line of SQE code. Carrying them as allow_failure: true was worse than deleting them: a permanently-yellow job reads as coverage, and two real main breakages hid behind exactly that. The suites themselves are unchanged and now run via make test-integration / make test-distributed (FILTER= selects one test, make test-integration-down tears the stack down).

Two defects in the local path were fixed while making it the gate, both of which would have made the new target a worse experience than the script it wraps. scripts/integration-test.sh defaults RUST_MIN_STACK to 8 MiB to mirror production WORKER_STACK_BYTES, but an unfiltered run passes --ignored, which force-runs the write e2e suites, and those SIGABRT below 32 MiB; the make target exports 33554432, the value every other coordinator suite in the repo uses. And a fixed-port collision surfaced as Bind for 0.0.0.0:18181 failed: port is already allocated fourteen lines into the compose bring-up without naming the holder, which is the single most common local failure because the bench, parity and quickstart rigs publish overlapping ports. scripts/integration-preflight.sh now names the container and the docker rm -f that clears it.

The preflight accepts ports held by THIS project's own containers, so the intended fast rerun (stack left up, idempotent bootstrap) is not blocked, and rejects everything else. Matching is against docker compose ps output rather than a name prefix, which is load-bearing: the stale rig that exposed the bug was sqlengine-rand-010-polaris-1, sharing the sqlengine- prefix while belonging to a different project. Both branches verified against a live conflict.

First real run of the new gate, and it found three things. 215 passed, 7 failed. Four of the seven were impossible by construction: rewrite_data_files_distributed_parity (three cases) and compaction_distributed_benchmark need live sqe-worker processes on :50052 and :50053, and the DISTRIBUTED=0 skip list only named test_distributed_select because it was written before those tests existed. So a default run reported four failures no healthy machine could avoid, which is the same disease as a permanently-yellow CI job. They are in the skip list now.

Two of the remaining three were stale-schema dead weight, the same class as the CLOSED #377: incremental_scan_e2e ordered by timestamp_ms, a table_snapshots column #320 removed in favour of committed_at. #377 only covered the v3_e2e pair, so these two survived its fix. One word per site.

The last one is issue #431, filed rather than fixed. test_error_classification_live expects CATALOG_ERROR for a DELETE on a missing table and gets TABLE_NOT_FOUND. The expectation looks like the stale side (the same test expects TABLE_NOT_FOUND for SELECT on a missing table, and the more specific code is the more useful one), but that is an inference: git log -L on those lines reaches only the b9e2094 main-wipe revert, so which side moved is not recoverable cheaply. Changing an expected error classification as a drive-by is a behaviour decision, not test cleanup.

Measured after the fixes: 217 passed, 1 failed, the failure being #431 alone. That is what makes the target a usable gate: green except for one tracked red, so anything else red is yours.

Still standing, deliberately not touched: scenario-test, access-control-test and scenario-test-aws use the same dind sidecar and carry the same zero signal on the shared runners. They were kept because they are the only CI-side coverage of the quickstarts and the policy path, and because removing the access-control gate was not in scope. scenario-test's main-push rule has no allow_failure, so it remains a red-main hazard on a broken runner.

FIXED 2026-08-17, aikido CI findings #430 / #392 / #394: Aikido fingerprints an issue on (rule, FILE), never the line, so unpinned-image + .gitlab-ci.yml is ONE issue showing whichever image the scanner reaches first. #393 was rust:slim, then #430 was alpine:3.24; pinning alpine alone would have retitled #430 to docker:29, not closed it. All six image: lines are now digest-pinned (!863). docker:29-dind is out of scope because services: entries are never scanned. #392 was NOT a duplicated-pin problem in this repo: the scanner walked the root file before its includes, so $AIKIDO_TOOLS_IMAGE read as unresolvable even though guardrails.yml pins it to a digest; fixed centrally with a two-pass gather (aikido!129), no ref: bump needed. #394 (curl | bash for cargo-binstall) is real and lives in aikido's test-rust.yml; the pinned+checksummed install is in the same MR but clears here only after an aikido release and a ref: bump, which would also restore Harbor mirroring (v0.9.5 predates aikido!126).

FIXED 2026-08-16, issue #390: TPC-DS q17/q29/q85 no longer depend on RNG luck. q17 and q29 plant the three-leg (store, return, catalog) coincidence; q29 forces the return date. customer_demographics marital/education is sk-derived; web order 1 is planted at $125 in 2000 for q85.

DOCUMENTED 2026-08-16, issue #414: IcebergScanExec still advertises one output partition. Parallel I/O is inside the scan. Wiring target_partitions flipped joins to CollectLeft and regressed TPC-DS q72 5-6x. parallel_probe_scan stays opt-in (TPC-DS +26% at SF1). Default-on waits for a cost gate.

DOCUMENTED 2026-08-16, issue #417: TPC-BB Flight SF1 2026-08-15 is 63.1s, q01 35.5s (20 rows), 10/10. Do not quote a TPC-BB total without naming q01.

DOCUMENTED 2026-08-16, issue #428 leftovers: #405/#411 shipped as restartable-not-HA and HashJoin-cannot-spill. #396 documented. #406/#407 first slices landed. #390 planted in !857. #412/#415 remain documented/spiked engine work, not leftover tracking. Remaining #428 items stay on the GitLab issue.

FIXED 2026-08-17: workspace rand 0.8 -> 0.10 after the #390 plants. API rename only (random_range / RngExt). rusqlite stays 0.39. SF1 generate + compare-trino on the 0.10 draw: TPC-DS 99/99, 0 vacuous-bug (q17=3, q29=1, q79=100, q85=1). SSB 13/13, TPC-BB 10/10, TPC-C 8/8, TPC-E 11/11, ClickBench 41/43 (2 both-empty, 0 vacuous-bug). TPC-H 20/22 with 0 vacuous-bug (q21 Trino timeout, q22 both timeout). Bank not compared (RustFS dropped mid first pass).

FIXED 2026-08-12, suite hygiene: make test-access-control leaked its own grants between runs, so two denial-baseline tests failed on any stack the suite had already used.

denied_before_any_grant and all_tables_in_schema_grant_covers_the_namespace both time out after 120 s with still allowed for alice with 3 rows. The cause is in Ranger, not in the assertion: a policy named grant-1786370165684 grants role analyst sixteen access types on sales_wh.ac.orders, and alice is in analyst.

ac_setup calls ranger.bootstrap(), which clears the sqe-ac-e2e- prefixed slate. SQE's own GRANT statement does not use that prefix: it writes grant-<epoch_ms>. So every grant the suite makes through SQL outlives the run that made it, and the next run starts pre-authorized. A test whose whole job is to prove access is denied BEFORE a grant cannot survive that.

Proven rather than argued: deleting the leaked grant-* policies and re-running the three tests turns all of them green, with no code change. It leaks on EVERY run, not just historically. Of the three deleted, two survived from 2026-08-10 15:56 and one (grant-1786440718652) was written by the run executed minutes earlier the same day. bootstrap-ranger.sh seeds only wildcard admin/baseline policies and never a namespace-ac grant, so these can only be test residue.

Fixed by having bootstrap() delete every coarse-gate (polaris) policy scoped inside the suite's own namespaces, in delete_suite_grants. Scoped by RESOURCE, not by a second name prefix: adding grant- would repeat the original mistake the first time Ranger changes that format or merges a grant into a policy someone else named. The resource coordinate is what makes a policy the suite's.

Measured on the live stack, before and after one run:

	before	after
polaris policies scoped to namespace ac	9	0
bootstrap catalog-wildcard grants	3	3
parity-demo acparity policies	6	6

The nine included grant-1786441368465 on sales_wh.ac.orders carrying roles=['analyst'], created 2026-08-11 09:42 and still alive through a run on 2026-08-12 09:04. Alice is in analyst, which is exactly why she read three rows in a test whose job was to prove she could not.

Live verdict: 41 passed, 0 failed (scripts/access-control-test.sh, 612 s). denied_before_any_grant and revoke_disables_access both pass; the same run failed both beforehand. Nine unit tests pin the scoping predicate's boundaries (wildcards, catalog-level, foreign catalogs, acparity, multi-valued resources, malformed JSON) because it decides what gets DELETED from a service the suite does not own.

Two things the fix deliberately does NOT do. It spares catalog-level and wildcard policies: bootstrap-ranger.sh seeds those and they are shared with the demo and Polaris itself. And it spares every other namespace, so the parity demo's acparity work is untouched, which is what lets the two suites share one Ranger.

Measured: there is no cache to flush. Policy propagation and table load are both sub-second, and the demo's 45 minutes were failing assertions, not latency.

Taken on the quickstart stack, each number the median of a direct measurement rather than an estimate:

What	Measured
sqe-cli round trip, SELECT 1	183-360 ms
SELECT on a 3-row governed table, repeated	177-245 ms
CREATE POLICY to mask visible to another user	227 ms
DROP POLICY to raw visible again	412 ms
REVOKE to denial	947 ms
GRANT to allow	2365 ms
One Spark probe (token mint + cache clear + JVM + query)	7013 ms, of which 244 ms is the token

Nothing needs a flush, because SQE already flushes. handle_grant and handle_policy_ddl both call invalidate_policy_cache(), so a policy authored through SQE SQL takes effect on the next query rather than waiting out the 30-second policy.ranger.cache-ttl-secs. That TTL bounds edits made in Ranger Admin BEHIND SQE's back, which is the case worth lowering it for. GRANT is the slowest step at ~2.4 s and that is Polaris's own embedded Ranger plugin polling at pollIntervalMs: 5000, outside SQE.

Table load is not slow either. A repeated SELECT costs the same 180 ms as SELECT 1, so essentially all of it is docker exec + CLI startup + the Flight handshake, and none of it is catalog work.

Where the demo's time actually goes: 25 comparisons times a 7-second Spark probe. A clean pass is 3m44s with 3 retry iterations in the entire transcript. The 45-minute runs were four failing steps each burning a 120-second AC_PARITY_POLICY_BUDGET, plus my own concurrent Spark probes contending for the same container. Retry budgets hide broken assertions as latency, which is exactly what happened.

OPEN DECISION: REVOKE SELECT does not stop reads, and the Databricks model says to split read out of write.

Closing the gate on one user in the parity demo took three statements, because grant-profile.json expands table-data-write to include table-data-read. A user holding INSERT keeps reading after REVOKE SELECT, and the statement reports success. Nobody reads an implication graph before believing a revoke.

Unity Catalog does not do this. MODIFY and SELECT are independent there: a principal with only MODIFY can write and cannot read, and MERGE requires both because it genuinely reads. Traversal is the part SQE already matches (USE CATALOG / USE SCHEMA against SQE's namespace-list / namespace-properties-read expansion).

MEASURED, and it kills the obvious fix: table-data-read is not what gates reading. Three probes against the live stack, each waiting out both the Ranger poll (5 s) and SQE's 30-second metadata cache, because a first attempt that skipped the second wait produced a false "read still works" from cache:

Access types granted on the table	Can the user read?
the full INSERT expansion (19 types)	yes
the same, minus table-data-read (18)	yes
table-data-write alone	no, 403 LOAD_TABLE
table-data-write + table-properties-read	yes

table-properties-read is what unlocks LOAD_TABLE. Once that succeeds Polaris vends storage credentials and the engine reads the files directly, so table-data-read is decorative on the read path. Every writer must hold table-properties-read to commit, which means INSERT inherently confers read at the Polaris layer, and no edit to the grant profile can change that.

Four changes, in the order they pay off:

1. Drop table-data-read from the table-data-write expansion. Infeasible as stated, per the table above. Separating read from write needs one of: Polaris gating loadTable / credential vending on a data-read privilege (upstream change, the correct place), or SQE enforcing the read/write split at gate TWO, its own plan rewriter over the query service. The rewriter can already deny (it injects lit(false) on every fail-closed path), but it would have to start reading policyType-0 access policies, which it deliberately ignores today: the shared service carries one blanket allow precisely so Kyuubi defers to Polaris. Changing that is a design decision about which gate owns object access, not a config flag. It would bind both engines, since both consume that service. Note the profile change would also be cross-repo regardless: grant-profile.json is generated by the platform's gen_grant_profile.py and its fixtures are the cross-writer contract.

2. DONE: REVOKE ALL PRIVILEGES ON <object> FROM <principal>. Closing a gate is now one statement instead of an audit of the implication graph. It reads the access types the grantee actually holds from Ranger rather than planning them from a privilege name, so grants written before provenance labels existed or straight through the Ranger console are removed too; it clears that grantee's DENY items at the coordinate; and it is idempotent. GRANT ALL still binds no deeper than the catalog, and that asymmetry is the point: granting everything at a coordinate needs a definition of everything and once wrote a catalog-wide policy from a single-table grant, while revoking everything only removes and cannot widen access.

   Measured live, in sequence: analyst holds SELECT + INSERT and bob reads; REVOKE SELECT leaves bob reading; REVOKE ALL PRIVILEGES produces 403 LOAD_TABLE; a second run succeeds as a no-op.

3. Assert with CHECK ACCESS, not with a query result. SQE already has CHECK ACCESS SELECT ON t FOR USER x, which answers "can this principal read this, and via which grant". It would have made this defect obvious immediately, where SHOW GRANTS did not: SHOW GRANTS lists the statements issued, and the operative fact was the expansion. The parity demo should CHECK ACCESS before asserting the denial.

4. Say what admin and ownership bypass. Unity Catalog owners keep their privileges and cannot be revoked out of them. SQE's admin_roles behave similarly and that is nowhere in the revoke story.

OPEN, dependency bumps: three renovate MRs cannot merge as written, and the reason was recorded only in a branch commit message.

Measured 2026-07-28 on experiment/dep-upgrade-cost (marked "not for merge", 275 commits behind main and now conflicting in four files, so it is not re-runnable). The finding outlives the branch:

• sqlx 0.9 needs rustc 1.94, and the project pins 1.92.
• rusqlite 0.40 hits a links=sqlite3 conflict through vendored iceberg-catalog-sql, which carries its own sqlx 0.8.1. Two crates claiming the same native library cannot coexist, so this is not a version-pick problem.
• reqwest renamed its rustls-tls feature to rustls.
• sha2 0.11 changes hex formatting at ten call sites; the experiment fixed one.

That is what !779 (rust 0.x breaking) is asking for, which is why it cannot go in as a bump. It also changes Cargo.toml WITHOUT regenerating Cargo.lock, the same shape that broke main twice (#386). !733 (DataFusion/Arrow v59) and !730 (object_store 0.14) share that defect and are 301 commits stale against a tree on DF 54.

Order that actually works: raise the rustc pin, or drop sqlx 0.9 from the set; de-vendor or align iceberg-catalog-sql's sqlx; then let renovate regenerate against current main so the lockfile comes with it.

SUPERSEDED: GRANT/REVOKE no longer posts to the unauthenticated plugin endpoint. Code uses /service/plugins/secure/services/{grant,revoke}/, which Ranger covers with isAuthenticated() rather than security="none". The measurement below is the 2.8.0 hole that transport closed. The Ranger 2.8 hold in renovate.json stays for Spark/Kyuubi: Kyuubi's bundled plugin still downloads policies from unauthenticated /service/plugins/policies/download/{service}, which 2.9.0 refuses. See the WANTED block at the top of this file.

Historical (OPEN, SECURITY as filed): SQE's GRANT/REVOKE wrote through a Ranger endpoint that required no authentication at all, and Ranger 2.9.0 closed that hole rather than breaking an API.

Measured on the shipped quickstart at Ranger 2.8.0. A POST /service/plugins/services/grant/polaris carrying no credentials whatsoever returned HTTP 200 and created a live policy granting dave table-data-read. The same request with no credentials against the public v2 API returns 401, so this is specific to the plugin grant/revoke endpoints, not a misconfigured server.

Ranger says so itself, in security-applicationContext.xml:

pattern="/service/plugins/services/grant/*"  security="none"
pattern="/service/plugins/services/revoke/*" security="none"

security="none" bypasses Spring Security entirely, so the basic auth SQE sent on that path was not rejected, it was simply never processed. No UserSessionBase is created, which is why ContextUtil.getCurrentUserSession() is null there. At the time of the finding, GRANT and REVOKE posted to that endpoint (and ranger-setup seeded through it too). Anyone with network reach to Ranger Admin could grant themselves any privilege on any resource, no credentials needed.

This is why 2.9.0 answers 400. ServiceREST.grantAccess calls bizUtil.failUnauthenticatedIfNotAllowed(), which throws when the session is null and ranger.admin.allow.unauthenticated.access is false (its default):

if (currentUserSession == null && !allowUnauthenticatedAccessInSecureEnvironment) {
    throw new Exception("Unauthenticated access not allowed");
}

RANGER-5635 in the 2.9.0 notes ("ranger.admin.allow.unauthenticated.download.access is honored only when Kerberos is enabled") is that class of fix: the check used to be skipped when Kerberos was off, which is exactly how our stack runs (KERBEROS_ENABLED=false). So 2.8.0 was not authenticating these calls and not enforcing the property either. 2.9.0 enforces it.

The original proposed fix was the public v2 policy API. That is not what shipped. GRANT/REVOKE moved to /service/plugins/secure/services/{grant,revoke}/, which keeps Ranger's server-side merge and the named-grantor delegateAdmin check. Setting ranger.admin.allow.unauthenticated.access=true would have made 2.9.0 work by keeping the hole open, and that was the wrong direction. The 2.8 hold that remains is the Spark/Kyuubi download path, not this grant transport.

Status as of 2026-08-12. The parity demo now runs on an EU retail bank fixture, and adding a persona to the quickstart stack turns out to touch five places rather than three.

The fixture was orders(id, region, amount, ssn, email, phone) with three rows. The mechanics were complete; the transcript did not show what they were for. It is now a 12-row customer register and a 24-row payment ledger: national_id, dob, iban, nationality, residency_region, pep_flag, risk_score, and cross-border payments carrying counterparty country, channel, and AML alerts. The old policy targets map over, so sections 1 to 7 keep their probe count and gain a meaning: the row filter is GDPR data residency, MASK_HASH on iban is a pseudonymous account key, MASK_NULL hides an internal risk score.

Two mask types join section 3 (MASK_DATE_SHOW_YEAR on dob, MASK on full_name), and two personas express shapes the analyst/engineer pair cannot: a fraud desk (fraud_analyst/erin) that sees every jurisdiction with no customer identity, and an auditor (auditor/frank) that reads the register unmasked but the ledger only inside a retention window. Their two new sections prove the things a one-table fixture cannot: one tag rule spanning both tables, a row filter staying scoped to the table its policy names, and a join carrying both a filter and five column masks. 34 comparisons, up from 27.

The trap: a new demo identity needs FIVE edits, and the fifth fails as a denial. Keycloak realm role plus user, the Ranger users loop, mkrole membership, the baseline traverse loop, and polaris/bootstrap-data.sh. Polaris federation resolves an EXISTING principal by preferred_username and never creates one, so a realm-only user mints a token successfully and then fails every read with 401 "Failed to resolve principal". Under a role-scoped policy that is indistinguishable from the denials this demo asserts deliberately, so it would have been diagnosed as a policy bug. preflight_role and preflight_principal now name both failures in the first seconds instead of twenty minutes in.

Assertions are aggregates over semantics, not pinned renderings. A count(*) plus sum(CASE WHEN ...) states the security claim without depending on how an engine prints a digest, a float, or a date, so two engines that mask correctly but print differently still agree, and one that does not mask at all fails loudly. Every derived number is checked against the seeded rows in code rather than reasoned: 12 customers / 7 EU, 24 payments / 18 in window, 4 alerts, 15 joined rows, the twelve distinct birth years, no dob on 1 January, all IBANs at most 28 characters. Two renderings come from source rather than guesswork: ranger_store.rs maps MASK_SHOW_LAST_4 to PartialMask{show_last: 4, digit: 'x'}, and sha256_udf.rs emits 64 lowercase hex characters.

CALIBRATED: 43 of 43 comparisons green on a live stack, first pass, exit 0. Both open inferences held, so neither became a third documented divergence: Kyuubi truncates MASK_DATE_SHOW_YEAR to 1 January exactly as SQE does (both returned dob_year_only = 12), and it applies a row filter AND column masks to a JOINED relation the way SQE does (both returned 15 | 15 | 15 | 0). The derived Spark rendering nnnnn9103 was exact. AC_PARITY_SECTIONS="3,8,9" gates comparisons only and never the GRANT/policy/tag actions, so a re-check of the new sections costs minutes rather than a full run.

The run contradicted one of my own captions, which is what running it is for. The five-way panel called carol "every row, every column"; she is sqe_admin AND engineer AND analyst in the realm, so the engineer policy applies to her and she reads the same four masked EU rows Bob does. Being an admin at the OBJECT gate is not an exemption from the DATA gate. Relabelled to say so, since that is the better lesson.

Adding a persona to this stack needs FIVE seeded sites, and a stack seeded before them needs keycloak-config, ranger-setup and polaris-setup force-recreated. All three are idempotent, so re-running them on a live stack is safe.

Two dead ends worth not re-walking. MASK_NONE cannot serve as a break-glass exemption: matching policies are unioned, so lifting a mask needs Ranger evaluation-order priorities SQE does not implement (ranger_store.rs:497, and access_control_e2e.rs:1747 already says so). Column restriction is not authorable at all: restricted_columns is populated only fail-closed, on an unsupported mask type, so "invisible denied columns" cannot be demonstrated through this surface.

Status as of 2026-08-11. OPEN: the parser rejects dbt-trino's view DDL, and SECURITY is not the only token it rejects.

view is dbt's default materialization, so every dbt model that does not explicitly set +materialized: table fails against SQE's Trino endpoint with Parse error: sql parser error: Expected: AS, found: security. The platform side already defaults its scaffold to +materialized: table, which unblocks the shipped template and nothing else, so this stays open engine-side. Telling users to avoid views is not a fix.

Probed SQE's parser directly (live, quickstart stack) and found TWO gaps, not one:

Statement	Result
CREATE OR REPLACE VIEW v AS SELECT 1	accepted
... v SECURITY DEFINER AS ...	Expected: AS, found: SECURITY
... v SECURITY INVOKER AS ...	Expected: AS, found: SECURITY
... v COMMENT 'c' AS ...	Expected: =, found: 'c'
... v COMMENT = 'c' AS ...	accepted

So Trino's COMMENT '<text>' view clause is rejected too: SQE only accepts the COMMENT = '<text>' form. Fixing SECURITY alone still leaves any described view model failing, which matters because dbt writes view comments from model descriptions when persist_docs is on.

The SECURITY decision is an authorization decision, and the direction of the risk is the useful part. Trino's SECURITY DEFINER runs a view with the definer's privileges, so a reader needs access to the view but not to its base tables. SQE expands views and authorizes the base table as the querying user, which is SECURITY INVOKER semantics (parity-demo step 33 asserts exactly that). Therefore:

• Accepting SECURITY INVOKER and ignoring it is a no-op. It already describes what SQE does.
• Accepting SECURITY DEFINER and ignoring it is STRICTER than requested, not laxer: the reader who was meant to be shielded from base-table grants gets denied instead. It fails closed. That is a usability break and a support burden, not a privilege escalation.

That asymmetry is why silently accepting DEFINER is defensible where silently accepting a laxer clause would not be. It still deserves a warning rather than silence, because the failure it produces (denials on a view that works in Trino) is otherwise unexplainable to the user.

The verbatim DDL is now captured, and it settles both unknowns. dbt emits:

create or replace view
      "ws_viewrepro2_1786444430"."dev"."stg_example"
    security definer
    as
      with source as ( select * from "..."."dev_raw"."example_table" ),
      renamed as ( select * from source )
      select * from renamed

Fed to SQE verbatim it reproduces the reported error exactly: Expected: AS, found: security at Line: 3, Column: 5. Delete only the security definer line and the same statement parses and reaches catalog resolution. So SECURITY is the ONLY parse blocker in what dbt actually emits: the CTE body, the quoted three-part name and create or replace are all fine. The COMMENT '<text>' gap found by probing is real but latent, and bites separately once persist_docs is on.

dbt emits definer, so accepting only INVOKER unblocks nothing. That was the fork the decision hung on, and it is now closed.

True DEFINER cannot be implemented in SQE's identity model, and that is the deciding constraint. DEFINER means running the view body as the view's creator. SQE has no service account by design: every query runs as the authenticated user via bearer-token passthrough to Polaris and S3. There is no credential to run as the definer with, so honouring the clause would mean introducing exactly the stored-credential the architecture exists to avoid. Iceberg's view spec has nowhere to put it either: view metadata carries versions, representations, schemas and a free-form properties map, and no security or owner concept.

That leaves three honest options, and only the third both unblocks dbt and keeps the record:

1. Reject precisely. Keep failing, but with SECURITY DEFINER is not supported; SQE evaluates views with the querying user's privileges instead of a parser error pointing at a column number. Honest, still blocks every dbt view model.
2. Accept and ignore. Unblocks dbt. Safe in direction, because SQE's INVOKER enforcement is STRICTER than DEFINER: the reader who was meant to be shielded from base-table grants is denied rather than let in. It fails closed. But the intent is silently discarded.
3. Accept, record, warn. Parse the clause, store it in the Iceberg view's properties map (sqe.view-security = "definer", plus the creating principal), keep enforcing INVOKER, and warn once at creation. create_view in rest_catalog.rs already sends a properties object, currently {}, so there is a place for it. The classification then travels with the view in Iceberg metadata rather than being lost, which is the same argument as sqe.column-tags, and a later DEFINER implementation or another engine can honour it.

Option 3 is now implemented. sqe_sql::view_compat folds both Trino clauses into shapes sqlparser already stores, so neither is invented and neither is dropped:

• COMMENT '<text>' becomes COMMENT = '<text>', landing in CreateView::comment
• SECURITY DEFINER becomes WITH (sqe_view_security = 'definer'), landing in CreateView::options

catalog_ops::view_properties then writes comment, sqe.view-security and sqe.view-definer onto the Iceberg view's properties map (rest_catalog::create_view sent a hardcoded {} before), and warns on definer that SQE enforces INVOKER so readers still need the base tables. The rewrite is parse-gated like ctas_compat, so SQL that already parses is byte-identical and a view broken for an unrelated reason keeps its own error.

Verified live against the quickstart, not just in unit tests: the dbt-shaped DDL creates the view, COMMENT 'eu orders' lands as the view's comment, Polaris returns sqe.view-security: definer and sqe.view-definer: carol in the view metadata, the warning reaches the log once, and the view reads back. The parity demo is 35/35 in 4m50s afterwards, which is what proves the new pre-parse stage is inert for the other 34 statements.

Three things the tests pin, because each was a real bug during implementation: the view NAME scan must be ident (. ident)* rather than "any word that is not AS", or it swallows the clauses themselves; a comment-only statement must not be discarded by an early return keyed on the security clause; and the injected WITH must precede COMMENT, because sqlparser accepts WITH (...) COMMENT = '...' and not the reverse.

Still open, and worth its own decision: SQE writes its view SQL with "dialect": "sqe", so another engine reading the same Iceberg view has no reason to trust the representation. If dbt-on-Trino-compat is a supported path, that dialect string matters. Also still open: the clauses are accepted on BOTH endpoints, not just the Trino one, matching how ctas_compat and alter_execute already sit in the shared pre-parse pipeline. Trino's grammar is a superset here, so accepting it on Flight costs nothing, but it is a choice rather than an accident.

Status as of 2026-08-10. Six failing comparisons in the consolidated parity demo came down to five unrelated causes, and one was a live engine regression that made every Iceberg view unreadable. Three of the six shared a single cause, which is only visible once the plan is read rather than the retry loop watched. Now 35/35, and the last holdout needed three revokes rather than one. A clean run is 3m44s wall-clock with 3 retry iterations across the whole transcript, measured at 34 comparisons; the earlier 45-minute runs were failing steps burning their 120-second budgets, not slow policy propagation.

A row filter that reads a tag-masked column is a real divergence, and it looked like a timing flake. The demo tagged region while region = 'EU' was the active row filter, and Spark returned ZERO rows for 120 seconds of "policies not settled" retries. EXPLAIN EXTENDED settled it: Kyuubi puts its masking Project BELOW RowFilterMarker, so the filter compares the mask literal XX instead of the stored value. SQE filters on stored values and masks the survivors. Nothing leaks either way. The fixture now keeps the filtered column and the tag-masked column apart (phone carries the tag), and section 5b re-creates the collision on purpose and asserts 2 against 0. The count is asserted rather than the empty result set, because an empty result is also what a failed query returns.

MASK_HASH is a second non-portable named mask, and the assertion had pinned the digest length. SQE emits a 64-character sha256, Kyuubi an unkeyed 32-character md5, so length(email) = 64 was silently false for Spark on every masked-cell count. Masked-cell predicates now assert "not the raw value, and hash-shaped".

Every Iceberg view was unreadable, and a unit test was asserting a message shape production never produces. The view path in SqeSchemaProvider::table() only falls back after a genuine table miss, gated on SqeErrorCode::TableNotFound. iceberg-rust's REST catalog says Unexpected => Tried to load a table that does not exist, with no "not found" and no 404, so it classified as a generic CatalogError and the fallback never ran. The guard's own test constructed "HTTP 404 Not Found" by hand and passed. classify_catalog_error now treats "does not exist" as absence, and the new test builds the error through catalog_src the way rest_catalog.rs does.

A revoke test that could never deny, and the third revoke is the one worth knowing about. Bob holds BOTH demo roles in the quickstart realm, so revoking engineer left the section-1 analyst grant carrying him. Revoking analyst SELECT was still not enough: grant-profile.json expands table-data-write to include table-data-read, so the INSERT granted back in section 2 kept conferring read. REVOKE SELECT reported success and the rows still came back. That is correct behaviour (a writer that cannot read its own table is useless) and it means closing the gate is "revoke every privilege that implies read", not "revoke SELECT". Measured one revoke at a time: engineer SELECT gone and Bob still reads 3 rows, analyst SELECT gone and Bob still reads, analyst INSERT gone and Polaris answers 403 LOAD_TABLE. SQE's allow was right at every step and the assertion was wrong.

A token that expired 45 minutes before the test that needed it. Keycloak issues 300-second access tokens while a full run takes tens of minutes, and the script minted them once at startup. An expired token makes Polaris answer 401 Not Authorized, which reads like a policy denial and scored as one against the loose check while failing the strict 403 LOAD_TABLE one. Tokens are now minted per Spark probe; a curl costs nothing beside the JVM start that follows it.

Editing the script while a run was in flight cost the whole run. Bash reads a script incrementally, so deleting a function mid-run shifts every later byte offset. The run finished and its output was unusable as evidence.

Status as of 2026-08-08. Two "access-control DDL defects" were one scan bug, and it was returning the wrong column's data.

The control experiment is the whole story. Both DDL findings were measured on a MASKED table, so both were filed against sqe-policy. Running the identical DDL and query with NO policy at all reproduced both, with masks=0 filters=0 restricted=0 in the log. The fix belonged in the scan.

iceberg_scan.rs matched projections against each data FILE's parquet names, not Iceberg field ids. Gate: no delete files and every file under 3 MB, so every small and freshly-created table and no merge-on-read table. That is why benchmarks never saw it and the access-control fixture hit it every run.

Three symptoms, and the third is the bad one. SELECT id, ssn, nickname after ADD COLUMN returned 2 columns silently. SELECT id, classified after a rename reset the Flight connection. And SELECT classified, where NO projected name matched the file, returned id's VALUES under the name classified: the empty index list was treated as COUNT(*), which reads parquet column 0, while the real count-star flag was false. Fixed by field-id resolution plus a typed-NULL backfill.

The published rename finding was wrong, and the correction is instructive. I had reported the engines breaking DIFFERENTLY (SQE dropping the column, Spark returning it raw). With the scan fixed they AGREE: both return it RAW. What remains is one shared defect, a rename silently unmasking a column, which is what I predicted BEFORE writing the original test and then abandoned because the measurement disagreed. The measurement was of a different layer.

A cross-engine comparison is structurally blind to what both engines get wrong together. Agreement is not correctness. sqe.column-tags keyed by column name is exactly that kind of shared assumption.

SHOW MASKING POLICIES shipped blind to tag policies. It walked only bundle.policies, never bundle.tag_policies, so a tag-masked deployment got an EMPTY listing: the same "nothing is masked here" hazard the refusing trait default was written to prevent, reached from the other side.

SET MASKING POLICY refuses resource policies on purpose. Ranger evaluates a policy's database/table/column lists as a CROSS PRODUCT, so appending a column would widen the mask to tables nobody named. Tag policies resolve to their tag and reuse the projector.

The quickstart's Spark no longer runs as root. A root-credentialed catalog alias defeats per-user identity for the whole session, because a per-user token governs only the alias it is attached to. parity-test.sh now asserts the property: a tokenless spark-sql must fail to load the table.

CI was not publishing to Harbor at all. Two guardrails failed in an early stage, so every build and publish job was SKIPPED. Both were false positives: gitleaks on a plan doc's quickstart password, and "license evidence missing" for our own crates because syft reads Cargo.lock (no licenses) and deps.dev cannot resolve a workspace member or a vendored fork.

NEXT: issue #391 tracks the open items. The rename defect is FIXED (!794): RENAME COLUMN and DROP COLUMN now carry sqe.column-tags with them in the same commit as the schema change. Next by weight: the worker scan path still resolves projections by parquet NAME (same class as the coordinator bug !784 fixed, unverified because the quickstart is single-node), then the mask-precedence decision (align SQE to Ranger's tag-first, or keep resource-first and document it).

Status as of 2026-08-06. Spark is subject to the same object-level access control as SQE, and it needed no engine code. Polaris already runs its own Ranger plugin keyed on the federated OIDC identity, so the whole thing was a credential swap: give Spark's Iceberg REST catalog a per-user Keycloak token and Polaris authorizes the end user instead of root. token-refresh-enabled=false is load-bearing, because Iceberg otherwise exchanges the external JWT against Polaris's own token endpoint and the identity silently reverts, at which point every denial test passes for the wrong reason.

Six live probes before a line of design, and the last two changed it. Probes 1 and 2 each disabled one tier, which is how the gap between them stayed hidden: run together, Kyuubi refuses BEFORE Polaris is consulted, and since SQE ignores policyType-0 entirely the two engines disagreed on every object-level grant. Object level belongs to Polaris, so the shared query service carries one deliberate blanket allow that makes Kyuubi defer. It cannot be a self-documenting named policy, because Ranger auto-generates all - database, table, column over that exact resource signature and refuses a second one with error 3010; the grant API merges an item into the existing match instead. Probe 6: Polaris refuses an unauthorized write at ADD_TABLE_SNAPSHOT, not at LOAD_TABLE, so a refused write can leave staged files behind.

The shared service is renamed hive -> query. The old name sent every reader looking for a metastore that is not there. The servicedef TYPE stays hive, because Kyuubi is hardwired to the database/table/column shape, and the Rust default stays hive so existing deployments are untouched. parity-test.sh passes 3/3 byte-exact through the renamed service, and the SQE suite still passes 31/31, which is what proves SQE really does ignore policyType-0 rather than it merely being assumed.

A mutation run caught the guard test asserting the wrong service. It checked the test-owned sqe_ac_hive, but Kyuubi reads whatever the container's ranger-spark-security.xml names, and the plugin's own cache file (sparkSql_query.json) gave it away. A precondition asserted against a service the engine never reads is worse than no precondition, because it reads as proof. The guard now re-reads the container config so the two cannot drift apart silently.

Phase 2a landed too: mask and row-filter parity asserted directly across both engines. Four cases, and the first two are each other's control: a portable CUSTOM mask (concat + substr, built-ins in both DataFusion and Spark) must render byte-identically, and Ranger's named MASK_SHOW_LAST_4 must NOT, because Kyuubi ignores the servicedef transformer. If the comparison ever reported equal regardless, the inequality assertion fails. Both engines are pointed at the test-owned frontend service, Kyuubi by a plugin conf written into the container per invocation, so mask policies never touch the service parity-test.sh cross-compares against. 11 Spark cases total, 0 failed.

Phase 2b landed: the tag projector closes the last fail-open. SET TAG now also writes the association into Ranger's tag store, so Spark masks a column SQE tagged. Measured first: PUT /service/tags/importservicetags with op: add_or_update writes tagdef + resource + association in one call and MERGES, and Kyuubi masks from the projection alone (no Atlas, no tagsync). On projection failure the Iceberg property is ROLLED BACK, because keeping it would mask in SQE while Spark returned raw and the statement would have reported success. project-tags is off by default; the quickstart enables it. 13 Spark cases, 0 failed.

Two vacuous tests caught by mutation, both mine. The rollback test was verified by removing the revert (0 tags before, 1 after). The tag-parity test PASSED with project-tags = false, because Ranger's tag store is global and persists across runs and the fixture only cleaned policies: it was reading a stale association. With tag-store cleanup added, the same mutation fails and shows the fail-open directly. Also fixed: spark_access_control_e2e contains the substring access_control_e2e, so make test-access-control had been silently force-running the Spark tests (38 instead of 31) against a stack that deliberately excludes Spark.

FIXED 2026-08-16, issue #421: CALL system.reproject_column_tags (also CALL sqe.system.reproject_column_tags) projects existing Iceberg sqe.column-tags into Ranger for tables tagged before the projector. Scope is exactly one of table, namespace, or catalog. Admin-only ([auth] admin_roles). Requires project-tags = true. Returns one row per table (projected / skipped / error). Does not write Iceberg, so there is no SET TAG rollback. Repair for a failed SET TAG projection remains: re-run SET TAG.

NEXT: owner-on-create and the dedicated always-running IO runtime, both unchanged. A tag-masked column on a table tagged before the projector, and not yet reprojected, is still protected in SQE and returned RAW by Spark. The hive/Spark mirror question is now answered: no dual-write of object grants, Polaris owns object level and the frontend service defers.

Status as of 2026-08-05. grant-profile v5 vendored: one file instead of two, and data-platform's #509 item B is unblocked. v5 folds the access-type implication graph into grant-profile.json as a top-level access_types map, so servicedef-polaris.json is no longer an input to planning and the vendored copy is deleted.

Checked before changed, which is what made it a ten-minute job. v5 is v4 plus one key: privileges (15), aliases (13), fixtures (26) and rejects (9) are byte-identical after a sorted dump, and the new 69-entry map is equivalent to the servicedef's (all 69 names in both, zero differing implied set). SQE's expansion was then replayed in Python over all 26 fixtures against BOTH sources -- 26/26 either way -- so the change was known to be a change of WHERE the graph is read from before any Rust moved. The Rust suite passed with no fixture edits.

The fold does not pre-expand privileges, and that distinction is the point. The original argument for two files was that finished access-type sets would make the fixtures self-satisfying: this code asserting it read what it read. SQE still computes the closure and compares against expect, which the platform computed with its own code, so the property the split protected survives the fold. Pre-expansion would not, and should be resisted if proposed.

servicedef-polaris.json did not go away. It is still the Ranger service DEFINITION, registered by both quickstarts' bootstrap-ranger.sh. Only the vendored planning copy is gone, and scripts/check-vendored-profile.sh no longer names it, so the platform can move or drop its shared copy without SQE's gate failing. That gate exits 2 (cannot compare) on a missing file and is not allow_failure, which is why the order mattered: SQE migrates first, then they delete.

One new guard: access_types is a REQUIRED serde field. Defaulted, a profile missing it would expand every seed to itself, so INSERT would confer table-data-write alone and every Iceberg commit would fail an authz check with nothing pointing at the profile. Under-granting is the safe direction and the undiagnosable one; a_profile_without_the_implication_graph_is_refused pins it, mutation-checked.

NEXT: unchanged. Owner-on-create (specced, docs/superpowers/specs/2026-08-04-owner-on-create-design.md), then the dedicated always-running IO runtime that would FIX rather than bound wedge cause 2, then the hive/Spark mirror question.

Status as of 2026-08-04 (bridge, written after the grant-authority entry below). The catalog-traversal wedge can no longer hang a process: runtime_bridge::block_on_compat waits on an OS-level deadline and returns a typed error instead of blocking forever. Cause 2 (the bridged future awaiting I/O registered with the parked caller's runtime) is still not FIXED, and cannot be from inside the bridge; it is now bounded and diagnosable, which is what a bridge can do.

The guard had to be OS-level, and the two obvious alternatives were already ruled out in this repo. tokio::time::timeout cannot fire on a runtime whose thread is synchronously blocked (learned at #195, and the access-control e2e test carried a comment claiming otherwise, now corrected). JoinHandle::join has no deadline at all. Receiver::recv_timeout fires regardless of what any runtime is driving, so the bridge now sends its result over a channel. On the deadline the worker thread is left detached, which is stated rather than hidden: nothing can safely cancel a future blocked in another reactor, and the count is bounded by the number of timeouts.

The return type changed from Option to Result<_, BridgeError>, and that is the honest half of the change. The third state meant "could not run" with no reason attached, so six call sites rendered every failure as "no tokio runtime available" -- a message that would have been a lie on a timeout, on a runtime that was present. Ten call sites now name the real cause, and contains_or_refresh loses a state it only had to carry that ambiguity.

Exposure is unchanged and remains test-and-embedded only: every deployed entry point builds a multi-thread runtime, whose bridge branch neither contends for a core nor parks the reactor. The dedicated always-running IO runtime that would actually fix cause 2 still wants its own spec.

Coverage: 4 bridge unit tests, the new one mutation-checked (restoring the unbounded wait makes it fail in 10s with its own message rather than hanging the suite, which is the property being bought).

Status as of 2026-08-04 (later). WITH GRANT OPTION is usable: [access_control] grant_authority = "ranger-delegate" hands the GRANT/REVOKE decision to Ranger's per-resource delegateAdmin, and a table owner can grant on their own table with no engine-wide admin role. Default stays admin-role, so an upgrade changes nobody's deployment.

The naive version of this change would have shipped a removed authz gate that enabled nothing. A live Ranger 2.8 probe (9/9 predictions, re-run to confirm the one deviation was my own leftover policy) settled it: delegateAdmin does NOT cascade upward. A grantor holding it on cat.ns.tbl gets 200 there and 403 on cat.ns AND on cat, for grant and revoke alike (separate endpoints, separate messages, verified independently rather than inferred), and 403 for an access type outside their delegate set. Since a table GRANT writes three policies outermost-first, a delegated grant fails on its very first call. So the relaxation only means something with the second half: SQE now SKIPS a traversal level the grantee already holds at that exact resource. Ranger merges access types, so the skipped POST would have changed nothing; what it removes is the one call the delegated grantor was not authorized to make. The named level is never skipped (it may still add access types or delegateAdmin), and the check is exact-resource on purpose, because a wrong "already covered" from wildcard matching would skip a level the grantee lacks and leave a grant that reports success and confers nothing.

What follows is the actual shape of delegated grants, documented rather than hidden: an admin onboards a principal to a catalog and namespace once, table owners manage their own tables from then on. An unseeded grantee gets an error naming the level that failed and the two USAGE statements that fix it.

Two things are safe by construction rather than by documentation. ranger-delegate is honoured only for a backend whose enforces_grantor_authority() is true, so asking for it against a backend that acts with SQE's identity keeps the gate instead of removing the last check (all four cells of that truth table are unit-tested, including the one that must not relax). And the Ranger backend now REFUSES a grant or revoke with no grantor rather than falling back to the configured admin user: with the coarse gate off, that fallback would have been full escalation from any authenticated session. Asserted as zero HTTP requests, because refusing after the catalog level had landed would be worse than not refusing.

DENY keeps its admin gate and ignores the setting. Ranger's grant/revoke endpoints cannot write a deny item at all, so DENY goes through the policy API, which authorizes the REST user and takes no grantor. There is nothing finer to hand over to, so relaxing it would leave the write unauthorized rather than authorized more finely.

Coverage: 4 wiremock tests (each mutation-checked: skip disabled, skip applied to the named level, admin fallback restored, level hint removed) and 3 live e2e cases covering the whole matrix. The live e2e is itself mutation-checked: disabling the skip makes dave's delegated grant 403 on the catalog level. Also fixed a doubled "Query execution error:" prefix in the grant failure message, which is the one message operators are asked to read.

NEXT: owner-on-create, specced at docs/superpowers/specs/2026-08-04-owner-on-create-design.md and deliberately not built: it is what makes delegated grants self-sustaining (a table creator currently owns nothing), and it needs a narrow, stated exception to "always act as the caller" plus a DROP story. Then the schema() wedge cause 2.

Status as of 2026-08-04. grant-profile v4 is adopted: the grant path plans from the vendored profile instead of a hand-written map (!770, !771), and SHOW SCHEMAS/SHOW TABLES no longer answer about the wrong catalog (!770).

The catalog-resolution bug was one comparison. show_catalog asked whether the named catalog differed from config.catalog.warehouse, the LEGACY single-catalog field, and used the session catalog when it matched. The session resolves through resolve_default_catalog(): query.default_catalog, or failing that the alphabetically FIRST entry of flattened_catalogs() (which sorts, for deterministic information_schema ordering). With two declared catalogs and [catalog] warehouse = "sales_wh", the session default sorts to ops_wh, so SHOW SCHEMAS FROM sales_wh listed ops_wh's namespaces and reported success; the other catalog returned the same rows by the discovery route, so the two were indistinguishable. It matters because SHOW SCHEMAS is how an operator confirms a grant landed, and it cost real time twice during the traversal work: once making a fixture table look absent, once making GRANT USAGE look insufficient for discovery when Polaris was logging 200. A second latent bug went with it: the by-name path used the discovery template, which clones one catalog's config and overrides only warehouse, so a declared catalog with its own url, auth or TTL was read with another's settings. Unknown catalogs now error rather than falling back.

Profile adoption deleted SQE's privilege vocabulary. Gone: map_sql_to_ranger_access{,_for}, READ_ACCESS, WRITE_ACCESS, VIEW_READ_ACCESS, MAPPED_PRIVILEGES, ResourceLevel, build_resource_map, reject_scope_deeper_than_level, plus eleven unit tests whose subject is now the 26 golden fixtures. REVOKE and DENY take the deepest planned policy, so the scope guard lives in one place and all three statements agree on scope by construction. check_access uses the deepest level's SEED rather than the first element of the sorted expansion, which for INSERT would have reported table-data-read and made a write privilege look like a read. GROUP grantees now work (they are Ranger roles of the identical name; the old refusal cited usersync, which this deployment does not use), and a partially written plan is now compensated innermost-first rather than left half-applied.

INSERT is narrower, and that is a security fix. It no longer confers table-location-set, table-uuid-assign, table-format-version-upgrade or table-properties-write, so an append-only grantee can no longer repoint a table's storage. Verified by asserting their absence AND that a write still commits, which is what makes the narrowing safe rather than merely smaller.

But adoption does NOT narrow existing grants, and that is the thing to know before deploying. Ranger's grant endpoint MERGES access types into the policy for a resource and REVOKE removes only the types it names, so a policy written by the old code keeps all four wider types and a narrower REVOKE INSERT cannot clear them. Observed: a fixture table granted by the old code still showed all 23 types including table-location-set after the change. Found because the first version of the assertion sat on a shared fixture table and failed on residue rather than behaviour; moving it to a table with no policy history is the correct scope for the claim. A bulk cleanup of live access-control policies wants its own change rather than being buried in a refactor.

Process note worth keeping. The planning algorithm was validated in Python against the profile's own fixtures before any Rust was written, which caught that a plan truncates at the level the statement NAMES (22/26 fixtures before that) in minutes rather than through a Rust rewrite. And an MR showed an empty diff not because of tooling but because its target branch had been deleted on merge while the work was in flight; GitLab reported zero changes rather than an error. Retargeting to main fixed it, and later MRs target main for that reason.

NEXT: wire scripts/check-vendored-profile.sh into CI, the only unfinished item of the platform handoff. Then the cleanup pass for over-broad grants already written. Then, needing their own specs: the hive/Spark plane. CORRECTED after checking the configs: the earlier framing here said a SQL grant is "invisible to Spark", which is too absolute and partly wrong about why. SQE writes ONLY the polaris plane. Spark's Kyuubi RangerSparkExtension runs with plugin.mode = ACTIVE against the hive service, so it ENFORCES authorization and default-denies without a matching hive allow policy. Whether the polaris plane gates Spark at all depends on the Spark identity mode: on the platform's user-bound Spark Connect path the user's own token reaches Polaris (spark-delegated-identity.md, keyed on preferred_username), so polaris policies DO apply per-user; on the service-principal path they are bypassed entirely, which is the case the platform's own AccessGrantService comment cites for mirroring coarse grants into hive. Either way a SQL-issued grant is not SUFFICIENT on the Spark path, because the hive plane must also allow and SQE never writes it. So the gap is narrower and better located than "parity layer": mirror coarse grants into hive the way AccessGrantService already does, reusing map_privilege_to_hive_access_types / map_polaris_access_types_to_hive as the contract. Worth copying their hard-won detail: that map was keyed on canonical privileges while callers passed raw strings, so GRANT DELETE silently mirrored nothing and was inert on Spark while the API answered 201 -- the profile's canonical_privilege gives SQE that for free, and an unmapped privilege must mean NO hive write rather than defaulting to select, which would turn USE into row-reading access), DENY as SQL (the backend already works; only a classifier arm is missing), and SqeCatalogProvider::schema() cause 2 (a dedicated IO runtime; test-and-embedded exposure only, since every SQE binary is multi-thread). Smaller and still open: row filters through narrow views, Ranger glob patterns, namespace flattening in resolve_policy_key. Still wanting a decision: relaxing require_admin on GRANT/REVOKE/DENY.

Status as of 2026-08-03 (later). GRANT now writes the full three-level plan grant-profile.json v4 specifies, and the provenance label prefix is fixed to the shared chm. Two corrections to work shipped earlier the same day, both found by reading the platform's handoff contract against what had actually landed rather than by any test.

The label prefix was sqe: and had to be chm:. The revoke-narrowing fix only works if both writers agree on the format. SQE and the data-platform control plane write to the SAME Ranger polaris service and both read these labels to decide what a REVOKE must hold back, so a private prefix left each blind to the other's provenance and falling straight back to the cascade the labels exist to prevent: SQE would strip a grant the platform made, and the platform would strip SQE's. The bug the fix removed, reintroduced across the tool boundary. Now mirrors provenance.py: prefix chm, grantee types USER and ROLE only (a GROUP is not a third kind, since the platform materialises every Keycloak group as a Ranger role of the identical name, so a group grantee is labelled ROLE), and the privilege must round-trip to one SQE actually maps. That last rule carries the security weight: the grant path deliberately lets an operator name a native Polaris access type directly, and accepting that back on the read path let any string reach the pass-through arm, so a forged or hand-edited label could make revoke hold back an access type nobody granted, permanently. An under-revoke is the one direction worse than the cascade, so it fails closed to "no provenance".

The catalog level is now auto-granted, reversing an earlier refusal. build_grant_plan writes catalog:[namespace-list] | namespace:[namespace-properties-read] | table:[...], which is v4's SELECT plan. The earlier version deliberately wrote only two levels on the grounds that catalog-wide namespace-list exposes sibling namespace NAMES unrelated to the granted table. That reasoning was sound and was overruled for a stronger one: the control plane generates its policies from v4, both tools write to the same Ranger service, and the drift gate that keeps them in step compares plans byte for byte -- a two-level plan fails it by construction. A SQL grant that produced different policies from the equivalent API call would make "who granted this, and does it mean the same thing" unanswerable. The widening is real, is now documented as a cost of every table grant rather than as a thing SQE refuses, and the honest mitigation is separate catalogs when namespace names are themselves sensitive. MANAGE / ALL stay single-policy: they bind at the catalog level already.

Outermost-first ordering, unchanged in spirit: Ranger has no transaction across three calls, and outermost-first fails to "can list, nothing readable" (inert) where innermost-first fails to "has table access, table unreachable", the symptom being removed. Revoke releases the DEEPEST level only, matching access/service.py:449, because catalog and namespace policies are shared with every other grant in that catalog and walking the plan backwards would strip discovery from unrelated grants. Traversal therefore accumulates and nothing cleans it up; that is the correct trade. Provenance is written at the deepest level only for the same reason, so the sqe:traversal: marker the earlier version put on shared policies is gone.

Verified: one GRANT SELECT on a table, with no catalog grant issued by hand, and dave (no role, so outside the quickstart's wildcard discovery) reads 3 rows. Mutation-checked twice -- disabling the catalog level and disabling the namespace level each fail the e2e case on their own assertion. All three levels are asserted as EQUALITY rather than contains, because writing more than the profile specifies is as much a drift as writing less. 238 sqe-policy unit tests, access-control e2e 25/25.

One process note. The first run of the three-level assertion failed on a dirty environment: catalog-list and catalog-properties-read left on dave's catalog policy by hand-editing during the traversal investigation survived every REVOKE, because SQE never grants them so no revoke removes them. The test now asserts its catalog-level pre-state explicitly and names that cause, so the next occurrence is self-diagnosing instead of surfacing as a confusing post-grant failure.

NEXT: SHOW SCHEMAS/SHOW TABLES catalog resolution. (The schema() wedge was called production-reachable in the earlier entry below and that was WRONG: both coordinator binaries and sqe-cli build multi-thread runtimes, and the multi-thread bridge branch neither contends for a core nor parks the reactor, so no served query hits it. Cause 1 of two is now fixed; the remainder is test-and-embedded exposure, no longer top of the queue.) Then profile adoption proper (§2/§8 of the handoff): vendor grant-profile.json + servicedef-polaris.json, replace SQE's hand-written access-type map with expand_access_types over the servicedef impliedGrants closure, and add the golden-fixture + vendored-bytes drift gate. That is what makes today's change match v4's expansion and not just its plan shape, and it subsumes the table-location-set divergence (v4 excludes it from INSERT after expansion). Also queued: GROUP grantees on the write path (a two-line fix; the current refusal cites Ranger usersync, which this deployment does not use), grant compensation on partial-plan failure, row filters through narrow views, namespace flattening, Ranger globs.

Status as of 2026-08-03. A table grant now writes the namespace visibility it needs, so reading a table takes two statements instead of three (branch feat/grant-ancestor-traversal). GRANT SELECT ON cat.ns.tbl used to write ONE Ranger policy, at the table level, and that policy is inert on its own: SqeCatalogProvider::schema() answers only for a namespace its per-namespace LOAD_NAMESPACE_METADATA probe could load, so without namespace-level namespace-properties-read the probe 403s, the namespace is hidden, and planning ends at "table not found" without ever attempting LOAD_TABLE. The grant reported success and the grantee still could not read. build_grant_plan now returns the plan ancestor-first and grant() walks it.

Only the namespace level is auto-written, and that asymmetry is the design. The namespace policy is an ancestor ON THE PATH to the table the operator named: required to reach it, conferring nothing about objects outside that path. Catalog-level namespace-list is categorically different, because it exposes sibling namespace NAMES unrelated to the granted table, so auto-adding it would be the same silent widening reject_scope_deeper_than_level refuses, except it would report success. It also gets granted once per role rather than once per table, so the ergonomic case for automating it is weak. Three statements became two and the remaining one is the one that costs something; the docs say that plainly rather than claiming the problem is gone. Ancestor-first ordering is deliberate too: Ranger has no transaction across two calls, and ancestor-first fails to "namespace visible, no table access" (inert) where table-first would fail to "has table access, table invisible", the exact symptom being removed. A primary-level failure now says the namespace grant was left in place. WITH GRANT OPTION applies to the named object only, so a grantee never gains authority to re-grant namespace visibility.

Revoke deliberately does NOT release the namespace policy. One namespace policy serves every table granted under it, so releasing it on the first REVOKE would break the grantee's access to the others; the discriminating case is two tables in one namespace, revoke one, the other still readable. The residue is visibility only and is marked sqe:traversal:<GRANTEE_TYPE>:<name>, which retained_access_types skips explicitly rather than letting parse_grant_label return None and log a corrupt-label warning on every revoke. Refcounting per originating table (sqe:USER:u:SELECT@cat.ns.tbl) would let revoke release it exactly when the last dependent goes away; that is the upgrade, not shipped.

The empirical basis, one variable at a time. dave holds no role, so the quickstart's wildcard discovery does not cover him. With catalog namespace-list plus a table SELECT grant he got table 'sales_wh.acdemo.orders' not found; adding ONLY namespace-properties-read at {catalog: sales_wh, namespace: acdemo} turned the same query into 3 rows, with carol reading 3 rows throughout so the table provably existed for every reading. 6 unit tests (plan shape and ancestor-first order, ancestor carries visibility only, ancestor never carries delegateAdmin, namespace/catalog privileges stay single-policy, the scope-widening guard still fires on ALL named against a table, traversal labels are not read as grant provenance) plus one_table_grant_writes_the_namespace_it_needs, mutation-checked: disabling the expansion fails it on the exact assertion.

Two pre-existing bugs found while establishing this, both reported and neither fixed here. SHOW SCHEMAS FROM <catalog> can answer about a DIFFERENT catalog: with [catalog] warehouse = "sales_wh", FROM sales_wh and FROM ops_wh both returned ops_wh's namespaces, while sales_wh actually holds sales, ac, acdemo per Polaris's own response. In show_catalog the explicit name is preferred and then the guard cat != self.config.catalog.warehouse discards it for the one case where the named catalog IS the configured default, falling through to session_catalog(session), which re-resolves from the session default. It matters because SHOW SCHEMAS is how an operator confirms a grant landed, and it cost time here twice: it made a fixture table look absent (it was not) and made USAGE look insufficient for discovery (it is sufficient; Polaris logged 200 on both the list and the probe). Second: SqeCatalogProvider::schema() HANGS rather than denying when a principal can list a catalog's namespaces while every per-namespace probe 403s -- contains_namespace -> runtime_bridge::block_on_compat -> pthread_join -> __ulock_wait, captured with sample, the same re-entrant-block_on family as #195. It reproduces on a current-thread runtime and not through the container, whose runtime is multi-threaded. NOTE, corrected 2026-08-03: this was described as production-reachable, and it is not -- every SQE binary builds a multi-thread runtime, so the exposure is tests and any single-threaded embedding. It is why the new e2e case asserts the Ranger policy instead of a read by dave: dave sits in exactly that state until the plugin polls, so the first read attempt hangs and eventually never retries.

NEXT: the two bugs above. (Superseded by the entry above: the schema() wedge is NOT production-reachable, since every SQE binary builds a multi-thread runtime. Corrected 2026-08-03.) Then SHOW SCHEMAS/SHOW TABLES catalog resolution. Then the remaining grant divergences: table-location-set in WRITE_ACCESS (an append-only grantee can repoint storage), row filters through narrow views, namespace flattening in resolve_policy_key, Ranger glob patterns. Still wanting an explicit decision: relaxing require_admin on GRANT/REVOKE/DENY, and group grantees on the write path.

Status as of 2026-08-03. Polaris upgraded to 1.7 and access control documented end to end. Five compose files moved apache/polaris:1.6.0 -> 1.7.0 and the whole access-control surface was re-validated from a destroyed-volume start (docker compose down -v, so Ranger's Postgres and the S3 bucket both began empty): scripts/access-control-demo.sh 32/32 and make test-access-control 23/23. The catalog-traversal finding was re-run end to end on 1.7 with the same verdict. One behavioural difference recorded: an ungranted LOAD_TABLE answers 403 on 1.7 rather than hiding behind a 404, though SQE still reports "table not found" because that message comes from its own planning path, not the Polaris status code. Version claims describing what SQE supports no longer name a minor (they had said 1.5 for two releases); claims recording WHEN something was established keep theirs, and blog posts and ebook chapters were left alone as history.

Testing 1.7 found a real bug that the demo had been hiding. CHECK ACCESS SELECT ... FOR USER "alice" answered false while SHOW GRANTS listed table-data-read for ROLE analyst, alice was a member, and alice was reading 4 rows. check_access passed an empty role list to evaluate_access under a comment claiming roles were unknown at that layer, when Ranger serves them at /service/public/v2/api/roles. Since role grants are the normal way to grant, the answer was wrong for the common case, and wrong in the dangerous direction for auditing: it looks authoritative, so an auditor concludes a table is closed while someone reads from it. Now resolves the target user's roles, follows nested roles breadth-first with a seen-set (Ranger does not prevent an operator creating a cycle, and a naive walk would hang the request rather than answer it), and reports degradation in the reason instead of a confident "no" if the lookup fails. Groups are still not resolved, deliberately: Ranger only knows them under usersync. The demo should have caught this a week earlier. Its matcher was the ERE alternation 'alice|true|ALLOW', which passes on any output containing "alice", so the step reported PASS while printing false -- the same vacuous-matcher class fixed once already in that script, second instance missed. Both introspection matchers now anchor on the value, with dave (in no role) as a negative control.

Two other defects found and shipped while assessing data-platform's grant-profile handoff. GRANT ALL ON wh.sales.orders wrote a CATALOG-wide policy: ALL binds to the catalog level, build_resource_map drops the keys below it, so one table was named, success was reported, and the grantee got catalog-content-manage over every table in wh. Silent in both directions. Now refused, naming the scope that would have been written; general rather than an ALL special case, because USAGE on a table and CREATE SCHEMA on a namespace widen identically (!761). And SHOW TABLES leaked the raw Polaris 403 (naming the operation AND the principal) where SHOW SCHEMAS returned a silent 0 rows for the same user; both now share namespaces_or_empty_on_denial, with a non-denial failure still erroring so a broken connection cannot read as an empty catalog (!763).

§7.1 of the platform handoff is answered (!762): vendor grant-profile v4, keep _TRAVERSE_CATALOG. Their spike found catalog namespace-list was not needed to reach a table and could not run a query to confirm. It reproduces at the REST layer (LOAD_TABLE 200 with a table-only grant) and does not survive the engine: SqeCatalogProvider::schema() answers only for a namespace in its cached list, which needs catalog LIST_NAMESPACES plus a per-namespace LOAD_NAMESPACE_METADATA visibility probe. Either failure ends planning at "table not found" without attempting LOAD_TABLE. The three levels derived empirically are exactly v4's SELECT plan, so the contract matches the engine and SQE's single-ResourceLevel map does not.

Docs. New features/access-control-tutorial.md splits the two gates the way they actually are (Polaris catalog gate, then SQE row/column/mask/tag), then combines them. Two published claims were actively wrong and are corrected: the matrix said view grants were unsupported and walked readers through an error !760 removed, and it listed a bare table grant as sufficient to read a table. Every statement in the tutorial ran on the clean 1.7 stack, including GRANT USAGE ON DATABASE (verified to write the catalog-level policy) and the full three-grant sequence as a user holding no role.

NEXT: the grant-profile adoption itself is now unblocked for the platform to pick a version. On the SQE side the open items are the remaining three divergences (single-level plans, table-location-set in WRITE_ACCESS, provenance labels), all of which want the vendored profile; plus relaxing require_admin for GRANT now that Ranger enforces per-resource delegate authority, which is a security-boundary change wanting its own MR. Separately: ~25 stale docs/ranger-*.md links across the book need a sweep.

Status as of 2026-07-31. Access control is now covered by a real integration test (branch test/ranger-access-control-e2e). Until now the only end-to-end coverage of Ranger/Polaris access control was quickstart/polaris-ranger-keycloak/test.sh, which classifies results by grepping CLI output: its denial check matches not found, the same string a typo'd table name produces, and its mask check only asserts the absence of digits. crates/sqe-coordinator/tests/it/access_control_e2e.rs replaces that with 20 cases in the Rust it tier (tag row filters included), asserting decoded Arrow values against an in-process QueryHandler wired to the real Ranger enforcer and Ranger grant backend, authenticating alice/bob/carol/dave through Keycloak ROPC. Denials are proven by running the identical SQL as carol first, so "denied" is distinguishable from "invalid statement". Coverage: grant enables / revoke disables, role vs direct user grants, write privileges separate from read (including a denied DROP that provably did not drop), Ranger deny precedence, resource column masks (amount -> NULL, ssn -> xxx-xx-1111), keyed HMAC-SHA256 hash masks asserted against an out-of-band digest (issue #37), resource row filters (previously untested end to end, because the demo drops its filter policy to keep the Spark mask cross-compare byte-comparable), tag-based masks and tag row filters via SET TAG DDL, tag fail-closed, SHOW GRANTS and CHECK ACCESS asserted per Arrow column. Run with make test-access-control; it brings up a subset of the quickstart stack (no sqe, data-seed or spark container, so the demo fixtures and parity-test.sh are untouched) and is gated on SQE_AC_E2E=1 so scripts/integration-test.sh cannot force-run it against the wrong stack, failing loudly rather than skipping when the gate is set and the stack is absent. One real defect found and fixed: Ranger's tag servicedef defines mask types ONLY in component-qualified form (hive:MASK_SHOW_LAST_4, hive:CUSTOM, trino:...), while ranger_store::map_mask matched bare names, so every tag-based mask fell through to the unsupported arm and the tagged column was restricted instead of masked -- fail-closed, so not a leak, but the feature was inert. normalize_mask_type now accepts the bare and hive: forms and leaves another component's prefix unmatched (still fail-closed). One platform behaviour understood and worked around, no upgrade needed: tag-based row filters looked impossible at first (Ranger rejects the policy with "tag policy can specify values for one of the following resource sets: does not have any resource hierarchies") because the tag servicedef ships an empty rowFilterDef: {}. Cause: Ranger propagates each component servicedef's dataMaskDef into the tag servicedef unconditionally, but rowFilterDef only when Ranger Admin sets ranger.servicedef.autopropagate.rowfilterdef.to.tag=true (AbstractServiceStore, default false). RangerAdmin::bootstrap now patches the capability in over REST (ensure_tag_rowfilter_support), and tag row filters work end to end: tag_row_filter_restricts_rows asserts bob sees exactly the 2 EU rows while alice sees 3, mutation-checked by pointing the policy at an unused tag. Note the aggregate tag servicedef does NOT round-trip through Ranger's own validator (duplicate ozone:assume_role access type + itemId 201209, elasticsearch implied grants naming access types the def does not declare), so the patch sanitizes those Ranger-generated defects first. That PUT is a test-environment patch reset by a Ranger upgrade or docker compose down -v. Closed as documentation, deliberately not as a compose change: the apache/ranger image mounts only install.properties, whose unknown keys never reach the generated ranger-admin-site.xml, and the two alternatives are worse than the gap. Porting the servicedef surgery into bootstrap-ranger.sh would mean reimplementing the dedupe-and-prune logic in jq-less sh, duplicating tested Rust with untested shell, against the servicedef parity-test.sh depends on; wrapping the image entrypoint to sed the xml after setup is fragile against an image we do not control. There is no testing gap either way, because RangerAdmin::bootstrap patches the capability in idempotently on every run. Operators get the property and its snippet from the quickstart README gotcha and the design note. The TODO(phase3) on the tagPolicies shape is retired: capture_live_tag_bundle (opt-in SQE_AC_CAPTURE=1) replaced the placeholder tag_bundle_live_sample.json with a real Ranger 2.8 capture and resolve_tag_policies_against_live_sample is no longer #[ignore]d. Also extracted build_grant_backend into policy_wiring (it was duplicated byte-identically in both coordinator binaries). Tag-state-unknown deny is now covered too (unknown_tag_state_denies): a handler whose TableMetadataCache has never seen the table gets column_tags -> None, and plan_rewriter logs "Tag state unknown (cache miss or disabled); denying access" and injects a deny-all filter. The test carries its own control (warm handler 3 rows, cold handler 0) and asserts no raw value survives. Worth knowing: that same cold-cache condition made a first attempt at a Ranger-outage test pass vacuously -- a second handler pointed at a dead Ranger returned 0 rows, but so did one pointed at a healthy Ranger, because the deny came from the tag branch, not the outage. The mutation check caught it. The genuine policy-breaker test now exists too (ranger_outage_fails_closed): it applies an ssn mask, warms the handler so the outage is the only variable, stops ranger-admin via a RangerOutage guard that restarts it on drop (including on panic, so a failure cannot poison the suite), asserts the query returns zero rows with no raw ssn anywhere, then asserts masking resumes after the restart -- proving the deny was the outage and not a latched breaker. Cache-TTL expiry is now its own case too (cache_ttl_bounds_policy_staleness), pinning the bounded over-permissive window that RangerPolicyConfig::cache_ttl_secs documents: two handlers differing in exactly one config value, the fresh one at the suite's 2s TTL and a second at 30s, driven by a mask edit on the test-owned hive service (which SQE reads directly, so the only timer in the path is the one under test -- a GRANT would have measured Polaris's plugin poll instead). All three edges are asserted: the fresh handler picks the mask up, the 30s handler is still serving its cached decision 10s after the edit, and that staleness ends. Two drafts of it passed vacuously before this one. The first probed ~0.5s after seeding, so it held for any TTL >= 1s; the second started its clock when a warm-up loop SUCCEEDED rather than when the cache entry was inserted, and since a cold TableMetadataCache makes a second handler fail closed for an indeterminate time (measured: 60s), the loop could succeed on a cache hit for an entry already most of a TTL old. Both were caught by mutation, not by reading the code. The fix is setup_ranger_handler_sharing, which reuses the warm table cache so the policy TTL is genuinely the only variable; the case now runs in 33s instead of 93s and the 5s-TTL mutation fails it. The Flight SQL smoke test landed too (crates/sqe-coordinator/tests/it/flight_sql_smoke.rs), closing the last gap in this batch: Arrow Flight SQL is how every client reaches SQE and no Rust test had ever started the server. audit_e2e_test.rs came closest by calling service methods with hand-built tonic::Requests, and states the limit itself -- do_handshake is unreachable that way because tonic::Streaming<HandshakeRequest> cannot be constructed without the server machinery. Two cases now run against a real socket: handshake -> set_token -> GetFlightInfo -> DoGet asserting the decoded Arrow value (SELECT 1 + 1 AS answer, so a literal echoed from the SQL text cannot satisfy it), and a tokenless statement refused on the gRPC status CODE rather than a message substring, because a substring match would also accept an Internal or Unavailable error that happened to mention authorization -- which is how a broken server reads as a working deny. Both mutation-checked (expect 3; expect NotFound), and the negative case's control is the positive one, the identical sequence with a token. It serves via serve_with_incoming on a 127.0.0.1:0 listener, the one divergence from production wiring: both entry points hand tonic a SocketAddr and let it bind, which cannot report an OS-assigned port back. No docker, no gate -- an AnonymousProvider supplies identity and the queries touch no catalog, so it runs on a bare cargo test in 0.3s. NEXT: nothing outstanding in this batch. The access-control CI job carries no signal while the dind runner is down, so its coverage is local-only until that is fixed.

Status as of 2026-07-28. POST /api/v1/catalogs/refresh is now registered unconditionally (branch feat/sqlengine-acl, item 6 of the ACL handoff). The control-plane invalidation hook shipped on 2026-07-23 was registered only inside the if state.web_ui route group. metrics.web_ui defaults to false and is TOML-only (no SQE_METRICS__* override), so on a default deployment the route 404'd while /healthz answered 200 on the same port — indistinguishable from a build predating the endpoint, and debug-level logging hid it. The platform's invalidation hook had therefore never once fired in the quickstart, and data-platform carried web_ui = true in quickstart/sqe/assets/sqe-config/sqe.toml purely as a workaround. The refresh route now sits in its own always-registered sub-router keeping its require_admin_bearer layer; the dashboard and /api/v1/queries* stay behind web_ui and still 404 when it is off. require_admin_bearer fails closed (no bearer provider or no auth config -> 401 "auth not configured"), which is what makes unconditional registration safe, and a test now pins that so a future refactor cannot open a control-plane endpoint by default. 5 new router tests (route present with web_ui = false at 401/403/200, dashboard routes absent, fail-closed with no auth wired) plus a healthz-with-dashboard-off test; all verified red against the pre-fix router (404) before the fix. Docs: operations/web-ui.md also corrected a false claim that the UI is on by default. data-platform can drop its web_ui = true workaround once this lands.

Item 1 also shipped: GRANT ... ON ALL TABLES IN SCHEMA was a silent no-op. extract_grant_statement mapped AllTablesInSchema to (catalog, namespace, None), a namespace-level resource. Namespace SELECT is namespace-list + namespace-properties-read and deliberately carries no table-data-read, and Ranger does not widen a namespace policy to the tables beneath it (no implicit isRecursive, no defaulted table wildcard), so the statement parsed, returned success, and conferred nothing on any table. Live-verified on the platform side: granting that resource shape (resource_key "main_warehouse//analytics_db//") still denied reads on both tables in the namespace. Meanwhile FutureTablesInSchema already mapped to table "*", which covers existing and future tables, so the two were effectively swapped: ON FUTURE did what ON ALL should and ON ALL did nothing. Both arms now share one match arm producing table "*". Fixed here and NOT in the grant profile on purpose: adding table-data-read at the namespace level would silently widen every namespace-scoped grant. Accepted parity limit, documented in design-notes/ranger-access-control.md: Ranger has no future-only resource so ON ALL and ON FUTURE necessarily collapse to the same policy, where Snowflake distinguishes them; SQE treats ON FUTURE as a superset covering existing tables rather than rejecting it. 5 tests (grant + revoke resolve the same shape, single-part schema, ALL/FUTURE collapse pinned, and a guard that plain ON SCHEMA stays namespace-level and is not over-widened), all verified red pre-fix with left: None, right: Some("*"). NEXT: item 7 (revoke not taking effect on a warm table), then items 2+5 (real grantor + WITH GRANT OPTION -> delegate_admin), item 3 (remove require_admin, only after 2), item 4 (profile-driven planning against grant-profile.json v2). Still owed cross-repo: a golden fixture for the ON ALL shape in data-platform/quickstart/assets/ranger/grant-profile.json (generated by gen_grant_profile.py, so a separate MR there, not a hand-edit here).

Status as of 2026-07-26. Bounded-memory Phase 2 foundation in progress (branch feat/bounded-memory-phase2). ScanMorsel + group_row_groups_into_morsels, versioned ScanTask (v1/v2) with row-group/byte-range fields and worker validate_version, worker applies with_row_groups for morsel tickets, coordinator max_bins raised to num_workers * 32. Footer-driven morsel planning and work-stealing still open. Stacks on Phase 0+1 ownership accounting. NEXT: wire footer row-group planning into query_handler (delete-aware gate), then Phase 3 SpillManager.

Status as of 2026-07-26. Bounded-memory Phase 1 shipped (branch feat/bounded-memory-phase1). New sqe-spill crate with pool-backed ByteBudget / Accounted (64 KiB units, wait-on-budget, fail-on-ItemTooLarge, permit Drop releases pool). Worker scan path no longer cumulative-try_grows: decoded batches are ownership-admitted, channel is mpsc<Accounted<RecordBatch>>, Flight holds the permit via AccountedEncodeStream until the encoder polls the next batch. Config: [worker.memory] sub-budgets with resolve/validate. Zero-pruning ≥20x-RAM and slow-consumer gates are green under a 64 MiB pool. NEXT: Phase 2 scan morsels, then Phase 3 SpillManager.

Status as of 2026-07-26. Bounded-memory Phase 0 shipped (branch feat/bounded-memory-phase0). Red safety gates and metrics for the multi-phase spill plan (docs/superpowers/plans/2026-07-25-bounded-memory-spill-execution.md). Worker metrics now expose ownership gauges (scan_*_resident_bytes, flight_*, shuffle_resident_bytes, spill counters, memory_backpressure_seconds). Integration tests under crates/sqe-worker/tests/{zero_pruning_memory,slow_consumer}.rs reproduce the four unsafe boundaries on a laptop (local Parquet + LocalFileSystem, 64 MiB pool): cumulative scan try_grow ResourcesExhausted at ~20x decoded volume, wide/narrow 16-batch queue ratio >>4x, item-bounded shuffle ≥10x a 4 MiB budget, and unknown join stats keeping HashJoinExec. Baseline JSON: benchmarks/results/bounded-memory-phase0-baseline.json. Future-green tests are #[ignore] until Phases 1/4/5. NEXT: Phase 1 — sqe-spill ByteBudget + Accounted batch ownership, remove cumulative fragment reservation, byte-admitted scan/Flight channels.

Status as of 2026-07-23. Event-driven catalog cache invalidation shipped (branch feat/admin-catalog-refresh-endpoint). A workspace catalog created or rebound out-of-band (the platform provisions it directly against Polaris) was invisible to SQE reads (table not found) until an unrelated write/DDL rebuilt the session: the #368 miss-triggered re-list only refreshes namespaces within an already-registered catalog, never the catalog set, so a read-poll could not self-heal it. Four complementary levers now close that gap. (1) POST /api/v1/catalogs/refresh on the health port, behind the existing require_admin_bearer gate, drops every session's cached SessionContext (invalidate_all_session_caches) plus the shared REST-catalog cache (invalidate_rest_catalog_cache_all); an optional {"username": "<u>"} body scopes the session drop to one user, a bodyless POST invalidates all (Option<Json<_>> so an empty body does not 422). This is the platform's instant, event-driven path. No new invalidation logic: it wires the two existing pub invalidators. (2) The SESSION_CONTEXT_CACHE TTL is now the passive backstop and was shortened 300s -> 60s, and (3) made configurable via coordinator.session_context_cache_ttl_secs (env SQE_COORDINATOR__SESSION_CONTEXT_CACHE_TTL_SECS, default 60), pushed into the process-global cache at startup through a new session_context::configure_session_cache_ttl. (4) CALL system.refresh_catalog_cache() gives a pure-SQL client a self-scoped refresh (drops ONLY the caller's own SessionContext, no process-global cache); it is deliberately self-scoped and bypasses the write-privilege gate like table_health, because a global flush reachable by any authenticated SQL user is a multi-tenant footgun and the table-scoped procedure auth model has no gate for a table-less op. The self-scoped path must not call the global invalidate_rest_catalog_cache_all / result-cache wipe (that reroute is what an earlier draft got wrong). On Polaris capability: the 1.0 event-listener framework is a server-side plugin (not a client-consumable pull API), and Iceberg REST offers ETag/If-None-Match only on loadTable (already used by TableMetadataCache), not on the list endpoints, so there is no cheaper catalog-set change-detection than event + TTL. Tests: 4 endpoint (401/403/200-empty/200-username), 1 procedure-parse, 2 config (default + env override), all green; cargo clippy --all-targets --all-features -D warnings clean on the three touched crates. Docs: sql-reference/procedures.md, deployment/configuration.md, operations/web-ui.md. NEXT: platform side (chameleon backend, separate repo) wires WorkspaceProvisioningService.{provision,attach_catalog,detach_catalog} to POST this endpoint via the existing PG LISTEN/NOTIFY cache-invalidation bus; and address the adjacent platform "Issue 6" count-check warn-passes once the endpoint is called on provision.

Status as of 2026-07-21. INSERT OVERWRITE ... SELECT shipped (issue #378, branch feat/insert-overwrite). SQE's write path was append-only and silently dropped sqlparser's overwrite flag, so INSERT OVERWRITE degraded to a plain append (stale rows retained, no error) and broke dbt's insert_overwrite incremental strategy. Now both INSERT entrypoints (handle_insert_streaming, handle_insert) route the flag through one new commit_written_files helper in write_handler.rs: append stays fast_append; overwrite commits an atomic rewrite_files().add_data_files(new).delete_files(removed) swap, reusing the DELETE CoW hand-rolled optimistic-concurrency retry loop. Unpartitioned = full replace; partitioned = dynamic overwrite (only partitions present in the SELECT output are replaced, untouched partitions preserved, Spark partitionOverwriteMode=dynamic / dbt semantics); zero-row overwrite = truncate (unpartitioned) or no-op (dynamic); the static Hive PARTITION (col=val) clause errors loudly (NotImplemented) rather than mishandle. The swap also drops position/equality-delete files covering the removed data files (reuses maintenance::{collect_live_delete_files,covered_position_deletes}), so the new path leaves no #376-style debris. Validation: 7/7 #[ignore] e2e tests pass against a live Polaris stack (insert_overwrite_e2e.rs), including the dynamic per-partition preservation assertion and MoR delete cleanup; run with RUST_MIN_STACK=33554432 (write-e2e thread-stack requirement shared by the existing suites). Trino has no INSERT OVERWRITE syntax (open request trinodb/trino#11602), so parity is N/A and this statement is beyond Trino's SQL surface. Existing DELETE/UPDATE/MERGE CoW paths untouched. NEXT: the remaining open issues from the #371 verification sweep — #377 (stale v3_e2e::{cdc_incremental_scan,for_version_as_of} querying #320-removed table_snapshots columns) and #376 (retrofit the superseded-delete cleanup onto the existing DELETE/UPDATE/MERGE CoW paths).

Status as of 2026-07-19. Victoria observability gaps closed (branch fix/victoria-observability). SQE now supports a trace-only OTLP endpoint for collectors whose logs and metrics use separate pipelines, including environment overrides for the endpoint and sampling rate. Streaming query and Iceberg spans live until stream completion instead of ending after setup. Iceberg plan counters are aggregated into bounded-cardinality Prometheus scan metrics, active query/session gauges are wired, local Iceberg bytes feed the S3 read counter, and EXPLAIN FULL.files_scanned reports executed files after pruning instead of copying the snapshot total. NEXT: enable SQE_METRICS__TRACES_OTLP_ENDPOINT=http://otel-collector:4317 in the data-platform quickstart and validate trace/log correlation against VictoriaTraces and VictoriaLogs.

Status as of 2026-07-18. Read-path unified benchmark harness complete (branch feat/unified-bench-harness-impl): Tasks 1-7 deliver sqe-bench run and scripts/benchmark.sh. Profile schema (Task 1: benchmarks/profiles/<name>.toml with credential injection from env/AWS); profile load (Task 3: runtime resolution of S3/Polaris creds, no committed secrets); CLI bootstrap (Task 2: argument validation, JSON output path); CLI integration (Task 4: subcommand wiring); run verb with read-suite attachment (Task 5: golden catalog attach + multi-suite test + compare); JSON reporter (Task 6: structured output to benchmarks/results/); and benchmark.sh infra shim (Task 7: coordinator lifecycle + environment setup). Usage: BENCH_PROFILE=local BENCH_SCALE=1 scripts/benchmark.sh tpch ssb tpcds clickbench. Read suites (tpch/ssb/tpcds/clickbench) attach golden (zero load) as delivered in the earlier Phase 1 branch. Write suites (tpcc/tpce) and write-suite provision/reset (new provision verb, reset verb with Polaris snapshot rollback) are out of scope for this harness and tracked as a follow-up plan (spec section "Follow-up"). Docs: docs/site/book/src/features/benchmarks.md (new section "Unified harness"), README mention, roadmap update TBD on Phase 10. NEXT: write-suite provision (build golden once per scale, record write baselines for parity) and reset (Polaris set-snapshot-ref rollback guarded by snapshot-id assert) on a follow-up branch after the read-path harness is verified against live parity runs.

Status as of 2026-07-15. Attached golden Iceberg tables cut per-run benchmark load to zero for the six read-only suites (branch feat/bench-attach-golden-catalog); write-suite shallow clone deferred. Every benchmark run used to rebuild Iceberg tables from scratch even though benchmark-publish-data.sh already made the underlying parquet reusable; for the big read suites (TPC-DS store_sales, TPC-H lineitem, SSB lineorder) that CTAS/load pass dominated wall-clock and measured nothing. Phase 0 spike proved a second iceberg_rest ATTACH against the same Polaris reaches the custom S3 endpoint on both the catalog and DataFusion's own FileIO. Shipped: ATTACH now carries S3 config inline (WAREHOUSE/TOKEN/S3_ENDPOINT/S3_REGION/S3_ACCESS_KEY/S3_SECRET_KEY/S3_PATH_STYLE, crates/sqe-catalog/src/mount.rs); scripts/benchmark-publish-iceberg.sh publishes tpch/ssb/tpcds/tpcbb/clickbench/bank once into a persistent Polaris via a throwaway golden-primary coordinator (idempotent, skip-if-present); scripts/benchmark-attach-golden.sh does the one-shot ATTACH via sqe-cli -e against an admin-capable coordinator (tests/benchmark-attach/coordinator-attach.toml, bearer_passthrough grants the fixed role ATTACH's require_admin gate needs); scripts/benchmark-test.sh gained BENCH_DATA_SOURCE=attach, which skips generate+load for the six read-only suites and queries golden.<ns>.<table> via --catalog golden instead. scripts/ci/attach-parity-smoke.sh proved result-neutrality at SF0.1: all 22 TPC-H queries return byte-identical row counts through the attached golden catalog vs the primary catalog on the same published tables. Deliberately out of scope this branch: TPC-C/TPC-E (write suites) still generate and load normally in attach mode; the shallow-clone step that would let them share golden data files while writing locally (system.register_table over a rewritten metadata.json) is Phase 2, gated on this Phase 1 landing first. No new committed benchmark baselines from this branch (the SF0.1 runs were correctness smokes, cleaned up). Docs: usage section in docs/site/book/src/features/benchmarks.md#fast-benchmark-runs-via-attached-golden-tables, roadmap bullet in docs/site/book/src/development/roadmap.md (Phase 10), design spec status updated at docs/superpowers/specs/2026-07-15-benchmark-attach-golden-catalog-design.md. NEXT: Phase 2 shallow-clone for tpcc/tpce on a follow-up branch, then a real SF10 attach-mode timing run to quantify the load-step savings the design set out to capture.

Status as of 2026-07-11 (b). MERGE follow-ups closed (branch fix/merge-followups): the six gaps left after !569/!576/!577. Correctness: (#372) a MERGE target row matched by more than one source row now errors with a Trino-worded cardinality violation instead of being silently duplicated by the FULL OUTER JOIN. build_cardinality_check_sql compares pair-count vs matched-count (no synthetic row id, stays a streaming aggregate), gated behind the new default-on merge_cardinality_check config flag and run only when a matched clause exists, via the shared check_merge_cardinality helper both merge paths call. (#374) Row-class detection uses injected presence-flag columns (SELECT *, TRUE AS <flag>) instead of first-column NULL sentinels, so a genuinely NULL first column can no longer misclassify a present row. (#375) Oracle sub-predicates (UPDATE ... WHERE / DELETE WHERE / INSERT ... WHERE) are rejected rather than silently dropped. (!569) the ON condition is rewritten with the identifier-aware replace_alias_qualifier on both paths. (!576) a data file missing from the delete-aware scan plan now hard-errors instead of silently raw-reading (which would resurrect deleted rows). Performance: (#373) handle_merge_equality gained the full clause surface via first-match-wins guards (COALESCE(pred, FALSE) AND NOT priors), one guarded query per clause, so predicated / multi-clause / NOT MATCHED BY SOURCE merges stay on merge-on-read (equality-delete + new-data in one RowDelta) whenever the table has a primary key; the merge_needs_cow CoW reroute is gone. New: in-process DataFusion execution tests (register MemTables, run the generated SQL, assert rows) covering first-match-wins, NULL-first-column classification, and cardinality detection. 631 coordinator lib tests green, all test targets compile, clippy clean on touched crates. NEXT: live-stack SCD2 round-trip (dbt-sqe snapshot: predicated matched-update close-out + BY SOURCE expiry) to exercise the MoR equality emission that unit tests can't reach.

Status as of 2026-07-10 (c). Runtime-filter-to-bloom row-group pruning (issue #369, branch feat/369-bloom-probe): sealed hash-join key sets now probe parquet bloom filters. The bloom-on-write lever (!554/!557) was inert for star joins because DataFusion consults SBBFs only for literal-equality predicates and the stats-based row-group IN pruning gives up above 200 literals — a sealed 65536-key runtime filter got no row-group pruning at all. New vendor patch family 8 (SbbfRowGroupEvaluator + reader hook after stats pruning, all sites SQE PATCH (sqe#369)): positive membership conjuncts (IN/= under AND only — OR/NOT ignored so negated equality-delete predicates can never prune unsoundly) are tested against each surviving row group's bloom, and the row group is pruned only when EVERY key is bloom-negative; blooms load lazily, missing bloom keeps. The key unlock is in physical_to_predicate.rs: a PARTITIONED join's sealed CASE-of-InLists filter now converts by unioning all arm IN sets into one Predicate::Set per column constrained by every arm (strict over-approximation; bails on ELSE true / lit(true) arms / mixed-column arms), so the bloom probe fires beyond CollectLeft joins. Config [catalog.runtime_filters] bloom_probe (default true) + bloom_max_values (default 65536) plumbed session_context -> catalog/schema/table provider -> IcebergScanExec; new row_groups_pruned_bloom counter shows in EXPLAIN ANALYZE. 18 behavioral tests in crates/sqe-catalog/tests/bloom_probe_369.rs incl. an e2e parquet write/scan asserting the metric. Expectation: SSB q4.1/q2.x will NOT move (their filter never reaches the fact scan — join structure); the win case is partitioned-join key sets pruning fact row groups. NEXT: SF10 rig off/on A/B (bloom_probe toggle vs --bloom-filter load) and commit the compare JSONs (issue #369 acceptance).

Status as of 2026-07-11. #371 verification found and fixed a CoW resurrection bug (branch fix/371-write-modes). Live-stack verification of write.{delete,update,merge}.mode on CTAS tables (post-!557, post-#370-refresh) confirmed the modes are read at write time: MoR DELETE emits position deletes without rewriting data files, absent properties default to CoW, and MoR UPDATE/MERGE take the documented CoW fallback (SQE DDL cannot declare identifier-field-ids, so the equality path is reachable only for externally created tables). The designed #179 round-trip caught real breakage: every DML rewrite read (read_parquet_via_table) decoded raw parquet, so a CoW UPDATE/DELETE/MERGE over a live delete manifest resurrected MoR-deleted rows (verified live: MoR DELETE id=1 -> CoW UPDATE id=2 -> id=1 back with its old value). Fix: rewrites now read through the Iceberg scan machinery (plan_delete_aware_read per commit attempt + read_tasks_to_arrow_with_metrics per file) so delete files are applied; the position-delete builder stays raw on purpose (physical offsets); the streaming MERGE target (B2) downgrades to the buffered delete-applying path when the snapshot carries delete files. Second find: cached metadata-TVF results (table_files/table_snapshots) were never invalidated by DML because the plan's TableScan carries the TVF name, so cache entries now index under the underlying Iceberg table via metadata_tvf_target_table. New ctas_write_modes_e2e integration suite (7 live tests incl. the resurrection round-trip) pins all of it. Pre-existing failures noted, not from this branch: sqlite-gated drop_secret_in_use_by_attached_catalog_errors, and v3_e2e::{cdc_incremental_scan,for_version_as_of} still query sequence_number/is_current_snapshot, columns the #320 table_snapshots schema removed. NEXT: rig-validate MoR at SF10; consider a follow-up to drop fully-superseded delete files during CoW rewrites (currently left as harmless manifest debris).

Status as of 2026-07-10 (c). Full MERGE INTO clause surface shipped (branch feat/merge-full-clauses-scd2): clause predicates, ordered multi-clause, NOT MATCHED BY SOURCE. Two gaps closed, one of them a silent correctness bug: MergeClause.predicate was IGNORED on both merge paths, so WHEN MATCHED AND <cond> THEN UPDATE updated every matched row; and WHEN NOT MATCHED BY SOURCE THEN UPDATE/DELETE was rejected. dbt SCD2 snapshots emit exactly these shapes (predicated matched-update closing the validity window + predicated insert). New sqe-coordinator::merge_sql module classifies statement-ordered clauses per row class (matched / source-only / target-only via the existing first-column NULL sentinels) and compiles ONE SELECT: per-column CASE with first-match-wins arms + a __sqe_merge_keep boolean filtered in an outer WHERE — which also replaces the old all-NULL marker rows for MATCHED DELETE (filter_merge_delete_rows deleted) and fixes a latent bug where source-only rows with NO insert clause were written as all-NULL rows. MoR dispatch reroutes shapes the equality path cannot express (any predicate, >1 clause per class, BY SOURCE) to CoW with an info log. 7 new unit tests on the SQL generator (dbt snapshot shape, arm ordering, by-source delete/update, needs-cow matrix); 619 coordinator lib tests green, clippy clean. NEXT: e2e MERGE parity run on a quickstart stack (SCD2 snapshot round-trip via dbt-sqe), then the Trino compat batch (#347 try(), #341 UNNEST ORDINALITY, #348 recursive aliases, #354 functions, #342 correlated/LATERAL).

Status as of 2026-07-10 (d). Trino compat batch #347/#348/#341 shipped (branch fix/trino-compat-347-348-341); #354 verified mostly-fixed; #342 = upstream DataFusion. Three new rewriters in sqe-sql::trino_compat (one AST walk, same visitor): (#347) try(expr) is lowered statically — the wrapper is removed, every inner CAST becomes TRY_CAST and every / % divisor gets NULLIF(d, 0), so invalid-cast and divide-by-zero yield NULL per Trino's contract (overflow-class errors still surface; try_cast( and quoted "try" untouched); (#348) a recursive CTE's declared column list is applied to its anchor projection (WITH RECURSIVE t(n) AS (SELECT 1 ...) -> SELECT 1 AS n) so the recursive term resolves the declared names — found separately: count(*) over ANY recursive CTE (aliased or not) hits a DataFusion "project index 0 out of bounds" bug, count(col)/SELECT * work; (#341) uncorrelated UNNEST(...) WITH ORDINALITY AS t(x, n) becomes (SELECT x, row_number() OVER () AS n FROM UNNEST(...) AS __sqe_unnest(x)) AS t — correlated arrays (column refs) are deliberately left on DataFusion's NotImplemented error since a global row number would be wrong and LATERAL can't plan anyway. Verified live via embedded CLI: try-forms return NULL, recursive count(n)=5, ordinality rows (7,1)(8,2)(9,3). #354 re-verified on main: count_if / element_at(array) / sequence / parse_datetime / listagg(+WITHIN GROUP via #340 rewrite) all pass — the ONLY remaining gap is lambdas (filter/transform/reduce), which need DataFusion-level higher-order function support. #342 (correlated scalar subquery in SELECT / LATERAL) is a DataFusion decorrelation limitation (ScalarSubqueryToJoin only rewrites filter-position subqueries); not locally fixable. 14 new sqe-sql tests; 424 sqe-sql tests green; clippy clean. NOTE: the embedded CLI (sqe-cli --embedded) does NOT apply rewrite_trino_compat, so these fixes are server-path only there — wiring the compat chain into embedded is a possible follow-up. NEXT: comment + close/scope the five issues; lambdas and #342 tracked as upstream/deferred.

Status as of 2026-07-10 (b). Vendored iceberg-rust refreshed to dev_rebase_main_20260303 @ 813e544 (issue #370, branch chore/370-vendor-refresh, stacked on !567). Twelve upstream commits via 3-way merge (vendor state committed onto c034b19, git merge 813e544, copy back): the #179 rewrite/overwrite DELETE-manifest fix (CoW DELETE/UPDATE path), memory-bounded manifest streaming on snapshot expiration / rewrite-manifests / append-overwrite validation (#170/#171/#172), position-delete sort by (file_path, pos) (#167), auto referenced_data_file (#169), delete files in snapshot summary (#166), rewrite_manifests target size (#174/#175), and Iceberg V3 Variant support (#145 — Type::Variant mapped to variant in information_schema). All 7 SQE patch families verified present post-merge (DynamicPredicate, SigV4, with_storage_factory, FileIOBuilder shims, loader feature gates, #358 current-schema projection, #367 DecodeGate) plus the #195 block_on fix and the 5 apache cherry-picks. Conflicts were confined to the vendor's earlier hand-backports of the manifest loaders (transaction/{remove_snapshots,rewrite_manifests,snapshot}.rs + the utils.rs->util/ rename) — resolved to upstream semantics; details in vendor/iceberg-rust/README.md. Validation: full workspace build, 16/16 lib test targets green (374 sqe-catalog / 604 sqe-coordinator), clippy --all-targets --all-features -D warnings clean. NEXT: issue #371 — verify write.{merge,delete,update}.mode honored e2e on a quickstart stack (the #179 fix touches exactly that path), plus a CoW DELETE round-trip and a TPC-H bench smoke.

Status as of 2026-07-10. Read-path memory safety (issue #367, branch fix/367-read-path-memory-tracking): scan decode is now bounded and pool-tracked. The 2026-07-09 campaign's two host OOM kills (coordinator at 18GB anon-rss under an 8GB pool showing ~4KB residue) came from a read path with zero MemoryReservation anywhere in sqe-catalog while parallel_scan default-on multiplied decode fan-out to target_partitions x num_cpus (each partition's vendored reader brings its own num_cpus semaphore). Fix is the read-side twin of the write path's TrackedBatchBuffer: new sqe-catalog::scan_memory::ScanDecodeGate gives each scan node ONE num_cpus permit budget shared across all partitions (deliberately per scan node, not process-global — a global semaphore can deadlock a join when parked probe-side decodes starve the build side), and every admitted decode reserves 4x its compressed bytes against the query pool fail-fast (try_grow, never a blocking wait), corrected to the actual decoded size on the direct-read path and released when the batches leave the scan. Vendored reader gained a minimal DecodeGate hook (patch family 7 in vendor/iceberg-rust/README.md; re-apply on the #370 refresh). Pool denials now classify as RESOURCE_EXHAUSTED (new classify_execution_error branch) so pressure fails one query typed instead of OOM-killing the node. Escape hatch: SQE_SCAN_DECODE_TRACKING=off (permits stay). Deferred: jemalloc-with-decay for the parked-RSS pattern, row-group subdivision in split_file_scan_task, SF10 rig validation. NEXT: rerun the SF10 bank compare under an 8GB pool on the rig — expect typed ResourceExhausted or clean pass, never a host OOM kill.

Status as of 2026-07-10. #366 mixed COUNT(DISTINCT) two-phase rewrite (branch fix/366-single-distinct-count-companion). New sqe_planner::SingleDistinctCountCompanionRule extends DataFusion's single-distinct-to-groupby rewrite to admit count() companions (COUNT(*) / COUNT(col)), which upstream rejects — that rejection dropped bank q03's whole aggregation onto per-group HashSet distinct state (8-12GB at SF10, degenerate spill). Count partials are re-aggregated with outer SUM + COALESCE(..,0) for the empty-input contract; rule is complementary (fires only on shapes the built-in rule skips), registered on the coordinator session context. Follow-ups: rig validation of bank q03 at SF10, embedded-CLI registration, upstream contribution.

Status as of 2026-07-10. Idle-timeout no longer kills correct spilling queries (branch fix/365-idle-timeout-operator-progress, issue #365). TrackedRecordBatchStream now fingerprints the plan's operator metrics (elapsed_compute + output_rows + spill activity) when the idle deadline fires and extends instead of aborting while the fingerprint advances; a wedged pipeline or abandoned client still aborts at the first deadline exactly as before (#75 unchanged). New with_query_deadline bounds those extensions with query.timeout_secs — the streaming path had NO total-runtime bound at all (the buffered path's tokio::time::timeout never wrapped stream consumption), so this closes that hole too. Rig follow-up: bank SF10 q03 at SQE_MEMORY_LIMIT=8GB should now finish slow instead of SqeFailed at 302s.

Status as of 2026-07-10. #368 fixed (branch fix/368-live-namespace-relist): externally committed namespaces become visible without a coordinator restart — SqeCatalogProvider::schema() now re-lists namespaces live on a snapshot miss (same visibility probes as construction, 5s per-provider cooldown, failed re-list keeps the old snapshot); the bench runner's post-bank-load coordinator bounce (!552) can be retired once merged.

Status as of 2026-07-10. #364 fixed (branch fix/364-groupby-limit-drop): sortless GROUP BY ... LIMIT no longer over-returns under the parallel-scan rules. LimitPushdown parks the fetch on the root CoalescePartitionsExec; the rules' EnforceDistribution re-run erased it from the tree entirely (clickbench q17 returned ~1M rows for LIMIT 10). Fix: both rules capture the pre-bump root-spine fetch (effective_root_fetch) and re-apply it at the root when the re-optimized tree lost it (reapply_erased_root_fetch); the stranded-fetch walk now shares one spine helper. Unit-tested against the erased-coalesce shape (fails without the fix). Upstream note: EnforceDistribution discarding a coalesce's fetch in remove_dist_changing_operators is a DataFusion bug worth filing.

Status as of 2026-07-09. Build-cycle tuning (branch perf/build-tuning): bench iteration drops from 7m33s to 44s. Timed the real loop (touch sqe-coordinator/src/lib.rs + release rebuild): 7m33s, of which the lib compile is only 46s — the rest is thin-LTO linking TWO binaries (sqe-server 403s + sqe-coordinator 255s) that release's lto=thin/codegen-units=4/non-incremental pays on every change. Fixes: (1) bench scripts (benchmark-test/split/load/matrix) now pass --bin sqe-bench --bin sqe-coordinator so the never-used sqe-server link is skipped; (2) the orphaned dev-release profile is wired in as PROFILE=dev-release (same opt-level, no LTO, incremental) — measured iteration cycle 43.6s after the one-time 11m cold build; committed baselines still come from PROFILE=release. Dep bloat also cut from default builds: sqe-cli no longer defaults aws on (it unified aws-sdk-glue/s3tables into every workspace cargo build/test via resolver-2), and the workspace no longer forces aws-sigv4 on iceberg-catalog-rest — new sqe-catalog feature rest-sigv4 (enabled by glue/s3tables/full-backends) carries it, so the pure-Polaris default now really has zero AWS SDK crates (docs claimed this; it was false until now). Standard Docker image (--no-default-features) thereby loses implicit SigV4-REST — AWS-endpoint users take Dockerfile.full. Upstream check same day: DF 54 = latest stable (arrow 59/object_store 0.14 blocked on DF 55); vendored RW fork is 12 commits behind dev_rebase_main_20260303 tip and the delta is valuable (DELETE-manifest rewrite/overwrite fix #179, memory-bounded manifest streaming #170/#171, position-delete sort fix #167) — vendor refresh is a good next MR. NEXT: vendor refresh of iceberg-rust to 813e544 (re-apply the 5 documented patch families + #195 block_on patch); the remaining serial long pole is the 45K-LOC sqe-coordinator lib — split along flight_sql / web-api / query-handler seams when it hurts again.

Status as of 2026-07-08. Bank benchmark with direct-to-Iceberg generation shipped (branch feat/bank-benchmark-iceberg-gen). New bank schema in sqe-bench (customer/account/kyc_profile dims + day-partitioned transaction/account_balance facts) built for the bank-demo story: bulk-create tens of TB of day-partitioned financial data, then run <14-day windowed queries that prune everything else. sqe-bench generate bank --sink iceberg writes zstd Parquet straight to the table's S3 location through iceberg-rust and commits one fast_append per trading day to the REST catalog (Polaris/Nessie) — the engine and the old staging+CTAS double-write are out of the loop entirely. Sizing is byte-target driven: --bytes-per-day 4t --days 12 runs a pilot calibration (measured ~51 compressed B/row; a SEPA-style e2e reference keeps rows from dictionary-compressing to nothing), derives rows/day and shards/day, and --dry-run prints the full plan with projected duration (48 TB: ~8.4h generation at 64 workers, 4.7h network floor at 25 Gbit — target env is a big cloud box + S3-compatible store). Determinism per (table, day, shard) unit (any unit regenerates identically; days can fan out across boxes); shards own disjoint account ranges and emit time-ordered rows so files get tight zone maps on t_ts+t_a_id with no sort step; per-worker memory is bounded (batch + row group + multipart) regardless of scale. Resume: day markers are table properties (sqe-bench.day.YYYY-MM-DD) committed atomically with the append — snapshot summaries alone are NOT durable on Nessie, which trims snapshot history from served metadata (found live). Validated e2e against the benchmark quickstart's Nessie+RustFS (create/write/commit, partition layout data/t_day=.../, --resume skips committed days, double-load guard bails without --resume; host access needs Nessie EXTERNAL_ENDPOINT since vended config overrides client S3 props). 8 windowed demo queries in benchmarks/queries/bank/. openspec change bank-benchmark-iceberg-gen. NEXT: run the 4 TB/day x 12 rig job on the real cloud box (release build; check the dry-run plan first), point sqe-bench test bank --namespace <ns> at it through SQE, and consider raising --customers (default 10M) so per-account activity stays plausible at 86B rows/day.

Status as of 2026-07-08. SSB SF10 reaches parity with Trino; both kernel-OOM classes closed; five MRs merged (!531-!535). The SSB fix was already in the tree: parallel_probe_scan (#235, shelved in June on a contended-box "perf-neutral" verdict) measured 26.7s -> 20.4s on the clean rig = parity with Trino's 21.4s, and tpch improves 4% (74.0s). It stays OPT-IN: tpcds pays +26% (276s vs 219s) under the flag, so the default flip waits for the memory clamp (tasks 1.2) and a cost gate. En route the flag exposed and we fixed a real correctness bug (MR !534/!535): both parallel rules re-ran only EnforceDistribution after bumping, stranding a redundant spilling sort below the new repartition (q67 OOM at 12GB) and leaving multi-partition roots whose per-partition TopK concatenated to 200 rows under LIMIT 100; fix = EnforceSorting re-run (stock DF order) + restore_single_partition_root (SortPreservingMerge/Coalesce carrying fetch), plus EXPLAIN parity for both rules. Memory arc closed by attribution instead of guesswork (MR !532/!533): SQE_MEMORY_LIMIT was documented but never read -- every "capped" run used the config's 64GB on a 31GB box, which explains both 2026-07-06 kernel kills; with a real 12GB cap the FULL 7-suite SF10 sweep runs in one coordinator, zero kills, and the tpcds load that died at "14GB" completes (write tracking measured ~99% pool coverage of RSS, so phase B of scan-throughput-memory-safety was already done). Post-load RSS parks at ~22GB via glibc arena retention; the DuckDB research (docs/internal/research/duckdb-memory-architecture.md) turned phase C into four items: env override [shipped], RAM-fraction default cap, tuned jemalloc (background purge + decay), caches as pool consumers; sort-merge and hash-join spill are tracked upstream, not built. NEXT: memory clamp on partition count (the q67-class pressure lever), phase D decode-efficiency profile of SSB q3.1 (the remaining per-cpu-second gap), jemalloc A/B on the rig.

Status as of 2026-07-06 (b). External S3 benchmark data source shipped (branch feat/bench-s3-data-source): generate once, load many. New scripts/benchmark-publish-data.sh generates datasets locally and publishes them to an S3 bucket as immutable source data (s3://<bucket>/<bench>/sf<scale>/<table>/*.parquet, skip-if-exists); benchmark-test.sh / benchmark-load.sh gained BENCH_DATA_SOURCE=s3://<bucket> (+ BENCH_S3_ENDPOINT/BENCH_S3_PROFILE) which skips the generate step and loads via read_parquet straight from the bucket — validated end-to-end against StorageGRID (s3://sqe-testlake, SF0.1, all 7 suites, 222/222 pass). Two engine defects surfaced by the first-ever foreign-endpoint TVF use, both fixed: (1) inline-credential S3 stores were only registered on the TVF's schema-inference context, so the actual scan resolved the bucket through the session fallback built from [storage] config — wrong endpoint and credentials whenever they differ (fixed by threading the session RuntimeEnv into read_parquet/read_csv/read_json and registering the store there too); (2) *.parquet globs never worked on object stores (DataFusion only glob-expands local paths; the star becomes a literal key) — the loader now uses the directory form for object-store data paths. Note: the SSRF endpoint gate ([storage.tvf] allowed_http_hosts) applies to any inline S3 endpoint; the scripts inject a temp config allowlisting the data endpoint host plus loopback. SF1 published same day; SF10 publish running. Warehouse-on-StorageGRID BUILT and validated same day: BENCH_WAREHOUSE=external + BENCH_WAREHOUSE_LOCATION put the Iceberg warehouse itself on the external endpoint (Polaris recreated with matching creds, SQE [storage] rewritten in the temp config, Trino comparison reads the same endpoint) — first genuinely fair local SQE-vs-Trino path since both engines fetch over the identical network. SF1 compare on all-StorageGRID: tpch 22/22 matched (SQE 62.3s vs Trino 58.6s), ssb 13/13 matched (27.2s vs 25.7s) — near-parity because the ~12MB/s VPN link paces both engines (IO-latency-bound regime; CPU-side differences reappear on a fast rig). Polaris gotchas burned into the scripts: S3-compatible endpoints without STS need SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION (CREATE TABLE fails with STS 405 otherwise), and with subscoping skipped Polaris' metadata-write client ignores both QUARKUS_S3_ENDPOINT_OVERRIDE and the catalog's storage endpoint — only SDK-level AWS_ENDPOINT_URL_S3 pins it (and the SDK env cannot force path-style, so the endpoint must support virtual-hosted addressing). SF100 needs the streaming SSB generator first. In-memory Polaris still means warehouse tables orphan on stack restart; Polaris-postgres persistence remains the open piece for load-once-test-many.

Status as of 2026-07-06. Dim-build-swap rule shipped (branch feat/dim-build-swap): SSB q4.1/q4.2's partkey filter now reaches the fact scan. Root cause measured via EXPLAIN statistics: cascaded join-cardinality underestimation (stream estimated 100,387 rows vs actual 2,433,461, 24x under) kept the fact stream as the CollectLeft build, and since dynamic filters flow build-to-probe, part's key set could never reach lineorder while a useless fact-side filter was pushed INTO the part scan. New DimBuildSwapRule (default on, query.dim_build_swap): when the build side is a join subtree (byte stats Absent by DF54 construction) and the probe side is a join-free scan with known bytes under the broadcast threshold, swap sides, then strip + re-run post-optimization FilterPushdown and coalesce the new build. Verified on ssb_sf10: outermost join now (p_partkey, lo_partkey), lineorder pushed_down_filters=4, scan output 2,433,461 -> 971,487 rows (2.5x survivor cut, matches selectivity arithmetic within 0.3%); 13/13 row-correct at SF1+SF10; dev-box wall -5% total / q4.3 -21% (bandwidth-confounded box; CPU-side win lands on the rig). Also fixed: EXPLAIN/EXPLAIN ANALYZE now apply the same post-planning plan-shape rules as execution (star-schema reorder + dim-build swap), so EXPLAIN no longer shows a plan that never runs. REMAINING: rig validation (TPC-H/TPC-DS regression sweep + true timing delta) before release.

Status as of 2026-07-06. COUNT(*) metadata fast-path shipped (branch feat/count-star-stats-fast-path). Unfiltered SELECT COUNT(*) now collapses to a literal via DataFusion's AggregateStatistics rule: manifest-aggregated row counts are stamped Precision::Exact when the snapshot has no live delete files, and IcebergScanExec::partition_statistics degrades Exact to Inexact whenever the scan carries any row-reducing filter (static predicate, df_filter, or pushed-down dynamic filter — policy row filters can never be bypassed). Verified live on ssb_sf10: COUNT(*) FROM lineorder returns 60,000,000 via ProjectionExec[60000000]/PlaceholderRowExec (was ~15s of per-row work at SF10); filtered counts keep the full scan plan. MoR tables (live position/equality deletes) stay on the scan path by design. Found during the SSB SF10 CPU-vs-IO probe (analysis MR !526).

Status as of 2026-07-05 (b). SSB SF10 root cause pinned AND SF10-verified; cap-fix candidate refuted by measurement; SF100 prep plan written. Full analysis in docs/evidence/perf/ssb-sf10-root-cause-sf100-prep.md (amended). SF10 loaded and instrumented on the freed dev box (60M rows, 4 files, sort-on-write OOM-failover observed live). Three layers: q1.x healthy; q4.1/q4.2 never get a partkey filter on the fact scan because part is the PROBE side of the outermost join (join structure — raising runtime_filter_inlist_max_values 65536->1M changed rows_decoded by ZERO; decode fraction = custkey x suppkey exactly at both scales); everything else is single-output-partition scan throughput with filters armed and working (q2.1/q3.1/q4.3 decode 0.3-3.4% of 60M; fetch_time above the scan 727ms SF1 -> 10.3s SF10). Ruled out empirically: type mismatch (zero casts, Int32 keys both sides), Iceberg storage, seal-race, InList cap, filter shape, raw decode. NEXT: (1) profile the single-stream funnel at SF10 (why was #235 perf-neutral) before re-attempting output partitioning; (2) make the partkey semijoin pushable (join reordering / stats for the side picker / predicate transfer — code investigation running); (3) SF100 blockers in order: streaming SSB generator port (~68GB resident today), sort-on-write spill, shuffle spill, worker backpressure.

Status as of 2026-07-02 (e). Write-path memory-safety program fully merged; compat docs reconciled. All three write-path MRs are on main: !508 (Layer A pool tracking + Layer B ingest/MERGE-B1 streaming), !509 (BoundedFanoutWriter wired + cow-keep-buffer tracking), !510 (MERGE B2 target streaming + fanout auto-tune). The streaming and bounded paths stay opt-in default-off (merge_target_streaming, fanout_max_open_writers, fanout_buffer_budget); write_buffer_tracking is on by default. Compat docs reconciled to the merged feature set (!511, commit 3da10da): Trino DDL/DML matrix + async statement protocol, DuckDB read_avro/read_delta corrections, Iceberg DF54 + write.merge.mode + maintenance/streaming/fanout. Wrote the stack-validation runbook at docs/internal/plans/2026-07-02-write-path-memory-safety-stack-validation.md so flipping the opt-in flags is a turnkey checklist (MERGE parity, fanout cutover -> rewrite_data_files round-trip, tiny-pool forcing, auto-derivation sanity). REMAINING (all stack/demo-gated, not code): run that runbook on a Polaris+S3 stack, then decide spec open #4 (defaulting the flags on) as a signed-off follow-up.

Status as of 2026-07-02 (d). Write-path memory safety: last two deferred items done (branch feat/write-path-merge-b2-fanout-autotune). (B2) copy-on-write MERGE can now stream its target from the pinned old_data_files instead of materialising the whole target into a MemTable: new merge_target_provider module (MergeTargetPartition = a DataFusion PartitionStream over the captured file set, read lazily one file at a time through the target's own FileIO — the same file_io().read() the buffered path uses, so no credential re-wiring; each batch normalised to the canonical Arrow schema and rebuilt with that exact Arc so StreamingTableExec per-batch validation passes). The target then flows through the merge join as governed/spillable operator memory. Gated behind new QueryConfig.merge_target_streaming (default false, requires write_buffer_tracking); default MERGE path unchanged. (Auto-tune) once bounded fanout mode is active, a fanout_* knob left at 0 auto-derives from the coordinator pool via auto_fanout_caps (max_open clamp [8,64], budget pool/8 floored at one row-group est); both-0 stays unbounded. clippy --all-targets clean, audit/deny green, 575 coordinator + 152 core lib tests (incl. 4 provider fs_io tests + auto-cap unit test). REMAINING (needs a Polaris+S3 stack, not code): end-to-end validation of the two opt-in paths — flip merge_target_streaming=true / set a fanout_* knob and check row/snapshot parity, cutover->rewrite_data_files round-trip, tiny-pool forcing; then consider defaulting them on (spec open decision #4 both-0=bounded flip) as a signed-off follow-up. All the memory-safety code is now in place.

Status as of 2026-07-02 (c). Write-path memory safety: deferred Layer B items 2 + 3 done (branch feat/write-path-fanout-wiring-cow-keep, off post-!508 main). (2) BoundedFanoutWriter is now wired into the streaming write path behind a FanoutLimits {max_open, byte_budget} gate: write_data_files_streaming's partitioned branch uses the bounded writer when is_bounded(), else the vendored unbounded TaskWriter (byte-for-byte unchanged). WriteHandler::fanout_limits() resolves the knobs from config (fanout_max_open_writers + parse_memory_limit(fanout_buffer_budget); a bad budget string warns and disables just the budget); all four streaming callers (CTAS, INSERT, ingest, MERGE B1) pass it. Default (both 0) = unbounded, so bounded mode is opt-in until stack-validated; cutovers emit one info line. (3) cow-keep-buffer: the per-file surviving-rows (CoW DELETE) and rewritten-rows (CoW UPDATE) accumulations are now gated TrackedBatchBuffers (decode reservation already released, so no double-count; honours write_buffer_tracking). Full clippy --all-targets --all-features clean, cargo audit/cargo deny green, sqe-coordinator 570 lib tests (incl. 8 writer/fanout tests). STILL DEFERRED to a Polaris+S3 stack: MERGE B2 input-side streaming (spec open decision #1); auto-deriving the fanout caps from pool size (open decision #4, "pending measurement"); and end-to-end validation of the wired bounded path (Iceberg commit of cutover output + cutover->rewrite_data_files round-trip + tiny-pool forcing + row/snapshot parity), which the local fs_io tests cannot cover.

Status as of 2026-07-02 (b). Write-path memory-safety hardening started (branch feat/write-path-memory-safety). The Iceberg write sink's buffers are invisible to the DataFusion memory pool (runtime.rs build_memory_pool only tracks DF operators), so a large write can OOM-kill the coordinator instead of failing cleanly. Design spec docs/internal/specs/2026-07-02-write-path-memory-safety-design.md extends subsystem A (the 2026-06-21 governor spec) rather than duplicating it. Four unbounded pool-invisible write buffers found, all coordinator-only: MERGE copy-on-write (worst: reads the whole target table into a Vec + collects the merged output), Flight DoPut ingest (try_collect of the whole upload), UPDATE/DELETE copy-on-write (read_parquet_via_table whole-file decode), partitioned fanout (one open writer per partition). CTAS/INSERT already stream (O(batch_size)) and are NOT a leak; the observed 167M-row CTAS 7.2GiB was governed operator state and the demo crash was RustFS at a 512MB cgroup cap. Design is two layers: A pool-track the buffers (MemoryConsumer+try_grow, mirroring sqe-worker/src/executor.rs:143) so a denied grow becomes a typed ResourceExhausted that fails one query not the node; B stream/bound where possible (stream ingest + MERGE output B1; MERGE input B2 needs credential + schema-cast wiring; a SQE-owned BoundedFanoutWriter cutover repaired by the existing system.rewrite_data_files in maintenance.rs). DONE (foundation + Layer A + most of Layer B, MR !508): crates/sqe-coordinator/src/write_memory.rs (TrackedBatchBuffer incl. untracked/gated modes + WriteReservation, 8 unit tests); MERGE target-read tracking; Layer B ingest streaming (handle_ingest_streaming feeds the Flight DoPut stream straight into write_data_files_streaming — no more try_collect of the whole upload); Layer B MERGE B1 output streaming (df.execute_stream() into the streaming sink; the WHEN MATCHED THEN DELETE all-NULL filter moved into the in-stream filter_merge_delete_rows adapter); Layer A UPDATE/DELETE/MoR-merge decode tracking (read_parquet_via_table gained a track flag reserving cow-file-bytes + cow-decode-buffer; handle_merge passes track=false to avoid double-counting merge-target-buffer; merge-equality now uses a tracked merge-eq-target-buffer); config knobs on QueryConfig (fanout_max_open_writers, fanout_buffer_budget, write_buffer_tracking — all #[serde(default)], snake_case since QueryConfig has no rename_all; the escape hatch is honoured all-or-nothing via TrackedBatchBuffer::gated); and BoundedFanoutWriter in writer.rs (LRW cutover, open-writer cap + byte budget, cutovers() counter, optional fanout-buffer reservation) with 4 fs_io+TempDir unit tests (one-file-per-partition, cap-1 reopen, byte-budget cutover, precise LRW eviction). Full workspace builds; cargo clippy clean; sqe-core 257 + sqe-coordinator 569 lib tests green (103 write-path). CONSTRAINT: write paths cannot be end-to-end tested locally (no Polaris+S3 stack); the streaming/decode changes are file-level-validated but their Iceberg commit + row/snapshot parity are integration-only. DEFERRED: (0) cow-keep-buffer — the per-file surviving-rows accumulation in the CoW UPDATE/DELETE rewrite loop is still an untracked Vec (spec named it a third Layer A buffer). Defensible for now: cow-decode-buffer already reserves the true per-file peak (compressed + full decode both live), and the kept set is a strict subset held only after that reservation releases; tracking it closes the residual. (1) MERGE input-side streaming (B2) — spec open decision #1 (ListingTable vs custom file-set provider + schema-evolution cast wiring over old_data_files), untestable credential path; the Layer A tracked merge-target-buffer is the sanctioned fallback until then. (2) Wiring BoundedFanoutWriter into the two partitioned write sites behind the fanout_* knobs (default 0 = current unbounded TaskWriter path unchanged) — the cutover→rewrite_data_files round-trip and the auto-derivation of the caps from the pool need a live catalog. NEXT: validate the whole branch against a Polaris+S3 stack (tiny-pool forcing per path, row/snapshot parity, cutover+compaction round-trip), then land B2 + fanout wiring.

Status as of 2026-07-02. Async Trino statement protocol shipped (issue #2), branch fix/trino-async-statement-protocol. POST /v1/statement no longer runs the query synchronously inside the HTTP handler: a 167.9M-row CTAS used to block one HTTP call for ~60s, tripping dbt-trino's hardcoded 30s request_timeout on the EnergyCo medallion. The POST now registers a query-state handle (QueryStatus registry on TrinoState, moka time_to_idle so an actively-polled long query is never reaped mid-flight, eviction listener aborts abandoned tasks), spawns Q::execute on a tokio task (guarded by a TerminalGuard Drop so a panic still yields a Failed poll instead of an infinite Running loop), waits a bounded maxWait (default 1s, cap 10s), and returns either the first page inline (fast queries, unchanged UX) or a QUEUED "started" response whose nextUri points at the new GET /v1/statement/queued/{id}/{token} poll route. Polls report RUNNING (incrementing token), redirect to the existing results-paging route on finish, or replay the mapped Trino error on failure (Retry-After preserved for RESOURCE_EXHAUSTED). DELETE aborts the in-flight task. Refactor pulled the dispatch + post-processing into run_statement/build_paginated_result shared by sync and async paths; route assembly extracted to build_statement_router with a build-time conflict test. 152 crate tests pass, clippy clean, coordinator builds. Spec docs/superpowers/specs/2026-07-02-trino-async-statement-protocol-design.md, plan docs/superpowers/plans/2026-07-02-trino-async-statement-protocol.md. NEXT: deploy the branch to the demo and re-run the EnergyCo dbt medallion at 167.9M (expect 19/19 with the default 30s dbt timeout), open the MR.

Status as of 2026-07-01 (b). #363 read_parquet directory "Corrupt footer" fixed: DataFusion 54 stale list_files_cache. The directory form (read_parquet('s3://bucket/dir/')) intermittently failed with "Invalid Parquet file. Corrupt footer" while the single-file form worked. Root cause is NOT the footer-fetch / RustFS suffix-range path the issue framing suggested. DataFusion 54 turns on list_files_cache (1 MiB) by default with an INFINITE TTL and no per-entry revalidation (CacheManagerConfig::default), and SQE never overrode it. When an external writer (the EnergyCo dbt medallion landing raw files) grows dir/part-0.parquet after SQE has listed the directory once, the cached ObjectMeta.size freezes at the mid-write size (~3032 B for customer). The next directory read computes the Parquet footer offset from that stale, smaller size and reads the "footer" from the middle of the now-larger file -> corrupt. Single-file reads never consult the listing cache, so they always saw the real size. Proven on the live demo: wire capture showed the failing read as range: bytes=0-3031; a demo-sqe restart (clears the in-memory cache) made both read_parquet('.../customer/') -> 5566 and .../meter_readings/ -> 167,932,474 return correct results, and the same restart cleared a separate stale-count manifestation (meter_readings had returned 72954). By elimination it must be list_files_cache: the statistics + footer-metadata caches both revalidate via is_valid_for(size + last_modified), so neither can serve a stale size. FIX (branch fix/read-parquet-directory-footer-363): new sqe_catalog::lazy_object_store::external_store_cache_config() = CacheManagerConfig::default().with_list_files_cache_limit(0), applied to the coordinator runtime (runtime.rs), the session_context.rs fallback + unlimited-memory runtimes, and the embedded CLI runtime (embedded.rs). Keeps the per-file stats/metadata caches (they self-heal on size/mtime change); Iceberg scans resolve files from immutable manifest snapshots, not directory listings, so they are unaffected. Two deterministic regression tests in read_parquet.rs: one asserts a grown file reads cleanly with the fix, one reproduces the exact "Corrupt footer" crash on the default caches and shows the fix resolves it. cargo test -p sqe-catalog + -p sqe-coordinator green (557 coordinator tests), clippy clean. NEXT: deploy this branch's image to the demo and re-run the EnergyCo medallion (bronze 6 -> silver 7 -> gold 6, 19/19) to close the E2E acceptance; the pre-built demo image still carries the bug (masked only until the cache re-poisons). Status as of 2026-07-01. Trino-compat parser/statement batch #351/#336/#335. Five tempto-surfaced parse/statement gaps, one themed branch fix/trino-parser-batch-351-336-335. FULLY FIXED: (#351a) SHOW CREATE SCHEMA <name> -- sqlparser 0.62 rejects the SCHEMA form at parse time (only models TABLE/VIEW/...), so it is detected by prefix in classifier.rs producing a new StatementKind::ShowCreateSchema(String), handled in query_handler.rs::handle_show_create_schema (mirrors SHOW CREATE TABLE: resolves the namespace via get_namespace, emits a single Create Schema column with CREATE SCHEMA <name> + optional WITH ( location = '...' ); a missing schema errors). (#351b) SET TIME ZONE '<tz>' -- parses as Statement::Set(Set::SetTimeZone) but hit the Utility fallthrough; now accepted as a documented no-op returning Ok(vec![]) (SQE has no per-session zone; the requested zone is logged, timestamps use the engine default), matching Trino's FINISHED-no-rows contract. Only the SetTimeZone variant is caught, so every other unsupported SET still errors. (#351c) bare TABLE <name> -- new parse-gated tokenizer rewrite sqe_sql::rewrite_bare_table (mirrors the #315 bare-VALUES rewrite) expands a leading TABLE keyword to SELECT * FROM, wired into the Trino server pre-parse chain; a no-op for CREATE/DROP/SHOW CREATE TABLE and SQL that already parses. (#335) nested/parameterized ROW-as-CAST (CAST(row(1, row(10)) AS row(a int, b row(x int)))) -- sqlparser cannot parse the nested ROW type, so the AST rewriter never sees it; new source-level sqe_sql::rewrite_nested_row_cast recursively expands the whole CAST into nested named_struct(...) (Trino's exact named-row semantics; named_struct serializes over the wire as a ROW, so no wire work needed). PARTIAL (#336): ALTER TABLE t DROP COLUMN a.b (dotted nested path) now surfaces a clear NotImplemented("dropping a nested column ('a.b') is not yet supported...") instead of sqlparser's baffling Expected: end of statement, found: .; the actual Iceberg nested-struct surgery (removing a subfield from a Struct field and re-committing the schema) is NOT implemented and is the remaining follow-up. Unit tests per fix in crates/sqe-sql (bare_table, nested_row_cast, classifier) + coordinator handler; the pre-existing drop_secret_in_use_by_attached_catalog_errors failure needs --features sql-sqlite and is not a regression. NEXT: implement the #336 nested-struct drop surgery in catalog_ops.rs (walk into the target Struct field, rebuild via TableUpdate::AddSchema preserving field IDs).

Status as of 2026-06-29. Tempto-based Trino/Iceberg compatibility harness shipped; surfaced one blocking response-shape bug. New testing/tempto/ runs the official upstream trino-product-tests Iceberg suite (published jar io.trino:trino-product-tests:465, no Trino source build) against SQE via tempto. It layers on the existing parity stack (docker-compose.test.yml + docker-compose.compare.yml) plus docker-compose.tempto.yml, which adds a Caddy TLS terminator (the Trino JDBC driver refuses Basic auth over plain HTTP; SQE has no native TLS) and a gradle:8.10.2-jdk23 runner. One command: scripts/tempto-test.sh (or --baseline to point the same suite at the real Trino). Catalog is iceberg for free via Config::LEGACY_CATALOG_NAME, so it reuses test_warehouse + scripts/bootstrap-test.sh. Hard-won setup facts (all in docs/internal/process/tempto-iceberg-compat.md): the JVM omits TLS SNI for the single-label host tls-proxy so Caddy needs default_sni; the product-tests jar registers an LDAP SuiteModuleProvider unconditionally so the tempto config needs a dummy ldap: block or Guice will not build; the jar pulls Confluent-hosted Kafka artifacts so the Gradle build needs the Confluent repo; resolution OOMs without a raised Gradle heap. HEADLINE FINDING (blocks every test): SQE's build_page_response (crates/sqe-trino-compat/src/server.rs ~627) emits data: [] (empty but non-null) for column-less DDL/update statements; the Trino 465 JDBC client's ResultRowsDecoder.toRows only early-returns when data == null, then requires !columns.isEmpty(), so it throws Columns must be set when decoding data on every CREATE/USE/INSERT. Real Trino omits data for updates; SQE's PREPARE path already does the equivalent. Suggested fix (deferred per owner): emit data: None when paginated.columns.is_empty(). Reproduction + root cause (primary sources both sides) in testing/tempto/exclusions.md. Branch test/tempto-iceberg-compat. NEXT: apply the data: None fix, then expand testing/tempto/allowlist.txt and triage the Iceberg suite (most upstream Iceberg tests are Spark/Hive/HDFS-coupled and stay excluded).

Status as of 2026-06-26. Trino-wire BI compatibility + mixed-auth listener + error/security hardening shipped (MRs !435-!441). Driven by a data-platform handoff (Metabase/Superset against SQE's Trino HTTP endpoint) plus a codebase-analysis triage. TRINO BI: (#1) PREPARE/DEALLOCATE are now short-circuited in submit_query (register the prepared SQL via the x-trino-added-prepare header, skip the executor) so the JDBC/Metabase connect test passes; combined with the existing EXECUTE <name> USING rewrite (resolves from X-Trino-Prepared-Statement, URL-encoded) the full round-trip works. (#2) an unqualified information_schema reference is qualified with the session catalog (X-Trino-Catalog) at the Trino boundary so it resolves to (and, under polaris-auto, discovers) that catalog instead of the engine default. (#3/#4) info_schema_compat translates DataFusion Arrow type display strings to Trino SQL names + scopes the catalog listing, and DESCRIBE is aliased to SHOW COLUMNS -- both already on main; the data-platform team's failing image was STALE (predated the BI-compat merge), so the fix there is a rebuild from main. (#5, partial) system.jdbc.catalogs enumerates all CONFIGURED catalogs (was default-only). AUTH: (#276) new opt-in fallthrough_on_reject on oidc_password + client_credentials_passthrough lets ROPC + client_id/secret + bearer share ONE listener (a clean token-endpoint rejection returns NotMyCredentials to defer to the next provider; infra errors still stop the chain); and an oidc_password provider with an empty client_secret now inherits [auth].client_secret (which SQE_AUTH__CLIENT_SECRET fills) -- this fixed a ROPC 401 the team hit after migrating to [[auth.providers]]. CORRECTNESS/SECURITY: (#268) new SqeError::Sourced { code, message, #[source] source } + catalog_src/execution_src/auth_src/config_src constructors preserve the cause chain (additive; 4 catalog boundaries migrated, ~697 String sites unchanged); (#269) parquet writer close() errors on empty-write paths now propagate; TWO SQL injections fixed (attacker-controlled X-Trino-Catalog in the info_schema qualify rewrite, and the table name in the SHOW COLUMNS/DESCRIBE info_schema query) -- both escape via standard SQL quoting. TRIAGE: the 2026-06-26 grep-based codebase analysis (docs/internal/reviews/) was filed as 49 GitLab issues, then a verification sweep found ~half false/stale (every panic/unwrap finding hit #[cfg(test)] code; the security-defaults cluster was already fixed) -- those closed, 11 confirmed-real labeled verified-real. #5 architectural finding: per-user enumeration of polaris-auto-DISCOVERED catalogs in system.jdbc.* is BLOCKED -- the Iceberg REST API is per-warehouse (no list), and listing via the Polaris MANAGEMENT API uses a service account, which would leak catalog existence across the authz boundary; do NOT build it naively. NEXT: the deferred #5 parts need a per-user catalog-list capability (upstream/design); flow SP role into SQE-side policy for masks on service principals; the remaining verified-real cleanup issues (oversized files, integration tests for 6 crates, dead_code, doc annotations).

Status as of 2026-06-25 (b). Service-principal OAuth extended to Trino + dbt, validated end to end (quickstart 10/10). Three changes building on the client_credentials_passthrough provider below. (1) ENGINE: the Trino-compat HTTP Basic-auth path now routes through the auth chain (it previously called the legacy Authenticator directly, bypassing all [[auth.providers]]). Both coordinator binaries' AuthenticatorAdapter (crates/sqe-coordinator/src/main.rs AND the DEPLOYED src/bin/sqe_server.rs -- the Dockerfile ENTRYPOINT is sqe-server, easy to miss) build FlightCredentials{username,password} and dispatch through the chain, so a service principal's client_id/secret reaches client_credentials_passthrough over Trino exactly as over Flight SQL. Shared identity_to_session helper extracted to crates/sqe-coordinator/src/auth_session.rs (3 unit tests) so Basic + bearer build sessions identically (preserving roles/refresh/expiry). Backward-compatible: empty [[auth.providers]] -> chain wraps the legacy Authenticator. (2) QUICKSTART: polaris-ranger-service-principal/sqe.toml adds a bearer_token provider alongside the passthrough one (they consume different credential fields, so both serve one listener); test.sh now also proves SELECT allow/deny over Trino HTTP Basic auth AND a client-fetched bearer token over Trino. 10/10 on a live stack; logs confirm the Trino requests hit client_credentials grant (passthrough) and bearer_token JWKS validation inside the trino.submit_query span. (3) dbt-sqe adapter (adapters/dbt-sqe): new method/client_id/client_secret/token profile fields. OAuth client_id/secret travel as Flight Basic auth (server runs the grant); a token sets adbc.flight.sql.authorization_header = Bearer .... Logic isolated in a dbt-free auth.py (flight_db_kwargs) with 7 unit tests runnable without the dbt runtime; sample_profiles.yml gains service_principal + bearer targets; new adapter README documents all three auth styles. NEXT: (optional) flow the SP role into SQE's own policy engine for SQE-side masks on service principals; the interactive Trino OAuth2 external (browser) flow already exists in oauth2.rs ([auth.external]) but is not demoed in a quickstart.

Status as of 2026-06-25. Per-connection service-principal auth shipped + a new quickstart, validated end to end (7/7). New auth provider client_credentials_passthrough (crates/sqe-auth/src/oidc_client_credentials.rs): a client connects to SQE presenting its OWN OAuth2 client_id/client_secret as Flight Basic auth (username = client_id, password = client_secret); SQE runs the client_credentials grant per connection and forwards the resulting bearer token to Polaris. This is the service-principal alternative to the ROPC user/password flow. It is distinct from the existing client_credentials backend (one server-baked identity, ignores the handshake) and from OidcM2mProvider (same). The token's roles come from realm_access.roles; user_id is the connecting client_id; the secret is cached in-memory keyed by client_id so refresh_catalog_token can re-run the grant (the grant issues no refresh token). New config variant AuthProviderConfig::ClientCredentialsPassthrough { token_url, roles_claim, subject_claim, scope } (NO client_id/secret in config) wired in factory.rs; 15 provider + 3 config tests, clippy clean. Constraint: consumes username/password so it CANNOT share a listener with oidc_password (service-principal-only); reachable over Flight SQL only (the Trino-compat HTTP Basic-auth path bypasses the provider chain). New quickstart quickstart/polaris-ranger-service-principal/ proves it on a live Keycloak + Polaris 1.5 + Ranger 2.8 stack: three SP confidential clients (sp-admin/sp-reader/sp-denied) with serviceAccountsEnabled + a hardcoded preferred_username mapper + an aud=account mapper (the profile client scope is excluded so its built-in username mapper does not collide with the hardcoded one), matching Polaris USER principals + Ranger USER grants; policy.engine = "passthrough" (authorization is enforced at the Polaris+Ranger boundary). test.sh mints each token and asserts preferred_username+aud, then proves per-connection identity: the SAME SELECT is allowed for sp-reader and denied for sp-denied, sp-reader is read-only, and a wrong secret is rejected. Branches feat/auth-client-credentials-passthrough (engine) + the quickstart. Spec: docs/superpowers/specs/2026-06-25-client-credentials-passthrough-design.md; handoff: docs/handoffs/sqe-client-credentials-auth-prompt.md. NEXT: (optional) flow the SP role into SQE's own policy engine so SQE-side column masks apply to service principals; Trino-compat HTTP support if needed.

Status as of 2026-06-20. ALTER TABLE SET TAGS column-tag authoring DDL shipped. Column tags are now authored with first-class DDL instead of hand-written SET TBLPROPERTIES('sqe.column-tags'=...) JSON. Two surfaces, both lowered to one internal SetTagsStatement: the SQE-native ALTER TABLE t SET TAGS (email = ('PII','GDPR'), salary = ('PII')) and UNSET TAGS (col), plus the Snowflake-compatible MODIFY|ALTER COLUMN col SET TAG name = 'val' / UNSET TAG name (the assigned value is ignored; the tag name is the label). SET merges: only the named columns change, tags within a column are unioned and deduped, other columns are untouched; UNSET TAGS (col) removes all tags on that column. A hand-rolled pre-parser in sqe-sql (tags.rs) lowers all four forms; the classifier routes StatementKind::SetTags; the coordinator reads the current sqe.column-tags map, applies apply_tag_ops merge logic, and commits one TableUpdate::SetProperties, reusing the same commit + cache-invalidation path as SET TBLPROPERTIES. The mask a tag triggers still lives in the Ranger tagPolicy; SET TAGS only authors which columns carry which label. CAVEAT: until the separate Iceberg-to-Ranger tag sync lands, other engines (Spark/Kyuubi) do not see these column tags. Docs updated: docs/ranger-fine-grained-enforcement.md (Authoring column tags), docs/ranger-tag-storage-decision.md. Branch feat/alter-table-set-tags. NEXT: Iceberg-to-Ranger tag sync for cross-engine tag parity; SHOW TAGS read-back.

Status as of 2026-06-19. FUTURE grants + conditional (sibling-column) masking shipped and documented. Two governance features: (1) GRANT/REVOKE ... ON FUTURE TABLES IN SCHEMA x translates to a Ranger policy with a table wildcard (table = "*"), covering existing AND future tables in the namespace with no follow-up grant. Ranger has no future-only resource, so unlike Snowflake the wildcard also covers tables that already exist; the documented difference is in docs/ranger-access-control.md. (2) A CUSTOM mask SQL expression can reference OTHER (sibling) columns of the same row, not only the masked column (e.g. CASE WHEN department = 'HR' THEN {col} ELSE '0' END on salary). Only BARE column names resolve against the scan schema; a qualified reference (t.col) fails to parse and SQE fails closed by restricting the column (dropped, not returned raw). Documented in docs/ranger-fine-grained-enforcement.md. Branch feat/future-grants-and-conditional-mask. Done.

Status as of 2026-06-19. Phase 2C (SQE <-> Spark Ranger mask parity) shipped + validated byte-exact (MR !386). The same Ranger hive-service policy on the same Polaris catalog produces identical masked output in SQE and standard Spark. Live result: SELECT id, ssn FROM sales_wh.sales.orders run as bob returns xxx-xx-1111 / xxx-xx-2222 / xxx-xx-3333 from BOTH engines, 3/3 byte-exact. Both apply the mask through their own plan-rewrite layer (SQE's PolicyEnforcer / PolicyPlanRewriter; Kyuubi's RangerSparkExtension) reading the same hive service-def + transformer templates, so results agree. The polaris-ranger-keycloak quickstart gains a spark service + parity-test.sh. Version matrix: Spark 3.5.4, iceberg-spark-runtime-3.5_2.12-1.8.1, kyuubi-spark-authz_2.12-1.11.1, Scala 2.12, Ranger 2.8. Required Spark config so it resolves the injected Hive mask UDF: spark.sql.catalogImplementation=hive (function resolution kept in the built-in/Hive registry). New governance reference docs landed: docs/ranger-access-control.md (catalog path), docs/ranger-fine-grained-enforcement.md (SQE-side row/column/mask/tag enforcement), docs/sqe-spark-ranger-parity.md (this parity result + scope). SCOPE: parity covers RESOURCE policies (named-column masks, row filters); tag-based masking is NOT cross-compared (Spark Authz reads tag associations from Ranger/Atlas, SQE reads them from the Iceberg sqe.column-tags property). Branch docs/ranger-governance-guides. NEXT: Spark 4 parity needs Kyuubi built from source (kyuubi-spark-authz_2.13 is unpublished); tag parity needs an Iceberg-to-Ranger tag sync; ALTER TABLE SET TAGS DDL sugar over SET TBLPROPERTIES('sqe.column-tags'=...).

Status as of 2026-06-19. Phase 3a (tag-based masking enforcement) shipped: tag source wired, full rewriter pipeline proven by executable tests. The TagSource trait (crates/sqe-policy/src/tag_source.rs) reads column-to-tags associations from Iceberg sqe.column-tags table properties; CacheTagSource is the production implementation. The mask-per-tag rule comes from Ranger tagPolicies via PolicyStore::resolve_tags. The PolicyPlanRewriter joins them: for each scan it calls tag_source.column_tags(catalog, full-namespace-vec, table) (the FULL namespace path split on ., not a truncated last component), receives resolve_tags -> (tag_masks, tag_filters, unmappable), and feeds merge_tag_masks which enforces: resource-mask wins over tag mask, restricted column stays restricted, unmappable tag fails closed (column dropped), tag row filters are ANDed with resource filters. Four new executable integration tests in crates/sqe-policy/tests/rewriter_integration.rs cover: (1) tag mask applied end to end (ssn tagged PII + Nullify -> all NULL), (2) FULL multi-level namespace identity -- FakeTagSource capture-asserts it receives ["ns1","ns2"] not ["ns2"] or ["ns1.ns2"] (the recurring identity leak), (3) resource-mask-wins precedence (Redact beats Hash), (4) unmappable-tag fail-closed (SECRET tag, no mask, ssn dropped from output). NOT yet live-demoed: the quickstart stack drifted to a separate compose project; the executable tests are the proof. Phase 3b remains open: tagging DDL (ALTER TABLE SET TAGS), Iceberg-to-Ranger tag sync, CUSTOM tag mask substitution. Branch feat/tag-based-masking. NEXT: Phase 3b tagging DDL + sync; Phase 2C dynamic transformer for arbitrary-N masks.

Status as of 2026-06-19. Phase 2B (session-context functions) shipped: role-conditional masking/filtering, the Snowflake model. New Immutable session-context UDFs (crates/sqe-policy/src/session_udf.rs) resolved from the authenticated session user: current_user(), is_role_in_session(role), current_available_roles(), current_database(), current_schema() (no current_role() — SQE has no primary-role concept). is_role_in_session(role) = membership in the flat token roles (SessionUser.roles), consistent with SQE-side enforcement (token roles, not Ranger membership). Usable INSIDE Ranger policy expressions (parse_sql_predicate registers them, bound to the policy's user) AND in user SQL (registered on the session context). KEY distribution-safety property: being Immutable with literal/no args, they CONST-FOLD to literals during coordinator-side optimization, so a row filter like is_role_in_session('admin') OR region = 'EU' ships to workers as pure literals + columns (no session UDF, no session state) — proven by an executable rewriter test (admin -> all rows, analyst -> EU-only, no residual is_role_in_session in the optimized plan). Always-available (not gated): resolved from the session user present on every query, avoids breaking SQL compat, never fails a policy that references them. current_user() WITH parens hits a sqlparser reserved-keyword quirk (not routed to the UDF); is_role_in_session(...) is the load-bearing primitive and works everywhere. Branch feat/session-context-functions. NEXT: Phase 2C (dynamic transformer for arbitrary-N masks + Spark/Kyuubi byte-exact parity); Phase 3 tag-based masking per docs/ranger-tag-storage-decision.md (associations in Iceberg/Polaris table properties, mask-per-tag rule in Ranger).

Status as of 2026-06-19. Fine-grained enforcement Phase 2A (mask vocabulary) shipped. The full Ranger hive built-in mask vocabulary is implemented and wired end to end. MaskType now covers MASK_NULL, MASK (full redact: X/x/n for upper/lower/digit), MASK_SHOW_LAST_4, MASK_SHOW_FIRST_4, MASK_HASH, MASK_DATE_SHOW_YEAR, and CUSTOM. RangerStore maps every standard dataMaskType string to the matching MaskType variant. The mask_partial DataFusion UDF applies the Hive char-level substitution rules: digits, uppercase, and lowercase each get their own replacement char; punctuation and non-ASCII pass through unchanged; Unicode scalar counting. The quickstart gains an ssn VARCHAR column on orders plus a MASK_SHOW_LAST_4 policy for role engineer: test.sh section 5 proves 111-11-1111 becomes xxx-xx-1111 for bob and stays raw for alice. Branch feat/ranger-mask-vocabulary. NEXT: Phase 2B: session-context SQL functions (current_user(), current_role()) inside filter expressions, requiring a richer SessionUser role model in sqe-auth. Phase 2C: dynamic transformer for arbitrary-N show-first/show-last masks; tag-based masking via Ranger tag policies.

Status as of 2026-06-18. Ranger fine-grained enforcement (Phase 1) shipped: row filters + column masks live in the quickstart. RangerStore: PolicyStore reads the hive Ranger service via GET /service/plugins/policies/download/hive and feeds SQE's existing PlanRewriter. [policy] engine = "ranger" + [policy.ranger] in sqe.toml activates it. The quickstart (quickstart/polaris-ranger-keycloak/) now creates a hive Ranger service instance with a MASK_NULL column-mask policy (role engineer, amount column) and a row-filter policy (role engineer, region = 'EU'). test.sh section 5 proves the enforcement: alice (analyst-only, not engineer) sees all rows and real amounts; bob (analyst + engineer) sees EU-only rows and NULL amounts. The coarse Polaris gate and SQE-side plan rewriting are independent: a query must pass both. Key mapping fact: SQE sends the LAST namespace component as the database resource (for sales_wh.sales.orders, database=sales; not sales_wh.sales). Phase 2A (mask vocabulary) now shipped on branch feat/ranger-mask-vocabulary. Branch feat/ranger-policy-store.

Status as of 2026-06-18. Apache Ranger access-control backend shipped and validated end to end. A second option for Polaris access control alongside the native backend: SQE's new ranger access-control backend (access_control.backend = "ranger") translates GRANT/REVOKE/SHOW GRANTS into Apache Ranger Admin REST calls (crates/sqe-policy/src/grants/ranger.rs, RangerConfig in sqe-core), and Polaris 1.5's embedded Ranger authorizer ENFORCES those policies. New quickstart quickstart/polaris-ranger-keycloak/ (Polaris 1.5 + Ranger 2.8 + Keycloak + RustFS) with test.sh passing 13/13 from a clean bring-up: a GRANT SELECT visibly enables a previously-denied read, REVOKE disables it, a Ranger DENY overrides an allow, negatives are denied, user vs role grants both work, SHOW GRANTS round-trips, and SQE-side fine-grained enforcement (row filter + column mask) is demonstrated. Hard-won enforcement findings (all in the quickstart OVERVIEW.md): Polaris sends root in every authz request so every policy needs root="*"; Polaris IGNORES the token's realm roles (they lack the PRINCIPAL_ROLE: prefix) so the user->role mapping lives in RANGER role membership (usersync in prod), not Polaris principal-roles (whose management ops are unmapped/always-denied in the 1.5.0 authorizer); the embedded authorizer does NOT honor service-def implied-grants so SQE emits the full explicit access-type set per privilege; the effective read gate is LOAD_TABLE (table-properties-read) because SQE reads parquet with its own S3 creds; grantee users/roles must pre-exist in Ranger. Spec/plan in docs/superpowers/. Branch feat/ranger-access-control-backend.

Status as of 2026-06-16. Clean-rig SF1+SF10 verdict (PR #3) + a memory-safe write fix it surfaced. Re-ran the full SQE-vs-Trino compare on a dedicated idle 8-core/31GB box, both engines containerized against the same Iceberg store, query cache OFF, single-node, so the suite totals are trustworthy (no contention, and not a cache effect since the compare runs each query once). SF1: SQE wins all three (TPC-H 2.3x, SSB 1.5x, TPC-DS 2.4x). SF10 is a real scaling CROSSOVER: SQE wins TPC-DS 1.22x (374s vs 455s, a breadth win that still loses the q72 monster at 0.7x) but TRAILS TPC-H 0.86x (q09 0.3x, q18 0.6x heavy hash joins) and SSB 0.53x (scan-bound); on large data Trino's vectorized and distributed hash joins scale better. The dynamic-filter snapshot fix (!371) HOLDS at scale: q10 4.6s, q12 4.9s, q17 13.5s, q20 3.1s, no explosions; correctness held 21/22, 13/13, 95/99. This corrects the contended-Mac "SQE wins TPC-H SF10" read. Loading SF10 exposed a write-path gap: a PARTITIONED CTAS with a sort-on-write hint fans the sort into one non-spillable ExternalSorterMerge per output partition and exhausts the pool (TPC-H lineitem 60M / ~84 monthly partitions OOMs where the unpartitioned SSB lineorder of the same size sorts fine). The bench loader now skips the redundant sort on already-partitioned tables (crates/sqe-bench/src/load.rs); the ENGINE-level fix (bounded or spillable partition writers) remains OPEN (#4). Memory budget on a 31GB box: SQE 12GB pool + Trino heap capped at 32 percent avoids the host OOM-kill that a 16GB pool + default Trino heap caused mid-tpcds-SF10. Branch fix/memory-safe-partitioned-write, results + README + docs/perf/sf10-slow-queries.md in PR #3. NEXT: the SF10 join/scan frontier (TPC-H q09/q18 heavy hash joins, the SSB scan-bound suite) where Trino's vectorized decode and distributed hash joins win; and the engine-level partitioned-write memory safety.

Status as of 2026-06-13 (correctness pass, in progress). Expected-row-count assertion shipped; the 7 TPC-DS vacuous-bug queries root-caused as distribution-fidelity (NOT plumbing). The two-engine compare scores agreement-on-nothing as "Match", so a generated-data bug hides as a vacuous pass. Now the compare harness asserts against a DuckDB canonical-count manifest (benchmarks/expected/canonical_rows_duckdb.json, 121 tpch+tpcds SF1 counts from CALL dbgen/dsdgen): a 0/0 result is ExpectedEmpty (PASS) when canonical==0, VacuousBug (FAIL) when canonical>0, and unchanged Vacuous when the manifest has no entry (classify_status in comparison.rs, 5 unit tests; BENCH_EXPECTED_ROWS overrides path, graceful no-op if absent; scale key sf{N}_official_rows so only SF1 asserts today). Definitive vacuous verdict for TPC-DS SF1: q17 is legitimately empty (DuckDB's own dsdgen returns 0 -> now asserted PASS), q08/q24/q25/q41/q54/q85/q91 are real bugs (official 5/1/1/4/1/1/2, ours 0). Root-caused: the FK/cardinality machinery is CORRECT (web_returns 100% join web_sales on item+order; catalog_returns carry returning_customer+call_center; refunded/returning cdemo/reason/addr populated; customer_demographics is the right 1.92M cross-product). The gap is dimension distribution fidelity -- dsdgen uses WEIGHTED .dst attribute/value-range distributions, ours are uniform, so the narrow multi-predicate intersections these queries probe come up empty (q41 per-manufacturer color×units×size combos; q85/q91 paired refunded/returning marital×education + price ranges; q08 zip ranges; q54/q25 cross-channel same-customer-same-item). Materially bigger than the tpch fix: reproduce dsdgen .dst weights across item-attribute / demographics-linkage / price-skew / zip generators, iterating regenerate->DuckDB-validate per query. Branch fix/bench-tpcds-correctness. NEXT (this pass): (1) the 7 distribution-fidelity fixes query-by-query vs DuckDB; (2) validate non-oracle vacuous (tpcbb q07, clickbench q27/q28 = fixed real hits dataset, tpch SF10 q11) vs their sources + extend the manifest to SF10/other scales; (3) then dual-engine EXPLAIN on every remaining failure/mismatch/slower with analytics.

Status as of 2026-06-13. Generator fidelity (TPC-H/SSB derived fields) + #131 scan parallelism FIXED. A fresh DuckDB-oracle sweep (validate-generator-tpcds.py vs CALL dbgen/dsdgen) caught a real TPC-H generator bug the two-engine compare could never see: q01 returned 6 (returnflag, linestatus) groups vs the spec's 4. Root cause was a bug class -- fields the spec DERIVES were drawn independent-uniform-random: returnflag/linestatus (now derived from the 1995-06-17 cutoff), the ship/commit/receipt date chain (receipt could precede ship), l_extendedprice baked in the discount, p_retailprice 100x too large, and lineitem's supplier formula off-by-one vs partsupp (25% of (partkey,suppkey) pairs failed the q09/q11 join). Fixed all five plus the inert SSB commitdate sibling; verified vs DuckDB at SF0.1+SF1 (q01 6->4, 0 missing partsupp pairs, receipt>ship always), 3 regression tests added. TPC-DS is healthy (the 21 SF0.1 vacuous queries are scale-noise: disjoint sets across scales, official counts 1-5). Branch fix/bench-generator-tpch-ssb-fidelity. Then #131: the SSB/TPC-H scan gap was a single-file fact table decoded on one thread (profile: RepartitionExec(RoundRobinBatch(11)) above the lineorder scan = fetch_time=910ms of a 969ms q4.1, files_matched=1). The to_arrow reader already splits a FileScanTask into byte-range subtasks (row groups by midpoint) and decodes them on spawned tasks, but the 128MB default split target left a 151MB file whole and TableScan had no knob. Fix threads task_split_target_size through TableScanBuilder->TableScan->to_arrow_with_metrics; IcebergScanExec sets 32MB. Scan stays UnknownPartitioning(1) so the q72 17s->100s CoalescePartitions regression (from the old target_partitions wiring) cannot recur. SF1: SSB 0.64x->1.45x (9.2s->5.5s, beats Trino), TPC-DS -14%, TPC-H -29%, q72 canary 797->768ms (no regression). SF10: TPC-H -22% + the q10 300s timeout resolved; SSB -15% (still ~2x behind -- at SF10 lineorder is already 4 files so the marginal win is smaller and the bottleneck shifts to I/O bandwidth). Branch perf/iceberg-scan-parallel-rowgroups. Blogs: docs/blog/2026-06-13-the-data-was-wrong-all-along.md, docs/blog/2026-06-13-one-file-one-thread.md. The >65536 IN-list cap is NOT the SSB lever (asked): SSB part filters are ~32K keys (under cap) and the part join is outermost so its dynamic filter never reaches the lineorder scan -- join-order, not threshold. NEXT: push SF10 SSB further (raise scan decode concurrency past num_cpus / async range I/O like Comet's parallel.io); then the standing list (worker scan backpressure, sorted loads, per-shape routing, DF upstream filing).

Status as of 2026-06-12 (late night). Key-set dynamic filters land pre-decode (feat/runtime-filter-keyset, stacked on feat/scan-profile-detail): SSB q2.2 decodes 98K rows instead of 60M. Three changes, no wire protocol touched. (1) New [query] runtime_filter_inlist_max_values = 65536 / runtime_filter_inlist_max_size = "4MB" raise DataFusion's IN-list materialization thresholds (its 150-value default sat below every SSB dimension filter, so scans got an opaque hash-table probe and useless min/max bounds); with real InListExpr snapshots, the EXISTING converters carry the key set into iceberg Predicate::is_in (single-node Tier-1) and into predicate_proto (workers) untouched. (2) New [catalog.runtime_filters] wait_ms = 100 bounds-waits at scan-stream open, after manifest planning, so sampled filters are sealed rather than lit(true) placeholders (bounded poll only; wait_complete() deadlocks probe-side scans). (3) The first validation run hit a landmine the 150-value default had been hiding: the vendored reader's PredicateConverter::r#in evaluated membership as an or(eq) loop -- one full-column kernel per list element per batch, 6.5K keys x 60M rows stalled every star query past timeout and starved the I/O runtime into S3 retry spirals. Fixed with a typed FnvHashSet membership evaluator (Int32/Int64/Date32/Utf8, or-eq fallback for the rest, null rows stay null). Validated on SSB SF10 single-node (debug rig): every lineorder scan now shows rows_decoded == output_rows, rows_filtered_dynamic=0 -- 100% of join selectivity applies inside the parquet reader pre-decode, Trino's exact behavior (its q2.2 ScanFilter: 60M in, 99.84% filtered, 731MB physical input; ours: 98.29K decoded, 765MB scanned). NEXT: (1) level-rig release-build SSB compare for the wall-clock number (expect 42.0s single / 53.6s dist-2w to move toward or past Trino's 28-41s; q4.1/q4.2 part filters carry 139-187K keys and still exceed the 65536 threshold -- evaluate raising it or a bloom path); (2) then the standing list: worker scan backpressure (q23/q37/q72/q82 distributed SF10), sorted bench loads, per-shape routing, DF upstream filing.

Status as of 2026-06-12 (night). Trino-grade scan visibility in query profiles (feat/scan-profile-detail), driven by the SSB structural diagnosis. Trino's EXPLAIN ANALYZE on SSB q2.2/q3.3 settled why SSB is the one suite Trino still wins: identical join order and broadcast strategy, but Trino collects the exact build-side key set (SortedRangeSet, e.g. 6494 point ranges for the brand-filtered part keys) in ~100ms, WAITS for it at split generation, and applies it row-level inside the scan: 99.84-99.99% of lineorder dies pre-join. SQE's range-only dynamic filters are structurally powerless on SSB's uniform FKs (min/max spans the whole key domain), so distributed scans ship all 60M rows and single-node decodes all 60M before the Tier-2 wrapper kills 99.8% post-decode. To make that one-profile-readable, IcebergScanExec now reports: bytes_planned/bytes_scanned (object-store bytes, both reader paths, Drop-flushed so LIMIT-terminated scans still report), rows_prefilter/rows_decoded (RowFilter vs decode kill rates), rows_filtered_dynamic + rows_passed_filter_pending (post-decode wrapper drops; rows that streamed through while a dynamic filter was still the lit(true) placeholder), dynamic_filters_resolved/pending, files_matched, planning_time (split-generation analog). Validated on SSB SF10: q2.2 single-node line reads rows_decoded=59.96M rows_filtered_dynamic=59.86M bytes_scanned=765MB vs Trino's Input: 60M, Filtered: 99.84%, Physical input: 731MB -- same bytes, wrong place to filter. NEXT: (1) build-side key-set/bloom dynamic filters: ship the membership set to workers (exact under threshold, bloom above; predicate_proto only carries range conjuncts today), apply pre-decode in the parquet RowFilter, bounded wait before fact-task open -- expected to move SSB 53.6s/42.0s toward or past Trino's 28-41s; then the prior NEXT list (worker scan backpressure, sorted loads, per-shape routing, DF upstream filing).

Status as of 2026-06-12 (evening). SF10 turned around in one day: parallel parquet decode (!352), level compare rig, greedy memory pool (!353). Morning SF10 numbers showed SQE 3-5x slower than Trino on every scan-bound query; profiles (first SF10 run with query_profile = "all") showed q06 waiting 6.3s of 6.4s on a scan decoding 8.5M rows on ONE core: iceberg-rust's try_buffer_unordered overlaps I/O but serializes decode onto the polling thread. !352 splits >=256MB whole-file parquet tasks into ~128MB byte-range subtasks (midpoint row-group assignment; overlap semantics would have double-read boundary row groups, regression-tested) each decoded on its own spawned runtime task, capped by the existing concurrency semaphore. Then the rig itself: host->Docker port-forward caps at ~96MB/s single / ~163MB/s aggregate while Trino read in-VM at ~320MB/s, so half the gap was the pipe; new tests/compare/sqe-singlenode.toml + compose rig runs both engines in-network with equal envelopes (8 VM CPUs, bounded heaps, 5GB/query). Then q39: failed at 8GB where Trino needs 5GB; NOT memory retention and NOT a cast bug: FairSpillPool hard-caps every registered spillable consumer at pool/N (q39's two CTE pipelines register ~90 -> ~95MB each) and the Partial aggregate cannot emit early because the optimizer derives PartiallySorted from the constant d_moy = 1 and GroupOrderingPartial::emit_to() never advances past a constant key (unfixed on DF main; #20445 only fixed the panic; upstream filing pending). !353: coordinator.memory_pool = "greedy" (default, TrackConsumersPool) / "fair" rollback. q39 21.8s/3864 rows at 8GB (Trino 29.2s). Final SF10 level-rig table (Trino 481): TPC-H single 130.5s / dist-2w 95.5s / Trino 106.4-138.6s (SQE dist WINS); SSB 42.0 / 53.6 / 28.0-41.1 (single-node right for star shapes); TPC-DS 543.9 / 338.3 / 328.4-468.0. q86 "0 rows" was an h2 GoAway transport flake, fixed with a compare retry. NEXT: (1) worker scan backpressure: q23/q37/q72/q82 fail distributed at SF10 when the scan reservation hits 2GB of the 4GB worker pool because parallel decode outruns Flight shipment; (2) make Tier-1 dynamic filters land before fact-task open (Trino waits 1s at split generation) so the single-threaded Tier-2 wrapper stops eating 20s on q09-class queries; (3) sorted bench loads (files_pruned_minmax=0 everywhere today); (4) per-shape routing single vs distributed; (5) file the DF upstream issue.

Status as of 2026-06-12. Generator fidelity v3 (DuckDB-validated) + dynamic filter pushdown into distributed scans. The DuckDB oracle (validate-generator-tpcds.py vs CALL dsdgen) proved 16 of the 29 TPC-DS SF0.1 vacuous compare queries were generator gaps, not scale artifacts, and TPC-C was fully broken at fractional scales (scale as i32 = 0 warehouses pinned every FK to a nonexistent w_id=0). Fixed with dsdgen-exact vocabularies and structures: real county list, official categories/classes/colors, (category, class)->brand-base correlation (q63's brand AND class conjunction is unsatisfiable without it), Midway/Williamson/TN stores, the deterministic 7200-row household_demographics cross product, log-uniform item prices, weekly half-item inventory snapshots, scale-aware web_page/warehouse null stripes. Validator failures 17 -> 5 at sf0.1 / 7 at sf1, every survivor a <=5-official-row correlation query (q04/q11/q17/q39/q74 sf0.1; q08/q24/q25/q41/q54/q85/q91 sf1 — these need cross-year/cross-channel/zip-overlap correlation machinery). TPC-DS SF1 vacuous 19 -> 8, TPC-C 2/8 -> 8/8 both scales. The one row-content DIFF across all 7 suites (q75, 57 vs 55 rows) is Trino rounding its DECIMAL(17,2) division to scale 2 (ratios 0.8983/0.8984 -> 0.90, dropped from < 0.9); DuckDB returns SQE's exact rows. Perf side, the new query profiles showed forced-distribution fact scans shipped EVERYTHING (SSB SF1 lineorder: 6M rows / 115MB per query): DistributedScanExec never received dynamic join filters (no pushdown hooks, and try_distribute swaps the scan node AFTER the optimizer deposited them on the Iceberg scan). Now it accepts them, carries them across the swap, waits up to 100ms for build sides (Trino-style), snapshots, converts to logical Exprs, and ANDs them into the ticket's predicate_proto (no wire change; worker RowFilter applies them) — SSB q3.3 ships 449 rows instead of 6M. find_iceberg_scan also picks the LARGEST scan by stats instead of first-DFS (q4.x was distributing a dimension while lineorder ran locally). tpce trade_result capped at LIMIT 1000 (21.6M-row result OOM-killed the Trino compare container twice). MRs: fix/bench-generator-fidelity-v3, perf/distributed-dynamic-filters. NEXT: correlation machinery for the 12 remaining vacuous queries (store-return -> catalog-repurchase chains, cross-year repeat customers, zip/store overlap), and per-query SSB SF1 parity under the forced-distribution rig (dispatch+S3 floor still loses to Trino's in-memory dims on sub-second queries).

Status as of 2026-06-11. Passive per-query profiling shipped ([query] query_profile = "off" | "slow" | "all"). DataFusion populates per-operator metrics during normal execution; we used to throw them away when the stream finished, which is part of why the q72 hunt took five days (per-operator timings were only visible by re-running under EXPLAIN ANALYZE). The StreamFinalizer now renders DisplayableExecutionPlan::with_metrics on success AND on error (failures always profile when the mode is not off), prefixed with elapsed/rows and an unpushed_scans=N full-scan flag (scan nodes displaying predicate=[]), capped at 64 KiB, logged once under the query_profile target, and stored on the QueryRecord (surfaced on /api/v1/queries/{id} detail only). DistributedScanExec now implements metrics() with BaselineMetrics around its stream so the profile shows real rows/elapsed on that row instead of blanks. Benchmark sweeps at SF1 now leave per-operator evidence for slow queries without interactive re-runs.

Status as of 2026-06-11. Differential testing made honest: generator fidelity + fail-fast + VACUOUS status (!333), idle-timeout now errors like Trino (!334), deltalake-core pinned (!335), stall tracked (#261). The SF1 full-suite compare looked healthy (133/134 "Match") but most of it validated nothing: the sqe-bench generators produced data the official queries cannot select (TPC-DS fact-table *_date_sk columns were 100% NULL because row builders emitted Date values into Int32 columns and cols_to_arrays silently coerced the mismatch to None; TPC-H p_type was a 15-of-150 hardcoded subset missing q08's literal and every customer had orders so q22's NOT EXISTS was empty; SSB brands were 3-digit instead of dbgen's MFGR#mcnn and cities were not the %-9.9s%d format q3.3/q3.4 probe), so both engines agreed on empty and the harness scored empty-vs-empty as Match. !333 fixes all four generators, makes the type-mismatch coercion a panic (the sweep test now generates every table of all 7 benchmarks fail-fast), and adds a Vacuous compare status so agreement-on-nothing is visible in every report. Validation on a fresh stack: TPC-H 22/22 with 0 vacuous, SSB 12/13 + 1 vacuous, TPC-DS 70/99 matched + 29 vacuous with 0 diffs and 0 failures; value-validated coverage went from ~54/134 to 104/134 queries. !334 fixes the sibling silent failure in the engine: the issue #75 stream idle-timeout guard ended a stalled query as a clean empty result; it now surfaces Query aborted: produced no results for 300s (idle timeout) through Flight and marks the query Failed (Trino's EXCEEDED_TIME_LIMIT semantics), proven in the wild when the intermittent distributed stall hit SSB q1.1 mid-suite. The stall itself (3 occurrences across 3 suites, ~1 per 100 sequential distributed queries on a long-lived stack, passes instantly in isolation, lost-wakeup suspected) is filed as #261 with a full evidence dossier. !335 pins deltalake-core = "=0.32.1": delta-rs deleted delta_datafusion::DeltaTableProvider in patch release 0.32.4, and the floating spec let a lockfile regeneration break the optional delta feature invisibly until an --all-features build. Remaining TPC-DS vacuous queries mostly need correlated sales-to-returns rows (q01/q17/q24-class); future generator fidelity work. NEXT: root-cause #261 (instrument the distributed scan/stream wakeup path), and migrate read_delta.rs to 0.32.4's TableProviderBuilder as a deliberate bump.

Status as of 2026-06-10. Distributed projection pushdown restored (follow-up to !327). The !327 "number of columns(2) must match number of fields(16)" failure was root-caused to the WORKER, not the coordinator schema contract: the streaming scan path (5bc4c02, 2026-05-15) returned builder.schema() (full parquet file schema) from open_parquet_stream while the built stream emits projected batches, so the Flight encoder advertised 16 fields and shipped 2-column batches; the coordinator's Flight decode failed before reassembly ran. The old buffering path used batches[0].schema() (projected), which is why the April distributed baseline was 22/22 WITH projection pushdown. Fix: worker takes the schema from the built ParquetRecordBatchStream; coordinator re-populates projected_columns/projected_field_ids (tested scan_task_projection() helper, all-or-nothing field IDs); reassemble_worker_batch hardened (equal width now also requires positional name equality, by-name reorder for parquet FILE-order batches, positional accept for renamed columns under field-ID projection, fail-closed otherwise). Validated vs Trino 465 on TPC-H SF0.1 (single worker, forced distribution): 22/22 matched on every run; median total 4794ms -> 1567ms (3.1x), scan-heavy q01/q06/q14/q15/q17/q19 subtotal 1800ms -> 415ms (4.3x; q06 9.0x, q14 10.3x). Workers now read only projected columns from S3 and ship only those over Flight. MR: fix/restore-projection-pushdown.

Status as of 2026-06-01 (updated). Web UI metrics dashboard extended: sparklines, tooltip, histogram legend, rows-out and latency time series. Each of the six Activity stat cards now renders a 36px sparkline of its 15-min bucket series (Total/Finished/Failed/Running in blue, Failed in red, Avg Latency and Rows Out in blue). The Query activity histogram now shows a legend (Completed in blue, Failed in red) and a "15-min buckets, last 12h" caption. All charts are hoverable: a single shared tooltip (#tip, event-delegated from document, lives outside the rewritten #overview subtree) shows HH:MM + value on mouseover of any bar, sparkline column, or gauge sparkline segment. Backend: MetricsSample extended with total_output_rows, finished_queries, exec_ms_sum; HistoryBucket replaced queriesCompleted/queriesFailed with total, finished, failed, rowsOut, avgLatencyMs. Histogram bars now stack finished + failed (previously double-counted failures via total+failed). New unit tests: bucket_samples_avg_latency_zero_when_no_finished, extended bucket_samples_two_buckets_delta and bucket_samples_clamps_negative_delta. All 26 affected tests pass; clippy clean.

Status as of 2026-06-01. Read-only web UI shipped. A network-gated ops dashboard is embedded in the coordinator's existing health server (metrics_port + 1): / serves a no-build single-page dashboard, and /api/v1/queries, /api/v1/queries/{id}, /api/v1/workers expose QueryTracker / WorkerRegistry state as JSON (Ballista/Trino-style). No login (protect at the network layer); toggle with [metrics] web_ui (default on). Spec/plan: docs/superpowers/specs/2026-06-01-sqe-web-ui-design.md, docs/superpowers/plans/2026-06-01-sqe-web-ui.md. NEXT (phase 2): an interactive SQL console + cancel + OIDC login on the UI.

Status as of 2026-05-31. Ballista wound down; bespoke distributed execution is the only engine. After driving the ballista opt-in path to functional parity on the common path, we measured it honestly and removed it. It was ~2.2x slower where it completed (TPC-H), could not finish the TPC-DS analytical core (an upstream datafusion-proto aggregate-serialization bug plus an executor-eviction-on-task-error bug), and its scheduler is less capable than our WeightedScheduler (Ballista 53 has no consistent-hash affinity, no scan locality, no straggler handling). The sqe-ballista crate, the [query] engine switch, and all integration wiring are removed; the ADBC unpadded-base64 Flight handshake fix (a real dbt-sqe connectivity fix found during the work) is kept. Decision, architecture notes, and borrowable ideas: docs/ballista-evaluation-learnings.md. Full historical detail (design, phases, divergence ledger D1-D13): docs/archive/ballista-evaluation/. NEXT: an SQE web UI (queries/tasks/workers, Ballista- and Trino-style) over the existing QueryTracker / FragmentInfo / WorkerRegistry state.

Status as of 2026-05-30 (SUPERSEDED by 2026-05-31). Ballista parity gate, criterion #1 (per-user bearer passthrough) code-complete. The user reframed the ballista relationship: SQE is the lakehouse SQL server (protocols, targets, speed, policy SQL are ours); ballista is narrowed to the distributed scheduler/task-management brain. Migration contract = Option 3 with parity-gated retirement (bespoke stays default, ballista opt-in, retire only at functional AND speed parity, functional blockers first). Spec: docs/archive/ballista-evaluation/2026-05-28-sqe-on-ballista-cutover-design.md ("Migration contract & parity gate"). First gate closed: the user bearer now threads through the PLAN (logical codec stamps it -> scheduler attaches to provider -> IcebergScanExec -> EncodedSqeScan -> executor mints a per-(user,table) FileIO, cached single-flight to keep D4's no-per-task-round-trip invariant), bypassing ballista's ConfigExtension propagation (D8). Trust model preserved (only the bearer travels). Unit-verified (wire round-trip, full-bearer cache keying, no-bearer fallback); per-user isolation is NOT E2E-verifiable on the single-principal dev stack. Plan: docs/archive/ballista-evaluation/2026-05-30-ballista-bearer-passthrough.md. NEXT on the gate: criterion #2 (policy-rewritten mask/row-filter plans survive the codec). Plus the standing E2E ballista-mode no-regression smoke (single-principal) and the multi-node speed gate (criterion #5, task 5b).

Status as of 2026-05-15. Four-wave audit-fix campaign: 130 issues filed, 19 themed MRs merged, ~110 issues closed. A separate audit pass produced 130 GitLab issues. We ran them through four sequential waves of parallel agents (MR !195 to !213). Wave 1 (4 MRs): critical policy correctness, tests infrastructure, auth hardening, Trino/Flight protocol completeness. Wave 2 (5 MRs): worker-side auth, SecretString migration, scheduler isolation, write-path correctness, policy on DELETE/UPDATE. Wave 3 (5 MRs in two batches): async hygiene, auth/session config, code-quality refactor, caching/perf, build hygiene + observability. Wave 4 (5 MRs in two batches): remaining correctness, test coverage, hygiene tail, operator tunability, type-safety polish. Total 108 commits. Five rebases needed: four structural (config.rs anchor collisions), one semantic (SecretString migration did not thread through with_worker_secret / start_credential_refresh_task builder signatures, caught at the next agent's first build, patched in MR !205). The only remaining audit issue is #2 (fix/bearer-concurrency-race, existing WIP branch). Blog write-up: docs/blog/2026-05-15-nineteen-mrs-four-waves.md.

Highlights from the campaign: SecretString newtype + sealed Session credentials, per-user FairSpillPool + query_semaphore, tonic channel pool, partition fan-out on IcebergScanExec + IncrementalScanExec, streaming worker output, mid-stream error termination, per-user TableMetadataCache keying (stops vended-cred leaks), field-ID parquet projection, time-travel provider scoping, MERGE namespace fix, CatalogCommitConflict retry, Drop-guard S3 cleanup on write cancel, HMAC mask key (opt-in via policy.mask_key), worker do_get + refresh_credentials auth gate, full TrinoStats + TrinoError fields, X-Trino-Set-* response headers, Flight SQL GetSqlInfo expansion, prepared-statement bind values, do_get_tables filter args, info_schema SQL-standard type names, AccessControlBackend + PolicyEngine enums, [workspace.package] + MSRV, default features flipped to rest-only with full-backends umbrella + Dockerfile.full, tonic HTTP/2 window + keepalive tuning, OPA circuit breaker + metrics, catalog roundtrip histogram, error_code label on sqe_query_count_total, audit tables_touched, per-worker WorkerLoadTracker reservation, idle-timeout for tracked streams, supervised tokio::spawn helper, constant-time API-key compare, base64 varbinary in Trino responses, decimal(20,0) for UInt64.

Status as of 2026-05-04. SQL surface lift: JSON + TIME + JDBC v3 live test, MoR confirmed already shipped (branch feat/iceberg-loader-s3tables). The doc audit revealed three "missing features" that turned out to already be implemented; the code changes ship the two that genuinely needed wiring. Score 163/189 (86.2%) -> 164/189 (86.8%).

• sqe:jdbc-catalog:v3 flips partial -> full. Added jdbc_postgres_v3_table_format_version_roundtrip in crates/sqe-catalog/tests/backends_integration.rs::sql_postgres: creates a format-version=3 table through the JDBC backend, drops the in-memory handle, reloads, and asserts the metadata still reports V3. Closes the engine-wiring caveat that has been on the cell since Phase L.
• JSON logical type shipped. SqlType::JSON -> Utf8 in sql_type_to_arrow. CAST(json_col AS BIGINT|VARCHAR|DOUBLE) rides DataFusion's built-in coercion; JSON extraction stays available via the existing json_extract / json_get_* UDFs. Trino-compat doc flips one ❌ to ✅ in the JSON section.
• TIME / TIME(p) shipped. Maps to Time64(Microsecond) for precisions 0..=6 (Iceberg's time is microsecond-only). localtime() now returns Time64 (was incorrectly returning Timestamp). extract_component handles Time64Microsecond + Time64Nanosecond arrays/scalars; hour() / minute() / second() work on TIME columns; year() / month() / day() raise a clear plan error per Trino spec. TIME WITH TIME ZONE rejects with NotImplemented pointing at TIMESTAMP WITH TIME ZONE.
• MoR DELETE was already wired. The Trino-compat doc claimed "MoR feasible but SQE uses CoW only". Reading handle_delete_dispatch shows that statement was stale: it has read write.delete.mode from table properties since Phase O+, routing to position-delete (no PK) or equality-delete (with PK) writers. Doc now reflects reality.
• Repaired tests broken by the loader refactor. Commit 378bd9f deleted crates/sqe-catalog/src/backends/{glue,hms,sql}.rs but left tests/backends_integration.rs referencing the removed types. Migrated mod glue and mod hms to the upstream GlueCatalogBuilder / HmsCatalogBuilder directly (same path the loader takes). Replaced mod sql with a builder smoke test for the new vendored iceberg-catalog-sql.
• Spark cross-engine read test (spark_reads_sqe_equality_delete_file) is plain #[test] and self-skips when docker is absent, not #[ignore]. Matrix evidence on sqe:equality-deletes:v2 updated. All 4 maintenance procedures have dedicated live tests; matrix notes on sqe:table-maintenance:v2/v3 updated.

Status as of 2026-04-30. Phase Q + Phase R: Unity OSS live test + bloom-filter footer probe shipped (MR !115 and !116). Phase Q flips sqe:unity-catalog:v2/v3 from partial to full via a read-only smoke against the bundled unity.default.marksheet_uniform table on the unitycatalog/unitycatalog:main-2f2e32d image. Phase R flips sqe:bloom-filters:v2/v3 to full by closing the last evidence gap with a self-contained footer-inspection test (writer_props_emit_bloom_filter_in_parquet_footer) and corrects the misleading "missing worker-side data writer" caveat (no separate worker writer exists). The bench-bloom-on-write negative result is now consolidated in docs/features/runtime-filter-pushdown.md. Score 158/189 (83.6%) -> 162/189 (85.7%). OSS release artifacts (SECURITY.md, .github/ISSUE_TEMPLATE, PULL_REQUEST_TEMPLATE) added; pre-public docs audit pass cleaned up the AWS profile leak in the catalogs blog.

Status as of 2026-04-29. Phase O + Phase P: live catalog matrix integrated (MR !113, branch feat/matrix-phase-o-live-catalogs). Five catalogs now have live integration tests in crates/sqe-catalog/tests/backends_integration.rs: Hive Metastore (apache/hive:standalone-metastore-4.1.0 over Thrift), Project Nessie (ghcr.io/projectnessie/nessie:0.107.5 over Iceberg REST), JDBC Postgres (docker-compose postgres), AWS Glue (real eu-central-1 account), AWS S3 Tables (federated Glue Iceberg REST endpoint with SigV4). Phase P added an aws-sigv4 cargo feature to the vendored iceberg-catalog-rest crate that swaps the OAuth/Bearer authenticator for an AWS SigV4 signer when rest.sigv4-enabled=true. Five matrix cells flip partial -> full (HMS v2/v3, Nessie v3, Glue v2/v3); rest-catalog and aws-glue-catalog cell notes enriched with the SigV4 path. Score 153/189 (81.0%) -> 158/189 (83.6%). Default sqe-catalog build now ships every supported backend compiled in (rest, sql-postgres, hms, glue, hadoop). Engine session-manager wiring gap is the only remaining caveat on non-REST cells; a coordinator built with --features hms/glue/sql can construct the catalogs but the engine still routes SQL through the REST path. The S3 Tables case is unaffected because S3 Tables IS Iceberg REST and rides the existing path.

Status as of 2026-04-28. Runtime filter pushdown into IcebergTableScan integrated (MR !112, branch feat/iceberg-scan-runtime-filter). New iceberg::expr::DynamicPredicate trait + TableScanBuilder::with_dynamic_predicate(...) in the vendored fork, plus an iceberg-datafusion bridge that absorbs DataFusion 53 runtime filters from HashJoinExec build sides and feeds them into the reader's existing row-group / page-index / row-filter pruning paths. TPC-H SF1: 18.4s -> 14.5s (-21.3%, 22/22 match). TPC-H SF10: 163.9s -> 143.6s (-12.4%, q15 RowDiff resolved). Five follow-up fix attempts at the per-task bind cost all reverted; the engineering log lives at docs/features/runtime-filter-pushdown.md. Upstream issue filed at apache/iceberg-rust#2376 with the API ask. Filed as MR !112; bloom-on-write branch (feat/bench-bloom-on-join-keys) deliberately stays unmerged because it regresses by +25.9s when layered on Path B-2 (bloom and runtime filters prune the same row groups, bloom adds eval overhead with no incremental benefit).

Status as of 2026-04-26. Iceberg matrix parity Phase N (partition-evolution) integrated. Matrix score 153/189 (81.0%), up from 151/189 (79.9%) after Phase M, 129/189 (68.3%) after Phase I, 99/189 (52.4%) baseline. Phase N adds ALTER TABLE ADD/DROP/REPLACE PARTITION FIELD end-to-end (pre-parser + classifier + coordinator handler + writer fix for unpartitioned-but-evolved specs); both partition-evolution:v2 and partition-evolution:v3 flip from partial to full. Phase M added PARTITIONED BY (...) for the six standard Iceberg transforms (identity, year, month, day, hour, bucket, truncate, void) with TaskWriter routing. Phase I (V3 path validation) flipped 16 V3 cells: table-creation, write-insert, read-support, copy-on-write, write-merge-update-delete, merge-on-read, position-deletes, equality-deletes, schema-evolution, statistics, cdc-support, time-travel, type-promotion, catalog-integration, polaris, rest-catalog. Root-cause fix: Iceberg REST CreateTableRequest has no dedicated format-version field, so SQE now forwards it through the reserved table property. CREATE TABLE TBLPROPERTIES are forwarded to the catalog and re-emitted via SHOW CREATE TABLE. FOR VERSION AS OF registers snapshot-pinned providers under a writable schema alias. 13/13 V3 e2e tests pass against docker-compose.test.yml. SQE wins 5 of 7 benchmark suites at SF1 vs Trino 465. DataFusion 53. Star-schema join reorder. Broadcast threshold 64MB. Dynamic filter type coercion. GRANT/REVOKE SQL via platform API. Open-source release prep complete. 222/222 queries pass at SF1 (TPC-H 22, TPC-DS 99, SSB 13, TPC-C 17, TPC-E 18, TPC-BB 10, ClickBench 43). Full suite runs in 154.8s. TPC-E SF10: 18/18 pass (trade_result_update_holding 10.9s). TPC-E SF100: 17/18 pass, trade_result_update_holding times out at the 120s harness cap under CoW (MoR path now available as an opt-in for this case). TPC-H SF1000 data generation in 6:23 on 32 cores (lineitem 4:43, 29x speedup vs serial, 91% scaling efficiency, 2.2 GiB peak RSS for 6B rows in flight via the bench-generate-parallel-streaming change). Streaming Flight SQL results path. 8 MiB tokio worker stack. sqe-trino-functions split out of coordinator for faster incremental builds. 1,334+ unit tests, 60/60 integration tests, 13/13 V3 e2e tests. 43/43 security audit findings resolved. Known limitation: q72 (15.5s vs Trino 1.4s, upstream DF#3843). Next: pluggable catalogs (HMS/Glue real implementations), worker-path bloom filters, OSS release.

Monitoring: OPA SPI refactor in Polaris (PR #3999, still draft) will affect Phase 5 OPA integration when it lands — do not implement OPA against Polaris until this stabilises. Remote S3 signing (Iceberg 1.12, not yet released) will affect the pluggable-catalogs design.

---

Step 1: Security and Functional Audit ✅

See AUDIT.md for the full report. Completed 2026-04-08.

Before starting new feature development, audit the current codebase against the design intent. Do this as a structured review, not just a code read.

1a. Security Audit

Area	What to Check	Files
Auth passthrough	Bearer token is never logged, never stored in memory longer than the session	sqe-auth/, sqe-coordinator/src/session.rs
Error messages	No stack traces, internal paths, or policy details leak to client	sqe-coordinator/src/error.rs
Query cancellation	In-flight queries are cleanly cancelled when client disconnects	sqe-coordinator/src/
Token validation	JWT expiry enforced; replay attacks mitigated	sqe-auth/src/
TLS	Flight SQL listener enforces TLS in non-dev mode	sqe-coordinator/src/server.rs
Rate limiting	Missing — not yet implemented (see Step 2)	—
Audit log	Missing — not yet implemented (see Step 2)	—
Config secrets	sqe.toml.example does not contain real credentials	sqe.toml.example

1b. Functional Audit

Area	What to Check	Files
EXPLAIN FULL	Metrics (elapsed_ms, output_rows) match actual query execution	sqe-coordinator/src/explain.rs
fmt_val	All Arrow data types render correctly (Utf8View, UInt32/64, Float32, decimals, dates)	sqe-cli/src/fmt_val.rs
Iceberg scan	Partition pruning is applied; snapshot time-travel works	sqe-catalog/src/
Policy rewriter	Column masks block predicate pushdown; row filters are transparent	sqe-policy/src/
Integration tests	All tests pass against a live Iceberg/S3 stack	make test-integration (local only, #387)
Docker build	docker build completes cleanly using pre-compiled binaries	Dockerfile

1c. Audit Commands

# Static analysis
cargo clippy --all-targets --all-features -- -D warnings

# Tests (unit)
cargo test --all

# Integration tests (brings up its own stack; local only, no CI job -- #387)
make test-integration

# Security advisory scan
cargo audit

# Check for unused dependencies
cargo +nightly udeps --all-targets

---

Step 2: Complete Core Engine Spec ✅ (99/103)

Spec: openspec/changes/sqe-core-engine/tasks.md

Step 2 is effectively complete. All implementation and test tasks are done. Only 4 tasks remain, all blocked on upstream:

Task	Ref	Status
DELETE FROM — CoW rewrite_files	8.4	✅ Done — via RisingWave fork rewrite_files()
MERGE INTO — CoW full-outer-join rewrite	8.5	✅ Done — via RisingWave fork rewrite_files()
Integration test: MERGE INTO	8.13	✅ Done
Integration test: DELETE FROM	8.14	✅ Done

All 103/103 tasks complete. DELETE, UPDATE, and MERGE INTO use Copy-on-Write via the RisingWave iceberg-rust fork's rewrite_files() transaction API.

Completed since last update (2026-03-22): distributed execution (7.6, 7.10, 7.11, 9.5, 9.6, 9.7), predicate pushdown (6.3), Trino pagination + headers (11.3, 11.7), worker metrics (12.3), OTel trace propagation (12.6), sqe-auth unit tests (2.5), Keycloak realm registration (13.3), all integration tests (2.6, 3.10, 3.11, 7.12, 7.13, 8.11, 8.12, 8.15, 8.16, 9.8, 10.5, 11.10, 13.4, 13.5), e2e test script.

---

Step 3: OSS Security Hardening ✅ (51/51)

Spec: openspec/changes/oss-security-hardening/

Step 3 is complete. All vendor-specific identifiers renamed and production security controls implemented.

Completed (2026-03-22): Keycloak → OIDC rename (oidc_password.rs + deprecated re-export), MinIO → generic S3 language, config validation (fail-fast on missing fields + port conflicts), TLS support ([coordinator.tls] with optional mTLS via ca_file), rate limiting (per-user + global via governor), query timeouts (per-role overrides), session lifecycle (idle + absolute timeouts with background sweeper), query cancellation (CancellationToken registry + Flight cancel handler), audit log enhancements (session_id, query_hash, client_ip), error sanitisation (client_message() + debug mode toggle), health endpoints (already existed).

---

Step 3b: Benchmark Suite ✅

Design: docs/superpowers/specs/2026-03-24-sqe-bench-design.md

Benchmark suite is complete. sqe-bench CLI provides generate/load/test pipeline for 6 benchmark suites. The read_parquet() TVF enables zero-copy Parquet → Iceberg loading.

Completed (2026-03-24):

• read_parquet() TVF — local filesystem and S3 with inline credentials; glob patterns; registered on every SessionContext
• sqe-bench generate — Parquet data generation for TPC-H (22q), TPC-DS (99q), SSB (13q), TPC-C (8q), TPC-E (11q), TPC-BB (10q)
• sqe-bench load — CTAS-based table loading via read_parquet(), namespace creation, --clean flag
• sqe-bench test — query runner with correctness validation (PASS/FAIL/DIFF/SKIP/ERROR), Flight SQL + Trino HTTP clients, JSON reports
• Scripts: benchmark-generate-all.sh, benchmark-load.sh, benchmark-test.sh
• Query files and expected results for all benchmarks

First results (TPC-H SF1, Flight SQL): 20/22 PASS, 1 DIFF (decimal precision), 1 SKIP (unsupported feature).

---

Step 4: Pluggable Auth

Plan: docs/superpowers/plans/2026-03-19-pluggable-auth.md Spec: openspec/changes/pluggable-auth/

Replace the single Keycloak ROPC provider with a composable AuthProvider trait chain.

Provider	Credential Detection	Use Case
OidcPasswordProvider	username + password, no eyJ prefix	JDBC/ODBC with OIDC password grant
BearerTokenProvider	password field starts with eyJ	pre-authenticated clients, CI/CD
ApiKeyProvider	password matches sqe_ prefix or configured prefix	scripting, service accounts
AnonymousProvider	no credentials	dev/read-only public data
MtlsProvider	mTLS client certificate	internal service-to-service

Config: [[auth.providers]] array; first-match chain; role mappings via [auth.role_mappings].

---

Step 5: Pluggable Catalogs

Plan: docs/superpowers/plans/2026-03-19-pluggable-catalogs.md Spec: openspec/changes/pluggable-catalogs/

Replace the hard-coded Polaris REST catalog with a CatalogBackend trait.

Backend	Notes
IcebergRestBackend	current default; Polaris, Lakeformation REST, any Iceberg REST
AwsGlueBackend	AWS SDK; IAM auth; read + write
NessieBackend	Project Nessie REST API; branch/tag awareness
HiveMetastoreBackend	Thrift HMS; for legacy Hive warehouse migration
StorageOnlyBackend	Scan base path for metadata/v*.metadata.json; no catalog server required

Multi-cloud storage via object_store: S3 (+ endpoint override for R2/Ceph/Garage), Azure ADLS Gen2/Blob, GCS, local filesystem.

Delta Lake support (delta-rs) as Cargo feature flag delta — Unity Catalog serves both Iceberg and Delta tables.

---

Step 6: Semantic AI Layer

Plan: docs/superpowers/plans/2026-03-19-semantic-ai-layer.md Spec: openspec/changes/semantic-ai-layer/

Four sub-systems that make SQE agent-native and semantically aware.

6a. RDF Triple Store on Iceberg (sqe-semantic)

• Convention: rdf.triples (subject, predicate, object, graph_name) Iceberg table, partitioned by predicate
• SPARQL 1.1 SELECT compiled to DataFusion LogicalPlan via spargebra + rdf-fusion
• SPARQL auto-detected when input starts with SELECT ?, CONSTRUCT, ASK, DESCRIBE
• Ontology time-travel via Iceberg snapshot + FOR SYSTEM_TIME AS OF

6b. Property Graph / ISO GQL (sqe-semantic)

• Convention: graph.nodes (id, labels[], properties json) + graph.edges (src_id, dst_id, label, properties json)
• graphlite embedded ISO GQL engine (ISO 39075:2024)
• Small graphs (<threshold): load into graphlite in-memory, execute GQL, return Arrow
• Large graphs: compile MATCH patterns to DataFusion recursive CTEs

6c. Vector Search (sqe-vector)

• lance + lance-datafusion for Arrow-native vector format on object storage
• LanceScanExec: DataFusion physical plan node reading Lance datasets
• vec_distance(col, query_vec, metric) UDF (cosine, l2, dot)
• embed(text) async UDF: HTTP POST to configurable embedding endpoint; SHA256 cache

6d. AI Agent Interfaces

• CLI-first (primary): sqe query, sqe schema search/describe/relationships/ontology, sqe explore; --output json|arrow|csv|table; --describe flag for self-documentation; piped output auto-selects JSON
• REST/OpenAPI (secondary): axum HTTP server; utoipa generates OpenAPI 3.1; /api/v1/openapi.json LLM-readable
• MCP (tertiary): thin stdio wrapper over REST API; tools generated from OpenAPI spec, not hand-coded
• TypeScript @sqe/client (npm): RestTransport (browser) + FlightTransport (Node.js, @grpc/grpc-js); auto-selects by env

---

Implementation Order Rationale

Step 1: audit               ✅ DONE (AUDIT.md: 1,218 tests, rsa removed, 5 config findings, no critical vulns)
Step 1+: OSS release        ✅ DONE (LICENSE, CONTRIBUTING, deny.toml, cliff.toml, CI pipelines, retro-tags, CHANGELOG, v0.15.0)
Step 2: core engine gaps    ✅ DONE (103/103 — DELETE, UPDATE, MERGE via CoW rewrite_files)
Step 3: security hardening  ✅ DONE (51/51 — TLS, rate limiting, timeouts, cancellation, audit, error sanitisation)
Step 3b: benchmark suite    ✅ DONE (sqe-bench: generate/load/test, 6 benchmarks, read_parquet() TVF, CI scripts)
Step 3c: hardening pass     ✅ DONE (type formatting, Flight SQL DoPut + metadata, clippy, decimal DIFF, token fingerprint)
Step 3d: query history+cache ✅ DONE (system.runtime.queries, in-memory history store, query result cache, config sections)
Step 3e: distributed wiring ✅ DONE (try_distribute in execute_query, fragment tracking, system.runtime.tasks shows workers)
Step 4: pluggable auth      ✅ DONE (11 providers: OIDC, bearer, API key, anonymous, mTLS, token exchange, AWS IAM, device code, auth code, OIDC discovery, chain)
Step 4b: streaming exec A   ✅ DONE (spill-to-disk, late materialization, scan planning, S3 I/O, SortMergeJoin — 21/22 TPC-H SF1 on 512MB)
Step 4c: streaming exec B   ✅ DONE (shuffle, distributed sort/join/aggregate, multi-endpoint Flight SQL, Trino function compat)
Step 4d: adaptive sort+metrics ✅ DONE (adaptive sort stripping, S3/auth/write Prometheus metrics)
Step 7.1: dbt-sqe adapter   ✅ DONE (ADBC Flight SQL, table/view/incremental/seed materializations)
Step 7.3: ALTER TABLE schema ✅ DONE (ADD/DROP/RENAME COLUMN, SET/DROP NOT NULL, type widening)
Step 8: Trino parity        ✅ DONE (compatibility matrix, sqe-bench compare, client testing scaffold, operational comparison)
Step 8b: Trino UDF blitz    ✅ DONE (70+ UDFs + engine features — ~95% SQL coverage)
Step 8c: Iceberg time travel ✅ DONE (FOR SYSTEM_TIME AS OF + 6 metadata TVFs + COMMENT ON + SHOW STATS)
Step 8d: MoR DELETE path    ✅ DONE (PositionDeleteFileWriter + FastAppendAction, alongside existing CoW)
Step 9: streaming + perf    ✅ DONE (streaming CTAS/INSERT, IN-subquery rewrite, safe sort order, --compare-trino benchmarks)
Step 9b: 5-layer caching    ✅ DONE (RestCatalog cache, table metadata cache, manifest cache, SessionContext cache, OAuth token cache — warm query <1ms)
Step 9c: DECIMAL + correctness ✅ DONE (parse_float_as_decimal=true, COUNT(*) crash fix, cache invalidation after DDL/DML, Int64 date returns)
Step 9d: safe defaults       ✅ DONE (sort_mode=partition_only, FairSpillPool fallback, spill_to_disk=true, trust_sort_order=false)
Step 9e: Trino comparison    ✅ DONE (SQE 2.5-8.8x faster than Trino 465 across all 7 suites, 221/222 match)
Step 9f: scale hardening     ✅ DONE (streaming result path, tuple-IN view-lifted semi-join, 8 MiB worker stack, pre-flight port check -- SF1 222/222 pass; TPC-E trade_result streams 21M rows in 8.7s without OOM; CoW DML with IN (subquery) scales to TPC-E SF10 34K tuples without stack overflow via `lift_in_subqueries`)
Step 5: pluggable catalogs  ✅ DONE for catalog backends (Phase O+P, MR !113): HMS, Nessie, JDBC postgres, AWS Glue (SDK path + federated REST), AWS S3 Tables (REST + SigV4), Hadoop storage-only -- all live-tested. Engine session-manager wiring (Section 11 of pluggable-catalogs/tasks.md) deferred to a follow-up phase; Delta Lake + Azure + GCS deferred to a separate multi-cloud-storage change.
Step 5c: dynamic Polaris catalog discovery ✅ DONE: `[query] catalog_discovery = "polaris-auto"` lazily resolves an undeclared Polaris warehouse at query time using the caller's bearer (same SqeCatalogProvider path; per-user session scoping; unauthorized/nonexistent -> "unknown catalog", no leak). Default stays `static`. Live-tested (lazy hit / miss / static / in-session reuse). Also fixed a latent bug: REST_CATALOG_CACHE now keys on warehouse (same-URL warehouses no longer collide). Spec/plan in docs/superpowers.
Step 9g: SF100 CoW DML scaling (`openspec/changes/cow-dml-parallel-streaming`) -- parallelise per-file rewrite + stream writes + drop double-WHERE; unblocks SF100 `trade_result_update_holding` (currently 120s timeout) and other super-linear UPDATEs (settlement 24x, executor 16x, status 13x for 10x data) <- NEXT
Step 6: semantic layer      (new crates; fully additive; no existing code broken)

Step 9g (cow-dml-parallel-streaming) is the immediate SF100 unblock. Step 5 (pluggable catalogs) follows. Step 6 is independent and fully additive.

Upstream watch list (refreshed 2026-04-29):

Resolved since the last refresh:

• ★ risingwavelabs/iceberg-rust caught up to DataFusion 53. Commit fb290e4c9 on the fork's main branch (2026-04-15) merges PR #148, which lands DF 53 + Arrow 58. SQE has been carrying a downstream rebase since Phase F; we can now align with the upstream fork on its next vendor refresh.

Blocking matrix v3 cells, still open:

• apache/iceberg-rust#2188 (Variant) — open, in active review, merge conflicts. Likely lands within weeks. Unblocks variant-type:v3.
• apache/arrow-rs#9790 (BorrowedShreddingState refactor) — opened 2026-04-22, no traction yet. Parent shredded variant work has effectively landed in arrow-rs; this is cleanup. Worth re-checking what arrow-rs version SQE pins. shredded-variant:v3 may already be partly reachable.
• apache/datafusion#12644 (User-defined types) — open since 2024-09-27. Long-running design discussion (geoarrow extension types). No merge in sight; geometry-type:v3 will not unblock soon via DataFusion proper. Practical path is to ride on arrow-rs extension-type metadata above DataFusion.
• Apache Iceberg V3 Java spec activity — heavy traffic on variant (#15385 predicate pushdown, #16133 row-group skip, #14297 shredded write) and lineage (#15776 ORC _row_id); geometry stalled (#12347 since 2025-09). V3 is still landing pieces; the multi-arg-transforms:v3, vector-type:v3, lineage:v3 cells track that progress.

SQE-filed, no upstream traction yet:

• apache/iceberg-rust#2376 (DynamicPredicate API) — SQE filed this; latest comment 2026-04-28 sharpens the cache-layer API ask (is_sealed() / generation()). MR !112 already shipped Path B-2 downstream so SQE is unblocked; the issue tracks getting the cache helper accepted upstream. 2026-06-15: this ask is now concrete. The probe-scan Tier-2 wrapper called DynamicFilterPhysicalExpr::current() once per batch, and for a partitioned-join CASE-of-IN-lists (~300K nodes) current() rebuilds the whole tree via transform_up (~10ms/call), making TPC-H q12/q17/q10 SF10 run 160-300s. Worked around downstream by caching the first sealed snapshot per scan (MR !371, iceberg_scan.rs): q12 161s->2.7s, q17 176s->7.1s, q10 300s-FAIL->3.3s, SSB also faster, default threshold unchanged. A generation()/is_sealed() cache hook would make the snapshot refresh precise instead of "cache the first sealed value"; see The Filter That Rebuilt Itself.

Affecting older watchlist items:

• apache/datafusion#21570 (ROLLUP empty GROUP BY) — open, an assignee took it on 2026-04-12 and committed to a PR. Should land in 1-2 release cycles. Still causes 6 TPC-DS DIFF results.
• apache/datafusion#20746 (MERGE INTO) — open umbrella issue, no in-flight PR. Don't expect MERGE in DataFusion soon; SQE keeps its CoW MERGE path.
• DataFusion IN (subquery) on MemTable-referenced columns — no specific upstream issue; closest open work (#14554, #15046) is stale. SQE's lift_in_subqueries workaround stays.

Pre-existing items (no change):

• iceberg-rust MoR (Epic #2186, Q3 2026) — could replace CoW DELETE/MERGE with a more efficient position-delete approach; matters for the longer-term follow-up to cow-dml-parallel-streaming.
• Polaris OPA SPI refactor (PR #3999) — must stabilise before Phase 5 OPA integration.
• Remote S3 signing (Iceberg 1.12) — will require revisiting credential vending in pluggable-catalogs once it ships.