Skip to content

feat(plan): estimate a pre-aggregation's groups, and judge the placement - #219

Merged
Hor911 merged 3 commits into
mainfrom
feat/aggregate-placement-gate
Oct 22, 2026
Merged

Hor911 merged 3 commits into
mainfrom
feat/aggregate-placement-gate

Conversation

@Hor911

@Hor911 Hor911 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Item G2a of ADR 0032, which splits G2 in two. The gate's decisive
quantity is how much a pre-grouping would reduce the rows it reads, and nothing estimated it: EstimateRows
returns a GroupAggregate's input rows as an upper bound (join_order.cc), which is exactly what a reduction
test cannot use. So DistinctValues estimates the groups a grouping of some columns over a subtree would build,
from each column's KeyDomain, capped at the subtree's rows — and the gate judges each legal placement on it.
Still no rewrite.

GroupAggregate keys=[customer.c_custkey] COUNT(orders.o_orderkey)
  [groupable below the right of join, 1 key, padding, worth 1500000->149999]

It admits exactly one placement of the 22 — Q13's, at the 10:1 the survey measured by hand — and refuses
every other.

An unknown estimate is a refusal, and the direction is the point. This quantity sits where a small value
argues for a rewrite, so an unknown group count must argue against one. That is the mirror of what #215
recorded, where an unknown domain divides and so had to stop reading as the largest.

Closes #

Type of change

  • feat: new SQL, CLI or engine capability
  • fix: bug fix
  • perf: performance improvement
  • refactor, test, docs, build, ci or chore
  • Breaking change (CLI, output format or semantics); also add the breaking-change label

Verification

$ pixi run ci
100% tests passed out of 2825

$ ANTB1_DIFF_SEED=20261025 pixi run diff-random
DIFF: PASS seed=20261025 queries=2000 failed=0 unsupported=0

$ clang-tidy -p build/tidy --warnings-as-errors='*' src/plan/aggregate_placement.cc src/plan/join_order.cc src/plan/tests/aggregate_placement_test.cc src/plan/explain.cc
clean

$ ASAN_OPTIONS=detect_leaks=0 ./build/ci-asan/bin/antb1_plan_tests
826 tests PASSED
`pixi run asan` cannot run on this host (pre-existing curl_global_init leak at discovery), so the binary was run
directly after generating the ci-asan fixtures.

$ pixi run lint
every hook Passed except shellcheck: 37 pre-existing SC2218 findings in files this PR does not touch.

Not run on this host: pixi run ci-gcc and codeql-build (the gcc environment is linux-64 only).

No performance change: nothing is rewritten. One CLI golden gains the verdict, a one-line diff reviewed before
updating.

Checklist

  • pixi run check passes locally — not ticked: stops at the pre-existing shellcheck findings. Both
    halves run separately and pass.
  • Tests cover the change (unit tests under src/<module>/tests/, or why none are needed)
  • Docs updated where behavior, commands or architecture changed (AGENTS.md, docs/, an ADR), or not needed
  • No ClickBench-derived data is committed (ADR-0006)
  • Nothing derived from TPC-H is committed (ADR-0006)
  • Changes to governance paths (see .github/CODEOWNERS) were agreed with a maintainer

The open item: docs/adr/0032-aggregate-placement.md, which this both extends with G2a and corrects on a
claim of its own
(below). Splitting G2 is recorded there with its reason, as ADR 0029 and 0030 record their
item updates.

The ADR is corrected on Q10, and the gate is why

ADR 0032 counted Q10 among the two queries this decision wins, at 0.945×. That measurement is at a placement
the decision does not describe
— it pre-groups two joins below the aggregate, where the ADR's exchange moves an
aggregate across one join. The placement the single exchange actually reaches, pre-grouping at the join to the
nation dimension by the seven columns condition 3 requires there, measures at SF 100, interleaved and rotated,
answers identical over 23 rows:

Q10 time peak
as written 3.155 s 5,164 MiB
the placement this decision reaches 3.926 s (1.245x slower) 6,819 MiB
the placement the ADR measured, two joins down 2.829 s (0.945x) 4,285 MiB

The gate refuses the reachable one, so nothing is wrong with the gate — what was wrong was counting Q10. On
its own evidence this decision is worth one query of the 22, and the honest case for it is Q13's 0.494× and
14,325 MiB → 1,554 MiB. The Consequences section now says that, and says plainly that this is not a
throughput item: against the 22 it is about 1% of total time.

An iterated exchange along a chain of joins would reach Q10's measured placement. It is not part of this ADR and
I have not proposed it.

Three of the six gate conditions are not decidable in a logical plan

Saying which is more useful than pretending otherwise, so the header does:

  • "Not a superkey" has no metadata behind it — Parquet carries no keys. Its effect falls out of the reduction
    test instead, for a measured reason rather than an asserted one.
  • Condition 5's second half ("a join between the two positions multiplies rows") needs to know whether a
    build is unique on its key, a runtime fact. I implemented the obvious plan-time proxy and it refused Q13:
    that join takes 150,000 rows to 1,500,000 and emits 1,500,000, multiplying nothing at all, while the rewrite
    is still worth 0.494× there — because what shrinks is the build and the final grouping's input, not the
    join's output. The proxy is gone and the reason is in the header.
  • Condition 6 is a property of the physical plan: ADR 0029's filters are attached long after this runs.

Also

A LEFT JOIN carries no row estimate — the binder keeps one only for the joins its order search ranked — so
the gate computes its own by ADR 0022's rule 3, locally. Changing what EstimateRows answers for a join would
move ChooseBuildSide, which is ADR 0026's decision and not this item's. Without this the gate could not judge
Q13 at all.

Tests

A catalog local to these cases, so declaring statistics cannot move another suite's expectations: a fact table
whose key takes ten values, one unique key, one column declaring nothing, and three dimensions carrying
different shares of those keys. Cases for the admitted shape and for each refusal the gate can decide, including
one where two conditions hold at once and the order of the checks decides which is reported.

Two test expectations I wrote turned out to be wrong rather than the code: a unique key against a ten-row
dimension is refused as "below a filter" before the reduction test is reached, which is correct, and isolating
"no reduction" needed a dimension carrying every value of that key.

AI assistance

  • No AI assistance
  • AI-assisted. Tools and what they did: Claude Code (Opus 5) built the estimate, the gate, the tests and the
    ADR update, and took the SF 100 measurements. The sequencing was the useful part: building the gate before the
    rewrite is what surfaced the Q10 correction and killed my own wrong proxy for condition 5, neither of which
    could have shown up as an answer because nothing is rewritten yet.
  • Accountable human (has read and understands the whole diff): @Hor911

Item G2a of ADR 0032, which splits G2 in two. The gate's decisive quantity
is how much a pre-grouping would reduce the rows it reads, and nothing
estimated it: EstimateRows returns a GroupAggregate's input rows as an
upper bound, which is exactly what a reduction test cannot use. So
DistinctValues estimates the groups a grouping of some columns over a
subtree would build, from each column's KeyDomain, capped at the subtree's
rows, and the gate judges each legal placement on that. Still no rewrite.

An unknown estimate is a refusal, and the direction is the point: this
quantity sits where a small value argues for a rewrite, so an unknown group
count must argue against one. That is the mirror of the lesson #215
recorded, where an unknown domain divides and so had to stop reading as the
largest.

It admits exactly one placement of the 22 -- Q13's, as worth 1500000 to
149999, which is the 10:1 the survey measured by hand -- and refuses every
other, including the one worth examining below.

Three of the ADR's six conditions are not decidable in a logical plan and
the header says which. "Not a superkey" has no metadata behind it, since
Parquet carries no keys; its effect falls out of the reduction test
instead. Condition 5's second half needs a build's key uniqueness, a
runtime fact, and the obvious plan-time proxy is wrong: Q13's join takes
150,000 rows to 1,500,000 and emits 1,500,000, multiplying nothing, while
the rewrite is still worth 0.494x there, because what shrinks is the build
and the final grouping's input rather than the join's output. I had
implemented that proxy and it refused Q13. Condition 6 is a property of the
physical plan, attached long after this runs.

A LEFT JOIN carries no row estimate, since the binder keeps one only for
the joins its order search ranked, so the gate computes its own by ADR
0022's rule 3 -- locally, because changing what EstimateRows answers for a
join would move ChooseBuildSide, which is ADR 0026's decision.

The ADR is corrected on Q10. Its 0.945x is at a placement two joins below
the aggregate, which this decision's single exchange does not reach; the
placement it does reach measures 1.245x slower at SF 100, 3.155 s to
3.926 s with peak 5,164 to 6,819 MiB, answers identical. The gate refuses
it, so the gate is right and the ADR's claim was wrong. On its own evidence
this decision is worth one query of the 22.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Hor911
Hor911 requested a review from lll-phill-lll as a code owner October 22, 2026 07:31
@claude

claude Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Claude finished @Hor911's task in 17s —— View job


Claude review: no problems found

I read the full diff against the AGENTS.md Code Review Rules: the DistinctValues estimate, the JoinRows and Judge gate, the EXPLAIN note, the new tests, the CLI golden and the ADR 0032 update. I did not build or run anything, so the PR's own test results are unchecked.

  • Correctness: no P0 or P1 problems.
    • The Int128 arithmetic stays within the project helpers.
    • An unknown estimate gives kUnknown, a refusal, which matches the stated direction.
    • The Judge checks run in a fixed order, and the tests cover that order.
  • Architecture: plan includes only its own join_order.h and common, so no module edge changes. exec does not use io.
  • Tests: the new cases cover the admitted shape and each decidable refusal, on a catalog local to those cases. The one CLI golden change is the verdict note.
  • P2 (performance): DistinctValues walks the whole subtree on every call, and PlacementOf also calls EstimateRows and JoinRows for each placement. That is plan-time only and the plans are small, so it is negligible.
  • "Ask a human first" paths changed: none from the AGENTS.md list. docs/adr/0032-aggregate-placement.md changes, and docs/adr/ is a CODEOWNERS path. Its Q10 correction rests on SF 100 measurements that I could not reproduce, so a maintainer should confirm it.
  • Not reviewed: the SF 100 timings and memory numbers in the PR body and ADR, and the ci-gcc and codeql-build legs, which the PR says it did not run.

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Coverage

Module Lines Floor Branches Floor Status
common 98.07% (203/207) 92.9% 92.44% (110/119) 90.8% ok
sql 99.08% (3244/3274) 97.3% 97.64% (2190/2243) 95.6% ok
plan 95.43% (9096/9532) 93.5% 89.79% (4890/5446) 89.5% ok
io 94.30% (1241/1316) 93.2% 90.24% (721/799) 86.6% ok
exec 97.05% (11194/11534) 96.1% 88.74% (6322/7124) 85.1% ok
engine 95.98% (478/498) 93.0% 88.52% (239/270) 85.1% ok
cli 94.59% (647/684) 93.4% 87.08% (283/325) 86.4% ok
total 96.52% (26103/27045) 90.38% (14755/16326)

Coverage gate: PASS (floors: tools/ci/coverage_thresholds.json; they only go up).

Clang source-based coverage of the hermetic tests over src/<module>/ (module unit tests excluded); N/A: nothing to cover. Reproduce with pixi run coverage (details: build/coverage/report.txt).

Hor911 and others added 2 commits October 22, 2026 11:08
The P2 of the review: DistinctValues walked the whole subtree per call,
and JoinRows called it once per key per side, so a two-key join walked
four overlapping subtrees where two would do. Plan-time only and the
plans are small, so this is tidiness rather than a measurable win.

EachDistinctValues answers for several columns in one walk and
DistinctValues is now its product. The split is also the more honest
primitive: a caller wanting the largest of several domains cannot take a
maximum out of a product, which is exactly what JoinRows needs and why it
was asking column by column in the first place.

Verdicts over the 22 are byte-identical before and after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Hor911
Hor911 merged commit 28dc21b into main Oct 22, 2026
16 checks passed
@Hor911
Hor911 deleted the feat/aggregate-placement-gate branch October 22, 2026 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant