ci: pin vgi-rpc-java past the LargeBinary fix, and re-pin VGI_REF

Two unrelated CI failures, both stale pins rather than Java bugs.

PlanClientRoundTripTest failed in CI and passed locally on the same JDK and
commit. Line 262 is `vgi.init(...)`, redeeming a plan split, and
InitRequest.split_tokens is a List<byte[]> annotated large_binary. vgi-rpc-java
v0.23.0's Marshalling.writeListElement has no LargeBinary case, so it hits
`default -> throw new IllegalArgumentException("unsupported list element type:
LargeBinary")`. Locally settings.gradle.kts composite-includes a checkout
sitting two commits past v0.23.1, one of which is 306e182 "support LargeBinary
list elements" — whose own message names this test. CI checked out v0.23.0 and
never saw it. Reproduced byte-for-byte by pointing the composite at v0.23.0.

Bumping to the newest TAG would not have fixed it: v0.23.1 is a0309fe, which
predates 306e182, and upstream tags stop there. So both workflows pin the
untagged SHA — the minimal revision carrying the fix.

Separately, all three integration lanes failed on cache/secret_ineligible.test
and macro/macros.test, both asserting duckdb_logs events. Those assertions
landed upstream on 2026-08-26 together with the C++ client code that emits them
(27b6ded, a0eb66b, f10b9bc). The lane took VGI_REF: main so it got the TESTS,
while the extension comes from `INSTALL vgi FROM community`, published from
40891ac — before all three. The binary simply doesn't emit those events. Both
files pass locally against a from-source extension, which clears the Java
worker: nothing is missing here. ci/README.md predicted this exact failure mode;
`VGI_REF: main` defeated it. Re-pinned to 40891ac, the commit the last green run
used, restoring the SHA pin the file originally had.

Bump VGI_REF by hand when the community channel republishes vgi; the
authoritative value is repo.ref in that channel's description.yml.

KNOWN ISSUE this exposed, not fixed here: vgi/build.gradle.kts declares
farm.query:vgirpc:0.23.0, and no PUBLISHED vgirpc carries the LargeBinary fix.
CI's composite substitution hides it, but released vgi-java 0.27.0 therefore
cannot redeem a table_function_plan split for a downstream JVM consumer. The
remedy is a vgi-rpc-java release >= 306e182, then bumping the declared
dependency and these two pins together.

Author: rustyconover

.github/workflows/ci.yml

@@ -22,12 +22,26 @@ concurrency:
   cancel-in-progress: true
 
 env:
-  # The vgi-rpc-java tag built from source via the composite include. It MUST
-  # match the `farm.query:vgirpc` version declared in vgi/build.gradle.kts:
-  # the composite substitutes that coordinate, so this ref — not the declared
-  # version -- is what :vgi:compileJava actually compiles against here, and a
-  # stale pin fails as `cannot find symbol` on whatever the newer vgirpc added.
-  VGI_RPC_JAVA_REF: v0.23.0
+  # The vgi-rpc-java revision built from source via the composite include. The
+  # composite substitutes the `farm.query:vgirpc` coordinate declared in
+  # vgi/build.gradle.kts, so this ref -- not the declared version -- is what
+  # :vgi:compileJava actually compiles and runs against here. A stale pin fails
+  # as `cannot find symbol` on whatever the newer vgirpc added, or, as below, as
+  # a runtime marshalling error.
+  #
+  # This is an UNTAGGED SHA on vgi-rpc-java main, not a release, because no
+  # release contains the fix these sources need. v0.23.0 and v0.23.1 both ship a
+  # `writeListElement` whose ArrowType switch has no LargeBinary case, so
+  # serializing InitRequest.split_tokens (a large_binary List<byte[]>) throws
+  # `unsupported list element type: LargeBinary` -- which is exactly how
+  # PlanClientRoundTripTest.plansAndRedeemsEverySplitAcrossPagination fails when
+  # it redeems a table_function_plan split. Fixed by vgi-rpc-java 306e182.
+  # Move this back to a tag once a vgirpc release >= that commit is cut, and
+  # bump `farm.query:vgirpc` in vgi/build.gradle.kts to it in the same change --
+  # until then the PUBLISHED vgi-java cannot redeem a split, because it resolves
+  # the published vgirpc 0.23.x. Keep in step with the identical pin in
+  # integration.yml.
+  VGI_RPC_JAVA_REF: 306e182e072a19fa753488241f141215c3b46e56
 
 jobs:
   test:

.github/workflows/integration.yml

@@ -25,21 +25,38 @@ concurrency:
   cancel-in-progress: true
 
 env:
-  # The Query-farm/vgi commit whose test/sql/integration suite we run. Pinned
-  # for reproducibility; bump deliberately and re-validate against the then-current
-  # community extension (see ci/README.md).
-  VGI_REF: main
+  # The Query-farm/vgi commit whose test/sql/integration suite we run.
+  #
+  # This MUST be the same commit the community channel's vgi extension is built
+  # from -- see `repo.ref` in Query-farm-haybarn/haybarn-community-extensions
+  # extensions/vgi/description.yml -- because the extension is installed live
+  # (`INSTALL vgi FROM community`) and is NOT pinned here. The .test files and
+  # the extension binary are two halves of one client; skewing them tests a
+  # client against another client's tests.
+  #
+  # It was `main` (floating), which is how the 2026-08-26 upstream trio
+  # 27b6ded / f10b9bc / a0eb66b broke all three lanes: those commits added the
+  # `reason=secret_dependent` result-cache assertion to
+  # cache/secret_ineligible.test and the shared callable-discovery RPC-count
+  # assertion to macro/macros.test, TOGETHER WITH the C++ client code that
+  # satisfies them. The suite picked up the tests; the published extension did
+  # not have the code. Neither is a Java worker defect -- both files pass
+  # against a from-source extension built at vgi main with this same worker.
+  #
+  # Bump deliberately, in lockstep with the community republish, and
+  # re-validate locally (see ci/README.md "Version pins").
+  VGI_REF: 40891ac19b84ad8d545e6d3853104cebfaef8cb3
   # The Haybarn release providing the prebuilt haybarn-unittest binary is
   # resolved to the latest release at run time (see the "Resolve latest haybarn
   # release" step) so the unittest host stays ABI-compatible with the rebuilt
   # community-published vgi extension, mirroring the vgi-typescript CI.
-  # The vgi-rpc-java tag built from source (composite include) for the worker's
-  # HTTP features. It MUST match the `farm.query:vgirpc` version declared in
-  # vgi/build.gradle.kts — the composite substitutes that coordinate, so this
-  # ref, not the declared version, is what the worker actually compiles
-  # against, and a stale pin fails as `cannot find symbol`. Keep it in step
-  # with the identical pin in ci.yml.
-  VGI_RPC_JAVA_REF: v0.23.0
+  # The vgi-rpc-java revision built from source (composite include) for the
+  # worker's HTTP features. The composite substitutes the `farm.query:vgirpc`
+  # coordinate declared in vgi/build.gradle.kts, so this ref, not the declared
+  # version, is what the worker actually compiles against. An untagged SHA
+  # rather than a tag: see the pin's full comment in ci.yml, and keep the two
+  # in step.
+  VGI_RPC_JAVA_REF: 306e182e072a19fa753488241f141215c3b46e56
 
 jobs:
   integration:

ci/README.md

@@ -114,9 +114,26 @@ Two pins live in the workflow's `env:` block:
 
 | Pin | What | Why |
 |-----|------|-----|
-| `VGI_REF` | the `Query-farm/vgi` commit supplying the `.test` files | reproducibility — bump deliberately |
+| `VGI_REF` | the `Query-farm/vgi` commit supplying the `.test` files | must equal the commit the community vgi extension is built from — bump deliberately |
+| `VGI_RPC_JAVA_REF` | the `Query-farm/vgi-rpc-java` revision built from source | currently an untagged SHA; no release has the `LargeBinary` list-element fix the sources need |
 | `HAYBARN_RELEASE` | the Haybarn release supplying `haybarn-unittest` | must be ABI-compatible with the community vgi extension (both `v1.5.4`) |
 
+**`VGI_REF` is not free-floating, and setting it to `main` is a bug.** The
+extension half of the client comes from the community channel; the test half
+comes from `VGI_REF`. Skewing them tests one client against another client's
+tests. The authoritative value is `repo.ref` in
+[`Query-farm-haybarn/haybarn-community-extensions`](https://github.com/Query-farm-haybarn/haybarn-community-extensions)
+`extensions/vgi/description.yml`. On 2026-08-27 `VGI_REF: main` broke all three
+lanes exactly this way: upstream `27b6ded` / `f10b9bc` / `a0eb66b` (2026-08-26)
+added the `reason=secret_dependent` assertion to
+`cache/secret_ineligible.test` and the callable-discovery RPC-count assertion
+to `macro/macros.test` *together with the C++ client code that satisfies them*.
+The suite picked up the tests, the published extension did not have the code,
+and both files fail against a Java worker that is in fact correct — they pass
+against a from-source extension built at vgi `main` with the same worker. When
+the community channel republishes, bump `VGI_REF` to the new `repo.ref` and
+re-validate locally.
+
 **The coupling to know about:** the vgi extension is pulled live from the
 community channel (`INSTALL vgi FROM community`), which always serves the
 *currently published* build — it is not version-pinned here. So CI verifies the