release: 0.24.0 — per-catalog attach options, and a gated ATTACH that answers

`attach/attach_options_required.test` was skipping here: upstream 1f1d7b1 split
the `required` assertions out of the shared attach_options_echo.test behind
VGI_ATTACH_OPTIONS_REQUIRED_WORKER, because the catalog they need existed only
in the Python fixture. This grows the same catalog.

- `Worker.ExtraCatalog` gains a 5th component, `attachOptions`: an auxiliary
  catalog advertises its own specs on its `catalog_catalogs()` row and enforces
  them at its own attach. A 4-arg convenience constructor keeps every existing
  call site source-compatible. Deliberately per-catalog rather than folded into
  `Worker.attachOptions(...)` — one worker serves a catalog that requires an
  option next to one that takes none, which is exactly what the test asserts.
- Fixture: `AttachOptionsFixture.requiredSpecs()` (`api_key` required, `region`
  defaulting) as the `attach_options_required` catalog of the attach-options
  worker, mirroring vgi-python's `RequiredAttachOptions`. ci/run-integration.sh
  points the env var at that same worker on both lanes, so the test runs.
- The refusal path had never been exercised in Java, and it HUNG.
  `AttachOptionRequirements` throws `RpcError("ValueError", …)`, whose
  `errorKind()` is null as documented, and vgirpc wrote that null into the error
  batch's metadata — throwing in the flatbuffer writer, so the batch never
  reached the wire and the client waited on a response that would never come.
  Fixed in vgirpc 0.21.1; pinned here (build.gradle.kts + VGI_RPC_JAVA_REF).

Global functions needed no work: they shipped in 0.22.0 and both
global_functions tests pass against an extension carrying vgi 21fdd4e. CI's red
on them is the community extension lagging the consuming side — see CLAUDE.md.

Local: launch 296 files / 268 pass / 28 skip / 0 fail (attach_options_required
now among the passes); the two attach tests also verified under shm and over
HTTP. `:vgi:test` green with a new ExtraCatalogAttachOptionsTest; javadoc clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: rustyconover

.github/workflows/integration.yml

@@ -36,7 +36,7 @@ env:
   # The vgi-rpc-java commit built from source (composite include) for the worker's
   # HTTP features. Drop once a vgirpc release with these features is published and
   # pinned in vgi/build.gradle.kts.
-  VGI_RPC_JAVA_REF: v0.21.0
+  VGI_RPC_JAVA_REF: v0.21.1
 
 jobs:
   integration:

CLAUDE.md

@@ -183,8 +183,8 @@ change to `~/Development/vgi-rpc-java/` doesn't show up, run
 ## Releasing (Maven Central)
 
 Published: **`farm.query:vgi`** (this repo) and **`farm.query:vgirpc`** /
-`vgirpc-oauth` (the sibling). Latest as of 2026-07-21: **vgi 0.20.0 → vgirpc
-0.17.0**. To cut a release: bump `version` in `build.gradle.kts`, push, then
+`vgirpc-oauth` (the sibling). Latest as of 2026-08-10: **vgi 0.24.0 → vgirpc
+0.21.1**. To cut a release: bump `version` in `build.gradle.kts`, push, then
 create a GitHub Release whose tag is the version (`v0.2.0` for `0.2.0`). The
 `release.yml` workflow (trigger: `release: published`) verifies tag == version,
 runs tests, and publishes. Both repos now set
@@ -288,6 +288,53 @@ older interfaces** (`TableFunction`, `TableInOutFunction`, etc.) — the
 `ScalarFn` style hasn't been extended to those because their richer
 lifecycle methods + per-execution state don't translate one-for-one.
 
+## State of play (as of 2026-08-10, gated attach options)
+
+**Global functions were already done** (0.22.0 — `Worker.registerGlobalFunctions` /
+`globalFunctionPrefix`, the two protocol-1.3.0 attach fields, the four
+`global_*` probes). Against a locally built extension carrying vgi `21fdd4e`,
+`global_functions/{basic,lifecycle}.test` both pass. **CI is red on them anyway,
+and it is not the worker:** the lane installs `vgi FROM community`, and the
+published build predates the consuming side — `vgi_global_functions()` doesn't
+exist and `ATTACH … (global_functions false)` is an unknown option. Verified by
+`FORCE INSTALL vgi FROM community` under the same `haybarn-v1.5.5-rc1` runner CI
+uses: `count(vgi_global_functions) = 0`. Nothing to do here until the community
+extension republishes; the same lag is what fails `attach_options_echo.test`'s
+`opt.required` read (upstream `1f1d7b1` has since moved those assertions out).
+
+**Per-catalog attach options** (this release). `Worker.ExtraCatalog` gained a
+5th component, `attachOptions` — auxiliary catalogs advertise their own specs on
+their `catalog_catalogs()` row and enforce them at their own attach. A 4-arg
+convenience constructor keeps every existing call site compiling. The main
+catalog's `Worker.attachOptions(...)` is unchanged and unrelated: one worker can
+serve a catalog that requires an option next to one that takes none.
+- Fixture: `AttachOptionsFixture.REQUIRED_CATALOG_NAME` /`requiredSpecs()`
+  (`api_key` required, `region` defaulting) registered as an extra catalog in
+  Main's attach_options worker mode, mirroring vgi-python's
+  `RequiredAttachOptions`. Drives `attach/attach_options_required.test`, which
+  upstream gated behind `VGI_ATTACH_OPTIONS_REQUIRED_WORKER` when the shared
+  file's required assertions broke every non-Python SDK. `ci/run-integration.sh`
+  points that var at the same attach-options worker on both lanes, so the test
+  runs rather than skipping.
+- **The refusal path had never been exercised, and it hung.**
+  `AttachOptionRequirements` throws `RpcError("ValueError", …)` — whose
+  `errorKind()` is null, as documented — and vgirpc's `Wire.errorMetadata` put
+  that null in the batch metadata unconditionally. It threw in the flatbuffer
+  key/value writer, so the error batch never reached the wire and the client
+  waited forever. Fixed in **vgirpc 0.21.1** (`Wire` skips a null kind,
+  `ErrorMetadataTest` pins it); this repo pins that version. Symptom to
+  recognise: a worker-side `NullPointerException … Utf8Safe.computeEncodedLength`
+  in `VGI_WORKER_STDERR`, with the client hung rather than erroring.
+- **Local:** launch 296 files / 268 pass / 28 skip / 0 fail; the two attach
+  tests also verified under shm and against an HTTP worker. `:vgi:test` green
+  (new `ExtraCatalogAttachOptionsTest`), `:vgi:javadoc` clean.
+
+> **Local CI-lane repro is misleading.** `ci/run-integration.sh` against the
+> homebrew `haybarn-unittest` fails ~every test at ATTACH ("field count differs:
+> expected 14, actual 17") — that binary's cached community extension is far
+> older than CI's. Use `~/Development/vgi/build/release/test/unittest` for local
+> work; only the GitHub lane says anything real about the published extension.
+
 ## State of play (as of 2026-07-23, result-cache same-name)
 
 **Cache-key schema disambiguation.** The extension fixed a result-cache-key bug:

build.gradle.kts

@@ -8,7 +8,7 @@ plugins {
 
 allprojects {
     group = "farm.query"
-    version = "0.23.0"
+    version = "0.24.0"
 
     repositories {
         mavenCentral()

ci/run-integration.sh

@@ -121,6 +121,10 @@ case "$TRANSPORT" in
     export VGI_VERSIONED_WORKER="launch:${HERE}/wrappers/vgi-worker-versioned"
     export VGI_VERSIONED_TABLES_WORKER="launch:${HERE}/wrappers/vgi-worker-versioned-tables"
     export VGI_ATTACH_OPTIONS_WORKER="launch:${HERE}/wrappers/vgi-worker-attach-options"
+    # Same worker: it also serves the `attach_options_required` catalog, whose
+    # gated ATTACH is asserted separately (upstream split those assertions out of
+    # the shared attach_options_echo.test behind this env var).
+    export VGI_ATTACH_OPTIONS_REQUIRED_WORKER="$VGI_ATTACH_OPTIONS_WORKER"
     # bad-enum fixture worker: serves the example catalog but advertises an
     # unrecognized null_handling enum for `double`, driving the C++ parser's
     # strict-enum rejection (bad_enum.test). Skipped over HTTP (like
@@ -167,6 +171,7 @@ case "$TRANSPORT" in
     # has an explicit "Pool / HTTP safety" section), so http serves it fine.
     ao_port="$(boot_http_worker "${HERE}/wrappers/vgi-worker-attach-options")"
     export VGI_ATTACH_OPTIONS_WORKER="http://localhost:${ao_port}"
+    export VGI_ATTACH_OPTIONS_REQUIRED_WORKER="$VGI_ATTACH_OPTIONS_WORKER"
     echo "attach_options http worker on ${VGI_ATTACH_OPTIONS_WORKER}"
     # NB: VGI_TEST_DEDICATED_WORKER stays UNSET here. The buffering crash /
     # pool-recovery tests SIGKILL their worker mid-process, which is only safe on

vgi-example-worker/src/main/java/farm/query/vgi/example/Main.java

@@ -307,7 +307,18 @@ public static void main(String[] args) {
                     .catalogName(catalogName)
                     .attachOptions(farm.query.vgi.example.table.AttachOptionsFixture
                             .declaredSpecs().toArray(new farm.query.vgi.AttachOptionSpec[0]))
-                    .registerTable(new farm.query.vgi.example.table.EchoAttachOptionsFunction());
+                    .registerTable(new farm.query.vgi.example.table.EchoAttachOptionsFunction())
+                    // Second catalog from the same worker, gated on an option the
+                    // caller must supply: it advertises that at discovery and
+                    // refuses the anonymous ATTACH, rather than attaching into
+                    // something that reads as an empty catalog. It carries no
+                    // functions of its own — the attach IS what's under test
+                    // (attach/attach_options_required.test).
+                    .registerExtraCatalog(new Worker.ExtraCatalog(
+                            farm.query.vgi.example.table.AttachOptionsFixture.REQUIRED_CATALOG_NAME,
+                            null, null,
+                            "Attach-options catalog gated on a required option",
+                            farm.query.vgi.example.table.AttachOptionsFixture.requiredSpecs()));
             runWorker(ao, args);
             return;
         }

vgi-example-worker/src/main/java/farm/query/vgi/example/table/AttachOptionsFixture.java

@@ -30,6 +30,14 @@ private AttachOptionsFixture() {}
 
     public static final String CATALOG_NAME = "attach_options";
 
+    /**
+     * A second catalog from the same worker, declaring one option the caller
+     * must supply. Kept separate from {@link #CATALOG_NAME} so the
+     * defaults-and-round-trip coverage there keeps attaching with no options at
+     * all. Mirrors vgi-python's {@code REQUIRED_CATALOG_NAME}.
+     */
+    public static final String REQUIRED_CATALOG_NAME = "attach_options_required";
+
     private static final ArrowType DATE32 = new ArrowType.Date(DateUnit.DAY);
     private static final ArrowType TIME64_US = new ArrowType.Time(TimeUnit.MICROSECOND, 64);
     private static final ArrowType TIMESTAMP_US = new ArrowType.Timestamp(TimeUnit.MICROSECOND, null);
@@ -83,4 +91,17 @@ public static List<AttachOptionSpec> declaredSpecs() {
                                 Schemas.nullable("b", Schemas.UTF8)),
                         Map.of("a", 1L, "b", "x")));
     }
+
+    /**
+     * Options of the {@link #REQUIRED_CATALOG_NAME} catalog: one the caller has
+     * to supply (no default to fall back on) next to one that defaults, so a
+     * test can tell the two apart in the discovery listing.
+     *
+     * @return the gated catalog's declared option specs
+     */
+    public static List<AttachOptionSpec> requiredSpecs() {
+        return List.of(
+                AttachOptionSpec.required("api_key", "API key", Schemas.UTF8),
+                AttachOptionSpec.of("region", "Region", Schemas.UTF8, "us-east-1"));
+    }
 }

vgi/build.gradle.kts

@@ -5,7 +5,7 @@ plugins {
 
 dependencies {
     // farm.query is the published group (see ../build.gradle.kts allprojects).
-    api("farm.query:vgirpc:0.21.0")
+    api("farm.query:vgirpc:0.21.1")
     implementation("org.slf4j:slf4j-api:2.0.16")
     // Cross-process aggregate state store. DuckDB spawns multiple worker
     // subprocesses for parallel aggregation; SQLite's file locking gives us

vgi/src/main/java/farm/query/vgi/Worker.java

@@ -43,9 +43,10 @@ public final class Worker {
      *  <p>1.3.0 added {@code global_functions} / {@code global_function_prefix}
      *  to the {@code catalog_attach} result (positions 14/15, before
      *  {@code resolved_data_version}): functions a worker asks the client to
-     *  publish into its global namespace. This port advertises none (empty
-     *  list, empty prefix), but must carry the fields — the extension matches
-     *  the response schema exactly. */
+     *  publish into its global namespace — see {@link #registerGlobalFunctions}
+     *  and {@link #globalFunctionPrefix(String)}. A worker that opts out still
+     *  carries the fields (empty list, empty prefix): the extension matches the
+     *  response schema exactly. */
     public static final String VGI_PROTOCOL_VERSION = "1.3.0";
 
     private String catalogName = "vgi";
@@ -404,9 +405,33 @@ public Worker schemaTags(String schema, Map<String, String> tags) {
      * @param implementationVersion the advertised/resolved implementation version
      * @param dataVersion the advertised {@code data_version_spec} and resolved data version
      * @param schemaComment the comment on the catalog's single {@code main} schema
+     * @param attachOptions ATTACH-time options this catalog alone declares, advertised on its
+     *                      {@code catalog_catalogs()} row and enforced at its attach. Separate from
+     *                      {@link #attachOptions(AttachOptionSpec...)}, which is the main catalog's:
+     *                      a worker may serve one catalog that requires an option and another that
+     *                      takes none.
      */
     public record ExtraCatalog(String name, String implementationVersion, String dataVersion,
-                               String schemaComment) {}
+                               String schemaComment, List<AttachOptionSpec> attachOptions) {
+
+        /** Defensive copy; a null option list reads as none declared. */
+        public ExtraCatalog {
+            attachOptions = attachOptions == null ? List.of() : List.copyOf(attachOptions);
+        }
+
+        /**
+         * An auxiliary catalog declaring no attach options of its own.
+         *
+         * @param name the catalog name used in {@code ATTACH '<name>' ...}
+         * @param implementationVersion the advertised/resolved implementation version
+         * @param dataVersion the advertised {@code data_version_spec} and resolved data version
+         * @param schemaComment the comment on the catalog's single {@code main} schema
+         */
+        public ExtraCatalog(String name, String implementationVersion, String dataVersion,
+                            String schemaComment) {
+            this(name, implementationVersion, dataVersion, schemaComment, List.of());
+        }
+    }
 
     private final Map<String, ExtraCatalog> extraCatalogs = new LinkedHashMap<>();
 

vgi/src/main/java/farm/query/vgi/internal/VgiServiceImpl.java

@@ -987,9 +987,13 @@ public ItemsResponse catalog_catalogs() {
                 worker.dataVersionSpec(), attachOptionBytes,
                 worker.releases(), worker.sourceUrl()));
         for (Worker.ExtraCatalog extra : worker.extraCatalogs().values()) {
+            List<byte[]> extraOptionBytes = new ArrayList<>();
+            for (farm.query.vgi.AttachOptionSpec spec : extra.attachOptions()) {
+                extraOptionBytes.add(AttachOptionSpecSerializer.serialize(spec));
+            }
             items.add(CatalogInfoSerializer.serialize(
                     extra.name(), extra.implementationVersion(), extra.dataVersion(),
-                    List.of(), List.of(), null));
+                    extraOptionBytes, List.of(), null));
         }
         return new ItemsResponse(items);
     }
@@ -1082,6 +1086,10 @@ public void catalog_transaction_rollback(byte[] attach_opaque_data, byte[] trans
     public CatalogAttachResult catalog_attach(CatalogAttachRequest request, CallContext ctx) {
         Worker.ExtraCatalog extra = worker.extraCatalogs().get(request.name());
         if (extra != null) {
+            // Enforced against THIS catalog's specs, not the main catalog's: the
+            // same worker may serve one catalog that requires an option and
+            // another that takes none.
+            AttachOptionRequirements.validate(request.name(), extra.attachOptions(), request.options());
             // MetaWorker-style auxiliary catalog: a random per-ATTACH opaque id
             // is the storage scope isolating this session's state; the client
             // persists and resends it, so it also survives worker restarts.

vgi/src/test/java/farm/query/vgi/internal/ExtraCatalogAttachOptionsTest.java

@@ -0,0 +1,94 @@
+// Copyright 2026 Query Farm LLC - https://query.farm
+
+package farm.query.vgi.internal;
+
+import farm.query.vgi.AttachOptionSpec;
+import farm.query.vgi.Worker;
+import farm.query.vgi.protocol.CatalogAttachRequest;
+import farm.query.vgi.protocol.ItemsResponse;
+import farm.query.vgi.types.Schemas;
+import farm.query.vgirpc.RpcError;
+import farm.query.vgirpc.wire.Allocators;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.complex.ListVector;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.arrow.vector.util.Text;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * ATTACH options declared by an auxiliary catalog rather than the worker's main
+ * one. A worker may serve a catalog that requires an option next to one that
+ * takes none, so both the discovery listing and the refusal are per-catalog.
+ */
+class ExtraCatalogAttachOptionsTest {
+
+    private static final String GATED = "gated";
+
+    private static Worker worker() {
+        return Worker.builder()
+                .catalogName("host")
+                .registerExtraCatalog(new Worker.ExtraCatalog(GATED, null, null, "gated catalog",
+                        List.of(
+                                AttachOptionSpec.required("api_key", "API key", Schemas.UTF8),
+                                AttachOptionSpec.of("region", "Region", Schemas.UTF8, "us-east-1"))));
+    }
+
+    private static VgiServiceImpl service(Worker w) {
+        return new VgiServiceImpl(w, w.scalars(), w.tables(), w.tableInOuts(), w.aggregates());
+    }
+
+    /** A one-row batch whose column names are the supplied option keys. */
+    private static byte[] options(String name, String value) {
+        Schema schema = Schemas.of(Schemas.nullable(name, Schemas.UTF8));
+        try (VectorSchemaRoot root = VectorSchemaRoot.create(schema, Allocators.root())) {
+            root.allocateNew();
+            ((VarCharVector) root.getVector(name)).setSafe(0, new Text(value));
+            root.setRowCount(1);
+            return BatchUtil.writeSingleBatch(root);
+        }
+    }
+
+    /** Count of {@code attach_option_specs} on a serialized CatalogInfo row. */
+    private static int declaredOptionCount(byte[] catalogInfo) {
+        return BatchUtil.withReadBatch(catalogInfo, Allocators.root(),
+                root -> ((ListVector) root.getVector("attach_option_specs")).getObject(0).size());
+    }
+
+    @Test
+    void auxiliaryCatalogAdvertisesItsOwnOptionsAtDiscovery() {
+        ItemsResponse catalogs = service(worker()).catalog_catalogs();
+        assertEquals(2, catalogs.items().size());
+        assertEquals(0, declaredOptionCount(catalogs.items().get(0)), "main catalog declares none");
+        assertEquals(2, declaredOptionCount(catalogs.items().get(1)));
+    }
+
+    @Test
+    void omittingARequiredOptionFailsTheAttach() {
+        RpcError e = assertThrows(RpcError.class, () -> service(worker())
+                .catalog_attach(new CatalogAttachRequest(GATED, null, null, null), null));
+        assertTrue(e.errorMessage().contains("required option 'api_key'"), e.errorMessage());
+    }
+
+    @Test
+    void supplyingItAttachesNormally() {
+        var result = service(worker()).catalog_attach(
+                new CatalogAttachRequest(GATED, options("api_key", "secret"), null, null), null);
+        assertTrue(result.attach_opaque_data().length > 0);
+    }
+
+    /** The gate belongs to the catalog that declared it: the host catalog
+     *  declares no options and still attaches with none supplied. */
+    @Test
+    void theHostCatalogIsUnaffected() {
+        var result = service(worker()).catalog_attach(
+                new CatalogAttachRequest("host", null, null, null), null);
+        assertTrue(result.attach_opaque_data().length > 0);
+    }
+}