Skip to content

[DoNotMerge] Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381 [DonNotMerge] - #40648

Draft
nitinware wants to merge 9 commits into
apache:masterfrom
nitinware:couchbaseio
Draft

nitinware wants to merge 9 commits into
apache:masterfrom
nitinware:couchbaseio

Conversation

@nitinware

@nitinware nitinware commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381.

Design discussion: https://lists.apache.org/thread/cvzpfnw717zbnf7z6slfkfbp6zg6604g

Design doc: https://docs.google.com/document/d/1ALlXaiVSa_Dn9jJxM9MsmhEhBLPW_FP8RswN8HG33iU

Implementation

The commits are organized by component to make the changes easier to review:

  1. Write: Concurrent, batched UPSERT operations using withIdFn and withContentFn, with bounded in-flight concurrency and flushing in @FinishBundle.
  2. JsonObjectCoder: UTF-8 JSON encoding for Couchbase SDK JsonObject values.
  3. Read: Bounded SQL++ query execution as a single unit of work, with an optional parsing function (defaulting to JsonObject), named/positional parameters, scan consistency, and opt-in withRedistribute() applied after parsing.
  4. Schema-aware reads: readRows(Schema) converts query results into Beam Row values using RowJson, respecting the supplied schema's nullability constraints.
  5. Documentation: Class Javadocs, CHANGES.md, and the connector documentation table.
  6. Review improvements: Display data, a build configuration comment, and an upsert idempotency test.

Testing

  • 39 unit tests across CouchbaseIOTest and JsonObjectCoderTest.
  • 15 Testcontainers integration tests across CouchbaseIOReadTest and CouchbaseIOWriteTest, using couchbase/server:enterprise-8.0.3.
  • Tests cover authentication failures (the password must not appear in errors, toString, or display data), type mismatches, missing required fields, and repeated UPSERT operations.
  • Local validation passes: ./gradlew :sdks:java:io:couchbase:build (Checkstyle, Javadoc, Checker Framework nullness validation, and dependency analysis).
  • checkJavaLinkage could not run locally because of a corporate repository proxy restriction; CI validation is pending.
  • TLS (couchbases://) is delegated to the Couchbase SDK but is not covered by an integration test.

Known limitations / follow-ups

  • Connection lifecycle: One Couchbase Cluster per DoFn instance. JVM-wide sharing may be considered after benchmarking.
  • Source parallelism: SQL++ reads execute as a single unit of work. Partitioned reads are deferred.
  • SchemaTransform / Managed I/O: Deferred pending further community feedback (design doc §12, question 2).
  • Additional capabilities: Advanced write modes, durability configuration, dead-letter handling, and DCP-based CDC are future work.

R: @kennknowles

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • If this contribution is large, please file an Apache Individual Contributor License Agreement.
  • Replace these instructions with a description of your change and why you have made it.
  • Mention the appropriate issue in your description (for example: addresses #123 or fixes #456).
  • Update CHANGES.md with noteworthy changes.

See CONTRIBUTING.md for more details.

Adds the sdks/java/io/couchbase module with ConnectionConfiguration and
CouchbaseIO.write(), which upserts each element as a JSON document in
batches, with bounded concurrency through the reactive API. Includes unit
tests, Testcontainers integration tests against Couchbase Server 8.0.3,
and a PreCommit workflow for the module.
Encodes a JsonObject as its JSON string, which is 2 to 3 times smaller
than the SerializableCoder Beam would otherwise infer. It will be the
default output coder for CouchbaseIO.read().
Adds CouchbaseIO.read(), which runs a SQL++ query through the reactive
API and streams the rows. Supports named and positional parameters, scan
consistency, a parsing function applied inside the read, an output coder,
and an optional redistribute after parsing. The default output is
JsonObject, encoded with JsonObjectCoder.
@nitinware nitinware changed the title Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381. Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381 [DonNotMerge] Oct 18, 2026
@nitinware nitinware changed the title Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381 [DonNotMerge] [DoNotMerge] Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381 [DonNotMerge] Oct 18, 2026
Comment thread .github/workflows/beam_PreCommit_Java_Couchbase_IO_Direct.yml Fixed
Comment thread .github/workflows/beam_PreCommit_Java_Couchbase_IO_Direct.yml Fixed
Comment thread .github/workflows/beam_PreCommit_Java_Couchbase_IO_Direct.yml Fixed
@kennknowles
kennknowles self-requested a review October 19, 2026 16:03
@github-actions github-actions Bot added the size/xl XL: >=1000 lines changed label Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CouchbaseIO

2 participants