Skip to content

Add vector support for non-Postgres database loaders #68

Description

@jamessewell

Context

The CLI loader now validates vector(n) schema columns and currently allows them only for ParadeDB/PostgreSQL. Other backends can load scalar CSV/Parquet data, but vector columns are rejected early because each target database needs backend-specific schema mapping, ingest conversion, and query/index behavior.

Scope

  • Define the supported vector representation per backend, including ClickHouse, Elasticsearch, OpenSearch, and MongoDB.
  • Add backend capability metadata instead of hard-coding vector support to Postgres-style backends.
  • Implement schema creation and ingest conversion for supported vector-capable databases.
  • Add CSV and Parquet vector fixtures for each backend that supports vectors.
  • Update the loader support matrix and dataset docs once support is added.

Acceptance criteria

  • vector(n) schemas either load successfully for a backend with documented mapping, or fail with a targeted unsupported-backend error.
  • Vector dimension validation is preserved across CSV and Parquet inputs.
  • Each supported backend has unit or integration coverage for vector ingest.
  • Docs list the exact vector type/mapping used by each database.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or requestpriority-highHigh priority issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions