Skip to content

feat(lancedb): enable GooseFS in the UI and add IVF_FLAT/SQ/RQ/BTree indexes - #870

Open
XuQianJin-Stars wants to merge 4 commits into
zilliztech:mainfrom
XuQianJin-Stars:feat/lancedb-goosefs-ui-support
Open

XuQianJin-Stars wants to merge 4 commits into
zilliztech:mainfrom
XuQianJin-Stars:feat/lancedb-goosefs-ui-support

Conversation

@XuQianJin-Stars

Copy link
Copy Markdown
Contributor

Summary

  • Expose GooseFS/COS storage_options in the LanceDB Run Test UI (same GOOSEFS_* / COS env resolution as CLI) and fill missing DB icons.
  • Add Lance IVF_FLAT, IVF_SQ, and IVF_RQ index configs, CLI commands (lancedbivfflat / lancedbivfsq / lancedbivfrq), and UI index options. IVF_HNSW_SQ/PQ remain available.
  • Return PartialInsertError after a partially committed table.add so concurrent load does not retry already-written IDs; treat storage_options={} as an explicit empty mapping.

Test plan

  • make format and make lint pass
  • tests/test_lancedb_config.py and tests/test_lancedb_client.py pass
  • UI: LanceDB Run Test shows GooseFS-related fields hidden/storage_options filled from env; index dropdown includes IVF_FLAT, IVF_PQ, IVF_SQ, IVF_RQ, IVF_HNSW_SQ, IVF_HNSW_PQ
  • CLI: vectordbbench lancedbivfflat|lancedbivfsq|lancedbivfrq --help shows the new commands
  • Optional: 10K GooseFS run with IVF_FLAT / IVF_SQ / IVF_RQ against a local GooseFS cluster

Share GooseFS/COS storage_options resolution so UI and CLI both pick up
GOOSEFS_* env vars, expose IVF_HNSW_SQ/PQ in Run Test, allow concurrent
loads, and reuse existing icons for DBs without dedicated assets.
…torage_options

Return PartialInsertError for already-committed fragment prefixes so
ConcurrentInsertRunner does not retry whole batches, and treat
storage_options={} as an explicit empty mapping.
@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: XuQianJin-Stars
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@XuQianJin-Stars

Copy link
Copy Markdown
Contributor Author

hi @XuanYang-cn all checks have passed and there are no conflicts. Could you help review this PR when you get a chance? Thanks!

@XuQianJin-Stars
XuQianJin-Stars force-pushed the feat/lancedb-goosefs-ui-support branch 2 times, most recently from cb733b0 to 9e9efba Compare September 11, 2026 10:17
Wire Lance BTree through CLI (`lancedbbtree`), UI, and create_index so filter cases can use a scalar index on id/label.
@XuQianJin-Stars XuQianJin-Stars changed the title feat(lancedb): enable GooseFS in the UI and add IVF_FLAT/SQ/RQ indexes feat(lancedb): enable GooseFS in the UI and add IVF_FLAT/SQ/RQ/BTree indexes Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants