feat(rag): add active generation schema and store (#438 PR1)

[sqhyz55]

Introduce nullable generation_id columns, active_generations table, and ActiveGenerationStore for crash-safe re-chunking. Keep legacy chunk/embedding merge keys until ingestion publishes generations in a follow-up PR.

[gemini-code-assist]

Code Review

This pull request introduces the ActiveGenerationStore and its LanceDB implementation to track published searchable generations, updating the schemas for chunks and embeddings tables to include generation_id and config_hash while providing migration utilities for existing data. Feedback recommends optimizing performance by caching table existence checks and improving the atomicity of the publication process to reduce database round-trips.

[gemini-code-assist]

The _ensure_table method is called on every database operation (publish, get, list). Since it performs a directory listing via list_table_names to check for table existence, this can become a performance bottleneck during high-volume ingestion. Caching the result of this check after the first successful execution would significantly reduce overhead.

Suggested change

	def _ensure_table(self) -> None:
	from ..LanceDB.schema_manager import ensure_active_generations_table
	ensure_active_generations_table(self._get_sync_connection())
	def _ensure_table(self) -> None:
	if getattr(self, "_table_ensured", False):
	return
	from ..LanceDB.schema_manager import ensure_active_generations_table
	ensure_active_generations_table(self._get_sync_connection())
	self._table_ensured = True

[gemini-code-assist]

The current implementation performs an extra round-trip to the database to fetch the existing record just to preserve the created_at timestamp. This can be achieved more efficiently and atomically using merge_insert with a specific when_matched_update clause that excludes created_at. This avoids the overhead of a separate search operation.

        try:
            now = datetime.now(timezone.utc).replace(tzinfo=None)
            record = {
                "collection": collection,
                "doc_id": doc_id,
                "parse_hash": parse_hash,
                "user_id": user_id,
                "model_tag": normalized_tag,
                "generation_id": generation_id,
                "config_hash": config_hash,
                "created_at": now,
                "updated_at": now,
                "published_at": now,
                "operator": operator or "unknown",
            }

            (
                table.merge_insert(
                    on=[
                        "collection",
                        "doc_id",
                        "parse_hash",
                        "user_id",
                        "model_tag",
                    ]
                )
                .when_matched_update(
                    updates={
                        "generation_id": "source.generation_id",
                        "config_hash": "source.config_hash",
                        "updated_at": "source.updated_at",
                        "published_at": "source.published_at",
                        "operator": "source.operator",
                    }
                )
                .when_not_matched_insert_all()
                .execute([record])
            )
        finally:
            _safe_close_table(table)

[sqhyz55]

当前依赖的 LanceDB 0.30.2 的 merge_insert 只有：

when_matched_update_all()
when_not_matched_insert_all()
when_not_matched_by_source_delete()
没有 when_matched_update(updates={...})

[rogercloud]

I found one issue in the active generation pointer store around nullable legacy user scope handling.

[rogercloud]

This breaks legacy scopes where user_id is None. active_generations.user_id is nullable and the RAG model still allows Optional[int]; when callers publish with user_id=None, the pre-read builds user_id == NULL (which does not match NULL rows), and this merge_insert also cannot match an existing NULL-key row because the on columns are joined by equality. Re-publishing the same legacy scope therefore inserts duplicate active pointers, while get_active_generation(..., None, ...) returns no row.

A no-schema-change fix would be to normalize the active-generation store's legacy scope to a non-null int sentinel before building filters and records, e.g. LEGACY_USER_ID = -1, and use that normalized value consistently in publish_active_generation, get_active_generation, and list_active_generations. Please add a regression test that publishes the same (collection, doc_id, parse_hash, None, model_tag) twice and asserts there is only one row and the second generation is returned.

[rogercloud]

Context for this PR relative to the KB refactor (epic #494)

Thanks for this — the schema and the ActiveGenerationStore contract are well-built and the test coverage is solid. My concern is timing and placement against the in-flight KB semantic-layer refactor, not the code quality. Writing the full picture up so the rework path is clear.

Where this PR sits today

This branch was cut when the storage layer was still the primary seam: stores were exposed through StorageFactory and surfaced via kb/storage_shim.py. Consistent with that design, the PR is purely additive:

• appends an ActiveGenerationStore ABC to storage/contracts.py
• implements it in storage/lancedb_stores.py
• registers get_active_generation_store() on storage/factory.py and forwards it through kb/storage_shim.py
• adds an active_generations table + generation_id column in LanceDB/schema_manager.py

Because there are no deletions, it merges cleanly and does not duplicate anything already on main — generation_id / active_generation / ActiveGenerationStore / published_at currently have zero hits on main. So this is genuinely new ground, not a collision.

Why landing it now is awkward

Since this branch's base, main has advanced ~48 commits and Phase 2 of #494 ("KBCollectionHandle Replacement") has landed the full kb/ semantic layer — kb/coordinator.py, kb/collection_handle.py, and the *_compatibility.py facades (H01–H03 merged; the search lifecycle #511 is in review). That creates three frictions:

1. Ownership is moving the opposite direction. This PR binds the generation store to the StorageFactory + kb/storage_shim.py global-singleton seam. Phase 2 is doing the reverse — pulling data-plane mechanics down into KBCollectionHandle (chunk lifecycle → ref(kb): H02 move parse and chunk lifecycle into handle #509, embedding lifecycle → ref(kb): H03 move embedding lifecycle into handle #510), turning the coordinator into routing/orchestration (ref(kb): H07 shrink coordinator to router #514) and shrinking the compatibility facades to thin adapters (ref(kb): H08 shrink compatibility adapters #515). Active-generation publishing happens during ingest / re-chunk, which is exactly the data-plane being moved into the handle. Mounting it on the factory seam now means it becomes one more facade that ref(kb): H08 shrink compatibility adapters #515 has to unwind later.

2. Ordering vs. the concurrency baseline. The contract here is sync+async throughout, in service of crash-safe re-chunking / async ingestion. Epic ref(kb): track semantic-layer coordinator and handle refactor #494 sequences ref(kb): H09 make store data-plane thread-safe for concurrent handle access #657 ("make store data-plane thread-safe for concurrent handle access") explicitly after ref(kb): H08 shrink compatibility adapters #515 and calls it out as the hardening prerequisite for any async ingestion path. Publishing async generation pointers on a store layer whose concurrency hardening is still scheduled downstream inverts that order.

3. Baseline drift. LanceDB/schema_manager.py and storage/contracts.py have been touched repeatedly by the handle migration since this branch's base. A clean text merge does not guarantee the schema_version, table definitions, migration chain, and schema tests still line up — they need re-validation on current main, not just a green merge.

Suggested path to make it fit

Option A (preferred) — re-anchor to the handle after #509/#510. Hold this until chunk/embedding lifecycle lands in KBCollectionHandle, then design active-generation ownership into the handle: publish the pointer from the handle's chunk/embedding write path, with the coordinator owning policy and orchestration. Most of this PR is directly reusable — the active_generations schema, the ActiveGenerationStore contract shape, and the tests. The change is the mounting point: from a StorageFactory singleton to a handle-owned store.

Option B — if crash-safe re-chunking is needed sooner. Align with the refactor owner first on (a) whether the generation pointer is owned by KBCollectionHandle or KBCoordinator, and (b) pulling #657's concurrency baseline forward so the async publish path rests on a thread-safe store. Avoid cementing it on the global singleton in the meantime.

Either way, please rebase onto current main and re-run the RAG schema / migration tests, so we are validating against the live baseline rather than the ~48-commit-old base this branch was cut from.