feat(kg-rag): KG-RAG semantic-graph layer on the new kb-server architecture#423
Closed
AdriaGuilera wants to merge 177 commits into
Closed
Conversation
Ports the KG-RAG / semantic-graph stack from the legacy
lamb-kb-server-stable into the new pluggable lamb-kb-server (port 9092)
and wires it through Knowledge Stores instead of legacy Knowledge Bases.
KB server:
- New /graph and /benchmarks routers, mounted only when KG_RAG_ENABLED.
- Neo4j graph_store + LLM concept extractor adapted to string
collection IDs and request-scoped OpenAI credentials.
- New graph_indexing helper feeds Neo4j from the ChromaDB backend after
vector ingestion succeeds (fails open).
- KG-RAG query augmentation hangs off query_service.query_with_plugin
('simple_query' | 'kg_rag_query'), preserving the trace contract.
- Optional 'kg-rag' extra adds neo4j + openai.
LAMB backend:
- New knowledge_store_graph_router proxies graph + benchmark calls
through KnowledgeStoreClient using per-org token/URL resolution.
- KnowledgeStoreCreate threads graph_enabled to the KB server.
Frontend:
- New graphService.js + benchmarkService.js axios clients.
- New KnowledgeStoreGraphView + KnowledgeStoreBenchmarkView Svelte
components, mounted as tabs on KnowledgeStoreDetail when KG-RAG is
on. Curation actions (approve / reject concepts and relationships)
proxy through the LAMB router.
- CreateKnowledgeWizard exposes a Graph RAG opt-in checkbox.
Infra:
- docker-compose.next.yaml gets an optional neo4j service under
--profile kg-rag.
- .env.next.example documents the KG_RAG_* block.
Tests:
- tests/unit/test_kg_rag.py covers config parsing, concept-extraction
helpers and the plugin's three graceful-degradation paths.
Quality gates: lamb-kb-server-stable untouched; KB server starts and
all existing unit + integration tests pass without Neo4j; graph
features are off by default per-server AND per-collection.
Blocking fixes: - Benchmark route now uses a single body model with embedded ``embedding_credentials``, so the LAMB proxy's flat JSON validates (previously FastAPI's Body(embed=True) required wrapping under ``request``). Adds an EmbeddingCredentialsBody schema. - init_db now runs lightweight ALTER TABLE migrations so existing installations get the new ``graph_enabled`` column without an ``OperationalError: no such column``. Uses a direct sqlite3 connection (not the SQLAlchemy engine) so the migration check doesn't leave WAL state that breaks fork-based lock tests. Only runs against pre-existing DBs to keep the hot path zero-cost. Significant: - Permalinks now flow through to Neo4j Chunk nodes (permalink_original / permalink_full_markdown / permalink_page). Snapshot endpoint exposes them on chunk node data for citations. Moderate: - Vector backend gets public ``get_chunks_by_id``, ``get_chunks_by_source``, and ``iter_all_chunks`` surfaces. KG-RAG plugin and migration endpoint now call those instead of reaching into ChromaDB's private client cache. Default ``[]`` / NotImplementedError lets non-chromadb backends degrade cleanly. - Graph migration endpoint surfaces unsupported-backend (e.g. qdrant) as a clear 400 with explanation; the frontend hides the migrate button and explains why instead of letting the user click into an error. - ConceptExtractor passes a configurable per-request OpenAI timeout (default 60s) so a hung vendor can't pin an ingestion worker indefinitely. New ``KG_RAG_OPENAI_TIMEOUT_SECONDS`` env knob. - Graph + Benchmark tabs are now gated on ``ks.graph_enabled``, with a graceful fallback to "show tabs if the store could be migrated". Documented ingest-time slowdown in .env.next.example. Nits: - Drop unused ``_collection_dict`` helper in routers/graph.py. - Lazy-import OrganizationConfigResolver in the LAMB proxy router to match existing convention. - Move the ``reload_config`` fixture into tests/unit/conftest.py so any unit test can pick it up. Tests: - New regression tests for the proxy-flat body shape and the schema migration. Idempotent re-run check is included. Full unit (308) + integration (179 + 1 skipped) suites pass.
…reate toggle, vendor/model picker, verified-only retrieval, Sigma full-graph view) and drop the in-app benchmark surface fix(kb-v2): per-collection LLM extraction config + plugin system (OpenAI/Ollama), cascade-aware curation actions, and Cypher tightened to use only verified concepts + relationships
The question-time entity extractor was reading KG_RAG_QUESTION_EXTRACTION_MODEL env and falling back to the server-level chat_model, ignoring the per-collection extraction config picked by the user at KS creation. As a result the entity names produced at query time could be normalised differently from those stored at build time, silently zeroing the question-entity seeding path. It now resolves vendor/model/endpoint from the collection's stored extraction_* fields via LLMExtractionRegistry.build(...), the same path used by the build-time concept extractor. Cache key is now (vendor, model, question) so cross-collection lookups don't collide.
…ed KSs and surface extracted entities in Test Query UI
…ration filters
and pagination
- Fix concept verification state to be scoped per Knowledge Store via
MENTIONS.verification_state instead of the org-level Concept node,
so approving concepts in one KS does not affect other KSs
- Fix change-detection in curation tx to compare against
MENTIONS.verification_state (the per-collection value) rather than
concept.verification_state, which could be 'verified' from another KS
and silently suppress the write
- Delete associated graph data (Document, Chunk nodes, orphaned Concepts)
when a linked library item is removed from a Knowledge Store
- Add node drag support and zoom-adaptive labels to the Sigma.js graph view
- Add verification status filter (All/Unverified/Verified/Rejected) to
Concepts and Relationships sections in the curation panel
- Add client-side pagination (20 items/page) to both curation lists
- Show percentage verified in stats cards and section headers
- Optimistic in-place mutation on verify/unverify toggle to eliminate flicker
…b-integration' into feat/kg-rag-new-kbserver-arch
…m KB server refactor Lamb-Project#334
…ile_rag validation
…to lightweight module
Replace Base.metadata.create_all with Alembic migrations run to head at startup (init_db -> _run_migrations). Migrations live in backend/migrations/ (named so as not to shadow the alembic package on sys.path) with a baseline revision capturing collections + ingestion_jobs and their indexes. env.py derives the SQLite URL from config.DB_PATH, enables batch mode and WAL/foreign-key pragmas. Pre-Alembic databases (tables present, no alembic_version) are stamped to the baseline before upgrade so existing data is preserved. Adds an upgrade/downgrade round-trip test.
Replace Base.metadata.create_all with Alembic migrations run to head at startup (init_db -> _run_migrations) and retire the _apply_lightweight_ migrations ad-hoc-ALTER shim. The baseline revision captures all six tables, their indexes, and FK cascades; the shim's only delta (content_items.folder_id) folds into the baseline. env.py derives the SQLite URL from config.DB_PATH, enables batch mode and WAL/foreign-key pragmas. Pre-Alembic databases (tables present, no alembic_version) are stamped to the baseline before upgrade so existing data is preserved. Adds an upgrade/downgrade round-trip test.
…ate Knowledge wizard Remove 58 orphaned i18n keys (step0-step5 namespaces plus scattered dead keys in step6-step9, libraryContent, libraryStep, draft.savePrompt, and stale top-level wizard keys) from all four locales, touching only the knowledge.wizard block so the rest of each file is byte-preserved. Drop the dead uploadedItems WizardState field, declare the used-but-undeclared pendingFileParams, and fix stale 'former Step0/Step2' comments. No behavior change.
…/ca/eu)
Add 80 i18n keys that were referenced only via inline English { default }
fallbacks and missing from all four locales, so the wizard, add-content modal,
indexing-progress modal, and locked-config / chunking-params panels render in
the selected language instead of English.
en values mirror the existing inline defaults; es/ca/eu are translated using the
established conventions (biblioteca, fragmentación/fragmentació/zatiketa,
Almacén/Magatzem/Ezagutza-biltegia) and the project's 'index' terminology for
the Knowledge Store step. Only the common/knowledge/knowledgeStores blocks are
touched; every other byte of each locale is preserved.
…edential retention
KB Server: retain a failed job's embedding credentials in memory across
attempts (retain on retryable failure, discard on success/cancel/exhaustion,
purge past KB_RETRY_CACHE_TTL_MINUTES via a cleanup loop) instead of popping
them on first pickup. Add POST /jobs/{id}/retry (re-queues, reusing the cached
payload+credentials; 409 if not failed/exhausted, 410 if the window elapsed)
and GET /jobs/{id}/retry-available.
LAMB: proxy retry_job / get_job_retry_available in the KS client and add
POST /creator/knowledge-stores/{ks}/content/{item}/retry, which flips the
content link back to pending and clears the prior error.
Frontend: retryIngestion() now calls the real endpoint and surfaces the
backend detail (e.g. the 410 re-add message); remove the 'coming soon' stub.
Add direct tests for the knowledge_store_rag processor: multi-KS fan-out and continuous citation numbering, per-store error handling, the no-collections / empty-collections / no-user-message / no-assistant guards, and the [N]-numbered context with sources aligned by n.
…ations-ks-rag feat(Lamb-Project#430): emit inline [N] citations in Knowledge Store RAG answers
…store-rag test(Lamb-Project#442): cover knowledge_store_rag _run() retrieval path
…-alembic chore(Lamb-Project#432): adopt Alembic for KB Server schema migrations
…anager-alembic chore(Lamb-Project#434): adopt Alembic for Library Manager migrations
…dge-wizard-cleanup refactor(Lamb-Project#436): remove dead artifacts from the legacy Create Knowledge wizard
…dge-store-wizard fix(Lamb-Project#438): localize Create Knowledge wizard and KS UI (es/ca/eu)
…g-retry feat(Lamb-Project#440): implement KS indexing retry with in-memory credential retention
…s panel Render cited sources to the student and make them openable without a login: - citation_sources.build_owi_sources maps RAG sources to OpenWebUI's native citations schema (name='N' so OWI auto-links inline [N], excerpt under the real filename, absolute signed view URL). Emitted as a sources SSE event before [DONE] in main.py (and on the non-streaming response). - permalink_signing: per-organization HMAC signing (no expiry). Cross-org forgery is impossible; rotating LAMB_PERMALINK_SIGNING_SECRET revokes all. - Two public, signature-authorized routes on LAMB (before the authenticated /docs catch-all): a 'view' page (filename title + rendered markdown + Download original) and an 'original' download, both proxying to the Library Manager so OpenWebUI never contacts it directly. - Drop the PPS's injected sources block (OWI renders the panel now) and carry the chunk text on each source for the citation excerpt. The authenticated /docs permalink is unchanged.
…n (OWI drops sources field) OpenWebUI's chat path forwards only choices[].delta.content from an external model and drops any top-level sources field, so the citations panel never rendered. Instead append a clickable Markdown 'Sources' section to the answer content just before [DONE]; OWI renders it. Sources are grouped per library item with every citation number shown (e.g. '[1][3][5] [cv.pdf](signed-url)') so each inline [N] resolves. build_owi_sources is kept for non-streaming / spec-compliant clients.
…ant (default off) Cited documents are no longer exposed automatically. Gate the clickable Sources section (and the OWI sources field) on a new per-assistant capability metadata.capabilities.expose_sources, read in the completion pipeline via _assistant_exposes_sources (defaults false, mirroring vision/image_generation). Existing assistants and any whose JSON lacks the key never expose sources until the creator opts in. Add a 'Let students open cited sources' toggle to the assistant builder (ConfigurationPanel) wired through the form state/submit and bound in AssistantForm, with i18n in en/es/ca/eu.
…re not exposed Previously, with expose_sources off, the answer still showed [1][2][3] markers (the cite instruction + [N] context prefixes were always fed to the model), so students saw citations they could not open. Gate the citation instruction on the assistant's expose_sources flag in the PPS, and strip the per-chunk [N] prefixes from the context when off, so no citation markers are produced at all unless the creator opted in.
…citation-links feat(Lamb-Project#445): opt-in clickable citation links to cited sources
Reorganize the flat tests into unit/integration/e2e tiers mirroring the Knowledge Store suite, and fill the gaps to 98% line+branch coverage. - Shared harness: boundary fakes (markitdown/firecrawl/fitz/openai), poll helpers, payload factories, auto-tagging by directory, uvicorn-subprocess and Docker e2e drivers. - Unit tier: direct module tests for config/db/models/schemas, every import plugin, the plugin + capability registries, and the service layer (which previously had zero direct tests). - Integration tier: in-process ASGI against the real DB + worker; asserts real worker concurrency cap, api-keys-never-persisted, lifespan, route ordering, pagination. - E2E tier: real HTTP crash-recovery, single-instance flock, graceful shutdown, multitenancy, full error matrix, and a Docker image smoke test. - Tooling: pytest-cov branch coverage with a >=95% gate, scripts/run_tests.sh, and tests/README.md tier contract. Pins two latent bugs as strict xfails (folder delete-collision reparent under a non-root parent; verify_token 500 on non-ASCII token). Updates stale test docs in CLAUDE.md and the library-manager README.
…ning parent in delete_folder (Lamb-Project#447) The autoflush triggered while _next_available_name queried siblings wrote the still-colliding name and violated uq_folder_sibling_name when reparenting subfolders under a non-root folder. Now compute the deduped name before reassigning parent_folder_id. Unmarks the strict-xfail regression test that pinned this bug.
…fy_token (Lamb-Project#448) hmac.compare_digest raised TypeError on non-ASCII str input, surfacing as HTTP 500 on the auth path. Encode both operands to bytes before comparing so any invalid token (including non-ASCII) yields 401 while keeping the valid path constant-time. Unmarks the strict-xfail regression test that pins this behaviour.
…folder-reparent-collision fix(library-manager): delete_folder reparent-collision under non-root parent (Lamb-Project#447)
…token-non-ascii fix(library-manager): verify_token 500 on non-ASCII bearer token (Lamb-Project#448)
…line The tests/.yt_cache/ transcripts were gitignored, so a fresh clone had no cassettes and the YouTube tests fell through to a live yt-dlp network fetch (non-deterministic / rate-limit prone). Commit the two small fixtures and stop ignoring them so the suite is fully hermetic on clone — no network required.
The 'my' sharing predicate used `is_owner !== false`, which also passed shared items (and any item whose is_owner was undefined), so clicking "Mine" showed all libraries. Add `&& !l.is_shared` for parity with the working KnowledgeStoresList predicate; "Mine" now means owned-and-private.
…ine-filter fix(frontend): Libraries "Mine" filter shows all instead of only owned
…ct#453) The README and tests/README claimed 99% line+branch coverage, but the combined three-tier run (unit + integration + e2e, 594 tests) measures ~94%. The shortfall is not a regression: the e2e tier exercises the backend over real HTTP in a uvicorn subprocess that the parent pytest-cov process cannot instrument, so those server-side paths are not counted, alongside a few structurally unreachable guards. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…main' into feat/kg-rag-new-kbserver-arch
Author
|
Closing in favor of #454. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR ports the KG-RAG (Knowledge-Graph Retrieval-Augmented Generation) capability onto the new pluggable
lamb-kb-serverarchitecture introduced inprojects/refactor/kbserver-lamb-integration.It adds a semantic-graph layer (Neo4j) on top of the existing vector store, a
kg_rag_queryplugin that fuses graph traversal with vector retrieval (RRF), an auditable concept-extraction & curation pipeline, and a Svelte UI to explore and verify the per-collection knowledge graph.What's included
lamb-kb-server/backend) — Neo4j-backedgraph_store,graph_indexing,concept_extractionservices andkg_rag_queryplugin; graph/benchmark/curation routers; LLM extraction backends.backend/) — knowledge-store client, RAG processor and graph-proxy routers wiring KG-RAG into the LAMB core.frontend/svelte-app) — Knowledge Store UI: Graph RAG toggle (locked-at-creation), drag/zoom graph view, per-collection concept verification/curation./queryis routed through the KG-RAG plugin automatically for graph-enabled collections.docker-compose.next.yaml,.env.*.example, andDocumentation/kg-rag-deployment.mddeployment guide.lamb-kb-server,backendandlibrary-manager(100% diff coverage on feature-added lines).How to test
See
Documentation/kg-rag-deployment.md. In short: bring up the stack with thekg-ragprofile andKG_RAG_ENABLED=true, create a graph-enabled collection, ingest documents, and query — graph-enabled collections route throughkg_rag_query.