Add Leiden community detection and LightRAG dual-level retrieval#2
Conversation
Extend the port with two optional subsystems from graphrag-rs: - Leiden community detection (Graph/Leiden.swift): greedy modularity local-moving plus a refinement pass that splits internally disconnected communities. Made deterministic (stable node ordering) and weighted (uses relationship confidence as edge weight); reports Newman modularity. Only result-affecting config is exposed. - LightRAG dual-level retrieval (LightRAG/): extract high-/low-level keywords from a query via the LLM with a deterministic offline fallback, search a low-level chunk store and a high-level per-community summary store, and merge (interleave/highFirst/lowFirst/weighted) with dedup by id. LightRAGEngine ties it together with detectCommunities/retrieve/ask. Wire both into GraphRAG via detectCommunities() and lightRAG(), document them in the README, and add unit tests covering community separation, determinism, keyword extraction/fallback, merge strategies, and an offline end-to-end ask. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces community detection using the Leiden algorithm and dual-level retrieval via LightRAG. The review feedback focuses on critical performance, correctness, and concurrency improvements. Key recommendations include reusing pre-computed chunk embeddings to avoid redundant API calls, ensuring determinism in the Leiden detector by sorting dictionary keys during iteration, using structured concurrency to run semantic searches and searcher builds in parallel, robustly parsing LLM responses by stripping thinking tags, and filtering out virtual community IDs from grounding sources.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
…bustness
- Searchers: InMemorySemanticSearcher.build now reuses each chunk's
precomputed embedding from the graph instead of re-embedding on every query
(community summaries still embed on demand). Avoids the async-`??` autoclosure
pitfall with an explicit branch.
- Leiden: build adjacency by node index with sorted neighbors, and sum
modularity in sorted community-id order, so neighbor ordering and
floating-point sums are deterministic across runs.
- Keywords: strip thinking tags before locating the JSON braces, so a `{` in a
<think> preamble can't derail parsing; deduplicate fallback keywords
case-insensitively in first-seen order.
- DualRetrieval / LightRAGEngine: run the two independent searches, and the two
store builds, concurrently via async let.
- LightRAGEngine: exclude virtual "community_<id>" ids from grounding sources so
they only contain real chunk ids.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 20408af629
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…proof merge - LightRAGEngine now builds the two dual-level stores once per engine (via a memoizing actor cache) instead of re-embedding the whole corpus on every retrieve/ask — with a remote embedder that was O(corpus) network calls per query. - Searchers thread the real chunk ids each document grounds to: a community summary grounds to the chunks mentioning its member entities, so answers built from high-level hits cite actual evidence chunks rather than synthetic community ids. `ask` grounds `sources` on these real chunk ids. - Merge deduplicates within a retrieval level only (keyed by level), so a chunk id that happens to collide with a "community_<id>" no longer silently evicts the community hit or vice versa. Tests: split the interleave test into within-level dedup + colliding cross-level id cases, and add community→member-chunk grounding coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f98e1ad031
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ck keywords - GraphRAG.lightRAG() and detectCommunities() now require isBuilt (throwing GraphRAGError.notInitialized otherwise), mirroring the ask/search guard, so callers can't get answers from a raw, stale, or partially rebuilt graph. - KeywordExtractor.extract now runs the non-LLM fallback through the same `capped` step, so a configured maxKeywords below 10 (or 0) is honored even without an LLM. Update the end-to-end test and README for the now-throwing lightRAG(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 869131784f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…iterations - KeywordExtractorConfig clamps maxKeywords to >= 0 (and capped() clamps at the slice site too, since the field is mutable), so a negative limit reads as "no keywords" instead of trapping in prefix(_:). - The no-LLM keyword fallback now filters by a small stopword set instead of a length cutoff, so short but meaningful names/acronyms (Ada, Bob, IBM, AI, ML) survive; if every token is a stopword it falls back to the raw tokens so retrieval still runs rather than searching an empty query. - Leiden iterates 0..<max(0, maxIterations), so maxIterations == 0 means "no local moving" (singleton/refinement-only baseline) and negatives can't trap. Tests: short-entity fallback, all-stopword raw fallback, negative-limit clamp, and zero-iteration singletons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0265f520e8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…uild race - Keyword fallback now populates both levels with the same terms, so the default no-LLM path actually searches the high-level community store instead of silently degrading global/theme questions to chunk search only. - communitySourceChunks derives grounding from both TextChunk.entities and the members' Entity.mentions, so externally-constructed graphs that carry only mention evidence still resolve high-level hits to real chunks. - LightRAGStoreCache memoizes an in-flight Task (stored before the first await) so concurrent first queries share one build instead of each re-embedding the whole corpus under actor reentrancy; a failed build is not cached. Tests: mention-based community grounding, and updated the fallback test for the now dual-populated offline keywords. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cef8821d71
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…, grounding - Keyword fallback splits tokens on non-alphanumerics (matching how BM25 and HashEmbedder tokenize the index), so terms like "graph-based" become ["graph","based"] and match indexed content instead of a collapsed "graphbased". - The fallback caps each level to maxKeywords independently instead of routing the duplicated terms through the combined cap, so a small limit no longer starves the low-level (chunk) store while filling the high-level one. - DualLevelRetriever.retrieve returns early for nonpositive topK, before forwarding it to stores that limit with prefix(topK) (which traps on negatives). - InMemorySemanticSearcher sizes its candidate pool to the corpus, so a topK above the default 100 cap can still return enough matches for large stores. - communitySourceChunks also includes Relationship.context chunks for internal edges, completing evidence coverage for externally-constructed graphs. Tests: punctuation splitting, per-level cap, and nonpositive-topK early return. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7f1fece1d0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…g dimension - LightRAGEngine.retrieve returns an empty result for topK <= 0 before building the stores, so a no-op request never triggers corpus-wide embedding (or a throw from a remote/unavailable embedder). Mirrors DualLevelRetriever's guard one level up. - InMemorySemanticSearcher.build reuses a precomputed chunk embedding only when its dimension matches the current embedder; on a mismatch (e.g. a graph saved under a different embedder) it re-embeds, since cosine similarity would otherwise silently return 0 and drop the vector signal. Test: retrieve(topK: 0) over an engine with a throwing embedder returns empty without embedding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
EmbeddingModel requires isAvailable(); the new test double omitted it, breaking the test build. Source is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 478d62339e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…Keywords=0 - GraphRAG.lightRAG() rejects approach == "keyword" with a clear error: LightRAG needs chunk embeddings, which a keyword-only build leaves nil, so fail fast instead of forcing corpus-wide re-embedding (which would throw outright on an unavailable embedder). - LightRAGEngine.retrieve also short-circuits when keywordConfig.maxKeywords <= 0 (guaranteed-empty queries) before building the stores — a plain config check, so no embedding or LLM keyword call happens. - Leiden: documented why local moving intentionally omits the empty-singleton candidate (keeps the port aligned with the simplified single-level upstream; splitting connected over-merges is a deliberate non-goal), so the review can skip re-flagging it. Tests: keyword-only rejection, and maxKeywords=0 short-circuit via a throwing embedder. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
The local DualRetrievalConfig parameter named `config` shadowed the actor's `self.config: Config`; qualify with `self.` so the keyword-approach guard reads the GraphRAG Config that actually has `approach`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 696770c333
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…dex review) Round of Codex P2s all point at one design question: should LightRAG inherit the owning GraphRAG's hybrid-path Config (approach / similarityThreshold / topKResults) and short-circuit empty queries before building stores? Deliberate decision: LightRAG is a self-contained dual-level retrieval subsystem configured via DualRetrievalConfig / KeywordExtractorConfig and a per-call topK. It intentionally does not inherit the GraphRAG hybrid-path Config, and empty queries are handled by the (cached, built-once) retriever returning no hits. Documented at both flagged sites so the decision is explicit rather than an omission; can be threaded through if inheritance is later preferred. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
@codex review |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
2 participants
Extends the Swift port with two optional subsystems from
graphrag-rsthat were previously out of scope: Leiden community detection and LightRAG dual-level retrieval.Leiden community detection
Sources/GraphRAG/Graph/Leiden.swift— partitions the knowledge graph into communities via greedy modularity local-moving plus a refinement pass that splits internally disconnected communities.Two deliberate improvements over the Rust source:
confidenceas an edge weight; the Rust version was unweighted.Reports Newman modularity of the final partition. Only result-affecting knobs are exposed via
LeidenConfig(resolution,maxIterations,minModularityGain) — no dead config.LightRAG dual-level retrieval
Sources/GraphRAG/LightRAG/— answers queries by searching two levels at once:KeywordExtractorsplits the query into high-/low-level keywords via the LLM, with a deterministic offline fallback (tokenize words ≥ 4 chars). Each level is searched independently and the hits are merged (interleavedefault /highFirst/lowFirst/weighted) with dedup by id.LightRAGEngineties it together withdetectCommunities/retrieve/ask.Wiring & docs
GraphRAG.detectCommunities()andGraphRAG.lightRAG()expose both subsystems from the actor.Tests
Tests/GraphRAGTests/LightRAGTests.swiftcovers:lightRAG().ask(...)throughGraphRAG.Everything runs fully offline and builds under Swift 6 strict concurrency.
🤖 Generated with Claude Code
https://claude.ai/code/session_01VUzx3BzstYuB68txcHvAUh
Generated by Claude Code