Skip to content

RAG-2: BM25 keyword index over scraped Live docs #67

Description

@nonogv

Context

Depends on #66 (RAG-1). Part of the v2 RAG milestone.

What

Implement a lightweight BM25 scorer (~80 lines, no external deps) over data/live-docs.json.

File: src/agent/docs-index.ts

Export:
```ts
/** Returns the top-k most relevant chunks for a query. Returns [] if index is absent. */
export function retrieveLiveDocs(query: string, topK?: number): DocChunk[];

/** True if the local docs index is older than maxAgeDays (default 30). */
export function isDocIndexStale(maxAgeDays?: number): boolean;
```

Implementation notes:

  • Tokenise query and corpus: lowercase, split on word boundaries, strip punctuation
  • BM25 with standard defaults: k1=1.5, b=0.75
  • Return top topK (default 3) chunks above a minimum relevance threshold
  • If data/live-docs.json is absent, return [] silently — never throw

Tests in src/agent/docs-index.test.ts:

  • Returns correct top result for known device queries ("Operator FM", "Glue Compressor ratio")
  • Returns [] when data file is absent
  • Respects topK limit
  • isDocIndexStale returns true for an index older than the threshold

Acceptance criteria

  • retrieveLiveDocs('Operator FM synthesis') returns a chunk mentioning Operator from the manual
  • retrieveLiveDocs('anything') returns [] when data/live-docs.json does not exist
  • No external runtime dependencies added
  • All existing tests still pass

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestv2v2 power features

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions