Feature: Document Collections + Source-Grounded Citations & Highlight Viewer

[hoangsonww]

Summary

Introduce Collections (grouped docs) and “Ask Across Docs” with source-grounded citations and an inline highlight viewer. Answers show sentence-level citations that link to exact passages in the original files, with side-by-side highlight previews and export to Markdown/PDF.

Motivation

• Users often need insights across multiple files (reports, contracts, research).
• Trust requires traceability: every answer should cite where it came from.
• Faster review: jump directly to highlighted passages without leaving the app.
• Shareable outputs: export answers with citations for teammates/auditors.

Goals

1. Create Collections (add/remove docs, reorder, share, permissions).
2. Ask Across Docs: one query over all docs in a Collection.
3. Grounded citations: sentence/claim lines map to (docId, page?, span start–end).
4. Highlight viewer: side panel shows the exact passage, with text & (for PDFs) page preview.
5. Exports: copy/Markdown/PDF with numbered citations and appendix of sources.
6. Works with existing backends (MongoDB/Firestore + Redis cache) and current RAG pipeline.

Non-Goals

• Full collaborative live editing (future).
• Legal-grade OCR of scanned images (initial OCR is best effort).

UX Outline

• Collections Page: create, rename, delete; add docs from uploads/Drive; visibility (private/shared link).

• Ask tab: prompt box + filters (time range, file types, “strict citations only”).

• Answer panel:

  • Body with superscripted [1], [2]… citations
  • Right sidebar: list of sources; clicking a citation opens an inline highlight (text snippet; for PDFs, a page preview with overlay box).

• Export button → Copy MD / Download PDF (includes references).

Data Model (MongoDB + cache)

Collection {
  _id, ownerId, title, docIds: string[], visibility: 'private'|'link',
  createdAt, updatedAt
}

GroundedAnswer {
  _id, collectionId, userId, query: string,
  answerMarkdown: string,
  citations: Array<{
    id: string,                  // "c1", "c2", ...
    docId: string,
    page?: number,               // for PDFs
    span: { start: number, end: number },  // offsets in normalized text
    text: string                 // cached snippet for fast display
  }>,
  createdAt
}

API (Express/GraphQL)

POST   /api/collections                   { title, docIds? }
GET    /api/collections/:id
PATCH  /api/collections/:id               { title?, docIds?, visibility? }
DELETE /api/collections/:id

POST   /api/ask                           { collectionId, query, options }
GET    /api/answers/:id                   // returns answer + citations
GET    /api/answers/:id/export?format=md|pdf

GraphQL (examples):

mutation CreateCollection($title: String!, $docIds: [ID!]) { ... }
query Ask($collectionId: ID!, $q: String!) { ask(collectionId:$collectionId, query:$q) { answerMarkdown citations { id docId page text } } }

Ingestion & Retrieval Notes

• Normalization: During ingestion, store a normalized text stream with a per-chunk byte/char index map back to the raw source for highlight spans.
• Chunking: 1–2k tokens with overlap; store (docId, chunkId, start, end).
• Retrieval: Top-k semantic + BM25 hybrid; rerank; sentence align to build spans that map cleanly to highlights.
• Citation building: For each answer sentence, attach the best-scoring span (include page for PDFs via PDF parser).
• Caching: Redis for recent answers and snippet payloads.
• OCR: If PDF has images only, run lightweight OCR (toggle via env) and flag low-confidence spans.

Security & Privacy

• Respect Collection visibility; only members with access can ask/export.
• Strip PII in exported text when DOCUTHINKER_REDACT_PII=true (basic regex/NLP redaction pass).
• Signed, time-limited URLs for preview images.

Exports

• Markdown:

  Answer text ... [1][2]
  
  ---
  References
  [1] <doc title> — p. 6: "quoted snippet..."
  [2] <doc title> — p. 12: "quoted snippet..."
  

• PDF: same layout, with page thumbnails for cited passages.

Configuration

• .env

  • RAG_TOP_K=8
  • RERANK_TOP_K=4
  • OCR_ENABLE=false
  • EXPORT_MAX_TOKENS=4000
  • CITATION_STRICT=true // drop any sentence that can’t be grounded
  • CACHE_TTL_ANSWERS_SECONDS=3600

Acceptance Criteria

• Users can create/edit/delete Collections and add docs from existing uploads/Drive.
• Ask Across Docs returns an answer within the Collection scope.
• Every answer sentence has at least one citation; clicking shows an inline highlight.
• PDF citations show page preview with overlay; text docs show in-context snippet.
• Export works (MD & PDF) with correctly numbered references.
• Permissions enforced (private vs link).
• Unit tests for retrieval, sentence alignment, and citation mapping; e2e test for ask→click citation→export.
• README updated with screenshots and env flags.

Implementation Plan (High-Level)

1. Models & Routes: Collections CRUD; ask/answers endpoints; exports.
2. Ingestion update: store normalized text & offset map; capture page indices for PDFs.
3. Retriever: hybrid search + rerank; sentence alignment & span mapping util.
4. Frontend: Collections UI; Ask panel; answer renderer with superscripts; sidebar highlight viewer; export button.
5. PDF preview: small service to render cited pages (cached images).
6. Tests & Docs: unit/integration/e2e; README & screenshots.

Future Enhancements

• Live citations during chat (streaming tokens with clickable refs).
• Workspace sharing (role-based Collection access).
• Cross-Collection search with filters (author/date/type).

[hoangsonww]

backlog for now