Feature: Document Version History, Semantic Diffing, and Change-Aware Analysis

[hoangsonww]

Summary

Add document revision history and semantic diffing so users can compare versions of the same document, understand what changed, and regenerate AI outputs only where the document changed materially.

Problem / Opportunity

DocuThinker currently treats each upload as a document to summarize, analyze, chat with, and store in history. In many real workflows, documents evolve: contracts receive redlines, policies are updated, research papers get new drafts, and reports are revised before publication.

Uploading each revision as an unrelated document loses continuity. Users need to know what changed, whether prior conclusions still hold, and which summaries or recommendations are stale.

Proposed Feature

Introduce versioned documents and change-aware AI workflows:

• Allow users to upload a new revision against an existing document.
• Store immutable version metadata, extracted text snapshots, source filename, uploader, timestamp, and processing status.
• Provide text-level and semantic diff views, including additions, removals, moved sections, and changed claims.
• Compare summaries, key insights, sentiment, recommendations, and chat answers across versions.
• Mark generated outputs as current, stale, or partially stale based on revision changes.
• Support incremental re-analysis where only changed chunks are re-indexed or reprocessed when possible.

Scope

• Backend document-version model and migration path from single-version documents.
• AI/ML pipeline for structural diffing and semantic change classification.
• Web UI for version timeline, side-by-side diff, and changed-insights view.
• API updates for fetching versions, uploading revisions, and requesting version comparisons.
• Tests for version lineage, diff output, stale-analysis status, and incremental processing behavior.

Acceptance Criteria

• A user can attach a new upload as a revision of an existing document instead of creating an unrelated document.
• Document detail pages show a version timeline with uploader, timestamp, filename, and processing state.
• Users can compare any two versions and see text-level additions/removals plus a semantic summary of changed claims.
• Existing AI outputs show whether they were generated from the latest version or an older version.
• Users can regenerate summaries, recommendations, and RAG/chat indexes for a selected version.
• Search and document history make the distinction between document lineage and individual revisions clear.
• APIs prevent cross-user or cross-workspace access to historical revisions.
• Tests cover revision upload, version listing, diff generation, stale output labeling, and deletion behavior.

Non-Goals

• Real-time collaborative editing.
• Native Word tracked-changes rendering.
• Git-like branching and merging of document versions.
• Reconstructing visual PDF layout diffs beyond text and section-level references.

Dependencies / Risks

• Current document storage may need a compatibility layer so existing documents appear as version 1.
• Diff quality depends on robust text extraction and chunk alignment across formats.
• Incremental reprocessing should not compromise retrieval accuracy when moved or rewritten sections affect context.

Open Questions

• Should deleting a document delete every revision, or should revision-level deletion be supported?
• Should users be able to compare versions across different documents when they are not formally linked?
• What minimum confidence should be required before marking prior analysis as stale automatically?