Feature: AI Quality Feedback, Golden Datasets, and Regression Evaluation Workbench

[hoangsonww]

Summary

Add an AI quality feedback and evaluation workbench that captures user feedback on generated outputs, curates regression datasets, and runs repeatable evaluations across prompts, providers, retrieval settings, and document workflows before changes ship.

Problem / Opportunity

DocuThinker already has many AI-powered paths: summarization, key ideas, recommendations, chat, RAG, agentic workflows, provider failover, token budgeting, and cost tracking. As those paths expand, quality can regress silently when prompts change, models are upgraded, retrieval settings shift, or document extraction behavior changes.

The repository has a lightweight continuous_learning.py feedback helper and context/cost observability, but there is no productized workflow for collecting structured user ratings, turning representative examples into golden test sets, comparing model or prompt variants, and blocking releases when answer quality drops. This feature would make AI behavior measurable and governable instead of anecdotal.

Proposed Feature

Create a cross-service quality loop for AI outputs:

• In-product feedback controls for summaries, chat answers, recommendations, rewritten content, and report sections.
• Structured feedback schema capturing rating, reason categories, optional comments, source document/version, prompt version, provider/model, retrieval metadata, trace ID, latency, token usage, and cost.
• Admin/evaluator workbench for reviewing feedback, promoting examples into curated golden datasets, and tagging them by workflow, document type, language, risk level, and expected behavior.
• Offline and on-demand evaluation runner that replays golden examples against selected prompt/model/retrieval configurations.
• Scoring outputs for groundedness, citation coverage where available, instruction following, semantic similarity to expected outputs, toxicity/safety flags, latency, and cost.
• Regression gates that can run in CI or pre-deploy checks and fail when quality, latency, or cost thresholds are breached.
• Dashboard/API surface for quality trends by workflow, prompt version, provider, language, and document category.

Scope

• Backend persistence for feedback events, golden examples, evaluation runs, scores, and prompt/model configuration metadata.
• API endpoints for submitting feedback, managing curated datasets, launching evaluations, and reading evaluation summaries.
• Orchestrator hooks to attach trace IDs, prompt versions, provider/model data, token/cost records, retrieval details, and tool execution metadata to AI responses.
• AI/ML evaluation utilities for scoring outputs and replaying workflows against fixed inputs.
• Web UI for end-user feedback capture and maintainer-facing review/evaluation workflows.
• CI integration that can run a small smoke dataset on pull requests and a larger scheduled benchmark.
• Documentation for dataset curation, scoring interpretation, threshold configuration, and privacy handling.

Acceptance Criteria

• Users can rate AI outputs with positive/negative feedback, reason categories, and optional comments without leaving the current document/chat/report workflow.
• Feedback records include enough metadata to reproduce the output: document ID/version, workflow type, prompt version, model/provider, retrieval configuration, trace ID, latency, token usage, and cost.
• Maintainers can review feedback and promote selected examples into named golden datasets with expected output notes or rubric criteria.
• An evaluation runner can replay at least summarization, chat/RAG, recommendations, and rewriting workflows against a chosen dataset.
• Evaluation results include quality scores, groundedness/citation coverage when sources exist, latency, token usage, and estimated cost by provider/model.
• Prompt/model/retrieval changes can be compared side by side against a baseline run.
• CI can execute a bounded smoke evaluation and fail when configured regression thresholds are exceeded.
• The UI/API clearly separates private user feedback from curated datasets and avoids exposing document content to unauthorized users.
• Tests cover feedback submission, dataset promotion, replay determinism controls, scoring aggregation, permission enforcement, and CI threshold behavior.

Non-Goals

• Fully automated model fine-tuning in the first version.
• Replacing existing unit, integration, or E2E tests.
• Guaranteeing a single universal quality score for every document type.
• Public leaderboard or benchmark publishing.

Dependencies / Risks

• Evaluation data may contain sensitive document excerpts, so dataset curation needs permission checks, redaction options, and retention rules.
• LLM-as-judge scoring can be noisy; rubric scoring should be versioned and combined with deterministic metrics where possible.
• Replay results may vary across providers unless temperature, prompt versions, retrieval snapshots, and model identifiers are pinned.
• CI evaluation must be bounded to avoid excessive provider cost and flaky release gates.

Open Questions

• Which workflows should be part of the first smoke dataset: summaries, chat/RAG, recommendations, rewriting, or report generation?
• Should golden datasets store full source text, redacted snippets, or references to immutable document versions?
• Which thresholds should block CI versus only warn in scheduled benchmark reports?
• Who should have access to the evaluator workbench in the initial permission model?