Research Reliability Evaluator

A subagent-group framework with shared audit memory for evaluating the reliability of AI-generated research reports.

Research Reliability Evaluator does not compete with Deep Research systems. It audits their outputs.

Instead of asking whether an AI can generate a report, this project asks whether the report preserves a complete evidence chain from question to sources, evidence, claims, reasoning, and conclusions.

Why Report Generation Is Not Enough

Research reports can sound confident while hiding weak sources, unsupported claims, citation drift, missing uncertainty, and reasoning gaps. This project treats a report as an object to audit. It accepts an original question, an externally generated report, and source documents, then produces structured findings and repair actions.

Project Highlights

Research Reliability Evaluator is not just a scorecard. It is an inspectable audit layer for research reports.

Key differences from many existing LLM evaluation setups:

• Reliability-first, not generation-first. The project does not generate reports. It evaluates whether a report is source-grounded, logically valid, aligned with the question, balanced, and traceable.
• Shared Audit Memory. Every agent reads and writes a structured AuditMemory object. The full audit state is serialized to disk, so users can inspect the exact claims, evidence checks, reasoning failures, trace paths, repair actions, and scores.
• Subagent audit group without chatroom-style instability. Agents are deterministic Python classes with run(memory) methods, not loosely coordinated conversational agents.
• Failure diagnosis and repair planning. The evaluator does not stop at "low score." It emits structured failures such as unsupported_claim, reasoning_gap, missing_traceability, and maps them to concrete repair actions.
• Question-to-source evidence-chain auditing. The system evaluates the path from original question to required dimensions, claims, retrieved evidence, source reliability, reasoning, and final conclusions.
• Runs without an API key. The fallback pipeline uses deterministic retrieval and rule-based checks, making the project runnable, testable, and reproducible without paid model calls.
• Benchmark plus human-baseline calibration. The repository includes scripts to audit all 400 DeepResearch Bench reports and to build topic-matched human-written baselines from cited public sources.

What This Adds Beyond Existing Work

Existing work such as DeepResearch Bench, ReportBench, FActScore, ALCE, RAGAS, and related factuality/citation benchmarks is valuable, but much of it focuses on aggregate evaluation, LLM-as-judge scoring, citation correctness, or factual precision.

This project focuses on a complementary gap:

Can a user inspect why a research report is unreliable, where the evidence chain breaks, and what should be repaired?

The output is therefore not only an overall score. It includes:

• claim-level audit records
• source reliability audits
• evidence support checks
• reasoning validity checks
• required-question-dimension coverage
• traceability paths
• structured failure taxonomy
• repair plan
• human-baseline comparison summaries

This makes the project useful as an engineering tool for auditing research reports, not only as a benchmark leaderboard.

Architecture

The evaluator uses a subagent audit group coordinated through shared audit memory:

External Report + Original Question + Source Documents
        |
Audit Orchestrator
        |
Shared Audit Memory
        |-- Claim Extraction Agent
        |-- Source Reliability Agent
        |-- Evidence Faithfulness Agent
        |-- Reasoning Validity Agent
        |-- Question Alignment Agent
        |-- Completeness & Balance Agent
        |-- Traceability Agent
        |-- Failure Diagnoser Agent
        |-- Repair Planner Agent
        |-- Reliability Scorer Agent

Agents are Python classes with run(memory) methods. They do not chat with each other. They read and write structured AuditMemory, which is serializable and saved to disk.

Six Reliability Dimensions

• Source Reliability: authority, accessibility, freshness, diversity, independence, relevance.
• Evidence Faithfulness: claim-evidence consistency, citation correctness, semantic distortion, overclaiming.
• Reasoning Validity: inference strength, causal reasoning, missing steps, unsupported forecasts.
• Question Alignment: objective fulfillment, scope alignment, topic drift.
• Completeness and Balance: risks, uncertainty, counterevidence, alternative interpretations.
• Traceability and Transparency: question-to-claim-to-evidence-to-source paths, confidence clarity, auditability.

Failure Taxonomy

The evaluator emits structured failures such as unsupported_claim, overclaiming, causal_leap, question_misalignment, missing_risk, missing_uncertainty, missing_counterevidence, missing_traceability, low_quality_source, and stale_source.

Scoring Method

Each dimension starts at 100. Penalties are applied by failure severity:

• critical: -25
• high: -15
• medium: -8
• low: -4

Overall score is a weighted average:

• Source Reliability: 15%
• Evidence Faithfulness: 25%
• Reasoning Validity: 20%
• Question Alignment: 15%
• Completeness & Balance: 15%
• Traceability & Transparency: 10%

Setup

python -m venv .venv
.venv\Scripts\activate
pip install -e .[dev]
copy .env.example .env

The project runs without an OpenAI API key. Optional LLM mode uses:

OPENAI_API_KEY=
OPENAI_MODEL=gpt-4o-mini
USE_LLM=true

Demo

python demos/run_demo.py

Or run the CLI directly:

python -m rre.cli audit \
  --question "Does this report provide a reliable evidence chain for its conclusion?" \
  --report data/input_reports/demo_report.md \
  --sources data/sources/demo_sources \
  --output outputs/demo \
  --no-llm

Import Public Deep Research Reports

You can import real generated reports from the public Hugging Face dataset muset-ai/DeepResearch-Bench-Dataset:

python scripts/import_deepresearch_bench.py --model openai --count 1
python scripts/import_deepresearch_bench.py --model gemini --count 1
python scripts/import_deepresearch_bench.py --model perplexity --count 1

Supported model keys are openai, gemini, perplexity, and grok.

The importer writes the benchmark prompt and generated article into data/input_reports/, creates a local citation-index source folder under data/sources/, and runs the audit into outputs/.

By default, the importer normalizes localized Chinese reference labels from the source dataset into English, such as converting reference-section headings to References and access-date labels to accessed April 28, 2025. Use --raw if you need byte-for-byte dataset text.

For a full batch run over all 100 tasks across all 4 models:

python scripts/batch_deepresearch_bench.py --count 100 --max-workers 4

This audits 400 generated reports and writes batch summaries to outputs/batch/.

The full per-case benchmark reports and audit folders are intentionally ignored by .gitignore because they are reproducible and can exceed hundreds of MB. Commit the batch summaries and scripts, not the generated per-case artifacts.

Important limitation: the benchmark report files include generated articles with citation markers, but not downloaded full source documents. The importer therefore creates a citation-index source file for pipeline testing. For a full evidence-faithfulness audit, collect the cited source documents as text files and place them in the case source folder.

Using Your Own Report

Place your report in a text or Markdown file and put source text files in a folder. Then run:

python -m rre.cli audit --question "YOUR QUESTION HERE" --report path/to/report.md --sources path/to/sources --output outputs/my_case --no-llm

Use --use-llm to enable OpenAI calls if your .env is configured.

Human-Written Baselines

For calibration, you can also audit public human-written articles or reports:

python scripts/import_human_report_url.py \
  --url https://ourworldindata.org/age-structure \
  --question "What are the major global trends in population aging and age structure?" \
  --case-id human_owid_age_structure

Human-written baselines are useful as positive controls, but they are not automatically perfect gold standards. Use the article's actual research question or scope as --question; otherwise the evaluator will correctly penalize question-alignment mismatch.

Current Benchmark Signals

The current repository has been run on:

• 400 AI-generated DeepResearch Bench reports: 100 tasks x 4 systems.
• 100 automatically selected topic-matched human-written baselines, chosen from URLs cited by the AI reports.
• A hand-checked task-51 pilot comparing Japan-aging AI reports with human-written synthesis/report pages.

Current batch-level results:

Group	Mean Overall Score
Auto-selected human baselines	84.73
Grok DeepResearch reports	82.84
OpenAI DeepResearch reports	78.73
Perplexity Research reports	78.46
Gemini DeepResearch reports	74.95

Interpretation: the automatic human baselines are topic-matched but not manually verified gold standards. They are useful for calibration, not final truth. The result is still informative: human-written synthesis sources score higher on average under the same audit pipeline, while AI reports show more evidence, traceability, and reasoning failures.

Generated summaries:

• outputs/batch/batch_summary.md
• outputs/batch/batch_model_summary.csv
• outputs/batch/batch_failure_type_summary.csv
• outputs/human_baselines/auto_human_vs_ai_by_model.md
• outputs/human_baselines/task51_human_vs_ai.md

A longer project report with method and result figures is available at:

• docs/research_reliability_evaluator_report.md

Output Files

The pipeline writes:

• audit_memory.json
• claims.jsonl
• source_audit.jsonl
• evidence_checks.jsonl
• reasoning_checks.jsonl
• alignment_checks.jsonl
• completeness_checks.jsonl
• traceability_checks.jsonl
• failures.jsonl
• repair_plan.jsonl
• dimension_scores.json
• reliability_score.json
• audit_report.md
• executive_summary.md

Streamlit App

streamlit run app/streamlit_app.py

The app supports pasting a question, pasting or uploading a report, uploading source text files, running the audit, and inspecting scorecards, failures, repair actions, report text, and shared audit memory JSON.

Roadmap

• Stronger citation parsing and claim-to-citation mapping.
• More precise contradiction detection.
• Source metadata extraction from PDFs and web pages.
• Human review workflow for high-severity failures.
• Pluggable retrieval backends and vector stores.
• Calibration datasets for scoring validation.