AI Daily Evaluation Agent

A rubric-based evaluation harness for AI news selection, report generation, and agent trajectory analysis.

This repository started as an AI daily report generator. It now adds a dedicated evaluation layer that treats the whole daily-report agent as an object to measure: which candidate items were selected, whether the generated report is high quality, and whether the agent's decision trajectory is traceable.

Motivation

An AI daily report should not be judged only by the final text. A polished summary can still hide weak source selection, unverifiable claims, inconsistent ranking, or a missing repair loop.

This project evaluates three connected surfaces:

• Candidate news selection: should this item enter the daily?
• Generated report quality: is the final report factual, dense, readable, useful, and safe?
• Agent trajectory: can we audit the path from raw items to final acceptance?

The goal is a portfolio-ready project for LLM Evaluation, Agent Evaluation, and Benchmark Engineering roles: clear rubrics, deterministic demo runs, structured outputs, and reproducible evaluation cases.

Architecture

flowchart LR
    A[Raw Items] --> B[Candidate Scoring]
    B --> C[Reranking]
    C --> D[Report Generation]
    D --> E[Critique]
    E --> F[Revision]
    F --> G[Evaluation Report]

    B --> H[Candidate Evaluation JSON]
    D --> I[Report Quality Evaluation JSON]
    F --> J[Trajectory Evaluation JSON]
    H --> G
    I --> G
    J --> G

Loading

The existing agent/ package still contains the daily-report pipeline:

• agent.cli: main CLI for running, replaying, publishing, and deterministic eval.
• agent.pipelines.daily_report: collect -> curate -> write -> critique -> publish -> eval -> repair.
• agent.harness.trace: append-only JSONL traces for stage, tool, and LLM events.
• agent.eval.metrics: deterministic checks for report artifacts.
• evals/news_heat: a focused benchmark for news ranking/reranking quality.
• evaluation.artifact_loader: maps stored artifacts/ runs into evaluation schemas.

The new evaluation/ package is a lightweight layer on top of that system. It does not rewrite the pipeline.

Evaluation Dimensions

Candidate News Evaluation

Scores whether a candidate AI news item deserves coverage.

Dimension	Meaning
impact	Importance for practitioners, researchers, founders, or product teams
novelty	Newness, rarity, and non-obviousness
urgency	Whether it needs to be covered now
audience_relevance	Fit for the target daily-report audience
technical_depth	Presence of methods, benchmark, API, repo, or implementation signal
source_reliability	Trustworthiness of the source
coverage_value	Final editorial value after overlap and usefulness checks

Generated Report Evaluation

Scores the generated daily report.

Dimension	Meaning
factuality	Source grounding and link consistency
information_density	Signal per paragraph
structure	Section clarity and ordering
readability	Clear prose for the target audience
actionability	Helps readers decide what to watch or do next
platform_fit	Fit for daily/social publishing
risk_control	Avoids hype, hallucination, and misleading certainty

Agent Trajectory Evaluation

Scores the full agent execution trace.

Dimension	Meaning
task_completion	Research, scoring, reranking, generation, critique, revision, finalization
decision_traceability	Inputs, outputs, decisions, and rationales are visible
evidence_usage	Source information is carried through the workflow
ranking_consistency	Final ranking aligns with scores and rationales
self_correction	Critique leads to meaningful revision

Quick Start

Install dependencies:

pip install -r requirements.txt

Run the evaluation demo without any API key:

python -m evaluation.runner --demo

The demo reads small juya-daily-derived evaluation cases:

• datasets/eval_cases/sample_news_items.json
• datasets/eval_cases/sample_reports.json
• datasets/eval_cases/sample_trajectories.json

You can also evaluate a stored real pipeline run from local artifacts:

python -m evaluation.runner --from-artifacts --run-id 2026-05-30

Both modes write:

• outputs/eval_runs/{run_id}/summary.json
• outputs/eval_runs/{run_id}/evaluation_report.md

The default judge is RuleBasedJudge, so the demo is deterministic and offline-friendly.

Optional LLM-as-a-judge entry point:

python -m evaluation.runner --demo --judge llm --provider mock

If no provider is configured, LLMJudge falls back to the rule-based judge so the project still runs.

Example Output

# Evaluation Report

- Evaluation Run ID: `eval-20260607-120000`
- Judge: `rule_based`
- Overall Score: **8.21/10**

## Top Candidate Items

1. `news_model_release_001` - 8.43/10 (keep)
2. `news_agent_benchmark_002` - 8.08/10 (keep)

## Failure Modes

- No major trajectory failure modes detected in this demo run.

Project Structure

evaluation/
  schemas.py          # Pydantic artifacts for candidates, reports, trajectories, run summaries
  rubrics.py          # Central rubric dimensions, weights, and scoring anchors
  judge.py            # RuleBasedJudge and optional LLMJudge
  runner.py           # CLI runner: python -m evaluation.runner --demo
  metrics.py          # Weighted score helpers
  trajectory.py       # Lightweight trajectory logger
  report_exporter.py  # JSON + Markdown export

datasets/eval_cases/
  sample_news_items.json
  sample_reports.json
  sample_trajectories.json

outputs/eval_runs/
  .gitkeep

docs/
  evaluation_design.md
  failure_modes.md

Why This Matters For LLM / Agent Evaluation

This project demonstrates:

• Benchmark construction: small, inspectable eval cases with expected evaluation surfaces.
• Rubric design: explicit criteria, score anchors, and weights instead of vague LLM scoring.
• LLM-as-a-judge readiness: a judge abstraction with prompts and JSON schema validation.
• Evaluation harness design: repeatable CLI, structured artifacts, and Markdown reporting.
• Trajectory logging: inputs, outputs, decisions, rationales, timing, and failure fields.
• Failure mode analysis: incomplete traces, missing rationales, weak evidence use, and ignored critique.
• Reproducible evaluation: no-key rule-based demo with stable sample datasets.

Existing Daily Pipeline

The original daily-report agent can still be run with:

python -m agent.cli run --provider mock

With real provider configuration:

cp .env.example .env
python -m agent.cli run --provider deepseek

Future Work

• Connect LLMJudge to a real model with calibration examples and judge agreement checks.
• Add more data sources and evaluation cases across model releases, papers, products, policy, and infra.
• Add human preference annotation for candidate selection and report quality.
• Build a larger benchmark suite with gold labels and model/system comparisons.
• Compare different models on the same daily-report task, including cost, latency, and failure modes.

License

MIT