🎯 SURE-EVAL

A reproducible, version-managed evaluation framework for speech & audio tasks.

🌐 English · 中文 · 📖 Docs · 🚀 Demo

---

✨ What is SURE-EVAL?

SURE-EVAL is a deterministic evaluation system for speech and audio benchmarks:

• 🧩 Pipeline routes — every metric is a declared chain of versioned nodes.
• 📊 Reproducible reports — every run writes report.json + pipeline_description.json.
• ⚖️ Fair comparison — same route + same inputs always produce the same score.

Use it as a CLI, a Python library, or a module in larger agent workflows.

---

🚀 30-Second Quick Start

# Clone and install the lightweight base package
git clone https://github.com/PigeonDan1/sure-evaluation.git
cd sure-evaluation
pip install -e .

# Check the installation
sure-eval doctor

# Prepare a tiny text-only ASR demo
printf "utt1\thello world\nutt2\tthis is a test\n" > /tmp/sure_ref.txt
printf "utt1\thello world\nutt2\tthis is test\n" > /tmp/sure_hyp.txt

# Inspect, describe, and run an in-process ASR metric
sure-eval agent plan asr --language en --metric wer --json
sure-eval metric describe asr --language en --metric wer --output /tmp/asr.json
sure-eval metric run --pipeline /tmp/asr.json \
  --ref-file /tmp/sure_ref.txt --hyp-file /tmp/sure_hyp.txt --output-dir /tmp/asr_eval

# View the score
cat /tmp/asr_eval/report.json | grep score

Input files are tab-separated: <key>\t<text>.

---

📋 Supported Tasks

Task	Metrics	Notes	Guide
ASR	WER, CER, MER	Text-only; canonical normalization routes require [canonical]	docs/tasks/asr.md
S2TT	BLEU, chrF2, XCOMET-XL, BLEURT-20	Base + optional heavy metrics	docs/tasks/s2tt.md
SD	DER	Requires [diarization]	docs/tasks/sd.md
SA-ASR	cpWER, DER	Requires [diarization]	docs/tasks/sa_asr.md
TTS	CER/WER, speaker similarity, MOS	Optional transcription + scoring nodes	docs/tasks/tts.md
VC	CER/WER, speaker similarity, MOS	Optional transcription + scoring nodes	docs/tasks/vc.md
SE	SI-SDR, STOI, PESQ, MOS	Full-reference + no-reference enhancement quality	docs/tasks/se.md
TSE	SI-SDR, speaker similarity, MOS, WER/CER	Signal + optional sim/MOS/ASR nodes	docs/tasks/tse.md
Classification / SER / GR	Accuracy	Text-only, base install	docs/tasks/classification.md
SLU	Accuracy	Text-only, base install	docs/tasks/slu.md
KWS	accuracy, macro_recall, precision, recall, F1, FRR, FAR	Base + optional node	docs/tasks/kws.md

Each guide lists the exact pipeline IDs, nodes, input formats, and CLI examples.

For a machine-readable catalog of every metric → pipeline → node mapping, see docs/pipeline_catalog.jsonl and docs/pipeline_catalog.md. For agent-facing route and environment readiness, see docs/agent_contract.md. Route variants with the same reported metric, such as ASR normalization variants, TTS Qwen3-ASR CER/WER, or spk_sim provider variants, are selected by exact pipeline_id.

Click any task in the CLI for its route:

sure-eval metric describe <task> --help

---

🛠️ Installing Optional Parts

# From a source checkout; the package is not currently published to PyPI.
git clone https://github.com/PigeonDan1/sure-evaluation.git
cd sure-evaluation
pip install -e .

# Development
pip install -e ".[dev]"

# Optional extras, from the repository root
pip install -e ".[diarization]"  # MeetEval for SD / SA-ASR
pip install -e ".[audio]"        # Local audio helpers
pip install -e ".[download]"     # Hugging Face / ModelScope download helpers
pip install -e ".[wetext]"       # compatibility no-op; wetext_norm uses node-local uv
pip install -e ".[canonical]"    # canonical ASR CER/MER/WER routes

# Put caches on a large disk
export SURE_EVAL_CACHE_DIR=/path/to/sure-eval-cache

Prepare a heavy metric environment:

sure-eval env list
sure-eval env setup --task asr --language zh --metric cer --dry-run
sure-eval env setup --task tts --language zh --metrics cer,dnsmos --dry-run
sure-eval env setup --node transcription/qwen3_asr_1_7b --dry-run
sure-eval env setup --task tts --language zh --metrics cer,dnsmos

See docs/installation.md and docs/environment.md for details.

---

📑 Pipeline Catalog

SURE-EVAL exposes every metric as a declarative pipeline. The machine-readable catalog maps each supported task + language + metric to its selected nodes, required input roles, relative route config paths, and Python entrypoints:

• docs/pipeline_catalog.jsonl — one JSON object per line
• docs/pipeline_catalog.md — schema and usage notes

pipeline_id names the computation as task.language.metric.node_version.... metric is the canonical reported metric (cer, wer, spk_sim, wv_mos, macro_recall, etc.). execution_metrics records accepted CLI/API selectors when a task needs a compatibility alias or method-specific selector such as tts_cer, sim/wavlm-large, or wv-mos. Bundle rows use pipeline_kind=bundle and list atomic members in member_pipeline_ids. task_config_path and route_config_path are repository-relative paths; script_entrypoint and executor link the route to runnable code. When a pipeline JSON contains an exact pipeline_id, sure-eval metric run --pipeline ... executes that selected route and rejects reports whose pipeline_id, pipeline_kind, member IDs, or computation nodes diverge from the description.

Example entries:

{
  "task": "ASR",
  "language": "zh",
  "metric": "cer",
  "pipeline_id": "asr.zh.cer.wetext_norm_zh_itn_v1.wenet_cer_v1",
  "pipeline_kind": "atomic",
  "member_pipeline_ids": [],
  "execution_metrics": ["cer"],
  "nodes": ["normalization/wetext_norm", "scoring/wenet_cer"],
  "computation_node_ids": ["normalization/wetext_norm", "scoring/wenet_cer"],
  "task_config_path": "src/sure_eval/evaluation/tasks/asr/manifest.yaml",
  "route_config_path": "src/sure_eval/evaluation/tasks/asr/routes.yaml",
  "describe_entrypoint": "sure_eval.evaluation.scripts.asr.describe_pipeline",
  "script_entrypoint": "sure_eval.evaluation.scripts.asr.run",
  "executor": "sure_eval.evaluation.tasks.asr.pipeline.evaluate_asr_files",
  "required_roles": ["hyp", "ref"]
}

Regenerate it after adding new routes:

python scripts/generate_pipeline_catalog.py

---

🐍 Python API

from sure_eval.evaluation.scripts import describe_pipeline, run_task

# Inspect the route
desc = describe_pipeline("asr", language="zh", metric="cer")
print(desc.node_ids)
# ('normalization/wetext_norm', 'scoring/wenet_cer')

# Run and get a report
report = run_task(
    "asr",
    ref_file="ref.txt",
    hyp_file="hyp.txt",
    language="zh",
    metric="cer",
    output_dir="/tmp/asr_eval",
)
print(report.score)

Legacy one-liner:

from sure_eval.evaluation import SUREEvaluator

evaluator = SUREEvaluator(language="zh")
print(evaluator.evaluate("asr", "ref.txt", "hyp.txt")["cer"])

---

🏗️ Design at a Glance

User request
    │
    ▼
sure-eval metric describe  ──►  pipeline JSON (route + nodes)
    │
    ▼
sure-eval metric run       ──►  report.json + pipeline_description.json

Every metric is a route: a declared combination of task, language, and versioned nodes. Routes live in tasks/<task>/routes.yaml; node metadata lives in nodes/<stage>/<name>/manifest.yaml and node_env.yaml.

This makes every score traceable to the exact code, config, and inputs that produced it.

---

🤝 How to Contribute

Start with docs/contributing.md. It routes each PR type to the right short guide:

• new task
• new metric
• new pipeline route
• node/tool/version change
• maintenance

Use docs/add_a_metric.md only when the category is unclear.

---

📄 License

MIT License. See LICENSE.