lamb-eval

Evaluation framework for LAMB (Learning Assistants Manager and Builder). Orchestrates 17 evaluation metrics across 4 domains using the DeepEval, RAGAS, and Opik judge frameworks (plus custom adapters) under a unified CLI, producing structured reports with versioned snapshots for reproducibility.

Quick Start

# Install (Python 3.11+, pip >= 25 required for --group flag / PEP 735)
pip install -e "." --group test

# Copy environment file and set your OpenRouter API key
cp .env.sample .env
# Edit .env and set OPENROUTER_API_KEY=sk-or-v1-...

# Sanity check — one case, local judge, no API cost
lamb-eval run --dataset datasets/fixtures/smoke_test.yaml --judges local --max-cases 1

# Run only local metrics on a small fixture — no judge LLM, zero API cost
lamb-eval run --dataset datasets/fixtures/simple_rag_en.yaml --judges none

# Standard run with the local LM Studio / Ollama judge (no API cost)
lamb-eval run --dataset datasets/gold/en/index.yaml --judges local

# Run with the OpenRouter API judge (requires OPENROUTER_API_KEY)
lamb-eval run --dataset datasets/gold/en/index.yaml --judges api

Web UI

A local, browser-based companion to the CLI: configure and launch a run, watch its progress live (each metric and each case as it finishes), browse snapshots with rich JSON rendering, and explore the gold datasets with faceted filtering. It is a thin UI over the framework — no auth, no database, not for production — and triggers runs by shelling out to lamb-eval run, so a metric crash can't take down the server.

# Containerized (recommended) — build + serve, then open http://localhost:8090
make webui                 # = docker compose up --build webui

# Local dev (run from the repo root so relative data dirs resolve)
pip install -e "." --group ui
make webui-local           # = uvicorn webui.app:app --reload --port 8000

v1 launches runs against saved response snapshots (the --snapshot flag) plus the local/none/api judge; live LAMB runs are a planned follow-up. The live view reflects each progress event within ~200 ms. See webui/.

Judges

The --judges flag (default local) selects how metrics are judged:

Value	Groups run	Judge	API cost
`none`	[L] only	No judge LLM — Group L local metrics only	None (offline)
`local` (default)	[L,A,B,C]	Local LM Studio / Ollama model	None
`api`	[L,A,B,C]	Remote API model (OpenRouter by default)	Yes (needs the provider API key)

Local backend (--judges local): --judge-backend ollama|lmstudio overrides the JUDGE_BACKEND env var (default lmstudio). The model is resolved per backend (see Configuration); override via OLLAMA_LOCAL_JUDGE_MODEL / LMSTUDIO_LOCAL_JUDGE_MODEL. At run start the configured backend is probed; if it is unreachable the run falls back to Ollama, and fails with a clear error if neither is up (never silently runs with no judge).
API model (--judges api): --judge-model MODEL overrides the judge model (default: evaluation.judge_model in config, then mistralai/mistral-small-3.1-24b-instruct).
API provider (--judges api): --judge-provider openrouter|openai|openai_compatible (or JUDGE_API_PROVIDER env, or evaluation.judge_provider in config) selects the remote endpoint. OpenRouter is the default; the unconfigured behavior is unchanged. Each provider has its own base-URL / API-key env vars: openrouter → OPENROUTER_BASE_URL / OPENROUTER_API_KEY; openai → OPENAI_BASE_URL / OPENAI_API_KEY; openai_compatible → JUDGE_API_BASE_URL (required) / JUDGE_API_KEY.

Cap and filter the case set with --max-cases N (stratified sample), --language, --domain, --split, and --scopes.

Environment Setup

Copy .env.sample to .env and configure:

cp .env.sample .env

Variable	Required	Description
`OPENROUTER_API_KEY`	For `--judges api` (default provider)	Default remote LLM judge provider. Get one at openrouter.ai/keys. Not needed for `--judges none` or `--judges local`.
`JUDGE_API_PROVIDER`	Optional	Remote judge provider for `--judges api`: `openrouter` (default), `openai`, or `openai_compatible`. Equivalent to `--judge-provider`. Provider-specific keys: `OPENAI_API_KEY` / `OPENAI_BASE_URL` (openai) and `JUDGE_API_KEY` / `JUDGE_API_BASE_URL` (openai_compatible).
`LAMB_SERVER_URL`	For live eval	LAMB backend URL (default: `http://localhost:9099`). In production, point this at the real host — every endpoint resolves through `core/endpoints.py` (env > localhost dev default). Not needed when using `--snapshot` or pre-populated datasets.
`LAMB_BEARER_TOKEN`	For live eval	LAMB API authentication token.
`LAMB_KB_URL`	For KB tooling	LAMB KB server URL (default: `http://localhost:9090`). Used by `scripts/build_calibration_assistant.py`.
`LMSTUDIO_BASE_URL` / `OLLAMA_BASE_URL`	For `--judges local`	Override the local judge endpoints (defaults `http://localhost:1234/v1` / `http://localhost:11434/v1`).

The framework validates the active provider's API key (default OPENROUTER_API_KEY) at startup when API-judged metrics (Groups A/B/C) are selected. If the key is missing, it prints a clear error and exits before wasting time on local metrics.

CLI Commands

`lamb-eval run`

Run an evaluation suite against a dataset.

lamb-eval run --dataset <path> [options]

Required (one of):
  -d, --dataset           Path to dataset YAML file
  -s, --snapshot          Use a saved response snapshot (e.g. v001) instead of calling LAMB

Judges:
  --judges                none (Group L local metrics only, no judge LLM),
                          local (all groups, local LM Studio/Ollama judge — default),
                          api (all groups, remote API judge — OpenRouter by default)
  --judge-backend         Local backend for --judges local: ollama or lmstudio
                          (overrides JUDGE_BACKEND env, default lmstudio)
  --judge-provider        Remote provider for --judges api: openrouter (default),
                          openai, or openai_compatible (overrides JUDGE_API_PROVIDER env)
  --judge-model           API judge model for --judges api
                          (default: evaluation.judge_model in config, then mistral-small-3.1)

Filtering / sampling:
  --max-cases N           Cap the number of cases evaluated (stratified sample)
  --language              Comma-separated language codes (e.g. en,es,ca,eu)
  --domain                Comma-separated domains
  --split                 Dataset split: train, dev, or test
  --scopes                Comma-separated: rag_core,multi_turn,safety,educational,nlp_baselines

Output:
  -o, --output-dir        Output directory (default: reports/)
  --output-formats        Comma-separated: json,html
  --case-concurrency      Parallel cases per metric (default from config)
  --assistant-id          Override the run-level default LAMB assistant
  --save-responses        Also save the LAMB responses fetched on this live run as a
                          reusable response snapshot (opt-in; no-op on --snapshot replay)
  --snapshot-description   Description stored with the snapshot saved via --save-responses
  --config                Path to an alternate config YAML (default: config/default.yaml)
  -v, --verbose           Enable debug logging
  -q, --quiet             Suppress INFO logs and the progress bar

`lamb-eval snapshot`

Manage versioned snapshots of LAMB responses and evaluation results.

# Capture LAMB responses as a versioned snapshot
lamb-eval snapshot capture --dataset datasets/fixtures/simple_rag_en.yaml

# Or persist responses as a snapshot during a normal live run — one pass, no
# separate capture step (opt-in; ignored on --snapshot replay):
lamb-eval run --dataset datasets/fixtures/simple_rag_en.yaml --judges local \
  --save-responses --snapshot-description "smoke run"

# List all snapshots (responses + eval results)
lamb-eval snapshot list
lamb-eval snapshot list --type responses
lamb-eval snapshot list --type results

# Re-evaluate using a saved snapshot (skip LAMB API calls)
lamb-eval run --dataset datasets/fixtures/simple_rag_en.yaml --snapshot v001

Selecting a LAMB assistant

lamb-eval auto-discovers a healthy assistant on the target LAMB instance — no flag needed in the common case. At run start it queries GET /v1/models, sends a 1-token chat-completions probe to each candidate (capped at 10), and uses the first one that returns HTTP 200. To override the auto-discovery, in priority order:

--assistant-id lamb_assistant.N (CLI flag, highest priority)
Top-level assistant_id: field in the dataset YAML
lamb.default_assistant_id in config/default.yaml

Whichever explicit value you supply is also probe-validated at startup, so a stale ID fails fast instead of after 1000 cases. If no source supplies an ID and auto-discovery turns up nothing, lamb-eval exits with code 2 and a message listing the resolution options.

`lamb-eval report`

Regenerate a report from a previous evaluation JSON.

lamb-eval report --input reports/eval_abc123_20260323.json --format html

Metrics (20 total)

Every metric is tagged with two axes for filtering:

Cost Group: L (local, free), A/B/C (API, increasing cost)
Evaluation Scope: rag_core, safety, educational, nlp_baselines

Judging is provided by the DeepEval, RAGAS, and Opik frameworks plus custom adapters.

RAG Core (5 metrics)

Metric	Group	Framework	What it measures
`faithfulness.deepeval`	A	DeepEval	Answer grounded in retrieval context (LLM judge)
`answer_relevancy.deepeval`	B	DeepEval	Answer addresses the question (LLM judge)
`factual_correctness.ragas`	A	RAGAS	Factual accuracy vs reference answer
`context_recall.ragas`	B	RAGAS	Fraction of reference info covered by retrieval
`context_precision.ragas`	B	RAGAS	Relevance of the retrieved chunks to the question (reference-free; informational)

Safety (4 metrics)

Metric	Group	Framework	What it measures
`toxicity.deepeval`	C	DeepEval	Toxic/harmful content detection (higher=safer)
`demographic_bias.opik`	C	Opik	Demographic stereotyping level (lower=less bias)
`prompt_injection.opik`	L	Opik	Prompt injection detection (1.0=safe)
`abstention_accuracy.custom`	A	Custom	Correctly abstains when KB lacks info (AbstentionBench)

Educational Quality (9 metrics)

Metric	Group	Framework	What it measures
`aitutor_quality.aitutor`	A	Custom	Combined tutoring quality: mistake identification, location, guidance, actionability
`scaffolding_quality.custom`	A	Custom	Progressive scaffolding (MRBench + KMP-Bench)
`socratic_quality.custom`	A	Custom	Socratic questioning form: presence, relevance, guided discovery, non-revelation (ERL4SIIP; diagnostic descriptor — excluded from the verdict)
`misconception_remediation.custom`	A	Custom	Misconception identification and correction (BEA 2025)
`tutor_tone.custom`	A	Custom	Encouraging vs neutral vs offensive tone (MRBench)
`grade_appropriateness.custom`	L	Custom	Distance from target grade level
`scope_adherence.custom`	A	Custom	Stays within system prompt knowledge scope
`response_coherence.custom`	A	Custom	Contextual coherence with student's question (MRBench)
`good_tutoring.custom`	A	Custom	Reference-free pedagogical quality: 0.01·socratic + 0.99·substance (LOO-fitted, n=32 single-rater pilot; reference-free; informational)

Multi-turn (1 metric)

Metric	Group	Framework	What it measures
`socratic_consistency.custom`	A	Custom	Cross-turn coherence, misconception tracking, and questioning consistency form over a conversation (judge + heuristic; diagnostic descriptor — excluded from the verdict)

Diagnostic descriptors. A metric whose metadata carries direction="context_dependent" and has no threshold (ungated, listed in INFORMATIONAL_METRICS) is a diagnostic form descriptor: fully scored and reported for reference, but excluded from the pass/fail verdict. The two socratic metrics (socratic_quality, socratic_consistency) follow this pattern — they measure Socratic form, not teaching quality, and higher form does not mean better teaching.

NLP Baselines (1 metric)

Metric	Group	Framework	What it measures
`semantic_similarity.ragas`	L	RAGAS	Semantic similarity between answer and reference (local embeddings)

Datasets

Datasets are YAML files validated against datasets/schema.json. Two types supported:

type: single_turn — standard question/answer evaluation
type: multi_turn — conversation evaluation with turn-by-turn scoring

The repo ships two dataset trees:

`datasets/gold/` — gold-standard evaluation corpus

The authoritative multi-language, pedagogically-rich evaluation corpus built for issue #3. Layout: datasets/gold/<lang>/<domain>/<subdomain>.yaml plus per-language safety_abstention.yaml and multi_turn.yaml, aggregated by datasets/gold/<lang>/index.yaml (one per language) and datasets/gold/index.yaml (global).

Languages: English (en), Spanish (es), Catalan (ca), Basque (eu) — matching the four locales LAMB itself supports.
Domains (10): mathematics, computer_science, software_engineering, natural_sciences, social_sciences, humanities, arts_and_design, architecture, language_and_writing, study_skills_meta. Each split into 6–22 subdomains (e.g. mathematics/calculus_optimization, natural_sciences/physics_quantum).
Cross-cutting: safety_abstention.yaml (~50 cases per language) for off-topic / harmful / out-of-scope / misconception probes; multi_turn.yaml (~40 conversations per language) split between Socratic and backtracking strategies.
Per-language scale: ≥500 single-turn cases + ≥40 multi-turn conversations.
Quality bar (enforced by tests/unit/test_gold_dataset_schema.py): every reference answer ≥150 chars, ≥20% of each file uses Socratic or backtracking strategy, ≤30% direct, ≥10% of cases per language have should_abstain: true, no duplicate IDs, cross-language ID parity (same <domain>_<subdomain>_NNN tests the same concept in every language).
Authoring guide: see datasets/gold/README.md and the dataset-authoring skill at .claude/skills/dataset-authoring/SKILL.md.

Use the gold tree by passing the per-language index (or the global one) and any of the new metadata filters:

lamb-eval run --dataset datasets/gold/en/index.yaml --judges local --max-cases 100
lamb-eval run --dataset datasets/gold/en/index.yaml --domain mathematics --split test
lamb-eval run --dataset datasets/gold/es/index.yaml --judges api
lamb-eval run --dataset datasets/gold/index.yaml --judges api

The --language, --domain, --split flags filter cases by case.metadata. Comma-separated values for --language and --domain.

`datasets/fixtures/` — test fixtures

Small files used by the test suite (kept untouched by gold authoring):

Fixture	Cases	Use
`smoke_test.yaml`	1	E2E smoke test (`tests/e2e/test_smoke.py`)
`simple_rag_en.yaml`	5	Integration tests (`tests/integration/test_end_to_end.py`)
`five_case_full.yaml`	5	Demonstrates the full metadata schema with all 21 optional fields

Metadata fields for specialized metrics

Some metrics require specific fields in case.metadata:

Field	Type	Required By
`system_prompt`	str	`scope_adherence.custom`
`target_grade_level`	int	`grade_appropriateness.custom`
`should_abstain`	bool	`abstention_accuracy.custom`
`student_misconception`	str	`misconception_remediation.custom`

Metrics gracefully return score=0.0 with a descriptive reason when required metadata is missing — they never crash.

Output Structure

Reports (`reports/`)

Transient working directory containing only the latest run's output. Old files are cleaned on each new run.

reports/
  eval_{run_id}_{timestamp}.json     # Machine-readable full results
  results_{run_id}_normal.txt        # Human-readable summary (one row per metric)
  results_{run_id}_extended.txt      # Detailed per-case scores
  results_latest_normal.txt          # Latest run symlink
  results_latest_extended.txt        # Latest run symlink
  history/
    index.json                       # Chronological log referencing snapshots

Snapshots (`snapshots/`)

Immutable versioned archive for reproducibility. Two independent version tracks:

snapshots/
  index.json                         # Master index of all snapshots
  lamb_responses/
    v0001_20260323_simple_rag/       # LAMB LLM outputs
      metadata.json
      responses.yaml
  eval_results/
    v0001_20260323_resplive/         # Evaluation results (4 files)
      metadata.json                  # Run metadata (judge models, filters, timing)
      eval_{run_id}_{timestamp}.json # Full evaluation JSON
      results_{run_id}_normal.txt    # Summary results
      results_{run_id}_extended.txt  # Detailed results

Each evaluation run automatically saves a snapshot. The history/index.json maintains a chronological log of all runs with references to their snapshots.

Configuration

File	Purpose
`config/default.yaml`	The only config file — thresholds, concurrency, sampling, groups, scopes, frameworks, `judge_model`, local-model settings, output formats
`.env`	API keys and LAMB connection (copy from `.env.sample`)
`datasets/schema.json`	JSON Schema for dataset validation

The evaluation: block in config/default.yaml carries concurrency, case_concurrency, metric_timeout, sampling, groups, scopes, frameworks, judge_model, judge_provider, thresholds, local_models. The judge_model key is the default model for --judges api; judge_provider selects the remote endpoint (openrouter default, openai, or openai_compatible).

The local judge for --judges local is resolved per JUDGE_BACKEND (--judge-backend overrides):

Backend	Default model	Override env var
`ollama`	`ollama/qwen2.5:0.5b`	`OLLAMA_LOCAL_JUDGE_MODEL`
`lmstudio`	`lmstudio/mlx-community/gemma-4-26b-a4b-it-4bit`	`LMSTUDIO_LOCAL_JUDGE_MODEL`

Environment variables in config files are substituted using ${VAR} or ${VAR:-default} syntax.

Architecture

Run Configuration and Metric Filtering

Runs are configured by config/default.yaml plus a flat set of run flags — there is no layered profile system. The --judges flag (none/local/api) selects judge routing; see the Judges section.

Metrics carry a cost group and an evaluation scope. --judges none restricts to Group L (local, zero API cost); --scopes rag_core,safety,... filters by evaluation scope. The active cost groups and scopes default from the evaluation: block in config/default.yaml.

Gold-tree case-level filters (--language, --domain, --split) slice the case list by case.metadata before evaluation, and --max-cases caps the result with a stratified sample.

Metric Adapters

Each metric is a MetricAdapter subclass in src/lamb_eval/metrics/<domain>/. They self-register via register() at module level and are auto-discovered at runtime via pkgutil.iter_modules.

Runner Pipeline

core/runner.py: discover metrics -> filter -> configure judge models -> populate LAMB responses -> evaluate (threads + subprocess isolation for metrics that declare thread_safe = False) -> aggregate -> generate reports -> save snapshot -> record history. The per-case scoring (core/scoring.py), subprocess evaluator (core/subprocess_eval.py), and report persistence (core/report_pipeline.py) are separate modules the runner composes.

Key Directories

Path	Purpose
`src/lamb_eval/core/`	Orchestration: config, run options + entrypoint (`run_options.py`, `api.py`), runner, scoring, models, filtering, snapshots, endpoints
`src/lamb_eval/metrics/`	20 metric adapters across 5 domains
`src/lamb_eval/metrics/{json_utils,judge_client,text_utils}.py`	Shared metric utilities: JSON parsing, LLM judge calls (with per-call timeout), text overlap (`_custom_utils.py` re-exports these for back-compat)
`src/lamb_eval/clients/`	Async HTTP client for LAMB API
`src/lamb_eval/reporting/`	JSON, HTML, and text report generation
`config/`	YAML config (`default.yaml`)
`datasets/`	Schema + example evaluation datasets
`tests/unit/`	494 tests, no infrastructure needed
`tests/integration/`	19 tests, requires running LAMB at localhost:9099

Important Conventions

Lazy imports: All framework dependencies (deepeval, ragas, opik, etc.) are imported inside _ensure_loaded(), never at module level. The package imports without heavy deps installed.
Graceful degradation: Metrics return MetricScore(score=0.0, reason="...") when data is missing. They never crash.
Heuristic fallbacks: All Group A metrics have a heuristic path that activates when no judge model is configured, enabling local smoke testing without API keys.
OpenRouter routing: All judge LLM calls route through OpenRouter. litellm_model_id() handles LiteLLM (Opik) format conversion. GPTModel factory handles DeepEval. LangChain ChatOpenAI handles RAGAS. call_judge_llm handles custom metrics.

Docker

The image is a multi-stage build (builder → ci → runtime) using uv for fast installs and CPU-only PyTorch wheels. Dependencies are pinned by uv.lock (consumed by the Docker builds via uv sync --locked), so image builds are reproducible. The final runtime image runs as a non-root user (uid/gid 1000) and ships with config/ and datasets/ baked in, so it is usable standalone via docker run without volume mounts.

Requires BuildKit (default in Docker ≥ 23). On older Docker, prefix builds with DOCKER_BUILDKIT=1.

# Build the image
docker compose build eval-runner

# Run evaluation
docker compose up

# Standalone run without compose (no volume mounts; uses baked-in config/datasets)
docker run --rm lamb-eval:local --help

Volume permissions: Compose mounts ./reports, ./config, and ./datasets from the host. The container writes as uid 1000. If your host user is a different uid, override with user: "${UID}:${GID}" in compose or --user $(id -u):$(id -g) on docker run.

Continuous Integration

There is no GitHub Actions / hosted CI for this repo (and therefore no CI badge — by design). Instead, the CI-safe quality gate runs inside a dedicated, dependency-complete Docker image so it is reproducible anywhere Docker is available. The gate mirrors the pre-commit-checks static + unit stages (the ones that need no live LAMB backend and no Ollama/LM Studio judge): ruff check, ruff format --check, bandit, the dependency CVE audit (scripts/security_audit.sh), and pytest tests/unit/. Integration and e2e tiers are deliberately excluded — they need infrastructure unavailable in a self-contained container and stay local-only.

# Run the full gate in a container (builds the lock-pinned `ci` image, streams logs)
docker compose up --build ci

# Same gate, with a propagated exit code (e.g. for scripting)
docker compose up --build --abort-on-container-exit --exit-code-from ci ci

The ci compose service bind-mounts the working tree read-only, so after the first build you can re-run docker compose up ci against live edits without rebuilding. The gate definition lives in one place — scripts/ci_gate.sh — which is also runnable directly in a local venv:

pip install -e "." --group test    # PEP 735, pip >= 25
bash scripts/ci_gate.sh

Dependency lock. uv.lock pins the container's dependency set. It is the only consumer of the lock and is scoped to linux; local/macOS development is unchanged and still installs via pip install -e "." --group test (not uv sync). Regenerate the lock whenever pyproject.toml dependencies change — the build uses --locked and fails if the lock is stale:

docker run --rm -v "$PWD":/w -w /w ghcr.io/astral-sh/uv:0.5-python3.11-bookworm-slim uv lock

Tests

Per-tier overview, runtime, and prerequisites are in tests/README.md. All tiers are gated by the pre-commit-checks skill, which must pass before any commit/push/PR. See §Contributing below for the full two-step gate.

# Unit tests (no external services, ~7s)
pytest tests/unit/ -v -m unit

# Integration tests (requires LAMB at localhost:9099, ~2min)
pytest tests/integration/ -v -m integration

# E2E smoke (requires local Ollama, ~15min)
pytest tests/e2e/ -v -m e2e

# Lint + complexity + google docstrings
ruff check src/ tests/

# Format
ruff format src/ tests/

Contributing

Every change follows a two-step gate — both steps must complete in order:

pre-commit-checks — static analysis (ruff lint + format + complexity + Google docstrings), security review, and the full unit/integration/e2e test suite. Must be green before any commit or push.
git-workflow — the full lifecycle: issue → branch → typed commits → PR → updates → cleanup. Invoke only after pre-commit-checks passes.

Full conventions (commit type taxonomy, branch naming, label palette, PR format) are in CLAUDE.md and the two skill files linked above.

Known Limitations

Local judge model quality: The default Ollama local judge (qwen2.5:0.5b) is too small for complex chain-of-thought prompts. The Opik bias metric produces unreliable scores with local judges. Use --judges api for meaningful results from these metrics.
Heuristic fallbacks are English-only: Keyword-based heuristics (abstention phrases, tone indicators) use English word lists. The LLM judge path handles multilingual content naturally.
Metadata-dependent metrics need dataset engineering: grade_appropriateness, misconception_remediation, scope_adherence, and abstention_accuracy all require specific metadata fields not found in standard RAG datasets.

Name		Name	Last commit message	Last commit date
Latest commit History 188 Commits
.claude/skills		.claude/skills
config		config
datasets		datasets
docs		docs
implementation_plans		implementation_plans
scripts		scripts
snapshots		snapshots
src/lamb_eval		src/lamb_eval
tests		tests
webui		webui
.dockerignore		.dockerignore
.env.sample		.env.sample
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
docker-compose.yaml		docker-compose.yaml
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

lamb-eval

Table of Contents

Quick Start

Web UI

Judges

Environment Setup

CLI Commands

lamb-eval run

lamb-eval snapshot

Selecting a LAMB assistant

lamb-eval report

Metrics (20 total)

RAG Core (5 metrics)

Safety (4 metrics)

Educational Quality (9 metrics)

Multi-turn (1 metric)

NLP Baselines (1 metric)

Datasets

datasets/gold/ — gold-standard evaluation corpus

datasets/fixtures/ — test fixtures

Metadata fields for specialized metrics

Output Structure

Reports (reports/)

Snapshots (snapshots/)

Configuration

Architecture

Run Configuration and Metric Filtering

Metric Adapters

Runner Pipeline

Key Directories

Important Conventions

Docker

Continuous Integration

Tests

Contributing

Known Limitations

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

`lamb-eval run`

`lamb-eval snapshot`

`lamb-eval report`

`datasets/gold/` — gold-standard evaluation corpus

`datasets/fixtures/` — test fixtures

Reports (`reports/`)

Snapshots (`snapshots/`)

Packages