Skip to content

EVALUATION.md

Latest commit

 

History

History
65 lines (47 loc) · 3.56 KB

EVALUATION.md

File metadata and controls

65 lines (47 loc) · 3.56 KB
Raw

FoodLM-Bench v0.1 mini

FoodLM-Bench is a committed, reproducible evaluation set for FoodLM's engine and future local model comparisons. The resolution, rule-derived substitution, and grounded generation-integrity suites are versioned under benchmarks/data/. Public values below come only from benchmarks/results/*.json through scripts/render_results.py.

Benchmark design

The resolution suite covers product demos, real label noise, E-numbers, aliases, process modifiers, typos, and adversarial input. Expected ingredients must resolve to the packaged ontology.

The substitution suite evaluates whether the engine's top candidates intersect the exact source-order rule list after hard goal filters. This is a rule-derived test, not a human preference study.

The generation suite validates plans against a committed JSON schema and scans every generated string for ontology ingredients outside the supplied engine context. In engine mode it measures integrity of a deterministic projection, not language-model quality. Base and FoodLM artifacts remain pending until both are evaluated locally on the declared Qwen3-8B weights and published adapter.

Metrics

  • Resolution precision, recall, and F1: micro-averaged set overlap between expected and resolved canonical ingredients.
  • Substitution acceptable@3: cases where the top three include a rule-derived acceptable candidate.
  • Substitution zero-unacceptable: cases where the top three avoid specified hard-filter failures.
  • Generation JSON-schema validity: plans satisfying the committed target schema.
  • Generation hallucination rate: out-of-context ontology mentions divided by all detected ontology mentions.
  • Generation constraint satisfaction: plans preserving the requested goal with no out-of-context ingredient mention.

Results

Artifact date: 2026-07-16.

Suite Metric Engine Base FoodLM
Resolution Precision 1.0000 pending pending
Resolution Recall 0.9905 pending pending
Resolution F1 0.9952 pending pending
Substitution Acceptable@3 rate 1.0000 pending pending
Substitution Zero-unacceptable rate 1.0000 pending pending
Generation integrity JSON-schema validity 1.0000 pending pending
Generation integrity Hallucination rate 0.0521 pending pending
Generation integrity Constraint satisfaction 0.7500 pending pending

Validated instruction records: 1,700. Engine data: 175 canonical ingredients, 209 aliases, and 49 rule targets.

Selected source artifacts:

  • benchmarks/results/generation_engine_2026-07-16.json
  • benchmarks/results/resolution_engine_2026-07-16.json
  • benchmarks/results/substitution_engine_2026-07-16.json

Engine values describe deterministic system behavior on this mini benchmark. They do not establish food-science correctness, product performance, safety, regulatory suitability, or model quality.

Reproduce

python benchmarks/run_eval.py --suite all --model engine
python scripts/render_results.py
python scripts/render_results.py --check
pytest -q

Base and FoodLM publication is local-only and requires CUDA. FoodLM additionally requires an explicit adapter:

python benchmarks/run_eval.py --suite all --model base
FOODLM_ADAPTER=smc3030/foodlm-8b python benchmarks/run_eval.py --suite all --model foodlm

The benchmark runner does not allow endpoint or Anthropic responses to be published as base or foodlm artifacts. Inspect each artifact's backend_type and model_identity before rendering it.