Barebones DVC implementation for mapping component

[SamuelHLewis]

Context

So far, we have run our evals through python scripts, with one orchestrator script calling other component-specific scripts. This PR implements DVC as the orchestration tool instead of python, which means that:

1. DVC's version control means that only the pipeline stages whose dependencies have changed are rerun.
2. All parameters are held in a params.yaml file rather than being scattered across scripts and env variables, making them easier to track and change.

Note: this PR only implements a DVC pipeline for the theme mapping component. The same pattern can be repeated for the other components, but these are left for later PRs.

Changes proposed in this pull request

1. Create a new dir for component-specific eval pipelines at /themefinder/evals/pipelines/
2. Create a separate dir for the mapping component eval pipeline at /themefinder/evals/pipelines/mapping/
3. Create separate scripts for the data download (/themefinder/evals/pipelines/mapping/download_data.py) and evaluation (/themefinder/evals/pipelines/mapping/evaluate.py) steps of the eval pipeline, based on the current code in /themefinder/evals/eval_mapping.py.
4. Implement DVC to orchestrate this pipeline, with pipeline stages and dependencies specified in /themefinder/evals/pipelines/mapping/dvc.yaml
5. Hold all parameters for all stages in a separate file that DVC refers to (/themefinder/evals/pipelines/mapping/params.yaml)
6. Add DVC as a new dev dependency in /themefinder/pyproject.toml

Guidance to review

1. Run make install from the base of the repo, and check that there are no errors
2. Move to the /themefinder/evals/pipelines/mapping/ dir
3. Run uv run --package themefinder --extra dev dvc dag and check that:
   • DVC runs successfully (which means that the themefinder-specific install of DVC has worked)
   • A directed-acyclic graph is displayed (which means that DVC is reading the dvc.yaml file correctly)
4. Run uv run --package themefinder --extra dev dvc repro and check that:
   • the data is written to /themefinder/evals/data/
   • the evaluate stage kicks off
   • Note: you may get an async error as the evaluate stage runs, which I will resolve in a later PR

Things to check

• I have added any new ENV vars in all deployed environments and updated the .env.test files in the repo

[linear-code]

PRO-679

[github-actions]

Claude finished @SamuelHLewis's task in 57s —— View job

---

PR Review: PRO-679 — DVC Eval Pipeline

• Read changed files
• Provide review feedback

---

Implements DVC orchestration for the theme-mapping eval pipeline, splitting the existing script into download_data.py + evaluate.py with params centralised in params.yaml.

---

🔴 DO — evaluate.py:161 — all_scores[f"{item_key}_output"] = output writes the full labels map into scores.json, mixing raw model output with metrics. DVC metrics files should only contain numeric scores. Fix this →

🟡 TRY — download_data.py:143 — write_dataset_items_to_local uses date.today().isoformat() to create a new timestamped subdirectory on every run. Since DVC tracks ../../data/${dataset}, this means the output always changes, defeating caching. Consider using a fixed subpath (e.g. outputs/mapping/latest/).

🟡 TRY — evaluate.py — _run_with_langfuse uses mapping_f1_evaluator (single F1 per item) while _run_local_fallback uses calculate_mapping_metrics (multiple metrics). These produce structurally different scores.json files depending on context, making results non-comparable. Fix this →

⚪️ NIT — download_data.py:47 — topics_df.rename(columns={"topic_id": "topic_id", "topic": "topic"}) is a no-op rename.

---

Good structure overall — DVC pipeline, param centralisation, and Langfuse/local fallback pattern are all clean.

[georgia-i-ai]

This looks fine as a PoC for how to use DVC. I've incorporated this into the eval framework design, and so happy for this code to be merged so we can use it as a starting point for the main refactor 👍

[georgia-i-ai]

Going forward we won't have these two paths separated out in this way, but as we've still got this formulation in the other eval scripts currently it's fine for now.