[Feedback] Evaluation and Optimization

[jdehorty]

Summary

Announce and gather feedback on the evaluation and optimization infrastructure that keeps the skills healthy as CesiumJS evolves.

Kind of feedback

Methodology or RFC

Area affected

The public evaluation and optimization branch is now feat/eval-and-optimization, with replacement draft PR #13: #13.

Older draft PR #6 and the old eval-and-optimization branch are superseded.

Context

CesiumJS evolves continuously. New APIs land, old ones deprecate, and behavior changes across versions. Without an automated quality bar, the skills in this repo silently drift. A user prompt that produced correct CesiumJS code last quarter may produce broken or deprecated code today, and no one notices until someone files a bug.

To stay healthy over time, the skills need two things:

1. Regression detection. When a SKILL.md edit, a CesiumJS API change, or a model upgrade affects what these skills produce, we want to catch the regression before users do.
2. Active quality improvement. Not just "did this break", but "could this be better". Manually tuning prose across 14 SKILL.md files based on hunches does not scale. Skills need a way to be proposed, evaluated, and either accepted or rejected on evidence.

Recent work in agentic AI gives us a sustainable path. Meta-Harness (Lee et al., Stanford 2025) and SPO (EMNLP 2025) describe propose, evaluate, decide loops driven by a coding agent with full filesystem access to prior iteration history. The same primitives used to write the skills can be turned around to evaluate and optimize them.

A public-ready version of that infrastructure is published as a standalone branch and draft PR. This is still an in-progress branch, not intended to merge into main immediately. It will continue to evolve as scenarios are added, the methodology is refined, and feedback comes in.

Proposal or current thinking

What is published:

• evaluation/: deterministic evaluation cases, schemas, fixtures, scorecard runner, browser scene-state capture, qualitative visual review, and review UI tooling.
• optimization/: optimization scenarios, candidate/history/result records, scorecard-guided focus selection, proposer loop, public-safe dashboards, and validation scripts.
• wiki/: public-facing architecture, local run instructions, public artifact policy, and evaluation workflow docs.
• .github/workflows/: deterministic evaluation CI plus separate visual/baseline-audit workflows.

How it works:

A coding agent reads the public prior-iteration history and proposes a revised SKILL.md. The runner generates code from that skill, executes it in a browser harness, captures deterministic scene evidence and screenshots, and then records whether the candidate should be accepted, flagged, or routed back through optimization. Programmatic checks gate correctness independently of the visual-quality signal.

Visual scenes use recognizable landmarks and public URL-backed assets where possible so reviewers and judges can reliably tell whether output matches the scenario's expected outcome.

What is intentionally not on the branch:

Raw local traces, ignored .env files, local run directories, and untracked generated artifacts are excluded. They can contain access tokens, local file paths, local server URLs, and prompt commentary. The branch keeps public-safe scenarios, scorecards, curated evidence, summaries, and dashboards.

Rationale or evidence

The earlier draft branch proved the methodology but was not ideal as the public branch because it used the older tuning/ layout and had local-development history that should not be public-facing. PR #13 is rebuilt from current main as a scrubbed publication branch with public-safe commit metadata and validated tracked artifacts.

Validation used for the replacement branch:

• optimization/scripts/check-secrets.sh
• python3 optimization/scripts/check-public-artifacts.py
• gitleaks detect --redact --log-opts="origin/main..HEAD"
• python3 evaluation/scripts/validate-evaluation.py
• python3 -m pytest evaluation/tests optimization/tests
• Manual history/content scans for personal paths, personal email, private-source markers, and token-shaped literals

Open questions

1. Persona coverage. Today's eval prompts do not vary persona voice. Adding beginner, intermediate, expert, and migrator variants per skill would broaden coverage cheaply. Worth doing now, or wait until the suite grows?
2. Migration scenarios for deprecated APIs. Worth a dedicated eval category, or fold them into existing scenarios?
3. Failure-mode taxonomy aggregated across iterations. Which categories would be most useful to track?
4. Synthetic prompt expansion. Currently 5 to 15 evals per skill. A larger suite via LLM-driven expansion is reasonable once the pipeline is validated. What size makes sense?
5. Additional recognizable landmarks worth including for future visual scenarios.

Pre-submission checks

• I searched existing issues for duplicates.
• The replacement public branch excludes local traces, ignored secrets, local paths, and untracked run artifacts.