feat(eval): add CompileAndRun scorer with sandbox + YAGNI-for-tests#215

Merged

Delqhi merged 1 commit into

mainfrom

Jun 16, 2026

Delqhi commented Jun 16, 2026

Collaborator

Closes #181

vercel Bot commented Jun 16, 2026 •

edited

Loading

Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
sin-code	Ready	Preview, Comment, Open in v0	Jun 16, 2026 7:38pm


          feat(eval): add CompileAndRun scorer with sandbox + YAGNI-for-tests

2ca4c5c

Closes #181

- Add CompileAndRun scorer to evalharness/scorer.go
- Add runner_extras.go with code extraction, per-language compile, and
  sandboxed self-check execution
- Extend EvalCase with Scorer config and ScorerFromConfig helper
- Extend dataset.TestCase with ScorerConfig and validation
- Wire scorer into dataset.Runner and eval_cmd.go CLI
- Add --scorer, --language, --self-check, --skip-test, --scorer-binary flags
- Update docs: scorer.doc.md, runner_extras.doc.md, dataset.doc.md,
  AGENTS.md §12, CHANGELOG.md
- Add tests: Python happy path, syntax error, self-check failure, skip-test,
  no code block, Go compile + self-check

Delqhi force-pushed the feat/issue-181 branch from ad1b8ad to 2ca4c5c Compare

June 16, 2026 19:37

Delqhi merged commit 48d4f9d into main

vercel Bot commented Jun 16, 2026

Contributor

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-young-hobby-team-24h").

Learn More: https://vercel.com/spriterfrau-5830s-projects?upgradeToPro=build-rate-limit

vercel Bot commented Jun 16, 2026

Contributor

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/cristianoronalodbau-9053s-projects?upgradeToPro=build-rate-limit

vercel Bot commented Jun 16, 2026

Contributor

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-young-hobby-team-24h").

Learn More: https://vercel.com/manheiler-8784s-projects?upgradeToPro=build-rate-limit

vercel Bot deployed to Preview – sin-code

June 16, 2026 19:38

View deployment

vercel Bot commented Jun 16, 2026

Contributor

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/mariobatrhs-9606s-projects?upgradeToPro=build-rate-limit

vercel Bot commented Jun 16, 2026

Contributor

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/einskarst-2238s-projects?upgradeToPro=build-rate-limit

vercel Bot commented Jun 16, 2026

Contributor

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/aragonheld-7756s-projects?upgradeToPro=build-rate-limit

vercel Bot commented Jun 16, 2026

Contributor

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/sagamundis-1139s-projects?upgradeToPro=build-rate-limit

vercel Bot commented Jun 16, 2026

Contributor

Deployment failed with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/ronaldokrause-9559s-projects?upgradeToPro=build-rate-limit

Delqhi pushed a commit that referenced this pull request


          merge: resolve main conflicts in PR #215 (issue #181)

360670d

github-actions Bot commented Jun 16, 2026

🏆 CEO Audit — A+ (100.0/100)

Metric	Value
Grade	A+
Score	100.0/100
Critical findings	0
High findings	0
Profile	`QUICK`
Min grade gate	B

📥 Download full report (Markdown)
📊 Download SARIF (for Code Scanning)

Run ~/.config/opencode/skills/ceo-audit/scripts/audit.sh . --profile=QUICK locally to reproduce.

github-actions Bot commented Jun 16, 2026

🏆 CEO Audit — A+ (100.0/100)

Metric	Value
Grade	A+
Score	100.0/100
Critical findings	0
High findings	0
Medium findings	0
Profile	`QUICK`
Min grade gate	B

📥 Download full report (Markdown)

Run ID: 27643161928 · Commit: ${github.sha}

Run ~/.config/opencode/skills/ceo-audit/scripts/audit.sh . --profile=QUICK locally to reproduce.

Delqhi mentioned this pull request

fix(eval): add missing closing brace for newEvalDiffCmd #224

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet