Skip to content

feat: add empirical difficulty evaluator - #2

Merged
Suiseiseki-2016 merged 2 commits into
Yuchuwa:devfrom
Suiseiseki-2016:feat/empirical-difficulty-evaluator
Aug 30, 2026
Merged

feat: add empirical difficulty evaluator#2
Suiseiseki-2016 merged 2 commits into
Yuchuwa:devfrom
Suiseiseki-2016:feat/empirical-difficulty-evaluator

Conversation

@Suiseiseki-2016

Copy link
Copy Markdown
Collaborator

Summary

  • add an independent empirical difficulty evaluator for Atom and Range subjects
  • evaluate four default NewAPI Qwen models under equal turn and timeout budgets
  • report verifier-confirmed success rate, turns, wall time, tool calls, score, label, and confidence as standalone JSON
  • isolate Range runs with temporary copies and keep canonical Atoms, templates, and verification records unchanged
  • expose cvelab difficulty atom and cvelab difficulty range CLI commands

Validation

  • Ruff passed for the changed evaluator, CLI, agent, and test files
  • 34 focused tests passed across evaluator, agent runner, and researcher tests
  • live smoke Range (dmz_simple, CVE-2012-1823): 4/4 successes, easy, score 10.05
  • live enterprise Range (enterprise_3tier): 0/4 successes, very_hard, score 80.0

Runtime caveat

The live evaluations used disposable noncanonical runtime-image rebindings because the historical repository-pinned images were unavailable. Digest provenance was recorded outside the repository; canonical Atoms, templates, Ranges, and committed code were not modified for those runs.

Suiseiseki-2016 and others added 2 commits August 28, 2026 10:11
Add independent Atom and Range evaluation reports with NewAPI model metrics.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add Chinese comments explaining metrics, isolation, reset behavior, and timeout handling.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@Suiseiseki-2016
Suiseiseki-2016 merged commit a7892ec into Yuchuwa:dev Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant