[hippoley]

Summary

Propose audited CUA Bench reliability fixes and reproducibility interfaces from a tested fork. This is intentionally a draft, not merge-ready.

Changes

• Fail closed on invalid configuration, worker identity, malformed evaluator rewards, or failed environment cleanup.
• Preserve executed step counts across failures; add deterministic privacy-minimal action event list with recomputable digest.
• Add reproducible dataset file manifest with drift detection.
• Integrate pytest-json-report, jsonschema, Hypothesis and CI artifacts for machine-readable test evidence.

Evidence

• Latest fork CI: https://github.com/hippoley/cua/actions/runs/38014219135 (build and regression succeeded).

Known gaps / requested review

• This branch is 68 commits ahead and approximately 2,037 commits behind upstream main; requires rebase and scope reduction into smaller PRs before merge.
• Action hashes prove reported-event consistency, not physical execution or independent oracle correctness. VM fault-injection/cleanup and cross-provider proofs are still missing.
• Please assess overlap with upstream feat(cua-driver): add metadata-only runtime conformance witness v0 #4883/testkit: reject action-truth claims that contradict E2E oracles #4884 (conformance and action-truth) and fix(cua-bench-rl): reap idle worker envs without deadlocking and restart their clients' episodes #4898/fix(cua-bench-rl): stop workers hanging once their output pipe fills #4899 (RL worker deadlocks) before suggesting merge.
• No claim of third-party adoption or upstream acceptance is made.

This draft opens early collaboration; I will prioritize small independently testable contributions and update the acceptance evidence accordingly.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.

[github-actions]

📦 Publishable packages changed

This comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Only owner-applied release:<service> labels can do that.

• ⏸️ pypi/bench — no owner-authorized release label

Ask the release owner to apply release:<service> labels to auto-release on merge (+ optional bump:minor or bump:major, default is patch).
Or add no-release to skip.