[Etherlabs-dev]

Summary

This PR completes the portfolio-grade execution and verification pass for eval-harness while preserving the evidence, architecture, and failure-mode work already on the branch.

Engineering changes

• Flattened the nested project into a root src/ios_eval package
• Replaced the environment-dump requirements.txt with a minimal pinned pyproject.toml and separate dev/notebook extras
• Replaced train/test evaluation with deterministic stratified train/validation/test separation
• Fits Amount scaling on training data only
• Applies SMOTE to Random Forest training data only
• Selects thresholds on validation labels and reports the held-out test split once
• Added strict schema, numeric, finite-value, label, probability, threshold, and configuration validation
• Added atomic machine-readable reporting with dataset/environment/config/model provenance
• Removed empty modules, notebooks/config, duplicate leaky scripts, stale result files, and obsolete marketing drafts
• Added Ruff, dependency auditing, Python 3.11 CI, Docker test execution, and generated-data/secret safeguards

Verification

• Clean CPython 3.11.15 install: pass
• pip check: pass
• pip-audit: no known vulnerabilities
• ruff check .: pass
• ruff format --check .: pass
• pytest -q: 32 passed
• GitHub Actions Python jobs: pass
• Docker image build and container tests: pass
• Committed-file secret/token scan: no findings
• Public dataset remains Git-ignored

Independently reproduced benchmark

Public Kaggle mlg-ulb/creditcardfraud dataset, SHA-256 76274b691b16a6c49d3f159c883398e03ccd6d1ee12d9d8ee38f4b4b98551a89:

Model	AP	ROC AUC	Precision	Recall	F1	FP	FN	Validation threshold
Logistic Regression	0.7192	0.9720	0.5526	0.8571	0.6720	68	14	0.99
SMOTE + Random Forest	0.7920	0.9768	0.7500	0.8265	0.7864	27	17	0.71
XGBoost	0.8756	0.9749	0.9011	0.8367	0.8677	9	16	0.54

Two full runs matched exactly apart from timestamp/source revision. Full provenance is committed in results/benchmark.json.

Evidence correction

The earlier workflow selected thresholds on final test labels. Its threshold-dependent precision, recall, F1, FP, and FN figures are superseded. Average precision is threshold-independent, but all current operating metrics now use a validation-selected threshold.

These are public-dataset benchmark results—not production performance, fraud-loss prevention, SLA evidence, or client outcomes.

Remaining technical debt

• Random stratification does not test temporal/out-of-time drift
• Validation F1 is not a real institution-specific cost function
• Repeated or nested validation would better characterize threshold variance
• Probability calibration is not evaluated
• One public dataset cannot establish generalization to live transaction streams

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c4c0b91c1a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reject splits too small for stratification

This check does not account for the configured partition sizes, so inputs it accepts can still fail during stratification. For example, a 100-row dataset with three minority rows and test_size=0.01 passes this condition, but the first train_test_split receives a one-row test partition and raises because it cannot represent both classes. Validate that each requested partition can support both classes before starting the split so configurable benchmark runs fail with the intended actionable error.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Resolve provenance independently of the caller directory

When the installed ios-eval command is invoked from another Git worktree using explicit data/config paths, this subprocess inherits that caller directory and records the unrelated repository's HEAD as git_revision; outside a worktree it records null. Because the report describes this field as the source revision, otherwise successful runs can silently contain false provenance. Resolve it relative to the harness source/build metadata, or omit it unless the current checkout is verified to be this project.

Useful? React with 👍 / 👎.