Portfolio upgrade: evaluation rigor, evidence, and architecture

.dockerignore

@@ -0,0 +1,14 @@
+.git
+.github
+.venv
+__pycache__
+*.py[cod]
+.pytest_cache
+.ruff_cache
+data
+results
+content
+docs
+*.ipynb
+.env*
+kaggle.json

.github/workflows/ci.yml

@@ -0,0 +1,40 @@
+name: CI
+
+on:
+  push:
+    branches: [main, "portfolio-upgrade/**"]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11.15"
+          cache: pip
+      - name: Install
+        run: python -m pip install ".[dev]"
+      - name: Validate dependencies
+        run: python -m pip check
+      - name: Audit dependencies
+        run: pip-audit
+      - name: Lint
+        run: ruff check .
+      - name: Format
+        run: ruff format --check .
+      - name: Test
+        run: pytest -q
+
+  docker-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build test image
+        run: docker build --tag ios-risk-eval:test .
+      - name: Run container tests
+        run: docker run --rm ios-risk-eval:test

.gitignore

@@ -14,16 +14,21 @@ __pycache__/
 .Python
 venv/
 env/
+.venv/
 .env
 *.egg-info/
 dist/
 build/
+.pytest_cache/
+.ruff_cache/
+.coverage
 
 # Data
 *.csv
 data/raw/
 data/processed/
 data/versioned/
+data/**/*.zip
 
 # ML experiment tracking
 wandb/
@@ -36,6 +41,12 @@ mlruns/
 kaggle.json
 *.key
 *_secret*
+*.pem
+*.p12
+
+# Generated benchmark outputs (the reviewed reference artifact is explicitly allowed)
+results/*
+!results/benchmark.json
 
 # OS
 .DS_Store

Dockerfile

@@ -0,0 +1,16 @@
+FROM python:3.11.15-slim-bookworm
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    PIP_DISABLE_PIP_VERSION_CHECK=1
+
+WORKDIR /app
+
+COPY pyproject.toml README.md LICENSE ./
+COPY src ./src
+COPY tests ./tests
+
+RUN python -m pip install --no-cache-dir ".[dev]" \
+    && python -m pip check
+
+CMD ["python", "-m", "pytest", "-q"]

README.md

@@ -1,49 +1,131 @@
-# IntelligenceOS — Eval Harness
+# IntelligenceOS Risk Evaluation Harness
 
-> Building a domain-specific AI platform for Finance Ops & Risk — in public, one project at a time.
-
-This repository contains the open-source evaluation and measurement infrastructure for **IntelligenceOS** — a 12-project build toward a production AI platform.
-
----
-
-## Projects
-
-| Project | Layer | Status | Description |
-|---------|-------|--------|-------------|
-| [IOS Risk Eval Harness](./ios-risk-eval-harness) | Evaluation Layer | ✅ Complete | ML evaluation framework for fraud detection. Benchmarks three model architectures. XGBoost: avg_precision 0.869. |
-| IOS Risk Data Foundry | Data Layer | 🔜 In Progress | Domain data pipeline — ingest, clean, label, version, export. |
-| IOS Domain Intelligence Core | Model Families | ⏳ Upcoming | Fine-tuned Llama/Mistral on finance & risk domain data. |
-| IOS Verification System | Governance Layer | ⏳ Upcoming | Structured, auditable, schema-validated model outputs. |
-| IOS Agent Runtime v1 | Orchestration | 🚀 First Deploy | Closed-loop agent: Perceive → Plan → Act → Verify → Learn. |
-
----
-
-## What Is IntelligenceOS
-
-IntelligenceOS is a domain-specific AI platform being built from the ground up for Finance Ops & Risk.
-
-The architecture: **Perceive → Understand → Plan → Act → Verify → Learn**
-
-Built as a closed-loop system — every output is measurable, auditable, and feeds back into improvement. Not a wrapper around a general model. A platform.
-
----
-
-## Build in Public
-
-Every project is documented publicly as it's built:
-
-- **GitHub** — all code, open source, Apache 2.0
-- **Substack** — deep dives on every project
-- **X / LinkedIn** — progress updates
-
----
+> Leakage-safe, reproducible evaluation of imbalanced financial-risk classifiers.
+
+**Evidence status:** Benchmarked Research Prototype
+**Domain:** Financial Risk · Fraud Detection · Model Evaluation
+
+This repository compares three classical model strategies on the public ULB/Kaggle credit-card fraud
+dataset. It is an engineering and benchmark artifact—not a deployed fraud service, client result, or
+production-performance claim.
+
+## Verified benchmark
+
+The execution pass independently reran every model with Python 3.11.15 and the pinned dependencies in
+[`pyproject.toml`](./pyproject.toml). Unlike the earlier result, the corrected workflow selects each F1
+threshold on validation labels and evaluates the chosen operating point once on a held-out test split.
+
+| Model | Average Precision | ROC AUC | Precision | Recall | F1 | FP | FN | Validation-selected threshold |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| Logistic Regression | 0.7192 | 0.9720 | 0.5526 | 0.8571 | 0.6720 | 68 | 14 | 0.99 |
+| SMOTE + Random Forest | 0.7920 | 0.9768 | 0.7500 | 0.8265 | 0.7864 | 27 | 17 | 0.71 |
+| XGBoost | **0.8756** | 0.9749 | **0.9011** | 0.8367 | **0.8677** | **9** | 16 | 0.54 |
+
+**Evidence label:** public-dataset benchmark. The machine-readable evidence is
+[`results/benchmark.json`](./results/benchmark.json), including the dataset SHA-256, split details,
+configuration, environment, model parameters, thresholds, and full metrics.
+
+The prior `0.8693` XGBoost result used final test labels to select its reporting threshold. Average
+precision itself is threshold-independent, but the old threshold-dependent precision/recall/F1/FP/FN
+figures were not defensible held-out estimates and have been superseded.
+
+## Leakage controls
+
+```text
+creditcard.csv
+    ↓ validate schema, labels, numeric/finite values, and fingerprint
+stratified train / validation / test split (60% / 20% / 20%)
+    ├── train: fit Amount scaler; fit model; apply SMOTE only here for RF
+    ├── validation: select operating threshold by F1
+    └── test: score the selected threshold once
+    ↓
+JSON evidence artifact + console comparison
+```
+
+These boundaries are independently testable. Changing final test labels cannot change the selected
+threshold, scaler metadata records `train_only`, and SMOTE provenance records its train-only scope.
+
+## Quick start
+
+```bash
+python3.11 -m venv .venv
+source .venv/bin/activate
+python -m pip install ".[dev]"
+python -m pip check
+ruff check .
+ruff format --check .
+pytest -q
+```
+
+Download the public dataset and place it at `data/raw/creditcard.csv`:
+
+```bash
+python -m pip install kaggle
+kaggle datasets download -d mlg-ulb/creditcardfraud -p data/raw --unzip
+```
+
+The verified file has 284,807 rows, 31 columns, and SHA-256:
+
+```text
+76274b691b16a6c49d3f159c883398e03ccd6d1ee12d9d8ee38f4b4b98551a89
+```
+
+Run all three models:
+
+```bash
+ios-eval \
+  --data data/raw/creditcard.csv \
+  --config configs/benchmark.json \
+  --output results/benchmark.json
+```
+
+The macOS XGBoost wheel requires the OpenMP runtime (`brew install libomp`). Linux wheels used by CI and
+the Docker image include their supported runtime path.
+
+## Reproducible checks
+
+```bash
+docker build -t ios-risk-eval:test .
+docker run --rm ios-risk-eval:test
+```
+
+GitHub Actions runs Python 3.11 dependency validation, Ruff lint/format checks, 32 automated tests, and
+the same test suite inside Docker. The public dataset is intentionally not required for CI and is never
+committed.
+
+## Repository structure
+
+```text
+src/ios_eval/
+├── data_loader.py   # validation, deterministic splits, train-fitted scaling
+├── metrics.py       # metrics, threshold sweeps, deterministic selection
+├── evaluation.py    # model fitting, train-only SMOTE, provenance orchestration
+├── report.py        # atomic JSON serialization and console comparison
+└── cli.py           # ios-eval command
+configs/benchmark.json
+tests/
+results/benchmark.json
+docs/
+pyproject.toml
+Dockerfile
+```
+
+The historical nested project wrapper, empty modules/notebooks/config, duplicate leaky scripts, oversized
+environment dump, stale single-model result, and obsolete marketing drafts were removed.
+
+## Evidence boundaries and limitations
+
+- One historical public dataset does not establish performance on another processor, geography, product,
+  fraud regime, or live client stream.
+- The selected threshold optimizes validation F1, not a real institution's fraud-loss/review-cost function.
+- This is a single deterministic split, not temporal or out-of-time validation.
+- No calibration, drift monitoring, serving latency, SLA, production deployment, or client outcome is claimed.
+- Average precision measures ranking quality; threshold-dependent metrics describe only this selected operating
+  point and held-out split.
+
+See [architecture](./docs/architecture.md), [evidence policy](./docs/evidence.md),
+[failure modes](./docs/failure-modes.md), and [benchmark reproduction](./docs/benchmark-reproduction.md).
 
 ## License
 
-Copyright 2026 Ugo Chukwu / Etherlabs
-
-Licensed under the Apache License, Version 2.0. See [LICENSE](./LICENSE) for full terms.
-
----
-
-*IntelligenceOS v0.1 — built with purpose.*
+Copyright 2026 Ugo Chukwu / Etherlabs. Licensed under the Apache License, Version 2.0.

configs/benchmark.json

@@ -0,0 +1,45 @@
+{
+  "random_state": 42,
+  "validation_size": 0.2,
+  "test_size": 0.2,
+  "threshold_metric": "f1",
+  "threshold_step": 0.01,
+  "models": {
+    "Logistic Regression": {
+      "type": "logistic_regression",
+      "estimator": {
+        "class_weight": "balanced",
+        "max_iter": 1000,
+        "random_state": 42
+      }
+    },
+    "SMOTE + Random Forest": {
+      "type": "smote_random_forest",
+      "smote": {
+        "random_state": 42,
+        "k_neighbors": 5
+      },
+      "estimator": {
+        "n_estimators": 100,
+        "max_depth": 10,
+        "class_weight": "balanced",
+        "random_state": 42,
+        "n_jobs": 1
+      }
+    },
+    "XGBoost": {
+      "type": "xgboost",
+      "estimator": {
+        "n_estimators": 200,
+        "max_depth": 6,
+        "learning_rate": 0.1,
+        "scale_pos_weight": "train_class_ratio",
+        "random_state": 42,
+        "eval_metric": "aucpr",
+        "tree_method": "hist",
+        "n_jobs": 1,
+        "verbosity": 0
+      }
+    }
+  }
+}

docs/architecture.md

@@ -0,0 +1,62 @@
+# Architecture
+
+## Purpose
+
+The package compares imbalanced binary classifiers while keeping training, operating-point selection, and
+final evaluation in distinct data boundaries.
+
+## Runtime flow
+
+```text
+CLI + benchmark JSON config
+          ↓
+schema and value validation
+          ↓
+deterministic stratified split
+          ├── train (60%)
+          │     ├── fit Amount scaler
+          │     ├── optional SMOTE for Random Forest
+          │     └── fit estimator
+          ├── validation (20%)
+          │     └── select F1 threshold
+          └── test (20%)
+                └── report selected threshold once
+          ↓
+atomic JSON evidence artifact + console table
+```
+
+`Time` is dropped because it is an elapsed sequence field rather than an anchored event timestamp. `Amount`
+is scaled using training statistics only. V1-V28 are already PCA-transformed source features.
+
+## Module boundaries
+
+| Module | Responsibility |
+|---|---|
+| `data_loader.py` | File fingerprinting, schema/value validation, deterministic splitting, train-fitted scaling |
+| `metrics.py` | Model-agnostic confusion/ranking metrics, threshold sweep and deterministic tie-breaking |
+| `evaluation.py` | Estimator construction, train-only SMOTE, validation threshold selection, provenance assembly |
+| `report.py` | Atomic JSON serialization and dependency-light text rendering |
+| `cli.py` | Configurable executable entry point |
+
+The deterministic logic lives in Python rather than notebooks or orchestration tools. No n8n workflow exists in
+this repository; adding one would not improve this local benchmark. A future integration may call `ios-eval` from
+n8n for schedules or notifications without moving metric or leakage controls into nodes.
+
+## Reproducibility contract
+
+The JSON artifact records:
+
+- evidence label and claim boundary;
+- dataset filename, source, row/column/class counts, and SHA-256;
+- random seed and exact train/validation/test composition;
+- preprocessing scope and fitted scaler statistics;
+- model types, resolved parameters, imbalance handling, and SMOTE scope;
+- threshold candidate count, selection split/metric, and validation metrics;
+- default and selected-threshold held-out test metrics;
+- Python, platform, and dependency versions;
+- configuration fingerprint and source Git revision.
+
+## Scope boundary
+
+The repository intentionally has no web API, database, queue, authentication layer, live network dependency,
+model registry, or production service. Those would add speculative surface without improving benchmark validity.