feat: LesNet 4.5.0 — data-sourcing module + S/M/L/XL model family (distillation, int8 quantisation)

.github/workflows/release.yml

@@ -0,0 +1,27 @@
+name: Release
+
+# On a version tag, ensure a GitHub Release exists with generated notes. The trained model
+# artifacts (LesNet.M4.5*.keras / .artifacts.json / .tflite) are uploaded to this release by
+# the model-build automation (gh release upload) — they are too large to live in the repo/CI.
+on:
+  push:
+    tags:
+      - 'v*'
+
+permissions:
+  contents: write
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Create or update the release
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          if gh release view "${GITHUB_REF_NAME}" >/dev/null 2>&1; then
+            gh release edit "${GITHUB_REF_NAME}" --generate-notes
+          else
+            gh release create "${GITHUB_REF_NAME}" --generate-notes --title "LesNet ${GITHUB_REF_NAME}"
+          fi

CLAUDE.md

@@ -35,12 +35,17 @@ lesnet/                        # the package
   __init__.py                  # main(): Pyramid WSGI app factory (pyramid imported lazily)
   routes.py                    # routes: home, supported-diagnoses, predict, labels
   config/model.py              # ModelConfig — released-model registry (MODEL_URLS) for the web app + downloader
-  ml/                          # the triage system (framework-light, pure-logic where possible)
+  data/                        # data subsystem (stage 1: source -> sort -> balance -> manifest; no TF)
+    config.py                  # SourcingConfig
+    sources/                   # isic, pad_ufes, fitzpatrick17k, ddi (+ registry) -> LesionRecords
+    taxonomy.py · canonical.py # diagnosis -> triage bucket; curated diagnosis-name canonicalisation
+    quality.py · sort.py · balance.py    # quality + pHash dedup; bucket/diagnosis folders; fair 1:1 balance
+    records.py · splits.py · manifest.py # LesionRecord + manifest IO; grouped leakage-free splits
+  ml/                          # the triage MODEL system (training-coupled, imports TF)
     config.py                  # PipelineConfig — all training/inference hyperparameters
-    datasets.py · splits.py    # manifest build/load, patient-grouped leakage-free splits, LesionRecord
     data_loader.py             # tf.data pipeline from a manifest
     preprocessing.py           # DullRazor hair removal + Shades-of-Gray colour constancy + resize/scale
-    features.py · taxonomy.py  # metadata encoding; diagnosis -> triage/fine label maps
+    features.py                # metadata encoding (age/sex/site/Fitzpatrick)
     model.py                   # build_triage_model (EfficientNetV2-S) + triage/fine/feature sub-models
     losses.py · metrics.py     # focal loss + class weights; clinical metrics (sens/spec/ROC/ECE/AURC)
     training.py                # orchestrator: fit (optionally metric-gated) -> calibrate -> conformal -> OOD -> save
@@ -53,8 +58,7 @@ lesnet/                        # the package
   views/{api,default,notfound}.py   # /predict + /labels (lazy TriagePredictor); home + supported-diagnoses; 404
   templates/*.jinja2 · static/      # web UI
 commands/                      # CLI scripts (run from repo root)
-  download_isic_full.py · fetch_isic_sample.py   # dataset download
-  run_build_dataset.py                           # build the grouped manifest
+  build_dataset.py                               # stage 1: source -> canonicalise -> quality -> balance -> sort -> manifest
   run_train_triage.py                            # train (--smoke for a CPU synthetic end-to-end run)
   run_evaluate.py                                # evaluation report + model card
   run_triage_inference.py · run_validation_check.py   # inference (single image / validation set)
@@ -69,8 +73,7 @@ Dockerfile / docker-compose.yml                  # container (production.ini); c
 
 ## Data → training → inference flow
 
-1. **Download:** `commands/download_isic_full.py` (resumable, multithreaded) or `fetch_isic_sample.py`.
-2. **Manifest:** `commands/run_build_dataset.py` builds a patient/lesion-grouped, leakage-free manifest CSV.
+1. **Source + curate (stage 1):** `commands/build_dataset.py` (→ `lesnet.data.pipeline`) downloads (ISIC malignant-maximal; PAD-UFES auto; Fitzpatrick17k/DDI on disk), canonicalises diagnosis names, quality-gates + pHash-dedupes, sorts into `benign/not_sure/malignant/<diagnosis>/`, balances ~1:1 benign:malignant (group-safe, fairness-aware), and writes a leakage-free `manifest.csv` + `report.json`.
 3. **Train:** `commands/run_train_triage.py --manifest …` → `lesnet.ml.training.train` (EfficientNetV2-S, focal loss, optional metric-gated rounds) → calibrate (temperature) → choose sensitivity-first operating point → split-conformal → fit Mahalanobis OOD → save `triage_model.keras` + `artifacts.json`.
 4. **Evaluate:** `commands/run_evaluate.py` writes `evaluation_report.json` + `model_card.md` (incl. fairness gate).
 5. **Infer:** web `POST /predict` (lazy `TriagePredictor`) or `commands/run_triage_inference.py`. Pipeline = quality gate → OOD gate → calibrated triage → abstention/conformal set + auxiliary fine-grained prediction. `GET /labels` returns the class list.
@@ -94,10 +97,14 @@ python -m pip install -e ".[testing]"
 # run the web app, http://localhost:6543
 pserve development.ini --reload
 
-# tests + lint
+# tests + lint (lesnet/data is gated at 100% coverage)
 python -m pytest
+python -m pytest --cov=lesnet.data --cov-fail-under=100
 ruff check
 
+# stage 1: source + build a balanced, sorted, leakage-safe dataset
+python commands/build_dataset.py --sources isic pad_ufes_20 fitzpatrick17k ddi --dest data/dataset
+
 # CPU smoke (synthetic end-to-end: train -> calibrate -> save -> load -> infer)
 python commands/run_train_triage.py --smoke --artifacts /tmp/lesnet_art
 
@@ -110,6 +117,7 @@ python commands/download_model.py -m M-4s
 
 ## Git / contributing
 
+- **Always branch off the latest `origin/main`** (`git fetch origin main && git checkout -B <branch> origin/main`) — never off a stale release/feature branch. `main` is the source of truth and may carry squash-merged PRs that raw branches lack.
 - CI (`.github/workflows/test.yaml`) runs `ruff check` + `pytest` on PRs to `main`; CodeQL also runs; Dependabot manages bumps.
 - Commit style: short subjects, often gitmoji + milestone/issue tag. Not strict Conventional Commits — match history.
 - `docs/model-redesign.md` is the authoritative design for the triage system. `docs/code-audit.md` is a historical (2026-06-09) audit of the now-removed legacy pipeline.

commands/build_dataset.py

@@ -0,0 +1,80 @@
+"""Stage 1 — source, canonicalise, quality-gate, balance, and sort a training dataset.
+
+  # malignant-maximal balanced build from ISIC (+ any sources present on disk):
+  python commands/build_dataset.py --sources isic pad_ufes_20 fitzpatrick17k ddi --dest data/dataset
+
+ISIC is sourced malignant-maximally: page the whole archive's metadata (cheap), then download
+images only for all malignant + suspicious + a benign sample matched to the target ratio.
+Other sources auto-download where the licence allows (DDI must be placed on disk manually).
+"""
+import argparse
+import json
+import os
+
+from lesnet.data import balance, sort
+from lesnet.data.config import SourcingConfig
+from lesnet.data.pipeline import run
+from lesnet.data.sources import isic
+
+
+def _isic_malignant_maximal_download(root, config):
+    """Populate ``root`` with ISIC metadata + a malignant-maximal, ratio-matched image set."""
+    resolution = 'full' if config.full_resolution else 'thumbnail_256'
+    isic.collect_metadata(root, resolution=resolution, limit=config.sample_limit)
+    url_by_id = isic.url_map(root)
+
+    annotated, _ = sort.annotate(isic.parse(root, config.sample_limit))
+    by_bucket = {'benign': [], 'not_sure': [], 'malignant': []}
+    for record in annotated:
+        by_bucket[record.triage_bucket].append(record)
+
+    benign_target = round(config.balance_ratio * len(by_bucket['malignant']))
+    benign_keep = balance._select_groups(by_bucket['benign'], benign_target, config.seed)
+    selected = by_bucket['malignant'] + by_bucket['not_sure'] + benign_keep
+    selected_ids = {os.path.splitext(os.path.basename(r.image_path))[0] for r in selected}
+
+    download_set = {isic_id: url_by_id[isic_id] for isic_id in selected_ids if isic_id in url_by_id}
+    print(f"ISIC: {len(annotated)} labelled · malignant={len(by_bucket['malignant'])} "
+          f"not_sure={len(by_bucket['not_sure'])} benign(kept)={len(benign_keep)} "
+          f"-> downloading {len(download_set)} images")
+    isic.download_images(root, download_set)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Build the LesNet triage dataset (stage 1).")
+    parser.add_argument('--sources', nargs='+', default=['isic'],
+                        choices=['isic', 'pad_ufes_20', 'fitzpatrick17k', 'ddi'])
+    parser.add_argument('--dest', default='data/dataset')
+    parser.add_argument('--raw-dir', default='data/raw')
+    parser.add_argument('--isic-root')
+    parser.add_argument('--pad-ufes-root')
+    parser.add_argument('--fitzpatrick17k-root')
+    parser.add_argument('--ddi-root')
+    parser.add_argument('--sample-limit', type=int, default=None)
+    parser.add_argument('--ratio', type=float, default=1.0, help="benign:malignant target.")
+    parser.add_argument('--no-dedupe', action='store_true')
+    parser.add_argument('--thumbnails', action='store_true', help="ISIC thumbnails (faster, lower-res).")
+    parser.add_argument('--seed', type=int, default=42)
+    args = parser.parse_args()
+
+    root_arg = {'isic': args.isic_root, 'pad_ufes_20': args.pad_ufes_root,
+                'fitzpatrick17k': args.fitzpatrick17k_root, 'ddi': args.ddi_root}
+    roots = {name: root_arg[name] for name in args.sources if root_arg.get(name)}
+    config = SourcingConfig(
+        sources=tuple(args.sources), roots=roots, raw_dir=args.raw_dir, dest=args.dest,
+        sample_limit=args.sample_limit, full_resolution=not args.thumbnails,
+        dedupe=not args.no_dedupe, balance_ratio=args.ratio, seed=args.seed)
+
+    if 'isic' in config.sources:
+        isic_root = config.roots.get('isic') or os.path.join(config.raw_dir, 'isic')
+        if not (os.path.isdir(isic_root) and os.listdir(isic_root)):
+            _isic_malignant_maximal_download(isic_root, config)
+
+    _records, report = run(config)
+    print(json.dumps({key: report[key] for key in
+                      ('final_total', 'by_bucket', 'benign_to_malignant_ratio', 'by_split')}, indent=2))
+    print(f"Dataset + manifest written to {config.dest}")
+
+
+if __name__ == '__main__':
+    main()

commands/build_models.py

@@ -0,0 +1,77 @@
+"""Train the full 4.5.0 model family end to end (paper §5, §7).
+
+  python commands/build_models.py --manifest data/dataset/manifest.csv --out models/4.5.0 --until-target
+
+1. Train the teacher M4.5XL (EfficientNetV2-L @ 512).
+2. Distil students M4.5L / M4.5m / M4.5s from the teacher (cross-resolution: the teacher is
+   rebuilt at each student's input size and its convolutional weights transferred).
+3. int8-quantise M4.5s to TFLite for the <500 MB live demo.
+Each model's artifacts (model.keras + artifacts.json) land in models/4.5.0/<name>/; package
+them for release with commands/package_model.py.
+"""
+import argparse
+import os
+from dataclasses import replace
+
+from lesnet.data.records import load_manifest
+from lesnet.data.taxonomy import build_fine_vocabulary
+from lesnet.ml import quantize, variants
+from lesnet.ml.config import PipelineConfig
+from lesnet.ml.data_loader import filter_valid, make_dataset
+from lesnet.ml.features import METADATA_DIM
+from lesnet.ml.model import build_triage_model
+from lesnet.ml.training import distill, train
+
+
+def _split(records):
+    buckets = {'train': [], 'val': [], 'test': []}
+    for record in records:
+        buckets.get(record.split, buckets['train']).append(record)
+    return buckets['train'], buckets['val'] or buckets['train']
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Train the LesNet 4.5.0 model family.")
+    parser.add_argument('--manifest', required=True)
+    parser.add_argument('--out', default='models/4.5.0')
+    parser.add_argument('--epochs', type=int, default=60)
+    parser.add_argument('--until-target', action='store_true', help="Metric-gated training to targets.")
+    parser.add_argument('--max-epochs', type=int, default=200)
+    parser.add_argument('--no-pretrained', action='store_true')
+    args = parser.parse_args()
+
+    train_records, val_records = _split(load_manifest(args.manifest))
+    fine_vocabulary = build_fine_vocabulary(train_records)
+    n_fine = max(len(fine_vocabulary), 1)
+    base = PipelineConfig(epochs=args.epochs, train_until_target=args.until_target,
+                          max_epochs=args.max_epochs, pretrained=not args.no_pretrained)
+
+    print("=== Teacher: M4.5XL ===")
+    teacher_config = variants.config_for(variants.TEACHER, base,
+                                         artifacts_dir=os.path.join(args.out, variants.TEACHER))
+    teacher_model, _ = train(teacher_config, train_records, val_records)
+    teacher_weights = os.path.join(args.out, 'teacher.weights.h5')
+    teacher_model.save_weights(teacher_weights)
+
+    for name in variants.STUDENTS:
+        print(f"=== Student: {name} (distilling from {variants.TEACHER}) ===")
+        student_config = variants.config_for(name, base, artifacts_dir=os.path.join(args.out, name))
+        # Teacher rebuilt at the student's resolution; conv weights transfer across input sizes.
+        teacher_at_resolution = build_triage_model(
+            replace(teacher_config, image_size=student_config.image_size), n_fine, METADATA_DIM)
+        teacher_at_resolution.load_weights(teacher_weights)
+        student_model, _bundle = distill(student_config, teacher_at_resolution, train_records, val_records)
+
+        if variants.VARIANTS[name].quantize:
+            print(f"=== Quantising {name} to int8 TFLite (live-demo model) ===")
+            representative, _, _ = make_dataset(
+                filter_valid(train_records), student_config, fine_vocabulary, training=False)
+            tflite_path = os.path.join(args.out, name, 'model_int8.tflite')
+            quantize.export_tflite(student_model, tflite_path, dataset=representative, mode='int8')
+            print(f"  {name} int8 TFLite: {quantize.model_size_mb(tflite_path):.1f} MB")
+
+    print(f"4.5.0 family complete in {args.out}: {variants.TEACHER} + {', '.join(variants.STUDENTS)}")
+
+
+if __name__ == '__main__':
+    main()