feat: model search (3 CatBoost configs × 4 high-pace leagues) - #12
Merged
Conversation
Adds a hyperparameter search to answer "architecture vs data" for KBL/WNBA/ LNB_DR/BSN, where V6-baseline showed no edge. - model.py: typed ModelHyperparams dataclass + build_classifier() + default_hyperparams() (sourced from settings.model, l2 unpinned). train_classifier gains an optional `hyperparams` override — V6 behaviour unchanged when omitted. - scripts/model_search_4_leagues.py: trains Baseline (V6) / Light (depth=3, iterations=500) / Conservative (l2_leaf_reg=10, lr=0.01) on the same 80/20 chrono split, scores all at a FIXED 0.54 threshold (apple-to-apple), prints a table (AUC / ROI / n / CI95) + analysis. - tests/test_model_hyperparams.py: 6 tests (baseline mirrors settings, overrides land, l2 unset by default, frozen). Result: only 1 positive cell (BSN/Light +2.70%) but with sub-coin-flip AUC and a CI straddling zero — noise, not edge. However Light (depth=3) reduces loss and lifts AUC on 3/4 leagues → V6 slightly overfits these small noisy leagues. No config clears zero on any league: the binding constraint is DATA (sample size + summer variance), not architecture. Tuning cuts the bleeding, doesn't create edge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…agues
Per-league grid search (BASE / +FATIGUE / +TEAM_ADV / +both) on the 4 new
high-pace leagues (game period, real closing lines), after mapping them to
Sofascore and running box-score enrichment.
- BSN -> BASE+FATIGUE: FATIGUE lifts AUC 0.496->0.557 (>0.52), ROI +2.3%.
TEAM_ADV HURT it badly (AUC->0.43, ROI -24..-32%) despite 91% box
score, so it is excluded.
- KBL -> BASE: FATIGUE helped (0.407->0.435) but stayed <0.52; TEAM_ADV
unavailable (Sofascore box score empty for KBL, 0% enriched).
- WNBA -> BASE: sub-coin-flip everywhere; TEAM_ADV properly tested (89% cov)
and FAILED (0.393->0.419, ROI still -15.6%).
- LNB_DR-> BASE: train<100 (guard-skipped), too small to evaluate.
Thresholds kept at the pre-registered 0.54 (no per-cell cherry-picking).
BSN is the only screening candidate (AUC>0.52 + positive ROI) but marginal
(+2.3% on 39 one-season bets) — needs a multi-season walk-forward before it
can be trusted (cf. the B.League within-season mirage).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…NBA/LNB_DR) Codex gap: the 4 new-league feature assignments had no tests. Add BSN (BASE+FATIGUE — fatigue visible, team_adv hidden) and extend the BASE-only parametrisation with KBL/WNBA/LNB_DR.
…vert speculative new-league combos Two coupled changes keeping the PR to reusable tooling + knowledge only: 1. Extract the per-league *data* (FeatureGroup enum, FEATURES_BY_LEAGUE, thresholds, group→column maps) into a new data module `src/evaluation/feature_config.py`. `feature_selector.py` becomes a thin logic façade that re-exports every public name, so all existing imports (model.py, grid_search/cell.py, periods.py, tests) keep working unchanged. feature_selector.py 247→135 lines; feature_config.py 148 lines — both well under the 250-line limit, leaving headroom to add leagues later. 2. Revert the BSN/KBL/WNBA/LNB_DR production assignments (b094680) and their tests (964ad4a). The 4 new leagues fall back to the safe BASE default until a multi-season walk-forward confirms an edge — the within-season BSN +2.3% ROI (n=39) is not enough after the B.League cross-season mirage lesson. Model-search infra (model.py hyperparams refactor, model_search_4_leagues.py) and the negative-research knowledge stay. Gate: ruff clean, mypy 114 files clean, pytest green, go vet+test green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Итог: переиспользуемая инфраструктура модель-сёрча + знание (без спекулятивного прод-конфига)
Спринт по 4 новым высокотемповым лигам (KBL, WNBA, LNB_DR, BSN) завершён. Вывод по существу: узкое место — данные (объём выборки, межсезонная дисперсия, бедный box-score), а не архитектура/фичи. Гипотеза «сложные фичи систематичнее» — опровергнута.
Что остаётся в PR (полезное, проверенное)
model.py— рефактор гиперпараметров:ModelHyperparams(frozen dataclass),default_hyperparams(),build_classifier();train_classifier(..., hyperparams=None). Переиспользуемая инфра для любого модель-сёрча.scripts/model_search_4_leagues.py— 3 CatBoost-конфига × 4 лиги, без магических чисел, типизирован.tests/test_model_hyperparams.py— 6 тестов.feature_selector→feature_config— данные (enumFeatureGroup,FEATURES_BY_LEAGUE, пороги, group→column-мапы) вынесены в новый data-модульfeature_config.py(SOLID: данные vs логика).feature_selector.py— тонкий фасад с ре-экспортом, все существующие импорты работают без изменений. 247→135 строк,feature_config.py148 строк — обе с запасом под лимит 250 для будущих лиг.feature_config.py): TEAM_ADV не помогает нигде среди новых лиг (вредит BSN AUC 0.56→0.43 при 91% покрытии; WNBA 0.393→0.419 при 89%); FATIGUE помогает только BSN (AUC 0.496→0.557, +2.3%, n=39); KBL — box-score на Sofascore пуст; LNB_DR — train<100.Что откатано (сознательно)
Прод-присвоения
FEATURES_BY_LEAGUEдля BSN/KBL/WNBA/LNB_DR и их тесты (b094680,964ad4a) откатаны. Новые лиги падают на безопасный дефолт BASE до валидации через multi-season walk-forward. Причина — дисциплина: within-season BSN +2.3% (n=39) недостаточно после урока B.League (within-season +19% → cross-season −5% мираж).Гейт (локально + CI)
ruff clean · mypy 114 файлов clean · pytest green · go vet+test green · все файлы ≤250 строк. CI: Python tests ✅, Go tests ✅.